Abraham took this grievously for his son.

“For her part, however, Sarah was not guilty of acting unreasonably; she acted even quite logically, so logically that even God agreed with the words she spoke. The patriarch, being affectionate and well disposed toward Ishmael, did not take kindly to her remarks. "This remark about his son … struck Abraham as severe." It was not, you see, that he took much interest in Hagar; rather, he was well disposed toward his son for the reason that he was then still in his youth. Consider in this case, however … the extraordinary considerateness of the loving God: when he saw Sarah apparently having a human problem in being distressed by the parity of esteem of the children and Abraham not taking kindly to the expulsion of Ishmael and the maidservant. (After all, even if in his great restraint he did not take issue with Sarah, it still struck him as severe, that is, harsh, repugnant, and oppressive.) At that point, in fidelity to his characteristic loving kindness and wishing to strengthen the bonds of harmony between them, the Lord said to Abraham, "Don't let the remark about the child and the maidservant strike you as severe. Whatever Sarah says to you, heed her."”

“AND THE THING WAS VERY GRIEVOUS IN ABRAHAM’s SIGHT ON ACCOUNT OF HIS SON. That is, for he had taken to degenerate ways. The plain meaning, however, is that he was grieved because she had told him to send him away. Thus the language of Rashi. The correct interpretation appears to me to be that Scripture is speaking in honor of Abraham, saying that the reason why the matter was very displeasing to him was not due to a craving for his concubine and his desire for her. Therefore, if she had told him to cast out only the maidservant, he would have done her will. But it was on account of his son that he was very much incensed and did not want to listen to her. But the Holy One, blessed be He, told him that he should not resent it at all, neither for the son nor for the maidservant, and that he should listen to Sarah’s bidding for it is through Isaac alone that his name will be carried on, while Ishmael will not be referred to as his offspring. Now because Abraham feared lest an accident happen to Ishmael upon his sending him away, He told him that He will make a nation of him and He will bless him since he is indeed his son.”

“And God said unto Abraham,.... Either by an articulate voice, or by an impulse on his mind, suggesting to him what he should do, being no doubt in great perplexity how to conduct between his wife and his son, but God determines the case for him, and makes him easy: let it not be grievous in thy sight because of the lad, and because of the bondwoman: that is, let not the motion displease thee, which Sarah has made, to turn out the bondwoman and her son; let not thine affection to the one and to the other hinder compliance with it; do not look upon it as an ill thing, or as an hard thing; it is but what is right and proper to be done, and leave the bondwoman and her son to me; I will take care of them, be under no concern for them and their welfare: in all that Sarah hath said unto thee, hearken to her voice; the Targum of Jonathan adds, for she is a prophetess: and indeed in this affair she spoke under a spirit of prophecy, according to the will of God; at least what she said became a divine oracle, and is called the Scripture, Gal 4:30; for the word "all" here must be restrained to what she had said concerning Hagar and Ishmael, and their ejection, and not to be extended to everything she had said, or should say to Abraham, to which he was always to be attentive: whereas on the other hand, it became her, as a wife, to hearken and be obedient to the voice of her husband: but in this particular Abraham is bid to listen to her, and do accordingly, for the following reason: for in Isaac shall thy seed be called; he, and those that descended from him, should be called and reckoned the seed of Abraham more especially; and Abraham's seed in his life should inherit the land of Canaan, given to him and his seed for an inheritance: and this is a good reason why the bondwoman and her son should be cast out, that they or their offspring might not inherit the land with Isaac, or his descendants; and particularly from Abraham in his line, and not in the line of Ishmael, should the Messiah spring, that seed in whom all the nations of the earth should be blessed; and therefore a separation was necessary, that this might abundantly appear.”