A foundational guideline on complimenting is that praise should not come from one's own mouth about oneself, as Mishlei 27:2 states that it should be another, not one's own mouth, that lauds a person — a principle oriented around third-party praise rather than self-praise.
When praising another person, Eruvin 18b teaches that only some of a person's praise should be said in his presence, while the fuller praise should be reserved for when he is not present, learning this distinction from how God addressed Noach directly versus how the Torah describes him in his absence.
The core attitude behind complimenting others is rooted in judging people favorably and seeking out their good qualities even when it is difficult, as Likutei Moharan 282:1 teaches that one must search for the small amount of good even in a very wicked person, and Tomer Devorah 1:14 similarly counsels that a person should let another's good qualities outweigh grievances against him, finding sufficiency in whatever good that person has done.
Pirkei Avot 1:15 frames the broader manner in which one should greet and relate to every person, instructing that one receive every person with a cheerful countenance, which bears on the spirit in which compliments and interactions should be offered.
A key guideline concerns the limits of truthfulness in praise: Ketubot 17a records the machlokes between Beit Shammai and Beit Hillel over how to praise a bride, with Beit Shammai insisting one describe her as she truly is out of concern for the prohibition of staying far from a false matter, while Beit Hillel rule that one always praises her as fair and attractive, reasoning from the example of praising an inferior purchase to enhance it in the buyer's eyes rather than diminish it, establishing that a person's mind should be pleasing to others even when the praise is not literally accurate.
Sotah 14a grounds the broader obligation to act kindly toward others, including through speech, in the imitation of God's own attributes, such as clothing the naked as God clothed Adam and his wife.
Mishneh Torah, Human Dispositions 6:3 codifies that it is a mitzvah to love every fellow Jew as oneself, and that this love obligates a person to speak in praise of others just as one would want to be spoken of, while one who elevates himself through demeaning another forfeits a share in the World to Come.
Shulchan Arukh, Yoreh De'ah 151:14 rules that it is forbidden to praise non-Jews, even merely to say how attractive a given gentile is in appearance, and certainly to praise their deeds or express fondness for something they said — unless the intent of the praise is to thank God for having created such a beautiful being, in which case it is permitted.
On the mystical dimension of praise, Sulam on Zohar, Bamidbar 11 teaches that one who speaks the praise of his fellow, whether regarding his children or his wealth, is obligated to bless him and to give thanks over the blessings, deriving this from Moshe's blessing of the Israelites, which consisted of two blessings followed by thanksgiving.
Tanya, Part I; Likkutei Amarim 32:1-2 situates the love and honor due to every Jew, which underlies appropriate praise of others, in the recognition that all souls share one root and source in God, making all Israelites true brothers regardless of their differing bodies.
Tzitz Eliezer 15:47 rules that praising someone's deeds or expressing appreciation for something they said can be permitted when the intent is really to praise and thank God for the good He has brought about through that person.
B'tzel HaChochma 1:46 rules that praising non-Jews, even simply calling them attractive, is forbidden, unless the praise is meant to acknowledge God for having created such a beautiful creature.
Mishneh Halachos 7:116 draws the same distinction, holding that direct praise of a non-Jew's person, appearance, deeds, or words is forbidden, while praise offered as thanksgiving to God, or mention of someone's expertise for a practical purpose, is permitted.
On the related question of flattery, Mishneh Halachos 10:284 and Mishneh Halachos 11:130 hold that flattery is permitted or even warranted for the sake of peace, to avoid undue pressure, or to draw a student, friend, spouse, or teacher toward Torah study, accepting rebuke, and fulfilling mitzvot, while flattery offered without such a constructive purpose, particularly toward the wicked, is prohibited and may lead a person to fall under their influence.