The root of the practice is the Gemara's account of Rav Nachman bar Yitzchak, whose mother covered his head so that the fear of Heaven would be upon him, a teaching that appears in Shabbat 156b and is codified almost verbatim in Kitzur Shulchan Arukh 3:6 and in Mishnah Berurah 2:11-12, which cites it in the name of the Magen Avraham as grounds for training even young children to cover their heads.
Shulchan Arukh, Orach Chayim 2:6 rules that one may not walk four amot with an uncovered head, and Shulchan Arukh HaRav, Orach Chayim 2:7 grounds this in the pasuk that the whole earth is full of God's glory, deriving that one must not carry himself as though oblivious to that glory, extending the stringency as a measure of piety to less than four amot and to sitting as well, and even applying it to covering the heads of young children so that fear of Heaven rest upon them.
Mishneh Torah, Human Dispositions 5:6 frames this as an expression of the great modesty Torah scholars practice upon themselves, that they not uncover their heads or their bodies.
Arukh HaShulchan, Orach Chaim 2:10 gathers further Talmudic sources — Rav Huna's statement that the Divine Presence rests above his head, Rav Huna's oath that he never walked four amot bareheaded, and an account from Masechet Kallah tying an uncovered head to a child's lineage — and explains that the head, containing the mind which is the source of wisdom and fear of Heaven, should not be exposed before God who fills the whole earth, just as one would not stand bareheaded in a holy place.
The obligation extends specifically to moments of sanctity: Shulchan Arukh, Orach Chayim 91:3 records the view that one may not utter God's name with an uncovered head and that there is reason to object to entering a synagogue bareheaded, while Sulam on Zohar, Pinchas 625 likewise ties covering the head to the Divine Presence resting upon it and states that this is all the more so required for a brachah or mention of God's name.
Later authorities work out the practical measure of this obligation: Kitzur Shulchan Arukh Yalkut Yosef, Orach Chaim 2:16 holds that it is a measure of piety to wear a large kippah covering most of the head, though a small kippah suffices by strict law provided it is visible from all sides, and one should be stringent to wear a hat or larger kippah specifically during Shema, prayer, and Birkat HaMazon; Mishnah Berurah 8:4 discusses covering the head together with the tallit as inducing fear of Heaven; and Teshuvos V'hanhagos 1:13 holds that covering even part of the head suffices to avoid the prohibition, since it shows one has not cast off the yoke of Heaven, citing the same instruction from Shabbat 156b to cover one's head.
Modern expositions reaffirm and apply these sources directly: Rav Eliezer Melamed (Peninei Halakhah, Likkutim I 7:1) retells both the account of Rav Huna's refusal to walk bareheaded and the story of Rav Nachman bar Yitzchak's mother as the basis for the practice, and Rav Eliezer Melamed (Peninei Halakhah, Women's Prayer 10:5) traces how the originally pious custom became a binding one as codified in Shulchan Arukh, Orach Chayim 2:6 and extended to the recitation of brachot and God's name in Shulchan Arukh, Orach Chayim 91:3; Rav Eliezer Melamed (Peninei Halakhah, Prayer 5:3) separately notes that although some say unmarried women should also cover their heads during prayer, the practice is not to be particular about this.
Rav Mordechai Eliyahu (Shut HaRav HaRashi 3:10) rules that even in places where people were accustomed to go without a kippah, it is preferable nowadays to wear one, since it adds fear of Heaven and Torah, invoking the same episode of Rav Nachman bar Yitzchak's mother, and in Rav Mordechai Eliyahu (Shut HaRav HaRashi 3:9) he rules that yeshiva students must follow the head covering practice set by the rosh yeshiva.
Additional sources on the sheet touch related aspects of the practice: Shaarei Halacha uMinhag 5:159 cites a Zohar passage in Parashat Naso concerning head covering and its reward; Sefer HaMinhagim, p. 9 identifies a source for the custom of pious people wearing a kippah beneath a hat, referring to Chullin 138a; Kitzur Shulchan Arukh 3:2 situates avoiding gentile customs of dress and hair as a broader context for distinctive Jewish dress; Kitzur Shulchan Arukh Yalkut Yosef, Orach Chaim 98:16 records the custom of yeshiva students to wear a hat and suit during prayer as fitting attire; Peri Tzadik, Shushan Purim 1 discusses the custom of wearing a special hat on Shabbat as alluding to the crown of Torah; and Shabbat 138b discusses the laws of a felt hat on Shabbat.
