The foundation of the practice is already laid in Mishnah Rosh Hashanah 4:1, which records that when Rosh Hashanah fell on Shabbos, the shofar was sounded in the Temple but not in the provinces.
The Gemara in Rosh Hashanah 29b supplies Rabbah's rationale: although everyone is obligated in shofar-sounding, not everyone is expert in it, so the Rabbis enacted a decree lest a person pick up the shofar, carry it to an expert to learn the technique, and in doing so carry it four cubits in a public domain — a Torah-level Shabbos violation; the same logic, the Gemara notes, underlies the rabbinic suspensions of lulav and Megillah on Shabbos.
Mishneh Torah, Shofar, Sukkah and Lulav 2:6 codifies this ruling, emphasizing that although shofar-sounding on Shabbos is technically only a rabbinic prohibition (shevut) and a Torah positive commandment would ordinarily override a rabbinic prohibition, the decree nevertheless stands because the risk of carrying the shofar four cubits in a public domain — or transporting it between domains — could bring a person to a sin punishable by stoning.
Mishneh Torah, Shofar, Sukkah and Lulav 2:9 adds that sounding was nonetheless permitted before the rabbinic court, since the court's members are diligent and would prevent the blowers from carrying the shofar through a public domain.
Tosafot on Rosh Hashanah 29b clarifies that shofar-sounding is itself prohibited on Shabbos by rabbinic enactment, and that were it not for Rabbah's specific decree — the fear of carrying — the prohibition alone would not have been sufficient to suppress the sounding entirely.
Tur, Orach Chayim 588 and Beit Yosef, Orach Chayim 588 both transmit this reasoning from the Mishnah and Gemara, with the Beit Yosef adding the Ran's distinction explaining why a similar precautionary decree was not applied to circumcision on Shabbos: everyone is occupied with shofar-sounding and no one would remind his neighbor of a danger, unlike circumcision.
Shulchan Arukh, Orach Chayim 588:5 rules definitively that the shofar is not sounded when Rosh Hashanah falls on Shabbos, and the Rema adds that it is even forbidden to handle the shofar, unless it is needed for its own use or for the sake of its place.
Arukh HaShulchan, Orach Chaim 588:7 elaborates that while the Rambam states the positive commandment would ordinarily override a rabbinic shevut, the specific shevut of the act of sounding itself is necessarily overridden on Yom Tov as well — the true operative concern remains the carrying decree of Rabbah.
A further dimension is noted by Rav Mordechai Eliyahu (Maamar Mordechai Hilchot Shabbat, Chelek 3 55:54) and Rav Mordechai Eliyahu (Maamar Mordechai Hilchot Shabbat, Chelek 3 55:59), who point to a Talmudic derivation from the juxtaposition of two pesukim — one referring to Rosh Hashanah as "zikhron teru'ah" (a remembrance of the blast) and the other as "yom teru'ah" (a day of sounding) — understood to indicate that when Rosh Hashanah falls on Shabbos the Torah itself prescribes only remembrance of the blast, not actual sounding, making the prohibition rooted in Torah law.
Rav Mordechai Eliyahu (Maamar Mordechai Hilchot Shabbat, Chelek 3 55:43) and Rav Mordechai Eliyahu (Maamar Mordechai Hilchot Shabbat, Chelek 3 55:55) confirm that the rabbinic decree based on Rabbah's concern — fear of carrying the shofar, lulav, or Megillah four cubits in a public domain — remains fully in force today.
Rav Mordechai Eliyahu (Maamar Mordechai Hilchot Shabbat, Chelek 3 55:52) further notes, in line with the Rambam and Shulchan Arukh, that in our time there is no shofar-sounding on Shabbos because there is neither the Temple nor an ordained rabbinic court.
Nitei Gavriel, Rosh Hashana, perek 73 addresses the related question of why shofar was sounded on the second day rather than the first when Rosh Hashanah fell on Yom Tov followed by another day of Yom Tov: in Temple times sounding was withheld until witnesses arrived to sanctify the new month, so when the first day was thus observed without sounding, the shofar — and accordingly the Shehecheyanu blessing — was associated with the second day.