The foundational picture is that the return was partial: Ezra 1–2 records Cyrus's call for "those among this people who wish to go up" to rebuild the Temple, with those who stayed behind instructed to support the project materially — already implying that remaining in Babylonia was a live and accepted option.
The numbers who actually made the journey were small, and the Sefer HaIkkarim (Maamar 3, 22:3) preserves a striking rabbinic tradition that Ezra deliberately left the most distinguished and learned Jews in Babylonia — making his ascending community, in effect, a spiritually "sifted" group — and that because the great Torah scholars remained behind, Ezra had no authority to alter anything in the Torah unilaterally, since his rulings would not be ratified by all those who had not ascended with him.
Even among the priestly families, the stay-behinds were the overwhelming majority: the Gemara (Arakhin 12b) records that only four of the twenty-four priestly watches returned from Babylonia, while twenty watches remained there, and it was the prophets among the returnees who had to reconstitute the full rota of twenty-four watches by lot.
The Levites present a parallel case: the Sefer HaIkkarim (Maamar 3, 22:3) notes explicitly that not a single Levite initially joined Ezra's ascent — citing the pasuk in which Ezra searches the assembled community and finds no Levites at all — until he sent emissaries back to Babylonia and obtained a small contingent of eighteen.
One prominent figure who remained in Babylonia was Mordechai, who Esther 2:5–6 describes as having been exiled from Jerusalem with Yechanyah, king of Yehudah — and Nechemia 7–8 (Nechemiah 7–8) lists those who did return by genealogical register, underscoring that the community in Babylonia and the community in the Land were treated as two distinct and carefully tracked populations.