The core machlokes is recorded in the Gemara itself: where a man fathered children as a gentile and later converted, Bekhorot 47a:2 brings Rabbi Yochanan ruling that he has no firstborn for inheritance purposes, since the earlier son already held 'the first of his strength' before the conversion, against Reish Lakish who holds that a convert is like a newborn, so the first son born after conversion counts as the firstborn for inheritance.
The Mishnah and Rambam align with the stricter reading on the father's side: Bekhorot 46a:8-10 and Mishneh Torah, Inheritances 2:12 both rule that once a Jewish man has fathered a son from a maidservant or a gentile woman, that son is not called 'his son' in the halachic sense, so the next son born to him from a Jewish wife is nevertheless a firstborn for inheritance and takes a double portion.
For pidyon haben the question turns instead on the mother's womb rather than the father's prior children: Shulchan Arukh, Yoreh De'ah 305:20-21 rules that a maidservant or gentile woman who conceives while still unfree or non-Jewish but gives birth after she is freed or converted bears a son obligated in redemption, since the birth itself occurred in sanctity, while one who already gave birth before converting or being freed and then gives birth again afterward bears a son who is exempt, as that later birth is no longer the 'first opening of the womb.'
Related rulings in Yoreh De'ah on pidyon haben confirm that the mother's status, not the father's, is decisive: Shulchan Arukh, Yoreh De'ah 305:18 holds that a kohenet or leviyah's son by a Jewish husband is not obligated in redemption because the law follows the mother, and conversely a leviyah's son from a gentile is exempt while a kohenet's son from a gentile becomes obligated because her priestly status was voided by the gentile's relations; Arukh HaShulchan, Yoreh De'ah 305:2 likewise explains that the mitzvah of pidyon haben follows peter rechem through the mother, which is the opposite principle from bechor lenachalah, which follows only the father.
Rav Eliezer Melamed extends this same womb-based logic to the case of a gentile woman who converts during her first pregnancy and gives birth as a Jew: Peninei Halakhah, Family 9:10:2 rules that even though the father is a gentile and the mother was a gentile at conception, the son is obligated in redemption because the birth occurred in sanctity.
Among the modern responsa, Igros Moshe YD 1:195 addresses the case of a son born to a Jewish mother and a non-Jewish father and rules that he should be redeemed, preferably through a limited zechiyah arrangement, while discussing whether the redemption brachah and Shehecheyanu are said; Tzitz Eliezer 9:28 takes a more hesitant position on the same configuration, citing authorities who hold the court is not obligated to redeem such a son and that no brachah should be recited, treating any redemption as doubtful until the son redeems himself as an adult without a brachah.
Teshuvos V'hanhagos 8:57 considers a related but distinct question, whether a later-born child can still be deemed a peter rechem for pidyon haben once an earlier child already emerged from that womb, distinguishing this from the laws governing a firstborn animal.
Supporting background sources establish the broader frame: the obligation of pidyon haben is rooted in Shemot 13:2 and Bamidbar 18:15, which tie redemption to the first issue of the womb; Sefer HaChinukh 392:6 likewise explains that the mitzvah depends on the mother's side, exempting even a Jew's son by a kohenet or leviyah; Shulchan Arukh, Yoreh De'ah 305:23 details which prior births void the 'first opening of the womb' status; and Kiddushin 8a:8-10 discusses the mechanics of redemption once the obligation applies. Shabbat 135a:3 and Responsa Chatam Sofer, Yoreh De'ah 296:7 touch on related conversion and redemption mechanics but do not alter this picture.