The earliest source establishing the legal category is Mishnah Sanhedrin 3:3, which lists one who plays with dice (המשחק בקוביא) among those disqualified from serving as a witness or judge, placing gamblers alongside usurers and those who trade in Shemittah produce.
Mishneh Torah, Robbery and Lost Property 6:10 explains the underlying prohibition: one who plays with sticks, pebbles, bones, or similar objects under a winner-takes-all condition has committed rabbinic-level theft — because even though the loser consented, taking another's money for nothing through sport and jest constitutes gezel.
Mishneh Torah, Robbery and Lost Property 6:11 adds a second, independent ground: gambling with a non-Jew, where the theft concern does not apply, still carries the prohibition of engaging in matters of idleness, since a person ought to occupy himself only with words of wisdom and the constructive needs of the world.
Shulchan Arukh, Choshen Mishpat 370 codifies both dice games and wagers on animal races under the same heading — rabbinic theft — and the Tur (Choshen Mishpat 370) confirms that all such competitions in which the winner takes from the loser fall within this category, with the Beit Yosef (Choshen Mishpat 370:1) tracing the ruling explicitly to the Gemara in Sanhedrin.