The Talmud defines who qualifies as an am ha'aretz, with one opinion stating that even one who reads Scripture and studies Mishnah but has not served Torah scholars is considered an am ha'aretz — the core source for Rabbi Elazar's statement.
תָּנוּ רַבָּנַן: אֵיזֶהוּ עַם הָאָרֶץ? — כֹּל שֶׁאֵינוֹ קוֹרֵא קְרִיאַת שְׁמַע עַרְבִית וְשַׁחֲרִית, דִּבְרֵי רַבִּי אֱלִיעֶזֶר. רַבִּי יְהוֹשֻׁעַ אוֹמֵר: כֹּל שֶׁאֵינוֹ מַנִּיחַ תְּפִילִּין. בֶּן עַזַּאי אוֹמֵר: כֹּל שֶׁאֵין לוֹ צִיצִית בְּבִגְדוֹ. רַבִּי נָתָן אוֹמֵר: כֹּל שֶׁאֵין מְזוּזָה עַל פִּתְחוֹ. רַבִּי נָתָן בַּר יוֹסֵף אוֹמֵר: כֹּל שֶׁיֵּשׁ לוֹ בָּנִים וְאֵינוֹ מְגַדְּלָם לְתַלְמוּד תּוֹרָה. אֲחֵרִים אוֹמְרִים: אֲפִילּוּ קָרָא וְשָׁנָה וְלֹא שִׁמֵּשׁ תַּלְמִידֵי חֲכָמִים הֲרֵי זֶה עַם הָאָרֶץ. אָמַר רַב הוּנָא: הֲלָכָה כַּאֲחֵרִים. רָמֵי בַּר חָמָא לָא אַזְמֵין עֲלֵיהּ דְּרַב מְנַשְּׁיָא בַּר תַּחְלִיפָא דְּתָנֵי סִיפְרָא וְסִפְרֵי וְהִלְכְתָא. כִּי נָח נַפְשֵׁיהּ דְּרָמֵי בַּר חָמָא, אָמַר רָבָא: לָא נָח נַפְשֵׁיהּ דְּרָמֵי בַּר חָמָא אֶלָּא דְּלָא אַזְמֵין אַרַב מְנַשְּׁיָא בַּר תַּחְלִיפָא. וְהָתַנְיָא — אֲחֵרִים אוֹמְרִים אֲפִילּוּ קָרָא וְשָׁנָה וְלֹא שִׁמֵּשׁ תַּלְמִידֵי חֲכָמִים הֲרֵי זֶה עַם הָאָרֶץ? שָׁאנֵי רַב מְנַשְּׁיָא בַּר תַּחְלִיפָא דִּמְשַׁמַּע לְהוּ לְרַבָּנַן, וְרָמֵי בַּר חָמָא הוּא דְּלָא דָּק אַבָּתְרֵיהּ. לִישָּׁנָא אַחֲרִינָא: דְּשָׁמַע שְׁמַעְתָּתָא מִפּוּמַּיְיהוּ דְרַבָּנַן וְגָרֵיס לְהוּ — כְּצוֹרְבָא מֵרַבָּנָן דָּמֵי.
The Gemara cites a baraita with additional opinions with regard to the defining characteristics of an am ha’aretz: The Sages taught: Who is an am ha’aretz? One who does not recite Shema in the evening and morning. This is the statement of Rabbi Eliezer. Rabbi Yehoshua says: An am ha’aretz is one who does not don phylacteries. Ben Azzai says: An am ha’aretz is one who does not have ritual fringes on his garment. Rabbi Natan says: An am ha’aretz is one who does not have a mezuza on his doorway. Rabbi Natan bar Yosef says: An am ha’aretz is one who has children but who does not want them to study Torah, so he does not raise them to engage in Torah study. Aḥerim say: Even if one read the Bible and studied Mishna and did not serve Torah scholars to learn from them the meaning of the Torah that he studied, that is an am ha’aretz. Rav Huna said: The halakha is in accordance with the opinion of Aḥerim. The Gemara relates: Rami bar Ḥama did not include Rav Menashya bar Taḥlifa, who studied Sifra, Sifrei, and halakhot, in a zimmun because he had merely studied and did not serve Torah scholars. When Rami bar Ḥama passed away, Rava said: Rami bar Ḥama died only because he did not include Rabbi Menashya bar Taḥlifa in a zimmun. The Gemara asks: Was it not taught in a baraita: Aḥerim say: Even if one read the Bible and studied mishna and did not serve Torah scholars, that is an am ha’aretz? Why, then, was Rami bar Ḥama punished? The Gemara answers: Rav Menashya bar Taḥlifa is different, as he served the Sages. And it was Rami bar Ḥama who was not precise in his efforts to check after him to ascertain his actions. Another version of the Gemara’s answer: Anyone who hears halakhot from the mouths of Sages and studies them is considered a Torah scholar.