The foundational rule appears in the Mishnah cited by the Gemara (Bava Kamma 92a): the rich brought their bikkurim in baskets of gold and silver while the poor brought theirs in wicker baskets of peeled willow — and whereas both the baskets and the first-fruits were given to the kohanim, the rich had their expensive baskets returned to them while the poor did not get their wicker baskets back.
The Jerusalem Talmud (Bikkurim 3:4) records the same rule in its own version of the Mishnah, confirming that it was the baskets and the bikkurim together that were handed to the kohanim — with no mention of any return of the wicker baskets to the poor.
Rashi (Bava Kamma 92a) makes the asymmetry explicit in his brief gloss: the wicker baskets of the poor were given to the kohen, while the gold and silver vessels of the rich were returned to them.
The Rambam (Mishneh Torah, Bikkurim 3:8) codifies the rule by framing it in terms of vessel material: if the bikkurim were brought in metal vessels the kohen takes the first-fruits and the vessel is returned to its owner, but if they were brought in willow-wicker or similar baskets, both the bikkurim and the baskets belong to the kohanim — with the additional qualification that if the bikkurim became ritually impure, the baskets do not go to the kohanim at all.
The Tosafot Yom Tov (Bikkurim 3:8) quotes and sharpens this language — the wicker baskets of the poor are given to the kohanim while the vessels of the rich are returned to them — and it is precisely from this disparity that the Gemara derives the folk saying that poverty chases the poor.
The Gemara (Bava Kamma 92a) uses this Mishnah as the proof-text for the adage that poverty follows the poor: the one who can least afford to lose the basket is the very one who does not get it back, illustrating how disadvantage compounds itself.
The Likkutei Sichos (vol. 29, Ki Tavo) analyzes this halachic distinction and concludes, on our reading of the cited page, that the poor person's wicker basket becomes intrinsically unified with the bikkurim themselves — unlike the rich person's metal vessel, which remains separable — and that this dynamic reflects a spiritual principle in which the humblest physical vessel can be wholly elevated to holiness.