The starting point is Pirkei Avot 2:7, which states plainly that increasing possessions increases worry, placing this within a series of parallel observations about excess in wealth, women, servants, and the corresponding excess in Torah, counsel, and charity.
Rabbeinu Bachya explains this mishnah by describing how the one who pursues wealth is never at ease, fearing the king may seize it, robbers may steal it on the road, or his ships may sink at sea, so that increased possessions bring only increased dread rather than increased calm.
Midrash Shmuel on Avot 2:7 develops the same teaching further, noting that one who amasses wealth often labors not even for himself but for those who will inherit after his death, and that such a person is anxious both about losing what he has and about failing to seize every opportunity to gain more.
This concern is echoed in Tanach itself: Kohelet 5:11 observes that the laborer sleeps sweetly regardless of how much he eats, but the rich man's abundance does not let him sleep, and Tehillim 39:7-8 describes a person amassing wealth in futility without knowing who will ultimately gather it in.
Mishlei 15:16 and Mishlei 30:8-9 frame the proper response to this danger not as rejecting wealth outright but as preferring a little with fear of God over great wealth with confusion, and asking instead for a moderate portion so as not to be drawn either to denial of God through satiety or to theft through poverty.
Mesillat Yesharim 13:20 applies this concern practically, describing how the pursuit of wealth to maintain one's table draws a person into the toil of commerce, and from there onward to wrongdoing, theft, and false oaths, ultimately removing him from divine service, Torah study, and prayer.
Sotah 48b roots the antidote in trust rather than asceticism, teaching that one who has bread for today yet worries about tomorrow is considered of little faith, implying that the anxiety attending increased possessions reflects a deficiency in trust in God's providence.