Actually, the most probable limit was the amount of the on-chip RAM to hold sprites. To my knowledge, sprite cirtuitrycircuitry, including sprite on-chip RAM (or just flip-flops) were responsible for the majority of silicon space in VIC-II chip.
In all the architectures @rwallace listed none were able to buffer even the single line before actually outputting it to TV: memory fetch was synchronous to the pixels outputting. Hence, the sprite data had to be fetched ahead of time and outputted just at the required moment when sprite coordinate comparators fire. The need to pre-fetch and store sprite data on-chip follows from this. Therefore silicon area, rather expensive at the time, limited the size and number of sprites available on those architectures.