For example, the VIC-II is a big complex chip for its time; at least in the early days, yield must have been significantly less than 100%.
Not really, while the VIC-II had a transistor count a bit larger than the original 8µm NMOS 6502. Not a lot, but it was manufactured in a 5 µm process, resulting in a smaller size and higher yield. The 1981 65C02 had almost three times the transistor count of the NMOS CPU, but due to being manufactured in 3 µm, its size was only a little over 1/4th (6 vs 21 mm^2) and 5 µm was already considered outdated in 1980.
Most of the area of the VIC-II is used for sprites.
I wouldn't call the 40 bytes of line buffer insignificant
That suggests the majority of reject chips would be perfectly usable for non-sprite display.
The whole idea of using rejected chips, especially when it's about rather small (even at that time) chips like the VIC, isn't working out in a commercial way. At least not if you're the original manufacturer and/or customer for such a 'less function' version.
Lets assume, for the sake of the argument, that the area used for sprites is one quarter (25%) of the chip, and the failure location is random. So of all chips with only one failure location, only the 25% with that failure can be used as 'without sprites' (again assuming given that it isn't a failure disabling disables the whole chip - like a contact between +5V and ground :)) For chips with two failures it's maybe another 5% that fall in the same area, therefore it isn't worth looking at even higher rates.
Next, this is all in relation to the fault rate of the production. Here it is important that the 5 µm process on which the VIC-II was produced was well proven and established. Not anywhere near cutting edge - that would have been around 1-2µm at that time. I think it is safe to assume that Commodore did get a yield of way more than 80% already at the first run, and they would have been able to get this to 95+% within a few months.
With numbers there is no business case for the manufacturer to go for 'less functional' versions. After all, at 80% good rate (which is rather bad), the best possible quote (if all failing chips had only a single failure) would be 5%. That's no production number even worth making a different stamp to mark them. Especially, as from a manufacturer's perspective, all to do is reducing the fault rate to improve sales of the main product.
Last but not least, thorough testing slows down production and costs a lot of money - much more when done into every detail like needed to select chips that are still working at a reduced level. Money that is wasted on good chips. So getting quality up and restricting tests to detect faults as fast as possible and then aborting makes the whole production more cost efficient.
From a customer's perspective (in this case the Commodore computer factories), it's even worse, as your ability to build such a machine on a random output quota from the manufacturer of a product he doesn't want to make in the first place. So with every iteration the VIC-II output gets improved, your supply goes away. To further output machines you must go ahead and use full functional VIC-IIs - which would mean that there you earn less money (assuming the final customer pays less for a less capable machine) from the same resource (VIC-II chip production).
[But then why did the 32 KiBit RAM chips surface?]
A simple matter of scale. For one, 64 KiBit RAM chips are about 10 times more complex than a VIC-II, while at the same time their structure is way more symmetric, enhancing the chance that a faults impact is truly local. Even more important, they were produced in (almost) infinitely larger numbers than for example VIC-IIs - just think, each C64 already had 8 of them compared to only one VIC-II. Also there were many more manufacturers. Each of them trying to get their production running and quality up - each of them going through a cycle from many faulty chips to less of them.
Also, fault-wise it was more simple with RAMs, as any chip with only a single fault that is not located in control or interface (which is less than 2% of the surface) can be used as 32 KiBit one - that makes, using the above reasoning, 98% of single faulty and 49% of double faulty. Sounds like a way better quota doesn't it?
It's further a matter of scaling. With many millions of 64 KiB chips, each factory, especially during run up, produced hundreds of thousands of partial faulties, making it a good business for others to buy this waste at next to nothing and spend money in testing and repackaging then as 32 KiBit.
Bottom line, it's about numbers and Commodore wasn't going anywhere near this.
Still useless for the C64, of course, but Commodore did (mistakenly, but still did) introduce the C16 as a spriteless lower-end machine.
I need to object here (even though not part of the core question). The C16 was neither a mistake, nor a bad idea. Its design was meant to counter Sinclair. Delivering a low end machine that could beat the Monster from the Island :)) Serious, the US home computer industry was shocked by the ZX-80/81 and the prospect of an even more capable future colour machine. The TED as C116 was directly set against that - and the C16 only a fast byproduct of a cheaper 'real' computer using VIC20/C64 parts.
Instead of basing it on the Plus 4, could they not have given it reject VIC-II chips and saved money?
As noted above, there wasn't any real business case to save money. More importantly, the TED had a much higher gate count than the VIC-II. Not least due to its 75 (instead of 40) byte line buffer; it also had a 16-bit timer, a sound channel and an I/O port. Except for the sprites, all its graphic capabilities surpassed the VIC-II. While being more complex it was at the same time targeting a way lower priced market, this should tell how little transistor count meant - at least in this region of rather low integrated ones.
Similarly the excellent SID (C64 sound chip). Intuitively it would seem a defect in one of the voices, say, should leave the chip still usable with a smaller repertoire in a lower-end machine like the C16. Is there a reason why this was not done?
Most likely again, not enough failing chips to make it worthwhile. See above.
But more important, it would have meant that the C16 would have had two chips, like the C64, making production way more expensive than it was targeted for. Beside more board space and traces, also another 64 drillings would have been needed, well, plus having to produce two chips instead of one.
The one historical case I know of where part-bad chips were successfully sold with reduced functionality is the 486, where units with a defective FPU were sold as the 486SX. Are there are any other cases?
If at all, only in the beginning. for most of their availability, 486SX were crippled on purpose to allow sales at lower prices, without hurting the profits on 486DX sales.