[W]hy not make the drives even cheaper by jumping straight to an even smaller form factor such as the 3.5" that was eventually settled on?
Because that needed a new technology. The move to 5.25 inch didn't change any technology involved. Everything stayed the same:
- Drive design
- Drive mechanics
- Electronics (including analogue)
- Material of floppies
- Each and every step of Manufacturing of floppies
The only difference was in size, a linear shrink with a factor of about 1.5 (*1), which means the needed surface and thus size shrunk in half (*2). All without any basic change, just relative minor adaptations. For example to manufacture the floppies only the punch to cut out the magnetic foil with its corresponding holes (same for sleeves) and folding brackets closing a disk sleeve and so on had to be made. So just new tools to be installed on existing machinery. To produce 3.5" ones, new steps, different materials and different handling had to be developed - quite a large investment, especially compared to retooling.
It's a bit like the Tick part of the often cited Tick-Tock strategy Intel follows for CPUs. A shrink of an existing design just in scale, not design or function. The following Tock was then again a step of design changes (*3).
(Beside, a 3.5 inch drive in an 8 inch bay would just look ridiculous :))
Some comments (for example jcarons) asked why not going smaller, like 3.5, but keeping technology the same. Beside all the technical reasons regarding recording (like density), it's much about mechanical reliability. For example due thinner materials the medium gets more sensitive to damage - and more so, a smaller hole in the middle reduces reliability for centering while increasing stress on the material (due a reduced lever) at the same time.
This isn't anything theoretical, as Mitsumi tried exactly this with their 2.8" variation. Beside having meager capacity of 64 KiB per side, they were extremely unreliable. For Nintendo's Famicom-Disk they created a redesigned version borrowing the centering mechanism (in plastic) as well as a hard case, making it match the setup of a 3.5" drive - sans the slider, keeping the need for sleeves.
So, no, going way below 5" by keeping the 8" technology wasn't a serious option.
*1 - It's sufficient close to square root of 2 (1.41)
*2 - drive height was kept the same thus half volume. Later developments cut the height as well in half, resulting in 1/4th volume of a slim line 5.25 compared with a full height 8 inch
*3 - In fact it was a competition of many designs: IBM's 4", Sony's 3.5", Matushita's 3", Mitsumis 2.8", Sharp's 2.5" and Fujitsu's 2". Not to mention spiral formats like Sony/Canon's 2" VideoFloppy