In practice, 5.25" drives equalled or exceeded the capacity of 8"
drives when 5.25" floppy drives started using HD media.
The Evolution of 5.25" Drive Systems
There are only two different recording media that were ever
commonly used in floppy drives: the original media (often called "DD") and
the later HD media. Note that when it comes to floppy diskette drive
systems the term "density" is somewhat confusing because it's used not only
for media (i.e., the composition of the recording surface) but also radial
track density (e.g. 48 or 96 tracks per inch (tpi): a property of the
drive, not the media) and the encoding of bits into flux transitions (a
property of the controller, not the drive or the media).
Over time, physical capacity was expanded in various ways. The following
roughly traces the evolution 5.25" diskette systems. The sizes are for
popular formats but could easily vary by 20% or more in either direction,
depending on sector size and other factors. (These answers mention
a variety of formatting changes used to put more data onto a floppy using
the same physical density specifications.)
- The original single-sided single-density systems used original media, 40
tracks (48 tpi) and FM encoding for a capacity of around 90 KB/diskette.
(Some early drives could address only 35 tracks.)
- Single-sided double-density (SSDD) systems used original media, 40
tracks and MFM encoding for a capacity of around 160 KB/diskette.
- Double-sided double-density (DSDD or sometimes "2D") systems obviously
doubled the above to about 320 KB/diskette.
- The move to 80-track (96 tpi) drives, sometimes called "quad-density" or
"2DD" doubled that again to 720 KB. But note that the double- to
quad-density increase was via a completely different method than the
single- to double-density increase, via a change to the drive head
rather than a change to how the controller wrote data to the diskette.
- HD drives used a different media that approximately doubled the linear
recording density, and they started out as 80-track drives, so they
stored about 1.2 to 1.44 MB/diskette. For reasons I won't get into the
most popular 5.25" format was 1.2 MB (less than double) even though 3.5"
systems (which are essentially slightly miniaturised 5.25" systems)
exactly doubled it to 1.44 MB.
8" vs. 5.25" Drive Systems
8" drive systems in common use followed the first three steps of this
progression, but at the same recording density offered both more tracks (77
instead of 40) and more sectors per track (typically 26 × 256 byte sectors
versus 16 × 256 byte sectors for 5.25"). Thus, in one common format a DSDD
8" diskette stored around 2×77×26×256/1024 = 1000 KB whereas a similarly
formatted 5.25" diskette stored 2×40×16×256/1024 = 320 KB. The most common
format soon changed slightly to increase the sizes to 1.2 MB for an 8"
diskette and 360 KB for a 5.25" diskette.
5.25" drives started their progress towards catching up when drives with
narrower heads running at 96 tpi were introduced, bringing their standard
capacity to 640 or 720 KB. But this was still considerably less than 8"
drives. It was when HD media was introduced that the standard 1.2 MB 2HD
5.25" format approximately equaled the capacity of the 8" DSDD format.¹²
Capacity was immediately extended beyond this for special cases such as
Windows installation diskettes; a change in formatting allowed 1.5-1.8 MB
on an HD diskette. But the same formatting tricks (mainly, larger sectors
for less overhead) were possible on 8" drive systems, though I don't know
if they were ever used.
¹ In the case of the NEC PC-9801 series, it exactly equaled the 8" format
because NEC for whatever reason decided that they would use exactly the 8"
format: 77 tracks (even though the drive supported 80), the same number and
size of sectors and, strangely enough, even running the drive at the
standard 8" rate of 360 RPM rather than the 300 RPM at which 5.25" drives
normally run.
² The 3.5" HD diskettes stored 1.44 MB instead of 1.2 MB because they used
18 instead of 15 sectors per track. They were capable of this because, as
part of the process of "shrinking" the 5.25" diskette to 3.5" whilst
maintaining exact format compatibility they had to increase the
coercivity to compensate for the smaller surface area. DD/HD
coercivity for 5.25" media was 300/600 Oersteds; for 3.5" media it was
660/720 Oe.)
Further Development
8" drives were perfectly capable of improving in the same way as the 5.25"
drives did, moving from 48 to 96 tpi and DD to HD media. Indeed, someone
had obviously manufactured such drives because Maxell manufactured media
for them. But by the time these improvements were available 8"
drives had become relatively expensive and inconvenient compared to even
5.25" drives, not to mention 3.5" drives, and this is probably the reason
why 96 tpi HD 8" drive systems were very rare. But if you don't take this
rarity into account, 5.25" drives never caught up with 8" drives in storage
capacity.
3.5" drive system capacity continued to increase, though HD media was the
last system that received truly widespread acceptance. 3.5" ED
drives ("Extended Density," with a typical formatted capacity of
2.88 MB) introduced new ED media and perpendicular recording and were
fairly widely available, though they never achieved the widespread
acceptance of HD drive systems. By this point 5.25" drive system
development, like 8" drive systems before it, had fallen off, but even so
there was a 5.25" 2.44 MB drive system (probably also using ED
media) standardised enough to be available from at least two manufacturers.
Eventually laser-servo designs bumped 3.5" capacity up to 20 MB and
beyond. The only format here that achieved any significant popularity were
the LS-120 and LS-240 SuperDisk formats (120 and 240 MB, respectively). I
don't know how difficult it would have been to transfer laser-servo
technology to 5.25" or 8" systems.
Further Information
A good source of many fascinating details about many kinds of floppy drive
systems is Herb Johnson's Tech information on floppy disks drives and
media web pages, which provided many of the references above. (But I may
be biased since I am, in a small way, a contributor to those pages.)