The chronology of some early floppy standards was: 80kb, 160kb, 360kb, 720kb, 1.2mb, finally 1.44mb. (There where less common sizes such as 250kb, 800kb, 1mb) mentioned by comments: 120kb, 2.88mb :)
There were not only many more, but also way relevant ones. Just think of the Apple II's 140 KiB format which at one point might have been the most used one. Or Commodore's 1541 with its 170 KiB format, for sure outselling many PCs of the same time frame.
Or Apples 800 KiB format for the Mac - it took many years until the so called SuperDrive could also read 1.44s - mainly for data transfer reasons.
The last size (and still the most common today) is 1.44mb.
Not really, as development already advanced when the 1.44s were making their way into the PC. At the time IBM introduced the 3.5" HD drives (1987) with their (somewhat) ill-fated PS/2, 2.88 ED drives where already available. It took many years to establish the 3.5" drive within the PC market, as its capacity increase over 1.2M drives was negligible, but more importantly they were incompatible - folding 5.25 disks wasn't a viable solution :))
In fact, it may be useful to add machine type and region, as well as business area to qualify on what this claim is based, as for example in Japan the 2.88 drive is/was the most common one in PCs. Or take Apple's legacy-free iMacs, which was almost always sold with a SuperDisk able not only to do 1.44s, but also up to 31 MiB on standard floppies (and 120 or 240 MiB on dedicated diskettes). Similarly, certain business areas did see standard use higher density formats, like flopticals and Bernoulli (Zip) drives. Usually due to the fact that there was a great need for data exchange with large amount of data per item - most prominent here may be publishing.
That's an incredible wide area putting atop a rather fuzzy definition (*1), so there may be many possible answers, so lets qualify it a bit and look at this in a greater detail. The firstmost might be to clarify about what defines as a floppy for this question.
By Strict Compatibility
To qualify a followup must not only use the same technology (see below) but also the same media - or at least media that can be formatted and used with 'old' drives. I'm not really sure if this is intended by the question, as it obviously mentions 8", 5,25" and 3,5" formats. Also by calling the 1.44s 'floppy', when, in other places they where called 'stiffy' :))
By Technology Similar to the Mentioned Formats
This category would contain all devices using magnetic recording on exchangeable rotating media. There may be subcategories as to include exchangeable drives or not.
By Usage:
Here any media that allows data exchange between machines by being physically carried (and meant to do so). While this seams a bit wide and may even include punch cards, the addition of allowing random access would already nail it down to what floppies where for practical purpose in the 80s and 90s.
A History of Exchange Media May Look Like This
(Starting with the 3,5"/1.44, and not counting tapes or large/mainframe removable drives that is)
1981/2 3,5" Floppy got introduced/standardized
1982 10 MiB Bernoulli drive using flexible 5.25" media
1983 CD-ROM gets standardized by Sony/Philips
1984 720 KiB 3,5" drives became available to end users
1986 IBM used 720 KiB 3,5" drive with the IBM PC Convertible
1986 44 MiB Removable disk SQ555 introduced by SyQuest
1987 1440 KiB HD format introduced with the PS/2 line
1987 2880 KiB ED format introduced
1987 20 MiB Bernoulli II drives which later evolved up to 230 MiB
1990 650 MiB CD-R 'drive' from Philips (35,000 USD)
1991 20 MiB Floptical (sold as 21 MB)
1991 88 MiB SQ800 by SyQuest
1991 Creative Labs' SB Pro brings CD interfaces into mainstream PC
1994 100 MiB Iomega introduced the 100 MB ZIP drive
1995 135 MiB SQ135 by SyQuest
1995 HP introduced the 4020i as first CD-R writer under 1000 USD
1996 120 MiB LS-120
1996 230 MiB SyQuest EZFlyer
1996 1 GiB Jaz Drive (Removable Disk)
1997 240 MiB LS-240 - capable to write 30 MiB to a standard HD floppy
1998 144 MiB Caleb UHD144 drive
1999 200 MiB Sony HiFD
1999 40 MiB Iomega places the PocketZIP as media for consumer devices
After 2000, next to all development for floppy-like media dropped as CD-R and the upcoming DVD-R had taken over that niche.
(A honorable mention should also go to MO drives as used in many professional backup solutions as well as the NeXT Cube - but not really much in the PC world. Together with CD-RW (Sony Mini-Disk and many solutions of easy to exchange disk drives under various *pack/*paq names)
So even way before IBM adopted the 1.44 drive, floppies did reach not only higher densities, but also wide range use. And while the 1.44 HD format was often the basic drive, there were many differences even considering only 'simple' drives adhering to the 3,5" mechanics and compatibility with the HD format.
Already before 1990 the 2.88 ED format became the standard in Japan and most parts of Asia.
The Floptical did gain some usage
During the late 90s and into the 2000s LS120/240 drives became widely used all around the world - again with great focus in Asia but also Europe. And as de facto standard drive for iMacs after 1998.
Last atempt here was the UHD144 and Sony's HiFD
Unlike the 2.88and the LS drives, none of the later did gain any market hold.
Similar in the non 3.5" HD compatible range of Syquest's removable disks and Iomega's Bernoulli/ZIP drives did gather a reasonable market share, but mostly as add on products (where 2.88 ans LS drives where also available as OEM choices).
All of this development finally ended when CD-R (writable) - and especially their media - became available and cheap. Mind you, where even the cheapest LS120 or UHD144 was 5 USD (and a ZIP still 10 USD), a (one time writable) CD-R got sold for less than 1 USD in a 25 pack.
The same way price did help the ZIP drives to succeed over the more compatible and reliable LS120/240 drives, it got killed by the even cheaper CD-R. After all, outside serious and repeated data transfer, no one really needs to rewrite a disk.
Was the limitation speed, storage density, cost, format standards, user demands, royalties, patents, rivalry, something else?
Not really. Again, there were many improvements made, even when restricting it to the most narrow definition of a 3.5" media compatible with HD drives. And all of them have had a wide application selling hundreds of millions of drives, adhering to widely licensed and acknowledged standards
If at all, a business case can be made.
For a serious look it might be good to rephrase from a seemingly technical question into one of usage. So when narrowing it down to the (more or less) standard PC and ignore most of the mentioned parts of the IT world, we may see three basic classes of users:
(Large) Professional/Institutional installations where PCs were network connected. Here FDs were at most needed for initial installation and maintenance. In such installations FDs were often disabled or otherwise locked. No need for any investment in bigger formats. (*2)
Small business (without the need for large chunk data exchange), here a FD was usually only needed for software installation. A rather rare task where handling of a dozen floppies was acceptable. While simple, incremental backup could be handled usually with compressed archives on floppies, only full backup might call for large removable media, but due the lack of any need for data exchange a wide variety of drives and standards have filled that nice. Ultimately CD and DVD writers have taken over the role for backup and data exchange - followed by an in-creep of network backup, formerly only found in large installations, due to the use of NAS systems.
Home use. To some degree similar to the small business case. Again it was the advent of the CD in 1991 that replaced the floppy as main installation media (And let's be serious, which home user did care enough to back up their machines?) - and in this case even as main run media when it came to games. After all, the inherent copy protection of a CD (at least before writers became cheap) was a major push to publish games on CD and for home users to buy machines with CD drives. Like before CD/DVD writers have taken, followed by flash drives (USB-Sticks) over plus various kind of online storage.
Again, not to forget, that flopticals (LS) and Bernoullies (ZIP) each sold many tens of millions of drives and quite a few companies (Dell, Compaq, Siemens, even IBM) offered such drives (usually LS) as standard setup with their machines. Not to mention the hundreds of millions of 2.88 drives sold in Japan and all over Asia.
Why did common floppies never advance past 1.4mb in size?
Beside the fact that they did and there were other, larger standard exchange medias, depending on region, machine and business?
Bottom line, there was much development over all ranges in flexible magnetic media in 3.5" housings as well a with non flexible, non magnetic, non 3.5", or even non rotating when it came to removable exchangeable media. And most of it was driven by open standards and patents, or closed but still successful designs, either accepted by major industrial players like IBM, Microsoft or Compaq.
*1 - Floppy is one of these seemingly simple and clear terms, that loses all definition when looking close.ly We all think we know what it is - until we go into details what is and what is not a floppy. Only 8" and 5.25", where media and case is 'floppy' or may either be 'hard'? Does the head have to touch the surface or may it fly? And so on.
*2 - Even today this type stays without removable media - floppy drives are replaced by disabled drivers for storage over USB.