The description of that format does not hint what exactly is used, but there are essentially four ways of stuffing more data into a track, all based on reducing the format overhead of regular sector formatting.
Increased Number of Sectors
All formats leave a lengthy gap at the end of a track to avoid overwriting the first sector. If a tat gap is large enough, one could instead write an additional sector.
Increased Sector Size
There is not only a gap between the last and the first sector of a track but also one between each sector as well as between sector header and sector data. Having less sectors not only means less gaps between then and between header and sector, but also less headers. That can be done in various ways:
A) Symmetric Increased Sector Size
All sectors are increased by the same factor, hoping that with enough reduction in sector number space for one more is provided.
B) Filling Up With Variable Size
Increased sector size may not always produce size allowing even distribution of a track, which would result in a gap too small for another sector. This method adds sectors of smaller size until that gap is filled as much as possible.
C) Writing a Single Sector Per Track
That's essentially putting #2 to the max. A sector equals a track. Having only one sector means also only a single gap between header and sector and a single gap at the end of the track. All overhead is reduced to the absolute minimum possible.
Writing a Track With Only Minimal Gap Size
Here Sectors are written as usual but the gap between header and data as well as between sectors are reduced to the absolute minimum needed for synchronisation. This can only be done when the whole track is written as one using a special command.
A lot can be done, isn't it? But all of these have their issues. And all of them are about timing - or better timing differences between writing and reading when using different drives. We tend to forget that a floppy recording is not only about encoding data into a time variant signal, but that this signal can be read back at a different timing.
Or in simple words, the drive writing a disk may have had a different rotational speed than the drive a disk is later read and rewritten.
And believe me, there was a lot of variation, especially early on. An SA-400, the grand daddy of all 5.25 drives, was considered in spec when rotation was +/-10%. This means no format should use more than 90% of a track, otherwise the recording may not be compatible with other drives. Since the speed variation does as well influence the space one had to provide for each sector - at least not if a faster drive should be able to rewrite a sector on a disk formatted by a slower drive.
As a result, basic formats usually stayed below 80 % of a track.
(Following all calculation will be done using the basic 5.25 inch, 40 track, MFM drive to keep it simple - application to later 1.44 follows the very same rules, just more tracks and sectors)
#1 is what IBM already did with the PC going from 320 to 360 KiB per disk by writing 9 sectors instead of 8. The original 320 KiB format was developed with a really high security margins, so it may work with the worst drives imaginable (+/- 10%). By the time the IBM-PC was created (1981) Speed variation was down to +/- 4.5%. Together with the previous margin this was enough to squeeze in another sector without increasing risk of overwrite.
Of course, this only works once as a general improvement. Then again, if one had a drive toward the fast side and no need for data exchange another (10th) sector could sometimes be fitted. This was quite common on machines like the Atari ST. Still fully DOS compatible, as DOS takes the number of sectors per track from a disks boot sector, but not always readable on all machines.
#2A is another easy bet, benefiting from reduced sector overhead. In fact it also played a role why IBM's 360 KiB format worked out fine. Standard MFM format is designed for 16 sectors of 256 bytes user data. By using 512 byte sectors IBM already halved the sector overhead.
One can increase that further. By using 1024 byte sectors instead of 512 byte on an IBM 5.25 drive would allow 5 sectors per track, holding 5 KiB instead of 4.5 KiB per track or 400 KiB per Disk. Using three 2048 byte sectors instead would yield 6 KiB but is too close to maximum track capacity to work reliable across all machines and for data exchange.
Increasing sector size would as well work fine with all DOS and all compatible OS as sector size is recorded on disk and blocking/deblocking is part of the DOS spec. Well in theory. Any practical use would be a test how well drivers implement this.
#2B sounds like a great way to solve this by closing in to maximum capacity. In case of an MFM drive a combination of one 4096 , one 1024 and a 512 Byte sector would add up to 5.5 Kib per track or 440 KiB 40 track 5.25 Disk.
But there's a caveat - DOS does not support variable sector size. THere is only a single variable for sector size and only that is used by DOS to calculate sector numbers for access. So any format using this would need very specific drivers, showing DOS a same size organization and do all conversion on it's own. Not exactly portable.
#2C would mean creating one sector with a size close to track capacity. With a brutto track capacity of about 6250 byte and some security margin this ends at a sector size of ~5.8 KiB. Cool. Except it's even worse than #2B, as DOS does not support odd sector sizes.
#3 now tackles the GAPs directly. A 512 byte sector including headers and gaps takes 658 byes on disk. Of those 106 are gap bytes. Another 330 are within the track header/trailer, giving a total of 1284 gap bytes. Even with keeping a fully compatible 512 byte sector size, one could put 11 sectors for the same 5.5 KiB per track (440 KiB per Disk) as in #2B. All readable by standard read command from every standard drive out there. This works as we haven't touches any sync bytes or headers.
(For a 1.44 MB Disk this is means an increase to 1.76 MB)
The only, but critical issue will be writing. When using a standard write command, the controller will seek as with reading but then write including all gaps, which will always overwrite the header of a following sector, thus destroying the disk. Writing a sector can only happen reading all sectors of that track, repackaging them in the condensed format, inserting the one to be written and then write the track at once. Not anything one can expect from DOS or any standard floppy driver.
Oh and there's another caveat: The write track command can not write data bytes containing the value of F5 or F6 - so this would need another hack to be discovered :)))
Long story short,t such a disk would be great to distribute software or do data backup, but not as a regular rewriteable exchange medium.
Pick your solution.