How does 2M for DOS (2 MB formatted floppy disks) achieve what it does?

Question

This is the program and technique in question:

https://dbpedia.org/page/2M_(DOS)

Also described here:

https://en.wikipedia.org/wiki/2M_(DOS)

And here is the unintelligible explanation from the website:

"2MGUI utilized bit banging and tricked the floppy controller into writing a full track of data as a single sector before resetting the floppy controller in order to avoid overwriting the next track, enabling the absolute maximum capacity possible at the physical level. A similar technique was also used by Vincent Joguin's Disk2FDI for Amiga floppies."

Ok, it gets direct access to the floppy controller (instead of using DOS interrupts), with you so far.

And then: "writing a full track of data as a single sector"

Uh, what?

A floppy disk looks like this:

A track consists of multiple sectors, how do you write a full track into a sector?

I can imagine that there would be used some better, more modern encoding scheme than initially used for 1.44 MB floppy disks to achieve higher density. However, if the floppy disk supports writing a full track worth of data into each sector, then why didn't they simply do this originally? Is the data more prone to corruption? Is it that simple -- you can just tell the floppy disk controller to stuff higher density data into a sector than usual, but at the cost of higher error rates?

I don't think it is this simple. Clearly there is a lot more to it than this simple explanation of just stuffing a track into a sector.

Answer 1

A typical floppy disk track will contain, depending upon the exact medium, between 8 and 24 repetitions of the pattern "sector header, sector data; sector header, sector data", etc. Each sector header contains a flux transition sequence that is only allowed to appear at the start of a sector header, followed by information about the sector. Each sector's data starts with a flux transition sequence that is only allowed to appear at the start of a sector's data, followed by the data itself.

In order to read a sector, the drive controller will look for a sector header marker, then read the information about the sector to see if it's the one of interest. If not, it will revert to looking for the next a header marker. Once the desired sector is found, it will watch, briefly, for the "Start of data" marker and once it is seen, read a sector's worth of data. If the start of data marker wasn't seen very soon after the sector header, the drive should revert to looking for another sector header.

To write a sector, the drive controller will look for a sector header as above, but once it has seen the right sector it will write out the sector data marker and a sector worth of data, blindly overwriting whatever passes under the head. In order to avoid overwriting the start of the next sector, the controller must be certain that it finishes writing a sector before the data for the next sector spins under the head. To accommodate the possibility that data might be written with a drive that's spinning faster than the drive used to write it originally, disks are formatted with some unused space between sectors.

The 512-byte sectors used on PC-DOS disks represent a trade-off between buffering requirements and inter-sector waste space. Using larger sectors would reduce the amount of space wasted between sectors, but because updating even a single byte in a file requires reading an entire sector, editing the data in RAM, and writing the entire sector back, using larger sectors would make it necessary to read or write more data at once.

Answer 2

The description of that format does not hint what exactly is used, but there are essentially four ways of stuffing more data into a track, all based on reducing the format overhead of regular sector formatting.

1. Increased Number of Sectors

   All formats leave a lengthy gap at the end of a track to avoid overwriting the first sector. If a tat gap is large enough, one could instead write an additional sector.

2. Increased Sector Size

   There is not only a gap between the last and the first sector of a track but also one between each sector as well as between sector header and sector data. Having less sectors not only means less gaps between then and between header and sector, but also less headers. That can be done in various ways:

   2. A) Symmetric Increased Sector Size

      All sectors are increased by the same factor, hoping that with enough reduction in sector number space for one more is provided.

   3. B) Filling Up With Variable Size

      Increased sector size may not always produce size allowing even distribution of a track, which would result in a gap too small for another sector. This method adds sectors of smaller size until that gap is filled as much as possible.

   4. C) Writing a Single Sector Per Track

      That's essentially putting #2 to the max. A sector equals a track. Having only one sector means also only a single gap between header and sector and a single gap at the end of the track. All overhead is reduced to the absolute minimum possible.

3. Writing a Track With Only Minimal Gap Size

   Here Sectors are written as usual but the gap between header and data as well as between sectors are reduced to the absolute minimum needed for synchronisation. This can only be done when the whole track is written as one using a special command.

A lot can be done, isn't it? But all of these have their issues. And all of them are about timing - or better timing differences between writing and reading when using different drives. We tend to forget that a floppy recording is not only about encoding data into a time variant signal, but that this signal can be read back at a different timing.

Or in simple words, the drive writing a disk may have had a different rotational speed than the drive a disk is later read and rewritten.

And believe me, there was a lot of variation, especially early on. An SA-400, the grand daddy of all 5.25 drives, was considered in spec when rotation was +/-10%. This means no format should use more than 90% of a track, otherwise the recording may not be compatible with other drives. Since the speed variation does as well influence the space one had to provide for each sector - at least not if a faster drive should be able to rewrite a sector on a disk formatted by a slower drive.

As a result, basic formats usually stayed below 80 % of a track.

(Following all calculation will be done using the basic 5.25 inch, 40 track, MFM drive to keep it simple - application to later 1.44 follows the very same rules, just more tracks and sectors)

#1 is what IBM already did with the PC going from 320 to 360 KiB per disk by writing 9 sectors instead of 8. The original 320 KiB format was developed with a really high security margins, so it may work with the worst drives imaginable (+/- 10%). By the time the IBM-PC was created (1981) Speed variation was down to +/- 4.5%. Together with the previous margin this was enough to squeeze in another sector without increasing risk of overwrite.

Of course, this only works once as a general improvement. Then again, if one had a drive toward the fast side and no need for data exchange another (10th) sector could sometimes be fitted. This was quite common on machines like the Atari ST. Still fully DOS compatible, as DOS takes the number of sectors per track from a disks boot sector, but not always readable on all machines.

#2A is another easy bet, benefiting from reduced sector overhead. In fact it also played a role why IBM's 360 KiB format worked out fine. Standard MFM format is designed for 16 sectors of 256 bytes user data. By using 512 byte sectors IBM already halved the sector overhead.

One can increase that further. By using 1024 byte sectors instead of 512 byte on an IBM 5.25 drive would allow 5 sectors per track, holding 5 KiB instead of 4.5 KiB per track or 400 KiB per Disk. Using three 2048 byte sectors instead would yield 6 KiB but is too close to maximum track capacity to work reliable across all machines and for data exchange.

Increasing sector size would as well work fine with all DOS and all compatible OS as sector size is recorded on disk and blocking/deblocking is part of the DOS spec. Well in theory. Any practical use would be a test how well drivers implement this.

#2B sounds like a great way to solve this by closing in to maximum capacity. In case of an MFM drive a combination of one 4096 , one 1024 and a 512 Byte sector would add up to 5.5 Kib per track or 440 KiB 40 track 5.25 Disk.

But there's a caveat - DOS does not support variable sector size. THere is only a single variable for sector size and only that is used by DOS to calculate sector numbers for access. So any format using this would need very specific drivers, showing DOS a same size organization and do all conversion on it's own. Not exactly portable.

#2C would mean creating one sector with a size close to track capacity. With a brutto track capacity of about 6250 byte and some security margin this ends at a sector size of ~5.8 KiB. Cool. Except it's even worse than #2B, as DOS does not support odd sector sizes.

Finally:

#3 now tackles the GAPs directly. A 512 byte sector including headers and gaps takes 658 byes on disk. Of those 106 are gap bytes. Another 330 are within the track header/trailer, giving a total of 1284 gap bytes. Even with keeping a fully compatible 512 byte sector size, one could put 11 sectors for the same 5.5 KiB per track (440 KiB per Disk) as in #2B. All readable by standard read command from every standard drive out there. This works as we haven't touches any sync bytes or headers.

(For a 1.44 MB Disk this is means an increase to 1.76 MB)

The only, but critical issue will be writing. When using a standard write command, the controller will seek as with reading but then write including all gaps, which will always overwrite the header of a following sector, thus destroying the disk. Writing a sector can only happen reading all sectors of that track, repackaging them in the condensed format, inserting the one to be written and then write the track at once. Not anything one can expect from DOS or any standard floppy driver.

Oh and there's another caveat: The write track command can not write data bytes containing the value of F5 or F6 - so this would need another hack to be discovered :)))

Long story short,t such a disk would be great to distribute software or do data backup, but not as a regular rewriteable exchange medium.

Pick your solution.