What is between the sectors of floppy disks?

Question

I've found this page which has an informative description of how sectors were encoded on floppy disks: http://nerdlypleasures.blogspot.com.au/2015/11/ibm-pc-floppy-disks-deeper-look-at-disk.html

Do the gap bytes span the entire space between sectors? Or are there sections between the sectors where "nothing" is written? I'm asking about regular formats, not exotic ones where sectors overlap. I.e, after the data CRC, do the gap bytes start immediately? The article says

Sometimes there may be some junk bytes between the CRC value and the next sequence of Gap bytes.

What about the other times? Can there be "nothing" or will it always be filled with data, even if it's not normally accessible? The article also says

I have never come across a disk that has had more than 10 x 512 byte sectors.

This means that a track must be able to hold at least 5320 bytes, assuming a single gap byte and a single 00 sync byte for each sector ID and data field. The 9-sector format as shown uses 5274 bytes per track, so there is room for at least 46 bytes more!

Answer 1

This calls for a lengthy Floppies 101.

For one, never assume a floppy as something with an inherently byte orientated structure. There is none. Just magnetic reversing areas. It's more like a magnetic tape (looped at the end). All orientation on this track is timing based. There are no cell markers or position markers or whatsoever. Well, there's a single marker that shows up once per revolution. It can be used to count them and maybe to start something (like a track), but it's not really needed and some quite low level implementations ignored it, so let's also ignore it (*1).

The disk is moved by mechanical means at a variable and less than perfectly controlled speed. This means that while drives may have a nominal revolution of 300 rpm, individual drives can rotate faster or slower, while at the same time a drive may also jitter during one round. It's not really favourable when doing timing based operations on the track.

To add complexity, it should be possible not only to read a single sector, but also to write just a sector, not always a whole track. To provide a reliable data storage media offering this and being also exchangeable over different machines, compensations for these effects are needed, thus two provisions are made:

1. Sync bytes to locate headers and data sectors
2. Three types of gaps to cover jitter and speed differences:
3. A gap between the header and the data sector
4. A gap between the data sector end and the next header
5. A gap between the last data sector and the first header

All together it's a form of prayer that everything will go well at the end.

The structure is not god or manufacturer given, but written on the disk when formatting. Formatting is a process where the whole track is written at once, including all headers, (dummy) sectors and gaps since there is no physical structure to check and format against. So the first prayer goes that total time needed to write the formatted track is shorter than a revolution. Otherwise it would overwrite its own beginning. How much shorter is based on the amount of speed spread the format developer has to expect from drives. Lets say that the drives are accurate at +/-3%, resulting in a maximum usable length of 97% of a track - in case the machine formatting got any of the unlucky 309 rpm drives.

The timing of one real revolution defines the maximum usable length of a track.

Formatting includes (writes) all gaps on the track. After that, these bytes aren't read or used again. They are exactly what the name suggests, a void between used sniplets. Their purpose is to allow timing variations and variable setup times for various stages of reading and writing. Their content is irrelevant.

During normal operations (i.e. not formatting), when a specific sector is needed, the controller looks out for sync bytes. They tell the speed a block (header or data) was written, so the controller can pick that speed up and use it to read the following bytes. After all, it could have been written on a different machine and/or a drive adjusted to a different speed (rpm). After synchronisation the following data can be read. Usually first looking for headers and checking them, until the desired address is found. On a read operation, the process gets repeated (waiting for sync and syncing) then the following block gets read and done. Gaps (point 2.1.) are completely ignored and not really needed - except for providing time to switch modes.

Now, when writing the picture changes. Here also the header is searched and compared. After that the controller switches for writing and starts to output the new sector data on his own timing. This includes sync bytes, data mark, data and CRC but not any gap bytes. The gap bytes between header and data have no real meaning at this stage, except, they should occupy the time span that the controller needs to switch from header check to writing.

Although during reading synchronisation was kept by the data encoded, in writing synchronisation comes from the controller. This means a new sector can be longer or shorter than the existing one - again depending on drive adjustment (and controller). If it's shorter fine, but when longer it needs more space - that's what the gap after a data sector (b.2) is meant for. With 3% drive variation and 512 Byte sectors formatting should have added some 15 gap bytes (these numbers are made up, check actual data sheets for real values) to make writing reliable.

Now your questions:

Do the gap bytes span the entire space between sectors?

No. The gap spans the space between data blocks (header or sector). There are no bytes, just rubbish.

Or are there sections between the sectors where "nothing" is written?

In the gaps and at the 'end'.

after the data CRC, do the gap bytes start immediately?

After formatting yes, later on hopefully.

Sometimes there may be some junk bytes between the CRC value and the next sequence of Gap bytes. What about the other times? Can there be "nothing" or will it always be filled with data, even if it's not normally accessible? The article also says

Gaps are gaps. Whatever is in the gap doesn't matter. Maybe it's remaining of previous sectors, or alien messages. Looking in there is as useful as playing records backward.

I have never come across a disk that has had more than 10 x 512 byte sectors. This means that a track must be able to hold at least 5320 bytes, assuming a single gap byte and a single 00 sync byte for each sector ID and data field. The 9-sector format as shown uses 5274 bytes per track, so there is room for at least 46 bytes more!

Repeat after me: there are no byte cells on floppies.

It's all about timing. On a physical level the maximum is defined by the flux changes the magnetic material can handle. Usually something like 6000 bpi. It's a property tied to length, not timing. So the outer tracks could hold more than the inner. But (normal PC) controllers handle disks at constant speed and constant data rate. For 5.25" MFM disks this is 250 kbps (=(decimal) kilobits per second). As the drive rotates at 300 rpm, or 5 rps (300 / 60), a 250 kbps data rate, the FDC can record exactly 50 (250 / 5) kbit (kilobits), or 6250 bytes, on a single track.

6250 bytes per track is the (rough) upper limit for fixed speed MFM 5.25" drives. The controller can't supply more and the media can't take more. With your kind of calculation this would offer not just 46 Bytes, but a whole whopping kilobyte :)) Well, of course, ignoring that there is also a track header which takes about 100 bytes off.

(A classic example of the need for gaps: I remember problems of PCs with a uPD765 controller when reading disks formatted by a WD1772 controller - the WD was quite fast to start writing after the track hole was detected, while some 765 implementation took time to think about before reading the first bytes, which would be the track block. Now, no hassles, as every sector also provides the track number, right? No, after track change the 765 insists on looking for the track header, and every time the index hole came, he went blind for a moment. But enough of wartime stories.)

The little problem here is that your calculation is based on dropping all gaps and syncs which would make the media unreadable. Again, A floppy isn't some certified stream in byte order where a single byte somewhere will do the trick. The gaps are needed to cover timing variations and setup times (otherwise they could be dropped entirely). And one sync isn't a sync at all, just another gap. Controller PLLs need maybe several byte times to sync reliably, also a sync byte may get damaged (due to writing), or reading may just start half a bit time too late. Also, you may need to look again at the sync pattern. They are not just zeros. There should be a byte called A1, that's the real sync mark, as it's not A1, but a modified 12 bit pattern that would be an A1 if there wouldn't be a 'clocking error' inside. Thus triggering the PLL. You'll need at least 3 of them - plus a bunch of zeros to lead in - to be sure synchronization happens.

But long story short: yes, you can cram more into a track if you're sure about the timing of your drives and your machine. Up to 11 512 Byte sectors have been found to work acceptably on standard MFM disk drives manufactured from the late '80s onwards.

If you really want to dig deeper, there's an awesome book called Scheibenkleister (*2) written about storage media for the Atari ST. And yes, it's a great read for PC users, as the basic floppy hardware of the original ST is just the same as on the PC, while Atari Formats are also just PC Disks. A disk was supplied with the book offering a 'Hyperformat' program allowing to play with various kind of track types and track numbers. 12 sectors per track proved to be reliable for most machines. And once formatted on the ST, the PC could read and write them (see above). So get that book.

Here is also a nice page about the very basic hardware of floppy drives and controllers

There is also another (newer) question about the historic development of floppies vs. disks worth reading next - not to mention many other great ones carrying the floppy tag :)

---

*1 - As cmm reminds in a comment, there are also hard-sectored disks. While this sounds like fixed cell, it's not really a property of the track, nor does it change the timing-based nature of recording itself. The benefit of sector markers is a much simpler electronic circuit. Keep in mind, floppies were developed in the late 1960s when adding several hundred gates was a big deal - as was saving a single one by using analogue tricks. Having hardware that reads and decodes headers on the fly, to find the right sector to read or write, would have been ridiculously complex and expensive. Something maybe appropriate for expensive disks, but for sure not for a device developed as low cost (and low speed) storage.

Having sector holes now allowed to have simple, 5 bit wide counter and comparator logic to 'find' a sector for reading or writing. The counter was (prepared to) reset whenever two pulses came in shorter than expected time (the index hole was placed between the holes for the last and first sector, so only one detector was needed) and incremented with every following pulse. Whenever the value was the same as looked for, comparator issued a pulse and the controller read whatever was coming along (and stopped by the next pulse in case of no data), or blurt out its data. When writing, no checks were made; when reading, sector headers were only checked after having read the whole sector to double check if it's the right track and sector.

Long story short, a hard-sectored disk can be read without looking for index and sector markers, as everything is on the track - it only needs a logic fast enough to make the decision to keep a sector when track/sector number is the one it's looking for. Similar writing can be done by using appropriate timers.

*2 - There have been English (and other language) translations, just don't ask me, I searched the net, but I couldn't find any hint.

Answer 2

There can be values between the sectors but you probably don't want them.

The problem affecting disks is that they don't quite spin at a constant rate. Motor fluctuations, aerodynamics, friction from the head, etc, together give a few percentage points of variability.

So reading disks means applying a phase-locked loop to try to recover the clock rate and therefore the data. And generally speaking, that's what the stuff between bits of data is: 4e or FF bytes to allow the PLL to settle and then 00s to allow proper framing.

When you first format a disk, you'll establish the initial ID marks and data bodies. Later you might want to rewrite the data. But you'll almost certainly be writing at a slightly different rate. So the controller will spot the existing ID mark, then write a new data section after it. It may well produce a fairly dirty slicing-in of the new data, and write it at a slightly different rate. So the stuff between the ID field and the data mark gives the PLL time to recover from a potential abrupt phase change, into a slightly different clock.

Then you might be spinning the disk slightly more quickly than the original drive. So your data rate will look lower. So the new data you write to replace the old might stretch into the gap you left after the data and before the next ID. But it can't go so far as not to give the PLL enough time to adjust to another splice and rate change.

There are plenty of machines with contemporaneous controllers that can see a lot of the stuff between sectors — anything with a WD1770 can do an okay job*, the Amiga can do so perfectly — but the stuff is there for the purpose of reliable operation.

Also, the archetypal size for an MFM track is usually given as 6250 bytes. But it'll depend on how your drive feels that day.

*) MFM sync words can arise naturally in ordinary data, so it can easily lose framing if your data happens to contain certain values.

Expanded detail on that claim, it having been queried:

One defined MFM sync word is the data value C2. Normally encoded that should be 0101 0010 **1**010 0100 but, as is normal for sync words, a clock bit is omitted to make 0101 0010 **0**010 0100.

Unfortunately, look at what happens if you validly encode the data value 000101001:

1. 0 -> [clock] 0
2. 0 -> 10
3. 0 -> 10
4. 1 -> 01
5. 0 -> 00
6. 1 -> 01
7. 0 -> 00
8. 0 -> 10
9. 1 -> 0 [data]

Putting it all together: 0101 0010 0010 0100. So what looks exactly like the C2 sync word occurs anywhere your actual disk data contains the nine bits 000101001. When the WD is performing its read track, it has its sync detector on and adjusts its framing accordingly. So if you have that sequence it will adjust its framing and definitely be wrong as it will have adjusted so that clock fluctuations are in the position that data fluctuations should be.

EDIT: external sources mentioning the 0x5224 sync word: Atari ST FD Hardware, the FDI disk image file format specification, the UAE source code, the MAME documentation. External sources mentioning its ambiguity: "All you always wanted to know about the WD1772" (an Atari forum post), Atari Floppy Disk Copy Protection. It crops up a lot in ST literature because that was a machine with sufficiently-fixed hardware for real hit-the-metal protection schemes, it uses the WD which can make framing errors during a read-whole-track command due to the ambiguity, and it was a popular machine which always had a disk drive, so that was the only avenue for software distribution and piracy was a significant risk.

Answer 3

Profit for vendors of preformatted disks :)

IIRC, many such disks failed when you actually reformatted them (certainly more often than those originally sold unformatted), making it appear like selling them preformatted actually was a method of reusing partially defective media respectively achieving higher production yields.