Understanding the DMK disk image file format used by TRS-80 emulators

Question

DMK is a low-level disk image file format used by TRS-80 Model I & Model III, Tandy Color Computer, Dragon 32 & 64, and MSX emulators. It's capable of representing copy-protected disks and all "WD-1771 and WD-19xx controller functions and formats". So far I'm only digging into the TRS-80 Model I/III side of things.

The DMK format seems to be a bit vague to me probably due to assuming a relatively deep level of knowledge about how the disk format and drive controller work at a low level. But maybe I'm just missing something.

The format starts with a 16-byte header that describes the disk geometry. Then there is the first 128-byte track header section containing a list of 16-bit "offsets" to each "IDAM" in the track. An IDAM is an "ID Address Mark". The two high bits of an offset are flags.

After this section the doc says:

The actual track data follows the header and can be viewed with a hex editor showing the raw data on the track. If the virtual disk doesn't have bits 6 or 7 set of byte 4 of the disk header then each single density data byte is written twice, this includes IDAMs and CRCs (the CRCs are calculated as if only 1 byte was written however). The IDAM and sector data each have CRCs, this is just like on a real disk.

I guess "raw data" is the key. Because the actual sector data starts at offset 0x0108 of the DMK file, which leaves 120 bytes between the IDAM table and the sector data, which must be internal/low-level "raw" data. But I can't find this part documented. How can I make sense of it?

UPDATE:

I'm even more confused. I found a tool for viewing DMK files that helped me realize a few more of my DMKs have this little directory track declaration, but their offsets occur at locations varying by a number of bytes from the start of the disk image. But it can't be very analog because the DMK files are still all the same size.

Answer 1

I support DMK in my MSX emulator, and bear in mind that it's a bit of a confused file format. It has a bunch of design deficiencies, and was clearly tightly coupled to the program that originally implemented it. But starting with the perfectly sensible stuff:

The first 16 bytes are the header, which you seem to be familiar with — write protection, geometry, track size, etc.

From there on are a bunch of track images. Each track image is the length you got from the header, and they're interleaved by side. So you can find track t on side s of a q-sided disk (i.e. 1-sided or 2-sided) with track lengths of n at file offset:

(t*q + s)*n + 16

i.e. immediately after the 16-byte header, on a double-sided disk, you'll see the track image for side 0, track 0. Then you'll see side 1, track 0. Then side 0, track 1; side 1, track 1; side 0, track 2, etc, etc.

The first 128 bytes of each track image are a list of IDAM pointers and single/double density flags. So the first 128 bytes of each track image aren't actually on the original floppy disk, they're just part of this file format.

As you note, an IDAM is an ID Address Mark. This relates to how floppy disks work — they're always spinning and a floppy controller just receives an incoming stream of data bits. Having stepped to the track it wants, to read a given sector the controller then has to sit and wait until that particular sector goes past.

How does it tell where the sectors begin? Through the presence of an IDAM. Standard IBM System 34 disk encoding as used by most computers is an ID Address Mark, then an address ending with a CRC, then a gap, then a data (or deleted data) mark, then sector data and another CRC, then another gap to the next IDAM.

This is where DMK starts to go off the rails a little:

Obviously repeating every byte twice for single-density disks is a complete fiction, likely invented so that the one-size-for-all-tracks model worked. When dealing with sectors marked as single-density, just discard every other byte from the original file.

On a real disk, address and data marks are specially-crafted so as to be unambiguous, by dropping a clock bit. This file format doesn't attempt to preserve that. Instead it tells you where the address marks are in a separate table. It doesn't tell you anything about where the data marks are. You have to make an educated guess.

The file format also records FM and MFM marks as looking the same. This is likely a misunderstanding of the author based on the control commands a WD1770 accepts. But:

• where an IDAM is recorded, you should see an 0xfe in the track image;
• there'll then be a standard address — one byte track address, one byte side number, one byte sector number, one byte length (log 2, minus seven*), then two bytes of CRC;
• some number of gap bytes will follow;
• eventually you'll see either a 0xfb or 0xf8. That's your data mark. 0xfb is regular data, 0xf8 is deleted data. I don't know about the specifics of the TRS-80 but it's very common not to use the deleted data mark as it's not that useful if you have an actual filing system;
• the contents of the sector are then the number of bytes after that corresponding to the sector size, followed by another two bytes for the CRC.

* i.e. if the byte value stored for sector length is m then the sector is 1 << (m + 7) bytes long. Or 128 << m if you prefer — if m is 0 then the sector is 128 bytes long, if m is 1 then the sector is 256 bytes long, etc.

So now you've got an algorithm for transcribing a DMK track back into an actual floppy disk:

1. skip to track location;
2. read IDAM table;
3. move to the end of the IDAM table;
4. output all bytes from there until the first indicated IDAM;
5. discard the byte at the IDAM and write an actual IDAM;
6. output the next six bytes, forming the address mark;
7. keep outputting whatever you find until you see a 0xfb or 0xf8. Remember to throw away every other byte if this is flagged as a single-density sector;
8. throw away the 0xfb or 0xf8 and write a data or deleted data mark;
9. output every following byte (or every other re: single density) for the declared length of the sector plus its CRC;
10. return to step 4, now with an eye on the next IDAM; or
11. if there is no next IDAM, just write out all remaining bytes until the end of the track.

Supposing you're interested in a tool just to extract a particular sector, you can jump to the appropriate track, check out the IDAM table, use it to look up all the address marks, check those addresses for the sector you're looking for and then search forward for the associated data if you find it.

That's why you're seeing stuff like directory contents at different locations in different DMKs. Within a track the sectors can come in any order. Even when they come in the same order, they can be rotated arbitrarily since this file format is not rigorous as to the location of the index hole. E.g. if a real floppy disk had sectors tagged as 1, 2, 3, 9, 5 then it is technically valid to encode that as a DMK as sectors in the order 1, 2, 3, 9, 5 or 2, 3, 9, 5, 1 or 3, 9, 5, 1, 2 or the other two rotations of that.

(Aside: it might be that the author intended the index hole always to be at the start of a track image, but the specification doesn't say that so your mileage may vary.)