14

The standard IBM System 34 floppy format supports two different types of sectors, "data" and "deleted data". In the sector header, data sectors are marked by an FB byte (Data Mark aka Data Address Mark or DAM), deleted data sectors by an F8 byte (Deleted Data Mark aka Deleted Data Address Mark or DDAM).

Many floppy disk controllers have separate commands for dealing with the two different sector types. The NEC uPD765 (used in the original IBM PC, and also some other machines such as the Amstrad CPC) and its compatible successors (Intel 82072, etc) have READ DATA/WRITE DATA commands to read/write data sectors, and READ DELETED DATA/WRITE DELETED DATA commands to read/write deleted data sectors.

My question is–did any software actually use "deleted sectors"? What did they use them for?

OpenVMS has an API for directly writing floppy disk sectors (IO$_WRITEPBLK) which accepts a modifier flag (IO$M_DELDATA) to cause it to write a deleted sector instead of a normal sector, but that still doesn't make clear whether any software actually used that API and what it used it for.

I've heard some suggestions they might have been used to mark bad sectors (although I don't think that was the original intention behind them). Minicomputer and mainframe systems (with record-oriented file systems) sometimes support marking individual records within a file as "deleted", much like deleting a row from a database. I suspect that was what IBM was thinking when they introduced this feature, but I'm sceptical it ever saw much use for its intended purpose.

I've also heard suggestions it has been used for copy protection. One source confirms it was used as part of the disk copy protection scheme for several BBC micro games. Likely it was also used for that purpose on other platforms, but would appreciate details.

One other use I've discovered is that LS-DOS (for the TRS-80) marked all the sectors on the directory cylinder as "deleted data". Possibly this was done to discourage user software from modifying it; I suppose it also could have played a role in locating the directory cylinder (if it was in a non-standard location) although I don't know if it was ever actually used in that way.

5
  • Seems like such a thing is only necessary if you lack a file system to tell you where the file data are. Maybe some sort of "mag tape replacement" use-case?
    – dave
    Commented Jul 8, 2023 at 23:36
  • As the format originates from IBM, my guess would be that only some IBM software did this, so that's where I'd look (so maybe Raffzahn knows). None of the mini, micro or home computer systems I know used this at all.
    – dirkt
    Commented Jul 9, 2023 at 5:16
  • 2
    If any kind of obscure floppy controller feature can be used for copy protection, there will be an 1980s IBM PC application that does use that feature. Commented Jul 9, 2023 at 8:00
  • @MichaelKarcher Thanks. I edited to add a source I found mentioning its use for copy protection on BBC micros. You are right that it was probably used as part of IBM PC copy protection schemes too, but would be nice if someone could cite specific examples Commented Jul 9, 2023 at 8:44
  • This seems like something that virus authors would have found tempting...? Commented Jul 11, 2023 at 3:40

2 Answers 2

6

Did any software use "deleted data" floppy sectors?

Yes. A lot.

... but it's complex, not at least as this isn't a software and especially not an application side issue but hardware and hardware-like as well.

The standard IBM System 34 floppy format supports two different types of sectors, "data" and "deleted data".

That definition is way older and came already with the basic 33FD format of 1973 (*1) used with the 3740 Data Entry System. What System /34 introduced were 53FD drives using DS DD (MFM).

So, yeah, lets go back way, way, way back, right to the creation of the world ... err, the floppy as storage media (*1).

In the sector header, [..] deleted data sectors by an F8 byte (Deleted Data Mark aka Deleted Data Address Mark or DDAM).

Sounds like wording from a certain PoV (*2). In basic floppy documentation this is called a Control Mark (CM), as it marked the sector as one containing control information. The 765 bit recording detection of an F8 address mark (ST2/D6) is accordingly named Control Mark. It's the basic way of Out-Of-Band-Signalling (*3): Sectors with FB build the data stream, that is all data content is user data, while F8 sectors hold meta information about how to handle/find the data stream.

What type of Control Field the data of that sector represented was encoded in the first data byte(s). IIRC there were several different use cases, all marked by

  • Deletion - letter D (x'C4' (*4))

    At least with 3740 compatible handling the remaining bytes were unchanged, allowing to 'undelete' a record later on (*5).

  • Unused Label - D as with deleted

    The first cylinder of a floppy is reserved for IPL and label (*6) information All sectors not used for label

  • Inserted (but not used) - D followed by Backslash \ and Spaces (x'C4E04040...')

    Some 374x allowed not only add, edit and deletion of records but as well inserting. This was done moving up all sectors by the number of sectors to be inserted and filling all 'freed' ones with D\ and spaces.

  • Overflow - D as with deleted

    An overflow control fiels told the controller to look for the next sector using the overflow area.... at that point memory gets blurry - could be that this wasn't with the 3740, but somewhat later)

  • Sequential Bad Sector - F (xC6)

    Mark single bad sectors. The following sector will contain the data.

  • Relocated Bad Sector - . (x'4B')

    Marks a bad sector, the replacement is to be found by looking track/sector number up in the bad sector list on Track 0. Usually the last two tracks of a disk were reserved for relocation.

  • Bad Cylinder - Done by ID field

    If a wohle cylinder is to be marked as bad all it's ID fields will contain the number FF.

The last isn't about a CM, but part of the same mechanic, as all of that was made to have the disk controller act independent from the system (CPU) when it comes to reading a given sector. After all, it simplifies error handling a lot when standard cases are already done by the hardware. To understand this one must note that the 3740 station had two discrete (*7) processors:

  • The main processor handling input, editing, display, etc. and executing the record program (*8)
  • The floppy processor handling the drive.

And, depending on mode, it is the floppy processor that does handle most of the above during read independent of the main processor.

When in sequential read a CM is encountered, the first byte is checked as well. If it's a D or F, reading will continue with the next sector as if the last has not been seen at all. In all modes an F will result in a continued read with the next sector. Same it's a .. Here the bad block list will be read and the replacement sector will be fetched. Same and independent of AM, if an ID x'FF' is found the head is advanced by one track and then looking again.

All of that enables rather simple operation at the main processor side as the most common cases are handled invisible (*9). Only if he cares to see deleted records (D) must be told - which is only the case in some editing modes. Like restore or insert.

All of his was set in stone long before the first mini computer - and even less micro - or integrated floppy controller touched a floppy disk.

Everything that came after was either using that standard in full or part or misusing it to reach individual goals (like copy protection). Wich of course may be as colorful as the human mind is when trying to come up with new applications :))

Many floppy disk controllers have separate commands for dealing with the two different sector types. The NEC uPD765 [...] and its compatible successors (Intel 82072, etc)

If looking for original handling, one should best start with WD 177x/9x series, as NEC's uPD765 is a later development.

have READ DATA/WRITE DATA commands to read/write data sectors, and READ DELETED DATA/WRITE DELETED DATA commands to read/write deleted data sectors.

Well, where the WD 17xx offer only basic ability of address mark handling(detecting i, writing it), the NEC 765 does implement the whole continuous read/skip mechanic IBM had in mind for CM. With the 'normal' Read Data command and the SK-Bit (D5 in the first command word) set to 1 the 765 will behave quite like a 3740 in sequential read: A sector found with an CM will not return an error but be skipped and the next following sector delivered instead. Read Deleted Data inverts that behaviour. Handy, isn't it?

My question is–did any software actually use "deleted sectors"? What did they use them for?

Tons. Starting with any software on minis or micros that had to read IBM diskettes for data transfer. After all, connecting to big iron or (god help) trying to replace it won't work without handling the interfaces it's to be linked in.

OpenVMS has an API for [...it...] but that still doesn't make clear whether any software actually used that API and what it used it for.

See above. DEC was the new comer to the show - without handling existing interfaces it would be an island noone would ever buy (*10). In fact did DEC support 3740 as early as 1975 with their first Drive (RX01) for PDP-8 and PDP-11 systems using the RX8/RX11 controllers.

While DEC operating systems did not use a CM in regular operation, the controllers were specially fitted to detect and produce a CM. Standard 3740 having deleted records by default makes support for reading as well as writing mandatory for any kind of data transfer using diskettes to and from IBM mainframes.

The use of low priced diskettes instead of tapes for small to mid sized data sets opened new markets for DEC - many departments liked the idea of buying lower priced DEC systems instead of renting 374x while at the same time improving data entry by using the more capable way of programming.

Same goes for engineering departments in companies and universities now able to move their data sets direct to mainframe without the need of expensive communication lines or tapes.

I've heard some suggestions they might have been used to mark bad sectors (although I don't think that was the original intention behind them).

It has and it was. Not just on floppies but disk as well - that's where it originated.

Minicomputer and mainframe systems (with record-oriented file systems) sometimes support marking individual records within a file as "deleted", much like deleting a row from a database.

Record, not row. Think punch cards, not databases (*11).

I suspect that was what IBM was thinking when they introduced this feature, but I'm sceptical it ever saw much use for its intended purpose.

Don't tell millions of users of mainframes that did use those features direct (like with 3740) or without knowing.

I've also heard suggestions it has been used for copy protection.

I would classify that as a misuse(*12). It's based on the fact that 765 read command is usually issued without SKIP (SK=0) resulting in an abortion standard disk copy software does not handle, doing he desired protection. Likewise with SKIP (SK=1), that sector will be skipped and the next delivered instead, so 'automatically' corrupting the copy by leaving out the one sector.

One other use I've discovered is that LS-DOS (for the TRS-80) marked all the sectors on the directory cylinder as "deleted data". Possibly this was done to discourage user software from modifying it;

Sounds like a great way to protect unintended alteration. Kind of a read lock - and a fast detection of corruption.

I suppose it also could have played a role in locating the directory cylinder (if it was in a non-standard location) although I don't know if it was ever actually used in that way.

Rather not, as it would mean one had to read the whole disk to find them.


*1 - If not with the 1971 23FD, but I'm not sure and didn't find any 23FD record definition with that detail.

*2 - Or 'grown' usage as the 3741 and 3742 stations allowed programming to keep the data of a deleted record untouched. Only the address mark was changed to F8. Those disk were of course incompatible with basic 3740 or a direct attached unit.

*3 - In-Band-Signalling would be if control information is embedded within the user data stream - like using escape sequenced. In band signalling has the advantage of less prerequisites (i.e. media definition) but the disadvantage of more complex data handling.

*4 - All encoding is in EBCDIC. Those systems were at foremost meant to replace punch card stations for main frames, thus all markings in mainframe code. This stayed true even with later systems using ASCII for data. Another advantage of Out-Of-Band-Signalling: Control encoding is complete Independent of any user data.

*5 - The missing character had to be entered again. To turn this into and advantage experienced designers of data entry schemata usually made the first byte a type indicator of the record, so anyone editing had less of a problem to reconstruct that even a long time later.

*6 - Label is IBM terminology for any kind of descriptive data, like disk geometry, block lists, directories or alike.

*7 - Discrete as in independent as well as in build from discrete components, not microprocessors or alike. Remember, the 3740 was introduced and delivered January 1973.

*8 - Don't get me going on that. It was horrible and brilliant in one. horrible in how to encode it and what all was not possible, but brilliant in what could be done with that wired stuff in terms of data entry process.

*9 - Doesn't sound different from Hard disks we know, does it? And in fact, the whole basic operation is copied from disk handling before the FD was invented. So nothing new here :))

*10 - At least not the ones that controlled the budget - _'Is it IBM compatible' was a notorious question way before the PC came along. IMHO unconscious transfer of that deep seated preconception was what made the PC what it became.

*11 - Not to mention that simple measures always beat complex ones :))

*12 - Then again, one could argue it's a valid use of a control field supplying information important to handle the data stream of that game :))

4
  • 1
    Although Unix likes to view files as monolithic streams of bytes, and such a view facilitates data interchange, the ideal formats for archiving or transporting data are often not the ideal format for actively manipulating them.
    – supercat
    Commented Jul 9, 2023 at 15:42
  • archive.org/details/… uses the terminology "address mark" and "control address mark". It talks about control records exclusively as deleted though; it says a deleted record normally has a D in first position, but it still classifies a control record without an initial D as a "deleted record". However archive.org/details/bitsavers_ibm3741374May75_25913459/page/… does mention using CAM followed by EBCDIC F to represent what you call a "sequential bad sector", but it calls that a "Bad Spot" Commented Jul 12, 2023 at 8:38
  • Also, according to archive.org/details/… if System/32 encountered any first character other than EBCDIC D or F, it would set the "invalid control record status bit" and terminate the READ DISKETTE DATA operation. Whereas, a D or F control record would be skipped–suggesting the System/32 only supported D or F control records, unless software was written to handle them specially Commented Jul 12, 2023 at 8:55
  • @SimonKissane Keep in mind that the Floppy Reference Manual covers commonalities of various systems, not all build to the same features.
    – Raffzahn
    Commented Jul 12, 2023 at 12:11
1

Partial answer:

The IBM Diskette General Information Manual from Aug 1979 says:

Various systems have the ability to modify records or the location of records. These modifications are as follows:

  • Logically delete a record
  • Move a record from a defective sector to the next sequential sector
  • Move a record from a defective sector to an alternative sector

These modifications are made by changing the contents of the address marker AM2 and the first character of the data field that immediately follows AM2. When the first character of the data field changes, the data field changes to a control field that designates what type of modification is made.

Note: The address marker AM2 usually contains a hex FB. When any of the three modifications is necessary, AM2 is changed to hex F8. F8 alerts the device to check the first character of the next field.

That should answer the question "what did they use them for".

Unfortunately, it doesn't say which "systems", and it might be very well something in the controllers and not actual software, but since the IBM Floppy was officially released 1971, this should restrict the number of "systems" to look at. The ability to handle records inside blocks should further restrict it.

7
  • Well, no. For one, the Floppy was released in 1973 with the 3740 (and above cited manual already in 1977). By 1979 there were more than 100 different systems spread over all of IBMs families using them, many with specific enhancements. Plus considerable more systems outside the IBM Product range. Think Wang, DEC, etc, whoever wanted to have their share of business computing and connect/replace IBM hardware.
    – Raffzahn
    Commented Jul 9, 2023 at 13:08
  • Sorry to be nitpicking, but there was no release in 1971. Neither as product nor in parts to the public. It was a strictly internal part of the 2835 disk system. Only with the 33FD a product was created in 1973 - and only he diskettes available as one. Compatible drives only came laterThe only floppy drive available before that was the Memorex 650 except that it was incompatible. Compatible drives emerged in 1974 with Shugarts SA800. It's compatibility with the 33FD is quite visible in CP/M's 128 byte record size :)
    – Raffzahn
    Commented Jul 9, 2023 at 14:02
  • Also, "records inside of blocks" wasn't anything the 3740 and compatible formats did. A sector was a record. Originally restricted to 128 bytes.
    – Raffzahn
    Commented Jul 9, 2023 at 14:03
  • @Raffzahn This here says "when introduced in 1971, the floppy...". It's from IBMs website. Maybe you are right and they are wrong, I don't know. I just copied it. As for competitors reading IBM floppies, I wouldn't be surprised if they just ignored the more complicated stuff (or provided hardware support, and then said "somebody needs to write software for it", and that got never written). That's how it usually goes". If you have an example of a system where you can show deleted sectors worked, all the better.
    – dirkt
    Commented Jul 9, 2023 at 17:45
  • Well, those pages got a lot of information, but are not free of simplification. The 23FD was only available as part systems and neither intended nor used for any data storage or exchange. By all logic any compatible could only emerge after the 2740 introduced the 33FD. Beside that the usage is with the basic definition and supported by all controllers, I'd say the OpenVMS cited in the question makes a great example, doesn't it. Likewise the very document you link shows two pages (p.22/23) of systems supporting those. A peek into the 3740 manual will not only show usage but as well handling.
    – Raffzahn
    Commented Jul 9, 2023 at 19:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .