Why does the FAT file system have separate ‘hidden’ and ‘system’ attributes?

Question

File systems used by DOS and Windows have used file attribute bits as a relatively prominent feature. The first of them, FAT, exposes four attributes to the user: read-only, archive, hidden and system. (There are a few more used internally, but they do not concern us here.) The function of ‘read-only’ is self-explanatory. ‘Archive’ is somewhat less obvious, but once you look it up, it also seems sensibly motivated: it denotes that the file has been modified since last backup. ‘Hidden’ makes the file not appear in normal directory listings and makes it harder to accidentally delete, as basic protection against inattentive or perhaps too adventurous users, while ‘system’… does the same.

For as long as I can remember, applying the ‘hidden’ and ‘system’ attributes has had virtually the same effect, and they were usually used together, often in conjunction with the ‘read-only’ attribute: most notably, for the DOS kernel files IO.SYS and MSDOS.SYS. As such, the attributes seem pretty much redundant to each other; although more recent releases of Windows seem to have started subtly differentiating between them, with ‘system’ acting as a ‘super hidden’ bit of sorts, whose hidden-in-listings effect has to be separately disabled, and which sometimes triggers strongly-worded discouragements when the user attempts to delete the file. But this doesn’t seem to be what motivated the addition of the ‘system’ bit in the first place, back in MS-DOS 2.0 (if not earlier). So what was the real motivation?

Answer 1

The attributes are documented in The MS-DOS Encyclopedia as follows:

The hidden bit (bit 1) is set to 1 to indicate that the entry is to be skipped in normal directory searches — that is, in directory searches that do not specifically request that hidden entries be included in the search. The system bit (bit 2) is set to 1 to indicate that the entry refers to a file used by the operating system. Like the hidden bit, the system bit excludes a directory entry from normal directory searches.

In practice their behaviour is the same, but their definitions aren’t redundant: a system file is a hidden file used by the operating system, whereas a hidden file is any file which shouldn’t appear in a regular directory listing.

Hidden files were added in version 1.10 (in Microsoft versioning; the version of DOS actually available as 1.10 was 1.25); bits 1 or 2 could be used to hide a file, and the source code comments didn’t distinguish them.

Answer 2

Although MS-DOS didn't come bundled with utilities to change the flags, it was useful to distinguish files that could be casually hidden and unhidden from those which shouldn't be. While I doubt this was the original intention, the ability to temporarily "hide" files among other things made it possible to exclude files from wildcard operations, and PC magazine published a utility called NO.COM for that purpose. The basic notion is that if one wanted to copy everything other than woozle.dat and foozle.dat from one disk to another, one could hide woozle.dat and foozle.dat, then copy *.*, and then unhide everything.

The usefulness of the system flag was further enhanced with disk defragmenting utilities came on the scene. Some files, including the boot files that were first marked by the "system" attribute, need to be located at certain locations on the disk in order to serve their intended purpose. Disk defragmenting utilities would generally use the "system" attribute as an indication that they shouldn't do anything at all with a particular file.

I don't know whether the designers of these attributes had such abilities in mind, but the ability to prevent the storage associated with files from being touched can serve a number of purposes:

1. It allows a simple disk boot sector to fetch boot files from a fixed location, without having to decode directory or FAT structures.

2. If parts of a disk will hold data in MS-DOS format, but parts will hold data in some other format, having the MS-DOS part of the disk contain a "system file" which is marked as using up all of the space that is used by the other disk format will let MS-DOS know that it should leave that space alone, without MS-DOS having to know or care what it's actually used for.

3. If parts of a disk develop errors, creating system files containing the associated blocks will prevent the system from trying to allocate those blocks to any other purpose.

4. Copy-protection schemes would often randomly select a few unused blocks on a hard drive, create a system file which occupies those blocks, encrypt a list of the involved blocks somehow, and store that list in the system file. This would ensure that anyone wanting to copy the associated program would need to arrange to place the same system file contents in the same locations on the target drive, something which today would seem trivial but at the time was a moderately effective deterrent to piracy.

None of those purposes would have been served effectively without a system/hidden distinction.

Answer 3

Like much other details the attributes are in part inherited from CP/M. CP/M stored them in the top bit of the file name and type (*1). This was possible as file names were defined as 7 bit ASCII, so the 8th bit was 'free' to be used (*2). In Detail these were:

• T1' (*3) Read-Only
• T2' System File
• T3' Archived

Well, or not. While it's common knowledge that PC-DOS is based on CP/M, this is not entirely true, as it's file system is based on the 8 bit FAT used with Microsoft's Stand Alone BASIC-80, which Patterson adapted for SCP's 8086 S100 board before creating 86-DOS which later became PC-DOS. BASIC-80's directory was based like CP/M on 16 byte entries, but with a different layout:

(Taken from the BASIC-80 5.0 Manual)

It already featured an attribute byte, but except for Read-Only (10h) none match any CP/M attributes. What Patterson did was increasing the file name length to 11 bytes for CP/M compatibility. It is not clear what attributes were included.

For 86-DOS the introduction of 32 byte directory entries is documented with version 0.42 of February 1981 as seen in the MS-DOS.ASM file. This was done on request by IBM to incorporate a file creation and last changed date as well as increased file size(*5). It is not clear how much Marc McDonald's MIDAS OS, which as well used 32 byte entries, influenced this. What's clear is that the 12 and 16 bit FAT structure was taken from there.

In addition, the very first known pre-release of PC-DOS, which may match 86-DOS 1.10 of July 1981, notes in point 2 of it's 'COMMENTS.TXT' file the introduction of 'a new attribute byte' whose flags prevent system files from being shown.

The source presented on Github as MS-DOS 1.25 may neither be MS-DOS nor PC-DOS but rather some version of 86-DOS from after MS took the source (Q-DOS 1.10) and independent of Microsoft's development. It's missing several functions, including GETRDONLY and SETATTRIB being a null function.

Comments in MS-DOS.ASM mention that hidden and system, each on its own, would make a file not being displayed. No other attributes are mentioned.

---

Considering all of this it seems plausible that IBM requested that flags from Microsoft to be able to handle system files and hidden separate, even though it was later only sparsely used.

---

*1 - More exact, CP/M 2.2 only used the 3 flags in the file type, it was up to MP/M-II and later CP/M 3.0 to also use the file name.

*2 - Of course file search operations had to mask them away before comparing.

*3 - CP/M developed it#s own notation here:

• T meaning file type byte,
• 1 meaning first byte (*4), and
• ' meaning the top bit.

*4 - Yes, numbering in original DR documentation was 1..3, not 0..2 as in some later cases.

*5 - 8 Bit FAT only provided a two byte length field, good for up to 64 Ki - 1 bytes.

Answer 4

I think to explain these features, you'd have to acknowledge that a common distinction exists between some sort of "system administrator" (someone generally responsible for the provision and oversight of the system) and a "user" (someone who uses the system without being an expert about all its internal details or workings).

When it comes to computer file systems, the space available has typically been shared between files that are created by the system administrator (i.e. Microsoft, or various other vendors) to support the operation of the system itself, and files that are created and modified by users.

The system flag is amongst the provisions made for distinguishing between system and user files. Other methods exist alongside nowadays, including special directories (such as the "Windows" folder).

How exactly these flags are used, and whether the system flag has any effect separate from the hidden flag, is probably something that has been subject to varying practices over the years.

There was probably not intended to be only one way in which these flags could ever be used - rather, they were provided to create flexibility for making distinctions between files, on an exact criteria that may only be decided later.

As others indicate, the meaning of "hidden" is not necessarily "to exclude from the view of the browsing user", but also "to exclude from the attention of various programs which operate on sets of files".

The "system" marking is completely orthogonal to that definition of "hidden", notwithstanding that later file browsers hid both from the user by default.

Once point-and-click GUIs became available for file management, the role of the hidden flag as being a mark against a file that alters its handling by programs that process multiple files, probably receded, to be replaced by selections that could be made more dynamically by the user in the GUI.

The use of the system flag to mark system files amongst a mixed collection has also receded, with directories more consistently used to keep the two kinds apart. When these flags were first designed however, additional directories meant more typing for commands and navigation.

What remains nowadays is more or less vestigial, and both hidden and system flags are used mainly to record that files should not be displayed to the browsing user by default, with little other practical function to distinguish the two.

Answer 5

I'm not particularly familiar with the actual history of how and when the "hidden" and "system" attributes were introduced (which fortunately has already been addressed in other answers), but to answer your question of why both of these attributes exist, I can certainly reconstruct a plausible chain of reasoning. It goes something like this:

• Problem 1: Since system files are stored in the same file system as user files, users can accidentally (or through ignorance of their purpose) delete, rename, overwrite or otherwise manipulate system files. This can cause the system to malfunction, annoying users and causing them to blame the system for being fragile and hard to use safely.

• Solution 1: Hide system files by default, so that users are not confused by their presence and cannot easily manipulate them by accident.

• Problem 2: Some users want to hide (and possibly later unhide) their own files too. This is now possible, but only by marking (and later unmarking) those files as system files using custom software tools. Users are now being trained and incentivized to treat the "system" file attribute as something they can toggle on and off at will, potentially causing problem 1 to re-emerge if users accidentally unhide actual system files or otherwise mix them up with their own files.

• Solution 2: Introduce a second attribute for marking user files as hidden and add built-in system support for toggling this attribute (and not the "system" file attribute) on and off. Users now have access to the "file hiding" feature without having to misuse the "system" file attribute for it, and are not tempted to install tools or learn habits that may cause them to accidentally unhide system files.

Notably, other operating systems such as Unix solved problem 1 in a different way, by having multiple user accounts and implementing file ownership and permissions so that only privileged administrator accounts (only intended to be accessible by a few properly trained users) can modify critical system files. Thus, they never had the same need to hide system files from accidental manipulation as single-user operating systems like MS-DOS did, and thus never ended up with these two file attributes.

While Unix did end up implementing its own way to hide user files from directory listings (by prefixing their name with a dot) and also using it to hide some user-specific system configuration files, the situation is still substantially different: even if a non-admin user on a Unix system accidentally deletes all their dotfiles, they'll still have a fully functional OS that will boot up and even let them log in and use the system, only losing a bunch of their personal settings (and of course possibly some data, depending on what else they deleted). While this may certainly be a frustrating situation for the user, the crucial difference is that the OS is still working, so the user is less likely to blame the OS just because it "suddenly broke for no reason."