26

File systems used by DOS and Windows have used file attribute bits as a relatively prominent feature. The first of them, FAT, exposes four attributes to the user: read-only, archive, hidden and system. (There are a few more used internally, but they do not concern us here.) The function of ‘read-only’ is self-explanatory. ‘Archive’ is somewhat less obvious, but once you look it up, it also seems sensibly motivated: it denotes that the file has been modified since last backup. ‘Hidden’ makes the file not appear in normal directory listings and makes it harder to accidentally delete, as basic protection against inattentive or perhaps too adventurous users, while ‘system’… does the same.

For as long as I can remember, applying the ‘hidden’ and ‘system’ attributes has had virtually the same effect, and they were usually used together, often in conjunction with the ‘read-only’ attribute: most notably, for the DOS kernel files IO.SYS and MSDOS.SYS. As such, the attributes seem pretty much redundant to each other; although more recent releases of Windows seem to have started subtly differentiating between them, with ‘system’ acting as a ‘super hidden’ bit of sorts, whose hidden-in-listings effect has to be separately disabled, and which sometimes triggers strongly-worded discouragements when the user attempts to delete the file. But this doesn’t seem to be what motivated the addition of the ‘system’ bit in the first place, back in MS-DOS 2.0 (if not earlier). So what was the real motivation?

8
  • 3
    For reference, the two other generally defined attributes in the attributes byte of the 86-DOS family are "Directory" and "Volume label". There is also "Part of an LFN chain" which is Volume label + Read only + Hidden + System.
    – ecm
    Dec 15, 2022 at 17:28
  • 3
    I remember also seeing ‘device’, used internally for character devices, but now that I think of it, it might be a FreeDOS-ism. Dec 15, 2022 at 17:34
  • 1
    The interrupt list supports your recall: "this call also returns successfully if given the name of a character device without wildcards. DOS 2.x returns attribute 00h, size 0, and the current date and time. DOS 3.0+ returns attribute 40h and the current date and time." Think there was also "Executable", that may have been a Netware specific use. List mentions this in fd.lod.bz/rbil/interrup/dos_kernel/214301.html#table-01420
    – ecm
    Dec 15, 2022 at 17:42
  • 1
    A common use for hidden files is temporary files that are not useful out-of-context, e.g. work files being used by a sort or intermediate files created by a multi-pass compiler. They are created and deleted during a run of the associated application. Whether or not that was the original motivation is up to a Magic 8 Ball.
    – HABO
    Dec 16, 2022 at 13:59
  • 2
    Isn't this just escalating conflict? Implement one bit for a specific use; 3rd-party programmers start using it for something else; implement another bit to distinguish the two cases. Did H and S appear at the same time or was one later? Dec 16, 2022 at 21:25

5 Answers 5

25

The attributes are documented in The MS-DOS Encyclopedia as follows:

The hidden bit (bit 1) is set to 1 to indicate that the entry is to be skipped in normal directory searches — that is, in directory searches that do not specifically request that hidden entries be included in the search. The system bit (bit 2) is set to 1 to indicate that the entry refers to a file used by the operating system. Like the hidden bit, the system bit excludes a directory entry from normal directory searches.

In practice their behaviour is the same, but their definitions aren’t redundant: a system file is a hidden file used by the operating system, whereas a hidden file is any file which shouldn’t appear in a regular directory listing.

Hidden files were added in version 1.10 (in Microsoft versioning; the version of DOS actually available as 1.10 was 1.25); bits 1 or 2 could be used to hide a file, and the source code comments didn’t distinguish them.

4
  • I'm guessing the intent of the System File bit was to allow Fat32 to, in the future, provide support for preventing "Delete System 32" problems. Dec 16, 2022 at 22:32
  • 6
    Would it be fair to say that a System file is hidden for System reasons, but a Hidden file is hidden for User reasons. Having two separate flags is useful to the user because they can hide and show files without accidentally messing up the system by resetting the System flag.
    – Ben
    Dec 18, 2022 at 0:35
  • 1
    @Ben, I think that's exactly the case. There's a vast difference between protecting OS files that affect every program and files that affect only the specific application. Well, given it's MS-DOS, for some very loose definition of "protection" :-)
    – paxdiablo
    Dec 18, 2022 at 23:47
  • 1
    @Ben that’s probably the general idea, yes; however it’s hard to know conclusively since the implementation doesn’t follow through. While DOS doesn’t have any protection, it does distinguish internal and public APIs; thus one could have imagined that the system attribute would only be modifiable through a DOS-private service, whereas the hidden attribute would be modifiable through the public APIs. Attributes as a whole weren’t really user-modifiable anyway when they were introduced, since ATTRIB was only added in DOS 3.0. Dec 19, 2022 at 12:10
20

Although MS-DOS didn't come bundled with utilities to change the flags, it was useful to distinguish files that could be casually hidden and unhidden from those which shouldn't be. While I doubt this was the original intention, the ability to temporarily "hide" files among other things made it possible to exclude files from wildcard operations, and PC magazine published a utility called NO.COM for that purpose. The basic notion is that if one wanted to copy everything other than woozle.dat and foozle.dat from one disk to another, one could hide woozle.dat and foozle.dat, then copy *.*, and then unhide everything.

The usefulness of the system flag was further enhanced with disk defragmenting utilities came on the scene. Some files, including the boot files that were first marked by the "system" attribute, need to be located at certain locations on the disk in order to serve their intended purpose. Disk defragmenting utilities would generally use the "system" attribute as an indication that they shouldn't do anything at all with a particular file.

I don't know whether the designers of these attributes had such abilities in mind, but the ability to prevent the storage associated with files from being touched can serve a number of purposes:

  1. It allows a simple disk boot sector to fetch boot files from a fixed location, without having to decode directory or FAT structures.

  2. If parts of a disk will hold data in MS-DOS format, but parts will hold data in some other format, having the MS-DOS part of the disk contain a "system file" which is marked as using up all of the space that is used by the other disk format will let MS-DOS know that it should leave that space alone, without MS-DOS having to know or care what it's actually used for.

  3. If parts of a disk develop errors, creating system files containing the associated blocks will prevent the system from trying to allocate those blocks to any other purpose.

  4. Copy-protection schemes would often randomly select a few unused blocks on a hard drive, create a system file which occupies those blocks, encrypt a list of the involved blocks somehow, and store that list in the system file. This would ensure that anyone wanting to copy the associated program would need to arrange to place the same system file contents in the same locations on the target drive, something which today would seem trivial but at the time was a moderately effective deterrent to piracy.

None of those purposes would have been served effectively without a system/hidden distinction.

7
  • 2
    "It allows a simple disk boot sector to fetch boot files from a fixed location, without having to decode directory or FAT structures." Note that all original MS-DOS boot sector loaders, including those of MSW 4, expect the first 1536 (MS-DOS v6 style) or 2048 (MS-DOS v7 style) bytes of the first loaded file (IO.SYS) to be contiguous from the first data cluster of that file. This is always true if the cluster size is at least 2 KiB but could be untrue otherwise, if the file also happens to get fragmented. Like your example this is to avoid having to access the FAT (which the next loader does).
    – ecm
    Dec 15, 2022 at 17:47
  • 3
    I think I noted that I specifically wanted to ask about the original reason for the attribute’s introduction, instead of uses that came after the fact (especially speculative ones). Although the defrag point seems valid. On the other hand, I have not heard of any copy protection scheme that did (4); as for (3), and even (2), a cluster can be marked as ‘bad’ in the FAT, in which case it will not be allocated to any file. I’m not sure this existed back in the times of MS-DOS 2.0, but the ‘system’ bit certainly wasn’t needed for the purpose. Dec 15, 2022 at 17:50
  • 3
    @AaronF ATTRIB was introduced with DOS 3.0, but it initially could only manipulate the read-only and archive attributes; starting with DOS 5.0 it could also manipulate the hidden and system attributes. Dec 16, 2022 at 9:41
  • 1
    @StephenKitt Now you mention it, I remember having to reformat disks with the /s switch in order to recreate the config.sys and io.sys with the system flags. So now I'm wondering how did I change the hidden attribute on v3.2... there must have been a 3rd-party utility like Norton Commander, or maybe I used a .com program typed in from a computer magazine using DEBUG (those always felt like casting a magic spell to my prepubescent brain)
    – Aaron F
    Dec 16, 2022 at 17:10
  • 1
    I think part of the reason behind separate flags is that back then not only did software often have to create temporary files on disk (to get round memory limitations - WordStar targeted 384K machines for a long time!) AND it had to run on floppy disks: there wasn't a central scratch space equivalent of /tmp. So the flags just mean what they appear to: system belong to the system, and hidden belong to the user.
    – Lou Knee
    Dec 17, 2022 at 13:19
12

Like much other details the attributes are in part inherited from CP/M. CP/M stored them in the top bit of the file name and type (*1). This was possible as file names were defined as 7 bit ASCII, so the 8th bit was 'free' to be used (*2). In Detail these were:

  • T1' (*3) Read-Only
  • T2' System File
  • T3' Archived

Well, or not. While it's common knowledge that PC-DOS is based on CP/M, this is not entirely true, as it's file system is based on the 8 bit FAT used with Microsoft's Stand Alone BASIC-80, which Patterson adapted for SCP's 8086 S100 board before creating 86-DOS which later became PC-DOS. BASIC-80's directory was based like CP/M on 16 byte entries, but with a different layout:

BASIC-80 Directory Format

(Taken from the BASIC-80 5.0 Manual)

It already featured an attribute byte, but except for Read-Only (10h) none match any CP/M attributes. What Patterson did was increasing the file name length to 11 bytes for CP/M compatibility. It is not clear what attributes were included.

For 86-DOS the introduction of 32 byte directory entries is documented with version 0.42 of February 1981 as seen in the MS-DOS.ASM file. This was done on request by IBM to incorporate a file creation and last changed date as well as increased file size(*5). It is not clear how much Marc McDonald's MIDAS OS, which as well used 32 byte entries, influenced this. What's clear is that the 12 and 16 bit FAT structure was taken from there.

In addition, the very first known pre-release of PC-DOS, which may match 86-DOS 1.10 of July 1981, notes in point 2 of it's 'COMMENTS.TXT' file the introduction of 'a new attribute byte' whose flags prevent system files from being shown.

The source presented on Github as MS-DOS 1.25 may neither be MS-DOS nor PC-DOS but rather some version of 86-DOS from after MS took the source (Q-DOS 1.10) and independent of Microsoft's development. It's missing several functions, including GETRDONLY and SETATTRIB being a null function.

Comments in MS-DOS.ASM mention that hidden and system, each on its own, would make a file not being displayed. No other attributes are mentioned.


Considering all of this it seems plausible that IBM requested that flags from Microsoft to be able to handle system files and hidden separate, even though it was later only sparsely used.


*1 - More exact, CP/M 2.2 only used the 3 flags in the file type, it was up to MP/M-II and later CP/M 3.0 to also use the file name.

*2 - Of course file search operations had to mask them away before comparing.

*3 - CP/M developed it#s own notation here:

  • T meaning file type byte,
  • 1 meaning first byte (*4), and
  • ' meaning the top bit.

*4 - Yes, numbering in original DR documentation was 1..3, not 0..2 as in some later cases.

*5 - 8 Bit FAT only provided a two byte length field, good for up to 64 Ki - 1 bytes.

2

I think to explain these features, you'd have to acknowledge that a common distinction exists between some sort of "system administrator" (someone generally responsible for the provision and oversight of the system) and a "user" (someone who uses the system without being an expert about all its internal details or workings).

When it comes to computer file systems, the space available has typically been shared between files that are created by the system administrator (i.e. Microsoft, or various other vendors) to support the operation of the system itself, and files that are created and modified by users.

The system flag is amongst the provisions made for distinguishing between system and user files. Other methods exist alongside nowadays, including special directories (such as the "Windows" folder).

How exactly these flags are used, and whether the system flag has any effect separate from the hidden flag, is probably something that has been subject to varying practices over the years.

There was probably not intended to be only one way in which these flags could ever be used - rather, they were provided to create flexibility for making distinctions between files, on an exact criteria that may only be decided later.

As others indicate, the meaning of "hidden" is not necessarily "to exclude from the view of the browsing user", but also "to exclude from the attention of various programs which operate on sets of files".

The "system" marking is completely orthogonal to that definition of "hidden", notwithstanding that later file browsers hid both from the user by default.

Once point-and-click GUIs became available for file management, the role of the hidden flag as being a mark against a file that alters its handling by programs that process multiple files, probably receded, to be replaced by selections that could be made more dynamically by the user in the GUI.

The use of the system flag to mark system files amongst a mixed collection has also receded, with directories more consistently used to keep the two kinds apart. When these flags were first designed however, additional directories meant more typing for commands and navigation.

What remains nowadays is more or less vestigial, and both hidden and system flags are used mainly to record that files should not be displayed to the browsing user by default, with little other practical function to distinguish the two.

0

I'm not particularly familiar with the actual history of how and when the "hidden" and "system" attributes were introduced (which fortunately has already been addressed in other answers), but to answer your question of why both of these attributes exist, I can certainly reconstruct a plausible chain of reasoning. It goes something like this:

  • Problem 1: Since system files are stored in the same file system as user files, users can accidentally (or through ignorance of their purpose) delete, rename, overwrite or otherwise manipulate system files. This can cause the system to malfunction, annoying users and causing them to blame the system for being fragile and hard to use safely.

  • Solution 1: Hide system files by default, so that users are not confused by their presence and cannot easily manipulate them by accident.

  • Problem 2: Some users want to hide (and possibly later unhide) their own files too. This is now possible, but only by marking (and later unmarking) those files as system files using custom software tools. Users are now being trained and incentivized to treat the "system" file attribute as something they can toggle on and off at will, potentially causing problem 1 to re-emerge if users accidentally unhide actual system files or otherwise mix them up with their own files.

  • Solution 2: Introduce a second attribute for marking user files as hidden and add built-in system support for toggling this attribute (and not the "system" file attribute) on and off. Users now have access to the "file hiding" feature without having to misuse the "system" file attribute for it, and are not tempted to install tools or learn habits that may cause them to accidentally unhide system files.

Notably, other operating systems such as Unix solved problem 1 in a different way, by having multiple user accounts and implementing file ownership and permissions so that only privileged administrator accounts (only intended to be accessible by a few properly trained users) can modify critical system files. Thus, they never had the same need to hide system files from accidental manipulation as single-user operating systems like MS-DOS did, and thus never ended up with these two file attributes.

While Unix did end up implementing its own way to hide user files from directory listings (by prefixing their name with a dot) and also using it to hide some user-specific system configuration files, the situation is still substantially different: even if a non-admin user on a Unix system accidentally deletes all their dotfiles, they'll still have a fully functional OS that will boot up and even let them log in and use the system, only losing a bunch of their personal settings (and of course possibly some data, depending on what else they deleted). While this may certainly be a frustrating situation for the user, the crucial difference is that the OS is still working, so the user is less likely to blame the OS just because it "suddenly broke for no reason."

1
  • 3
    Other answers are authoritative, and yours is speculative.
    – RonJohn
    Dec 16, 2022 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.