22

Continuing my nostalgia reading of Dan Gookin's DOS For Dummies, there is a section that literally advises against compression programs like DriveSpace and calls it a solution to a problem not some glorious road that one must take.

On more recent operating systems, compression is not something dangerous to the best of my knowledge. So what was the dangerous part?

7 Answers 7

43

The dangerous part was that your disk contents would be compressed.

i.e., you have data that can only be read by the compression engine.

If the compression engine was buggy, no data for you. If other buggy software corrupted the container file, no data for you.

If you had disk problems or file system corruption, you'd be limited to recovery tools that incorporated the same decompression algorithms. Contrast that to the fairly wide-open landscape of plain FAT tools.

7
  • 16
    Also, a single flipped bit would result in a whole compressed block being corrupted. Old drives and buses were not that reliable, and e.g. IDE bus didn't have any checksums before UltraDMA was introduced in 1998.
    – jpa
    Jan 15 at 9:12
  • So if I understand correctly, it's volume level compression while more recent operating systems are at a file level Jan 15 at 9:43
  • 6
    @user10191234 no, the issue isn't one of volume-level vs. file-level compression. The issue is that the DOS-based encryption was not transparent to applications with low-level access to file (which was common at the time) - so they could unknowingly corrupt a compressed file - and that the underlying physical drives and buses were much less reliable than todays', making data loss due to compression much more likely than today.
    – Dreamer
    Jan 16 at 13:09
  • 2
    I remember pretty well the fun I had with DriveSpace on old Windows. Yes, it was volume-level compression, but I didn't use it for long enough to encounter the issues. Still, it was impressive for the era.
    – IS4
    Jan 17 at 10:25
  • 1
    Decryption? You mean decompression? But yes, the file system used the CPU to compress data, and used the CPU to decompress it. The net effect of this is to make I/O take longer, from the application's point of view. Jan 17 at 23:07
39

The main problem was that DOS drive compression programs such as DriveSpace, Stacker, or SuperStor created a virtual drive and stored its contents in a large file on the original drive.

This created two risks:

  • the original drive could still be accessed, and any tool (or user) attempting to manipulate the file containing the compressed data could wreak havoc;
  • all the compressed drive’s data and metadata was stored in a proprietary format, which meant that familiar disk repair tools couldn’t be used.

Errors would generally result in the loss of multiple files, or even the whole volume, and were difficult if not impossible to repair.

Modern file systems supporting compression and write access tend to compress files individually inside a regularly-structured file system, reducing the risk.

4
  • 4
    Not only does per-file compression reduce the risk, it also allows you to compress files only when there's a real benefit to so doing. For example, a large file that you don't need to access very often, versus something that's needed very frequently. Besides which, disk space is essentially free these days, at least on desktop systems :-) Jan 15 at 16:50
  • 2
    I am not sure whether buying a larger harddrive could be considered essentially free.
    – IS4
    Jan 15 at 18:58
  • 7
    5 cents/gigabyte is pretty close to 'free', and you can get it even cheaper. This is the retrocomputing forum; we remember $1,000/megabyte (in 1969 dollars, maybe $10,000 /megabyte in current US dollars). Jan 16 at 3:31
  • 2
    indeed current hard drive prices are about $15-$20 per terabyte. Which will store 10 copies of Cyberpunk 2077.
    – user253751
    Jan 17 at 11:46
14

In addition to the compatibility problems and the data recovery problems mentioned in the other answers, there is one inherent problem with the disk compression schemes:

They are sometimes quite off from the efficiency promise they offer.

The promise is that the added time for decompressing a file is somewhat offset by the reduced time for reading the now-smaller file from disk. One can even expect an overall speedup if the disk is slow and the CPU is fast. So far, so good.

On the other hand, if one tries random access to a file (i.e. not reading it as a whole from the start to the end) the promise somewhat breaks apart.

Writing is not quite as fast either. In contrast, for a non-compressed filesystem, writing is almost as fast as reading.

And things go really, really bad if one tries random-writing in a file (e.g. a database-like workload). The usual result is that the random write in a file is either impressively slow (100x is not unusual), or the file grows larger than the real data contained inside, or both, depending on the particular compression scheme used.

Then, there is the issue of completely arbitrary reporting of the free space available. Some software actually used these numbers and failed when they were inflated too much.

13

I'll add some context to the good technical answers already given.

When the first on-the-fly disk compression software for PCs (Stacker) came out, it had a huge impact. It became prevalent faster than any software I'd ever seen, or would see. Storage was expensive, and this software was manna from heaven.

This even reached to point where computer ads started quoting machines with 80MB HDDs and "160MB" machines, machines with 100MB HDDs as "200MB" machines, and so on. This was circa 1992, I think, and you could flip through hundreds of pages of Computer Shopper magazine without seeing an ad that didn't use this trick.

I think the book you're quoting is (quite reasonably) saying "slow down." Yes, Stacker was an amazing program, but there are tradeoffs involved. It's a gentle reminder to an MS-DOS user community that very much needed it at the time, and you wouldn't know that unless you'd lived through it.

1
  • 3
    Another factor that made Stacker so popular were deals with computer stores to prominently display boxes of Stacker in their stores.
    – rcgldr
    Jan 16 at 17:18
3

There was an issue with using it for floppies. (You remember those, right? Weird shaped thin packages that you stuck in a slot on your computer to carry a small amount of data from one place to another. This was before such things as WiFi and so on. How primitive!)

You would get used to how long it took to write the data to the floppy. And you would wait that long. And the active light would go out. So you would pull the floppy out to head to the other machine. But the reason the light went out was because it had timed out waiting for the compression program to finish compressing the rest of the file. It still had a lot to write. Now the write has failed. Your floppy is corrupted. If you put the floppy back in, it probably does not help.

Even better, the error message is sitting there on the source machine. Unless you stood around and waited for the compression program to start to write again, and have it time out, you might not notice your floppy was corrupted. You might get all the way to the client's office with a corrupted floppy. Since you were in a hurry to get the data to the client by the deadline, probably you were not waiting around for this.

The corrupted floppy would sometimes be resistant to being uncorrupted. The corrupted file might, or might not, just let you delete it. You might need to reformat the floppy to get the media back.

After a few cycles of this kind, the company decided not to apply compression to floppies.

1

In those times Norton Disc Doctor could handle & fix corruptions of most disk compression algorithms (being on filesystem level and volume level).

Some compression systems used a file which contained the whole virtual volume, other systems made a new partition with the partition type set to some identifier which wasn't used in that time.

Some problems could occur, or the virtual volume became corrupted, or the identifier in the partition table became corrupted. Or worse the whole partition table became corrupted.

I remember one case recovering such a compressed "disk" completely low level hex editing a partition table because the partition table was corrupted and the compressed partition was gone.

On the other hand, due the nature of compression and depending on the algorithm used it was sometimes possible to recover (parts of) those volumes depending on the redundant info that was contained in the volume. Some compression schemes contained redundant info to make it less error prone. (One could compare it with the technique used in some raid systems where redundant info is contained on disk. If one disk fails one can rebuild the raid from scratch)

0

Compressed drives also require more work to use/own.

When a drive is compressed, the host drive contains a CONFIG.SYS, AUTOEXEC.BAT, a container file that holds the compressed user data and any other files referenced by CONFIG.SYS and AUTOEXEC.BAT. While booting, it's these CONFIG.SYS and AUTOEXEC.BAT files that are used. After the compression driver is loaded by CONFIG.SYS, it's assigned a new drive (e.g. D:) exposing the data in the container by compressing writes and decompressing reads for that drive. After mounting the container file, it swaps it with the original host drive letter (what was D: become C: and visa-versa). The problem is that when you install new software or make a configuration change, the CONFIG.SYS and AUTOEXEC.BAT files in the container file are updated and not the ones used during the boot. It requires the files on the boot drive to be synchronized with the copies in the compressed container.

Compression amplifies the problem of fragmentation. A normal FAT volume allocates in fixed size clusters. Compressed data is allocated using variably sized blocks with the granularity of a sector (or smaller, Stacker 4.0 could pack multiple small bocks into a single sector). Fragmentation is at the sector level and not just the cluster level. Without regular defragmentation, even small files could require multiple reads with data spread across different tracks undoing the gains from the smaller (compressed) i/o.

3
  • I just checked with MS-DOS 6.22 and DriveSpace. The uncompressed physical host drive has no CONFIG.SYS or AUTOEXEC.BAT so only the files from compressed logical volume is used.
    – Justme
    Jan 18 at 19:42
  • Awesome you looked - are you sure they aren't hidden? It's been decades since I've looked at 6.0 (I have never looked at 6.22) and it absolutely loaded config.sys from the uncompressed drive. Stacker certainly does.
    – Tony Lee
    Jan 18 at 21:45
  • They were not hidden in MS-DOS 6.22 using DriveSpace. Checked with DIR /A and ATTRIB commands on the host drive.
    – Justme
    Jan 18 at 22:24

Not the answer you're looking for? Browse other questions tagged or ask your own question.