23

After reading Charles Duffy's comment on Was there a clearly identifiable "first computer" to use or demonstrate the use of virtual memory?:

I had a computer science professor around 1999 or therebouts who still bemoaned that virtual memory had won out over use of dropfiles (completely serializing/deserializing process state). My class at CSU Chico learned more about virtual memory in our systems architecture class than we did in our operating system courses... – Charles Duffy

I tried to find out what a "dropfile" was. Unfortunately, I was unable to do so, probably because the words "drop" and "file" are so often used to refer to the more modern virtual memory / swap combination that, once those were filtered out, my search engine returned only irrelevant results about config files passed to BBS games.

What actually are dropfiles, what were they used for, and why?

  • 2
    They are about the same as what is termed a Hibernation file in Windows or hiberfil.sys. Needs a bit more citation to turn this into an answer... – Brian Tompsett - 汤莱恩 Nov 9 at 17:47
34

TL;DR;

Dropfiles are somewhat related to virtualization as they allow to remove a process complete from execution and restart it later. Agent_L describes it quite head on as 'per-process hibernation'. A whole process will be taken off a machine with the option to restart it later on the same or another machine under the same or another user.

Dropfiles are originated from a complete different view of computer operation. They exist in environments that try to avoid the overhead of virtual memory. Usually these were machines where a process occupied the whole machine, running without interruption.


The Long Read

A dropfile is exactly what essentially Charles Duffy said, a complete serialization of a process' state. Important detail to be added: Issued by the OS. It's essentially a recording of all state of a process to a disk file. Depending on the OS, this includes

  • Registers
  • Allocated Memory Content
  • OS Values, like
    • Assigned Devices
    • File Tables
    • Job Stack (*1)

To a degree, it's much like a dump, except written to file and reloadable.

The main reason for a dropfile to be written is either processing timeout (assigned total CPU time or run time exhausted) or an operator issued abort - like when needing space to load a different job.

While dropfiles quickly went out of use (see below) and were replaced by virtualization for commercial mainframes, they remained a major thing on scientific mainframes. CDC's operating systems were based around dropfiles. There was a whole set of utilities to handle dropfiles, especially to move them between production environment and development.

To understand this, it's important to keep in mind that these machines were about number crunching, not data processing. Here, any addition of virtual memory meant slowing down the machine, so they worked without such fancy stuff (see this answer about Cray machines) . Also they (often) got a fixed number of processes that could run concurrently, defined by the number of 'PCs'. Processes got a fixed amount of memory assigned, as well as a total run time. Each process was expected to work full time, with not much I/O at all - after all, that's what a supercomputer was for: crunching numbers. Such jobs needed the whole attention of a machine - and having the ability to operate more than one program at the same time at high speed was a major USP of Cray's designs.

If the assigned run time was reached, a dropfile got written, memory freed and its PC assigned to the next job in line. Usually a job was not expected to reach that limit, but this depends on the way jobs were structured. There was no automatic restart; it was always done manually - after all, that job already exceeded what was assumed to be the time needed, so there must be something wrong. Tools (like SWITCHP on VSOS (*2)) allowed developers to load a dropfile to do a 'live' system diagnosis.

Of course, researchers quickly learned about the benefits for their 'private' projects. Stuff that didn't get enough processing time was lined up at very low priority, having the operator (or operator scripts) restarting their job whenever there was no other to run.

Dropfiles are a quite efficient way to handle a few jobs with long run times and low exchange rates. Compared to virtualization, run time overhead is negligible.

Once dropfiles were available, they became a great tool to not only suspend a job for diagnosis or later restart, but also to move it between different (compatible) machines or even sites. A programs could be developed on some small machine, debugged until it seemed to work fine, dropped and restarted on a faster production machine. Or copied and put onto several machines in parallel A whole job became a tradeable commodity like a single program. In some way, dropfiles (when handled accordingly) can even be seen as a form of containerization (*3).

So while dropfiles are somewhat related to virtualization, they are result of a completely different view on computer operation.


Modern Dropfile Implementation(s)

[While knowing I'm going totally off-topic here, I feel it might be helpful to see how these ideas may be still relevant in modern computing]

CRIU (Checkpoint/Restore In Userspace) is a project implementing something quite similar to dropfiles for Linux (as mentioned here by noɥʇʎԀʎzɐɹƆ). CRIU enables the packaging of all content and state of a Linux user process in a set of files and restoration from there. Like often with modern developments, incorrect historic names are reused - see below for the original meaning of checkpoint.

There are/have been several other similar atempts like BLCR (Berkeley Lab Checkpoint/Restart), but they went all, AFAICT, into hibernation many years ago.


Checkpoints

Checkpoints are a somewhat similar concept developed prior to dropfiles and persistent in commercial operation way into the 90s.

Where dropfiles capture the whole process state for an automated and complete restart, checkfiles keep 'only' data (pointers) enabling continued operation after the application is loaded again.

Here a process can request a 'checkpoint' written with data it needs for an optional restart. An OS managed one checkpoint file per process (if requested): whenever a new checkpoint was written the prior was deleted.

Unlike with a dropfile, restarting from a checkpoint required cooperation of the program to be restarted. The job got started again as with the prior run, but the program checked for a checkpoint and, if present, read it, setup all data as before and proceed from there.

The whole operation is much like what modern office software (like Libre Office) does when being started after a crash.


Modern Checkpointing

While with a similar goal to CRIU, DMTCP (Distributed MultiThreaded CheckPointing) walks some middle line between checkpointing and dropfile. Like a dropfile, the whole application memory space gets saved; but like checkpointing, it is done by the application using specific interfaces (called plugins) to handle all resources that need to be saved. It feels like a very interesting project, recovering (and rediscovering) things that have worked well in ye olde times to be used for today's tasks.


*1 - No, that's not a CPU stack, but the commencing punch card job - today's analogy would be 'the script' :)

*2 - Which in fact as based on virtual memory operation - still the job structure was still based around long running batch operation.

*3 - As I like to say: There isn't anything really new when it comes to computing - just improved reuse :)

  • 1
    @JD No, not really, this would be quite a stretch. A dropfile is about a single process, not a whole machine and it's really more of a memory dump plus a a little state information (most is already part of the dump). It isn't much of a mechanism either. – Raffzahn Nov 11 at 7:52
  • 2
    So, it's like per-process hibernation? – Agent_L Nov 11 at 14:58
  • 2
    @Agent_L Yes, that could be said. In fact, that comes as close as a comparsion can be made with today's systems. I think I'll add that to the answer. – Raffzahn Nov 11 at 15:21
  • 1
    CRIU implements this. – noɥʇʎԀʎzɐɹƆ Nov 11 at 16:05
  • 2
    From what I recall, at least a couple of systems in my past (TOPS-10, GEORGE 3) supported a "save" command that would save the address space as an executable file that could be subsequently run. This sounds like the same result as a "dropfile" though it was conceptualized quite differently. – another-dave Nov 13 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.