Return to Answer

##TL;DR;

##The Long Read

##Modern Dropfile Implementation(s)

##Checkpoints

##Modern Checkpointing

TL;DR;

The Long Read

Modern Dropfile Implementation(s)

Checkpoints

Modern Checkpointing

Dropfiles are somewhat related to virtualization as they allow to remove a process complete from execution and restart it later. Agent_L describes it quite head on as 'per-process hybernation'. A whole process will be taken of a machine with the option to restart it later on the same or another machine under the same or another user.

Dropfiles are originated in a complete different view on computer operation. They exist in environments that try to avoid the overhead of virtual memory. Usually these were machines where a process occupied the whole machine, running without interruption.

A dropfile is exactly what essentially Charles Duffy said, a complete serialization of a process' state. Important detail to be added: Issued by the OS. Essentially a recording of all states of a process to a disk file. Depending on the OS this includes

• Registers
• Allocated Memory Content
• OS Values like
  • Assigned Devices
  • File Tables
  • Job Stack (*1)

In some degree it's much like a dump, except written to file and reloadable.

Main reasons for a dropfile to be written is either processing timeout (assigned total CPU time or run time exhausted) or an operator issued abort - like when needing space to load a different job.

While dropfiles went quick out of use (see below) and being replaced by virtualization for commercial mainframes, they kept a major thing on scientific mainframes. CDC's operating systems were based around dropfiles. There was a whole set of utilities to handle dropfiles, especially to move them beteen production environment and development.

To understand this it's important to keep in mind that these machines were about number crunching, not data processing. Here any addition of virtual memory meant slowing down the machine, so they worked without such fancy stuff (see this answer about Cray machines) . Also they (often) got a fixed number of processes that could run concurrent, defined by the number of 'PCs'. Processes got fixed amount of memory assigned as well as a total run time. Process was expected to work full time, not much I/O at all - after all, that's what a supercomputer was for: Crunching numbers. Such jobs needed the whole attention of a machine - and having the ability to operate more than one program at the same time at high speed was a major USPs of Cray's designs.

If the assigned run time was reached, a dropfile got written, memory freed and its PC assigned to the next job in line. Usually a job was not expected to reach that limit, but this ofc depends on the way jobs were structured. There was no automatic restart, it was always done manual - after all, that job already exceeded what was assumed to be the time needed. So there must be something wrong. Tools (like SWITCHP on VSOS (*2)) allowed developers to load a dropfile to do a 'live' system diagnosis.

Of course, researchers quickly learned about the benefits for their 'private' projects. Stuff were they didn't get enough processing time was lined up at very low priority, having the operator (or operator scripts) restating their job whenever there was no other to run.

Dropfiles are a quite efficient way to handle a few jobs with long run times and low exchange rates. Compared to virtualization run time overhead is negliable.

Once dropfiles were available, they became as well a great tool not only to suspend a job for diagnosis or later restart, but as well to move it between different (compatible) machines or even sites. Programs could be developed on some small machine, debugged until it seamed to work fine, dropped and restarted on a faster production machine. Or as well copied and put onto several machines in parallel A whole job became a tradeable commodity like a single program. In some way, dropfiles (when handeled accordingly) can even be seen as a form of containerization (*3).

So while dropfiles are somewhat related to virtualization, they are result of a complete different view on computer operation.

[While knowing to go totally off-topic here, I feel it might be helpful to see how these ideas may be still relevant in modern computing]

CRIU (Checkpoint/Restore In Userspace) is a project implementing something quite similar to dropfiles for Linux (As mentioned here by an 'undecodable' user). CRIU enables the packaging of all content and states of a Linux user process in a set of files and restore it from there. Like often with modern developments, incorrect historic names are reused - see below for the original meaning of checkpoint.

There are/have been several other similar atempts like BLCR (Berkeley Lab Checkpoint/Restart), but they went all, AFAICT, already many years ago in hibernation

Here a process can request a 'checkpoint' written with data it needs for an optional restart. An OS managed one checkpoint file per process (if requested), whenever a new checkpoint was written the prior was deleted.

While with a similar goal as CRIU, DMTCP (Distributed MultiThreaded CheckPointing) walks some middle line between checkpointing and dropfile. Much like with a dropfile the whole application memory space gets saved, but much like with checkpointing it is done by the application using specific interfaces (called plugins) to monitor handle all resources that need to be saved. It feels like a very interesting project, recovering (and rediscovering) things that have worked well in ye olde times to be used for today's tasks.

Dropfiles are somewhat related to virtualization as they allow to remove a process complete from execution and restart it later. Agent_L describes it quite head on as 'per-process hibernation'. A whole process will be taken off a machine with the option to restart it later on the same or another machine under the same or another user.

Dropfiles are originated from a complete different view of computer operation. They exist in environments that try to avoid the overhead of virtual memory. Usually these were machines where a process occupied the whole machine, running without interruption.

A dropfile is exactly what essentially Charles Duffy said, a complete serialization of a process' state. Important detail to be added: Issued by the OS. It's essentially a recording of all state of a process to a disk file. Depending on the OS, this includes

• Registers
• Allocated Memory Content
• OS Values, like
  • Assigned Devices
  • File Tables
  • Job Stack (*1)

To a degree, it's much like a dump, except written to file and reloadable.

The main reason for a dropfile to be written is either processing timeout (assigned total CPU time or run time exhausted) or an operator issued abort - like when needing space to load a different job.

While dropfiles quickly went out of use (see below) and were replaced by virtualization for commercial mainframes, they remained a major thing on scientific mainframes. CDC's operating systems were based around dropfiles. There was a whole set of utilities to handle dropfiles, especially to move them between production environment and development.

To understand this, it's important to keep in mind that these machines were about number crunching, not data processing. Here, any addition of virtual memory meant slowing down the machine, so they worked without such fancy stuff (see this answer about Cray machines) . Also they (often) got a fixed number of processes that could run concurrently, defined by the number of 'PCs'. Processes got a fixed amount of memory assigned, as well as a total run time. Each process was expected to work full time, with not much I/O at all - after all, that's what a supercomputer was for: crunching numbers. Such jobs needed the whole attention of a machine - and having the ability to operate more than one program at the same time at high speed was a major USP of Cray's designs.

If the assigned run time was reached, a dropfile got written, memory freed and its PC assigned to the next job in line. Usually a job was not expected to reach that limit, but this depends on the way jobs were structured. There was no automatic restart; it was always done manually - after all, that job already exceeded what was assumed to be the time needed, so there must be something wrong. Tools (like SWITCHP on VSOS (*2)) allowed developers to load a dropfile to do a 'live' system diagnosis.

Of course, researchers quickly learned about the benefits for their 'private' projects. Stuff that didn't get enough processing time was lined up at very low priority, having the operator (or operator scripts) restarting their job whenever there was no other to run.

Dropfiles are a quite efficient way to handle a few jobs with long run times and low exchange rates. Compared to virtualization, run time overhead is negligible.

Once dropfiles were available, they became a great tool to not only suspend a job for diagnosis or later restart, but also to move it between different (compatible) machines or even sites. A programs could be developed on some small machine, debugged until it seemed to work fine, dropped and restarted on a faster production machine. Or copied and put onto several machines in parallel A whole job became a tradeable commodity like a single program. In some way, dropfiles (when handled accordingly) can even be seen as a form of containerization (*3).

So while dropfiles are somewhat related to virtualization, they are result of a completely different view on computer operation.

[While knowing I'm going totally off-topic here, I feel it might be helpful to see how these ideas may be still relevant in modern computing]

CRIU (Checkpoint/Restore In Userspace) is a project implementing something quite similar to dropfiles for Linux (as mentioned here by noɥʇʎԀʎzɐɹƆ). CRIU enables the packaging of all content and state of a Linux user process in a set of files and restoration from there. Like often with modern developments, incorrect historic names are reused - see below for the original meaning of checkpoint.

There are/have been several other similar atempts like BLCR (Berkeley Lab Checkpoint/Restart), but they went all, AFAICT, into hibernation many years ago.

Here a process can request a 'checkpoint' written with data it needs for an optional restart. An OS managed one checkpoint file per process (if requested): whenever a new checkpoint was written the prior was deleted.

While with a similar goal to CRIU, DMTCP (Distributed MultiThreaded CheckPointing) walks some middle line between checkpointing and dropfile. Like a dropfile, the whole application memory space gets saved; but like checkpointing, it is done by the application using specific interfaces (called plugins) to handle all resources that need to be saved. It feels like a very interesting project, recovering (and rediscovering) things that have worked well in ye olde times to be used for today's tasks.

Dropfiles are somewhat related to virtualization as they allow to remove a process complete from execution and restart it later. Agent_L describes it quite head on as 'per-process hybernation'. A whole process will be taken of a machine with the option to restart it later on the same or another machien under the same or another user.

Once dropfiles were available, they became as well a great tool not only to suspend a job for diagnosis or later restart, but as well to move it between different (compatible) machines or even sites. Programs could be deveoped on some small machine, debugged until it seamed to work fine, dropped and restarted on a faster production machine. Or as well copied and put onto several machines in parallel A whole job became a tradeable commodity like a single program. In some way, dropfiles (when handeled accordingly) can even be seen as a form of containerization (*3).

Unlike with a dropfile, restarting from a checkpoint required cooperation of the program to be restarted. The job got started again as with the prior run, but the progam checked for a checkpoint and, if present, read it and *2 - No, that's not a CPU stack, but the commencing punch card job - today's analogy would be 'the script' :)

Dropfiles are somewhat related to virtualization as they allow to remove a process complete from execution and restart it later. Agent_L describes it quite head on as 'per-process hybernation'. A whole process will be taken of a machine with the option to restart it later on the same or another machine under the same or another user.

Once dropfiles were available, they became as well a great tool not only to suspend a job for diagnosis or later restart, but as well to move it between different (compatible) machines or even sites. Programs could be developed on some small machine, debugged until it seamed to work fine, dropped and restarted on a faster production machine. Or as well copied and put onto several machines in parallel A whole job became a tradeable commodity like a single program. In some way, dropfiles (when handeled accordingly) can even be seen as a form of containerization (*3).

Unlike with a dropfile, restarting from a checkpoint required cooperation of the program to be restarted. The job got started again as with the prior run, but the program checked for a checkpoint and, if present, read it, setup all data as before and proceed from there.