What are the "virtual machines" that were running on 80386 and later x86 CPUs before full hardware virtualization?

Question

When reading techy articles about older Windows versions I often come across the term "Virtual machine". Apparently even Windows 3.11 had some sort of virtual-machine-thing going on in it, not to mention later Windows versions. Except instead of running a full OS, they used it to provide isolation among programs or something. I've never seen a clear explanation of this, it's only been mentioned in passing.

But at the same time, consumer-level virtual machines (as in VirtualBox, VMWare, Virtual PC, etc.) only became a thing around mid-2000s, when the mainstream CPUs added dedicated hardware support for that. I remember that a few years before that there was a minor cheer when somebody (VMWare, I think?) figured out a way to do it without hardware support and thus started the whole virtual machine ball rolling. Their way did however have reduced performance and wasn't as stable as the hardware-based solutions that followed.

So what were the "virtual machines" that Windows used before this? I suspect that this term ("virtual machine") means something different it this context, but what?

Answer 1

I would think the articles you've read were most likely about the Virtual 8086 Mode introduced with the 386.

Here a host OS (running at privilege 0) would create a standard protected process, but mark it as VM86 when starting. In turn the process will be restricted to real mode addressing (16 bit segment and offset) and a 1 MiB address space.

Despite the naming (8086 mode) the code is not restricted to pure 16 bit. The CPU acts still as 386 enabling use of all operations with 32 bit registers or operands, by applying the usual prefixes. Restrictions comes only due operation encoding (like only BX, BP, SI and DI can be used for addressing) and address calculation. Any offset generated by an address calculation (like base plus index) must be result in a 16 bit value — otherwise a protection fault is issued (*1)

Virtual 8086 mode is essentially like the way a 386 comes out of reset, except it provides no way to access an virtual memory functions. All 'real' address generation is restricted to 20 bit, all critical instructions (*2) and I/O (*3) would be trapped, avoiding any break out attempt while allowing the host OS to redirect/emulate all hardware reaction.

In theory it would have been possible to load any arbitrary 16 bit (8086) OS and have it run flawless. For practical reasons higher level abstractions were preferred. After all, a 16 MHz 386 might be fast, but emulating byte wise disk access would it pull well down to C64/1541 level :))

VM86 was introduced with the 80386 and is still available today. Next to all OS adapted to 80386 and later supported this, starting with DR Concurrent DOS and Windows /386 (*4). All Windows supported it for DOS/16 bit programs, as well as OS/2 and Linux (for DOSEMU).

So unlike virtualization that 'multiplies' the existing, 'real' CPU with all its features, VM86 presents a limited CPU model for virtualized tasks.

---

The OS-Dev Wiki provides two pages to further dive into this interesting mode:

• Virtual 8086 Mode for the basic description and
• Virtual Monitor for consideration about a supervisor

---

*1 - The VM handler can of course relieve this by emulating a real mode address wrap. This may of course result in a hefty performance penalty ... as with an dirty code using fringe cases.

*2 - Trapped instructions are

• PUSHF/POPF - To avoid escaping the VM and emulation if flag status
• CLI/STI - To manage interrupt forwarding (and avoid locking the machine)
• INT - So the Host OS can catch and emulate software interrupts
• IRET - Again to avoid leaving the VM (IRET loads the EFLAGS rgister)

*3 - I/O privilege can be made available, which is quite handy when using a ROM BIOS from protected mode.

*4 - Yes, that's Win 2.1, so way before Win 3.x

Answer 2

"Virtual machine" has a long and varied history, not always meaning exactly what it means today.

Early designers of timesharing systems viewed what they were providing to their users was a virtual machine. In the terminology of the time, the user could act like he had a computer all to himself. This computer wasn't identical to the hardware; it may have had less access to certain machine resources, but on the plus side it had "instructions" (system calls) that the hardware did not and could not provide.

The hardware didn't need to provide much more than process isolation through memory management, and a "user mode" that restricted access to certain instructions that would allow it to break out of its isolation.

The goal was not to provide a programming environment like bare metal, but to provide something better than bare metal (and for a fraction of the cost). These days we tend to say "process" rather than "virtual machine".

For Intel processors, it's a little simpler. The operating systems of the time, such as MS-DOS, were single-user single-programming operating systems. MS-DOS thought it owned the entire machine and ran it in real mode. Virtual-8086 mode in the 80386 gave a protected-mode operating system the means to simulate multiple real-mode processors, and thus run multiple MS-DOS systems at the same time.

Notice that the virtual machine here was a subset (real mode) of the hardware machine.

In a separate path of development, full hardware emulation was developed. Somewhere in the 1960s, IBM in their Cambridge (MA, USA) research facility were building what is more like a modern virtual machine system, initially on a System/360 model 40. This eventually saw the light of day as VM/370.

In this sort of scheme, a hypervisor running on the bare metal uses hardware facilities to manifest machines that are "close enough" to the hardware to be able to run standard operating systems ("supervisors" in other terminology). "Virtual machine" now starts to mean a simulation of a real machine.

At some point in the early 2000s, "commodity CPUs" became powerful enough to support this sort of virtual machine, and the underlying hardware started to add features to make the virtualization more efficient.

tl;dr - the meaning of "virtual machine" depends on context.

Answer 3

In addition to the answers already given, the term virtual machine can also refer to a process virtual machine, i.e. a runtime environment executing platform independent bytecode[1]. The best known example today is probably the Java Virtual Machine (a version of which existed for Windows 3), but process virtual machines go back much further. Prominent historical examples include the UCSD p-System and Infocom's z-machine, both from the late 70's.

[1] see e.g. here for the distinction between system-level and process-level virtual machines

Answer 4

Windows 3.x in 386 enhanced mode ran processes in unprivileged sandboxes. They weren't as well isolated as the processes of modern OSes, but they had distinct virtual address spaces, and if they tried to directly use system-level features like segment descriptors and page tables, they would crash. A process could ask the actual kernel to perform similar operations on its behalf using DPMI, but it was the process's responsibility to know to do that.

The heroic measures that VMware and other software virtualizers had to resort to in the old days were only necessary because most OS kernels were only designed to run on bare hardware, and patching them all was not seen as a viable option. If there had been a standard paravirtualization interface similar to DPMI that was supported by every operating system that people wanted to virtualize, then most of that complexity (and the silicon cost of the virtualization support in modern CPUs) could have been avoided.