I've read that modern OSes rely on hardware-powered process isolation to prevent processes (and/or users) from clobbering each others' RAM. But on Intel processors, this hardware was first included in the 80286 (protected mode), so Linux required a minimum of a 80386 to run.
Was there a way to run a memory safe POSIX on a 8086 or 80286?
The short answer would be "No", since there is no way to prevent a user process from accessing privileged address space (of the OS or other processes) without some form of memory protection. Usually, this memory protection has to be implemented in hardware of the processor, such as you pointed out with 80286 protected mode.
Some alternatives would be:
Since POSIX is built on the older C-standard for heap usage (i.e. malloc/free), it would be possible to have user processes that cooperate on an 8086, through these API's, to guarantee they only access their own memory. Of course, bugs being a reality, this would not be as good as hardware protection of memory. Systems such as the Amiga and Macintosh (using the Motorola 68000) that used this strategy of software convention suffered with stability problems created by memory access bugs.
A computer using an 8086 can provide memory protection by using an external memory management unit. This would be a chip or a circuit that sits between the CPU and the memory and provides an additional layer of memory translation, sends interrupts if out-of-range memory is accessed, and so on. I don't know if this was commonly done on the 8086 (I've never seen such a system described, but then I've not looked for one either), but was very common for workstations based on early revisions of the 68000.
(Edit: at least some systems were produced that used this approach, although as @RossRidge points out in the comments it was a little easier on the 68000 due to specific support designed into the processor, which is probably why it was more common there.)
For an 80286, the standard 286 protected mode provides all the isolation that you'd need to run a POSIX compliant operating system with memory safety.
(It wouldn't be a very good POSIX system, because memory allocations would need to be limited to 64K to fit inside segment limits, but POSIX allows for sizes to be limited as low as _POSIX_SSIZE_MAX, which is defined as 32KiB, so this is fine)
There have been a number of Unix-like operating system that run on the 8086 and 80286, including Minix, which is usually considered the forerunner of Linux (it is the system that Torvalds used at the time he developed the first versions of Linux and influenced the early development quite a bit) and Xenix. There is also a port of Linux to 16-bit systems called ELKS, although I don't know whether it supports memory protection or not (looking at the source suggests it probably does, but I've never really done anything with it so can't be sure).
Technically yes because the 8086 instruction set is Turing-complete. Here is Linux running very slowly on an ARMv5 emulation on an 8-bit RISC microcontroller (also mentioned here). But if you want process isolation, I would look for other, less extreme solutions first!
I would say no. Even if external memory protection hardware is added, the processor lacks the concept of a user and supervisor or privileged state. As a result, there's no way to stop a program from disabling interrupts or accessing I/O ports, like those in the MMU.
Now if we set aside the need to isolate a malicious program, the MMU might be enough. Only problem with that thinking is that buggy programs can be pretty malicious even if the author is not.
So it would seem that we are back to no!
If you want process isolation, you need something more modern and a well written operating kernel.
The 8086,80186,8088 and 80188 all lacked any real memory protection although switching segment registers would protect against accidental overwriting. The 80286 did support protection so a POSIX OS with hardware enforced memory protection could be written.
The NEC V20 and V30 were 8086 clones with a 8080 emulation mode. Since it wouldn't be able to address more than 64K in 8080 mode one could presumably write a POSIX OS where the kernel ran in 8086 mode while userspace ran in 8080 mode switching between them to make system calls. Presumably it would still be possible to address the first 256 I/O ports directly which would mean user process could talk directly to some fairly important hardware if the V20/V30 were embedded in a standard IBM PC clone.
Yes, but it's not easy. There are at least two possible approaches:
This one is the canonical/classical solution. Essentially, you write an emulator/interpreter for some sort of virtual machine that does have kernel/user privilege modes and memory protection. You need to ensure (or assume) your interpreter has no vm-escape bugs.
Write the program-loader not to accept arbitrary 8086 machine code, but instead only a highly structured subset with enforcement of memory safety. This requires designing such a subset, and again you need to ensure or assume your implementation doesn't have bugs that break the necessary invariants.
Either way, to do POSIX or even something POSIX-like you're going to need lots of supplemental memory. There's no way to implement POSIX in the main memory size supported by the 8086, and both of these options will greatly further increase the required memory (and decrease speed).
The 8086 is a 16-bit processor. One possibility for implementing some form of process isolation is to use the processor's segment registers (CS, DS, SS, ES). These allow a process's stack (SS), heap (DS, ES), and code (CS) reside in specific 64kB areas of a 1MB address space. This works by left shifting the 16-bit segment register by four bits and adding to that the 16-bit stack pointer (SS << 4 + SP), instruction pointer (CS << 4 + IP), or data address (e.g. CS << 4 + SI), to obtain the 20 bits of the physical address.
Thus, through a suitable segment register setup one can isolate a process to at most 64kB, provided the process follows the convention of not altering the segment registers. For the requirements of C programs, where the heap and the stack must be addressable through the same 16-bit pointers, this convention restricts them to 64kB of data and 64kB of code. Although this might sound overly restrictive, remember that early Unix run on a PDP-11 with 64kB of RAM. Consequently, providing a 1MB memory for multiple processes with up to 64kB of code and 64kB of data is more than generous.
Furthermore, by manipulating segment registers and copying memory regions, a supervisor program can dynamically readjust memory regions as processes are created and destroyed in a way that's transparent to running processes. Early versions of Andrew Tannembaum's MINIX operating system relied on some of these ideas.
IN and OUT instructions. Those restrictions cannot be enforced in hardware but validation of the user code before transferring control would be possible. Similar just-in-time code validation is used in other places such as Java byte code, some versions of VM Ware, and the NaCl sandbox.
The Minix operating system implemented Virtual Memory Management on 8086 in software. Minix source code is available.
Can one isolate processes on a 8086?
Was there a way to run a memory safe POSIX on a 8086 or 80286?
To start with, 8086 or 286 (non protected mode), usually run DOS and thus only one process.
And the general answer would be no ...
... but it is possible to run either CPU in single step mode.
Sure, it'll slow down operation quite a lot, but for testing purpose it's a great way to monitor everything. With a few additional restrictions the downside can be (somewhat) kept in check. By making all used memory segments non overlapping and 64 KiB in size, the ISR only needs to check if an instruction loading a segment register is about to be executed, any only do detailed checking then, otherwise directly return.
I used that when searching for a memory problem in a rather complex communication application. In fact, with this (and other) optimizations it was fast enough to even cape up with average work load, fine to have it running for long enough to finally catch the bug - about 3 weeks later.
Bottom Line: Yes, it's possible, but you want to do it a lot.
Yes, you can with external hardware. The MC6829 MMU for the 6809 is a good example of how to do it. It detects when the CPU answers an interrupt request (software or hardware), and puts itself into a supervisor state when it can be modified. The you touch the "fuse register" just before a return from interrupt to enter user state, where it's completely invisible to the programmer and CPU.
