How did the original Apple Macintosh and Atari ST use protected mode?

Question

The Macintosh, ST, and Amiga were all 68000 CPU systems released in 1984-85, along with their respective new Operating Systems. While this was 5 years before protected mode became commonplace on Intel PC (80286/80386) OS's, the 68000 had a protected mode as well (i.e. a special mode allowing the use of privileged/supervisor instructions not normally available to user code).

The Amiga Exec kernel eschewed the use of protected mode in order to provide efficient multitasking and inter-process communications. The original Mac OS and Atari TOS did not really have these features, and I think they made use of protected mode for something.

How was the protected mode used in Mac OS and TOS? What advantages did this provide?

Answer 1

The (plain) 68k never had anything directly comparable to the Intel x86 range's Protected Mode. When Intel introduced the Protected mode (PM) to its x86 range of CPUs, this lifted a number of restrictions, though, that the 68k range never had:

1. PM allowed the OS designer to protect (hence the name...) certain address ranges from access from non-privileged code, basically layering the software into privileged (OS) and non-privileged (applications) layers with certain access limitations on memory areas and certain instructions. The 68k from the very beginning supported supervisor and user modes to do a similar thing (but needing support from circuitry external to the CPU to define and impose those protected ranges). The Atari ST, for example, would execute a software trap triggered by its MMU (**) and GLUE custom chips when applications in user mode tried to access certain hardware register areas that were only allowed for Supervisor Mode code.
2. Starting with the 386, PM provided a 32-bit linear address range to end user programs and allowed OS and applications writers to get rid of the quirky segmented architecture the original 8086 had (with "tiny", "large" and "huge" memory models and all the software quirks that came with them). The 68k never had this limitation as it provided linear 32-bit addressing from the very beginning (but see further down below (*))
3. The 386 PM introduced a fully capable MMU and address translation between virtual, linear and physical memory addresses - The original 68000 didn't have this and relied on external circuitry to do such things. A similar capability was introduced to the Motorola CPU range with the 68010 and 68020 CPUs, until the first "full MMU" implementation followed in the 68030.

The Atari ST generally ran applications in User Mode, in order to allow memory (and I/O) protection.

The Sinclair QL's QDOS did as well, effectively switching off multitasking when in Supervisor Mode (or rather, running OS code). Thus restricting Supervisor Mode code to OS code or hardware specific code like device drivers and (obviously) interrupt service routines. This also meant that user applications had to restrict the time they spent in Supervisor Mode as much as possible, in order to not severely harm proper time-slicing. Another reason not to stay in SM for too long was the very limited resources (only 100 bytes of supervisor stack, for example) assigned to this mode by the OS.

The 68k Mac, to my knowledge, didn't actually care much about UM and SM and applications were allowed to run in Supervisor mode all the time.

PalmOS also had a strong distinction between Supervisor and User Mode, reliant on co-operative multitasking, it effectively switched off application switching in Supervisor mode in order to be able to do some of it's complicated memory management magic.

(*) The fact that the 68k range had a non-segmented architecture is only partially valid: It is true that the 68000 did have linear 32-bit addressing from the very beginning, but this was only valid for direct addressing modes - indirect (register-relative) addressing was limited to +-32k in the original 68000, this limitation was lifted only later by the 68020.

This lead to the interesting fact that some computer architectures for the 68000 (examples being MacOS 68k, QDOS for the Sinclair QL and PalmOS for Palm Handhelds) implemented a software-segmented architecture, that introduced artificial segmentation into 64k segments by only allowing register- and PC-relative addressing, which allowed them to get rid of an otherwise needed relocating loader to adapt absolute addresses in the code to the load address of the binary.

This limited intrasegment calls to +-32k and actually introduced a sort of segmentation similar to what the 8086 had, but made life for the OS writers quite a bit easier (but for applications writers more complicated). Imposing the limit of fully position-independent code and data onto applications allowed the OS to do much more effective memory management (entirely needed on early 68k Macs that were limited to 128k of memory and PalmOS devices whose main memory also had to accommodate for mass data storage). The OS could shift applications' data and code around arbitrarily and only had to adjust one single base register in the application - Much like it had a hardware MMU. Applications writers, however, had to introduce jump islands or other helpers in their code (typically done on MacOS and PalmOS) in order to be able to reach a location further than 32k away, or, as an alternative, had to resort to having a relocation loader embedded into the application, where possible (Sinclair QL).

The Amiga and the Atari ST operating systems did have a relocating loader and supported full absolute 32-bit addressing in applications programs. This did, however, mean that once an application was loaded and relocated in memory by the OS, it couldn't easily be moved. (I'm not an Amiga expert, but do believe the Amiga used register-relative addressing in libraries, though).

The Sinclair QL allowed both approaches, supporting both so-called pure and impure programs, the former fully movable in memory and allowing to run several task ("job") instances on top of the very same code, the latter supporting in-built relocating loaders.

(**) The Atari ST custom chip referred to as "MMU" here (that was the name Atari christened it) was not what we would call an MMU today. Yes, it did "manage memory", but not in the sense of virtual memory and address translation.

Answer 2

So far as I recall, the Macintosh System Software didn't bother with the user versus supervisor mode distinction. Even after the release of System 7, which supported virtual memory, virtually everything ran in supervisor mode.

I think the Apple Lisa, which had a more minicomputer-like (or what we'd today call Unix-like) operating system design, as well as an MMU, did use supervisor for the operating system itself and user mode for application level code.

Answer 3

68000 had a protected mode as well (i.e. a special mode allowing the use of privileged/supervisor instructions not normally available to user code).

Err, no. At least don't call it protected mode, call it supervisor mode or privileged mode. "Protected mode" implies something like the >=80286 protected mode which had hardware memory protection. This did not exist with the original 68K.

The 68000 supervisor mode meant two things:

• certain instructions were not allowed in user mode, for example enabling and disabling interrupts
• supervisor mode had a separate stack pointer

There was no built in support for memory protection at all, and I know for a fact (bitter experience), that early TOS on the Atari treated the memory space as a single shared resource for all programs. However, the 68000 had memory mapped IO and I think I'm correct in stating that the Atari MMU (a separate chip to the 68000) stopped user mode programs from accessing memory locations used by the registers of the IO devices. But that was the limit of memory protection.

I think roughly the same rules applied to Mac OS because one of the big deals when comparing it to Windows 9x was the lack of protection for processes (or pre-emptive multitasking). Mac OS X was the first virtual memory pre-emptive multitasking OS that achieved any traction in the Mac world.

With the 68000 you could not "eschew" supervisor mode. Any hardware interrupt or software trap automatically put the processor into supervisor mode, so you always had to deal with it - even on the Commodore Amiga.