The (plain) 68k never had anything directly comparable to the Intel x86 range's Protected Mode. When Intel introduced the Protected mode (PM) to its x86 range of CPUs, this lifted a number of restrictions, though, that the 68k range never had:
- PM allowed the OS designer to protect (hence the name...) certain address ranges from access from non-privileged code, basically layering the software into privileged (OS) and non-privileged (applications) layers with certain access limitations on memory areas and certain instructions. The 68k from the very beginning supported supervisor and user modes to do a similar thing (but needing support from circuitry external to the CPU to define and impose those protected ranges). The Atari ST, for example, would execute a software trap triggered by its MMU (**) and GLUE custom chips when applications in user mode tried to access certain hardware register areas that were only allowed for Supervisor Mode code.
- Starting with the 386, PM provided a 32-bit linear address range to end user programs and allowed OS and applications writers to get rid of the quirky segmented architecture the original 8086 had (with "tiny", "large" and "huge" memory models and all the software quirks that came with them). The 68k never had this limitation as it provided linear 32-bit addressing from the very beginning (but see further down below (*))
- The 386 PM introduced a fully capable MMU and address translation between virtual, linear and physical memory addresses - The original 68000 didn't have this and relied on external circuitry to do such things. A similar capability was introduced to the Motorola CPU range with the 68010 and 68020 CPUs, until the first "full MMU" implementation followed in the 68030.
The Atari ST generally ran applications in User Mode, in order to allow memory (and I/O) protection.
The Sinclair QL's QDOS did as well, effectively switching off multitasking when in Supervisor Mode (or rather, running OS code). Thus restricting Supervisor Mode code to OS code or hardware specific code like device drivers and (obviously) interrupt service routines. This also meant that user applications had to restrict the time they spent in Supervisor Mode as much as possible, in order to not severely harm proper time-slicing. Another reason not to stay in SM for too long was the very limited resources (only 100 bytes of supervisor stack, for example) assigned to this mode by the OS.
The 68k Mac, to my knowledge, didn't actually care much about UM and SM and applications were allowed to run in Supervisor mode all the time.
PalmOS also had a strong distinction between Supervisor and User Mode, reliant on co-operative multitasking, it effectively switched off application switching in Supervisor mode in order to be able to do some of it's complicated memory management magic.
(*) The fact that the 68k range had a non-segmented architecture is only partially valid: It is true that the 68000 did have linear 32-bit addressing from the very beginning, but this was only valid for direct addressing modes - indirect (register-relative) addressing was limited to +-32k in the original 68000, this limitation was lifted only later by the 68020.
This lead to the interesting fact that some computer architectures for the 68000 (examples being MacOS 68k, QDOS for the Sinclair QL and PalmOS for Palm Handhelds) implemented a software-segmented architecture, that introduced artificial segmentation into 64k segments by only allowing register- and PC-relative addressing, which allowed them to get rid of an otherwise needed relocating loader to adapt absolute addresses in the code to the load address of the binary.
This limited intrasegment calls to +-32k and actually introduced a sort of segmentation similar to what the 8086 had, but made life for the OS writers quite a bit easier (but for applications writers more complicated). Imposing the limit of fully position-independent code and data onto applications allowed the OS to do much more effective memory management (entirely needed on early 68k Macs that were limited to 128k of memory and PalmOS devices whose main memory also had to accommodate for mass data storage). The OS could shift applications' data and code around arbitrarily and only had to adjust one single base register in the application - Much like it had a hardware MMU. Applications writers, however, had to introduce jump islands or other helpers in their code (typically done on MacOS and PalmOS) in order to be able to reach a location further than 32k away, or, as an alternative, had to resort to having a relocation loader embedded into the application, where possible (Sinclair QL).
The Amiga and the Atari ST operating systems did have a relocating loader and supported full absolute 32-bit addressing in applications programs. This did, however, mean that once an application was loaded and relocated in memory by the OS, it couldn't easily be moved. (I'm not an Amiga expert, but do believe the Amiga used register-relative addressing in libraries, though).
The Sinclair QL allowed both approaches, supporting both so-called pure and impure programs, the former fully movable in memory and allowing to run several task ("job") instances on top of the very same code, the latter supporting in-built relocating loaders.
(**) The Atari ST custom chip referred to as "MMU" here (that was the name Atari christened it) was not what we would call an MMU today. Yes, it did "manage memory", but not in the sense of virtual memory and address translation.