It's been a while, but I've read in a system programming book that you could switch your Intel 80286 CPU from the normal real mode to a more powerful protected mode. I clearly remember that they said it was impossible to switch back though, unless you fully restarted the computer.
Why was this (made?) impossible? Was it a bug, intention by the DOS developers or a hardware restriction of the CPU?
This was intentional so that the CPU would support secure operating systems. In a secure operating system with rigorous memory access protections you could not allow any software - user or kernel extension or driver - to switch back to the full freedom of real mode.
They had a lot of interesting memory management hardware on the '286: rings and call gates - swiped nearly directly from the most secure hardware/software platform of the time: Multics. And they had new extensions too: task segments and task gates for multiprocessing.
These features never got used as they wanted. One major reason was that due to compatibility with all the peripherals - including disk controllers and graphics hardware that in those days were nowhere near sufficiently standardized - OS and other software that needed computer model/manufacturer independence needed to use the BIOS to access peripherals - and the BIOS was real mode - and the hacked-up switch to real mode (as discussed in other answers) was slow. Another major reason was that calling through call gates, and using hardware tasks, was much much slower than just implementing normal procedure calls and concurrent threads. Hardware task switching in particular was several times slower than just saving/restoring register context with normal instructions. So no OS software got written for these special modes.
These capabilities were also available when 32-bit processors were made - starting with the '386 (and they're still there in the x86 architecture to this day) - but with 32-bit addressing it turns out it is much easier to use paging hardware (address translation tables and so on) to get OS security - also much higher performance. No 32-bit OS used the stuff: Windows, OS/2, and all the Unixes used paging for process isolation and security.
I was actually saddened by this back in the day: I loved Multics, and the Multics architecture. The Intel designer's hearts were in the right place, and it was brilliant work sticking all that stuff in a commodity microprocessor, but it turned out to be the wrong solution for the problem. Operating system engineering had progressed past the Multics days and all that special hardware just did not lead to an economic solution.
My guess is that it was merely a design decision based upon the assumption that once a protected mode OS is started, there is no need to go back. Most microprocessors at that time already booted in its most privileged mode and had at least two levels of protection. The 80286 had to boot in real mode to keep compatibility with DOS and I think they thought DOS would reduce itself to just a minimal procedure to boot the main OS.
It seems that Intel engineers didn't realize that DOS was going to live for about ten years after launching the 286, and software engineers along with motherboard manufacturers would figure out a way to switch the CPU back to real mode in order to call DOS services from a protected mode program (who decided that the keyboard controller is a good place to put a register with a bit to reset the CPU and another one to enable CPU addresses beyond 1MB?)
By the time the 80386 was produced, they added the feature to switch the CPU back to real mode, and the dirty trick used with the 80286 -to reset it with a magic number at a certain memory location so the BIOS could read it and jump to some predefined code to resume operation- was not needed any more.
Then, the undocumented LOADALL instruction was spoiled on the USNet, which could allow not only switching from and to real mode, but allow non standard CPU states, such as the so called "unreal" mode, which allowed a real mode DOS program to access memory beyond the 1MB barrier.
This article from the OS/2 museum discuss the use of LOADALL to switch back from protected mode, and how Microsoft used it in HIMEM.SYS to allow fast above-1MB-memory copying without having to leave real mode.
Just speculating here, but it might have been a product decision to encourage writing code for protected mode. It's also possible it was a combination of technical difficulties and product priorities.
The CPU is put into protected mode by setting the PE bit in MSW using the LMSW or LOADALL instructions. Clearing the PE bit has no effect using either of those instructions, thus it is not possible to switch back to real mode. The fact that LOADALL allows undocumented behaviours like "unreal mode" (by loading values into descriptor caches to access memory outside of 1MB) and even allows you to put the CPU into an unusable state by loading nonsensical values, but disallows clearing the PE bit makes it likely it was a deliberate product decision. OTOH, LOADALL is undocumented, so maybe there really was a technical problem.
Just like switching from real to protected mode, going back to real mode is a carefully choreographed dance that needs to make sure all memory accesses go to defined memory locations (especially instruction fetches). The documentation has a strange passage that says "After executing LMSW instruction to set PE, the 80286 must immediately execute an intra-segment JMP instruction to clear the instruction queue of instructions decoded in real address mode" hinting that there is more to it than just setting or clearing a bit somewhere. It probably wasn't completely trivial to implement (not very hard either, but it definitely would have had a cost if you're counting transistors or microcode ops. The 80286 had ~134,000 transistors, and using a couple 100 transistors to implement an unnecessary feature would probably have meant dropping other, deemed more important features). Since they didn't anticipate a need for it, maybe they weren't willing to pay the "silicon tax".
There are still ways to get back to real mode by resetting the CPU without restarting the computer. On the PC, this could be done by putting a magical value into a special memory location to prevent the BIOS from reinitializing the computer after a reset, and then causing a reset through the i8042 keyboard controller which would externally reset the CPU, or by generating a triple fault (faulting in a fault handler, usually by invalidating the IDTR and causing an interrupt).
Using the triple fault is usually much faster. Going through the keyboard controller can take almost a millisecond, the triple fault only a couple hundred microseconds. Even with the triple fault, the reset circuitry is still external, but it doesn't have to be processed by the very slow i8042 (some PCs of the time "short-circuited" the reset bit in the 8042 port to directly reset the CPU without going through the i8042, in that case, there's no noticeable difference).
The reasons to switch back to real mode are all due to technical debt, you wouldn't need it in a clean system that's designed for protected mode from the ground up. In reality, people kept writing software for real mode and were unwilling (or unable) to quickly port stuff like device drivers to protected mode, and the 2 don't mix. Maybe Intel was trying to "encourage" people to port their drivers to protected mode by making it necessary, but as we know, this didn't work out.
Intel didn't make that mistake again, and the 80386 fixed the glitch both by allowing mode switches in all directions, and by adding a virtual 8086 mode that was essentially a protected mode that looked like real mode to the application.
