I’m aware of three main techniques used to exit protected mode on 286s; they all involve resetting the CPU, they vary on how that is done:
- using the keyboard controller (which IBM endowed with the ability to reset the CPU);
- using a chipset-specific “fast reset”;
- using a triple fault.
There’s a detailed discussion of the first and last approaches on Robert Collins’ site. To reset using the keyboard controller, you write 0xFE (the code for the shutdown command) to the 8042 status port (0x64). To reset using a triple fault, you set the limit of the IDTR to 0, and generate an interrupt; since the segment is now invalid, the CPU faults, but that also involves an invalid segment, so the CPU faults again twice before resetting.
When the CPU resets, it starts running from near the top of memory, in the BIOS; the BIOS checks a code in the CMOS, or a value in RAM, to determine whether it’s running a full boot-up or it’s recovering from a protected mode exit. To set this up, store the restart address (the real mode address your program will resume from) in the DWORD at 0x0040:0x0067, and set the CMOS shutdown type to 5 (see
SET_SHUTDOWN_TYPE for details; these run in real mode before the switch to protected mode, but the setup can also be done in protected mode). After resetting, the program needs to restore the CPU state properly (in particular, it needs to reload all the segment registers and the stack pointer); see Robert Collins’ documentation for details.