"Protection error" is primarily a memory management violation. A process executes with its own address space. Particular pages (range of virtual addresses) may or may not be assigned to the address space. The hardware triggers a fault on an attempt to reference an address not assigned to the address space. The OS then aborts the process, if this was a user-mode fault. If the bad access was in kernel mode code, then typically that is "not supposed to happen", and the only thing that can be aborted is the kernel -- i.e., the system crashes.
Simplistic example: if your process has a virtual address range of 0 to 1000000, and it tries to read address 1005000, then that will result in an exception, and the OS will kill the process.
(This is overly simplified, since it ignores the mechanics of paged virtual memory, and how in some cases invalid addresses can be supplied, for example stack expansion).
There are other exceptions, such as the divide-by-zero case, that are also detected in hardware. Similarly, they might cause process termination or kernel crash, depending on where they happen.
Windows 9x, by virtue of its DOS-based heritage, did not have total separation between user and kernel, and it's possible that some user-mode errors could render the system unstable enough to warrant a reboot.
Having said that, this is not exactly retro material: it's how processor memory management still works. The OS sets up descriptions of the virtual address space for a process; the hardware detects accesses outside that assigned space; and the OS then responds to detected violations.
