I know that Virtual 8086 mode was added on the 80386 to make it possible to run old applications in a Protected Mode environment.
The 80286 had a lot of shortcomings, such as not being able to switch back to Real Mode from Protected Mode (except by reset), but its PM environment wasn't that bad by itself. What made it unusable was its incompatibility with the old applications, which expected to be able to do segment arithmetic and other tricks that can be used in Real Mode.
But why wasn't this issue addressed in software at the time? People chose to ditch PM completely until the 80386 came along and instead use RM with dirty hacks to access extended memory (Unreal Mode or switching to PM to access memory and then reset).
Here's a way that I think they could have done it (but I'm not sure it would work in practice). The OS (or DOS extender or whatever) sets up a GDT (and possibly LDTs) which contain the descriptors required by each application (on the 286 you cannot have flat descriptors that cover all memory, because registers are 16-bit). However, all of these descriptors are marked as Not Present in the GDT/LDT except for the time slices in which the applications actually run. Of course, there's no need to create all these descriptors in a single-tasking system like MS-DOS. The only descriptors that are marked Present are the Ring 0 ones (kernel stuff, basically).
When an old 8086 application is run, allocate segments for it as required. If the segments are in extended memory, use
LOADALL to set the segment registers to them, since there is no entry in the GDT that can be used. If the application starts resetting segment registers (either by using segment register instructions, far calls or whatever), the CPU will trap and generate a fault, since regardless of what value the segment register takes, it won't be a valid GDT entry. The virtual monitor can then use
LOADALL to set the descriptor caches to the new segment (possibly in extended memory), while keeping the value in the visible segment register itself as it was set by the 8086 application.
TLDR: If the program ever reads the segment register, it will get what it expects. If it tries to write to the segment register, the virtual monitor will handle the case by using
LOADALL to load segments above 1MB in the descriptor caches while keeping the unrelated values set by the software inside the visible part of the segment registers.
Was this too slow to be practical? Did DOS programs perform so much segment reloading that the
LOADALLs would slow the system down to a crawl?
If DOS was extended to run in PM when the 80286 came out, while keeping compatibility with 8086 programs, programmers could have started making a lot of 16-bit PM applications instead of RM ones, which would reduce the overhead of accessing extended memory for all applications, not just applications which used DOS extenders.