According to the Oracle VirtualBox 6.0 manual, "Certain rare guest operating systems like OS/2 make use of very esoteric processor instructions that are not supported with our software virtualization. For virtual machines that are configured to contain such an operating system, hardware virtualization is enabled automatically." Any idea which "very esoteric processor instructions" would be meant by this?
As far as I’m aware the difficulty in virtualising OS/2 isn’t due to esoteric processor instructions, but rather esoteric processor features. Specifically, OS/2 uses all the protected mode features available on 286 (OS/2 1.x) and 386 (OS/2 2.0 and later) PCs: segment limits, paging, protection rings... The latter has commonly been given as the main difficulty in supporting OS/2: while most protected-mode PC operating systems use only rings 0 (kernel) and 3 (user-mode), OS/2 also uses ring 2 for privileged code which couldn’t touch the kernel but could access some pieces of hardware (for printers and displays).
Because support for x86-style rings other than 0 and 3 isn’t needed for many operating systems, it probably isn’t a priority for developers of virtualisation tools; bear in mind that most virtualisation tools are developed mostly by companies, and their goal is typically to support specific workloads rather than provide a complete emulation of the original hardware platform. By skipping some support for unused rings, they only lost support for operating systems such as OS/2, and some features of DR DOS (DPMS specifically). On top of that, at least some virtualisation tools rely on protection rings themselves to virtualise the operating system they’re running: if they can’t use hardware-assisted virtualisation, they’ll run their virtual machine manager in ring 0, and run the guest operating system in ring 1 (instead of ring 0). Doing that doesn’t leave much room for the guest operating system to do anything clever with protection rings itself.
This probably isn’t the only feature which causes issues. OS/2 runs in both 16-bit and 32-bit protected mode simultaneously, with frequent switches from one to the other, even when running 32-bit applications; this requires specific support in virtualisation environments and assistance from the host operating system (at least, on 64-bit PC operating systems). OS/2 1.x relies on triple-faulting to leave protected mode, and OS/2 in general uses call gates for system calls, both of which require specific handling in the virtual machine manager and are much easier to implement with hardware assistance. (Other operating systems also use call gates, so that’s not an OS/2-specific requirement.)
All this is specifically a problem when virtualising on x86 without hardware virtualisation (AMD-V and VT-x). The goal when virtualising is to set the host environment up such that the guest can run directly on the CPU, with no translation, but with all privileged operations intercepted by the “hypervisor”. “Software virtualisation” on x86 is deficient for these purposes, and requires a lot of work in the hypervisor to make up for the architecture’s deficiencies. Hardware-assisted hypervisors can run OS/2 without difficulty, relying on the hardware to provide a complete protected-mode virtualisation; as can emulators (such as Bochs and QEMU in non-KVM mode), relying on their CPU emulation to simulate the protected-mode environment. Keith Adams’ and Ole Agesen’s 2006 paper, A Comparison of Software and Hardware Techniques for x86 Virtualization, explains the complexity of x86 virtualisation in detail.