Wikipedia's Intel 80186 entrymentions
The 80186 would have been a natural successor to the 8086 in personal computers. However, because its integrated hardware was incompatible with the hardware used in the original IBM PC, the 80286 was used as the successor instead in the IBM PC/AT.
That explains why most of the PC compatibles didn't use it either.
What exactly did the 186 do that was incompatible with the IBM PC?
The main issue with the 80186 isn’t with the CPU core itself, but with its integrated peripherals: they aren’t compatible with those used in the IBM PC, and they aren’t integrated in the same way either.
The IBM PC uses an 8237 DMA controller at offset 0x00 in the I/O address space, an 8259 PIC at offset 0x20, and an 8253 PIT at offset 0x40. The 80186’s integrated peripherals are mapped using a “peripheral control block”, which can be moved around in the I/O address space (it defaults to 0xFF00), but their offsets are fixed and don’t match the PC’s: while the PIC is at offset 0x20, the PIT is at offset 0x50, and the DMA descriptors are at offset 0xC0. The DMA control registers aren’t the same as the 8237’s, so code expecting an 8237 won’t work with the 80186. The 80186’s PIC and PIT are more sophisticated than the 8253 and 8259, and controlled rather differently too.
The 80186 also has more reserved hardware interrupts than the 8086.
A number of more-or-less compatible computers were made using the 80186, including one which has already been discussed here (Why is the RM Nimbus PC-186 not IBM PC compatible?), and famously, the Tandy 2000, Siemens PC-D, and a few of HP’s line of palmtop PCs (100LX, 200LX, 1000CX). These were at most DOS- and Windows-compatible, thanks to DOS’ and Windows’ ability to run on non-PC-compatible systems (with appropriate hardware interface layers). Many such computers took advantage of their non-PC-compatibility to provide improvements compared to the PC, e.g. more memory for DOS programs.
Stephens Answer points out most details, I belive it's worth to mention that the 80186 is not incompatible with the IBM-PC's structure/hardware per se. The CPU core works for all details like a 286 in real mode, with the same additional instructions and exceptions, as there are:
Instructions:
Exceptions:
So far software will run into the same incompatibilities as with an 286.
Further it's incompatible additional hardware aren't as much of an issue, depending on the design choosen.
As Stephen said, the included peripherals are superior to IBM's choice of 8 bit components. This is especially true for the DMA controller, able to transfer to any location at any length (up to 64 KiB).
While the address structure of the I/O block is complete different from the PC, it doesn't get in the way of any PC hardware as it it located at FF00h after reset. An area no PC-hardware (I know) occupies. Not even later one. It can be moved to any location in IO or memory address space.
The most obvious way to reach IBM-PC compatibility would adding everything exactly s the PC did. While this would eliminate many of the advantages of having integrated peripherals, but that sounds worse as it is, as most of a PC-I/O could be added as a single southbridge chip.
The only remaining incompatibility would be the Interrupt Controller (PIC), as it's located on a different address.
This is where the quite handy address decoder could be used. By connecting it's output to NMI and setting it to cover the I/O address range 0000h..03FFh, which is the range all I/O in the original PC was located, it would generate an NMI (*1) with each access (*2). Now an NMI handler can decode the offending instruction (*3) and translate it to the real hardware and back.
With 80186 systems running at least at 6 MHz or above, the translation layer's performance impact was tolerable. In fact, I do only remember one usage which could not be emulated, and that's active sound generation, that's were the CPU essentially handles the speaker in software. something already critical on genuine hardware.
The Olivetti Prodest PC1 of 1987 (!), based on a NEC V40 (a SoC much like an 80188, but with PC like timer/UART/PIO) did for use an NMI handler to emulate a PC compatible 8237 DMA controller while using the build in 20 bit DMA.
Long story short: It's quite possible to use an 80186 and its advantages while being mostly compatible.
The mentioned PC-D, being conceived as Unix workstation, did offer a different way. Here all memory and I/O was required to handle the READY signal, controlled by a watchdog timer provided by the 80186 as well. While this in theory could be used for emulation as well (I did so), the performance impact was rather heavy, as it only fired after about 1 millisecond. Eons in CPU time. The nice part was that the system ROM, much like MS-Windows, routed any NMI by default into the ROM based debugger. From there it was just a few commands until one could patch the offending software to run flawless :)
*1 - NMI is a strange fellow on the IBM-PC AT, anyway, as it could be masked. Jep, that's something one needs to read twice, as the very feature of an NMI is to be not maskable, so it can be used to report critical conditions (like memory error).
*2 - The same can be archived by simply selecting all I/O with top 6 bits zero, which needs a single TTL, leaving the PCS signals to other use.
*3 - While decoding an offending memory instruction, with just the address of the next instruction, as the NMI provides, is an quite complicated and rather error prone process, it's straight forward when it comes to I/O, as there are only 4 (well, 8 with INS/OUTS) opcodes to detect, and they have fixed formats.
To only possible ambiguity would be direct address (8 bit port number) instructions, but luckily That address range (0Exh) is unused on the PC. Similar the INS/OUTS do not collide either - well, they aren't to be expected with 8088 software at all.
Working from memory and in addition to what everybody else has said, the order of the memory access and SP increment/decrement in the push and pop opcodes changed relative to the 8086 and 8088.
This was the standard way of checking whether a CPU was an 88/86 or a 188/186. You could distinguish between the 88/188 and 86/186 by using self-modifying code to explore the length of the prefetch queue, and some related technique allowed the V20/30 to be identified.
I forget for the moment how the code that distinguished between a 188/186 and a 286 worked, and from the '386 onwards there was a CPUID opcode.
CPUID was only added in the Intel Pentium, and retro-fitted to the last 486s. Detecting a 386 or a 486 involved checking for new flags in the flags register. In the checks I’m aware of (and used back in the day), the PUSH SP difference was used to distinguish between <286 and 286, not 808x and 18x; the latter were distinguished using the SHL behaviour (truncating CL to five bits). Put another way, PUSH SP behaves the same on 808x and 18x, SHL behaves the same on 18x and 286.
Commented
Sep 7, 2020 at 14:56