While Win 3.x operating system stuck to the 80286 platform,
Not really as Windows was since 2.1 available in two versions:
- Windows/286 and
- Windows/386
It got only unified by name with Windows 3.0. Unlike the name suggests, the difference wasn't 286 vs. 386, but real mode vs. protected mode. Protected mode was only used on 80386-class CPUs and only for Windows itself. On a 286 Windows ran the same code (and mode) as on a 8086. Applications always used (virtual) real mode.
did applications compiled to Win 3.x have to use 80286 instructions explicitly
They could use the added user-mode instructions in either mode
or be aware of the 80286 memory layout?
Not at all. Beside that it's the whole idea of Windows to hide those things, memory layout on a 286 was exactly the same as for 8086 - and in all practical means for 386, as here user tasks did run in virtual 86 mode, so no chance to 'see' anything difference in memory layout.
Or as long as the source code didn't use 80286-exclusive assembly, could applications be compiled with a 8086 compiler
Well, the default compile option for the SDK was 8086, but easy to override. Which I'm pretty sure many did to take advantage of any possible speedup.
and leave the protect-mode stuff to the OS and be transparent?
User processes were never to access any protected mode instructions.
Windows offered a standardized environment independent of hardware including CPU, a single set of calls for everything about the environment, from memory to graphics.
All Windows interfaces were 16 bit and its memory layout was 8086 alike no matter on what (sub)version.
It wasn't until Win 3.1 offered the (optional) Win32s interface that it started to support 32-bit calls.
This is not to say there were no differences. The 386 variant did offer a native EMS emulation mode, while the (2)86 used an existing HIMEM driver. While either is transparent to the user program, I do remember that it was considerably faster on a 386.
If the latter is true, how often was the case?
That's hard to guess. Since it was up to each developer to decide which CPU to target, only a look at each and every package would help here. Due to the abstraction offered by Windows, it didn't really matter. What mattered was speed and accessible memory, so lifted requirements were at that time less about the CPU itself, but the way extended/expanded memory was handled.
There seems to be a little mix-up between '286 Instructions', 'Memory Layout' and 'Protected Mode', as well as 'compiling with/without 286 features'. These are independent issues.
286 Instructions
The 286 not only brought its protected mode and the (privileged) instructions that go with it but it also extended user mode instructions with
- Array Checking (BOUND)
- Extended Stack Handling
- ENTER, LEAVE, PUSHA, POPA, PUSH immediate
- Speed Up I/O
- Immediate Shifts / Rotates
- Immediate IMUL
So applications could have a benefit from being compiled for 286 (or anything other than basic 86) without conflicting with Windows in any way. In fact, that worked already under any prior version.