Does the Intel 8086 CPU have user mode and kernel mode as modern CPUs do? and if it doesn't have user mode and kernel mode, does that mean that any user program written for the Intel 8086 CPU could do anything that the OS can do?
No. There is no mechanism for any privilege levels or protection in 8086. As a consequence, there is nothing special about OS code, and thus user applications are allowed to do everything, including reading and writing to any physical memory address, directly access any I/O port, and enable/disable interrupts at will.
Protected mode was introduced in 80286. Some non-PC compatible systems with 8086/8088 might have had external hardware that was used to provide protection and distinction between kernel and userland code, but I haven’t found really good sources for those, so I can’t tell how it worked.
To answer the second part of the question, yes, any program could do what the OS could do. Some of the useful reasons included:
Directly access hardware, especially to provide support for third-party expansion cards.
Directly access graphics memory, to draw graphics.
Overwrite system calls, to alter or extend the functionality of the OS. Certain TSR programs that did this were especially popular.
Does the Intel 8086 CPU have user mode and kernel mode…
…as modern CPUs do?
I guess you mean "modern" x86 CPUs, the offspring of the 8086, right? Alas, the idea of an ISA with privilege levels is way older than the 8086.
and if it doesn't have user mode and kernel mode, does that mean that any user program written for the Intel 8086 CPU could do anything that the OS can do?
And that's what brought us the mess of direct screen addressing, user I/O, mangling with interrupts, and so on. These things made life hard for any proper OS to offer compatibility with DOS programs—at least until CPUs got fast enough to just emulate everything.
The 8086 has no programmer-visible supervisor/user model in the instruction set architecture.
However, it exposes a lot of state information through its pins, such as pins S0, S1 and S2 that interface to a bus controller (like the Intel 8288).
The S0, S1 and S2 pins indicate what kind of access the 8086 is making: code, data, I/O and so on. Together with the address pins, you can get an idea of what the CPU is doing: in what area of memory is it executing code and what it's trying to access.
It seems very promising, therefore, that a supervisor/user-like protection scheme is possible if we control the board design. We could design a piece of hardware (to augment or replace the bus controller) which decodes the processor state from the S1-S2 pins, address pins and perhaps others, and acts as a gatekeeper to the external bus, preventing accesses to certain areas by unprivileged code. That piece of hardware could itself be programmable in various ways, and only by kernel code. It could be told to reveal certain address ranges, or close off access. It could deliver an external interrupt to the processor if it attempts an illegal access.
No, the original 8086 only has real mode, which is maintained in all later x86 CPUs. In this mode, memory is only accessible using physical addresses, although it's still not exactly a flat address space because of segmentation.