Using INT comes not only natural due the way the 8086 is designed, but was as well intended by Intel as OS entry point, much like a Supervisor Call (SVC) on /360 type mainframes:
(Excerpt from the October 1979 Intel 8086 Family User's Manual page 2-28.)
Software-initiated interrupt procedures may be used as service routines ("supervisor calls") for other programs in the system. In this case, the interrupt procedure is activated when a program, rather than an external device, needs attention. The "attention" might be to search a file for a record, send a message to another program, request an allocation of free memory, etc. Software interrupt procedures can be advantageous in systems that dynamically relocate programs during execution. Since the interrupt pointer table is at a fixed storage location, procedures may "call" each other through the table by using software interrupt instructions . This provides a stable communication "exchange" that is independent of procedure addresses. The interrupt procedures may themselves be moved so long as the interrupt pointer table always is updated to provide the linkage from the "calling" program via the interrupt type code.
INT is intended to move address dependencies from physical to logical, offering an abstract interface to services. Exactly what BIOS and DOS is. Using INTs for either is simply as it should be.
The Long Read
Access to the DOS API was done through the INT 21h x86 instruction. This was always counter-intuitive to me, coming from 8-bit systems that accessed system services by calling subroutines through a jump table.
For one, this is as well known for 8 bit, like with 8080/85/Z80 systems using the RST instruction. But more important, the interrupts are exactly this, an indirect subroutine jump using a jump table. It got several advantages:
- Short two byte opcode vs. six byte for indirect far call
- Fast execution due to fixed address
- Portability as the address is not coded within the user program
- Use of a logical number that could be redirected in future versions
- Executing an INT is independent from the address mode the application or the OS runs in
- Taking the table out of user memory improves compatibility
- It's the most upward compatible design possible, as new CPU generations can use this as hook for task switch and alike without breaking compatibility
The only 'cost' may seem that the flag register gets saved in addition. In reality this simplifies the OS interface even more, as the flag word is now located at a fixed address (SP+4), so its content can easy be manipulated for return information - like carry set for any error. The function handler has not to take care of producing the right flags from some artificial source just before return (like on many 8 bit OS's), but simply sets a bit in a defined memory location, the rest is done by hardware.
This simple system seems to give the same benefits from indirection as using software interrupts on x86.
Not really, as for a call the user software needs to know the address of said table, which makes it quite hard to move or virtualize it in future CPU/OS versions.
An example of such a system is the kernel (or "Kernal", for Commodore purists) used on Commodore's 8-bit machines.
Commodore is a great example how fixed entry points complicates development of software. Most 6500 based commodore Kernals provide the same functionalities but located at different entry points. Software needs to be ported for each machine. Given, it often can be done with a few switches and recompilation, but writing software that can run on more than one or two machines needs to bring a compatibility layer.
Additionally, the reliance on software interrupts might be a reason for why the transition to using Protected mode, and accessing more than 640KiB of memory, was so slow and difficult for MS-DOS users.
Why? Do you have any proof thereof? The INTs were of no problem, as they work quite well in protected mode. The CPU handles everything necessary - which is exactly the reason why INT had to be used in the first place, it allows a simple hook for upward compatibility. Use of INT instructions for any userland -> OS call in applications is the base for transparent move to a protected mode system.
The real issue with DOS applications wasn't the INT system, or anything about the CPU, but applications breaking two basic rules of well behaving: Hardware assumptions about memory management and direct hardware access in general, both without using any protocol. Especially the first one is what made most of the DOS problems, the assumption that
Segment+1 is the same as
Offset+16. Within a protected mode system this is no longer true. As a result, any program trying to 'outsmart' the OS will fail.
Why didn't MS-DOS just use the CALL instruction with a jump table instead of software interrupts?
See above, using a call table would give up all the advantages - and in most cases welcome the associated disadvantages.
Did this choice impact MS-DOS programs being able to run in protected mode on the 80286+?
No. It was, as said, direct hardware access, including memory management. Disrespecting the logic structure of the CPU is what broke code and made the move to protected mode basically impossible.
Bottom line: Adding the INT instruction is one of the best decisions when making the 8086, full in line with the goal to create a CPU made for complex high level software. Using it was the right way to do.
Maybe Intel should have called it 'SVC' like 20+ years before Amdahl did for the IBM /360, where opcode X'0A'
SVC worked similar (*1) - and enabled compatibility across OS versions over many decades from real mode single CPU machines all the way to virtual multi-processor systems and 64 bit code.
Then again, the 8086 is a simple microprocessor, so it makes sense to combine hardware interrupts and CPU exceptions with OS/function calls into a single mechanic.
The only dark shadow, cast over the INT use, was due IBM's decision to use the interrupts below 20h for BIOS functions, as they were reserved by Intel for CPU exceptions.
*1 - Well, not completely, at least on early machines it used the default interrupt and the OS had to decode and jump.