Self-modifying code was one of the few ways one could invoke a dynamically-chosen software interrupt.

High-level language toolchains targetting the MS-DOS platform sometimes provided an ‘invoke interrupt’ subroutine in their standard libraries that allowed user code to invoke arbitrary software interrupts with register values of the programmer’s choosing without writing any assembly code. This was int86 in Borland and Microsoft C, and Intr in Borland’s Pascal compilers. Although the interrupt number was usually a hardcoded constant in source code, compilers of the time did not have sophisticated inlining, register allocation or constant-folding that would allow emitting an appropriately hardcoded interrupt instruction directly; those were regular subroutine calls. This means an implementation of such a subroutine had to somehow invoke an interrupt vector whose number was passed as a function argument.

However, x86 has no instruction that allows calling an interrupt whose number is stored in a register or memory operand. Given that, such a dynamic interrupt invocation routine had either to use self-modifying code to generate an appropriate int instruction at runtime, or to reimplement the opcode’s logic in terms of a pushf / call far pair. The latter technique, however, later turned out to play poorly with the virtual 8086 mode, in which int instructions triggered traps into the virtual 8086 mode monitor, while pushf / call far pairs did not. As such, implementations of this kind of routine tended to prefer self-modifying code.

This fact is remarked upon in Unauthorized Windows 95 (pp. 277–278, 281); Intel must have been aware of it as well. While it was possibly not the only reason to support self-modifying code, I suspect it was probably the single most common use thereof, and it alone would have been decisive even just on its own.

Self-modifying code was one of the few ways one could invoke a dynamically-chosen software interrupt.

High-level language toolchains targetting the MS-DOS platform sometimes provided an ‘invoke interrupt’ subroutine in their standard libraries that allowed user code to invoke arbitrary software interrupts with register values of the programmer’s choosing without writing any assembly code. This was int86 in Borland and Microsoft C, and Intr in Borland’s Pascal compilers. Although the interrupt number was usually a hardcoded constant in source code, compilers of the time did not have sophisticated inlining, register allocation or constant-folding that would allow emitting an appropriately hardcoded interrupt instruction directly; those were regular subroutine calls. This means an implementation of such a subroutine had to somehow invoke an interrupt vector whose number was passed as a function argument.

However, x86 has no instruction that allows calling an interrupt whose number is stored in a register or memory operand. Given that, such a dynamic interrupt invocation routine had either to use self-modifying code to generate an appropriate int instruction at runtime, or to reimplement the opcode’s logic in terms of a pushf / call far pair. The latter technique, however, later turned out to play poorly with the virtual 8086 mode, in which int instructions triggered traps into the virtual 8086 mode monitor, while pushf / call far pairs did not. As such, implementations of this kind of routine tended to prefer self-modifying code.

This fact is remarked upon in Unauthorized Windows 95 (pp. 277–278, 281); Intel must have been aware of it as well. While it was possibly not the only reason to support self-modifying code (AARD code comes to mind as well), I suspect it was probably the single most common use thereof, and it alone would have been decisive even just on its own.