28

In this question, by 'self-modifying code', I mean software that writes to a section of code that the CPU will very soon fetch and attempt to execute.

I am not here talking about the software engineering implications of self-modifying code, or about the security implications, but only about the implications for the design of fast, correct CPUs.

On the simplest microprocessors, self-modifying code is not a special problem. On an 8080, you can write just ahead of the instruction pointer, and a few clock cycles later, the instruction you just wrote, will be fetched and executed as though it had been there all along. But as pipelines and instruction cache are introduced, this becomes more problematic.

https://people.computing.clemson.edu/~mark/330/colwell/case_486.html

fetch - fetch 16 bytes of instructions from the single physically-addressed 4-way set associative 8KB cache into a prefetch buffer (providing about five instructions per fetch); use the two 16-byte buffers in a double buffered manner or use one for prefetching down a branch target path

On hardware like that, self-modifying code will not work by default. You could write new instructions to memory just ahead of the IP, only to find the CPU doesn't notice, because it has already fetched the next few instructions from that location.

On some CPUs, that's how it is, and self-modifying code needs an explicit flush to work reliably. But x86 makes it work transparently (or at least hitherto did; nowadays there is a push for W^X; but that's for security reasons, which is a separate issue). To that end, the CPU needs to incorporate extra circuitry that constantly checks for such writes, taking up a small amount of die area, and using up a small amount of electricity on every memory write.

Why?

Backward compatibility, is the obvious general answer. But backward compatibility with what software exactly?

A couple decades later, the answer would be obvious: JavaScript JIT compilers. But the 486 predates them.

For what software was the 486 in 1989, spending resources supporting self-modifying code? Was it just based on the general feeling that there was likely to be something out there that would care? Or was there a particular program, or category of programs, that used self-modifying code for some particular purpose, and was widely used?

18
  • 17
    I think you have to back up much further. The 486 was meant to be backward compatible with the 386 in protected mode, and with the 8086 in real mode. So in the latter case, you must ask yourself about the popularity of self-modifying code in 1978. Commented Dec 3, 2022 at 4:03
  • 17
    I don't think Intel was thinking about backward compatibility in terms of specific software. Compatibility was the main reason people were using x86, so it had to be backward compatible with all software. I suspect they learned their lesson with the 286, when a seemingly harmless departure from x86 behavior led to PCs having to implement the awful A20 gate hack. Commented Dec 3, 2022 at 4:05
  • 9
    A better question might be, why didn't Intel kill self-modifying code in the 386 protected mode, which didn't have to be backward compatible with anything. That takes us back to 1985. Ditto, why didn't AMD kill it for AMD64. Commented Dec 3, 2022 at 4:07
  • 4
    I can't confirm it now, but I have a vague recollection that, perhaps as far back as 8086, you had to perform a jump before executing the modified code, which would flush the prefetch queue. So the central problem for a modern CPU might be that we don't want to do a cache flush on every jump, and thus we resort to more elaborate mechanisms. Commented Dec 3, 2022 at 4:33
  • 3
    Note that paging on modern CPUs and operating systems is some sort of self-modifying code in a certain sense as well.
    – tofro
    Commented Dec 3, 2022 at 14:06

8 Answers 8

16

Self-modifying code was one of the few ways one could invoke a dynamically-chosen software interrupt.

High-level language toolchains targetting the MS-DOS platform sometimes provided an ‘invoke interrupt’ subroutine in their standard libraries that allowed user code to invoke arbitrary software interrupts with register values of the programmer’s choosing without writing any assembly code. This was int86 in Borland and Microsoft C, and Intr in Borland’s Pascal compilers. Although the interrupt number was usually a hardcoded constant in source code, compilers of the time did not have sophisticated inlining, register allocation or constant-folding that would allow emitting an appropriately hardcoded interrupt instruction directly; those were regular subroutine calls. This means an implementation of such a subroutine had to somehow invoke an interrupt vector whose number was passed as a function argument.

However, x86 has no instruction that allows calling an interrupt whose number is stored in a register or memory operand. Given that, such a dynamic interrupt invocation routine had either to use self-modifying code to generate an appropriate int instruction at runtime, or to reimplement the opcode’s logic in terms of a pushf / call far pair. The latter technique, however, later turned out to play poorly with the virtual 8086 mode, in which int instructions triggered traps into the virtual 8086 mode monitor, while pushf / call far pairs did not. As such, implementations of this kind of routine tended to prefer self-modifying code.

This fact is remarked upon in Unauthorized Windows 95 (pp. 277–278, 281); Intel must have been aware of it as well. While it was possibly not the only reason to support self-modifying code (AARD code comes to mind as well), I suspect it was probably the single most common use thereof, and it alone would have been decisive even just on its own.

35

Raffzahn has the main idea: once self-modifying code had been supported in the past, it had to continue to be supported in perpetuity.

But just to go back a little further, https://stackoverflow.com/questions/57115764/how-does-the-jmp-instruction-affect-the-8086s-working has some good information.

The 8086 had a natural reason to allow self-modifying code because, well, it was 1978 and memory cost several dollars per kilobyte. The only requirement was you had to execute a jump to flush the prefetch queue, which was about the only cache-like mechanism that the 8086 had.

Fast forward to 1985: the 80386 and its 32-bit mode was the next place where 8086 compatibility was deliberately broken. At this point, it would have been possible to forbid self-modifying code. It probably wasn't so popular by then. But there also wouldn't have been any clear reason to disallow it. The 80386 still didn't have L1 cache or pipelines or any such thing, so self-modifying code still just worked as long as you did the jump, without really requiring any extra effort from the CPU designers to support it. And from that time forward, we were stuck with it; Intel learned from the 286 that there was so much x86 code out there that every obscure wart was sure to be essential to some program somewhere, and so even the most minute incompatible change was perilous.

Things got tough with later CPUs as we picked up instruction and data caches, out-of-order execution, uop caches, and all that. The semantics had to be maintained: self-modifying code couldn't require anything more than a jump. But by this point, you couldn't possibly afford to flush all that state at every jump. So we got fancier mechanisms like cache snooping to detect which writes actually did modify upcoming code, and only in that case did the expensive "pipeline nuke" have to happen.

18
  • 11
    Anything that loads code into memory is 'self-modifying' in some sense, so you can't actually forbid it. But explicit flushing is not generally a problem in such cases.
    – dave
    Commented Dec 3, 2022 at 13:06
  • 9
    Prior to the Pentium CPUs, the most common form of self-modifying code was floating-point emulation: the program would be compiled with interrupt calls everywhere that a floating-point instruction was needed. When one of those interrupts was hit, it would either be replaced with the appropriate FPU instruction (if an FPU was present), or the emulation code would run (if it wasn't). The Pentium made this obsolete by always including an FPU.
    – Mark
    Commented Dec 3, 2022 at 21:55
  • 6
    @another-dave According to reverseengineering.stackexchange.com/a/12277 the original 8086 didn't have a trap for invalid instructions.
    – Neil
    Commented Dec 4, 2022 at 0:54
  • 3
    @Joshua: What kernels do you have in mind that actually emulate FPU instructions? That's not at all how it works under Linux; when a rare piece of code (like software RAID5 or RAID6) wants to use those registers, it calls kernel_fpu_begin() to trigger a save of the FPU/SIMD state. Then SIMD instructions like vpxor can run without corrupting user-space state. See Why am I able to perform floating point operations inside a Linux kernel module? - getting this wrong leads to silent corruption of user-space, not trapping to emulation. Commented Dec 4, 2022 at 13:16
  • 3
    @Joshua: Is it possible you were misinterpreting some details or something, and it was actually saying BSD was changing from lazy to eager FP context saving? So it saves/restores on every user-space context switch (but still not on interrupts or system calls). Linux used to do lazy FP context switching, and set a control register bit so FP instructions would fault and trigger restore of the FP state for this process. But with more processes using SSE2 all the time (e.g. memcpy), and higher cost of interrupts vs. extra stores, it doesn't futz around with that. I'd expect BSD does the same. Commented Dec 4, 2022 at 20:01
22

The support for self-modifying code was not specifically built as a feature, so it was not specifically allowed or prevented.

It just was possible due to the architecture of the CPU and the system. To put it in more general fashion, there was no distinction between code and data memory spaces, which of course allows to build specific systems with code in ROM and data in RAM and no way to load code to RAM for execution, but it also allows to build a generic system which allows executable code to be loaded from some storage.

And it was already more complex to do self-modifying code on x86 CPUs than on many other CPUs, due to the memory being read into a prefetch queue first before executing the code from prefetch queue. Many other CPUs simply fetched the opcode bytes directly when needed.

Even on the 8088, you had to specifically organize your code so that self-modifying code works as expected, and same code would not work on 8086 or 80286 or 80386 unless the code was adapted to support them due to the prefetch queue size differences.

Also, you can't really assume that a JIT compiler would somehow work right on the edge and generate native CPU opcodes one at a time, in real-time, right before they are fetched for execution. They obviously work in larger chunks at a time, such as per-file, per-function or per some suitable block of code, as the JIT compiler would have to anyway run itself to generate the executable opcodes, and maybe even analyzing what blocks are run often enough to spend time optimizing them and keep them in memory, and which blocks are not important so they don't need to be optimized or kept in memory.

19

I'm going to dispute the basis of the question, to at least a minimal degree.

The original 8086 and 8088 both included prefetched instruction queues. If you modified an instruction that was already in the PIQ, your modification was not take into account--the instruction in the PIQ was still executed as it was originally fetched.

In fact, if you needed to determine whether code is executing an 8086 or 8088, the standard way to do so depended on the fact that the 8088 has only an 4-byte PIQ, while the 8086 has a 6-byte PIQ.

So, to distinguish between them, you'd typically use self-modifying code to change the target of a jmp 4 bytes ahead of the current location, so the offset of the jmp was in one (or both) of the two bytes that would be in an 8086 PIQ, but not in an 8088 PIQ.

So, the code looked vaguely like this (probably not working as it stands--last time I wrote code like this was probably at least 35 years ago).

mov ax, offset _88 - branch_offset
mov jump_offset+1, ax
nop ; enough NOPs to fill all but the last slot in the 8088 PIQ
nop
nop
jump_offset:
; The tricky part: when the preceding `mov` executed, the offset of this
; jmp was in the 8086 PIQ, but not in an 8088 PIQ
jmp _86 ; so on an 8088, the CPU will use the modified offset,
        ; and jump to _88

xor ax, ax ; we should never get here
ret

_86:
    mov ax, 8086
    ret

_88:
    mov ax, 8088
    ret
5
  • 6
    Right. Self modfiying code often used techniques that flushed the prefetch queue (e.g. running JMP $+1, which IIRC worked at least up to the 80286, as it was included in the standard code to switch to protected mode) to ensure they ran the updated code.
    – occipita
    Commented Dec 3, 2022 at 23:30
  • 3
    Yes, that twist needs to be remembered as well. after all, self modifying code always had to take care of cache, prefetch and pipeline - on all CPU and alwasys.
    – Raffzahn
    Commented Dec 4, 2022 at 0:07
  • 2
    @occipita That should be jmp short $ + 2, not plus 1. The answer is also wrong because its example is missing the branch_offset label after the jump instruction, plus it depends on the jmp being near where an optimising assembler may easily turn it into a short jump.
    – ecm
    Commented Dec 4, 2022 at 11:19
  • I guess you'd do this with interrupts disabled, otherwise that could flush the prefetch queue earlier than expected on 8086. (Also, not that it matters, but instead of modifying a jmp rel8, it could be an add al, 0 or 2. In real code you'd probably be branching on the result instead of returning an integer so this is a good example.) Commented Dec 11, 2022 at 22:01
  • 2
    @PeterCordes: Yeah, you'd want interrupts disabled. As far as the details go...yup, lots of variation possible. Commented Dec 11, 2022 at 22:57
12

Preface:

Avoiding '(self) modified' code is impossible on computers allowing to load code (see section at the End)


The Question

Backward compatibility, is the obvious general answer. But backward compatibility with what software exactly?

There may have been some PC/PC-XT .COM program using self modifying code to shave a byte or a cycle. But I don't think they had any special software they had in mind. After all, those enhanced CPUs were designed way before any incompatibility could have been detected.

Also you don't have to look for specific 486 code, that box was already opened with 8088 PCs. At least in case of IBM it's easy to see the see the effort they invested to emulate 8088 ISA access with the AT or the addition of the ever dreaded A20 gate. In Either case it would have been easy to dismiss the fringe cases those hacks fixed. With compatibility being a holy grail, continued support for such practice was a must.

It was the fear of problems due incompatibility not any experienced/known incompatibilities

Now, backward compatibility aside, generated code is a very common feature in non trivial software. Think RegExp and data base queries (*1).


Computers Use Self Modifying Code All the Time

It's important to keep in mind, that any computer that allows to load an arbitrary program already needs the ability to modify code.

  • Loading a program, modifies that locations. Usually they did hold code prior - like the last program run.

This is not only about programs loaded by a monitor, but as well

  • overlays - which in extreme may mean calling a function at exactly the same memory location as the calling instruction.

Aside from such generic replacements there is generated code - that's not just some fancy modern JIT (*1):

  • Every compiler produces new code to be executed
  • Likewise using a debugger

While one could argue that all of these cases have many layers of execution inbetween generation an calling(*2), there are other quite common ways of runtime generation within applications falling into the JIT category - except predating JavaScript by several decades - as what we call now JIT was present in many applications that do have to work with ad hoc queries on large amount of data. Examples are

  • Regular Expressions - already in 1968 QED for MULTICS generated code so evaluation could be speed up
  • Data Base Queries - it was (and maybe still is) rather common to translate queries at run time into machine code

or, more modern, but still 1980s,

  • Microsofts CBLT or Compiled BLock Transfer - To make BitBliting somewhat performant, Windows GDI generated tight screen handling loops on stack (see this description) (*3).

At the lowest end, there is in addition the

  • need to modify immediate operands of instructions

That case might be most prominent on x86 with the interrupt number when intending to call an arbitrary service via an INT instruction (*4).

In fact, it can be assumed that, in many cases, such modification only happened quite short before the invocation, using something like

        ...
        MOV AL,DesiredService
        MOV INTNR,AL
        MOV AX,axContent
        INT 0FFh
        ORG *-1
INTNR   DB  0
        ...

*1 - JavaScript it rather a late comer in that.

*2 - I wouldn't, as there is no guarantee that such generator isn't running into the generated code right after modifying the last byte (which as well can be the first in that section).

*3 - Thanks to occipita for adding this example.

*3 - Thanks to Natan Eldredge for reminding us.

4
  • 8
    One that I remember is Turbo C++'s int86() function. Since the INT instruction takes an immediate, but int86 wants the interrupt number as a runtime argument, they use self-modifying code. The alternative is to waste some 1024 bytes by coding out all 256 possibilities and jumping to the right one. Commented Dec 3, 2022 at 4:35
  • 2
    @NateEldredge: Another alternative approach for implementing INT would be to push the flags and desired return address, followed by the contents of the interrupt vector to use, and then perform a far return.
    – supercat
    Commented Dec 3, 2022 at 7:33
  • @NateEldredge Jup, that's the usual case of generated code - thanks for the reminder.
    – Raffzahn
    Commented Dec 3, 2022 at 15:32
  • 7
    Probably the most commonly-used example of self-modifying/generated code on 16-bit PCs was Windows' BitBlt API, used for drawing images and copying areas of screen around, which used code generated on the stack segment to perform the requested drawing operations. See Raymond Chen's description here: devblogs.microsoft.com/oldnewthing/20180209-00/?p=97995
    – occipita
    Commented Dec 3, 2022 at 23:38
6

"Real" self-modifying code is seldom used on x86. There are some exceptions, like the int86() example in Raffzahn answer but in most cases, it is not quite self modifying as some 8bits CPUs had to do, things like changing on the fly instructions immediate parameters for optimisations.

Sometimes programs were patched "live", this was the case for fast FPU emulation, and maybe also early VMWare virtualisation which had to do many very complex things to work. But I'd argue that it's not really self-modifying code, its some external program or system library (such as FPU library) that modifies another program. Debuggers also modified the code to insert INT3 breakpoint instructions.

Each time a program is loaded, that the linker updates branch addresses, or code is generated by a JIT, some data memory becomes executable code. Besides eventual MMU update to allow the execution of that code, a variety of operations need to be done to ensure that the new code will be correctly loaded, it goes with synchronisation instructions, flushing prefetch buffers,... and also some way to ensure cache coherency between instructions and data. In some simple RISC CPUs, that coherency is managed by explicit cache flushes, in x86 and all modern, high performance CPUs, cache coherency is managed by the hardware which is far more efficient than forced flushes.

x86 CPUs need to be compatible with the earliest members of the family, which had no cache (8088/86, i286 and i386). So cache controller had to be clever enough to hide coherency issues between instructions and data to maintain compatibility. In the i486 the problem was not yet very complex as it had a shared instruction and data cache. But it became a bit more elaborate with later models with split caches.

The initial requirement for compatibility became an needed feature for efficient JIT execution and overall better performance.

1
  • 3
    live patching is also very commonly done nowadays, with Windows having mov edi, edi and Linux having NOPs at the function entries
    – phuclv
    Commented Dec 4, 2022 at 14:33
5

A couple decades later, the answer would be obvious: JavaScript JIT compilers. But the 486 predates them.

It predates Javascript JIT compilers, but there were plenty of other virtual machines for dynamic languages that had similar capabilities. And even without JIT, such VMs still regularly used self-modifying code, for example in the use of inline caching:

The canonical implementation [1] is a register load of a constant followed by a call instruction. The "uninitialized" state is better called "unlinked". The register is loaded with the message selector (typically the address of some object) and the call is to the run-time routine that will look-up the message in the class of the current receiver, using the first-level method lookup cache above. The run-time routine then rewrites the instructions, changing the load instruction to load the register with the type of the current receiver, and the call instruction to call the preamble of the target method, now "linking" the call site to the target method.

This technique was first used in the VM of Smalltalk, which was originally developed in the late 70s and commercialized in 1980 and thus predates the development of the 80486 by many years. The use of such techniques would have been considered by the designers of the 486.

3

I first learned to compute in the late sixties. Here are some personal memories of what ordinary programmers were doing at the time.

Back in the seventies, many machines had (or appeared to have) a Von Neumann architecture, with one bus for data and addressing. The PDP-11 had a particularly neat instruction set for the time. A lot of programming was done in PDP-11 machine code.

The other alternative was the Harvard architecture, with a separate bus for address and data. This would seem more efficient. However, programs often used to write all variables but the handful it could keep in registers (or cache if you were lucky enough to have one) to drum or disk memory. I wrote programs in the early eighties on a HP machine where you could re-segment a stripe on the disc so the number you wanted should be just coming around on the disc when wanted for it. Having two buses did not make things that much faster.

Back then there was a serious discussion whether compilers were a good thing. They were good for housekeeping, but the hard calculations were so much faster in machine code, and machine code could not be wrong because it was what the machine did (moving a program from one machine to another was hard back then). I remember a Xerox PARC paper from the seventies (had a paper copy but threw it out years ago, sorry) that proposed using machine code only. If I were laying out an x86 chip back then, I would try hard to make it able to cross-compile the millions of lines of PDP-11 code and have it run exactly as the PDP-11 did. A new processor is only useful when you have code to run on it.

In all these years I only remember one genuinely useful case of rapidly self-modifying code in commercial software (reading images off tape, PDP-11, late eighties), and I replaced that with a byte addressed look-up table, folded to be addressed by signed bytes. Distinguishing an 8086 from an 8088 (I did use both but never met this) is number two.

Then things went pretty much as the other answers describe. Processors got faster, and could emulate self modifying code by flushing the cache, and still match the old compute speeds.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .