In this question, by 'self-modifying code', I mean software that writes to a section of code that the CPU will very soon fetch and attempt to execute.
I am not here talking about the software engineering implications of self-modifying code, or about the security implications, but only about the implications for the design of fast, correct CPUs.
On the simplest microprocessors, self-modifying code is not a special problem. On an 8080, you can write just ahead of the instruction pointer, and a few clock cycles later, the instruction you just wrote, will be fetched and executed as though it had been there all along. But as pipelines and instruction cache are introduced, this becomes more problematic.
fetch - fetch 16 bytes of instructions from the single physically-addressed 4-way set associative 8KB cache into a prefetch buffer (providing about five instructions per fetch); use the two 16-byte buffers in a double buffered manner or use one for prefetching down a branch target path
On hardware like that, self-modifying code will not work by default. You could write new instructions to memory just ahead of the IP, only to find the CPU doesn't notice, because it has already fetched the next few instructions from that location.
On some CPUs, that's how it is, and self-modifying code needs an explicit flush to work reliably. But x86 makes it work transparently (or at least hitherto did; nowadays there is a push for W^X; but that's for security reasons, which is a separate issue). To that end, the CPU needs to incorporate extra circuitry that constantly checks for such writes, taking up a small amount of die area, and using up a small amount of electricity on every memory write.
Backward compatibility, is the obvious general answer. But backward compatibility with what software exactly?
For what software was the 486 in 1989, spending resources supporting self-modifying code? Was it just based on the general feeling that there was likely to be something out there that would care? Or was there a particular program, or category of programs, that used self-modifying code for some particular purpose, and was widely used?