46

Preface: This question does in part intersect with Use of undocumented opcodes, but targets especially the 8086 instruction handling.


I was reading Tanenbaum's "Operating Systems, Design and Implementation" 3rd edition (The MINIX Book), where I encountered the following quote that surprised me:

For instance, the 8086 and 8088 processors do not support detection of illegal instruction operation codes, but this capability is available on the 286 and above, which trap on an attempt to execute an illegal opcode.

What did the 8086 and 8088 do when it an encountered an illegal instruction?

  • 2
    @StephenKitt I'd err on the side of not closing it if the questions themselves aren't identical; it's possible that there are answers to this one that wouldn't be answers to the others, and I know for sure that the other way around also applies. – wizzwizz4 Jun 11 at 17:31
  • 2
    @wizzwizz4 I guess that’s something we haven’t decided yet as a community. Most SEs seem to take the approach that a question is a dupe if the dupe target’s answers provide an answer to the question (in its entirety), even if the questions aren’t identical; this corresponds to the introductory text in the yellow frame on duplicates, “This question already has an answer here:”. In this case, the question is fully answered by the target’s answers. The target itself wouldn’t be marked as a dupe so the lack of a bijection doesn’t matter. – Stephen Kitt Jun 11 at 17:42
  • 2
    @wizzwizz4 Basically, if I were to copy-paste my answer to the target question here, then IMO that would be a perfectly valid answer here too. In my mind that means either my answer is inappropriate for the target question, or this question is a dupe. – Stephen Kitt Jun 11 at 17:44
  • @wizzwizz4 If you prefer I can copy-paste my answer here, but it seems a bit of a waste. – Stephen Kitt Jun 11 at 17:44
  • 1
    @Raffzahn yes, I agree, I retracted my vote a couple of days ago. – Stephen Kitt 2 days ago
50

Illegal opcodes were just instructions that hadn't been fully defined by the chip designers – a little like Undefined Behaviour in C, but much more predictable. Many people called these "undocumented instructions", because they functioned just like ordinary instructions, on the particular versions of the particular chips on which they are found. There was no special handling to prevent these instructions from executing.

Most of them were just NOPs (either because they weren't wired up to anything or because they did stuff like writing a register to itself) or duplicates of other instructions (because the instruction decoder ignored some bits when it didn't need to pay attention to them), but some of them were more interesting.

For instance, SALC, which does:

if (carry flag set)
  AL = 0xFF;
else
  AL = 0x00;

This can be used as a translation layer between C code and certain assembly returning conventions. It is equivalent to SBB AL, AL, except that it does not clobber flags.

For more information, see:

40

It might be better to think about it this way:

On the 80186 and above, a new thing was defined called an "illegal instruction", and this new thing came with a new behavior -- a #UD exception that was generated when one was encountered. Before the 80186/80286, there was no such thing as an illegal instruction, just undocumented ones.

Your question then becomes, "what did the 8086 and 8088 do when encountering undocumented instructions" and the answer is simply that the behavior was undocumented. Some of them appeared to do nothing (i.e. they produced the same apparent result as a NOP instruction), while others did odd things, or simply the same thing as another opcode.

Here's a WayBack link that may shed more light on the situation: http://web.archive.org/web/20190321200321/http://www.os2museum.com/wp/undocumented-8086-opcodes-part-i/

  • 2
    I found that "POP CS" is useless kinda funny. I used it. It can be used to perform a long jmp instruction. It's most useful when moving your code's base. At the end of the base move you would do "POP CS" to jump to the next instruction in your code in the new location. – Joshua Jun 12 at 17:37
  • @Joshua The "pop cs" idiom was common in many dos viruses, for the reason you give. – Hasse1987 Jun 13 at 23:17
3

To understand this, you need a general idea of how processors work at the raw hardware level (or at least how they worked before the concept of microcode was developed).

Basically, as the processor would read the bytes of a single assembler language instruction, those bytes would be fed as an input to a network of logic gates called the instruction decoder; this network was designed to produce as output the signals to enable or disable various internal components of the processor according to what the instruction was supposed to do, so that the processor ends up doing the required operation.

As a result, if you viewed the instructions of older, simpler processors as bit patterns, you might notice certain groupings; for example, if the top three bits of the instructions would be set so, you might know that it must be a jump instruction of some type; other patterns might identify other general types of instructions.

The design team for a new processor architecture would develop a reasonable set of instructions, the various networks of logic gates to implement them, and the instruction decoder to manage them all. Back when the 8086 and 8088 processors were designed, it was not yet economically viable to add extra logic to the instruction decoder to exclude any "undocumented instructions", so any bit pattern fed into the instruction decoder would do something, but only the documented instructions would be guaranteed to exist and work the same also in the future members of the same processor family.

Sometimes the undocumented instructions actually ended up doing something useful, but relying on them was risky, as it might mean any program using a particular undocumented instructions might only run successfully that particular processor model. On a newer model, the previously undocumented instructions might actually been used for some new functionality, or they might have become completely useless as a result of a redesign of the instruction decoder.

With 80186 or above, using a number of logic gates to trap the undocumented instructions was economically feasible, and it became a compatibility feature: a program might prefer to use a new instruction available in newer processors only, but could provide a routine to emulate that instruction in software within an "illegal instruction" trap handler.

  • Those patterns in the instruction encoding are very obvious in MIPS, for example, which was designed so the bits of the instruction encoding could be used fairly directly as internal control signals. And it only has a few instruction formats (3-register ALU, or I-type with a 16-bit immediate, or J-type for unconditional jumps). It's word-oriented not byte-oriented. But basically all ISAs with fixed-width instructions have some opcode bits and some register-number bits in consistent locations across many instructions. – Peter Cordes Jun 14 at 4:02
  • For more about x86 opcode patterns, see Are x86 opcodes arbitrary? Also, 3-bit groups come up in some, so octal is apparently helpful. – Peter Cordes Jun 14 at 4:04
0

Based on the other answers, the proper answer is: they do whatever the instructions bits tell them too. Since apparently they aren't validated, they always do something, even if that something is nothing.

Same as trying to turn on a lamp on an illuminated room- you won't notice it is there and it is on.

New contributor
pedro is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • 1
    You could ask your question as a new question — as it is, your answer is likely to be flagged as “not an answer” ;-). – Stephen Kitt Jun 12 at 15:12
  • 1
    It knows its length the same way it "knows" what the instruction does: the wiring causes it to increment the instruction register by however much it needs to. – wizzwizz4 Jun 12 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.