52

Preface: This question does in part intersect with Use of undocumented opcodes, but targets especially the 8086 instruction handling.

I was reading Tanenbaum's "Operating Systems, Design and Implementation" 3rd edition (The MINIX Book), where I encountered the following quote that surprised me:

For instance, the 8086 and 8088 processors do not support detection of illegal instruction operation codes, but this capability is available on the 286 and above, which trap on an attempt to execute an illegal opcode.

What did the 8086 and 8088 do when it an encountered an illegal instruction?

Improve this question
7
  • 3
    @StephenKitt I'd err on the side of not closing it if the questions themselves aren't identical; it's possible that there are answers to this one that wouldn't be answers to the others, and I know for sure that the other way around also applies.
    – wizzwizz4 ♦
    Jun 11, 2019 at 17:31
  • @StephenKitt Could you write up a Retrocomputing Meta question? You have a very good point, and I have a slightly-less-good rebuttal, but we really need to decide on this as a community.
    – wizzwizz4 ♦
    Jun 11, 2019 at 17:45
  • I keep wanting this question to be the lead line for a joke...
    – Tim Holt
    Jun 14, 2019 at 0:37
  • I just got it this for close review, and I have a hard time to decide. For one, it does tackle the issue for a specific CPU, while the other is more generic. But Stephen is as well right, both may trigger the same answers. But not due the way this one if phrased, but rather because the other is way to broad. Marking it as double would be good to target future readers toward the already given answers - better than copying. But then again, asking exactly for the 8086 is a question in its own right. No idea for a middle way, I vote leave open - maybe to incooperate the link into the question.
    – Raffzahn
    Jun 15, 2019 at 8:58
  • @StephenKitt After a bit rereading, I no longer see it as a duplicate, as the liked question is specific about undocumented opcodes and their usage in relation to documentes, while this is strictly about the 8086 and its handling.
    – Raffzahn
    Jun 15, 2019 at 9:30

4 Answers 4

Reset to default
57

Illegal opcodes were just instructions that hadn't been fully defined by the chip designers – a little like Undefined Behaviour in C, but much more predictable. Many people called these "undocumented instructions", because they functioned just like ordinary instructions, on the particular versions of the particular chips on which they are found. There was no special handling to prevent these instructions from executing.

Most of them were just NOPs (either because they weren't wired up to anything or because they did stuff like writing a register to itself) or duplicates of other instructions (because the instruction decoder ignored some bits when it didn't need to pay attention to them), but some of them were more interesting.

For instance, SALC, which does:

if (carry flag set)
  AL = 0xFF;
else
  AL = 0x00;

This can be used as a translation layer between C code and certain assembly returning conventions. It is equivalent to SBB AL, AL, except that it does not clobber flags.

For more information, see:

Improve this answer
46

It might be better to think about it this way:

On the 80186 and above, a new thing was defined called an "illegal instruction", and this new thing came with a new behavior -- a #UD exception that was generated when one was encountered. Before the 80186/80286, there was no such thing as an illegal instruction, just undocumented ones.

Your question then becomes, "what did the 8086 and 8088 do when encountering undocumented instructions" and the answer is simply that the behavior was undocumented. Some of them appeared to do nothing (i.e. they produced the same apparent result as a NOP instruction), while others did odd things, or simply the same thing as another opcode.

Here's a WayBack link that may shed more light on the situation: http://web.archive.org/web/20190321200321/http://www.os2museum.com/wp/undocumented-8086-opcodes-part-i/

Improve this answer
2
  • 3
    I found that "POP CS" is useless kinda funny. I used it. It can be used to perform a long jmp instruction. It's most useful when moving your code's base. At the end of the base move you would do "POP CS" to jump to the next instruction in your code in the new location.
    – Joshua
    Jun 12, 2019 at 17:37
  • @Joshua The "pop cs" idiom was common in many dos viruses, for the reason you give.
    – Hasse1987
    Jun 13, 2019 at 23:17
8

To understand this, you need a general idea of how processors work at the raw hardware level (or at least how they worked before the concept of microcode was developed).

Basically, as the processor would read the bytes of a single assembler language instruction, those bytes would be fed as an input to a network of logic gates called the instruction decoder; this network was designed to produce as output the signals to enable or disable various internal components of the processor according to what the instruction was supposed to do, so that the processor ends up doing the required operation.

As a result, if you viewed the instructions of older, simpler processors as bit patterns, you might notice certain groupings; for example, if the top three bits of the instructions would be set so, you might know that it must be a jump instruction of some type; other patterns might identify other general types of instructions.

The design team for a new processor architecture would develop a reasonable set of instructions, the various networks of logic gates to implement them, and the instruction decoder to manage them all. Back when the 8086 and 8088 processors were designed, it was not yet economically viable to add extra logic to the instruction decoder to exclude any "undocumented instructions", so any bit pattern fed into the instruction decoder would do something, but only the documented instructions would be guaranteed to exist and work the same also in the future members of the same processor family.

Sometimes the undocumented instructions actually ended up doing something useful, but relying on them was risky, as it might mean any program using a particular undocumented instructions might only run successfully that particular processor model. On a newer model, the previously undocumented instructions might actually been used for some new functionality, or they might have become completely useless as a result of a redesign of the instruction decoder.

With 80186 or above, using a number of logic gates to trap the undocumented instructions was economically feasible, and it became a compatibility feature: a program might prefer to use a new instruction available in newer processors only, but could provide a routine to emulate that instruction in software within an "illegal instruction" trap handler.

Improve this answer
2
  • 3
    Those patterns in the instruction encoding are very obvious in MIPS, for example, which was designed so the bits of the instruction encoding could be used fairly directly as internal control signals. And it only has a few instruction formats (3-register ALU, or I-type with a 16-bit immediate, or J-type for unconditional jumps). It's word-oriented not byte-oriented. But basically all ISAs with fixed-width instructions have some opcode bits and some register-number bits in consistent locations across many instructions.
    – Peter Cordes
    Jun 14, 2019 at 4:02
  • For more about x86 opcode patterns, see Are x86 opcodes arbitrary? Also, 3-bit groups come up in some, so octal is apparently helpful.
    – Peter Cordes
    Jun 14, 2019 at 4:04
1

Based on the other answers, the proper answer is: they do whatever the instructions bits tell them too. Since apparently they aren't validated, they always do something, even if that something is nothing.

Same as trying to turn on a lamp on an illuminated room- you won't notice it is there and it is on.

EDIT: I had a question in the answer which I removed, which was "Another question you may ask is: if an instruction does not exist, how does the processor knows it length?", which was replied in the comments below.

Improve this answer
2
  • 1
    You could ask your question as a new question — as it is, your answer is likely to be flagged as “not an answer” ;-).
    – Stephen Kitt
    Jun 12, 2019 at 15:12
  • 2
    It knows its length the same way it "knows" what the instruction does: the wiring causes it to increment the instruction register by however much it needs to.
    – wizzwizz4 ♦
    Jun 12, 2019 at 16:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .