The reason I am asking this question is because I was trying to compile a list of all 8086 opcodes, regardless of whether or not they are documented. The list is just for fun, the documented opcodes + the well known undocumented ones such as SALC are more than enough for any serious code designed for the 8086.

There is quite a lot of documentation online on single-byte undocumented instructions of the 8086/88, such as SALC (D6) or POP CS (0F), but there seems to be little info on undocumented/invalid multi-byte opcodes. The 8086 doesn't have an invalid opcode exception, so every opcode combination must necessarily do something.

Let's take an interesting instruction, for example: LEA. This instruction computes a memory address using regular addressing modes (BX + SI, BP + DI + disp etc.) and stores that address in a general-purpose register. All instructions that compute addresses this way use a ModR/M byte that follows the instruction opcode (in this case 8D).

The ModR/M byte has the format xxrrrmmm, where xx is the mode, mmm is the memory/register operand, while rrr is the other register operand (notice how the 8086 never uses 2 memory operands in the same instruction). The mode defines the kind of memory/register operand: 00, 01, 10 are different types of memory operands, while 11 is a register operand.

If we assemble LEA DX, [BP + DI], we get 8D 11. The ModR/M byte is 0x11, or 00 010 001. The rrr=010, which means that the destination register is DX. The xx=00, so we have a memory operand without a displacement. Finally, the mmm=001, which stands for BP + DI.

The manual doesn't specify what happens when you use mode xx=11 (both operands are registers). For example, a ModR/M byte 11 010 001 would make the instruction LEA DX, CX, which doesn't make much sense. The question is what does the 8086 do when it encounters such weird combinations in the ModR/M byte?

LEA is not the only instruction that has invalid combinations of ModR/M. There are also some instructions which don't use the rrr field at all, because the second operand is an immediate value (or because there is no second operand). In such cases, the manual says that the rrr field must be 000.

There are also "group" instructions. Instructions that are part of a group (such as shift, rotate, multiply, divide etc.) use the same opcode for the first byte, but never use the rrr field. To distinguish between instructions, the rrr field is the index of the instruction in the group, and the ModR/M byte is usually written as /i, with i being from 0 to 7 and representing the rrr field. For example, D2 /0 means ROL r/m8, CL, while D2 /1 means ROR r/m8, CL. Some of these indexes, however, are not assigned to any instruction, for example GRP3 /1 and most of GRP4, and are therefore undefined.

Here is a list of instructions that can have an illegal ModR/M byte (it aims to be exhaustive, but probably isn't):

  • Instructions that by design can only take a memory operand (xx=11 is not documented by the manual):

    • 8D: LEA r16, addr (technically it's not m16 because it doesn't access memory)
    • C4: LES r16, m16:16
    • C5: LDS r16, m16:16
  • Instructions that have only one non-immediate operand (the rrr field is unused and the manual says it should always be 000, almost as if they were group instructions with index /0):

    • 8F: POP r/m16
    • C6: MOV r/m8, imm8
    • C7: MOV r/m16, imm16
  • "Group" instructions with an undocumented index in the rrr field:

    • GRP2 /6 (this list of undocumented opcodes says it is a "set −1" instruction, but on a modern x86 CPU it executes as SHL/SAL, which is more plausible, since this is where SAL would be if it wasn't an alias for SHL)
    • GRP3 /1
    • GRP4 /2~7 (this group only has 2 documented instructions, INC and DEC)
    • GRP5 /7
  • 3
    I imagine you’re aware of this list of undocumented opcodes (but it’s mostly single-byte opcodes). There’s also this brief analysis of the microcode. None of these address your question! Sep 8 '21 at 13:41
  • Except for LEA, I would expect most of those opcodes would simply be interpreted as two-byte ways of performing operations that can also be done with a single byte form. Not sure what LEA would do, though.
    – supercat
    Sep 8 '21 at 14:42
  • @StephenKitt Yes, I am aware of those. The first one is probably the most interesting, since it claims to be Part I, while Parts II and III (which were never posted) have titles that could answer this question and many others.
    – DarkAtom
    Sep 8 '21 at 15:15
  • 1
    @supercat I extended the list to include the group instructions and also specified how each instruction has undefined ModR/M combinations as you suggested. I suppose the key to the answer is probably the microcode, which for the 8086/88 was thankfully disassembled already. But for that one would probably have to spend quite a few hours of analysis.
    – DarkAtom
    Sep 8 '21 at 21:06
  • 1
    On modern computers, LES and LDS with mod == 11 are VEX prefixes encoding AVX instructions. Not that this would be relevant to the 8086 in any way.
    – fuz
    Sep 11 '21 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.