Undocumented ModR/M byte combinations

Question

The reason I am asking this question is because I was trying to compile a list of all 8086 opcodes, regardless of whether or not they are documented. The list is just for fun, the documented opcodes + the well known undocumented ones such as SALC are more than enough for any serious code designed for the 8086.

There is quite a lot of documentation online on single-byte undocumented instructions of the 8086/88, such as SALC (D6) or POP CS (0F), but there seems to be little info on undocumented/invalid multi-byte opcodes. The 8086 doesn't have an invalid opcode exception, so every opcode combination must necessarily do something.

Let's take an interesting instruction, for example: LEA. This instruction computes a memory address using regular addressing modes (BX + SI, BP + DI + disp etc.) and stores that address in a general-purpose register. All instructions that compute addresses this way use a ModR/M byte that follows the instruction opcode (in this case 8D).

The ModR/M byte has the format xxrrrmmm, where xx is the mode, mmm is the memory/register operand, while rrr is the other register operand (notice how the 8086 never uses 2 memory operands in the same instruction). The mode defines the kind of memory/register operand: 00, 01, 10 are different types of memory operands, while 11 is a register operand.

If we assemble LEA DX, [BP + DI], we get 8D 11. The ModR/M byte is 0x11, or 00 010 001. The rrr=010, which means that the destination register is DX. The xx=00, so we have a memory operand without a displacement. Finally, the mmm=001, which stands for BP + DI.

The manual doesn't specify what happens when you use mode xx=11 (both operands are registers). For example, a ModR/M byte 11 010 001 would make the instruction LEA DX, CX, which doesn't make much sense. The question is what does the 8086 do when it encounters such weird combinations in the ModR/M byte?

LEA is not the only instruction that has invalid combinations of ModR/M. There are also some instructions which don't use the rrr field at all, because the second operand is an immediate value (or because there is no second operand). In such cases, the manual says that the rrr field must be 000.

There are also "group" instructions. Instructions that are part of a group (such as shift, rotate, multiply, divide etc.) use the same opcode for the first byte, but never use the rrr field. To distinguish between instructions, the rrr field is the index of the instruction in the group, and the ModR/M byte is usually written as /i, with i being from 0 to 7 and representing the rrr field. For example, D2 /0 means ROL r/m8, CL, while D2 /1 means ROR r/m8, CL. Some of these indexes, however, are not assigned to any instruction, for example GRP3 /1 and most of GRP4, and are therefore undefined.

Here is a list of instructions that can have an illegal ModR/M byte (it aims to be exhaustive, but probably isn't):

• Instructions that by design can only take a memory operand (xx=11 is not documented by the manual):

  • 8D: LEA r16, addr (technically it's not m16 because it doesn't access memory)
  • C4: LES r16, m16:16
  • C5: LDS r16, m16:16

• Instructions that have only one non-immediate operand (the rrr field is unused and the manual says it should always be 000, almost as if they were group instructions with index /0):

  • 8F: POP r/m16
  • C6: MOV r/m8, imm8
  • C7: MOV r/m16, imm16

• "Group" instructions with an undocumented index in the rrr field:

  • GRP2 /6 (this list of undocumented opcodes says it is a "set −1" instruction, but on a modern x86 CPU it executes as SHL/SAL, which is more plausible, since this is where SAL would be if it wasn't an alias for SHL)
  • GRP3 /1
  • GRP4 /2~7 (this group only has 2 documented instructions, INC and DEC)
  • GRP5 /7

Answer 1

As part of his series about the 8086 micro code Ken Shirriff has just added a blog post where he analyses some of the mentioned cases:

Undocumented 8086 instructions, explained by the microcode

It explains for example

• LEA reg,reg loads a value from an internal register called IND, which in normal cases would be filled with whatever the EA calculation yields - except, when ModR/M points to a register no EA calculation is done. As a result giving access to previous IND.

• LES/LDS will act similar, except two internal registers OPR and IND are used. EA handling would have loaded the effective address into IND and loaded OPR from that address. LES/DS would now load OPR into the target register, increment IND by two and load the segment register with the word pointed to by IND. As a result OPR can be accessed, while destroying the segment register.

• JMP FAR with a register would act likewise, except now loading an internal tmpB into PC and whatever IND+2 points to into CS - thus giving (hard to use) access to tmpB.

Register meaning

• Indirect (IND) holds an address within a segment (16 bit offset)
• Operand (OPR) holds the data value read or written
• tmpB holds the second operand of any ALU operation

For more see the blog entry - although, the C6/C7 group is not covered.