In Appendix C (on software compatibility to the 8086), the Intel 286 users manual states the following about instruction length (Page C-2):
Do not Duplicate Prefixes.
The 80286 sets an instruction length limit of 10 bytes. The only way to violate this limit is by duplicating a prefix two or more times before an instruction. Exception 6 occurs if the instruction length limit is violated. The 8086/8088 has no instruction length limit.
This seems to be wrong, though. The correct exception is exception 13 (0Dh). It is correctly documented in the manual on Page B-9:
#GP 13 General Protection (Selector or Zero Error Code)
This exception is generated for all protection violations not covered by the other exceptions in this section. Examples of this include:
An attempt to address a memory location by using an offset that exceeds the limit for the segment involved.
An attempt to jump to a data segment.
An attempt to load SS with a selector for a read-only segment.
An attempt to write to a read-only segment.
Exceeding the maximum instruction length of 10 bytes.
The limit has been raised by the 80386 to 15 bytes, because operands and addresses may be 32 bit instead of 16 bit, and you also have extra prefixes. You can find it in the 80386 programmers manual, page 168:
9.8.13 Interrupt 13 ── General Protection Exception
All protection violations that do not cause another exception cause ageneral protection exception. This includes (but is not limited to):
- Exceeding segment limit when using CS, DS, ES, FS, or GS
- Exceeding the instruction length limit of 15 bytes (this can occur only if redundant prefixes are placed before an instruction)
The limit of 15 bytes is valid until today.
So if we just consider x86 processors, the last processor that did not implement a logic to detect oversized instructions is the 8086 or the 80186. I can not find any reference to a 10-byte limit in the iAPX 86/88/186/188 reference manual, so I am confident to claim that the last Intel x86 CPU that allowed instructions of unlimited size is the 80186/80188. The instruction size limit became part of the architecture with the 286 and was changed with the transition from 16 bits to 32 bits.
As all AMD processors up to the (original, non-enhanced) Am486 are licensed copies of the Intel Design, the same applies to AMD processors. For the 16-bit processors, there are other third-party manufacturers like Siemens and Harris, they are also licensed copies of the respective Intel models, so no behavioural difference is to be expected.
There are 80186-compatible processor cores in microcontrollers produced until 2000s at least. They most likely do not include the instruction size limit. The same applies to the V30 and microcontrollers based on it. Basically everything that doesn't have the protected mode (and doesn't have the general protection fault) also doesn't have the instruction size limit.