To understand this, you need a general idea of how processors work at the raw hardware level (or at least how they worked before the concept of microcode was developed).
Basically, as the processor would read the bytes of a single assembler language instruction, those bytes would be fed as an input to a network of logic gates called the instruction decoder; this network was designed to produce as output the signals to enable or disable various internal components of the processor according to what the instruction was supposed to do, so that the processor ends up doing the required operation.
As a result, if you viewed the instructions of older, simpler processors as bit patterns, you might notice certain groupings; for example, if the top three bits of the instructions would be set so, you might know that it must be a jump instruction of some type; other patterns might identify other general types of instructions.
The design team for a new processor architecture would develop a reasonable set of instructions, the various networks of logic gates to implement them, and the instruction decoder to manage them all. Back when the 8086 and 8088 processors were designed, it was not yet economically viable to add extra logic to the instruction decoder to exclude any "undocumented instructions", so any bit pattern fed into the instruction decoder would do something, but only the documented instructions would be guaranteed to exist and work the same also in the future members of the same processor family.
Sometimes the undocumented instructions actually ended up doing something useful, but relying on them was risky, as it might mean any program using a particular undocumented instructions might only run successfully that particular processor model. On a newer model, the previously undocumented instructions might actually been used for some new functionality, or they might have become completely useless as a result of a redesign of the instruction decoder.
With 80186 or above, using a number of logic gates to trap the undocumented instructions was economically feasible, and it became a compatibility feature: a program might prefer to use a new instruction available in newer processors only, but could provide a routine to emulate that instruction in software within an "illegal instruction" trap handler.