Retrocomputing Stack Exchange is a question and answer site for vintage-computer hobbyists interested in restoring, preserving, and using the classic computer and gaming systems of yesteryear. It only takes a minute to sign up.
Modern x86 CPUs are binary compatible with 8086. You can literally run 8086 binaries on a modern PC, in real mode. (The species analogy is a stretch here, but works if you look at forward compat instead of backwards compat: old x86 chips can't run AVX / AVX2 / FMA / AVX512 code, so you could look at each ISA extension as a speciation event.)
Other than changed immediate operand lengths (to support 64-bit addresses) and register encodings (to support the new registers), which obviously make practically all code incompatible at the binary level, there are a few other major changes in long mode:
Instruction prefix handling is changed; older processors allowed arbitrary sequences of prefixes (originally) or limited instruction length to 15 bytes (from the 80386 onwards, IIRC) but otherwise allowed any combination of prefixes. Long mode only allows one prefix of each of 4 groups, and results are undefined if this restriction isn't followed.
Handling of segmentation is completely replaced. Attempts to load segment registers are effectively ignored, as are CS, DS and ES segment override prefixes. The meaning of FS and GS prefixes is also somewhat different (they refer to fixed base addresses defined in new registers).
The single-byte encodings of the DEC and INC instructions are no longer valid (they are reused for prefixes that activate the new addressing modes)
As noted in the question, the AA* (ASCII Adjust) and DA* (Decimal Adjust) instructions are not permitted and raise an exception if used.
BOUND (array bounds check), CALL FAR and JMP FAR, INTO (Interrupt Overflow), LAHF (Load AH with Flags), SAHF (Store AH into Flags), LDS/LES (load far pointer using DS/ES segment registers), PUSH/POP instructions with CS, DS, ES or SS as target, PUSHA/POPA (push/pop all 16-bit registers), and PUSHAD/POPAD (push/pop all 32-bit registers) are all likewise unavailable and raise exceptions if used.
The 16-bit JCXZ instruction isn't available (its opcode is used for JRCXZ instead).
ARPL (Adjust Requested Privilege Level) isn't available; its opcode is reused for MOVSXD.
Instructions that load descriptors and tables have new extended data formats
The behaviour of SYSCALL and SYSRET is substantially different.