Each of the 68K series CPUs had dedicated address-generation hardware which was wired more directly to the A registers and had only limited access to the D registers. Conversely, the main ALU was more directly wired to the D registers than the A registers. It thus became a performance enhancement, allowing the main ALU and the addressing logic to operate in parallel without conflicting in the register bank.
In the above die-shot with the sections helpfully labelled, you can clearly see there are separate sections of the chip for processing addresses and data. Notice also that there is no section marked "register bank"; the registers are physically entwined with their respective execution units.
In the 68040 and 68060, these separate execution units became distinct stages in the CPU's pipeline(s). The EA (Effective Address) was calculated in two stages, whose use was repeated as necessary for some of the more complex addressing modes, and the main ALU existed in another pipeline stage which came after these.
If you carefully examine the 68K instruction set, you should notice that actually, instructions which modify the A registers have different mnemonics than those for the D registers, even when they perform the same function. They decode to completely different sections of the microcode ROM (marked µROM in the floorplan) which activate the appropriate parts of the correct execution unit.
Both sets of instructions have access to addressing modes (selected by the six-bit field at the right-hand end of the instruction word) that include using either A or D registers as the second operand. Addressing modes that refer to memory are all based around A registers, with only indexed modes permitting the use of a D register in the address equation. Indexed modes take correspondingly longer, as they require an access cycle across an internal bus bridging the two execution units.
This duplication of circuitry stands in direct contrast with simpler CPUs such as the 6502 family, in which the same ALU was used for both accumulator and address-indexing arithmetic, including relative branches. Only address operations which required merely an increment or decrement (such as advancing the program counter) had logic separate from the main ALU to increase internal parallelism at minimal cost.