As explained on the Wikipedia Game Boy article and Is the Game Boy Sharp LR35902 object-compatible with the 8080/Z-80?, the Game Boy used a custom CPU that was fairly close to being a superset of the 8080 or subset of the Z80.
This processor is similar to an Intel 8080 in that none of the registers introduced in the Z80 are present. However, some of the Z80's instruction set enhancements over the 8080, particularly bit manipulation, are present. Features removed from the Intel 8080 instruction set include the parity flag, half of the conditional jumps, and I/O instructions. I/O is instead performed through memory load/store instructions. Still, several features are added relative to both the 8080 and the Z80, such as new load and store instructions to optimize access to memory-mapped registers. The IC also contains integrated sound generation.
Did they start by licensing the 8080 core from Intel, then add features? Or did they start with the desired instruction set, and design their own chip layout from scratch?
The former seems difficult to reconcile with such drastic changes. It is difficult to save space by deleting features from an existing chip layout, without redoing the rest of the layout to fill in the holes.
The latter raises the question of why, in that case, not design a completely new instruction set? But then, maybe Nintendo and Sharp decided they were not in the ISA design business, and basing their work on the 8080 instruction set allowed reuse of an existing assembler toolchain.
So was it a clean sheet layout, or was it based on a licensed core?
