The 8051 is a Harvard-based microcontroller,
Not really. It's a modified Von-Neumann design with non overlapping address spaces for program, data and I/O. There is no simultaneous access to program and data at the same time, which is the basic feature for any Harvard design (*1).
A modified Von-Neumann architecture combines the (software) advantage of separate address spaces with the need of only a single bus for data and program combined, which simplifies hardware design, quite important to save chip area on (early) microcontrollers.
What is the main reason it was not used widely in 80's home computers?
A main reason? Then it will be the same for all microcontrollers: It's a microcontroller.
Microcontrollers are about providing as many programmable ports as possible. The 8051 came in a 40-pin device offering 32 of the 40 as free accessible I/O, organized as 4 ports. With external memory, 3 of the 4 are used for control lines and address/data, leaving a single 8-bit port.
With external RAM/ROM, it gets reduced to a tiny (128 Bytes) RAM, 4 KiB of ROM, an 8-bit port and two timers. While not enough to define the core of a home computer of the early 1980s, it would need exactly the same external components as a standard CPU like an 8085: external latches, address decoding, RAM/ROM for program/data, video RAM, video controller and so on.
Considering that an MCU is usually more expensive than a MPU, using the 8051 that way would make it more expensive than using an 8085 - and less flexible as well.
Was 8051 too late (1980)?
Most definitely. Especially when considering that it would have taken 2-3 more years until a complex device, like a home computer, would be ready for market - that's about the time the Famicom/NES hit the market - with specs, a 8051 couldn't compete without at all.
Or has it too slow access to the external memory?
Not just that, but incrediblly slow in general, even more so in 1980. Its maximum clock frequency of 12 MHz may seem a lot, but that gets less impressive when considering that a machine cycle takes 12 clock cycles. Also, the full 12 MHz can only be sustained on external RAM/ROM with an access time of less than 166 ns. With RAM like that, a 6502 could run at 3 MHz or a Z80 at 8 MHz.
RAM speed is about cost, so it would be more likely that the 8051 would have run at only 4 MHz, as its memory interface had no way of extending access time when needed. The practical result would have been an even slower computer, delivering mediocre performance.
But...
At that point it's important, that while random RAM access is quite slow on the 8051, this doesn't influence real world application as much. Most access will still be for code. A real setup would keep core access function in on board ROM, making execution fast and equal important, all heavy used data would be held in on board RAM, as well accessible in full speed - much like the zero/direct page of 6500/6800 type MCUs. In addition, 8051 code is quite compact, thus as well reducing impact of slowdown for external code.
Bottom line for speed: While the 8051 is quite slow in itself, when running some BASIC program, performance would still be competive.
Or is there any lack of instruction set?
It's almost never the instruction set. The fun about programmable devices is that everything can be done.
Or is there any other reason?
As said, it's not meant to work as a general purpose computer, but a controller. Using it as such is much like using a sporty convertible (imagine here a 1980s Fiat X15 or Alpha Spider) as the family car.
Now having said that, 8051 have been used in many computer like devices, especially various 'kid computers' of the 1990s. For the early, 1980 time frame, it was rather the 8048, the direct predecessor (*2) was used in several computer like devices:
- IMSAI 8048 Control Computer, a very early (1977) single boarder, expandable to 64(!) KiB RAM.
- Philips Videopac Computer G7000 (also known as Magnavox Odyssey 2) - eventually the most known one.
- Kosmos Computer Praxis CP1, a course system. Here the 8049 runs a virtual decimal (!) processor.
- 8052AH BASIC - almost a computer - it's basically a 8051 with double the RAM (256 Bytes) and an 8 KiB BASIC interpreter in ROM. As Busybee already mentioned, all it needs is some external RAM and a terminal (*3) .
The G700 is eventually the closest to a (home) computer here. It features a full keyboard and colour video capabilities. With a basic cartridge it could for sure compete with a ZX80/81 of the same time.
*1 - Harvard isn't about separate address spaces. They are just a side effect of parallel access to program and data store - which in turn is intended to allow independent read and faster operations (It as well relaxes the need to make program and data word size the same).
Logical a Harvard type CPU can still merge both into one address space (to simplify handling), without leaving its origin. Similar a Von-Neumann can separate them logically without turning into Harvard, as the access is still common. And 8051 is an example of such a modified Von-Neumann.
*2 - In fact, the very first data sheet puts the 8051 into the MCS-48 family.
*3 - In the late 70s to early 80s several manufacturers offered a MCU with BASIC embedded.
i8048
in keyboards but they are very similar toi8051
...