I was at college 1979-83, studying computing, and remember being told in a microprocessor course that the Z8000 wasn't very good. After a browse through the user's manual I can see why.
The memory model seems to have been designed by a hardware engineer who didn't think very much about software. Indeed, the whole architecture seems designed to do the same jobs as the Z80, only better, unlike its more successful competitors. The crucial difference was that most 8-bit software was written in assembly languages, while 16-bit and later software was mostly written in compiled high-level languages.
The 68000 family was clearly designed with an eye towards high-level language programming. The architecture was 32-bit in concept, although the original implementations were 16-bit. It had "flat" unsegmented memory addressing, which is always easier for compilers, and could address 16MiB of RAM, which increased to 4GiB in later members of the family.
The 8086 family was less ambitious, but was practical. It could only address 1MiB of RAM, and it dealt with it in 64KiB segments, but you could place those segments anywhere in memory. This made it practical, if inconvenient, to deal with data structures larger than 64KiB. Once 16-bit MS-DOS had begun to dominate the business microcomputer market, there was plenty of motive to improve the programming tools.
The Z8000 family came in two models. The Z8002 could only deal with 64KiB RAM, although you could stretch that by using separate 64KiB address spaces for code, data and stack. However, writing an operating system that works with that is gratuitously difficult. Operating systems have to be able to handle program code as data when they load programs. The multiple address space trick is mostly useful for embedded systems without general-purpose operating systems.
The Z8001 was the large memory version, able to handle 8MiB RAM in each address space. It deals with the memory in 64KiB segments, but it has a crucial flaw. The segments are at fixed addresses. Segment 0 is at address 0, segment 1 is at address 64Ki, segment 2 is at address 128Ki, segment 3 at address 192Ki, and so on. The segments can't overlap, and you can't change their positions.
This makes handling data structures larger than 64KiB very cumbersome. It also restricts the use of smaller data structures, because you really don't want them to cross segment boundaries. This creates arbitrary restrictions, makes linkers and loaders more complicated, and generally reduces your flexibility in the use of memory. Operating system people hate that, and so do compiler writers. The combination of that, plus buggy initial chips, doomed the Z8000.
Zilog acknowledged the memory model had been a mistake with the Z80000, which had a mode with a 4GiB linear address space. It also supported Z8002 and Z8001 modes, with more and larger segments available in the Z8001 mode. However, by the time it appeared in 1986, the x86 and 68000 were well-established and the fashion in new processors was RISC. The Z80000 was very much non-RISC, and doesn't seem to have got anywhere.