Back in the 8-bit days, I used 6502 computers, where the story about memory access was easy to understand. RAM chips of the late seventies and early eighties could do 2 MHz (or a bit more e.g. 2.6 in the datasheet I found), a 1 MHz 6502 took half the bandwidth leaving the other half for video which translated into a maximum of about 10K of video RAM. (Variation on a theme: the BBC Micro used the new fast RAM chips, which allowed a doubling of all those numbers.)
I'm writing a story that involves among other things a fictional Z80-based home computer at the end of the seventies; the protagonists selected that CPU in the belief that it was slightly faster than the 6502 (takes an average of perhaps three times as many clock cycles to do things, but has four times the clock speed). The computer has 16K of RAM (about the max affordable at the time) and graphics similar to those of the Sinclair Spectrum. I'm belatedly checking the situation with regard to sharing memory access, and...
http://www.worldofspectrum.org/faq/reference/48kreference.htm#Contention
... it's looking surprisingly terrible? The video seems to be using 3/4 of the memory bandwidth rather than the expected 1/2 even though it's only 6K of video RAM. The Z80 needs memory access at what are in this context effectively random intervals, which means it will very often trip over the video chip. Worse yet:
https://en.wikipedia.org/wiki/ZX_Spectrum_Contended_Memory
"However, the ULA must wait for a Z80 read or write to complete before it can stop the Z80's clock and access the memory. As the ULA has no capacity to prefetch and store data in advance, it relies upon the memory being available to read to produce a clean and true video display. If an existing Z80 access is in progress when the ULA must read video data, the read is missed and the display shows blank white pixels in place of the correct video pixels. On the next frame, they may well be read correctly. The consequences of this often appeared as a flickering of missing pixels on the display, or 'snow' as it was also called. To avoid snow, some programs and games only copied data to the display on interrupt, as the Spectrum maskable interrupt was generated by the display vertical refresh signal."
... that seems to mean that a program that spends most of its time updating the display (like games, which, let's be honest, are the main consumers of CPU cycles on home computers), must effectively shut down operations while the screen is being displayed, which is most of the time. The combination of these would seem to imply that you'll only actually see a small fraction of the nominal performance of the CPU. Even the separate memory banks on the 48K Spectrum won't help all that much, because the CPU does still need to update video memory.
I don't think it can really be that bad. If it was, nobody would ever have used a Z80 for any computer with a graphical display, and in practice, the Spectrum, Amstrad, MSX and probably some other machines I'm forgetting, did happily use the Z80. What am I missing?