It's not just the systems you mentioned; in almost everything from the era, other than the IBM PC (with its discrete - and not particularly high performance, merely higher performance than hanging something directly off the 8088 bus - graphics cards) and the MSX/Console type machines using a TMS VRAM graphics system, or an Amiga equipped with FastRAM, the CPU had to contend with the video system for access to RAM.
There's various reasons for this. One major driver was that of sheer cost. If you run two buses in your machine, one with main memory accessible by the CPU alone, another with video memory that's looked after by a discrete chip, you greatly increase the complexity, and thus the design and building cost of your machine. It also increases the potential points of failure, which is an undue risk if you're making a simple and cheap home starter system. One bus with everything hanging off it, and a somewhat simpler Glue ULA / PLC / etc gating the access on a timeshare basis, especially if the "video chip" is little more than a shift register that has video data serially DMA'd into it, is far more streamlined and straightfoward, and can be banged out for minimal cost without suffering too much in the way of reliability issues.
Another is, well... need. Video bandwidth in the early days really wasn't that much. The sheer amount of available memory (the PC's 16KB CGA was positively massive for its launch era, especially considering the machine's base RAM wasn't any larger, and the MDA only had 4KB; by the time of the Mac, ST and Amiga some years later, the screenbuffer size had only increased, respectively, to 1.33, 2.0, or 2.5x (default; max of 10x for the rather less frequently used, slow, memory squashing, flickery hi-rez interlace), with the competing EGA coming with 64KB by default) limited the practical resolution and colour depth you could deliver, and that reduced how much data you had to stream from memory to screen per second*. The best you tended to get with any 8-bit or other early 80s machine was in the realm of 256 or 320 pixels on a line and 2 bits per pixel (direct, or equivalent via block attributes). 512 or 640 bits, a little under 16,000 times per second, isn't that much... it's in the realm of 8 to 10MBit, or 1.0 to 1.25MByte/s (actually a little higher because of blanking, but let's not split hairs for the sake of a demo; we can consider it doesn't go above 2MB/s, and probably settles around 1.5).
OK, when you've got a 6502 system running at all of 1 or 2MHz, or even a Z80 at 4MHz (both limited as much by the memory speed as anything else - in fact, a decrease in the cost of high-frequency DRAM is literally what made it possible to run the BBC Micro at 2MHz and have it deliver an 80-column display with graphics capability, when the original design only produced half of those specs), this still sounds like quite a lot. But, there's more; you have to consider the CPU demand as well. And early processors just weren't very efficient when it came to memory access. Modern chips can easily completely saturate the memory bus with continual access, and indeed often end up momentarily paused waiting for the next RAM cycle if they get multiple cache misses in a row. Not so the earlier generation of chips. You'd be lucky to get 4 clocks per cycle a lot of the time (excepting some oddities of efficiency like the 6502's zero-page mode), and ignoring the other specifics it was typical for the bus to sit idle a good 50% of the time; this was even true of the 8086 (the 8088 hammered the bus a bit more as it was trying to fit a 16-bit architecture into 8 bits), and persisted into at least the 68000/68010 era, maybe the 80286 as well. Anything sitting idle in a computer, generally, is just a waste of silicon and electricity. What's to be done with it?
Well, er... we could fit the video access into it. If things are arranged with sufficient grace, the impact on processing could be anything from "nothing" or "so minor that you wouldn't notice outside of detailed benchmarks", to "moderate, but still not worth adding an extra bus and chipset to overcome", depending on machine and, indeed, the video mode. We can even trace the decisions regarding default video modes to these arrangements, somewhat.
Later examples which are the ones I'm most familiar with are the aforementioned trio of Mac, ST and Amiga. All use the 68000 CPU at 7 to 8MHz, and pixel clocks in the 7 to 32MHz range depending on mode (higher clock/resolution meaning lower bit depth, so roughly the same bandwidth - including the Amiga's interlace mode, which essentially halves the framerate). The 68000 has a 4-clock memory cycle, which is, nominally, continually active when filling the prefetch queue or running through strings of short instructions... but in reality it's only active on the bus for two out of every four cycles, maximum, and it's designed for memory rated at only half the CPU's clock speed. If your computer's Glue logic is in any way competent, and you have full-speed memory (which by the time these machines came around was fairly affordable), it can fool the processor into thinking it's got exclusive use of the bus, when in fact the graphics system (and/or other DMAs) have free reign of it at an effective 3.5MHz or more. Meaning you can run a 7 to 8MB/sec graphics mode using shared memory with almost no performance penalty whatsoever - the only times the processor is made to wait is if it's just finished processing one of the relatively rare longer instructions that runs for a total number of cycles not divisible by four... in which case that longer instruction is extended by one, two or three clocks. But even in the worst case scenario (say, adding 3 clocks onto a 5-clock instruction), it's not quite a 50% slowdown, and the greater majority of instructions are either 4-clock aligned, or take so long to process that the extra latency after each one causes only a very fractional increase to processing time. That bandwidth is what gets you the common 320x200p (or x256p, 400i, 512i) 16-colour or 640x200p (etc) 4-colour ST or default Amiga modes, 640x400p (72Hz) mono ST mode, and is in fact rather in excess of the Mac's relatively lowly 512x342p (60Hz) mono which, in concert with somewhat reduced blanking, needs half as much bandwidth again (possibly down to it being derived from the lower frequency Lisa, where 720x364p 60Hz used the full available bandwidth, and maybe intended to run at 4MHz or less vs the Lisa's 5MHz). We also see evidence of it in the Amiga's higher quality modes, which demand an extra 25% (lo-rez 32-colour), 50% (lo-rez 64-colour/hi-rez 8-colour) or even 100% (hi-rez 16-colour) bandwidth, and sap the processor speed accordingly (in the latter mode, it can ONLY run during the border/blanking periods of the screen scan, being completely HALTed during the active parts of the video, audio and disc DMA cycle). In those modes, the ability for the custom chipset to compensate for the hobbled CPU, or even for the CPU to run on a separate bus with its own "Fast" RAM (one of the many reasons it was a comparitively expensive machine), become absolutely essential.
Backtracking to the 8-bits, we can still see similar, though there's often more of a tradeoff in terms of speed vs cost. The Spectrum needs about 14Mbit/s video bandwidth (=1.75MByte), with 7MHz pixels at 2bpp equivalent. The system clock spins at 3.5MHz, transferring 8 bits every other clock assuming ~2MHz memory and the usual 2-clock DRAM R/W cycle. So that's just enough to support the normal graphics mode (or the double rez, colourless option in its Timex sibling), with the processor halted during the active video period. This may be why the machine's video appears in a relatively small window with thick borders; you can still deliver reasonably good resolution this way, with high-ish clocks and a PAL signal, without entirely crippling the processor. There's exactly 224 clock ticks per line, of which 128 are taken up by video access, so you lose a little over half the potential memory accesses in this scheme... but, also, only 192 of the 312 scanlines are active. Overall, only 35% of the actual scanned frame is active video. If your programmer is sufficiently wizard-grade with their use of machine code and cycle counting, and can arrange so that the processor is fed a long-running instruction just before the start of the active video period (then writes out the result as soon as it's allowed, and spins through some shorter ones in the blanking period), you may only really lose a third of the available bandwidth... less still, if they can do it on every line. This, I expect, would have been considered entirely acceptable in the pursuit of a lowest-possible-cost Colour Computer that still gave acceptable (rather than "blazing") performance, especially as it was using what would otherwise have been considered a pretty speedy processor only a few years earlier when even pretty expensive CP/M based micros were using things like 2.5MHz i8080s. Compared to those, it had performance to burn. And if you were really bothered more by processing than visuals, there was likely still the option of turning the display off (except for syncs) altogether, such as what happened with the ZX81's "Fast" mode, or whenever you pressed a key on the ZX80. Or you could just buy a machine that cost more than £149 fully assembled...
(In fact, sanity checking my memory via Wikipedia to make sure it didn't run the bus at 7MHz instead with 4MHz memory... turns out the Spectrum improved on the ZX81 in more ways than just adding colour, sound, and a larger internal RAM/ROM. It actually WAS a split-bus design... maybe not fully a true two-bus system, but there was an at least notionally Amiga-like separation of "memory accessible by the video/Glue system AND the CPU", being the lower 16KB (or only 16KB, in the base model, where the above dissection would still hold true), and "memory accessible only by the CPU", being the upper 32KB (or 112KB) in the larger-memory models. In which case, if you were writing 16KB code, you'd have to be exquisitely careful about your timing in order to squeeze out the best performance, but it's not like you were going to write anything THAT complicated in the approx 9KB of usable program RAM available to you anyway so optimisation was rather a given; if you were writing 48K or 128K code, you just had to make sure anything timing-critical lived above that 16K boundary, and save the spare 9-and-a-bit K of lower memory for less important stuff such as infrequently referenced data that could be specifically called upon during vsync, maybe a double-buffer for the screen or a larger scrollable area, music replay routines, etc... this would be analogous to using a higher Amiga video mode - EHB low rez in deep overscan, or 16-colour hi-rez with a reduced viewport - on a system with only 512K or even the very base 256K of the launch model, but 2x or more that amount hanging off the FastRAM bus, with the same considerations about what goes where, including whether the AV system could directly access it or not)
C64 I'm fairly sure I read as actually having GREATER memory bandwidth... like, the processor might have only run at about 1MHz, but the bus / RAM was clocked somewhat higher, and a typical video mode tended to share access on about a 2:1 basis (video:CPU) during the active periods. Atari 8-bits similar. These systems of course, particularly the Ataris, had the additional advantage of offering multiple screen modes, with contention varying - much like in the Amiga - according to the combination of text vs graphics, resolution and colour depth, as well as having hardware sprites that meant you could produce the illusion of a visually richer display (through use of a limited number of small, but higher resolution and higher colour depth elements, on top of an otherwise more rudimentary playfield; again, this helped the Amiga no end, especially in its later years when VGA-equipped PCs, later Macs, and 16-bit consoles became its main competitor, rather than the ST, 8-bit consoles and mono Macs) without having to significantly increase actual video bandwidth.
(OK, I haven't done as well sanity-checking THAT, but what wiki knowledge I have dredged up says that the commie's VIC-II "shares the RAM with the CPU, each accessing it on alternate half-cycles, other than so-called BadLines where the VIC blocks the CPU". Which, all in all, sounds rather like how the 68000 vs Video works in the later machines; as the VIC has a dedicated 4-bit mini-bus connection to the half-kilobyte "colour RAM" chip (presumably accessed at a higher frequency derived from the master 14MHz colourburst crystal?), which lies outside of the normal 64KB memory map, and that is responsible entirely for holding information about the attributes of the background character/tilemap, one assumes that in normal use the VIC alternates reading actual character information from RAM with attribute information from the ColourRAM (during phases where the CPU has memory access), and BadLines are... IDK... where the ColourRAM has to be updated from normal RAM, via the VIC, as it can't be directly written by the CPU? Perhaps it's refreshed on one out of every eight scanlines, always copying the data afresh with each new frame as that's a lot simpler than trying to determine when and what to copy, and as the characters are blocks of 8x8 (or 4x8) pixels the colours read from the first line are good for the other seven; in this way, the processor runs at full speed almost all of the time, only being halted for the active width of one scanline out of eight through the active height of the display... or in other words, about 80 out of 114 ticks, on 25 out of 262 or 312 lines per frame = 5.6 to 6.7 percent of the time, which again could be ameliorated slightly by clever time-sensitive machine language programming. Having to run just four short additional traces to one very small, simple, cheap additional chip on the board, to enable the CPU to run at about 93 to 95% of its theoretical maximum potential, with an otherwise maximally simplified design, seems like a reasonable compromise, vs either massively slowing it without that small additional buffer (losing more like 45 to 54% of its speed), or having to provide a wholly separate memory bank just for the video system in order to recover that last 6% or so.)
(Indeed, though it was not a conscious bit of copying, a thought-exercise design I came up with for a more comprehensively enhanced STe vs what Atari actually delivered incorporated something vaguely similar in order to allow more Amiga-like graphics modes without a second bus or increased clock frequencies; essentially, a small SRAM buffer which could be DMA'd into in the usual video-readout fashion, starting immediately after the end of an actual display period, but not stopping - and, in the real life machine, essentially idling/wasting cycles, not even recovering them (as the Amiga does) for Blitter or Disk transfers - during the border/blank area. The only slightly modified (to add extra modes) main video chip then reading out of that buffer during the usual display period, starting just as it becomes completely full up, and FIFOing for the remainder of the display time (reading out at a greater rate than it's being written into, but discontinuously), until it's emptied at the end of the active frame. In this way, a fairly cheap and simple 16KB SRAM, a mildly upgraded "shifter", and a minimal tweak - not addition - to the existing Glue logic would allow a 50% increase of either resolution or colour depth (enough to finally produce a 16-colour 80-column display, albeit with only a 6x8 font, or with 4 colours/greys in the 400-line mode, or a true 640x200 display with 8 colours, or 320x200 with 64... or, say, 20% to one and 25% to the other... or a reduction of one with a greater than otherwise expected increase to the other) over what could be provided by the usual 32KB screenbuffer, and without causing any additional processor slowdown (UNLIKE the Amiga case), simply by taking advantage of the unused and "wasted" areas of the video scan. It's not quite the same idea as the C64 colour RAM / Badlines, but there's a good bit of overlap between them)
(Atari... IDK what their approach was in that regard, but everything was a bit more modular and separate anyway, so there may have been full separation. But the wide proliferation of video modes, including some pretty weird ones with e.g. "1.5 bit" colour depth, means that it was probably all shared, and left up to the programmer to decide on an appropriate division of system bandwidth between AV and CPU)
Later on is where this model started to fall apart, with systems like the A1200, Mac LC, Atari Falcon (somewhat; its architecture was largely designed to extend and make the most of a shared-memory model with a focus on improved graphics, but still suffered in the higher modes) and even the PCjr / Tandy series, not to mention a good bunch of peri-millennial PC systems which used AGP to share system memory with graphics, all showing increasing levels of system hamstringing with use of high quality graphics modes, particularly anything beyond the level of Generic VGA (ie, 320x200 256-colour thru 640x480 16-colour... in some cases the latter having to be interlaced to retain speed, unless the former was scandoubled anyway). But the fact that their makers thought that it was still a viable idea - and essentially provided for programmers and users to, as before, decide on the best mix of graphical quality and processor speed on a case-by-case basis (and, in the LC and A1200 at least, still provided for multiple memory areas so a well-behaved operating system or bootloaded program could keep time-critical code away from the AV memory) - shows that they had got well-used to it being an entirely workable concept with their earlier machines.
Oh, and another couple of considerations, which are certainly relevant to both the PC, and the TMS based machines: Code complexity, cross compatibility, and speed of video updating. Having to send data through a different chip, possibly at the end of a whole additional bus, makes for less efficient code than simply dropping it into a certain place in main memory, or moving it from one memory address to another. You have to write more lines in your programs to carry out the same task, you may have to duplicate data to a greater extent (having some things both in your main RAM and in the video RAM, for speed of access or just displaying it at all, and paging things in and out as needed), and it takes longer. You can't just MOV a bunch of bytes using the same mechanisms as any other transfer, but instead have to shuttle them through another chip, with particular instructions, acknowledgements, maybe a slower bus in-between, and that other chip isn't guaranteed to be particularly efficient or work as fast as the rest of your system. Plus in most of these early examples the available maximum VRAM is pretty limited (typically 16KB) and it doesn't always make the best use of it... EG often the only settable modes use the entire memory as a single display page... and even if they don't, the half-page capabilities are so poor that you wouldn't realistically employ them for anything but the most simplistic purposes. So you have to suffer a much slower full-screen update, by streaming the entire new screen across whatever bus is between your main memory and the video controller, and avoid doing that instead of rewriting small portions (or, where the facility even exists, moving sprites around on top of a static background), whereas on a lot, if not all of the shared-memory systems mentioned above, the screenbuffer is quite a bit smaller than the total memory space (and crucially, a good bit smaller than the range accessible by the AV system), so you can have multiple video pages (a la Hercules, EGA, VGA and later, more memory-dense PC video cards, or the graphics memory of later and more advanced consoles) within the shared memory area and flit between them with a simple single register write - that is, changing the start address of the active video area. Boom, double buffering and all its advantages, such as moderately smooth scrolling and/or sprite movements without "tearing", zipper-effects, orphaned-pixel trails/smears, or "snow", are yours for the taking, if you even need to use them anyway because of the increased write speed to the video memory and much improved ease of blitting sprites in from nearby offscreen areas, or shifting the entire image around by a few pixels (or chunky-scrolling by whole character spaces in the hybrid text/graphics models), if indeed the video chip itself doesn't allow hardware sprites and scrolling (for which a larger-than-the-screen memory page is absolutely essential) in the first place, obviating some of those procedures entirely. This in part is why a lot of early PC games are single-screeners; the background is fixed and everything that moves does so against that static background, with any change of scene being a flip-screen affair. EG, Alley Cat, Round 42, or any number of first-gen arcade conversions (early arcade boards themselves being pretty simplistic and having very little memory). Also why almost anything that uses the TMS system or a derivative - including both the NES and SMS, as well as the first generation of MSX machines - operates on a tilemap (ie, background is essentially harware-scrollable textmode, but with a customised graphical character set) and sprite basis, rather than using bitmaps, because that's the only way to produce a dynamic, scrolling display with the available memory and update bandwidth (and usually they cheat somewhat by using the cartridge as a character ROM, instead of relying on having one in the machine, or using internal RAM to hold that data, so the notional memory space is a bit larger)... otherwise they'd be stuck with single/flipscreen action as well.
*(ironically, textmode could often be what made the highest demand, and was one reason CGA had to be a discrete card; 8 graphics pixels in monochrome use no more bandwidth than an 8-bit text character of the same width, and in 2-bit/4-colour mode they use as much as a character plus simple attributes... and on top of that the character/attribute data has to be converted into actual pixels by feeding through a font ROM and colour gating system... the only saving is memory amount because that same character is read out from the same RAM address several times to build the image one scanline at a time. An integrated system probably could have dealt with CGA's graphics mode, and 40-col attributed or 80-col unattributed text, but the 80x25 full colour textmode that formed the basis for all standard PC text displays for the next 30+ years would have been unsustainable... hence the original problem of having to choose between slow / flickery screen-blanking Vsynced updates when scrolling, or putting up with "snow" as data transfer interferred with read-out to the screen)
PS. The Spectrum's method of producing separate memory areas is likely rather simpler, and less costly than it would be in the Amiga... it is after all already a very pared-down architecture with the minimum of chips. Having a bank of 16KB and another of 32KB means you only need 14 (or maybe 7, if they're time-multiplexed) address lines going one way, and 15 (or 8) the other. Or if the lower bank shares space with the ROM (backwards to how you'd usually expect it - often, ROM is freely accessible by the CPU, but Sinclair was often found going against the grain, and in any case you're gonna need access to the character glyphs), each of them only needs 15 lines, instead of the 16 for the whole system. It's a small saving, but it's still a saving. In the 68000 system you'd have a minimum of 18 on the shared side (assuming 512KB and word rather than byte width access; more likely 20) and potentially 22~24 for the CPU-only side. Of course, the data lines would only count 8 on both legs, rather than 16, and I wouldn't be entirely surprised to find that they're actually shared and time-multiplexed somehow (like the chips are strobed one clock tick apart and the data coming off them gated, buffered inside the ULA, and delivered in just-in-time fashion to their respective destinations - which is somewhat like how the Atari ST Glue logic works when arbitrating CPU vs Video memory access; the access request from one chip is woven around the response to the other, and the actual addressing AND data is buffered through the chipset). And, in any case, the lines are WAY shorter; the Amiga motherboard version may find itself routed through 16 (1x256K, so each data line has a different length and path, alongside the address lines) internal chips for the video part, and off to an arbitrarily-placed external connector on the FastRAM side. The Spectrum one? A single 8x16K chip on one side, and two on the other, each getting all of the data lines and most of the address lines, making the routing short and uncomplicated... And of course the C64 one is simpler still. Just four (eight?) extremely short traces between the VIC and the CRAM. Comparing either of those (or, say, the Falcon's Videl) to the Amiga, or the entirely separate video and VRAM subsystem in a TMS design, let alone the complete multi-slot expansion bus in a PC, is like apples to oranges.
PPS FWIW the BBC used 4MHz memory. It clocked the memory bus at twice what the processor and video sections ran at, allowing zero-contention access by both to the same memory areas. It just so happened to exist at a time when that speed of memory had become affordable, but the processor speeds hadn't yet increased to match. Thus CGA-like graphics (but with greater height, slightly better choice over palette, and a higher-colour/lower-resolution option, with no "snow" or other silliness, at the expense of its most colourful textmode being limited to a "mere" 40x25 in 8 colours) and a fairly nippy 6502 (probably acquitting itself quite well against the PC's 5MHz 8088 in a straight fight), all on a cost-efficient shared system architecture, leaving overhead in the budget for all the additional nice-to-haves like the coprocessor bus, sideways ROMs, disk operations, decent BASIC, surprisingly good soundchip (at the time, only really beaten by the SID), relatively professional keyboard, monitor output, more ports than you can shake a stick at, EcoNet capabilities, surprising memory expansion capability, etc. Showing that, at least with an extra year or two of development and serendipity, you need not make something along the same lines as, or costing as much as, the IBM in order to enjoy similar performance.