The TMS9918 video display processor was designed in the 70s
To start with, the 9918 was not being designed as a general purpose chip, especially not with Z80 systems in mind. It was a special design, made to fit for the TI 99/4. Before the 9918, there was the 9917 VTC (Video Timers/Controlers). A device comperable to the 6845 VDC or similar, capable of reading character codes just in time from a screen RAM, filter them thru a character ROM and build a screen frame.
To change content of the screen RAM from a CPU, that RAM needed to have a second port. And if that RAM should be a dynamic one, a third access path is needed for refresh. All much like with other contemporary VDC.
For the TI99/4 its designer didn't want to add a complicated mux logic for dual port access as well as they shunned the idea of building/adding a DRAM controller. Also some graphics abilities would be nice - after all, a homecomputer would look better with then without.
So the 9918 was born, integrating all of these components. Graphics, DRAM refresh and a second access port to make the RAM reachable by the CPU - plus some other components to reduce the 99/4s chip count. At it's core it's a special chip to fit the 99/4 design, not targeted at other designs. Having it also generating the CRU clock for the GROMs is not the least hint (*1,2).
Bottom line: The 9918 isn't your of the shelf general purpose beefed up VDC, but a special to type design for the 99/4A. Much like the VIC is made to fit the C64 - except that computer was a less advanced design. Imagine if Commodore hat used the VIC (or TED) not only in their machines, but also offered for sale like TI did.
in 15 memory cycles out of every 16, which can limit throughput somewhat
Keep in mind, these are VDP memory cycles, not CPU which in average were way slower.
especially if connected to faster than average processors: a 3.5MHz Z80 (e.g. as used in the MSX) might need a small delay between each access in a tight loop 1, but the VDP was still used into the mid-80s, by which time 8MHz Z80s were commonly available, and such a delay would be very limiting for such a processor.
Not really, this is a somewhat screwed assumption, as the 9918 memory is its own address space, not main memory. Of the 5 memory cycles of an ODIR (String output) of a Z80, which would be the appropriate use to shovel large amounts of data from or to VDP memory only one. that means only i one of 5 cases it has to wait. While that LDIR example may during some time break down effective speed to act more like a 3.5 MHz one, it even doesn't do so always (only in graphics mode and during line display).
In a real setup a CPU doen't usually just shovel big amounts of data in a continuous stream. For example to move some (non sprite) block, it will read one (or two) bytes, do some bitblitting then write them back and continue the next line until the block is done. This includes many non video memory operations, where the VDP RAM access time is not involved. Even more so, for everything else like sprite moves or such only single access cycles are needed - inbetween many many other. So in a real word application an 8 MHz Z80 is virtually not slowed down at all.
Bottom line: There is no essential performance gain in having a full dual port memory design, but a lot of additional cost.
The VDP is designed to use typical low-cost DRAM available in the mid-70s,
At that time not low cost, but the only available ones.
[... Argument about using a different RAM architecture and hidden access ...]
Sure, one could, but that would have made the 9918 a different chip with a different architecture. Plus needing to redo all software using the existing architecture - which is eventually way more expensive than 'just' a few chips.
It also would have opened a whole snake bin about addressing. A 9918 can be used by any CPU that could handle a single port plus a single Mode (address). No matter if it's a microcontroller or a 16 Bit CPU. And for classic 8 bit machines with their 64 Ki address space it offered the great advantage of having a separate video memory not cluttering the main address space - in case of MSX allowing the machines to have a full 64KiB RAM and all of it available to OS and application. Video was essential just an I/O device like a terminal ... except faster.
In case of MSX, that concept even enabled to soon reach 64 KiB of video memory and finally 192 with much improved capabilities and not taking away a single bit of main memory. Doing so in a shared memory with an 16 bit address space would have required rather complex and unsatisfying bank switching, adding slow software layers to make it available to anything but brute force assembly.
Bottom Line: Switching the interface and access strategy would have increased cost, reduced usability and brought no real world performance gain.
Were there any systems that did this, or if not is there a reason I'm not seeing why they either couldn't do it or it would have been too difficult to do?
No, the true question is why should they? The dual port logic was already provided by the 9918, so access was possible, there is no real life penalty by using it, but a huge gain in upward compatibility and extension, not to mention preserving the full 64 KiB address for the main CPU.
On a wider perspective than just the memory access issue, the TMS9918 can be seen as the genuine ancestor to all of today's video systems. While others like Apple, Atari, Commodore and Amiga extended a main memory concept (who needs 64 KiB anyway, lets use some for graphics), did the 9918 leapfrog all of this and offered a complete separated graphics subsystem. Much like IBM did with VGA (*2) and every of today's graphics cards provide a clear API and no display memory wrangling.
Using a separate and well defined hardware interface not only enabled easy and independent upgrade for larger memory size, but also adding high level screen manipulation functions like window related coordinates and regional memory handling but also line drawing and alike. All without changing much in the interface or main CPU memory.
*1 - The 9918 was also tasked with generating the CPU clock (hence the
CPUCLK pin) for the planned single phase 9985 CPU, which should run at 3.58 MHz. Using the 9900 with it's 4 phase clock required a 9904, so the 'CPU'-clock was only used for the 9919 sound generator.
*2 - Another great hint is the missing of a wait signal (or
READY in case of the 9900 interface). With the CPU speed intended for the 99/4 it would have been impossible to overrun the access time anyway. If it would have been intended as a general purpose chip, outputing a wait signal would have been way more useful as CPU or GROM clock. Adapting other CPUs would have been made transparent due hardware toggling (wait cycles) instead of cycle counting.
*3 Well, they did add a banked shared memory access, but here again the CPU was slowed down when video access was done - exactly the same way as with any other dual port design since 9918 :)
Primary intention of the designers was for user s access graphics only thru their high level interface. Offering that banked access mode was shooting themself in the knee.