Well, since the question itself is way too broad for any useful answer not being a list answer, let's add some FM7 background
The FM7's Sub-CPU, the one doing the graphics, had its own a Sub-Sub-CPU, a 4 bit controller doing the keyboard :))
Having multiple CPU's each operating a dedicated part was quite common for professional systems, not just the IBM-PC having an 8 bit keyboard controller, or the Superbrain having a second Z80 CPU to handle it's FD system. And a professional system is where the FM7 inherited it: The FM8 introduced in 1981 - soon to be replaces by the FM7 as "low end" and the FM11 as business system. Well, not so "low end" with a price of ~600 GBP at a time when the 48 KiB Spectrum, a true home computer, sold at less than a third of that (175 GBP).
Being a business system and as such poised to handle the high variety of Japanese character displays, the FM8 got a dedicated graphics system with it's own 6809. Not so much for speed concern, although it helped, but to simplify software design and, most important, leaving enough usable RAM for applications - after all, the display had to be plain bitmap, thus already a B&W one would need 16 KiB, having 8 colours tripled that. So no chance to have that within the main address space.
As feature reduced version of the FM8 the FM7 inherited that feature, still being an upper end machine with 64 + 48 KiB RAM.
Comparing the Sub-CPU to a GPU replacing, including replacement of classic video controllers like VIC or 6845 does not really match up. Yes, the 6809 does a lot of bit fiddling, but there is still dedicated hardware outputting RAM bitmap to video. So it's more like taking a CGA (or VGA) and add a CPU. The picture is displayed anyway, without any help of that CPU, but with it, much more tasks can be offloaded.
Since the FM series Sub-CPU also had its Sub-Sub-CPU for the keyboard, the whole system is more like a close coupled terminal. An idea used by several other machines as well. Just think of various Z80 cards for the Apple II, where the host machine is 'reduced' to being a terminal, handling not just video, but all I/O including disks (*1) or sound as well(*2).
In fact, there is another quite popular UK machine which made that even a systematic feature: The BBC Micro. With a second CPU added, the BBC becomes a sub-system to whatever second CPU is plugged in - including another 6502. So quite like the FM machines.
Lesser known, but quite admired back then was the 1978 MicroAngelo card by SCION, a S100 bus high resolution graphics card with local memory and its own Z80 CPU to execute graphics primitives. Its manual may be a good read to gather ideas about instruction building.
So, how does it work?
Since both are complete systems, all it needs is a communication linke. That can be a serial line, like with a classic terminal, a pair of 8 bit wide ports (think two 6522 in handshake mode), some FIFO (speeding communication quite up), or some dual ported RAM. The later being what is used in the FM7.
There is a 128 byte RAM located at $FC80 used to send commands to the Sub-CPU. Whenever the CPU wants to do so, it
- checks a port bit to see if the Sub-CPU is still executing, or idle
- when idle the Main-CPU halts the Sub-CPU
- writes a command byte and optional parameters to the dual port RAM
- releases the Sub-CPU which in turn
- sets the busy flag
- takes the command byte and executes whatever is requested.
- optional returning data
- finally setting the busy flag and
- return to idle operation.
Call this 'Theory of Operation', but there is no secret trick or incredible complex operation, only two independent computer systems talking via a communication link.
I guess at minimum that would be three commands like
- Initialize System; maybe accompanied with basic settings for foreground/background/border colour and alike
- Clear Screen
- Output a Character
From there on, add whatever might be useful, like
- Output a String; speeding up communication
- Position Cursor
- Set Character set
- Load Character Definition
- Set Character Orientation; like for writing at arbitrary angles
- Set a Graphics Cursor
- a Line
- a Rectangle filled/unfilled
- a Circle
- a Series of Lines
- a Circle Section
- ... whatever
- add BitBlit with off screen sources for sprites etc.
- add memory read/write to manage such resources
- or even upload custom code.
I guess you get the picture by now. No dark secret magic, just outsourcing the very functions your programs need.
P.S.: Unlike the FM I'd rather not build my own video output hardware but use a ready available 9958, to spare the additional development, even if just used in a fixed bitmap mode.
P.P.S.: Oh, and if you're add it, think about a third CPU (well, fourth considering keyboard&mouse :)) dedicated to sound. Since sound needs to run continuous and uninterrupted it will benefit even more from having it's own CPU. If drawing lags at times it'll be a minor nuisance, if sound lags, it'll be a major one.
Communication might be done much like with video. Also, using an existing synthesizer chip (FM preferred) will simplify design and give the CPU more room to handle complex higher level stuff, like preparing instrument tables, translating MIDI, mixing sounds or even creating 3D positioning. The Sky is the limit :))
*1 - Another example is the 1979 Superbrain employing a secondary Z80 to handle the Floppy drives. Of course going tha way also leads to all those machines that had 'intelligent' peripherals, from HP and Commodore all the way to Apple.
*2 - The main differences in case of the Apple II is that both CPU share the same memory and only one can work at any given time.