As I understood, you want to make something that behaves like any possible NES cartridge and can be changed (at least for simulated ROM content) on the fly (in real time)?
And it should be cheap and open source?
Quick Answer: Dream On
The Full Monty
What you're asking for is something like test equipment for chip manufacturing. There are units that would satisfy sour requirements and speed - but they are neither open source, nor cheap - think 100k EUR for entry level. More important, none will fit in a NES cartridge.
Of course such a beast would be able to not only react to all your needed transfers in time but also emulate next to any special circuitry.
Lets Shovel
It's a matter of speed (and memory). You already touch on this with your rough numbers
(1.x MHz NES -> 1.79 * 72 * margin-factor MHz microcontroller? Or less because only a subset of all those pins are needed)
That would give some 128 Mbit/s (or 8 MByte). Except, that's only if all data is prepared in sequence or requests. In reality, the CPU sets up a request and wants a reaction within less than 200ns. So even when only looking at the data lines, it needs to transfer 72 bit to the emulation thingy, and 8 bit back all within 200 ns, cranking the needed connection up to ~800 Mbit/s.
Just to transfer the data we are already near a PCIe 1.1 line not counting the overhead - and with no necessary processing on the PC side included. Not to mention all the various latencies a PC may have to feed the data in time.
The fastest Pi Network is Gbit with the Pi 3 B+. So that alone wouldn't make it. Also a Pi wouldn't be able to do it locally.
A Smaller Size
Well, if we ignore all special hardware cartridges and go just for something with ROM and the ability to change that data at runtime, the picture becomes very different. One could use some dual port RAM and a microcontroller. One side connected to the NES interface, maybe with the usual banking logic added, while the other side can be filled (or read) from said microcontroller.
To the NES this looks like a perfectly normal ROM cartridge, even (simple) RAM as with Super Mario Bros 3 can be done. From the PC side, each cell can be read or written.
Of course, it's again a matter of speed. But now it depends more on the amount of data you want/need to change within a certain time. This again depends on the programs you want to write. A high speed USB 2.0 will already be able to deliver data faster than the NES can read it - as long as it's semi streamed, not totally random.
In fact, if the data is read from only a few cells in sequence under program control even a USB 1.1 full speed can do the trick, as even with the fastest access a 6502 needs 4 clocks to read a single byte (LDA <ABS>
), and many more if it's doing something with the data. This turns the needed bandwidth down to less than 500 KiB/s - USB 1.1 full speed can deliver up to 1 MiB/s.
With some thought it might be possible to write a low overhead protocol to deliver your data just in time.
And no, Dual Port RAMs aren't overly expensive. 1 Mib (like in 128Kib x8) is around 30 Euro per piece. For a one off design, chip prices do not really matter in comparison to the time invested.
The Gist of the Matter
The game would copy the same sprite or background tile from the same memory address but get something different than a previous cycle.
Hmm, when reading this, it sounds as a single pipe could also do the trick?
So why not doing some more common FLASH based ROM emulator (maybe with a Arduino for reload without unplugging) and a FTDI FT232x based USB-FIFO? From the NES side this could look like a single memory location delivering a new byte each time it is accessed, while the PC sees a high speed serial connection. The internal buffer of up to 2 KiB would cover next to all possible timing issues. And the back channel could be used to send commands to the PC.
In Lockstep
(I'll worry about how to synchronize and not be in the middle of a memory copy later)
Synchronizing the game to the USB clock would make a great start.