Your question seems to have an inherent contradiction in it. You appear to be saying that you don't want to "[d]isassemble the original game and rewrite the...routines," but all the examples you give (intercept a call to a memory location, intercept a software interrupt when a register has a specific value) are ones that would require the kind of knowledge of the code you get only through disassembly (or access to source code) and, once you have that, the easiest way to implement changes is usually just to change the code directly.
I doubt you'll find an emulator that provides that kind of "intercept" facility because that capability is already easily available in the machine itself: just change the code at a memory location or an interrupt handler to do what you want. (This is in fact often exactly how debuggers do this.)
Note that this technique does not limit your capabilities in any way: you can use it to call new code in any language runnining within the emulated machine or, by using the I/O facilities provided by the emulator, communicate with code running outside the emulated machine. (Your new routine needs to provide the API provided by the original routine, but it can of course marshal arguments for the substitute routine as necessary, e.g., push information in registers on to the stack to call compiled C code or call into a Python interpreter, and when that returns, unmarshal the results back into registers to return to the game code.)
Let's look at your examples:
If the subroutine at 0x1234 returns a key code whose value is used for movement (say, one of the two arrow keys) and you wish to use a gamepad, replace the routine with one that calls the emulator's I/O routines for gamepad input and translates the left/right presses on the gamepad to the appropriate key codes.
If you have a machine learning model running in another process outside the emulator, attach the communications channel between these two processes as an I/O device into the emulator and, in your substitute routine(s), write information out to that I/O device to send it to the other process, reading from that I/O device to get whatever information the other process wants to send back.
To replace only some of the code for a software interrupt handler (or anything else, for that matter), copy that routine elsewhere and replace it with your own code that calls your new routine if AH = 0x3F or calls the original code otherwise.
Changing the display of a side-scroller to have a bigger screen area can't be done by just replacing a few routines. As Bruce Abbott points out, this typically involves substantial changes to the game code itself because it won't be drawing more than the screen area and a little bit around it on a continuous basis, as you can see in the examples given at 32:32 and later in Michael Steil's The Ultimate Game Boy Talk. (The link I give you starts at 29:15; to understand the difficulties with what you're proposing you very much need to understand everything in the Pixel Processing Unit section of that video.)
Loading more data than fits into the memory/address space limitations of the original hardware has similar deep issues: the game doesn't know how to access the additional memory you wish to add, and you need to change the game itself to understand that the additional memory is there and how to access it. There's no way an emulator can do this for you.
Speeding up parts of the program to make it do more (e.g., allow more players in a multiplayer game) is an even deeper problem yet: remember that most action game code outside of modern platforms has assumptions about machine timing deeply embedded throughout the entire program: this is why older IBM PC clones used to include a "turbo" button to slow the machine down to original IBM PC speed, and why emulators take very great care not to run instructions faster or slower than the original hardware they're emulating.
It's certainly possible to write a "scriptable debugger" (as flavio aptly terms it) integrated into an emulator, but I doubt anybody would see much point, since once you've decided to change something beyond the API provided by the hardware I/O devices in the system it's easier and more reliable just to change the code you're running.
Because you're asking here about a particular implementation of a solution to a seemingly vague and general problem that you don't describe in detail, on reflection I think this question should probably be closed as too broad. I'd suggest you create a new question stating exactly what game you're trying to modify and describing specifically what you want to change about it, and let others suggest the best way of implementing that change. Once you have a few questions like this, you may be able to start asking about more general techniques for dealing with that type of problem. (For things like this, it's almost invariably better to generalize from specific solved cases, rather than try to generalize before anybody's seen specific problems and solutions.)
An Example Change
Your question appears to be an example of an "XY problem": you're asking "how can I use X to do Y" when the easiest solution is "don't use X, use something else." Here I provide a small example that I hope will help clarify this.
Let's say a game uses the arrow keys to move left and right because when it was originally designed the computer it ran on didn't have a gamepad. A gamepad was later made available on that system (as an expansion card or added to a later version of the computer) and the emulator emulates this.
You've disassembled and analyzed the game to the degree that you know it calls a routine at
$3210 to check for input, and expects
2 to be returned in the
A register indicating nothing, left or right being held down at that moment. The routine itself writes and reads I/O ports to scan the keyboard.
Your suggestion for doing this seems to be to add code to the emulator that keeps a list of addresses at which it should stop emulation, and call host code that is dynamically linked in in some way. That code then needs to use internal APIs (if they even exist) to interface with the emulator's code that handles device input and to get the gamepad status, load the emulated machine's
A register with the appropriate
2 result, and somehow externally reset the machine's state to return from that call to
$3210. This is quite tricky; perhaps it could fake an instruction load of
RET instead of letting the emulator load whatever instruction the PC currently points to? And keep in mind that you may have to maintain timing dependencies, too, by delaying for however long the original routine would have taken.
That's the hard way of doing it. The easy way is simply to replace the routine at that location with one that reads different I/O ports supplied by the emulator to read the gamepad instead of the keyboard. This requires no changes to the emulator at all, no access to the emulator's internal APIs or internal code, and is much less code and much less complex code overall. Even if the emulator doesn't offer the emulated device you need, it's far easier to add just the emulation for that device than to add what's essentially a debugger and go grovelling around in the internals of the emulator's code. Note that the emulator side of the device can do anything it wants on the host machine, even starting and communicating with other processes or the like.
You're not likely to find emulators that already support your solution because why would the developers of an emulator go to all the work of adding this pseudo-debugger and internal APIs and the like when there's a much simpler solution available?
If it's really important that an unpatched binary be loaded, you might tweak the emulator to load and apply patches after the original binary is loaded, but before emulation starts.