2

EDIT: previously I added a lot of examples to this questions. I removed the examples, because the question was voted to be "too broad", so I put more emphasis on the question again.

Question:

Is there an example of a library written in a "modern" multi-purpose language (C++, Python, Java, Go) that loads binaries of an old platform (e.g. 8-bit, 16-bit or 20-bit DOS) and emulates it partially until a given condition, like a break point?

I am looking for a library that allows mixing emulated old binaries with a modern language that doesn't have the same limitations.
I am not looking for an emulator with a scriptable debugger, unless the debugger can be scripted in a common multi-purpose language without restrictions and can be invoked in an automated way.
I am also not looking for software disassembled and ported to a new language.
I am also not looking for porting a modern language to an old system.

  • 2
    Not saying that it is an easy way, but MAME takes care of this. Emulates a lot of platforms, is open source and you can tweak and recompile the platforms you like to change. – UncleBod Aug 28 at 14:26
  • 1
    @UncleBod As mentioned in point 1), I am not looking for an emulator alone. If you can give an example of how MAME is used as a library you can make this an answer. – Helena Aug 28 at 14:59
  • 1
    @RichF As mentioned in point 1), I am not looking for an emulator alone. If you can give an example of how DOSBox is used as a library you can make this an answer. – Helena Aug 28 at 14:59
  • 3
    I'm also failing to see the suitability of MAME/DOSBox here given the scope of changes it sounds like Helena has in mind. Suppose she wanted to add a particle effect when Mario jumps — if you're writing Mario-specific code and tapped in to the full state of the hardware you know which sprite he is so you have his location, you can spot entry into the jump routine, so throw on some particles. You couldn't do that on real hardware due to sprite limits and constrained processing resources. So you couldn't easily patch it into an emulator without being able to do an end-run around the emulation. – Tommy Aug 28 at 15:16
  • 1
    What you are describing is not the function of a library IMHO. You are asking for an emulator (library or not) with an scriptable debugger, OR the integration of a debugger and a decompiler/patcher (IDA + HexRays here) But you specifically said no disassembly, so... I believe you can use MAME source code as a library to emulate and patch the game, but it will be far better if you just add those capabilities to the existing debugger (but this is no easy task, i think). – flavio Aug 28 at 18:16
2

I am imagining to combine the best of both worlds by having a emulation library that allows me to do everything an emulator does, but also gives me the opportunity to only emulate parts of the code (e.g. run a subroutine until it returns), or have callbacks that give the control back to the language I am calling it from.

This is what emulators already do, since at some point they have to translate virtual machine operations into real OS calls and operations that access the actual hardware. Many emulators have a built-in debugger with breakpoints that can be set at code locations or when certain operations occur, which could be a good place to add your own callback hooks.

However this doesn't mean you can just tap into the emulated code at any point and jam stuff into it expecting it to work. Most games for home computers such as Amstrad CPC etc. were written in assembly language (or even raw machine code) without the use of standard libraries or APIs other than perhaps file and keyboard I/O. They often used tricks such as self-modifying code to squeeze more out of the limited memory and processing power. And they were intimately tied to the hardware, wringing the most out of it with whatever worked (sometimes using undocumented 'features' that have taken emulator writers years to reproduce accurately).

Once you start playing with the code all bets are off, unless you know exactly what it is doing and how it can be safely modified. Even something as 'simple' as increasing the number of characters or making the screen bigger is likely to break the code that uses that data. Graphics may be stored in a non-standard format which is specific to the particular game, with a structure that needs to be understood and preserved.

What this means is that before you can consider using emulated code as a 'library', you need to understand what its requirements and dependencies are. Making significant enhancements such as larger screen size or more objects may require extensive modifications which increase the chances of creating bugs.

is there any examples of games that are extensions of old games that use a hybrid approach of emulating code of the original platform and running code on the emulating host platform.

Closest I can think of is the SPEC256, which is a ZX Spectrum emulator enhanced to provide 256 color graphics. To handle the extra memory required it runs a '64 bit Z80' in parallel with the normal 8 bit CPU emulation. The developers have created tools for extracting monochrome graphics from a game and integrating the 256 color equivalents. However they haven't released the tools or any source code, and I'm betting that each game needs a lot of custom patches to get the 256 color graphics working properly.

  • Thanks for bringing up SPEC256, this looks indeed a partial match to what I am looking for, as it goes beyond the limitations of the original platform. – Helena Aug 29 at 6:33
  • Yeah, I wrote a Spec256-alike a while ago, and avoiding custom code is almost impossible. I targetted the game Dan Dare, and one thing it does is it reverses its sprites using a 256 byte lookup table. So you're not really going to succeed with a parallel 64-bit Z80 alone. I had to trap the sprite flipping routine and implement that in native code. – Tommy Aug 29 at 22:30
  • See the The Day of the Tentacle remake post-mortem here: youtu.be/HqWrdIf69M8 it emulates the DOS version and then snoops it's memory to generate the improved graphics. – user3570736 Sep 1 at 10:44
2

Your question seems to have an inherent contradiction in it. You appear to be saying that you don't want to "[d]isassemble the original game and rewrite the...routines," but all the examples you give (intercept a call to a memory location, intercept a software interrupt when a register has a specific value) are ones that would require the kind of knowledge of the code you get only through disassembly (or access to source code) and, once you have that, the easiest way to implement changes is usually just to change the code directly.

I doubt you'll find an emulator that provides that kind of "intercept" facility because that capability is already easily available in the machine itself: just change the code at a memory location or an interrupt handler to do what you want. (This is in fact often exactly how debuggers do this.)

Note that this technique does not limit your capabilities in any way: you can use it to call new code in any language runnining within the emulated machine or, by using the I/O facilities provided by the emulator, communicate with code running outside the emulated machine. (Your new routine needs to provide the API provided by the original routine, but it can of course marshal arguments for the substitute routine as necessary, e.g., push information in registers on to the stack to call compiled C code or call into a Python interpreter, and when that returns, unmarshal the results back into registers to return to the game code.)

Let's look at your examples:

  1. If the subroutine at 0x1234 returns a key code whose value is used for movement (say, one of the two arrow keys) and you wish to use a gamepad, replace the routine with one that calls the emulator's I/O routines for gamepad input and translates the left/right presses on the gamepad to the appropriate key codes.

  2. If you have a machine learning model running in another process outside the emulator, attach the communications channel between these two processes as an I/O device into the emulator and, in your substitute routine(s), write information out to that I/O device to send it to the other process, reading from that I/O device to get whatever information the other process wants to send back.

  3. To replace only some of the code for a software interrupt handler (or anything else, for that matter), copy that routine elsewhere and replace it with your own code that calls your new routine if AH = 0x3F or calls the original code otherwise.

  4. Changing the display of a side-scroller to have a bigger screen area can't be done by just replacing a few routines. As Bruce Abbott points out, this typically involves substantial changes to the game code itself because it won't be drawing more than the screen area and a little bit around it on a continuous basis, as you can see in the examples given at 32:32 and later in Michael Steil's The Ultimate Game Boy Talk. (The link I give you starts at 29:15; to understand the difficulties with what you're proposing you very much need to understand everything in the Pixel Processing Unit section of that video.)

  5. Loading more data than fits into the memory/address space limitations of the original hardware has similar deep issues: the game doesn't know how to access the additional memory you wish to add, and you need to change the game itself to understand that the additional memory is there and how to access it. There's no way an emulator can do this for you.

  6. Speeding up parts of the program to make it do more (e.g., allow more players in a multiplayer game) is an even deeper problem yet: remember that most action game code outside of modern platforms has assumptions about machine timing deeply embedded throughout the entire program: this is why older IBM PC clones used to include a "turbo" button to slow the machine down to original IBM PC speed, and why emulators take very great care not to run instructions faster or slower than the original hardware they're emulating.

It's certainly possible to write a "scriptable debugger" (as flavio aptly terms it) integrated into an emulator, but I doubt anybody would see much point, since once you've decided to change something beyond the API provided by the hardware I/O devices in the system it's easier and more reliable just to change the code you're running.

Because you're asking here about a particular implementation of a solution to a seemingly vague and general problem that you don't describe in detail, on reflection I think this question should probably be closed as too broad. I'd suggest you create a new question stating exactly what game you're trying to modify and describing specifically what you want to change about it, and let others suggest the best way of implementing that change. Once you have a few questions like this, you may be able to start asking about more general techniques for dealing with that type of problem. (For things like this, it's almost invariably better to generalize from specific solved cases, rather than try to generalize before anybody's seen specific problems and solutions.)

An Example Change

Your question appears to be an example of an "XY problem": you're asking "how can I use X to do Y" when the easiest solution is "don't use X, use something else." Here I provide a small example that I hope will help clarify this.

Let's say a game uses the arrow keys to move left and right because when it was originally designed the computer it ran on didn't have a gamepad. A gamepad was later made available on that system (as an expansion card or added to a later version of the computer) and the emulator emulates this.

You've disassembled and analyzed the game to the degree that you know it calls a routine at $3210 to check for input, and expects 0, 1 or 2 to be returned in the A register indicating nothing, left or right being held down at that moment. The routine itself writes and reads I/O ports to scan the keyboard.

Your suggestion for doing this seems to be to add code to the emulator that keeps a list of addresses at which it should stop emulation, and call host code that is dynamically linked in in some way. That code then needs to use internal APIs (if they even exist) to interface with the emulator's code that handles device input and to get the gamepad status, load the emulated machine's A register with the appropriate 0/1/2 result, and somehow externally reset the machine's state to return from that call to $3210. This is quite tricky; perhaps it could fake an instruction load of RET instead of letting the emulator load whatever instruction the PC currently points to? And keep in mind that you may have to maintain timing dependencies, too, by delaying for however long the original routine would have taken.

That's the hard way of doing it. The easy way is simply to replace the routine at that location with one that reads different I/O ports supplied by the emulator to read the gamepad instead of the keyboard. This requires no changes to the emulator at all, no access to the emulator's internal APIs or internal code, and is much less code and much less complex code overall. Even if the emulator doesn't offer the emulated device you need, it's far easier to add just the emulation for that device than to add what's essentially a debugger and go grovelling around in the internals of the emulator's code. Note that the emulator side of the device can do anything it wants on the host machine, even starting and communicating with other processes or the like.

You're not likely to find emulators that already support your solution because why would the developers of an emulator go to all the work of adding this pseudo-debugger and internal APIs and the like when there's a much simpler solution available?

If it's really important that an unpatched binary be loaded, you might tweak the emulator to load and apply patches after the original binary is loaded, but before emulation starts.

  • 1
    This doesn't really answer my question. As mentioned, I do not want to extend the game by writing new machine code for the old platform. I have been asking for an approach that allows using a modern programming language that can make full use of the host system. Using IO to communicate with a separate thread is an interesting idea, but I would have to write machine code to communicate the game information to the outside. But I am taking your feedback that it is unclear what I want, so I added more examples and what I am thinking of doing with this library. – Helena Aug 28 at 21:32
  • 1
    @Helena The technique I provided does exactly allow you to use a modern programming language and full use of the host system; please re-read my answer carefully with this in mind. I've boldfaced a couple of important points in my question. It's true that you still have to use a tiny amount of assembler code, but the level of understanding you need to have of the original game's code is such that it will be trivial to write this minimal amount of assembler to interface with your modifications written in another langauge such as C or Python, whether running within the emulator or outside it. – Curt J. Sampson Aug 29 at 3:04
  • @Helena I've also added points 4, 5 and 6 to address your new examples; from those new examples you give it's pretty clear at this point you don't understand very well the problem you're trying to solve, and therefore you don't understand why the solution you're proposing doesn't make sense. What you want to do is a lot more complex than you think it is. This is why I suggest you pick a specific game and a specific modification you'd like to make to it and actually try to implement that: this will give you the understanding of the problem that you're currently missing. – Curt J. Sampson Aug 29 at 3:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.