16

Lately I've been interested in how old machines work, in particular an NES. While there are quite a few resources on the basic operations and even some games that have been totally broken down byte by byte, there isn't much information on how this is done.

So far, I've found discussions on two different methods to accomplish this, but neither seems very efficient, so I'm curious to see if there is a better option. Firstly there's converting the hex ROM file into assembly code, but this is difficult to follow and deals with RAM operations, making it difficult to precisely where in the ROM the information is being drawn from. Secondly, you can go in and systematically delete or corrupt parts of the ROM file just to see the effects. I've personally had more success with this, but it takes forever, and is entirely guesswork and doesn't lead to a broader understanding of how the program actually operates.

Are these the only two viable options to doing this? Are there any patterns or behaviors I should be aware of? Thanks.

  • Welcome to Retrocomputing Stack Exchange. This is a really interesting question; I've tried to do this before but haven't got far. Are you looking for all data or just certain bits of data? – wizzwizz4 Apr 26 '17 at 6:01
  • 8
    That's a very broad question, and it basically amounts to "how is reverse engineering done". I guess everyone has a preferred way; I usually start out with an assembly listing, look at data, split it into routines, try to guess what some routines do, and then work my way up. With emulators, you have a lot more options, both for initial guesses, and finding out details. And yes, it takes forever. – dirkt Apr 26 '17 at 6:02
  • 3
    I agree with @dirkt. Another interesting option with games is to look for sprite data and other graphical data; they can be spotted quite easily in ROM dumps. – Stephen Kitt Apr 26 '17 at 8:18
  • @wizzwizz4 I've been searching for enemy behavior data, I've been looking at Kirby for NES and trying to find how enemies transfer their characteristics to Kirby. I've been able to find text and tile data easily enough, but determining subroutines is much trickier. – Aquova Apr 26 '17 at 13:21
  • Emulators enable powerful analysis, though not many do as much as they could. Here's a video of a feature I was playing with a few years ago in an Apple II emulator. A few other emulators have similar features, but I'm not aware of any NES ones. – Nick Westgate May 1 '17 at 23:54
17

I've done this partially with Commodore PET Space Invaders which I used as a test program to debug my Commodore PET emulator.

I used a disassembler to convert the program to assembly language and I then went through the code annotating it as I found out what it does. When I understood what a bit of code did, I would look for its entry point (my disassembler created labels for the target of every JMP and JSR) and rename the label to something useful. That would help me with other bits of code that called the piece I had just reverse engineered.

It helped that, having written an emulator, I was quite familiar with the memory mappings of the IO chips and certain important bit patterns. So, for instance, if I saw a bit of code writing a certain bit pattern to a certain address that I knew was mapped to one of the PIA ports, I knew it would be scanning the keyboard.

It also helped that I could run my emulator in trace mode (albeit very slowly) so every instruction is printed out as it is executed. So, for instance, when a space invader started appearing on the screen, I could look at the trace and see which bits of code were being executed.

It took some time and I basically suspended work on it when I found a bug that was caused by my emulator being too fast. I haven't figured out a way to make it run accurately at the correct speed yet.


To expand a bit, I used the disassembler (da65) from the cc65 tool chain. It has the handy ability to accept an "info file" so you don't need to annotate the generated source from the disassembler. So, for example, the first two bytes of a PET program file are actually the load address. I wrote the following annotation in the info file

RANGE { 
    NAME "loadAddress";
    START $03FF;    
    END   $0400;
    TYPE ByteTable;      
};

which make the assembler treat the first two bytes as data and not garbage assembler instructions.

The next bit was a one line BASIC program that calls the machine code routine that is the start of the program proper. Again, I forced the disassember to treat it as data

RANGE { 
    NAME "basicBootstrap";
    COMMENT "SYS(1039)";
    START $0401;    
    END   $040E;
    TYPE ByteTable;      
};

This time it also causes the disassembler to emit a comment that tells you what it is.

And then it's just a case of laboriously going through the code identifying the function of various bits. For example, I found a short loop that tested a certain bit on one of the PIA registers which I know is the vertical retrace, so it is clearly syncing with the monitor. I add this annotation

LABEL { 
    NAME "waitForRetraceLow";     
    ADDR  $09B0; 
    size $8;
    COMMENT "Wait for the screen to lower its retrace signal";  
};

and the disassembler creates a label for the routine as well as using the label in JSR and JMP from elsewhere.

2

I just answered a similar question of yours on stackoverflow.com which may help: https://stackoverflow.com/questions/43573269/how-does-a-6502-processor-transfer-data-between-rom-and-ram

I am not quite clear what you mean by "deals with RAM operations". Perhaps you could clarify?

Yes, reverse engineering is a long process. I am working on deciphering a games cartridge I wrote 35 years ago and am finding it difficult. Understanding what the code does is one thing but understanding why it does it is far more complicated and requires a good understanding of the console's hardware and the logical operation of the game.

Getting back to your real question though, somewhere in the code it must be accessing the ROM (and again I refer you to my response on stackoverflow.com). What I would do would be to search the disassembled code for any reference to the ROM area. So, for example if your ROM starts at $C000, run Find commands from your text editor for $C, $D, $E, $F. Hopefully that will locate something that might get you started.

1

The reverse engineering Robotron 2084 for the Apple II topic on forum.6502.org discusses, through several dozen posts over a few months, forum user fschuhi's reverse-engineering of the Robotron: 2084 game on the Apple II.

He did this using execution tracing of with Apple II emulator (written in Python), what appears to be a homebrew disassembler, xlwings to interface Python with MS Excel, and further homebrew Excel code to help with the disassembly and analysis, including creating heat maps of memory accesses and DOT code to describe call graphs. He's essentially created a workbench that lets him run parts of the code, analyze what happened during the run and annotate a disassembly with this information. It's brilliant work.

Sadly, the code for all this is not available. (fschuhi said he might refactor it and upload it to GitHub at some point, but the project seems to have been set aside for the moment, so I wouldn't hold my breath waiting for that to happen.)

That said, just looking at the images and reading the text provides a lot of good ideas.

Here's the top level of the workbench, with a bit of disassembled code:

Splash1 (Workbench).jpg

Here's a call graph of the code that generates the title screen:

call_tree.dot3.png

And here's a heat map showing reads and writes to a small area of memory. (These can be zoomed out to let you easily see all memory accesses done by any particular routine you emulate.)

reads top.jpg

1

If you want to get proficient with the first option, that is disassembling a ROM and finding out which part does what, you should get used to an interactive analysis tool that responds immediately to adding label names, retyping data, declaring something as code and shows proper cross-referencing. The two best-known freely available tools that include 6502 support are:

  • IDA 3.7. It works on modern computers inside DOSBox, but as this software is quite dated, the user interface also is. If you want to run it inside DOSBox, consider getting the 3rd party video mode utilities to set a 132-character video mode. Newer Freeware versions of IDA do not support anything except x86 code. The commercial edition is an option though, but the price likely is prohibitive for intermittent hobby use.
  • Ghidra. This a heavy-weight generic reverse engineering tool that even includes a decompiler. I did not check the quality of the decompiler output for 6502 code, but the disassembler should be fine anyway.

None of these tools comes with NES-specific support (like names for NES hardware registers) in the standard install (at least I didn't stumble across it during the quick evalution for this answer), but both support scripting. I suspect that (at least for IDA, due to its age) NES analysis support scripts are available.

Getting used to static analysis with tools like this takes some time, and using them in conjunction with an emulator with debugger is likely the easiest way to get started. With some experience, these tools add a lot of value though by you being able to document your analysis results (mostly by adding comments and renaming things) and the cross-referencing functionality ("show me where this variable is written from").

  • Note that even "non-interactive" tools (such as da65 and f9dasm) where you edit an "info" file that, fed into the disassembler along with the object code, produces the result, can be used in a somewhat interactive way if you use multiple windows. I keep the info file open in an editor in one window and run a command in a loop that runs the disassembly and displays the output file in an editor in another windows; exiting the editor brings up the latest version of the disassembly. – Curt J. Sampson Nov 8 at 10:17
  • 1
    @CurtJSampson Good point. While the workflow you describe is more cumbersome than using an interactive tool, using the old IDA interface in DOSBox also adds a considerable amount of cumbersomeness, as does the Java GUI of Ghidra that integrates only moderately well into native Operating System customs (like the context menu key not popping up the context menu in Ghidra in Windows 10). Especially for small programs (like NES ROMs) where re-disassembling is not prohibitively slow, your approach is also working well. It's worth trying both approaches and stick with the one that works best. – Michael Karcher Nov 8 at 11:55
  • I'm curious how you feel 6502bench SourceGen stacks up (full disclosure: I wrote it). I added a NES definition file to it but haven't tried disassembling a NES ROM. – fadden Nov 8 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.