I've done this partially with Commodore PET Space Invaders which I used as a test program to debug my Commodore PET emulator.
I used a disassembler to convert the program to assembly language and I then went through the code annotating it as I found out what it does. When I understood what a bit of code did, I would look for its entry point (my disassembler created labels for the target of every
JSR) and rename the label to something useful. That would help me with other bits of code that called the piece I had just reverse engineered.
It helped that, having written an emulator, I was quite familiar with the memory mappings of the IO chips and certain important bit patterns. So, for instance, if I saw a bit of code writing a certain bit pattern to a certain address that I knew was mapped to one of the PIA ports, I knew it would be scanning the keyboard.
It also helped that I could run my emulator in trace mode (albeit very slowly) so every instruction is printed out as it is executed. So, for instance, when a space invader started appearing on the screen, I could look at the trace and see which bits of code were being executed.
It took some time and I basically suspended work on it when I found a bug that was caused by my emulator being too fast. I haven't figured out a way to make it run accurately at the correct speed yet.
To expand a bit, I used the disassembler (da65) from the cc65 tool chain. It has the handy ability to accept an "info file" so you don't need to annotate the generated source from the disassembler. So, for example, the first two bytes of a PET program file are actually the load address. I wrote the following annotation in the info file
which make the assembler treat the first two bytes as data and not garbage assembler instructions.
The next bit was a one line BASIC program that calls the machine code routine that is the start of the program proper. Again, I forced the disassember to treat it as data
This time it also causes the disassembler to emit a comment that tells you what it is.
And then it's just a case of laboriously going through the code identifying the function of various bits. For example, I found a short loop that tested a certain bit on one of the PIA registers which I know is the vertical retrace, so it is clearly syncing with the monitor. I add this annotation
COMMENT "Wait for the screen to lower its retrace signal";
and the disassembler creates a label for the routine as well as using the label in
JMP from elsewhere.