Self-modifying code in commercial games for the (S)NES, Gameboy, Genesis/MD, PC Engine, Atari, etc

Question

I'm curious if there are any extant examples of commercial games for classic consoles (let's say, pre-32-bit era here) which make uses of self-modifying code. As far as I know, all the systems I've listed are capable of executing from RAM. I know this for a fact, in the case of the Gameboy, and I can't conceive of anything which would limit the others from doing so.

Answer 1

I know about one Atari 8bit game called Robbo. There are source codes published where in file R1.ASM you can see:

     LDA LMVT,X
     STA JMP1+1
     LDA LMVT+4,X
     STA JMP1+2
JMP1 JMP *

LMVT DTA L(UP_)
     DTA L(DW_)
     DTA L(LT_)
     DTA L(RT_)

     DTA H(UP_)
     DTA H(DW_)
     DTA H(LT_)
     DTA H(RT_)

LDA gets address from table indexed by X register, STA stores this address in JMP instruction (after OPCODE, so JMP1+1). JMP jumps to different locations depending on value in X register. I think this is quite typical situation for self modifying code - modify only parameters of instruction not whole instruction.

Answer 2

I rarely had occasion to use self-modifying code. Doing that had a lot of overhead both in ROM space and RAM space, and I rarelysaw an occasion where it would be worth it. I did do it on the IBM PC, where I had bunches of block moves to do that were not contiguous, and doing so eliminated looping overhead during a critical time, at the expense of some RAM.

I did, however, use a variant for copy protection. We ROM programmers were always in an arms race with people who cracked games. Usually, copy protection was bolted on after-the-fact by adding code that had no purpose but copy protection... typically a loop that wrote garbage into the ROM area (having no effect on ROM but destroying a RAM copy). It was easy enough for the crackers to disassemble the code and unbolt them.

On the other hand, games have many tables of gameplay parameters. Some were binary: The robots do not shoot on level 1, but do on level 3. If there were 30 levels, that was 30 bytes per parameter unless you wanted to do a binary unpack. ROM space was tight.

So I would store 3 binary tables in the same byte array: in bits 0, 6 and 7, and other values in bits 1-5. To get to bit 0, I did

 ROR ($TABLE_REF),Y

And the value would pop into the carry flag. (Y has the level number, loaded earlier. table_ref is a zero page location preloaded with the table location). For bits 6 and 7, I would ASL - bit 7 drops into carry and bit 6 drops into sign.

This operation writes the shifted value back to RAM. That has no effect on ROM. But in a RAM copy, it corrupts the gameplay table. Playability is ruined: The robots would never shoot.

By using only a 2-byte instruction, the cracker couldn't just replace it with a 3-byte JSR to a replacement subroutine. It required rewriting that section of the code.

Answer 3

RAM and ROM are reachable by the CPU for execution, so most consoles can execute code from RAM.

The consoles using cartridges usually don't have a lot of RAM, so I think it's VERY rare if someone uses the small RAM used for data to store code just to perform self-modifying code.

On computers on the other hand it happened all the time, just because

1. it's sometimes easier than creating a table
2. it saves cycles and an indirection.

A CD console like the Amiga CD32 is a console but cannot run the code from CD obviously, so it has to load it into its 2MB RAM.

Same can be said for the Commodore CDTV, an earlier version, less powerful, which inherited a lot of Amiga games too.

Since those 2 consoles inherited a lot of Amiga games with very little adaptation, and some amiga games use self-modifying code, you can say that those consoles can use self-modifying code.

The CD32 console can also run CDTV titles. The fact that the processor is a 680EC20 could cause issues, so when detecting a CDTV title, the CD32 console turns the caches off, to handle CPU-dependent issues such as CPU-busy loops and cache issues (i.e. self-modifying code).

It's difficult to find an example of a real game that used self-modifying code on an official CD32 game but I found one: Zool CD32

Disassembling the executable, we found several occurrences of this instruction in the code, (setting a label plus 2 is fishy enough and often a sign of self-modifying code):

MOVE.L  #LAB_0411,LAB_0404+2    ;00cefe: 23fc0000b2fa0000abd6

now at LAB_0404 we find:

LAB_0404:
    JSR LAB_0868        ;00abd4: 4eb90001538c
    CLR.L   LAB_002F+2      ;00abda: 42b9000004ea

So the code changes the jump address depending on the situation (without flushing the caches afterwards). The 68020 instruction cache isn't big enough for this to be a (visible) issue.

---

Another completely unrelated example is the Pengo arcade video game. I reverse engineered the set 1 back in 2010 quite extensively, and browsing my notes I came across this piece of Z80 code:

;; install a modifiable routine in $8C24 (self-modifying code used
;; for maze path drawing)

2DB6: 06 85         ld   b,$0D
2DB8: 11 11 2E      ld   de,$2E39
2DBB: 21 24 8C      ld   hl,$8C24
2DBE: 1A            ld   a,(de)
2DBF: 77            ld   (hl),a
2DC0: 13            inc  de
2DC1: 23            inc  hl
2DC2: 10 7A         djnz $2DBE

As we can see, the game installs some code into $8C24 which is in RAM (as opposed to the lower half like the above code which is in ROM). The code is installed in RAM so self-modifying code will work (this is a maze drawing algorithm, something quite complex to code in Z80 so if they had something working which used self-modifying code, so be it, they're going to use it)

Part of the code using this small smc routine:

    ;; those routines below use a self-modifying code technique to
    ;; change opcode and op value (bit & res and value)
    ;; 30CA tests if way is clear
    ;; 30D2 clears bits to mark that way is clear
    ;;
    ;; this kind of technique is used in tree scan recursive algorithms

    ;; returns z if way is clear, nz if not
is_way_clear_30CA
    : 16 CE         ld   d,$46  ; z80 operand mask
30CC: 18 8E         jr   $30D4
    ;; 30CE is not used
30CE: 16 EE         ld   d,$C6  ; z80 operand mask
30D0: 18 8A         jr   $30D4

set_way_clear_30D2
     16 86         ld   d,$86   ; z80 operand mask
30D4: 21 AF 8C      ld   hl,maze_hole_wall_bit_table_8C27
30D7: 79            ld   a,c
30D8: 2F            cpl
30D9: E6 07         and  $07
30DB: 07            rlca
30DC: 07            rlca
30DD: 07            rlca
30DE: B2            or   d
30DF: 32 0D A4      ld   ($8C25),a ; change operand of the bit/res opcode
30E2: 16 88         ld   d,$00
30E4: 58            ld   e,b
30E5: 19            add  hl,de
30E6: C3 0C A4      jp   $8C24  ; calls self-modifying code bit/res test routine!!

Answer 4

The NES game Gimmick! uses self modifying code as part of its anti-piracy check failure response. 96 bytes are decrypted while being copied from ROM to RAM and then jumped to. This is to hide a routine that halts the game and displays the message BLACK HOLE when certain conditions are met.

https://tcrf.net/Notes:Gimmick!

Answer 5

(This is slightly off-topic, but related.)

The 65816 CPU has a pair of block move instructions, MVN and MVP, which allow you to copy up to 64KiB from one 24-bit address to another. The instructions work like this:

ldx #{16-bit source address}
ldy #{16-bit destination address}
lda #{16-bit count - 1}
mvn src_bank, dest_bank

(The "bank" is the high 8 bits of the 24-bit address.)

The tricky part is the instruction itself, because part of the address is embedded in it. If you happen to know the bank addresses at assembly time (e.g. you know source and destination are both in bank 0) you're fine, but in the general case you're required to use self-modifying code.

If you want to use these instructions from ROM code, you have to set up a move instruction somewhere in RAM and call it. If your code is reentrant (due to interrupts) you can't simply use a fixed location. IIRC the Apple IIgs ROM pushed the instruction onto the stack and called it there.