Note: This answer mainly focuses on the NES, since that's what I'm most familiar with.
Yes; this is called static recompilation or static binary translation, and it is theoretically possible -- jamulator by Andrew Kelly does it. However, recompilation can be incredibly difficult (to the point that falling back to interpretation at runtime may be required in some cases). Additionally, the system's non-CPU hardware (such as graphics and sound hardware) still must be emulated.
Statically recompiling machine code is incredibly difficult; determining the behavior of a program by analyzing its code ahead-of-time is in some cases provably impossible. Some of the problems faced when statically recompiling a retro video game ROM include:
How do we determine what parts of the ROM are executable code and what parts are data?
The "obvious" way to recompile a NES ROM is to take a single pass, reading a stream of 6502 instructions and outputting a stream of x86 instructions. However, the ROM also includes data bytes, which will produce garbage if interpreted as instructions. These garbage instructions should never be executed, since the program would never jump to that address, but the presence of non-instruction data poses two problems:
1) The program must be able to read its own ROM. This isn't really a hard problem at all; we just need to include an uncompiled copy of the ROM in the compiled program binary to read data bytes from.
2) We can't write a simple single-pass recompiler. Instructions can be multiple bytes wide, so if we try to recompile a block of non-executable data, we might end up misaligned when recompiling subsequent instructions. (As a simple (somewhat contrived) example, in 6502 machine code, the sequence of bytes 69 09 0A is ADC #09; ASL, while the sequence of bytes A5 69 09 0A is LDA $69; ORA #0A. The point at which we start executing drastically affects our results.)
So this means we have to perform much more complex code analysis in order to determine the basic blocks of the program and compile them individually. jamulator does this by starting at the interrupt vectors and following all possible branches from there. However, this approach still has problems with:
Dynamic jumps
This is where the address to execute is computed at runtime. In some cases, the runtime-computed address can be looked up and the corresponding x86 code can be found. However, if the only way to access a basic block is via an indirect jump, the recompiler would likely have assumed that block was not executable code and thus not compiled it.
jamulator mitigates this problem using a hack: hard-coded support in the recompiler to recognize jump table implementations in specific games. This works, but is clearly not a general-purpose solution.
Bank switching
Although jamulator only supports NROM games, many NES games with more complicated mappers could switch regions of code and data in and out of the accessible address space. This means that each jump could go to dozens of different locations in the ROM, depending on which bank is mapped to that address at runtime.
Dynamically generated code/self-modifying code
Although this was uncommon for NES games due to the small amount of RAM, if I remember correctly C64 software would occasionally generate and execute code at runtime, or modify code mid-execution. This would be nearly impossible to statically predict.
Other hardware such as graphics and audio
The NES Picture Processing Unit and Audio Processing Unit still must be emulated. jamulator includes an emulator for the non-CPU hardware in a runtime library, and the generated x86 code calls a library function to handle writes to memory addresses mapped to I/O operations.
Many NES games relied on precise timing between the CPU and PPU, so the generated CPU code must count the number of 6502 clock cycles taken by the executed code. jamulator implements this relatively simply: after each NES instruction, it calls out to a runtime library which runs the rest of the hardware for a specific number of cycles. This approach is simple to implement, but has a few disadvantages:
- CPU writes don't quite happen on the exactly correct cycle relative to the rest of the hardware. The timing is only accurate to the nearest instruction, not the nearest cycle.
- Emulating the PPU and APU after every CPU instruction greatly impairs performance, since the recompiler loses opportunities to remove, combine, or reorder instructions for better performance. Frequent switching between operations also harms the x86 CPU's cache performance and branch prediction accuracy.
A more efficient (but much more complex) approach is to predict the times at which accurate synchronization between the CPU and the rest of the hardware is needed, and use these predictions to switch between fast and cycle-accurate emulation modes as needed.
Although static recompilation is often possible, it can be extremely difficult. jamulator sometimes falls back to interpretation when the game does something not properly handled by the static recompiler.
The accuracy of static recompilation could be improved by running the game inside of something like FCEUX's code-data logger, which emulates the game while recording the code paths it takes. Data from an actual run of the game can be used to greatly improve the accuracy of static recompilation. However, the "test run" recorded by the code-data logger must comprehensively exercise the possible code paths taken by the game in order to be useful.
Emulators of newer systems, such as the Dolphin Emulator (which emulates the GameCube and Wii) frequently use just-in-time compilation, where the emulator recompiles sections of the game's code at runtime. This generally provides the best of both worlds: we get the performance improvements of recompiled code, and the improved insight of being able to analyze the game's code at runtime.
