The page you linked on The Cutting Room Floor offers its own explanation:
Sometimes this is to pad out a disc or ROM to fill any empty space, other times it's just funky compiler behavior...
"Funky compiler behaviour" is the most likely explanation for the older games. More specifically, as explained in Muzer's comment, it's likely because memory was reused without erasing it first or completely overwriting it with new data. One can imagine this resulting in source code ending up in a game in one of two ways. Either a compiler/assembler that ends up reusing part of its own memory for both storing the source code input and the machine code output, or a linker that ends up using memory previously used by a compiler/assembler.
The first case case could happen in a scenario something like the following:
- The program allocates an input buffer
- The source code is read into the input buffer
- The source code is converted into a parse tree or some other intermediate representation
- The input buffer is freed
- The program allocates an output buffer which happens to overlap the the location in memory where the input buffer was
- The program converts the intermediate representation to machine code and stores it in the output buffer
- The entire output buffer is written out
If the output buffer isn't cleared in step 5 or entirely overwritten in step 6 it could end up containing part of the source code read into memory in step 2.
Older single-user operating systems like MS-DOS didn't (and/or couldn't) provide any sort of real process isolation and there's was no security to speak of. While a modern multitasking OS will erase any memory it provides to a process so it can't see any data that might have been stored there by other processes, these older OS'es didn't do that. So the second case mentioned above could happen in a scenario like the following:
- The compiler/assembler allocates space for an input buffer
- The source code is read into the input buffer
- The source code is compiled/assembled into an object file
- The compiler/assembler exits
- The linker allocates space for an output buffer which happens to overlap the location in memory where the compiler/assembler allocated the input buffer
- The game's object files are linked together and stored in the output buffer
- The entire output buffer is written out
Again, if the output buffer isn't cleared in step 5 or entirely overwritten in step 6 it could end up containing part of the source code read into memory in step 2.
This later scenario is probably more likely, at least for anything other than the earliest of games. The program is only likely to write an entire buffer regardless of how much of that buffer was used when the output needs to be a certain size like when creating a ROM image. A smaller game might use assembler that produces the ROM image directly, but bigger games are more likely to use separate compilation and produce the ROM image using a linker.
Padding out a disc or ROM is a much less likely explanation, though you see something similar in the newer games listed on that The Cutting Room Floor page. Instead of explicit padding, or even actual game source code, you often see build scripts and build configuration files included in games. These are files that happened to be in the build output directory for use by the game's build process but aren't actually needed at runtime by the game itself. They end up being included in the game because the entire build output directory ends being included in the game's disc (or flash cart) file system. Essentially they turn that entire directory into an ISO and use it as the image they send to manufacturing.