What tools were used in late MS-DOS era for reverse engineering and patching binary executables?

Question

In the late MS-DOS era, what were the state of the art software tools for reverse engineering and patching binaries, i.e. .exe and .com files?

A simple list of the tools that were "state of the art", widely used, or considered excellent, and what they were used for, for DOS would suffice. However, the answer to this question might also depend on the answer to "What do you want to do with such a tool?". For this reason I have provided a concrete example with a use case, consider the following:

I have an old game I would like to run, but my DOS machine has too much RAM. The game issues a simple error: "Program too big to fit in memory". I have reason to suspect that it is indeed extended memory which is "insufficient", and not simply a lack of conventional memory which would typically be the case. This is beyond the point anyway, but this might be due to an integer overflow or similar issue. I want to find whatever code is doing this and just NOP it out, patch the binary, and see if that makes the problem go away.

I would like a tool that allows me to search for strings in memory, in particular that above-mentioned error. It would be nice if the tool could list references to the string. Obviously I would want to be able to have basic debugging functionality as well, breakpoints, watchpoints, etc.

Finally, I would like to be able to patch the .exe file and save it.

Obviously, I could use modern tools for this particular problem, such as IDA pro, on a modern system, but I would like to do it on the DOS machine, partly just to play around with such tools.

Answer 1

As others have mentioned, “Program too big to fit in memory” means that DOS can’t find a large enough memory block to fit the amount of memory that your program’s header requests. This can be either because you have too little available conventional memory (extended memory isn’t taken into account), or because your executable is corrupted.

To answer your question about state-of-the-art tools which people would have used in the late DOS era to debug, disassemble and patch executables, here are a few which I would recommend:

• Sourcer was one of the best non-interactive disassemblers; it also included binary patching tools — so you’d disassemble your program, figure out where it needed fixing, and write a binary patch for it.
• IDA is an excellent interactive disassembler, and early versions were available for DOS (see here).
• Debuggers would commonly be used to explore the inner workings of a program. The best debugger is the one you’re most familiar with, so for most people that would be the debugger included in their development environment — Turbo Debugger for Borland users, CodeView for Microsoft users, the Watcom debugger etc. There was some cross-over, and Turbo Debugger in particular seemed quite popular back then. Other famous debuggers include SoftICE and Periscope (which was a real ICE, i.e. hardware-based). I tend to use 386SWAT nowadays; it wasn’t famous back in the late DOS days, since it was an internal Qualitas tool (used to develop the 386MAX memory manager), but it was released to the general public at the turn of the century and is very capable.
• Other useful tools include hex viewers and editors; examples include QView, BIEW, HIEW etc. (see this list). Some of these also include disassemblers, and executable header viewers (which would allow you to easily view the required memory in your case, and perhaps patch that). You could use one of these to actually patch your executable, after determining what byte sequence to look for, either in a disassembler or debugger.
• Once you’ve figured out what to patch, you might need an unpacker first — DOS executables were often compressed, so you wouldn’t be able to patch them directly. There were quite a few tools to handle this; I’d generally use UNP (see this list).

Answer 2

The simplest way to patch a .com file is DEBUG, which comes with any DOS installation.

DEBUG is partially usable for .exe files as well, but cannot reverse their relocation process, and so cannot save them once they have been relocated (If you load them as a plain binary file instead of an executable, you can, however, save them, but have limitations with debugging them).

Answer 3

a bit off topic to your actual question but to make your game/exe whatever usable again:

1. Too much memory

common on MS-DOS for 32MByte and 64MByte (IIRC some versions use 32 some 64). The memory manager reports negative value of free or total memory causing this problem. To repair simply use Smart Drive and fill memory to it until only 32MByte or slightly less is left for the OS (IIRC the buffers could be incrementally increased). This work on all exe I used at the time even if 128 or 256 or 512 MByte of RAM memory was present.

In case your out of memory error is due to missing memory manager like: himem.sys or emm386/qemm386 just install it into startup (config.sys).

2. CRT runtime error 200

this is caused due to using old TP7 crt unit which on init use some CPU measuring loop. On fast computers it overflows and causes this error. There are 3 approaches to fix.

• use crt lib patch (not working for some packed executables) it was called tppatch.exe
• use slow commandline tool (also not 100%)
• turn off CPU/MB caches in BIOS Setup (this has 100% success rate for me)

See this dedicated question for more details.

3. bad startup config

demanding MS-DOS programs where picky on the startup config you chose. Some programs need only himem.sys (to boost BASE memory) other need XMS/EMS. So if you know which one you need then configure your startup propperly. The easiest is to have startup boot menu in autoexec.bat/config.sys to select which one you want quickly.

For more info on this topic see (my startup config included:):

• Run Turbo C++ in Freedos

Now some additional tools you should have (the comments are opinion based!):

• Volkov Commander (file manager with file associations and context menus is a must)

• Hiew (txt viewer editor disassembler is a must)

• UniVBE (VESA driver for also non VESA gfx cards can enhance resolutions is amust)

• RAR,PKZIP,PKLITE,UC2,ARJ (packers/unpackers is a must especially pklite for executables)

• NEOPAINT (gfx editor)

• Derive (algebraic calculator and plotter now it is called derive for windows)

• PV.EXE (gfx extractor/decoder and viewer from any files) here link to it from my old archive

• QV (quick view - multimedia viewer can play also DivX,avi,mov,fli,flc,... the best I saw ever)

• QPV (image viewer also the best of its kind I know of)

• MODPLAY,DOSAMP (sound players)

• TCPP or TP7 (borland compilers nad IDE with turbo debuger and linkers is a must for some patching)

• tppatch (CRT error patcher is a must)

• NASM (assembly compiler the best in its class)

• gmouse (genius mouse driver)

• load_sys (loader of drivers without restart during runtime is a must)

• DOS4GV (protected mode engine is a must)

• FAKECD (CDROM emulator is a must)

• MSCDEX (CDROM driver)

• UNERASE,UNDELETE

• TBAV (the best antivirus of its class)

• EZ Drive,NDD,PQ8,FHFORMAT (HDD and FDD utilities)

There where also TCPP and TP7 disassemblers and discompilers (that recreate the source from compiled exe) but can not remember how they called.

Here is first link from Google with some of them included for download:

• MS-DOS utilities

Here some stuff like mouse,gmouse and USB drivers for w9x related to other question

• MS-DOS (g)mouse and w9x USB stuff

Answer 4

The state of the art in assembly-level debugging of DOS programs is almost certainly SoftICE. I've never used it, but my understanding is that it was for a very long time the tool of choice of software copyright protection scheme crackers, as well as low-level operating system/device driver developers.

That said, it may well be overkill for your problem. You do not specify how you have determined that the problem is not a shortage of conventional memory, but it should be noted that the error you quote is IIRC the standard error used by MSDOS when there is not enough conventional memory for the size specified in the .EXE file ("MZ") header. I would begin by opening the .EXE file in a hex editor and calculating:

• code_size = 512*(word at 0x0004)
• bss_size = 16*(word at 0x000A)
• total_size = code_size + bss_size

and comparing that against your available memory (as reported by mem, if you're on DOS 4+). If it's larger (or even quite close), you may well not have enough conventional memory, even if you seem to have quite a lot. I've seen programs before that required very nearly all of the conventional memory on a 640K system to load. You may have one of these - or, as suggested by Anonymous above, you may have a corrupt file that has nonsense in one of these fields.

Answer 5

I have an old game I would like to run, but my DOS machine has too much RAM.

The most proper way to run DOS applications in Windows is using DOSbox.

"Program too big to fit in memory"

It may happen that you have corrupt executable.

Obviously I would want to be able to have basic debugging functionality as well, breakpoints, watchpoints, etc.

In order to debug something, you must have an understanding of how it works. There were great disassembly tools, there was Turbo Debugger from Borland, however it is a good question if you will be able to use them properly without initial training.