8

In the late MS-DOS era, what were the state of the art software tools for reverse engineering and patching binaries, i.e. .exe and .com files?

A simple list of the tools that were "state of the art", widely used, or considered excellent, and what they were used for, for DOS would suffice. However, the answer to this question might also depend on the answer to "What do you want to do with such a tool?". For this reason I have provided a concrete example with a use case, consider the following:

I have an old game I would like to run, but my DOS machine has too much RAM. The game issues a simple error: "Program too big to fit in memory". I have reason to suspect that it is indeed extended memory which is "insufficient", and not simply a lack of conventional memory which would typically be the case. This is beyond the point anyway, but this might be due to an integer overflow or similar issue. I want to find whatever code is doing this and just NOP it out, patch the binary, and see if that makes the problem go away.

I would like a tool that allows me to search for strings in memory, in particular that above-mentioned error. It would be nice if the tool could list references to the string. Obviously I would want to be able to have basic debugging functionality as well, breakpoints, watchpoints, etc.

Finally, I would like to be able to patch the .exe file and save it.

Obviously, I could use modern tools for this particular problem, such as IDA pro, on a modern system, but I would like to do it on the DOS machine, partly just to play around with such tools.

  • 2
    It's highly likely that your use case isn't just a simple patch. Remember EMM386.EXE, HIMEM.SYS and QEMM, designed to squeeze every last byte out of the high-memory area so that these games would run? – Robert Harvey Apr 13 '18 at 22:26
  • 7
    "Program too big to fit in memory" strongly suggests lack of conventional memory is the problem, as it's MS-DOS and not the game, that's printing the message. A lot of MS-DOS games had very high conventional memory requirements, you probably need to adjust your configuration, like Robert Harvey suggested, to get the game to run. What game are you trying to play? – Ross Ridge Apr 13 '18 at 23:11
  • 2
    "I want to find whatever code is doing this and just NOP it out, patch the binary, and see if that makes the problem go away." - yeah, that'll work :/ Maximize Conventional Memory Under DOS – Bruce Abbott Apr 14 '18 at 8:17
  • and should you want to share your patch, use the oldskool binary patch formats :) -- yurisk.info/2017/05/22/… – sendmoreinfo Apr 14 '18 at 12:40
  • This is actually very common today, where virtual machines do all kinds of tricks with the binaries run inside. – Thorbjørn Ravn Andersen Nov 12 '18 at 17:56
12

As others have mentioned, “Program too big to fit in memory” means that DOS can’t find a large enough memory block to fit the amount of memory that your program’s header requests. This can be either because you have too little available conventional memory (extended memory isn’t taken into account), or because your executable is corrupted.

To answer your question about state-of-the-art tools which people would have used in the late DOS era to debug, disassemble and patch executables, here are a few which I would recommend:

  • Sourcer was one of the best non-interactive disassemblers; it also included binary patching tools — so you’d disassemble your program, figure out where it needed fixing, and write a binary patch for it.
  • IDA is an excellent interactive disassembler, and early versions were available for DOS (see here).
  • Debuggers would commonly be used to explore the inner workings of a program. The best debugger is the one you’re most familiar with, so for most people that would be the debugger included in their development environment — Turbo Debugger for Borland users, CodeView for Microsoft users, the Watcom debugger etc. There was some cross-over, and Turbo Debugger in particular seemed quite popular back then. Other famous debuggers include SoftICE and Periscope (which was a real ICE, i.e. hardware-based). I tend to use 386SWAT nowadays; it wasn’t famous back in the late DOS days, since it was an internal Qualitas tool (used to develop the 386MAX memory manager), but it was released to the general public at the turn of the century and is very capable.
  • Other useful tools include hex viewers and editors; examples include QView, BIEW, HIEW etc. (see this list). Some of these also include disassemblers, and executable header viewers (which would allow you to easily view the required memory in your case, and perhaps patch that). You could use one of these to actually patch your executable, after determining what byte sequence to look for, either in a disassembler or debugger.
  • Once you’ve figured out what to patch, you might need an unpacker first — DOS executables were often compressed, so you wouldn’t be able to patch them directly. There were quite a few tools to handle this; I’d generally use UNP (see this list).
  • 3
    I'm sure "The best debugger is the one you’re most familiar with" partly applies, but Borland's Turbo Debugger was -- if not the best -- at least the first I saw to fully leverage the power of the 386's protected mode to simplify a lot of things (breakpoint when an address or port was written to etc.) – TripeHound Apr 16 '18 at 14:07
  • @TripeHound I liked TD too; IIRC though SoftICE beat it by a few years with its 386 and hardware debugging support. – Stephen Kitt Apr 16 '18 at 14:38
  • Although I never used it, IIRC (and it's a long time ago) SoftICE was originally (or at least had the option of) a hardware module that let you do things that couldn't be done (at least in real time) in pre-386 days. TD386 gave you most of that ability in a software-only solution. Presumably SoftICE still won for certain tasks (e.g. possibly protected-mode code; possibly if reverse-engineering 386-debug-register-aware code). – TripeHound Apr 16 '18 at 15:05
  • 1
    @TripeHound SoftICE was always software-only (hence the name). Debuggers with hardware support were products such as Periscope, or the various Microtek MICE. – Stephen Kitt Apr 16 '18 at 15:17
  • That makes sense ... somehow I'd got the notion it had (or could have) a hardware component. The grey cells aren't what they used to be! – TripeHound Apr 16 '18 at 15:21
6

a bit off topic to your actual question but to make your game/exe whatever usable again:

  1. Too much memory

    common on MS-DOS for 32MByte and 64MByte (IIRC some versions use 32 some 64). The memory manager reports negative value of free or total memory causing this problem. To repair simply use Smart Drive and fill memory to it until only 32MByte or slightly less is left for the OS (IIRC the buffers could be incrementally increased). This work on all exe I used at the time even if 128 or 256 or 512 MByte of RAM memory was present.

    In case your out of memory error is due to missing memory manager like: himem.sys or emm386/qemm386 just install it into startup (config.sys).

  2. CRT runtime error 200

    this is caused due to using old TP7 crt unit which on init use some CPU measuring loop. On fast computers it overflows and causes this error. There are 3 approaches to fix.

    • use crt lib patch (not working for some packed executables) it was called tppatch.exe
    • use slow commandline tool (also not 100%)
    • turn off CPU/MB caches in BIOS Setup (this has 100% success rate for me)
  3. bad startup config

    demanding MS-DOS programs where picky on the startup config you chose. Some programs need only himem.sys (to boost BASE memory) other need XMS/EMS. So if you know which one you need then configure your startup propperly. The easiest is to have startup boot menu in autoexec.bat/config.sys to select which one you want quickly.

For more info on this topic see (my startup config included:):

Now some additional tools you should have (the comments are opinion based!):

  • Volkov Commander (file manager with file associations and context menus is a must)
  • Hiew (txt viewer editor disassembler is a must)
  • UniVBE (VESA driver for also non VESA gfx cards can enhance resolutions is amust)
  • RAR,PKZIP,PKLITE,UC2,ARJ (packers/unpackers is a must especially pklite for executables)
  • NEOPAINT (gfx editor)
  • Derive (algebraic calculator and plotter now it is called derive for windows)
  • PV (gfx extractor/decoder and viewer from any files)
  • QV (quick view - multimedia viewer can play also DivX,avi,mov,fli,flc,... the best I saw ever)
  • QPV (image viewer also the best of its kind I know of)
  • MODPLAY,DOSAMP (sound players)
  • TCPP or TP7 (borland compilers nad IDE with turbo debuger and linkers is a must for some patching)
  • tppatch (CRT error patcher is a must)
  • NASM (assembly compiler the best in its class)

  • gmouse (genius mouse driver)

  • load_sys (loader of drivers without restart during runtime is a must)
  • DOS4GV (protected mode engine is a must)
  • FAKECD (CDROM emulator is a must)
  • MSCDEX (CDROM driver)
  • UNERASE,UNDELETE
  • TBAV (the best antivirus of its class)

  • EZ Drive,FHFORMAT,NDD,PQ8 (disc utilities)

There where also TCPP and TP7 disassemblers and discompilers (that recreate the source from compiled exe) but can not remember how they called.

Here is first link from Google with some of them included for download:

  • 2
    "CPU measuring loop. On fast computers it overflows" Actually, as I recall the problem was that it underflowed. Specifically, the runtime would measure the time required to run through a loop of fixed length, and when the computer was "too fast", that time measured as zero. This amount of time was then used as a value to divide by, to get a scaling factor. Since you can't divide a real number by zero and get a meaningful result, the result of this was a "division by zero" error. (Frustratingly, it happened even if the program never used any delay instructions at all...) – a CVn Jun 22 '18 at 22:25
  • 1
    s/Volcov/Volkov/. you can also add a link to vvv.kiev.ua/download – Igor Skochinsky Nov 15 '18 at 14:10
  • @IgorSkochinsky thx I added it along wit EZ-Drive related to another QA I dealt with recently... – Spektre Nov 15 '18 at 15:16
  • you still have 'c' instead of 'k' – Igor Skochinsky Nov 15 '18 at 16:07
  • 1
    @IgorSkochinsky ow I miss that s/Volcov/Volkov/ of yours completely was seing only the link ... :) – Spektre Nov 15 '18 at 18:36
5

The simplest way to patch a .com file is DEBUG, which comes with any DOS installation.

DEBUG is partially usable for .exe files as well, but cannot reverse their relocation process, and so cannot save them once they have been relocated (If you load them as a plain binary file instead of an executable, you can, however, save them, but have limitations with debugging them).

3

I have an old game I would like to run, but my DOS machine has too much RAM.

The most proper way to run DOS applications in Windows is using DOSbox.

"Program too big to fit in memory"

It may happen that you have corrupt executable.

Obviously I would want to be able to have basic debugging functionality as well, breakpoints, watchpoints, etc.

In order to debug something, you must have an understanding of how it works. There were great disassembly tools, there was Turbo Debugger from Borland, however it is a good question if you will be able to use them properly without initial training.

  • DOSbox is good, and is very easy to use which is a bonus, but I'd definitely not go as far as to say it's "the most proper way". There are compatibility issues -- specifically, it runs its own DOS version that isn't quite 100% compatible with MSDOS, and there are a few programs that misbehave because of that. – Jules Jun 22 '18 at 8:39
3

The state of the art in assembly-level debugging of DOS programs is almost certainly SoftICE. I've never used it, but my understanding is that it was for a very long time the tool of choice of software copyright protection scheme crackers, as well as low-level operating system/device driver developers.

That said, it may well be overkill for your problem. You do not specify how you have determined that the problem is not a shortage of conventional memory, but it should be noted that the error you quote is IIRC the standard error used by MSDOS when there is not enough conventional memory for the size specified in the .EXE file ("MZ") header. I would begin by opening the .EXE file in a hex editor and calculating:

  • code_size = 512*(word at 0x0004)
  • bss_size = 16*(word at 0x000A)
  • total_size = code_size + bss_size

and comparing that against your available memory (as reported by mem, if you're on DOS 4+). If it's larger (or even quite close), you may well not have enough conventional memory, even if you seem to have quite a lot. I've seen programs before that required very nearly all of the conventional memory on a 640K system to load. You may have one of these - or, as suggested by Anonymous above, you may have a corrupt file that has nonsense in one of these fields.

  • 1
    I broke hundreds of software protection routines with Turbo Debugger. – Rui F Ribeiro Apr 20 '18 at 19:17
  • @RuiFRibeiro - it's a pretty good system, but as I understand it, SoftIce is able to debug generic protected mode apps, which I believe TD can only do if they use the same DPMI interface that Borland used for their compilers. I believe it can also relocate itself almost entirely out of conventional memory, which can be helpful for those applications that are very close to memory limits. – Jules Apr 20 '18 at 19:25

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.