3

Windows and DOS binary executable files with the .EXE extension have an MZ header in them and nowadays also a PE header.

But before these there used to also be .COM binary executable files and they had no internal header, they were just raw x86 machine code.

But Intel's x86 processor family has been around for ages from the 8-bit era through to the 64-bit era with the instruction set and the memory model changing several times. Memory models had to do with how to access more memory beyond the processor's address bus could directly address. There used to be multiple pages of memory and segment registers

So back to .COM files, did they use "real mode" or "protected mode"? Or were both used? If both were used, was there any way to tell which was needed?

I want to disassemble some old .COM executables and the tool asks me whether the files are real mode or protected mode.

(I did assembly and machine code programming on Z80 and Motorola 680x0 and switched to PCs in the 486 (32-bit) era but found the assembly syntax and segmented memory models ugly and didn't continue with assembly language or even understand the stuff about memory segments and such.)

  • 1
    Certainly it's possible for code written as real-mode code to execute in protected mode, in the V86 submode. – another-dave Apr 25 at 12:40
15

DOS programs always start in real mode (or an emulation thereof), so it’s best to start disassembling them assuming that. When disassembling, you should assume real mode, with 16-bit data and 16-bit addresses, until the code you’re disassembling changes that. The DOS-based disassemblers I’ve used generally know about the executable formats involved, and don’t ask. (They need to know about the formats to be able to set the segments up appropriately.)

In DOS, there are three executable layouts, and the extension doesn’t matter.

The first is the “.COM” layout, which is loaded into memory as-is after the PSP, with all free memory allocated to the process, and all segments set to point at the PSP. Execution starts at offset 0100h, which maps to offset 0 in the file.

The second is the device driver layout, which I won’t detail here.

The third is the MZ layout, with a header used to tell the loader how to load the executable. This header specifies how much of the executable to load initially, how much memory to allocate, the segment layout, relocations to perform, and where execution starts.

All three layouts leave plenty of room for a switch to protected mode, so ideally your disassembler should be able to recognise that.

Non-DOS executables have a real-mode DOS stub; if you disassemble assuming real-mode MZ, that‘s what you’ll see. Again, most disassemblers can interpret the format correctly and will open LE, NE, PE etc. appropriately.

The age of your executable can be a useful piece of information: protected mode DOS programs were unusual until at least 1993.

With the freeware version of IDA 7, you can analyse COM files by loading them as binary files, changing the loading offset to 0x100, and selecting 16-bit mode.

| improve this answer | |
  • I'm using Ghidra, which works with many executable formats and CPU architectures, but its loaders seem to ignore file extensions and detect file types only by their internal format. But as .com files have no internal format it doesn't recognize them, and I suppose these are not the kinds of binaries that are the primary target of Ghidra anyway. Dunno. My executable is an old ZX Spectrum emulator called Spec256 from the '00s or '90s. – hippietrail Apr 25 at 9:28
  • 1
    Right, Ghidra doesn’t know how to handle COM files unfortunately (although I imagine it would be possible to teach it how). Give the demo of IDA a try, it should work much better for this type of analysis. – Stephen Kitt Apr 25 at 10:19
  • 4
    DOS didn’t exist pre-16-bit, it was always a 16-bit operating system. (See Can x86 processors run 8-bit applications?) It also didn’t make the migration to 32 bits. You should always assume 16-bit data and addresses, until the program being analysed issues instructions (or prefixes) to change that. One common feature was for 386+ programs to use the 32-bit registers in real mode, but you’d see the corresponding prefix. For Windows programs things are different, but they all have headers. – Stephen Kitt Apr 27 at 8:03
  • 1
    It might be important to mention that the file extension is normally ignored by DOS when determining what type of file the executable is, instead it uses the magic number signature in the MZ file header for this decision: COMMAND.COM and EDIT.COM actually are .EXE files on some DOS versions. – tofro Oct 6 at 6:53
  • 1
    Way back I was doing some stuff which involved generating an NE-format file, and then running it through a binder to make a binary to load onto a bare-metal '386 in protected mode. I found it extremely useful to also generate a file to tell Sourcer about all the special bits (descriptor tables etc.), and usually used Sourcer's output in preference to ordinary listing files when debugging with a logic analyser. – Mark Morgan Lloyd Oct 9 at 15:59
5

com files are not segmented (its just single segment). They have limitation that it can not cross 64K of code (filesize).

They always starts in real mode but I do not think there is any restriction to switch to protected from the code.

So while disassembling set real mode any switching from the code should be recognized by the tool. However that is improbable as in 64K of code there is not much room to make protected mode stuff as you need to make own OS like environment. There where used DOS extentions like DOS/4GW for this but newer saw it used along with *.com file.

| improve this answer | |
  • The earliest versions of DOS would read .COM files into a sequential region of up to 65,280 bytes, starting at offset 256 of an arbitrary segment, and make a call to offset 256 of that segment, while .EXE files, which had to start with the byte sequence 0x4D 0x5A [character codes "MZ"], would be processed in a more complicated fashion. Later versions, however, process both .COM or .EXE files by checking whether they start with "MZ", and selectthe loading method based upon that. So it might help to clarify what is meant by "com files". – supercat Oct 8 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.