13

Windows and DOS binary executable files with the .EXE extension have an MZ header in them and nowadays also a PE header.

But before these there used to also be .COM binary executable files and they had no internal header, they were just raw x86 machine code.

But Intel's x86 processor family has been around for ages from the 8-bit era through to the 64-bit era with the instruction set and the memory model changing several times. Memory models had to do with how to access more memory beyond the processor's address bus could directly address. There used to be multiple pages of memory and segment registers

So back to .COM files, did they use "real mode" or "protected mode"? Or were both used? If both were used, was there any way to tell which was needed?

I want to disassemble some old .COM executables and the tool asks me whether the files are real mode or protected mode.

(I did assembly and machine code programming on Z80 and Motorola 680x0 and switched to PCs in the 486 (32-bit) era but found the assembly syntax and segmented memory models ugly and didn't continue with assembly language or even understand the stuff about memory segments and such.)

Improve this question
2
  • 2
    Certainly it's possible for code written as real-mode code to execute in protected mode, in the V86 submode.
    – dave
    Commented Apr 25, 2020 at 12:40
  • .com files: I assume he's talking pre-386. No VMs. BTW, com files had to run in a single 64K segment.
    – Miss Understands
    Commented Jun 20 at 5:02

2 Answers 2

Reset to default
25

DOS programs always start in real mode (or an emulation thereof), so it’s best to start disassembling them assuming that. When disassembling, you should assume real mode, with 16-bit data and 16-bit addresses, until the code you’re disassembling changes that. The DOS-based disassemblers I’ve used generally know about the executable formats involved, and don’t ask. (They need to know about the formats to be able to set the segments up appropriately.)

In DOS, there are three executable layouts, and the extension doesn’t matter.

The first is the “.COM” layout, which is loaded into memory as-is after the Program Segment Prefix, with all free memory allocated to the process, and all segments set to point at the PSP. Execution starts at offset 0100h, which maps to offset 0 in the file.

The second is the device driver layout, which I won’t detail here.

The third is the MZ layout, with a header used to tell the loader how to load the executable. This header specifies how much of the executable to load initially, how much memory to allocate, the segment layout, relocations to perform, and where execution starts.

All three layouts leave plenty of room for a switch to protected mode, so ideally your disassembler should be able to recognise that.

Non-DOS executables have a real-mode DOS stub; if you disassemble assuming real-mode MZ, that‘s what you’ll see. Again, most disassemblers can interpret the format correctly and will open LE, NE, PE etc. appropriately.

The age of your executable can be a useful piece of information: protected mode DOS programs were unusual until at least 1993.

With the freeware version of IDA 7, you can analyse COM files by loading them as binary files, changing the loading offset to 0x100, and selecting 16-bit mode.

Improve this answer
11
  • 1
    Right, Ghidra doesn’t know how to handle COM files unfortunately (although I imagine it would be possible to teach it how). Give the demo of IDA a try, it should work much better for this type of analysis.
    – Stephen Kitt
    Commented Apr 25, 2020 at 10:19
  • 6
    DOS didn’t exist pre-16-bit, it was always a 16-bit operating system. (See Can x86 processors run 8-bit applications?) It also didn’t make the migration to 32 bits. You should always assume 16-bit data and addresses, until the program being analysed issues instructions (or prefixes) to change that. One common feature was for 386+ programs to use the 32-bit registers in real mode, but you’d see the corresponding prefix. For Windows programs things are different, but they all have headers.
    – Stephen Kitt
    Commented Apr 27, 2020 at 8:03
  • 1
    Way back I was doing some stuff which involved generating an NE-format file, and then running it through a binder to make a binary to load onto a bare-metal '386 in protected mode. I found it extremely useful to also generate a file to tell Sourcer about all the special bits (descriptor tables etc.), and usually used Sourcer's output in preference to ordinary listing files when debugging with a logic analyser.
    – Mark Morgan Lloyd
    Commented Oct 9, 2020 at 15:59
  • 2
    Technically, device drivers (usually named with a .SYS filename extension) can occur either in flat format files (similar to .COM files but loaded at offset 0 in device mode) or in MZ relocated format files. And a file can even be a valid executable (.COM or MZ) and a device driver.
    – ecm
    Commented Oct 11, 2020 at 15:16
  • 1
    Note that Microsoft eventually changed its loader to look at the file's contents instead of its extension, so that a PE file with .COM extension works just fine. This was possible because no legitimate x86 program starts with "MZ": the corresponding instructions (DEC BP; POP DX) underflow the stack.
    – dan04
    Commented Jun 24, 2021 at 23:47
9

com files are not segmented (they have just a single segment). They have a limitation that they cannot have more than 64K of code (filesize).

They always start in real mode but I do not think there is any restriction on switching to protected mode from the code.

So while disassembling, set real mode. Any switching from the code should be recognized by the tool. However that is improbable as in 64K of code there is not much room to make protected mode stuff as you need to make your own OS-like environment. There were DOS extensions like DOS/4GW used for this but I never saw them used along with *.com files.

Improve this answer
6
  • 1
    The earliest versions of DOS would read .COM files into a sequential region of up to 65,280 bytes, starting at offset 256 of an arbitrary segment, and make a call to offset 256 of that segment, while .EXE files, which had to start with the byte sequence 0x4D 0x5A [character codes "MZ"], would be processed in a more complicated fashion. Later versions, however, process both .COM or .EXE files by checking whether they start with "MZ", and selectthe loading method based upon that. So it might help to clarify what is meant by "com files".
    – supercat
    Commented Oct 8, 2020 at 15:52
  • There's actually nothing in later DOS versions that limits a .COM file to 64k. Try renaming a huge .EXE file to .COM and it will very probably still work.
    – tofro
    Commented Feb 26, 2022 at 15:21
  • 4
    @tofro: that's only if you consider the definition of a COM file to be a file with the .com extension. I suspect changing the extension from .exe to .com doesn't make it a COM file any more than changing it to .txt makes it a text file.
    – paxdiablo
    Commented Feb 27, 2022 at 19:43
  • @paxdiablo A .COM file has no other traits than its extension - because it has no meta information whatsoever in it beyond the binary code.
    – tofro
    Commented Feb 28, 2022 at 11:40
  • 2
    @tofro but other executable files such as .EXE do have other traits, notably the MZ header so renaming an MZ .EXE to .COM ought to mean that the header would be interpreted as machine code by the tool or OS unless it checks for the case of EXE format without an .EXE filename extension.
    – hippietrail
    Commented Mar 1, 2022 at 6:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .