8

Windows and DOS binary executable files with the .EXE extension have an MZ header in them and nowadays also a PE header.

But before these there used to also be .COM binary executable files and they had no internal header, they were just raw x86 machine code.

But Intel's x86 processor family has been around for ages from the 8-bit era through to the 64-bit era with the instruction set and the memory model changing several times. Memory models had to do with how to access more memory beyond the processor's address bus could directly address. There used to be multiple pages of memory and segment registers

So back to .COM files, did they use "real mode" or "protected mode"? Or were both used? If both were used, was there any way to tell which was needed?

I want to disassemble some old .COM executables and the tool asks me whether the files are real mode or protected mode.

(I did assembly and machine code programming on Z80 and Motorola 680x0 and switched to PCs in the 486 (32-bit) era but found the assembly syntax and segmented memory models ugly and didn't continue with assembly language or even understand the stuff about memory segments and such.)

Improve this question
1
  • 2
    Certainly it's possible for code written as real-mode code to execute in protected mode, in the V86 submode.
    – another-dave
    Apr 25, 2020 at 12:40

2 Answers 2

Reset to default
20

DOS programs always start in real mode (or an emulation thereof), so it’s best to start disassembling them assuming that. When disassembling, you should assume real mode, with 16-bit data and 16-bit addresses, until the code you’re disassembling changes that. The DOS-based disassemblers I’ve used generally know about the executable formats involved, and don’t ask. (They need to know about the formats to be able to set the segments up appropriately.)

In DOS, there are three executable layouts, and the extension doesn’t matter.

The first is the “.COM” layout, which is loaded into memory as-is after the Program Segment Prefix, with all free memory allocated to the process, and all segments set to point at the PSP. Execution starts at offset 0100h, which maps to offset 0 in the file.

The second is the device driver layout, which I won’t detail here.

The third is the MZ layout, with a header used to tell the loader how to load the executable. This header specifies how much of the executable to load initially, how much memory to allocate, the segment layout, relocations to perform, and where execution starts.

All three layouts leave plenty of room for a switch to protected mode, so ideally your disassembler should be able to recognise that.

Non-DOS executables have a real-mode DOS stub; if you disassemble assuming real-mode MZ, that‘s what you’ll see. Again, most disassemblers can interpret the format correctly and will open LE, NE, PE etc. appropriately.

The age of your executable can be a useful piece of information: protected mode DOS programs were unusual until at least 1993.

With the freeware version of IDA 7, you can analyse COM files by loading them as binary files, changing the loading offset to 0x100, and selecting 16-bit mode.

Improve this answer
10
  • I'm using Ghidra, which works with many executable formats and CPU architectures, but its loaders seem to ignore file extensions and detect file types only by their internal format. But as .com files have no internal format it doesn't recognize them, and I suppose these are not the kinds of binaries that are the primary target of Ghidra anyway. Dunno. My executable is an old ZX Spectrum emulator called Spec256 from the '00s or '90s.
    – hippietrail
    Apr 25, 2020 at 9:28
  • 1
    Right, Ghidra doesn’t know how to handle COM files unfortunately (although I imagine it would be possible to teach it how). Give the demo of IDA a try, it should work much better for this type of analysis.
    – Stephen Kitt
    Apr 25, 2020 at 10:19
  • 4
    DOS didn’t exist pre-16-bit, it was always a 16-bit operating system. (See Can x86 processors run 8-bit applications?) It also didn’t make the migration to 32 bits. You should always assume 16-bit data and addresses, until the program being analysed issues instructions (or prefixes) to change that. One common feature was for 386+ programs to use the 32-bit registers in real mode, but you’d see the corresponding prefix. For Windows programs things are different, but they all have headers.
    – Stephen Kitt
    Apr 27, 2020 at 8:03
  • 1
    Way back I was doing some stuff which involved generating an NE-format file, and then running it through a binder to make a binary to load onto a bare-metal '386 in protected mode. I found it extremely useful to also generate a file to tell Sourcer about all the special bits (descriptor tables etc.), and usually used Sourcer's output in preference to ordinary listing files when debugging with a logic analyser.
    – Mark Morgan Lloyd
    Oct 9, 2020 at 15:59
  • 1
    Technically, device drivers (usually named with a .SYS filename extension) can occur either in flat format files (similar to .COM files but loaded at offset 0 in device mode) or in MZ relocated format files. And a file can even be a valid executable (.COM or MZ) and a device driver.
    – ecm
    Oct 11, 2020 at 15:16
7

com files are not segmented (its just single segment). They have limitation that it can not cross 64K of code (filesize).

They always starts in real mode but I do not think there is any restriction to switch to protected from the code.

So while disassembling set real mode any switching from the code should be recognized by the tool. However that is improbable as in 64K of code there is not much room to make protected mode stuff as you need to make own OS like environment. There where used DOS extentions like DOS/4GW for this but newer saw it used along with *.com file.

Improve this answer
5
  • 1
    The earliest versions of DOS would read .COM files into a sequential region of up to 65,280 bytes, starting at offset 256 of an arbitrary segment, and make a call to offset 256 of that segment, while .EXE files, which had to start with the byte sequence 0x4D 0x5A [character codes "MZ"], would be processed in a more complicated fashion. Later versions, however, process both .COM or .EXE files by checking whether they start with "MZ", and selectthe loading method based upon that. So it might help to clarify what is meant by "com files".
    – supercat
    Oct 8, 2020 at 15:52
  • There's actually nothing in later DOS versions that limits a .COM file to 64k. Try renaming a huge .EXE file to .COM and it will very probably still work.
    – tofro
    Feb 26, 2022 at 15:21
  • 2
    @tofro: that's only if you consider the definition of a COM file to be a file with the .com extension. I suspect changing the extension from .exe to .com doesn't make it a COM file any more than changing it to .txt makes it a text file.
    – paxdiablo
    Feb 27, 2022 at 19:43
  • @paxdiablo A .COM file has no other traits than its extension - because it has no meta information whatsoever in it beyond the binary code.
    – tofro
    Feb 28, 2022 at 11:40
  • @tofro but other executable files such as .EXE do have other traits, notably the MZ header so renaming an MZ .EXE to .COM ought to mean that the header would be interpreted as machine code by the tool or OS unless it checks for the case of EXE format without an .EXE filename extension.
    – hippietrail
    Mar 1, 2022 at 6:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .