10

I'm looking for a way to identify CP/M-80 (i.e. 8080 code) program files (.COM) from their content. A process usually known as Magic Number or File Signature detection, intended to identify a file by looking at their first few bytes for certain values and/or structure.

To narrow this down a bit, it will be fine if the distinction is good enough to separate CP/M-80 .COM files from their counterpart under MS-DOS(*1). That means, while a positive detection of a .COM file being 8080 code is great, already a negative detection of it not being 8086 code will do the trick.

I couldn't find any information. In addition, what I could figure out so far is rather limited:

  • CP/M 3 file may start with an optional header starting with a C9h (RET) marking a new, optional header used for RSX linking.

Basic CP/M programs often start with

  • Loading the stack pointer LXI SP,const (31 xx xx)
  • A variant thereof is LHLD adr/SPHL (2A xx xx F9)
  • Some even save SP first with LXI H,0/DAD SP/SHLD xxx/LXI SP,const
  • Often this is prefixed by a jump around some header JMP const/....suff.../LXI ... (C3 xx xx ...)

But I also found next to any other instruction when scanning files I have at hand - including pushing all registers onto the OS stack first...

The good part is that on a x86

  • C3h is a RET,
  • 31h is a XOR r/m16,r16,
  • 2Ah is a SUB r8,r/m8

The first is rather useless as a program start, but might be used to make prevent data files from execution, while the second may be unusual, but not unheard of to clear a register during setup. The last one is most definitive unusual for a program start, but as a 2...4 byte instruction any combination may show up as 4th byte, so a check for F9h is even weaker.

So either are rather weak indicators.


Question: Is there any (somewhat reliable) fast way for detecting CP/M-80 program files.


Background

From comments it seems that it would be of help to add why I'm looking for that information:

I'm in some NEC V20 nostalgia phase, here about the only useful improvement beside speed and 80186 instruction extension: The 8080 emulation mode. I'm tinkering with first drafts of a modified MS-DOS COMMAND.COM that would - or as of now should - detect when a .COM file is selected for execution if that one is possibly an 8080 one for CP/M-80. If this is the case it would

  • prepare the PSP according to CP/M (won't need much change),
  • add a BDOS to MS-DOS translation stub to top of memory (right below 64KiB),
  • Setup emulation vectors
  • invoke emulation mode and
  • clean up changed vectors after exit.

All of that not a big deal and already working fine, except for the detection part. And yes, I know, this has been done back then several times in form of loaders (which mine already is), but my attempt is to make it seamless, much like it was possible on CP/M-86.

Oh, and all of that should of course work on a 4.77 MHz XT without much delay - remember kids, grampa had to shovel his data with a toy sized scoop, both ways up hill and in the snow.


*1 - Using the same extension as CP/M-80 was a real stupid decision. CP/M-86 was at least so kind to go with .CMD - and drop memory images totally.

35
  • 1
    You likely need to examine more of the file after making sure it isn't something else (like an .exe renamed .com). Scanning for DOS/BIOS and CPM/BDOS call sites is one option. The values around a CD opcode (INT on x86 and CALL on 8080) strongly suggest the OS.
    – Brian
    Oct 25, 2022 at 12:48
  • 3
    An interesting alternate way of looking at this is: Is it possible to create an executable that will run on both CP/M-80 and MS-DOS? Some specific combination of opcodes at the beginning that would effectively jump to one location in the file for CP/M-80 and a different location for MS-DOS and then execute appropriate (8080 or 8086) code? I have seen plenty of programming contests/puzzles/etc. over the years where the goal is to run one piece of code in multiple languages, the difference here is this is object code, not source code. Oct 25, 2022 at 14:42
  • 2
    @manassehkatz-Moving2Codidact What a fun idea. Would it make a good puzzle on the codegolf site? Oct 25, 2022 at 15:49
  • 5
    @manassehkatz-Moving2Codidact The short answer is 'yes': en.wikipedia.org/wiki/Fat_binary#CP/M_and_DOS
    – john_e
    Oct 25, 2022 at 16:04
  • 2
    @thebusybee: A lot of programs for both CP/M and DOS were assembled rather than compiled, so a search for the signatures of common compilers would draw a blank on a great many programs.
    – john_e
    Oct 26, 2022 at 9:12

3 Answers 3

17

At user Raffzahn's request, I've expanded a comment into an overly complex statistical answer. In summary, if a random *.COM file is 27392 bytes or fewer and starts with one of twelve byte values, it is very probably a CP/M-80 executable.

There is no practical way to use “magic numbers” to identify CP/M-80 executable files. The Linux file command typically identifies them as one of the following MIME types:

  • application/octet-stream
  • application/x-dbt
  • application/x-dosexec
  • application/x-matlab-data
  • application/x-tex-tfm
  • application/zlib
  • image/x-tga
  • text/plain

In a review of 5510 unique *.COM files lying about my hard drive (mostly from the Oakland CP/M Archive CD-ROM, the Walnut Creek CP/M CD-ROM and z80pack), with potential MS-DOS or other non-8080-compatible executables weeded out, I found:

Size

  • 25% are 1792 bytes or fewer
  • 50% are 4864 bytes or fewer
  • 75% are 12288 bytes or fewer
  • 95% are 27392 bytes or fewer
  • All of them are 53504 bytes or fewer.

First byte

  • 60% have a first byte (c3)
  • 75% have a first byte (c3|21|2a)
  • 80% have a first byte (c3|21|2a|31)
  • 90% have a first byte (c3|21|2a|31|18|ed|00|cd)
  • 95% have a first byte (c3|21|2a|31|18|ed|00|cd|0e|01|11|3a)

MS-DOS

Using all 3508 unique *.COM files on the Power DOS CD ROM (Walnut Creek) (July 1995) and the Simtel MSDOS 1996-09 CDs, I found:

Size

  • 25% are 768 bytes or fewer
  • 50% are 3072 bytes or fewer
  • 75% are 9984 bytes or fewer
  • 80% are 12800 bytes or fewer
  • 85% are 15872 bytes or fewer
  • 90% are 21760 bytes or fewer
  • 95% are 32768 bytes or fewer
  • All of them are 187904 bytes or fewer. These must have some additional payload to be this large.

(So the typical file size could be smaller under MS-DOS. Perhaps some of these have been packed, a technology that wasn't as effective or widespread under CP/M.)

First byte

  • 45% have a first byte (e9)
  • 65% have a first byte (e9|eb|8c)
  • 75% have a first byte (e9|eb|8c|b4|4d)
  • 80% have a first byte (e9|eb|8c|b4|4d|b8)
  • 90% have a first byte (e9|eb|8c|b4|4d|b8|e8|fc|24|bc|bb)
  • 95% have a first byte (e9|eb|8c|b4|4d|b8|e8|fc|24|bc|bb|be|ba|fb|9c|b0|1e|33)

Note that none of these common MS-DOS first bytes are shared with CP/M-80 *.COM files. Checking the first byte may be all you need to do. Like all statistical assumptions, however, you won't catch every last outlier.

19
  • 2
    sample corpus and methodology can be made available, if anyone cares enough about this
    – scruss
    Oct 27, 2022 at 13:28
  • 2
    Statistical fingerprinting. Nice. You could also train an AI on a labelled set of DOS .com files and CP/M .com files. Oct 27, 2022 at 14:21
  • 1
    @scruss How lovely. You're awesome. To to make sure I read this right, the slashes are meant as or, meaning 60% start by C3h JMP a16, 15% with 21h LXI H,i16 or 2Ah LHLD a16, 5% with 31h LXI SP,i16 and so on? 18h and EDh are as well nice finds as these are most likely Z80 programs. 01h&11h might be variants loading SP via BC or DE - assuming the next is some way to move that into SP. As the saying goes, no good deed goes unpunished, have you tried to look at the next instruction as well? Again, that statistic is a real great help. Thanks you.
    – Raffzahn
    Oct 27, 2022 at 14:47
  • 1
    @Raffzahn - oops, I guess I'm too used to using egrep. Also, I'm using a percentage exceedance (can't help it; former due diligence engineer). The 75% line means: of all the probably CP/M COM files I could find, ¾ or more of them start with the byte c3 or with the byte 21 or with the byte 2a. Consequently, less than ¼ of the COM files start with any of the other 253 byte values. I wouldn't infer anything from the intervals between the %age steps: it won't be exactly 15% that have first byte 21 or 2a. I can add raw percentages later
    – scruss
    Oct 27, 2022 at 15:49
  • 1
    @scruss "it would melt Raffzahn's poor NEC V20" - so far it runs remarkably cool at 4.77 MHz .. 9.54 MHz (*1) ... Cool as a Sloth :)) *1 -I 'pimped' that machine back then not only with a V20 but as well a variable speed clock generator which configures maximum clock speed between 4.77 and 9.54, depending on which memory address is accessed. A Sloth riding a Ferrari :)
    – Raffzahn
    Oct 27, 2022 at 16:11
5

A process usually known as Magic Number or File Signature detection, intended to identify a file by looking at their first few bytes for certain values and/or structure.

As you have found out, there is nothing like a Magic Number like you'd find in an ELF binary or anything like that. Rather, the fact it's executable is determined by CP/M by looking at the filename's extension. Yes, it's another bad idea, but that's how it worked.

Question: Is there any (somewhat reliable) fast way for detecting CP/M-80 program files.

You could look at Markov Chains; essentially read the instruction stream and see which of your models best model your observed state transitions. So then you'd see if your program is an 80186 MS-DOS program, or an CP/M-80 program, or this, or that, or the other. The technique would normally require heavy use of floating point, so maybe not exactly fast if you're intending to do this detection on the V20 itself.

I have no idea if this is a workable idea.

I'm tinkering with first drafts of a modified MS-DOS COMMAND.COM that would - or as of now should - detect when a .COM file is selected for execution if that one is possibly an 8080 one for CP/M-80.

Cool idea! The best of luck with it. Depending on where you want to take this idea, you could consider using some kind of meta-data. See if there's an unused bit in the filesystem, or use a different filename extension, or keep a reference to the executable in another file, or something along those lines.

Or, require the user to explicitly invoke your compatibility layer, just like a Linux user has to type wine minesweeper.exe. Just typing minesweeper.exe on Linux will result in a permissions error.

Using the same extension as CP/M-80 was a real stupid decision. CP/M-86 was at least so kind to go with .CMD - and drop memory images totally.

This part reminded me of a question I asked some years ago which, while being about UNIX and so not directly relating to anything like a solution for you, has some insights of various kinds in the answers. In particular, finding that a binary was for the wrong architecture, I surmise was not a problem that anyone was facing.

2
  • 5
    Typing minesweeper.exe on Linux won't necessarily result in a permissions error. If you've put minesweeper.exe in your PATH (otherwise, you need ./minesweeper.exe) and run chmod +x minesweeper.exe and then registered the MZ magic number with the kernel's binfmt_misc system, then it'll just transparently hand it off to Wine.
    – ssokolow
    Oct 26, 2022 at 11:25
  • 2
    @ssokolow Actually that is how it works in WSL2. If you execute a Windows binary it is run normally. Oct 28, 2022 at 12:00
2

Write two small interpreters that understand a subset of 8080 and 8086 opcodes, and use each to trace the first few dozen bytes of proposed execution of the program.

Note that you don't need to have much understanding of the opcodes: for a great many of them, perhaps vast majority, all you need to know about the instruction is the length so you can skip past it. For example all or almost all of the 128 instructions from $40 - $BF for the 8080 can be interpreted as "ignored 1-byte instruction."

You will need to be able to follow jumps, calls and the like, and you'll need to know a few other instructions depending on how exactly you implement heuristics as suggested below.

There are some simple 8080 heuristics you can look for to make a judgement.

  1. Obviously invalid instructions (e.g., any of 08 10 18 20 28 30 38 CB D9 DD ED FD) can immediately disqualify the binary as 8080. (This would be more difficult if you were trying to identify Z80 code as well, but since the V20/V30 don't run Z80 code, you don't need to worry about that.)

  2. Look for call $0005 (especially when preceded by ld c,BB). These indicate a call into the BDOS, and many CP/M programs are likely to do at least one call fairly early on.

  3. Also consider ld hl,($0006), which gives the maximum RAM address (plus one) that a CP/M program can use. Being followed by an ld sp,hl instruction would probably seal the deal.

There will be similar heuristics you can use for MS-DOS programs, such as calling into the INT routines that make BIOS and MS-DOS calls, and so on.

I suspect that without too much work on this you can get fairly good detection of both types of programs. Perhaps more importantly, searching for these positive indications is likely to result in far fewer programs that get mis-detected and run anyway, usually causing a crash. Instead you'll get a notice that your heuristics couldn't positively detect either 8080 or 8086 code, and when you find such binaries you can go disassemble them, figure out what they are, and see if there are any reasonable extensions to your heuristic that would produce an identification for binaries like that.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .