Executable ASCII files before x86?

Question

I've known about a technique allowing to bootstrap arbitrary 16-bit x86 code from a subset of instructions representable as printable ASCII bytes since the early 1990s.

The first example of an ASCII executable I saw was a short text that could be prepended to a uuencoded file resulting in an MS-DOS .COM executable that would uudecode and probably run itself.

That one I couldn't find, but here's an example of a post reminiscing about x86 ASCII executables, with a few sample files to play with.

For example, an ASCII executable to convert .COM files to executable ASCII starts as follows:

T_OOWW3=XXWX5 2PY5w3P_-l.P-KD1Ep-OLPZ-pJP-pw40PQX5fsPu
ASDWERT/Nide5Fe,xPQX-=.PQX-MQP-xx4_P5rjP5Z2P-jE,JP=
5O2,APQX5R8P-rJPPRX5iBP-x=PRX5TsP59DHHP5rIHP-w64ZP=
40-2APQX-MiP-trP5_WP-pBP51w,pPTYPZPZP__z1t3w.FNtKptDCZ
LGcP4mCC558taMjL.4Hh0.44r5tNNAbs55p4VGsO5n_55LlC8zp_rk
gS5_pOiq.AIkgWub7GwtcOI.C9xO7PC2aPf.stA2.yGQ5JGvMvc4O_

What is the history of this technique? Was it invented for the x86 instruction set or earlier? Which existing instruction set architectures are known to allow it?

Seeing how many people have misinterpreted the question, a clarification:

The main usage of this trick was to publish binaries on USENET for people (or send them by e-mail to people) who don't use Unix and access "the cyberspace" from an MS-DOS machine. They may have no idea what uudecode is. With an ASCII-only executable the instructions are: copy the message in its entirety to a file, delete all lines up to and including the -- cut here -- line, rename the file to whatever.com (for example, uudecode.com) and run it.

In order for it to work, the file, which consists only of printable ASCII — bytes 10 (LF), 13 (CR), 32 to 126 (space to ~), and definitely no bytes with the high bit set (might not pass through e-mail/USENET), and no other control bytes, especially no ^Z — has to begin with a cleverly constructed sequence of machine commands that doesn't use any of the forbidden bytes yet manages to convert the rest of the text from ASCII to the binary intended to be executed.

Alternatively, using ASCII executables could be a way to enter machine code by typing it in more efficiently than hex.

So the question is, is it possible to have a similar converter for 8080/Z80, 6502, or another microcomputer platform, and if yes, has it been used and for what purposes, if not transmitting executables through e-mail?

Answer 1

If you go back a lot before the x86, this technique wasn't unusual at all. In fact, writing programs using printable letters and symbols was pretty much the norm for early computers, except that there was a number of encodings for words of varying bit size, and that encoding was not ASCII.

Examples:

• On the IBM 1401 (1959), a program that looked like

  ,008015,022029,036043,050054,055062,063065,069080/333/M0792502F1.065HELLO WORLD

  would print "HELLO WORLD". Here , (set word mark), / (clear storage), M (move) etc. were opcodes, and the rest was operands. Wikipedia has a list of characters and corresponding opcodes.

• On the Olivetti P101 "desktop computer" (1965), a program like

  b ↑
  B ↑
  b ↓
  B ÷
  A ◇

  would read two numbers, divide them and print the result. More examples in the manual. This machine didn't even have character sets with all Latin letters.

• There was another early computer where the poor programmers had to translate assembler instructions into two characters of a rather random teletype-like charset on papertape, because initially there was no proper assembler, so "writing gibberish" was actually the proper method to program this computer. Unfortunately, I can't remember at the moment which computer that was (will edit answer when I do).

And there's probably a lot more examples.

So the technique itself is quite old. Coming back to encoding machine language into ASCII, in principle, one can apply it to any kind of ISA, one just has to define what part of ASCII one considers as admissible, which part of the instructions set of the particular CPU they match, and then it becomes an exercise to encode what you want in this restricted manner.

Answer 2

It was standard practice on the Sinclair ZX80 & ZX81 to put executable code into a REM statement at the beginning of a BASIC program.

REM statements are, of course, text comments, so this meets the spirit of your requirement for executable ASCII.

The ZX80 (1980) and ZX81 (1981) predate your question about the early 1990s by about 10 years and used the Z80 processor.

There is a guide on how to put executable code into REM statements here. Essentially poking assembler codes back into a reserved space.

This is just one example of the use of this technique. It was also used later in the 1980s on the BBC Micro. Small embedded subroutines were also put into REM statements on the HP9845, mostly to accelerate arithmetic calculations.

Answer 3

Short Answer

It can be done in any environment that:

1. Allows the remarking of data files into program files,
2. Has a loader format that's either primitive enough or all readable
3. Has a character set (doesn't have to be ASCII) that has a sufficient number of encodings that produce valid opcodes
4. Has an address space layout that fits the possible encodings
5. Necessary OS calls can be encoded (subset of 3.)

Numbers 4. and 5. can be circumvented depending on the machine, the OS and available text encodings.

Since some small machines have 256 printable characters, the difference between a binary and a text file is negligible anyway.

History here is a bit hard to grasp, as the necessity didn't arose in ye old days. Who needs tricks when you've got full access? But there was a somewhat similar situation for early mainframes - while there was a 'binary' mode for punch cards for many machines, one could also punch 'text' cards with encodings outside of the (usual) readable range. This worked, since the translation of 12 hole punch card code into 8 Bit EBCDIC worked according to fixed rules.

Remember, real men always had a handpunch near, or carry at least a porta-a-punch.

Answer 4

I remember doing this on the university mainframe around 1975. This was on an ICL1904S. Note that the 1900 series had been around for more than 10 years at that time. I don't know when the feature came out but it had been around for some time.

You could list out any executable in card reader format. It would produce the executable in 6-bit characters in lines of 80 characters. Not really ASCII - the whole system ran on 6-bit a character set. These could then be embedded in the GEORGE 3/4 batch files.

It was absolutely brilliant because the uni ran a cleanup every 2-3 weeks. The OS would go through and delete all the executables and intermediate object files but it left the batch files alone, whatever their size.

Answer 5

For a slightly interesting twist on this concept, consider Control Data mainframes.

These beasts included not only a CPU, but a "peripheral processing unit" (PPU)¹--and the CPU sent commands to the PPU via normal I/O channels.

The CPU was a 60-bit processor that used 6-bit character codes. The PPUs were 12-bit processors, so the CPU sent a stream of 2 characters to the PPU to send it commands. The PPU commands all required that the first character of the string be a 0.

In most CDC character sets (they had a few, since each one only supported 64 characters), a 0 character was a colon, and letters started at 1, so A=1, B=2, etc.

One semi-popular trick when I was in college was to get a user to execute a program that tried to print the string :D to the screen. As it happened, PPU command 4 was "log off user"...

---

1. In the higher-end machines, this was officially a "peripheral processor" (PP) instead, but the concept remained essentially similar.

Answer 6

This certainly seems to be possible on the 6502. While several seemingly crucial instructions (like STA, STX and STY) exist only with the 8th bit set, it's still possible to construct arbitrary bytes in RAM using SEC with the read-modify-write forms of ROL, ROR and/or LSR, provided the RAM addresses are printable ASCII. The full set of ADC/EOR/AND opcodes are also available to speed up construction of arbitrary bytes, then both JMP and JSR are available, as are BPL, BVC, BVS.

It's straightforward to see how this could be used to construct a small program in zero-page, which could in turn accept hex digits or even Base64, and translate that into a full program.

Answer 7

The first example of an ASCII executable you saw is in the Google Usenet archive here

Answer 8

Is 1949 early enough?

The Manchester Mark 1 had 20-bit wide instructions which were conventionally written as four 5-bit characters using a variation on teleprinter code which Alan Turing adapted for the purpose by replacing control codes with printable characters so that all instructions and data could be written as text.

One might suggest this is cheating because all machine code can be written in e.g. hexadecimal to avoid ASCII control codes, or because it's not ASCII (which didn't exist until the 1960s). However, since the entire point of the exercise was to be able to enter text which can be executed directly, it surely counts.

Similar adaptations of ASCII also exist, e.g. ye olde CP437 which has glyphs for all 256 values. Some very large C=64 programs also shove code and data into the text-mode buffer (what with it being the only spare memory left) and you can watch the PETSCII characters twinkle on the screen as it executes.