60

From the the two methods of encoding 8-bit data as human-readable ASCII, for a time, uuencode format was more popular. USENET 'binaries' groups were filled with uuencoded posts with whatever goodies were shared. The format was quite robust, insensitive to line breaks (if your mail program reflowed the text, for uuencode you could still decode the file) and the uuencode/uudecode programs were quite user-friendly.

Base64 was not nearly so well liked. Some people would post base64-encoded binaries, arousing mild ire from these, who didn't have decoders. It was sensitive to formatting and white spaces. I'm not entirely sure, but I think it generated a little bigger output too.

Then I was off-the-loop for a time, and when I came back to the Unix and Linux world, uuencode was dead, and wherever 7-bit was still needed, Base64 ruled, and rules to this day.

What happened? What events led to base64 winning the format war?

  • 3
    uuencode IIRC had leading/trailing whitespace as being meaningful, which could cause issues AFAIK. – Muzer Jun 1 '17 at 14:59
  • 2
    uuencode was a de-facto standard that never made it into any RFC. Base64 was a real standard that made it into the earliest MIME RFCs and ended up in a dedicated (RFC3548, RFC3648) one. – tofro Jun 1 '17 at 17:35
  • 1
    @tofro somehow I think if the MIME designers had wanted UUencode, they’d have added it to the RFC ;-). – Stephen Kitt Jun 1 '17 at 18:32
  • 4
    Actually, reflowed uuencode, although theoretically recoverable, was not handled well by the tools. Conversely, the standard Base64 decoding algorithm doesn't care about spaces or line breaks at all. – Leo B. Jun 1 '17 at 23:22
  • 2
    Base64 is not sensitive to whitespace formatting at all as whitespace is specified to have a very specific meaning in base64: it means ignore this byte and read the next byte. – slebetman Jun 3 '17 at 21:13
46

I’m not sure about specific events, but I think the main reason Base64 “won” is that it’s one of the binary encodings supported by MIME, and MIME took over.

So perhaps the question then becomes two-fold:

  • Why did MIME pick Base64 over UUencode? Possibly because Base64 is actually more resistant than UUencode: it only uses alphanumeric characters plus two other characters to encode content (‘+’ and ‘/’ in MIME), and one character for padding (‘=’).
  • Why did MIME become the dominant mail/news content wrapper? I guess it boils down to convenience, especially once most MUAs and news agents supported it (ah, the days of slrn and Forte Agent...).
  • 4
    @Nick MIME was invented in 1990 or 1991, and codified in 1992; URLs were invented in 1992 and codified in 1994. So URL-safety didn’t exist when MIME was invented. (That doesn’t answer the non-path aspect of course since file system paths already used /.) – Stephen Kitt Jun 1 '17 at 22:47
  • 2
    The sad thing is by the time MIME was codified, non-8-bit clean mail transports were mostly a thing of the past. Yet email today still often gets base64 encoded in chance it ends up getting sent through BITNET. – user722 Jun 2 '17 at 16:06
  • 3
    @RossRidge SMTP is a line oriented text based protocol. Even if the whole mail system was 8-bit clean, you'd still need to encode binary files. For example, if a mail server sees the sequence 0x0d 0x0a 0x0d 0x0a 0x2e (a blank line followed by a .) it "knows" that is the end of the message being sent. Such a sequence embedded in a binary file will cause the message to be terminated at that point. – JeremyP Jun 5 '17 at 9:02
  • 3
    @JeremyP The whole mail system is 8-bit clean. You don't need to encode a text message that isn't plain 7-bit ASCII text with the horribly inefficient base64 encoding or even the quoted-printable anymore. The only limitation on the body STMP messages is the 1000 character line length limit. The "dot" problem you mentioned is a non-issue. It's solved by the same dot-stuffing algorithm that lets you send a plain text message with a single period on a line. In practice binary files still need to be encoded (because of poor support for RFC 3030's BINARYMIME), but not most text messages. – user722 Jun 5 '17 at 17:29
  • 5
    @RossRidge SMTP is a line oriented text protocol. Mail servers are at liberty to insert line feeds if they wish to shorten lines. All "8 bit clean" means is that eight bit characters will not lose the top bit when being transferred, it does not mean that you can safely transmit a binary file without some encoding. – JeremyP Jun 6 '17 at 8:58
25

The problem with uuencode is that the format was not robust in the face of some of the really crufty mail software and gateways into and out of proprietary non-SMTP and non-ASCII mail systems of the day. Just to liven things up further, there were multiple EBCDIC variants which had different code points for some ASCII characters used by uuencode, opening up another route for data corruption. For example, the character $ has code point 74 in code page 285 used in the UK, but code point 91 in code page 037 used in the USA.

This corruption would have been one of the driving forces behind the design of MIME, and its character set would have been carefully chosen to minimise problems with such gateways.

  • 1
    Perhaps you could explain why the variation between character assignments in various EBCDIC code pages is more of a problem for uuencode than for base64? – John Bollinger Jun 1 '17 at 21:22
  • 2
    @JohnBollinger: Adding up the letters a-z and A-Z, along with the digits 0-9, yields 62 characters. Thus, base64 only requires two more characters to get to 64, and finding two characters that behave consistently on all systems isn't too hard. – supercat Jun 1 '17 at 21:26
  • 2
    @JohnBollinger, UUEncode uses the upper-case letters, the numbers, and a wide range of punctuation. Base64 uses the upper- and lower-case letters, the numbers, and two punctuation characters. Looking at a selection of EBCDIC Latin-1 code pages, the letters and numbers have consistent code points, but the punctuation code points vary wildly. – Mark Jun 1 '17 at 22:30
  • 4
    @JohnBollinger You are misunderstanding the problem. I uuencode a binary into ASCII text containing a dollar sign. I send that to some gateway that understands it is getting ASCII and converts to EBDIC. The gateway then forwards to another EBCDIC machine that fails to handle the code-page conversion (so the dollar has been mangled to something else). Finally the second EBCDIC machine converts back to ASCII and sends the text to you. You try to uudecode and it all goes horribly wrong. – Martin Bonner supports Monica Jun 2 '17 at 11:29
  • 7
    @JohnBollinger Base64 is not "predicated on the ASCII mappings", except vacuously in the fact that the characters exist at all (vs uuencode which uses the low six bits of a set of 64 consecutive ASCII characters), and the earliest formal description of, RFC 2045, does explicitly mention EBCDIC. And, in fact, "This subset has the important property that it is represented identically in all versions of ISO 646, including US-ASCII, and all characters in the subset are also represented identically in all versions of EBCDIC. " – Random832 Jun 2 '17 at 13:57
12

Base64 is slightly more compact as it does not use a character indicating line length at the beginning of each line:

% dd bs=1k count=1024 < /dev/urandom | uuencode /dev/stdout | wc -с
1444736
% dd bs=1k count=1024 < /dev/urandom | uuencode -m /dev/stdout | wc -c
1421440

Overall, Base64 is about 1.5% better.

  • 2
    this is not an answer. this is a comment – Jasen Feb 24 '19 at 22:35
9

Some of the reasons base64 was disliked was because uuencode stored the original file name and file mode of the encoded data. Also, uuencode had been around longer and was more established, which meant that many people had a uudecode program available but they did not have a base64 decoder. Keep in mind at that time, many people were using systems that did not have a C compiler (the C compiler was often sold as an expensive add-on if it was available at all, and this was before GCC was widely available) so acquiring and compiling their own base64 decoder was a significant effort.

But in certain contexts you didn't need a file name or mode (e.g. inline encoding of the body of an email message), and uuencoded data was particularly vulnerable to corruption because at that time it was not uncommon for a mail gateway somewhere along the line to insert an unwanted newline somewhere within your message, or for character set translation to corrupt something. The extra newlines were usually easy to fix, and the uuencode format made it easy to see where they had occurred, but corruption due to character set translation was much harder to fix (sometimes impossible without trial-and-error testing). Base64 encoding solved these problems and was therefore a better choice for use within the MIME email encoding standard.

The decline in popularity of terminal-mode access compared to GUI access is what really killed uuencode. Users who were using graphical email clients on a PC or on a Workstation or X Terminal, found base64-encoded MIME attachments more convenient than uuencoding, and web browsers allowed you to download files without needing any encoding at all (shifting the common method of binary file transfer away from mail and news, towards the use of FTP and HTTP instead). Uuencoding is still an easy way to send a file when both the sender and receiver are using text-only terminals and can't use FTP, but today this is almost never the case.

  • I don't agree with the comment about C compilers. I can remember when Sun announced that Solaris would no longer come with a C compiler and whilst there was much outrage, in practice we shrugged our shoulders and installed gcc. I think the issue is more likely that most people didn't want to have to write a base64 decoder. Whilst it's not hard, getting it right is not completely trivial and probably out of reach for non programmers. – JeremyP Jan 2 at 16:05
  • uuencode was in common use when Solaris did not yet exist, it was still SunOS. The systems that did not come with bundled C compilers were running things like System III / System V / Xenix. possibly also vendor-specific versions like AIX / HP-UX but I'm not sure. also at that time it was difficult to install gcc if you didn't have a C compiler already, because you needed an existing C compiler to compile gcc to start with. then step two would be to use gcc to recompile itself, and step three would be to repeat and confirm you got identical binaries. – Ken Gober Jan 3 at 19:32
  • My recollection is that all of those versions of Unix you list had C compilers bundled. I may be misremembering, but I worked with System V, Xenix, AIX and HP-UX in ancient days and don't remember us having to buy the C compiler. – JeremyP Jan 6 at 10:21
  • @KenGober There was a tarball of a Solaris gcc available (GNU mirrors perhaps) for the exact purpose of providing the first compiler to bootstrap a new version of gcc. – Thorbjørn Ravn Andersen Jan 9 at 18:51
3

I can't definitively say whether it is cause or effect so am somewhat chancing my arm by promoting it to an answer but: the only way of forming a data URL (i.e. one that has the data directly within it†) is as base64.

Since all moderately substantial application environments supports URLs, even if they don't explicitly support base64 encoding and decoding then they at least support decoding just by forming the data URL. So it's just really easy for developers to support.

Therefore I think base64's usage in URLs may have contributed to its ascension, in the same way that its use in the IBM PC helped the x86 — it's not where the thing came from or why it was designed, but it led to a substantial propagation.

† e.g. this tiny document icon that I cribbed from this site, which doesn't identify a remote resource but itself contains a local resource. You might need to copy and paste it into your browser bar if yours is anything like mine, as trying to follow it like a link from here inevitably leads to the error that it's not a functioning link. Which is the point.

  • 1
    I'd say Base64 was already en vogue (for e-mail) when nobody even knew what an URL was supposed to be... – tofro Jun 1 '17 at 16:28
  • The relevant RFC is from 1998 — tools.ietf.org/html/rfc2397 — when do we think uuencode lost dominance? – Tommy Jun 1 '17 at 16:58
  • 1
    Base64 encoding is already defined (even if the name wasn't established yet) in RFC1421 from 1993 – tofro Jun 1 '17 at 17:31
  • 3
    You've got "cause" and "effect" backwards here. Base64 (RFC 1421) pre-dates URLs (RFC 1738) by almost two years, data: URIs (RFC 2397) by five and a half years, and widespread use of data: URIs by a decade or more. – Mark Jun 1 '17 at 22:37
  • 1
    @Mark I'm fully aware of the sequence of events; my suggestion was that by selecting base64 from the competing options for data: URLs, the IETF gave it a substantial boost. Like saying that x86 is prevalent because it was used in the IBM PC, even though it predates the IBM PC and the market's winnowing down to the IBM PC. – Tommy Jun 2 '17 at 15:32
-2

This also may have a history with multi-byte vs. unicode character. Multi-byte was a stop-gap support for unicode a while back when uuencode was invented. Supporting multi-byte is not really needed unless you have some really old backups.

  • 1
    "..unless you have some really old backups" - or unless you use Java. But who sane would program in Java anyway? Though, technically speaking, surrogates are not only multibyte, but multi-word. And, even more technically speaking, UTF-8 is in fact multibyte... This does not answer the question, though. – Radovan Garabík Dec 30 '20 at 9:03
  • @RadovanGarabík There are plenty of good reasons to program in Java (and many not to) - saneness is not relevant there. It is very good if you need to build cathedrals that need to stand for "centuries", but not if you just need shelter for the night. – Thorbjørn Ravn Andersen Dec 31 '20 at 10:48
  • @ThorbjørnRavnAndersen From what I've seen (admittedly, just a tiny fraction of the wonderful of programming, but over almost 40 years) is that while COBOL has been used to build cathedrals that will stand for centuries (though I've never programmed a line of COBOL myself, ever), Java, by and large, will not last that long. Java, the language, will be around a long time, but I can't see it producing the robust "forever" programs. – manassehkatz-Moving 2 Codidact Jan 4 at 16:10
  • 1
    @manassehkatz-Moving2Codidact Let's have this discussion again in 20 years. – Thorbjørn Ravn Andersen Jan 10 at 5:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.