Why does the default base64 encoding use forward slash /? [closed]

Question

As anyone who has been bitten by using base64 instead of base64url is quite well aware, the "original" base64 alphabet uses alphanumeric, +, = (both perfectly cromulent URL characters), and the dreaded /. I want to know how this came about, because it seems that using / in an encoding alphabet is extremely short-sighted.

I have been able to track the origin of this through:

• RFC4648, which points to
• RFC3548, which points to
• RFC1421, at which point, the base64 trail goes cold.

We can however look to RFC821/RFC822, the famous Simple Mail Transfer Protocol (SMTP). There is no mention of base64 encoding, or any binary-to-text encoding for that matter. I'm not even sure the idea of attaching/sending binary data via email existed at the time, and it wasn't until 1992 that the first MIME email attachment was sent.

I estimate, given the 1993 RFC, that the decision of what would go into the base64 encoding occurred sometime in the mid-80s. Borne shell and awk would have been a thing. ASCII and EBCDIC would be fairly established. I'm guessing it would have happened after uuencode was invented in 1980 (which includes _), though it may have been made without knowledge of uuencode.

Regardless, all the folks working on this sort early tech of were surely familiar with UNIX to some degree, and the convention of using / for path separation. Further, the use of the / symbol as a separator of the form "something / something_else" goes way back, probably to the 18th century at least. I'm guessing this usage is probably what subconsciously influence the choice of / as the UNIX path separator. Surely, one might consider the idea of base64blob1/base64blob2. All this put together makes / a really strange choice for the fledgling base64 encoding alphabet, even in a pre-URL age, especially since better alternatives like - and _ are right there. Heck, even ,.@!$&% are all better candidates than / in my book. / is literally the only character besides \0 disallowed in UNIX filenames.

I guess the motivation for this is, in my mind (as someone who started writing code in the 90s), there is a continuum of characters from "most word-like" to "most code/delimeter-like" (i.e. more like control characters than actual identifiers).

• a-zA-Z the exemplar of word-like
• 0-9 still quite word-like, valid in identifiers
• _ basically the "I need a space but it needs to be an identifier"
• - word-like in some contexts
• @#$%&+= - vaguely word-like but also code-like
• !?:;., - definitely have that delimiter feel
• ^*| - more code-like than word-like
• ""''<>[](){} ok these are definitely delimiters/brackets
• \/ and backtick - unabashedly separators

I wasn't there, but a lot of these "tropes" originate in the 60's and 70's, so I would imagine 80's developers would have similar intuition. / just "feels wrong".

So how did this come about?

Edit: ARPA Text Internet Messages grammar gives special meaning to ()<>@,;:\.[], so it might make sense why those would be excluded, albeit they can be string-quoted. But that still leaves -_, which is present in ASCII 1965 and EBCDIC. Is there some other early character encoding that lacks these that would sway the decision? Perhaps PETSCII?

Edit 2: Let us assume that the character set needed to be common between ASCII63, EBDCIC invariant, and PETSCII. If we take the intersection of these, subtract the SMTP special characters and the obvious alphanumerics, that leaves the candidates =-%/?*&+, of which we need 3. =+- makes a ton of sense, has some nice symmetry to it. ?%& also seem fairly reasonable (pre-URL). / and * feel like the least promising. Is there some other character set or protocol restriction out there which may have ruled out -?%&, necessitating /?

Answer 1

I'm not aware of a (published) rationale for the choice of '+' and '/' as encoding characters, as well as '=' for padding / end-of-message, and I strongly suspect there isn't one.

Base64 was designed as a tool for encoding (8-bit) binary data so that it passes safely and reliably through systems that can handle only 7-bit-ASCII printable characters, and that may insert, delete or modify whitespace along the way[1][2].

Like the earlier uuencode, it does so by taking three 8-bit words, and chopping them into four 6-bit words, each of which is then assigned a printable, non-whitespace character. Unlike uuencode, base64 has the nice property that all the characters used exist in all variations of ASCII and EBCDIC in use at the time[3]. Having used upper case letters, lower case letters and digits, the designers were still two characters short, and made a choice. A different encoding scheme, xxencode, chose '+' and '-' (and a different arrangement of the other characters).

It's worth bearing in mind that base64 encoding was never intended to be used in filenames, and URLs wouldn't exist for another seven years[4].

[1] e.g., by switching between CR LF, CR and LF for line breaks

[2] such as e-mail, at the time

[3] unlike, say, the underscore, which does not exist in the 1963 version of ASCII and Commodore's PETSCII derived from that, rather than the 1967 version of ASCII everybody else was using in the 80s

[4] counting from RFC 989 / Feb 1987 to RFC 1738 / Dec 1994

Answer 2

This answer is speculation but it's too long for a comment and I suspect any answer is likely to involve some speculation.

We can however look to RFC821, the famous Simple Mail Transfer Protocol (SMTP).

RFC821 defines the mechanism used to transmit SMTP messages across the Internet. The content of the messages is outside of its scope. The original definition of the content and structure of a message is in RFC822. However, both of these RFCs long predate base64.

All this put together makes / a really strange choice for the fledgling base64 encoding alphabet, even in a pre-URL age

Not really. After all, it was perfectly possible to encode a path in base64. The only problem arises when people try to read a base64 field as unencoded text and nobody would be that stupid, would they?

Anyway, in the era when base64 was created, the most likely use of it would be in the body of an email, or perhaps in a header of an email. Even today you are more likely to see it in the body of an HTTP message or the headers than in the URL^*. Several of the characters you listed ,.@!$&% have special meanings in the context of an SMTP header, as do two others suggested in another answer i.e. :;. : is used to denote the end of the header key. ; is often used in headers to separate tokens. @, ! and . are used in email addresses. They probably chose the least worst characters in the context of the time in which base64 was created and in the context for which it was designed.

^*I would regard base64 encoded data in a resource path to be an abuse of the standard. Possibly there's a case for get parameters to be base64 encoded but there are better ways even for that. Why would you want to embed binary data in something that's supposed to be human readable?