Why wasn't ASCII designed with a contiguous alphanumeric character order?

Question

Anyone who has dealt with strings at a low level (e.g., writing a parser in C), knows that doing so tends to involve frequent checks of—either manually, or through isalpha(), isalnum(), etc—whether a character is a case-insensitive alphabetic character...

(*c >= 'A' && *c <= 'Z') || (*c >= 'a' && *c <= 'z')

...or an alphanumeric character...

(*c >= '0' && *c <= '9') || (*c >= 'A' && *c <= 'Z') || (*c >= 'a' && *c <= 'z')

...because the above ranges are non-contiguous in the ASCII order of characters (and many subsequent character encodings such as UTF-8). If only these 3 ranges were adjacent to each other, a much more succinct...

*c >= '0' && *c <= 'z'

...would save countless keystrokes and once-precious clock cycles since the dawn of unix time (or actually a number of years earlier — signed integer).

I guess other considerations were considered more pressing at the time. Does anyone remember what they might have been?

Answer 1

Why is ASCII this way?

First of all, there is no one best sorting order for everything. For example, should UPPER or lower case be first? Should numbers be before or after letters? Too many choices, and no way to please everyone. So they came up with specific pieces that "made sense":

• Numerals

0x30–0x39 - Easy bit mask to get your integer value.

• UPPER case letters

0x41–0x5A - Another easy bit mask to get your letter relative value. They could have started with 0x40, or put the space at the beginning. They ended up putting space at the beginning of all printable characters (0x20), which makes a lot of sense. So we ended up with @ at 0x40 - no specific logic to that particular character that I know of, but having something there and starting the letters at 0x41 makes sense to me for the times you need a placeholder of some sort to mark "right before the letters".

• lower case letters

0x61–0x7a - Again a simple bit mask to get the letter relative value. Plus, if you want to turn UPPER into lower or vice versa, just flip one bit.

• Control codes

These could have been anywhere. But placing them at the beginning has the nice advantage that extensions to the character set - from 128 to 256 and beyond - can treat everything >= 0x20 as printable.

• Everything else

Everything else got filled in - 0x21–0x2f, 0x3a–0x3f, 0x5b–0x5f, 0x7b–0x7f. "Matching" characters generally next to each other, like ( and ), or with one character separating them, like < = > and [ \ ]. Most of the more universally used characters are earlier in the character set. The last character - 0x7f (delete or rubout) is another special case because it has all 7 bits set - see Delete character for all the gory details.

Why is C (and most other languages) this way?

A high-level language should be designed to be machine independent, or rather to make it possible to implement on different architectures with minimal changes. A language may be far more efficient on one architecture than another, but it should be possible to implement reasonably well on different architectures. A common example of this is Endianness, but another example is character sets. Most character sets - yes, even EBCDIC, do some logical grouping of letters and numbers. In the case of EBCDIC, the letters are in sequence, but lower case is before UPPER case and each alphabet is split into 3 chunks. So isalpha(), isalnum() and similar functions play a vital role. If you use (*c >= '0' && *c <= '9') || (*c >= 'A' && *c <= 'Z') || (*c >= 'a' && *c <= 'z') on an ASCII system it will be correct, but on an EBCDIC system it will not be correct - it will have quite a few False Positives. And while *c >= '0' && *c <= 'z' would have lots of False Positives in ASCII, it will totally fail in EBCDIC.

Arguably, a "perfect text sorting character set" could be created that would fit your "ideal", but it would inevitably be less-than-ideal for some other use. Every character set is a compromise.

Answer 2

According to ASA X3.4-1963 Appendix A, one of the design considerations was:

(7) Ease in the identification of classes of characters

Furthermore:

A4.4 The character set was structured to enable the easy identification of classes of graphics and controls.

And on page 8:

A6.3 To simplify the design of typewriter-like devices, it is desirable that there be only a common 1-bit difference between characters normally paired on keytops. This, together with the requirement for a contiguous alphabet, the collating requirements outlined above, and international considerations, resulted in the placement of the alphabet in the last two columns of the graphic subset. This left the second column of the graphic subset for the numerals.

There is a considerable amount of other information about the structure of ASCII in that appendix.

Answer 3

man 7 ascii of Linux Programmer's Manual says,

Uppercase and lowercase characters differ by just one bit and the ASCII character 2 differs from the double quote by just one bit, too. That made it much easier to encode characters mechanically or with a non microcontroller-based electronic keyboard and that pairing was found on old teletypes.

As supplement information, Eric S. Raymond authored Things Every Hacker Once Knew a few years ago. It has a section on the purposes of various designs in ASCII. Reading it is strongly recommended.

ASCII, the American Standard Code for Information Interchange, evolved in the early 1960s out of a family of character codes used on teletypes.

ASCII, unlike a lot of other early character encodings, is likely to live forever - because by design the low 127 code points of Unicode are ASCII. If you know what UTF-8 is (and you should) every ASCII file is correct UTF-8 as well.

The following table describes ASCII-1967, the version in use today. This is the 16x4 format given in most references.

Dec Hex    Dec Hex    Dec Hex  Dec Hex  Dec Hex  Dec Hex   Dec Hex   Dec Hex
  0 00 NUL  16 10 DLE  32 20    48 30 0  64 40 @  80 50 P   96 60 `  112 70 p
  1 01 SOH  17 11 DC1  33 21 !  49 31 1  65 41 A  81 51 Q   97 61 a  113 71 q
  2 02 STX  18 12 DC2  34 22 "  50 32 2  66 42 B  82 52 R   98 62 b  114 72 r
  3 03 ETX  19 13 DC3  35 23 #  51 33 3  67 43 C  83 53 S   99 63 c  115 73 s
  4 04 EOT  20 14 DC4  36 24 $  52 34 4  68 44 D  84 54 T  100 64 d  116 74 t
  5 05 ENQ  21 15 NAK  37 25 %  53 35 5  69 45 E  85 55 U  101 65 e  117 75 u
  6 06 ACK  22 16 SYN  38 26 &  54 36 6  70 46 F  86 56 V  102 66 f  118 76 v
  7 07 BEL  23 17 ETB  39 27 '  55 37 7  71 47 G  87 57 W  103 67 g  119 77 w
  8 08 BS   24 18 CAN  40 28 (  56 38 8  72 48 H  88 58 X  104 68 h  120 78 x
  9 09 HT   25 19 EM   41 29 )  57 39 9  73 49 I  89 59 Y  105 69 i  121 79 y
 10 0A LF   26 1A SUB  42 2A *  58 3A :  74 4A J  90 5A Z  106 6A j  122 7A z
 11 0B VT   27 1B ESC  43 2B +  59 3B ;  75 4B K  91 5B [  107 6B k  123 7B {
 12 0C FF   28 1C FS   44 2C ,  60 3C <  76 4C L  92 5C \  108 6C l  124 7C |
 13 0D CR   29 1D GS   45 2D -  61 3D =  77 4D M  93 5D ]  109 6D m  125 7D }
 14 0E SO   30 1E RS   46 2E .  62 3E >  78 4E N  94 5E ^  110 6E n  126 7E ~
 15 0F SI   31 1F US   47 2F /  63 3F ?  79 4F O  95 5F _  111 6F o  127 7F DEL

However, this format - less used because the shape is inconvenient - probably does more to explain the encoding:

   0000000 NUL    0100000      1000000 @    1100000 `
   0000001 SOH    0100001 !    1000001 A    1100001 a
   0000010 STX    0100010 "    1000010 B    1100010 b
   0000011 ETX    0100011 #    1000011 C    1100011 c
   0000100 EOT    0100100 $    1000100 D    1100100 d
   0000101 ENQ    0100101 %    1000101 E    1100101 e
   0000110 ACK    0100110 &    1000110 F    1100110 f
   0000111 BEL    0100111 '    1000111 G    1100111 g
   0001000 BS     0101000 (    1001000 H    1101000 h
   0001001 HT     0101001 )    1001001 I    1101001 i
   0001010 LF     0101010 *    1001010 J    1101010 j
   0001011 VT     0101011 +    1001011 K    1101011 k
   0001100 FF     0101100 ,    1001100 L    1101100 l
   0001101 CR     0101101 -    1001101 M    1101101 m
   0001110 SO     0101110 .    1001110 N    1101110 n
   0001111 SI     0101111 /    1001111 O    1101111 o
   0010000 DLE    0110000 0    1010000 P    1110000 p
   0010001 DC1    0110001 1    1010001 Q    1110001 q
   0010010 DC2    0110010 2    1010010 R    1110010 r
   0010011 DC3    0110011 3    1010011 S    1110011 s
   0010100 DC4    0110100 4    1010100 T    1110100 t
   0010101 NAK    0110101 5    1010101 U    1110101 u
   0010110 SYN    0110110 6    1010110 V    1110110 v
   0010111 ETB    0110111 7    1010111 W    1110111 w
   0011000 CAN    0111000 8    1011000 X    1111000 x
   0011001 EM     0111001 9    1011001 Y    1111001 y
   0011010 SUB    0111010 :    1011010 Z    1111010 z
   0011011 ESC    0111011 ;    1011011 [    1111011 {
   0011100 FS     0111100 <    1011100 \    1111100 |
   0011101 GS     0111101 =    1011101 ]    1111101 }
   0011110 RS     0111110 >    1011110 ^    1111110 ~
   0011111 US     0111111 ?    1011111 _    1111111 DEL

Using the second table, it’s easier to understand a couple of things:

• The Control modifier on your keyboard basically clears the top three bits of whatever character you type, leaving the bottom five and mapping it to the 0..31 range. So, for example, Ctrl-SPACE, Ctrl-@, and Ctrl-` all mean the same thing: NUL.

• Very old keyboards used to do Shift just by toggling the 32 or 16 bit, depending on the key; this is why the relationship between small and capital letters in ASCII is so regular, and the relationship between numbers and symbols, and some pairs of symbols, is sort of regular if you squint at it. The ASR-33, which was an all-uppercase terminal, even let you generate some punctuation characters it didn’t have keys for by shifting the 16 bit; thus, for example, Shift-K (0x4B) became a [ (0x5B)

It used to be common knowledge that the original 1963 ASCII had been sightly different. It lacked tilde and vertical bar; 5E was an up-arrow rather than a caret, and 5F was a left arrow rather than underscore. Some early adopters (notably DEC) held to the 1963 version.

If you learned your chops after 1990 or so, the mysterious part of this is likely the control characters, code points 0-31. You probably know that C uses NUL as a string terminator. Others, notably LF = Line Feed and HT = Horizontal Tab, show up in plain text. But what about the rest?

Many of these are remnants from teletype protocols that have either been dead for a very long time or, if still live, are completely unknown in computing circles. A few had conventional meanings that were half-forgotten even before Internet times. A very few are still used in binary data protocols today.

Here’s a tour of the meanings these had in older computing, or retain today. If you feel an urge to send me more, remember that the emphasis here is on what was common knowledge back in the day. If I don’t know it now, we probably didn’t generally know it then.

Full text.

Answer 4

This chart (showing the hexadecimal values of ASCII characters) outlines manassehkatz's answer graphically:

• Numbers are at 0x30 + the value of the number
• Capital letters are at 0x40 + the value of the letter (A=1, B=2 etc)
• Lowercase letters are at 0x60 + the value of the letter.

Answer 5

None of the existing answers consider the context of the development of ASCII. Remember, the first version of the standard was released in 1963.

1. At the time, hexadecimal notation was neither popular nor standardized. Most systems used either binary-coded-decimal or octal, for which the digits 0-9 are sufficient.

   As described in this question, at least four other ways of representing the hexadecimal digits 10-15 were used. The choice of A-F -- which may seem obvious today -- was not introduced until the IBM System/360, which first came out in 1964, after the first version of the ASCII standard.

   Thus, hexadecimal notation was not even a consideration by the ASCII committee.

2. The ASCII committee decided early on that the lowest four bits of a decimal digit's encoding must be the actual digit. In other words, 0 must encode as XXX0000, and 9 must encode as XXX1001.

   Criterion 6. The numerics should have bit patterns such that the four low-order bits shall be the binary coded decimal representation of numerics.

   Coded Character Sets, History and Development, PDF page 257, logical page 235.

   So far, your proposal is consistent with the standard. It would place 0 at XXX0000, 9 at XXX1001, A at XXX1010, and F at XXX1111.

3. Another decision of the committee was that the alphabetic characters all had to be contiguous:

   Criterion 10. The alphabetics should have contiguous bit patterns.

   ibid

   Considering your proposal, A through F are already contiguous. G would have to continue in a new group of 16, (XXX+1)0000. V would be (XXX+1)1111. W would continue into a third group of 16, (XXX+2)0000. Z would be (XXX+2)0011.

4. Where your proposal fails is the committee's requirement that all alphabetic characters must fit into 5 bits:

   Criterion 9: The alphabetics A through Z, and some code positions contiguous to the code position of Z, should be contained in a 5 bit subset.

   ibid

   At the time of the standard's creation, every bit was precious. Packing customer names into 5-bit fields instead of 6 could significantly shrink the size of a business database.

   As we previously noted, your proposal places A at XXX1010 and Z at (XXX+2)0011. Such an encoding overlaps a 5-bit boundary, and thus does not meet the committee's criteria.

Answer 6

Oldstyle ASR-33 teletype machines (telex machines) only handled 7-bit codes. They only handled uppercase English-language characters, the ten digits, and some punctuation.

They printed with this little cylindrical print head with a limited number of characters available.

Later, tons of terminals, both printing and screen-based, came on the market using the same code.

Lowercase English letters were more or less an afterthought. Various terminal products (not ASR-33s) started using the 8th bit. Making the uppercase and lowercase letters contiguous in the code space would have created a new code incompatible with the old ASR-33 code. So, instead, they made the lowercase letter codes by ORing in the 0x40 bit to the existing codes.

That's why the codes aren't contiguous: backward compatibility with teletypes.

I remember getting my first ADM-3a terminal with 8-bit support. It was awesome for the time. It made UNIX useful. I had to flip some dip switches on the PDP-11 serial port card to support it. Yeah, back then a serial port card was bigger than a Raspberry Pi is now, and drew far more power.

Back in the day of serial ports and modems a big hassle was setting them up right. You needed to know the number of bits in the character, the parity, and the number of stop bits of the other end of the connection. Here's the serial port config screen from Windows 3.1:

Answer 7

The argument you make with resorting to C expression complexity can be recast as follows:

If anyone really wanted to classify the ASCII characters so fast that the machine code output from a competent compiler wasn't enough, they'd be needing an FPGA or an ASIC anyway, and at that point it's irrelevant what the code is, since the most efficient encoding of the code ranges in the FPGA fabric or gate arrays will be nothing like C and vastly more efficient in energy use per each character classified, etc.

IOW: If speed matters that much, you'll be dealing with FPGA or custom silicon, and then nobody cares what it looks like in C. Both FPGAs and silicon can deal with some logic patterns that are terribly inefficient in C, like content-addressable memory, etc.

And, in any case, the way most character classification is done, the contiguity wouldn't help much, since there's a large variety of character classes that are entirely application-dependent, e.g. a lexer or parser might use a character class that finds no use outside of programming language front-ends, and nobody would be arguing to sort ASCII so that lexing C or Python is "cheap".

Furthermore, I'd dare say at this point there's more character classification done on Unicode code points, and most likely those of the Chinese character set / kanji, than ASCII, so there's that too.