66

Anyone who has dealt with strings at a low level (e.g., writing a parser in C), knows that doing so tends to involve frequent checks of—either manually, or through isalpha(), isalnum(), etc—whether a character is a case-insensitive alphabetic character...

(*c >= 'A' && *c <= 'Z') || (*c >= 'a' && *c <= 'z')

...or an alphanumeric character...

(*c >= '0' && *c <= '9') || (*c >= 'A' && *c <= 'Z') || (*c >= 'a' && *c <= 'z')

...because the above ranges are non-contiguous in the ASCII order of characters (and many subsequent character encodings such as UTF-8). If only these 3 ranges were adjacent to each other, a much more succinct...

*c >= '0' && *c <= 'z'

...would save countless keystrokes and once-precious clock cycles since the dawn of unix time (or actually a number of years earlier — signed integer).

I guess other considerations were considered more pressing at the time. Does anyone remember what they might have been?

  • 64
    Have you seen EBCDIC? I think we're lucky that ASCII has the alphabet in one contiguous sequence! – Greg Hewgill Jun 26 at 1:22
  • 15
    If you're writing in C, you should probably be using isalnum(). And if you're implementing isalnum() on all but tiny-memory systems, you'd do well to use a lookup table -- so the call just turns into something like return tbl[ch] & (ISALPHA|ISDIGIT). – another-dave Jun 26 at 12:09
  • 7
    How would you think localization into this? – Thorbjørn Ravn Andersen Jun 26 at 15:05
  • 1
    @GregHewgill recently had to convert something from EBCDIC to ASCII at work (dd if=blah.ebcdic conv=ascii > blah.txt for the unfortunate like me) and noticed those funky things about it as well. I've asked a similar question hoping to get some good insight on it. – Captain Man Jun 26 at 16:08
  • 1
    @another-dave and now you’ve made the original question irrelevant. – Thorbjørn Ravn Andersen Jun 27 at 7:27
114

Why is ASCII this way?

First of all, there is no one best sorting order for everything. For example, should UPPER or lower case be first? Should numbers be before or after letters? Too many choices, and no way to please everyone. So they came up with specific pieces that "made sense":

  • Numerals

0x30–0x39 - Easy bit mask to get your integer value.

  • UPPER case letters

0x41–0x5A - Another easy bit mask to get your letter relative value. They could have started with 0x40, or put the space at the beginning. They ended up putting space at the beginning of all printable characters (0x20), which makes a lot of sense. So we ended up with @ at 0x40 - no specific logic to that particular character that I know of, but having something there and starting the letters at 0x41 makes sense to me for the times you need a placeholder of some sort to mark "right before the letters".

  • lower case letters

0x61–0x7a - Again a simple bit mask to get the letter relative value. Plus, if you want to turn UPPER into lower or vice versa, just flip one bit.

  • Control codes

These could have been anywhere. But placing them at the beginning has the nice advantage that extensions to the character set - from 128 to 256 and beyond - can treat everything >= 0x20 as printable.

  • Everything else

Everything else got filled in - 0x21–0x2f, 0x3a–0x3f, 0x5b–0x5f, 0x7b–0x7f. "Matching" characters generally next to each other, like ( and ), or with one character separating them, like < = > and [ \ ]. Most of the more universally used characters are earlier in the character set. The last character - 0x7f (delete or rubout) is another special case because it has all 7 bits set - see Delete character for all the gory details.

Why is C (and most other languages) this way?

A high-level language should be designed to be machine independent, or rather to make it possible to implement on different architectures with minimal changes. A language may be far more efficient on one architecture than another, but it should be possible to implement reasonably well on different architectures. A common example of this is Endianness, but another example is character sets. Most character sets - yes, even EBCDIC, do some logical grouping of letters and numbers. In the case of EBCDIC, the letters are in sequence, but lower case is before UPPER case and each alphabet is split into 3 chunks. So isalpha(), isalnum() and similar functions play a vital role. If you use (*c >= '0' && *c <= '9') || (*c >= 'A' && *c <= 'Z') || (*c >= 'a' && *c <= 'z') on an ASCII system it will be correct, but on an EBCDIC system it will not be correct - it will have quite a few False Positives. And while *c >= '0' && *c <= 'z' would have lots of False Positives in ASCII, it will totally fail in EBCDIC.

Arguably, a "perfect text sorting character set" could be created that would fit your "ideal", but it would inevitably be less-than-ideal for some other use. Every character set is a compromise.

  • 10
    @Raffzahn Of course numbers come before letters. With greater-than-10 radii it's a must, everywhere else it's just convent. – Agent_L Jun 26 at 10:49
  • 4
    @Agent_L More than that, they should come immediately before the letters, making hex-conversion (and, in fact, all the way up to hexatridecimal conversion) much easier! – TripeHound Jun 26 at 13:11
  • 7
    A minor addition: C, C++, Java, and probably a bunch more languages require the codes for '0'..'9' to be contiguous and increasing, so identifying the digits 0..9 with *c >= '0' && *c <= '9' always works in those languages. – Pete Becker Jun 26 at 14:03
  • 6
    Amusingly, the [\] sequence has accidentally (mostly) saved people using regexes to filter out punctuation. I've seen people in Python use '['+string.punctuation+']' to construct a regex character class to find/replace punctuation, and it mostly works, because the backslash preceding ] escapes it, preventing it from being seen as the end of the character class (it just fails to find the backslash itself, which is a bug, but a much less noticeable one). – ShadowRanger Jun 26 at 16:11
  • 7
    ... just flip one bit ... many of the older hardware terminals worked like that, just connect the A-Z keys to bits 0-4, and the shift key to bit 5. – Guntram Blohm supports Monica Jun 26 at 17:34
59

According to ASA X3.4-1963 Appendix A, one of the design considerations was:

(7) Ease in the identification of classes of characters

Furthermore:

A4.4 The character set was structured to enable the easy identification of classes of graphics and controls.

And on page 8:

A6.3 To simplify the design of typewriter-like devices, it is desirable that there be only a common 1-bit difference between characters normally paired on keytops. This, together with the requirement for a contiguous alphabet, the collating requirements outlined above, and international considerations, resulted in the placement of the alphabet in the last two columns of the graphic subset. This left the second column of the graphic subset for the numerals.

There is a considerable amount of other information about the structure of ASCII in that appendix.

  • 3
    Teletypes were typewriter like devices, and they began using ASCII before computers did. – Walter Mitty Jun 26 at 10:38
  • 3
    +1 This, as Wikipedia says: Locating the lowercase letters in sticks 6 and 7 caused the characters to differ in bit pattern from the upper case by a single bit, which simplified case-insensitive character matching and the construction of keyboards and printers. – rexkogitans Jun 26 at 11:57
  • 5
    IOW, the X3.4 people knew what they were doing. – RonJohn Jun 26 at 14:14
  • 1
    @RonJohn: Indeed. And it's so rare that we can answer a "Why (did|didn't) they do X?" question with such a succinct quote from a document written over 50 years ago. – Greg Hewgill Jun 26 at 18:12
  • 4
    On the contrary, it's my experience that most of "Why did they design this old piece of crap like this??" (and I've seen a lot, not just on RC) have very good reasons easily discoverable on the Intarweb (though you might not know the right Google Fu). – RonJohn Jun 26 at 18:22
27

man 7 ascii of Linux Programmer's Manual says,

Uppercase and lowercase characters differ by just one bit and the ASCII character 2 differs from the double quote by just one bit, too. That made it much easier to encode characters mechanically or with a non microcontroller-based electronic keyboard and that pairing was found on old teletypes.

As supplement information, Eric S. Raymond authored Things Every Hacker Once Knew a few years ago. It has a section on the purposes of various designs in ASCII. Reading it is strongly recommended.

ASCII, the American Standard Code for Information Interchange, evolved in the early 1960s out of a family of character codes used on teletypes.

ASCII, unlike a lot of other early character encodings, is likely to live forever - because by design the low 127 code points of Unicode are ASCII. If you know what UTF-8 is (and you should) every ASCII file is correct UTF-8 as well.

The following table describes ASCII-1967, the version in use today. This is the 16x4 format given in most references.

Dec Hex    Dec Hex    Dec Hex  Dec Hex  Dec Hex  Dec Hex   Dec Hex   Dec Hex
  0 00 NUL  16 10 DLE  32 20    48 30 0  64 40 @  80 50 P   96 60 `  112 70 p
  1 01 SOH  17 11 DC1  33 21 !  49 31 1  65 41 A  81 51 Q   97 61 a  113 71 q
  2 02 STX  18 12 DC2  34 22 "  50 32 2  66 42 B  82 52 R   98 62 b  114 72 r
  3 03 ETX  19 13 DC3  35 23 #  51 33 3  67 43 C  83 53 S   99 63 c  115 73 s
  4 04 EOT  20 14 DC4  36 24 $  52 34 4  68 44 D  84 54 T  100 64 d  116 74 t
  5 05 ENQ  21 15 NAK  37 25 %  53 35 5  69 45 E  85 55 U  101 65 e  117 75 u
  6 06 ACK  22 16 SYN  38 26 &  54 36 6  70 46 F  86 56 V  102 66 f  118 76 v
  7 07 BEL  23 17 ETB  39 27 '  55 37 7  71 47 G  87 57 W  103 67 g  119 77 w
  8 08 BS   24 18 CAN  40 28 (  56 38 8  72 48 H  88 58 X  104 68 h  120 78 x
  9 09 HT   25 19 EM   41 29 )  57 39 9  73 49 I  89 59 Y  105 69 i  121 79 y
 10 0A LF   26 1A SUB  42 2A *  58 3A :  74 4A J  90 5A Z  106 6A j  122 7A z
 11 0B VT   27 1B ESC  43 2B +  59 3B ;  75 4B K  91 5B [  107 6B k  123 7B {
 12 0C FF   28 1C FS   44 2C ,  60 3C <  76 4C L  92 5C \  108 6C l  124 7C |
 13 0D CR   29 1D GS   45 2D -  61 3D =  77 4D M  93 5D ]  109 6D m  125 7D }
 14 0E SO   30 1E RS   46 2E .  62 3E >  78 4E N  94 5E ^  110 6E n  126 7E ~
 15 0F SI   31 1F US   47 2F /  63 3F ?  79 4F O  95 5F _  111 6F o  127 7F DEL

However, this format - less used because the shape is inconvenient - probably does more to explain the encoding:

   0000000 NUL    0100000      1000000 @    1100000 `
   0000001 SOH    0100001 !    1000001 A    1100001 a
   0000010 STX    0100010 "    1000010 B    1100010 b
   0000011 ETX    0100011 #    1000011 C    1100011 c
   0000100 EOT    0100100 $    1000100 D    1100100 d
   0000101 ENQ    0100101 %    1000101 E    1100101 e
   0000110 ACK    0100110 &    1000110 F    1100110 f
   0000111 BEL    0100111 '    1000111 G    1100111 g
   0001000 BS     0101000 (    1001000 H    1101000 h
   0001001 HT     0101001 )    1001001 I    1101001 i
   0001010 LF     0101010 *    1001010 J    1101010 j
   0001011 VT     0101011 +    1001011 K    1101011 k
   0001100 FF     0101100 ,    1001100 L    1101100 l
   0001101 CR     0101101 -    1001101 M    1101101 m
   0001110 SO     0101110 .    1001110 N    1101110 n
   0001111 SI     0101111 /    1001111 O    1101111 o
   0010000 DLE    0110000 0    1010000 P    1110000 p
   0010001 DC1    0110001 1    1010001 Q    1110001 q
   0010010 DC2    0110010 2    1010010 R    1110010 r
   0010011 DC3    0110011 3    1010011 S    1110011 s
   0010100 DC4    0110100 4    1010100 T    1110100 t
   0010101 NAK    0110101 5    1010101 U    1110101 u
   0010110 SYN    0110110 6    1010110 V    1110110 v
   0010111 ETB    0110111 7    1010111 W    1110111 w
   0011000 CAN    0111000 8    1011000 X    1111000 x
   0011001 EM     0111001 9    1011001 Y    1111001 y
   0011010 SUB    0111010 :    1011010 Z    1111010 z
   0011011 ESC    0111011 ;    1011011 [    1111011 {
   0011100 FS     0111100 <    1011100 \    1111100 |
   0011101 GS     0111101 =    1011101 ]    1111101 }
   0011110 RS     0111110 >    1011110 ^    1111110 ~
   0011111 US     0111111 ?    1011111 _    1111111 DEL

Using the second table, it’s easier to understand a couple of things:

  • The Control modifier on your keyboard basically clears the top three bits of whatever character you type, leaving the bottom five and mapping it to the 0..31 range. So, for example, Ctrl-SPACE, Ctrl-@, and Ctrl-` all mean the same thing: NUL.

  • Very old keyboards used to do Shift just by toggling the 32 or 16 bit, depending on the key; this is why the relationship between small and capital letters in ASCII is so regular, and the relationship between numbers and symbols, and some pairs of symbols, is sort of regular if you squint at it. The ASR-33, which was an all-uppercase terminal, even let you generate some punctuation characters it didn’t have keys for by shifting the 16 bit; thus, for example, Shift-K (0x4B) became a [ (0x5B)

It used to be common knowledge that the original 1963 ASCII had been sightly different. It lacked tilde and vertical bar; 5E was an up-arrow rather than a caret, and 5F was a left arrow rather than underscore. Some early adopters (notably DEC) held to the 1963 version.

If you learned your chops after 1990 or so, the mysterious part of this is likely the control characters, code points 0-31. You probably know that C uses NUL as a string terminator. Others, notably LF = Line Feed and HT = Horizontal Tab, show up in plain text. But what about the rest?

Many of these are remnants from teletype protocols that have either been dead for a very long time or, if still live, are completely unknown in computing circles. A few had conventional meanings that were half-forgotten even before Internet times. A very few are still used in binary data protocols today.

Here’s a tour of the meanings these had in older computing, or retain today. If you feel an urge to send me more, remember that the emphasis here is on what was common knowledge back in the day. If I don’t know it now, we probably didn’t generally know it then.

Full text.

16

Hex chart of ASCII characters

This chart (showing the hexadecimal values of ASCII characters) outlines manassehkatz's answer graphically:

  • Numbers are at 0x30 + the value of the number
  • Capital letters are at 0x40 + the value of the letter (A=1, B=2 etc)
  • Lowercase letters are at 0x60 + the value of the letter.
  • thanks, it 's very nice to know that, – Luke Jun 27 at 0:43
6

Oldstyle ASR-33 teletype machines (telex machines) only handled 7-bit codes. They only handled uppercase English-language characters, the ten digits, and some punctuation.

They printed with this little cylindrical print head with a limited number of characters available. ASR-33 print head cylinder

Later, tonnage of terminals, both printing and screen-based, came on the market using the same code.

Lowercase English letters were more or less an afterthought. Various terminal products (not ASR-33s) started using the 8th bit. Making the uppercase and lowercase letters contiguous in the code space would have created a new code incompatible with the old ASR-33 code. So, instead, they made the lowercase letter codes by ORing in the 0x40 bit to the existing codes.

That's why the codes aren't contiguous: backward compatibility with teletypes.

I remember getting my first ADM-3a terminal with 8-bit support. It was awesome for the time. It made UNIX useful. Had to go flip some dip switches on the PDP-11 serial port card to support it. Yeah, back then a serial port card was bigger than a raspberry pi is now, and drew far more power.

enter image description here

Back in the day of serial ports and modems a big hassle was setting them up right. You needed to know the number of bits in the character, the parity, and the number of stop bits of the other end of the connection. Here's the serial port config screen from Windows 3.1.

enter image description here

  • 2
    Did they really started using the 8th bit? The end result was all in 7-bits, and in fact at the time (and for quite a while after), many communication protocols only supported 7 bits with the 8th bit as parity. – manassehkatz-Reinstate Monica Jun 27 at 14:49
  • They sure did start using it. See my edit. – O. Jones Jun 27 at 15:15
  • 1
    My point wasn't about using 8 bits for serial communications in the microcomputer era - that is clearly the case and led to "Code Pages" and "Extended ASCII" and various other ways to use the upper 128 characters of 8-bit bytes used for the (nominally 7 bit) ASCII code. I am questioning whether 8th bit was ever used for lower case on an ASR 33 (or similar device in the 1960s). A Windows 95 port configuration was about 30 years after ASCII and the ASR 33. – manassehkatz-Reinstate Monica Jun 27 at 15:19
  • 2
    @manassehkatz, Totally agree--ASCII was and still is a 7-bit code space--but when you say, "the parity bit" you're mixing protocol levels: There is no parity bit in ASCII. If you're talking about a PC COM port, then the parity bit, when its enabled, exists only in the async-serial protocol that carries the 7-bit ASCII codes over the wire from one UART to another. – Solomon Slow Jun 27 at 15:35
  • 1
    As far as DEL = 7F, see en.wikipedia.org/wiki/Delete_character (already in my answer above) - essentially a useful artifact of punched paper tape. – manassehkatz-Reinstate Monica Jun 27 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.