(By "Teletype" I mean all teletypewriters compatible with the protocol used by the Teletype.)
The Teletype was the dumbest of all dumb terminals, which meant that it was cheap. This means that it was widely spread, and so that computers would be designed to be compatible with it, then other terminals would be designed to be compatible with them, then other computers would be designed to be compatible with them etc. creating a standard.
What is the protocol used by Teletypes?
Aside from the character code (which I'll get to later) about the only "protocol" has to do with character framing, until very late in Teletype machines' timeline.
Quick summary: The signal from a sending teletype to a receiving one is a single line, usually a current loop (two wires), which is always in either "marking" (current flowing) or "spacing" (current not flowing) condition (1 or 0, respectively). An idle line is in "marking". To send a character, the sender puts the line in "spacing" condition for one bit time (this is the "start bit"), then sends the bits of the character least-significant-bit first, then holds the line in "marking" condition for at least two bit times. The character itself is encoded with one of the five-bit codes, likely ITA2, or the six-bit TTS code, or ASCII, depending on the models of the machines. That's it. That's all the "protocol" there is.
Details:
Teletypes used a single "current loop" for their I/O interface. (Machines with RS-232 interfaces came very late in their history.) There are only two electrical terminals. As in a traditional analog telephone (which also has two terminals) the same pair of wires handles data in both directions at the same time—and as with a simple analog voice telephone, if the parties at both ends try to type at the same time, the result is garbled data. (There is no protocol in the machines to resolve this. It's up to the operators to not type over each other.)
In a simple point-to-point teletypewriter circuit, a two-conductor cable runs from one machine to the other, with a battery or other current source somewhere in between. So both machines and the battery are all in series in one loop (just as in a point-to-point telegraph circuit). Current in the loop is either flowing (which is called the "marking condition", binary 1) or not ("spacing condition", binary 0; also called "breaking condition"). The current level was generally 60 milliamps in longer distance service, but 20 mA in later machines connected directly to a computer. An idle line, with machines properly connected, is always in the "marking" condition: Current is flowing in the loop. Interrupting the loop, anywhere in the loop, causes current to stop flowing everywhere in the loop—as with any pure series circuit.
Aside: The terms "marking" and "spacing" came from Morse code telegraphy. Originally the inventors did not anticipate that Morse operators would learn the "sound of Morse", or that that would be reliable. Instead, the receiver side of Morse circuits was a "pen recorder". This involved a quill pen point hung over a paper disc which was continuously rotated by a clockwork mechanism—or a length of paper tape, also advanced by clockwork. When the sender's key was pressed, completing the circuit, current flowed, and a solenoid in the receiver caused the pen to touch the paper, making a "mark". When the sender's key was not pressed, current stopped flowing and spring tension withdrew the pen from the paper, leaving a "space" as the paper continued to move under it. Hence marks (current is flowing) and spaces (current is not flowing). The marks, which looked like "dots" and "dashes", on the paper could then be "decoded" at leisure. But Morse operators had to know the code to send it, and they soon learned to read the short and long clicks made by the pen recorder as dits and dahs and were able to write down what they heard as it came in without referring to the pen record. The paper disc did have the advantage that it could be saved as a permanent record, but eventually, for some circuits, the whole pen mechanism was discarded in favor of a much simpler electromagnetic "sounder". The terms "marking" and "spacing" persisted into teletypewriter usage as they used the same sorts of circuits. Just like in a teletypewriter current-loop circuit, in a Morse circuit your receiver has to maintain "marking condition" (by closing the "shorting bar" on their key) or you won't be able to send them anything.
Further aside: Such "pen recorders" were also used for monitoring dialed telephone lines, to see what number the subscriber was calling.
The keyboard sends a character by interrupting the current in a specific pattern. These interruptions in turn inform the printer to print. The keyboard and printer in each machine were connected in series, and in series with the keyboard and printer in the machine at the other end. So typing a key causes printing on both the local and the remote machine. (This used to be called "local echo".)
There are no headers or wrappers on messages, no checksums, no addresses, ... nothing. If you're expecting a "protocol" like IP or something, forget it. You type a key and the character is printed by both machines. That's it.
Heck, they didn't even think about parity bits until the 8-bit machines came along in the early 70s (model 33, etc.). And even then, if a received character doesn't pass parity checks, there was no provision for asking the sender to send again, or anything like that. However there is much to say about ...
It is easy to imagine that the line is changed back and forth from one state to the other to encode a succession of bits, and that some number of such bits would comprise a character. Not terribly unlike Morse code, really. But "reading" Morse, identifying the boundaries of characters and words, was not easy enough to be done mechanically when teletypes were first invented in the 1900s (by which I mean the first decade of the 20th C. - yes, really that long ago!). The bits comprising each character have to be bounded, or "framed", in some way so that an electromechanical receiver can easily and reliably recognize when each new character begins and ends.
Enter "start bits" and "stop bits". Character framing is done by, first, agreeing on a fixed number of bits and bit times per character. (Unlike Morse code, in which different characters have different durations.) And second, by wrapping the bits of each character in a leading "start bit" and a trailing "stop bit". Or two.
When a key is struck, the keyboard starts a cycle during which it interrupts the current in a precisely timed manner. First, the line goes from marking (current flowing) to spacing (no current) condition for one bit time. This is the "start bit". It conveys no information; it's just for timing. The transition from marking to spacing state alerts the receiver that a character follows.
The start bit is followed by the bits that encode the character for the key, 1 = marking, 0 = spacing, least significant bit first. The original teletype machines used a five-bit code. This eventually progressed to eight-bit codes—details later. What is important here is that the number of data bits doesn't change.
Then the line is held at "marking condition" (current flowing) for a minimum of two bit times. In modern parlance we refer to the latter as "two stop bits".
After the required two bit times' worth of "marking" condition, ie the stop bits, the line can continue to be held at marking condition forever. That is, after all, the "idle state" of the line. The marking-to-spacing condition that signals the start bit of the next character can happen at any time after the required two stop bits. In particular, it does not have to happen at any integral number of bit times. This is one reason this is called asynchronous serial communication.
When the line first goes from marking to spacing condition this indicates to a receiver (printer) that a new character has begun to arrive. The receiver starts a cycle during which it reads the state of the line approximately half way through each bit period. In the electromechanical machines this is done by what is called a "distributor", not unlike the car part of the same name. It is basically a rotary selector switch driven by a motor. It starts rotating when the start bit is received, and during the ensuing bit times, it routes the incoming "1 or 0" to individual wires.
The current on these wires change the position of "stops" or "bails", one per data bit, that will cause the printer, when cycled, to print the appropriate character.
After this is done for the last of the data bits and the line goes to "marking" condition with the first of the stop bits, the printer's main cycle clutch is engaged and the printer begins to print the selected character. It's the detection of the first of the stop bits, the "marking" condition during the bit that follows the last of the data bits, that triggers printing.
A print cycle takes just about one full character time, so if the next character follows soon enough after, that main clutch never disengages. This is good - it wears the clutch less. The printer is designed so that once a print cycle begins the bails can be positioned for the next character as the next character is in the process of arriving.
If a line is disconnected or otherwise goes "open", the result is a never-ending "spacing" condition, which results in a "free running" condition in the receiver: the distributor is running but (because no stop bits are ever seen) the printer's clutch is never engaged, so it never starts a print cycle. (This makes a distinctive sound, much more quiet than printing, as the only thing turning is the distributor.) Later Teletypes have a key marked "break" that simply opens the line, causing this condition. This allowed operators to get the attention of the person at the other end. (It also interrupted communication for as long as the "break" key was depressed.)
In switched dialup teletype networks like TWX the switching was all done externally to the teletype machines themselves, and was essentially a much smaller version of the public switched telephone network (PSTN). These networks were commonly used to send messages, not unlike email messages. But there was no "protocol" enforced by the equipment for framing messages, for message headers like "To" and "from" info, etc. The various messaging services that used this equipment each had their own conventions for this sort of information, once two machines were connected. And within each service, actual practice often varied significantly from one operator to the next.
Now, about the character code: A couple of people have mentioned Baudot code, which was indeed the first code used with teleprinters. But you might not recognize the "sending unit" that used Baudot code! It had no typewriter-like keyboard; it was manually operated with five keys for the five bits, so the operator had to know the code. The code was designed to be relatively easy to learn, with the most-often-used characters needing the fewest keys depressed.
Five bits permitted only 32 different characters, so the code included "letters" and "figures" codes that put the receiver into two different "shift" modes. For example, a character with only the "1" bit set would be an "A" in letters mode, but a "1" in figures mode. This "figures vs letters shift" idea has persisted throughout all subsequent five-bit teletypewriter codes (and the six-bit TTS code). The "Figures" set included some special characters (period, comma, etc.) as well as digits, and both sets included space, carriage return, bell, NULL (all zeroes), etc. So there is no room in the code set for the lowercase alphabet.
Although it is common to refer to five-bit teletypes as using "Baudot" code, the actual code invented by Émile Baudot was soon replaced. Very little actual Baudot-code equipment survives today outside of museums.
Later five-bit codes were the Murray code (pretty much a complete redesign of the code set, intended to make a typewriter-style keyboard easier to make and more reliable), the Western Union code (which was basically the Murray code with a couple of minor changes), and the ITA2 code, introduced in 1924. All of these were five-bit codes and all included the "figures and letters shift" codes.
ITA2 with minor variations became very widely used until 7-bit ASCII arrived in about 1963. By "widely used" I mean by networks like Telex and TWX. This code was also used on commercial (and amateur!) radioteletype (RTTY) links.
Any teletype that uses a five-bit code is called a "five-level" machine, "levels" referring to the five possible positions of holes (excluding the feed holes) across the width of punched paper tape. Another characteristic of all of them (I think) is that their keyboards have only three rows of keys (instead of four as was common on typewriters, and on the later ASCII teletypewriters). This was because the numeric digits ("figures") and other special characters did not have their own codes; they were shifted off of the letter keys, so they did not have their own "row" of keys either.
Accordingly, in place of a typewriter's "shift" keys the five-level machines have keys labeled "figs" and "ltrs". These sent their own characters out of the 32 possible; they did not modify the codes sent by other keys, as would be the case with the "Shift" keys on an ASCII machine. The relationships between letters and digits were chosen so that the standard QWERTY keyboard layout had digits 1 through 0, in order, on the QWERTY row.
Here is a picture of a three-row "Baudot" or five-level machine, an ASR32. (Yes, "thirty-two", not 33. The ASR33 was the ASCII equivalent.) If you enlarge the picture you can clearly see the "Figs" and "Ltrs" keys.
(Aside: Today it would be trivial to build a keyboard with a layout nearly identical to that of a modern computer keyboard (except for some missing special characters) but which would send ITA2 or other five-level codes. This may have been done during the last years of five-level codes, so it's possible that a machine with four rows of keys might be a "five-level" machine. Certainly a computer program that sends a five-level code can use a standard modern four-row keyboard, sending the "Figs" and "Ltrs" codes automatically as needed.)
There is very little "five-level" equipment in operational service today, outside of a tiny fraction of ham operators who operate "RTTY" (radio teletype) and enjoy running the authentic machines as part of their hobby. But even among RTTY fans there are probably far more hams now sending and receiving the five-level codes with their computer than with the old electromechanical Teletypes. Although they were amazingly rugged and reliable, time does take its toll; the machines are old, require fairly frequent cleaning and oiling to stay reliable, and of course replacement parts can only be had by scavenging from another machine, or by machining them from scratch.
In a time period overlapping much of the usage of the "five-level" teletypes there was also teletype equipment using a 6-bit code, the TTS - "TeleTypesetting" - code. These machines looked much like their five-bit equivalents and someone not familiar with them probably wouldn't notice the difference. But a close look would reveal that they were printing both upper and lower case!
The TTS code and the machines that went with them were used by the news "wire services". I've written a lengthy answer about those here. You probably never encountered such a machine unless you worked for a newspaper or a radio or TV station - or a wire service. These circuits and codes continued to be used until the late 70s or possibly the early 80s. There are far fewer operating examples of six-level teletype equipment today than five-level because there is virtually no hobbyist interest in it. (One of the "enthusiast" sites says there is one Teletype Model 20 known to be in working condition.)
Flash forward to 1963 or so, when ASCII entered the Teletype world. The famous Teletype model 33 uses the same concept of start bit, data bits, two stop bits, but at a faster bit rate (110 bits/sec, commonly called "110 baud") and with eight data bits instead of five. Hence "8-level", though that term fell into disuse as these machines went everywhere and largely displaced the five-level machines, so there was no longer a need to distinguish them.
Their keyboards have four rows of keys in what has become known as a "bit-paired" arrangement for the special characters. e.g. 2 shifts to the double-quote mark ", not to @. The "figs" and "ltrs" keys, and the underlying concept, are gone. They do have shift keys but the machines send and print only upper case letters, so the shift keys work only on the number keys and on a handful of special-character-only keys. (I don't know what an ASR33 would do if it received lowercase letters. I believe it prints them as their uppercase equivalents.)
Their keyboard arrangement became known as "bit-paired" because for any given key the shift keys make a single-bit change in the generated character code (though which bit it is differs for different keys.) This was advantageous as the bit coding from these keyboards was still partly done with mechanical linkages, as it had been on the post-Baudot three-row machines.
This is not the case with common computer keyboards of today; they almost always use an arrangement called "typewriter-paired" wherein e.g. 2 shifts to @. This arrangement follows that introduced by IBM with some of their Executive Typewriters, and later with the very successful Selectric. (As noted in one of the comments, many of these "pairings" are locale-dependent; what I describe here is for North American keyboards.)
The model 33 was a light-duty machine with a lot of plastic and metal stampings instead of cast metal, so it was less reliable than its predecessors - but much cheaper. The very similar-looking model 32 was a five-level (so-called "Baudot") machine with a traditional three-row keyboard, intended to replace the older five-level machines in messaging systems. There was also the model 35, which like the 33 was an ASCII machine but built more along the lines of the older, more rugged five- and six-level teletypes.
The ASR33 (automatic send receive) model included a paper tape reader and punch. An operator could prepare a paper tape offline and send it to a receiving station later. The paper tape punch could also make a tape from incoming data.
The ability to read and write paper tape (although slowly, at only 10 char/sec), as well as provide keyboard input and printed output, made the ASR33 a mainstay in the early days of microcomputers.
This machine may have been the first teletypewriter to implement a form of flow control: The ASCII control codes DC1 and DC3 (device control 1 and device control 3, typed as ctrl-Q and ctrl-S respectively) were adopted as XON and XOFF. Sending an XOFF would cause the machine at the other end to stop its paper tape reader (if it was running) and sending an XON would turn it back on again. Thus an operator could pause a lengthy transmission if, for example, it was noticed that the local machine was about to run out of paper. Similarly, DC2 and DC4 would turn the remote machine's paper tape punch on and off.
XON and XOFF were soon adopted by operating system developers for software flow control: A terminal operator could pause a lengthy output from the computer by typing control-S and then resume it via ctrl-Q. Serial-interfaced printers and terminals evolved to the point where they sometimes had their own reasons to exert flow control over the host computer, and XON/XOFF flow control became a de facto standard.
Eventually Teletype Corporation introduced the model 37, which was basically a model 33 extended to the full ASCII-7 character set, including both lower and upper case alphabets. But it was hardly "letter quality print" and it still only ran at 15 characters per second - faster than the ASR33, but not by much. Soon these were all largely displaced by printing terminals using "daisy wheel" and dot matrix mechanisms, at 30 char/s and sometimes faster, and by video display terminals for applications that didn't need hardcopy.
Since the "eight-level" machines only support a 128-character ASCII code set (at most) they really only make use of seven of the data bits. The eighth data bit might be sent always 0, always 1, or with even or odd parity, depending on the customer's choice.
As with the five- and six-row machines, a start bit that is not followed by a stop bit after the expected number of data bits is regarded as a "framing error", which can be used to signal an interrupt of sorts to the other end. In other words the "break key" on a terminal keyboard does not send any ordinary "character"; you will not find "break" in the ASCII code set. Instead it causes what is better thought of as a line condition.
Many operating systems treated a framing error as a signal to "interrupt current program". DEC's OSs tended to stick with ctrl-C.
If a teletype is converted to RS232 (really "TIA-232-F" these days) the keyboard and printer are electrically separated. The "transmitted data" pin of the RS232 connector carries the data sent from the keyboard and the "received data" pin sends data to the printer; these two "sides" of the machine can operate completely independently. In RS232 a negative voltage indicates "mark" (a "1" bit on the data circuits, "deasserted" on the control circuits) and positive indicates "space". So an idle line has negative voltage on it.
What is possibly unexpected about this is that - unlike the case with current loop interfaces - if you just cross-connect two such teletypes to each other, hitting a key on one causes printing on the other one, but not on the local printer! Must such teletypes were connected to simple modems and most such modems had a button labeled "half duplex", which really should have been labeled "local print", which causes the characters typed locally to be sent to the local printer as well as to the other end.
Alternately, computers to which these machines were connected could echo the received character back to the teletype. If the computer was set to do this and your modem was set to half duplex you would often see "doubled" characters when you typed things. Many computer systems made some use of the fact that the user would see nothing if the computer didn't echo their typing back - for example, they would turn off the echoing while you were typing your password.
If the only purpose of the stop bit(s) was character framing on the line, one stop bit would really have been enough. Two were used with teletypes to give the mechanical printer enough time to complete a cycle, especially for lengthy operations like "carriage return". (Carriage returns actually got an entire extra character time to complete because the CR was always followed by a line feed (LF).) When non-mechanical devices like CRT terminals that could run at much higher speeds appeared, designers realized that these did not need such an accommodation so a single stop bit became the norm on higher-speed links.
p.s.: Technically, only machines made by the Teletype Corporation should be called "Teletypes". Many other companies made compatible equipment and generically they are all called "teletypewriters", or "teleprinters" if they are receive-only.
Lots more information at baudot.net .
Per this site about using one with Linux:
It'll simply output every character received, and transmit every key pressed.
Teletypes use the five bit Baudot code. Although the machines themselves are obsolete, Baudot code still lives on in the form of RTTY which is still used by radio amateurs.
When I was a lot younger, I used to own an teletype - which worked as a computer printer! I bought a surplus teletype from the local GPO telephone exchange. (GPO - United Kingdom Post Office). This thing took two people to carry it - it was industrially built and had a huge metal base. It had a hard coded identifier which as I remember might have been a hotel. When I got it home I hooked it up as a printer on a Tandy TRS-80. One of the magazines had produced the schematic to do this. Take the cassette output line, hook it up to a relay. the relay swings the current running through a HUGE transformer, which goes into the teletype. If your code switches the relay on and off at exactly the correct rate for a Baudot signal the teletype prints!
This may have been one of the few printers ever on a TRS80 Model 1
Lots of good information in these answers; very interesting stuff. However, you asked what protocol the Teletype machines used and I haven't seen anyone give the actual answer yet. The protocol that Teletype machines used was simply called TTY protocol. Here's the definition from PCmag.com's encyclopedia of term:
TTY protocol (TeleTYpewriter protocol): A low-speed asynchronous communications protocol with limited or no error checking.
So, to answer your question, Teletype machines used the TTY (TeleTYpe) protocol. I used to run a BBS which you called using a terminal program that could emulate all sorts of terminals. To transmit the text of the regular board menus and whatnot, ANSI/TTY was always used. When you would go to transfer a file, upload or download, it was necessary to then use a DATA protocol, such as xmodem, ymodem or zmodem (or HS-Link!) instead of a TEXT protocol such as TTY. If you read Rick Brant's awesome description, under the "sending" heading he literally describes the TTY protocol. Does this clear up any of the confusion? Hope it helps :)
Also, there was nothing really "compatable" with the TTY protocol, per se. Teletype machines did one thing- when you typed a character, it caused the other teletype to print the character you typed. Much like how when you type a character on a computer keyboard, then that character is printed on your screen. Later, terminals would emulate TTY in order to send character text. Later still, when computers were connected to each other through a modem connection, you would dial the other with a terminal program that emulated the many types of terminals (ANSI,VT100, etc) which themselves were emulating a teletype machine.
btoa could deal with a few variations of Base-95. It's good to know the name of the TTY protocol.