15

While debugging uart on some microcontroller board, I've made a simple firmware that simply responds on every byte received dumping that byte in HEX like "0x**". Then, using different terminal emulators in linux, I've found that there are many different escape encodings for the following keyboard keys:

for F1..F5:

  • "\033[["+'A'..'E' in linux text console
  • "\033O"+'P','Q','R','S' (for F1..F4 only) in different gui terminals like mate-terminal and xterm, also running from screen or tmux
  • "\033[11~".."\033[15~" in putty (run in windows XP)
  • "\033[15~" for F5 only in different gui terminals (mate-terminal, xterm), also in screen or tmux
  • "\033[16~" for F5 when using minicom in linux (that seems to recode "\033[[E" and "\033[15~" into that sequence)

for BACKSPACE:

  • 0x08 or 0x7F

for HOME/END:

  • "\033[1~", "\033[4~" in text console, putty and running from screen or tmux
  • "\033O"+'H','F' in mate-terminal
  • "\033[H","\033[F" in xterm
  • "\033[1~", "\033O"+'F' recoded by minicom from gui terminals

When not explicitly noted, I was using either picocom or socat to interact with /dev/ttyUSB0. Those tools seem not to recode any codes generated by terminal emulator, unlike the minicom.

So the questions are:

  • What historic computer systems are inherited in every listed encoding for that keyboard keys?
  • What are the historic reasons (if any) for the minicom to change F5 code as well as recode HOME/END?
4
  • 6
    The short answer is, for the most part the varied encodings are simply due to the lack of an available standard, so each manufacturer came up with their own. Eventually the ANSI standard was defined but by then there were already many systems already in place using the older encodings.
    – Ken Gober
    Feb 23, 2019 at 18:34
  • 7
    The difference between backspace (0x08) and delete (0x7f) is a special case; they originally meant different things. Backspace was a typewriter/printer carriage control character and moved the printhead one space to the left, which was useful if you wanted to overprint one character on top of another. Delete was used to 'overpunch' a character on paper tape and the convention was to ignore them when reading a tape, this was a convenient way to 'rub out' a typo on a paper tape without having to throw away the whole tape and start over (which is why 0x7f is also known as 'rubout').
    – Ken Gober
    Feb 23, 2019 at 18:36
  • 1
    @KenGober - further to your explanation. I suspect that if your keyboard comes from a paper-tape lineage, it will have a "rubout/delete" key. If your keyboard comes from a typewriter tradition, it will have "backspace". In the latter case, and of course on "glass ttys", one can understand why the tty driver writers wanted to treat backspace as erase-last-character. The PC keyboard falls into this category inasmuch as the most convenient key is labelled "backspace". (By upbringing, I fall into the papertape camp on this one). Feb 24, 2019 at 3:30
  • 1
    This is why one should always take comment-answers with a large pinch of salt. The control sequences in the question do not result from multiple manufacturers or a lack of standardization. They pretty much all come from one manufacturer, and are quite regular.
    – JdeBP
    Feb 26, 2019 at 17:17

3 Answers 3

19

It's one single source, not multiple vendors.

Nearly all of what you mention actually comes from one source, DEC VTs. The various different control sequences generated by the function, cursor, editing, and calculator keypads that you mention pretty much all, with a couple of exceptions, have their roots in DEC VTs.

A DEC VT had a slightly different (native layout) keyboard to the IBM PC/AT keyboard descendent that you are likely looking at. Most notably it had the PF1 to PF4 keys on the calculator keypad, had the Find and Select keys on the editing keypad, had 20 function keys rather than 12, and lacked the concept of ⇭ Num Lock.

This is a VT320 keyboard

Rather, it had the concept of modes. The cursor keypad and the calculator keypad could be operated in two modes, application and normal modes. The keys on these keypads would send different control sequences to the host according to the current mode, which was controlled by a control sequence (a different one for each keypad) sent from the host to the terminal.

The DEC VT system is actually fairly logical, although this has to be teased out of the docs slightly. A terminal keyboard basically generates two sorts of control sequence: a variable length one beginning with the Control Sequence Introducer (CSI) character and a two-character sequence beginning with the Shift State 3 (SS3) character.

(You have to know, at this point, that these C1 control characters are the underlying control characters that are involved. The [, O, and N sequences, that you may have thought were the control codes, are in fact what ECMA-48 terms 7-bit code extensions. They are ways of encoding the C1 control characters for transmission paths that are not 8-bit clean. All C1 characters have a 7-bit encoding. Ironically, given how long these 7-bit encodings have lived on, terminal I/O has been 8-bit clean for decades now.)

There are a lot of CSI control sequences, varying according to what intermediate characters, parameters, and final characters they have. ECMA-48 documents them, their standard structure, and provides a way for vendors to have private-use control sequences. ECMA-48 even defines a specific control sequence for encoding function keys, FNK.

DEC VTs do not use FNK.

DECFNK and application mode

Instead, they use DECFNK. It's one of those private-use extensions, a CSI control sequence that has a final character, ~, that is in the range reserved by ECMA-48 for private-use control sequences. DECFNK control sequences have two parameters, the first denoting the key number of the main key and the second denoting the ancillary modifier keys (⇧ Shift, ⎈ Control, and ⎇ Alt) that were also pressed at the time.

The numbering of the first parameter seems crazy, with odd gaps, until one looks at a DEC VT keyboard. Then one realizes that the numbering simply follows the physical positions on the keyboard:

  • The editing keypad comes first, proceeding from left to right, from Find as 1 to Next as 6.
  • The cursor keypad comes next, with the cursor keys numbered from 7 to 10. DECFNK is only used in certain circumstances for these keys, but these are nonetheless their documented numberings.
  • Then comes the function row, with 20 function keys numbered from 11 to 34. This includes the 18 function keys that are labelled as such, and the Help and Do keys that are on that row. The gaps in the numbering simply correspond to actual physical gaps in the function key row.

The calculator keypad is not covered by this because it has no need of such control sequences in normal mode. In that mode, all that it generates are ordinary numeric and punctuation characters, after all.

The cursor and calculator keypads on a real DEC VT can be switched to application modes. The original idea behind this is that mainframe/minicomputer applications connected to DEC terminals could switch the terminal into application mode and thereby give the user what amounted to a huge bunch of extra application-defined function keys.

This is where the two-character sequences come in. The SS3 control character is a single shift, a temporary change of "shift state" to apply to one immediately following character. It is this that is used to encode cursor and calculator keypad keys when in application mode. Again, the system is fairly logical, with the "shifted" character being a different lowercase letter of the alphabet for each key on the keypad, in roughly bottom-to-top order of the physical key positions.

Of course, the editing keypad on an IBM PC/AT descendant keyboard does not have Find, Select, and so forth. It has Insert, Delete, and so forth, in the same positions. Similarly, an IBM PC/AT descendant keyboard has a calculator keypad that has the concept of numeric lock that turns it into a bunch of cursor and editing keys. Then there is the fact that on a DEC VT, the F1 to F5 function keys were (usually) local only, and never generated input to the host. So most terminal emulators have coöpted the codes from the PF1 to PF4 keys on a DEC VT's calculator keypad for those function keys.

These all complicate things somewhat. I won't go into the complexities because they do not affect the core thesis of this answer. The keys still end up generating DEC VT terminal control sequences.

So to apply all these to some concrete examples from the question:

  • DECFNK 1 (CSI 1 ~) is the control sequence generated by the Find key on the editing keypad.
  • DECFNK 17 (CSI 1 7 ~) is the control sequence generated by the F6 function key.
  • DECFNK 15 (CSI 1 5 ~) would be the control sequence generated by the F5 function key, were it not for the fact that on most of the DEC VT range it cannot be taken out of local mode at all.
  • Note that DEC VTs have a gap between F5 and F6. Hence there is no key that generates DECFNK 16. That minicom sends that for F5 is an outright error. But it's an error probably inspired by the fact that the gap is between F4 and F5 on a PC/AT keyboard, and exacerbated by the fact that one has to look quite carefully in the DEC docs to find the code for F5.
  • DECFNK 11 (CSI 1 1 ~) is the control sequence generated by the real F1 function key on a DEC VT, which could be changed out of local mode.
  • SS3 P is the control sequence generated by the PF1 calculator keypad key on a DEC VT, used by terminal emulators (running on PCs) for F1.
  • ⌫ Backspace generating either or DEL is actually a switchable thing on a real DEC VT. There's a DECBKM private-use control sequence that a host can send to control it.
  • CSI A, CSI B, CSI C, and CSI D are Up, Down, Right, and Left when the cursor keypad is in normal mode. These are almost, but not quite, the same as the output control sequences sent by the host that cause cursor motions. (The difference is beyond the scope of this answer, though.)
  • SS3 A, SS3 B, SS3 C, and SS3 D are Up, Down, Right, and Left when the cursor keypad is in application mode.

… and the rest is SCO.

I did say that there were exceptions. These exceptions largely exist in the terminal emulators built into operating system kernels like Linux and the FreeBSD/NetBSD/OpenBSD kernels. They also largely have a common ancestor, though: SCO Multiscreen and the SCO console.

Some of the control sequences that you'll see sometimes from combinations of modifiers and the 12 IBM PC/AT function keys in PC terminal emulators are still the ones from the SCO console. But there is more that just that. On the SCO console (and also in a DEC VT's "SCO Console Emulation" mode):

  • CSI H is ⇱ Home.
  • CSI F is ⇲ End.

There's even more to it than that. SCO Multiscreen used ⎇ Alt+Fn for switching between screens, for example. And UnixWare keyboard map files will look quite familiar to FreeBSD users.

Further reading

4
  • I always wondered why Application mode existed and now I know. A great answer, with so many tantalizing tidbits. Now I want to know what a "real F1 key" is, why 8-bit codes aren't standard, why F5 has a DECFNK reserved for it, and why reserving DECFNK 16 for the gap between the F5 and F6 keys was an error.
    – hackerb9
    Jan 10 at 18:03
  • Oh, wait, I just figured out that the reference to DECFNK 16 being an error was referring to the original question which mentioned that Minicom sends it for F5. I'll edit the answer to make that more clear.
    – hackerb9
    Jan 10 at 18:05
  • 1
    @hackerb9 "real F1 key" = the one that says "F1" on the label. See the earlier note about how some of the DEC F-keys not normally getting passed through to the remote system, so PC app authors tended to map PC F1-F4 to DEC PF1-PF4, because the remote apps would be more likely to have bindings for those.
    – hobbs
    Jan 11 at 16:20
  • @hobbs Thanks. I have a VT340 that I've been learning from, but it doesn't have any "F1" key on it. The key that is in the "F1" location is labelled Hold Screen. I'm guessing the issue is that I have the "Word Processing" ("gold key") version of the keyboard.
    – hackerb9
    Jan 18 at 2:40
7

Partial answer:

General notes: ESC [ is "control sequence introducer", CSI. ESC O is "single-shift-3", SS3.

DEC VT200 terminals sent CSI n~ for function keys F6 onwards, where n = the F-key number plus 11 decimal. F1 to F5 were local only, but it is a reasonable extension to have them send similar sequences with n = 12 to 16. n = 11 to 15 seems like an off-by-one error from this viewpoint.

DEC VT100 terminals sent SS3 P,Q,R,S for four keys labelled PF1 to PF4 on the numeric keypad. Probably VT200 too.

DEC VT100s used CSI 1 to 4 ~ for the so-called editing keypad.

For PC-connected keyboards, of course, the keyboard sends scan codes that are mapped to character sequences by software, so there's plenty of room for latitude here. It is not necessarily true that the PC software was emulating any actual terminal; it just required that the key-mapping software provided something that was potentially useful to application code.

Backspace/delete was a common confusion on terminals of the day. Sometimes a <x]key sent backspace, sometimes it sent delete. Software tended to declare one of those to be the appropriate character-delete operation.

The summary, then, is that many of these came in principle from DEC keyboards, but PC keyboards don't necessarily have the same key layouts, so the terminal emulators do a best-effort (and subjective) job of mapping the keys they need onto the keys they've got.

2
  • I would like to point that it has actually nothing to do with PC keyboards, since in the OS like windows or linux internal representation (scancodes given to the processes which would like to receive events or keyboard state matrix) has nothing to do with what actual keyboard send, be it either PS/2 keyboard, USB keyboard or some custom key matrix that is represented as keyboard by some driver.
    – lvd
    Feb 23, 2019 at 17:08
  • 1
    My point was that a 'terminal' sends characters in some character code (in the VT2xx case, per DEC std 169, a superset of ASCII) whereas a 'PC keyboard' sends codes which must be translated into characters. True, I represented it as a two-stage procedure, whereas it's likely three stage: scan-codes to virtual keycodes to characters, the first in the OS, the second in the terminal emulator. Feb 23, 2019 at 17:17
3

True terminals used to vary a lot in the code sequences they accepted for their control and code sequences they returned when a key was pressed. And keyboards were less standardized than nowadays, so the keys available used also to vary a lot.

Programs and libraries (the best known is probably curses) which had to interact with several kinds of terminals used a database giving that information. The database is still available here if you really want to find out the origin of the code sequences used by today's terminal emulators.

I won't do that research but just add that they tend to emulate the sequence of popular DEC VTxx series (whose structure has been standardized as ECMA-48 -- there is an equivalent ISO standard, ISO/IEC 6429, but the ECMA one has the advantage of being available for free on the web; one sometimes use "ANSI sequences" either to refer to the ECMA-48 one, or as a more generic way to refer to control sequences), but different models in the series sometimes used different sequences. Terminal emulator also sometimes code sequences having no historical sources for features having little historical precedent.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .