Fix Keyboard Input on Terminals - Please

I want terminal keypresses to Just Work. What do I mean Just Work? I mean programs in a simple orthogonal way, can determine the key that was pressed, and this model should easily map to the user's expectation.

Why doesn't it currently? Currently, there are a number of classes of physically-distinct key presses that terminals encode using the same bytes.

• Special names for some ASCII characters collide with Ctrl-modified letters. For example, Tab is Ctrl-I.
• The Ctrl modifier encodes lower or upper-case letters identically; Ctrl-I is Ctrl-Shift-I.
• UTF-8 collides with 8bit-high characters; e.g. Alt-C might be encoded as 0xc3, which is the first UTF-8 byte of many Unicode characters, such as é.
• Sometimes programs rely on timing information to distinguish, for example, Alt-C from the two separate keypresses of Escape followed by C. This results in delays to recognise a plain Escape key if the time is too long, or erroneously recognising Escape followed by a letter as an Alt-modified letter if typed too quickly, or if the time limit is too short.

Even when keys produce unique unambiguous byte sequences, it may be that some programs cannot recognise them.

• The way terminals encode modified keypresses is an extension to the original methods used in the 1970s, and some older code cannot parse them correctly. Press Ctrl-Up on a modern xterm and watch a program fail and die in all sorts of interesting ways because they don't have real CSI parsers.

How can we fix this?

By having a sane and sensible model on BOTH ends of the terminal interaction, and a well-defined way of communicating. How exactly we go about this really depends who you are:

Authors of Programs that Read Terminal Keypresses

The simplest way to go about this is to use the key input handling features of my terminal UI library, libtickit. Failing this, if you're using some existing system of keyboard input, such as GNU Readline, I suggest you apply pressure to the maintainers of that system until they adopt this method of reading extended keypresses.

Authors of Terminal Emulators

Making a terminal send the correct key encodings should be a relatively easy task, given the encoding scheme already splits modifiers and keysyms in a way likely to be similar to the underlying input system at work, such as X11 or Win32.

End-users of Programs

Even if you're not the author of a program like this, you can still help fix the situation. Write to the maintainer of your program, report a bug. Explain that you wanted to use such a keybinding, or whatever the problem was. Explain that, because their program doesn't correctly handle these cases, it wasn't possible, but if they were to adopt this scheme, then it would be and your problem would be solved. With enough voices, even the most stubbon developer who believes it's 1970 and everyone lives behind a green-phosphor glass teletype on a 9,600 baud modem, can still change his ways.

Encoding Specification

The primary motivation of this scheme is to encode any possible keypress uniquely; that a keypress maps to one possible sequence of bytes, and a valid sequence of bytes encodes only one keypress. For backward compability, it is also required that any keypress that can be represented without this scheme is also represented by the same bytes within it; that is, this scheme is an extension of existing encodings, not a replacement of.

Modifiers

The two encoding forms used by this specification to encode modifiers both put a number as their second parameter whose value is 1 + a bitmask encoding the modifiers. We add 1 because by convention, a missing CSI parameter is equivalent to 1, so in the "unmodified" state the modifier field would be 1 - we must add 1 to it so that no-bits-set (unmodified) is encoded as 1 (though, in practice, we rarely use the 1 form explicitly).

Shift	1
Alt	2
Ctrl	4

Thus, the various encoding forms using these modifiers will look like:

unmodified	Shift+	Alt+	Alt+Shift+	Ctrl+	Ctrl+Shift+	Ctrl+Alt+	Ctrl+Alt+Shift+
(direct UTF-8)	CSI ...;2 u	CSI ...;3 u	CSI ...;4 u	CSI ...;5 u	CSI ...;6 u	CSI ...;7 u	CSI ...;8 u
CSI ... ~	CSI ...;2 ~	CSI ...;3 ~	CSI ...;4 ~	CSI ...;5 ~	CSI ...;6 ~	CSI ...;7 ~	CSI ...;8 ~
CSI X	CSI 1;2 X	CSI 1;3 X	CSI 1;4 X	CSI 1;5 X	CSI 1;6 X	CSI 1;7 X	CSI 1;8 X

These various forms are explained more below.

Unicode Characters

When unmodified, just send UTF-8 bytes as normal. No UTF-8 sequence of a regular character starts with a C0 or C1 byte, so these will be unique. The Shift key should not be considered as a modifier for Unicode characters, because it is most likely used to obtain the character in the first place (e.g. the shift key is often required to obtain the ! symbol).
Certain key presses have special behaviours with modifiers. Alt tends to prefix Escape. Ctrl tends to mask with 0x1f. When the simplest form of these keys are pressed, they should be encoded as previously. Existing schemes lack a way to encode more general modifiers with Unicode codepoints at present, so we are at liberty to invent something new for these. The CSI u command lies in the private-use area, so has no fixed meaning at present.

CSI [codepoint];[modifier] u

We can use this to represent the problematic combinations of keys; for example variations on the letter "A" (ASCII decimal value 65) and "a":

	unmodified	Alt+	Ctrl+	Ctrl+Alt+
without Shift	a	ESC a	0x01	ESC 0x01
with Shift	A	ESC A	CSI 65;5 u	CSI 65;7 u

This same pattern is continued for the other letters, except for the awkward cases of We can now represent the usual problematic keys unambiguously:

Ctrl-I = CSI 105;5 u	Ctrl-Shift-I = CSI 73;5 u	Tab = 0x09
Ctrl-M = CSI 109;5 u	Ctrl-Shift-M = CSI 77;5 u	Enter = 0x0d
Ctrl-[ = CSI 91;5 u	Ctrl-{ = CSI 123;5 u	Escape = 0x1b
Ctrl-@ = CSI 64;5u		Ctrl-Space = 0x00

Because other C0 bytes do not have usual alternative names or keypresses, these can continue to be sent using the simple single-byte encoding. This is essential to ensuring that legacy systems continue to interpret them correctly (e.g. termios still sends SIGINT on Ctrl-C). The Space key is unique among the printing Unicode keys in that its behaviour doesn't normally change as a result of the Shift modifier. We can capture this by encoding it using CSIu - uniquely, the only printing Unicode key for which we use this:

Ctrl-Space = 0x00

Ctrl-Shift-Space = CSI 32;6 u

Modified C0 controls

The problematic keys listed above plus Backspace can be represented using the CSI u encoding when any modifiers are present. Because these symbols are almost universally found on their own physical keycap, there is no problem with encoding the Shift modifier as well, with the value 1. We encode these keys using their normal ASCII codepoint values.

Enter = 0x0d	Shift-Enter = CSI 13;2 u	Ctrl-Enter = CSI 13;5 u	...
Escape = 0x1b	Shift-Escape = CSI 27;2 u	Ctrl-Escape = CSI 27;5 u
Backspace = 0x7f	Shift-Backspace = CSI 127;2 u	Ctrl-Backspace = CSI 127;5 u

The Space key is trickier, because it's traditionally used with the Ctrl modifier to obtain the NUL byte encoding.

Space = 0x20

Shift-Space = CSI 32;2 u

Ctrl-Space = 0x00

Ctrl-Shift-Space = CSI 32;6 u

Since most terminals repesent Shift-Tab as CSI Z, that needs handling specially as well.

Tab = 0x09

Shift-Tab = CSI Z

Ctrl-Tab = CSI 9;5 u

Ctrl-Shift-Tab = CSI 1;5 Z

...

Special keys

Existing schemes usually use CSI ~ to encode special (i.e. non-Unicode) keypresses. XTerm and other terminals use the second CSI parameter to pass encoding the set of modifiers in effect.

CSI [number];[modifier] ~

If the modifier value is 1; i.e. there are no modifiers, then for backward-compatibility to send the same bytes as previously, it should be omitted, sending simply

CSI [number] ~

Key	Number
Insert	2
Delete	3
Page Up	5
Page Down	6
Home (*)	7
End (*)	8
F1 (*)	11
F2 (*)	12
F3 (*)	13
F4 (*)	14
F5	15
F6	17 (**)
F7	18
F8	19
F9	20
F10	21
F11	23
F12	24

(*): Note that some terminals prefer to encode Home, End and the F1 to F4 function keys using shorter CSI letter encodings. Parsers or other receiving programs should accept either encoding identically. (**): Note also the discontinuity of numbered function keys. After every group of 5 F keys there is a missing number; so every 6th number is not used by an F key (10, 16, 22, 28,...). This is purely for historic reasons; the numbered F keys on the original VTxx keyboards had physical gaps in these locations.

Really special keypresses

Some terminals encode certain keys outside of the CSI ~ encoding, instead picking a different CSI command. This too can accomodate the modifier in its second position.

CSI 1;[modifier] {ABCDFHPQRS}

Key	Code
Up	A
Down	B
Right	C
Left	D
End	F
Home	H
F1	P
F2	Q
F3	R
F4	S

As before, if the modifier is 1 then the modifier parameter should be omitted. Since the leading number is also implied, this can be omitted too.

CSI {ABCDFHPQRS}

Advantages: solves above problems. Can be fitted into existing programs with a minimum of fuss and without breaking back-compat.

Extension: We can take it one step further, and require CSI to be sent as the single 8bit code 0x9b and never as 7bit multibyte 0x1b 0x4b (Escape [). This reserves the single 0x1b byte exclusively for the Escape key. If this is known to be in effect, then it is no longer required to use timing information to recover accurate keypress information, as every possible keypress now has a unique encoding. Even if this is not known to be in effect, programs ought to accept the single 0x9b byte encoding of CSI anyway, in anticipation of a time when terminals may send it encoded this way by default.