Fix Keyboard Input on Terminals - Please

Keyboard input on Terminals has many deficiencies to it. I want them all fixed. I have a plan on how to do it but it Needs Your Help

I want terminal keypresses to Just Work. What do I mean Just Work? I mean programs in a simple orthogonal way, can determine the key that was pressed, and this model should easily map to the user's expectation.

Why doesn't it currently? Currently, there are a number of classes of physically-distinct key presses that terminals encode using the same bytes.

Even when keys produce unique unambiguous byte sequences, it may be that some programs cannot recognise them.

How can we fix this?

By having a sane and sensible model on BOTH ends of the terminal interaction, and a well-defined way of communicating. How exactly we go about this really depends who you are:

Authors of Programs that Read Terminal Keypresses

The simplest way to go about this is to use the key input handling features of my terminal UI library, libtickit. Failing this, if you're using some existing system of keyboard input, such as GNU Readline, I suggest you apply pressure to the maintainers of that system until they adopt this method of reading extended keypresses.

Authors of Terminal Emulators

Making a terminal send the correct key encodings should be a relatively easy task, given the encoding scheme already splits modifiers and keysyms in a way likely to be similar to the underlying input system at work, such as X11 or Win32.

End-users of Programs

Even if you're not the author of a program like this, you can still help fix the situation. Write to the maintainer of your program, report a bug. Explain that you wanted to use such a keybinding, or whatever the problem was. Explain that, because their program doesn't correctly handle these cases, it wasn't possible, but if they were to adopt this scheme, then it would be and your problem would be solved. With enough voices, even the most stubbon developer who believes it's 1970 and everyone lives behind a green-phosphor glass teletype on a 9,600 baud modem, can still change his ways.

Encoding Specification

The primary motivation of this scheme is to encode any possible keypress uniquely; that a keypress maps to one possible sequence of bytes, and a valid sequence of bytes encodes only one keypress. For backward compability, it is also required that any keypress that can be represented without this scheme is also represented by the same bytes within it; that is, this scheme is an extension of existing encodings, not a replacement of.

The two encoding forms used by this specification to encode modifiers both put a number as their second parameter whose value is 1 + a bitmask encoding the modifiers. We add 1 because by convention, a missing CSI parameter is equivalent to 1, so in the "unmodified" state the modifier field would be 1 - we must add 1 to it so that no-bits-set (unmodified) is encoded as 1 (though, in practice, we rarely use the 1 form explicitly).
Alt 2
Ctrl 4
Thus, the various encoding forms using these modifiers will look like:
(direct UTF-8)CSI ...;2 uCSI ...;3 uCSI ...;4 uCSI ...;5 uCSI ...;6 uCSI ...;7 uCSI ...;8 u
CSI ... ~CSI ...;2 ~CSI ...;3 ~CSI ...;4 ~CSI ...;5 ~CSI ...;6 ~CSI ...;7 ~CSI ...;8 ~
CSI XCSI 1;2 XCSI 1;3 XCSI 1;4 XCSI 1;5 XCSI 1;6 XCSI 1;7 XCSI 1;8 X
These various forms are explained more below.
Unicode Characters
When unmodified, just send UTF-8 bytes as normal. No UTF-8 sequence of a regular character starts with a C0 or C1 byte, so these will be unique. The Shift key should not be considered as a modifier for Unicode characters, because it is most likely used to obtain the character in the first place (e.g. the shift key is often required to obtain the ! symbol).
Certain key presses have special behaviours with modifiers. Alt tends to prefix Escape. Ctrl tends to mask with 0x1f. When the simplest form of these keys are pressed, they should be encoded as previously. Existing schemes lack a way to encode more general modifiers with Unicode codepoints at present, so we are at liberty to invent something new for these. The CSI u command lies in the private-use area, so has no fixed meaning at present.
CSI [codepoint];[modifier] u
We can use this to represent the problematic combinations of keys; for example variations on the letter "A" (ASCII decimal value 65) and "a":
without ShiftaESC a0x01ESC 0x01
with ShiftAESC ACSI 65;5 uCSI 65;7 u
This same pattern is continued for the other letters, except for the awkward cases of We can now represent the usual problematic keys unambiguously:
Ctrl-I = CSI 105;5 uCtrl-Shift-I = CSI 73;5 uTab = 0x09
Ctrl-M = CSI 109;5 uCtrl-Shift-M = CSI 77;5 uEnter = 0x0d
Ctrl-[ = CSI 91;5 uCtrl-{ = CSI 123;5 uEscape = 0x1b
Ctrl-@ = CSI 64;5u Ctrl-Space = 0x00
Because other C0 bytes do not have usual alternative names or keypresses, these can continue to be sent using the simple single-byte encoding. This is essential to ensuring that legacy systems continue to interpret them correctly (e.g. termios still sends SIGINT on Ctrl-C). The Space key is unique among the printing Unicode keys in that its behaviour doesn't normally change as a result of the Shift modifier. We can capture this by encoding it using CSIu - uniquely, the only printing Unicode key for which we use this:
Ctrl-Space = 0x00Ctrl-Shift-Space = CSI 32;6 u
Modified C0 controls
The problematic keys listed above plus Backspace can be represented using the CSI u encoding when any modifiers are present. Because these symbols are almost universally found on their own physical keycap, there is no problem with encoding the Shift modifier as well, with the value 1. We encode these keys using their normal ASCII codepoint values.
Enter = 0x0dShift-Enter = CSI 13;2 uCtrl-Enter = CSI 13;5 u...
Escape = 0x1bShift-Escape = CSI 27;2 uCtrl-Escape = CSI 27;5 u
Backspace = 0x7fShift-Backspace = CSI 127;2 uCtrl-Backspace = CSI 127;5 u
The Space key is trickier, because it's traditionally used with the Ctrl modifier to obtain the NUL byte encoding.
Space = 0x20Shift-Space = CSI 32;2 uCtrl-Space = 0x00Ctrl-Shift-Space = CSI 32;6 u
Since most terminals repesent Shift-Tab as CSI Z, that needs handling specially as well.
Tab = 0x09Shift-Tab = CSI ZCtrl-Tab = CSI 9;5 uCtrl-Shift-Tab = CSI 1;5 Z...
Special keys
Existing schemes usually use CSI ~ to encode special (i.e. non-Unicode) keypresses. XTerm and other terminals use the second CSI parameter to pass encoding the set of modifiers in effect.
CSI [number];[modifier] ~
If the modifier value is 1; i.e. there are no modifiers, then for backward-compatibility to send the same bytes as previously, it should be omitted, sending simply
CSI [number] ~
Insert 2
Delete 3
Page Up 5
Page Down 6
Home (*) 7
End (*) 8
F1 (*) 11
F2 (*) 12
F3 (*) 13
F4 (*) 14
F5 15
F6 17 (**)
F7 18
F8 19
F9 20
F10 21
F11 23
F12 24
(*): Note that some terminals prefer to encode Home, End and the F1 to F4 function keys using shorter CSI letter encodings. Parsers or other receiving programs should accept either encoding identically. (**): Note also the discontinuity of numbered function keys. After every group of 5 F keys there is a missing number; so every 6th number is not used by an F key (10, 16, 22, 28,...). This is purely for historic reasons; the numbered F keys on the original VTxx keyboards had physical gaps in these locations.
Really special keypresses
Some terminals encode certain keys outside of the CSI ~ encoding, instead picking a different CSI command. This too can accomodate the modifier in its second position.
CSI 1;[modifier] {ABCDFHPQRS}
Up A
Down B
Left D
End F
Home H
F1 P
F2 Q
F3 R
F4 S
As before, if the modifier is 1 then the modifier parameter should be omitted. Since the leading number is also implied, this can be omitted too.

Advantages: solves above problems. Can be fitted into existing programs with a minimum of fuss and without breaking back-compat.

Extension: We can take it one step further, and require CSI to be sent as the single 8bit code 0x9b and never as 7bit multibyte 0x1b 0x4b (Escape [). This reserves the single 0x1b byte exclusively for the Escape key. If this is known to be in effect, then it is no longer required to use timing information to recover accurate keypress information, as every possible keypress now has a unique encoding. Even if this is not known to be in effect, programs ought to accept the single 0x9b byte encoding of CSI anyway, in anticipation of a time when terminals may send it encoded this way by default.