LeoNerd.org.uk logo LeoNerd.org.uk LeoNerd.org.uk logo

Fix Keyboard Input on Terminals - Please

Keyboard input on Terminals has many deficiencies to it. I want them all fixed. I have a plan on how to do it but it Needs Your Help

I want terminal keypresses to Just Work. What do I mean Just Work? I mean programs in a simple orthogonal way, can determine the key that was pressed, and this model should easily map to the user's expectation.

Why doesn't it currently? Currently, there are a number of classes of physically-distinct key presses that terminals encode using the same bytes.

Even when keys produce unique unambiguous byte sequences, it may be that some programs cannot recognise them.

How can we fix this?

By having a sane and sensible model on BOTH ends of the terminal interaction, and a well-defined way of communicating. How exactly we go about this really depends who you are:

Authors of Programs that Read Terminal Keypresses

The simplest way to go about this is to use my terminal key input library, libtermkey. Failing this, if you're using some existing system of keyboard input, such as GNU Readline, I suggest you apply pressure to the maintainers of that system until they adopt this method of reading extended keypresses.

Authors of Terminal Emulators

Making a terminal send the correct key encodings should be a relatively easy task, given the encoding scheme already splits modifiers and keysyms in a way likely to be similar to the underlying input system at work, such as X11 or Win32.

End-users of Programs

Even if you're not the author of a program like this, you can still help fix the situation. Write to the maintainer of your program, report a bug. Explain that you wanted to use such a keybinding, or whatever the problem was. Explain that, because their program doesn't correctly handle these cases, it wasn't possible, but if they were to adopt this scheme, then it would be and your problem would be solved. With enough voices, even the most stubbon developer who believes it's 1970 and everyone lives behind a green-phosphor glass teletype on a 9,600 baud modem, can still change his ways.

Encoding Specification

The primary motivation of this scheme is to encode any possible keypress uniquely; that a keypress maps to one possible sequence of bytes, and a valid sequence of bytes encodes only one keypress. For backward compability, it is also required that any keypress that can be represented without this scheme is also represented by the same bytes within it; that is, this scheme is an extension of existing encodings, not a replacement of.

Unmodified Unicode
Just send UTF-8 bytes as normal. No UTF-8 sequence of a regular character starts with a C0 or C1 byte, so these will be unique
Modified Unicode
Certain key presses have special behaviours with modifiers. Alt tends to prefix Escape or set the top bit. Ctrl ends to mask with 0x1f. When the simplest form of these keys are pressed, they should be encoded as previously. Existing schemes lack a way to encode more general modifiers with Unicode codepoints at present, so we are at liberty to invent something new for these.
CSI [codepoint];[modifier] u
The u CSI command lies in the private-use area, so has no fixed meaning at present. The modifier is value 1 + a bitmask encoding the modifiers. 2 for Alt, 4 for Ctrl. Because most keyboard layouts require the use of the Shift modifier to obtain some unicode keypoints (for example Shift-1 to obtain !) we must ignore the value of the Shift modifier here.
We can now represent the usual problematic keys unambiguously:
Ctrl-H = CSI 104;5 uCtrl-Shift-H = CSI 72;5 u
Ctrl-I = CSI 105;5 uCtrl-Shift-I = CSI 73;5 uTab = 0x09
Ctrl-M = CSI 109;5 uCtrl-Shift-M = CSI 77;5 uEnter = 0x0d
Ctrl-[ = CSI 91;5 uCtrl-{ = CSI 123;5 uEscape = 0x1b
Because other C0 bytes do not have usual alternative names or keypresses, these can continue to be sent using the simple single-byte encoding. This is essential to ensuring that legacy systems continue to interpret them correctly (e.g. termios still sends SIGINT on Ctrl-C).
Ctrl-A = 0x01Ctrl-Shift-A = CSI 65;5 u
Ctrl-B = 0x02Ctrl-Shift-B = CSI 66;5 u
...
Modified C0 controls
The three problematic keys listed above, plus Space and Backspace can be represented using the CSI u encoding when any modifiers are present. Because these symbols are almost universally found on their own physical keycap, there is no problem with encoding the Shift modifier as well, with the value 1. We encode these four keys using their normal ASCII codepoint values.
Enter = 0x0dShift-Enter = CSI 13;2 uCtrl-Enter = CSI 13;5 u...
Escape = 0x1bShift-Escape = CSI 27;2 uCtrl-Escape = CSI 27;5 u
Space = 0x20Shift-Space = CSI 32;2 uCtrl-Space = CSI 32;5 u
Backspace = 0x7fShift-Backspace = CSI 127;2 uCtrl-Backspace = CSI 127;5 u
Since most terminals repesent Shift-Tab as CSI Z, that needs handling specially as well.
Tab = 0x09Shift-Tab = CSI ZCtrl-Tab = CSI 9;5 uCtrl-Shift-Tab = CSI 1;5 Z...
Special keys
Existing schemes usually use CSI ~ to encode special (i.e. non-Unicode) keypresses. XTerm and other terminals use the second CSI parameter to pass encoding the set of modifiers in effect.
CSI [number];[modifier] ~
If the modifier value is 1; i.e. there are no modifiers, then for backward-compatibility to send the same bytes as previously, it should be omitted, sending simply
CSI [number] ~
KeyNumber
Insert 2
Delete 3
Page Up 5
Page Down 6
Home (*) 7
End (*) 8
F1 (*) 11
F2 (*) 12
F3 (*) 13
F4 (*) 14
F5 15
F6 17 (**)
F7 18
F8 19
F9 20
F10 21
F11 23
F12 24
(*): Note that some terminals prefer to encode Home, End and the F1 to F4 function keys using shorter CSI letter encodings. Parsers or other receiving programs should accept either encoding identically. (**): Note also the discontinuity of numbered function keys. After every group of 5 F keys there is a missing number; so every 6th number is not used by an F key (10, 16, 22, 28,...). This is purely for historic reasons; the numbered F keys on the original VTxx keyboards had physical gaps in these locations.
Really special keypresses
Some terminals encode certain keys outside of the CSI ~ encoding, instead picking a different CSI command. This too can accomodate the modifier in its second position.
CSI 1;[modifier] {ABCDFHPQRS}
KeyCode
Up A
Down B
RightC
Left D
End F
Home H
F1 P
F2 Q
F3 R
F4 S
As before, if the modifier is 1 then the modifier parameter should be omitted. Since the leading number is also implied, this can be omitted too.
CSI {ABCDFHPQRS}

Advantages: solves above problems. Can be fitted into existing programs with a minimum of fuss and without breaking back-compat.

Extension: We can take it one step further, and require CSI to be sent as the single 8bit code 0x9b and never as 7bit multibyte 0x1b 0x4b (Escape [). This reserves the single 0x1b byte exclusively for the Escape key. If this is known to be in effect, then it is no longer required to use timing information to recover accurate keypress information, as every possible keypress now has a unique encoding. Even if this is not known to be in effect, programs ought to accept the single 0x9b byte encoding of CSI anyway, in anticipation of a time when terminals may send it encoded this way by default.