[textinput] Support UTF-8 at the prompt - #23085
Conversation
|
Cool! And would fix https://its.cern.ch/jira/browse/ROOT-9562 Maybe also https://root-forum.cern.ch/t/c-20-std-format-fails/ ? |
|
this is remarkable. |
96498a7 to
06fbd6d
Compare
Test Results 23 files 23 suites 3d 18h 22m 38s ⏱️ For more details on these failures, see this check. Results for commit 7c5883e. ♻️ This comment has been updated with latest results. |
|
🎉 Great! Now we are one step closer to being able to count past 255. 🚀 |
06fbd6d to
e110a60
Compare
textinput assumed one line-buffer element is one byte and one terminal column. Both assumptions break on UTF-8, so anything outside ASCII corrupted the buffer on input and misplaced the cursor on output. Text now stores std::u32string, so every index in the library counts characters; the editor's existing index arithmetic (cursor motions, cut/paste, ...) becomes correct unchanged. GetText() still returns UTF-8, cached with a character-to-byte offset table, so Getline's C interface, the interpreter and the history file are unaffected. Since a character occupies 0, 1 or 2 terminal columns, the display now lays out lines by accumulated character width (CharWidth() in the new UTF8.h) instead of index arithmetic, and moves a double-width character that would straddle the margin to the next line. The readers now decode multi-byte input: StreamReaderUnix assembles UTF-8 sequences (previously bytes >= 0x80 went negative, colliding with the EOF sentinel and hitting undefined isprint()); StreamReaderWin reads through ReadConsoleInputW, combining surrogate pairs and keeping the IME, dead-key and AltGr events it used to drop, and TerminalDisplayWin writes through WriteConsoleW. Also fixed: byte-vs-character index mixups in HandleControl(), Getline_color.cxx and tab completion (whose byte-space cursor could end up past the end of the line and abort), a pre-existing wrong-variable bug in the colorizer's trailing-space trim, and stale screen contents when deleting a combining mark. Verified on Linux with unit tests for the encoding/width layer and by driving root.exe through a pty: painting, cursor columns, wrapping, editing, history, reverse search and tab completion over accented, combining, CJK and emoji input. The Windows sources are only type-checked, not built. 🤖 Done with the help of AI.
e110a60 to
7c5883e
Compare
|
1000 lines added in the component every interactive root session goes through, but AI-assisted. The |
textinput assumed one line-buffer element is one byte and one terminal column. Both assumptions break on UTF-8, so anything outside ASCII corrupted the buffer on input and misplaced the cursor on output.
Text now stores std::u32string, so every index in the library counts characters; the editor's existing index arithmetic (cursor motions, cut/paste, ...) becomes correct unchanged. GetText() still returns UTF-8, cached with a character-to-byte offset table, so Getline's C interface, the interpreter and the history file are unaffected. Since a character occupies 0, 1 or 2 terminal columns, the display now lays out lines by accumulated character width (CharWidth() in the new UTF8.h) instead of index arithmetic, and moves a double-width character that would straddle the margin to the next line.
The readers now decode multi-byte input: StreamReaderUnix assembles UTF-8 sequences (previously bytes >= 0x80 went negative, colliding with the EOF sentinel and hitting undefined isprint()); StreamReaderWin reads through ReadConsoleInputW, combining surrogate pairs and keeping the IME, dead-key and AltGr events it used to drop, and TerminalDisplayWin writes through WriteConsoleW. Also fixed: byte-vs-character index mixups in HandleControl(), Getline_color.cxx and tab completion (whose byte-space cursor could end up past the end of the line and abort), a pre-existing wrong-variable bug in the colorizer's trailing-space trim, and stale screen contents when deleting a combining mark.
Verified on Linux with unit tests for the encoding/width layer and by driving root.exe through a pty: painting, cursor columns, wrapping, editing, history, reverse search and tab completion over accented, combining, CJK and emoji input. The Windows sources are only type-checked, not built.
🤖 Done with the help of AI.
Fixes ROOT-9562