Skip to content

[textinput] Support UTF-8 at the prompt - #23085

Merged
guitargeek merged 1 commit into
root-project:masterfrom
guitargeek:textinput-utf8-prompt
Sep 3, 2026
Merged

[textinput] Support UTF-8 at the prompt#23085
guitargeek merged 1 commit into
root-project:masterfrom
guitargeek:textinput-utf8-prompt

Conversation

@guitargeek

@guitargeek guitargeek commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

textinput assumed one line-buffer element is one byte and one terminal column. Both assumptions break on UTF-8, so anything outside ASCII corrupted the buffer on input and misplaced the cursor on output.

Text now stores std::u32string, so every index in the library counts characters; the editor's existing index arithmetic (cursor motions, cut/paste, ...) becomes correct unchanged. GetText() still returns UTF-8, cached with a character-to-byte offset table, so Getline's C interface, the interpreter and the history file are unaffected. Since a character occupies 0, 1 or 2 terminal columns, the display now lays out lines by accumulated character width (CharWidth() in the new UTF8.h) instead of index arithmetic, and moves a double-width character that would straddle the margin to the next line.

The readers now decode multi-byte input: StreamReaderUnix assembles UTF-8 sequences (previously bytes >= 0x80 went negative, colliding with the EOF sentinel and hitting undefined isprint()); StreamReaderWin reads through ReadConsoleInputW, combining surrogate pairs and keeping the IME, dead-key and AltGr events it used to drop, and TerminalDisplayWin writes through WriteConsoleW. Also fixed: byte-vs-character index mixups in HandleControl(), Getline_color.cxx and tab completion (whose byte-space cursor could end up past the end of the line and abort), a pre-existing wrong-variable bug in the colorizer's trailing-space trim, and stale screen contents when deleting a combining mark.

Verified on Linux with unit tests for the encoding/width layer and by driving root.exe through a pty: painting, cursor columns, wrapping, editing, history, reverse search and tab completion over accented, combining, CJK and emoji input. The Windows sources are only type-checked, not built.

🤖 Done with the help of AI.

unicode

Fixes ROOT-9562

@ferdymercury

ferdymercury commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Comment thread core/textinput/src/textinput/UTF8.h
Comment thread core/textinput/src/textinput/UTF8.cpp
@dpiparo

dpiparo commented Aug 17, 2026

Copy link
Copy Markdown
Member

this is remarkable.

@guitargeek
guitargeek force-pushed the textinput-utf8-prompt branch from 96498a7 to 06fbd6d Compare August 17, 2026 11:52
@github-actions

github-actions Bot commented Aug 17, 2026

Copy link
Copy Markdown

Test Results

    23 files      23 suites   3d 18h 22m 38s ⏱️
 3 859 tests  3 858 ✅ 0 💤 1 ❌
79 522 runs  79 520 ✅ 1 💤 1 ❌

For more details on these failures, see this check.

Results for commit 7c5883e.

♻️ This comment has been updated with latest results.

@wacfrr

wacfrr commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

🎉 Great! Now we are one step closer to being able to count past 255. 🚀
🎉 太好了!距离我们能数到255以上又近了一步。🚀

@guitargeek
guitargeek force-pushed the textinput-utf8-prompt branch from 06fbd6d to e110a60 Compare August 18, 2026 21:54
textinput assumed one line-buffer element is one byte and one terminal
column. Both assumptions break on UTF-8, so anything outside ASCII
corrupted the buffer on input and misplaced the cursor on output.

Text now stores std::u32string, so every index in the library counts
characters; the editor's existing index arithmetic (cursor motions,
cut/paste, ...) becomes correct unchanged. GetText() still returns UTF-8,
cached with a character-to-byte offset table, so Getline's C interface,
the interpreter and the history file are unaffected. Since a character
occupies 0, 1 or 2 terminal columns, the display now lays out lines by
accumulated character width (CharWidth() in the new UTF8.h) instead of
index arithmetic, and moves a double-width character that would straddle
the margin to the next line.

The readers now decode multi-byte input: StreamReaderUnix assembles UTF-8
sequences (previously bytes >= 0x80 went negative, colliding with the EOF
sentinel and hitting undefined isprint()); StreamReaderWin reads through
ReadConsoleInputW, combining surrogate pairs and keeping the IME, dead-key
and AltGr events it used to drop, and TerminalDisplayWin writes through
WriteConsoleW. Also fixed: byte-vs-character index mixups in
HandleControl(), Getline_color.cxx and tab completion (whose byte-space
cursor could end up past the end of the line and abort), a pre-existing
wrong-variable bug in the colorizer's trailing-space trim, and stale
screen contents when deleting a combining mark.

Verified on Linux with unit tests for the encoding/width layer and by
driving root.exe through a pty: painting, cursor columns, wrapping,
editing, history, reverse search and tab completion over accented,
combining, CJK and emoji input. The Windows sources are only
type-checked, not built.

🤖 Done with the help of AI.
@guitargeek
guitargeek force-pushed the textinput-utf8-prompt branch from e110a60 to 7c5883e Compare August 19, 2026 09:37
@guitargeek
guitargeek marked this pull request as ready for review September 2, 2026 14:50
@guitargeek

Copy link
Copy Markdown
Contributor Author

1000 lines added in the component every interactive root session goes through, but AI-assisted. The pty-driven testing described and local tests are reassuring, but this is the kind of change you want to land early in a dev cycle so it soaks on master and get some testing from developers and power users, not right before a release branch is cut (which will happen in November).

@guitargeek
guitargeek merged commit db68792 into root-project:master Sep 3, 2026
31 of 34 checks passed
@guitargeek
guitargeek deleted the textinput-utf8-prompt branch September 3, 2026 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants