Terminal: compose combining marks, and keep a grapheme cluster in two cells - #2904
Open
insjang wants to merge 4 commits into
Open
Terminal: compose combining marks, and keep a grapheme cluster in two cells#2904insjang wants to merge 4 commits into
insjang wants to merge 4 commits into
Conversation
The emulator assumed every character occupies one cell, so Hangul, Han and Kana text, fullwidth forms and emoji were placed one column short per character and the screen fell apart as soon as a program laid text out for a real terminal (line editors, curses UIs, Ink based CLIs). Add CharWidth, a UAX eclipse-platform#11 East Asian Width lookup: Wide and Fullwidth count as two columns, combining marks and controls as zero, Ambiguous as one, as UAX eclipse-platform#11 recommends outside an East Asian legacy context. The emulator advances the cursor by that width and stores a NUL filler in the second cell of a wide character, never splits one across the right margin, blanks the other half when either half is overwritten and counts insert mode in cells. The renderer skips the fillers so a fixed width font draws a wide glyph over both cells, falls back to placing each character at its own cell when the font does not advance exactly one cell per column, and draws a character beyond the BMP whole. A partial repaint that starts on the second cell of a wide character is widened to its first, and copying drops the fillers. Tests cover the width table, placement, the margin, overwriting halves and insert mode.
… cells A combining mark has no cell of its own. It was dropped, which lost data: an accent typed as a separate mark, a Hangul syllable sent as conjoining jamo, a Japanese voicing mark. A mark now composes with the character before it (NFC) whenever the two have a single form, which covers all of those. What has no composed form is a grapheme cluster: an emoji joined to others with a zero width joiner, one with a skin tone, a character with a presentation selector, a keycap, a flag made of two regional indicators. Windows Terminal (1.22+) and the libraries programs use to lay text out give such a cluster two cells however many characters it runs to, so the emulator has to count the same way or the columns drift. The cells keep the cluster's first character, which is what gets drawn; the whole cluster is kept beside the line and copied out whole, so what the user copies is what the program wrote. Writing either cell forgets the cluster, and it moves with its line when lines scroll or are copied. A narrow character asked to be shown as an emoji (VS16, as in a red heart) takes a second cell, as those terminals give it. Regional indicator pairs join into one flag. Marks that compose still compose first, so Hangul and accented Latin are unchanged. This builds on the East Asian Width change (CharWidth and the two-cell placement), and adds getCluster/setCluster to the text data API, hence the version bump.
New @SInCE 1.2 API was added to the package without bumping its Export-Package version, which API Tools flags as an error.
…size Matches the existing @SuppressWarnings("unchecked") convention used for the same generic array pattern in newClusters().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #2894 (East Asian Width): builds on
CharWidthand the two-cell placement of wide characters. This PR shows that commit as well until #2894 is merged; only the second commit is new here.A combining mark has no cell of its own. It was dropped, which lost data: an accent typed as a separate mark, a Hangul syllable sent as conjoining jamo, a Japanese voicing mark. A mark now composes with the character before it (NFC) whenever the two have a single form, which covers all of those.
What has no composed form is a grapheme cluster: an emoji joined to others with a zero-width joiner, one with a skin tone, a character with a presentation selector, a keycap, a flag made of two regional indicators. Windows Terminal (1.22+) and the libraries programs use to lay text out give such a cluster two cells however many characters it runs to, so the emulator has to count the same way or the columns drift.
API:
ITerminalTextDataReadOnly.getCluster/ITerminalTextData.setCluster(default methods,@since 1.2, bundle version 1.2.0).Tests: combining marks and clusters in
VT100EmulatorBackendTest; cluster bookkeeping (overwrite, scroll, clean, copy, window) inAbstractITerminalTextDataTestfor every data implementation.