diff --git a/CHANGELOG.md b/CHANGELOG.md index 454e66b0c..bd881ce9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,20 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ## [Unreleased] +### Added +- **Two separate thumb tracks (experimental)** — an opt-in gesture mode that feeds an earlier part of a word and the part you're swiping now to the recognizer as two *separate* thumb strokes instead of splicing them into one long trail, removing the invented connector movement between them. Under Two-Thumb Typing → Recognition. (#135) +- **Redraw earlier word parts cleanly (experimental)** — optionally replaces the earlier part of a multi-part word with a tidy path through its key centres, and turns a single tap into a small swipe, so the recognizer sees one believable whole-word gesture. Includes tunable trail-speed and pause controls. (#135) + +### Fixed +- **Gesture typing no longer silently returns zero suggestions** when a stroke's touch points never carry pointer id 0 — reachable in two-thumb use (thumb A down, thumb B down, thumb A lifts, thumb B swipes on). Raw MotionEvent pointer ids are now renumbered in first-seen order onto the two per-pointer tracks the native decoder actually reads. (#135) + +### Reliability & testing +- Added a native gesture **two-pointer track harness** (`jni/tests/replay/two_pointer_track_test.cpp`) that drives the real AOSP `ProximityInfoState` on the host, with tunable pointer-id / time-policy / tap-promotion knobs and a printed sweep table. Runs in CI alongside the existing native suite. (#135) +- The multi-part trail merge moved behind a pure, unit-tested `StrokeAligner` seam whose defaults reproduce the previous behaviour exactly. (#135) + +### Changed +- Documented the two-thumb decoder research in `docs/TWO_THUMB_TEMPORAL_ALIGNMENT.md`: the native decoder models two pointer tracks, track membership is decided by pointer id rather than timing, and deliberately overlapping stroke timestamps measurably corrupts the decoder's speed features. (#135) + ## [0.2.0] - 2026-08-06 ### Upstream diff --git a/app/src/main/java/helium314/keyboard/keyboard/internal/BatchInputArbiter.java b/app/src/main/java/helium314/keyboard/keyboard/internal/BatchInputArbiter.java index 8b7ba6001..edcdeb28a 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/internal/BatchInputArbiter.java +++ b/app/src/main/java/helium314/keyboard/keyboard/internal/BatchInputArbiter.java @@ -35,6 +35,13 @@ void onUpdateBatchInput( private static int sLastRecognitionPointSize = 0; // synchronized using sAggregatedPointers private static long sLastRecognitionTime = 0; // synchronized using sAggregatedPointers + // Renumbers raw MotionEvent pointer ids onto the dense track slots the native decoder reads. + // Without this, a gesture whose points never carry id 0 (e.g. thumb A down, thumb B down, + // thumb A lifts, thumb B swipes on with id 1) leaves the decoder's track 0 unused and + // Suggest::initializeSearch bails out with zero suggestions. Guarded by sAggregatedPointers + // like the other statics here. + private static final PointerIdNormalizer sPointerIdNormalizer = new PointerIdNormalizer(); + // ---- Two-thumb typing: autospace grace period (#1.2) ---- // When the last finger of a gesture lifts and the user has configured a non-zero grace // window, we delay the actual commit (the "autospace grace period"). If another finger @@ -75,8 +82,10 @@ public interface DeferredCommit { } private final GestureStrokeRecognitionPoints mRecognitionPoints; + private final int mPointerId; public BatchInputArbiter(final int pointerId, final GestureStrokeRecognitionParams params) { + mPointerId = pointerId; mRecognitionPoints = new GestureStrokeRecognitionPoints(pointerId, params); } @@ -154,6 +163,7 @@ public boolean mayStartBatchInput(final BatchInputArbiterListener listener) { sAggregatedPointers.reset(); sLastRecognitionPointSize = 0; sLastRecognitionTime = 0; + sPointerIdNormalizer.reset(); listener.onStartBatchInput(); } return true; @@ -184,7 +194,8 @@ public void updateBatchInputByTimer(final long syntheticMoveEventTime, public void updateBatchInput(final long moveEventTime, final BatchInputArbiterListener listener) { synchronized (sAggregatedPointers) { - mRecognitionPoints.appendIncrementalBatchPoints(sAggregatedPointers); + mRecognitionPoints.appendIncrementalBatchPoints(sAggregatedPointers, + sPointerIdNormalizer.slotFor(mPointerId)); final int size = sAggregatedPointers.getPointerSize(); if (size > sLastRecognitionPointSize && mRecognitionPoints.hasRecognitionTimePast( moveEventTime, sLastRecognitionTime)) { @@ -229,7 +240,8 @@ public boolean mayEndBatchInput(final long upEventTime, final int activePointerC final int graceMs, final BatchInputArbiterListener listener, final DeferredCommit deferredCommit) { synchronized (sAggregatedPointers) { - mRecognitionPoints.appendAllBatchPoints(sAggregatedPointers); + mRecognitionPoints.appendAllBatchPoints(sAggregatedPointers, + sPointerIdNormalizer.slotFor(mPointerId)); if (activePointerCount != 1) { // Other fingers are still down — gesture continues, no commit yet. return false; diff --git a/app/src/main/java/helium314/keyboard/keyboard/internal/GestureStrokeRecognitionPoints.java b/app/src/main/java/helium314/keyboard/keyboard/internal/GestureStrokeRecognitionPoints.java index 4d00765ed..c3253cc3a 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/internal/GestureStrokeRecognitionPoints.java +++ b/app/src/main/java/helium314/keyboard/keyboard/internal/GestureStrokeRecognitionPoints.java @@ -303,20 +303,36 @@ public boolean hasRecognitionTimePast( // TODO: Make this package private public void appendAllBatchPoints(final InputPointers out) { - appendBatchPoints(out, getLength()); + appendBatchPoints(out, getLength(), mPointerId); + } + + /** + * Same as {@link #appendAllBatchPoints(InputPointers)} but stamps the points with an explicit + * track slot instead of the raw MotionEvent pointer id. See {@link PointerIdNormalizer}. + */ + public void appendAllBatchPoints(final InputPointers out, final int pointerIdOverride) { + appendBatchPoints(out, getLength(), pointerIdOverride); } // TODO: Make this package private public void appendIncrementalBatchPoints(final InputPointers out) { - appendBatchPoints(out, mIncrementalRecognitionSize); + appendBatchPoints(out, mIncrementalRecognitionSize, mPointerId); + } + + /** + * Same as {@link #appendIncrementalBatchPoints(InputPointers)} but stamps the points with an + * explicit track slot instead of the raw MotionEvent pointer id. + */ + public void appendIncrementalBatchPoints(final InputPointers out, final int pointerIdOverride) { + appendBatchPoints(out, mIncrementalRecognitionSize, pointerIdOverride); } - private void appendBatchPoints(final InputPointers out, final int size) { + private void appendBatchPoints(final InputPointers out, final int size, final int pointerId) { final int length = size - mLastIncrementalBatchSize; if (length <= 0) { return; } - out.append(mPointerId, mEventTimes, mXCoordinates, mYCoordinates, + out.append(pointerId, mEventTimes, mXCoordinates, mYCoordinates, mLastIncrementalBatchSize, length); mLastIncrementalBatchSize = size; } diff --git a/app/src/main/java/helium314/keyboard/keyboard/internal/PointerIdNormalizer.java b/app/src/main/java/helium314/keyboard/keyboard/internal/PointerIdNormalizer.java new file mode 100644 index 000000000..2cfd5f86b --- /dev/null +++ b/app/src/main/java/helium314/keyboard/keyboard/internal/PointerIdNormalizer.java @@ -0,0 +1,74 @@ +/* + * SPDX-License-Identifier: GPL-3.0-only + */ + +package helium314.keyboard.keyboard.internal; + +/** + * Maps raw {@link android.view.MotionEvent} pointer ids onto the dense track slots the native + * gesture decoder actually reads. + * + *

Why this exists. The native decoder keeps exactly {@code MAX_POINTER_COUNT_G == 2} + * per-pointer tracks ({@code jni/src/defines.h}). {@code DicTraverseSession} seeds track i + * with pointer id i, and {@code ProximityInfoStateUtils::updateTouchPoints} keeps only the + * points whose {@code pointerIds[k] == i}. Two consequences follow, both measured in + * {@code jni/tests/replay/two_pointer_track_test.cpp}: + * + *

+ * + *

Android assigns pointer ids as the lowest currently-free index, so ids are neither guaranteed + * to start at 0 for a given stroke nor to be contiguous. This class removes that dependency by + * renumbering ids in first-seen order within a gesture: the first pointer to contribute + * becomes slot 0, the second becomes slot 1, and so on. + * + *

For the overwhelmingly common cases the mapping is the identity (single finger 0 → 0; two + * fingers 0,1 → 0,1), so this is a no-op in normal use and only repairs the broken cases. Slots + * {@code >= 2} are still dropped by the native side exactly as before — this class deliberately + * does not merge a third finger into an existing track, which would change recognition behaviour. + * + *

Not thread-safe: like the rest of the batch-input machinery it is only touched from the + * keyboard view's UI thread. + */ +public final class PointerIdNormalizer { + + /** Beyond this many distinct pointers in one gesture we stop renumbering and pass ids through. */ + private static final int MAX_TRACKED_POINTERS = 8; + + private final int[] mRawIds = new int[MAX_TRACKED_POINTERS]; + private int mCount; + + /** Forget every mapping; call at the start of each gesture. */ + public void reset() { + mCount = 0; + } + + /** + * @return the dense slot for {@code rawPointerId}, allocating one in first-seen order if this + * is the first time the id is seen in the current gesture. Returns {@code rawPointerId} + * unchanged if more than {@link #MAX_TRACKED_POINTERS} distinct pointers appear (which the + * decoder would discard anyway). + */ + public int slotFor(final int rawPointerId) { + for (int i = 0; i < mCount; i++) { + if (mRawIds[i] == rawPointerId) { + return i; + } + } + if (mCount >= MAX_TRACKED_POINTERS) { + return rawPointerId; + } + mRawIds[mCount] = rawPointerId; + return mCount++; + } + + /** @return how many distinct pointers have been seen since the last {@link #reset()}. */ + public int trackedPointerCount() { + return mCount; + } +} diff --git a/app/src/main/java/helium314/keyboard/latin/WordComposer.java b/app/src/main/java/helium314/keyboard/latin/WordComposer.java index e8d93e519..eea455fac 100644 --- a/app/src/main/java/helium314/keyboard/latin/WordComposer.java +++ b/app/src/main/java/helium314/keyboard/latin/WordComposer.java @@ -18,6 +18,8 @@ import helium314.keyboard.latin.common.StringUtils; import helium314.keyboard.latin.define.DebugFlags; import helium314.keyboard.latin.define.DecoderSpecificConstants; +import helium314.keyboard.latin.gesture.StrokeAligner; +import helium314.keyboard.latin.settings.Settings; import java.util.ArrayList; import java.util.Collections; @@ -61,13 +63,6 @@ public final class WordComposer { // huge time discontinuity at the prefix/swipe boundary and confuse the recognizer. private final InputPointers mExtendBatchInputBase = new InputPointers(MAX_WORD_LENGTH); private boolean mExtendBatchInputBaseSet; - // Inter-point interval used when synthesising timestamps for the base. Roughly the - // sampling rate of a fast hand-drawn swipe; chosen to look like natural gesture speed. - private static final int EXTEND_BASE_POINT_INTERVAL_MS = 25; - // Gap inserted between the last synthetic base point and the first real point of the - // current gesture. Pretends the user briefly paused at the prefix endpoint before - // continuing the stroke — within the recogniser's "single stroke" tolerance. - private static final int EXTEND_BASE_GAP_BEFORE_NEW_MS = 60; // Cache these values for performance private CharSequence mTypedWordCache; @@ -285,20 +280,10 @@ public void setBatchInputPointers(final InputPointers batchPointers) { if (mExtendBatchInputBaseSet && mExtendBatchInputBase.getPointerSize() > 0 && batchPointers.getPointerSize() > 0) { // Multi-part composition: feed the lib the merged trail (prior fragments + - // current gesture) with synthesised timestamps so the base looks like a - // natural continuation of the new gesture. - final int baseSize = mExtendBatchInputBase.getPointerSize(); - final int[] baseX = mExtendBatchInputBase.getXCoordinates(); - final int[] baseY = mExtendBatchInputBase.getYCoordinates(); - final int firstNewTime = batchPointers.getTimes()[0]; - final int baseLastTime = firstNewTime - EXTEND_BASE_GAP_BEFORE_NEW_MS; - final int baseFirstTime = baseLastTime - (baseSize - 1) * EXTEND_BASE_POINT_INTERVAL_MS; - mInputPointers.reset(); - for (int i = 0; i < baseSize; i++) { - mInputPointers.addPointer(baseX[i], baseY[i], 0, - baseFirstTime + i * EXTEND_BASE_POINT_INTERVAL_MS); - } - mInputPointers.appendAll(batchPointers); + // current gesture). StrokeAligner owns the re-timing and the pointer-id policy — + // see docs/TWO_THUMB_TEMPORAL_ALIGNMENT.md. + StrokeAligner.merge(mInputPointers, mExtendBatchInputBase, batchPointers, + Settings.getValues().mStrokeAlignParams); } else { mInputPointers.set(batchPointers); } diff --git a/app/src/main/java/helium314/keyboard/latin/common/InputPointers.java b/app/src/main/java/helium314/keyboard/latin/common/InputPointers.java index f1e4d0a1a..c20fca4ce 100644 --- a/app/src/main/java/helium314/keyboard/latin/common/InputPointers.java +++ b/app/src/main/java/helium314/keyboard/latin/common/InputPointers.java @@ -107,15 +107,48 @@ public void shift(final int elementCount) { } /** - * Append all pointers from {@code other} to the end of this. Pointer ids are forced to - * 0 since multi-part gesture composition doesn't preserve pointer identity across - * separate strokes. + * Append all pointers from {@code other} to the end of this, forcing pointer id 0. + * + *

Historically this was the only merge path, which is why the decoder's second pointer + * track was never populated by multi-part composition. Prefer + * {@link #appendAll(InputPointers, int)} when the caller knows which track the points belong + * to — see {@link helium314.keyboard.latin.gesture.StrokeAligner}. */ public void appendAll(@NonNull final InputPointers other) { - append(0, other.mTimes, other.mXCoordinates, other.mYCoordinates, 0, + appendAll(other, 0); + } + + /** + * Append all pointers from {@code other} to the end of this, stamping them with + * {@code pointerId}. + * + *

The native decoder keeps one {@code ProximityInfoState} per pointer id (two of them, + * {@code MAX_POINTER_COUNT_G}) and each state ingests only the points carrying its own + * id. So this argument decides which decoder track the appended stroke lands in. Ids outside + * {@code [0, 1]} reach no track at all. + */ + public void appendAll(@NonNull final InputPointers other, final int pointerId) { + append(pointerId, other.mTimes, other.mXCoordinates, other.mYCoordinates, 0, other.getPointerSize()); } + /** + * Append all pointers from {@code other}, keeping each point's own pointer id. + * + *

Used when {@code other} is already a genuine multi-pointer stroke whose track assignment + * must survive the merge. + */ + public void appendAllPreservingIds(@NonNull final InputPointers other) { + final int length = other.getPointerSize(); + if (length == 0) { + return; + } + mXCoordinates.append(other.mXCoordinates, 0, length); + mYCoordinates.append(other.mYCoordinates, 0, length); + mPointerIds.append(other.mPointerIds, 0, length); + mTimes.append(other.mTimes, 0, length); + } + public void reset() { final int defaultCapacity = mDefaultCapacity; mXCoordinates.reset(defaultCapacity); diff --git a/app/src/main/java/helium314/keyboard/latin/gesture/IdealPrefixTrailBuilder.java b/app/src/main/java/helium314/keyboard/latin/gesture/IdealPrefixTrailBuilder.java new file mode 100644 index 000000000..6d9f0da95 --- /dev/null +++ b/app/src/main/java/helium314/keyboard/latin/gesture/IdealPrefixTrailBuilder.java @@ -0,0 +1,120 @@ +/* + * SPDX-License-Identifier: GPL-3.0-only + */ + +package helium314.keyboard.latin.gesture; + +import helium314.keyboard.keyboard.Key; +import helium314.keyboard.keyboard.Keyboard; +import helium314.keyboard.latin.common.InputPointers; + +import java.util.HashMap; +import java.util.Map; + +/** + * Synthesises an "ideal" gesture trail for a composing prefix by tracing the prefix's key centres, + * so the merged stream handed to the decoder looks like one plausible whole-word swipe. + * + *

Motivation: the prior-fragment base that {@link StrokeAligner} prepends is otherwise the + * raw trail of what came before — which is sparse and un-stroke-like for a tap (a single + * coordinate) and noisy for a partial swipe. Replacing it with a clean key-centre path gives the + * recognizer the shape it was trained on. + * + *

Tap promotion. A single-letter prefix becomes a small out-and-back micro-stroke around + * the key centre rather than a lone point, so the recognizer sees a vertex — this is the + * "promote taps into small swipes" idea, and it is why an isolated tap coordinate no longer has to + * masquerade as a stroke. + * + *

Only coordinates matter here: {@link StrokeAligner#merge} discards the base's timestamps and + * re-synthesises them relative to the incoming stroke, so the times written below are placeholders. + * + *

Originally written for issue #99 (B7b) and gated to a side-by-side {@code swipetest} build; + * it is now reachable at runtime via {@code PREF_STROKE_IDEAL_PREFIX}. + */ +public final class IdealPrefixTrailBuilder { + + private IdealPrefixTrailBuilder() {} + + /** Roughly one sample per (keyWidth / SPACING_DIVISOR) px along each inter-key segment. */ + private static final int SPACING_DIVISOR = 4; + /** Micro-stroke radius for a tap prefix, as a fraction of key width. */ + private static final int TAP_ARC_RADIUS_DIVISOR = 6; + /** Fallback key width when the keyboard reports none. */ + private static final int FALLBACK_KEY_WIDTH = 40; + + /** + * @return a key-centre trail for {@code word}, or {@code null} if it can't be built — an empty + * word, no keyboard, or any letter that isn't on this keyboard. Returning null + * rather than a partial path matters: a hole in the synthetic trail is worse for + * recognition than the raw trail the caller falls back to. + */ + public static InputPointers build(final String word, final Keyboard keyboard) { + if (word == null || word.isEmpty() || keyboard == null) return null; + + final Map keyByCode = new HashMap<>(); + int keyWidth = 0; + for (final Key key : keyboard.getSortedKeys()) { + final int code = key.getCode(); + if (code <= 0 || key.isModifier() || !Character.isLetter(code)) continue; + keyByCode.put(Character.toLowerCase(code), key); + if (keyWidth == 0) keyWidth = key.getWidth(); + } + if (keyByCode.isEmpty()) return null; + + final int len = word.length(); + final int[] cx = new int[len]; + final int[] cy = new int[len]; + int count = 0; + for (int i = 0; i < len; ) { + final int cp = word.codePointAt(i); + i += Character.charCount(cp); + final Key key = keyByCode.get(Character.toLowerCase(cp)); + if (key != null) { + cx[count] = key.getX() + key.getWidth() / 2; + cy[count] = key.getY() + key.getHeight() / 2; + count++; + continue; + } + if (Character.isLetter(cp) || Character.getType(cp) == Character.NON_SPACING_MARK) { + // A letter we cannot place would leave a hole in the synthetic path, which is + // worse than the raw trail. Bail out and let the caller fall back. Covers popup + // letters, accented/combining forms and unsupported scripts. + return null; + } + // Non-letters (apostrophes, digits, punctuation) are legitimately not on the trail. + } + if (count == 0) return null; + + final int effectiveKeyWidth = keyWidth > 0 ? keyWidth : FALLBACK_KEY_WIDTH; + final InputPointers out = new InputPointers(64); + + if (count == 1) { + // Tap prefix → small out-and-back micro-stroke so the recognizer sees a vertex + // instead of an isolated point. + final int r = Math.max(1, effectiveKeyWidth / TAP_ARC_RADIUS_DIVISOR); + addPoint(out, cx[0] - r, cy[0]); + addPoint(out, cx[0], cy[0]); + addPoint(out, cx[0] + r, cy[0]); + addPoint(out, cx[0], cy[0]); + return out; + } + + final int step = Math.max(1, effectiveKeyWidth / SPACING_DIVISOR); + addPoint(out, cx[0], cy[0]); + for (int i = 1; i < count; i++) { + final int x0 = cx[i - 1], y0 = cy[i - 1], x1 = cx[i], y1 = cy[i]; + final double dist = Math.hypot(x1 - x0, y1 - y0); + final int samples = Math.max(1, (int) (dist / step)); + for (int s = 1; s <= samples; s++) { + final float t = (float) s / samples; + addPoint(out, Math.round(x0 + (x1 - x0) * t), Math.round(y0 + (y1 - y0) * t)); + } + } + return out; + } + + private static void addPoint(final InputPointers out, final int x, final int y) { + // pointerId 0 / time 0: StrokeAligner re-stamps both when it merges the base. + out.addPointer(x, y, StrokeAligner.BASE_POINTER_ID, 0); + } +} diff --git a/app/src/main/java/helium314/keyboard/latin/gesture/StrokeAligner.java b/app/src/main/java/helium314/keyboard/latin/gesture/StrokeAligner.java new file mode 100644 index 000000000..05f20538c --- /dev/null +++ b/app/src/main/java/helium314/keyboard/latin/gesture/StrokeAligner.java @@ -0,0 +1,182 @@ +/* + * SPDX-License-Identifier: GPL-3.0-only + */ + +package helium314.keyboard.latin.gesture; + +import helium314.keyboard.latin.common.InputPointers; + +/** + * Merges a multi-part word's prior-fragment trail ("the base") with the stroke currently being + * gestured, into the single {@link InputPointers} stream handed to the gesture decoder. + * + *

This is the seam where the fork decides what shape the decoder thinks the user drew, + * and it is deliberately parameterised so the alternatives can be A/B'd on device. See + * {@code docs/TWO_THUMB_TEMPORAL_ALIGNMENT.md} for the measurements behind the defaults. + * + *

The two modes

+ * + * {@link Mode#CONNECTOR} (default, the historical behaviour): everything is stamped with + * pointer id 0, so the decoder sees one long single-pointer glide. The jump from the base's last + * point to the new stroke's first point is an implicit "connector" that the decoder reads as real + * movement — which is where hallucinated middle letters ({@code techcolony}) come from. + * + *

{@link Mode#DUAL_POINTER}: the base keeps pointer id 0 and the current stroke gets + * pointer id 1, so the two land in separate decoder tracks + * ({@code ProximityInfoState[0]} and {@code [1]}). There is then no connector to hallucinate + * across, and the decoder's built-in two-pointer search can spell the word by alternating between + * the tracks. Measured in {@code jni/tests/replay/two_pointer_track_test.cpp}. + * + *

Invariants this class must preserve

+ * + *
    + *
  1. Track 0 must be non-empty. {@code Suggest::initializeSearch} early-returns when + * {@code ProximityInfoState(0)} is unused, yielding zero suggestions. The base always takes + * id 0, and callers only reach the merge path with a non-empty base.
    2. + *
  2. Only ids 0 and 1 are ever emitted. Anything higher reaches no track at all. With + * three or more fragments the older ones stay collapsed into track 0 (joined by connectors, + * exactly as today) and only the newest stroke gets track 1.
    3. + *
  3. Timestamps stay globally monotonic. The decoder's speed and beeline features walk + * the raw arrays across the track boundary, so a decreasing timestamp there yields a + * negative duration and a garbage speed rate. The base is therefore always re-timed to end + * {@code gapBeforeNewMs} before the current stroke begins — in both modes.
    4. + *
  4. Ids are stable for a given raw index across incremental recognition. + * {@code checkAndReturnIsContinuousSuggestionPossible} compares x/y/time but not pointer + * ids, so a point must never change track mid-gesture. The base is fixed for the duration of + * a gesture and the current stroke only grows at the tail, so this holds.
    5. + *
+ * + *

Called on the input path, so it allocates nothing beyond what {@link InputPointers} itself + * needs to grow. + */ +public final class StrokeAligner { + + private StrokeAligner() {} + + /** Pointer id of the prior-fragment base. Must be 0 — see invariant 1. */ + public static final int BASE_POINTER_ID = 0; + /** Pointer id of the in-flight stroke under {@link Mode#DUAL_POINTER}. */ + public static final int CURRENT_POINTER_ID = 1; + + public enum Mode { + /** One merged single-pointer trail (historical behaviour). */ + CONNECTOR, + /** Base on decoder track 0, current stroke on track 1. */ + DUAL_POINTER; + + /** Parses the stored preference value, falling back to {@link #CONNECTOR}. */ + public static Mode fromPrefValue(final String value) { + if ("dual_pointer".equals(value)) return DUAL_POINTER; + return CONNECTOR; + } + } + + /** Tunable knobs. Defaults reproduce the historical behaviour exactly. */ + public static final class Params { + /** Inter-point interval when synthesising timestamps for the base. */ + public static final int DEFAULT_INTERVAL_MS = 25; + /** Gap between the base's last synthetic point and the stroke's first real point. */ + public static final int DEFAULT_GAP_MS = 60; + + public final Mode mode; + public final int basePointIntervalMs; + public final int gapBeforeNewMs; + + public Params(final Mode mode, final int basePointIntervalMs, final int gapBeforeNewMs) { + this.mode = mode == null ? Mode.CONNECTOR : mode; + // Clamp to sane values: a non-positive interval would make the base's timestamps + // non-increasing, which is exactly the negative-duration hazard invariant 3 exists to + // avoid. + this.basePointIntervalMs = Math.max(1, basePointIntervalMs); + this.gapBeforeNewMs = Math.max(1, gapBeforeNewMs); + } + + public static Params defaults() { + return new Params(Mode.CONNECTOR, DEFAULT_INTERVAL_MS, DEFAULT_GAP_MS); + } + + /** + * The timing knobs are only surfaced in the UI for {@link Mode#DUAL_POINTER}, so + * {@link Mode#CONNECTOR} pins them to the historical constants. Without this, tuning the + * sliders in dual mode and switching back would silently leave "one joined trail" behaving + * differently from how it always has. + */ + int effectiveIntervalMs() { + return mode == Mode.DUAL_POINTER ? basePointIntervalMs : DEFAULT_INTERVAL_MS; + } + + int effectiveGapMs() { + return mode == Mode.DUAL_POINTER ? gapBeforeNewMs : DEFAULT_GAP_MS; + } + } + + /** + * Merge {@code base} and {@code current} into {@code out}, which is reset first. + * + *

The base's own timestamps are discarded and re-synthesised backwards from the current + * stroke's first point, because base coordinates can come from taps (which carry a {@code 0} + * time sentinel) or from an earlier gesture on an unrelated clock. Only the base's geometry is + * meaningful. + * + * @param out receives the merged stream; must not alias {@code base} or {@code current}. + * @param base prior fragments' trail. If empty, {@code current} is copied through unchanged. + * @param current the stroke being gestured now. + * @param params tuning knobs; {@code null} means {@link Params#defaults()}. + */ + public static void merge(final InputPointers out, final InputPointers base, + final InputPointers current, final Params params) { + final Params p = params == null ? Params.defaults() : params; + final int baseSize = base == null ? 0 : base.getPointerSize(); + final int currentSize = current == null ? 0 : current.getPointerSize(); + + if (baseSize == 0 || currentSize == 0) { + // Nothing to merge — a lone stroke keeps whatever pointer ids it already carries, so + // genuinely simultaneous two-thumb input is untouched by this class. + out.reset(); + if (currentSize > 0) { + out.set(current); + } else if (baseSize > 0) { + out.set(base); + } + return; + } + + final int[] baseX = base.getXCoordinates(); + final int[] baseY = base.getYCoordinates(); + final int intervalMs = p.effectiveIntervalMs(); + final int firstNewTime = current.getTimes()[0]; + final int baseLastTime = firstNewTime - p.effectiveGapMs(); + final int baseFirstTime = baseLastTime - (baseSize - 1) * intervalMs; + + out.reset(); + for (int i = 0; i < baseSize; i++) { + out.addPointer(baseX[i], baseY[i], BASE_POINTER_ID, baseFirstTime + i * intervalMs); + } + + if (p.mode != Mode.DUAL_POINTER) { + out.appendAll(current, BASE_POINTER_ID); + return; + } + // Dual-pointer: the current stroke normally takes track 1 wholesale. But if it is ITSELF a + // simultaneous two-thumb stroke it already occupies both tracks, and flattening it onto + // track 1 would destroy that structure. In that case keep its own ids and let the base + // share track 0 — which reads coherently as "track 0 = this thumb plus the word so far". + if (isMultiPointer(current)) { + out.appendAllPreservingIds(current); + } else { + out.appendAll(current, CURRENT_POINTER_ID); + } + } + + /** @return true if {@code pointers} carries more than one distinct pointer id. */ + private static boolean isMultiPointer(final InputPointers pointers) { + final int size = pointers.getPointerSize(); + if (size < 2) return false; + final int[] ids = pointers.getPointerIds(); + final int first = ids[0]; + for (int i = 1; i < size; i++) { + if (ids[i] != first) return true; + } + return false; + } +} diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index 90907637a..604c77469 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -54,6 +54,7 @@ import helium314.keyboard.latin.common.StringUtilsKt; import helium314.keyboard.latin.common.SuggestionSpanUtilsKt; import helium314.keyboard.latin.define.DebugFlags; +import helium314.keyboard.latin.gesture.IdealPrefixTrailBuilder; import helium314.keyboard.latin.settings.Settings; import helium314.keyboard.latin.settings.SettingsValues; import helium314.keyboard.latin.settings.SpacingAndPunctuations; @@ -821,7 +822,17 @@ public void onStartBatchInput(final SettingsValues settingsValues, // word simply stays open until the user taps space), see // SettingsValues#isMultipartComposeActive. if (settingsValues.isMultipartComposeActive()) { - mWordComposer.setExtendBatchInputBase(mWordComposer.getInputPointers()); + // With the ideal-prefix knob on, replace the raw prior trail with a clean + // key-centre path for the composing prefix (and promote a one-letter prefix + // to a micro-stroke). Falls back to the raw trail whenever the synthetic one + // can't be built, so a fragment is never lost. + InputPointers base = null; + if (settingsValues.mStrokeIdealPrefix) { + base = IdealPrefixTrailBuilder.build(mWordComposer.getTypedWord(), + keyboardSwitcher.getKeyboard()); + } + mWordComposer.setExtendBatchInputBase( + base != null ? base : mWordComposer.getInputPointers()); } } else if (mWordComposer.isSingleLetter() && !isInlineEmojiSearchAction()) { // We auto-correct the previous (typed, not gestured) string iff it's one diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt index b6c6d59f3..876baec53 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt +++ b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt @@ -194,6 +194,10 @@ object Defaults { const val PREF_MULTIPART_FULL_WORD_SUGGESTIONS = true const val PREF_MULTIPART_TAP_SEED_GESTURE = true const val PREF_MULTIPART_RERECOGNIZE_TAPS = false + const val PREF_STROKE_ALIGN_MODE = "connector" + const val PREF_STROKE_ALIGN_INTERVAL_MS = 25 + const val PREF_STROKE_ALIGN_GAP_MS = 60 + const val PREF_STROKE_IDEAL_PREFIX = false const val PREF_SHOW_SETUP_WIZARD_ICON = true const val PREF_USE_CONTACTS = false const val PREF_USE_APPS = false diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java index 94bb9e240..4f5da38ec 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java @@ -210,6 +210,12 @@ public final class Settings implements SharedPreferences.OnSharedPreferenceChang // re-recognize the whole word, instead of literally appending it to a (possibly // mis-resolved) fragment. Makes a slow tap-after-swipe behave like a fast one. Default off. public static final String PREF_MULTIPART_RERECOGNIZE_TAPS = "multipart_rerecognize_taps"; + // Stroke alignment (#135): how the prior-fragment base and the in-flight stroke are merged + // before the decoder sees them. See docs/TWO_THUMB_TEMPORAL_ALIGNMENT.md. + public static final String PREF_STROKE_ALIGN_MODE = "stroke_align_mode"; + public static final String PREF_STROKE_ALIGN_INTERVAL_MS = "stroke_align_interval_ms"; + public static final String PREF_STROKE_ALIGN_GAP_MS = "stroke_align_gap_ms"; + public static final String PREF_STROKE_IDEAL_PREFIX = "stroke_ideal_prefix"; public static final String PREF_SHOW_SETUP_WIZARD_ICON = "show_setup_wizard_icon"; public static final String PREF_USE_CONTACTS = "use_contacts"; public static final String PREF_USE_APPS = "use_apps"; diff --git a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java index 4df0ffde1..c56ce869f 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java @@ -25,6 +25,7 @@ import helium314.keyboard.latin.R; import helium314.keyboard.latin.RichInputMethodManager; import helium314.keyboard.latin.common.Colors; +import helium314.keyboard.latin.gesture.StrokeAligner; import helium314.keyboard.latin.permissions.PermissionsUtil; import helium314.keyboard.latin.utils.InputTypeUtils; import helium314.keyboard.latin.utils.JniUtils; @@ -157,6 +158,9 @@ public class SettingsValues { public final boolean mMultipartFullWordSuggestions; public final boolean mMultipartTapSeedGesture; public final boolean mMultipartRerecognizeTaps; + // Stroke alignment (#135): pre-built so the input path never parses prefs per gesture. + public final StrokeAligner.Params mStrokeAlignParams; + public final boolean mStrokeIdealPrefix; public final boolean mSlidingKeyInputPreviewEnabled; public final boolean mRecordInputTraces; public final int mKeyLongpressTimeout; @@ -474,6 +478,16 @@ public SettingsValues(final Context context, final SharedPreferences prefs, fina mMultipartRerecognizeTaps = prefs.getBoolean( Settings.PREF_MULTIPART_RERECOGNIZE_TAPS, Defaults.PREF_MULTIPART_RERECOGNIZE_TAPS); + mStrokeAlignParams = new StrokeAligner.Params( + StrokeAligner.Mode.fromPrefValue(prefs.getString( + Settings.PREF_STROKE_ALIGN_MODE, + Defaults.PREF_STROKE_ALIGN_MODE)), + prefs.getInt(Settings.PREF_STROKE_ALIGN_INTERVAL_MS, + Defaults.PREF_STROKE_ALIGN_INTERVAL_MS), + prefs.getInt(Settings.PREF_STROKE_ALIGN_GAP_MS, + Defaults.PREF_STROKE_ALIGN_GAP_MS)); + mStrokeIdealPrefix = prefs.getBoolean(Settings.PREF_STROKE_IDEAL_PREFIX, + Defaults.PREF_STROKE_IDEAL_PREFIX); mSuggestionStripHiddenPerUserSettings = mToolbarMode == ToolbarMode.HIDDEN || mToolbarMode == ToolbarMode.TOOLBAR_KEYS; mOverrideShowingSuggestions = mInputAttributes.mMayOverrideShowingSuggestions diff --git a/app/src/main/java/helium314/keyboard/settings/screens/TwoThumbTypingScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/TwoThumbTypingScreen.kt index e73389788..60a2e121c 100644 --- a/app/src/main/java/helium314/keyboard/settings/screens/TwoThumbTypingScreen.kt +++ b/app/src/main/java/helium314/keyboard/settings/screens/TwoThumbTypingScreen.kt @@ -59,6 +59,7 @@ fun TwoThumbTypingScreen( val backspaceBehavior = currentBackspaceBehavior(prefs) val dualThumbHinting = prefs.getBoolean(Settings.PREF_GESTURE_DUAL_THUMB_HINTING, Defaults.PREF_GESTURE_DUAL_THUMB_HINTING) val debugDrawPoints = prefs.getBoolean(Settings.PREF_GESTURE_DEBUG_DRAW_POINTS, Defaults.PREF_GESTURE_DEBUG_DRAW_POINTS) + val strokeAlignMode = prefs.getString(Settings.PREF_STROKE_ALIGN_MODE, Defaults.PREF_STROKE_ALIGN_MODE) val items = buildList { add(R.string.settings_category_two_thumb_typing_words) @@ -82,6 +83,16 @@ fun TwoThumbTypingScreen( } add(R.string.settings_category_two_thumb_typing_recognition) + // DUAL_POINTER only means anything to the native decoder; the Java fallback engine + // ignores pointer ids entirely, so offering the choice there would be misleading. + if (JniUtils.sHaveNativeGestureLib) { + add(Settings.PREF_STROKE_ALIGN_MODE) + } + add(Settings.PREF_STROKE_IDEAL_PREFIX) + if (JniUtils.sHaveNativeGestureLib && strokeAlignMode == "dual_pointer") { + add(Settings.PREF_STROKE_ALIGN_INTERVAL_MS) + add(Settings.PREF_STROKE_ALIGN_GAP_MS) + } add(Settings.PREF_GESTURE_DUAL_THUMB_HINTING) if (dualThumbHinting) { add(Settings.PREF_GESTURE_DUAL_THUMB_MIDLINE_PCT) @@ -201,6 +212,37 @@ fun createTwoThumbTypingSettings(context: Context) = listOf( R.string.two_thumb_point_hinting, R.string.two_thumb_point_hinting_summary) { SwitchPreference(it, Defaults.PREF_GESTURE_DUAL_THUMB_HINTING) }, + Setting(context, Settings.PREF_STROKE_ALIGN_MODE, + R.string.stroke_align_mode, R.string.stroke_align_mode_summary) { def -> val items = listOf( + stringResource(R.string.stroke_align_mode_connector) to "connector", + stringResource(R.string.stroke_align_mode_dual_pointer) to "dual_pointer", + ) + ListPreference(def, items, Defaults.PREF_STROKE_ALIGN_MODE) + }, + Setting(context, Settings.PREF_STROKE_IDEAL_PREFIX, + R.string.stroke_ideal_prefix, R.string.stroke_ideal_prefix_summary) { + SwitchPreference(it, Defaults.PREF_STROKE_IDEAL_PREFIX) + }, + Setting(context, Settings.PREF_STROKE_ALIGN_INTERVAL_MS, + R.string.stroke_align_interval, R.string.stroke_align_interval_summary) { def -> + SliderPreference( + name = def.title, + key = def.key, + default = Defaults.PREF_STROKE_ALIGN_INTERVAL_MS, + range = 5f..60f, + description = { value -> "${value.toInt()} ms" } + ) + }, + Setting(context, Settings.PREF_STROKE_ALIGN_GAP_MS, + R.string.stroke_align_gap, R.string.stroke_align_gap_summary) { def -> + SliderPreference( + name = def.title, + key = def.key, + default = Defaults.PREF_STROKE_ALIGN_GAP_MS, + range = 5f..200f, + description = { value -> "${value.toInt()} ms" } + ) + }, Setting(context, Settings.PREF_GESTURE_DUAL_THUMB_MIDLINE_PCT, R.string.gesture_dual_thumb_midline) { def -> SliderPreference( name = def.title, diff --git a/app/src/main/jni/tests/replay/two_pointer_track_test.cpp b/app/src/main/jni/tests/replay/two_pointer_track_test.cpp new file mode 100644 index 000000000..06e00728a --- /dev/null +++ b/app/src/main/jni/tests/replay/two_pointer_track_test.cpp @@ -0,0 +1,424 @@ +// SPDX-License-Identifier: Apache-2.0 +// +// Two-pointer track experiment harness — "does the AOSP gesture pipeline actually ingest two +// simultaneous strokes as two tracks, and what does pointer-id assignment do to that?" +// +// WHY THIS EXISTS +// --------------- +// LeanType composes one word out of several thumb fragments (tap->swipe, swipe->swipe). Today +// that is done SPATIALLY: WordComposer.setBatchInputPointers prepends the prior fragment's trail, +// re-timed, and InputPointers.appendAll() forces EVERY point to pointer id 0, so the recognizer +// sees one long single-pointer glide with a synthetic connector. +// +// But AOSP models TWO pointer tracks for gesture input (defines.h MAX_POINTER_COUNT_G == 2): +// DicTraverseSession holds ProximityInfoState[2] and seeds state i with pointerId i +// (dic_traverse_session.cpp initializeProximityInfoStates), and updateTouchPoints() keeps only the +// points whose pointerIds[k] == pointerId (proximity_info_state_utils.cpp). So pointer id -- not +// timing -- is what splits strokes into tracks. +// +// These tests drive the REAL ProximityInfoState (the same open AOSP preprocessing that is compiled +// into libjni_latinimegoogle.so) so the claims are measured, not inferred. What they CANNOT do is +// produce a recognized word: this tree has no gesture suggest policy +// (gesture_suggest_policy_factory.cpp returns null), so decoding quality is still device-only. +// +// TUNABLE: see TrackParams below -- pointer-id assignment, time policy, overlap, gap/interval, and +// tap->micro-stroke promotion are all knobs, and `TwoPointerSweep` prints a table across them. + +#include + +#include +#include +#include +#include + +#include "defines.h" +#include "suggest/core/layout/proximity_info.h" +#include "suggest/core/layout/proximity_info_state.h" +#include "suggest/policyimpl/typing/scoring_params.h" + +namespace latinime { +namespace replay { +namespace { + +// --------------------------------------------------------------------------- +// Keyboard model: 1080x310 QWERTY, same geometry as GestureReplayHostSeamTest. +// --------------------------------------------------------------------------- + +constexpr int kKeyboardWidth = 1080; +constexpr int kKeyboardHeight = 310; +constexpr int kGridWidth = 32; +constexpr int kGridHeight = 16; +constexpr int kKeyWidth = 108; +constexpr int kKeyHeight = 90; +constexpr int kKeyCount = 26; +constexpr const char *kLetters = "qwertyuiopasdfghjklzxcvbnm"; + +struct KeyGeometry { + int xs[kKeyCount], ys[kKeyCount], widths[kKeyCount], heights[kKeyCount], codes[kKeyCount]; + float sweetXs[kKeyCount], sweetYs[kKeyCount], radii[kKeyCount]; +}; + +KeyGeometry buildKeyGeometry() { + KeyGeometry g{}; + for (int i = 0; i < kKeyCount; ++i) { + const int row = i < 10 ? 0 : (i < 19 ? 1 : 2); + const int col = i < 10 ? i : (i < 19 ? i - 10 : i - 19); + const int rowOffset = row == 0 ? 0 : (row == 1 ? kKeyWidth / 2 : kKeyWidth); + g.xs[i] = rowOffset + col * kKeyWidth; + g.ys[i] = row * kKeyHeight; + g.widths[i] = kKeyWidth; + g.heights[i] = kKeyHeight; + g.codes[i] = kLetters[i]; + g.sweetXs[i] = g.xs[i] + kKeyWidth / 2.0f; + g.sweetYs[i] = g.ys[i] + kKeyHeight / 2.0f; + g.radii[i] = kKeyWidth / 2.0f; + } + return g; +} + +// Owns the arrays ProximityInfo borrows. +class Qwerty { + public: + Qwerty() + : mGeom(buildKeyGeometry()), + mProximityChars(kGridWidth * kGridHeight * MAX_PROXIMITY_CHARS_SIZE, + NOT_A_CODE_POINT), + mInfo(kKeyboardWidth, kKeyboardHeight, kGridWidth, kGridHeight, kKeyWidth, kKeyHeight, + mProximityChars.data(), static_cast(mProximityChars.size()), kKeyCount, + mGeom.xs, mGeom.ys, mGeom.widths, mGeom.heights, mGeom.codes, + mGeom.sweetXs, mGeom.sweetYs, mGeom.radii) {} + + const ProximityInfo *info() const { return &mInfo; } + + void centerOf(const char c, int *outX, int *outY) const { + for (int i = 0; i < kKeyCount; ++i) { + if (kLetters[i] == c) { + *outX = mGeom.xs[i] + kKeyWidth / 2; + *outY = mGeom.ys[i] + kKeyHeight / 2; + return; + } + } + *outX = -1; + *outY = -1; + } + + private: + KeyGeometry mGeom; + std::vector mProximityChars; + ProximityInfo mInfo; +}; + +// --------------------------------------------------------------------------- +// TUNABLE PARAMETERS <-- the knobs to play with +// --------------------------------------------------------------------------- + +struct TrackParams { + // How pointer ids are assigned to the two fragments. + enum PointerMode { + ALL_ZERO, // today's behaviour: InputPointers.appendAll() forces id 0 for everything + SPLIT_0_1, // the proposal: fragment A -> id 0, fragment B -> id 1 + SPLIT_1_0, // reversed, to test the "state 0 must be used" constraint + ALL_ONE, // pathological: nothing carries id 0 + ALL_TWO, // pathological: a third finger (id >= 2) + }; + // How the two fragments are laid out on the time axis. + enum TimeMode { + GLOBAL_MONOTONIC, // B starts after A ends (today, via the re-timed extend base) + PER_POINTER_RESTART, // B's clock restarts at 0 (what raw per-stroke stamps look like) + OVERLAPPED, // B overlaps A by overlapPct of A's duration ("simultaneous") + }; + + PointerMode pointerMode = ALL_ZERO; + TimeMode timeMode = GLOBAL_MONOTONIC; + int overlapPct = 0; // OVERLAPPED only: 0 = sequential, 100 = fully co-timed + int gapMs = 60; // WordComposer.EXTEND_BASE_GAP_BEFORE_NEW_MS + int intervalMs = 25; // WordComposer.EXTEND_BASE_POINT_INTERVAL_MS + int samplesPerKeyHop = 4; // densification along each inter-key segment + // Tap -> micro-stroke promotion (IdealPrefixTrailBuilder, issue #99/B7b). + bool promoteTaps = true; + int tapArcRadiusDivisor = 6; // radius = keyWidth / divisor +}; + +struct Trace { + std::vector xs, ys, times, ids; + int fragmentASize = 0; + int size() const { return static_cast(xs.size()); } +}; + +// Trace a word's key centres, densified — mirrors IdealPrefixTrailBuilder.build(). +void appendWordPath(const Qwerty &kb, const std::string &word, const TrackParams ¶ms, + std::vector *xs, std::vector *ys) { + std::vector cx, cy; + for (const char c : word) { + int x = 0, y = 0; + kb.centerOf(c, &x, &y); + if (x < 0) continue; + cx.push_back(x); + cy.push_back(y); + } + if (cx.empty()) return; + if (cx.size() == 1) { + if (params.promoteTaps) { + // Out-and-back micro-stroke so the recognizer sees a vertex, not a lone point. + const int r = std::max(1, kKeyWidth / std::max(1, params.tapArcRadiusDivisor)); + const int pxs[4] = {cx[0] - r, cx[0], cx[0] + r, cx[0]}; + for (int i = 0; i < 4; ++i) { + xs->push_back(pxs[i]); + ys->push_back(cy[0]); + } + } else { + xs->push_back(cx[0]); + ys->push_back(cy[0]); + } + return; + } + xs->push_back(cx[0]); + ys->push_back(cy[0]); + for (std::size_t i = 1; i < cx.size(); ++i) { + const int steps = std::max(1, params.samplesPerKeyHop); + for (int s = 1; s <= steps; ++s) { + const float t = static_cast(s) / steps; + xs->push_back(static_cast(std::lround(cx[i - 1] + (cx[i] - cx[i - 1]) * t))); + ys->push_back(static_cast(std::lround(cy[i - 1] + (cy[i] - cy[i - 1]) * t))); + } + } +} + +Trace buildTwoFragmentTrace(const Qwerty &kb, const std::string &fragA, const std::string &fragB, + const TrackParams ¶ms) { + Trace t; + std::vector ax, ay, bx, by; + appendWordPath(kb, fragA, params, &ax, &ay); + appendWordPath(kb, fragB, params, &bx, &by); + + const int nA = static_cast(ax.size()); + const int nB = static_cast(bx.size()); + t.fragmentASize = nA; + + // Fragment A always runs 0, interval, 2*interval, ... + const int aDuration = std::max(0, (nA - 1) * params.intervalMs); + int bStart = 0; + switch (params.timeMode) { + case TrackParams::GLOBAL_MONOTONIC: + bStart = aDuration + params.gapMs; + break; + case TrackParams::PER_POINTER_RESTART: + bStart = 0; + break; + case TrackParams::OVERLAPPED: + bStart = aDuration - (aDuration * params.overlapPct) / 100; + break; + } + + int idA = 0, idB = 0; + switch (params.pointerMode) { + case TrackParams::ALL_ZERO: idA = 0; idB = 0; break; + case TrackParams::SPLIT_0_1: idA = 0; idB = 1; break; + case TrackParams::SPLIT_1_0: idA = 1; idB = 0; break; + case TrackParams::ALL_ONE: idA = 1; idB = 1; break; + case TrackParams::ALL_TWO: idA = 2; idB = 2; break; + } + + for (int i = 0; i < nA; ++i) { + t.xs.push_back(ax[i]); + t.ys.push_back(ay[i]); + t.times.push_back(i * params.intervalMs); + t.ids.push_back(idA); + } + for (int i = 0; i < nB; ++i) { + t.xs.push_back(bx[i]); + t.ys.push_back(by[i]); + t.times.push_back(bStart + i * params.intervalMs); + t.ids.push_back(idB); + } + return t; +} + +struct TrackStats { + bool used = false; + int sampledSize = 0; + int minRawIndex = -1; + int maxRawIndex = -1; + float minSpeedRate = 0.0f; +}; + +// Drive the REAL AOSP preprocessing for one pointer track. +TrackStats analyzeTrack(const Qwerty &kb, const Trace &trace, const int pointerId) { + // Heap-allocated: ProximityInfoState is large. + auto state = std::unique_ptr(new ProximityInfoState()); + const std::vector locale; + std::vector inputCodes(trace.size(), NOT_A_CODE_POINT); + + state->initInputParams(pointerId, ScoringParams::MAX_SPATIAL_DISTANCE, kb.info(), + inputCodes.data(), trace.size(), trace.xs.data(), trace.ys.data(), trace.times.data(), + trace.ids.data(), true /* isGeometric */, &locale); + + TrackStats s; + s.used = state->isUsed(); + s.sampledSize = state->size(); + for (int i = 0; i < s.sampledSize; ++i) { + const int raw = state->getInputIndexOfSampledPoint(i); + if (s.minRawIndex < 0 || raw < s.minRawIndex) s.minRawIndex = raw; + if (raw > s.maxRawIndex) s.maxRawIndex = raw; + const float rate = state->getSpeedRate(i); + if (i == 0 || rate < s.minSpeedRate) s.minSpeedRate = rate; + } + return s; +} + +// "technology" split the way LeanType composes it: swipe "tech", then swipe "nology". +const char *kFragA = "tech"; +const char *kFragB = "nology"; + +// ============================================================================= +// 1. Today's behaviour: everything is pointer 0, so the second track is dead. +// ============================================================================= + +TEST(TwoPointerTrackTest, AllPointsPointerZeroLeavesSecondTrackUnused) { + const Qwerty kb; + TrackParams params; + params.pointerMode = TrackParams::ALL_ZERO; + const Trace trace = buildTwoFragmentTrace(kb, kFragA, kFragB, params); + + const TrackStats t0 = analyzeTrack(kb, trace, 0); + const TrackStats t1 = analyzeTrack(kb, trace, 1); + + EXPECT_TRUE(t0.used) << "track 0 must absorb the whole merged trail"; + EXPECT_GT(t0.sampledSize, 0); + // Track 0 spans BOTH fragments — one long glide with a connector jump in the middle. + EXPECT_LT(t0.minRawIndex, trace.fragmentASize); + EXPECT_GE(t0.maxRawIndex, trace.fragmentASize); + + // This is the finding: InputPointers.appendAll()'s hardcoded id 0 makes the decoder's + // second track permanently unused, so the whole multi-part problem has to be solved + // spatially (connectors) instead. + EXPECT_FALSE(t1.used) << "track 1 must be empty when every point carries id 0"; + EXPECT_EQ(0, t1.sampledSize); +} + +// ============================================================================= +// 2. The proposal: split ids 0/1 and BOTH native tracks light up. +// ============================================================================= + +TEST(TwoPointerTrackTest, SplitPointerIdsPopulateBothTracks) { + const Qwerty kb; + TrackParams params; + params.pointerMode = TrackParams::SPLIT_0_1; + const Trace trace = buildTwoFragmentTrace(kb, kFragA, kFragB, params); + + const TrackStats t0 = analyzeTrack(kb, trace, 0); + const TrackStats t1 = analyzeTrack(kb, trace, 1); + + ASSERT_TRUE(t0.used); + ASSERT_TRUE(t1.used) << "track 1 SHOULD be populated once fragment B carries pointer id 1"; + EXPECT_GT(t0.sampledSize, 0); + EXPECT_GT(t1.sampledSize, 0); + + // Each track sees only its own fragment: no connector, no spatial jump. + EXPECT_LT(t0.maxRawIndex, trace.fragmentASize) << "track 0 must not contain fragment B points"; + EXPECT_GE(t1.minRawIndex, trace.fragmentASize) << "track 1 must not contain fragment A points"; +} + +// ============================================================================= +// 3. Pathological id assignments (why normalisation is mandatory). +// ============================================================================= + +// suggest.cpp: `if (!traverseSession->getProximityInfoState(0)->isUsed()) return;` +// If no point carries id 0 the whole search bails out and returns zero suggestions. +TEST(TwoPointerTrackTest, NoPointerZeroLeavesTrackZeroEmpty) { + const Qwerty kb; + TrackParams params; + params.pointerMode = TrackParams::ALL_ONE; + const Trace trace = buildTwoFragmentTrace(kb, kFragA, kFragB, params); + + EXPECT_FALSE(analyzeTrack(kb, trace, 0).used) + << "track 0 empty => Suggest::initializeSearch early-returns => no suggestions"; + EXPECT_TRUE(analyzeTrack(kb, trace, 1).used); +} + +// Only states 0 and 1 exist (MAX_POINTER_COUNT_G == 2), so a third finger is dropped silently. +TEST(TwoPointerTrackTest, PointerIdTwoIsSilentlyDropped) { + const Qwerty kb; + TrackParams params; + params.pointerMode = TrackParams::ALL_TWO; + const Trace trace = buildTwoFragmentTrace(kb, kFragA, kFragB, params); + + EXPECT_FALSE(analyzeTrack(kb, trace, 0).used); + EXPECT_FALSE(analyzeTrack(kb, trace, 1).used) << "id >= 2 reaches no track at all"; +} + +// ============================================================================= +// 4. Time policy: does it change track membership at all? +// ============================================================================= + +TEST(TwoPointerTrackTest, TimePolicyDoesNotChangeTrackMembership) { + const Qwerty kb; + const TrackParams::TimeMode modes[] = {TrackParams::GLOBAL_MONOTONIC, + TrackParams::PER_POINTER_RESTART, TrackParams::OVERLAPPED}; + + int baselineT0 = -1, baselineT1 = -1; + for (const auto mode : modes) { + TrackParams params; + params.pointerMode = TrackParams::SPLIT_0_1; + params.timeMode = mode; + params.overlapPct = 100; + const Trace trace = buildTwoFragmentTrace(kb, kFragA, kFragB, params); + + const TrackStats t0 = analyzeTrack(kb, trace, 0); + const TrackStats t1 = analyzeTrack(kb, trace, 1); + EXPECT_TRUE(t0.used); + EXPECT_TRUE(t1.used); + if (baselineT0 < 0) { + baselineT0 = t0.sampledSize; + baselineT1 = t1.sampledSize; + } + // Membership is decided purely by pointer id: shifting/overlapping the clocks cannot + // move a point from one track to the other. + EXPECT_EQ(baselineT0, t0.sampledSize) << "time mode changed track-0 membership"; + EXPECT_EQ(baselineT1, t1.sampledSize) << "time mode changed track-1 membership"; + } +} + +// ============================================================================= +// 5. Sweep: prints the parameter table so the knobs can be explored by hand. +// Run with: ctest --test-dir -R TwoPointerSweep --output-on-failure +// ============================================================================= + +TEST(TwoPointerSweep, PrintTrackTable) { + const Qwerty kb; + struct Row { const char *name; TrackParams::PointerMode pm; TrackParams::TimeMode tm; int overlap; }; + const Row rows[] = { + {"today (all id 0, monotonic)", TrackParams::ALL_ZERO, TrackParams::GLOBAL_MONOTONIC, 0}, + {"today (all id 0, restart) ", TrackParams::ALL_ZERO, TrackParams::PER_POINTER_RESTART, 0}, + {"split (0/1, monotonic) ", TrackParams::SPLIT_0_1, TrackParams::GLOBAL_MONOTONIC, 0}, + {"split (0/1, restart) ", TrackParams::SPLIT_0_1, TrackParams::PER_POINTER_RESTART, 0}, + {"split (0/1, overlap 50%) ", TrackParams::SPLIT_0_1, TrackParams::OVERLAPPED, 50}, + {"split (0/1, overlap 100%) ", TrackParams::SPLIT_0_1, TrackParams::OVERLAPPED, 100}, + {"split (1/0 reversed) ", TrackParams::SPLIT_1_0, TrackParams::GLOBAL_MONOTONIC, 0}, + {"all id 1 ", TrackParams::ALL_ONE, TrackParams::GLOBAL_MONOTONIC, 0}, + {"all id 2 ", TrackParams::ALL_TWO, TrackParams::GLOBAL_MONOTONIC, 0}, + }; + std::printf("\n%-30s | t0.used t0.n [raw range] minSpeed | t1.used t1.n [raw range] minSpeed\n", + "config"); + std::printf("%s\n", std::string(120, '-').c_str()); + for (const Row &r : rows) { + TrackParams params; + params.pointerMode = r.pm; + params.timeMode = r.tm; + params.overlapPct = r.overlap; + const Trace trace = buildTwoFragmentTrace(kb, kFragA, kFragB, params); + const TrackStats t0 = analyzeTrack(kb, trace, 0); + const TrackStats t1 = analyzeTrack(kb, trace, 1); + std::printf("%-30s | %d %3d [%3d..%3d] %8.3f | %d %3d [%3d..%3d] %8.3f\n", + r.name, t0.used ? 1 : 0, t0.sampledSize, t0.minRawIndex, t0.maxRawIndex, + t0.minSpeedRate, t1.used ? 1 : 0, t1.sampledSize, t1.minRawIndex, t1.maxRawIndex, + t1.minSpeedRate); + } + std::printf("\n(raw index range shows which raw samples reached each track; fragment A is " + "raw [0..%d))\n\n", buildTwoFragmentTrace(kb, kFragA, kFragB, TrackParams()).fragmentASize); + SUCCEED(); +} + +} // namespace +} // namespace replay +} // namespace latinime diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml index 4be327fbd..4e0b628d2 100644 --- a/app/src/main/res/values/strings.xml +++ b/app/src/main/res/values/strings.xml @@ -378,6 +378,15 @@ Delete whole word Improve two-thumb recognition (experimental) Adds synthetic hints for the recognizer when both thumbs are used. It may help two-thumb gestures, but can hurt accuracy if the hand split is wrong. + Joining word parts + How an earlier part of the word is fed to the recognizer together with the part you are swiping now. + One joined trail + Two separate thumb tracks (experimental) + Word-part trail speed Spacing between the replayed points of the earlier word part. Lower is a faster imagined swipe. + Pause before the new part + Imagined pause between the earlier word part and the part you are swiping now. + Redraw earlier word parts cleanly + Replaces the earlier part of the word with a tidy path through its key centres, and turns a single tap into a small swipe, so the recognizer sees one believable whole-word gesture. Keep insight across word parts Keep the trail visible across the parts of one word, clearing when the next word starts. Turn off to clear at each new swipe. diff --git a/app/src/test/java/helium314/keyboard/keyboard/internal/PointerIdNormalizerTest.kt b/app/src/test/java/helium314/keyboard/keyboard/internal/PointerIdNormalizerTest.kt new file mode 100644 index 000000000..fd9575f7c --- /dev/null +++ b/app/src/test/java/helium314/keyboard/keyboard/internal/PointerIdNormalizerTest.kt @@ -0,0 +1,98 @@ +package helium314.keyboard.keyboard.internal + +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertNotEquals + +/** + * Pins the pointer-id → decoder-track-slot mapping. + * + * The native gesture decoder keeps exactly two per-pointer tracks and seeds track *i* with pointer + * id *i* (`jni/src/defines.h` MAX_POINTER_COUNT_G, `dic_traverse_session.cpp`). Two failure modes + * follow, both measured against the real AOSP preprocessing in + * `jni/tests/replay/two_pointer_track_test.cpp`: + * + * - no point carrying id 0 ⇒ track 0 unused ⇒ `Suggest::initializeSearch` returns early ⇒ + * **zero suggestions**; + * - any id >= 2 reaches no track at all. + * + * Android hands out the lowest free pointer index, so a stroke's raw id is not guaranteed to be 0. + */ +class PointerIdNormalizerTest { + + @Test + fun `single finger keeps slot zero`() { + val n = PointerIdNormalizer() + assertEquals(0, n.slotFor(0)) + assertEquals(0, n.slotFor(0)) + assertEquals(1, n.trackedPointerCount()) + } + + @Test + fun `two thumbs in order map to slots zero and one`() { + val n = PointerIdNormalizer() + assertEquals(0, n.slotFor(0)) + assertEquals(1, n.slotFor(1)) + // Stable across repeated lookups — ids must not drift mid-gesture, because + // checkAndReturnIsContinuousSuggestionPossible does not compare pointer ids. + assertEquals(0, n.slotFor(0)) + assertEquals(1, n.slotFor(1)) + assertEquals(2, n.trackedPointerCount()) + } + + /** + * The regression this class exists for: thumb A goes down (id 0), thumb B goes down (id 1), + * thumb A lifts, and thumb B swipes on alone still carrying raw id 1. Before normalisation + * every aggregated point carried id 1, track 0 stayed empty and the gesture produced no + * suggestions at all. + */ + @Test + fun `gesture whose only pointer is raw id one still anchors track zero`() { + val n = PointerIdNormalizer() + val slot = n.slotFor(1) + assertEquals(0, slot, "the first pointer to contribute must anchor track 0") + assertNotEquals(1, slot) + } + + @Test + fun `first seen order decides the slot, not the raw id value`() { + val n = PointerIdNormalizer() + assertEquals(0, n.slotFor(3)) + assertEquals(1, n.slotFor(2)) + assertEquals(0, n.slotFor(3)) + assertEquals(1, n.slotFor(2)) + } + + /** A third finger still falls outside the decoder's two tracks — unchanged behaviour. */ + @Test + fun `third distinct pointer gets a slot the decoder will ignore`() { + val n = PointerIdNormalizer() + n.slotFor(0) + n.slotFor(1) + assertEquals(2, n.slotFor(7)) + assertEquals(3, n.trackedPointerCount()) + } + + @Test + fun `reset forgets the mapping so the next gesture starts at slot zero`() { + val n = PointerIdNormalizer() + n.slotFor(4) + n.slotFor(9) + assertEquals(2, n.trackedPointerCount()) + n.reset() + assertEquals(0, n.trackedPointerCount()) + assertEquals(0, n.slotFor(9), "a fresh gesture must re-anchor track 0") + } + + @Test + fun `passes ids through once more than eight distinct pointers appear`() { + val n = PointerIdNormalizer() + for (i in 0 until 8) { + assertEquals(i, n.slotFor(100 + i)) + } + // Beyond the tracked capacity we stop renumbering; the decoder discards these anyway. + assertEquals(500, n.slotFor(500)) + // Already-tracked ids keep their slots. + assertEquals(0, n.slotFor(100)) + } +} diff --git a/app/src/test/java/helium314/keyboard/latin/gesture/IdealPrefixTrailBuilderTest.kt b/app/src/test/java/helium314/keyboard/latin/gesture/IdealPrefixTrailBuilderTest.kt new file mode 100644 index 000000000..e34884586 --- /dev/null +++ b/app/src/test/java/helium314/keyboard/latin/gesture/IdealPrefixTrailBuilderTest.kt @@ -0,0 +1,146 @@ +package helium314.keyboard.latin.gesture + +import helium314.keyboard.ShadowInputMethodManager2 +import helium314.keyboard.ShadowProximityInfo +import helium314.keyboard.keyboard.Key +import helium314.keyboard.keyboard.Keyboard +import helium314.keyboard.keyboard.KeyboardId +import helium314.keyboard.keyboard.KeyboardLayoutSet +import helium314.keyboard.keyboard.internal.KeyboardParams +import helium314.keyboard.latin.common.InputPointers +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertNotNull +import kotlin.test.assertNull +import kotlin.test.assertTrue +import org.junit.runner.RunWith +import org.robolectric.RobolectricTestRunner +import org.robolectric.annotation.Config + +/** + * Covers the synthetic prefix trail, including the tap → micro-stroke promotion. + * + * Recognition *quality* is not testable here (there is no gesture policy in this tree), so these + * assert the geometry contract only: a tap becomes a stroke with a vertex, multi-letter prefixes + * are densified, and unbuildable inputs return null so the caller falls back to the raw trail. + */ +@RunWith(RobolectricTestRunner::class) +@Config(shadows = [ShadowInputMethodManager2::class, ShadowProximityInfo::class]) +class IdealPrefixTrailBuilderTest { + + /** A single row of 100x100 letter keys, laid out left to right. */ + private fun keyboardFor(letters: String): Keyboard { + val params = KeyboardParams().apply { + mId = KeyboardLayoutSet.getFakeKeyboardId(KeyboardId.ELEMENT_ALPHABET) + mOccupiedWidth = letters.length * 100 + mOccupiedHeight = 100 + mBaseWidth = mOccupiedWidth + mBaseHeight = mOccupiedHeight + mMostCommonKeyWidth = 100 + mMostCommonKeyHeight = 100 + GRID_WIDTH = letters.length + GRID_HEIGHT = 1 + } + letters.forEachIndexed { index, letter -> + params.onAddKey(Key( + letter.toString(), null, letter.code, null, null, + 0, Key.BACKGROUND_TYPE_NORMAL, + index * 100, 0, 100, 100, 0, 0, + )) + } + return Keyboard(params) + } + + private fun keyboard() = keyboardFor("techsplo") + + private fun InputPointers.xs() = xCoordinates.take(pointerSize) + private fun InputPointers.ys() = yCoordinates.take(pointerSize) + private fun InputPointers.ids() = pointerIds.take(pointerSize) + + @Test + fun `null or empty input yields null so the caller keeps the raw trail`() { + val kb = keyboard() + assertNull(IdealPrefixTrailBuilder.build(null, kb)) + assertNull(IdealPrefixTrailBuilder.build("", kb)) + assertNull(IdealPrefixTrailBuilder.build("hello", null)) + } + + @Test + fun `a word with no mappable letters yields null`() { + assertNull(IdealPrefixTrailBuilder.build("123", keyboard())) + } + + /** The tap-promotion claim: one letter must come back as a stroke, not a point. */ + @Test + fun `a single letter prefix becomes an out-and-back micro-stroke`() { + val trail = assertNotNull(IdealPrefixTrailBuilder.build("s", keyboard())) + + assertEquals(4, trail.pointerSize, "a tap must be promoted to a 4-point micro-stroke") + val xs = trail.xs() + val ys = trail.ys() + // Out and back around the key centre: left, centre, right, centre. + assertEquals(xs[1], xs[3], "the micro-stroke must return to the key centre") + assertTrue(xs[0] < xs[1], "first point must sit left of centre") + assertTrue(xs[2] > xs[1], "third point must sit right of centre") + assertTrue(ys.all { it == ys[0] }, "the micro-arc stays on one row") + // A real vertex, not a degenerate zero-length wiggle. + assertTrue(xs[2] - xs[0] > 1, "the micro-stroke must have non-trivial extent") + } + + @Test + fun `a multi letter prefix is densified beyond one point per key`() { + val trail = assertNotNull(IdealPrefixTrailBuilder.build("tech", keyboard())) + assertTrue(trail.pointerSize > 4, + "expected interpolated samples between key centres, got ${trail.pointerSize}") + } + + @Test + fun `every synthesised point is on the base track`() { + for (word in listOf("s", "tech", "hello")) { + val trail = assertNotNull(IdealPrefixTrailBuilder.build(word, keyboard())) + assertTrue(trail.ids().all { it == StrokeAligner.BASE_POINTER_ID }, + "$word: the prefix trail must stay on decoder track 0") + } + } + + @Test + fun `punctuation is skipped but an unmappable letter forces a fallback`() { + // Apostrophes legitimately have no place on the trail. + val withApostrophe = assertNotNull(IdealPrefixTrailBuilder.build("to'p", keyboard())) + val without = assertNotNull(IdealPrefixTrailBuilder.build("top", keyboard())) + assertEquals(without.pointerSize, withApostrophe.pointerSize) + assertEquals(without.xs(), withApostrophe.xs()) + + // A letter that isn't on this keyboard would leave a hole in the synthetic path, which is + // worse than the raw trail — so the builder bails out and the caller falls back. + assertNull(IdealPrefixTrailBuilder.build("tzch", keyboard())) + assertNull(IdealPrefixTrailBuilder.build("téch", keyboard())) + } + + @Test + fun `case does not change the produced geometry`() { + val lower = assertNotNull(IdealPrefixTrailBuilder.build("tech", keyboard())) + val upper = assertNotNull(IdealPrefixTrailBuilder.build("TECH", keyboard())) + assertEquals(lower.xs(), upper.xs()) + assertEquals(lower.ys(), upper.ys()) + } + + /** The whole point of the builder: feed StrokeAligner a stroke-like base. */ + @Test + fun `the synthesised tap trail survives a StrokeAligner merge as a real stroke`() { + val trail = assertNotNull(IdealPrefixTrailBuilder.build("s", keyboard())) + val current = InputPointers(8).apply { + addPointer(500, 100, 0, 1000) + addPointer(520, 105, 0, 1025) + } + val out = InputPointers(16) + StrokeAligner.merge(out, trail, current, + StrokeAligner.Params(StrokeAligner.Mode.DUAL_POINTER, 25, 60)) + + assertEquals(6, out.pointerSize) + assertEquals(listOf(0, 0, 0, 0, 1, 1), out.ids()) + out.times.take(out.pointerSize).zipWithNext().forEach { (a, b) -> + assertTrue(b >= a, "merged trail must stay monotonic in time") + } + } +} diff --git a/app/src/test/java/helium314/keyboard/latin/gesture/StrokeAlignerTest.kt b/app/src/test/java/helium314/keyboard/latin/gesture/StrokeAlignerTest.kt new file mode 100644 index 000000000..03192fc4d --- /dev/null +++ b/app/src/test/java/helium314/keyboard/latin/gesture/StrokeAlignerTest.kt @@ -0,0 +1,276 @@ +package helium314.keyboard.latin.gesture + +import helium314.keyboard.latin.common.InputPointers +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertTrue + +/** + * Pins [StrokeAligner]'s merge contract. + * + * The invariants asserted here are not stylistic — each maps to a measured property of the native + * decoder (`jni/tests/replay/two_pointer_track_test.cpp`, `docs/TWO_THUMB_TEMPORAL_ALIGNMENT.md`): + * + * - track 0 must be non-empty or `Suggest::initializeSearch` returns zero suggestions; + * - only pointer ids 0 and 1 reach a decoder track at all; + * - timestamps must be globally monotonic, because the decoder's speed/beeline features walk the + * raw arrays across the track boundary and a decreasing timestamp yields a negative duration. + */ +class StrokeAlignerTest { + + private fun pointers(vararg triples: Triple, id: Int = 0) = + InputPointers(16).apply { + triples.forEach { (x, y, t) -> addPointer(x, y, id, t) } + } + + private fun base() = pointers( + Triple(10, 10, 0), + Triple(20, 12, 0), + Triple(30, 14, 0), // tap-sourced coords carry a time=0 sentinel + ) + + private fun current() = pointers( + Triple(100, 50, 1000), + Triple(120, 55, 1025), + Triple(140, 60, 1050), + ) + + private fun InputPointers.idsList() = pointerIds.take(pointerSize) + private fun InputPointers.timesList() = times.take(pointerSize) + private fun InputPointers.xsList() = xCoordinates.take(pointerSize) + + // ---- shared invariants ------------------------------------------------- + + @Test + fun `connector mode keeps every point on track zero`() { + val out = InputPointers(16) + StrokeAligner.merge(out, base(), current(), + StrokeAligner.Params(StrokeAligner.Mode.CONNECTOR, 25, 60)) + + assertEquals(6, out.pointerSize) + assertTrue(out.idsList().all { it == 0 }, "connector mode must not split tracks") + } + + @Test + fun `dual pointer mode puts the base on track zero and the new stroke on track one`() { + val out = InputPointers(16) + StrokeAligner.merge(out, base(), current(), + StrokeAligner.Params(StrokeAligner.Mode.DUAL_POINTER, 25, 60)) + + assertEquals(listOf(0, 0, 0, 1, 1, 1), out.idsList()) + } + + @Test + fun `no mode ever emits a pointer id the decoder would discard`() { + for (mode in StrokeAligner.Mode.entries) { + val out = InputPointers(16) + StrokeAligner.merge(out, base(), current(), StrokeAligner.Params(mode, 25, 60)) + assertTrue(out.idsList().all { it == 0 || it == 1 }, + "$mode emitted an id outside [0,1]; those reach no decoder track") + } + } + + @Test + fun `track zero is always populated so the search does not bail out`() { + for (mode in StrokeAligner.Mode.entries) { + val out = InputPointers(16) + StrokeAligner.merge(out, base(), current(), StrokeAligner.Params(mode, 25, 60)) + assertTrue(out.idsList().contains(0), + "$mode left track 0 empty; Suggest::initializeSearch would return no suggestions") + } + } + + @Test + fun `timestamps are globally monotonic in every mode`() { + for (mode in StrokeAligner.Mode.entries) { + val out = InputPointers(16) + StrokeAligner.merge(out, base(), current(), StrokeAligner.Params(mode, 25, 60)) + val times = out.timesList() + times.zipWithNext().forEach { (a, b) -> + assertTrue(b >= a, "$mode produced a decreasing timestamp ($a -> $b); " + + "the decoder's speed features would compute a negative duration") + } + } + } + + @Test + fun `base timestamps are re-synthesised to land just before the new stroke`() { + val out = InputPointers(16) + StrokeAligner.merge(out, base(), current(), + StrokeAligner.Params(StrokeAligner.Mode.CONNECTOR, 25, 60)) + + // current starts at 1000, gap 60 => base ends at 940, interval 25 over 3 points. + assertEquals(listOf(890, 915, 940, 1000, 1025, 1050), out.timesList()) + } + + @Test + fun `geometry is preserved and ordered base-then-current`() { + val out = InputPointers(16) + StrokeAligner.merge(out, base(), current(), + StrokeAligner.Params(StrokeAligner.Mode.DUAL_POINTER, 25, 60)) + + assertEquals(listOf(10, 20, 30, 100, 120, 140), out.xsList()) + } + + // ---- knobs ------------------------------------------------------------- + + @Test + fun `interval and gap knobs move the base timeline in dual pointer mode`() { + val out = InputPointers(16) + StrokeAligner.merge(out, base(), current(), + StrokeAligner.Params(StrokeAligner.Mode.DUAL_POINTER, 10, 100)) + + // gap 100 => base ends at 900; interval 10 over 3 points. + assertEquals(listOf(880, 890, 900, 1000, 1025, 1050), out.timesList()) + } + + @Test + fun `non-positive knobs are clamped so the base cannot go non-monotonic`() { + val params = StrokeAligner.Params(StrokeAligner.Mode.CONNECTOR, 0, -5) + assertTrue(params.basePointIntervalMs >= 1) + assertTrue(params.gapBeforeNewMs >= 1) + + val out = InputPointers(16) + StrokeAligner.merge(out, base(), current(), params) + out.timesList().zipWithNext().forEach { (a, b) -> assertTrue(b >= a) } + } + + @Test + fun `defaults reproduce the historical connector behaviour`() { + val defaults = StrokeAligner.Params.defaults() + assertEquals(StrokeAligner.Mode.CONNECTOR, defaults.mode) + assertEquals(25, defaults.basePointIntervalMs) + assertEquals(60, defaults.gapBeforeNewMs) + } + + @Test + fun `unknown or missing pref values fall back to connector`() { + assertEquals(StrokeAligner.Mode.CONNECTOR, StrokeAligner.Mode.fromPrefValue("connector")) + assertEquals(StrokeAligner.Mode.CONNECTOR, StrokeAligner.Mode.fromPrefValue("nonsense")) + assertEquals(StrokeAligner.Mode.CONNECTOR, StrokeAligner.Mode.fromPrefValue(null)) + assertEquals(StrokeAligner.Mode.DUAL_POINTER, + StrokeAligner.Mode.fromPrefValue("dual_pointer")) + } + + // ---- degenerate inputs ------------------------------------------------- + + @Test + fun `mode changes ids only, never geometry or timing`() { + // The Java SwipeGestureEngine fallback (used when the native lib is absent, e.g. the + // offlinelite flavor) flattens all points into one path and ignores pointer ids, so + // DUAL_POINTER must be invisible to it. + val connector = InputPointers(16) + val dual = InputPointers(16) + StrokeAligner.merge(connector, base(), current(), + StrokeAligner.Params(StrokeAligner.Mode.CONNECTOR, 25, 60)) + StrokeAligner.merge(dual, base(), current(), + StrokeAligner.Params(StrokeAligner.Mode.DUAL_POINTER, 25, 60)) + + assertEquals(connector.pointerSize, dual.pointerSize) + assertEquals(connector.xsList(), dual.xsList()) + assertEquals(connector.ys().take(connector.pointerSize), dual.ys().take(dual.pointerSize)) + assertEquals(connector.timesList(), dual.timesList()) + assertTrue(connector.idsList() != dual.idsList(), "only the ids should differ") + } + + private fun InputPointers.ys() = yCoordinates.toList() + + @Test + fun `connector mode ignores the timing knobs so it always means historical behaviour`() { + // The sliders are only shown for DUAL_POINTER; tuning them there and switching back must + // not silently change what "one joined trail" does. + val tuned = InputPointers(16) + StrokeAligner.merge(tuned, base(), current(), + StrokeAligner.Params(StrokeAligner.Mode.CONNECTOR, 5, 200)) + + assertEquals(listOf(890, 915, 940, 1000, 1025, 1050), tuned.timesList()) + } + + @Test + fun `dual pointer preserves an already multi-pointer current stroke`() { + // A genuinely simultaneous two-thumb stroke already occupies both decoder tracks; + // flattening it onto track 1 would destroy that structure. + val simultaneous = InputPointers(8).apply { + addPointer(100, 50, 0, 1000) + addPointer(300, 50, 1, 1005) + addPointer(120, 55, 0, 1025) + addPointer(320, 55, 1, 1030) + } + val out = InputPointers(16) + StrokeAligner.merge(out, base(), simultaneous, + StrokeAligner.Params(StrokeAligner.Mode.DUAL_POINTER, 25, 60)) + + assertEquals(listOf(0, 0, 0, 0, 1, 0, 1), out.idsList()) + assertTrue(out.idsList().all { it == 0 || it == 1 }) + out.timesList().zipWithNext().forEach { (a, b) -> assertTrue(b >= a) } + } + + @Test + fun `an empty base copies the current stroke through untouched`() { + // This is the ordinary single-swipe path, including genuinely simultaneous two-thumb + // input, whose real MotionEvent pointer ids must survive unchanged. + val simultaneous = InputPointers(16).apply { + addPointer(1, 1, 0, 10) + addPointer(2, 2, 1, 12) + } + val out = InputPointers(16) + StrokeAligner.merge(out, InputPointers(4), simultaneous, + StrokeAligner.Params(StrokeAligner.Mode.DUAL_POINTER, 25, 60)) + + assertEquals(2, out.pointerSize) + assertEquals(listOf(0, 1), out.idsList()) + assertEquals(listOf(10, 12), out.timesList()) + } + + @Test + fun `an empty current stroke yields just the base`() { + val out = InputPointers(16) + StrokeAligner.merge(out, base(), InputPointers(4), + StrokeAligner.Params(StrokeAligner.Mode.DUAL_POINTER, 25, 60)) + + assertEquals(3, out.pointerSize) + assertTrue(out.idsList().contains(0)) + } + + @Test + fun `merging into a non-empty output resets it first`() { + val out = InputPointers(16).apply { addPointer(999, 999, 1, 999) } + StrokeAligner.merge(out, base(), current(), StrokeAligner.Params.defaults()) + + assertEquals(6, out.pointerSize, "stale points must not survive the merge") + assertEquals(10, out.xsList().first()) + } + + @Test + fun `null params behave as defaults`() { + val withNull = InputPointers(16) + val withDefaults = InputPointers(16) + StrokeAligner.merge(withNull, base(), current(), null) + StrokeAligner.merge(withDefaults, base(), current(), StrokeAligner.Params.defaults()) + + assertEquals(withDefaults.timesList(), withNull.timesList()) + assertEquals(withDefaults.idsList(), withNull.idsList()) + } + + /** + * Invariant 4: a given raw index must not change track as the stroke grows, because + * `checkAndReturnIsContinuousSuggestionPossible` compares x/y/time but not pointer ids. + */ + @Test + fun `pointer ids for existing points are stable as the stroke grows`() { + val params = StrokeAligner.Params(StrokeAligner.Mode.DUAL_POINTER, 25, 60) + val first = InputPointers(16) + StrokeAligner.merge(first, base(), current(), params) + + val grown = current().apply { addPointer(160, 65, 0, 1075) } + val second = InputPointers(16) + StrokeAligner.merge(second, base(), grown, params) + + val firstIds = first.idsList() + val secondIds = second.idsList() + assertEquals(firstIds, secondIds.take(firstIds.size), + "an already-seen point changed decoder track mid-gesture") + assertEquals(first.timesList(), second.timesList().take(firstIds.size), + "an already-seen point was re-timed mid-gesture") + } +} diff --git a/app/src/test/java/helium314/keyboard/settings/SettingsContainerTest.kt b/app/src/test/java/helium314/keyboard/settings/SettingsContainerTest.kt index 38db92813..7fd6b70a5 100644 --- a/app/src/test/java/helium314/keyboard/settings/SettingsContainerTest.kt +++ b/app/src/test/java/helium314/keyboard/settings/SettingsContainerTest.kt @@ -3,6 +3,7 @@ package helium314.keyboard.settings import android.content.Context import androidx.test.core.app.ApplicationProvider import helium314.keyboard.latin.R +import helium314.keyboard.latin.settings.Defaults import helium314.keyboard.latin.settings.Settings import org.junit.Assert.assertEquals import org.junit.Assert.assertNull @@ -88,8 +89,31 @@ class SettingsContainerTest { } @Test - fun twoThumbFragmentBackspaceLabelMatchesBehavior() { - val context = ApplicationProvider.getApplicationContext() + fun strokeAlignmentSettingsAreRegistered() { + // All four are conditionally rendered on the Two-Thumb screen, so without an entry in the + // screen's Setting{} list they would silently vanish from settings search. + assertEquals(Settings.PREF_STROKE_ALIGN_MODE, + container[Settings.PREF_STROKE_ALIGN_MODE]?.key) + assertEquals(Settings.PREF_STROKE_IDEAL_PREFIX, + container[Settings.PREF_STROKE_IDEAL_PREFIX]?.key) + assertEquals(Settings.PREF_STROKE_ALIGN_INTERVAL_MS, + container[Settings.PREF_STROKE_ALIGN_INTERVAL_MS]?.key) + assertEquals(Settings.PREF_STROKE_ALIGN_GAP_MS, + container[Settings.PREF_STROKE_ALIGN_GAP_MS]?.key) + } + + @Test + fun strokeAlignmentDefaultsPreserveHistoricalBehaviour() { + // The experimental modes must stay opt-in: DUAL_POINTER and the synthetic prefix trail + // both change what the recognizer sees. + assertEquals("connector", Defaults.PREF_STROKE_ALIGN_MODE) + assertEquals(false, Defaults.PREF_STROKE_IDEAL_PREFIX) + assertEquals(25, Defaults.PREF_STROKE_ALIGN_INTERVAL_MS) + assertEquals(60, Defaults.PREF_STROKE_ALIGN_GAP_MS) + } + + @Test + fun twoThumbFragmentBackspaceLabelMatchesBehavior() { val context = ApplicationProvider.getApplicationContext() assertEquals("Delete last fragment", context.getString(R.string.two_thumb_backspace_fragment)) } diff --git a/docs/TWO_THUMB_TEMPORAL_ALIGNMENT.md b/docs/TWO_THUMB_TEMPORAL_ALIGNMENT.md new file mode 100644 index 000000000..928750b88 --- /dev/null +++ b/docs/TWO_THUMB_TEMPORAL_ALIGNMENT.md @@ -0,0 +1,365 @@ +# Two-thumb typing & the native gesture decoder — temporal alignment vs. pointer attribution + +Research findings for the hypothesis *"temporally shift the two thumbs' swipes so the AOSP gesture +library accepts them as simultaneous."* + +**Status:** research complete, measured against the real AOSP preprocessing code. +**Harness:** `app/src/main/jni/tests/replay/two_pointer_track_test.cpp` (runs in CI via +`.github/workflows/native-tests.yml`). + +--- + +## TL;DR — verdict + +| Claim | Verdict | +| --- | --- | +| The AOSP library can handle two simultaneous strokes | ✅ **Confirmed.** It models exactly two pointer tracks and decodes a word by alternating between them. | +| We should *temporally shift* strokes so it accepts them as simultaneous | ❌ **Falsified.** Track membership is decided **purely by pointer id**. Shifting or overlapping timestamps moves **zero** points between tracks — and overlapping actively **corrupts** the decoder's speed features. | +| Taps should be promoted to micro-swipes | ✅ **Sound, and already implemented** — `IdealPrefixTrailBuilder` on branch `b7a-prefix-aware-stripping` (issue #99), never merged. | +| There is a better lever than the current connector hack | ✅ **Yes: pointer-id attribution.** It is *necessary* to reach the second track, and measurably cleaner than the merged trail — but *not sufficient* on its own (four constraints in §2.4). | + +**The hypothesis is directionally right and mechanically wrong.** The goal — "make the library see +one genuine two-pointer gesture" — is achievable and natively supported. But the knob is +**`pointerIds[]`**, not the clock. Time still matters, in a *supporting* role: the concatenated +array must stay **globally monotonic**, and deliberately overlapping strokes is the one temporal +change that provably makes things worse. + +--- + +## 1. Q1 — Ground truth about the native decoder + +### 1.1 What is and isn't in this tree + +The gesture **scoring policy** is absent: `GestureSuggestPolicyFactory::sGestureSuggestFactoryMethod` +is initialised to `0` (`jni/src/suggest/policyimpl/gesture/gesture_suggest_policy_factory.cpp:20`). +Glide typing therefore requires the closed `libjni_latinimegoogle.so`, loaded by +`JniUtils.java:88-107`; the built-in library sets `sHaveNativeGestureLib = false`. +`jni/tests/replay/gesture_replay_test.cpp:11-29` already documents this — its replay test is +`DISABLED_` for exactly this reason. + +But "the decoder is closed source" is **imprecise**, and the distinction is what makes this research +possible: + +| Component | Open in this tree? | +| --- | --- | +| Gesture **Traversal / Weighting / Scoring** policy | ❌ closed | +| Search core (`Suggest`, `DicNode`, `DicTraverseSession`) | ✅ open | +| **Input preprocessing** (`ProximityInfoState`, `ProximityInfoStateUtils`) | ✅ open | + +The blob is *AOSP LatinIME + Google's private policy*, so the preprocessing that decides **what the +scorer is even allowed to see** is stock AOSP — readable, and (crucially) **host-executable**. + +### 1.2 The decoder models exactly two pointer tracks + +| Evidence | Location | +| --- | --- | +| `#define MAX_POINTER_COUNT 1` / `#define MAX_POINTER_COUNT_G 2` | `jni/src/defines.h:276-277` | +| `ProximityInfoState mProximityInfoStates[MAX_POINTER_COUNT_G]` — an array of **two** | `dic_traverse_session.h:178` | +| `for (i = 0; i < maxPointerCount; ++i) mProximityInfoStates[i].initInputParams(i, …)` — state *i* is seeded with pointer id *i* | `dic_traverse_session.cpp:69-79` | +| `updateTouchPoints` keeps **only** points where `pointerIds[i] == pointerId` | `proximity_info_state_utils.cpp:96-135` (esp. 98, 102) | +| `getProximityTypeG` loops over **both** used tracks, each at *its own* cursor `dicNode->getInputIndex(i)`, and returns MATCH if **either** matches | `dic_traverse_session.h:109-126` | +| The trie search node carries a **separate cursor per track** | `dic_node_state_input.h:89-91` | + +So a word can be spelled by **alternating between the two thumbs' trails**. That is Nintype-style +two-thumb decoding built into AOSP — not something we need to synthesise. + +### 1.3 Gesture input always runs with `maxPointerCount == 2` + +`dic_traverse_session.cpp:69-77` passes `maxPointerCount == MAX_POINTER_COUNT_G` **as the +`isGeometric` flag**, with an AOSP comment admitting the trick is "hacky and incorrect". If the +gesture traversal returned 1, `isGeometric` would be false for gestures and the entire geometric +pipeline (speed rates, beeline rates, `updateAlignPointProbabilities`) would never run — glide +typing would be broken. It isn't. ⇒ **both tracks are initialised on every gesture.** + +### 1.4 How time and pointer identity actually affect scoring + +- **Pointer identity → track membership.** Binary and absolute (`…utils.cpp:102`). +- **Time → geometric features only**, computed *within* a track: `refreshSpeedRates` + (`…utils.cpp:218-267`), `refreshBeelineSpeedRates` (`…:277-292`), and sampling decisions in + `pushTouchPoint`. These feed the closed weighting via `ProximityInfoState`'s public getters + (`proximity_info_state.h:156-174`). +- **There is no cross-track temporal comparator anywhere** — nothing in the open pipeline asks + "did these two strokes overlap?". + +--- + +## 2. Q2 — Assessing the temporal-shift idea + +### 2.1 The experiment + +`two_pointer_track_test.cpp` drives the **real** `ProximityInfoState::initInputParams` — the same +code compiled into the Google blob — with a two-fragment trace (`tech` + `nology`, the canonical +multi-part case from `TWO_THUMB_TYPING_INTERNALS.md` §5), sweeping pointer-id assignment and time +policy independently. + +`TwoPointerSweep.PrintTrackTable` output (`minSpeed` = minimum `getSpeedRate()` across the track's +sampled points; fragment A is raw `[0..13)`): + +``` +config | t0.used t0.n [raw range] minSpeed | t1.used t1.n [raw range] minSpeed +------------------------------------------------------------------------------------------------------ +today (all id 0, monotonic) | 1 28 [ 0.. 33] 0.491 | 0 0 [ -1.. -1] 0.000 +today (all id 0, restart) | 1 28 [ 0.. 33] -0.177 | 0 0 [ -1.. -1] 0.000 +split (0/1, monotonic) | 1 13 [ 0.. 12] 0.884 | 1 15 [ 13.. 33] 0.521 +split (0/1, restart) | 1 13 [ 0.. 12] -0.415 | 1 15 [ 13.. 33] -0.499 +split (0/1, overlap 50%) | 1 13 [ 0.. 12] -1.037 | 1 15 [ 13.. 33] -1.247 +split (0/1, overlap 100%) | 1 13 [ 0.. 12] -0.415 | 1 15 [ 13.. 33] -0.499 +split (1/0 reversed) | 1 15 [ 13.. 33] 0.521 | 1 13 [ 0.. 12] 0.884 +all id 1 | 0 0 [ -1.. -1] 0.000 | 1 28 [ 0.. 33] 0.491 +all id 2 | 0 0 [ -1.. -1] 0.000 | 0 0 [ -1.. -1] 0.000 +``` + +### 2.2 What it proves + +1. **Today the second track is dead.** With every point on id 0 (what + `InputPointers.appendAll` forces — see §2.3), track 0 absorbs raw `[0..33]` (both fragments, + with a spatial jump in the middle) and **track 1 gets zero points**. +2. **Splitting ids engages both tracks, cleanly.** `split (0/1)` gives track 0 exactly fragment A + (`raw [0..12]`) and track 1 exactly fragment B (`raw [13..33]`). No connector, no jump. +3. **Time cannot move a point between tracks.** + `TwoPointerTrackTest.TimePolicyDoesNotChangeTrackMembership` asserts that + `GLOBAL_MONOTONIC`, `PER_POINTER_RESTART` and `OVERLAPPED(100%)` all yield *identical* track + membership. **This is the direct falsification of the temporal-shift hypothesis.** +4. **Overlapping timestamps actively harms the decoder.** Look at the `minSpeed` column: healthy + configurations are positive; `overlap 50%` reaches **−1.037 / −1.247**. A negative speed rate is + arithmetically impossible from real input (`speed = length / duration`, `length ≥ 0`) — it means + `duration < 0`, i.e. the feature is garbage. So the one temporal change the hypothesis proposes + is the one that measurably degrades the decoder's inputs. +5. **Global monotonicity is required.** `restart` (per-stroke clocks) goes negative even *with* + correct ids (−0.415 / −0.499); `monotonic` is clean and in fact **better than today's merged + trail** (0.884 / 0.521 vs 0.491). + +### 2.3 Why time leaks across tracks at all (the F7 mechanism) + +`refreshSpeedRates` (`…utils.cpp:231-259`) walks **raw** input indices `j`/`j+1` +(`duration += times[j+1] - times[j]`), guarded only by +`if (i < sampledInputSize - 1 && j >= (*sampledInputIndice)[i+1]) break;`. For raw blocks +`[p0 p0 | p1 p1]`: + +- at track 0's **last** sampled point, `i < sampledInputSize - 1` is false ⇒ the forward guard is + disabled ⇒ the boundary edge is consumed; +- at track 1's **first** sampled point, `i > 0` is false ⇒ the backward guard is disabled ⇒ the same + edge is consumed again. + +So the window straddles the pointer boundary and reads a cross-thumb distance and a possibly +negative duration. `calculateBeelineSpeedRate` (`…:475-560`) and the raw-neighbour **angle** +computation (`…:109-115`) leak the same way. AOSP's own debug assertion at `…:61-71` treats +decreasing raw times as invalid input, confirming this is out-of-contract. + +**Consequence:** re-timestamping is still needed — but to enforce *monotonicity*, not to create +*overlap*. + +### 2.4 Where it would hook in, and what breaks + +The brief guessed `BatchInputArbiter`. That is the right seam for **truly simultaneous** input, +where ids are already correct (`GestureStrokeRecognitionPoints.java:314-320` appends with the +tracker's real MotionEvent id, `PointerTracker.java:434-437`). It is the **wrong** seam for the +fork's *sequential* fragments, which are merged much later in +`WordComposer.setBatchInputPointers` (`WordComposer.java:284-304`) — and that is where identity is +destroyed: + +```java +// InputPointers.java:109-117 +/** … Pointer ids are forced to 0 since multi-part gesture composition doesn't + * preserve pointer identity across separate strokes. */ +public void appendAll(@NonNull final InputPointers other) { + append(0, other.mTimes, other.mXCoordinates, other.mYCoordinates, 0, other.getPointerSize()); +} +``` + +Four constraints make id-remapping *necessary but not sufficient*: + +1. **Track 0 must anchor the word.** `suggest.cpp:81-84` early-returns when track 0 is unused ⇒ + **zero suggestions**. Measured: `NoPointerZeroLeavesTrackZeroEmpty`. +2. **Only ids 0 and 1 may be emitted.** Anything else reaches no track at all. Measured: + `PointerIdTwoIsSilentlyDropped`. (Reachable today: a third finger, or thumb B keeping id 1 after + thumb A lifts.) +3. **Ids must be stable across incremental recognition.** + `checkAndReturnIsContinuousSuggestionPossible` (`…utils.cpp:904-929`) compares x/y/time but + **not** pointer ids, so reassigning ids mid-gesture can silently reuse stale per-track state. +4. **Only two fragments fit.** The fork's combining mode routinely produces three or more. A third + fragment must reuse an id, which re-creates the spatial-jump problem the connector exists to + solve. **A hybrid is the likely endgame: ids for the first two fragments, connector beyond.** + +Also note the two-pointer path, while real, is **under-exercised**: several methods are hard-coded +to pointer 0 (`dic_node.h:192-197`, `suggest.cpp:245-249`) and partial commit explicitly does not +support multiple pointers (`suggestions_output_utils.cpp:63-65`). + +--- + +## 3. Q3 — Tap-to-micro-swipe promotion + +**Already built, and stranded.** `IdealPrefixTrailBuilder` (branch `b7a-prefix-aware-stripping`, +commit `e4724109d`, issue #99/B7b) synthesises an ideal key-centre trail for the composing prefix +and turns a single-letter (tap) prefix into a small **out-and-back micro-stroke**: + +- radius `keyWidth / 6`, **4 points** (`c−r`, `c`, `c+r`, `c`) — gives the recognizer a vertex + instead of an isolated point; +- multi-letter prefixes: key centres densified to ~`keyWidth / 4` spacing. + +It is gated behind `BuildConfig.FAKE_TRACK_V2` in a dedicated **`swipetest` build type** for +on-device A/B, and was never merged to `dev`. Its dev-log records the honest limitation: *"B7b +changes what the NATIVE recognizer returns, so it is not JVM-testable; verification is on-device +A/B only."* + +That geometry is reproduced in this harness (`TrackParams::promoteTaps`, +`tapArcRadiusDivisor`) so it can be swept alongside the pointer/time knobs. **Note its final line +still writes `pointerId 0`** — even the "fake-track" work never touched pointer identity. + +**Assessment:** sound and worth merging *independently* of the pointer-id question — it addresses a +different failure (sparse tap geometry), and 4 points at `keyWidth/6` is a reasonable starting +shape. It should be re-validated on device rather than assumed. + +--- + +## 4. Q4 — Prior art + +### 4.1 Inside this repo (the important part) + +An entire epic already exists and is **open**: + +| Issue | Title | State | +| --- | --- | --- | +| #97 | **[Epic] B7: Multi-part fake-track synthesis** | OPEN | +| #98 | B7a: Prefix-aware gesture result stripping | OPEN — built on `b7a-prefix-aware-stripping`, **not on `dev`** | +| #99 | B7b: Ideal prefix trail (tap → micro-stroke) | OPEN — built (`IdealPrefixTrailBuilder`), **not on `dev`** | +| #100 | **B7c: Adaptive connector bridge + distance-based re-timing** | OPEN — **never built** | +| #101 | B7d: Hybrid raw-vs-ideal prefix selection | OPEN — never built | +| #29 | B4: Per-thumb pointer attribution (true simultaneous) | OPEN — never built | +| #30 | B5: Tap geometry as weighted recognizer hints | OPEN — folded into #99 | + +**#100 is the maintainer's hypothesis, already specified months ago**: *"replace fixed 25 ms/60 ms +with distance-aware timing… large gap ⇒ **teleport** — no intermediate points, short dt (12–25 ms)… +re-time by `dt = clamp(distance / velocity, 8, 28 ms)`."* This research says #100 is worth doing +**for monotonicity and connector-hallucination reasons** (`techcolony`), but it will not make the +decoder treat the strokes as two tracks — only ids do that. + +**The key architectural finding:** epic #97 explicitly classifies #29 as +*"different problem (concurrent strokes), not composition"*. **That separation is wrong.** The +decoder's second track is exactly the "fake track" #97 is trying to synthesise — a real one, free. +Sequential composition and simultaneous two-thumb are the *same* mechanism at the decoder level. +No issue in this repo currently proposes using `mProximityInfoStates[1]`; even #29 proposes routing +the tapping thumb through Java-side live-converge instead. + +### 4.2 Decoder replacement (rejected / pending) + +- **SHARK²/statistical decoder**: a full FlorisBoard-derived `StatisticalSwipeDecoder` exists on + `feat/statistical-swipe-decoder` (commit `6afc07850`) with 21 JVM tests — **ruled out on quality** + per #97 ("we stay on Google's decoder and attack the track synthesis instead"). +- **NLnet open gesture recognizer** (#75, NGI Mobifree grant 101135795): data-gathering phase only, + library does not exist publicly; gathering ends "end of 2026 latest". Two-thumb work should be + designed to sit on top of it eventually. +- The in-tree Java fallback `SwipeGestureEngine` **ignores `pointerIds` and `times` entirely** — + `rankByIndex` flattens all points into one path (`SwipeGestureEngine.java:395-422`). It cannot + validate anything in this document. + +### 4.3 External + +- **HeliBoard #291** "Improving simultaneous/two-finger swiping" is the upstream request this + fork's two-thumb work answers (`TWO_THUMB_TYPING_INTERNALS.md` §intro). +- **Nintype** popularised two-thumb overlapping strokes; it uses its own decoder, so its design is + inspirational rather than transferable. +- The AOSP two-pointer design (`MAX_POINTER_COUNT_G`) dates from the original Google gesture work + and has never been publicly documented as a supported feature. + +--- + +## 5. Q5 — The harness, and what it can and cannot prove + +`app/src/main/jni/tests/replay/two_pointer_track_test.cpp`. Tunable knobs in `TrackParams`: + +| Knob | Values | Meaning | +| --- | --- | --- | +| `pointerMode` | `ALL_ZERO`, `SPLIT_0_1`, `SPLIT_1_0`, `ALL_ONE`, `ALL_TWO` | pointer-id assignment | +| `timeMode` | `GLOBAL_MONOTONIC`, `PER_POINTER_RESTART`, `OVERLAPPED` | time-axis policy | +| `overlapPct` | 0–100 | overlap amount for `OVERLAPPED` | +| `gapMs` / `intervalMs` | default 60 / 25 | today's `EXTEND_BASE_*` constants | +| `samplesPerKeyHop` | default 4 | trail densification (`IdealPrefixTrailBuilder` uses `keyWidth/4`) | +| `promoteTaps` / `tapArcRadiusDivisor` | on, 6 | tap → micro-stroke (B7b geometry) | + +Run: + +```bash +cmake -S app/src/main/jni -B ~/lt-host -DCMAKE_BUILD_TYPE=Release +cmake --build ~/lt-host -j +~/lt-host/latinime_host_unittests --gtest_filter='TwoPointer*' # sweep table +ctest --test-dir ~/lt-host -R TwoPointer # assertions +``` + +*(On Windows use WSL — the host build hits a MinGW `mkdir()` signature mismatch in an unrelated +v402 dictionary file.)* + +### What each tier proves + +| Tier | Proves | Does **not** prove | +| --- | --- | --- | +| **A. This harness** — real AOSP `ProximityInfoState`, no fidelity gap | Exactly which points reach each track; that time cannot change membership; that overlap corrupts speed features | The decoded **word** | +| **B. JVM unit tests** | Any Java-side transform (id assignment, monotonicity, micro-arc geometry) | Anything about recognition | +| **C. `SwipeGestureEngine` fallback** | *Spatial* path shape only | **Nothing** about ids or time — it ignores both | +| **D. On-device, with the blob** | Actual recognition quality | — | + +**Stated plainly: nothing below tier D produces a recognized word.** The *research* question +("does the decoder ingest two strokes as two tracks, and is time the lever?") is fully answered at +tier A. The *product* question ("does it recognise better?") remains device-only, and the existing +`swipetest` build type (#99) is the right vehicle for that A/B. + +--- + +## 6. Recommendation + +**Do not pursue temporal alignment as the mechanism.** It is falsified: timestamps cannot move a +point between tracks, and deliberate overlap is the single change measured to degrade the decoder's +inputs. + +Recommended order instead: + +1. **Fix the pointer-id hazards regardless of any redesign** (cheap, independent, likely + user-visible): guarantee at least one point carries id 0 and none carries id ≥ 2. Today a + two-thumb sequence where thumb A lifts first leaves thumb B on id 1 ⇒ track 0 empty ⇒ + `suggest.cpp:81-84` returns **no suggestions at all**. ✅ *done — `PointerIdNormalizer`.* +2. **Merge the stranded B7a + B7b work** (#98, #99). ✅ *B7b done — `IdealPrefixTrailBuilder` is + ported and reachable at runtime via `PREF_STROKE_IDEAL_PREFIX` (default off).* +3. **Prototype `SPLIT_0_1`** for the *two-fragment* case, with global-monotonic timestamps, subject + to the four constraints in §2.4. ✅ *done — `StrokeAligner`'s `DUAL_POINTER` mode, default off.* + **Still needs the on-device A/B to decide whether it ships as the default.** +4. **Keep the connector** for three-or-more fragments (it remains the default), and take #100's + distance-aware re-timing for its *own* merits (monotonicity, fewer `techcolony` hallucinations) + — not as a simultaneity mechanism. +5. **Reclassify #29.** It is not a "different problem" from #97; it is the same lever. Consider + folding them. + +**Confidence:** high for §1–2 (measured against real code, in CI). Medium for the recommendation — +whether two tracks *score* better than one merged trail is decided by the closed weighting policy +and can only be settled on-device. + +--- + +## 6a. What shipped, and how to try it + +Everything below defaults to **today's behaviour**, so nothing changes until a knob is moved. + +| Where | Knob | Default | Effect | +| --- | --- | :---: | --- | +| Two-Thumb Typing → Recognition | **Joining word parts** | One joined trail | `CONNECTOR` vs `DUAL_POINTER` (base → decoder track 0, current stroke → track 1) | +| ” | **Redraw earlier word parts cleanly** | off | `IdealPrefixTrailBuilder`: key-centre prefix path + tap → micro-stroke | +| ” | **Word-part trail speed** | 25 ms | base inter-point interval (`DUAL_POINTER` only) | +| ” | **Pause before the new part** | 60 ms | base→stroke gap (`DUAL_POINTER` only) | + +Implementation: `latin/gesture/StrokeAligner.java` (the merge seam, called from +`WordComposer.setBatchInputPointers`) and `latin/gesture/IdealPrefixTrailBuilder.java` (armed in +`InputLogic.onStartBatchInput`). `keyboard/internal/PointerIdNormalizer.java` fixes the id hazard +on the live gesture path independently of all of the above. + +**On-device A/B to run:** with two-thumb spacing on, compare *One joined trail* against *Two +separate thumb tracks* on the `TWO_THUMB_TYPING_INTERNALS.md` §5 matrix — especially +`tech`+`nology`, `te`+`chnology` and `s`+`ilo` — and watch for connector hallucinations +(`techcolony`) disappearing or top-1 accuracy regressing. + +--- + +## 7. Appendix — the one-line summary of the bug behind it all + +```java +// InputPointers.java:114 — this `0` is why the decoder's second track has never been used. +append(0, other.mTimes, other.mXCoordinates, other.mYCoordinates, 0, other.getPointerSize()); +``` diff --git a/docs/TWO_THUMB_TYPING_INTERNALS.md b/docs/TWO_THUMB_TYPING_INTERNALS.md index 588b14f88..a4757b390 100644 --- a/docs/TWO_THUMB_TYPING_INTERNALS.md +++ b/docs/TWO_THUMB_TYPING_INTERNALS.md @@ -762,6 +762,7 @@ the raw key-event path. Useful for the user-imported "power" symbol layout ## 8. Known caveats / future work - **Gesture-recognition accuracy with two thumbs** is bounded by the native glide-typing library. The PR's seed + concat trick fixes the common single-thumb-tap-then-swipe case (`"silo"`, `"technology"`) but a simultaneous two-thumb gesture where one thumb taps mid-swipe of the other can still produce odd results — that's where `PREF_GESTURE_DUAL_THUMB_HINTING` and `PREF_GESTURE_DEBUG_DRAW_POINTS` come in, and they remain experimental. + - **Update (research, see [`TWO_THUMB_TEMPORAL_ALIGNMENT.md`](TWO_THUMB_TEMPORAL_ALIGNMENT.md)):** the native decoder actually models **two** pointer tracks (`MAX_POINTER_COUNT_G == 2`) and can spell a word by alternating between them. We have never used the second track, because `InputPointers.appendAll` forces every merged point to pointer id 0. Track membership is decided by **pointer id, not timing** — measured in `jni/tests/replay/two_pointer_track_test.cpp`. - **Simultaneous tap-while-swiping recognition** now relies on combining/multi-part composition and the experimental point hinter rather than a separate suppression preference. Remaining odd recognizer outputs should be investigated in the gesture data / hinting path. - **`alternatives_then_next_word` mode** eats the first space tap to swap the strip. The current implementation doesn't restart the combining-mode timer for that synthetic event (since no composing word exists at that point). Probably correct, but worth keeping an eye on. - **`tryFragmentBackspace`** (manual-spacing sub-feature) was kept from wave 1. It is independent of the combining-mode timer and only fires under manual spacing — no conflict, but it does mean two backspace-pop mechanisms coexist (one for fragments under manual spacing, one for the gesture-committed-whole-word under combining mode).