chore(tools): gate test-result analysis instead of trusting it - #139
Merged
Conversation
Three false conclusions were drawn from Gradle test output in a single session, and none of them was flagged by the test run itself: - Gradle served results from an earlier run because the task was UP-TO-DATE, so a "passing" report described code that never executed. - An ad-hoc XML reader under-counted, reporting 4 failures where there were 12. Under-reporting is the dangerous direction: it looks like good news, and nothing contradicts it. - Failures were attributed to a code change when one of them reaches the network over HTTP and can flip with no code change at all. Each is a habit that has to be remembered, so each is now a check that runs whether or not anyone remembers. tools/check_test_results.py: - refuses to report if any result file predates the run (--started-after) - counts <testcase> elements AND sums the tests=/failures= attributes the suites declare, and refuses to report if the two disagree, so an under-counting reader cannot pass silently - diffs failing test NAMES against a checked-in baseline rather than comparing pass counts - quarantines tests marked `net:` in the baseline, which are reported but never counted as a regression or as an attributable fix Exit codes are distinct: 0 clean, 1 new failures, 2 untrustworthy results. Wired into the Unit tests workflow as the authoritative gate. The Gradle step becomes continue-on-error so the checker decides; it is strictly stronger than the exit code it replaces, since a green Gradle run over stale results now fails. The checker's own tests run in the same job. tools/tests/test_check_test_results.py covers each failure mode with fixtures, including a file holding several <testsuite> elements and one whose declared totals exceed what it lists. Verified against real output: on the results where the ad-hoc reader said 4 failures, the checker reports 320 tests / 12 failed, matching the totals the suites declare. Exit codes confirmed 2 / 0 / 1 for stale, clean, and new-failure runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a88bd77f-d993-44c5-9efc-c7124f0d825e
The pycache directories were committed by mistake; ignore them instead. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a88bd77f-d993-44c5-9efc-c7124f0d825e
AsafMah
added a commit
that referenced
this pull request
Aug 20, 2026
Device testing did not confirm the paragraph-merge theory this branch was originally built around, so the unverified behaviour change is parked on `feat/backspace-paragraph-keyevent` and only the provable fixes remain here. On a Samsung SM-S936B, on the unfixed build, both Microsoft Word and Samsung Notes merged paragraphs correctly in every flow tried: typed Enter then backspace, caret moved into the paragraph by tap, empty paragraphs, and a held backspace across the boundary. The premise that those editors ignore deleteSurroundingText across a block boundary is therefore not established, and routing '\n' deletion through KEYCODE_DEL is not justified on this evidence. What remains is independently correct: - The accelerated (held-backspace) second deletion measured its length from codePointBeforeCursor -- the code point read *before* the first deletion -- instead of codePointBeforeCursorToDeleteAgain. When the second character back is a multi-code-point emoji that picks the wrong length and can split it; in the opposite case it can over-delete. Present in upstream v4.1.2 as well, reported as LeanBitLab#423. - Two latent test-harness bugs. deleteSurroundingText did not clamp to the available text and drove selectionStart negative on an empty field. setText accepted a requireIdle parameter and then called handleMessages() without it, so reset() could not tolerate a leftover delayed message -- harmless until adding tests reshuffled JUnit's hash-based ordering, at which point it failed an unrelated test during setup rather than in its own logic. The fake InputConnection can still simulate a block-based editor (paragraph-scoped reads, cross-paragraph deletes that report success while doing nothing). The two tests that assert the parked behaviour are @ignore'd with the device findings recorded, so the hypothesis and the evidence against it stay together. Suite: 324 tests, 4 failed -- the documented Windows-only ParserTest baseline, confirmed with tools/check_test_results.py (#139). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a88bd77f-d993-44c5-9efc-c7124f0d825e
This was referenced Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns three recurring analysis mistakes into checks a machine enforces, per the repo's own "prefer a gate to a reminder" principle.
Why
In a single session I drew three confident, wrong conclusions from Gradle test output. None was flagged by the test run itself:
UP-TO-DATE. The report looked green; it described code that never executed.XLinkTest > otherLinks) reaches Codeberg over the network and can flip with no code change at all.Each of those is a habit that has to be remembered. Each is now a check that runs whether or not anyone remembers.
What
tools/check_test_results.py--started-after, fails if any result file predates the run. Catches theUP-TO-DATEtrap.<testcase>elements and sums thetests=/failures=attributes the suites declare, and refuses to report anything if the two disagree. An under-counting reader can no longer pass silently, whatever the mechanism.net:are reported but never counted as a regression or as an attributable fix.Exit codes are distinct so CI can tell the cases apart:
0clean,1new failures,2untrustworthy results.CI wiring. The checker becomes the authoritative gate in the Unit tests workflow. The Gradle step becomes
continue-on-errorso the checker decides — it is strictly stronger than the exit code it replaces, because a green Gradle run over stale results now fails. The checker's own tests run in the same job.Baselines.
tools/test_baselines/runTests-linux.txt(CI; intentionally empty) andrunTests-windows.txt(the fourParserTestfailures that don't reproduce on Linux). Both carry a header telling the next person not to paste new failures in without establishing they aren't theirs.Verification
tools/tests/, one per failure mode, including a file holding several<testsuite>elements and one whose declared totals exceed what it lists.python -m unittest discover -s tools/tests→ OK.320 tests, 12 failed— matching the totals the suites declare.2, clean →0, new failures →1.Note
tools/__pycache__/release.cpython-313.pycwas already tracked in the repo. It's removed here and__pycache__/added to.gitignore— committed bytecode, unrelated to this change but not worth a separate PR.Not addressed
This gates analysis of test results. It does nothing about the larger gap: green JVM tests are still not evidence that input behaviour works on a device. That remains manual.