fix(ci): bound the identity-ladder harness so the fast lane cannot time out - #333
Merged
Conversation
…me out tests/test_identity_ladder.py::test_harness_zero_false_accept_all_configs intermittently exceeded its 900s pytest-timeout in the required fast `test` job (runs 30750630903 and 30760532379 failed; 30757998188 passed between them). Two compounding causes, both runner-speed-scaled: 1. Redundant recorded-frame OCR. The harness rebuilt the recorded anchor -- including a full-frame RapidOCR pass over the same recorded frame -- for every one of the ~22 verdicts per pair, when the anchor depends only on the recorded frame and the substrate flags. ~300 of ~320 full-frame OCR passes computed results already computed. On a fast dev box that is ~350s total; on a slow shared runner it crosses 900s. 2. An unbounded corpus in the fast lane. The sweep grows linearly with COLLAPSE_PAIRS (14 pairs x 22 verdicts x 5 configs), so every corpus addition pushed the fast lane closer to its budget. Fixes, scheduling/budget only -- the never-false-accept invariant, the tier stack, the five configs, and the verdict logic are untouched: - Build each recorded anchor once per (pair, substrate) and share the recorded frame's OCR lines across variants; identifier crops get per-pair filenames so cached anchors never read another pair's crop. - The fast lane now runs test_harness_zero_false_accept_bounded_configs: a class-covering subset (first pair of every (glyph_class, flank) collapse class -- 6 of 14 pairs) through the same production tier stack. 98s on an M2 Max vs 350s+ before; growing COLLAPSE_PAIRS can no longer grow the fast lane's work unless a new collapse class appears. - The exhaustive 14-pair sweep moves to the nightly/dispatch full-matrix lane via OPENADAPT_IDENTITY_LADDER_EXHAUSTIVE=1 on the canonical Ubuntu leg. A new CI-contract test pins the flag to exactly that leg, so the exhaustive sweep can neither silently stop running nor creep back into the fast lane. Measured after the change: bounded 98s, exhaustive 230s (was ~350s) on an M2 Max; both assert the identical zero-false-accept invariant and both pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ane change The wheel gate's release-consistency check pins a reviewed SHA-256 over public artifacts including .github/workflows/ci.yml; the exhaustive-lane env-var addition changed that hash. Reviewed: the only inventory delta is the ci.yml row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
tests/test_identity_ladder.py::test_harness_zero_false_accept_all_configsintermittently exceeds its 900s pytest-timeout in the required fasttestjob on main (runs 30750630903 and 30760532379 failed on 2026-08-02; 30757998188 passed between them). Pre-existing intermittency, unrelated to the day's merges.Root cause
Two compounding, runner-speed-scaled costs:
_verdictrebuilt the recorded anchor — including a full-frame RapidOCR pass over the recorded frame — for every one of the ~22 verdicts per pair, though the anchor depends only on the recorded frame and the substrate flags. ~300 of ~320 full-frame OCR passes recomputed identical results. Measured on an M2 Max: ~0.7s per full-frame OCR, ~0.65s per Playwright render → ~350s total; a 2–3× slower shared runner crosses 900s.COLLAPSE_PAIRS(14 pairs × 22 verdicts × 5 configs), so every corpus addition pushed the fast required gate closer to its budget.Fix (scheduling/budget only — the invariant is untouched)
The never-false-accept invariant, the production tier stack (
Replayer._verify_identity), the five substrate configs, and the verdict logic are all unchanged. Only harness scheduling changes:idcrop_<label>.png) so cached anchors never read another pair's crop from the shared bundle dir.test_harness_zero_false_accept_bounded_configs— the first pair of every(glyph_class, flank)collapse class (6 of 14 pairs: O/0 and l/1 × digit-flanked/alpha-flanked/purely-numeric) through the same production stack under all five configs. The test asserts the subset covers every class in the corpus, so growingCOLLAPSE_PAIRScannot grow the fast lane's work unless a genuinely new collapse class appears.test_harness_zero_false_accept_all_configsnow runs where the slow lane already runs — the nightly/dispatch full matrix — gated byOPENADAPT_IDENTITY_LADDER_EXHAUSTIVE=1on the canonical Ubuntu leg. A new CI-contract test (test_exhaustive_identity_ladder_corpus_runs_in_the_slow_lane_only) pins the flag to exactly that leg: the exhaustive sweep can neither silently stop running anywhere nor creep back into the fast lane. The existing macOS deselect and its contract test are unchanged.Measured
Both runs assert the identical zero-false-accept invariant; both pass.
ruff check openadapt_flow,ruff format --check openadapt_flow tests, lenient whole-packagemypy,tests/test_identity_ladder.py(22 passed), andtests/test_ci_workflow_contract.py(7 passed) are green locally. The bounded harness exercises the same safety-path code paths as the exhaustive one, so the 85% safety-path coverage floor is unaffected.🤖 Generated with Claude Code