Skip to content

fix(ci): bound the identity-ladder harness so the fast lane cannot time out - #333

Merged
abrichr merged 2 commits into
mainfrom
fix/identity-ladder-harness-timeout
Aug 2, 2026
Merged

fix(ci): bound the identity-ladder harness so the fast lane cannot time out#333
abrichr merged 2 commits into
mainfrom
fix/identity-ladder-harness-timeout

Conversation

@abrichr

@abrichr abrichr commented Aug 2, 2026

Copy link
Copy Markdown
Member

Problem

tests/test_identity_ladder.py::test_harness_zero_false_accept_all_configs intermittently exceeds its 900s pytest-timeout in the required fast test job on main (runs 30750630903 and 30760532379 failed on 2026-08-02; 30757998188 passed between them). Pre-existing intermittency, unrelated to the day's merges.

Root cause

Two compounding, runner-speed-scaled costs:

  1. Redundant recorded-frame OCR. _verdict rebuilt the recorded anchor — including a full-frame RapidOCR pass over the recorded frame — for every one of the ~22 verdicts per pair, though the anchor depends only on the recorded frame and the substrate flags. ~300 of ~320 full-frame OCR passes recomputed identical results. Measured on an M2 Max: ~0.7s per full-frame OCR, ~0.65s per Playwright render → ~350s total; a 2–3× slower shared runner crosses 900s.
  2. Unbounded corpus in the fast lane. The sweep grows linearly with COLLAPSE_PAIRS (14 pairs × 22 verdicts × 5 configs), so every corpus addition pushed the fast required gate closer to its budget.

Fix (scheduling/budget only — the invariant is untouched)

The never-false-accept invariant, the production tier stack (Replayer._verify_identity), the five substrate configs, and the verdict logic are all unchanged. Only harness scheduling changes:

  • Anchor caching: each recorded anchor is built once per (pair, substrate) and the recorded frame's OCR lines are shared across substrate variants. Identifier crops get per-pair filenames (idcrop_<label>.png) so cached anchors never read another pair's crop from the shared bundle dir.
  • Bounded fast-lane run: the fast suite now runs test_harness_zero_false_accept_bounded_configs — the first pair of every (glyph_class, flank) collapse class (6 of 14 pairs: O/0 and l/1 × digit-flanked/alpha-flanked/purely-numeric) through the same production stack under all five configs. The test asserts the subset covers every class in the corpus, so growing COLLAPSE_PAIRS cannot grow the fast lane's work unless a genuinely new collapse class appears.
  • Exhaustive sweep moves to the slow lane: test_harness_zero_false_accept_all_configs now runs where the slow lane already runs — the nightly/dispatch full matrix — gated by OPENADAPT_IDENTITY_LADDER_EXHAUSTIVE=1 on the canonical Ubuntu leg. A new CI-contract test (test_exhaustive_identity_ladder_corpus_runs_in_the_slow_lane_only) pins the flag to exactly that leg: the exhaustive sweep can neither silently stop running anywhere nor creep back into the fast lane. The existing macOS deselect and its contract test are unchanged.

Measured

run before after
fast lane (M2 Max) ~350s, unbounded 98s, bounded by construction
exhaustive (M2 Max) ~350s 230s

Both runs assert the identical zero-false-accept invariant; both pass. ruff check openadapt_flow, ruff format --check openadapt_flow tests, lenient whole-package mypy, tests/test_identity_ladder.py (22 passed), and tests/test_ci_workflow_contract.py (7 passed) are green locally. The bounded harness exercises the same safety-path code paths as the exhaustive one, so the 85% safety-path coverage floor is unaffected.

🤖 Generated with Claude Code

abrichr and others added 2 commits August 2, 2026 23:14
…me out

tests/test_identity_ladder.py::test_harness_zero_false_accept_all_configs
intermittently exceeded its 900s pytest-timeout in the required fast `test`
job (runs 30750630903 and 30760532379 failed; 30757998188 passed between
them). Two compounding causes, both runner-speed-scaled:

1. Redundant recorded-frame OCR. The harness rebuilt the recorded anchor --
   including a full-frame RapidOCR pass over the same recorded frame -- for
   every one of the ~22 verdicts per pair, when the anchor depends only on
   the recorded frame and the substrate flags. ~300 of ~320 full-frame OCR
   passes computed results already computed. On a fast dev box that is
   ~350s total; on a slow shared runner it crosses 900s.
2. An unbounded corpus in the fast lane. The sweep grows linearly with
   COLLAPSE_PAIRS (14 pairs x 22 verdicts x 5 configs), so every corpus
   addition pushed the fast lane closer to its budget.

Fixes, scheduling/budget only -- the never-false-accept invariant, the tier
stack, the five configs, and the verdict logic are untouched:

- Build each recorded anchor once per (pair, substrate) and share the
  recorded frame's OCR lines across variants; identifier crops get per-pair
  filenames so cached anchors never read another pair's crop.
- The fast lane now runs test_harness_zero_false_accept_bounded_configs: a
  class-covering subset (first pair of every (glyph_class, flank) collapse
  class -- 6 of 14 pairs) through the same production tier stack. 98s on an
  M2 Max vs 350s+ before; growing COLLAPSE_PAIRS can no longer grow the
  fast lane's work unless a new collapse class appears.
- The exhaustive 14-pair sweep moves to the nightly/dispatch full-matrix
  lane via OPENADAPT_IDENTITY_LADDER_EXHAUSTIVE=1 on the canonical Ubuntu
  leg. A new CI-contract test pins the flag to exactly that leg, so the
  exhaustive sweep can neither silently stop running nor creep back into
  the fast lane.

Measured after the change: bounded 98s, exhaustive 230s (was ~350s) on an
M2 Max; both assert the identical zero-false-accept invariant and both
pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ane change

The wheel gate's release-consistency check pins a reviewed SHA-256 over
public artifacts including .github/workflows/ci.yml; the exhaustive-lane
env-var addition changed that hash. Reviewed: the only inventory delta is
the ci.yml row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@abrichr
abrichr merged commit ddbbd64 into main Aug 2, 2026
15 checks passed
@abrichr
abrichr deleted the fix/identity-ladder-harness-timeout branch August 2, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant