Skip to content

flow: take extra looks before stopping on a look-alike record ID - #1333

Merged
abrichr merged 7 commits into
mainfrom
claude/pixel-identity-c1eef9
Oct 19, 2026
Merged

abrichr merged 7 commits into
mainfrom
claude/pixel-identity-c1eef9

Conversation

@abrichr

@abrichr abrichr commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

What

On pixel surfaces (RDP, Citrix, canvas), the right-record check abstains when a record ID has a look-alike glyph (1/l, 0/O). Before this PR the run then stopped. Now it takes extra looks first, and stops only if they don't settle it:

  • the same OCR read on 2 fresh screenshots (each must read the same look-alike identifier);
  • the existing 2x re-read (must not name another record);
  • the known-ID list, when configured (already shipped);
  • where the surface allows it, a select-and-copy of the identifier's own text box, read from the clipboard (runtime.identity_copy_look in deployment.yaml, off by default).

Each look is recorded on the step's identity check (IdentityCheck.looks).

Fail-closed rules

  • OCR looks never accept on their own: a look-alike twin can read the same on every frame and at every size.
  • Only the known-ID list, or a copied ID that equals a run parameter in the identity band, can settle the record.
  • A recorded band is OCR text and can be a misread (A1001 recorded as Alo01), so in recorded-record mode the copy can stop a run but never accept one.
  • Any look that names another record is final (mismatch).
  • The copy drags across only the one text box holding the identifier. If the screen changes anywhere else during the look, the run halts (look_halt).
  • The copy look is input, so only the step-entry check takes it. The check just before the click takes no input; it reuses the copy only when the identifier's pixels are unchanged.
  • Governed and qualification runs never take the copy look.
  • A run-parameter mismatch that is only a look-alike misread (C0X1234 read as COX1234) gets the same looks; the mismatch stands unless the list or an exact copy of the run value settles it. It is never softened into an abstain.
  • A split ID (103 045 against 103045) is matched as one string, against the read and the list.

Tests

  • tests/test_identity_looks.py: decision rule, copy look, crop matching (pure).
  • tests/test_identity_looks_replayer.py: real render and real OCR on the replayer path. Correct row verifies with copy + run parameter; look-alike twin AC5OO61 stops; recorded mode cannot accept; a screen change outside the cell halts; governed runs take no copy; the pre-click check reuses the copy only on matching pixels.
  • tests/test_known_ids_replayer.py updated: the list now needs 3 agreeing frames instead of 2.
  • Local run of 43 identity/deployment/replayer test files: 1278 passed. The 2 failures (test_pixel_identity_probe) are a local scikit-image/numpy ABI mismatch and also fail on main.

Measurement

python -m openadapt_flow.validation.identity_looks_dense drives the real replayer against a live browser render of the dense record list, with real mouse drags. On 288 correct-row clicks, false stops went from 232 to 24 with the copy look and the ID as a run parameter, and wrong-record accepts stayed 0 of 288 in every configuration. Numbers and docs: #1340.

No new commands.

🤖 Generated with Claude Code

abrichr and others added 4 commits October 19, 2026 09:21
On a pixel surface, an OCR identity read that abstains only because the
record ID has a look-alike glyph (1/l, 0/O) now takes extra looks before
the run stops: two fresh frames, the 2x re-read, the known-ID list, and,
where configured, a select-and-copy of the identifier's text box. Only the
list or a copy equal to a run parameter can settle the record; any look
that names another record is final. Each look is recorded on the step's
identity check. The copy look is input, so only the step-entry check takes
it; the actuation check reuses it only on matching pixels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The copy drag now runs to the edges of the identifier's text box; it
  stopped inside the last glyph and copied one character short.
- A split identifier (103 045 against 103045) is matched as one string,
  against the read and the known-ID list.
- The check just before the click reuses the step-entry verdict when the
  identifier shows the same pixels, instead of re-reading the selected text.
- A run-parameter mismatch that is only a look-alike misread (C0X1234 read
  as COX1234) gets the same looks. The mismatch stands unless the list or
  an exact copy of the run value settles the record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abrichr and others added 2 commits October 19, 2026 15:00
- identity_looks_dense: pass a typed click point and ActionKind (mypy).
- LIMITS.md: name identity_looks.py by its package path, which the docs
  consistency check resolves.
- The look-alike twin test asserts the stop; it accepts a read that is
  already final, as Linux OCR builds read it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr
abrichr merged commit 3a821fe into main Oct 19, 2026
41 checks passed
@abrichr
abrichr deleted the claude/pixel-identity-c1eef9 branch October 19, 2026 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant