Skip to content

Fix scoring of equivalent public table answers - #49

Closed
lanyue-llk wants to merge 4 commits into
mainfrom
codex/public-table-scoring
Closed

lanyue-llk wants to merge 4 commits into
mainfrom
codex/public-table-scoring

Conversation

@lanyue-llk

@lanyue-llk lanyue-llk commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Valid table answers were rejected when headers included controls on a second visual line, when an extractor split a header from its adjacent body, or when a mixed Markdown answer retained an HTML table. Reuse the shared structured-table module to preserve these forms, including same-row fields and links. Composite table checks receive the original answer so HTML ownership survives visible-text normalization. No case-ID or provider exceptions are added.

Ten synthetic table controls bind three root causes in evaluation/scoring-regressions.json. They cover valid equivalents and wrong headers, missing fields, intervening sections, code examples, and hidden HTML. The new positive controls fail against main and pass with this change.

The public scorer also treated missing/mismatched final URLs as a hard acceptance gate and excluded missing-URL answers from the evaluated denominator. Final-URL agreement now remains diagnostic. HTML language tags also remain diagnostic; neither URL nor lang metadata changes content acceptance or the evaluated denominator. URL-based relative-link resolution remains intact. Fourteen paired controls cover missing/mismatched URLs and missing, empty or mismatched language tags with complete and incomplete answers. Summary tests verify the evaluated denominator.

HTML heading text is normalized before applying declared table section boundaries. Normal and linked tables share the escaped-pipe cell parser with stable table rows. Eighteen paired controls cover equivalent HTML/Markdown sections, out-of-section rows, literal pipes and actual extra columns; value checks confirm literal pipes survive extraction. The original scorer rejects the four new valid HTML controls.

Validation:

  • Focused scoring, metrics, composite/stable acceptance, revision, lifecycle and WCXB tests: 194 passed; 719 subtests passed.
  • Offline impact check scored all 600 retained dynamic attempts (6 methods × 100). All 600 lack HTML lang metadata and all remain evaluated. Content pass counts: Moli 45, Lightpanda 54, Obscura 39, Chromium 40, Kitesurf 31, Moli PR919 23. Altering URL/lang diagnostic fields leaves every verdict and evaluated denominator unchanged. No provider calls or saved-score edits.
  • Track naming, changed-file F lint, diff check and fixture CLI validation/run passed. The fixture's declared negative case remained a failure.

Rescoring the six requested immutable fixed-Moli answers, without provider calls or saved-verdict edits:

  • dynamic-017-en-stable-eui-table: passes, 10 complete unique rows.
  • replacement-fix2-ar-mantine-rtl, replacement-fix2-ar-shadcn-rtl, replacement-fix3-ar-antd-rtl: pass, 5 complete unique rows each.
  • risk-diverse-104: whole task passes with 87 repository records (minimum 70). Missing final URL is retained as an unavailable diagnostic.
  • risk-diverse-153: whole task passes with 26 problem records (minimum 21). Missing final URL is retained as an unavailable diagnostic.

All six have confirmed scoring defects and pass the shared task predicate after this correction. The last two were additionally blocked by the incorrect final-URL gate. Historical attempts and scores remain unchanged. No CI wait or merge is requested.

@lanyue-llk

Copy link
Copy Markdown
Contributor Author

All four fixes and their regression coverage are represented in #42 by the conflict-resolved aggregate commits ec289437, 06cf6767, 309a6975, and 8132e73c. Keeping this PR open would leave a duplicate integration path, so #42 is the sole delivery PR.

@lanyue-llk lanyue-llk closed this Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant