Repository navigation
Fix scoring of equivalent public table answers - #49
Closed
lanyue-llk wants to merge 4 commits into
Closed
lanyue-llk wants to merge 4 commits into
lanyue-llk wants to merge 4 commits into
Conversation
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Valid table answers were rejected when headers included controls on a second visual line, when an extractor split a header from its adjacent body, or when a mixed Markdown answer retained an HTML table. Reuse the shared structured-table module to preserve these forms, including same-row fields and links. Composite table checks receive the original answer so HTML ownership survives visible-text normalization. No case-ID or provider exceptions are added.
Ten synthetic table controls bind three root causes in
evaluation/scoring-regressions.json. They cover valid equivalents and wrong headers, missing fields, intervening sections, code examples, and hidden HTML. The new positive controls fail against main and pass with this change.The public scorer also treated missing/mismatched final URLs as a hard acceptance gate and excluded missing-URL answers from the evaluated denominator. Final-URL agreement now remains diagnostic. HTML language tags also remain diagnostic; neither URL nor lang metadata changes content acceptance or the evaluated denominator. URL-based relative-link resolution remains intact. Fourteen paired controls cover missing/mismatched URLs and missing, empty or mismatched language tags with complete and incomplete answers. Summary tests verify the evaluated denominator.
HTML heading text is normalized before applying declared table section boundaries. Normal and linked tables share the escaped-pipe cell parser with stable table rows. Eighteen paired controls cover equivalent HTML/Markdown sections, out-of-section rows, literal pipes and actual extra columns; value checks confirm literal pipes survive extraction. The original scorer rejects the four new valid HTML controls.
Validation:
Rescoring the six requested immutable fixed-Moli answers, without provider calls or saved-verdict edits:
dynamic-017-en-stable-eui-table: passes, 10 complete unique rows.replacement-fix2-ar-mantine-rtl,replacement-fix2-ar-shadcn-rtl,replacement-fix3-ar-antd-rtl: pass, 5 complete unique rows each.risk-diverse-104: whole task passes with 87 repository records (minimum 70). Missing final URL is retained as an unavailable diagnostic.risk-diverse-153: whole task passes with 26 problem records (minimum 21). Missing final URL is retained as an unavailable diagnostic.All six have confirmed scoring defects and pass the shared task predicate after this correction. The last two were additionally blocked by the incorrect final-URL gate. Historical attempts and scores remain unchanged. No CI wait or merge is requested.