Skip to content

Add certified full 545-case WebMainBench report - #31

Closed
lanyue-llk wants to merge 1 commit into
mainfrom
codex/webmainbench-545-full-report
Closed

lanyue-llk wants to merge 1 commit into
mainfrom
codex/webmainbench-545-full-report

Conversation

@lanyue-llk

Copy link
Copy Markdown
Contributor

Result

Adds a separate certified WebMainBench report for the complete 545-case official frozen HTML dataset, covering Moli 1.1.9, Lightpanda 0.4.1, Obscura 0.2.2, Chrome for Testing 154.0.8037.57, and Kitesurf. The previous 200-case report remains intact.

Published report: https://artifact-site.lexmount.cn/v/Avz3qzo6EEujI0iln1LAzCUF93I47OEC

The five methods produced 2,725 logical results from 2,747 physical calls. All 17 final request failures used the declared single retry; five earlier failures recovered. The report keeps complete per-case inputs, references, outputs, original errors, and individual diagnosis. The 28 machine issues (17 request failures, 10 empty bodies, one browser error page) received evidence-bound review. Low text similarity is an automatic diagnostic, not an independently reviewed failure.

Scoring correction

The pinned official TEDS implementation passes serialized strings to APTED, so its default child traversal cannot see the actual table tree. This report applies the repository's explicit-node exact tree scorer to the preserved raw outputs, retains the original official values, and records all 292 changed scores. The separate upstream scorer fix is WebMainBench #76.

Validation

  • Certified complete matrix and source hash checks, before and after provider calls.
  • TEDS correction review passed without new provider calls.
  • tests/test_webmainbench_full_report.py, tests/test_certified_webmainbench_full.py, tests/test_webmainbench_runner.py: 10 passed.
  • Report build release gate passed; local link check: 10 logical routes, zero broken links.
  • Desktop and mobile, light and dark report views were exercised. Hosted failure detail opened its exact log and navigated back to the report root.

Measured provider artifacts, report prose, figures, and review payloads remain outside Git. The three compact bindings and the profile in this PR identify those frozen inputs.

@lanyue-llk

Copy link
Copy Markdown
Contributor Author

Superseded by #33, which starts from current main and includes the reviewed cause matrix and same-case output comparison. The 545-case report is updated in place.

@lanyue-llk lanyue-llk closed this Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant