Repository navigation
Add certified full 545-case WebMainBench report - #31
Closed
lanyue-llk wants to merge 1 commit into
Closed
lanyue-llk wants to merge 1 commit into
lanyue-llk wants to merge 1 commit into
Conversation
Contributor
Author
|
Superseded by #33, which starts from current main and includes the reviewed cause matrix and same-case output comparison. The 545-case report is updated in place. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Result
Adds a separate certified WebMainBench report for the complete 545-case official frozen HTML dataset, covering Moli 1.1.9, Lightpanda 0.4.1, Obscura 0.2.2, Chrome for Testing 154.0.8037.57, and Kitesurf. The previous 200-case report remains intact.
Published report: https://artifact-site.lexmount.cn/v/Avz3qzo6EEujI0iln1LAzCUF93I47OEC
The five methods produced 2,725 logical results from 2,747 physical calls. All 17 final request failures used the declared single retry; five earlier failures recovered. The report keeps complete per-case inputs, references, outputs, original errors, and individual diagnosis. The 28 machine issues (17 request failures, 10 empty bodies, one browser error page) received evidence-bound review. Low text similarity is an automatic diagnostic, not an independently reviewed failure.
Scoring correction
The pinned official TEDS implementation passes serialized strings to APTED, so its default child traversal cannot see the actual table tree. This report applies the repository's explicit-node exact tree scorer to the preserved raw outputs, retains the original official values, and records all 292 changed scores. The separate upstream scorer fix is WebMainBench #76.
Validation
tests/test_webmainbench_full_report.py,tests/test_certified_webmainbench_full.py,tests/test_webmainbench_runner.py: 10 passed.Measured provider artifacts, report prose, figures, and review payloads remain outside Git. The three compact bindings and the profile in this PR identify those frozen inputs.