Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
172 commits
Select commit Hold shift + click to select a range
49d2513
Use native Web Fetch interfaces and frozen benchmark inputs
Oct 8, 2026
c0ee2ae
Run frozen resource targets without repeating source qualification
lanyue-llk Oct 10, 2026
26833a4
fix(webfetch): bind native variant evidence and complete resource pop…
lanyue-llk Oct 10, 2026
7641bbe
chore(reports): preserve unchanged release record formatting
lanyue-llk Oct 10, 2026
3c19b47
fix(publication): avoid rehashing unchanged payloads for every stagin…
lanyue-llk Oct 10, 2026
de62032
fix(publication): recover uncommitted operations from HTTP 404 responses
lanyue-llk Oct 10, 2026
3132955
chore(webfetch): bind the published six-configuration report version
lanyue-llk Oct 10, 2026
3ee2f78
fix(report): classify risk-controlled body metrics consistently
lanyue-llk Oct 10, 2026
c6be370
test(report): enforce each body-family publication boundary
lanyue-llk Oct 10, 2026
c1385fc
chore(report): bind verified body-metrics publication
lanyue-llk Oct 10, 2026
71b6ca7
fix(report): distinguish measured Moli release and source-build versions
lanyue-llk Oct 12, 2026
7286b01
fix(report): keep presentation release separate from scoring revision
lanyue-llk Oct 12, 2026
49e360a
fix(report): localize reliability state labels after rebase
lanyue-llk Oct 12, 2026
b0234b9
fix(report): translate raw output label in result details
lanyue-llk Oct 12, 2026
172f4b5
fix(report): localize failure labels and report all language gaps
lanyue-llk Oct 12, 2026
041db95
fix(report): retain one measured-report translation
lanyue-llk Oct 12, 2026
bb3885f
chore(report): bind verified Moli method-version publication
lanyue-llk Oct 12, 2026
5fd5e81
Fix scoped Markdown list and product acceptance
lanyue-llk Oct 12, 2026
7ec325a
test(report): select Chinese for bound Chinese article fixture
lanyue-llk Oct 12, 2026
ac5914b
fix(reference): exclude fully transparent rendered text
lanyue-llk Oct 12, 2026
96c07c6
test: compare reference words independently of DOM line breaks
lanyue-llk Oct 12, 2026
2dbdf49
test(scoring): reproduce mixed HTML body projection errors
lanyue-llk Oct 12, 2026
07332ec
style: preserve scoring registry formatting
lanyue-llk Oct 12, 2026
0c30c4f
fix(scoring): project visible prose from mixed HTML answers
lanyue-llk Oct 12, 2026
f07d746
test(scoring): guard mixed HTML code and hidden payload boundaries
lanyue-llk Oct 12, 2026
6c2857c
fix(scoring): preserve autolinks beside raw HTML prose
lanyue-llk Oct 12, 2026
e52cdfc
fix(scoring): declare serialized output scope for structural patterns
lanyue-llk Oct 12, 2026
35b2bbe
test(scoring): distinguish prose patterns from output structure
lanyue-llk Oct 12, 2026
0d6c6b7
docs(scoring): define prose and output structure boundaries
lanyue-llk Oct 12, 2026
9257ccb
fix(scoring): preserve Markdown-escaped HTML examples
lanyue-llk Oct 12, 2026
534c46e
fix(scoring): preserve fenced code in Markdown table cells
lanyue-llk Oct 12, 2026
d017bad
fix(scoring): retain Markdown record boundaries around inline HTML
lanyue-llk Oct 12, 2026
0515745
fix(markdown): escape literal HTML syntax in prose and link labels
lanyue-llk Oct 12, 2026
451bc73
feat(benchmark): support certified serial alternating case calls
lanyue-llk Oct 12, 2026
27146ef
fix(webfetch): separate Moli answer capture from download capacity
lanyue-llk Oct 12, 2026
787c309
fix(scoring): bind named Markdown table records by declared columns
lanyue-llk Oct 12, 2026
d3c099f
test(scoring): preserve names with declared auxiliary link exclusions
lanyue-llk Oct 12, 2026
8993072
Bind reported browser routes to measured metadata
lanyue-llk Oct 12, 2026
a0922b4
Restore pinned report inputs from packed snapshots
lanyue-llk Oct 12, 2026
75efa0b
Allow native routes for every Web Fetch task family
lanyue-llk Oct 12, 2026
87c2e6b
Document native routes for the stacked Moli build
lanyue-llk Oct 12, 2026
7395bb6
Reject projected datasets as benchmark inputs
lanyue-llk Oct 12, 2026
80c127e
Certify paired native comparison evidence
lanyue-llk Oct 12, 2026
c582aec
Require complete comparison diagnostics
lanyue-llk Oct 12, 2026
9d4c443
Bind Web Fetch candidate to stack PR 1432
lanyue-llk Oct 12, 2026
e907254
Localize updated Web Fetch methodology
lanyue-llk Oct 12, 2026
e4abccc
Preserve half-escaped HTML examples in scoring
lanyue-llk Oct 12, 2026
d6c5563
Assert the current Moli stack label
lanyue-llk Oct 12, 2026
2cd60cd
Complete stacked Moli report configuration
lanyue-llk Oct 12, 2026
6ad5590
Localize Moli native Markdown interface copy
lanyue-llk Oct 12, 2026
530fb74
Complete Web Fetch report localization
lanyue-llk Oct 12, 2026
4291b7a
Correct Moli document capability disclosure
lanyue-llk Oct 12, 2026
7135ed1
Guard rendered result scoring from source examples
lanyue-llk Oct 12, 2026
42b9079
Separate scorer and task rule revisions
lanyue-llk Oct 12, 2026
9f4918b
Keep required output patterns in final acceptance
lanyue-llk Oct 12, 2026
a1d9e36
Use valid contracts in rendered result controls
lanyue-llk Oct 12, 2026
2e349a6
Audit diagnostic native comparison batches
lanyue-llk Oct 12, 2026
82cb8be
fix(webfetch): correct regression reference contracts
lanyue-llk Oct 12, 2026
ca56645
Reject incompatible headless provider configs
lanyue-llk Oct 13, 2026
240b4d5
Use reader-facing risk-control page label
lanyue-llk Oct 13, 2026
36286a6
Fix forbidden phrase boundaries and guard scoped link ownership
lanyue-llk Oct 13, 2026
3232ac5
Fix equivalent public table answer scoring
lanyue-llk Oct 13, 2026
c1aa83e
Keep final URL checks diagnostic in shared scoring
lanyue-llk Oct 13, 2026
11c084f
Keep HTML language metadata diagnostic for content tasks
lanyue-llk Oct 13, 2026
684ec77
Separate risk-control body acceptance from page-shell diagnostics
lanyue-llk Oct 13, 2026
8aa7406
Remove page identity from body acceptance documentation
lanyue-llk Oct 13, 2026
b5aecfc
Resolve combined scoring regression registry
lanyue-llk Oct 13, 2026
2d9f04c
Align body contract tests with diagnostic page identity
lanyue-llk Oct 13, 2026
a6f9ddd
Keep page identity diagnostic in report explanations
lanyue-llk Oct 13, 2026
9849343
Preserve visible table scopes and escaped pipe cells
lanyue-llk Oct 13, 2026
72c88df
Bind corrected risk-control navigation scope
lanyue-llk Oct 13, 2026
82a2507
Align report copy with diagnostic text roles
lanyue-llk Oct 13, 2026
17c645d
fix(webfetch): keep Moli outputs native by task
lanyue-llk Oct 13, 2026
019714f
fix(webfetch): preserve body table column continuity
lanyue-llk Oct 13, 2026
29fa4f1
fix(webfetch): map shared native frame contracts
lanyue-llk Oct 13, 2026
853e19b
fix(webfetch): revise stale current-page contracts
lanyue-llk Oct 13, 2026
763507c
fix(webfetch): bind native frame completion contracts
lanyue-llk Oct 13, 2026
0d9aac1
fix(webfetch): use Lightpanda native HTML for frames
lanyue-llk Oct 13, 2026
30de3d5
fix(webfetch): verify native contracts against loaded tasks
lanyue-llk Oct 13, 2026
1cbd69d
fix(webfetch): preregister Lightpanda dynamic HTML
lanyue-llk Oct 13, 2026
21a7cc7
Revert "fix(webfetch): preregister Lightpanda dynamic HTML"
lanyue-llk Oct 13, 2026
242c62b
fix(webfetch): enforce native Markdown body routes
lanyue-llk Oct 13, 2026
c9c28a0
fix(webfetch): preserve native frame contracts
lanyue-llk Oct 13, 2026
50121f3
fix(webfetch): bind worker contracts to case identity
lanyue-llk Oct 13, 2026
fc1ad08
fix(webfetch): bind authoritative 430 dataset assembly
lanyue-llk Oct 13, 2026
2477271
fix(webfetch): name the fixed Moli lineage
lanyue-llk Oct 13, 2026
e122e3f
fix(webfetch): preserve native task identity
lanyue-llk Oct 13, 2026
cc82f41
fix(webfetch): wait for complete delayed records
lanyue-llk Oct 13, 2026
3447265
fix(webfetch): label both measured Moli lineages
lanyue-llk Oct 13, 2026
01630fe
fix(webfetch): report Lightpanda script wait limits
lanyue-llk Oct 13, 2026
1e1dfe6
fix(webfetch): include native frame resources for Moli
lanyue-llk Oct 13, 2026
596b916
fix(webfetch): group media tasks by deliverable
lanyue-llk Oct 13, 2026
515a0fc
feat(webfetch): preserve mixed Moli implementation evidence
lanyue-llk Oct 13, 2026
7e7f861
fix(webfetch): disclose mixed Moli result versions
lanyue-llk Oct 13, 2026
90bf857
fix(webfetch): name the six measured configurations
lanyue-llk Oct 13, 2026
0f532ed
feat(webfetch): bind layered correction sources
lanyue-llk Oct 13, 2026
576f297
fix(webfetch): require full reruns for version upgrades
lanyue-llk Oct 13, 2026
de335e8
docs(webfetch): describe current Moli native routes
lanyue-llk Oct 13, 2026
5d9cc19
feat(webfetch): record absent same-window speed evidence
lanyue-llk Oct 13, 2026
7c382d4
fix(webfetch): bind correction execution identities
lanyue-llk Oct 13, 2026
9995e44
docs(webfetch): distinguish page media from direct files
lanyue-llk Oct 13, 2026
6ba4e57
fix(webfetch): state media and document deliverables
lanyue-llk Oct 13, 2026
ab0d32e
fix(webfetch): reject invalid native fetch evidence
lanyue-llk Oct 13, 2026
4ecf375
fix(webfetch): disclose media and timeout contracts
lanyue-llk Oct 13, 2026
d59f566
fix(webfetch): keep document scoring native
lanyue-llk Oct 13, 2026
7635c70
fix(webfetch): localize native delivery contracts
lanyue-llk Oct 13, 2026
b016242
fix(webfetch): keep Moli on native Markdown
lanyue-llk Oct 13, 2026
87bce7b
fix(webfetch): apply hash-bound case corrections
lanyue-llk Oct 13, 2026
da5993f
fix(webfetch): reject irreproducible assembled verdicts
lanyue-llk Oct 13, 2026
d33a0bc
fix(webfetch): make report checks self-contained
lanyue-llk Oct 13, 2026
2547bd0
fix(webfetch): reserve Moli deadline cleanup budget
lanyue-llk Oct 13, 2026
ba0cad5
fix(report): describe recorded Moli timeout policy
lanyue-llk Oct 13, 2026
e50b0c7
fix(webfetch): correct table and dynamic scoring
lanyue-llk Oct 13, 2026
f985c92
fix(webfetch): bind correction scopes before rescoring
lanyue-llk Oct 13, 2026
0a9ed15
perf(webfetch): rescore corrected reference scope only
lanyue-llk Oct 13, 2026
32c0653
fix(webfetch): use native dynamic completion by family
lanyue-llk Oct 13, 2026
e94ff8f
fix(webfetch): preserve owned native answer surfaces
lanyue-llk Oct 13, 2026
57231b8
test(webfetch): require uniform native comparison parameters
lanyue-llk Oct 13, 2026
7d651b6
fix(webfetch): retire invalid body requirements
lanyue-llk Oct 13, 2026
5035abc
fix(webfetch): preserve scoped markdown card ownership
lanyue-llk Oct 13, 2026
d2c588d
fix(webfetch): close independent qualification gaps
lanyue-llk Oct 13, 2026
7bd1588
fix(webfetch): preserve image-owned card boundaries
lanyue-llk Oct 13, 2026
fc4e090
fix(webfetch): accept bound native route receipts
lanyue-llk Oct 13, 2026
016b4df
fix(webfetch): bind Moli resource frame receipts
lanyue-llk Oct 13, 2026
6be3954
fix(webfetch): retain provider replacement scope history
lanyue-llk Oct 13, 2026
4b1f020
fix(webfetch): bind the certified 430-case report
lanyue-llk Oct 13, 2026
0cfc985
fix(webfetch): finalize the public 430-case release
lanyue-llk Oct 13, 2026
8ba0599
fix(webfetch): freeze fair six-method rerun contract
lanyue-llk Oct 14, 2026
391881d
fix(webfetch): pass retries through certified runs
lanyue-llk Oct 14, 2026
42983d5
chore(webfetch): freeze fixed Moli build identity
lanyue-llk Oct 14, 2026
7dfcdff
fix(webfetch): bind Moli public DNS environment
lanyue-llk Oct 14, 2026
622693d
fix(webfetch): align local browser DNS environment
lanyue-llk Oct 14, 2026
a5e4ca3
fix(webfetch): use native family waits and correct shared card scoring
lanyue-llk Oct 14, 2026
959cd77
docs(webfetch): document native family completion policies
lanyue-llk Oct 14, 2026
de79ad6
fix(scoring): retain fields owned by native Markdown records
lanyue-llk Oct 14, 2026
ad3e33b
fix(webfetch): bind native field ownership and qualified page scope
lanyue-llk Oct 14, 2026
7bb062f
test(webfetch): preserve homepage correction execution scope
lanyue-llk Oct 14, 2026
3c73503
fix(webfetch): retain one fixed homepage reference
lanyue-llk Oct 14, 2026
c953261
fix(webfetch): bind alternatives to frozen page scope
lanyue-llk Oct 14, 2026
ce0a8a7
fix(webfetch): disclose partial rerun build provenance
lanyue-llk Oct 14, 2026
187f20d
fix(webfetch): align task wording with frozen anchors
lanyue-llk Oct 14, 2026
029849b
fix(webfetch): separate report notes from retrieval inputs
lanyue-llk Oct 14, 2026
d3819e9
fix(webfetch): enforce native routes across dynamic task kinds
lanyue-llk Oct 14, 2026
83d3f18
fix(webfetch): validate batch coverage and preserve historical bindings
lanyue-llk Oct 14, 2026
7f55fbe
fix(webfetch): exclude independently observed rotating tagline
lanyue-llk Oct 14, 2026
8e31600
fix(webfetch): require complete review and report measured interfaces
lanyue-llk Oct 14, 2026
9a0c5c5
fix(webfetch): preserve canonical full review through aggregation
lanyue-llk Oct 14, 2026
e353f0f
fix(webfetch): verify current review at the publication boundary
lanyue-llk Oct 14, 2026
7b64be9
fix(webfetch): label general web tasks by their actual scope
lanyue-llk Oct 14, 2026
c25809e
fix(webfetch): preserve native Moli delivery failures
lanyue-llk Oct 15, 2026
0ee8870
fix(report): name the general web family accurately
lanyue-llk Oct 15, 2026
1d4f8ce
Revert "fix(report): name the general web family accurately"
lanyue-llk Oct 15, 2026
e210ef2
feat(webfetch): publish reviewed 430-case comparison
lanyue-llk Oct 15, 2026
59086e0
fix(webfetch): diagnose failures and correct native fetch calls
lanyue-llk Oct 15, 2026
c7b301b
fix(webfetch): redact truncated sensitive field excerpts
lanyue-llk Oct 15, 2026
c11ea7b
fix(webfetch): enforce interpretation checks for measured reports
lanyue-llk Oct 15, 2026
9482e5c
docs(webfetch): bind published failure-review report version
lanyue-llk Oct 15, 2026
12ce41c
fix(webfetch): correct visible-name scoring and clarify report interp…
lanyue-llk Oct 15, 2026
3cfebc8
docs(webfetch): bind revised interpretation publication
lanyue-llk Oct 15, 2026
863613d
fix(report): restore reference-style Web Fetch interpretation
lanyue-llk Oct 15, 2026
74e60e4
test(webfetch): keep CI fixtures portable and internally consistent
lanyue-llk Oct 15, 2026
f34ff82
fix(report): reject ambiguous release translations and refresh CI con…
lanyue-llk Oct 15, 2026
d3def72
test(webfetch): restore scoped projection test import
lanyue-llk Oct 15, 2026
2edb193
test(package): declare fake browser smoke as loopback-only
lanyue-llk Oct 15, 2026
1c2ce5e
test(package): use portable numeric address for fake CLI
lanyue-llk Oct 15, 2026
0fdeebd
fix(scoring): preserve HTML record field ownership
lanyue-llk Oct 16, 2026
f99333e
fix(chromium): classify navigation transport failures for shared retries
lanyue-llk Oct 16, 2026
816be5b
fix(webfetch): preserve HTML layout invariance and partial DNS calls
lanyue-llk Oct 16, 2026
202ec91
fix(webfetch): validate DOM records before deduplication
lanyue-llk Oct 16, 2026
b61c880
test(webfetch): align branch coverage with shared retry policy
lanyue-llk Oct 16, 2026
cd7e005
chore(webfetch): remove superseded benchmark snapshots
lanyue-llk Oct 16, 2026
17c5ea6
refactor(webfetch): decouple report instances from git
lanyue-llk Oct 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/benchmark-editorial/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Treat feedback as editorial input, not publishable copy. Identify the reader pro

## Evidence and argument

Read the supplied reports and relevant reference articles, including their charts, cases and methodology. Track material that could not be inspected privately; do not claim full coverage from a landing page. Study how reference articles develop their reasoning as well as how they look. Their facts are not evidence for this benchmark.
Read the supplied reports and relevant reference articles, including their charts, cases and methodology. Track material that could not be inspected privately; do not claim full coverage from a landing page. Study how reference articles develop their reasoning as well as how they look. Their facts are not evidence for this benchmark. When reproducing a supplied reference, compare its actual opening, result charts, specimen interactions and conclusion with the delivered page. Reuse the owning presentation components where available; matching headings and factual claims alone does not establish presentation parity.

For this repository, use `evaluation/current-report.json`, the normalized persisted results and the current scoring implementation. Read [benchmark-report](../benchmark-report/SKILL.md) when building or auditing them. Keep a private ledger for consequential claims: evidence, calculation, interpretation and limits. Do not rerun providers, alter scores or select better attempts to improve a narrative.

Expand Down
6 changes: 4 additions & 2 deletions .agents/skills/benchmark-report/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Design, run, audit, revise, and release evidence-backed benchmark r

# Benchmark report

Produce a report whose conclusions can be traced to frozen inputs, measured outputs, scoring rules, and an explicit result review. Optimize for a reader deciding what the benchmark means, not for preserving the author's workflow.
Produce a report whose conclusions can be traced to frozen inputs, measured outputs, scoring rules, and the declared result-review coverage. Optimize for a reader deciding what the benchmark means, not for preserving the author's workflow.

Read the repository's `AGENTS.md` first. Project rules override this general skill. Before changing a benchmark, read [evaluation design](references/evaluation-design.md). Before reviewing scores or deciding whether to rerun, read [audit and revision](references/audit-and-revision.md). Before building or publishing a report, read [report and release](references/report-and-release.md).

Expand All @@ -18,13 +18,15 @@ An explicitly requested diagnostic report may be delivered separately from a cer
Public report CLIs default to English where supported and expose `--language en|zh`. Preserve the selected language on rebuild. English builds retain the matching Chinese edition; keep measured material in its source language. Historical Chinese-only renderers must disclose that limit and reject unsupported requests. Bind generated languages and the selected entry in report metadata. English navigation opens a not-generated notice when absent; never publish stale English pages. Validate each generated edition and retain route, value and evidence-link parity checks when both exist.

1. Establish the decision being supported, the measured unit, cohort, versions, execution location, freshness policy, and intended deliverable. Inspect the actual dataset, runner, outputs, scorer, prior review evidence, and current report. Do not infer them from screenshots or prose.
2. Freeze a machine-readable evaluation contract before expensive calls. It must identify the exact cases and methods, run count, timeout, concurrency policy, input and output contract, scoring provenance, reference state, and publication target when one exists. For source-dependent tasks, fail closed: verify and freeze references immediately before calls, verify the same declared scope afterward, and certify the cohort only when both checks pass. Partial or uncertified cohorts remain diagnostic evidence and cannot enter a release.
2. Freeze a machine-readable evaluation contract before expensive calls. It must identify the exact cases and methods, run count, timeout, concurrency policy, input and output contract, scoring provenance, reference state, and publication target when one exists. Existing compiled reference answers are frozen by default. Source metadata, capture dates, and absent newer acceptance-contract fields do not imply a live-reference task. Only tasks explicitly declaring `source_verification.reference_mode: live_exact` require live checks. For those tasks, fail closed: verify and freeze references immediately before calls, verify the same declared scope afterward, and certify the cohort only when both checks pass. An unchanged compiled acceptance-contract dataset is already the benchmark input: freeze its task and rule bytes for execution, without requiring historical source-qualification raw files or re-auditing cohort admission. Qualify sources when adding tasks or changing references/acceptance rules; do not turn an ordinary version rerun into dataset reconstruction. Partial or uncertified cohorts remain diagnostic evidence and cannot enter a release.
3. Let each method use its strongest documented route to produce the required output. Equalize the task, inputs, output contract, call budget, and scoring. Do not require identical internals. Record material asymmetries such as local versus remote execution, cold versus warm start, bundled conversion, network transit, account tier, and billable time.
4. Preserve evidence as an append-only chain: task definition, reference, provider call, raw artifact, normalized artifact, score, review, revision, and release. Keep provider output and call identity immutable. A presentation or scoring change must not masquerade as a new provider result.
5. Separate retrieval success, task acceptance, content quality, latency, cost, and freshness. A missing or partial reference makes the affected metric unavailable or scoped; it does not prove success or failure. Do not derive binary acceptance from an unrelated aggregate similarity score.
6. For heterogeneous same-window latency evidence, preserve the full predeclared task-by-method matrix. Show all-call latency on that common denominator separately from successful-call latency on each method's conditional subset. Treat the all-method shared-pass intersection as a diagnostic, never as a post-outcome selector or ranking sample.
7. At aggregate and publication boundaries, copy only explicit hash-bound evidence and public audit files. Reject symlinks and exclude worker trees, host paths, credentials, and internal run logs. Verify primary, auxiliary, delivered-file, and retained retry artifacts before release.
8. Independent result review is opt-in: do not start it unless the user explicitly requests review. When requested without a broader scope, review machine failures only; review machine passes or every attempt only when the user explicitly asks for full review. Keep the initial verdict blind, but do not equate agreement with correctness: run the anomaly investigation in [audit and revision](references/audit-and-revision.md) for triggered results before calling the review complete. For a large independent review that is safely batchable, prefer resumable Codex CLI jobs with an exact model ID, read-only sandbox, strict output schema, non-overlapping packets, and bounded concurrency; persist each completed packet so a retry never repeats substantive review. Record the exact reviewed/total coverage, accepted unreviewed passes, reviewer, disagreements, adjudications, anomalies, and unresolved items. A review summary never replaces per-attempt evidence.
For an explicitly requested full review, set `independent_result_review_scope: all` in the report profile and carry it into the assembled dataset comparison. The release boundary must reject failure-only review, missing attempts and accepted unreviewed passes. Reuse an existing content judgment only when the raw answer and acceptance contract are unchanged; verify each new call identity separately. For failed calls, inspect the current runtime evidence even when the answer is unchanged or empty: the same empty output can hide different failures. Changed evidence needs supplemental review, not another review of the unchanged population.

9. Rerun only calls whose inputs, provider route/version, runtime contract, or source validity changed. Recompute downstream scores and reports without provider calls when only scoring, annotations, or presentation changed. Never retry merely to obtain a better result. A reference or verdict correction must identify the shared root cause and add executable positive/negative controls through the production scorer; bind those regression IDs in the revision receipt instead of patching stored verdicts.
10. Build the public report from the frozen evidence. Every page should answer a reader question. Show task and run identifiers, the exact expected-versus-actual evidence behind a verdict, denominators and unavailable metrics, method differences, limits, concise reproduction instructions, and a compact review history.
11. Release only after code tests, evidence integrity, report links, responsive/visual checks, secret scans, target-site checks, and deployed-output verification pass. Preserve earlier sites unless replacement was explicitly requested.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ release manifest + deployed bytes

Hash and bind transitions where accidental mixing is plausible. Validate exact cardinality and uniqueness, not only totals. Treat raw outputs, timestamps, request status, method version, and call IDs as immutable measured evidence.

Source-dependent calls require a fail-closed boundary: capture and freeze the declared reference scope immediately before execution, repeat the independent observation after execution, and bind both checks to the complete cohort. A process exit, partial attempt count, changed reference, or missing post-check makes the cohort diagnostic rather than publishable; never salvage its successful rows into the release matrix.
Live exact-reference calls require a fail-closed boundary: capture and freeze the declared reference scope immediately before execution, repeat the independent observation after execution, and bind both checks to the complete cohort. A process exit, partial attempt count, changed reference, or missing post-check makes the cohort diagnostic rather than publishable; never salvage its successful rows into the release matrix.

## Independent result review

Expand Down Expand Up @@ -49,6 +49,10 @@ Use dependency impact, not convenience:

Never overwrite old evidence or select a later response because it scores better. Assemble a new complete evidence view atomically, record inherited versus rerun attempts, and verify all downstream bindings.

Before a cohort-wide offline recomputation, finish the paired controls and compare the known affected output family with its independent judgments. Cover equivalent serializers and adjacent invalid forms together, including visible text outside a link, image alternatives, and record ownership. Batch the shared corrections before the expensive sweep; do not rerun unchanged browser calls. Bound both worker count and queued payloads when saved outputs or logs are large.

A subresource network error does not establish a main-navigation failure. Failure categories must agree with the reviewed cause and its actual stage; keep missing embedded content, unsupported direct-file conversion, empty output, and process crashes distinct. Interpretation pages show complete distributions and representative evidence. Full per-result diagnoses belong in the existing result inventory and details.

## Revision receipts

A score-only revision must record old and new dataset/result hashes, changed fields, decision changes, provider call count of zero, reviewer binding, and predecessor history hash. A new provider call must receive a new call identity and must not be described as a scoring correction.
Expand Down Expand Up @@ -78,3 +82,4 @@ private evidence. Never rescue an empty declared body from a nonempty display
message. Reproject retained full responses without new product calls when only
this selection changed; missing historical envelopes require fresh calls across
the full declared correction scope before claiming revised benchmark scores.
Reruns of unchanged compiled acceptance-contract tasks freeze the existing executable dataset. They do not require recovering historical qualification raw files or repeating admission review. Source qualification is part of task authoring and reference/rule correction, not a prerequisite for every provider-version comparison. Preserve historical receipts without applying their evidence-file requirements to a new execution protocol.
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,7 @@ Ground truth must be independent of the provider under test. Retain source locat
## Machine-readable current profile

Keep one current profile that a fresh session can validate before running, scoring, rebuilding, or publishing. Include exact case IDs or their canonical digest, the ordered method/version set, run count, timeout, concurrency policy, output contract, reference state, review requirement, and release target. Fail closed on drift instead of reconstructing intent from old reports.

For native-content browser comparisons, use task-appropriate official interfaces and score the returned deliverable. A final URL or HTML language tag is diagnostic unless the benchmark explicitly evaluates that metadata. Do not repair missing text using JSON DOM conversion or external PDF/DOCX parsers. The declared Chromium HTML-to-Markdown exception does not authorize document parsers. Bind native script versions, generate the interface matrix from the runner route selector, and require route validation at aggregation and publication. Keep negative controls for wrong content alongside positive controls for correct content with missing metadata.

Keep browser engines distinct from execution configurations. Released and source-build variants use separate method identities in the execution registry and share the engine's native route constraints. Derive interface columns and participant counts from that registry. A strict native-mode marker selects default completion; it supplies no per-case selector or wait and must survive both dataset conversion and runner validation. Resource projection must preserve single URL values and tokenize only srcset candidates, including literal commas inside URLs. Protect these boundaries with runner, report and actual-browser controls.
Loading
Loading