Publish complete 545-case WebMainBench report

AGENTS.md

@@ -21,6 +21,7 @@ These requirements apply to **all** report tracks; browser-specific rules below
 - Run repository Python only through `make` targets or `uv run --frozen python`. `.python-version`, `uv.lock`, the Makefile and CI are the runtime contract; never fall back to system Python.
 - Run every publishable provider cohort through `scripts/run_certified_benchmark.py`, which freezes references before calls and verifies them again afterward. Direct `run_benchmark.py` output and incomplete cohorts are diagnostic evidence only; never assemble or publish them.
 - For the default Web Fetch report, treat `evaluation/current-report.json` as the matrix and publication-target authority. The independent WCXB report uses only `evaluation/wcxb/current-report.json` for its matrix and target. Browser Use and Browser Trajectory Replay have separate contracts under their own `evaluation/` namespaces; never reuse a publication target across reports or tracks. Publishing must use the relevant bound site and expected remote version; do not copy target IDs from prose, old receipts or earlier reports.
+- For the complete WebMainBench report, `evaluation/webmainbench-full/current-report.json` is the immutable five-method pre-call profile bound by its cohort certificate. The repaired Moli sixth version has its own pre-call profile and certificate at `evaluation/webmainbench-full/moli-fixed-report.json`; the report joins both complete 545-page cohorts by a hash-bound six-method receipt. Their `publication_state` records frozen pre-call state. The live site, share URL, and expected remote version are bound in `evaluation/webmainbench-full/publication-target.json` together with both profile hashes; update this target after publication without changing or re-certifying historical calls.
 - For the current WCXB report, the source-qualified 504-case manifest is the entire evaluation population for quality, delivery, and latency. Keep source-invalid cases outside the current manifest, scored evidence, public package, and future provider calls; preserve earlier physical calls only in private historical evidence. The dataset page owns frozen task inputs, source URLs and human reference scope; it must not show measured browser scores. The home page owns aggregate comparisons, the result analysis owns the filterable method-case inventory, and case result pages own full responses. Keep a direct link from each dataset row to a safe frozen-page preview and its human reference. Default result filters must show a nonempty, clearly labeled diagnostic population when one exists, and filtered tables must shrink rather than leave a large blank viewport.
 - Persist provider calls outside Git under `~/.browser-eval/` (or `BROWSER_EVAL_DATA_HOME`). Git may contain compact benchmark definitions, annotations, independent source evidence and public release metadata, but never measured outputs, execution manifests, provider artifacts, rerun directories, credentials or internal work logs.
 - Web Fetch keeps one current authoritative logical result per method and task. A declared failure retry retains every physical call, retains total retry execution cost separately from the profile-defined primary latency and uses the final response, never the best response. A proven wrong command, version, scorer, reference or timing is a correction: preserve the prior evidence, supersede the complete declared scope and disclose score, denominator and latency effects.

evaluation/webmainbench-full/case-correction-binding.json

@@ -0,0 +1,5 @@
+{
+  "schema_version": 1,
+  "relative_path": "evidence/webmainbench-545-origin-correction-20260924",
+  "sha256": "24ec49dba0948b85c0d980b3caf04b016f8fd486ceaa36eef43385b18cb4c9ab"
+}

evaluation/webmainbench-full/content-binding.json

@@ -0,0 +1,5 @@
+{
+  "schema_version": 1,
+  "relative_path": "report-content.json",
+  "sha256": "c5094f07e653f323a2d4914b69a4533f510fe9e0cb9d7fb75041007ddae6e1cb"
+}

evaluation/webmainbench-full/current-report.json

@@ -0,0 +1,32 @@
+{
+  "schema_version": 1,
+  "dataset_id": "webmainbench-545-full-five-methods",
+  "comparison_kind": "webmainbench_frozen_html_to_markdown",
+  "case_count": 545,
+  "case_ids_sha256": "5c0cf5810602b70e8f7c1e16723e24a49c3ea207d54865df5c03c38f3d88ffd5",
+  "source_dataset_sha256": "0efaa4b49a45e320a27fe6e5a0b6aad5b57259fc3321ac3448519cacc74c537e",
+  "providers": ["moli", "lightpanda", "obscura", "chromium", "kitesurf"],
+  "versions": {
+    "kitesurf": "Cloudflare hosted stable backend; version undisclosed",
+    "lightpanda": "0.4.1",
+    "obscura": "0.2.2",
+    "moli": "1.1.9",
+    "chromium": "Google Chrome for Testing 154.0.8037.57"
+  },
+  "timeout_seconds": 45,
+  "runs_per_case": 1,
+  "concurrency": 1,
+  "max_failure_retries": 1,
+  "reference_binding_mode": "complete_webmainbench_545_official_frozen_html_and_groundtruth_content",
+  "official_scorer_commit": "9d991bdc00c57b57521499494d96be85c31317ba",
+  "metric_config": {"use_llm": false},
+  "official_metrics": ["overall", "text_edit", "code_edit", "formula_edit", "table_edit", "table_TEDS"],
+  "input_contract": {
+    "body": "official html field encoded as UTF-8 without mutation",
+    "execution": "inline scripts may run; non-loopback subresources and data access are blocked; local engines receive the same body from an OS-isolated loopback HTTPS fixture with a per-run CA, Kitesurf receives the same body through Cloudflare Browser Rendering html input with external resource types rejected",
+    "original_url": "retained as evidence metadata only"
+  },
+  "publication_state": "pending",
+  "require_independent_result_review": false,
+  "conversion": "Native Markdown for Kitesurf, Lightpanda, Obscura and Moli; Chromium whole-DOM HTML via node-html-markdown 2.0.0"
+}

evaluation/webmainbench-full/failure-analysis-binding.json

@@ -0,0 +1,5 @@
+{
+  "schema_version": 1,
+  "relative_path": "failure-analysis.json",
+  "sha256": "b04c476bf39b094260a869fbc42917fa62ad0e3fc3738ac53dcd70235b51bbbc"
+}

evaluation/webmainbench-full/illustrations.json

@@ -0,0 +1,17 @@
+{
+  "schema_version": 1,
+  "assets": [
+    {
+      "name": "overview.png",
+      "sha256": "1166017830d43cf56876e7cfc771d34dcd2b77998cac38d07bb84fa4e869d177",
+      "generator": "ImageGen",
+      "scope": "six methods after certified origin correction"
+    },
+    {
+      "name": "error-mechanisms.png",
+      "sha256": "c53b86571b5c20976f9a54ccfe33d065111732399fceeb4a148c74c520d3d843",
+      "generator": "ImageGen",
+      "scope": "six methods after certified origin correction"
+    }
+  ]
+}

evaluation/webmainbench-full/moli-fixed-report.json

@@ -0,0 +1,28 @@
+{
+  "schema_version": 1,
+  "dataset_id": "webmainbench-545-moli-fixed",
+  "comparison_kind": "webmainbench_frozen_html_to_markdown",
+  "case_count": 545,
+  "case_ids_sha256": "5c0cf5810602b70e8f7c1e16723e24a49c3ea207d54865df5c03c38f3d88ffd5",
+  "source_dataset_sha256": "0efaa4b49a45e320a27fe6e5a0b6aad5b57259fc3321ac3448519cacc74c537e",
+  "providers": ["moli"],
+  "versions": {"moli": "1.1.9"},
+  "binary_sha256": "48f45a608ce41d3c6600ca88df964e580aecf353289e028e8d9f7703224ab3b1",
+  "source_commit": "a50182cbe4eef20b230d3f01e452d9de4f3146ba",
+  "timeout_seconds": 45,
+  "runs_per_case": 1,
+  "concurrency": 1,
+  "max_failure_retries": 1,
+  "reference_binding_mode": "complete_webmainbench_545_official_frozen_html_and_groundtruth_content",
+  "official_scorer_commit": "9d991bdc00c57b57521499494d96be85c31317ba",
+  "metric_config": {"use_llm": false},
+  "official_metrics": ["overall", "text_edit", "code_edit", "formula_edit", "table_edit", "table_TEDS"],
+  "input_contract": {
+    "body": "official html field encoded as UTF-8 without mutation",
+    "execution": "inline scripts may run; non-loopback subresources and data access are blocked; local engine receives the same body from an OS-isolated loopback HTTPS fixture with a per-run CA",
+    "original_url": "retained as evidence metadata only"
+  },
+  "publication_state": "pending",
+  "require_independent_result_review": false,
+  "conversion": "Native Markdown from Moli fixed build"
+}

evaluation/webmainbench-full/publication-target.json

@@ -0,0 +1,9 @@
+{
+  "schema_version": 1,
+  "profile_sha256": "8cbceaf9694e88e7c1ddbba3263573dcccf69b1ac89d7e9ea5aa098f2a3bab02",
+  "moli_fixed_profile_sha256": "db8e7fa408edf3dc2e19d2b0f35645448d0e1efd3ac86926d0bcc9e1a3fb3848",
+  "host": "https://artifact-site.lexmount.cn",
+  "site_slug": "zrqPjO672dgg",
+  "share_url": "https://artifact-site.lexmount.cn/v/Avz3qzo6EEujI0iln1LAzCUF93I47OEC",
+  "expected_remote_version": "ver_8cbf0496af25497283fdcb750756b422"
+}

harness/web_fetch/providers/headless.py

@@ -157,6 +157,7 @@ def cli_command(
     retrieval_plan=None,
     allow_private_network=False,
     ca_cert=None,
+    http_proxy=None,
 ):
     ms = str(max(1, math.ceil(timeout * 1000)))
     if output_mode not in {'html', 'markdown', 'document', 'file'}:
@@ -169,16 +170,23 @@ def cli_command(
         # rejects those responses before extraction can begin. Version 1.1.3
         # documents this behavior explicitly; 1.0.6 and 1.1.1 are bound to
         # release binaries and covered by route probes and end-to-end tests.
-        return [binary, 'fetch', '--timeout', ms, '--http-timeout', ms, url]
+        command = [binary, 'fetch']
+        if http_proxy:
+            command += ['--http-proxy', http_proxy]
+        return command + ['--timeout', ms, '--http-timeout', ms, url]
     if output_mode == 'markdown':
         completion = _cli_completion_arguments(engine, retrieval_plan)
         if engine == 'obscura':
             # Obscura's documented default is load plus an adaptive, at-most
             # five-second quiescence wait. Do not replace it with a fixed delay.
             prefix = [binary, '--allow-private-network'] if allow_private_network else [binary]
+            if http_proxy:
+                prefix += ['--proxy', http_proxy]
             return prefix + ['fetch', '--dump', 'markdown', *(completion or []), '--timeout',
                     str(math.ceil(timeout)), '--quiet', url]
         common = [binary, 'fetch', '--dump', 'markdown']
+        if http_proxy:
+            common += ['--http-proxy', http_proxy]
         common += completion or ['--wait-until', wait_until or 'done']
         if engine == 'moli':
             return common + (['--ca-cert', str(ca_cert)] if ca_cert else []) + ['--timeout', ms, '--http-timeout', ms, url]
@@ -187,6 +195,8 @@ def cli_command(
         raise ValueError('Not a native Markdown CLI engine: ' + engine)
     completion = _cli_completion_arguments(engine, retrieval_plan)
     common = [binary, 'fetch', '--dump', 'html']
+    if http_proxy:
+        common += ['--http-proxy' if engine in {'moli', 'lightpanda'} else '--proxy', http_proxy]
     common += completion or ['--wait-until', 'domcontentloaded']
     if engine == 'obscura':
         return common + ['--timeout', str(math.ceil(timeout)), '--quiet', url]
@@ -400,6 +410,7 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult:
                    'retrieval_plan': normalized_plan,
                    'allow_private_network': bool(self.config.get('allow_private_network')),
                    'ca_cert': self.config.get('ca_cert'),
+                   'http_proxy': self.config.get('http_proxy'),
                    'network_policy': self.config.get('network_policy', 'unrestricted'),
                    'endpoint': self.config.get('endpoint',
                    'wss://kitesurf.cloudflare.app/devtools/browser')}