Publish the fair six-method 430-task Web Fetch comparison by lanyue-llk · Pull Request #42 · lexmount/browser-eval

lanyue-llk · 2026-10-08T13:33:55Z

The earlier Web Fetch comparison mixed browser-specific retrieval behavior, task acceptance rules, and report assumptions. This PR moves those decisions into the shared pipeline and publishes one complete comparison of six registered methods across the same 430 tasks and seven task families.

Result

Run each method through its declared native interface for a complete task family. Chromium alone uses the pinned shared HTML-to-Markdown conversion; no provider output is repaired after retrieval.
Keep final URL and page-language observations diagnostic. They do not decide the verdict.
Require readable returned text for document tasks. Raw bytes, download URLs, and harness-side document parsing do not pass.
Generate the 7×6 interface matrix, overall results, seven family results, and method diagnoses from the participant registry.
Recompute reference or scoring corrections through the shared score_attempt path and hash-bound revision receipts. Physical calls remain immutable.
Project public evidence through the shared privacy boundary, including nested HTML-escaped query parameters, without changing measurements.

The authoritative population contains 2,580 results: 430 tasks for each of six methods, with no missing result. Final totals are:

Method	Passed
Moli 1.1.17 fixed snapshot	282/430
Lightpanda	254/430
Moli 1.1.15 source build	216/430
Chromium	206/430
Obscura	202/430
Kitesurf	183/430

The two Moli rows are exact measured binaries rather than moving release labels:

Moli 1.1.15: source 98be1f105728, binary SHA-256 5696cc....
Moli 1.1.17 fixed snapshot: source 60773df391ad, binary SHA-256 a276fc....

The current Moli implementation aggregate is reviewed separately in lexmount/moli#1501. Its later rebased HEAD is not relabeled as the binary measured here.

Evidence and publication

Dataset SHA-256: 67f080b71958c2e7b9dfe0240eab8663e733e3c754eda46250857432e9e16cb2
Results SHA-256: c99eddea915d5c873b464ed66d69f51915df2993059b60ffb0e88ea9a485c77d
Published version: ver_e7d5d84083f54d96a2e384f5c747384a
Report: https://artifact-site.lexmount.cn/v/IqGpMUYHWat-dw4n8usQttTptoPzboAo

Validation

Shared Web Fetch tests: 178 passed, 3 skipped, with 47 subtests passed.
Publication privacy and profile tests: 38 passed, with 54 subtests passed.
Release check: 430 tasks, 2,580 results, six methods, 3,028 logical routes, and zero broken links.
The package retained 2,402 scored outputs byte-for-byte and privacy-projected 178 public copies; the public sensitive-data scan passed.
Desktop, narrow-width, light-theme, and dark-theme report paths were inspected, including one success detail and one failure diagnosis round trip.
git diff --check passes.

All earlier split PRs created for this work were audited against this branch. Every still-valid browser-eval change is present here; superseded or duplicate PRs #47–#51 are closed.

github-actions · 2026-10-09T18:13:02Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

github-actions · 2026-10-10T02:59:31Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

github-actions · 2026-10-10T03:15:46Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

github-actions · 2026-10-10T03:39:33Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

github-actions · 2026-10-10T03:41:51Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

github-actions · 2026-10-10T03:53:32Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

github-actions · 2026-10-10T04:09:52Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

github-actions · 2026-10-10T04:22:28Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

github-actions · 2026-10-10T04:29:55Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

github-actions · 2026-10-10T04:36:59Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

github-actions · 2026-10-10T04:38:56Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

github-actions · 2026-10-10T04:54:26Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

lanyue-llk

审查结论：建议修正下列问题后再合入。

本 PR 将 Web Fetch 扩展为五浏览器、七类任务、每浏览器 430 题，并统一原生接口、正文评分和发布校验。共享任务投影与统一路由选择器是合适的抽象；保留动态任务分类也合理，但调用校验没有同步迁移。资源投影还把单个 URL 当成 srcset 列表解析，导致合法资源地址损坏。

确认两处功能回退，均做了基线对照：目标分支通过，PR 失败；在隔离副本中验证最小修复后，相关测试 63 项通过。现有相关测试分两组共 478 项通过、13 项跳过；启用四种真实本地浏览器后，资源、取消和文件测试 24 项通过、1 项失败，失败原因见第三条。进程清理失败后的隔离行为符合现有安全合同，不列为缺陷。

这次审查覆盖代码、调用合同及相关测试，不改变报告中独立逐结果复核 0/2,150 的范围。

lanyue-llk · 2026-10-10T05:45:44Z

            raise ValueError('Native fetch cannot discard declared retrieval hints: ' + case['id'])
        if case.get('track') == 'dynamic':
-            case['track'] = 'web'
+            case['retrieval_hints'] = {'version': 3, 'mode': 'native'}


[P1] 同步固定原生调用校验，避免动态任务整批无法执行

native_fetch_dataset() 现在给动态任务写入非空的 retrieval_hints={"version":3,"mode":"native"}。但 Provider.check_native_call() 对 uniform_native_parameters=True 的方法仍拒绝所有非空 hints，run_evaluation() 又在任何调用前执行该校验。因此认证入口的 --retrieval-mode native 遇到动态任务时，会直接报 fixed native calls prohibit per-case retrieval hints，Moli、Firecrawl、Tavily 等固定原生方法都受影响。相同场景在 main 可以通过。

最小修复是在共享校验中通过现有解析器识别并允许严格的 v3 native 标记，继续拒绝选择器、延迟和其他逐题辅助参数。补一个从原生数据集转换一直跑到 runner 校验的动态任务测试，避免只测 plan_for_case() 而漏掉调用边界。

lanyue-llk · 2026-10-10T05:45:45Z

+  const urls = new Set();
+  const add = (value, base = location.href) => {
+    if (!value) return;
+    for (const part of String(value).split(',')) {


[P2] 只对 srcset 解析候选列表，保留单个资源 URL 中的逗号

add() 对 src、href、poster 等单个 URL 也执行 split(',')。真实 Chromium 页面中，<img src="/image/a,b.png"> 被返回成 /image/a 和 /b.png；/image/thumb.png?crop=10,20 被返回成 ?crop=10 和 /20。这些都是合法 URL，当前投影丢掉真实资源并制造不存在的地址。main 的 Chromium 路径能保留两条完整地址，PR 的共享投影失败；现在其他使用该投影的浏览器也受影响。

把单 URL 属性和 srcset 候选解析分开：前者按完整属性值解析，后者按候选与描述符解析。补路径含逗号、查询含逗号和多候选 srcset 的功能测试。

lanyue-llk · 2026-10-10T05:45:45Z

+        self._run_shared(ChromiumFileDeliveryTests().test_inline_redirect_attachment_and_http_error_use_native_navigation)
+
+    def test_pdf_and_inline_attachment_mislabeled_docx_use_shared_extraction(self):
+        self._run_shared(ChromiumDocumentDeliveryTests().test_native_document_bytes_use_the_shared_text_extractor)


[P2] 将 Moli 文档实测迁移到当前正文合同

这项新增测试复用 output_mode='document' 的下载字节加外部解析测试，但 _run_shared() 实际通过 MoliWebFetchProvider.fetch() 调用当前路由。该路由对直接 PDF 选择原生 Markdown，已不再走共享文档解析器。使用本次评测的真实 Moli 1.1.15 运行，测试在 /fixture.pdf 失败，返回 tool_error：raw download output only supports automatic output or --dump json。未设置 MOLI_BINARY 时此测试被跳过，所以普通 CI 的通过不能证明文档实测通过。

应按当前交付合同拆开原始文件和文档正文的场景：原文件继续验证字节，正文验证原生文本及浏览器查看器文本；原生接口不提供正文的直接文档应保留失败，不恢复外部解析来满足旧断言。删除已退役解析路径专用的断言，并实际启用二进制运行这些测试。

github-actions · 2026-10-10T06:28:05Z

Claude encountered an error after 3s —— View job

I'll analyze this and get back to you.

Rebase the session changes onto main 03c9a5e and extend its shared certification, execution, evidence, assembly and reporting owners. Keep command entry points thin, package the participant registry, remove retired runners and historical qualification recovery, and enforce task-selected native outputs without external answer repair. Preserve measured evidence and require complete declared correction scopes. Add functional controls for native documents, media URL parsing, interrupted evidence recovery, fixed versus explicitly live references, scoring regressions, report bindings and release presentation.

Separate resource admission from execution of hash-bound fixed benchmark inputs. Keep source qualification mandatory for new or changed resource tasks, explicit live references and invalid sources. Exercise the public certified entry point with frozen body and resource answers whose historical raw files are absent, preserve the original task bytes, and retain strict admission controls.

…ulations

…g packet

lanyue-llk changed the title ~~fix(webfetch): preserve cohort contracts and bound Markdown parsing~~ Fix frozen Web Fetch reruns and native dynamic execution Oct 9, 2026

lanyue-llk changed the title ~~Fix frozen Web Fetch reruns and native dynamic execution~~ Fix frozen browser benchmark reruns and fresh report cohorts Oct 9, 2026

lanyue-llk force-pushed the codex/webfetch-430-refresh branch from 1a7f2e3 to da3f01b Compare October 9, 2026 16:44

lanyue-llk changed the title ~~Fix frozen browser benchmark reruns and fresh report cohorts~~ Correct native Web Fetch routes and publish five-browser 430-task report Oct 9, 2026

lanyue-llk marked this pull request as ready for review October 9, 2026 18:12

lanyue-llk changed the title ~~Correct native Web Fetch routes and publish five-browser 430-task report~~ 扩充五浏览器430题评测并硬化原生接口与发布规则 Oct 10, 2026

lanyue-llk commented Oct 10, 2026

View reviewed changes

lanyue-llk force-pushed the codex/webfetch-430-refresh branch from 7607b18 to 2166c94 Compare October 10, 2026 08:29

lanyue-llk changed the title ~~扩充五浏览器430题评测并硬化原生接口与发布规则~~ Use native Web Fetch interfaces and frozen benchmark inputs Oct 10, 2026

lanyue-llk force-pushed the codex/webfetch-430-refresh branch from cd0b669 to 0598c6a Compare October 10, 2026 11:08

lanyue-llk changed the title ~~Use native Web Fetch interfaces and frozen benchmark inputs~~ Use native Web Fetch interfaces and publish the complete 430-task comparison Oct 10, 2026

llk and others added 6 commits October 12, 2026 11:18

fix(webfetch): bind native variant evidence and complete resource pop…

54180b6

…ulations

chore(reports): preserve unchanged release record formatting

db91996

fix(publication): avoid rehashing unchanged payloads for every stagin…

2e52abf

…g packet

fix(publication): recover uncommitted operations from HTTP 404 responses

b028f70

lanyue-llk added 8 commits October 13, 2026 19:53

fix(webfetch): reject invalid native fetch evidence

075b90e

fix(webfetch): disclose media and timeout contracts

54b83a2

fix(webfetch): keep document scoring native

980524c

fix(webfetch): localize native delivery contracts

815b7f5

fix(webfetch): keep Moli on native Markdown

d7332e7

fix(webfetch): apply hash-bound case corrections

267bafc

fix(webfetch): reject irreproducible assembled verdicts

e4cfb98

fix(webfetch): make report checks self-contained

25ba64e

This was referenced Oct 13, 2026

Fix forbidden phrase boundaries and guard scoped link ownership #48

Closed

Fix scoring of equivalent public table answers #49

Closed

Accept risk-control body output against body and record contracts #50

Closed

lanyue-llk added 3 commits October 13, 2026 23:43

fix(webfetch): reserve Moli deadline cleanup budget

835d834

fix(report): describe recorded Moli timeout policy

59af2ce

fix(webfetch): correct table and dynamic scoring

a75c361

lanyue-llk mentioned this pull request Oct 13, 2026

Maintain the reviewed 430-task ten-method Web Fetch report #51

Closed

lanyue-llk added 14 commits October 14, 2026 00:33

fix(webfetch): bind correction scopes before rescoring

293f390

perf(webfetch): rescore corrected reference scope only

cb5c434

fix(webfetch): use native dynamic completion by family

5b0be70

fix(webfetch): preserve owned native answer surfaces

e0addb4

test(webfetch): require uniform native comparison parameters

3e28a5c

fix(webfetch): retire invalid body requirements

2058ea6

fix(webfetch): preserve scoped markdown card ownership

1e0437a

fix(webfetch): close independent qualification gaps

40be94a

fix(webfetch): preserve image-owned card boundaries

f22b04b

fix(webfetch): accept bound native route receipts

6cd9319

fix(webfetch): bind Moli resource frame receipts

1b0da97

fix(webfetch): retain provider replacement scope history

4545705

fix(webfetch): bind the certified 430-case report

742c782

fix(webfetch): finalize the public 430-case release

f6dac02

lanyue-llk changed the title ~~Use native Web Fetch interfaces for the complete 430-task comparison~~ Publish the fair six-method 430-task Web Fetch comparison Oct 13, 2026

Conversation

lanyue-llk commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Result

Evidence and publication

Validation

Uh oh!

github-actions Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lanyue-llk left a comment

Choose a reason for hiding this comment

Uh oh!

lanyue-llk Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

lanyue-llk Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

lanyue-llk Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

lanyue-llk commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 9, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading