Repository navigation
Publish complete 545-case WebMainBench report - #33
Conversation
|
Claude finished @lanyue-llk's task in 4m 24s —— View job Review 结论不通过（因 P1 问题，需修复后合并） 问题及建议
其余部分评估
|
|
Claude finished @lanyue-llk's task in 2m 26s —— View job PR Review: Publish complete 545-case WebMainBench report
Finding 1 — Bug:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35867517312. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 5m 43s —— View job PR Review
Both bugs from the previous review are now fixed. A new latent bug was found in the gate code, and one P1 from the first review remains. Fixed since last review
Still present — P1 carryover
The PR body states the Artifact Site was already updated to New bug — cause matrix gate is stricter than the renderer
expected_cause_cells = {(kind, provider) for kind, _ in cause_counts for provider in providers}
This does not trigger in the current certified run (all 7 causes apparently have occurrences — the gate passed), but it is a fragility: a future update to # current (line 130) — only counts causes with ≥1 occurrence:
expected_cause_cells = {(kind, provider) for kind, _ in cause_counts for provider in providers}
# fix — mirror what the renderer actually emits:
expected_cause_cells = {(cause, provider)
for cause in cause_order
for provider in providers}No security issues found. HTML generation uses |
|
Claude finished @lanyue-llk's task in 2m 50s —— View job Review 结论不通过（P1 问题需合并前修复） 问题及建议P1
P2
其余评估（通过）
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35870310028. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 3m 21s —— View job Review 结论不通过（P1 问题合并前需修复） 问题及建议
已确认通过的部分
|
|
Claude finished @lanyue-llk's task in 2m 15s —— View job PR Review
Two previously reported issues are still open. The Moli follow-up additions in the latest commit ( Still present — P1 (3rd carryover)
The publication info ( "publication_state": "published",
"artifact_hub_expected_version": "ver_...",
"artifact_hub_share_url": "https://artifact-site.lexmount.cn/v/..."The corresponding values are already in Still present — cause matrix gate fragility (2nd carryover)
expected_cause_cells = {(kind, provider) for kind, _ in cause_counts for provider in providers}
# replace line 176 with:
expected_cause_cells = {(cause, provider)
for cause in cause_order
for provider in providers}where No security issues found. No new bugs introduced by |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35957388415. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 2m 49s —— View job Review 结论不通过（P1 问题第五轮仍未修复，需合并前处理） 问题及建议
已确认通过的部分
|
|
Claude finished @lanyue-llk's task in 1m 52s —— View job PR Review
Latest commit ( P1 — still present (5th carryover)
The share URL is stable across versions ( P2 — still present (3rd carryover)
expected_cause_cells = {(kind, provider) for kind, _ in cause_counts for provider in providers}
expected_cause_cells = {(cause, provider)
for cause in cause_order
for provider in providers}No security issues. No new bugs in |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35957678100. Please check Claude's inline feedback and summary comment for details.
Review 进行中
|
|
Claude finished @lanyue-llk's task in 1m 54s —— View job PR Review
Latest commit ( Fixed — P2 cause matrix gate fragility
The Still present — P1 (6th carryover)
The PR body argues that "publication_state": "published",
"artifact_hub_expected_version": "ver_91722963e35744c7920e47bc2578f090",
"artifact_hub_share_url": "https://artifact-site.lexmount.cn/v/Avz3qzo6EEujI0iln1LAzCUF93I47OEC"
No security issues. No new bugs in |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35958053288. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 36s —— View job PR Review
Both previously flagged issues are now resolved. No new bugs found. P1 — ResolvedThe latest commit (
The previous reviews compared against the WCXB pattern ( P2 — Resolved (carryover from
|
|
Claude finished @lanyue-llk's task in 1m 55s —— View job Review 结论通过 问题及建议未发现需要阻塞合并的问题。 关于持续 6 轮的 P1（publication_state）：前序所有 bot review 将 WCXB 模式直接套用到 WebMainBench，未读取本 PR 对 AGENTS.md 的变更。 关于 P2（cause matrix 脆弱性）：commit 残余风险： |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35958478223. Please check Claude's inline feedback and summary comment for details.
Review 进行中
|
|
Claude finished @lanyue-llk's task in 5m 0s —— View job PR Review
No blocking bugs found. One latent fragility noted below. Latent — physical attempt uniqueness not validated after aliasing
Logical results get an explicit uniqueness check after aliasing: if len({row["attempt_id"] for row in rows}) != 545:
raise ValueError("Fixed logical result identity is not unique")
# …and later:
if len(combined) != 3270 or len({row["attempt_id"] for row in combined}) != 3270:
raise ValueError("Six-version result matrix is incomplete")But the combined physical list written at line 259 has no equivalent: physical = read_jsonl(baseline / "physical-attempts.jsonl") + attempts
(output / "physical-attempts.jsonl").write_text(...)The This does not affect the current certified run. The P2 cause matrix fragility fixed in commit Security: No issues found. HTML uses Previously flagged issues: Both resolved — |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35966495947. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 5m 17s —— View job PR Review
Latest commits ( Fixed — P1 and P2 from prior rounds
Latest commits
Latent — physical attempt uniqueness not validated (2nd carryover)
physical = read_jsonl(baseline / "physical-attempts.jsonl") + attempts
(output / "physical-attempts.jsonl").write_text(...)The logical result matrix has an explicit uniqueness check (line 87: No security issues found. HTML uses |
Review 进行中
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35967076241. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 17s —— View job PR Review
Latest commit ( Latent — physical attempt uniqueness not validated (3rd carryover)
physical = read_jsonl(baseline / "physical-attempts.jsonl") + attempts
(output / "physical-attempts.jsonl").write_text(...)The combined physical list is written without a uniqueness check. For contrast, Previously flagged issues — all resolved:
No security issues. No new bugs in |
|
Claude finished @lanyue-llk's task in 2m 42s —— View job Review 结论通过 问题及建议未发现需要阻塞合并的问题。 残余风险（P2）： 各历史问题最终状态
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35967809790. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 4m 13s —— View job Review 结论通过 问题及建议未发现需要阻塞合并的问题。 残余风险（最低）： 本轮关键变更（
|
| 问题 | 状态 |
|---|---|
| P1 publication_state | ✓ AGENTS.md line 24 明确了 split-authority 设计；gate 在 check_webmainbench_full_report.py:55-60 验证绑定完整性 |
| P2 cause matrix 脆弱性 | ✓ WEBMAINBENCH_FULL_CAUSE_OWNERS 为共享模块级常量，renderer 与 gate 同源 |
| P2 physical attempt 唯一性 | ✓ 本轮 commit a59f05f 新增 require_unique_attempt_ids 并补测试 |
| Finding 1 Linux 路径脱敏 | ✓ 早前已修复 |
| Finding 2 文件单侧存在 ValueError | ✓ 早前已修复 |
|
Claude finished @lanyue-llk's task in 1m 1s —— View job 关于
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35976698968. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 11m 57s —— View job Review 结论通过 问题及建议未发现需要阻塞合并的问题。 残余风险（P2）： 各历史问题最终状态（独立核实）
其他关键设计独立核实
|
|
Claude finished @lanyue-llk's task in 4m 44s —— View job PR Review
One latent bug found in the new assembly script. Bug —
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35977918901. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 4m 38s —— View job Review 结论通过 问题及建议未发现需要阻塞合并的问题。 残余风险（P2）： 本轮独立核查重点最新 bot review 的"latent bug"——判定为误报最近一轮英文 bot review 指出 "results_sha256": digest(output / "results.jsonl"),认为此处 此结论不正确。 实际执行链如下：
两者均为六方法合并版的 SHA256，完全相同 ✓。bot 将
最新 commit
|
| 问题 | 状态 |
|---|---|
P1 publication_state: "pending" |
✓ AGENTS.md:24 明确 split-authority 设计；gate 在 check_webmainbench_full_report.py:55-59 验证双 profile SHA256、expected_remote_version、share_url、site_slug 绑定完整性 |
| P2 cause matrix 脆弱性 | ✓ WEBMAINBENCH_FULL_CAUSE_OWNERS 为模块级常量，renderer 与 gate 共用同一数据源 |
| P2 physical attempt 唯一性 | ✓ require_unique_attempt_ids 在三处调用，含六方法物理调用 |
| Finding 1 Linux 路径脱敏 | ✓ assemble_webmainbench_full.py:28-30 |
Finding 2 文件单侧存在 ValueError |
✓ review_webmainbench_teds_correction.py:46-48 |
"latent bug" results_sha256 |
误报，无需修复（见上方分析） |
|
Claude finished @lanyue-llk's task in 3m 19s —— View job PR Review
Latest commit ( Still present —
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35985531040. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 2m 31s —— View job Review 结论通过 问题及建议未发现需要阻塞合并的问题。 本轮独立核查：
|
| 问题 | 状态 |
|---|---|
P1 publication_state: "pending" |
✓ AGENTS.md:24 明确 split-authority 设计；check_webmainbench_full_report.py:55-59 验证双 profile SHA256、版本、URL 绑定完整性 |
| P2 cause matrix 脆弱性 | ✓ WEBMAINBENCH_FULL_CAUSE_OWNERS 为 report.py 模块级常量，renderer 与 gate 共用同一数据源 |
| P2 physical attempt 唯一性 | ✓ require_unique_attempt_ids 在三处调用（含六方法物理调用） |
| Finding 1 Linux 路径脱敏 | ✓ assemble_webmainbench_full.py:28-30 |
Finding 2 文件单侧存在 ValueError |
✓ review_webmainbench_teds_correction.py:46-48 |
results_sha256 "bug" |
误报，original-results.jsonl 是六方法合并版的副本，hash 恒等，无需修复 |
未使用的 moli-followup-binding.json |
✓ 本轮 df3d7a8 已删除 |
|
Claude finished @lanyue-llk's task in 1m 50s —— View job PR Review
通过。未发现阻塞合并的问题。
|
| 问题 | 状态 |
|---|---|
P1 publication_state: "pending" |
✓ AGENTS.md line 24 明确 split-authority 设计；gate 验证 publication-target.json 双 profile SHA256 和发布字段 |
| P2 cause matrix 脆弱性 | ✓ WEBMAINBENCH_FULL_CAUSE_OWNERS 为模块级常量，renderer 与 gate 共用同一数据源 |
| P2 physical attempt 唯一性 | ✓ require_unique_attempt_ids 在三处调用，含六方法物理调用 |
results_sha256 "bug" |
误报，无需修复；df3d7a8 已补注释消除歧义 |
| Finding 1 Linux 路径脱敏 | ✓ 早前已修复 |
Finding 2 文件单侧存在 ValueError |
✓ 早前已修复 |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35987454867. Please check Claude's inline feedback and summary comment for details.
1 participant
完整报告
原网址夹具修正及六方法补测
图示与验证
发布目标版本由 evaluation/webmainbench-full/publication-target.json 记录。