diff --git a/.agents/skills/benchmark-editorial/SKILL.md b/.agents/skills/benchmark-editorial/SKILL.md index e9e6b2e55..152dc0cda 100644 --- a/.agents/skills/benchmark-editorial/SKILL.md +++ b/.agents/skills/benchmark-editorial/SKILL.md @@ -17,7 +17,7 @@ Treat feedback as editorial input, not publishable copy. Identify the reader pro ## Evidence and argument -Read the supplied reports and relevant reference articles, including their charts, cases and methodology. Track material that could not be inspected privately; do not claim full coverage from a landing page. Study how reference articles develop their reasoning as well as how they look. Their facts are not evidence for this benchmark. +Read the supplied reports and relevant reference articles, including their charts, cases and methodology. Track material that could not be inspected privately; do not claim full coverage from a landing page. Study how reference articles develop their reasoning as well as how they look. Their facts are not evidence for this benchmark. When reproducing a supplied reference, compare its actual opening, result charts, specimen interactions and conclusion with the delivered page. Reuse the owning presentation components where available; matching headings and factual claims alone does not establish presentation parity. For this repository, use `evaluation/current-report.json`, the normalized persisted results and the current scoring implementation. Read [benchmark-report](../benchmark-report/SKILL.md) when building or auditing them. Keep a private ledger for consequential claims: evidence, calculation, interpretation and limits. Do not rerun providers, alter scores or select better attempts to improve a narrative. diff --git a/.agents/skills/benchmark-report/SKILL.md b/.agents/skills/benchmark-report/SKILL.md index c9861c70a..5e63bc9fc 100644 --- a/.agents/skills/benchmark-report/SKILL.md +++ b/.agents/skills/benchmark-report/SKILL.md @@ -5,7 +5,7 @@ description: Design, run, audit, revise, and release evidence-backed benchmark r # Benchmark report -Produce a report whose conclusions can be traced to frozen inputs, measured outputs, scoring rules, and an explicit result review. Optimize for a reader deciding what the benchmark means, not for preserving the author's workflow. +Produce a report whose conclusions can be traced to frozen inputs, measured outputs, scoring rules, and the declared result-review coverage. Optimize for a reader deciding what the benchmark means, not for preserving the author's workflow. Read the repository's `AGENTS.md` first. Project rules override this general skill. Before changing a benchmark, read [evaluation design](references/evaluation-design.md). Before reviewing scores or deciding whether to rerun, read [audit and revision](references/audit-and-revision.md). Before building or publishing a report, read [report and release](references/report-and-release.md). @@ -18,13 +18,15 @@ An explicitly requested diagnostic report may be delivered separately from a cer Public report CLIs default to English where supported and expose `--language en|zh`. Preserve the selected language on rebuild. English builds retain the matching Chinese edition; keep measured material in its source language. Historical Chinese-only renderers must disclose that limit and reject unsupported requests. Bind generated languages and the selected entry in report metadata. English navigation opens a not-generated notice when absent; never publish stale English pages. Validate each generated edition and retain route, value and evidence-link parity checks when both exist. 1. Establish the decision being supported, the measured unit, cohort, versions, execution location, freshness policy, and intended deliverable. Inspect the actual dataset, runner, outputs, scorer, prior review evidence, and current report. Do not infer them from screenshots or prose. -2. Freeze a machine-readable evaluation contract before expensive calls. It must identify the exact cases and methods, run count, timeout, concurrency policy, input and output contract, scoring provenance, reference state, and publication target when one exists. For source-dependent tasks, fail closed: verify and freeze references immediately before calls, verify the same declared scope afterward, and certify the cohort only when both checks pass. Partial or uncertified cohorts remain diagnostic evidence and cannot enter a release. +2. Freeze a machine-readable evaluation contract before expensive calls. It must identify the exact cases and methods, run count, timeout, concurrency policy, input and output contract, scoring provenance, reference state, and publication target when one exists. Existing compiled reference answers are frozen by default. Source metadata, capture dates, and absent newer acceptance-contract fields do not imply a live-reference task. Only tasks explicitly declaring `source_verification.reference_mode: live_exact` require live checks. For those tasks, fail closed: verify and freeze references immediately before calls, verify the same declared scope afterward, and certify the cohort only when both checks pass. An unchanged compiled acceptance-contract dataset is already the benchmark input: freeze its task and rule bytes for execution, without requiring historical source-qualification raw files or re-auditing cohort admission. Qualify sources when adding tasks or changing references/acceptance rules; do not turn an ordinary version rerun into dataset reconstruction. Partial or uncertified cohorts remain diagnostic evidence and cannot enter a release. 3. Let each method use its strongest documented route to produce the required output. Equalize the task, inputs, output contract, call budget, and scoring. Do not require identical internals. Record material asymmetries such as local versus remote execution, cold versus warm start, bundled conversion, network transit, account tier, and billable time. 4. Preserve evidence as an append-only chain: task definition, reference, provider call, raw artifact, normalized artifact, score, review, revision, and release. Keep provider output and call identity immutable. A presentation or scoring change must not masquerade as a new provider result. 5. Separate retrieval success, task acceptance, content quality, latency, cost, and freshness. A missing or partial reference makes the affected metric unavailable or scoped; it does not prove success or failure. Do not derive binary acceptance from an unrelated aggregate similarity score. 6. For heterogeneous same-window latency evidence, preserve the full predeclared task-by-method matrix. Show all-call latency on that common denominator separately from successful-call latency on each method's conditional subset. Treat the all-method shared-pass intersection as a diagnostic, never as a post-outcome selector or ranking sample. 7. At aggregate and publication boundaries, copy only explicit hash-bound evidence and public audit files. Reject symlinks and exclude worker trees, host paths, credentials, and internal run logs. Verify primary, auxiliary, delivered-file, and retained retry artifacts before release. 8. Independent result review is opt-in: do not start it unless the user explicitly requests review. When requested without a broader scope, review machine failures only; review machine passes or every attempt only when the user explicitly asks for full review. Keep the initial verdict blind, but do not equate agreement with correctness: run the anomaly investigation in [audit and revision](references/audit-and-revision.md) for triggered results before calling the review complete. For a large independent review that is safely batchable, prefer resumable Codex CLI jobs with an exact model ID, read-only sandbox, strict output schema, non-overlapping packets, and bounded concurrency; persist each completed packet so a retry never repeats substantive review. Record the exact reviewed/total coverage, accepted unreviewed passes, reviewer, disagreements, adjudications, anomalies, and unresolved items. A review summary never replaces per-attempt evidence. +For an explicitly requested full review, set `independent_result_review_scope: all` in the report profile and carry it into the assembled dataset comparison. The release boundary must reject failure-only review, missing attempts and accepted unreviewed passes. Reuse an existing content judgment only when the raw answer and acceptance contract are unchanged; verify each new call identity separately. For failed calls, inspect the current runtime evidence even when the answer is unchanged or empty: the same empty output can hide different failures. Changed evidence needs supplemental review, not another review of the unchanged population. + 9. Rerun only calls whose inputs, provider route/version, runtime contract, or source validity changed. Recompute downstream scores and reports without provider calls when only scoring, annotations, or presentation changed. Never retry merely to obtain a better result. A reference or verdict correction must identify the shared root cause and add executable positive/negative controls through the production scorer; bind those regression IDs in the revision receipt instead of patching stored verdicts. 10. Build the public report from the frozen evidence. Every page should answer a reader question. Show task and run identifiers, the exact expected-versus-actual evidence behind a verdict, denominators and unavailable metrics, method differences, limits, concise reproduction instructions, and a compact review history. 11. Release only after code tests, evidence integrity, report links, responsive/visual checks, secret scans, target-site checks, and deployed-output verification pass. Preserve earlier sites unless replacement was explicitly requested. diff --git a/.agents/skills/benchmark-report/references/audit-and-revision.md b/.agents/skills/benchmark-report/references/audit-and-revision.md index 5ad926e2e..fbd856f8e 100644 --- a/.agents/skills/benchmark-report/references/audit-and-revision.md +++ b/.agents/skills/benchmark-report/references/audit-and-revision.md @@ -18,7 +18,7 @@ release manifest + deployed bytes Hash and bind transitions where accidental mixing is plausible. Validate exact cardinality and uniqueness, not only totals. Treat raw outputs, timestamps, request status, method version, and call IDs as immutable measured evidence. -Source-dependent calls require a fail-closed boundary: capture and freeze the declared reference scope immediately before execution, repeat the independent observation after execution, and bind both checks to the complete cohort. A process exit, partial attempt count, changed reference, or missing post-check makes the cohort diagnostic rather than publishable; never salvage its successful rows into the release matrix. +Live exact-reference calls require a fail-closed boundary: capture and freeze the declared reference scope immediately before execution, repeat the independent observation after execution, and bind both checks to the complete cohort. A process exit, partial attempt count, changed reference, or missing post-check makes the cohort diagnostic rather than publishable; never salvage its successful rows into the release matrix. ## Independent result review @@ -49,6 +49,10 @@ Use dependency impact, not convenience: Never overwrite old evidence or select a later response because it scores better. Assemble a new complete evidence view atomically, record inherited versus rerun attempts, and verify all downstream bindings. +Before a cohort-wide offline recomputation, finish the paired controls and compare the known affected output family with its independent judgments. Cover equivalent serializers and adjacent invalid forms together, including visible text outside a link, image alternatives, and record ownership. Batch the shared corrections before the expensive sweep; do not rerun unchanged browser calls. Bound both worker count and queued payloads when saved outputs or logs are large. + +A subresource network error does not establish a main-navigation failure. Failure categories must agree with the reviewed cause and its actual stage; keep missing embedded content, unsupported direct-file conversion, empty output, and process crashes distinct. Interpretation pages show complete distributions and representative evidence. Full per-result diagnoses belong in the existing result inventory and details. + ## Revision receipts A score-only revision must record old and new dataset/result hashes, changed fields, decision changes, provider call count of zero, reviewer binding, and predecessor history hash. A new provider call must receive a new call identity and must not be described as a scoring correction. @@ -78,3 +82,4 @@ private evidence. Never rescue an empty declared body from a nonempty display message. Reproject retained full responses without new product calls when only this selection changed; missing historical envelopes require fresh calls across the full declared correction scope before claiming revised benchmark scores. +Reruns of unchanged compiled acceptance-contract tasks freeze the existing executable dataset. They do not require recovering historical qualification raw files or repeating admission review. Source qualification is part of task authoring and reference/rule correction, not a prerequisite for every provider-version comparison. Preserve historical receipts without applying their evidence-file requirements to a new execution protocol. diff --git a/.agents/skills/benchmark-report/references/evaluation-design.md b/.agents/skills/benchmark-report/references/evaluation-design.md index ec68c72ce..ea412c7ea 100644 --- a/.agents/skills/benchmark-report/references/evaluation-design.md +++ b/.agents/skills/benchmark-report/references/evaluation-design.md @@ -43,3 +43,7 @@ Ground truth must be independent of the provider under test. Retain source locat ## Machine-readable current profile Keep one current profile that a fresh session can validate before running, scoring, rebuilding, or publishing. Include exact case IDs or their canonical digest, the ordered method/version set, run count, timeout, concurrency policy, output contract, reference state, review requirement, and release target. Fail closed on drift instead of reconstructing intent from old reports. + +For native-content browser comparisons, use task-appropriate official interfaces and score the returned deliverable. A final URL or HTML language tag is diagnostic unless the benchmark explicitly evaluates that metadata. Do not repair missing text using JSON DOM conversion or external PDF/DOCX parsers. The declared Chromium HTML-to-Markdown exception does not authorize document parsers. Bind native script versions, generate the interface matrix from the runner route selector, and require route validation at aggregation and publication. Keep negative controls for wrong content alongside positive controls for correct content with missing metadata. + +Keep browser engines distinct from execution configurations. Released and source-build variants use separate method identities in the execution registry and share the engine's native route constraints. Derive interface columns and participant counts from that registry. A strict native-mode marker selects default completion; it supplies no per-case selector or wait and must survive both dataset conversion and runner validation. Resource projection must preserve single URL values and tokenize only srcset candidates, including literal commas inside URLs. Protect these boundaries with runner, report and actual-browser controls. diff --git a/.agents/skills/benchmark-report/references/report-and-release.md b/.agents/skills/benchmark-report/references/report-and-release.md index 0501b5a2a..3caed87d1 100644 --- a/.agents/skills/benchmark-report/references/report-and-release.md +++ b/.agents/skills/benchmark-report/references/report-and-release.md @@ -12,7 +12,7 @@ Diagnostic failure analysis derives unmet requirements and expected/actual value Fresh diagnostic cohorts can supply a `webfetch_diagnostic_inputs/1` JSON file to the same build entry. It names complete saved run directories, independent whole-page browser observations and per-case reviews bound to dataset/results/observation hashes. The owner validates call and output identities and adapts these inputs to the existing model; no separate renderer is needed. Public copies may redact sensitive fields using the shared publication projector, with original/published hashes and an explicit local scope note; never alter the scored originals or present a redacted view as byte-identical. Review all requested passes as well as failures, retain access restrictions as observed limitations, and use actual requirements and source errors in interpretation specimens rather than storage metadata. -For live stable-contract cohorts, invoke `scripts/run_certified_benchmark.py --diagnostic`: invariant scoring rules alone do not qualify the current source. The generic entry rejects an unqualified stable-contract cohort before external work unless this diagnostic state is explicit. `--retrieval-mode native` selects ordinary WebFetch completion for a native experiment while preserving scenario categories and every acceptance requirement; it must reject declared retrieval hints instead of dropping them. Independent whole-page auditing uses the existing browser capturer with `--observation-scope visible_page --checkpoint-dir ...`; checkpoint identities include capture inputs and observer code, so resumption cannot silently reuse observations from another protocol. +For reruns of unchanged compiled stable-contract datasets, invoke `scripts/run_certified_benchmark.py` normally. Freeze existing task definitions and acceptance rules; do not require historical qualification raw files or repeat admission review. Source qualification belongs to adding tasks or changing references/acceptance rules. Live exact-reference cases retain pre/post reference checks. Reserve `--diagnostic` for explicitly diagnostic runs, whose outputs cannot be upgraded to certified evidence. `--retrieval-mode native` selects ordinary WebFetch completion while preserving scenario categories and every acceptance requirement; it rejects selector-guided retrieval hints instead of dropping them. Independent whole-page auditing, when requested, uses the existing browser capturer with `--observation-scope visible_page --checkpoint-dir ...`; checkpoint identities include capture inputs and observer code so observations from another protocol cannot be reused silently. For a diagnostic release, freeze the complete intended population in the track profile's `cohort_plan` (group IDs, case counts and SHA-256 of sorted case IDs joined by newlines with a trailing newline), independently of the report's selected inputs. Bind a uniform implementation when required. Pass that profile with `--profile` to both `scripts/build_report.py --diagnostic-cohorts` and `scripts/check_diagnostic_report.py` before publishing. A self-consistent subtotal is not evidence that the requested matrix is complete; reject missing groups, substituted cases and mixed implementations. Template examples without a deployment profile remain local examples, not a substitute for this release gate. @@ -66,6 +66,10 @@ Never publish user prompts, assistant reasoning, plans, internal notes, conversa Write every title and paragraph for the reader, not as a reply to author feedback. Convert corrections into neutral artifact facts; never mention who requested a change, quote review conversation, or use a rebuttal-style heading when a descriptive heading works. Review history may record an objective finding and resolution, while release history records only reader-visible effects. +For a fresh browser cohort, extend the maintained Web Fetch interpretation mode with `comparison.report_interpretation: {path, sha256}`. Keep its JSON prose outside Git in the evidence bundle. The `webfetch_browser_interpretation/1` schema binds the normalized task/result input hash, a title, introduction, scoring explanation, conclusions, and one explanation plus specimen case ID for every participant. The owner computes aggregate and family results, all-call and successful-call latency, observed versions, complete program-error distributions, and evidence links. A participant with failures must show a failed specimen. Historical prose bindings remain scoped to their original evidence; do not replace their hashes to reuse old conclusions. `scripts/check_report.py` verifies current-cohort numerical cells, populations, unique highlights, chapters and live result targets on both the interpretation and standalone routes, with negative controls in `tests/test_headless_report.py`. + +A composed cohort can use a compact current-report profile with `frozen_inputs` from `harness.web_fetch.storage.frozen_input_bindings` and an ordered per-track `case_inventory`. This binds the frozen dataset bytes and existing source/list-reference asset tree without duplicating whole datasets into a new repository folder. Both the build CLI and release checker validate these bindings and the complete provider-by-case matrix. The report package preserves original dataset bytes. These are publication-integrity checks on current inputs, not historical source-qualification requirements. + ## Claims and rankings Pair every ranking with its task family, denominator, acceptance meaning, a typical input-to-output route, and limits. Keep task acceptance separate from continuous quality, latency, cost, and title/body diagnostics. Do not call conditional averages global quality or explain a correlation as a verified mechanism. @@ -121,3 +125,5 @@ These rules apply to every public benchmark report, regardless of provider, task Repository adapter: `harness/reporting/presentation.py` provides track-independent checks over numeric cells marked with `data-metric`, `data-metric-direction`, and `data-sort`, and detail wrappers marked with `data-report-detail`. The current WebMainBench release gate invokes it. New or substantially rewritten tracks must integrate this checker (or equivalent semantic checks) in their own release command; adding a rule to this document is not sufficient. Existing tracks must not be claimed as covered until their gates actually invoke the checks. The local-audit variant also supports a new uniform native local measurement. Bind the exact committed case file, scorer, providers, one run, 45-second outer timeout and zero retries before calls. Preserve product-default retrieval without per-case wait/extraction hints. Validate completeness and every output/log hash before rendering through the existing report owner. Complete retained debug logs may be exposed through sanitized local report assets; do not publish raw private diagnostics or claim inherited independent review. + +Keep unpublished correction notes outside public release history. Schema 3 requires an explicit publication state for every release; an unmarked record fails the build. When a structured release is staged, mark it `publication_state: unpublished`; the public history selector and revision-binding gate must exclude it. Preserve its private correction receipt and measured calls. diff --git a/.dockerignore b/.dockerignore index 365cc8cb0..10ee128cf 100644 --- a/.dockerignore +++ b/.dockerignore @@ -15,3 +15,9 @@ configs browser_harness_eval.egg-info **/__pycache__ **/*.pyc +.env +.env.* +**/.env +**/.env.* +node_modules +**/node_modules diff --git a/.env.example b/.env.example index 1507f9e05..8fc860082 100644 --- a/.env.example +++ b/.env.example @@ -13,6 +13,13 @@ MOLI_BINARY=moli # Optional compiled Darwin resolver for explicitly frozen public_doh runs. MOLI_NATIVE_DNS_LIBRARY= MOLI_PR643_BINARY= +# Optional release/candidate comparison slots. Pin identity before calls. +MOLI_RELEASE_BINARY= +MOLI_RELEASE_REPORTED_VERSION= +MOLI_RELEASE_BINARY_SHA256= +MOLI_CANDIDATE_BINARY= +MOLI_CANDIDATE_REPORTED_VERSION= +MOLI_CANDIDATE_BINARY_SHA256= MOLI_1_1_11_BINARY=moli-1.1.11 MOLI_1_1_7_BINARY=moli-1.1.7 MOLI_1_0_6_BINARY=moli-1.0.6 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5a6de6306..e9df22ead 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -44,3 +44,9 @@ jobs: cache: npm - run: make setup-test - run: make ci + - name: Exercise rendered reference capture + run: | + uv sync --frozen --extra test --extra headless-eval + uv run --frozen --no-sync playwright install --with-deps chromium + export BROWSER_EVAL_TEST_BROWSER_BINARY="$(uv run --frozen --no-sync python -c 'from playwright.sync_api import sync_playwright; p = sync_playwright().start(); print(p.chromium.executable_path); p.stop()')" + uv run --frozen --no-sync python -m pytest tests/test_browser_observation_contracts.py -q diff --git a/AGENTS.md b/AGENTS.md index c1312079e..5d9c4e9bd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,7 +36,7 @@ These requirements apply to **all** report tracks; browser-specific rules below - For benchmark design, execution, scoring, result audit, report generation or release, read `.agents/skills/benchmark-report/SKILL.md`. Use `.agents/skills/benchmark-editorial/SKILL.md` additionally only for a new or substantially rewritten standalone interpretation article, official blog or public-account article. - Keep one report implementation in `harness/web_fetch/report.py`. The default `scripts/build_report.py` build uses persistent measured evidence (`~/.browser-eval/evidence/current/` by default, or `WEB_FETCH_EVIDENCE_DIR`) and writes `reports/webfetch/`; the independently bound WCXB build uses `--wcxb` and writes `reports/wcxb/`. Do not create versioned report implementations, compatibility renderers or redirect builds. - Run repository Python only through `make` targets or `uv run --frozen python`. `.python-version`, `uv.lock`, the Makefile and CI are the runtime contract; never fall back to system Python. -- Run every publishable provider cohort through `scripts/run_certified_benchmark.py`, which freezes references before calls and verifies them again afterward. Direct `run_benchmark.py` output and incomplete cohorts are diagnostic evidence only; never assemble or publish them. +- Run every publishable provider cohort through `scripts/run_certified_benchmark.py`, which freezes executable task inputs. Existing compiled reference answers stay frozen; pre/post source checks require an explicit `source_verification.reference_mode: live_exact` task declaration. Missing newer acceptance-contract fields never imply a live-reference task. Rerunning an unchanged compiled dataset must not require historical source-qualification raw files or reopen cohort admission; source qualification belongs to adding tasks or changing references/acceptance rules. Direct `run_benchmark.py` output and incomplete cohorts are diagnostic evidence only; never assemble or publish them. - For the default Web Fetch report, treat `evaluation/current-report.json` as the matrix and publication-target authority. The independent WCXB report uses only `evaluation/wcxb/current-report.json` for its matrix and target. Browser Use and Browser Trajectory Replay have separate contracts under their own `evaluation/` namespaces; never reuse a publication target across reports or tracks. Publishing must use the relevant bound site and expected remote version; do not copy target IDs from prose, old receipts or earlier reports. - For the complete WebMainBench report, `evaluation/webmainbench-full/current-report.json` is the immutable baseline pre-call profile bound by its cohort certificate. Independently certified variant cohorts are registered in `evaluation/webmainbench-full/`; discover the current variant registry and each variant's profile and certificate from that directory rather than hard-coding names. The method-binding file joins the baseline and all certified variant cohorts through locked profile hashes. Their `publication_state` records frozen pre-call state. The live site, share URL, and expected remote version are bound in `evaluation/webmainbench-full/publication-target.json` together with the baseline and all variant profile hashes; update this target after publication without changing or re-certifying historical calls. - For the current WCXB report, the source-qualified manifest is the entire evaluation population for quality, delivery, and latency. Keep source-invalid cases outside the current manifest, scored evidence, public package, and future provider calls; preserve earlier physical calls only in private historical evidence. The dataset page owns frozen task inputs, source URLs and human reference scope; it must not show measured browser scores. The home page owns aggregate comparisons, the result analysis owns the filterable method-case inventory, and case result pages own full responses. Keep a direct link from each dataset row to a safe frozen-page preview and its human reference. Default result filters must show a nonempty, clearly labeled diagnostic population when one exists, and filtered tables must shrink rather than leave a large blank viewport. @@ -53,8 +53,8 @@ These requirements apply to **all** report tracks; browser-specific rules below - Popular-site stable text must be independently observed and task-relevant. Do not freeze volatile ranks, prices, member counts or current list membership. Title diagnostics join returned entities to independent names by detail URL and never substitute for body quality, recall, freshness or task acceptance. - Keep pass, fail and all result filters, defaulting to fail, with failure classification. Every ranking must state its task family, denominator, acceptance meaning, typical input-to-output route and local limits. Preserve every registered provider's interface examples and adoption statistics. - When a current report has CPU, memory or network evidence, show the same grouped resource comparison on the home page and interpretation page. Lead with the evidence-derived winner for every metric where lower consumption has a clear meaning, mark winning cells in the tables, and state that resource leadership is separate from task pass rate. Do not label utilization or transfer rate as “better” merely because it is lower. Release checks must fail if the winner summary, grouped tables or winner markers are missing in either language. -- Maintain `research/report-changes.json` for substantive releases and `research/provider-pricing.json` for public pricing. Release notes explain reader-visible or result-visible effects without narrating assistant workflow; prices retain verification dates, commitments, billing units and unknowns. -- Render Web Fetch release history, including WebMainBench and the public benchmark research edition, from `research/report-changes.json`; keep Chinese and English release text together in each entry. Do not duplicate release prose or past measurements in report renderers or exact-string translation maps. Release checks compare structured records with rendered routes rather than matching fixed phrases. Browser Use and trajectory replay keep their existing structured release models. +- Keep `research/provider-pricing.json` as a reusable product-price catalog: retain verification dates, commitments, billing units and unknowns, but do not describe one run or report there. New report release history belongs in hash-bound `report-assets/report-changes.json` beside the evidence. `research/report-changes.json` is a legacy fallback while older report owners migrate; do not add a new report's results, prose, review scope or publication URL to it. +- Render a report's release history from its bound report asset when present; keep Chinese and English release text together in each entry. Do not duplicate release prose or past measurements in report renderers or exact-string translation maps. Release checks compare structured records with rendered routes rather than matching fixed phrases. Browser Use and trajectory replay keep their existing structured release models. - Use `Browser Use` for the autonomous track and `Browser Trajectory Replay` (Chinese: `轨迹重放`) for the fixed BrowserTool-action track. Their implementation roots are `harness/browser_use/` and `harness/trajectory_replay/`; do not add aliases or compatibility paths using the retired names. - The public WebArena human-recording replay is isolated under `harness/trajectory_replay/public_webarena/` with its own examples, tests, `evaluation/browser-use-public/current-report.json` and release script. It must not change autonomous `harness/browser_use/` or share a report target or denominator with the repository-curated replay suite. - Browser Trajectory Replay canonical inputs must not contain run-local snapshot refs. Compile each ref to a hash-bound stable selector while preserving the original ref action semantics; do not silently replace a DOM ref click/fill/select with BrowserTool's Playwright selector fallback because that adds different viewport, overlay and navigation-wait behavior. diff --git a/configs/providers.headless.json b/configs/providers.headless.json index 1aac71c17..65fc4063d 100644 --- a/configs/providers.headless.json +++ b/configs/providers.headless.json @@ -42,6 +42,24 @@ "engine": "moli", "binary": "${MOLI_PR643_BINARY}" }, + "moli-release": { + "type": "moli_webfetch", + "engine": "moli", + "label": "Moli release build", + "source_url": "https://github.com/lexmount/moli/releases", + "binary": "${MOLI_RELEASE_BINARY}", + "expected_reported_version": "${MOLI_RELEASE_REPORTED_VERSION}", + "expected_binary_sha256": "${MOLI_RELEASE_BINARY_SHA256}" + }, + "moli-candidate": { + "type": "moli_webfetch", + "engine": "moli", + "label": "Moli candidate build", + "source_url": "https://github.com/lexmount/moli", + "binary": "${MOLI_CANDIDATE_BINARY}", + "expected_reported_version": "${MOLI_CANDIDATE_REPORTED_VERSION}", + "expected_binary_sha256": "${MOLI_CANDIDATE_BINARY_SHA256}" + }, "moli-1.1.3": { "type": "headless_markdown", "engine": "moli", diff --git a/evaluation/dataset-catalog.json b/evaluation/dataset-catalog.json index f6e49728c..e047b749c 100644 --- a/evaluation/dataset-catalog.json +++ b/evaluation/dataset-catalog.json @@ -5,7 +5,7 @@ "name": "Web Fetch 评测", "track": "web_fetch", "dataset": "dataset.json", - "dataset_sha256": "9511b16bd27cfe6c2d7913917992cec574f73a8b5b642c19283ea542f90dab81", + "dataset_sha256": "5f751146143dcbe9daf46f0fb2511da1147508b3c8c4cd3377a6f3cceb82264e", "case_count": 150, "coverage_plan": { "web": [ @@ -610,7 +610,7 @@ "category": "public_institution", "track": "web", "task_prompt": "读取给定网页当前首屏加载完成后的整页正文;保留标题、正文、列表和表格,不要求翻页或点击展开。项目仓库页的正文目标为完整项目说明(README),不要求侧栏统计。", - "definition_sha256": "f9e3e58875e927521f93f67461f11c84756842464431e3cb4587d98a6317abe5", + "definition_sha256": "418e6f2b5bdd966d38e42f415c63ab22adf4b70e03daf66cf418b9ecfc984bcf", "reference_sha256": "ab62f0273ab9f9b792c99d2d021bb1fe7f6ff7fcf6af9935de7b65cc4443d2e8" }, { diff --git a/evaluation/dynamic-content-100-moli-webfetch/dataset-catalog.json b/evaluation/dynamic-content-100-moli-webfetch/dataset-catalog.json index d9c6e8c8d..d4be6c9d8 100644 --- a/evaluation/dynamic-content-100-moli-webfetch/dataset-catalog.json +++ b/evaluation/dynamic-content-100-moli-webfetch/dataset-catalog.json @@ -9,7 +9,7 @@ ] }, "dataset": "dataset.json", - "dataset_sha256": "b32c7ad678ddef0377868f88d8fb82b0f84c09b708f67e5688317db96eab9571", + "dataset_sha256": "0da2bb30aaabe3ca2b93ec17b4cb88cdc1cf3ea46336bb3777fdb21c1f211b84", "case_count": 100, "cases": [ { @@ -119,7 +119,7 @@ "category": "dynamic_content", "track": "dynamic", "task_prompt": "提取声明网址首次自动加载后的真实文字、列表条目、对应链接和正文。保留页面可见稳定文字;不点击、输入、滚动、翻页或使用OCR。动态成员不固定历史名单;同条目字段须对应完整。", - "definition_sha256": "4e9f21f4b2a39c1282197690083a275d505767249df6c14b3233c382c508dd62", + "definition_sha256": "bc740d6c0db8af1eabe3364600e9190ba62ceb0feed0ff589196b99bcf1a17ad", "reference_sha256": null }, { @@ -159,7 +159,7 @@ "category": "dynamic_content", "track": "dynamic", "task_prompt": "提取首次自动加载完成后的目标正文;不点击、不输入、不滚动、不搜索、不翻页。", - "definition_sha256": "590812f60d9eebe4db53ef19036d4bcb8d39c2a5e727a5cfcac729b0d444ec1c", + "definition_sha256": "cfd57dd7273d49da4a9792c5802eaab43969d1cf3412f85c55f377f397861187", "reference_sha256": "7d03bf2e3f739f5ac451815a0bba26082210da0d995c26516c87288f813df5c9" }, { @@ -349,7 +349,7 @@ "category": "dynamic_content", "track": "dynamic", "task_prompt": "提取首次自动加载完成后的目标正文;不点击、不输入、不滚动、不搜索、不翻页。", - "definition_sha256": "cfec76b111bd77e66e24efe11f74690bfd6de4b1dea4af9cff55c9ab9a4a6cf9", + "definition_sha256": "617fbaf8e8d8458d7457a7fc0c93ab44f97f58401fda21b0c9572faea9e39409", "reference_sha256": null }, { @@ -429,7 +429,7 @@ "category": "dynamic_content", "track": "dynamic", "task_prompt": "提取声明 URL 首次自动加载完成后的目标正文;不得登录、点击、输入、搜索、滚动、翻页或接受同意墙。排除导航、广告、推荐、页脚、Cookie、登录和订阅提示。", - "definition_sha256": "60177d04d06bc3bcf4ca43fddde548cc03f382e541b66bb1fe37bb978ad4c819", + "definition_sha256": "f3dfb5832ac20c50a2246db9991fa255594a126ae0a8f4ad584e2434c269b60e", "reference_sha256": "9429d55770b19f44d152aaeae72630e567326e675ae712f22844b86a54817c3b" }, { @@ -659,7 +659,7 @@ "category": "delayed_render", "track": "dynamic", "task_prompt": "读取初次自动加载完成后的 10 条记录,保留全部文字字段;不要求滚动、翻页或点击。允许渲染、解析内嵌数据或获取公开接口,不限制实现路径。", - "definition_sha256": "9af86098b6303bf838ae58438322b5ab38d6de51ddd8f0585c014376d64f68c0", + "definition_sha256": "fb358c76b50d670d547bc8a4b86839bc1cfbd6c343f289d13cfc33aa57d2fd5b", "reference_sha256": "1b0d230de9e4e8f91674cd3db383c68b17519c94da70b6d1dda382848fae4780" }, { @@ -719,7 +719,7 @@ "category": "embedded_frame", "track": "dynamic", "task_prompt": "父页自动加载的 iframe 中全部 14 个龟类科名称;不进入详情,不要求图片或按钮文字。允许渲染、解析内嵌数据、加载子文档或获取公开接口,不限制实现路径。", - "definition_sha256": "f9d3f2e96fa994b41f30d1c0eae566ff7357476b50ecff2c3555a5533e090370", + "definition_sha256": "d612b823798a1e3409a30c407a1e7cb52b22e7f4cf98608ed6095c1e3d164222", "reference_sha256": "e4183696f6e74f9016c0668ee9809eb2fe7b03c244bfdae2231d8e1a1326f9ca" }, { diff --git a/evaluation/dynamic-content-100-moli-webfetch/source-inputs.json b/evaluation/dynamic-content-100-moli-webfetch/source-inputs.json index 9ed46e436..9f61e28cb 100644 --- a/evaluation/dynamic-content-100-moli-webfetch/source-inputs.json +++ b/evaluation/dynamic-content-100-moli-webfetch/source-inputs.json @@ -1,13 +1,17 @@ { "schema_version": 2, "kind": "frozen_reference_inputs", - "id": "dynamic-content-100-moli-webfetch-b32c7ad678dd-v2", - "dataset": "dataset.json", - "dataset_sha256": "b32c7ad678ddef0377868f88d8fb82b0f84c09b708f67e5688317db96eab9571", + "id": "dynamic-content-100-moli-webfetch-0da2bb30aaab-v5", "catalog": "dataset-catalog.json", - "catalog_sha256": "81703f392b586931624692cb42a006843a9cbf02250fd9ea36d4ecb65ee1cb6e", + "catalog_sha256": "ca1572aa80fc2324b2f00800a79b053cfd584d1831ffb16e6906ae0981522c6c", + "dataset": "dataset.json", + "dataset_sha256": "0da2bb30aaabe3ca2b93ec17b4cb88cdc1cf3ea46336bb3777fdb21c1f211b84", "files": { - "dataset.json": "b32c7ad678ddef0377868f88d8fb82b0f84c09b708f67e5688317db96eab9571" + "dataset.json": "0da2bb30aaabe3ca2b93ec17b4cb88cdc1cf3ea46336bb3777fdb21c1f211b84" }, - "trees": {} + "trees": { + "sources": { + "dynamic-082-allocine-films/current-browser-observation.json": "717a0d1a864f561ae80c601c5733fccdfda16120e37b2ff4d0837c2718d86038" + } + } } diff --git a/evaluation/native-retrieval-contracts.json b/evaluation/native-retrieval-contracts.json new file mode 100644 index 000000000..4811fb0ed --- /dev/null +++ b/evaluation/native-retrieval-contracts.json @@ -0,0 +1,73 @@ +{ + "schema_version": 1, + "kind": "native_retrieval_contract_registry", + "contracts": { + "delayed-ten-records": { + "case_id": "dynamic-delayed", + "task_definition_sha256": "fb358c76b50d670d547bc8a4b86839bc1cfbd6c343f289d13cfc33aa57d2fd5b", + "source_authority": { + "path": "evaluation/sources/dynamic-delayed/render-audit.json", + "sha256": "cb72f36b479827c25816f1df5e2448885c0935ab61c0ad130bc145a2d466cc93", + "provider_output_used": false + }, + "capabilities": { + "completion": { + "selector": ".quote", + "minimum_count": 10, + "minimum_text_fragments": 3 + } + } + }, + "iframe-fourteen-family-headings": { + "case_id": "dynamic-iframe", + "task_definition_sha256": "d612b823798a1e3409a30c407a1e7cb52b22e7f4cf98608ed6095c1e3d164222", + "source_authority": { + "path": "evaluation/sources/dynamic-iframe/initial.html.txt", + "sha256": "c15cf12b2fa2c6d8c25c36b598fbcaca516fc93f87409d92eb9932a3644cca77", + "provider_output_used": false + }, + "capabilities": { + "include_subdocuments": true, + "completion": { + "frame_selector": "iframe", + "selector": "h3.family-name", + "minimum_count": 14 + } + } + }, + "iframe-list-one-hundred-items": { + "case_id": "dynamic-015-015-testpages-iframe-list", + "task_definition_sha256": "f3dfb5832ac20c50a2246db9991fa255594a126ae0a8f4ad584e2434c269b60e", + "source_authority": { + "path": "evaluation/sources/dynamic-015-015-testpages-iframe-list/mixed-session-stable-text.json", + "sha256": "cc1c90aec8f0550b7dc91b747abbe591e346ec4195fa75693a0c8e4ebb048800", + "provider_output_used": false + }, + "capabilities": { + "include_subdocuments": true, + "completion": { + "frame_selector": "iframe", + "selector": "li", + "minimum_count": 100 + } + } + }, + "iframe-one-form": { + "case_id": "dynamic-019-selenium-dev-selenium", + "task_definition_sha256": "cfd57dd7273d49da4a9792c5802eaab43969d1cf3412f85c55f377f397861187", + "source_authority": { + "path": "evaluation/sources/dynamic-019-selenium-dev-selenium/mixed-session-stable-text.json", + "sha256": "2931d10a0dec9fd1ba467e3e8dab7f311d4e996d1d3792cdf70c6e1ff2a41020", + "provider_output_used": false + }, + "capabilities": { + "include_subdocuments": true, + "completion": { + "frame_selector": "iframe", + "selector": "form", + "minimum_count": 1 + } + } + } + } +} diff --git a/evaluation/popular-sites/dataset-catalog.json b/evaluation/popular-sites/dataset-catalog.json index ebb27be7f..4090a6094 100644 --- a/evaluation/popular-sites/dataset-catalog.json +++ b/evaluation/popular-sites/dataset-catalog.json @@ -9,7 +9,7 @@ ] }, "dataset": "dataset.json", - "dataset_sha256": "0703badca3978cafe4f6db67fb9723f433781e1b87b4ef2af343756126e72cb5", + "dataset_sha256": "85544512d4cfd9459c22af1f70b1eb400844742dbf3179807662b76db2dcfc2e", "case_count": 100, "cases": [ { @@ -48,8 +48,8 @@ "url": "https://yuque.com/", "category": "popular_site", "track": "challenge", - "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:画板;互动交流;文档协同;权限管理;特色卡片;目录编排;知识管理;个人免费使用;团队免费使用;组件化编辑器;专业好用的编辑器;像书一样的知识库;Edge 插件;Chrome 插件;Help Center;Security;Terms;Join us;Pricing;Support;Welfare;About Us;Download;Feedback;Homepage;Developer;Contact us;Media reports;Product information;Template Gallery;Login / Register。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", - "definition_sha256": "fa553521956c988b34fda49a713d4236cb74d19922a5e9ecc2a6388e904f3032", + "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:画板;互动交流;文档协同;权限管理;特色卡片;目录编排;知识管理;个人免费使用;团队免费使用;组件化编辑器;专业好用的编辑器;像书一样的知识库;Edge 插件;Chrome 插件;Help Center;Security;Terms;Join us;Pricing;Support;Download/下载;Feedback;Developer;Contact us;Media reports;Product information;Login / Register。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", + "definition_sha256": "54a9ac13bc98bdcad886dea102e6b82af94fc6841776b50460ddf83716defe77", "reference_sha256": null }, { @@ -58,8 +58,8 @@ "url": "https://whatsapp.com/", "category": "global_homepage", "track": "challenge", - "task_prompt": "直接获取这个热门网站页面,并保留这些独立观察中确认的稳定普通网页文字:Apps;Blog;Log In;Mobile;Tablet;Careers;Desktop;Sitemap;About us;Download;Features;Contact Us;Learn more;Need help?;What we do;Who we are;Help Center;Brand Center;For Business;Use WhatsApp;WhatsApp Web;Keep in touch;Message privately;Never miss a moment with voice and video calls;Security Advisories;Terms & Privacy Policy。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", - "definition_sha256": "dcfd8b1e54ea580e56fd3cf5d4f7d120ec0e84e97bce1977fd882f132a837db1", + "task_prompt": "直接获取这个热门网站首次加载后可见的稳定正文,保留产品、功能、栏目和支持入口文字。不固定轮播内容、当前条目、顺序、价格、排名或数量;不点击子页面、不翻页、不滚动触发更多内容、不登录,也不执行其他 Browser Use 操作。图片文字、隐藏名称、网页标题和脚本文字不能代替正文。", + "definition_sha256": "7ee2656ed20ae532286dd7c9029aa79d72602ef1b26fbae7f86a34567d0b67ea", "reference_sha256": null }, { @@ -68,8 +68,8 @@ "url": "https://www.bing.com/?cc=us&setlang=en-us", "category": "global_homepage", "track": "challenge", - "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:Help;Maps;News;Legal;Images;Videos;Sign in;DISCOVER;Feedback;Shopping;Advertise;Text creator;About our ads;Image creator;Bing homepage quiz;Privacy and Cookies;Your Privacy Choices;Consumer Health Privacy。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", - "definition_sha256": "f8230467f8c7a831a354c342cead877524f30f562814f8b4f03af599d636e3ac", + "task_prompt": "直接获取这个热门网站首次加载后可见的稳定正文,保留产品、功能、栏目和支持入口文字。不固定轮播内容、当前条目、顺序、价格、排名或数量;不点击子页面、不翻页、不滚动触发更多内容、不登录,也不执行其他 Browser Use 操作。图片文字、隐藏名称、网页标题和脚本文字不能代替正文。", + "definition_sha256": "b08fe8228eb15dbc36695fd8fed6e5feaa081c755451f4e1f7f7ccb81f62e052", "reference_sha256": null }, { @@ -388,8 +388,8 @@ "url": "https://figma.com/", "category": "popular_site", "track": "challenge", - "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:Blog;Plans;Events;FigJam;Log in;Status;Careers;Company;Pricing;Product;Support;Dev Mode;Newsroom;Customers;Downloads;Resources;Templates;UI design;Use cases;UX design;Developers;Enterprise;Figma Design;Figma Slides;Integrations;Prototyping;Wireframing;Contact sales;Release notes;Legal and privacy;Online whiteboard;Build with intention;Design anything you can imagine;The intelligent canvas for infinite creativity。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", - "definition_sha256": "2a58e240ad573f15b77a457d3f1e94ac6a2b000136f450ea0d7e97339782e107", + "task_prompt": "直接获取这个热门网站首次加载后可见的稳定正文,保留产品、功能、栏目和支持入口文字。不固定轮播内容、当前条目、顺序、价格、排名或数量;不点击子页面、不翻页、不滚动触发更多内容、不登录,也不执行其他 Browser Use 操作。图片文字、隐藏名称、网页标题和脚本文字不能代替正文。", + "definition_sha256": "dfa4452952722c73a8ae1315edee484c9757a35ad7552058e0e03567dd71236c", "reference_sha256": null }, { @@ -398,8 +398,8 @@ "url": "https://www.deepseek.com/en/", "category": "global_homepage", "track": "challenge", - "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:More;Careers;Join Us;API Docs;Products;Research;API Pricing;DeepSeek App;DeepSeek Web;Terms of Use;Transparency;Privacy Policy;Service Status;DeepSeek Harness;Into the Unknown;Legal & Security;Use API Platform;Chat with DeepSeek;Report Security Issue。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", - "definition_sha256": "b2717adcdee3c3142defdb392ce3e5ec68e4ecc1c6f8046c7db9d805bce4f8d2", + "task_prompt": "直接获取这个热门网站首次加载后可见的稳定正文,保留产品、功能、栏目和支持入口文字。不固定轮播内容、当前条目、顺序、价格、排名或数量;不点击子页面、不翻页、不滚动触发更多内容、不登录,也不执行其他 Browser Use 操作。图片文字、隐藏名称、网页标题和脚本文字不能代替正文。", + "definition_sha256": "3fc32f903fa8c070c1b7c04dbe4a6d2b2c40f89383820b63249ea843a6380ced", "reference_sha256": null }, { @@ -488,8 +488,8 @@ "url": "https://usps.com/", "category": "global_homepage", "track": "challenge", - "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:FAQs;FOIA;Send;Shop;Careers;English;Español;Receive;Sign In;Support;Business;Feedback;Newsroom;PO BOXES;PASSPORTS;PostalPro;USPS Jobs;Contact Us;FREE BOXES;Site Index;Text Scams;Email Scams;Quick Tools;USPS Stamps;Click-N-Ship;Terms of Use;Top Searches;USPS Updates;Helpful Links;International;Privacy Policy;About USPS Home;Online Services;Postal Explorer;Search USPS.com;Informed Delivery;Inspector General;Stamps & Supplies;Track a Package;Find USPS Locations;Buy Stamps。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", - "definition_sha256": "f142bc76021b8d4176481ed6525950608b58a57c1e5b258202d6ee2154cf1437", + "task_prompt": "直接获取这个热门网站首次加载后可见的稳定正文,保留产品、功能、栏目和支持入口文字。不固定轮播内容、当前条目、顺序、价格、排名或数量;不点击子页面、不翻页、不滚动触发更多内容、不登录,也不执行其他 Browser Use 操作。图片文字、隐藏名称、网页标题和脚本文字不能代替正文。", + "definition_sha256": "975309122d6870f8e1713a06e4b4232feb0988e6195d13978e08dabe6b1bb4dc", "reference_sha256": null }, { @@ -608,8 +608,8 @@ "url": "https://feishu.cn/", "category": "popular_site", "track": "challenge", - "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:定价;审批;邮箱;云文档;知识库;下载飞书;产品功能;企业服务;关于飞书;加入我们;协同办公;博客中心;即时消息;友情链接;合作伙伴;多维表格;安全合规;帮助中心;应用目录;开放平台;技术支持;探索飞书;更新日志;服务协议;查看案例;管理后台;组织管理;联系我们;联系销售;营业执照;视频会议;解决方案;隐私政策;飞书人事;飞书招聘;飞书绩效;飞书项目;举报与反馈;飞书 AI;智能会议纪要;字节跳动旗下 AI 工作平台。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", - "definition_sha256": "314266967664c009389d3fc298b402aaa310d30eb2d737c41e1fba72cbce7078", + "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:定价;审批;邮箱;云文档;知识库;下载飞书;产品功能;企业服务;关于飞书;加入我们;协同办公;博客中心;即时消息;友情链接;合作伙伴;多维表格;安全合规;帮助中心;应用目录;开放平台;技术支持;探索飞书;更新日志;服务协议;查看案例/了解更多客户案例;管理后台;组织管理;联系我们;联系销售;营业执照;视频会议;解决方案;隐私政策;飞书人事;飞书招聘;飞书绩效;飞书项目;举报与反馈;飞书 AI;智能会议纪要;字节跳动旗下 AI 工作平台。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", + "definition_sha256": "b11303b1cccafbd5933c46e5f3de52c6b31e711efbc5599ca044fe73e8c469ba", "reference_sha256": null }, { @@ -668,8 +668,8 @@ "url": "https://dingtalk.com/", "category": "popular_site", "track": "challenge", - "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:专业版;专属版;中国站;标准版;下载钉钉;产品功能;付费产品;企业培训;企业百科;使用手册;加入我们;合作伙伴;商业版本;客户案例;市场活动;帮助中心;廉正举报;开放平台;服务协议;法律声明;注册钉钉;管理后台;组织管理;视频会议;钉钉动态;钉钉招聘;钉钉教育;钉钉文档;钉钉项目;支持与服务;服务与学习;安全合规中心;市场商务合作;钉钉隐私政策;案例与解决方案;钉钉服务状态中心。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", - "definition_sha256": "21b85dbce749d9d7ecd59901a839d5fb73d9d75e47553f7f8454f69f3bb8ffce", + "task_prompt": "直接获取这个热门网站首次加载后可见的稳定正文,保留产品、功能、栏目和支持入口文字。不固定轮播内容、当前条目、顺序、价格、排名或数量;不点击子页面、不翻页、不滚动触发更多内容、不登录,也不执行其他 Browser Use 操作。图片文字、隐藏名称、网页标题和脚本文字不能代替正文。", + "definition_sha256": "f78521edf38c4f11ae7c58fede9238ba4e47e77673d0daffa5d5f195e2ab99df", "reference_sha256": null }, { @@ -698,8 +698,8 @@ "url": "https://quark.cn/", "category": "popular_site", "track": "challenge", - "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:客户端;移动端;网页版;反馈入口;夸克网盘;校园招聘;用户协议;隐私协议;AI浏览器;iPad端;多设备体验;夸克PDF;夸克扫描王;浏览器插件;超级播放器;AI侧边栏:;PDF编辑:;搜索API;联系方式:;违法和不良信息举报;未成年人有害信息举报。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", - "definition_sha256": "9c01d5d31555e91efe12156819fddf266aa9e058913ec0d6575c7ef57af09618", + "task_prompt": "直接获取这个热门网站首次加载后可见的稳定正文,保留产品、功能、栏目和支持入口文字。不固定轮播内容、当前条目、顺序、价格、排名或数量;不点击子页面、不翻页、不滚动触发更多内容、不登录,也不执行其他 Browser Use 操作。图片文字、隐藏名称、网页标题和脚本文字不能代替正文。", + "definition_sha256": "8857d0962fa0e03e11198936e27ba4a59d55ac93d2da8744d971b0668dc91164", "reference_sha256": null }, { @@ -758,8 +758,8 @@ "url": "https://douyu.com/", "category": "popular_site", "track": "challenge", - "task_prompt": "直接获取这个热门网站页面,保留这些三次独立观察均出现的稳定普通网页文字:首页;直播;分类;赛事;视频;鱼吧;游戏;娱乐;户外;美食;关注;历史;消息;开播;下载;帮助。同时返回至少 10 条不同的具名详情入口;每条包含有意义的名称和 HTTP(S) 链接,链接路径完整匹配 /[0-9]+。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;不固定当前条目、顺序、价格、排名或数量。", - "definition_sha256": "3c757be2dccc1088c78772ab40864b3ba37bf483cc6399fecf7bddc11b898f8b", + "task_prompt": "直接获取这个热门网站首次加载后可见的稳定正文,保留产品、功能、栏目和支持入口文字。不固定轮播内容、当前条目、顺序、价格、排名或数量;不点击子页面、不翻页、不滚动触发更多内容、不登录,也不执行其他 Browser Use 操作。图片文字、隐藏名称、网页标题和脚本文字不能代替正文。同时返回至少 10 条不同的具名详情入口;每条包含有意义的名称和 HTTP(S) 链接,链接路径完整匹配 /[0-9]+。", + "definition_sha256": "c448cb3c4e2485921b0d7f073829d51e5fa9fac618d756ed617f72dff6e3dc42", "reference_sha256": null }, { @@ -878,8 +878,8 @@ "url": "https://lingxing.com/", "category": "popular_site", "track": "challenge", - "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:中文;价格;咨询;演示;首页;登录;白皮书;免费试用;关于领星;加入我们;快速响应;新闻中心;服务中心;核心功能;案例中心;物流追踪;立即体验;立即试用;获取报价;解决方案;跨境资讯;全周期服务;全方位保障;全生态产品;资源与支持;亚马逊ERP;多平台ERP;海外仓WMS;财务精准管理;进销存一体管理;领星ERP。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", - "definition_sha256": "228d4a87238776675c1603bbe7a818868e0e3e62c64ea648086f11aeb4d1ccba", + "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:中文;价格;咨询;演示;首页;登录;白皮书;免费试用;关于领星;加入我们;快速响应;新闻中心;服务中心;核心功能;案例中心;物流追踪;获取报价;解决方案;跨境资讯;全周期服务;全方位保障;全生态产品;资源与支持;亚马逊ERP;多平台ERP;海外仓WMS;财务精准管理;进销存一体管理;领星ERP。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", + "definition_sha256": "527b0968907c50b171456dac149b92a962b077bc1784776ca4a80a61830ad8d6", "reference_sha256": null }, { @@ -1008,8 +1008,8 @@ "url": "https://www.shopify.com/", "category": "popular_site", "track": "challenge", - "task_prompt": "直接获取这个热门网站页面,并保留这些三次独立观察都出现的稳定普通网页文字:Blog;Legal;Taxes;Global;Guides;Log in;Online;Themes;Careers;Courses;Domains;Pricing;Sitemap;Support;Payments;Products;Shipping;Shop App;Shop Pay;Sidekick;Analytics;App Store;Changelog;Discounts;Ecosystem;Investors;Resources;Affiliates;Build apps;Enterprise;Free Tools;Shopify POS;Shopify.dev;Theme Store;Why Shopify;Developer Docs;Hire a Partner;Privacy Policy;Service Status;Start for free;Sustainability;Community Forum;Compare Shopify;Website Builder;Shopify Checkout;Terms of Service;What is Shopify?;Shopify Help Center。不点击子页面、翻页、滚动触发更多内容、登录或执行其他 Browser Use 操作;图片文字、按钮控件文字、隐藏名称、网页标题和脚本文字不能代替这些文字。", - "definition_sha256": "697e494de49f481ae0834478820f03e67f68ccb1283a5b60d705a5d55b4beab4", + "task_prompt": "直接获取这个热门网站首次加载后可见的稳定正文,保留产品、功能、栏目和支持入口文字。不固定轮播内容、当前条目、顺序、价格、排名或数量;不点击子页面、不翻页、不滚动触发更多内容、不登录,也不执行其他 Browser Use 操作。图片文字、隐藏名称、网页标题和脚本文字不能代替正文。", + "definition_sha256": "15a609144a72478cdb24ab0e7d958c37384130f45053369864567e2413316edb", "reference_sha256": null } ] diff --git a/evaluation/popular-sites/review.json b/evaluation/popular-sites/review.json index a3bdb1bad..2622130a6 100644 --- a/evaluation/popular-sites/review.json +++ b/evaluation/popular-sites/review.json @@ -43,5 +43,23 @@ "reason": "This correction addresses only the clipped heading. The retained input-area prompt and other inherited criteria were not newly certified across historical page states." } ], - "case_rules_sha256": "c8226b15aa6c3544817b64e98b5a81e38d5ae665836ad667e93870580dae2089" + "case_rules_sha256": "40530454766310cefa68972792c3ee66cab94056264dac040bad3ab46b4dfb95", + "reference_contract_revision": { + "provider_calls": 0, + "case_ids": [ + "china-homepage-033-dingtalk-com", + "china-homepage-040-quark-cn", + "china-homepage-048-douyu-com", + "china-homepage-085-shopify-com", + "china-homepage-088-figma-com", + "global-homepage-008-whatsapp-com", + "global-homepage-010-bing-com", + "global-homepage-067-deepseek-com", + "global-homepage-086-usps-com" + ], + "source_qualification": "task-relevant intersection of two same-region independent first loads", + "provider_outputs_used_as_source": false, + "saved_call_url_route_runtime_changed": false, + "verdict_impact_bound_separately": "research/current-page-reference-contract-revision.json" + } } diff --git a/evaluation/popular-sites/source-inputs.json b/evaluation/popular-sites/source-inputs.json index 640950369..b165b42bb 100644 --- a/evaluation/popular-sites/source-inputs.json +++ b/evaluation/popular-sites/source-inputs.json @@ -1,13 +1,13 @@ { "schema_version": 2, "kind": "frozen_reference_inputs", - "id": "popular-sites-0703badca397-v2", - "dataset": "dataset.json", - "dataset_sha256": "0703badca3978cafe4f6db67fb9723f433781e1b87b4ef2af343756126e72cb5", + "id": "popular-sites-85544512d4cf-v6", "catalog": "dataset-catalog.json", - "catalog_sha256": "cf9b31ffe8486c158579cd543ae3f0ccf969ee4a3df8817b98b679192d087406", + "catalog_sha256": "f2b9dfb550230cdf8dbbba544d58158ac957620c1d064173790ead70754c178d", + "dataset": "dataset.json", + "dataset_sha256": "85544512d4cfd9459c22af1f70b1eb400844742dbf3179807662b76db2dcfc2e", "files": { - "dataset.json": "0703badca3978cafe4f6db67fb9723f433781e1b87b4ef2af343756126e72cb5" + "dataset.json": "85544512d4cfd9459c22af1f70b1eb400844742dbf3179807662b76db2dcfc2e" }, "trees": { "sources": { @@ -25,6 +25,7 @@ "challenge-10-justwatch/stable-page.txt": "1fdaee01d8cc8f0bf8f7fa82881643597555a15331463c73977d3a88948e5cfc", "china-homepage-008-doubao-com/evidence.json": "0a90ab08b6ccccc7fea5d649d90d76a1a96e2c93a215f711e8163372eda52698", "china-homepage-008-doubao-com/stable-page.txt": "76530bdcce1fb6759272e4a54593f485658cd8da00c27959284b41901fa7814a", + "china-homepage-012-feishu-cn/correction-observation-20261012.json": "4a1b14268f283ab5f57199c67837accc13da3d6dabc25d4990ddec85d413862b", "china-homepage-012-feishu-cn/evidence.json": "4e0d01e4972aedd70df3089b2f7e562ddb6fda9844c16f7a1ebf8bec13d7dfa8", "china-homepage-012-feishu-cn/stable-page.txt": "84297664cc2d46a139129346600d069a87a06ce7d042d42eb57d23b41fb04c40", "china-homepage-018-163-com/evidence.json": "6cf2d3528ec20b1b6cde576328381799aff02305c010c49416f41c0143ae0386", @@ -37,14 +38,16 @@ "china-homepage-028-weibo-com/stable-page.txt": "94e7e4c1a0f660f89fba469f516c4e20370ec86b06ce3d467d5456f30cf06e76", "china-homepage-029-360-com/evidence.json": "eddd0fd260c311f3947b20a6f229ec524af9444380e63664eb061a94d9d60e59", "china-homepage-029-360-com/stable-page.txt": "1e3452776fa5ce96fa9f9dd3b5ae22502564944c247a0ec26ed7d422b26ed2bb", - "china-homepage-033-dingtalk-com/evidence.json": "09d44b99430d84c559ff78ef33beb6562e4f46bb2255d3256ff69e3106c7a3f8", - "china-homepage-033-dingtalk-com/stable-page.txt": "7316417e47fb3c67eb3313d4b4248e3ee286e0c68394119546b386eb4e490e60", + "china-homepage-033-dingtalk-com/current-qualified-observations.json": "b124d67b50f17db6bc15b988ff9714c454c59165a5701fdcb0bbc557c0ce4bf8", + "china-homepage-033-dingtalk-com/evidence.json": "0c6c99e4d1a03653a48f1f061a63a9144a57f028905974bdf3f4f0da343f5b9b", + "china-homepage-033-dingtalk-com/stable-page.txt": "e0baff15f795c0d7ec2d6fe8575e0cca62a36bbfa4675ea48162e73aa324a5c3", "china-homepage-036-hao123-com/evidence.json": "a1aa8e3273f353fca54a3a1fb7587d021e25bc89315c92a2edecd7d177c8a3c0", "china-homepage-036-hao123-com/stable-page.txt": "dffd4da9eabd339f558f4916564f330f30aad006d80f304e905f66b0cf213f12", "china-homepage-039-1688-com/evidence.json": "61cb30bb5f148d5e5ec547db065731af558d2e1d969bc82901bb97d5c6052b6d", "china-homepage-039-1688-com/stable-page.txt": "f1449a6b781b629725722ca86dd6d5bc9145e1663dffa05974a2c98b56599d30", - "china-homepage-040-quark-cn/evidence.json": "1f4bcf0483c4d9939f752a95f894aa07221d51edfa2a4d5390dd7f9075c12354", - "china-homepage-040-quark-cn/stable-page.txt": "3206f822994495f731d6a477c4debc8211197a924657c082c3f447243ececcdc", + "china-homepage-040-quark-cn/current-qualified-observations.json": "4882f58b1dd814b7d4682343923e48c7dee9c3533b3908e22fdcd4e1659439c1", + "china-homepage-040-quark-cn/evidence.json": "47716e1234444171426adea72f57ed53ecdfec945be087b53db5bd33831ab4e6", + "china-homepage-040-quark-cn/stable-page.txt": "a4b50a21a1c4343176f5b622ca138200535f7818ce9b43d4c4c526ae0a2e9b09", "china-homepage-042-kdocs-cn/evidence.json": "92ce946484ecb71dcd2bbb067a362da610c9e98d7e4ce3e7ae9089feab7eb848", "china-homepage-042-kdocs-cn/stable-page.txt": "e490ca50c2664fb56cf3e4debf282fb2200e3767c3dcf3098a286987c42ecad6", "china-homepage-044-kimi-com/evidence.json": "a8f24f29dc974b244e5a03b2bc0ee2674d19cd5a2d3579069159b363c1393fd9", @@ -55,8 +58,9 @@ "china-homepage-046-wedoany-com/stable-page.txt": "d2c0e5746d11542b8f02c2ae0a255e12b7ed3e0c851b767708774c8fdc22600b", "china-homepage-047-qianwen-com/evidence.json": "ccf37aa501b9b650006c12839c519a1f0fbb06f62437baeb360a515c4988fe12", "china-homepage-047-qianwen-com/stable-page.txt": "0c4be8c605782e4e39bf4f16aab21dd75cbb5f203fb234004c5418fe4d3e3cd2", - "china-homepage-048-douyu-com/evidence.json": "77268705a97470660c5e894b287e1542dbe3ecc14e1e31ebf5da602a2c51c593", - "china-homepage-048-douyu-com/stable-page.txt": "24d412a26c0a7a0d46f4b8edad517d4b2026ed16b534b5771c4f0660eaf645f0", + "china-homepage-048-douyu-com/current-qualified-observations.json": "3239f658eaf00a0e9289757df24d5d575ffb015cb14108ec8033927194c2137a", + "china-homepage-048-douyu-com/evidence.json": "ac6f185790ed8e66b38fb868bc354e38bc0f1ffb8a85cae89a86b8bf499ad007", + "china-homepage-048-douyu-com/stable-page.txt": "d2dd3d0dbf5f4d72751fa360150c9ead29e8c01e82b26c2f001644e28c33febd", "china-homepage-050-openai-com/evidence.json": "18d36f820d5a1ce3cdfce8e160aa12db7470d879d7fb645f4d6266a704e57e94", "china-homepage-050-openai-com/stable-page.txt": "93bbcc875aceb84c71e7a114282e5539033ef42eecfaa88389f954aafd5a320b", "china-homepage-052-alipay-com/evidence.json": "f6254b37aa121c40a1ab3adeb6f5314ae87952e6ddbd68950344d0af3a05272b", @@ -79,6 +83,7 @@ "china-homepage-063-sina-com-cn/stable-page.txt": "6ad2e890ee0a4e0c374d896d9f82c0bf2308b6c93683dbfe6c9da664af2cd8a8", "china-homepage-064-gamer520-com/evidence.json": "6f0c4844c10118f1d65c13c7a0f49e5ab8915d86671f3f69b21a985096dd4202", "china-homepage-064-gamer520-com/stable-page.txt": "48af874737edfa5487ac28bd9880e754eb3b3bae026bbe928bd451509ebcd799", + "china-homepage-065-lingxing-com/correction-observation-20261012.json": "1518608faddd298bcd8afa15f3194824ef300d0e6e9c28be55216150fc14ef8c", "china-homepage-065-lingxing-com/evidence.json": "189ac9635512cf25f9a43f24d63a4a1992a6c0ac4afa8aefb47e0b6ad50b1bd7", "china-homepage-065-lingxing-com/stable-page.txt": "d8726d87e456ced66dff453ac5ce7f8a9dca69ed95af62924b133f8284c48f07", "china-homepage-067-alibaba-com/evidence.json": "416ba8de2d332a2d3b112aa37b6b26885e45530cfbb16494217f5dc6ac0d7e7e", @@ -105,12 +110,15 @@ "china-homepage-082-huawei-com/stable-page.txt": "609fb1909e70af8c0227d4428a11a3615f4766fb6edd70f4eacdbc5ab6388b6a", "china-homepage-083-grok-com/evidence.json": "e99f06310153bba8c9a372bd8fc9a173cca6defae75020e7387894b62b4e397d", "china-homepage-083-grok-com/stable-page.txt": "7af7419795e4d8c9199a413c6df023030ac3f00eb28d3c9799d1cd0cb7d2930b", - "china-homepage-085-shopify-com/evidence.json": "dc3096a3e99e9022b7e8dfc728ec4c5463b1b847426f2d717833424f20759ef1", - "china-homepage-085-shopify-com/stable-page.txt": "23e8ccf180acd0d78fafcbe06fac19e8aafe754f47bf75092932d96a557cd9cb", + "china-homepage-085-shopify-com/current-qualified-observations.json": "db803b5d4ce0b5628c39497f6e7750523eda54e8b40500e295b46cf26ff05d06", + "china-homepage-085-shopify-com/evidence.json": "fa8daffa836135c90dd5c53b5dfa404ff63667663a968eb51089d5d3a80ff5df", + "china-homepage-085-shopify-com/stable-page.txt": "301348f5d47149de15f6e94bfefffc14a9cb2750ef6aa0b23d98c7e1b6d217ac", "china-homepage-087-gamersky-com/evidence.json": "48e10145c49801329fa9a49c4b94e37c4f44f148e1100f667753f2685f216653", "china-homepage-087-gamersky-com/stable-page.txt": "ee53a5fad5eb1bfe0c794bfdd8df5cc25122b25249f2212ec25be1f75e1b09d3", - "china-homepage-088-figma-com/evidence.json": "539dcce9c42fdc307ffbb87a73464cddb3d419c756a879e97fddfb7f0d5c5aec", - "china-homepage-088-figma-com/stable-page.txt": "d90df27d57dac68552dc0a272ad1fa0280e7ae0e10ecb923a87beb82c00b319f", + "china-homepage-088-figma-com/current-qualified-observations.json": "bb868fdfb8ccaf39f6ad734207d1ec7f195e162c4598efc17bf1391c4e78cc44", + "china-homepage-088-figma-com/evidence.json": "65f2aa1c2218e158f803a3e6d10071c8a13a12972a9a0788a6313cc28ec2b707", + "china-homepage-088-figma-com/stable-page.txt": "4e6039ccd6cbe4c2d7a9fe75a8bdb6d72f7677b866e9d61bcd7a20f9d0817101", + "china-homepage-089-yuque-com/correction-observation-20261012.json": "390555ccd00108f6fc5dcb3d39767d3eca1b6510b72c9960d8d317e4e299807f", "china-homepage-089-yuque-com/evidence.json": "3d5b628c641ff70fedd166d02e5a0df21ddc187ce6a8feed1abcf52c42cd805d", "china-homepage-089-yuque-com/stable-page.txt": "b25d34580413cd6303983e5d4f12cbcdceaee759191b9172f8f5776a448f3809", "china-homepage-090-toutiao-com/evidence.json": "8da295bd282df6734a1a0fd3d9abef26da8519950298643da869fcc852f3eee6", @@ -121,10 +129,12 @@ "global-homepage-002-youtube-com/stable-page.txt": "d631d5da5efccfd75f3d1b5d233a12e0b2d012a3e071c5dd1195ff753a26a5ba", "global-homepage-005-chatgpt-com/evidence.json": "e9810f82224031b06e73dd2f8c446e30684c1b81b0be8d464e5699325b1780ed", "global-homepage-005-chatgpt-com/stable-page.txt": "b858ebd9b68bb0bd57eba393ce24398ca89617e5d654eeb69f83e45f554ae792", - "global-homepage-008-whatsapp-com/evidence.json": "49cf7e78f494e52fd463e354bf65ab6421abd3a714db3bba37ba9e4a398dfe18", - "global-homepage-008-whatsapp-com/stable-page.txt": "a576b8bc14483d3ca37a1d9785ddb7633bf53193b270a8d7b8acf154154ee69c", - "global-homepage-010-bing-com/evidence.json": "109c2c1f105cc8e4d143d079b1fcdd45b63a5ac422f1835e9e5216d4cc237ddb", - "global-homepage-010-bing-com/stable-page.txt": "ce637158bf39da31accbb45048be6bd203196e57aeb2e7c99440ffcb33d6ee58", + "global-homepage-008-whatsapp-com/current-qualified-observations.json": "3d379cb138e21709683e314d07bac7576279bfedea8796af61c4baecbaff0584", + "global-homepage-008-whatsapp-com/evidence.json": "5a84bb342c0d3c305212f85f0cd1963ceafe9a605a8a0f8bf3ffbc59e2a23109", + "global-homepage-008-whatsapp-com/stable-page.txt": "9c31244c560263a85aa50a119714d46a94cf8a1df4b5ae40dbcd62917862f736", + "global-homepage-010-bing-com/current-qualified-observations.json": "76782d2f51b5f1b47ccd1cdbc461515247f2041067ee3e4c49524d0e16789205", + "global-homepage-010-bing-com/evidence.json": "2559c3dea074a30bfc8f18f0f42c63f8cba2017bd8a488a02c1bec0e0b1204a7", + "global-homepage-010-bing-com/stable-page.txt": "8c0125d722106272f53fcadb131ecf4301cbec62259e1ca11fc02e01db195244", "global-homepage-011-wikipedia-org/evidence.json": "b74ddb43c6b9b2f2b8d42df033dd803cd6c48f54a59d1d4de1cbf3f416cdb1e2", "global-homepage-011-wikipedia-org/stable-page.txt": "9222ed451b4beee228a9e7e89b9a4bc721ea50d77da72e43dad3c853addf84da", "global-homepage-012-gemini-google-com/evidence.json": "97df3691ca324a26bcaed27ee795ef482127a3ce1282b23366c89fc932f8eb5e", @@ -187,8 +197,9 @@ "global-homepage-062-bbc-com/stable-page.txt": "3699ad1fb9f20bfbed633a77378aa2e9ca300dff9c31c5625bf67e812d86d482", "global-homepage-063-paypal-com/evidence.json": "4e08872b3f34ffa06fd3638679c65c2b5e494ceabb04191903c2fad32e29fdfa", "global-homepage-063-paypal-com/stable-page.txt": "b75caa76c1dec7b440d999da48ad8b36ada40d720ba159400e54219991a05dfe", - "global-homepage-067-deepseek-com/evidence.json": "ac6b9d1e223d9bbfdc4aa66f21ff8b96cd554e86b6f49aff4ef088f4f23adadc", - "global-homepage-067-deepseek-com/stable-page.txt": "233605615307da3fcf2cdc68ec872e7f69ef8c314c29a3bdaf1ea7fb2ed90ed2", + "global-homepage-067-deepseek-com/current-qualified-observations.json": "2ebbde4f22e3a45c6c3a771406609c6443fc884b78fcedaa02fb786095a9b093", + "global-homepage-067-deepseek-com/evidence.json": "286c630b63653be900e7513da5b9b12f4ed55f9a38f0cd5bd856b6a691f4244c", + "global-homepage-067-deepseek-com/stable-page.txt": "610313e71b04aaa781010e3f55a5a69cacdad3fad0b362f01f664e323a2234eb", "global-homepage-068-indeed-com/evidence.json": "28093bd02ad0fabcb84c72a45b91110341121ba4743c46f32e413051240e4aab", "global-homepage-068-indeed-com/stable-page.txt": "100062ef93cc1fa5a6cd361cef9a80cf5e043a1ed92f6ce47054651876dfb8b4", "global-homepage-072-rakuten-co-jp/evidence.json": "34e00b297f4c8fdae7b11c01ec8d709124bea0e9f7de9dad1327d65f559a2d5d", @@ -205,8 +216,9 @@ "global-homepage-084-qq-com/stable-page.txt": "9f2bb4267630ea8e88fa48615a0d44103794136e173cff1203b4a78d5728ce9d", "global-homepage-085-douyin-com/evidence.json": "8487abcc33fb10090ce9f2107181d44d82bdc142ac840ab479d6566ef0508197", "global-homepage-085-douyin-com/stable-page.txt": "5fc11c6c89dc547fe0bbada6f3d4107f6c936a8cc95dc3aeb9bd6087f222cdc5", - "global-homepage-086-usps-com/evidence.json": "8575340ba753b687210ff05565ef083f06270d46808d1fcce4111ef452b79000", - "global-homepage-086-usps-com/stable-page.txt": "b9d2e87bb0fd1e90f0de44ee8af80cf64f2437ce1d47170c46de888a9f44f331", + "global-homepage-086-usps-com/current-qualified-observations.json": "556031894e8d797f93fff54372a616cf0618f1cbd664fa7d5cf093fc0acfcd2b", + "global-homepage-086-usps-com/evidence.json": "9724f803aab787e981bf27852a6b42b815aa260242a9a044fbf3bed8ede38322", + "global-homepage-086-usps-com/stable-page.txt": "50f93f88e7308cd90fa9f727e730c00b30201b38598b1be8d184c8a6807ff444", "global-homepage-087-instructure-com/evidence.json": "f79426c53dda1fd78353f3c0068c3feba531c3464c241bd582d145805ae69434", "global-homepage-087-instructure-com/stable-page.txt": "e360081a38a406698504d6b1071bb99bf0d4e9f45fef719158d363e67e355700", "global-homepage-090-rutube-ru/evidence.json": "50dd43e30d5663cce790e71bf6731050c640a131448ae04872d9ad38f06eda40", diff --git a/evaluation/publication-archives.json b/evaluation/publication-archives.json new file mode 100644 index 000000000..554db217b --- /dev/null +++ b/evaluation/publication-archives.json @@ -0,0 +1,11 @@ +{ + "schema_version": 1, + "archives": [ + { + "id": "webfetch-browser-native-430-2026-10-15", + "site_url": "https://artifact-site.lexmount.cn/v/IqGpMUYHWat-dw4n8usQttTptoPzboAo", + "artifact_version": "ver_b5584a3d77524f5cbced6edd93105445", + "profile_sha256": "4144fec0ca5a3d0cccf3c31075ac1c55c0bef2bc3ec27f4c49b40b838250b272" + } + ] +} diff --git a/evaluation/risk-control/README.md b/evaluation/risk-control/README.md index ebe2f8895..861c825f5 100644 --- a/evaluation/risk-control/README.md +++ b/evaluation/risk-control/README.md @@ -4,7 +4,7 @@ This cohort asks whether Moli 1.1.11 can retrieve the declared content present o The scope is 100 URL cases across 74 distinct sites. The executable `dataset.json` is external and bound by `source-inputs.json`; `dataset-catalog.json` retains the inventory and definition hashes. Import approved full references using the [reference-input guide](../../docs/reference-inputs.md); they are not distributed with source. -Every case requires all declared stable text from initial load, including registered navigation, buttons and sidebars. Missing any required phrase fails. Another 79 cases require complete body regions; 21 cases use 27 independent count groups for names, real links, paths, deduplication and same-record fields. Human rules, machine contracts, qualification and replacement provenance are held in `first-30-rule-review.md`, `remaining-70-rule-review.md`, `body-contracts.json`, `list-contracts.json` and `qualification/`. Qualification explains historical cohort admission; it is not a scoring input. Raw investigations remain outside Git. `run-profile.json` binds authoritative inputs, provider configuration, execution parameters and binary identity. +Body extraction is accepted against the declared body regions and composite record contracts. The compiler marks stable text as diagnostic for the 97 cases with either contract; the remaining three cases still require every stable-text group. All 1,586 independently observed stable-text items remain available for diagnostics. There are 79 body-region cases and 23 composite cases, including 21 quantity cases with 27 independent count groups for names, real links, paths, deduplication and same-record fields. Human rules, machine contracts, qualification and replacement provenance are held in `first-30-rule-review.md`, `remaining-70-rule-review.md`, `body-contracts.json`, `list-contracts.json` and `qualification/`. Qualification explains historical cohort admission; it is not a scoring input. Raw investigations remain outside Git. `run-profile.json` binds authoritative inputs, provider configuration, execution parameters and binary identity. ## Deterministic build and validation without calls @@ -13,7 +13,7 @@ uv run --frozen python scripts/build_risk_control_dataset.py --output /tmp/risk- uv run --frozen python scripts/run_certified_benchmark.py risk-control --env-file .env ``` -The first command rebuilds from hash-bound independent evidence under `BROWSER_EVAL_DATA_HOME` (default `~/.browser-eval`). Output must match the bound external dataset byte for byte. Supply a new output path; frozen inputs cannot be overwritten. The second command validates the profile, dataset, Moli 1.1.11 version and binary hash without provider calls. +The first command rebuilds from hash-bound independent evidence under `BROWSER_EVAL_DATA_HOME` (default `~/.browser-eval`). The compiler emits the corrected stable-text roles. The historical profile and bound external dataset retain their original bytes; they must not be overwritten or treated as a certificate for the revised contract. Supply a new output path; frozen inputs cannot be overwritten. The second command validates the profile, dataset, Moli 1.1.11 version and binary hash without provider calls. The compiler is `harness.web_fetch.risk_control_dataset`. When installed or called from another directory, pass `--definition-dir` with an explicit copy of this directory's rules and qualification manifests. Frozen-rule hashes are still enforced. @@ -25,17 +25,26 @@ uv run --frozen python scripts/run_certified_benchmark.py risk-control \ --output "$BROWSER_EVAL_DATA_HOME/evidence/risk-control/" ``` -Output must be a new directory under the data root. The certified entry point is `scripts/run_certified_benchmark.py risk-control`; direct `--execute` on the execution script is rejected. A new run under the current profile has exactly 100 logical results and no automatic retries, retaining complete hash-bound debug stderr outside Git. Historical evidence additionally contains three user-authorized calls after DNS failures: 103 physical calls in total. The final response is authoritative, all call costs remain recorded and no best-response selection is allowed. +输出目录必须是新的,并位于数据目录下。正式入口固定为 +`scripts/run_certified_benchmark.py risk-control`;直接对执行脚本使用 `--execute` 会被拒绝。 +当前 profile 的新正式运行恰有 100 个逻辑结果,且不自动重试,并为每次调用在 Git 外保留完整、 +哈希绑定的 debug stderr。保留的历史证据另有 3 个 DNS 失败后的用户授权补调,因此保留 103 次 +物理调用;每例固定采用最后一次响应,全部调用耗时均计入,不能择优。运行前后 Moli 身份必须 +一致;profile、已编译的数据集、规则和 provider 配置也会在调用前后核验。 +普通重跑不读取历史资格原文,也不重做入选审核;修改题目、参考或规则时,才由构建器 +核验相应来源。新运行生成 `frozen-input-verification.json` 和 v2 认证回执。 +生成报告前校验 `certification.json` 对 profile、dataset、results、manifest、execution、 +冻结输入核验回执和完整日志的绑定。历史 v1 回执继续按原证据契约读取,不改写或升级。 Moli identity must remain unchanged before and after calls. Each of the 100 declared references is checked before and after execution; missing, changed or incomplete scope stops certification. Report builds verify `certification.json` bindings to the profile, dataset, results, manifest, execution, reference-verification receipts and complete logs. This directory contains no measured Moli outputs, raw provider artifacts, debug logs or credentials. Use the existing report builder with complete certified evidence. Historical results lacking pre/post-call reference certification cannot be promoted by writing a receipt afterward. -“Full page” means the declared stable text, body and list requirements within initial load. It does not claim every undeclared character or content revealed by interaction. Registered navigation text is as mandatory as body text. +“Page body” means the declared body and record requirements within initial load. Missing navigation, login links or footer text cannot reject content that satisfies those requirements. Blocked-page detection, record counts, same-record fields and real links remain acceptance gates. Cases without a body or composite contract retain their stable-text gate. ## Current contract and historical results -The current contract requires 1,600 stable-text items. `research/risk-control-full-page-rule-revision.json` binds prior contracts and the full-page revision, retaining all registered peripheral text without rewriting earlier receipts or provider outputs. The historical 31/100 belongs to earlier inputs and evidence and is not a score for the current contract. +The historical full-page contract required all 1,600 stable-text items. The body-scope correction assigns their acceptance role without changing the observed text or reference content. `research/risk-control-full-page-rule-revision.json` binds prior contracts and the full-page revision, retaining all registered peripheral text without rewriting earlier receipts or provider outputs. The historical 31/100 belongs to earlier inputs and evidence and is not a score for the current contract. The current adapter obtains identity from `final_url`, status and final HTML in the same `fetch --dump json` response, then converts to Markdown within the same deadline. GET completion order in old debug logs cannot prove redirect ownership; insufficient identity evidence remains unknown. A changed call route needs new execution over the complete affected scope for formal scores and timing; old-route logs and timings cannot substitute. diff --git a/evaluation/risk-control/body-contracts.json b/evaluation/risk-control/body-contracts.json index 5133aee33..607d06089 100644 --- a/evaluation/risk-control/body-contracts.json +++ b/evaluation/risk-control/body-contracts.json @@ -363,6 +363,7 @@ ] }, "risk-diverse-042": { + "stable_text_role": "diagnostic", "regions": [ { "lines": [ @@ -1199,6 +1200,7 @@ ] }, "risk-diverse-011": { + "stable_text_role": "required", "regions": [ { "lines": [ diff --git a/evaluation/risk-control/current/dataset-catalog.json b/evaluation/risk-control/current/dataset-catalog.json new file mode 100644 index 000000000..9152103ba --- /dev/null +++ b/evaluation/risk-control/current/dataset-catalog.json @@ -0,0 +1,1011 @@ +{ + "schema_version": 1, + "kind": "frozen_dataset_catalog", + "id": "risk-control-audited-100", + "name": "风控网站正文评测", + "dataset": "dataset.json", + "dataset_sha256": "165b888133e859304b035d74183dd40bf7ab609cac78bb6ae6e75b645c1de426", + "case_count": 100, + "cases": [ + { + "id": "risk-diverse-002", + "name": "知乎:人工智能应用场景回答", + "url": "https://www.zhihu.com/question/282715644/answer/3629347314", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "4e591a4595a0554b7f86a0eba1c601531f7fed481500da14494036310e974f70", + "reference_sha256": null + }, + { + "id": "risk-001", + "name": "Flipkart:Acer Aspire Lite 商品详情", + "url": "https://www.flipkart.com/acer-aspire-lite-amd-ryzen-3-quad-core-5400u-8-gb-256-gb-ssd-windows-11-home-al15-41-thin-light-laptop/p/itm1bc0bcb4598e7", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "119d84a124f4aaa94f60f41383c27b0b74b7a8185a003567ddbfb05b39b89d40", + "reference_sha256": null + }, + { + "id": "risk-006", + "name": "微博热搜", + "url": "https://s.weibo.com/top/summary", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "65eff51f32d754895c70737b8ce336e47606a575b7826b738cbcc4e4ec763e10", + "reference_sha256": null + }, + { + "id": "risk-011", + "name": "今日头条:关于人工智能,我们了解多少", + "url": "https://www.toutiao.com/article/7595028554912399898/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "f899d00a5c7a6303492637698931df6c99167bc66ed6066f15ae3e14959fcd0e", + "reference_sha256": null + }, + { + "id": "risk-061", + "name": "Dribbble:web-design 搜索结果", + "url": "https://dribbble.com/search/web-design", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "fd97c97d7512ccec16bb9c2ac9ed18fd3c86b87ee3fb7ab306feb33cf655d67e", + "reference_sha256": null + }, + { + "id": "risk-101", + "name": "哔哩哔哩:综合热门", + "url": "https://www.bilibili.com/v/popular/all/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "82de10eea401cff30b9c7302a18c112063a1e0f2fc44f3121844594b1429879a", + "reference_sha256": null + }, + { + "id": "risk-102", + "name": "腾讯视频:电影频道", + "url": "https://v.qq.com/channel/movie", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "a1a5341179ecce8397ee75b09b292ea72f5e184a936419b2e9cead7528d7e142", + "reference_sha256": null + }, + { + "id": "risk-diverse-271", + "name": "About Sanctum repository service", + "url": "https://dev.sanctum.geek.nz/cgit/?p=about", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "8f7e99e10858cf25406a2fa9bbb199b50befe2dd120e0915574a59e08ab31438", + "reference_sha256": null + }, + { + "id": "risk-diverse-027", + "name": "Linux Kernel Git", + "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "29ff371f48f33a097d310f071ef25194d302532ca923861f7625b8d228bf0060", + "reference_sha256": null + }, + { + "id": "risk-new-102", + "name": "DebianInstall", + "url": "https://wiki.debian.org/DebianInstall", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "9518f4ff3218de56c9da64cc9948c8acdcebd874aa02ac7fcd5c0b5f4f42af0f", + "reference_sha256": null + }, + { + "id": "risk-diverse-030", + "name": "Wine Bugzilla", + "url": "https://bugs.winehq.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "520ec13a5e8f641a324ab421e3118d3df3159dee4aa61bebca0fbbe5b85b3391", + "reference_sha256": null + }, + { + "id": "risk-diverse-031", + "name": "Science Olympiad Student Center Wiki", + "url": "https://scioly.org/wiki/index.php/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "f243ee8a571dc97e840eb5b5e5cb00a304bd2727987b4e659777a07224cbbe15", + "reference_sha256": null + }, + { + "id": "risk-diverse-032", + "name": "Enlightenment Git", + "url": "https://git.enlightenment.org/enlightenment/enlightenment", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "75827072b43b9bb530435287df09ce05eb9aa63035d08730922df86093f7b5e8", + "reference_sha256": null + }, + { + "id": "risk-diverse-033", + "name": "FreeCAD Forum", + "url": "https://forum.freecad.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "338297148e1b45116d6ef782ed93ae02bec71f5de2846ebbcdbc3c0bcdacaee3", + "reference_sha256": null + }, + { + "id": "risk-diverse-034", + "name": "ScummVM Wiki", + "url": "https://wiki.scummvm.org/index.php/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "09277073909b6f4e6889f83cd0773450bdbe7fb385a934140451fc34b6954be2", + "reference_sha256": null + }, + { + "id": "risk-diverse-036", + "name": "OpenWrt Wiki", + "url": "https://openwrt.org/start", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "efc2a8c10e8a109cbfe8df740392b423fe314037caac42fe2d1d7c1ce2fc626d", + "reference_sha256": null + }, + { + "id": "risk-diverse-038", + "name": "ArchWiki", + "url": "https://wiki.archlinux.org/title/Main_page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "30a2268fda5aa866597e5a75453e0e45fcbda67cca4961ab2f4278ed48e1707a", + "reference_sha256": null + }, + { + "id": "risk-diverse-039", + "name": "FreeBSD SVNweb", + "url": "https://svnweb.freebsd.org/base/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "4599e13e815f3c5595df7d573341e0fdd08780daf2761db24d489d3a1c5f0e01", + "reference_sha256": null + }, + { + "id": "risk-diverse-040", + "name": "Purism GitLab:Phosh", + "url": "https://source.puri.sm/Librem5/phosh", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "b6ca53feac1c9ff38ff47ca7ff872cd1c5b44a199edcf138689b3157c34dda37", + "reference_sha256": null + }, + { + "id": "risk-diverse-041", + "name": "Haiku Trac", + "url": "https://dev.haiku-os.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "cbff3640b4e71d5c49fcd1af6e767f93760f8133de56a7ab7f3b5f29d58c84c7", + "reference_sha256": null + }, + { + "id": "risk-diverse-042", + "name": "Devuan Git:installer-iso", + "url": "https://git.devuan.org/devuan/installer-iso", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "34d554128a209a16c28b091b87ae22ee2ad63f91b62212c51a76d3fcf4887269", + "reference_sha256": null + }, + { + "id": "risk-diverse-045", + "name": "CFA Archive", + "url": "https://www.cfaarchive.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "75c4ba4f48dc99fc2661f8ae3fc9fe2803b2c633ce6380d83066b9ba759b7858", + "reference_sha256": null + }, + { + "id": "risk-diverse-047", + "name": "Proxmox Bugzilla", + "url": "https://bugzilla.proxmox.com/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "024b598aa2297dd6da4bb324f2440f76939cd003e1d8100507998cbc7c610b8d", + "reference_sha256": null + }, + { + "id": "risk-diverse-048", + "name": "ReactOS Wiki", + "url": "https://reactos.org/wiki/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "6e297e1ffb8738d6f1057f189f88df128505dea03df8240228a40c0e160371f2", + "reference_sha256": null + }, + { + "id": "risk-diverse-049", + "name": "Arch Linux 32 Packages", + "url": "https://www.archlinux32.org/packages/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "36a18f11e670411c7969a26b4095de6a5b1bfe2cea37c111dbce440764e11991", + "reference_sha256": null + }, + { + "id": "risk-diverse-050", + "name": "Slackware Git", + "url": "https://git.slackware.nl/current/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "f77638953c120ea4628d342aa73693d91e9ed7c98a40a27991dd9e181a719223", + "reference_sha256": null + }, + { + "id": "risk-diverse-051", + "name": "Duke Digital Repository", + "url": "https://repository.duke.edu/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "7c67f961f5afbd2e9f343bc5eca6f5a783ebaa63354b5832f89376d61857aefd", + "reference_sha256": null + }, + { + "id": "risk-diverse-055", + "name": "hebis Discovery System", + "url": "https://tufind.hds.hebis.de/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "670c2f60030ecc27bf314eade1ac6116c2b2697803156c9483ccdfefff87444b", + "reference_sha256": null + }, + { + "id": "risk-diverse-057", + "name": "Xe Iaso", + "url": "https://xeiaso.net/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "db60c0ebf6aad7b76632735d6acf208e7f3d8043399efb5488a1df686460d756", + "reference_sha256": null + }, + { + "id": "risk-diverse-059", + "name": "Superlove", + "url": "https://superlove.sayitditto.net/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "5a5131232354f0e34134945124cccdf809ba7d5e23ae710683d35063d53bc4b8", + "reference_sha256": null + }, + { + "id": "risk-diverse-074", + "name": "Ticketmaster concerts", + "url": "https://www.ticketmaster.com/discover/concerts", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "93a4aec34c617dda96b5f4f3e06a77a6d8dd3abb53fe2b0c2953d5f668f744ee", + "reference_sha256": null + }, + { + "id": "risk-diverse-101", + "name": "Azur Lane Wiki", + "url": "https://azurlane.koumakan.jp/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "fc3104cfd5f22fd321503e226b57f54088c6c059feafb509bb703c79d438a52b", + "reference_sha256": null + }, + { + "id": "risk-diverse-104", + "name": "Sanctum Development", + "url": "https://dev.sanctum.geek.nz/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "33532667a459ec873b69c0b104d8d6b3e51c6f0a41702ad4d1803901875d4a29", + "reference_sha256": null + }, + { + "id": "risk-diverse-106", + "name": "TYPO3 Extension Repository", + "url": "https://extensions.typo3.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "a59ca8301806bdb7172cf6ea4c21f222448e231c24855932c91a2a2b32e46524", + "reference_sha256": null + }, + { + "id": "risk-diverse-113", + "name": "MozillaZine", + "url": "https://mozillazine.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "0b37f86d1c5102eb5a3d656e5454f9016a639c9f9d28810c24650120d6903eda", + "reference_sha256": null + }, + { + "id": "risk-diverse-114", + "name": "Pluralpedia", + "url": "https://pluralpedia.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "c8d96f1796c4871b36eb5be85de6e81c6beb5ea7727263d6a31f6b37a6730e19", + "reference_sha256": null + }, + { + "id": "risk-diverse-117", + "name": "RPM Fusion", + "url": "https://rpmfusion.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "670fb01613742e048a7a8e4a1fdda5cb00d6b05108c35f38b8a65354679c34eb", + "reference_sha256": null + }, + { + "id": "risk-diverse-118", + "name": "SquirrelJME", + "url": "https://squirreljme.cc/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3ed4808cbe6fd9abb5321ec9fb889a6f9ea96fb986203d9911435cca4ff9deae", + "reference_sha256": null + }, + { + "id": "risk-diverse-121", + "name": "Dolphin Emulator Forums", + "url": "https://forums.dolphin-emu.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "dc00ef5fe7fc8871581e1618ddcf7004b80c4f1fa209632019e7b05139632728", + "reference_sha256": null + }, + { + "id": "risk-diverse-122", + "name": "Free Pascal Wiki", + "url": "https://wiki.freepascal.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "9c7916dafe96857c32d60fabe4769fc7df4eade65fded00f3e093e340af43163", + "reference_sha256": null + }, + { + "id": "risk-diverse-123", + "name": "Koha Community Wiki", + "url": "https://wiki.koha-community.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "889884b2d86f23bea6bd99b7a856fd2f43edb567fdd76ae0b591f728252296a6", + "reference_sha256": null + }, + { + "id": "risk-diverse-129", + "name": "HydroOJ FAQs", + "url": "https://ws.hydrooj.com/d/faqs/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "e00b6b37cc32127897a7a48740a2a50516022c268ea7a7768b4ca184cabdf16e", + "reference_sha256": null + }, + { + "id": "risk-diverse-134", + "name": "Zillow Homes for Sale", + "url": "https://www.zillow.com/seattle-wa/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "478c7dc7335cf4316c7b0dfae4753cfbe9b1b72bc7953059613a4a0a43f64820", + "reference_sha256": null + }, + { + "id": "risk-diverse-135", + "name": "Go.Compare", + "url": "https://www.gocompare.com/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "116b80a20713f4629f41da494fcec44eb945b7197719150bfa5f63b1b5f2afd3", + "reference_sha256": null + }, + { + "id": "risk-diverse-138", + "name": "D Wiki", + "url": "https://wiki.dlang.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "96840e0deebc9ceea29abf68ffc68f5de228864e500767f0da65d75a693597c6", + "reference_sha256": null + }, + { + "id": "risk-diverse-139", + "name": "DSource", + "url": "https://dsource.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "884c24446fca078cf14d202d6f63b2c9187d5e1053b75e2a6df02ae3f4e5882d", + "reference_sha256": null + }, + { + "id": "risk-diverse-153", + "name": "LZNOIP Training", + "url": "https://www.lznoip.club/training/66358a5906b752f133d95f9f", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "9857521a0b6a1ec9b2a80313f994346b68c942524b45ffdbf4531c7811acdad3", + "reference_sha256": null + }, + { + "id": "risk-diverse-158", + "name": "Scribus", + "url": "https://www.scribus.net/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "f80964a03dfcee684872a0480903f799847dfdf19c0f3edcf8aaae37096e45ab", + "reference_sha256": null + }, + { + "id": "risk-diverse-159", + "name": "HAL Open Science", + "url": "https://hal.science/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "8aa2129bad9a21c2eed3941ad306b37c52b9bbf20c222823803279dcb928520d", + "reference_sha256": null + }, + { + "id": "risk-diverse-160", + "name": "Observation.org", + "url": "https://observation.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3b8e333095bdd62ed23078e671b039448e5eb47e98289f85a4f8cee64287ee07", + "reference_sha256": null + }, + { + "id": "risk-diverse-170", + "name": "Linkpuff Git", + "url": "https://git.linkpuff.me/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "4c332d88622387867e20c127f09ebc129ddbbf2f18be1948282074ca19640173", + "reference_sha256": null + }, + { + "id": "risk-diverse-177", + "name": "Egosoft Forum", + "url": "https://forum.egosoft.com/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "6544835745d971406a3c1bf03f373d87f3391f092fa04e3f241dd3b1c94c29f9", + "reference_sha256": null + }, + { + "id": "risk-diverse-181", + "name": "Wikimedia XTools Article Info", + "url": "https://xtools.wmcloud.org/articleinfo/en.wikipedia.org/Artificial_intelligence", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "57ccb3ad6f18a060b7709e3d7532cd8dd7a8f31fd7ce182a3b457d2501b6a31e", + "reference_sha256": null + }, + { + "id": "risk-diverse-182", + "name": "SP Codex Wiki", + "url": "https://spcodex.wiki/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "39ae5b459718f20831fe1ac68f37d4c0a289080d655c746a37bc03fb2d9dfb2e", + "reference_sha256": null + }, + { + "id": "risk-diverse-185", + "name": "Lumiera API Documentation", + "url": "https://lumiera.org/doxy/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "6e8a7188b25c90f84b21d90a96f4704f5ef0f013417d81f2cb97a8d4fe5135e7", + "reference_sha256": null + }, + { + "id": "risk-diverse-192", + "name": "ibiblio", + "url": "https://ibiblio.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "9606cad0f429c0662335f8ab1eae4d5a2963f24f13ae4827e5125406f22785d6", + "reference_sha256": null + }, + { + "id": "risk-diverse-193", + "name": "Fedora Documentation", + "url": "https://docs.fedoraproject.org/en-US/docs/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "2db31431ffe3dc39ff463c429719777f843cc709829f0f2d07c1db2af4c00983", + "reference_sha256": null + }, + { + "id": "risk-diverse-219", + "name": "QEMU Wiki", + "url": "https://wiki.qemu.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "46e5b068b86a789be248807115c6cf8effd714a1184a3eff6b62e1a5c70d922c", + "reference_sha256": null + }, + { + "id": "risk-diverse-220", + "name": "MusicBrainz Wiki", + "url": "https://wiki.musicbrainz.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "a354eb1a57b4b8524f3995266425641277f13d694dbc82522214b7073525804f", + "reference_sha256": null + }, + { + "id": "risk-diverse-221", + "name": "SABRE Roads", + "url": "https://www.sabre-roads.org.uk/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d3a79c0deb9b67f4f56658812c83749387d264561a575e1a1e07183627bf3e6a", + "reference_sha256": null + }, + { + "id": "risk-diverse-222", + "name": "Wikibase World", + "url": "https://wikibase.world/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "812e2762770ad7592308b4d287ad40914a9ff4998d50e250325c386eba3dcfa2", + "reference_sha256": null + }, + { + "id": "risk-diverse-223", + "name": "Noisebridge", + "url": "https://www.noisebridge.net/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "bee0dabe5f58b72fdb6d98dfaa79d02ee009a4ac754630cd213a8632b12e3dfd", + "reference_sha256": null + }, + { + "id": "risk-diverse-225", + "name": "Anton Paar Wiki", + "url": "https://wiki.anton-paar.com/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "e5ff8417c616f09baf1a777b97836fceade0f645346257a70f5a1354dec356b4", + "reference_sha256": null + }, + { + "id": "risk-diverse-226", + "name": "Team Fortress Wiki", + "url": "https://wiki.teamfortress.com/wiki/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "0345be81c157a69d3146ae3f64f4a1061e81d1b386bb8f13e47f9d342ce4e572", + "reference_sha256": null + }, + { + "id": "risk-diverse-011", + "name": "Threads:NASA 公开帖子", + "url": "https://www.threads.com/@nasa/post/DdEwHNClPdc?hl=zh-cn", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d0aead382084f4ad0a3242f01b0b5c2514a68e09bd90bb00b698dee103607637", + "reference_sha256": null + }, + { + "id": "risk-diverse-012", + "name": "Goodreads:84 Charing Cross Road", + "url": "https://www.goodreads.com/book/show/953716.84_Charing_Cross_Road", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "7e107faed4275e67cc4de09371c2be23de120cc0f369f162a5007bb5b69f2355", + "reference_sha256": null + }, + { + "id": "risk-diverse-241", + "name": "POWER-Z 测试仪购买方式", + "url": "https://www.power-z.cn/blogs/新闻/chargerlab-power-z-测试仪购买方式", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d8a0e2a125965a94112c1936b9225bb08c7428c0a8aba14694bcc3c8b66d670b", + "reference_sha256": null + }, + { + "id": "risk-diverse-242", + "name": "TIB English homepage", + "url": "https://www.tib.eu/en/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "51286793c385a614bb706fa23c6ee7303c31a9ca8ab254dd2a66cf1e65b8324a", + "reference_sha256": null + }, + { + "id": "risk-diverse-244", + "name": "Sciencemadness Wiki", + "url": "https://www.sciencemadness.org/smwiki/index.php/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "b65f4cc85529cfd1f190195dc3f693a7fabcc9111701e6b30b0278ced9d3b726", + "reference_sha256": null + }, + { + "id": "risk-diverse-245", + "name": "About xeno-canto", + "url": "https://xeno-canto.org/about/xeno-canto", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3041a5cfa0bee8f1d90a4828441c2f4debf39c60c92e3582a9be5aee6a2b96bb", + "reference_sha256": null + }, + { + "id": "risk-diverse-248", + "name": "The Global European", + "url": "https://theglobaleuropean.com/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3aeb4f3285949ea8f07b9f9a2cc03d40a0075b1adaee20e9aad5f8a316ce2d95", + "reference_sha256": null + }, + { + "id": "risk-diverse-262", + "name": "Hacklab public wiki", + "url": "https://knowledge.hacklab.to/wiki/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3b1628ec8e7b8c6f76054371cf226a9ca8447248f0393ec6d2f0af79a585b710", + "reference_sha256": null + }, + { + "id": "risk-diverse-263", + "name": "freedesktop wiki", + "url": "https://www.freedesktop.org/wiki/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "956e3912d35a1f3ff9ef81c3c5f1149c8ea7753db9085f321679f4e8ceaddbe2", + "reference_sha256": null + }, + { + "id": "risk-diverse-267", + "name": "openSUSE Wiki", + "url": "https://en.opensuse.org/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d4cca1263e33a9d2d16c1b65472777c70ba20f9f9cbd9b1c15b8795ccdab51e5", + "reference_sha256": null + }, + { + "id": "risk-diverse-268", + "name": "Alpine Linux Wiki", + "url": "https://wiki.alpinelinux.org/wiki/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "866e2946b9c8f5dd5926bcacc92e10cff875fc216876a2d0709857db39374f77", + "reference_sha256": null + }, + { + "id": "risk-new-006", + "name": "Dribbble:TAE Web Design for Travel & Tourism", + "url": "https://dribbble.com/shots/16508584-TAE-Web-Design-for-Travel-Tourism", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "cbfd2e5f5527d8e48997e9700a40b11b7569f4a77d70ffffd22402871d7309c3", + "reference_sha256": null + }, + { + "id": "risk-diverse-199", + "name": "About ScummVM", + "url": "https://wiki.scummvm.org/index.php?title=About", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "8b0aeb87110dd66c4df883be6917e7c6a0b3290dcc7a84d88d0663dc168f94bd", + "reference_sha256": null + }, + { + "id": "risk-diverse-200", + "name": "OpenWrt quick start", + "url": "https://openwrt.org/docs/guide-quick-start/start", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "80fcdc9d37936ff4663edc5626337431767c55eae1e516aac7847adcfa87a3d2", + "reference_sha256": null + }, + { + "id": "risk-diverse-201", + "name": "Arch Linux installation guide", + "url": "https://wiki.archlinux.org/title/Installation_guide", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "1086c03375e7d627c4b7cfbbcc1e33ea1df47542f98caddfc353402320e5d8ee", + "reference_sha256": null + }, + { + "id": "risk-diverse-202", + "name": "Haiku Trac guide", + "url": "https://dev.haiku-os.org/wiki/TracGuide", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3c4ed8565dae61a808d92cc120b91ac7520080506f347c1bc6a1e28ba8eb6bb9", + "reference_sha256": null + }, + { + "id": "risk-diverse-204", + "name": "ReactOS overview article", + "url": "https://reactos.org/wiki/ReactOS", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "c7ddcda583d479e4e496b80b3f7578df7cee834d94f6292c2ca8e6c11f3ed468", + "reference_sha256": null + }, + { + "id": "risk-diverse-205", + "name": "Superlove about", + "url": "https://superlove.sayitditto.net/about", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "7f6d0d8ffe32b61e4c9a69f5d2f447777cda9c2db1662ae6f14f3ec286542230", + "reference_sha256": null + }, + { + "id": "risk-diverse-208", + "name": "About Pluralpedia", + "url": "https://pluralpedia.org/w/Project:About", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3d376ae1c12f8ca7b3749aeebde59611b23b0b3cf6e9d88117f16554a30f6c05", + "reference_sha256": null + }, + { + "id": "risk-diverse-209", + "name": "RPM Fusion FAQ", + "url": "https://rpmfusion.org/FAQ", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "064a87a20cae59ad2d80433b40febe2b52b77b46f0fd6d35c3e43e63436ed686", + "reference_sha256": null + }, + { + "id": "risk-diverse-210", + "name": "Installing Free Pascal", + "url": "https://wiki.freepascal.org/Installing_the_Free_Pascal_Compiler", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "134dc199b0e024090cf15d8f45207cbcab8e789d88f94c39e2b504d0e680221a", + "reference_sha256": null + }, + { + "id": "risk-diverse-211", + "name": "Koha getting involved", + "url": "https://wiki.koha-community.org/wiki/Getting_involved", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "9983060a609fc693a8879b2f7841f7115b7ebfdb838923d709c6c8f484c99ef1", + "reference_sha256": null + }, + { + "id": "risk-diverse-212", + "name": "D language getting started", + "url": "https://wiki.dlang.org/Getting_Started", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "939ec0e69b155ab421c71a8d0db769abb23e9b6cc28f7b0eb414b187c66399c5", + "reference_sha256": null + }, + { + "id": "risk-diverse-213", + "name": "About Scribus", + "url": "https://www.scribus.net/about/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d2a933d9d5376df6b1002bdaca9e4b6f0548b585552dfd4df2467bffd9998663", + "reference_sha256": null + }, + { + "id": "risk-diverse-214", + "name": "About Observation", + "url": "https://observation.org/pages/about/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d3880e7d05dc5127914ff2af4db1c0475648abd842320393e6c777c6378c2514", + "reference_sha256": null + }, + { + "id": "risk-diverse-215", + "name": "About SPCodex", + "url": "https://spcodex.wiki/SPCodex:About", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d1a508d18a9e2e1516bf0cb3ff78f12a7e368df27b5da11030c2c18a70744ef8", + "reference_sha256": null + }, + { + "id": "risk-diverse-216", + "name": "Fedora beginners guide", + "url": "https://docs.fedoraproject.org/en-US/beginners-guide/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "69f07f2aeb03d28e5e4215a71be8f9e24ad5ce90522aab8cf5a7ad85541c76cd", + "reference_sha256": null + }, + { + "id": "risk-diverse-228", + "name": "Linux kernel README", + "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/about/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "c32a6ea8e233c75f035dc6ca519b4355c1913baec38d29d7477724d5bffdc284", + "reference_sha256": null + }, + { + "id": "risk-diverse-230", + "name": "Scioly getting started", + "url": "https://scioly.org/wiki/Help:Getting_started", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "f05d66afc127e26d9c17a6c6513e0eef5195896bbc39ca24dabbaa754494fd7e", + "reference_sha256": null + }, + { + "id": "risk-diverse-231", + "name": "FreeCAD forum FAQ", + "url": "https://forum.freecad.org/app.php/help/faq", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "361d8e0a7742190983185246ffd7e618df7ff31d9cc4f7a3059bf43bce14f64d", + "reference_sha256": null + }, + { + "id": "risk-diverse-234", + "name": "What are TYPO3 extensions", + "url": "https://extensions.typo3.org/about-extension-repository/what-are-extensions", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "07deed8cf2ea29f3c6c8d90e0167c9ff414e0eecf4a7e89b6d0e0803eee06b0f", + "reference_sha256": null + }, + { + "id": "risk-diverse-236", + "name": "SquirrelJME user guide", + "url": "https://multiphasicapps.net/doc/ckout/assets/user-guide/readme.mkd", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "6f0f4ba4208f9ac4691b3d19779ec1cddcd2abfcead3bf7cb088f1d10121568e", + "reference_sha256": null + }, + { + "id": "risk-diverse-237", + "name": "Dolphin FAQ", + "url": "https://dolphin-emu.org/docs/faq/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "a8cef268823caf2a1432adce663ac39878e6bb6d36cd45d2bcd4c7abffe4d1f3", + "reference_sha256": null + }, + { + "id": "risk-diverse-238", + "name": "Egosoft forum FAQ", + "url": "https://forum.egosoft.com/app.php/help/faq", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "82bdfa31fc84ecafd797dc3c809f80a4a5a20f0d0779915eccc16638003b9914", + "reference_sha256": null + }, + { + "id": "risk-diverse-239", + "name": "About ibiblio", + "url": "https://www.ibiblio.org/about/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "8c09ff83e5736ebd7bbf4854a6077830d94d74c2d4787e8ee754d73c061a20a9", + "reference_sha256": null + }, + { + "id": "risk-diverse-250", + "name": "Contributing to QEMU", + "url": "https://wiki.qemu.org/Contribute", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "f2c5c6629dfac5ba5f1b39f5e5f198038978113b02d7fa6429057d7b99d15876", + "reference_sha256": null + } + ] +} diff --git a/evaluation/risk-control/current/source-inputs.json b/evaluation/risk-control/current/source-inputs.json new file mode 100644 index 000000000..45619d88a --- /dev/null +++ b/evaluation/risk-control/current/source-inputs.json @@ -0,0 +1,13 @@ +{ + "schema_version": 2, + "kind": "frozen_reference_inputs", + "id": "risk-control-165b888133e8-v6", + "dataset": "dataset.json", + "dataset_sha256": "165b888133e859304b035d74183dd40bf7ab609cac78bb6ae6e75b645c1de426", + "catalog": "dataset-catalog.json", + "catalog_sha256": "52ec968b2ada26e5e061ccb25cbe4a73747539e556fa88f31bece6d7fb79b3cb", + "files": { + "dataset.json": "165b888133e859304b035d74183dd40bf7ab609cac78bb6ae6e75b645c1de426" + }, + "trees": {} +} diff --git a/evaluation/risk-control/dataset-catalog.json b/evaluation/risk-control/dataset-catalog.json index f01df28cb..89bbf5fe0 100644 --- a/evaluation/risk-control/dataset-catalog.json +++ b/evaluation/risk-control/dataset-catalog.json @@ -2,9 +2,9 @@ "schema_version": 1, "kind": "frozen_dataset_catalog", "id": "risk-control-audited-100", - "name": "曾观察到访问限制的100个网址", + "name": "风控网站正文评测", "dataset": "dataset.json", - "dataset_sha256": "d0d038bae0114be107116edd94e501996ba6cda852ac29244c089e15a3e37437", + "dataset_sha256": "d3f18edd2b29f8ef19eb496841f91c433d7f9adfe33c591874d963db55760d7c", "case_count": 100, "cases": [ { @@ -174,7 +174,7 @@ "category": "risk_control", "track": "risk_control", "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", - "definition_sha256": "e2f62cd594e62a8532c8619feddc73b7d1223ef57c223d58c9fd574beb6a2858", + "definition_sha256": "30a2268fda5aa866597e5a75453e0e45fcbda67cca4961ab2f4278ed48e1707a", "reference_sha256": null }, { @@ -374,7 +374,7 @@ "category": "risk_control", "track": "risk_control", "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", - "definition_sha256": "af3cf6a1bb67192436aa23f494bf0ef88f15d5ed260900d45e67b483cd28c455", + "definition_sha256": "670fb01613742e048a7a8e4a1fdda5cb00d6b05108c35f38b8a65354679c34eb", "reference_sha256": null }, { @@ -634,7 +634,7 @@ "category": "risk_control", "track": "risk_control", "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", - "definition_sha256": "a3a504844c884eec0738a4f09a4d96a4b20fcbafbd3af67b92705301a3c73ff5", + "definition_sha256": "e5ff8417c616f09baf1a777b97836fceade0f645346257a70f5a1354dec356b4", "reference_sha256": null }, { @@ -674,7 +674,7 @@ "category": "risk_control", "track": "risk_control", "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", - "definition_sha256": "1114f12794678c618328774b456b87df176abd8738d00f94fd8adcc3ce40f6cb", + "definition_sha256": "d8a0e2a125965a94112c1936b9225bb08c7428c0a8aba14694bcc3c8b66d670b", "reference_sha256": null }, { @@ -754,7 +754,7 @@ "category": "risk_control", "track": "risk_control", "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", - "definition_sha256": "24c8cfda691df49fe7a2b5a7aff11535fc7ad0a75fce13aa0c5bf5df74bb75fa", + "definition_sha256": "866e2946b9c8f5dd5926bcacc92e10cff875fc216876a2d0709857db39374f77", "reference_sha256": null }, { @@ -794,7 +794,7 @@ "category": "risk_control", "track": "risk_control", "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", - "definition_sha256": "4f6bcbe618aa5269babc4854ef318c2ff4530490995280ff7b6b45f245d6bdf9", + "definition_sha256": "1086c03375e7d627c4b7cfbbcc1e33ea1df47542f98caddfc353402320e5d8ee", "reference_sha256": null }, { @@ -844,7 +844,7 @@ "category": "risk_control", "track": "risk_control", "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", - "definition_sha256": "a57af07b4bf14172356d25b6a870b09f62c48cbef17702b16e9145e64091859b", + "definition_sha256": "064a87a20cae59ad2d80433b40febe2b52b77b46f0fd6d35c3e43e63436ed686", "reference_sha256": null }, { @@ -974,7 +974,7 @@ "category": "risk_control", "track": "risk_control", "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", - "definition_sha256": "d8df38c8d1867e1148a5dfadd4a3e4a4a0f46c7f0594a02ee227a41a7cd2dedf", + "definition_sha256": "a8cef268823caf2a1432adce663ac39878e6bb6d36cd45d2bcd4c7abffe4d1f3", "reference_sha256": null }, { diff --git a/evaluation/risk-control/first-30-rule-review.md b/evaluation/risk-control/first-30-rule-review.md index cb11a6614..fe68f03c9 100644 --- a/evaluation/risk-control/first-30-rule-review.md +++ b/evaluation/risk-control/first-30-rule-review.md @@ -221,10 +221,10 @@ [打开目标网页](https://wiki.archlinux.org/title/Main_page) - **网址、语言与目标:** 输入和观察最终网址相同;英文;ArchWiki 主页面介绍和主要导航内容。 -- **稳定文字:** 顶部站点导航 `Home`、`Packages`、`Forums`、`Wiki`、`GitLab`、`Security`、`AUR`、`Download`;页面与侧栏 `ArchWiki`、`Main page`、`Table of contents`、`Getting involved`、`Help`、`Contributing`、`Recent changes`、`Search`、`Appearance`;页面操作 `Read`、`View source`、`View history`;主体分别要求 `Welcome to the`、`your source for Arch Linux documentation on the web.`、`The distribution`、`Our community`、`Wiki interaction`。欢迎句中的 ArchWiki 是单独显示的链接文字,已由前面的 ArchWiki 组检查,不再把三段拼成一个稳定文字。所有文字都在独立保存的首次页面文本中,分布于顶部、侧栏、页面操作和完整主体,全部必需。 +- **稳定文字:** 顶部站点导航 `Home`、`Packages`、`Forums`、`Wiki`、`GitLab`、`Security`、`AUR`、`Download`;页面与侧栏 `ArchWiki`、`Main page`、`Table of contents`、`Getting involved`、`Help`、`Contributing`、`Search`、`Appearance`;页面操作 `Read`、`View source`、`View history`;主体分别要求 `Welcome to the`、`your source for Arch Linux documentation on the web.`、`The distribution`、`Our community`、`Wiki interaction`。欢迎句中的 ArchWiki 是单独显示的链接文字,已由前面的 ArchWiki 组检查,不再把三段拼成一个稳定文字。所有文字都在独立保存的首次页面文本中,分布于顶部、侧栏、页面操作和完整主体,全部必需。 - **name-link:** 不采用数量列表。 - **正文参考:** 三区为:欢迎语和 `Visit the Table of contents for a listing of article categories.`;The distribution 下 Overview、FAQ、Arch compared、Installation guide、General recommendations、List of applications 及说明;Our community 与 Wiki interaction 下各固定入口及说明。 -- **排除:** 语言数量、旧版本号、最后编辑时间、当前登录状态和外观选项的当前选中值。`Color (beta)` 及 Small/Standard/Large 等皮肤选项不冻结,因为名称和可用项可能随 MediaWiki 皮肤版本变化。3 区均需 60%,至少 2 区需 80%;只返回顶部导航、侧栏或主体其中一部分均不能通过。 +- **排除:** 语言数量、旧版本号、最后编辑时间、当前登录状态和外观选项的当前选中值。`Color (beta)` 及 Small/Standard/Large 等皮肤选项不冻结,因为名称和可用项可能随 MediaWiki 皮肤版本变化。3 区均需 60%,至少 2 区需 80%;只返回顶部导航、侧栏或主体其中一部分均不能通过;响应式导航中的 `Recent changes` 不参与通过判定。 ## 18. risk-diverse-039|FreeBSD SVNweb diff --git a/evaluation/risk-control/history/v5/dataset-catalog.json b/evaluation/risk-control/history/v5/dataset-catalog.json new file mode 100644 index 000000000..1efd6e419 --- /dev/null +++ b/evaluation/risk-control/history/v5/dataset-catalog.json @@ -0,0 +1,1011 @@ +{ + "schema_version": 1, + "kind": "frozen_dataset_catalog", + "id": "risk-control-audited-100", + "name": "风控网站正文评测", + "dataset": "dataset.json", + "dataset_sha256": "43717c2f8d5e9d3568f84670ba4975fbd22e05468fd2353c3a13f6c48983cdae", + "case_count": 100, + "cases": [ + { + "id": "risk-diverse-002", + "name": "知乎:人工智能应用场景回答", + "url": "https://www.zhihu.com/question/282715644/answer/3629347314", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "4e591a4595a0554b7f86a0eba1c601531f7fed481500da14494036310e974f70", + "reference_sha256": null + }, + { + "id": "risk-001", + "name": "Flipkart:Acer Aspire Lite 商品详情", + "url": "https://www.flipkart.com/acer-aspire-lite-amd-ryzen-3-quad-core-5400u-8-gb-256-gb-ssd-windows-11-home-al15-41-thin-light-laptop/p/itm1bc0bcb4598e7", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "119d84a124f4aaa94f60f41383c27b0b74b7a8185a003567ddbfb05b39b89d40", + "reference_sha256": null + }, + { + "id": "risk-006", + "name": "微博热搜", + "url": "https://s.weibo.com/top/summary", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "65eff51f32d754895c70737b8ce336e47606a575b7826b738cbcc4e4ec763e10", + "reference_sha256": null + }, + { + "id": "risk-011", + "name": "今日头条:关于人工智能,我们了解多少", + "url": "https://www.toutiao.com/article/7595028554912399898/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "f899d00a5c7a6303492637698931df6c99167bc66ed6066f15ae3e14959fcd0e", + "reference_sha256": null + }, + { + "id": "risk-061", + "name": "Dribbble:web-design 搜索结果", + "url": "https://dribbble.com/search/web-design", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "fd97c97d7512ccec16bb9c2ac9ed18fd3c86b87ee3fb7ab306feb33cf655d67e", + "reference_sha256": null + }, + { + "id": "risk-101", + "name": "哔哩哔哩:综合热门", + "url": "https://www.bilibili.com/v/popular/all/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "82de10eea401cff30b9c7302a18c112063a1e0f2fc44f3121844594b1429879a", + "reference_sha256": null + }, + { + "id": "risk-102", + "name": "腾讯视频:电影频道", + "url": "https://v.qq.com/channel/movie", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "a1a5341179ecce8397ee75b09b292ea72f5e184a936419b2e9cead7528d7e142", + "reference_sha256": null + }, + { + "id": "risk-diverse-271", + "name": "About Sanctum repository service", + "url": "https://dev.sanctum.geek.nz/cgit/?p=about", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "8f7e99e10858cf25406a2fa9bbb199b50befe2dd120e0915574a59e08ab31438", + "reference_sha256": null + }, + { + "id": "risk-diverse-027", + "name": "Linux Kernel Git", + "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "29ff371f48f33a097d310f071ef25194d302532ca923861f7625b8d228bf0060", + "reference_sha256": null + }, + { + "id": "risk-new-102", + "name": "DebianInstall", + "url": "https://wiki.debian.org/DebianInstall", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "9518f4ff3218de56c9da64cc9948c8acdcebd874aa02ac7fcd5c0b5f4f42af0f", + "reference_sha256": null + }, + { + "id": "risk-diverse-030", + "name": "Wine Bugzilla", + "url": "https://bugs.winehq.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "520ec13a5e8f641a324ab421e3118d3df3159dee4aa61bebca0fbbe5b85b3391", + "reference_sha256": null + }, + { + "id": "risk-diverse-031", + "name": "Science Olympiad Student Center Wiki", + "url": "https://scioly.org/wiki/index.php/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "f243ee8a571dc97e840eb5b5e5cb00a304bd2727987b4e659777a07224cbbe15", + "reference_sha256": null + }, + { + "id": "risk-diverse-032", + "name": "Enlightenment Git", + "url": "https://git.enlightenment.org/enlightenment/enlightenment", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "75827072b43b9bb530435287df09ce05eb9aa63035d08730922df86093f7b5e8", + "reference_sha256": null + }, + { + "id": "risk-diverse-033", + "name": "FreeCAD Forum", + "url": "https://forum.freecad.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "338297148e1b45116d6ef782ed93ae02bec71f5de2846ebbcdbc3c0bcdacaee3", + "reference_sha256": null + }, + { + "id": "risk-diverse-034", + "name": "ScummVM Wiki", + "url": "https://wiki.scummvm.org/index.php/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "09277073909b6f4e6889f83cd0773450bdbe7fb385a934140451fc34b6954be2", + "reference_sha256": null + }, + { + "id": "risk-diverse-036", + "name": "OpenWrt Wiki", + "url": "https://openwrt.org/start", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "efc2a8c10e8a109cbfe8df740392b423fe314037caac42fe2d1d7c1ce2fc626d", + "reference_sha256": null + }, + { + "id": "risk-diverse-038", + "name": "ArchWiki", + "url": "https://wiki.archlinux.org/title/Main_page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "30a2268fda5aa866597e5a75453e0e45fcbda67cca4961ab2f4278ed48e1707a", + "reference_sha256": null + }, + { + "id": "risk-diverse-039", + "name": "FreeBSD SVNweb", + "url": "https://svnweb.freebsd.org/base/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "4599e13e815f3c5595df7d573341e0fdd08780daf2761db24d489d3a1c5f0e01", + "reference_sha256": null + }, + { + "id": "risk-diverse-040", + "name": "Purism GitLab:Phosh", + "url": "https://source.puri.sm/Librem5/phosh", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "b6ca53feac1c9ff38ff47ca7ff872cd1c5b44a199edcf138689b3157c34dda37", + "reference_sha256": null + }, + { + "id": "risk-diverse-041", + "name": "Haiku Trac", + "url": "https://dev.haiku-os.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "cbff3640b4e71d5c49fcd1af6e767f93760f8133de56a7ab7f3b5f29d58c84c7", + "reference_sha256": null + }, + { + "id": "risk-diverse-042", + "name": "Devuan Git:installer-iso", + "url": "https://git.devuan.org/devuan/installer-iso", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "6aba3f75d126638c5a852af5b0be872afbb5bf44f732d5ce4dd26a84f94ae5d6", + "reference_sha256": null + }, + { + "id": "risk-diverse-045", + "name": "CFA Archive", + "url": "https://www.cfaarchive.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "75c4ba4f48dc99fc2661f8ae3fc9fe2803b2c633ce6380d83066b9ba759b7858", + "reference_sha256": null + }, + { + "id": "risk-diverse-047", + "name": "Proxmox Bugzilla", + "url": "https://bugzilla.proxmox.com/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "024b598aa2297dd6da4bb324f2440f76939cd003e1d8100507998cbc7c610b8d", + "reference_sha256": null + }, + { + "id": "risk-diverse-048", + "name": "ReactOS Wiki", + "url": "https://reactos.org/wiki/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "6e297e1ffb8738d6f1057f189f88df128505dea03df8240228a40c0e160371f2", + "reference_sha256": null + }, + { + "id": "risk-diverse-049", + "name": "Arch Linux 32 Packages", + "url": "https://www.archlinux32.org/packages/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "36a18f11e670411c7969a26b4095de6a5b1bfe2cea37c111dbce440764e11991", + "reference_sha256": null + }, + { + "id": "risk-diverse-050", + "name": "Slackware Git", + "url": "https://git.slackware.nl/current/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "f77638953c120ea4628d342aa73693d91e9ed7c98a40a27991dd9e181a719223", + "reference_sha256": null + }, + { + "id": "risk-diverse-051", + "name": "Duke Digital Repository", + "url": "https://repository.duke.edu/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "7c67f961f5afbd2e9f343bc5eca6f5a783ebaa63354b5832f89376d61857aefd", + "reference_sha256": null + }, + { + "id": "risk-diverse-055", + "name": "hebis Discovery System", + "url": "https://tufind.hds.hebis.de/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "670c2f60030ecc27bf314eade1ac6116c2b2697803156c9483ccdfefff87444b", + "reference_sha256": null + }, + { + "id": "risk-diverse-057", + "name": "Xe Iaso", + "url": "https://xeiaso.net/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "db60c0ebf6aad7b76632735d6acf208e7f3d8043399efb5488a1df686460d756", + "reference_sha256": null + }, + { + "id": "risk-diverse-059", + "name": "Superlove", + "url": "https://superlove.sayitditto.net/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "5a5131232354f0e34134945124cccdf809ba7d5e23ae710683d35063d53bc4b8", + "reference_sha256": null + }, + { + "id": "risk-diverse-074", + "name": "Ticketmaster concerts", + "url": "https://www.ticketmaster.com/discover/concerts", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "93a4aec34c617dda96b5f4f3e06a77a6d8dd3abb53fe2b0c2953d5f668f744ee", + "reference_sha256": null + }, + { + "id": "risk-diverse-101", + "name": "Azur Lane Wiki", + "url": "https://azurlane.koumakan.jp/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "fc3104cfd5f22fd321503e226b57f54088c6c059feafb509bb703c79d438a52b", + "reference_sha256": null + }, + { + "id": "risk-diverse-104", + "name": "Sanctum Development", + "url": "https://dev.sanctum.geek.nz/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "33532667a459ec873b69c0b104d8d6b3e51c6f0a41702ad4d1803901875d4a29", + "reference_sha256": null + }, + { + "id": "risk-diverse-106", + "name": "TYPO3 Extension Repository", + "url": "https://extensions.typo3.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "a59ca8301806bdb7172cf6ea4c21f222448e231c24855932c91a2a2b32e46524", + "reference_sha256": null + }, + { + "id": "risk-diverse-113", + "name": "MozillaZine", + "url": "https://mozillazine.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "0b37f86d1c5102eb5a3d656e5454f9016a639c9f9d28810c24650120d6903eda", + "reference_sha256": null + }, + { + "id": "risk-diverse-114", + "name": "Pluralpedia", + "url": "https://pluralpedia.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "c8d96f1796c4871b36eb5be85de6e81c6beb5ea7727263d6a31f6b37a6730e19", + "reference_sha256": null + }, + { + "id": "risk-diverse-117", + "name": "RPM Fusion", + "url": "https://rpmfusion.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "670fb01613742e048a7a8e4a1fdda5cb00d6b05108c35f38b8a65354679c34eb", + "reference_sha256": null + }, + { + "id": "risk-diverse-118", + "name": "SquirrelJME", + "url": "https://squirreljme.cc/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3ed4808cbe6fd9abb5321ec9fb889a6f9ea96fb986203d9911435cca4ff9deae", + "reference_sha256": null + }, + { + "id": "risk-diverse-121", + "name": "Dolphin Emulator Forums", + "url": "https://forums.dolphin-emu.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "dc00ef5fe7fc8871581e1618ddcf7004b80c4f1fa209632019e7b05139632728", + "reference_sha256": null + }, + { + "id": "risk-diverse-122", + "name": "Free Pascal Wiki", + "url": "https://wiki.freepascal.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "9c7916dafe96857c32d60fabe4769fc7df4eade65fded00f3e093e340af43163", + "reference_sha256": null + }, + { + "id": "risk-diverse-123", + "name": "Koha Community Wiki", + "url": "https://wiki.koha-community.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "889884b2d86f23bea6bd99b7a856fd2f43edb567fdd76ae0b591f728252296a6", + "reference_sha256": null + }, + { + "id": "risk-diverse-129", + "name": "HydroOJ FAQs", + "url": "https://ws.hydrooj.com/d/faqs/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "e00b6b37cc32127897a7a48740a2a50516022c268ea7a7768b4ca184cabdf16e", + "reference_sha256": null + }, + { + "id": "risk-diverse-134", + "name": "Zillow Homes for Sale", + "url": "https://www.zillow.com/seattle-wa/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "478c7dc7335cf4316c7b0dfae4753cfbe9b1b72bc7953059613a4a0a43f64820", + "reference_sha256": null + }, + { + "id": "risk-diverse-135", + "name": "Go.Compare", + "url": "https://www.gocompare.com/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "116b80a20713f4629f41da494fcec44eb945b7197719150bfa5f63b1b5f2afd3", + "reference_sha256": null + }, + { + "id": "risk-diverse-138", + "name": "D Wiki", + "url": "https://wiki.dlang.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "96840e0deebc9ceea29abf68ffc68f5de228864e500767f0da65d75a693597c6", + "reference_sha256": null + }, + { + "id": "risk-diverse-139", + "name": "DSource", + "url": "https://dsource.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "884c24446fca078cf14d202d6f63b2c9187d5e1053b75e2a6df02ae3f4e5882d", + "reference_sha256": null + }, + { + "id": "risk-diverse-153", + "name": "LZNOIP Training", + "url": "https://www.lznoip.club/training/66358a5906b752f133d95f9f", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "9857521a0b6a1ec9b2a80313f994346b68c942524b45ffdbf4531c7811acdad3", + "reference_sha256": null + }, + { + "id": "risk-diverse-158", + "name": "Scribus", + "url": "https://www.scribus.net/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "f80964a03dfcee684872a0480903f799847dfdf19c0f3edcf8aaae37096e45ab", + "reference_sha256": null + }, + { + "id": "risk-diverse-159", + "name": "HAL Open Science", + "url": "https://hal.science/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "8aa2129bad9a21c2eed3941ad306b37c52b9bbf20c222823803279dcb928520d", + "reference_sha256": null + }, + { + "id": "risk-diverse-160", + "name": "Observation.org", + "url": "https://observation.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3b8e333095bdd62ed23078e671b039448e5eb47e98289f85a4f8cee64287ee07", + "reference_sha256": null + }, + { + "id": "risk-diverse-170", + "name": "Linkpuff Git", + "url": "https://git.linkpuff.me/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "4c332d88622387867e20c127f09ebc129ddbbf2f18be1948282074ca19640173", + "reference_sha256": null + }, + { + "id": "risk-diverse-177", + "name": "Egosoft Forum", + "url": "https://forum.egosoft.com/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "6544835745d971406a3c1bf03f373d87f3391f092fa04e3f241dd3b1c94c29f9", + "reference_sha256": null + }, + { + "id": "risk-diverse-181", + "name": "Wikimedia XTools Article Info", + "url": "https://xtools.wmcloud.org/articleinfo/en.wikipedia.org/Artificial_intelligence", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "57ccb3ad6f18a060b7709e3d7532cd8dd7a8f31fd7ce182a3b457d2501b6a31e", + "reference_sha256": null + }, + { + "id": "risk-diverse-182", + "name": "SP Codex Wiki", + "url": "https://spcodex.wiki/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "39ae5b459718f20831fe1ac68f37d4c0a289080d655c746a37bc03fb2d9dfb2e", + "reference_sha256": null + }, + { + "id": "risk-diverse-185", + "name": "Lumiera API Documentation", + "url": "https://lumiera.org/doxy/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "6e8a7188b25c90f84b21d90a96f4704f5ef0f013417d81f2cb97a8d4fe5135e7", + "reference_sha256": null + }, + { + "id": "risk-diverse-192", + "name": "ibiblio", + "url": "https://ibiblio.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "e5aa019d864c6248715f6183e159bed1751aeaee849d29179723b743eb27111b", + "reference_sha256": null + }, + { + "id": "risk-diverse-193", + "name": "Fedora Documentation", + "url": "https://docs.fedoraproject.org/en-US/docs/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "2db31431ffe3dc39ff463c429719777f843cc709829f0f2d07c1db2af4c00983", + "reference_sha256": null + }, + { + "id": "risk-diverse-219", + "name": "QEMU Wiki", + "url": "https://wiki.qemu.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "46e5b068b86a789be248807115c6cf8effd714a1184a3eff6b62e1a5c70d922c", + "reference_sha256": null + }, + { + "id": "risk-diverse-220", + "name": "MusicBrainz Wiki", + "url": "https://wiki.musicbrainz.org/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "a354eb1a57b4b8524f3995266425641277f13d694dbc82522214b7073525804f", + "reference_sha256": null + }, + { + "id": "risk-diverse-221", + "name": "SABRE Roads", + "url": "https://www.sabre-roads.org.uk/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d3a79c0deb9b67f4f56658812c83749387d264561a575e1a1e07183627bf3e6a", + "reference_sha256": null + }, + { + "id": "risk-diverse-222", + "name": "Wikibase World", + "url": "https://wikibase.world/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "812e2762770ad7592308b4d287ad40914a9ff4998d50e250325c386eba3dcfa2", + "reference_sha256": null + }, + { + "id": "risk-diverse-223", + "name": "Noisebridge", + "url": "https://www.noisebridge.net/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "bee0dabe5f58b72fdb6d98dfaa79d02ee009a4ac754630cd213a8632b12e3dfd", + "reference_sha256": null + }, + { + "id": "risk-diverse-225", + "name": "Anton Paar Wiki", + "url": "https://wiki.anton-paar.com/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "e5ff8417c616f09baf1a777b97836fceade0f645346257a70f5a1354dec356b4", + "reference_sha256": null + }, + { + "id": "risk-diverse-226", + "name": "Team Fortress Wiki", + "url": "https://wiki.teamfortress.com/wiki/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "0345be81c157a69d3146ae3f64f4a1061e81d1b386bb8f13e47f9d342ce4e572", + "reference_sha256": null + }, + { + "id": "risk-diverse-011", + "name": "Threads:NASA 公开帖子", + "url": "https://www.threads.com/@nasa/post/DdEwHNClPdc?hl=zh-cn", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "a1dc25366197b23bb314d20dcb50d3ebb82c83d6403d45a46634518e8d8707b6", + "reference_sha256": null + }, + { + "id": "risk-diverse-012", + "name": "Goodreads:84 Charing Cross Road", + "url": "https://www.goodreads.com/book/show/953716.84_Charing_Cross_Road", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "7e107faed4275e67cc4de09371c2be23de120cc0f369f162a5007bb5b69f2355", + "reference_sha256": null + }, + { + "id": "risk-diverse-241", + "name": "POWER-Z 测试仪购买方式", + "url": "https://www.power-z.cn/blogs/新闻/chargerlab-power-z-测试仪购买方式", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d8a0e2a125965a94112c1936b9225bb08c7428c0a8aba14694bcc3c8b66d670b", + "reference_sha256": null + }, + { + "id": "risk-diverse-242", + "name": "TIB English homepage", + "url": "https://www.tib.eu/en/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "51286793c385a614bb706fa23c6ee7303c31a9ca8ab254dd2a66cf1e65b8324a", + "reference_sha256": null + }, + { + "id": "risk-diverse-244", + "name": "Sciencemadness Wiki", + "url": "https://www.sciencemadness.org/smwiki/index.php/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "b65f4cc85529cfd1f190195dc3f693a7fabcc9111701e6b30b0278ced9d3b726", + "reference_sha256": null + }, + { + "id": "risk-diverse-245", + "name": "About xeno-canto", + "url": "https://xeno-canto.org/about/xeno-canto", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3041a5cfa0bee8f1d90a4828441c2f4debf39c60c92e3582a9be5aee6a2b96bb", + "reference_sha256": null + }, + { + "id": "risk-diverse-248", + "name": "The Global European", + "url": "https://theglobaleuropean.com/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3aeb4f3285949ea8f07b9f9a2cc03d40a0075b1adaee20e9aad5f8a316ce2d95", + "reference_sha256": null + }, + { + "id": "risk-diverse-262", + "name": "Hacklab public wiki", + "url": "https://knowledge.hacklab.to/wiki/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3b1628ec8e7b8c6f76054371cf226a9ca8447248f0393ec6d2f0af79a585b710", + "reference_sha256": null + }, + { + "id": "risk-diverse-263", + "name": "freedesktop wiki", + "url": "https://www.freedesktop.org/wiki/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "956e3912d35a1f3ff9ef81c3c5f1149c8ea7753db9085f321679f4e8ceaddbe2", + "reference_sha256": null + }, + { + "id": "risk-diverse-267", + "name": "openSUSE Wiki", + "url": "https://en.opensuse.org/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d4cca1263e33a9d2d16c1b65472777c70ba20f9f9cbd9b1c15b8795ccdab51e5", + "reference_sha256": null + }, + { + "id": "risk-diverse-268", + "name": "Alpine Linux Wiki", + "url": "https://wiki.alpinelinux.org/wiki/Main_Page", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "866e2946b9c8f5dd5926bcacc92e10cff875fc216876a2d0709857db39374f77", + "reference_sha256": null + }, + { + "id": "risk-new-006", + "name": "Dribbble:TAE Web Design for Travel & Tourism", + "url": "https://dribbble.com/shots/16508584-TAE-Web-Design-for-Travel-Tourism", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "cbfd2e5f5527d8e48997e9700a40b11b7569f4a77d70ffffd22402871d7309c3", + "reference_sha256": null + }, + { + "id": "risk-diverse-199", + "name": "About ScummVM", + "url": "https://wiki.scummvm.org/index.php?title=About", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "8b0aeb87110dd66c4df883be6917e7c6a0b3290dcc7a84d88d0663dc168f94bd", + "reference_sha256": null + }, + { + "id": "risk-diverse-200", + "name": "OpenWrt quick start", + "url": "https://openwrt.org/docs/guide-quick-start/start", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "80fcdc9d37936ff4663edc5626337431767c55eae1e516aac7847adcfa87a3d2", + "reference_sha256": null + }, + { + "id": "risk-diverse-201", + "name": "Arch Linux installation guide", + "url": "https://wiki.archlinux.org/title/Installation_guide", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "1086c03375e7d627c4b7cfbbcc1e33ea1df47542f98caddfc353402320e5d8ee", + "reference_sha256": null + }, + { + "id": "risk-diverse-202", + "name": "Haiku Trac guide", + "url": "https://dev.haiku-os.org/wiki/TracGuide", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3c4ed8565dae61a808d92cc120b91ac7520080506f347c1bc6a1e28ba8eb6bb9", + "reference_sha256": null + }, + { + "id": "risk-diverse-204", + "name": "ReactOS overview article", + "url": "https://reactos.org/wiki/ReactOS", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "c7ddcda583d479e4e496b80b3f7578df7cee834d94f6292c2ca8e6c11f3ed468", + "reference_sha256": null + }, + { + "id": "risk-diverse-205", + "name": "Superlove about", + "url": "https://superlove.sayitditto.net/about", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "7f6d0d8ffe32b61e4c9a69f5d2f447777cda9c2db1662ae6f14f3ec286542230", + "reference_sha256": null + }, + { + "id": "risk-diverse-208", + "name": "About Pluralpedia", + "url": "https://pluralpedia.org/w/Project:About", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "3d376ae1c12f8ca7b3749aeebde59611b23b0b3cf6e9d88117f16554a30f6c05", + "reference_sha256": null + }, + { + "id": "risk-diverse-209", + "name": "RPM Fusion FAQ", + "url": "https://rpmfusion.org/FAQ", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "064a87a20cae59ad2d80433b40febe2b52b77b46f0fd6d35c3e43e63436ed686", + "reference_sha256": null + }, + { + "id": "risk-diverse-210", + "name": "Installing Free Pascal", + "url": "https://wiki.freepascal.org/Installing_the_Free_Pascal_Compiler", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "134dc199b0e024090cf15d8f45207cbcab8e789d88f94c39e2b504d0e680221a", + "reference_sha256": null + }, + { + "id": "risk-diverse-211", + "name": "Koha getting involved", + "url": "https://wiki.koha-community.org/wiki/Getting_involved", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "9983060a609fc693a8879b2f7841f7115b7ebfdb838923d709c6c8f484c99ef1", + "reference_sha256": null + }, + { + "id": "risk-diverse-212", + "name": "D language getting started", + "url": "https://wiki.dlang.org/Getting_Started", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "939ec0e69b155ab421c71a8d0db769abb23e9b6cc28f7b0eb414b187c66399c5", + "reference_sha256": null + }, + { + "id": "risk-diverse-213", + "name": "About Scribus", + "url": "https://www.scribus.net/about/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d2a933d9d5376df6b1002bdaca9e4b6f0548b585552dfd4df2467bffd9998663", + "reference_sha256": null + }, + { + "id": "risk-diverse-214", + "name": "About Observation", + "url": "https://observation.org/pages/about/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d3880e7d05dc5127914ff2af4db1c0475648abd842320393e6c777c6378c2514", + "reference_sha256": null + }, + { + "id": "risk-diverse-215", + "name": "About SPCodex", + "url": "https://spcodex.wiki/SPCodex:About", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "d1a508d18a9e2e1516bf0cb3ff78f12a7e368df27b5da11030c2c18a70744ef8", + "reference_sha256": null + }, + { + "id": "risk-diverse-216", + "name": "Fedora beginners guide", + "url": "https://docs.fedoraproject.org/en-US/beginners-guide/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "69f07f2aeb03d28e5e4215a71be8f9e24ad5ce90522aab8cf5a7ad85541c76cd", + "reference_sha256": null + }, + { + "id": "risk-diverse-228", + "name": "Linux kernel README", + "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/about/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "c32a6ea8e233c75f035dc6ca519b4355c1913baec38d29d7477724d5bffdc284", + "reference_sha256": null + }, + { + "id": "risk-diverse-230", + "name": "Scioly getting started", + "url": "https://scioly.org/wiki/Help:Getting_started", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "f05d66afc127e26d9c17a6c6513e0eef5195896bbc39ca24dabbaa754494fd7e", + "reference_sha256": null + }, + { + "id": "risk-diverse-231", + "name": "FreeCAD forum FAQ", + "url": "https://forum.freecad.org/app.php/help/faq", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "361d8e0a7742190983185246ffd7e618df7ff31d9cc4f7a3059bf43bce14f64d", + "reference_sha256": null + }, + { + "id": "risk-diverse-234", + "name": "What are TYPO3 extensions", + "url": "https://extensions.typo3.org/about-extension-repository/what-are-extensions", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "07deed8cf2ea29f3c6c8d90e0167c9ff414e0eecf4a7e89b6d0e0803eee06b0f", + "reference_sha256": null + }, + { + "id": "risk-diverse-236", + "name": "SquirrelJME user guide", + "url": "https://multiphasicapps.net/doc/ckout/assets/user-guide/readme.mkd", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "6f0f4ba4208f9ac4691b3d19779ec1cddcd2abfcead3bf7cb088f1d10121568e", + "reference_sha256": null + }, + { + "id": "risk-diverse-237", + "name": "Dolphin FAQ", + "url": "https://dolphin-emu.org/docs/faq/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "a8cef268823caf2a1432adce663ac39878e6bb6d36cd45d2bcd4c7abffe4d1f3", + "reference_sha256": null + }, + { + "id": "risk-diverse-238", + "name": "Egosoft forum FAQ", + "url": "https://forum.egosoft.com/app.php/help/faq", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "82bdfa31fc84ecafd797dc3c809f80a4a5a20f0d0779915eccc16638003b9914", + "reference_sha256": null + }, + { + "id": "risk-diverse-239", + "name": "About ibiblio", + "url": "https://www.ibiblio.org/about/", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "8c09ff83e5736ebd7bbf4854a6077830d94d74c2d4787e8ee754d73c061a20a9", + "reference_sha256": null + }, + { + "id": "risk-diverse-250", + "name": "Contributing to QEMU", + "url": "https://wiki.qemu.org/Contribute", + "category": "risk_control", + "track": "risk_control", + "task_prompt": "提取声明网址首次自动加载出的目标正文、列表和真实链接;不点击、不输入、不滚动、不翻页、不登录,也不使用图片文字。", + "definition_sha256": "f2c5c6629dfac5ba5f1b39f5e5f198038978113b02d7fa6429057d7b99d15876", + "reference_sha256": null + } + ] +} diff --git a/evaluation/risk-control/list-contracts.json b/evaluation/risk-control/list-contracts.json index d74383e4a..27950f125 100644 --- a/evaluation/risk-control/list-contracts.json +++ b/evaluation/risk-control/list-contracts.json @@ -555,8 +555,10 @@ "risk-diverse-181": { "groups": [ { - "id": "report-fields", + "id": "general-statistics", "kind": "field_report", + "start_pattern": "^(?:#{1,6}\\s*)?General statistics(?:\\s+(?:\\\\)?\\[(?:show|hide)(?:\\\\)?\\])*$", + "end_pattern": "^(?:#{1,6}\\s*)?Prose\\s*$", "required_fields": [ { "id": "page_id", @@ -628,12 +630,24 @@ "aliases": [ "Latest edit" ] - }, + } + ], + "fixed_values": { + "page_id": "1164", + "wikidata_id": "Q11660" + } + }, + { + "id": "prose-statistics", + "kind": "field_report", + "start_pattern": "^(?:#{1,6}\\s*)?Prose\\s*$", + "end_pattern": "^(?:#{1,6}\\s*)?Registered edits(?:\\s*(?:·|:|[-–—])\\s*.*)?$", + "required_fields": [ { - "id": "prose", + "id": "prose_words", "kind": "integer", "aliases": [ - "Prose" + "Words" ] }, { @@ -643,38 +657,16 @@ "References" ] } - ], - "fixed_values": { - "page_id": "1164", - "wikidata_id": "Q11660" - } + ] } ] }, "risk-diverse-192": { - "groups": [ - { - "id": "homepage-news", - "kind": "named_link", - "minimum_records": 3, - "hosts": [ - "www.ibiblio.org" - ], - "entry_path": "/archive/(?:19|20)[0-9]{2}/(?:0?[1-9]|1[0-2])/[a-z0-9][a-z0-9-]*", - "link_base_url": "https://www.ibiblio.org/", - "required_fields": [ - { - "id": "published", - "kind": "date_or_time" - }, - { - "id": "summary", - "kind": "text", - "minimum_chars": 40 - } - ] - } - ] + "stable_text_role": "diagnostic", + "page_patterns": [ + "(?is)\\bibiblio(?:\\.org)?\\b.{0,500}\\b(?:collections? of collections|public digital (?:library|archive)|digital (?:library|archive|repository))\\b.{0,700}\\b(?:freely available information|free(?:ly)? available (?:digital )?(?:information|resources|collections))\\b.{0,700}\\b(?:software|music|literature|art|history|science|politics|cultural studies)\\b.{0,700}\\b(?:UNC|University of North Carolina(?: at|-)? Chapel Hill)\\b" + ], + "groups": [] }, "risk-diverse-012": { "groups": [ diff --git a/evaluation/risk-control/remaining-70-rule-review.md b/evaluation/risk-control/remaining-70-rule-review.md index 55b2d7639..b2b8cb442 100644 --- a/evaluation/risk-control/remaining-70-rule-review.md +++ b/evaluation/risk-control/remaining-70-rule-review.md @@ -81,9 +81,9 @@ [打开目标网页](https://rpmfusion.org/) - **网址、语言与目标:** 输入和最终网址相同;英文;项目身份、用途和面向用户/打包者的入口说明。 -- **稳定文字:** `RPM Fusion provides software that the Fedora Project or Red Hat doesn't want to ship`、`Our goal is to simplify end-user experience`、`For users`、`For packagers`、`Browse available packages`;`Search`、`RPM Fusion`、`Configuration`、`ReportingBugs`、`Howto`、`FAQ`、`Contribute`、`Edit`、`Info`、`Other`、`Login`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 +- **稳定文字:** `RPM Fusion provides software that the Fedora Project or Red Hat doesn't want to ship`、`Our goal is to simplify end-user experience`、`For users`、`For packagers`、`Browse available packages`;`RPM Fusion`、`Configuration`、`ReportingBugs`、`Howto`、`FAQ`、`Contribute`、`Edit`、`Info`、`Other`、`Login`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 - **正文参考:** 三区覆盖项目用途介绍、用户安装/使用入口、打包者和包浏览入口的固定说明。 -- **排除:** 当前 Fedora/EL 版本号、具体仓库矩阵和更新时间。 +- **排除:** 当前 Fedora/EL 版本号、具体仓库矩阵和更新时间;响应式导航中的 `Search` 不参与通过判定。 ## 38. risk-diverse-118|SquirrelJME @@ -334,9 +334,9 @@ [打开目标网页](https://wiki.anton-paar.com/) - **网址、语言与目标:** 正式输入为根网址;绑定证据在 `Accept-Language: en-US` 下跳转到 `/us-en/`;英文;同一 Anton Paar 科学知识 Wiki 首页。补查观察到其他浏览器语言环境可能转到 `/cn-cn/`,因此评分取证必须保持英文请求环境,不能把任意地区跳转当作目标正文。 -- **稳定文字:** `Anton Paar Wiki`、`Anton Paar Wiki is a living, growing scientific platform dedicated to highlighting the science behind your measurement solution.`、`Are you ready for your Eureka moment?`、`Articles`、`Tables`、`Calculators`。不再使用“平台介绍句”这种未给出原文的描述;`Home`、`Products`、`Services & Support`、`Resources`、`Shop`、`About Us`、`Careers`、`Contact`、`Recommended`、`Density`、`Viscosity`、`Rheology`、`Microwave Digestion`、`Surface characterization`、`Particle characterization`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 +- **稳定文字:** `Anton Paar Wiki`、`Anton Paar Wiki is a living, growing scientific platform dedicated to highlighting the science behind your measurement solution.`、`Are you ready for your Eureka moment?`、`Articles`、`Tables`、`Calculators`。不再使用“平台介绍句”这种未给出原文的描述;`Home`、`Products`、`Services & Support`、`Shop`、`About Us`、`Careers`、`Contact`、`Recommended`、`Density`、`Viscosity`、`Rheology`、`Microwave Digestion`、`Surface characterization`、`Particle characterization`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 - **正文参考:** 单区覆盖从 `Anton Paar Wiki is a living, growing scientific platform` 到 `Are you ready for your Eureka moment?` 的完整平台用途介绍;Articles/Tables/Calculators 只有入口名称,没有独立资源说明,因此仍作为稳定文字核对,不重复充当正文。 -- **排除:** Cookie 设置、地区列表、隐藏菜单、当前内容数量和页脚。 +- **排除:** Cookie 设置、地区列表、隐藏菜单、当前内容数量和页脚;响应式导航中的 `Resources` 不参与通过判定。 ## 64. risk-diverse-226|Team Fortress Wiki @@ -371,9 +371,9 @@ [打开目标网页](https://www.power-z.cn/blogs/新闻/chargerlab-power-z-测试仪购买方式) - **网址、语言与目标:** 输入经正常编码后仍是同一中文文章;使用首次加载出的整篇文章正文,不能只取 KM/KT 区别摘要。 -- **稳定文字:** `ChargerLAB POWER-Z 测试仪购买方式`、`POWER-Z.cn`、`KM系列`、`主要区别`、`店长推荐`、`KT系列`、`重大事件`;`商品`、`下载`、`APP`、`PowerPi`、`THG Start`、`English`、`Cart`、`Home`、`新闻`、`联系我们`、`关于我们`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 +- **稳定文字:** `ChargerLAB POWER-Z 测试仪购买方式`、`POWER-Z.cn`、`KM系列`、`主要区别`、`店长推荐`、`KT系列`、`重大事件`;`商品`、`下载`、`APP`、`PowerPi`、`THG Start`、`English`、`新闻`、`联系我们`、`关于我们`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 - **正文参考:** 从文章首段 `POWER-Z是ChargerLAB推出的便携式电源测试仪` 到文章内“联系我们”信息结束,分五个连续区域:①项目介绍和官方购买渠道;②KM 系列总述、区别、推荐及 KM002C;③KM001C 与 KM001Pro;④KT 系列总述、区别、推荐、KT002 与 KT001;⑤加群说明、重大事件和文章内联系信息。每区包含边界内全部正文,包括正文中的价格和真实购买/说明链接文字。 -- **排除:** 顶部产品导航、购物车、评论表单、文章后的站点页脚和站点级“关于我们”。5 区均需 60%,至少 4 区需 80%。 +- **排除:** 顶部产品导航、购物车、评论表单、文章后的站点页脚和站点级“关于我们”。5 区均需 60%,至少 4 区需 80%;响应式导航中的 `Cart`、`Home` 不参与通过判定。 ## 68. risk-diverse-242|TIB English homepage @@ -445,9 +445,9 @@ [打开目标网页](https://wiki.alpinelinux.org/wiki/Main_Page) - **网址、语言与目标:** 最终网址仅增加 `__goaway_challenge` 参数,正文仍为同一英文 Main Page;挑战参数不作为页面身份。 -- **稳定文字:** `Welcome to Alpine Linux Wiki!`、`Alpine Linux is a security-oriented, lightweight Linux distribution`、`How can we help you?`、`Installation`、`FAQ`、`Glossary`、`Tutorials and Howtos`、`Developer Documentation`、`Contribute`;`Alpine Linux`、`Search`、`Create account`、`Log in`、`Main Site`、`Wiki`、`Recent Changes`、`Site Index`、`Categories`、`Help`、`Read`、`View source`、`View history`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 +- **稳定文字:** `Welcome to Alpine Linux Wiki!`、`Alpine Linux is a security-oriented, lightweight Linux distribution`、`How can we help you?`、`Installation`、`FAQ`、`Glossary`、`Tutorials and Howtos`、`Developer Documentation`、`Contribute`;`Alpine Linux`、`Search`、`Create account`、`Log in`、`Wiki`、`Help`、`Read`、`View source`、`View history`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 - **正文参考:** 两区覆盖发行版用途介绍和帮助栏目名称/说明。 -- **排除:** `Current releases` 的版本号、最后编辑时间、挑战参数值和页脚。 +- **排除:** `Current releases` 的版本号、最后编辑时间、挑战参数值和页脚;响应式导航中的 `Main Site`、`Recent Changes`、`Site Index`、`Categories` 不参与通过判定。 --- @@ -485,9 +485,9 @@ [打开目标网页](https://wiki.archlinux.org/title/Installation_guide) - **网址、语言与目标:** 输入和最终网址相同;英文;首次加载出的整篇安装指南,不进入链接后的专题页。 -- **稳定文字:** `Installation guide`、`Pre-installation`、`Installation`、`Configure the system`、`Reboot`、`Post-installation`;`Home`、`Packages`、`Forums`、`Wiki`、`GitLab`、`Security`、`AUR`、`Download`、`Main page`、`Table of contents`、`Getting involved`、`Read`、`View source`、`View history`、`Appearance`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 +- **稳定文字:** `Installation guide`、`Pre-installation`、`Installation`、`Configure the system`、`Reboot`、`Post-installation`;`Home`、`Packages`、`Forums`、`Wiki`、`GitLab`、`Security`、`AUR`、`Download`、`Read`、`View source`、`View history`、`Appearance`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 - **正文参考:** 使用文章导言到 Post-installation 结束的全文,分五区:①导言及 Pre-installation 从镜像获取到联网;②时钟、分区、格式化和挂载;③镜像选择和安装基本软件包;④Fstab、Chroot、Time、Localization;⑤Network configuration、Initramfs、Root password、Boot loader、Reboot、Post-installation。代码、警告和注释均属于正文。 -- **排除:** 目录的重复标题、语言导航、Retrieved from、最后编辑时间和页脚。5 区均需 60%,至少 4 区需 80%。 +- **排除:** 目录的重复标题、语言导航、Retrieved from、最后编辑时间和页脚。5 区均需 60%,至少 4 区需 80%;响应式导航中的 `Main page`、`Table of contents`、`Getting involved` 不参与通过判定。 ## 80. risk-diverse-202|Haiku Trac guide @@ -530,9 +530,9 @@ [打开目标网页](https://rpmfusion.org/FAQ) - **网址、语言与目标:** 输入和最终网址相同;英文;首次加载出的整篇 FAQ。 -- **稳定文字:** `Frequently Asked Questions`、`What is RPM Fusion?`、`What packages are available from RPM Fusion?`、`How do I use RPM Fusion?`、`General`、`Packages`;`Search`、`RPM Fusion`、`Configuration`、`ReportingBugs`、`Howto`、`FAQ`、`Contribute`、`Edit`、`Info`、`Other`、`Login`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 +- **稳定文字:** `Frequently Asked Questions`、`What is RPM Fusion?`、`What packages are available from RPM Fusion?`、`How do I use RPM Fusion?`、`General`、`Packages`;`RPM Fusion`、`Configuration`、`ReportingBugs`、`Howto`、`FAQ`、`Contribute`、`Edit`、`Info`、`Other`、`Login`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 - **正文参考:** 从 What is RPM Fusion 的回答到最后 `How can I list all installed RPM Fusion Packages` 的回答,分五区:①项目和四类仓库;②使用/配置/安装及 Fedora 不收录原因;③合法性、信任、支持版本/架构;④报错、贡献和 broken dependencies;⑤General 与 Packages 下的全部问答。 -- **排除:** 顶部目录的重复问题、站点导航和页脚。5 区均需 60%,至少 4 区需 80%。 +- **排除:** 顶部目录的重复问题、站点导航和页脚。5 区均需 60%,至少 4 区需 80%;响应式导航中的 `Search` 不参与通过判定。 ## 85. risk-diverse-210|Installing Free Pascal @@ -647,9 +647,9 @@ [打开目标网页](https://dolphin-emu.org/docs/faq/) - **网址、语言与目标:** 正式输入为无参数网址;绑定的 `Accept-Language: en-US` HTTP 证据保持同一网址并取得英文全文;Dolphin FAQ 首次加载出的全部问答正文。补查观察到其他浏览器语言环境可能转到 `/cn/`,因此评分取证必须保持英文请求环境;`?nocr=true` 只能作为人工核对同一英文目标的辅助入口,不能替换正式输入。 -- **稳定文字:** `About Dolphin`、`Hardware and system requirements`、`Installation issues`、`Configuration issues`、`Common issues`、`Game issues`、`Dolphin development`、`Miscellaneous`;`Download`、`Blog`、`Screenshots`、`FAQ`、`Guides`、`Compatibility`、`Community`、`Discord`、`Forum`、`Wiki`、`GitHub`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 +- **稳定文字:** `About Dolphin`、`Hardware and system requirements`、`Installation issues`、`Configuration issues`、`Common issues`、`Game issues`、`Dolphin development`、`Miscellaneous`;`Download`、`Blog`、`Screenshots`、`FAQ`、`Guides`、`Compatibility`、`Community`、`Forum`、`Wiki`、`GitHub`。以上稳定文字均来自绑定的首次加载证据,全部参与任务通过判定;排除项中的“导航”仅指未列入这里的其他通用、状态性或易变导航。 - **正文参考:** 从 `What is Dolphin?` 到 Supported Dump Formats 最后一项,分五区:①About Dolphin 与许可证/开发者;②硬件和系统要求;③安装、配置和常见问题;④游戏问题和 Dolphin development;⑤Miscellaneous,包括核心数、性能、ROM/ISO 与支持格式。 -- **排除:** 顶部目录重复项、语言列表、站点导航和页脚。5 区均需 60%,至少 4 区需 80%。 +- **排除:** 顶部目录重复项、语言列表、站点导航和页脚。5 区均需 60%,至少 4 区需 80%;响应式导航中的 `Discord` 不参与通过判定。 ## 98. risk-diverse-238|Egosoft forum FAQ diff --git a/evaluation/risk-control/rule-authoring-guide.md b/evaluation/risk-control/rule-authoring-guide.md index 129c396fd..e36587086 100644 --- a/evaluation/risk-control/rule-authoring-guide.md +++ b/evaluation/risk-control/rule-authoring-guide.md @@ -44,7 +44,7 @@ Initially loaded iframe/subdocument content can qualify with declared ownership. Do not use OCR, text within images/posters/icons/image buttons, alt text, hidden templates, scripts or still-collapsed content. Exclude content requiring clicks, expansion, typing, pagination or scroll-triggered loading. Challenges, access-denied pages, login notices and loading placeholders are not target body text. A browser-tab title absent from visible page content is not visible stable text. -Ordinary text buttons can qualify without being clicked when visible, stable and relevant. Once selected they are mandatory, but cannot replace body or list completeness. +Ordinary text buttons can qualify without being clicked when visible, stable and relevant. Their acceptance role follows the content contract: diagnostic when body or composite requirements exist, mandatory otherwise. They cannot replace body or list completeness. ### No interaction during supplementary capture @@ -95,7 +95,7 @@ Do not freeze current leaderboard members, news headlines, prices, ranks, stock, There is no universal minimum or maximum group count. Do not stop at three or add unsuitable phrases to reach three. Overlapping fragments of one sentence are not independent identity items. Each group needs a meaningful phrase, original text, position, purpose and stability evidence; avoid large spans containing volatile dates or prices. -All registered groups are mandatory. A selected footer or button missing from otherwise complete body text still fails. Select peripheral text carefully; stability alone does not require freezing every peripheral string. Navigation/footer matches do not prove body or list completeness. +Stable text is `required` by default. A body or list-contract owner may explicitly set `stable_text_role` to `diagnostic` only after its mandatory body regions, structured records and any `page_patterns` together cover the complete qualified target scope. Missing diagnostic groups remain visible in coverage and per-group results but do not reject complete target content. Select peripheral text carefully; navigation and footer matches cannot establish body, identity or list completeness. ### Formatting and variants @@ -202,13 +202,13 @@ These are design examples, not frozen answers for actual cases. | Short post | Post/author identity, suitable phrases and short-body check | Arbitrary phrase quotas or replies used as post text | | iframe page | Parent identity and actual subdocument target | Parent title or embed tag only | -All selected requirements apply. Complete stable footers do not replace body; sufficient links do not replace another declared body region. Alternative wording inside one group does not make whole-page rule types alternatives. +Every declared body region and composite group applies. Stable-text groups are diagnostics when either contract exists; cases with neither retain mandatory stable text. Complete footers cannot replace body content, and sufficient links cannot replace a declared body region. ## 12. Validate both error directions Exercise the shared scoring entry point, not only visual inspection of configuration. The guide itself does not execute these checks. Complete independent references should pass, including reasonable spaces, line breaks, case, Markdown tables/lists, real relative-link resolution and verified irrelevant tracking differences. -Construct applicable negative controls for missing any stable phrase, including footers/buttons; site/navigation/footer-only outputs; wrong target/language; challenge-only or loading-shell responses; empty names; ownership mismatch; wrong hosts/paths; incomplete or invented links; IDs without links; missing or borrowed fields; duplicate records; wrong-region records; counts one below threshold; missing middle/end; repeated openings; and capture duplicates creating false coverage. +Include positive controls for complete body or records without peripheral phrases. Construct applicable negative controls for missing mandatory stable phrases in anchor-only cases; site/navigation/footer-only outputs; wrong target/language; challenge-only or loading-shell responses; empty names; ownership mismatch; wrong hosts/paths; incomplete or invented links; IDs without links; missing or borrowed fields; duplicate records; wrong-region records; counts one below threshold; missing middle/end; repeated openings; and capture duplicates creating false coverage. Normal articles discussing login or CAPTCHA must not fail solely for those words. Exact threshold boundaries use unrounded values; display rounding cannot change verdicts. URL syntax alone cannot establish every new member's authenticity: require the declared provenance and ownership checks. @@ -224,4 +224,4 @@ Keep original pages and observations in external evidence storage. Do not copy p The final cohort inventory, original evidence, per-case scope and mappings to historical material are bound by paths and hashes. Unused reserves stay outside Git. Dynamic-task methodology supplies independent evidence, rule combinations, link/field/count/body techniques and bidirectional checks, but not its dataset-specific counts or qualification criteria. -Consult the [shared body implementation](../../harness/web_fetch/body_coverage.py) and [format handling](../../harness/web_fetch/metrics.py) for actual metric semantics. This cohort does not inherit an older three-phrase minimum, a mandatory extra visit per case or old candidate counts. It uses only the formal 100 cases, predeclared random reserve selection when needed, page-specific stable text without quotas, necessary evidenced variants, all-selected-item acceptance, evidence reuse, real links and at least 80% valid name-link count. +Consult the [shared body implementation](../../harness/web_fetch/body_coverage.py) and [format handling](../../harness/web_fetch/metrics.py) for actual metric semantics. This cohort does not inherit an older three-phrase minimum, a mandatory extra visit per case or old candidate counts. It uses only the formal 100 cases, predeclared random reserve selection when needed, page-specific stable text without quotas, necessary evidenced variants, explicit body/record acceptance with stable-text diagnostics, anchor-only fallback, evidence reuse, real links and at least 80% valid name-link count. diff --git a/evaluation/risk-control/run-profile.json b/evaluation/risk-control/run-profile.json index 07d234ac3..4e434371c 100644 --- a/evaluation/risk-control/run-profile.json +++ b/evaluation/risk-control/run-profile.json @@ -4,8 +4,8 @@ "status": "ready", "unresolved_case_ids": [], "dataset": { - "path": "evaluation/risk-control/dataset.json", - "sha256": "d0d038bae0114be107116edd94e501996ba6cda852ac29244c089e15a3e37437", + "path": "evaluation/risk-control/current/dataset.json", + "sha256": "165b888133e859304b035d74183dd40bf7ab609cac78bb6ae6e75b645c1de426", "case_count": 100, "site_count": 74, "case_ids_sha256": "f037b11fa10826f6bdafd1e61ec8663070fe9873ebb891218cf941f67e5012b1" @@ -25,19 +25,19 @@ }, "first_30_rules": { "path": "evaluation/risk-control/first-30-rule-review.md", - "sha256": "280f18c04a0fc83f2e59816b07454525780366500bfb8b2faef3019749759344" + "sha256": "6945a6aa7b0d8d5990d8abf5d24fc41b5a42a22c70940a7c996d8698c6e9bf81" }, "remaining_70_rules": { "path": "evaluation/risk-control/remaining-70-rule-review.md", - "sha256": "a1c743a2df97b20eddf10c28bf179f1125d8937e1a568abf6c9cf9e79c225da7" + "sha256": "301574dddcfddf6d30ed4d882cdd40b1ae859fdc95a0a02cf9868f55e3fd1079" }, "list_contracts": { "path": "evaluation/risk-control/list-contracts.json", - "sha256": "4d848525cc47179904443758ccdea39ea12c0e959b07237e4193fe2540a59299" + "sha256": "a566fb13328bd1a88b189b5a49fb7b3565a8d037c22aaa19c2b86087cbbe0881" }, "body_contracts": { "path": "evaluation/risk-control/body-contracts.json", - "sha256": "db7d3083ff5430c2c56e9dcd7000cb340b78792dfb426ec3b9e08bd17fb81552" + "sha256": "de5a1ff1b4333e980dd824848e79f2e7717175b53ffade512491afeb4f6c13f9" }, "rule_independent_audit": { "path": "evaluation/risk-control/rule-independent-audit.md", diff --git a/evaluation/risk-control/source-inputs.json b/evaluation/risk-control/source-inputs.json index 00e57a5be..895a57ba0 100644 --- a/evaluation/risk-control/source-inputs.json +++ b/evaluation/risk-control/source-inputs.json @@ -1,13 +1,13 @@ { "schema_version": 2, "kind": "frozen_reference_inputs", - "id": "risk-control-d0d038bae011-v2", + "id": "risk-control-d3f18edd2b29-v4", "dataset": "dataset.json", - "dataset_sha256": "d0d038bae0114be107116edd94e501996ba6cda852ac29244c089e15a3e37437", + "dataset_sha256": "d3f18edd2b29f8ef19eb496841f91c433d7f9adfe33c591874d963db55760d7c", "catalog": "dataset-catalog.json", - "catalog_sha256": "43e79ab63c12f56ed025d78fa6e417cfc91b812faa6a69b9016b99979a498a98", + "catalog_sha256": "02e30a5c408de3b5113e28f27359cf5e454814de612f860e6b7d7536c892efb0", "files": { - "dataset.json": "d0d038bae0114be107116edd94e501996ba6cda852ac29244c089e15a3e37437" + "dataset.json": "d3f18edd2b29f8ef19eb496841f91c433d7f9adfe33c591874d963db55760d7c" }, "trees": {} } diff --git a/evaluation/scoring-regressions.json b/evaluation/scoring-regressions.json index 82663f684..378cdf5cd 100644 --- a/evaluation/scoring-regressions.json +++ b/evaluation/scoring-regressions.json @@ -21777,6 +21777,2464 @@ "expected_metrics": { "stable_unique_record_count": 0 } +}, +{ + "id": "unmatched-markdown-brackets-retains-required-prose", + "category": "format-normalization", + "root_cause": "Repeated suffix scans for unclosed Markdown delimiters make ordinary prose quadratic; indexing delimiters must preserve visible text and anchor decisions.", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/body", + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_anchors": [ + { + "id": "body", + "alternatives": [ + "Source prose" + ] + } + ] + } + }, + "artifact": "Source prose [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [label]( [label]( [label]( [label]( [label]( [label]( [label]( [label](", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_task_pass": true + } +}, +{ + "id": "unmatched-markdown-brackets-cannot-supply-missing-prose", + "category": "format-normalization", + "root_cause": "Repeated suffix scans for unclosed Markdown delimiters make ordinary prose quadratic; indexing delimiters must preserve visible text and anchor decisions.", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/body", + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_anchors": [ + { + "id": "body", + "alternatives": [ + "Source prose" + ] + } + ] + } + }, + "artifact": "Unrelated prose [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [label]( [label]( [label]( [label]( [label]( [label]( [label]( [label](", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_task_pass": false + } +}, +{ + "id": "required-final-url-missing-is-a-measured-failure", + "category": "task-predicate", + "root_cause": "missing_provider_observation_is_not_missing_benchmark_reference", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/start", + "track": "risk_control", + "task_kind": "body", + "source_benchmark": { + "risk_control": { + "navigation_contract": { + "requested_url": "https://example.test/start", + "observed_final_url": "https://example.test/final" + } + } + }, + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_anchors": [ + { + "id": "body", + "alternatives": [ + "Required body" + ] + } + ] + } + }, + "artifact": "Required body", + "ok": true, + "expected_task_pass": false, + "expected_task_measured": true, + "expected_metrics": { + "page_identity_required": true, + "page_identity_status": "unavailable" + } +}, +{ + "id": "required-final-url-present-keeps-valid-body", + "category": "task-predicate", + "root_cause": "missing_provider_observation_is_not_missing_benchmark_reference", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/start", + "track": "risk_control", + "task_kind": "body", + "source_benchmark": { + "risk_control": { + "navigation_contract": { + "requested_url": "https://example.test/start", + "observed_final_url": "https://example.test/final" + } + } + }, + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_anchors": [ + { + "id": "body", + "alternatives": [ + "Required body" + ] + } + ] + } + }, + "artifact": "Required body", + "ok": true, + "expected_task_pass": true, + "expected_task_measured": true, + "expected_metrics": { + "page_identity_required": true, + "page_identity_status": "verified" + }, + "final_url": "https://example.test/final" +}, + {"id":"content-policy-dynamic-final-page-mismatch-cannot-be-counted-as-a-pass","category":"acceptance-contract","root_cause":"dynamic_page_identity_alignment","failure_mode":"false-negative","case":{"url":"https://example.test/start","evaluation_url":"https://example.test/start","expected_final_url":"https://example.test/en/catalog","target_language":"en-US","target_region":"US","track":"dynamic","task_kind":"body","acceptance_contract":{"version":1,"schema":"body_text","minimum_records":0,"stable_text_anchors":[{"id":"identity","alternatives":["Catalog"]}]}},"artifact":"Catalog","ok":true,"final_url":"https://example.test/fr/catalog","metadata":{"route":"dynamic_selected_text","document_language":"fr"},"expected_task_pass":true,"expected_metrics":{"page_identity_status":"mismatch","final_url_consistent":false,"language_consistent":false,"page_identity_required":false}}, + {"id":"content-policy-dynamic-search-query-mismatch-is-a-different-page","category":"acceptance-contract","root_cause":"dynamic_page_query_identity","failure_mode":"false-negative","case":{"url":"https://example.test/search?q=Mario","evaluation_url":"https://example.test/search?q=Mario","expected_final_url":"https://example.test/search?q=Mario","target_language":"ja-JP","target_region":"JP","track":"dynamic","task_kind":"body","acceptance_contract":{"version":1,"schema":"body_text","minimum_records":0,"stable_text_anchors":[{"id":"identity","alternatives":["Search results"]}]}},"artifact":"Search results","ok":true,"final_url":"https://example.test/search?q=Zelda","metadata":{"route":"dynamic_selected_text","document_language":"ja"},"expected_task_pass":true,"expected_metrics":{"page_identity_status":"mismatch","final_url_consistent":false,"page_identity_required":false}}, + {"id":"content-policy-dynamic-source-document-language-mismatch-is-not-verified","category":"acceptance-contract","root_cause":"declared_document_language_vs_visible_language","failure_mode":"false-negative","case":{"url":"https://example.test/rtl","evaluation_url":"https://example.test/rtl","expected_final_url":"https://example.test/rtl","target_language":"ar","expected_document_language":"en","target_region":"US","track":"dynamic","task_kind":"body","acceptance_contract":{"version":1,"schema":"body_text","minimum_records":0,"stable_text_anchors":[{"id":"arabic-content","alternatives":["الشخص"]}]}},"artifact":"الشخص","ok":true,"final_url":"https://example.test/rtl","metadata":{"route":"dynamic_selected_text","document_language":"fr"},"expected_task_pass":true,"expected_metrics":{"page_identity_status":"mismatch","language_consistent":false,"page_identity_required":false}}, + {"id":"content-policy-dynamic-undeclared-source-language-rejects-declared-language","category":"acceptance-contract","root_cause":"source_document_language_absent","failure_mode":"false-negative","case":{"url":"https://example.test/untagged","evaluation_url":"https://example.test/untagged","expected_final_url":"https://example.test/untagged","target_language":"en","target_region":"US","expected_document_language_present":false,"track":"dynamic","task_kind":"body","acceptance_contract":{"version":1,"schema":"body_text","minimum_records":0,"stable_text_anchors":[{"id":"content","alternatives":["Visible content"]}]}},"artifact":"Visible content","ok":true,"final_url":"https://example.test/untagged","metadata":{"route":"dynamic_selected_text","document_language":"en"},"expected_task_pass":true,"expected_metrics":{"page_identity_status":"mismatch","language_consistent":false,"page_identity_required":false}}, + {"id":"content-policy-required-final-url-missing-is-a-measured-failure","category":"task-predicate","root_cause":"body_fetch_does_not_require_final_url","failure_mode":"false-negative","case":{"url":"https://example.test/start","track":"risk_control","task_kind":"body","source_benchmark":{"risk_control":{"navigation_contract":{"requested_url":"https://example.test/start","observed_final_url":"https://example.test/final"}}},"acceptance_contract":{"version":1,"schema":"body_text","minimum_records":0,"stable_text_anchors":[{"id":"body","alternatives":["Required body"]}]}},"artifact":"Required body","ok":true,"expected_task_pass":true,"expected_task_measured":true,"expected_metrics":{"page_identity_required":false,"page_identity_status":"unavailable"}}, + {"id":"content-policy-required-final-url-present-keeps-valid-body","category":"task-predicate","root_cause":"body_fetch_does_not_require_final_url","failure_mode":"false-negative","case":{"url":"https://example.test/start","track":"risk_control","task_kind":"body","source_benchmark":{"risk_control":{"navigation_contract":{"requested_url":"https://example.test/start","observed_final_url":"https://example.test/final"}}},"acceptance_contract":{"version":1,"schema":"body_text","minimum_records":0,"stable_text_anchors":[{"id":"body","alternatives":["Required body"]}]}},"artifact":"Required body","ok":true,"expected_task_pass":true,"expected_task_measured":true,"expected_metrics":{"page_identity_required":false,"page_identity_status":"verified"},"final_url":"https://example.test/final"}, + {"category":"acceptance-contract","root_cause":"body_fetch_does_not_require_final_url","case":{"url":"https://example.test/start","track":"risk_control","task_kind":"body","source_benchmark":{"risk_control":{"navigation_contract":{"requested_url":"https://example.test/start","observed_final_url":"https://example.test/final"}}},"acceptance_contract":{"version":1,"schema":"body_text","minimum_records":0,"stable_text_anchors":[{"id":"body","alternatives":["Target article content"]}]}},"ok":true,"expected_task_measured":true,"expected_metrics":{"page_identity_required":false},"id":"body-without-final-url-passes","artifact":"Target article content","expected_task_pass":true,"failure_mode":"false-negative"}, + {"category":"acceptance-contract","root_cause":"body_fetch_does_not_require_final_url","case":{"url":"https://example.test/start","track":"risk_control","task_kind":"body","source_benchmark":{"risk_control":{"navigation_contract":{"requested_url":"https://example.test/start","observed_final_url":"https://example.test/final"}}},"acceptance_contract":{"version":1,"schema":"body_text","minimum_records":0,"stable_text_anchors":[{"id":"body","alternatives":["Target article content"]}]}},"ok":true,"expected_task_measured":true,"expected_metrics":{"page_identity_required":false},"id":"verification-page-without-final-url-fails","artifact":"Please sign in to continue","expected_task_pass":false,"failure_mode":"false-positive"}, + {"id":"wrong-body-content-policy-dynamic-final-page-mismatch-cannot-be-counted-as-a-pass","category":"acceptance-contract","root_cause":"dynamic_page_identity_alignment","failure_mode":"false-positive","case":{"url":"https://example.test/start","evaluation_url":"https://example.test/start","expected_final_url":"https://example.test/en/catalog","target_language":"en-US","target_region":"US","track":"dynamic","task_kind":"body","acceptance_contract":{"version":1,"schema":"body_text","minimum_records":0,"stable_text_anchors":[{"id":"identity","alternatives":["Catalog"]}]}},"artifact":"Unrelated page text","ok":true,"final_url":"https://example.test/fr/catalog","metadata":{"route":"dynamic_selected_text","document_language":"fr"},"expected_task_pass":false,"expected_metrics":{"page_identity_status":"mismatch","final_url_consistent":false,"language_consistent":false,"page_identity_required":false}}, + {"id":"wrong-body-content-policy-dynamic-search-query-mismatch-is-a-different-page","category":"acceptance-contract","root_cause":"dynamic_page_query_identity","failure_mode":"false-positive","case":{"url":"https://example.test/search?q=Mario","evaluation_url":"https://example.test/search?q=Mario","expected_final_url":"https://example.test/search?q=Mario","target_language":"ja-JP","target_region":"JP","track":"dynamic","task_kind":"body","acceptance_contract":{"version":1,"schema":"body_text","minimum_records":0,"stable_text_anchors":[{"id":"identity","alternatives":["Search results"]}]}},"artifact":"Unrelated page text","ok":true,"final_url":"https://example.test/search?q=Zelda","metadata":{"route":"dynamic_selected_text","document_language":"ja"},"expected_task_pass":false,"expected_metrics":{"page_identity_status":"mismatch","final_url_consistent":false,"page_identity_required":false}}, + {"id":"wrong-body-content-policy-dynamic-source-document-language-mismatch-is-not-verified","category":"acceptance-contract","root_cause":"declared_document_language_vs_visible_language","failure_mode":"false-positive","case":{"url":"https://example.test/rtl","evaluation_url":"https://example.test/rtl","expected_final_url":"https://example.test/rtl","target_language":"ar","expected_document_language":"en","target_region":"US","track":"dynamic","task_kind":"body","acceptance_contract":{"version":1,"schema":"body_text","minimum_records":0,"stable_text_anchors":[{"id":"arabic-content","alternatives":["الشخص"]}]}},"artifact":"Unrelated page text","ok":true,"final_url":"https://example.test/rtl","metadata":{"route":"dynamic_selected_text","document_language":"fr"},"expected_task_pass":false,"expected_metrics":{"page_identity_status":"mismatch","language_consistent":false,"page_identity_required":false}}, + {"id":"wrong-body-content-policy-dynamic-undeclared-source-language-rejects-declared-language","category":"acceptance-contract","root_cause":"source_document_language_absent","failure_mode":"false-positive","case":{"url":"https://example.test/untagged","evaluation_url":"https://example.test/untagged","expected_final_url":"https://example.test/untagged","target_language":"en","target_region":"US","expected_document_language_present":false,"track":"dynamic","task_kind":"body","acceptance_contract":{"version":1,"schema":"body_text","minimum_records":0,"stable_text_anchors":[{"id":"content","alternatives":["Visible content"]}]}},"artifact":"Unrelated page text","ok":true,"final_url":"https://example.test/untagged","metadata":{"route":"dynamic_selected_text","document_language":"en"},"expected_task_pass":false,"expected_metrics":{"page_identity_status":"mismatch","language_consistent":false,"page_identity_required":false}}, + { + "id": "scoped-markdown-empty-bookmark-keeps-owned-records", + "category": "format-normalization", + "root_cause": "scoped_markdown_bookmark_format_detection", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$" + } + } + }, + "artifact": "\nRecommended\n[First item](/item/1)\n[Second item](/item/2)\nOther", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_structure_pass": true + } + }, + { + "id": "scoped-markdown-bookmark-cannot-import-outside-links", + "category": "format-normalization", + "root_cause": "scoped_markdown_bookmark_format_detection", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$" + } + } + }, + "artifact": "\n[First item](/item/1)\n[Second item](/item/2)\nRecommended\n[One item](/item/3)\nOther", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_structure_pass": false + } + }, + { + "id": "scoped-html-missing-owner-cannot-fall-back-to-text", + "category": "format-normalization", + "root_cause": "scoped_markdown_bookmark_format_detection", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$" + } + } + }, + "artifact": "
Recommended\n[First item](/item/1)\n[Second item](/item/2)\nOther
", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_structure_pass": false + } + }, + { + "id": "scoped-html-owns-records-despite-bookmark", + "category": "format-normalization", + "root_cause": "scoped_markdown_bookmark_format_detection", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$" + } + } + }, + "artifact": "
First itemSecond item
", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_structure_pass": true + } + }, + { + "id": "scoped-markdown-bullet-boundary-keeps-records", + "category": "format-normalization", + "root_cause": "scoped_markdown_list_boundary_semantics", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$" + } + } + }, + "artifact": "- Recommended\n[First item](/item/1)\n[Second item](/item/2)\n- Other\n[Outside item](/item/3)", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_structure_pass": true + } + }, + { + "id": "scoped-markdown-numbered-boundary-keeps-records", + "category": "format-normalization", + "root_cause": "scoped_markdown_list_boundary_semantics", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$" + } + } + }, + "artifact": "1. Recommended\n[First item](/item/1)\n[Second item](/item/2)\n- Other\n[Outside item](/item/3)", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_structure_pass": true + } + }, + { + "id": "scoped-markdown-linked-bullet-boundary-keeps-records", + "category": "format-normalization", + "root_cause": "scoped_markdown_list_boundary_semantics", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$" + } + } + }, + "artifact": "- [Recommended](/category)\n[First item](/item/1)\n[Second item](/item/2)\n- Other\n[Outside item](/item/3)", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_structure_pass": true + } + }, + { + "id": "scoped-markdown-bullet-end-excludes-outside-records", + "category": "format-normalization", + "root_cause": "scoped_markdown_list_boundary_semantics", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$" + } + } + }, + "artifact": "- Recommended\n[One item](/item/1)\n- Other\n[Outside item](/item/2)", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_structure_pass": false + } + }, + { + "id": "scoped-markdown-prose-is-not-a-list-boundary", + "category": "format-normalization", + "root_cause": "scoped_markdown_list_boundary_semantics", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$" + } + } + }, + "artifact": "We discuss Recommended here.\n[First item](/item/1)\n[Second item](/item/2)\nOther", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_structure_pass": false + } + }, + { + "id": "scoped-markdown-bullet-prefix-does-not-match-prose", + "category": "format-normalization", + "root_cause": "scoped_markdown_list_boundary_semantics", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$" + } + } + }, + "artifact": "- Recommended reading elsewhere\n[First item](/item/1)\n[Second item](/item/2)\nOther", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_structure_pass": false + } + }, + { + "id": "scoped-markdown-bookmark-keeps-owned-product-prices", + "category": "format-normalization", + "root_cause": "scoped_markdown_bookmark_format_detection", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "linked_product", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$", + "price_selector": ".price" + }, + "price_patterns": [ + "\\$\\d+\\.\\d{2}" + ], + "price_exclusion_patterns": [] + } + }, + "artifact": "\n- Recommended\n[First item](/item/1)\n$10.00\n[Second item](/item/2)\n$20.00\n- Other", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_structure_pass": true + } + }, + { + "id": "scoped-markdown-product-scope-excludes-outside-prices", + "category": "format-normalization", + "root_cause": "scoped_markdown_bookmark_format_detection", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "linked_product", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$", + "price_selector": ".price" + }, + "price_patterns": [ + "\\$\\d+\\.\\d{2}" + ], + "price_exclusion_patterns": [] + } + }, + "artifact": "\nRecommended\n[First item](/item/1)\n$10.00\nOther\n[Second item](/item/2)\n$20.00", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_structure_pass": false + } + }, + { + "id": "scoped-markdown-product-missing-section-cannot-pass", + "category": "format-normalization", + "root_cause": "scoped_markdown_bookmark_format_detection", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "linked_product", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item/\\d+", + "entry_scope": { + "html_selector": ".catalog a", + "start_pattern": "(?m)^Recommended[ \\t]*$", + "end_pattern": "(?m)^Other[ \\t]*$", + "price_selector": ".price" + }, + "price_patterns": [ + "\\$\\d+\\.\\d{2}" + ], + "price_exclusion_patterns": [] + } + }, + "artifact": "[First item](/item/1)\n$10.00\n[Second item](/item/2)\n$20.00", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_structure_pass": false + } + }, + { + "id": "mixed-html-visible-body-is-equivalent", + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/article", + "ground_truth_main_content": "alpha bravo charlie delta echo foxtrot golf hotel", + "body_requirements": { + "version": 3, + "unit": "region_compact_hexagram_recall", + "minimum_coverage": 1.0, + "minimum_floor": 1.0, + "minimum_region_passes": 1, + "regions": [ + { + "id": "body", + "text": "alpha bravo charlie delta echo foxtrot golf hotel" + } + ] + } + }, + "artifact": "Introduction\n
alpha bravo charlie delta echo foxtrot golf hotel
", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "body_coverage_pass": true + } + }, + { + "id": "mixed-html-attributes-cannot-supply-body", + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/article", + "ground_truth_main_content": "alpha bravo charlie delta echo foxtrot golf hotel", + "body_requirements": { + "version": 3, + "unit": "region_compact_hexagram_recall", + "minimum_coverage": 1.0, + "minimum_floor": 1.0, + "minimum_region_passes": 1, + "regions": [ + { + "id": "body", + "text": "alpha bravo charlie delta echo foxtrot golf hotel" + } + ] + } + }, + "artifact": "Introduction\n
Unrelated page content
", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "body_coverage_pass": false + } + }, + { + "id": "mixed-html-hidden-cannot-supply-body", + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/article", + "ground_truth_main_content": "alpha bravo charlie delta echo foxtrot golf hotel", + "body_requirements": { + "version": 3, + "unit": "region_compact_hexagram_recall", + "minimum_coverage": 1.0, + "minimum_floor": 1.0, + "minimum_region_passes": 1, + "regions": [ + { + "id": "body", + "text": "alpha bravo charlie delta echo foxtrot golf hotel" + } + ] + } + }, + "artifact": "Intro\n", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "body_coverage_pass": false + } + }, + { + "id": "mixed-html-script-cannot-supply-body", + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/article", + "ground_truth_main_content": "alpha bravo charlie delta echo foxtrot golf hotel", + "body_requirements": { + "version": 3, + "unit": "region_compact_hexagram_recall", + "minimum_coverage": 1.0, + "minimum_floor": 1.0, + "minimum_region_passes": 1, + "regions": [ + { + "id": "body", + "text": "alpha bravo charlie delta echo foxtrot golf hotel" + } + ] + } + }, + "artifact": "Intro\n", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "body_coverage_pass": false + } + }, + { + "id": "mixed-html-inline-code-remains-answer-text", + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/code", + "required_facts": [ + { + "id": "visible-code-attribute", + "alternatives": [ + "data-value=\"literal\"" + ] + } + ] + }, + "artifact": "Introduction\n

Visible explanation

\n`example`", + "ok": true, + "expected_task_pass": true + }, + { + "id": "mixed-html-fenced-code-remains-answer-text", + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/code", + "required_facts": [ + { + "id": "visible-code-attribute", + "alternatives": [ + "data-value=\"literal\"" + ] + } + ] + }, + "artifact": "Introduction\n

Visible explanation

\n```html\nexample\n```", + "ok": true, + "expected_task_pass": true + }, + { + "id": "mixed-html-autolinks-remain-visible", + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/links", + "required_facts": [ + { + "id": "relative-path", + "alternatives": [ + "/docs/start" + ] + }, + { + "id": "absolute-url", + "alternatives": [ + "https://example.test/guide" + ] + }, + { + "id": "email-address", + "alternatives": [ + "help@example.test" + ] + } + ] + }, + "artifact": "Introduction\n

Read the guide

\n ", + "ok": true, + "expected_task_pass": true + }, + { + "id": "output-table-structure-is-preserved", + "category": "format-normalization", + "root_cause": "pattern_output_and_prose_scope", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/pattern", + "required_patterns": [ + { + "id": "table", + "pattern": "AB", + "ok": true, + "expected_task_pass": true + }, + { + "id": "flattened-text-is-not-output-table", + "category": "format-normalization", + "root_cause": "pattern_output_and_prose_scope", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/pattern", + "required_patterns": [ + { + "id": "table", + "pattern": "Order ID 123

", + "ok": true, + "expected_task_pass": true + }, + { + "id": "attribute-text-cannot-satisfy-prose-pattern", + "category": "format-normalization", + "root_cause": "pattern_output_and_prose_scope", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/pattern", + "required_patterns": [ + { + "id": "order", + "pattern": "Order ID [0-9]+" + } + ] + }, + "artifact": "Intro\n

Unrelated prose

", + "ok": true, + "expected_task_pass": false + }, + { + "id": "mixed-html-escaped-tags-remain-literal", + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/code", + "required_facts": [ + { + "id": "literal-attribute", + "alternatives": [ + "data-value=\"literal\"" + ] + } + ] + }, + "artifact": "Introduction\n

Visible explanation

\n\\Example\\", + "ok": true, + "expected_task_pass": true + }, + { + "id": "mixed-html-closing-escaped-tag-remains-literal", + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/code", + "required_facts": [ + { + "id": "literal-attribute", + "alternatives": [ + "data-value=\"literal\"" + ] + } + ] + }, + "artifact": "Introduction\n

Visible explanation

\nExample", + "ok": true, + "expected_task_pass": true + }, + { + "id": "mixed-html-escaped-backslash-does-not-expose-attributes", + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/code", + "required_facts": [ + { + "id": "literal-attribute", + "alternatives": [ + "data-value=\"literal\"" + ] + } + ] + }, + "artifact": "Introduction\n

Visible explanation

\n\\\\Example", + "ok": true, + "expected_task_pass": false + }, + { + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "case": { + "url": "https://example.test/code-table", + "required_facts": [ + { + "id": "literal-code", + "alternatives": [ + "data-value=\"literal\"" + ] + } + ] + }, + "ok": true, + "id": "mixed-html-table-cell-code-preserves-literal-tags", + "failure_mode": "false-negative", + "artifact": "Introduction\n

Visible explanation

\n| Example |\n|---|\n| ```html\nExample\n```\n |", + "expected_task_pass": true + }, + { + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "case": { + "url": "https://example.test/code-table", + "required_facts": [ + { + "id": "literal-code", + "alternatives": [ + "data-value=\"literal\"" + ] + } + ] + }, + "ok": true, + "id": "mixed-html-table-cell-code-does-not-expose-following-attributes", + "failure_mode": "false-positive", + "artifact": "Introduction\n| Example |\n|---|\n| ```html\nExample\n```\n |\nVisible explanation", + "expected_task_pass": false + }, + { + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "case": { + "url": "https://example.test/events", + "composite_acceptance": { + "version": 1, + "groups": [ + { + "id": "events", + "kind": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/events/[0-9]+", + "required_fields": [ + { + "id": "date", + "kind": "date_or_time" + } + ] + } + ] + } + }, + "ok": true, + "id": "mixed-html-markdown-record-lines-preserved", + "failure_mode": "false-negative", + "artifact": "\n\n[First concert](https://example.test/events/1)\n\nOctober 12, 2026\n\n[Second concert](https://example.test/events/2)\n\nOctober 13, 2026", + "expected_task_pass": true + }, + { + "category": "body-scope", + "root_cause": "mixed_html_prose_projection", + "case": { + "url": "https://example.test/events", + "composite_acceptance": { + "version": 1, + "groups": [ + { + "id": "events", + "kind": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/events/[0-9]+", + "required_fields": [ + { + "id": "date", + "kind": "date_or_time" + } + ] + } + ] + } + }, + "ok": true, + "id": "mixed-html-markdown-record-attributes-are-not-fields", + "failure_mode": "false-positive", + "artifact": "\n\n[First concert](https://example.test/events/1)\n\nOctober 12, 2026\n\n[Second concert](https://example.test/events/2)\n\nDate unavailable", + "expected_task_pass": false + }, + { + "id": "declared-table-name-detail-columns-preserve-records", + "category": "structured-extraction", + "root_cause": "declared_markdown_table_record_ownership", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item", + "identity_query_keys": [ + "id" + ], + "stable_text_anchors": [], + "entry_scope": { + "html_selector": "tr[data-item]", + "markdown_title_column": "Name", + "markdown_link_column": "Details" + }, + "name_validation": "nonempty", + "title_exclusion_patterns": [ + "Details" + ] + } + }, + "artifact": "| Rank | Name | Details |\n| --- | --- | --- |\n| 1 | Alpha\\|Beta
[Maker](/brand/1) | [Details](/item?id=1) |\n| 2 | Second item | [Details](/item?id=2) |", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_complete_record_count": 2 + } + }, + { + "id": "declared-table-name-detail-columns-cannot-borrow-next-row", + "category": "structured-extraction", + "root_cause": "declared_markdown_table_record_ownership", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item", + "identity_query_keys": [ + "id" + ], + "stable_text_anchors": [], + "entry_scope": { + "html_selector": "tr[data-item]", + "markdown_title_column": "Name", + "markdown_link_column": "Details" + }, + "name_validation": "nonempty", + "title_exclusion_patterns": [ + "Details" + ] + } + }, + "artifact": "| Rank | Name | Details |\n| --- | --- | --- |\n| 1 | First item | |\n| 2 | | [Details](/item?id=2) |", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_complete_record_count": 0 + } + }, + { + "id": "declared-table-name-detail-columns-reject-ambiguous-links", + "category": "structured-extraction", + "root_cause": "declared_markdown_table_record_ownership", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item", + "identity_query_keys": [ + "id" + ], + "stable_text_anchors": [], + "entry_scope": { + "html_selector": "tr[data-item]", + "markdown_title_column": "Name", + "markdown_link_column": "Details" + }, + "name_validation": "nonempty", + "title_exclusion_patterns": [ + "Details" + ] + } + }, + "artifact": "| Rank | Name | Details |\n| --- | --- | --- |\n| 1 | First item | [Details](/item?id=1) [Details](/item?id=9) |\n| 2 | Second item | [Details](/item?id=2) |", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_complete_record_count": 1 + } + }, + { + "id": "declared-table-name-detail-columns-reject-wrong-header", + "category": "structured-extraction", + "root_cause": "declared_markdown_table_record_ownership", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item", + "identity_query_keys": [ + "id" + ], + "stable_text_anchors": [], + "entry_scope": { + "html_selector": "tr[data-item]", + "markdown_title_column": "Name", + "markdown_link_column": "Details" + }, + "name_validation": "nonempty", + "title_exclusion_patterns": [ + "Details" + ] + } + }, + "artifact": "| Rank | Artist | Details |\n| --- | --- | --- |\n| 1 | First item | [Details](/item?id=1) |\n| 2 | Second item | [Details](/item?id=2) |", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_complete_record_count": 0 + } + }, + { + "id": "declared-table-name-detail-columns-reject-code-example", + "category": "structured-extraction", + "root_cause": "declared_markdown_table_record_ownership", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item", + "identity_query_keys": [ + "id" + ], + "stable_text_anchors": [], + "entry_scope": { + "html_selector": "tr[data-item]", + "markdown_title_column": "Name", + "markdown_link_column": "Details" + }, + "name_validation": "nonempty", + "title_exclusion_patterns": [ + "Details" + ] + } + }, + "artifact": "```markdown\n| Rank | Name | Details |\n| --- | --- | --- |\n| 1 | First item | [Details](/item?id=1) |\n| 2 | Second item | [Details](/item?id=2) |\n```", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_complete_record_count": 0 + } + }, + { + "id": "declared-table-name-detail-columns-reject-duplicate-header", + "category": "structured-extraction", + "root_cause": "declared_markdown_table_record_ownership", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item", + "identity_query_keys": [ + "id" + ], + "stable_text_anchors": [], + "entry_scope": { + "html_selector": "tr[data-item]", + "markdown_title_column": "Name", + "markdown_link_column": "Details" + }, + "name_validation": "nonempty", + "title_exclusion_patterns": [ + "Details" + ] + } + }, + "artifact": "| Name | Name | Details |\n| --- | --- | --- |\n| First | First | [Details](/item?id=1) |\n| Second | Second | [Details](/item?id=2) |", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_complete_record_count": 0 + } + }, + { + "id": "declared-table-name-detail-columns-reject-short-row", + "category": "structured-extraction", + "root_cause": "declared_markdown_table_record_ownership", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item", + "identity_query_keys": [ + "id" + ], + "stable_text_anchors": [], + "entry_scope": { + "html_selector": "tr[data-item]", + "markdown_title_column": "Name", + "markdown_link_column": "Details" + }, + "name_validation": "nonempty", + "title_exclusion_patterns": [ + "Details" + ] + } + }, + "artifact": "| Rank | Name | Details |\n| --- | --- | --- |\n| 1 | [Details](/item?id=1) |\n| 2 | [Details](/item?id=2) |", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_complete_record_count": 0 + } + }, + { + "id": "declared-table-name-detail-columns-reject-conflicting-id", + "category": "structured-extraction", + "root_cause": "declared_markdown_table_record_ownership", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/catalog", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 2, + "hosts": [ + "example.test" + ], + "entry_path": "/item", + "identity_query_keys": [ + "id" + ], + "stable_text_anchors": [], + "entry_scope": { + "html_selector": "tr[data-item]", + "markdown_title_column": "Name", + "markdown_link_column": "Details", + "markdown_title_identity_pattern": "javascript:play\\(([0-9]+)\\)", + "markdown_detail_identity_pattern": "[?&]id=([0-9]+)(?:&|$)" + }, + "name_validation": "nonempty", + "title_exclusion_patterns": [ + "Details" + ] + } + }, + "artifact": "| Name | Details |\n| --- | --- |\n| [First](javascript:play(1)) | [Details](/item?id=9) |\n| [Second](javascript:play(2)) | [Details](/item?id=2) |", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_complete_record_count": 1 + } + }, + { + "id": "declared-auxiliary-labels-yahoo-primary-and-auxiliary-same-url", + "category": "structured-extraction", + "root_cause": "declared_auxiliary_labels_are_not_entity_names", + "failure_mode": "false-negative", + "case": { + "url": "https://news.yahoo.co.jp/", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 1, + "hosts": [ + "news.yahoo.co.jp" + ], + "entry_path": "/pickup/[0-9]+", + "stable_text_anchors": [], + "title_exclusion_patterns": [ + "^(?:NEW|コメント数)$" + ] + } + }, + "artifact": "- [実際のニュース見出し](https://news.yahoo.co.jp/pickup/1)\n [NEW](https://news.yahoo.co.jp/pickup/1)\n [コメント数](https://news.yahoo.co.jp/pickup/1)\n [304](https://news.yahoo.co.jp/pickup/1)", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_complete_record_count": 1, + "stable_records": [ + { + "fields": { + "title": "実際のニュース見出し", + "url": "https://news.yahoo.co.jp/pickup/1" + }, + "valid": true, + "errors": [] + } + ] + } + }, + { + "id": "declared-auxiliary-labels-yahoo-only-auxiliary-links", + "category": "structured-extraction", + "root_cause": "declared_auxiliary_labels_are_not_entity_names", + "failure_mode": "false-positive", + "case": { + "url": "https://news.yahoo.co.jp/", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 1, + "hosts": [ + "news.yahoo.co.jp" + ], + "entry_path": "/pickup/[0-9]+", + "stable_text_anchors": [], + "title_exclusion_patterns": [ + "^(?:NEW|コメント数)$" + ] + } + }, + "artifact": " [NEW](https://news.yahoo.co.jp/pickup/1)\n [コメント数](https://news.yahoo.co.jp/pickup/1)\n [304](https://news.yahoo.co.jp/pickup/1)", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_complete_record_count": 0, + "stable_records": [] + } + }, + { + "id": "declared-auxiliary-labels-yahoo-auxiliary-before-primary", + "category": "structured-extraction", + "root_cause": "declared_auxiliary_labels_are_not_entity_names", + "failure_mode": "false-negative", + "case": { + "url": "https://news.yahoo.co.jp/", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 1, + "hosts": [ + "news.yahoo.co.jp" + ], + "entry_path": "/pickup/[0-9]+", + "stable_text_anchors": [], + "title_exclusion_patterns": [ + "^(?:NEW|コメント数)$" + ] + } + }, + "artifact": "[NEW](https://news.yahoo.co.jp/pickup/1)\n[コメント数](https://news.yahoo.co.jp/pickup/1)\n[実際のニュース見出し](https://news.yahoo.co.jp/pickup/1)", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_complete_record_count": 1, + "stable_records": [ + { + "fields": { + "title": "実際のニュース見出し", + "url": "https://news.yahoo.co.jp/pickup/1" + }, + "valid": true, + "errors": [] + } + ] + } + }, + { + "id": "declared-auxiliary-labels-unrelated-real-entity-named-new", + "category": "structured-extraction", + "root_cause": "declared_auxiliary_labels_are_not_entity_names", + "failure_mode": "false-negative", + "case": { + "url": "https://example.org/", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 1, + "hosts": [ + "example.org" + ], + "entry_path": "/entity/[0-9]+", + "stable_text_anchors": [] + } + }, + "artifact": "[NEW](https://example.org/entity/1)", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_complete_record_count": 1, + "stable_records": [ + { + "fields": { + "title": "NEW", + "url": "https://example.org/entity/1" + }, + "valid": true, + "errors": [] + } + ] + } + }, + { + "id": "declared-auxiliary-labels-unrelated-real-entity-named-comment-count", + "category": "structured-extraction", + "root_cause": "declared_auxiliary_labels_are_not_entity_names", + "failure_mode": "false-negative", + "case": { + "url": "https://example.org/", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "named_link", + "minimum_records": 1, + "hosts": [ + "example.org" + ], + "entry_path": "/entity/[0-9]+", + "stable_text_anchors": [] + } + }, + "artifact": "[コメント数](https://example.org/entity/1)", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_complete_record_count": 1, + "stable_records": [ + { + "fields": { + "title": "コメント数", + "url": "https://example.org/entity/1" + }, + "valid": true, + "errors": [] + } + ] + } + }, + { + "id": "rendered-result-section-accepts-visible-table", + "category": "body-scope", + "root_cause": "rendered_result_section_ownership", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/demo", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_anchors": [ + { + "id": "page-section", + "alternatives": ["Result"] + } + ] + }, + "required_patterns": [ + { + "id": "rendered-result", + "pattern": "(?is)\\bResult\\b\\s+(?!Loading\\b).*?\\bName\\b.*?\\bEmail\\b.*?\\bJohn\\b.*?john@example\\.com" + } + ] + }, + "artifact": "Live Editor\n```js\nconst rows = [[\"John\", \"john@example.com\"]];\n```\nResult\nName Email\nJohn john@example.com", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "content_pass": true + } + }, + { + "id": "rendered-result-section-rejects-source-only-values", + "category": "body-scope", + "root_cause": "rendered_result_section_ownership", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/demo", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_anchors": [ + { + "id": "page-section", + "alternatives": ["Result"] + } + ] + }, + "required_patterns": [ + { + "id": "rendered-result", + "pattern": "(?is)\\bResult\\b\\s+(?!Loading\\b).*?\\bName\\b.*?\\bEmail\\b.*?\\bJohn\\b.*?john@example\\.com" + } + ] + }, + "artifact": "Live Editor\nconst rows = [[\"John\", \"john@example.com\"]];\n```js\nconst rows = [[\"John\", \"john@example.com\"]];\n```\nResult\nLoading...", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "content_pass": false + } + }, + { + "id": "localized-equivalent-stable-anchor-passes", + "category": "acceptance-contract", + "root_cause": "equivalent_visible_label_alternatives", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/home", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_anchors": [ + { + "id": "download", + "alternatives": [ + "Download", + "下载" + ] + } + ] + } + }, + "artifact": "下载", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_anchor_hits": 1, + "stable_task_pass": true + } + }, + { + "id": "unrelated-label-does-not-satisfy-equivalent-stable-anchor", + "category": "acceptance-contract", + "root_cause": "equivalent_visible_label_alternatives", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/home", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_anchors": [ + { + "id": "download", + "alternatives": [ + "Download", + "下载" + ] + } + ] + } + }, + "artifact": "Get the application", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_anchor_hits": 0, + "stable_task_pass": false + } + }, + { + "id": "independent-stable-anchors-pass-in-any-document-order", + "category": "acceptance-contract", + "root_cause": "independent_stable_anchor_order", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/topics", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_anchors": [ + { + "id": "education", + "alternatives": [ + "Education" + ] + }, + { + "id": "research", + "alternatives": [ + "Research" + ] + }, + { + "id": "weather", + "alternatives": [ + "Weather" + ] + } + ] + } + }, + "artifact": "Weather\nEducation\nResearch", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_anchor_hits": 3, + "stable_task_pass": true + } + }, + { + "id": "independent-stable-anchors-still-require-every-group", + "category": "acceptance-contract", + "root_cause": "independent_stable_anchor_order", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/topics", + "task_kind": "body", + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_anchors": [ + { + "id": "education", + "alternatives": [ + "Education" + ] + }, + { + "id": "research", + "alternatives": [ + "Research" + ] + }, + { + "id": "weather", + "alternatives": [ + "Weather" + ] + } + ] + } + }, + "artifact": "Education\nResearch", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_anchor_hits": 2, + "stable_task_pass": false + } + }, + { + "id": "complete-body-passes-without-responsive-navigation-labels", + "category": "acceptance-contract", + "root_cause": "task_relevant_visible_text_scope", + "failure_mode": "false-negative", + "case": { + "url": "https://example.test/article", + "task_kind": "body", + "ground_truth_main_content": "Article title alpha bravo charlie delta echo foxtrot golf hotel", + "body_requirements": { + "version": 3, + "unit": "region_compact_hexagram_recall", + "minimum_coverage": 0.8, + "minimum_floor": 0.6, + "minimum_region_passes": 2, + "regions": [ + { + "id": "opening", + "text": "alpha bravo charlie delta" + }, + { + "id": "ending", + "text": "echo foxtrot golf hotel" + } + ] + }, + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_anchors": [ + { + "id": "article-title", + "alternatives": [ + "Article title" + ] + } + ] + } + }, + "artifact": "Article title\nalpha bravo charlie delta\necho foxtrot golf hotel", + "ok": true, + "expected_task_pass": true, + "expected_metrics": { + "stable_task_pass": true, + "body_coverage_pass": true, + "content_pass": true + } + }, + { + "id": "page-title-without-required-body-still-fails", + "category": "acceptance-contract", + "root_cause": "task_relevant_visible_text_scope", + "failure_mode": "false-positive", + "case": { + "url": "https://example.test/article", + "task_kind": "body", + "ground_truth_main_content": "Article title alpha bravo charlie delta echo foxtrot golf hotel", + "body_requirements": { + "version": 3, + "unit": "region_compact_hexagram_recall", + "minimum_coverage": 0.8, + "minimum_floor": 0.6, + "minimum_region_passes": 2, + "regions": [ + { + "id": "opening", + "text": "alpha bravo charlie delta" + }, + { + "id": "ending", + "text": "echo foxtrot golf hotel" + } + ] + }, + "acceptance_contract": { + "version": 1, + "schema": "body_text", + "minimum_records": 0, + "stable_text_anchors": [ + { + "id": "article-title", + "alternatives": [ + "Article title" + ] + } + ] + } + }, + "artifact": "Article title", + "ok": true, + "expected_task_pass": false, + "expected_metrics": { + "stable_task_pass": true, + "body_coverage_pass": false, + "content_pass": false + } + }, + {"id":"forbidden-term-longer-label-is-not-error","category":"format-normalization","root_cause":"forbidden_phrase_substring_collision","failure_mode":"false-negative","case":{"url":"https://example.test/news","task_kind":"body","required_facts":[{"alternatives":["Latest news"]}],"forbidden_any":["Not Found"],"acceptance_contract":{"version":1,"schema":"named_link","minimum_records":1,"hosts":["example.test"],"entry_path":"/story/[^/]+"}},"artifact":"Latest news\n[Story](https://example.test/story/one)\n[Page Not FoundExtra](https://example.test/column)","ok":true,"expected_task_pass":true,"expected_metrics":{"forbidden_hits":[]}}, + {"id":"forbidden-term-complete-phrase-still-fails","category":"format-normalization","root_cause":"forbidden_phrase_substring_collision","failure_mode":"false-positive","case":{"url":"https://example.test/news","task_kind":"body","required_facts":[{"alternatives":["Latest news"]}],"forbidden_any":["Not Found"],"acceptance_contract":{"version":1,"schema":"named_link","minimum_records":1,"hosts":["example.test"],"entry_path":"/story/[^/]+"}},"artifact":"Latest news\n[Story](https://example.test/story/one)\nPage Not Found.","ok":true,"expected_task_pass":false,"expected_metrics":{"forbidden_hits":["Not Found"]}}, + {"id":"forbidden-term-prefixed-word-is-not-error","category":"format-normalization","root_cause":"forbidden_phrase_substring_collision","failure_mode":"false-negative","case":{"url":"https://example.test/news","task_kind":"body","required_facts":[{"alternatives":["Latest news"]}],"forbidden_any":["Not Found"],"acceptance_contract":{"version":1,"schema":"named_link","minimum_records":1,"hosts":["example.test"],"entry_path":"/story/[^/]+"}},"artifact":"Latest news\n[Story](https://example.test/story/one)\nLatestNot Found","ok":true,"expected_task_pass":true,"expected_metrics":{"forbidden_hits":[]}}, + {"id":"named-caption-owned-by-link","category":"structured-extraction","root_cause":"scoped_markdown_caption_ownership","failure_mode":"false-negative","case":{"url":"https://example.test/shows","task_kind":"body","acceptance_contract":{"version":1,"schema":"named_link","minimum_records":1,"hosts":["example.test"],"entry_path":"/series/[^/]+","name_validation":"nonempty","entry_scope":{"html_selector":"main a:has(p)","markdown_label_pattern":"(?s)
\\s*(?P