fix(runner): stream --ids sources and cap selection to the pinned set

[SebTardif]

What Problem This Solves

Fixes an issue where clawscan benchmark --ids could exhaust memory. readBenchmarkIDSource did an unbounded io.ReadAll on HTTP sources (and os.ReadFile on local files). Follow-up to #45, which @steipete closed: a 256 KiB HTTP-only cap would reject the documented full SkillTrustBench ID list (5,520 cases, about 1.3 MiB of canonical JSONL) and left files unbounded.

The first head still kept every parsed id in memory up to the 5,520-count cap. A hostile source can send 5,520 unique records that are each far larger than a real case id. This tip also caps one extracted id at 256 bytes and retained id text at 256 KiB. Those are not JSONL file-size limits, so the documented 5,520-id / 1.3 MiB SkillTrustBench stream still parses.

Why This Change Was Made

Stream file and HTTP --ids sources through the same line parser. Bound the selection by case count (5,520 unique IDs, the pinned SkillTrustBench full set), one-id length (256 bytes), and retained id text (256 KiB). Do not copy the 256 KiB plugin-manifest limit onto the JSONL stream.

User Impact

A hostile or accidental multi-gigabyte --ids URL no longer has to be fully buffered. A valid full-set JSONL list still loads. More than 5,520 unique IDs, a single id longer than 256 bytes, or more than 256 KiB of retained id text is rejected before the benchmark run starts.

Evidence

Terminal output from the public CLI built at 6baa16f. Selection load rejects the new retained-id bounds and still accepts a 299600-byte JSONL file (larger than 256 KiB):

$ /tmp/clawscan-47 benchmark SkillTrustBench --ids /tmp/clawscan-huge-id.txt --scanner clawscan-static --output /tmp/clawscan-ids-out.json
--ids source /tmp/clawscan-huge-id.txt line 1 exceeds the 256-byte benchmark id limit
exit status 1

$ /tmp/clawscan-47 benchmark SkillTrustBench --ids /tmp/clawscan-agg-ids.txt --scanner clawscan-static --output /tmp/clawscan-ids-out.json
--ids source /tmp/clawscan-agg-ids.txt exceeds the 262144-byte retained-id budget
exit status 1

$ /tmp/clawscan-47 benchmark SkillTrustBench --ids /tmp/clawscan-too-many-ids.txt --scanner clawscan-static --output /tmp/clawscan-ids-out.json
--ids source /tmp/clawscan-too-many-ids.txt exceeds the 5520-id SkillTrustBench selection limit
exit status 1

A 400-row JSONL fixture (299600 bytes) loads from a file. The CLI then looks up selected ids in SkillTrustBench, which proves selection finished and the 256 KiB stream cap is not applied:

$ wc -c /tmp/clawscan-large-ids.jsonl
299600 /tmp/clawscan-large-ids.jsonl
$ /tmp/clawscan-47 benchmark SkillTrustBench --ids /tmp/clawscan-large-ids.jsonl --scanner clawscan-static --output /tmp/clawscan-large-out.json
--ids requested benchmark id case_00046, but it is missing from SkillTrustBench split benchmark
exit status 1

The same 400-id selection also loads from HTTP (loaded count=400, matching sha256 835384f6bb8da14b7a7db4cdf6fc1c844331eaa3a05244064f9351b58d69e1bb).

make docs-site on this tree: Built 7 docs page(s) in dist/docs-site. Generated dist/ was not committed.

Real behavior proof

• Behavior or issue addressed: --ids no longer does an unbounded full-body read. One extracted id is capped at 256 bytes and retained id text at 256 KiB. A valid JSONL list larger than 256 KiB still loads.

• Real environment tested: macOS, Go toolchain, branch fix/ids-stream-semantic-bound at 6baa16f, public CLI binary /tmp/clawscan-47 built from that tree.

• Exact steps or command run after this patch:

  python3 -c "from pathlib import Path; Path('/tmp/clawscan-huge-id.txt').write_text('a'*257+'\n')"
  /tmp/clawscan-47 benchmark SkillTrustBench --ids /tmp/clawscan-huge-id.txt --scanner clawscan-static --output /tmp/clawscan-ids-out.json
  /tmp/clawscan-47 benchmark SkillTrustBench --ids /tmp/clawscan-agg-ids.txt --scanner clawscan-static --output /tmp/clawscan-ids-out.json
  /tmp/clawscan-47 benchmark SkillTrustBench --ids /tmp/clawscan-too-many-ids.txt --scanner clawscan-static --output /tmp/clawscan-ids-out.json
  /tmp/clawscan-47 benchmark SkillTrustBench --ids /tmp/clawscan-large-ids.jsonl --scanner clawscan-static --output /tmp/clawscan-large-out.json

• Evidence after fix: terminal output from the public CLI on 6baa16f:

  $ /tmp/clawscan-47 benchmark SkillTrustBench --ids /tmp/clawscan-huge-id.txt --scanner clawscan-static --output /tmp/clawscan-ids-out.json
  --ids source /tmp/clawscan-huge-id.txt line 1 exceeds the 256-byte benchmark id limit
  exit status 1
  $ /tmp/clawscan-47 benchmark SkillTrustBench --ids /tmp/clawscan-agg-ids.txt --scanner clawscan-static --output /tmp/clawscan-ids-out.json
  --ids source /tmp/clawscan-agg-ids.txt exceeds the 262144-byte retained-id budget
  exit status 1
  $ /tmp/clawscan-47 benchmark SkillTrustBench --ids /tmp/clawscan-too-many-ids.txt --scanner clawscan-static --output /tmp/clawscan-ids-out.json
  --ids source /tmp/clawscan-too-many-ids.txt exceeds the 5520-id SkillTrustBench selection limit
  exit status 1
  $ wc -c /tmp/clawscan-large-ids.jsonl
  299600 /tmp/clawscan-large-ids.jsonl
  $ /tmp/clawscan-47 benchmark SkillTrustBench --ids /tmp/clawscan-large-ids.jsonl --scanner clawscan-static --output /tmp/clawscan-large-out.json
  --ids requested benchmark id case_00046, but it is missing from SkillTrustBench split benchmark
  exit status 1

• Observed result after fix: The CLI exits 1 at selection load for a 257-byte id, for 2000 retained ids over the 262144-byte budget, and for 5521 ids. A 299600-byte JSONL source is accepted and the next error is a missing SkillTrustBench row, not a stream-size reject. Hugging Face row fetch is not started on the reject paths.

• What was not tested: A live Hugging Face --ids URL for the full 5,520-row JSONL (about 1.3 MiB). Coverage for that size class is the 400-row file and HTTP fixture (299600 bytes).

Summary

• What changed: stream --ids file and HTTP sources; cap unique IDs at the pinned SkillTrustBench set size; cap one id at 256 bytes and retained id text at 256 KiB.
• Why: unbounded ReadAll/ReadFile can exhaust memory; a 256 KiB HTTP file-size cap rejects valid documented input; a count-only cap still lets 5,520 huge records stay in memory.

Scope

• CLI behavior
• Judge/profile/benchmark behavior

Security / Trust Impact

• Security/trust impact explained

Unbounded remote --ids bodies are no longer fully buffered. Limits are the SkillTrustBench case count, a 256-byte per-id cap, and a 256 KiB retained-id-text budget, documented in docs/benchmarks.md.

Verification

• Focused scanner/benchmark/manual proof: public CLI reject of a 257-byte id, a 262144-byte retained-id budget overflow, and 5521 IDs; public CLI accept of a 299600-byte JSONL source (above)
• Docs site proof (make docs-site) or N/A: Built 7 docs page(s) in dist/docs-site (generated dist/ not committed)

Ref #45

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs changes before merge. Reviewed September 7, 2026, 11:54 AM ET / 15:54 UTC.

ClawSweeper review

What this changes

The PR streams benchmark ID sources from files and HTTP, adds selection count and byte limits, and updates tests, CLI help, and usage guidance.

Merge readiness

⛔ Needs changes before merge - 2 items remain

This PR remains useful: current main and v0.1.7 still buffer entire ID sources. The supplied runtime proof is sufficient; the previously reported documentation omission remains unresolved.

Priority: P2
Reviewed head: 6baa16f5fe70e10ca1f69925ae1fe25d034a2b9a

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused, useful patch with sufficient runtime evidence and one outstanding documentation repair.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): The captured macOS transcript exercises the reviewed loader through the public CLI, shows all three limit rejections, and demonstrates that a 299600-byte JSONL file passes selection loading. The later missing-row error is downstream of the changed behavior; a complete benchmark run is unnecessary for this proof.
Patch quality	🐚 platinum hermit (4/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (terminal): The captured macOS transcript exercises the reviewed loader through the public CLI, shows all three limit rejections, and demonstrates that a 299600-byte JSONL file passes selection loading. The later missing-row error is downstream of the changed behavior; a complete benchmark run is unnecessary for this proof.
Evidence reviewed	8 items	Applicable repository policy: Read the complete root AGENTS.md. No additional ancestor-scoped AGENTS.md or maintainer notes were found for the changed paths. Applied benchmark, public CLI, validation, and generated-artifact guidance; did not run the autoreview helper under the read-only review contract. Still needed on current main and latest release: Current main retains io.ReadAll for HTTP and os.ReadFile for local ID sources. The inspected v0.1.7 tag resolves to the same main commit, so the release does not contain this change. Release identity: The local v0.1.7 tag identifies commit 6432c40, matching the supplied latest-release metadata.
Findings	1 actionable finding	[P3] Document the per-ID and retained-text limits
Security	None	None.

How this fits together

ClawScan's benchmark loader reads selected case IDs before running scanners and an optional judge. It preserves selection order and records selection metadata in the benchmark artifact.

flowchart LR
  A[File or HTTP source] --> B[Stream text or JSONL]
  B --> C[Validate IDs and limits]
  C --> D[Reject invalid selection]
  C --> E[Select benchmark cases]
  E --> F[Run scanners and judge]
  F --> G[Benchmark artifact]

Loading

Before merge

• Document the per-ID and retained-text limits (P3) - The parser now rejects IDs longer than 256 bytes and aggregate ID text above 256 KiB, but this paragraph and the CLI skill describe only the 5,520-ID maximum. Operators cannot determine all acceptance limits from the guidance. Document both byte limits and distinguish them from total streamed source size. This previously reported finding remains unresolved at the same head.
• Complete next step (P2) - Document both byte limits in the benchmark guide and CLI skill, distinguishing retained ID text from total source size.

Findings

• [P3] Document the per-ID and retained-text limits — docs/benchmarks.md:15-17

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production and test LOC	production +24 net; tests +85; guidance +3 net	The production growth supports streaming and selection limits, with three focused regression tests.

Root-cause cluster

Relationship: canonical
Canonical: #47
Summary: This is the open streaming follow-up to a closed, unmerged HTTP-only cap proposal.

Members:

• same_root_cause: fix(runner): cap HTTP --ids response bodies #45 - Both address unbounded buffering of HTTP ID sources; the earlier proposal did not merge.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Technical review

Best possible solution:

Keep the shared streaming loader and publish all enforced selection limits, clearly distinguishing retained ID text from total source size.

Do we have a high-confidence way to reproduce the issue?

Yes, from source: current main buffers an entire file or HTTP body before parsing, allowing input-proportional memory growth. No runtime reproduction or tests were executed during this read-only review.

Is this the best way to solve the issue?

Yes: sharing the streaming parser avoids an incompatible total-JSONL size cap while preserving selection semantics; the remaining repair is documentation.

Full review comments:

• [P3] Document the per-ID and retained-text limits — docs/benchmarks.md:15-17
  The parser now rejects IDs longer than 256 bytes and aggregate ID text above 256 KiB, but this paragraph and the CLI skill describe only the 5,520-ID maximum. Operators cannot determine all acceptance limits from the guidance. Document both byte limits and distinguish them from total streamed source size. This previously reported finding remains unresolved at the same head.
  Confidence: 0.99

Overall correctness: patch is correct
Overall confidence: 0.92

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 6432c40f2340.

Labels

Label justifications:

• P2: This addresses excessive memory use in the optional benchmark ID-loading workflow.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The captured macOS transcript exercises the reviewed loader through the public CLI, shows all three limit rejections, and demonstrates that a 299600-byte JSONL file passes selection loading. The later missing-row error is downstream of the changed behavior; a complete benchmark run is unnecessary for this proof.
• proof: sufficient: Contributor real behavior proof is sufficient. The captured macOS transcript exercises the reviewed loader through the public CLI, shows all three limit rejections, and demonstrates that a 299600-byte JSONL file passes selection loading. The later missing-row error is downstream of the changed behavior; a complete benchmark run is unnecessary for this proof.

Evidence

Acceptance criteria:

• [P1] make docs-site.
• [P1] git diff --check.

What I checked:

• Applicable repository policy: Read the complete root AGENTS.md. No additional ancestor-scoped AGENTS.md or maintainer notes were found for the changed paths. Applied benchmark, public CLI, validation, and generated-artifact guidance; did not run the autoreview helper under the read-only review contract. (AGENTS.md:1, 6baa16f5fe70)
• Still needed on current main and latest release: Current main retains io.ReadAll for HTTP and os.ReadFile for local ID sources. The inspected v0.1.7 tag resolves to the same main commit, so the release does not contain this change. (internal/runner/benchmark.go:343, 6432c40f2340)
• Release identity: The local v0.1.7 tag identifies commit 6432c40, matching the supplied latest-release metadata. (6432c40f2340)
• Introduced implementation and regression coverage: The verified merge-base-to-head diff streams both source types, closes readers, preserves the existing parser and selection hash, and adds three tests covering larger JSONL sources and rejection limits. (internal/runner/benchmark.go:322, 6baa16f5fe70)
• Prior documentation finding remains: The guide and CLI skill still describe only the 5,520-ID limit, while the parser also enforces 256 bytes per ID and 256 KiB of aggregate ID text. Comparison against the previous reviewed SHA is empty; this is the same outstanding finding. (docs/benchmarks.md:15, 6baa16f5fe70)
• After-fix production CLI evidence: The supplied complete PR body, captured under sourceRevision 9e48e20999e1a3a6abb323d7c593a3de215676819949f99081450d4488a23ddf, records a macOS CLI built from the reviewed head rejecting all three limits. A 299600-byte JSONL source reaches the later missing-benchmark-row error, demonstrating successful selection parsing. The body also records HTTP selection success and explicitly discloses that the full live Hugging Face list was not exercised. The contributor's follow-up confirms this refreshed evidence: fix(runner): stream --ids sources and cap selection to the pinned set #47 (comment). (internal/runner/benchmark_registry.go:326, 6baa16f5fe70)

Likely related people:

• Patrick Erichsen: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• SebTardif: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Document the 256-byte per-ID and 256-KiB aggregate ID-text limits in the benchmark guide and CLI skill, distinguishing them from source size.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (51 earlier review cycles; latest 8 shown)

• reviewed 2026-08-30T22:36:10.622Z sha 6baa16f :: needs changes before merge. :: [P3] Document every enforced --ids limit
• reviewed 2026-08-31T05:33:23.076Z sha 6baa16f :: needs changes before merge. :: [P3] Document every enforced --ids limit
• reviewed 2026-08-31T07:58:44.770Z sha 6baa16f :: needs changes before merge. :: [P3] Document the per-ID and retained-text limits
• reviewed 2026-09-02T19:41:19.588Z sha 6baa16f :: needs changes before merge. :: [P3] Document the per-ID and retained-text limits
• reviewed 2026-09-03T15:43:06.156Z sha 6baa16f :: needs changes before merge. :: [P3] Document the per-ID and retained-text limits
• reviewed 2026-09-04T11:53:58.834Z sha 6baa16f :: blocked before merge. :: [P3] Document the per-ID and retained-text limits
• reviewed 2026-09-05T12:54:44.023Z sha 6baa16f :: needs changes before merge. :: [P3] Document the per-ID and retained-text limits
• reviewed 2026-09-06T01:18:24.786Z sha 6baa16f :: needs changes before merge. :: [P3] Document the per-ID and retained-text limits

[SebTardif]

@clawsweeper

[P2] Bound total retained ID bytes
The 5,520-item guard runs after each arbitrary ID has been inserted into both ids and seen; because the scanner permits records up to 1 MiB and IDs only reject whitespace, a hostile HTTP source can still retain several GiB across 5,520 unique records. Add an aggregate byte or ID-length bound that still accepts the documented

Pushed 6baa16f. One extracted id is capped at 256 bytes. Retained id text is capped at 256 KiB. That is not a JSONL file-size limit, so the documented 5,520-id / 1.3 MiB SkillTrustBench stream still parses.

@clawsweeper re-review

[SebTardif]

@clawsweeper

The PR has useful terminal proof at b501d31, but the final retained-text limits were added in 6baa16f and need a redacted current-head runtime transcript.

Refreshed the PR body with public CLI output from 6baa16f: 256-byte id reject, 262144-byte retained-id reject, 5520-id reject, and a 299600-byte JSONL source that loads then fails on a missing SkillTrustBench row. Also ran make docs-site (Built 7 docs page(s) in dist/docs-site); generated dist/ was not committed.

@clawsweeper re-review