[KAVentures]

Actual gap

Issue #226 has an offline scorer but no reproducible external-agent study runner. The previous benchmark missed the entire point when C did not call OWG.

Implementation

• Add scripts/run_issue226_agent_study.py: launch an explicit operator-supplied agent executable (no shell) for shuffled, matched A/B/C case/replicate trials; isolate a fresh temporary directory and loopback HTTP evidence broker per trial.
• A cannot read source or OWG. B and C can request the exact independently reviewed source snapshot; C alone can request matching OWG records, optionally backed by the real locally authenticated /v1/record-graph API. Wrong IDs/arms/credentials cannot produce successful tool telemetry.
• Broker records actual HTTP invocations and SHA-256 source digests, never trusts agent-self-reported tool logs, and omits stdout/stderr/raw source data from reports. This is an executable agent harness, not a simulated agent performance claim.
• Fix deterministic scorer: cases with resource_ref=null must correctly abstain from B/C source reads rather than being unfairly marked as a confound; preserve fail-closed requirements elsewhere.
• Add substantive HTTP broker, subprocess adapter, access isolation, explicit source outage and no-reference regression tests, plus detailed private operator instructions.

Claims and remaining manual work

No model/provider calls are made in CI; actual AI CLI adapter, privacy-reviewed held-out cases, source snapshots, independent reviewer, and signed desktop/provider auth acceptance still must be provided and executed. Local OWG HTTP endpoint is not a native MCP plugin invocation, so native plugin smoke needs separate testing. This PR touches only scripts/, tests/ and a separate documentation file and has zero file overlap with PR #240. Keep draft until complete CI and manual real-agent integration test; do not close #226 solely on these tests.