Add Shared-Memory Governance Benchmark (SMGB): dataset + evaluation

[mrrahman1517]

What & why

Single-user memory benchmarks (LongMemEval, LoCoMo, DMR — the ones in PR #31) only score recall. They can't tell a governed shared-memory system from a naive one. This PR adds the Shared-Memory Governance Benchmark (SMGB): a public, system-agnostic benchmark for the governance of shared agent memory.

The one-table argument

Reference runners on the seed set (python -m benchmarks.governance.run_governance --reference):

system        queries mean_recall leak_rate total_leaks isolation_violations stale_leaks
-----------------------------------------------------------------------------------------
oracle        14      1.000       0.000     0           0                    0
naive_shared  14      1.000       0.500     13          0                    1
naive_global  14      1.000       0.643     15          2                    1

All three baselines score mean_recall = 1.000 — a recall-only benchmark would rate them identical. Only the leakage / isolation / stale-propagation axes reveal that naive_shared leaks 13 facts and naive_global also breaches tenant isolation. Recall alone cannot distinguish governed memory from naive; SMGB can.

Design highlights

• No LLM judge. Ground truth is derived deterministically from a scope + policy graph, so scoring is reproducible and free. (PiSAs 2607.05318 is LLM-judged; ArgusFleet 2606.24535 is proprietary — SMGB removes both blockers.)
• System-agnostic. A "run" is just {query_id: [ranked memory_ids]}. Any memory system can be scored; no SDK or network needed.
• Self-checking data. The loader validates every hand annotation against the policy-derived labels and rejects mismatches, so dataset authoring bugs can't slip in.
• Time is first-class. Every query has an as_of; promotion / supersession / deletion are timeline events. The same question before vs after a promotion has different correct answers.

Seven axes

utility, leakage, isolation, promotion, conflict (supersession), deletion, provenance.

What's included

• benchmarks/governance/ — schema.py, policy.py (deterministic core), scorer.py, loader.py, run_governance.py (CLI), package README.md.
• benchmarks/governance/data/seed_scenarios.jsonl — 6 scenarios / 14 queries covering all axes (incl. the two private->shared promotion cases from the whiteboard), reproducible via _generate_seed.py.
• benchmarks/tests/test_governance_{policy,scorer,seed}.py — 16 tests.
• Docs/shared_memory_governance_benchmark.md — formal design + verified gap analysis, for team circulation.

Verification

• python -m pytest benchmarks/tests -q -> 16 governance tests pass
• ruff check benchmarks/governance -> clean
• CLI --reference, --validate-only, --json-out all verified

Scope

Self-contained and based directly on main (introduces the benchmarks/ package scaffold). A companion multi-agent consistency harness — durability/staleness of shared writes under concurrency, per Golab et al. — is tracked separately and can land later; this PR does not depend on it.

Open questions for review

Scenario coverage priorities, scope taxonomy, whether to ship the live Cosmos adapter in v1, provenance-chain depth, and public naming — see the "Open questions" section of the design doc.

[Copilot]

Pull request overview

Adds the Shared-Memory Governance Benchmark (SMGB) to evaluate shared-memory governance (authorization, isolation, promotion, supersession, deletion, provenance) in a deterministic, system-agnostic way, complementing the existing single-user recall-focused benchmarks.

Changes:

• Introduces a new benchmarks.governance package: schema + deterministic policy + scorer + dataset loader + CLI runner.
• Checks in a seed JSONL dataset (6 scenarios / 14 queries) plus a reproducible generator script.
• Adds unit/integrity tests for policy labeling, scorer discrimination, and seed dataset validity; documents the benchmark design in Docs/.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
Docs/shared_memory_governance_benchmark.md	Design + gap analysis write-up for SMGB.
Docs/README.md	Adds SMGB design doc to the documentation index.
benchmarks/init.py	Adds top-level benchmarks package marker.
benchmarks/governance/init.py	Exposes SMGB public API surface.
benchmarks/governance/README.md	Usage docs, dataset format, and quickstart commands.
benchmarks/governance/schema.py	Dataclass schema + time parsing + axis/event enums.
benchmarks/governance/policy.py	Deterministic authorization/label derivation logic.
benchmarks/governance/scorer.py	Scoring + reporting + reference runners.
benchmarks/governance/loader.py	JSONL loader + referential integrity + hand-label validation.
benchmarks/governance/run_governance.py	CLI entrypoint for validation + scoring + JSON output.
benchmarks/governance/data/seed_scenarios.jsonl	Seed dataset (scenarios/queries).
benchmarks/governance/data/_generate_seed.py	Reproducible seed dataset generator script.
benchmarks/tests/init.py	Test package marker.
benchmarks/tests/test_governance_policy.py	Unit tests for deterministic policy labeling.
benchmarks/tests/test_governance_scorer.py	Unit tests for scorer + reference runners.
benchmarks/tests/test_governance_seed.py	Integrity tests for checked-in seed dataset.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.