Add ECEB enterprise context benchmark

[copyleftdev]

What changed

• defines the ECEB-0.1 quality-constrained enterprise context benchmark
• adds benchmark, run, and report JSON Schemas
• adds a Rust evaluator and evaluate-eceb CLI command
• adds a synthetic cross-source enterprise seed spanning code, RFC, runbook, PR history, and restricted policy data
• adds deterministic expected metrics and negative gates for policy leakage, stale evidence, token overflow, and nondeterministic context
• exposes the benchmark through repository documentation and the Pages artifact

Why

Symgliph needs to test the minimum verified, policy-permitted context that preserves answer quality—not merely whether a model can answer with an unrestricted corpus. ECEB separates research claims from conformance fixtures and makes policy violations hard failures.

Impact

Implementations can now submit repeatable run records and receive per-task token, cost, quality, provenance, policy, and repeatability vectors. The checked-in seed is explicitly synthetic and does not claim enterprise-scale model performance.

Validation

• cargo fmt --all --check
• cargo clippy --all-features --all-targets -- -D warnings
• cargo test --all-features --all-targets --locked
• spec/validate.sh
• scripts/eceb-conformance.sh
• scripts/site-check.sh
• npx --yes html-validate site/index.html site/404.html
• cargo doc --all-features --no-deps --locked
• cargo package --locked

Summary by CodeRabbit

• New Features

  • Added the ECEB-0.1 enterprise efficiency benchmark, including specifications, schemas, sample datasets, and evaluation reports.
  • Added a command-line workflow for evaluating benchmark runs and generating reports.
  • Added validation for quality, provenance, policy compliance, repeatability, token usage, and cost metrics.
  • Published the benchmark specification and expanded website documentation.

• Documentation

  • Added usage instructions, benchmark guidance, authentication scenarios, and claim limitations.

• Tests

  • Added automated conformance checks covering invalid evidence, stale data, budget overflows, and nondeterministic results.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e9cc8e18-ff03-4748-a792-75b9e136ab76

📥 Commits

Reviewing files that changed from the base of the PR and between 4469f2d and e08e33e.

📒 Files selected for processing (25)

• .github/workflows/ci.yml
• .github/workflows/pages.yml
• CHANGELOG.md
• README.md
• datasets/eceb-v0.1/README.md
• datasets/eceb-v0.1/benchmark.json
• datasets/eceb-v0.1/corpus/code/auth.rs
• datasets/eceb-v0.1/corpus/history/pr-1842.md
• datasets/eceb-v0.1/corpus/hr/payroll-token-rotation.md
• datasets/eceb-v0.1/corpus/ops/auth-service-runbook.md
• datasets/eceb-v0.1/corpus/rfc/0007-jwt-retry-policy.md
• datasets/eceb-v0.1/expected-report.json
• datasets/eceb-v0.1/runs/conformance-pass.json
• scripts/eceb-conformance.sh
• scripts/site-check.sh
• site/index.html
• site/styles.css
• spec/README.md
• spec/eceb-v0.1.md
• spec/schema/eceb-benchmark-v0.1.schema.json
• spec/schema/eceb-report-v0.1.schema.json
• spec/schema/eceb-run-v0.1.schema.json
• src/eceb.rs
• src/lib.rs
• src/main.rs

---

📝 Walkthrough

Walkthrough

Added ECEB-0.1 schemas, specification, evaluator, CLI, synthetic authentication benchmark, conformance fixtures, negative validation gates, CI integration, and published documentation.

Changes

ECEB benchmark and conformance

Layer / File(s)	Summary
ECEB specification and data contracts spec/eceb-v0.1.md, spec/schema/*, datasets/eceb-v0.1/benchmark.json, spec/README.md	Defines benchmark, run, and report structures, corpus provenance, task policies, strategies, quality limits, budgets, and report constraints.
Evaluator and public API src/eceb.rs, src/lib.rs	Implements corpus verification, structural validation, budget selection, provenance and policy gates, repeatability checks, metrics, reports, tests, and public re-exports.
Enterprise conformance seed and fixtures datasets/eceb-v0.1/corpus/*, datasets/eceb-v0.1/runs/*, datasets/eceb-v0.1/expected-report.json, datasets/eceb-v0.1/README.md	Adds the synthetic authentication corpus, repeated observations, expected report, and benchmark documentation.
CLI execution and conformance gates src/main.rs, scripts/eceb-conformance.sh, .github/workflows/ci.yml	Adds the ECEB evaluation command and validates successful and intentionally invalid benchmark runs.
Documentation and site publication README.md, CHANGELOG.md, site/*, spec/README.md, .github/workflows/pages.yml, scripts/site-check.sh	Documents ECEB, publishes the specification, adds site links and styling, and checks the published specification link.

Estimated code review effort: 5 (Critical) | ~90 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant Corpus
  participant Evaluator
  participant Report
  CLI->>Corpus: verify benchmark corpus
  CLI->>Evaluator: evaluate benchmark and run
  Evaluator->>Evaluator: validate evidence, policy, quality, and repetition gates
  Evaluator->>Report: build ECEB report
  Report-->>CLI: write report or return exit status

Loading

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/eceb-v0-1

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}