Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ jobs:
run: scripts/eceb-conformance.sh
- name: Replay public ECEB study
run: scripts/eceb-study-check.sh
- name: Replay adaptive ECEB study
run: scripts/eceb-adaptive-study-check.sh
- name: Build documentation
run: cargo doc --all-features --no-deps --locked
- name: Package
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/pages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ jobs:
cp -a site/. _site/
cp spec/symgliph-context-glyph-v0.1.md _site/spec/SCGP-0.1.md
cp spec/eceb-v0.1.md _site/spec/ECEB-0.1.md
cp spec/adaptive-context-v0.2.md _site/spec/SACP-0.2.md
- name: Configure Pages
uses: actions/configure-pages@v5
- name: Upload Pages artifact
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ the project intends to use Semantic Versioning after its first stable release.
conformance seed, executable evaluator, and negative policy/provenance gates.
- Deterministic 30-task Multi-SWE-bench Rust study suite, four-arm runner,
comparative study schemas, raw golden run records, and offline replay gate.
- Experimental SACP-0.2 adaptive context profile, deterministic expansion
engine, CLI and JSON-RPC entry points, auditable stopping trace, schema, and
optional Multi-SWE-bench study arm. The replay evaluator binds every trace to
its declared ladder and sufficiency policy and rejects premature limit claims.

### Changed

Expand All @@ -34,5 +38,12 @@ the project intends to use Semantic Versioning after its first stable release.
- The Multi-SWE-bench Rust v0.1 study reduced paired prompt tokens by 97.58% and
localization-model spend by 98.54% with Symgliph compiled context, while exact
fix-path recall declined from 53.32% to 41.76%; lexical BM25 retained 46.87%.
- A retrieval-only adaptive threshold sweep selected 50% query-term coverage as
the lowest tested setting that retained the fixed compiled arm's 51.85%
evidence recall. This is in-sample calibration, not model-quality evidence.
- The adaptive 30-task follow-up retained 51.85% evidence recall, reduced mean
compiled prompt tokens from 5,138 to 4,345, improved exact fix-path recall
from 41.76% to 45.09%, and reduced recorded localization spend from $0.0668
to $0.0573. It remains below BM25 and full-context recall.

[Unreleased]: https://github.com/copyleftdev/symgliph/commits/main
55 changes: 53 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,39 @@ this run, but it did **not** preserve quality parity: exact fix-path recall was
lexical BM25. The study supports the efficiency hypothesis and identifies
retrieval quality as the next research problem.

## Adaptive context experiment

The experimental SACP-0.2 profile addresses that gap by walking an ascending
token and semantic-node ladder. It stops at the first packet that passes a
declared evidence-sufficiency heuristic, when candidates are exhausted, or when
hard limits are reached. Every attempted context digest and stopping condition
is retained for evaluation.

```text
query -> retrieve -> pack -> estimate sufficiency
├── pass: emit packet
└── fail: expand budget and evidence
```

The built-in estimator measures query-term coverage, evidence count, and
optional excerpt completeness. It is deliberately named `estimated_sufficient`:
passing it does not establish retrieval recall or answer quality. A follow-up
Multi-SWE-bench run tested the policy against full context, BM25, and the fixed
compiled arm.

The provisional 50% query-term threshold was selected by an exploratory replay
of the existing 30-task traces. It preserved the fixed compiled arm's evidence
recall while reducing selected context, but it is not independent validation.
See the [calibration record](docs/adaptive-calibration.md).

In that follow-up, adaptive context retained the fixed arm's 51.85% evidence
recall while reducing mean prompt tokens from 5,138 to 4,345 and improving
exact fix-path recall from 41.76% to 45.09%. It still trailed BM25 and full
context. Read the [adaptive study and claim boundary](docs/eceb-adaptive-study.md).

Read the experimental
[`Adaptive Context Compilation 0.2`](spec/adaptive-context-v0.2.md) profile.

Read the [methodology and claim boundary](docs/eceb-multiswe-study.md), inspect
the [raw report](benchmarks/results/eceb-multiswe-rust-v0.1/report.json), or
recompute it without API access:
Expand All @@ -90,6 +123,13 @@ cargo run -- pack ./my-corpus \
--query "where is authorization checked?" \
--max-tokens 4000

# Start small and expand until the declared evidence estimator passes.
cargo run -- adaptive-pack ./my-corpus \
--query "where is authorization checked?" \
--min-tokens 1000 \
--max-tokens 8000 \
--require-sufficient

# Recompute the corpus and semantic roots against current source.
cargo run -- verify ./my-corpus
```
Expand Down Expand Up @@ -126,9 +166,13 @@ Every evidence record identifies its semantic node, source path, byte range,
full-file digest, and excerpt digest. If a selected file changed after
compilation, packing fails rather than emitting stale evidence.

`ContextEngine::pack_adaptive` accepts an `AdaptiveContextRequest` and returns
the selected packet plus the complete expansion trace. A failed estimate still
returns a valid verified packet unless the caller elects fail-closed behavior.

For local model and agent bridges, `symgliph serve` exposes
`context.describe`, `context.pack`, and `context.proof` over newline-delimited
JSON-RPC:
`context.describe`, `context.pack`, `context.adaptive`, and `context.proof` over
newline-delimited JSON-RPC:

```json
{"jsonrpc":"2.0","id":1,"method":"context.pack","params":{"query":"authorization checks","max_tokens":2000,"max_nodes":8}}
Expand Down Expand Up @@ -160,6 +204,7 @@ Run the independent gates:
spec/validate.sh # SCGP artifact and packet conformance
scripts/eceb-conformance.sh # ECEB policy, provenance, and budget gates
scripts/eceb-study-check.sh # replay the public comparative study
scripts/eceb-adaptive-study-check.sh # replay the adaptive follow-up
```

## Artifact model
Expand Down Expand Up @@ -202,6 +247,12 @@ project code or build scripts. Additional analyzers can implement the
A/B runs, Fabric instruction recovery, and blind semantic discovery.
- [ECEB-0.1](spec/eceb-v0.1.md): a quality-, policy-, provenance-, and
budget-aware measurement protocol.
- [SACP-0.2](spec/adaptive-context-v0.2.md): deterministic expansion, explicit
stopping semantics, and auditable evidence-sufficiency estimates.
- [Adaptive calibration](docs/adaptive-calibration.md): threshold sweep,
negative findings, and the preregistered next-run setting.
- [Adaptive ECEB study](docs/eceb-adaptive-study.md): the prospective model
run, complete traces, measured improvement, and remaining quality gap.
- [Multi-SWE-bench Rust study](docs/eceb-multiswe-study.md): the current
four-arm public result and raw observations.
- [Semantic-routing golden set](docs/golden-set.md): linked corpora, queries,
Expand Down
Loading