Skip to content

context.md

Latest commit

 

History

History
239 lines (184 loc) · 9.57 KB

context.md

File metadata and controls

239 lines (184 loc) · 9.57 KB
Raw

Source-verifiable AI context

Symgliph can reduce model input tokens, but the visual glyph is an identity and navigation surface—not a secret alternate language a pretrained model can decode. The token-saving component is a manifest-backed retrieval tool:

question -> lexical seeds -> semantic neighbors -> verified source -> token pack

The model receives ordinary text, so no model training or custom tokenizer is required. The compact artifact stays outside the prompt and controls which exact source ranges are materialized.

Contract

For request q, budget B, manifest M, and current source tree X, ContextEngine::pack returns packet P with these checked properties:

  1. tokens_o200k(P.prompt) <= B;
  2. every evidence path belongs to M;
  3. each complete source file still matches its BLAKE3 digest in M;
  4. each excerpt digest commits to the exact bytes included in P.prompt;
  5. byte ranges are half-open and remain on UTF-8 boundaries; and
  6. omitted or shortened evidence sets P.truncated = true.

Source blocks are labelled as untrusted data. Provenance protects integrity; it does not make repository text safe as model instructions.

Adaptive compilation

ContextEngine::pack_adaptive runs the ordinary verified packer across an ascending token and semantic-node ladder. Each round records its ceilings, actual prompt tokens, new evidence count, exact-context BLAKE3 digest, and a deterministic evidence-sufficiency estimate.

The reference estimator checks query-term coverage, minimum evidence count, and—when requested—whether all included excerpts are complete. It stops with one of three explicit outcomes:

  • estimated_sufficient: the declared heuristic passed;
  • candidate_exhausted: all candidates fit but the heuristic failed; or
  • hard_limit_reached: omitted material remained at both hard ceilings.

This loop reduces needless fixed-budget padding and makes expansion behavior auditable. It does not observe model correctness, so estimator success is not quality verification. The experimental contract is SACP-0.2; ECEB remains the independent quality and noninferiority layer.

cargo run -- adaptive-pack . \
  --query "where is retry policy enforced?" \
  --min-tokens 1000 \
  --max-tokens 8000 \
  --require-sufficient

Measured proof

On 2026-08-01, this repository was compiled and queried with:

scripts/context-proof.sh . "How does CorpusBuilder build and verify the semantic graph?"

The fail-closed gate produced:

Measurement Result
Full non-binary corpus 63,941 tokens
Retrieved prompt budget 4,000 tokens
Retrieved prompt actual 3,976 tokens
Verified excerpts 11
Tokens excluded 59,965
Packed/full ratio 6.21%

The top evidence included compile_graph, CorpusBuilder::build, semantic_root, and a graph-expanded resolve_references excerpt. The gate requires at least one evidence item, exact budget compliance, and a packed/full ratio no greater than 25% by default.

This is a measured reduction of input text for one repository and query, not a guarantee of equal answer quality or a universal cost percentage. Total cost is approximately:

initial packed tokens + tool protocol overhead + follow-up expansion tokens

Broad questions may need more expansions. Exact whole-corpus transformations, global audits, and questions whose relevant semantics are absent from the analyzers may not benefit and should fall back to wider retrieval.

Multi-model A/B result

The provider-backed proof runner executes three fixed code questions against full and packed context with the same system instructions and output limit. It alternates which answer is labelled A, then asks an independent judge to score both against five-item ground-truth rubrics. A pair passes when packed input is smaller and its score is no more than one point below the full-context score. The aggregate must also reduce billed answer-call cost.

On 2026-08-01, nine comparisons across Claude Haiku 4.5, Gemini 2.5 Flash, and GPT-4.1 Mini produced:

Measurement Full context Packed context
Native provider prompt tokens 629,204 63,386
Billed answer-call credits 0.356578 0.045534
Mean blinded rubric score 10.00/10 9.44/10

That run removed 565,818 native prompt tokens (89.93%) and reduced aggregate answer-call cost by 87.23%. All nine token and quality assertions passed. One individual Gemini call cost slightly more because completion behavior can outweigh input savings on a single short request; this is why the report keeps input tokens and total billed cost as separate measurements.

The JSON report contains complete answers, judgments, native usage, actual routed model IDs, latencies, and evidence node IDs. Reproduce it with:

cargo run --features openrouter --bin symgliph-openrouter-proof -- . \
  --env-file /path/to/openrouter.env

The credential parser reads only OPENROUTER_API_KEY and OPENROUTER_BASE_URL; neither value is written to the report.

External Fabric-pattern result

To test beyond Symgliph's own source, the same A/B framework was run against Daniel Miessler's Fabric prompt library at commit befdfeefb2402db706f4b54165b8a47ef2cedbad. The runner selected 24 of 237 patterns by SHA-256 ordering of pattern names. Six patterns were then selected by an independent BLAKE3 ordering for questions, producing 18 comparisons over the three answer models.

Each answerer was asked to recover a named pattern's identity, steps, exact output format, input expectations, and critical constraints. Only the blinded judge received the complete gold system.md. This first experiment isolates verified instruction recovery; the blind-discovery experiment below tests choosing a pattern without receiving its name.

Measurement Full 24-pattern context Packed context
Native provider prompt tokens 591,924 53,274
Billed answer-call credits 0.302911 0.048925
Mean blinded rubric score 9.17/10 9.56/10

Packed context removed 538,650 native prompt tokens (91.00%) and reduced aggregate answer-call cost by 83.85%. All 18 token, completion, cost, and per-pair quality assertions passed.

The first run exposed a genuine failure on Fabric's 51 KB write_micro_essay pattern: a whole-document excerpt consumed the budget before late output constraints. The final packer reserves budget across at least four evidence nodes and skips a truncated Markdown document in favor of its heading-scoped sections. A regression test now asserts that late Markdown constraints survive packing.

Reproduce the result with:

scripts/fabric-proof.sh /path/to/Fabric /path/to/openrouter.env

Blind semantic discovery result

The stronger experiment supplies only six natural-language user goals from benchmarks/fabric-discovery.json. It rejects any case whose normalized goal contains its hidden pattern directory name. The answer model never receives the gold label or gold document through the question or judge reference.

The discovery pipeline is:

unnamed goal
  -> one batched embedding request over 24 patterns and six goals
  -> cosine-ranked top-three pattern names
  -> verified lexical/graph context packing
  -> answer with exact selected directory

The embedding documents contain each directory name plus a bounded head/tail excerpt of its system.md. The candidate names are retrieval hints, not source evidence: the normal packer must still resolve them to digest-verified source ranges under its hard token budget. Embedding tokens and cost are charged once to the packed pipeline. Four independent gates prevent a soft score from hiding discovery failure:

  1. the hidden target must occur in the semantic top three;
  2. verified evidence from that target must occur in the packet;
  3. every packed answer must contain the exact target directory name; and
  4. every comparison must retain token reduction, finish normally, and stay within the one-point noninferiority margin, while aggregate cost falls.

On 2026-08-01, the same Fabric commit and 24-pattern subset produced 18 blind comparisons across Claude Haiku 4.5, Gemini 2.5 Flash, and GPT-4.1 Mini:

Measurement Full 24-pattern context Packed discovery pipeline
Native prompt/input tokens 594,113 84,035
Chat prompt tokens only 594,113 67,614
One-time embedding tokens 0 16,421
Billed credits 0.351108 0.054952
Mean blinded rubric score 9.33/10 9.72/10

All six targets ranked first semantically, all 18 packets retrieved the hidden target, and all 18 packed answers selected its exact name. End-to-end prompt input fell by 510,078 tokens (85.86%) and cost fell by 84.35%, including the embedding pass. These findings prove the pipeline on this fixed subset and case set; they do not establish general retrieval accuracy over all Fabric patterns or arbitrary goals.

Reproduce the result with:

scripts/fabric-discovery-proof.sh /path/to/Fabric /path/to/openrouter.env

JSONL tool protocol

Start the local server:

cargo run -- serve ./my-corpus

Send one JSON-RPC object per line. Supported methods:

  • context.describe: corpus roots, counts, tokenizer, and method discovery;
  • context.pack: verified evidence under max_tokens and max_nodes; and
  • context.proof: the packet plus its full-corpus token comparison.

Application failures are returned as JSON-RPC errors and do not terminate the stream. A malformed line also receives an error, allowing a long-lived model tool process to continue serving later requests.