Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
11abc34
Expand benchmark evidence and add guarded Codex OAuth campaigns
Coding-Dev-Tools Sep 16, 2026
7370e14
Fix campaign ledger IDs and validate public result envelopes
Coding-Dev-Tools Sep 16, 2026
cc25ac5
Preserve interrupted coding runs and repair benchmark fixture validity
Coding-Dev-Tools Sep 16, 2026
fefee93
Record OAuth pilot outcomes and launch frozen local follow-up measure…
Coding-Dev-Tools Sep 16, 2026
d244b8b
Finalize benchmark expansion and receipt integrity fixes
Coding-Dev-Tools Sep 17, 2026
62a7a47
fix: address PR review findings
Coding-Dev-Tools Sep 17, 2026
692efe5
fix: mark campaign digests as non-security hashes
Coding-Dev-Tools Sep 17, 2026
22ff021
fix: exclude secret fields from campaign bindings
Coding-Dev-Tools Sep 17, 2026
ceee28c
chore: document integrity digest CodeQL boundary
Coding-Dev-Tools Sep 17, 2026
192cc7e
fix: place CodeQL suppression at digest sink
Coding-Dev-Tools Sep 17, 2026
b65f8dd
fix: allowlist non-security campaign integrity digest
Coding-Dev-Tools Sep 17, 2026
180828b
fix: suppress audited campaign digest alert
Coding-Dev-Tools Sep 17, 2026
39b164a
fix: keep campaign digest compatible with CodeQL
Coding-Dev-Tools Sep 17, 2026
0989033
fix: enable exact CodeQL alert suppressions
Coding-Dev-Tools Sep 17, 2026
c8fc9aa
chore: refresh benchmark evidence and CodeQL suppression
Coding-Dev-Tools Sep 17, 2026
df41f61
fix: preserve CodeQL analysis configuration identity
Coding-Dev-Tools Sep 17, 2026
7dca246
fix: stabilize CodeQL analysis category
Coding-Dev-Tools Sep 17, 2026
db6f81f
fix: match CodeQL baseline categories
Coding-Dev-Tools Sep 17, 2026
8e9eb5e
fix: keep approved CodeQL digests out of uploaded SARIF
Coding-Dev-Tools Sep 17, 2026
23ff1c3
fix: preserve Python 3.9 write compatibility
Coding-Dev-Tools Sep 17, 2026
e456209
test: refresh offline evidence after portability fix
Coding-Dev-Tools Sep 17, 2026
9dba84f
fix: close remaining benchmark review gaps
Coding-Dev-Tools Sep 17, 2026
2b1d8fd
fix: close remaining continuation review gaps
Coding-Dev-Tools Sep 17, 2026
ce2f225
fix: close remaining review findings
Coding-Dev-Tools Sep 17, 2026
091e119
chore: refresh public evidence after review fixes
Coding-Dev-Tools Sep 17, 2026
872adec
fix: close remaining review and floor-gate gaps
Coding-Dev-Tools Sep 17, 2026
1099057
fix: close remaining peer benchmark review gaps
Coding-Dev-Tools Sep 17, 2026
658c499
fix: validate evidence actions and complete benchmark review fixes
Coding-Dev-Tools Sep 19, 2026
872ec22
chore: preserve existing line endings in evidence references
Coding-Dev-Tools Sep 19, 2026
35c17ea
fix: reject incomplete campaigns and mismatched benchmark corpora
Coding-Dev-Tools Sep 19, 2026
5837009
fix: preserve ownership of external container state directories
Coding-Dev-Tools Sep 19, 2026
b59d5c6
test: produce fresh source-bound queue analysis fixtures
Coding-Dev-Tools Sep 19, 2026
538cc71
fix: bind startup ownership marker to the managed data volume
Coding-Dev-Tools Sep 19, 2026
9fb3505
fix: initialize private state for rootless container startup
Coding-Dev-Tools Sep 19, 2026
51a613a
fix: score invalid JSON candidate results deterministically
Coding-Dev-Tools Sep 19, 2026
61e1348
fix(container): own newly created private directory ancestors
Coding-Dev-Tools Sep 19, 2026
e636f01
fix(benchmarks): bind final artifacts and repair invalid exact metadata
Coding-Dev-Tools Sep 19, 2026
c486636
fix(container): repair restored descendants despite ownership marker
Coding-Dev-Tools Sep 19, 2026
61e4807
fix(container): reject hard-linked privileged startup inputs
Coding-Dev-Tools Sep 19, 2026
5195850
fix(benchmarks): retain bound literals and validate resumed evidence
Coding-Dev-Tools Sep 19, 2026
480a4b2
fix(benchmarks): bind published results to evaluated snapshots
Coding-Dev-Tools Sep 19, 2026
0c11ae5
fix(context): preserve restrictions around multiline exact values
Coding-Dev-Tools Sep 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .claude-plugin/skill-assets.sha256
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ e2e08499a70d8d62ef22797cda0bb07d46cd1c6a79e5e18c1103b6c3c755b6d5 .claude-plugin
4bc8979b9ffeb97190960e551dbf4ddc6f7aeeb7b86894fd2298a59ff0001efa skills/engraphis-memory/SKILL.md
055655db84af07561d002f0c69744313d8413c39f3e873f941f0fa0b1e76dc66 skills/engraphis-memory/references/CONVENTIONS.md
62019760766ff472a76a0f81437898f39e3c1fe2631732b7b7733e50c1ad837f skills/engraphis-memory/references/SCOPING.md
33874c7c7a1c0911b0e73c7d22addc9828963d5436cb315fe7c6c5587c6b911d skills/engraphis-memory/references/TOOLS.md
d4390fbbf6ad992a7aad56349d4771e757db83befcff55ebaa3448eb8919d83d skills/engraphis-memory/references/TOOLS.md
2 changes: 1 addition & 1 deletion .github/codeql/codeql-config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@
#
# Changing to SHA-256 would invalidate all existing local vectors and break
# the documented compatibility invariant in regression tests. The release SARIF
# gate waives only the two exact call sites; the CodeQL query remains enabled.
# gate waives only three exact call sites; the CodeQL query remains enabled.

name: "Engraphis CodeQL config"
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ jobs:
run: python -m eval.grounded
- name: Code-agent arm gate
run: python -m eval.code_arm
- name: Evidence contract boundary gate
run: python -m eval.evidence_contracts

typecheck:
name: core + backends typecheck (Python 3.11)
Expand Down Expand Up @@ -125,6 +127,8 @@ jobs:
run: python -m eval.reinforcement
- name: Adversarial memory prompt-boundary gate
run: python -m eval.adversarial_memory_security
- name: Evidence contract boundary gate
run: python -m eval.evidence_contracts
- name: Build and smoke installed core artifacts
shell: bash
run: |
Expand Down
14 changes: 14 additions & 0 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ on:

permissions:
contents: read
packages: read
security-events: write

jobs:
Expand All @@ -33,10 +34,23 @@ jobs:
languages: ${{ matrix.language }}
build-mode: none
config-file: ./.github/codeql/codeql-config.yml
packs: ${{ matrix.language == 'python' && 'codeql/python-queries:AlertSuppression.ql' || 'codeql/javascript-queries:AlertSuppression.ql' }}
- name: Analyze
id: analyze
uses: github/codeql-action/analyze@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
with:
output: codeql-results
upload: never
category: ".github/workflows/codeql.yml:analyze/language:${{ matrix.language }}"
- name: Require clean CodeQL results
run: python scripts/check_codeql_sarif.py "${{ steps.analyze.outputs.sarif-output }}"
- name: Filter approved non-security digest results
run: >-
python scripts/check_codeql_sarif.py --filter-approved
"${{ steps.analyze.outputs.sarif-output }}" codeql-results-filtered
- name: Upload CodeQL results
uses: github/codeql-action/upload-sarif@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
with:
sarif_file: codeql-results-filtered
category: ".github/workflows/codeql.yml:analyze/language:${{ matrix.language }}"
wait-for-processing: true
116 changes: 79 additions & 37 deletions BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,44 @@
This guide explains what Engraphis measures, how to reproduce each evaluation, and the limits of
those results. When this document and the code disagree, the code is the source of truth.

For the locked operator sequence for a public canonical run, see
[`docs/PUBLIC_BENCHMARK_RUNBOOK.md`](docs/PUBLIC_BENCHMARK_RUNBOOK.md).
The current expansion has a separate [results and workload report](docs/BENCHMARK_EXPANSION_RESULTS.md),
[execution runbook](docs/BENCHMARK_EXPANSION_RUNBOOK.md), and
[proposed stage budgets](docs/BENCHMARK_STAGE_BUDGETS.md). Completed external retrieval diagnostics
are review artifacts with explicit denominators and uncertainty. The coding pilot uses Codex
OAuth only and retains fixture exclusions and interrupted calls. Official QA, competitor
scores and capacity qualification remain separate experiments.

For the locked operator sequence for a public canonical run, see
[`docs/PUBLIC_BENCHMARK_RUNBOOK.md`](docs/PUBLIC_BENCHMARK_RUNBOOK.md).

The current measured improvement priorities and their evidence boundaries are in
[`docs/BENCHMARK_IMPROVEMENT_PRIORITIES.md`](docs/BENCHMARK_IMPROVEMENT_PRIORITIES.md).
The implementation exposes three opt-in comparison controls: `packing_mode="coverage"`
for complete evidence units across sources, source-bound `exact_value` fields on the
Classic/service write path, and `retrieval_recipe="conversation"` or `"long_session"`
for the measured depth/budget starting points. `"legacy"` packing and `"default"`
retrieval remain the defaults until development, validation and untouched-holdout gates
show a workload-specific benefit.

`python -m eval.evidence_contracts` checks exact-action validation and compares
legacy and coverage packing on small deterministic development fixtures. It runs
in the full offline CI matrix and the NumPy-only Python 3.9 job. These fixtures
test boundary correctness; they do not estimate external QA or model task success. The
[current diagnostic](docs/benchmark-evidence/evidence-contracts-20260919-v5.json)
also preserves a verified literal and nearby conditions from an oversized sentence
inside a 24-token context. This is one development case, not a general quality estimate.

### Public numeric evidence registry

Every exact public aggregate retained below comes from the checked-in, public-safe
[`offline-fixtures-v9.json`](docs/benchmark-evidence/offline-fixtures-v9.json) artifact. Its
SHA-256 is
`455fc9d32a236e582a49aaaf9b84f30cae2573dc6ed982f4dd7dd845afcaf24c`, also recorded in the
adjacent `.sha256` file. The artifact contains no raw questions, answers, prompts, customer data,
or per-record content fingerprints.
The fixture-suite digest is
`c8976cf0989e82f57810b99bf19f2ee3f860d07726bcb28479a319a19f00f94f`. The artifact defines
Every exact public aggregate retained below comes from the checked-in, public-safe
[`offline-fixtures-v45.json`](docs/benchmark-evidence/offline-fixtures-v45.json) artifact. Its
SHA-256 is
`35865f892fe5b6fd85009c01ee199e640d0e0f93efc7dcea1d70f30ccd71b7d4`, also recorded in the
adjacent `.sha256` file. The artifact contains no raw questions, answers, prompts, customer data,
or per-record content fingerprints.

The fixture-suite digest is
`6460533735eb17757b8705762b92346eba1b496afb4380b48d62f03d2bb364a8`. The artifact defines
the digest algorithm and records the SHA-256 of every suite and dataset file. Each evidence ID
also binds its exact command through `sha256(UTF-8 exact command)`:

Expand All @@ -26,10 +50,24 @@ also binds its exact command through `sha256(UTF-8 exact command)`:
| `offline-performance` | `python -m eval.performance --dataset eval/datasets/codemem.jsonl --k 5 --iterations 10 --json` | `bbe4aca81e58d4830e50a8fc7729a1d15b71d97a6299bccd79432b7f119677d7` |
| `offline-grounded` | `python -m eval.grounded` | `590442e51e3642c10489165759919dc86ffac62c182937330c153e7f8d5fc26f` |

External, model-dependent, latency, consolidation, and productivity numbers are not published
until a redacted immutable artifact with the same three bindings exists. Use the
[public benchmark runbook](docs/PUBLIC_BENCHMARK_RUNBOOK.md) to produce that evidence; absence
from this registry means no public number is claimed.
External, model-dependent, latency, consolidation, and productivity numbers are not included in
this offline registry unless a redacted immutable artifact with the same three bindings exists. Use
the [public benchmark runbook](docs/PUBLIC_BENCHMARK_RUNBOOK.md) to produce registry evidence.
Completed retrieval-only diagnostics are documented separately in the
[benchmark expansion results](docs/BENCHMARK_EXPANSION_RESULTS.md); absence from this registry
means no number is claimed in this offline registry.

The context-efficiency chart is generated from the registry values and the selected report schema.
Historical LoCoMo, graph, handoff, consolidation, and security figures remain preserved in their
source artifacts but are omitted from the current chart until each has a matching immutable,
public-safe artifact. The chart labels coding outcomes, external datasets, and operational
capacity as pending evaluation tracks rather than implying scores. Regenerate it with
`python scripts/render_benchmark_report.py --report docs/benchmark-evidence/offline-fixtures-v45.json --output docs/images/context-efficiency.svg` after selecting the report to publish.

The companion examples are also generated from that artifact with
`python -m scripts.render_benchmark_examples --report docs/benchmark-evidence/offline-fixtures-v45.json --output docs/images/evidence-backed-agent-examples.svg`.
The historical-to-executable mapping is in
[`docs/BENCHMARK_CHANGE_COVERAGE.md`](docs/BENCHMARK_CHANGE_COVERAGE.md).

## What we measure today (all offline, no API key)

Expand Down Expand Up @@ -69,13 +107,17 @@ frontier-model QA score.
lexical + graph + fusion + scoring + rerank + packing path after warmup, with reinforcement
disabled so repeated measurements do not mutate their corpus. It reports p50/p95/p99 latency,
retrieval quality, packed context tokens, and full/compact JSON-shape payload proxies in one
JSON-safe schema. Payload proxies are sampled once per question, independently of the number
of timed iterations; they are not serialized MCP envelopes or transport responses. In the
registered CodeMem run, 26 payload samples total **23,810** full-proxy
`engraphis.regex.v1` tokens versus **10,982** compact-proxy tokens, avoiding **12,828** proxy
tokens (**53.88% lower**), while 260 recalls are timed. Packed context across the same 26
JSON-safe schema. Its legacy `quality` fields score all candidate chunks returned before
context packing; additive `packed_quality` fields score only chunks admitted to reader context.
Payload proxies are sampled once per question, independently of the number of timed iterations;
they are not serialized MCP envelopes or transport responses. In the
registered CodeMem run, 26 payload samples total **24,590** full-proxy
`engraphis.regex.v1` tokens versus **11,138** compact-proxy tokens, avoiding **13,452** proxy
tokens (**54.71% lower**), while 260 recalls are timed. Packed context across the same 26
samples averages **85.38** tokens and reaches **108** under a 1,500-token cap; Recall@5,
hit@5, and answer-token recall remain 1.000. These aggregates are evidence ID
hit@5, and answer-token recall remain 1.000 for the legacy candidate-page view. The registered
v9 artifact predates `packed_quality`, so no packed-quality aggregate is published from it.
These aggregates are evidence ID
`offline-performance` in the registry above. `--filler-memories`, `--candidate-k`, and
`--retrieval-profile` make scaling and routing experiments executable, but their results need
separate evidence before publication.
Expand Down Expand Up @@ -149,8 +191,9 @@ python -m eval.performance --dataset fixed-1000-plus.jsonl --acceptance-matrix -
# External retrieval diagnostics (downloads all-MiniLM-L6-v2; not QA/leaderboard results)
python -m eval.external --dataset longmemeval_s.json --format longmemeval --k 10
python -m eval.external --dataset locomo10.json --format locomo --k 10
# Complete external-dataset coverage with an immutable embedding revision. This remains a
# private diagnostic; it is not an official benchmark-harness or public evidence artifact.
# Complete external-dataset coverage with an immutable embedding revision. These runs are
# retrieval-only diagnostics, not official benchmark-harness or leaderboard results. Completed
# public-safe artifacts and measured results are listed in the benchmark expansion report.
python -m eval.external --dataset longmemeval_s.json --format longmemeval --canonical \
--embed-revision <40-character-model-commit> --json external-longmemeval.json
python -m eval.external --dataset locomo10.json --format locomo --canonical --no-resolve \
Expand All @@ -173,12 +216,11 @@ names every remaining replacement/removal, must be fully consumed, and is record
report with its own hash. Any source update, unused repair, or unresolved ID fails the run. This
repairs retrieval references only; it does not claim to correct LoCoMo's semantic answer labels.

A private pinned retrieval diagnostic was inspected during development, but its result artifact is
not checked into the public evidence registry. This document therefore publishes none of that
run's workload counts or scores. Reproduce it from the hash-bound source and repair manifest,
export a public-safe immutable artifact, and validate its checksum before adding quantitative
claims. Any future values remain evidence-retrieval metrics, not end-to-end QA accuracy or an
official LoCoMo leaderboard score.
The earlier private pinned retrieval diagnostic was the pre-publication state. Current complete
LoCoMo and LongMemEval retrieval diagnostics are retained as separate public-safe artifacts in the
[benchmark expansion report](docs/BENCHMARK_EXPANSION_RESULTS.md), with source, model, configuration
and checksum boundaries. Those values remain evidence-retrieval metrics, not end-to-end QA accuracy
or an official LoCoMo leaderboard score.

## What we do NOT yet claim

Expand Down Expand Up @@ -314,15 +356,15 @@ Use `--artifact` on any of these commands to write a redacted, immutable evidenc
an adjacent SHA256 file. The ordinary console/`--json` report is private run material and may
contain source questions for debugging.

### Upstream-data diagnostics awaiting public artifacts
### Upstream-data diagnostics and publication scope

The LoCoMo-Plus, MemoryAgentBench, and Mem2ActBench adapters have been exercised against upstream
data and exposed useful product gaps. Their earlier local envelopes are not present in the
checked-in evidence registry, so this document withholds their case counts, retrieval scores,
token coverage, and throughput measurements. Rerun each adapter with `--artifact`, publish the
redacted immutable envelope and checksum, and add its suite/config binding before quoting a
number. Until then these lanes demonstrate executable plumbing only, not leaderboard,
answer-quality, or marketing results.
The LoCoMo-Plus and MemoryAgentBench adapters have been exercised against upstream data and remain
queued for their own public-safe retrieval artifacts. Rerun each pending adapter with `--artifact`,
publish the redacted immutable envelope and checksum, and add its suite/config binding before
quoting a number. Mem2ActBench's declared small retrieval diagnostic is complete and has a checked-in
artifact; its exclusion and memory-cardinality figures are source-preparation metadata in the
public source lock, not product or action-success metrics. None of these lanes is an official
leaderboard, answer-quality, or marketing result.

The MemoryAgentBench loader accepts both its aligned public JSON export and the Hugging Face
dataset-server `rows[].row` envelope. Rows without gold evidence remain useful for answer-token
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ All notable changes to Engraphis are documented here. Format loosely follows

## [Unreleased]

- Receipt-chain structural corruption remains fail-closed at the Store boundary without
bricking a completed service operation: affected responses now carry a content-free
`receipt_warning`, and graph/import workers preserve their completed state.

## [1.7.4] - 2026-09-13

- Writable SQLite files now default to WAL plus FULL synchronization, with an explicit
Expand Down
6 changes: 5 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@ ENV PYTHONUNBUFFERED=1 \
# Customer-side cloud session and entitlement display cache. Keep it on /data rather
# than the container's ephemeral home so reconnects do not lose rotated credentials.
# License issuance, trial state, leases, and revocations remain private services.
ENGRAPHIS_STATE_DIR=/data/.engraphis
ENGRAPHIS_STATE_DIR=/data/.engraphis \
# Dashboard-managed non-secret settings must survive a Railway redeploy with the volume.
ENGRAPHIS_ENV_FILE=/data/.engraphis/config.env

WORKDIR /app

Expand All @@ -33,6 +35,8 @@ RUN apt-get update \
COPY pyproject.toml README.md LICENSE NOTICE ./
COPY engraphis ./engraphis
COPY scripts ./scripts
# The declared distribution license assets are part of the package build metadata.
COPY deploy ./deploy

# Railway runs CPU workloads. Install the CPU-only PyTorch wheel before the embedding
# stack so pip cannot select PyPI's multi-gigabyte CUDA dependency chain. The public
Expand Down
Loading
Loading