context-graph-eval: LongMemEval v1 to Golden converter (map #297) - #311
Open
antejavor wants to merge 25 commits into
Open
context-graph-eval: LongMemEval v1 to Golden converter (map #297)#311antejavor wants to merge 25 commits into
antejavor wants to merge 25 commits into
Conversation
First slice of eval Tier 1. New workspace member context-graph/eval, holding the evaluation loop for the Context Graph family. to_golden() maps a LongMemEval record onto a deepeval Golden. Golden.context takes only turns flagged has_answer, not whole evidence sessions -- otherwise ContextualRecall would score retrieval against facts the answer never needed. Goldens carry tier and question_type metadata so Tier 1 and Tier 2 can be scored separately (#303), and abstention questions are marked so they can be split out. to_session_fixtures() converts the haystack into injectable fixtures, keeping distractor sessions deliberately: they are what give retrieval precision, and so the payload-size efficiency metric, anything to measure. Decisions: #302 corpus format, #303 tiers/authorship, #304 metrics, #308 benchmark choice, #309 efficiency and isolation.
) Second slice of eval Tier 1: turns the converter into something that produces a real committed corpus file. Own JSONL writer instead of deepeval's save_as. Correcting #302, which claimed persistence needed no new code: deepeval's jsonl branch writes only input/actual_output/expected_output/retrieval_context/context, dropping additional_metadata (the tier, without which #303's two tiers cannot be scored apart), dropping name/source_file (upstream traceability), and joining context on '|' -- corrupting any entry containing that character. fetch() pins an explicit upstream revision. A moving ref would silently change the corpus between runs, breaking the cross-version comparison the committed corpus exists to support. The oracle variant is refused: no distractors means retrieval precision and payload-size efficiency score well by construction. Sampling is deterministic, stratified by (question_type, abstention), and proportional with a per-stratum floor. Three bugs found by running against the real dataset rather than hand-built records, each invisible to the unit tests as written: - Counting questions are answered upstream with a bare int, aborting the whole corpus build against Golden's str-typed expected_output. - Abstention questions are marked by an '_abs' question_id suffix, NOT by question_type as the upstream README states -- 30 of 500 records, none with that type. The original check could never have fired. - Those records sit in contiguous runs at the end of each type block, so stratifying on question_type alone sampled zero of them, silently dropping the category where a confident answer is itself the failure. Sampling then needed proportional quotas: equal-weighting strata turned a 6% upstream abstention rate into 40% of a 60-question sample, and reconciling rounding drift against a single stratum left two equally-sized categories at 17 vs 11.
…tance (map #297) Third slice of eval Tier 1, and the first that touches Memgraph. Tested e2e against a real instance rather than a mocked client, per CONTEXT-MAP's testing tiers: asserting on a query string passed to a mock proves nothing about whether real Memgraph accepts it. Corrects #309's isolation mechanism. Per-batch databases are Memgraph multi-tenancy and need an Enterprise licence -- 'SHOW DATABASES' is refused outright on community. The ticket reasoned from the existence of a MEMGRAPH_DATABASE config key that database isolation was free; the key exists but cannot select a second database without a licence. Replaced by a dedicated eval instance cleared before each batch: same known-fixed-state guarantee, no Enterprise dependency, and clearing is safe only because nothing else lives there. Correction recorded on the ticket and the map. SessionFixture.turns now carries Turn(role, content) rather than pre-formatted 'role: content' strings -- injection records each turn as a Message with its own role, and flattening early threw that away. Two more bugs found by running against the real dataset: - Upstream draws distractors from a shared pool and reuses them across questions: 3,942 of 23,867 haystack session ids are duplicates. Injected raw, colliding ids MERGE onto one Session node, piling different questions' turns together and duplicating content on every reuse. Session ids are now namespaced per question. - actions-graph validates session_id against ^[a-zA-Z0-9_-]{1,128}$, so the obvious ':' separator is rejected at injection. Uses '--' instead; no real question id contains one and the longest real pairing is 46 characters. holds_evidence is deliberately never written to the graph -- it is corpus-side bookkeeping, and storing it would tell retrieval where the answer lives. Verified against real data: 3 questions -> 148 sessions, 1,564 turns, all left reconciliation_status='pending', no evidence marking leaked.
…s (map #297) Fourth slice of eval Tier 1. reconcile_batch() drives sessions-graph's reconcile_session over an injected batch: the same pass a real harness session gets, one LLM call extracting entities into Chunks and a second producing the Episode. Kept separate from injection because it is LLM-backed and slow -- folding it in would make staging a batch cost as much as scoring one. Bounded via limit, since a full run's sessions are far too many to distil in one pass. Partial failure is reported, not raised: a score only means something if you know how much of the graph is populated, so one undistillable session must not abandon the rest of the batch. LLM credentials resolve from context-graph's config file (ADR 0002) before falling back to the environment, so eval runs standalone without exported vars. Also reverses this branch's session-id namespacing. Namespacing per question was added on the evidence that 3,942 of 23,867 haystack ids repeat across questions. Checking the content behind those repeats shows *zero* carry differing content -- a repeated id genuinely is the same session. So the ids were never the bug; re-appending turns on reuse was. Fixtures are now deduplicated by session_id at injection, which keeps ids verbatim, lets a shared session be one node as it would be in a real organizational graph, and avoids about 4,600 redundant LLM-backed reconciliations over a full run. Verified against real data: 10 questions -> 496 fixture slots -> 493 distinct sessions injected, all pending. Coverage gap, stated plainly: the four tests asserting real distillation (pending cleared, Chunks linked to source Actions, Episode written, per-session outcomes) are gated on an LLM key and SKIP here -- config.toml has openai_api_key set to an empty string. They are written but unverified. The non-LLM behaviour (pending discovery, limiting, batch resilience via an injected failing wrapper, empty-batch no-op) is verified.
…etween runs (map #297) Fifth slice of eval Tier 1. retrieval.py is #300's baseline as decided: the agent gets the graph schema and writes its own read-only Cypher, with no ranking, templates, or vector search. Those stay deferred until this baseline's failures say what they should be. Retrieved carries retrieval_context (what the graph actually returned -- what ContextualRecall scores and what #309's efficiency metric counts tokens over), the queries it ran (so a score is diagnosable), and failed queries recorded rather than raised, so one bad statement costs a question its answer instead of aborting a batch. Writes are refused: retrieval must not alter the graph it is scored against. The step budget is bounded for the same family of reason -- retrieval cost is itself scored, so unlimited querying could buy coverage with an unbounded payload. The write guard duplicates mcp_memgraph's is_write_query rather than importing it: that module binds a client registry to global env config at import time and would point retrieval at whatever Memgraph the environment names rather than the dedicated eval instance. Noted in-code as a sync obligation. Also fixes a real bug in the previous slice. Injection used ActionsGraph.clear(), which only deletes Session|Agent|Action|Tool and leaves Chunk, Entity, Episode and Memory standing -- exactly what reconciliation produces. A batch therefore inherited the previous run's *distilled memory*, so a question could be answered from the last run rather than this batch's fixtures: the leak #309 exists to prevent, and one that would inflate scores invisibly. Injection now wipes the whole graph, safe only because the eval instance is dedicated. Coverage gap, stated plainly: the one test that exercises #300's actual premise -- can a real model reach an injected fact given only schema and Cypher -- is LLM-gated and SKIPS here, as do the four reconciliation tests. The loop's plumbing (write refusal, context capture, query recording, error handling, schema description) is verified against real Memgraph with a stub model. Incidental finding worth knowing before scoring: turn text is not an Action property but JSON inside Action.properties, so retrieval has to dig for it. That is a real difficulty signal about the baseline, not a defect.
…ema (map #297) Final slice of eval Tier 1's pipeline. scoring.py implements #304's split: the LLM judges quality (ContextualRecall over retrieval, plus one GEval coverage rubric over the answer), plain code counts cost. Faithfulness and AnswerRelevancy stay omitted -- both exist mainly for the no-ground-truth case, and each extra metric is another judge call per question, multiplied again per schema candidate. Coverage hard-gates and efficiency ranks within it (#309): a failing question is dropped rather than ranked, so a zero-token failure cannot top the ranking. Aggregation is per tier with no blended headline field at all (#303) -- a single cross-tier number is precisely what would let an organizational-recall regression hide behind a personal-memory gain, so the type simply does not offer one. Efficiency uses a median rather than a mean so one pathological payload cannot drag the number that gets compared across schema versions, and the tokenizer is pinned for the same reason #304 pins the judge. Abstention questions are reported apart: there a confident answer is the failure, and averaging them into ordinary recall hides that. Also closes a real divergence from #300 flagged during review. Retrieval was using a hand-rolled labels-and-relationship-types summary rather than the existing query surface. graph_schema now calls memgraph_toolbox's SearchSchemaTool -- the same implementation the MCP server's search_schema tool wraps -- so the baseline genuinely is the existing surface, pointed at the dedicated eval instance. It degrades to the simple view when Memgraph runs without --schema-info-enabled rather than crashing. This matters beyond fidelity: injected turn text lives inside Action.properties as JSON, not in a content property, so an agent given only label names would have to guess that. The eval instance should be started with --schema-info-enabled.
Completes eval Tier 1's pipeline: inject -> reconcile -> retrieve -> score, driven end to end, with a 'run' subcommand. On the division of labour: deepeval does ship a runner, but it runs metrics over test cases and knows nothing about injection, reconciliation, or retrieval -- all of which must happen before an actual_output exists to score. So the runner owns the pipeline loop and deepeval owns the scoring loop underneath it. Judging goes through deepeval's evaluate() rather than a hand-rolled metric loop, with async/display/error configs set explicitly. Ordering is the runner's real responsibility: retrieving before injection would query an empty graph and score every question a miss; scoring before reconciliation would score raw turns rather than emerged memory. check_offline() refuses to start when CONFIDENT_API_KEY is set. deepeval uploads a test run whenever a Confident AI key is present, and #302 kept eval data out of a vendor cloud on this project's own owned-IP grounds -- a stray env var should fail loudly rather than quietly export results. A question whose retrieval raises is reported as a miss rather than propagating: a coverage rate computed over a silently shortened corpus is wrong, not merely noisy. Reconciliation is separable via --skip-reconcile since it dominates cost. The gated efficiency median is empty without a judge (nothing clears an unscored gate), so the CLI additionally prints an explicitly UNGATED payload median -- labelled so it is never mistaken for the comparable number, which #309 defines only within the gate. Verified end to end against real LongMemEval data with a stub model: 4 questions -> 174 sessions injected, abstention questions detected, per-question efficiency measured. Real reconciliation, real retrieval and judging remain LLM-gated and unverified.
…l dispatch (map #297) Two tiers, split on cost and determinism. tests.yaml gains test-context-graph-eval, running on every PR touching the package. It provides NO LLM key deliberately: the LLM-backed tests skip themselves without one, so the job stays fast, free and deterministic. That covers 74 of the package's 79 tests -- converter, corpus, sampling, scoring composition, injection e2e, and retrieval plumbing -- against a real Memgraph. eval-run.yaml runs a real eval batch on workflow_dispatch only, following test-live-graph-model.yaml's precedent. Not on push/PR for three separate reasons: 1. Cost -- reconciliation is ~2 LLM calls per session and a 20-question batch injects ~900 sessions. 2. Non-determinism -- judged scores vary run to run; #304 handles that with repeat-and-compare calibration, not a pass/fail gate. Gating on a noisy metric produces flaky CI that teaches people to ignore it. 3. It would contradict #299, which chose human-gated promotion: eval produces a comparison report a person reads. A CI gate on the score is automatic promotion wearing a different hat. Both jobs start Memgraph with --schema-info-enabled, which is required rather than cosmetic: retrieval's schema description goes through memgraph-toolbox's SearchSchemaTool -- the same implementation the MCP server's search_schema wraps -- which refuses without it, and the agent would silently fall back to bare label names, testing a different path than production. The dispatch job runs the retrieval agent on OpenAI and the judge on Anthropic, keeping #304's deliberate provider split so the judge does not share the extraction pipeline's blind spots. It uploads the report as an artifact rather than asserting on it -- the score is input to a human decision, not a gate. This effectively answers #306 (what triggers an eval run, and how often).
This was referenced Aug 25, 2026
…305) The report a person reads to decide whether to promote a candidate. #299 chose human-gated promotion, so this does not decide -- it makes the decision makeable, which means saying what changed and whether the change is real. Two behaviours carry most of the value: It REFUSES to compare runs measured differently -- different corpus revision, judge model, or tokenizer. #302 pinned the corpus and #304 pinned the judge and tokenizer precisely so two runs would be comparable; a delta computed across different pins measures the pin change as though it were the change under test and reports it confidently. A refusal, not a warning. Without calibration it will not call any delta real. Judged scores vary run to run, so a bare 12/20 -> 13/20 invites reading a win into noise. The noise floor comes from #304's repeat-and-compare check; absent one the report says NOT CALIBRATED and returns inconclusive. coverage_is_real is deliberately three-valued -- 'not real' and 'cannot tell' are different answers to a human deciding. A real coverage regression decides the verdict even when efficiency improved: coverage is the gate (#309) and a cheaper answer missing facts is not better. Efficiency alone never declares an improvement, since 'coverage held' cannot be established inside the noise floor. Regressions are named, not just counted -- a rate says something moved, only names say where to look. CLI: 'run --save' persists a run, 'compare' renders the report. Sizing caveat found by rendering real numbers and worth carrying into #304's calibration work: at 20 questions one question is 5pp, so any single flip clears a +/-4pp floor. Coverage granularity is coarser than a plausible noise floor at small corpus sizes -- an argument for scaling the corpus before trusting small coverage deltas.
…#297, #307) One question, per #307: a fact that exists only inside a subagent. Top-level recall is already covered by Tier 1's fixtures, whereas the nested carrier has a demonstrated silent-failure mode -- #281 found get_session_actions() does a single-hop HAS_ACTION match, so once subagent activity moved under (:Agent), reconciliation would stop seeing it with no entities, no Episode mention, and no error. Caught once by reading code; as a gold-slice question it is caught automatically from now on. Building it corrected #307's own framing. A fact planted in the session prompt CANNOT be subagent-only: to reach the subagent the top-level prompt must state it, which puts it in a top-level Action as well, so recall would pass even with nesting completely broken. The fact therefore has to originate inside the subagent's execution. It reads an existing repo file -- no scratch fixture to maintain, and stable ground truth. The fact is the pinned corpus revision, chosen because it must be UNGUESSABLE. An obvious candidate like the declared licence fails: a model answers 'MIT' from priors, and recall that can be guessed tests nothing. A hex revision cannot be. A test asserts the answer tracks DEFAULT_REVISION, so bumping the pin fails loudly here instead of surfacing later as a mysterious regression. evidence_is_nested() is checked BEFORE trusting a recall result: if the model declines to delegate, the fact lands top-level, recall succeeds trivially, and the question silently stops testing nesting -- a false pass indistinguishable from a real one. The live-session driver is deliberately NOT included. It cannot simply set MEMGRAPH_URL: per ADR 0002 hook subprocesses resolve configuration only from ~/.config/context-graph/config.toml and ignore env vars at hook runtime, so the driver must back up, rewrite and restore the user's real config file pointing at the eval instance -- exactly what scripts/dev-memgraph.sh hooks-local already does. Writing that blind, unable to run a billed Claude Code session to verify it, against a path that mutates the user's live config, is not worth the risk. Documented in the README as the remaining step, with the reuse target named.
#297) The 5 LLM-gated tests had never run. Running them found three real bugs, two of them serious. Full suite now 102 passed, 0 skipped, against real models. 1. reconcile_batch passed working_dir=None into LightRAG, which guards on the KEY being present rather than its value, so None reached os.path.exists() and raised TypeError before any reconciliation happened. Defaults to ./lightrag_storage, matching sessions-graph reconcile's own default. 2. LightRAG's Memgraph storage backends resolve their connection from the ENVIRONMENT, not from the client passed in. Unset, they refuse to start. Set to something else, they write reconciliation output to THAT graph -- so with an ambient MEMGRAPH_URL pointing at a dev instance, an eval batch would have distilled straight into it. That is precisely the pollution #309's dedicated-instance decision exists to prevent, with nothing to indicate it had happened. reconcile_batch now takes memgraph_url and sets the environment LightRAG actually follows, refusing to run when neither is available. 3. A query that was valid Cypher but matched nothing was invisible to the retrieval agent -- the loop recorded errors only, so the model saw no difference between 'I have not queried yet' and 'my assumption was wrong'. Observed against a real model: it invented an action_type from the question's wording ('adopt_dog'), ran that same idea four times, got zero rows each time with nothing to contradict it, and answered 'not in memory'. Empty results are now reported back. Finding, more important than the fixes: even with (3) fixed, the retrieval baseline reaches the fact in only about two runs of three -- on the EASIEST possible case, two sessions and one distractor. The test now asserts success at least once in three rather than pretending determinism, and documents the rate. At ~2/3 on the easy case, a Tier 1 coverage score would largely measure whether the agent guessed workable Cypher rather than whether the memory is any good. This is the concrete evidence #300 deferred retrieval v2 to obtain, and it points at the first thing to try: a schema description carrying real property VALUES, not just keys. The consistent failure is the model inventing a domain-shaped action_type because nothing in the schema dump contradicts it.
…/3 to 6/6 (map #297) The retrieval baseline was reaching an injected fact about two runs in three, on the easiest possible case. Diagnosis: the model was not reasoning badly, it was reasoning correctly from a schema that told it nothing about where content lives. It saw that action_type existed, never what it contained, so it invented a domain verb from the question's wording and queried that. graph_schema now describes property VALUES, not just keys: (:Action:Message:UserMessage) action_type: user_message properties: JSON string, keys: content, message_id, model, role, usage (values are free text) Measured effect on the same question: 6/6, up from 2/3. The test now asserts a majority of three rather than one of three -- strengthened, but not to 'always', since asserting determinism against a real model makes the test flaky rather than the model reliable. The safety constraint is what shapes the implementation. Sampling values out of the graph under test could hand the agent the answer, and retrieval would then 'succeed' without retrieving anything -- the eval measuring its own prompt. So a property's values are shown only when it is BOTH low-cardinality and short. Cardinality alone is not enough: in a small graph a free-text field easily has only two distinct values, and printing those leaks exactly the content being asked about. Length is what separates an enum from a sentence. A test asserts the injected content never appears in the schema. JSON-valued properties get their inner KEYS shown and their values withheld, on the same principle -- structure is schema, content is data. Also fixes a Memgraph incompatibility found on the way: ORDER BY on a list value is rejected outright, so label combinations are sorted in Python.
…ry (map #297) Measured at scale, 4 of 20 questions issued ZERO queries and answered from an empty context -- one of them a question whose answer was sitting in the graph untouched. Cause: a model reply with no recognisable Cypher was treated as 'I have enough' unconditionally, so an opening line of prose ('Sure! Let me look that up') ended the loop before it began. A reply without a query now means 'done' only once rows have actually been retrieved. Before that it means ask again, and the prompt says so explicitly rather than inviting prose. Found while measuring retrieval reliability against real distractor volume (936 sessions, 20 questions). That measurement is NOT reported as a result: scoring by substring containment against gold answers was the wrong instrument for this corpus and invalidated the run. Gold answers are full sentences, and abstention questions whose gold answer means 'not in memory' were marked wrong when the agent correctly said exactly that. #304 chose an LLM judge over exact matching for precisely this reason; bypassing it to save calls cost more than it saved.
…all (map #297) Measured at scale with the judge: abstention scored 0/8 -- while the agent had correctly answered "not in memory" on at least four of them. Cause: coverage takes the weakest metric, and ContextualRecall asks whether the retrieved context supports the expected output. For a question whose correct answer is "that isn't in memory", the correct retrieved context is EMPTY, so that metric scores near zero by construction and min() then guarantees failure however well the agent behaved. Roughly 40% of the sampled corpus was unpassable by design. Abstention questions are now judged by the rubric alone, which already knows to require a refusal. The runner judges the two groups in separate passes. Also fixes three things found while fixing that: - reconcile_batch demanded memgraph_url even when the CALLER supplied a LightRAG wrapper, in which case no LightRAG client is constructed and the variable is irrelevant. It refused to run a batch that needed nothing from it. - deepeval's Anthropic judge could not be constructed at all. _build_client resolves settings.ANTHROPIC_API_KEY before self._anthropic_api_key, and the settings value is a pydantic SecretStr that is truthy and therefore always wins; httpx then rejects it before any request is made. Clearing the env var is not enough either -- settings cache at first access, and merely constructing another deepeval model earlier in the process populates them, so the cached object has to be corrected directly. #304 chose Anthropic as the judge specifically so it would not share the pipeline's blind spots, and eval-run.yaml supplies the key as an environment variable, so the CI eval job would have failed on its first dispatch. Also pins a judge model that exists: deepeval defaults to claude-3-7-sonnet-latest, which 404s. - .deepeval/ is now gitignored. deepeval writes a run cache there that reached 1 MB and tripped the repo's large-file hook when it was accidentally staged.
… tests (map #297) Both would have degraded results quietly rather than failing, which is the mode this eval exists to catch. 1. The judge silently disappeared on a second build. _clear_deepeval_anthropic_secret popped ANTHROPIC_API_KEY from the process environment as belt-and-braces alongside clearing deepeval's cached settings. Clearing the settings is what actually fixes the SecretStr bug; removing the env var only meant the key was gone for every later caller, so a second _build_model returned None and the run reported "not judged" instead of failing. Verified: two consecutive builds now both return a judge, where the second previously returned None. 2. The tokenizer name recorded on a run could be a lie. efficiency_tokens fell back to whitespace splitting when tiktoken was missing -- producing numbers roughly a third smaller -- while RunMeta still recorded the pinned tokenizer's name. compare() checks that name to refuse incomparable runs, so two runs counted in different units would have compared cleanly and reported a confident delta. tiktoken was also only a transitive dependency, so the fallback was genuinely reachable. It is now declared explicitly, the fallback is gone (an efficiency figure that quietly changes units is worse than one that fails), and the run records the tokenizer it verified rather than the one it was configured with. Neither was caught by the suite: the first needs two calls in one process, the second needs an absent transitive dependency. Both were found by reading the code asking "what fails quietly here".
… (map #297) Measured on the easy case: 6/6 hits at 1.3 steps average, from 2/6 at 4.0. Payload per successful answer roughly halved. The step-budget waste this started out chasing was a SYMPTOM. The agent spent its whole budget because it kept failing, not because it was profligate. Watching the actual queries showed two real causes: 1. Vocabulary mismatch. It searched `properties CONTAINS 'adopt' AND properties CONTAINS 'dog'` against text reading "I adopted a beagle named Max" -- which contains "adopt" but not "dog", so the conjunction matched nothing. Verified directly: 'adopt' alone matches, 'beagle' alone matches, the AND returns zero. The model reasons from the question's vocabulary and assumes the stored text shares it. 2. Wrong dialect. It reached for `apoc.convert.fromJsonMap` to parse the JSON in `properties`. APOC is Neo4j; Memgraph does not have it, so the query errored. The prompt now states both: APOC is unavailable so JSON properties must be substring-matched, and stored wording rarely matches the question's so one broad term beats several ANDed together. A named STOP token is also offered once rows are in hand. On its own it changed nothing -- the agent was not failing to stop, it was failing to succeed -- but it is the right shape now that queries land, and average steps fell to 1.3. Note on the illustration in that prompt: the first version used the same words as a test question's answer ("a question about a dog may be stored as 'beagle'"), which put the answer into the prompt -- the agent could have "retrieved" what it had just been told, precisely the leak the schema description is written to avoid. Replaced with a domain no corpus question touches, and re-measured afterwards: 6/6 held, so the gain was the guidance and not the leak. Full suite green against real models: 112 passed, 0 skipped.
Measured at scale, median payload 18,893 -> 5,032 tokens (-73%) and max 1,067,650 -> 15,611, with coverage unchanged at 5/20. The tail is gone at no cost to the score. The explosion was self-inflicted. Telling the agent to "prefer ONE broad term over several ANDed together" is what made queries land at all, and also what made them match enormous row sets. Efficiency is a scored axis (#309), so an unbounded payload is not merely untidy -- and beyond the score, a million-token payload risks exhausting the judge's context window and costs real money per question. Two mechanisms, because the prompt alone is only advisory: - A hard cap in the loop, sized in characters via a conservative chars-per-token ratio. This runs on every row of every query, so an approximation that never under-estimates is worth more than an exact count. - A LIMIT instruction in the prompt, as the counterweight to the broad-term guidance -- broadening the match is what made queries succeed, and the LIMIT stops that from paying for itself in payload. Truncation is REPORTED, never silent: the agent is told how many rows were dropped and why. Silently truncating would cost coverage for a reason nothing records -- the agent would believe it had seen everything its query matched and stop looking. Same failure shape as the steps=0 and zero-row bugs: the loop knowing something the agent does not. The cap sits at 20,000 tokens, deliberately ABOVE the observed median rather than below it. Its job is clipping the pathological tail, not squeezing the typical case; tightening the median belongs to retrieval v2 and should follow measurement rather than a guess made today. Unresolved, and recorded rather than papered over: at least one question (118b2229) produced an answer exactly matching its gold answer -- "45 minutes each way" -- and scored 0, because ContextualRecall judged the retrieved rows as not supporting it. That is either the metric doing its job on an ungrounded lucky guess, or the metric being harsh. The two are not distinguishable without the human-graded calibration anchor #304 already calls for, so the 25% headline may undercount. Full suite green against real models: 115 passed, 0 skipped.
#297) CI lint was failing because `ruff format --check .` formats Python inside markdown fences, and the README's aligned trailing comments in doc examples do not survive it. Only src/ and tests/ had been formatted locally. Three verified bugs from the review, all mine: - conftest's eval_graph fixture tore down with ActionsGraph.clear(), which removes only Session|Agent|Action|Tool and leaves Chunk, Entity, Episode and Memory standing. The reconciliation tests create exactly those, so a test's distilled memory survived into the next one -- the same leak inject._wipe exists to prevent, reintroduced in the fixture. - is_write_query upper-cased the query before matching, which silently disabled every pattern containing lowercase letters. The apoc rule could therefore never fire, and `CALL apoc.refactor.rename.label(...)` passed the guard -- 'refactor' is matched by no other pattern. Now matched case-insensitively; verified it does not over-block reads (word boundaries keep 'sunset' and 'preset' safe). - report.compare refused runs whose corpus, judge or tokenizer differed but not those with different question counts. Coverage is reported as a rate, so a 20-question baseline and a 60-question candidate produced comparable-looking percentages over different corpora -- the sampling change measured as though it were the change under test. Adds the CONTEXT.md every sibling in this family has, and the CONTEXT-MAP entry listing eval among the Contexts. CONTEXT-MAP's testing-tier policy binds "every package in this family", so a family member missing from the map is a real gap. Also corrects an over-claim of my own. test_a_real_model_can_find_an_injected_fact was strengthened to require a majority of three attempts on the strength of a single 6/6 measurement; a later run of the same question scored 1/3. Reverted to "at least once in three", with the history recorded in the docstring. Reading one favourable sample as a settled improvement is precisely the mistake this eval exists to make harder, and the review was right to flag it. 118 passed, 0 skipped, against real models. Both lint gates green repo-wide.
… (map #297) Addresses the two sharpest findings from the PR review. ## The corpus is now in git, and the run reads it #302 put the corpus in git so that two runs being compared provably answer the same questions. That mechanic existed -- build-corpus wrote a JSONL -- but nothing read it: `run` re-fetched and re-converted upstream in-process every time, which proves nothing. A change in sampling, conversion, or upstream would silently alter the question set between a baseline and its candidate. corpus/tier1-longmemeval.jsonl (19.3 KB, 20 questions) is committed, and `run` reads it. Rebuilding is now a deliberate act via build-corpus. Departed from #302's literal wording on one point. It says to commit "the Golden JSONL, plus injection-ready session fixtures". Measured: at 20 questions those fixtures are 9.54 MB of reshaped upstream text -- 19x the repo's 500 KB pre-commit limit, and precisely the "upstream blobs" the same ticket rejected vendoring on size and licence grounds. The answer key's immutability comes from git; the fixtures' comes from the pinned revision. Same guarantee, two mechanisms, and following the reasoning rather than the sentence. ## Dead code wired read_corpus is now the run's input path. gate_and_rank drives new cheapest / costliest lines per tier, which makes #309's gate-then-rank visible rather than merely implemented -- a median says something moved, those say where to look. gold_slice_goldens is reachable behind --gold-slice. ## A trap found by running it --gold-slice initially reported "Tier 2: coverage 0/1". That is not a recall failure: the gold slice's fixture is planted by a real Claude Code session (#307), that driver does not exist, so the fact was never in the graph. A guaranteed zero dressed as a measurement is the exact false signal this eval exists to remove. Added evidence_is_planted, and the run now refuses --gold-slice when the fixture is absent rather than scoring it. Worth recording that the flag cannot succeed yet for a second reason either: inject_batch wipes the graph, so even a correctly planted session would be destroyed by Tier 1's injection in the same batch. Reconciling those two is part of building #307's driver, not something to paper over here. evidence_is_nested remains uncalled for the same reason -- it needs that driver, and wiring it to anything else would be pretence. 120 passed, 0 skipped, against real models. Both lint gates green repo-wide.
…297) The PR review found the retrieval baseline had been "tuned until it scored", which erases the signal #300 deferred retrieval v2 to be shaped by. That was largely right, and this reverts the part of it that was actually strategy. Kept, because these are defects rather than strategy -- #300 defers "ranking, query templates, vector search", not a prompt that is simply wrong: - zero-row results are reported to the agent (a valid-but-empty query used to be indistinguishable from not having queried); - prose no longer ends the loop before a single query (4 of 20 questions used to issue none); - the schema describes property values, not just keys; - the agent is told which database dialect it is querying, having reached for Neo4j's APOC. Reverted, because these are search strategy: - "prefer ONE broad term over several ANDed together" - "always add a LIMIT" Measured at scale (20 questions, ~936 sessions, judge, no reconciliation): crude tuned coverage 4/20 5/20 median payload 2,679 5,032 The hints bought ONE question and doubled the payload. One question is 5pp at n=20, coarser than any plausible noise floor, so this was never a demonstrated gain. What made them look valuable was an easy-case 6/6 against 2/6 -- a single favourable sample, later contradicted by a 1/3 run of the same question. It also confirms the two were entangled: the LIMIT advice existed only to contain the payload explosion the broad-term advice caused. Removing both halves the median payload by itself. The payload cap stays -- unbounded cost is a safety concern independent of strategy -- but now has no prompt counterweight, so truncation will be more frequent. That is itself signal about what v2 must handle. The recovered failure modes are recorded on #300 as evidence rather than left in commit messages: vocabulary mismatch (the fundamental one), content not being addressable inside a JSON blob, unbounded result sets, and no relevance ordering.
Clears the last three PR-review findings. ## Attribution kept, not collapsed coverage was min(ContextualRecall, GEval), which gates correctly but threw away which stage failed -- ContextualRecall scores retrieval, the rubric scores the answer. #304 had noted that attribution "falls out for nothing"; collapsing to one number was discarding it. Scored now carries metric_scores alongside coverage. min() still gates -- passing one check while failing another is not a pass -- and the report prints which metric was the weakest link across failures, so a run failing mostly on retrieval is visibly a different problem from one failing on answering. ## Calibration is now measured, not typed #304 chose repeat-and-compare over trusting a single run, because temperature=0 is not deterministic on hosted APIs. Until now --noise-floor was a number typed by hand, which is the guess the mechanism exists to avoid. `calibrate` derives it from repeated runs of the same questions. It uses the full range rather than a standard deviation: the floor's job is to stop someone believing a delta the judge would have produced anyway, so it should cover the movement actually observed. It refuses fewer than two runs (one has no spread, and reporting 0.0 would assert that any delta is real), and refuses runs covering different questions, whose spread would measure the corpus rather than the judge. This covers the NOISE half of #304's calibration. The BIAS half -- roughly 25 human-graded items, to catch a judge that is stable and consistently wrong -- is not something code can supply and remains outstanding. It is also what would settle whether 118b2229's exactly-correct answer scoring 0 is the metric working or the metric being harsh. ## Test dedup Seven near-identical scripted-LLM stubs (StubLLM x3, ChattyLLM, Modest, Capturing, DoneAfterOne, _Firehose) collapse into one conftest ScriptedLLM: the shape was always "count calls, cypher then prose", only the script differed, so the script is the parameter. requires_openai_key was also defined twice and now lives in conftest. 124 passed, 0 skipped, against real models. Both lint gates green repo-wide.
…make it safe (#307) The gold slice now works end to end. Verified against a real session: the fact lands inside an Explore subagent (1 action) and nowhere at top level (0). ## ADR 0003: CONTEXT_GRAPH_CONFIG selects the config FILE Hooks resolve configuration only from a config file (ADR 0002), and that file's path was a single global. So pointing one session's hooks at the eval instance meant rewriting it -- which redirects EVERY Claude Code session on the machine. Not hypothetical: while building this, an unrelated session's activity was recorded into the graph under test. An eval graph contaminated by ambient sessions is exactly the pollution a dedicated instance exists to prevent, so the mechanism defeated its own purpose. The override selects which file to read, never what is in it, so ADR 0002's guarantee is intact. Its reasoning was about *ambient* environment -- hook subprocesses do not source shell profiles, so `doctor` and hooks disagreed. A path handed down by the process spawning the session is the opposite case: explicit, and with nothing ambient to drift from. The driver now writes a throwaway config and passes its path, so nothing outside the child's environment is modified -- no backup, no restore, and no window in which a crash leaves the machine misconfigured. Existing tests improved with it: they monkeypatched _identity._CONFIG_DIR and _CONFIG_FILE, and now use the supported override, exercising the same path a real isolated session takes rather than a shape only tests can produce. ## Four failures, each hidden behind the last Every one of these exited 0 or looked like success, and none was visible from a green test suite. 1. Session ran, nothing recorded -- read as the model declining to delegate. 2. The transcript disproved that (subagent spawned, fact found), so the fault was hooks. drive_session had been DISCARDING the transcript on success; test-graph-model already documents why not to, and that lesson was not carried over. A billed run had produced no diagnosis. 3. --strict revealed ImportError: skills-graph is required. `uv run --package agent-context-graph` re-resolves the environment to that package's own closure, which excludes the connector packages -- so every hook raised, the runner swallowed it (correctly: a broken hook must not break the harness), and exited 0 having written nothing. Dropping --package fixes it. 4. Hooks then worked, but the fact appeared at top level too. A subagent's report is recorded as the parent's Task ToolResult, so ANY fact a subagent is asked to report lands at both depths by construction -- and retrieval could have answered without traversing HAS_AGENT. The question would have passed with nesting completely broken. ## Three gates, none of them a recall result evidence_is_planted, evidence_is_nested, and the new evidence_is_top_level decide whether a recall result would mean anything, and fail for different reasons. The prompt now asks the subagent to confirm without quoting the value; whether a model obeys is model-decided, so the prompt is a request and the check is the guarantee. A run that leaks anyway is reported void rather than scored. Also strips ANTHROPIC_API_KEY from the session environment: #304 puts the judge on Anthropic, so the eval process legitimately holds one, and inheriting it made the CLI refuse to start. The judge's credential has no business steering the session under test. 135 eval tests, 111 agent-context-graph tests, both lint gates green.
…n target (map #297) ## Subsampling Reconciliation cost scales with *sessions* while coverage needs *questions*, and upstream couples them at roughly 47:1 -- so an affordable full-pipeline run would otherwise collapse to two questions, at which point one flip is 50pp and coverage says nothing at all. --max-sessions-per-question trims each haystack. Evidence sessions are never dropped: losing one makes its question unanswerable for a reason unrelated to recall, and the resulting miss would be indistinguishable from a real failure. Selection is deterministic (upstream order, not random) because the subsample is part of what a run measured -- two runs of the same corpus must inject the same graph or their scores are not comparable, the same reasoning that pinned the corpus and the judge. Off by default. The full haystack is the honest difficulty; a trimmed one has fewer distractors and therefore flatters retrieval, so any score measured this way is an UPPER BOUND and is not comparable to a full-haystack run. Flattering the number has to be asked for explicitly. ## reconcile_batch's target is now plumbed through run_batch called reconcile_batch without memgraph_url, so the first real full-pipeline run stopped immediately on the guard added when LightRAG's storage backends turned out to resolve their connection from the environment rather than the client passed in. Without that guard reconciliation would have distilled into whatever MEMGRAPH_URL happened to name -- plausibly the dev instance -- and the eval graph would have shown no Chunk or Episode nodes, reading as "reconciliation is broken" while the dev graph quietly filled with eval data. A test now asserts the URL reaches reconciliation, since the failure it prevents is silent and writes to the wrong place. 134 passed, 5 skipped. Both lint gates green.
…#297) A run whose judge errored printed "coverage 0/2 (0%)" -- an outage rendered as a measurement, indistinguishable from a real regression. Observed live when the judge provider ran out of credit: every metric errored and the run exited 0 with a confident, wrong number. A Scored row with no metric_scores was never judged, so it is now excluded from the rate and surfaced separately as `unscored`. coverage_rate is None rather than 0.0 when nothing in a tier could be scored -- better to say nothing than to report a score that was never taken. The warning fires only when a judge was actually configured; without one, unscored is expected. Also here: - `run --limit` was silently ignored after `run` switched to reading the committed corpus. A run asked for 2 questions did all 20, which is how a "minimal" check turned out to be 40 sessions of reconciliation. - Reconciliation prints per-session progress. The loop is sequential at ~2 LLM calls per session, so a modest batch runs for many minutes; silent until done is indistinguishable from hung, which is how two runs were abandoned without knowing whether they were progressing. - First tests for the report printer. It had none, and this commit's own first draft moved the efficiency line inside the nothing-was-scored branch, so ordinary runs would have stopped printing it. Claude-Session: https://claude.ai/code/session_01MjKpCYyk2v2N9DMqTBUJiy
The set of questions a run asks is what every number downstream is relative to, and it was silently wrong for a while: --limit stopped being consumed when `run` switched to reading the committed corpus, so a run asked for 2 questions did all 20. That is ~40 sessions of reconciliation at two LLM calls each -- expensive, slow, and indistinguishable from a hang, so it was killed twice before the cause was found. The previous commit fixed the behaviour inline, where nothing could test it: _run needs Memgraph, a network fetch of the haystack, and two model credentials before it reaches that line. Pulled out as select_goldens so the selection can be tested on its own, including that a Tier 1 --limit does not trim the Tier 2 gold slice appended after it (#303). Claude-Session: https://claude.ai/code/session_01MjKpCYyk2v2N9DMqTBUJiy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First slice of eval Tier 1, from Map: Context-graph emergence pipeline.
Adds a new workspace member,
context-graph/eval(packagecontext_graph_eval) — the evaluation loop for the Context Graph family. This PR contains the corpus converter only; nothing touches a database yet.Why a new package
The eval code depends on
sessions-graph/actions-graph/unstructured2graphplusdeepeval. Putting it inmemgraph-toolbox(wheredeepevaland the existingCoherenceEmbeddingsBasedMetriclive) would invert the dependency direction, since every context-graph package already depends on toolbox. It is a workspace member but is not published to PyPI — consumers of context-graph don't need the eval corpus.What's here
to_golden(record) -> Golden— maps one LongMemEval v1 record onto a deepevalGolden:question→input,answer→expected_outputcontexttakes only turns flaggedhas_answer, not whole evidence sessions. Whole sessions would drag in surrounding chatter, andContextualRecallMetricwould then score retrieval against facts the answer never actually needed.additional_metadatacarriestierandquestion_type, so Tier 1 (adopted) and Tier 2 (authored) can be scored separately per #303 — a blended score would let an organizational-recall regression hide behind a personal-memory gain.name/source_filekeep each Golden traceable to its upstream record, which matters under #302's fetch-and-convert model.to_session_fixtures(record) -> list[SessionFixture]— converts the haystack into injectable fixtures. Distractor sessions are kept deliberately. They are what give retrieval precision — and therefore the payload-size efficiency metric from #309 — anything to measure; a haystack of evidence alone would score well by construction.SessionFixture.holds_evidenceis bookkeeping only and must never reach retrieval, which would hand the answer's location to the thing under test.Decisions this implements
Golden→ JSONL in git; upstream is fetch-and-convert, not vendoredContextualRecall+ oneGEval; quality judged, cost countedTesting
10 unit tests, all green; ruff lint and format clean.
Pure transformation with no I/O, so these stay unit tests per the family's testing-tier policy in
CONTEXT-MAP.md— nothing here asserts Cypher correctness or needs a live Memgraph.uv run --package context-graph-eval --extra test pytest context-graph/eval/testsTest records are hand-built to the schema documented in the LongMemEval repo, not recomputed the way the converter computes them.
Note
Ruff's
B905caught a genuine bug rather than a style nit:haystack_session_ids,haystack_dates, andhaystack_sessionsare parallel arrays, and a non-strictzipwould silently truncate on any upstream mismatch — dropping sessions and corrupting the corpus with no visible error. Nowstrict=True.Not in this PR
Fetch script for the pinned upstream release, JSONL corpus writer, fixture injection into a dedicated eval database, reconciliation trigger, the retrieval agent over
run_cypher_query, and the metrics themselves.Benchmark survey backing the LongMemEval choice (including why LoCoMo is ruled out on its CC BY-NC licence):
docs/research/2026-08-memory-benchmarks.mdon branchresearch/memory-benchmarks.