Scope stores to the project, and measure the semantic backend - #3
Merged
Conversation
Two changes aimed at making this usable day to day rather than just
correct on a benchmark.
Per-project stores
- The default was a single global ~/.agent_memory/store.json shared by
every project, and all three README configs pointed at it. Recall
matches on similarity alone, so one project's "how do we deploy"
answer could surface while working on another.
- The store now resolves as: AGENT_MEMORY_PATH, then .agent_memory/
store.json in the enclosing git repository, then the global file when
outside a repo. `.git` is tested for existence, so worktrees and
submodules (where it is a file) work too.
- `--global` opts back into the shared store, `agent-memory stats` prints
which store is in use, and a one-time stderr notice points at existing
global memories so upgrading does not look like data loss. It goes to
stderr because on the MCP server stdout carries the protocol.
Measured the sentence-transformers backend
- It shipped unmeasured with a guessed relevance floor. Now scored on the
same benchmark and published in results_sentence_transformers.md:
hashing MiniLM
developer phrasing 0.93 0.86
paraphrase 0.43 0.79
off-topic rejected 6/12 12/12
Not a clean sweep, which is the interesting part: exact word matching
genuinely wins when the words match. MiniLM wins where real use lives.
The README now recommends the `real` extra for daily use and keeps
hashing as the offline/CI default.
- Calibrated its floor from the sweep plus observed behaviour: off-topic
queries are fully rejected by 0.15 and recall is flat to 0.35, so the
sweep alone cannot choose. Set to 0.20, just under the 0.22 scored by a
real paraphrase against the memory that answers it. The previous 0.25
guess would have rejected it.
- Fixed a latent break: sentence-transformers 5.x renamed
get_sentence_embedding_dimension. Both names are now supported — the
same failure mode as the mcp 1.x/2.x rename.
- eval/run_eval.py takes --embedder and writes the two runs to separate
files, so a machine with the model installed cannot overwrite the
results CI regenerates and diffs.
Tests 65 -> 74, plus 9 sentence-transformers tests that skip without the
extra. Verified over a real JSON-RPC stdio session: 8 tools, paraphrased
query resolved, off-topic rejected, no protocol corruption.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 918992997d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| | Developer phrasing — 40% word overlap | **0.93** | 0.86 | | ||
| | Outsider paraphrase — 3% word overlap | 0.43 | **0.79** | | ||
| | Average of the two | 0.68 | **0.82** | | ||
| | Off-topic queries correctly rejected | 6/12 | **12/12** | |
There was a problem hiding this comment.
The evaluation runs only four off-topic queries, while min_score_sweep counts returned memories and uses k * len(off_topic) (12 candidate slots) as the denominator. Consequently, 6/12 and 12/12 cannot represent “queries correctly rejected”; this incorrectly suggests that twelve off-topic questions were tested and does not reveal how many queries returned no memories. Report a query-level count or relabel the row as rejected candidate-memory slots.
Useful? React with 👍 / 👎.
The page had grown by accretion — heavy evaluation analysis before the reader knew what the thing was, quickstart buried two thirds down, and install instructions in three different places. Reordered to try it -> understand it -> evidence -> reference: - Quick start first, with a real terminal session whose output is copied from an actual run: a paraphrased query that shares no words with the memory it finds, and an off-topic query that returns nothing. - One install table replacing three scattered pip lines. - "How it works" as a three-row tool table before the sequence diagram. - Evaluation consolidated under "Does it actually work?", keeping the random control, the embedder comparison and the caveats. - New Reference section: CLI commands, memory types, environment variables, Python API, migration — previously scattered or unwritten. - Added CI/Python/licence badges. Every number in the page is checked against eval/results*.json, the shipped floors, the test count and the tool count. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The two changes from the "what would make this actually work day to day" list. (#3, session hooks, deliberately left for later.)
1. Memories were shared across every project
The default store was a single global
~/.agent_memory/store.json, and all three MCP configs in the README pointed at it. Since recall matches on similarity alone, your booking app's answer to "how do we deploy?" could surface while you're working on your portfolio site. Invisible on day one, confusing by month three.Resolution order is now:
AGENT_MEMORY_PATHif set.agent_memory/store.jsonin the enclosing git repo — the normal case~/.agent_memory/store.jsonoutside a repo.gitis tested for existence, not directory-ness, so worktrees and submodules work.--globalopts back in,agent-memory statsprints the active store, and a one-time notice points at existing global memories so upgrading doesn't look like data loss — on stderr, because stdout carries the MCP protocol.2. The semantic backend shipped unmeasured
SentenceTransformerEmbedderexisted but had never been scored, and its relevance floor was a guess. Now measured on the same benchmark:It isn't a clean sweep, and that's the useful finding. Exact word matching genuinely wins when the words match — hashing is better and needs no model download. But real use is the second row: you write a memory in March and ask in July, in different words. The README now recommends the
realextra for daily use and keeps hashing as the offline/CI default.The floor was recalibrated against evidence, not just the sweep. Off-topic queries are fully rejected from 0.15 and labelled recall is flat to 0.35, so the sweep alone can't pick a value. Probing real paraphrases found "which AI model answers customer questions" scoring 0.22 against the memory that answers it — which the previous 0.25 guess would have thrown away. Floor set to 0.20.
Also fixed a latent break:
sentence-transformers5.x renamedget_sentence_embedding_dimension. Both names now work — the same failure mode as themcp1.x/2.x rename that broke the server before.eval/run_eval.pytakes--embedderand writes the two runs to separate files, so a machine with the model installed can't overwrite the results CI regenerates and diffs.Verification
memory_bootresolved a paraphrased query correctly, off-topic returned nothing, clean exit, zero protocol corruptionKnown limits, now stated in the README
Nothing writes memories for you yet — the agent must choose to call
memory_write/memory_handoff. That's the next piece, and it's what session hooks would fix.🤖 Generated with Claude Code