Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@
},
"metadata": {
"description": "Official Basic Memory plugins from the canonical basic-memory repository",
"version": "0.22.1"
"version": "0.23.0"
},
"plugins": [
{
"name": "basic-memory",
"source": "./plugins/claude-code",
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
"version": "0.22.1",
"version": "0.23.0",
"author": {
"name": "Basic Machines"
},
Expand Down
26 changes: 24 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Unreleased

## v0.23.0 (2026-08-XX)
## v0.23.0 (2026-08-23)

Semantic search grows up and concurrent writes stop deadlocking. Search gains
opt-in cross-encoder reranking, pluggable vector indexes with a first-party
Expand Down Expand Up @@ -229,6 +229,11 @@ search entries.
- **#1073**: `edit_note` with a `memory://` URL routes to the target
project instead of creating phantom notes (**#1066**), and scoped
`memory://` URL paths are preserved (**#1092**).
- **#1299**: `build_context` bounds its traversal requests — primary pages
cap at 50 results, related context at 100 per page, and negative
`max_related` is rejected instead of acting as an unbounded SQL limit —
so a traversal call can no longer become a multi-megabyte vault export
(**#1296**). Callers paginate or follow returned `memory://` links.
- **#1285**: Glob-filtered directory listings traverse subdirectories
again: `file_name_glob` filters results instead of accidentally pruning
recursion, so `*.md` with depth 2 finds files inside subdirectories.
Expand Down Expand Up @@ -261,6 +266,15 @@ search entries.
`create_time` — timestamps fall back to `update_time`, then the earliest
message time, so the whole archive imports (**#1276**); undecodable
import uploads return a 400 with the parse error instead of a 500.
- **#1298**: The update check no longer reports "up to date" when the
Homebrew probe fails, no longer auto-runs `brew upgrade` for an update
inferred from PyPI metadata, and names the actual Homebrew target
version it will install.
- **#1300 / #1302**: The multilingual-E5 prefix contract is documented —
asymmetric FastEmbed models need `semantic_embedding_query_prefix` /
`semantic_embedding_document_prefix`, which already drive reindex through
the embedding identity — along with custom-model selection, the installed
FastEmbed catalog, and dimensions (**#1264**).
- **#1058 / #1094**: `bm doctor` never prints a blank failure message, and
migrations adapt to existing event loops (**#1027**).
- **#1080**: Loading config no longer recreates an empty `~/basic-memory`
Expand All @@ -279,7 +293,9 @@ search entries.
registration APIs instead of private PluginManager writes, preserving the
lifecycle ownership that cleans them up on provider unload (**#1257**);
the slash-command monkeypatch docs are split by Hermes version so modern
installs skip the legacy collector workaround (**#1278**).
installs skip the legacy collector workaround (**#1278**), and the install
docs require a Hermes release with managed manifest-v2 installation
instead of the nonexistent `--path` option (**#1280**, **#1303**).

### Maintenance

Expand All @@ -292,6 +308,12 @@ search entries.
- Kept Milvus as a first-party optional vector backend while removing the
unused Python entry-point registry for separately packaged vector
adapters.
- **#1292 / #1293**: The dev-release pipeline publishes again via PyPI
trusted publishing — its version gate imported a hardcoded module version
and had silently skipped every dev publish since the 0.22.1 bump.
- **#1304 / #1307**: The concurrent-write convergence benchmark is ported
into the canonical `/benchmarks` package, and the read-load benchmark
runs against the current MCP SDK's typed result fields.

## v0.22.1 (2026-06-12)

Expand Down
13 changes: 6 additions & 7 deletions benchmarks/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ directory for comparing Basic Memory against other memory systems.

Primary goals:
- Deterministic retrieval benchmarks
- Optional LLM-as-a-judge benchmarks
- End-to-end QA scoring with fixed answerer and judge models
- Public, reproducible artifact publication (including provenance metadata)

The benchmark package keeps its own `pyproject.toml` and lockfile so benchmark
Expand All @@ -16,7 +16,6 @@ dependencies do not pollute the Core product environment.
## Build / Test Commands

- Install: `uv sync --group dev`
- Install judge extras: `uv sync --group dev --extra judge`
- Run tests: `uv run pytest -q`
- Lint: `uv run ruff check .`
- Type check: `uv run pyright`
Expand All @@ -37,8 +36,8 @@ Dataset and conversion:
Run retrieval:
- `uv run bm-bench run retrieval --providers bm-local,mem0-local --dataset-id locomo --dataset-path benchmarks/datasets/locomo/locomo10.json --corpus-dir benchmarks/generated/locomo/docs --queries-path benchmarks/generated/locomo/queries.json --output-root benchmarks/runs --allow-provider-skip`

Run judge (optional):
- `uv run bm-bench run judge --run-dir benchmarks/runs/<run-id>`
Run end-to-end QA scoring:
- `uv run bm-bench run qa --run-dir benchmarks/runs/<run-id> --answerer claude:claude-haiku-4-5 --judge claude:claude-sonnet-4-6`

Validate and publish:
- `uv run bm-bench validate-artifacts --run-dir benchmarks/runs/<run-id>`
Expand All @@ -53,15 +52,15 @@ Validate and publish:
- `just bench-run-bm-local`
- `just bench-run-mem0-local`
- `just bench-run-full`
- `just bench-judge RUN_DIR=benchmarks/runs/<run-id>`
- `just bench-publish RUN_DIR=benchmarks/runs/<run-id>`
- `just bench-qa benchmarks/runs/<run-id>`
- `just bench-publish benchmarks/runs/<run-id>`

## Repository Layout

- `src/basic_memory_benchmarks/cli.py` - CLI surface
- `src/basic_memory_benchmarks/runner.py` - run orchestration
- `src/basic_memory_benchmarks/providers/` - provider adapters (`bm-local`, `bm-cloud`, `mem0-local`, `zep-reference`)
- `src/basic_memory_benchmarks/scoring/` - retrieval + judge scoring
- `src/basic_memory_benchmarks/scoring/` - retrieval + end-to-end QA scoring
- `src/basic_memory_benchmarks/reporting/` - artifact writers / comparison helpers
- `src/basic_memory_benchmarks/converters/` - dataset conversion logic
- `src/basic_memory_benchmarks/datasets/` - dataset fetch/load helpers
Expand Down
25 changes: 4 additions & 21 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ dependencies remain isolated from the product environment.
## Goals

- Deterministic retrieval benchmarks (Recall@5/10, MRR, Precision@5, content-hit, latency)
- Optional LLM-as-judge scoring (Pydantic Evals)
- End-to-end QA scoring with fixed answerer and judge models
- Public artifacts with provenance and reproducibility metadata
- Clean dependency isolation from the core Basic Memory package

Expand All @@ -29,12 +29,6 @@ dependencies remain isolated from the product environment.
uv sync --group dev
```

Optional judge dependencies:

```bash
uv sync --group dev --extra judge
```

## Quickstart

### 1) Fetch LoCoMo dataset
Expand Down Expand Up @@ -85,16 +79,7 @@ know") when the retrieved memories don't contain the answer, and abstention is
graded correct only when the gold answer marks the question unanswerable
(LoCoMo adversarial cases).

### 5) Optional retrieval-context judge (legacy)

Scores whether the *retrieved context* contains the expected answer, without
answer generation:

```bash
uv run bm-bench run judge --run-dir benchmarks/runs/<run-id>
```

### 6) Publish run artifacts
### 5) Publish run artifacts

```bash
uv run bm-bench publish --run-dir benchmarks/runs/<run-id>
Expand Down Expand Up @@ -290,8 +275,6 @@ Per run (`benchmarks/runs/<run-id>/`):
- `retrieval-summary.json`
- `per-query-qa.jsonl` (optional)
- `qa-summary.json` (optional)
- `per-query-judge.jsonl` (optional)
- `judge-summary.json` (optional)
- `summary.md`

## Just commands
Expand All @@ -303,8 +286,8 @@ just bench-convert-locomo
just bench-run-bm-local
just bench-run-mem0-local
just bench-run-full
just bench-judge
just bench-publish RUN_DIR=benchmarks/runs/<run-id>
just bench-qa benchmarks/runs/<run-id>
just bench-publish benchmarks/runs/<run-id>
```

## Notes on dataset publication
Expand Down
40 changes: 19 additions & 21 deletions benchmarks/docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ It covers:

| Area | Status |
| --- | --- |
| Single run execution (`run retrieval`, `run full`, `run judge`) | Implemented |
| Single run execution (`run retrieval`, `run full`, `run qa`) | Implemented |
| Concurrent write convergence (`run concurrent-write`) | Implemented |
| `just` one-command pipelines (`bench-full`, `bench-full-judge`) | Implemented |
| `just` retrieval and QA workflows (`bench-full`, `bench-qa`) | Implemented |
| Artifact generation and publish/compare commands | Implemented |
| Manual BM revision comparison via worktrees + `--bm-local-path` | Implemented workflow, manual orchestration |
| `bm-bench run revision-matrix` | Planned, not implemented yet |
Expand Down Expand Up @@ -58,12 +58,6 @@ cd /path/to/basic-memory/benchmarks
just sync
```

If you plan to run judge metrics:

```bash
just sync-judge
```

### Dataset assumptions

LoCoMo source and converted outputs are created by:
Expand All @@ -78,15 +72,14 @@ just bench-prepare-long
### `just` commands (current)

- `bench-full`
- `bench-full-judge`
- `bench-qa`
- `bench-concurrent-write-smoke`
- `bench-concurrent-write-load`
- `bench-prepare-short`
- `bench-prepare-long`
- `bench-run-short`
- `bench-run-long`
- `bench-run-full`
- `bench-judge`
- `bench-validate`
- `bench-publish`
- `bench-compare`
Expand All @@ -101,7 +94,9 @@ Top-level commands:
- `run retrieval`
- `run concurrent-write`
- `run full`
- `run judge`
- `run qa`
- `run rejudge`
- `run review`
- `compare`
- `validate-artifacts`
- `publish`
Expand All @@ -120,17 +115,15 @@ This runs:
2. `just bench-prepare-long`
3. `just bench-run-full`

### One-command full retrieval + judge
### End-to-end QA scoring

```bash
cd /path/to/basic-memory/benchmarks
just bench-full-judge
just bench-qa benchmarks/runs/<run_id>
```

This runs:
1. `just sync-judge`
2. `just bench-prepare-long`
3. `just bench-run-full-judge`
This generates answers from each provider's retrieved context, applies the
same judge to every provider, and writes QA artifacts into the retrieval run.

### Short vs long workflows

Expand Down Expand Up @@ -196,10 +189,15 @@ Required files:
- `retrieval-summary.json`
- `summary.md`

Optional judge files:
Optional QA files:

- `per-query-judge.jsonl`
- `judge-summary.json`
- `per-query-qa.jsonl`
- `qa-summary.json`
- `per-query-qa-rejudge.jsonl`
- `qa-rejudge-summary.json`
- `qa-rejudge-flips.json`
- `review.html`
- `qa-diagnosis.json`

### Key provenance fields

Expand Down Expand Up @@ -483,7 +481,7 @@ Dry-run checks:
just --dry-run bench-run-short
just --dry-run bench-run-long
just --dry-run bench-full
just --dry-run bench-full-judge
just --dry-run bench-qa benchmarks/runs/<run_id>
```

Artifact field checks:
Expand Down
29 changes: 5 additions & 24 deletions benchmarks/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,6 @@ longmemeval_dev_output_dir := "benchmarks/generated/longmemeval-s-dev"
sync:
uv sync --group dev

sync-judge:
uv sync --group dev --extra judge

test:
uv run pytest -q

Expand Down Expand Up @@ -110,13 +107,6 @@ bench-full:
just bench-prepare-long
just bench-run-full

# Full retrieval + judge pipeline:
# 1) sync deps (+judge extras), 2) fetch+convert long dataset, 3) run full with judge
bench-full-judge model="gpt-4o-mini":
just sync-judge
just bench-prepare-long
just bench-run-full-judge model="{{model}}"

# --- Benchmark execution ---

bench-smoke:
Expand Down Expand Up @@ -198,18 +188,6 @@ bench-run-full:
{{bm_local_path_flag}} \
--allow-provider-skip

bench-run-full-judge model="gpt-4o-mini":
uv run bm-bench run full \
--dataset-id locomo \
--dataset-path {{locomo_dataset_path}} \
--corpus-dir benchmarks/generated/locomo/docs \
--queries-path benchmarks/generated/locomo/queries.json \
--providers bm-local,mem0-local \
{{bm_local_path_flag}} \
--allow-provider-skip \
--judge \
--judge-model "{{model}}"

# --- Concurrency benchmark (basic-memory#1248) ---

# Small-scale smoke: 4 writers x 25 notes; strict so divergence fails the command
Expand Down Expand Up @@ -244,8 +222,11 @@ bench-latest-run:
set -euo pipefail
ls -1dt benchmarks/runs/* | head -n 1

bench-judge run_dir model="gpt-4o-mini":
uv run bm-bench run judge --run-dir "{{run_dir}}" --model "{{model}}"
bench-qa run_dir answerer="claude:claude-haiku-4-5" judge="claude:claude-sonnet-4-6":
uv run bm-bench run qa \
--run-dir "{{run_dir}}" \
--answerer "{{answerer}}" \
--judge "{{judge}}"

bench-validate run_dir:
uv run bm-bench validate-artifacts --run-dir "{{run_dir}}"
Expand Down
5 changes: 0 additions & 5 deletions benchmarks/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,6 @@ dependencies = [
"typer>=0.16.1",
]

[project.optional-dependencies]
judge = [
"pydantic-evals>=0.4.0",
]

[project.scripts]
bm-bench = "basic_memory_benchmarks.cli:main"
basic-memory-benchmarks = "basic_memory_benchmarks.cli:main"
Expand Down
Loading
Loading