diff --git a/.planning/PROJECT.md b/.planning/PROJECT.md index 2782c15..5ebdb9c 100644 --- a/.planning/PROJECT.md +++ b/.planning/PROJECT.md @@ -39,6 +39,12 @@ An agent can persist a structured memory and retrieve the right one later — ac - ✓ Pre-built binaries (macOS arm64/x86_64, Linux x86_64/aarch64 gnu) + Homebrew formula — Phase 2 - ✓ Input validation at the shared MemoryService seam — limit (1–200) and ttl_secs (1–3155760000) rejected as REST 400 / MCP invalid_params, never 500, never a silent clamp (API-02) — Validated in Phase 3 (v1.1) - ✓ Exact tag matching via json_each equality on all three query paths — 'rustling' no longer matches tag=rust (API-03, behavioral change recorded for v0.1.0 notes) — Validated in Phase 3 (v1.1) +- ✓ `memory_update` merge-patch with same-transaction FTS re-mirror + vector re-embed; unknown id is clean not-found data (MCP-06) — Validated in Phase 4 (v1.1) +- ✓ Flat typed relations `memory_link`/`memory_unlink` (relates_to symmetric, supersedes/caused_by directed), opt-in 1-hop expansion, cascade on forget/TTL (MCP-07) — Validated in Phase 4 (v1.1) +- ✓ Migration 0003 with hygiene: frozen 0001/0002 fixture test, pre-migration online backup, friendly newer-schema refusal (STORE-05) — Validated in Phase 4 (v1.1) +- ✓ Decay-aware hybrid RRF search (`search_mode: "hybrid"`, k=60), keyword degrade byte-identical to v1.0, forgiving FTS5 query handling (SEARCH-04, SEARCH-05) — Validated in Phase 4 (v1.1) +- ✓ `agent-memory export` writes format-v1 JSONL (versioned header with embedding model/dim and counts, memory + link records keyed by export-local ids, no vectors), stdout or atomic 0600 `--output` (DIST-04) — Validated in Phase 5 (v1.1) +- ✓ `agent-memory import --from jsonl` restores idempotently into empty or non-empty databases with key→id link remapping, expired rows skipped and counted, rows landing `embedding_status = 0` for the sweep backfill, malformed files rejected as `InvalidArgument("line N: …")` with nothing written (INTEROP-02) — Validated in Phase 5 (v1.1) ### Active @@ -53,7 +59,7 @@ An agent can persist a structured memory and retrieve the right one later — ac ## Context -- **Current state (v1.1 Phase 3 complete, 2026-07-12):** API hardening landed on the shipped v0.0.1 codebase (validation seam + exact tag matching, 7/7 must-haves verified); CI revived on GitHub-hosted `ubuntu-latest` (D-12); cross-compile spike answered the milestone's feasibility unknown — all 4 legs GREEN, Phase 6 adopts cargo-xwin 0.23.0/msvc with mingw-w64/gnu fallback and the musl CFLAGS shim proven. Next: Phase 4 (memory_update, relations, hybrid RRF). +- **Current state (v1.1 Phase 5 complete, 2026-10-25):** Phases 3–5 landed on the shipped v0.0.1 codebase: validation seam + exact tag matching (Phase 3); `memory_update`, flat typed relations, migration 0003 hygiene and decay-aware hybrid RRF (Phase 4, 28/28 must-haves); portable JSONL export/import with format v1 frozen, plus storage-error redaction on every service path (Phase 5, 35/35 must-haves, live-Ollama round trip observed). CI on GitHub-hosted `ubuntu-latest` (D-12); release pipeline on GitHub-hosted runners with SLSA provenance (quick task 261009-hc1). Suite: 287 passed / 0 failed / 2 ignored, 95.6 % line coverage. Next: Phase 6 (musl + Windows binaries on the spike-chosen toolchains, v0.1.0 release). - **v1.0 baseline (shipped 2026-07-12):** v0.0.1 released publicly — GitHub Release with 4 checksummed target tarballs (macOS arm64/x86_64, Linux x86_64/aarch64 gnu) + Homebrew tap (`brew install unityinflow/tap/agent-memory`). ~5,400 LOC Rust across `agent-memory-core` + `agent-memory` crates; clippy `-D warnings` clean; >80% coverage CI gate; 25/25 STRIDE threats closed. Known v2 debt: musl binaries (sqlite-vec BSD typedefs), Windows (cfg(unix) refactor), REST boundary-value hardening, SEARCH-04 hybrid RRF, MCP-06 update/relations, DIST-04 export. - **Ecosystem position:** Tool 10 of 20 in UnityInFlow. Phase 3. Its own `CLAUDE.md` marks it *"Planned — no strict blocking dependencies,"* so it can proceed in parallel with the active Phase 2 close-out (budget-breaker starter, kore v0.1.0). - **Why it exists:** Every agent tool reinvents state storage (GSD has STATE.md, Superpowers has skill context, RTK has its own SQLite DB). Switching runtimes loses all project context. A standard MCP memory API works across tools and survives sessions. @@ -91,6 +97,10 @@ An agent can persist a structured memory and retrieve the right one later — ac | Two-tier error taxonomy at the store seam (InvalidQuery → 400/invalid_params; internal → 500/internal_error) | Bad client input must never read as a server fault, on either transport | ✓ Good — proven live at 4 layers (02-05) | | Phase 6 Windows toolchain: cargo-xwin 0.23.0/msvc first, mingw-w64/gnu as proven fallback; musl via zigbuild + `-Du_int*_t` CFLAGS shim | Phase 3 spike ran all 4 legs GREEN in one hosted dispatch (run 29196226289) — feasibility unknown answered before distribution work | ✓ Verdict recorded in spike-cross-compile.yml header | | CI on GitHub-hosted `ubuntu-latest` for this public repo (ecosystem D-02 exception) | OPS-02 makes org self-hosted runners unreachable from public repos; the gate was silently dead | ✓ Good — first green runs since going public (D-12) | +| Export/import are CLI-only in v0.1.0; malformed files are `InvalidArgument` in core (Phase 5 D-01/D-11) | An MCP export would route the corpus through the JSON-RPC stdout channel and a REST import is an arbitrary-file-read primitive; keeping the error in core preserves the two-tier taxonomy for any later transport | ✓ Good — strict all-or-nothing parser, nothing written on failure, 25-row malformed table | +| JSONL format v1 is a frozen one-way contract: header (format/version/model/dim/schema/counts), memory and link records, derived fields and vectors excluded (Phase 5 D-03..D-06) | A shipped tool's export is a compatibility contract from the first file written; vectors are model-pinned derived data the sweep rebuilds for free | ✓ Good — goldens pin the wire order; any change needs `version: 2` + a reader for 1 | +| Import identity stays `(source, mem_type, content)`, links remap through a key→id map, created_at/expires_at/access_count preserved, last_accessed refreshed, expired rows skipped-and-counted, rows land pending and never embed inline (Phase 5 D-07..D-10, D-13) | Zero new schema; a preserved stale `last_accessed` would make a restore unfindable by rank; one re-embed mechanism, zero mixing risk | ✓ Good — review-driven hardening: post-remap self-links dropped, `access_count` bounded + saturating bump, dedup probe ignores expired-but-unswept rows | +| Storage-engine error text is redacted at the `MemoryService` seam on every public method, both transports and CLI (Phase 5 D-14, closes 04-REVIEW IN-01) | One seam beats two transport mappers; `SEARCH_STORAGE_FAILURE` text kept byte-identical so Phase 4 goldens hold | ✓ Good — 10 fault-injection tests incl. export/import, `/health`, MCP | ## Evolution @@ -110,4 +120,4 @@ This document evolves at phase transitions and milestone boundaries. 4. Update Context with current state --- -*Last updated: 2026-07-12 after v1.1 Phase 3 (API hardening & toolchain spikes complete)* +*Last updated: 2026-10-25 after v1.1 Phase 5 (portable export & import complete; Phase 4 validations recorded retroactively)* diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 128f285..08e4719 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -29,8 +29,8 @@ Each maps to a roadmap phase. IDs continue from the v1.0 numbering (archived in ### Portability (DIST / INTEROP) -- [ ] **DIST-04**: User can export all memories to a versioned JSONL file (header records format version + embedding model/dim; embeddings excluded as derived data; relation edges included with export-local key remapping) -- [ ] **INTEROP-02**: User can import that JSONL on another machine — idempotent re-run, imported rows land `embedding_status = 0` and re-embed via the existing sweep backfill, expired items skipped and counted +- [x] **DIST-04**: User can export all memories to a versioned JSONL file (header records format version + embedding model/dim; embeddings excluded as derived data; relation edges included with export-local key remapping) +- [x] **INTEROP-02**: User can import that JSONL on another machine — idempotent re-run, imported rows land `embedding_status = 0` and re-embed via the existing sweep backfill, expired items skipped and counted ### Distribution (DIST) @@ -69,8 +69,8 @@ Mapped by roadmap creation 2026-07-12. Coverage: 12/12 v1.1 requirements mapped | STORE-05 | Phase 4 | Complete | | SEARCH-04 | Phase 4 | Complete | | SEARCH-05 | Phase 4 | Complete | -| DIST-04 | Phase 5 | Pending | -| INTEROP-02 | Phase 5 | Pending | +| DIST-04 | Phase 5 | Complete | +| INTEROP-02 | Phase 5 | Complete | | DIST-05 | Phase 6 | Pending | | DIST-03 | Phase 6 | Pending | | DIST-06 | Phase 6 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index cb3db38..4daf72e 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -28,7 +28,7 @@ Full phase details: [milestones/v1.0-ROADMAP.md](milestones/v1.0-ROADMAP.md) - [x] **Phase 3: API Hardening & Toolchain Spikes** - Out-of-bounds limit/ttl rejected as client errors and exact tag matching at the shared service seam, plus early Windows/musl cross-compile spikes answering the milestone's only feasibility unknown (completed 2026-07-12) - [x] **Phase 4: Memory Update, Relations & Hybrid Search** - `memory_update` with same-transaction re-embed, flat typed links with cascade, migration 0003 hygiene, and decay-aware hybrid RRF as a parallel track (completed 2026-10-23) -- [ ] **Phase 5: Portable Export & Import** - Versioned JSONL export (link edges included, embeddings excluded) and idempotent cross-machine import with sweep-backfill re-embedding +- [x] **Phase 5: Portable Export & Import** - Versioned JSONL export (link edges included, embeddings excluded) and idempotent cross-machine import with sweep-backfill re-embedding (completed 2026-10-25) - [ ] **Phase 6: Distribution & v0.1.0 Release** - musl + Windows binaries on the spike-chosen toolchains and the checksummed v0.1.0 release with Homebrew update and behavioral-change release notes ## Phase Details @@ -104,7 +104,28 @@ Plans: 3. Imported rows land with `embedding_status = 0` and become semantically searchable after the existing sweep backfill re-embeds them — no embedding vectors travel in the file. 4. Expired items in the file are skipped and counted, and the import report states imported/skipped/duplicate counts; malformed files (bad header/version/rows) are rejected as client errors through the Phase 3 seam, never a 500 or crash. -**Plans**: TBD +**Plans**: 5/5 plans executed (4 executed + 1 gap closure) + +Plans: +**Wave 1** + +- [x] 05-01-PLAN.md — D-14 storage-error redaction on every service path (tracer: seam → REST `/health` + `POST` → MCP → README) + the store contract both later tracers consume: `Store::find_by_key`, `LinkOutcome::Linked { created }`, `ImportedMemory` / `Store::insert_imported` (INTEROP-02) + +**Wave 2** *(blocked on Wave 1)* + +- [x] 05-02-PLAN.md — `agent-memory export`: format v1 `Record` enum + golden tests, one-snapshot `Store::export_rows`, streaming `MemoryService::export`, stdout default + atomic 0600 `--output` (DIST-04) + +**Wave 3** *(blocked on Wave 2)* + +- [x] 05-03-PLAN.md — `agent-memory import --from jsonl`: strict two-pass `parse_jsonl` + `import_jsonl` with key→id remapping, expiry skip, pending re-embed, six-count report; PITFALLS 18/19 kill-tests + malformed-file table (INTEROP-02, DIST-04) + +**Wave 4** *(blocked on Wave 3)* + +- [x] 05-04-PLAN.md — README "Export & import" section (format v1 spec, timestamp policy, pending embeddings, backup-vs-export) + spawned-binary CLI round trip (DIST-04, INTEROP-02) + +**Gap closure** *(from 05-VERIFICATION.md `human_verification` items — converts both human items into explicit, re-runnable evidence)* + +- [x] 05-05-PLAN.md — Gap wave 5: held-out race test for the 05-02 one-snapshot backstop truth (`a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key`: a memory + link committed mid-export are invisible to that snapshot and present in the next) + recorded live-Ollama realism run (export A → import B → first-sweep backfill → hybrid paraphrase search, `05-05-realism.sh`) (DIST-04, INTEROP-02) ### Phase 6: Distribution & v0.1.0 Release @@ -131,5 +152,5 @@ Phases execute in numeric order: 3 → 4 → 5 → 6 | 2. Semantic Search, Interop & Release | v1.0 | 5/5 | Complete | 2026-07-12 | | 3. API Hardening & Toolchain Spikes | v1.1 | 2/2 | Complete | 2026-07-12 | | 4. Memory Update, Relations & Hybrid Search | v1.1 | 7/7 | Complete | 2026-10-23 | -| 5. Portable Export & Import | v1.1 | 0/? | Not started | - | +| 5. Portable Export & Import | v1.1 | 5/5 | Complete | 2026-10-25 | | 6. Distribution & v0.1.0 Release | v1.1 | 0/? | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index ddb4e59..5428776 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,19 +2,19 @@ gsd_state_version: 1.0 milestone: v1.1 milestone_name: Hardening & Interop -current_phase: 5 -current_phase_name: Portable Export & Import +current_phase: 6 +current_phase_name: Distribution & v0.1.0 Release status: planning -stopped_at: Phase 04 complete, ready to plan Phase 5 -last_updated: "2026-10-23T20:25:00.455Z" -last_activity: 2026-10-23 -last_activity_desc: Phase 04 complete, transitioned to Phase 5 -state_head: 4441bdafab5d420a1fbcf3ba348c0589d72bdc11 +stopped_at: Phase 5 complete, ready to plan Phase 6 +last_updated: "2026-10-25T13:51:46.548Z" +last_activity: 2026-10-25 +last_activity_desc: Phase 5 complete, transitioned to Phase 6 +state_head: fe18db796014dc9d885859771ee089ce7ba895df progress: total_phases: 4 - completed_phases: 2 - total_plans: 9 - completed_plans: 9 + completed_phases: 3 + total_plans: 14 + completed_plans: 14 --- # Project State @@ -24,14 +24,14 @@ progress: See: .planning/PROJECT.md (updated 2026-07-12) **Core value:** An agent can persist a structured memory and retrieve the right one later — across sessions and tools — over a standard MCP interface, with no cloud. -**Current focus:** Phase 04 — Memory Update, Relations & Hybrid Search +**Current focus:** Phase 5 — Portable Export & Import ## Current Position -Phase: 5 — Portable Export & Import +Phase: 6 — Distribution & v0.1.0 Release Plan: Not started Status: Ready to plan -Last activity: 2026-10-23 - Completed quick task 261023-v6w: WR-04: reject memory_list expand_links without limit +Last activity: 2026-10-25 — Phase 5 complete, transitioned to Phase 6 Progress: [░░░░░░░░░░] 0% (0/4 v1.1 phases) @@ -39,7 +39,7 @@ Progress: [░░░░░░░░░░] 0% (0/4 v1.1 phases) **Velocity:** -- Total plans completed: 17 +- Total plans completed: 22 - Average duration: ~28 min - Total execution time: ~1.4 hours @@ -51,6 +51,7 @@ Progress: [░░░░░░░░░░] 0% (0/4 v1.1 phases) | 02 | 5 | - | - | | 03 | 2 | - | - | | 04 | 7 | - | - | +| 5 | 5 | - | - | **Recent Trend:** @@ -76,6 +77,11 @@ Progress: [░░░░░░░░░░] 0% (0/4 v1.1 phases) | Phase 04 P05 | 10 min | 3 tasks | 12 files | | Phase 04 P06 | 7 min | 3 tasks | 9 files | | Phase 04 P07 | 12 min | 3 tasks | 7 files | +| Phase 05 P01 | 14 min | 3 tasks | 10 files | +| Phase 05 P02 | 13 min | 2 tasks | 8 files | +| Phase 05 P03 | 16min | 2 tasks | 6 files | +| Phase 05 P04 | 9 min | 2 tasks | 2 files | +| Phase 05 P05 | 6 min | 2 tasks | 2 files | ## Accumulated Context @@ -123,6 +129,19 @@ Recent decisions affecting current work: - [Phase 04]: 04-07: deny_unknown_fields added to UpdateArgs/UpdateBody only (new v0.1.0 surfaces, REVIEW WR-02); the [03-01] decision stands for the v0.0.1 DTOs, so the v0.1.0 release-notes delta for existing surfaces is unchanged. Over MCP, rmcp 1.8 reports the rejection as a tool error result (isError: true, "failed to deserialize parameters: unknown field ..."), not a JSON-RPC -32602; REST is a 400 {"error"}. - [Phase 04]: 04-07: backup temp is pid-qualified and a process-wide BACKUP_LOCK serializes backups within one process, so same-process openers never share a temp; prune tolerates NotFound from a concurrent opener - [Phase 04]: 04-07: WR-01 closed with verify-then-rename + tolerate-by-re-probe (no file lock, no new dep); cross-process exclusive lock stays deferred +- [Phase 05-01]: redact_storage_error(op, e) at the MemoryService seam on all nine methods (store, list, search, import, forget, link, unlink, update, sweep) via *_unredacted inner fns; transport mappers byte-identical; closes 04-REVIEW IN-01 (D-14) +- [Phase 05-01]: Store::exists -> Store::find_by_key(Option) with the same NULL-safe (source, mem_type, content) key (D-07); LinkOutcome::Linked { created: bool } from tx.changes() across 7 sites, storage-only not a wire change (D-12) +- [Phase 05-01]: ImportedMemory + Store::insert_imported (no embedding parameter, always embedding_status 0, last_accessed = now, created_at/access_count/expires_at preserved) behind one shared insert_row helper; importer skips expires_at <= now, sweep deletes < now (D-10, D-13) +- [Phase 5]: [Phase 05-02]: Format v1 frozen in export.rs as one strict tagged Record enum (record first, declaration order, deny_unknown_fields; header carries exported_rows/exported_links); wire order golden-tested; D-03 one-way rating recorded as pre-approved +- [Phase 5]: [Phase 05-02]: Store::export_rows streams every row from ONE pooled connection + ONE read transaction (rusqlite unchecked_transaction) — meta pin, counts, memories ORDER BY id, links ORDER BY from_id,to_id,kind — via a visitor, never a Vec; MemoryService::export flushes inside spawn_blocking and redacts with op export (tenth op string) +- [Phase 5]: [Phase 05-02]: Research correction: stdout export sink is BufWriter::new(std::io::stdout()) — StdoutLock is !Send and cannot be a Box; --output publishes via export_file::PendingExport ({target}.export-tmp-{pid} beside the target, create_new + 0600, sync_all then rename, one abort covers every failure) +- [Phase 5]: 05-03: jsonl importer is add-alongside beside gsd-state — two parser modules, two tolerance policies (strict vs tolerant), shared seam/dedup/writer lane; a third format forces an ImportSource enum +- [Phase 5]: 05-03: D-11 messages — HeaderProbe format/version are Option (non-header line 1 → 'first record must be the header'), serde's ' at line 1 column N' suffix stripped, InvalidArgument payload lifted before the 'line N:' prefix +- [Phase 5]: 05-03: import_jsonl runs passes 2+3 in ONE spawn_blocking hop with per-record writer transactions; parse_jsonl is the all-or-nothing gate; the jsonl path never calls the embedder +- [Phase 05]: 05-04: README Export & import section is one ## with ten ### subsections in plan order; sweep cadence documented as the shipped 'then hourly' (SWEEP_INTERVAL 3600 s); tests/cli.rs is realism-only (zero coverage by design, like tests/rest.rs) and pins the exact report strings +- [Phase 05]: 05-04: phase-end gates recorded — cargo llvm-cov 95.61% lines (floor 80), D-01 transport grep empty, sql/ + Cargo.lock + .github/ unchanged across all phase-5 commits +- [Phase 5]: [05-05]: Gap 1 (DIST-04 backstop truth 2.8) closed with a channel-ordered race test driving Store::export_rows directly — WAL snapshot isolation proven, no production change; every wait bounded by recv_timeout +- [Phase 5]: [05-05]: Gap 2 (INTEROP-02 live re-embed) closed by a recorded run of the real binary against local nomic-embed-text (05-05-realism.sh, exit 0: backfilled=3 on first sweep tick, search_mode hybrid, target rank 0) — stays a script, never a cargo test ### Pending Todos @@ -164,8 +183,8 @@ Items acknowledged and carried forward: ## Session Continuity -Last session: 2026-10-23T20:12:50.720Z -Stopped at: Phase 04 complete, ready to plan Phase 5 +Last session: 2026-10-25T13:42:17.195Z +Stopped at: Phase 5 complete, ready to plan Phase 6 Resume file: None ## Operator Next Steps diff --git a/.planning/WINDOWS.md b/.planning/WINDOWS.md new file mode 100644 index 0000000..18c30f5 --- /dev/null +++ b/.planning/WINDOWS.md @@ -0,0 +1,35 @@ +--- +schema_version: 1 +open_count: 1 +waived_count: 0 +fixed_count: 0 +total_count: 1 +last_updated: 2026-10-25T12:22:24.215Z +--- + +# Broken Windows Ledger + +> Cross-phase defect register. With `workflow.windows_enforce` enabled, `/gsd-ship` blocks while `open_count > 0`. +> Waive with `gsd-tools windows waive ""` (reason required). +> Mark fixed with `gsd-tools windows fixed `. + +| id | phase | kind | file | line | description | status | reason | recorded_at | resolved_at | +|----|-------|------|------|------|-------------|--------|--------|-------------|-------------| +| 1 | 05 | deviation | crates/agent-memory-core/src/import/jsonl.rs | | 05-03: D-11 message wording differs from the plan's interface contract in three recorded ways (Option probe fields, serde position suffix stripped, InvalidArgument payload lifted) — documented in 05-03-SUMMARY.md | open | | 2026-10-25T12:22:24.215Z | | + +````json +[ + { + "id": 1, + "kind": "deviation", + "phase": "05", + "file": "crates/agent-memory-core/src/import/jsonl.rs", + "line": null, + "description": "05-03: D-11 message wording differs from the plan's interface contract in three recorded ways (Option probe fields, serde position suffix stripped, InvalidArgument payload lifted) — documented in 05-03-SUMMARY.md", + "status": "open", + "reason": "", + "recorded_at": "2026-10-25T12:22:24.215Z", + "resolved_at": null + } +] +```` diff --git a/.planning/phases/05-portable-export-import/05-01-PLAN.md b/.planning/phases/05-portable-export-import/05-01-PLAN.md new file mode 100644 index 0000000..e761c2a --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-01-PLAN.md @@ -0,0 +1,358 @@ +--- +phase: 05-portable-export-import +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - crates/agent-memory-core/src/service.rs + - crates/agent-memory-core/src/domain.rs + - crates/agent-memory-core/src/store/mod.rs + - crates/agent-memory-core/src/store/sqlite.rs + - crates/agent-memory-core/tests/redaction.rs + - crates/agent-memory-core/tests/relations.rs + - crates/agent-memory-core/tests/store.rs + - crates/agent-memory/src/rest/handlers.rs + - crates/agent-memory/src/mcp.rs + - README.md +autonomous: true +requirements: [INTEROP-02] + +estimate: + tokens: 55000 + raw_tokens: 55000 + tasks: 3 + confidence: low + +must_haves: + truths: + - "A storage fault on any MemoryService path — store, list, search, import, forget, link, unlink, update, sweep — reaches the caller only as MemoryError::Internal carrying ` failed due to a storage error (details are in the server log)`; the SQLite detail is logged at error level and never returned; SEARCH_STORAGE_FAILURE is byte-identical to the op = search text so every Phase 4 golden assertion still holds (D-14)" + - "GET /health and POST /api/memories against a store whose memories table is offline return HTTP 500 with a JSON error body containing no `sqlite` substring, and memory_store over MCP returns internal_error with no `sqlite` substring — one change at the service seam covers REST, MCP and the CLI (D-14, closes 04-REVIEW IN-01)" + - "Store::find_by_key returns Some(id) of the row matching (source, mem_type, content) with NULL-safe source equality (source IS ?1, CR-01) and None otherwise; MemoryService::import dedups through it, so a duplicate can later map an export-local key onto the existing row (D-07)" + - "Store::link reports LinkOutcome::Linked { created: true } for a freshly inserted edge and LinkOutcome::Linked { created: false } for a duplicate or a symmetric relates_to re-link; the MCP and REST link JSON is unchanged (D-12 countability)" + - "Store::insert_imported lands a row with preserved created_at, access_count and expires_at, last_accessed equal to import-time now, decay_score 1.0, base_weight from the type pin, embedding_status 0 and no vector row — the one store path a restored memory takes (D-10, D-13)" + - "The README scopes storage-error redaction to every request the daemon serves and every CLI import/export, and lists the redaction as behavioral change 4 for v0.1.0 (D-14)" + artifacts: + - path: "crates/agent-memory-core/src/service.rs" + provides: "redact_storage_error(op, e) applied to every public MemoryService method via the *_unredacted inner-fn shape" + contains: "fn redact_storage_error" + - path: "crates/agent-memory-core/tests/redaction.rs" + provides: "fault-injection proof (ALTER TABLE ... RENAME) for the eight non-search paths" + contains: "failed due to a storage error" + - path: "crates/agent-memory-core/src/domain.rs" + provides: "ImportedMemory (restored-row shape) and LinkOutcome::Linked { created }" + contains: "pub struct ImportedMemory" + - path: "crates/agent-memory-core/src/store/mod.rs" + provides: "Store::find_by_key and Store::insert_imported trait methods" + contains: "fn insert_imported" + - path: "crates/agent-memory-core/src/store/sqlite.rs" + provides: "one shared insert_row helper behind insert and insert_imported; link reads tx.changes()" + contains: "fn insert_row" + - path: "README.md" + provides: "redaction sentence widened to every surface; behavioral change 4" + contains: "failed due to a storage error" + key_links: + - from: "crates/agent-memory-core/src/service.rs public methods" + to: "redact_storage_error" + via: "each public fn delegates to its *_unredacted body and maps the error with the op string" + pattern: "redact_storage_error\\(\"(store|list|search|import|forget|link|unlink|update|sweep)\"" + - from: "crates/agent-memory-core/src/service.rs import()" + to: "Store::find_by_key" + via: "the dedup probe now returns the matching id" + pattern: "find_by_key\\(" + - from: "crates/agent-memory-core/src/store/sqlite.rs link()" + to: "LinkOutcome::Linked { created }" + via: "tx.changes() read before commit" + pattern: "LinkOutcome::Linked \\{ created" + - from: "crates/agent-memory-core/src/store/sqlite.rs insert_imported()" + to: "insert_row" + via: "the same INSERT statement insert() uses, with preserved columns bound" + pattern: "insert_row\\(" + prohibitions: + - statement: "Shipped migrations 0001–0003 and LATEST_SCHEMA_VERSION must not change — this phase ships no migration; the fixture divergence test stays green" + status: resolved + verification: test + - statement: "The (source, mem_type, content) idempotency contract must not change — find_by_key keeps the exact three-column NULL-safe key, no new identity column" + status: resolved + verification: test + - statement: "The existing gsd-state importer's tolerance rules must not change — parse_gsd_state and its tests are untouched" + status: resolved + verification: test +--- + +# Plan 05-01: Redact storage-engine text on every service path; teach the store the format-v1 contract + +## Phase Goal + +**As a** user, **I want to** export all my memories to a portable file and import them on another machine, **so that** my agent's memory moves with me — across machines, with no cloud sync. + +This plan ships the user-approved scope addition first and lays the store contract both Wave-2/3 tracers build on: + +- **D-14 (tracer):** today only `search` redacts storage-engine text; `store`, `list` (and therefore `/health`), `update`, `link`/`unlink`, `forget`, `import` and `sweep` still send `sqlite error: …` with table and column names to the client (04-REVIEW IN-01). Import adds another write path through the same mapping, so redaction lands on EVERY public `MemoryService` method before any new path exists — and every later test in this phase asserts redacted text. +- **Store contract (expansion):** `Store::find_by_key` (D-07: a duplicate must map its export key to the *existing* row id), `LinkOutcome::Linked { created }` (D-12: `links_skipped_duplicates` must be countable through the existing `Store::link`), and `ImportedMemory` + `Store::insert_imported` (D-10/D-13: preserved timestamps, always pending). These touch `domain.rs`, `store/mod.rs`, `sqlite.rs` and ripple into `mcp.rs`/`handlers.rs`/`relations.rs` — the same files D-14 touches — so they land here in one wave instead of forcing plans 02 and 03 to each carry partial store edits. + + +1. Generalise `redact_search_error` into `redact_storage_error(op, e)` and apply it to every public `MemoryService` method through the `*_unredacted` inner-fn shape `search` already uses. Prove it in-process for a write path and for `/health` on both transports, mirroring T-04-19. +2. Replace `Store::exists -> bool` with `Store::find_by_key -> Option` (one caller, one impl) and make `Store::link` report whether it inserted (`LinkOutcome::Linked { created: bool }`, seven sites, not a wire change). +3. Add `ImportedMemory` and `Store::insert_imported`, refactoring the single `memories` INSERT into one private `insert_row` helper so `insert` and `insert_imported` can never drift. + +Purpose: clients never see storage-engine text on any surface (the IN-01 deferral is closed), and the import/export tracers in plans 02 and 03 compile against a tested store contract instead of inventing one. +Output: `redact_storage_error` on nine methods with core/REST/MCP proofs; `tests/redaction.rs`; `find_by_key`; `Linked { created }`; `ImportedMemory` + `insert_imported`; README sentence + behavioral change 4. + + + +@$HOME/.claude/gsd-core/workflows/execute-plan.md +@$HOME/.claude/gsd-core/templates/summary.md + + + +@.planning/STATE.md +@.planning/phases/05-portable-export-import/05-CONTEXT.md +@.planning/phases/05-portable-export-import/05-RESEARCH.md +@.planning/phases/05-portable-export-import/05-PATTERNS.md +@.planning/phases/04-memory-update-relations-hybrid-search/04-REVIEW.md +@crates/agent-memory-core/src/service.rs +@crates/agent-memory-core/src/store/sqlite.rs + + +## Execution notes for every task + +- Gate before EVERY commit: `cargo fmt --all`, then `cargo clippy --workspace --all-targets --locked -- -D warnings`, then `cargo test --workspace --locked`. Run clippy and the full test suite in the background with a log file (`cargo test --workspace --locked > /tmp/gsd-05-test.log 2>&1 &`) and poll; the executor watchdog is ~600 s and a cold workspace build can take several minutes. +- One commit per task. Stage explicit paths only. NEVER stage `CLAUDE.md`, `.claude/`, `.mcp.json`, `.gitignore`, `.gsd/`, `docs/`, `10-agent-memory.md`, `claude-code-harness-engineering-guide-v2.md`, `.planning/research/.cache/`, `.planning/config.json`, `.planning/milestone.lock`. +- No `unwrap()`/`expect()` in production code (tests may use `expect`). No new crate dependency; `Cargo.lock` must stay byte-identical (`--locked` enforces it). +- Fault injection in tests uses `ALTER TABLE RENAME TO _offline` on a raw side connection — the repo's bash hook blocks shell text containing a table-drop statement, and the T-04-19 recipe is RENAME anyway. +- Never change the CI runner or add a GitHub Actions workflow. Never push, tag or release. + +## Interface contracts (established by this plan) + +**`redact_storage_error`** (`crates/agent-memory-core/src/service.rs`) + +```text +fn redact_storage_error(op: &str, e: MemoryError) -> MemoryError +``` +- `InvalidType | InvalidQuery | InvalidArgument` → returned unchanged (client tier). +- `Internal | SchemaTooNew { .. }` → returned unchanged (crate-authored text). +- `Sqlite | Pool | Join | Migration` → `tracing::error!(op, error = %e, "storage error")` then `MemoryError::Internal(format!("{op} failed due to a storage error (details are in the server log)"))`. +- No catch-all arm. `pub const SEARCH_STORAGE_FAILURE` keeps its exact current text; a unit test pins `redact_storage_error("search", )` to `Internal(SEARCH_STORAGE_FAILURE)`. +- Op strings, exactly: `"store"`, `"list"`, `"search"`, `"import"`, `"forget"`, `"link"`, `"unlink"`, `"update"`, `"sweep"`. Plans 02/03 add `"export"` and reuse `"import"`. +- Shape per method: the existing body moves to `async fn _unredacted(...)`; the public `async fn (...)` is `self._unredacted(..).await.map_err(|e| redact_storage_error("", e))`. A tail `.map_err` on a `?`-laden body is NOT acceptable — early returns would escape unredacted. + +**`Store::find_by_key`** (`store/mod.rs`, replaces `exists`) + +```text +fn find_by_key(&self, source: Option<&str>, mem_type: MemoryType, content: &str) -> Result, MemoryError>; +``` +SQL: `SELECT id FROM memories WHERE source IS ?1 AND mem_type = ?2 AND content = ?3 LIMIT 1` on the read pool, `.optional()?`. The CR-01 NULL-safety paragraph stays in the doc. + +**`LinkOutcome::Linked { created: bool }`** (`domain.rs`) + +`created` is `tx.changes() == 1`, read before `commit`. Transports match `LinkOutcome::Linked { .. }` and emit the same JSON as today. Tests assert `created: true` on the first insert and `created: false` on a duplicate or symmetric re-link. + +**`ImportedMemory`** (`domain.rs`, beside `NewMemory`) and **`Store::insert_imported`** + +```text +pub struct ImportedMemory { + pub content: String, pub mem_type: MemoryType, pub tags: Vec, + pub source: Option, pub scope: Option, + pub created_at: i64, pub access_count: i64, pub expires_at: Option, +} +fn insert_imported(&self, m: ImportedMemory, now: i64) -> Result; +``` +Row semantics: `created_at`/`access_count`/`expires_at` preserved from the file; `last_accessed = now`; `decay_score = 1.0`; `base_weight` = 2.0 if `mem_type.is_pinned()` else 1.0; `embedding_status = 0` ALWAYS (no embedding parameter exists); `tags` serialised with `serde_json::to_string` like `insert`. Same writer-lock + single-transaction shape as `insert`. The caller owns the import-time `expires_at <= now` skip (D-10); the sweep's own delete predicate is `expires_at < now` — both stated in the trait doc. + +**`insert_row`** (private, `sqlite.rs`): one private `struct MemoryRow<'a>` carrying `mem_type, content, tags_json, source, scope, access_count, created_at, last_accessed, expires_at, embedding_status`, and `fn insert_row(tx: &rusqlite::Transaction<'_>, row: &MemoryRow<'_>) -> Result` holding THE one `memories` INSERT. `insert` calls it with `access_count = 0`, `created_at = last_accessed = now`, `embedding_status = i64::from(embedding.is_some())`; `insert_imported` calls it with the preserved values, `last_accessed = now`, `embedding_status = 0`. Using a struct keeps the helper under clippy's `too_many_arguments` limit. + +## Artifacts this phase produces (this plan's share) + +- `service.rs`: `fn redact_storage_error(op: &str, e: MemoryError) -> MemoryError`; `async fn store_unredacted`, `list_unredacted`, `import_unredacted`, `forget_unredacted`, `link_unredacted`, `unlink_unredacted`, `update_unredacted`, `sweep_unredacted` (beside the existing `search_unredacted`); unit tests `redact_storage_error_search_text_equals_the_constant`, `redact_storage_error_passes_client_and_crate_text_through`. +- `domain.rs`: `pub struct ImportedMemory`; `LinkOutcome::Linked { created: bool }`. +- `store/mod.rs`: `fn find_by_key(...)` (replaces `exists`); `fn insert_imported(...)`. +- `sqlite.rs`: `struct MemoryRow<'a>`, `fn insert_row(...)`; `find_by_key` impl; `insert_imported` impl; `link` returns `created`. +- `tests/redaction.rs` (new): eight fault-injection tests. +- `tests/store.rs`: `find_by_key_returns_the_matching_id_null_safely`, `insert_imported_preserves_provenance_and_lands_pending`. +- `tests/relations.rs`: four asserts upgraded to `Linked { created: … }`. +- `rest/handlers.rs` tests: `health_storage_failure_is_500_without_sqlite_text`, `store_storage_failure_is_500_without_sqlite_text`; `mcp.rs` test: `memory_store_storage_failure_is_internal_error_without_sqlite_text`. +- README: widened redaction sentence; "Behavioral changes in v0.1.0" item 4. + + + + + Task 1 (tracer): A storage fault on any service path reaches REST, MCP and the CLI only as the redacted internal error + crates/agent-memory-core/src/service.rs, crates/agent-memory-core/tests/redaction.rs, crates/agent-memory/src/rest/handlers.rs, crates/agent-memory/src/mcp.rs, README.md + + - crates/agent-memory-core/src/service.rs: lines 33-60 (`SEARCH_STORAGE_FAILURE` + `redact_search_error`), 181-200 (`MemoryService`), 209-232 (`store`), 244-265 (`list`), 296-312 (`search` / `search_unredacted` — the shape to replicate), 394-463 (`import`), 573-666 (`forget`, `link`, `unlink`, `update`), 688-749 (`sweep`, `BACKFILL_BATCH`); check whether a `#[cfg(test)] mod tests` already exists (the `rrf_fuse` tests) and extend it rather than adding a second one. + - crates/agent-memory-core/src/domain.rs: lines 319-332 (`MemoryPatch`, needed to call `update` in a test), 375-426 (`MemoryError` variants and Display strings). + - crates/agent-memory-core/tests/hybrid.rs: lines 712-770 (the RENAME fault-injection recipe and the `SEARCH_STORAGE_FAILURE` assertions at 729 and 760 that must stay green). + - crates/agent-memory-core/tests/relations.rs: lines 20-104 (`Harness`, `harness()`, `harness_with()`, `new_memory`, `store`, `raw_conn`, `edge_count`, `edges_touching` — copy into the new test file). + - crates/agent-memory/src/rest/handlers.rs: lines 165-182 (`map_memory_error`, unchanged), 437-454 (`health_handler`), 478-500 (`test_state`/`state_with`), 851-889 (`search_storage_failure_is_500_without_sqlite_text` — copy its shape), and the existing store-handler test helpers near it. + - crates/agent-memory/src/mcp.rs: lines 33-50 (`map_mcp_error`, unchanged), 522-545 (`test_server`/`server_with`), 1309-1344 (`memory_search_storage_failure_is_internal_error_without_sqlite_text` — copy its shape), plus `store_one` and the `memory_store` tool signature. + - README.md: lines 455-463 (the search-only redaction sentence), 486-512 ("Behavioral changes in v0.1.0"). + - .planning/phases/04-memory-update-relations-hybrid-search/04-REVIEW.md: IN-01. + - .planning/phases/05-portable-export-import/05-RESEARCH.md: Pattern 8 and Code Example 7. + + + - Unit (service.rs): `redact_storage_error("search", MemoryError::Sqlite(rusqlite::Error::QueryReturnedNoRows))` yields `MemoryError::Internal(m)` with `m == SEARCH_STORAGE_FAILURE`; `redact_storage_error("store", )` yields `Internal("store failed due to a storage error (details are in the server log)")`; `InvalidArgument("x")`, `InvalidType("x")`, `InvalidQuery("x")`, `Internal("y")` and `SchemaTooNew { db_version: 9, binary_version: 3 }` come back with their `to_string()` unchanged. + - Integration (`crates/agent-memory-core/tests/redaction.rs`, failing `FakeEmbedder`, real temp DB): eight `#[tokio::test]`s — one per op. Setup per op: `store` two rows where an id is needed; rename `memories` → `memories_offline` (for `link`/`unlink`: rename `memory_links` → `memory_links_offline` instead, so the endpoint pre-check still passes and the INSERT/DELETE is what fails); then call `store(new_memory)`, `list(ListArgs::default())`, `import(vec![draft])`, `forget(id)`, `link(a, b, LinkKind::RelatesTo)`, `unlink(a, b, LinkKind::RelatesTo)`, `update(id, patch with Some(content))`, `sweep(h.clock.now())`. Each must `expect_err` and satisfy `matches!(err, MemoryError::Internal(ref m) if m == &format!("{op} failed due to a storage error (details are in the server log)"))` AND `!err.to_string().to_lowercase().contains("sqlite")`. + - REST in-process (handlers.rs tests): `health_storage_failure_is_500_without_sqlite_text` — `test_state`, rename `memories`, call `health_handler(State(state))`, convert the `Err(ApiError)` into a response → status 500, JSON body `error` contains `list failed due to a storage error` and not `sqlite`. `store_storage_failure_is_500_without_sqlite_text` — same rename, then the store handler with a valid body → 500, `error` contains `store failed due to a storage error`, no `sqlite`. The existing `search_storage_failure_is_500_without_sqlite_text` stays green unchanged. + - MCP in-process (mcp.rs tests): `memory_store_storage_failure_is_internal_error_without_sqlite_text` — `test_server`, rename `memories`, call `memory_store` → `err.code == McpError::internal_error("x", None).code`, `err.message` contains `store failed due to a storage error`, no `sqlite`. The existing search redaction test stays green unchanged. + - Every Phase 4 golden stays green: `tests/hybrid.rs:729,760` (equality on `SEARCH_STORAGE_FAILURE`), `handlers.rs:881`, `mcp.rs:1336` (`contains("search failed due to a storage error")`). + + + Implements D-14 (user-approved scope addition closing 04-REVIEW IN-01) per RESEARCH Pattern 8. + + 1. **service.rs — the helper.** Rename `redact_search_error` to `redact_storage_error` and give it an `op: &str` parameter per "Interface contracts". Keep the exhaustive match with no catch-all arm (a new `MemoryError` variant must fail to compile until tiered). The storage-tier arm logs `tracing::error!(op, error = %e, "storage error")` and returns `MemoryError::Internal(format!("{op} failed due to a storage error (details are in the server log)"))`. Keep `pub const SEARCH_STORAGE_FAILURE` with its exact text; reword its doc to say it equals the template at `op = "search"` and is pinned by a unit test. Update the helper's doc: it tiers every service-path error for the wire — the ONE place storage-engine text is stripped — and explain why the seam (not the transport mappers) is the right place: MCP, REST including `/health`, and the CLI are all covered by one change (Phase 3 D-06 / Phase 4 T-04-19 pattern). + 2. **service.rs — apply to every public method.** For `store`, `list`, `import`, `forget`, `link`, `unlink`, `update` and `sweep`: move the current body into `async fn _unredacted(...)` with the same signature, and make the public fn delegate via `.await.map_err(|e| redact_storage_error("", e))`. Change `search` to pass `"search"`. Do not alter any validation order, hop structure or return value. Update each public doc comment with one sentence: errors are tiered for the wire by `redact_storage_error` (D-14). The `sweep` background task in `main.rs` keeps logging the (now redacted) error — the detail already went to the error log inside the helper, so nothing is lost. + 3. **service.rs — unit tests.** In the existing `#[cfg(test)] mod tests` (or a new one if none exists), add `redact_storage_error_search_text_equals_the_constant` and `redact_storage_error_passes_client_and_crate_text_through` per ``. Build the storage variant with `MemoryError::Sqlite(rusqlite::Error::QueryReturnedNoRows)`. + 4. **tests/redaction.rs (new).** Module doc: D-14 proof — storage faults on every non-search path are redacted at the seam (T-04-19 pattern generalised). Copy the relations.rs harness (`Harness`, `harness`, `harness_with`, `new_memory`, `store`, `raw_conn`). Add a helper `fn take_offline(db_path: &Path, table: &str)` that runs the RENAME through `raw_conn`, and a helper `fn assert_redacted(err: &MemoryError, op: &str)` doing both assertions. Then the eight tests per ``. For `update`, build the patch from the `MemoryPatch` fields you read in domain.rs (content `Some(..)`, every other field absent/`None`). + 5. **handlers.rs + mcp.rs tests** per ``, modelled on the existing search redaction tests (same `register_vec_extension()` + raw `rusqlite::Connection::open` + `execute_batch` rename). Do NOT touch `map_memory_error` or `map_mcp_error` — both must still compile exhaustively, unchanged. + 6. **README.md.** Replace the sentence "A storage failure during search reports a generic internal error (HTTP 500 / MCP `internal_error`, `search failed due to a storage error`); the detail goes only to the server's stderr log." with: "A storage failure on any request the daemon serves — and on any CLI `import`/`export` — reports a generic internal error (HTTP 500 / MCP `internal_error`) shaped ` failed due to a storage error (details are in the server log)`; the engine detail (SQLite error text, table and column names) goes only to the server's stderr log." In "Behavioral changes in v0.1.0", change "Three deliberate contract changes" to "Four" and append item 4: storage faults on every surface now report that generic message instead of echoing the storage engine's text; v0.0.1 returned `sqlite error: …` strings on non-search 500s / `internal_error`s, and clients parsing them must stop — the detail is in the server log. + 7. `cargo fmt --all`; clippy; full tests (background + poll); commit with explicit paths: `feat(05-01): redact storage-engine text on every service path (D-14)`. + + The redaction is one helper at one seam; the transport mappers and every message template outside the storage tier are unchanged, so widening or narrowing the set of redacted methods later is a local edit. + + cargo test -p agent-memory-core --lib redact_storage_error && cargo test -p agent-memory-core --test redaction && cargo test -p agent-memory-core --test hybrid && cargo test -p agent-memory --bin agent-memory storage_failure && cargo clippy --workspace --all-targets --locked -- -D warnings + + + - `grep -c "fn redact_storage_error" crates/agent-memory-core/src/service.rs` outputs 1 and `grep -c "async fn [a-z_]*_unredacted" crates/agent-memory-core/src/service.rs` outputs 9. + - `grep -c 'redact_storage_error("' crates/agent-memory-core/src/service.rs` outputs at least 9 (one call per public method: store, list, search, import, forget, link, unlink, update, sweep). + - `grep -c "pub const SEARCH_STORAGE_FAILURE" crates/agent-memory-core/src/service.rs` outputs 1 and the constant's text is unchanged: `grep -c '"search failed due to a storage error (details are in the server log)"' crates/agent-memory-core/src/service.rs` outputs 1. + - `cargo test -p agent-memory-core --test redaction` runs 8 tests, all green. + - `cargo test -p agent-memory --bin agent-memory storage_failure` runs at least 4 tests (the two existing search tests plus health, store on REST, and memory_store on MCP), all green. + - `cargo test -p agent-memory-core --test hybrid` is green (the `SEARCH_STORAGE_FAILURE` equality assertions at lines 729 and 760 are unchanged). + - `grep -c "failed due to a storage error (details are in the server log)" README.md` outputs at least 1 and `grep -c "^4\. " README.md` outputs at least 1 inside the "Behavioral changes in v0.1.0" section (check with `sed -n '/^## Behavioral changes/,/^## /p' README.md | grep -c '^4\. '` → 1). + - `git diff --stat -- crates/agent-memory/src/rest/handlers.rs crates/agent-memory/src/mcp.rs` shows changes only inside the `#[cfg(test)]` modules (the error mappers are byte-identical). + - fmt, clippy `-D warnings` and `cargo test --workspace --locked` are green. + + A storage fault on store, list, import, forget, link, unlink, update, sweep or search reaches any client — REST (including /health), MCP, CLI — only as ` failed due to a storage error (details are in the server log)`, with the engine detail in the stderr log, and the README says so for every surface. + + + + Task 2: The dedup probe returns the matching row id, and a link insert reports whether it created the edge + crates/agent-memory-core/src/domain.rs, crates/agent-memory-core/src/store/mod.rs, crates/agent-memory-core/src/store/sqlite.rs, crates/agent-memory-core/src/service.rs, crates/agent-memory/src/mcp.rs, crates/agent-memory/src/rest/handlers.rs, crates/agent-memory-core/tests/relations.rs, crates/agent-memory-core/tests/store.rs + + - crates/agent-memory-core/src/domain.rs: lines 173-183 (`LinkOutcome`), 95-150 (`LinkKind`, `canonical_endpoints`). + - crates/agent-memory-core/src/store/mod.rs: lines 81-96 (`exists` doc + signature), 174-206 (`link` doc — the "still returns Linked" sentence to reword). + - crates/agent-memory-core/src/store/sqlite.rs: lines 751-772 (`exists` impl — keep the CR-01 comment and `source IS ?1`), 945-990 (`link` impl — `INSERT OR IGNORE`, where `changes()` must be read). + - crates/agent-memory-core/src/service.rs: the one `store.exists(` call inside `import_unredacted` (was line 413 before Task 1). + - crates/agent-memory/src/mcp.rs line 438 and crates/agent-memory/src/rest/handlers.rs line 370 (the `LinkOutcome::Linked =>` arms and the JSON they emit). + - crates/agent-memory-core/tests/relations.rs lines 130-150 and 785-800 (the four `LinkOutcome::Linked` asserts; read the surrounding comments to see which call is the first insert and which is the duplicate / symmetric re-link). + - crates/agent-memory-core/tests/store.rs lines 1-60 (`new_memory`, `service_for` — the harness your new test extends; you will also need `use agent_memory_core::store::Store;` to call trait methods on `SqliteStore` directly). + - .planning/phases/05-portable-export-import/05-RESEARCH.md: Patterns 6 and 6b. + + + - `tests/store.rs::find_by_key_returns_the_matching_id_null_safely`: open a `SqliteStore` on a temp path; `insert` row A (`source: None`, Decision, "alpha") and row B (`source: Some("s")`, Decision, "beta") with `embedding: None, now: 1_000`; then `find_by_key(None, Decision, "alpha") == Some(id_a)`, `find_by_key(Some("s"), Decision, "beta") == Some(id_b)`, `find_by_key(Some(""), Decision, "alpha") == None` (NULL is not the empty string — CR-01), `find_by_key(None, Pattern, "alpha") == None`, `find_by_key(None, Decision, "gamma") == None`. + - `tests/relations.rs`: the first `link` of a pair asserts `LinkOutcome::Linked { created: true }`; the D-02 duplicate asserts `LinkOutcome::Linked { created: false }`; in the symmetric test the forward `a→b relates_to` is `created: true` and the reverse `b→a` is `created: false`; `edge_count` assertions are unchanged. + - `tests/import.rs` (unchanged file) stays green — dedup now runs through `find_by_key(...)?.is_some()`. + - MCP/REST link tests stay green: the JSON (`"linked": true`, echoed request ids) is byte-identical. + + + Implements D-07 (id-returning dedup probe) and the D-12 countability prerequisite per RESEARCH Patterns 6 and 6b. + + 1. **domain.rs:** change `LinkOutcome::Linked` to `LinkOutcome::Linked { created: bool }`, keeping `#[derive(Debug, Clone, Copy, PartialEq, Eq)]`. Doc: `created` is true when the row was inserted, false when the stored edge already existed (the D-02 idempotent no-op; the JSONL importer counts the latter as `links_skipped_duplicates`). + 2. **sqlite.rs `link`:** after the `INSERT OR IGNORE`, read `let created = tx.changes() == 1;` BEFORE `tx.commit()?`, and return `Ok(LinkOutcome::Linked { created })`. Update the comment above the statement: a duplicate is still a success, now reported as `created: false`. + 3. **store/mod.rs:** rename `exists` to `find_by_key` returning `Result, MemoryError>`. Keep the CR-01 NULL-safe paragraph verbatim and add: the matching row's id is returned so an importer can map an export-local key onto the existing row instead of inserting a second copy (D-07); `None` means no such row. Reword the `link` doc sentence to "A duplicate of the stored row is an idempotent no-op that returns `LinkOutcome::Linked { created: false }`". + 4. **sqlite.rs `exists` → `find_by_key`:** same read-pool connection and the same comment block; change `SELECT 1` to `SELECT id`, the row mapper to `|row| row.get::<_, i64>(0)`, keep `.optional()?`, return `Ok(found)`. + 5. **service.rs `import_unredacted`:** the dedup probe becomes `store.find_by_key(draft.source.as_deref(), draft.mem_type, &draft.content)?.is_some()`. Keep the CR-01 comment. + 6. **mcp.rs:438 and handlers.rs:370:** the arm becomes `LinkOutcome::Linked { .. } =>` with the JSON body unchanged. + 7. **tests/relations.rs:** upgrade the four asserts per `` — these become stronger idempotency proofs, not weaker ones. + 8. **tests/store.rs:** add `find_by_key_returns_the_matching_id_null_safely` per `` using the `Store` trait directly on `SqliteStore` (no service needed). + 9. `cargo fmt --all`; clippy; full tests (background + poll); commit with explicit paths: `feat(05-01): id-returning dedup probe and countable link insert (D-07, D-12)`. + + `LinkOutcome` is a public core enum matched at seven sites (store impl, two transports, four tests); reverting the `created` field touches all of them. It is not a wire change — transports echo request ids — so no released client depends on it. + + cargo test -p agent-memory-core --test store find_by_key && cargo test -p agent-memory-core --test relations && cargo test -p agent-memory-core --test import && cargo test -p agent-memory --bin agent-memory link && cargo clippy --workspace --all-targets --locked -- -D warnings + + + - `grep -c "fn find_by_key" crates/agent-memory-core/src/store/mod.rs` outputs 1 and `grep -c "fn find_by_key" crates/agent-memory-core/src/store/sqlite.rs` outputs 1; `grep -c "fn exists" crates/agent-memory-core/src/store/mod.rs` outputs 0. + - `grep -c "SELECT id FROM memories" crates/agent-memory-core/src/store/sqlite.rs` outputs at least 1 and `grep -c "source IS ?1" crates/agent-memory-core/src/store/sqlite.rs` outputs at least 1 (the NULL-safe probe survives the rename). + - `grep -c "Linked { created: bool }" crates/agent-memory-core/src/domain.rs` outputs 1; `grep -c "tx.changes()" crates/agent-memory-core/src/store/sqlite.rs` outputs at least 1. + - `grep -c "Linked { created: true }" crates/agent-memory-core/tests/relations.rs` outputs at least 2 and `grep -c "Linked { created: false }" crates/agent-memory-core/tests/relations.rs` outputs at least 2. + - `cargo test -p agent-memory-core --test store find_by_key` passes; `cargo test -p agent-memory-core --test relations` and `--test import` are fully green. + - `cargo test -p agent-memory --bin agent-memory link` is green with no JSON assertion changed in mcp.rs or handlers.rs tests. + - fmt, clippy `-D warnings` and `cargo test --workspace --locked` are green. + + An importer can ask the store "which existing row is this?" and get an id; a link insert says whether it created an edge; the wire shape of every existing surface is unchanged. + + + + Task 3: The store accepts a restored memory with its provenance preserved and always pending embedding + crates/agent-memory-core/src/domain.rs, crates/agent-memory-core/src/store/mod.rs, crates/agent-memory-core/src/store/sqlite.rs, crates/agent-memory-core/tests/store.rs + + - crates/agent-memory-core/src/domain.rs: lines 240-253 (`NewMemory` — the struct `ImportedMemory` sits directly after and mirrors; do NOT extend `NewMemory`, it has 15 construction sites). + - crates/agent-memory-core/src/store/mod.rs: lines 21-32 (`insert` doc — mirror its style), 219-228 (`sweep_expired` doc with the `expires_at < now` predicate you must contrast against). + - crates/agent-memory-core/src/store/sqlite.rs: lines 596-644 (`insert` — the writer-lock + transaction shape and the INSERT whose column list and bindings move into `insert_row`), 603 (`base_weight` from `is_pinned`). + - crates/agent-memory-core/tests/store.rs lines 1-60 (harness) and crates/agent-memory-core/tests/backfill_race.rs lines 195-213 (`vec_rows` / `status` raw-connection helpers to copy). + - crates/agent-memory-core/sql/0001_init.sql (the `memories` columns and the `memories_ai` FTS trigger — confirm the column list you bind) and sql/0002_embeddings.sql (`embedding_status`). + - .planning/phases/05-portable-export-import/05-RESEARCH.md: Pattern 5; 05-CONTEXT.md D-10 and D-13. + + + - `tests/store.rs::insert_imported_preserves_provenance_and_lands_pending`: open a `SqliteStore`; `insert_imported(ImportedMemory { content: "restored decision", mem_type: Decision, tags: ["a","b"], source: Some("gsd-state"), scope: Some("proj"), created_at: 500, access_count: 7, expires_at: Some(9_000) }, now = 1_000)` returns an id; through a raw connection (`register_vec_extension()` first) the row has `created_at == 500`, `last_accessed == 1_000`, `access_count == 7`, `expires_at == 9_000`, `embedding_status == 0`, `decay_score == 1.0`, `base_weight == 2.0` (Decision is pinned), `tags == "[\"a\",\"b\"]"`, `source == "gsd-state"`, `scope == "proj"`, `mem_type == "DECISION"`; `SELECT count(*) FROM vec_memories WHERE memory_id = ?1` is 0. A second `insert_imported` with `mem_type: Pattern, source: None, scope: None, expires_at: None` has `base_weight == 1.0`, NULL `source`/`scope`/`expires_at`. + - The imported row is visible through the normal read path: a `MemoryService` over the same store lists it, and a keyword `search` for "restored" finds it (the FTS trigger fired on the shared INSERT). + - Every pre-existing test in `tests/store.rs`, `tests/ttl.rs`, `tests/decay.rs`, `tests/update.rs`, `tests/backfill_race.rs` stays green after the `insert_row` refactor (the `insert` path is byte-equivalent in behaviour). + + + Implements the Claude's-Discretion choice from 05-CONTEXT.md (dedicated `ImportedMemory` + `Store::insert_imported`, zero existing call sites touched — RESEARCH Pattern 5) carrying D-10 (preserved vs refreshed columns) and D-13 (always pending). + + 1. **domain.rs:** add `pub struct ImportedMemory` per "Interface contracts" directly after `NewMemory`, `#[derive(Debug, Clone, PartialEq)]`. Doc comment: a memory restored from a format-v1 file (INTEROP-02); unlike `NewMemory` it carries absolute timestamps — `created_at`, `access_count`, `expires_at` are preserved as provenance, `last_accessed` is NOT a field because it is refreshed to import-time `now` so restored memories start decay-fresh (D-10, PITFALLS 19), and the row always lands `embedding_status = 0` for the sweep backfill (D-13). + 2. **store/mod.rs:** add `fn insert_imported(&self, m: ImportedMemory, now: i64) -> Result;` after `insert`. Doc: same single writer transaction as `insert`; no embedding parameter — the row ALWAYS lands pending (D-13); `last_accessed = now`; preserved columns listed; the caller (the JSONL importer) owns the import-time rule `expires_at <= now → skip and count`, whereas `sweep_expired` deletes `expires_at < now` — both predicates named so the boundary is deliberate. + 3. **sqlite.rs — `insert_row`:** add the private `struct MemoryRow<'a>` and `fn insert_row(tx, row) -> Result` per "Interface contracts". Move the existing INSERT text and `params![]` into it unchanged except that `access_count`, `created_at` and `last_accessed` become three separately bound parameters (today `?7` is bound once for both timestamps and `access_count` is the literal `0`). Compute `base_weight` inside from `row.mem_type.is_pinned()`. Return `tx.last_insert_rowid()`. + 4. **sqlite.rs — `insert`:** keep its signature, writer lock, transaction and vector INSERT; build a `MemoryRow` with `access_count: 0, created_at: now, last_accessed: now, embedding_status: i64::from(embedding.is_some())` and call `insert_row`. Behaviour must be identical. + 5. **sqlite.rs — `insert_imported`:** same writer-lock + `conn.transaction()?` shape; `MemoryRow` from the `ImportedMemory` with `last_accessed: now`, `embedding_status: 0`; no vector write; `tx.commit()?`; return the id. Comment: restored rows are re-embedded only by the sweep backfill against the local model pin (PITFALLS 17). + 6. **tests/store.rs:** add `insert_imported_preserves_provenance_and_lands_pending` per `` (raw `rusqlite::Connection` reads after `register_vec_extension()`; `use agent_memory_core::store::Store;`). + 7. `cargo fmt --all`; clippy; full tests (background + poll); commit with explicit paths: `feat(05-01): ImportedMemory + Store::insert_imported for restored rows (D-10, D-13)`. + + Additive type and trait method with one implementor and no existing callers; removing them later deletes two definitions and one test. + + cargo test -p agent-memory-core --test store && cargo test -p agent-memory-core --test ttl && cargo test -p agent-memory-core --test backfill_race && cargo test -p agent-memory-core --test update && cargo clippy -p agent-memory-core --all-targets --locked -- -D warnings + + + - `grep -c "pub struct ImportedMemory" crates/agent-memory-core/src/domain.rs` outputs 1; `grep -c "fn insert_imported" crates/agent-memory-core/src/store/mod.rs` outputs 1 and the same grep on sqlite.rs outputs 1. + - `grep -c "fn insert_row" crates/agent-memory-core/src/store/sqlite.rs` outputs 1, and the `memories` table has exactly one INSERT statement in sqlite.rs production code: `grep -v '^\s*//' crates/agent-memory-core/src/store/sqlite.rs | grep -c "INTO memories "` outputs 1. + - `cargo test -p agent-memory-core --test store insert_imported_preserves_provenance_and_lands_pending` passes, asserting `created_at == 500`, `last_accessed == 1_000`, `access_count == 7`, `embedding_status == 0` and zero `vec_memories` rows. + - `cargo test -p agent-memory-core --test store`, `--test ttl`, `--test decay`, `--test update`, `--test backfill_race` are fully green (the `insert` refactor changed no behaviour). + - `git diff --stat -- crates/agent-memory-core/sql/` is empty and `grep -c "LATEST_SCHEMA_VERSION: i64 = 3" crates/agent-memory-core/src/store/migrations.rs` outputs 1 (no migration this phase). + - fmt, clippy `-D warnings` and `cargo test --workspace --locked` are green. + + The store has exactly one way to write a `memories` row, and a restored memory goes through it with its provenance preserved, its freshness reset, and its embedding pending for the sweep. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| MCP stdio / REST loopback client → service error text | an untrusted reader of error bodies; today receives storage-engine detail on non-search paths | +| service → stderr log | trusted operator channel; the only place engine detail may go | +| store writer lane | single-writer transaction boundary that `insert_imported` must share with `insert` | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan | +|-----------|----------|-----------|----------|-------------|-----------------| +| T-05-01 | Information Disclosure | `sqlite error: …` (table/column names, engine text) returned by store/list/update/link/unlink/forget/import/sweep and by `/health` | medium | mitigate | `redact_storage_error(op, e)` at the service seam on all nine methods; engine detail only in the `tracing::error!` log; in-process tests on core, REST `/health` + `POST /api/memories`, MCP `memory_store` assert no `sqlite` substring (Task 1) | +| T-05-14 | Information Disclosure | `SqliteStore::open` failures printed by the CLI (anyhow text on local stderr) | low | accept | Startup, operator-facing, outside the `MemoryService` seam; no transport carries it (orchestrator resolution 4). README scopes redaction to "every request the daemon serves and every CLI import/export" | +| T-05-15 | Tampering (integrity) | two divergent `memories` INSERT statements drifting apart (column order, defaults) | low | mitigate | one private `insert_row` behind `insert` and `insert_imported`; acceptance grep asserts exactly one INSERT into `memories` (Task 3) | +| T-05-16 | Repudiation | a restored row reported "imported" but swept away later | low | mitigate (contract) | `insert_imported` doc fixes the two predicates (`<= now` skip at import, `< now` sweep); the importer in plan 03 enforces the skip | +| T-05-SC | Tampering | npm/pip/cargo installs | low | accept | Zero new dependencies; `--locked` builds fail if `Cargo.lock` changes; the rusqlite 0.39 / libsqlite3-sys 0.37 pin triangle is untouched | + +No `high` severity threat is open. ASVS L1, block_on: high. + + + +- `cargo fmt --all --check`, `cargo clippy --workspace --all-targets --locked -- -D warnings`, `cargo test --workspace --locked` all green after the third commit. +- `cargo test -p agent-memory-core --test redaction` — 8 green; `cargo test -p agent-memory --bin agent-memory storage_failure` — REST health/store and MCP store redacted. +- `cargo test -p agent-memory-core --test relations --test import --test store` — green with the stronger `created` asserts and the two new store tests. +- `git diff --stat -- crates/agent-memory-core/sql/` empty; `Cargo.lock` unchanged. + + + +- Every public `MemoryService` method redacts storage-engine text; Phase 4 goldens untouched; README widened and behavioral change 4 added. +- `Store::find_by_key`, `LinkOutcome::Linked { created }`, `ImportedMemory` and `Store::insert_imported` exist, are documented, and are covered by in-process tests. +- Three commits on `main`, each with fmt + clippy + full tests green; nothing pushed. + + + +Create `.planning/phases/05-portable-export-import/05-01-SUMMARY.md` when done. Record: the nine op strings, the `insert_row` refactor, the `LinkOutcome` change scope (7 sites, not a wire change), and the full-suite run count. + diff --git a/.planning/phases/05-portable-export-import/05-01-SUMMARY.md b/.planning/phases/05-portable-export-import/05-01-SUMMARY.md new file mode 100644 index 0000000..53fface --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-01-SUMMARY.md @@ -0,0 +1,253 @@ +--- +phase: 05-portable-export-import +plan: 01 +subsystem: core-service, store +tags: [rust, sqlite, rusqlite, redaction, error-tiering, store-contract, import, relations] + +# Dependency graph +requires: + - phase: 04-memory-update-relations-hybrid-search + provides: "redact_search_error / SEARCH_STORAGE_FAILURE (T-04-19), Store::exists with NULL-safe source IS ?1 (CR-01), LinkOutcome + INSERT OR IGNORE link (D-02), the two exhaustive transport error mappers" + - phase: 03-api-hardening-toolchain-spikes + provides: "MemoryError::InvalidArgument client tier at the MemoryService seam (D-04/D-06)" +provides: + - "redact_storage_error(op, e) applied to every public MemoryService method (store, list, search, import, forget, link, unlink, update, sweep) via the *_unredacted inner-fn shape — closes 04-REVIEW IN-01 (D-14)" + - "tests/redaction.rs: eight fault-injection proofs (ALTER TABLE ... RENAME) plus REST /health, POST /api/memories and MCP memory_store in-process proofs with no `sqlite` substring" + - "Store::find_by_key(source, mem_type, content) -> Option replacing Store::exists (D-07)" + - "LinkOutcome::Linked { created: bool } read from tx.changes() before commit (D-12 countability)" + - "ImportedMemory + Store::insert_imported(m, now): preserved created_at/access_count/expires_at, last_accessed = now, embedding_status = 0, no vector row (D-10, D-13)" + - "one private insert_row(tx, &MemoryRow) helper holding THE single `memories` INSERT behind insert and insert_imported (T-05-15)" + - "README: redaction sentence widened to every surface; 'Behavioral changes in v0.1.0' item 4" +affects: [05-02 export tracer, 05-03 import tracer, 05-04 docs + CLI realism, phase-6 release notes] + +# Actuals (#2632) — same estimateTokens scale as the plan's `estimate` (chars/4 over the realized diff) +actuals: + tokens: 10400 + tasks: 3 + commits: 3 + +# Tech tracking +tech-stack: + added: [] + patterns: + - "Service-seam error tiering: every public MemoryService method is `self._unredacted(..).await.map_err(|e| redact_storage_error(\"\", e))`; a tail map_err on a ?-laden body is never acceptable (early returns would escape unredacted)" + - "Fault injection for storage-tier tests: ALTER TABLE RENAME TO _offline on a raw side connection (never DROP); rename memory_links instead of memories when the endpoint pre-check must still pass" + - "One INSERT per table: a private row struct + helper (MemoryRow / insert_row) shared by every insert path so column order and defaults cannot drift" + +key-files: + created: + - crates/agent-memory-core/tests/redaction.rs + modified: + - crates/agent-memory-core/src/service.rs + - crates/agent-memory-core/src/domain.rs + - crates/agent-memory-core/src/store/mod.rs + - crates/agent-memory-core/src/store/sqlite.rs + - crates/agent-memory-core/tests/relations.rs + - crates/agent-memory-core/tests/store.rs + - crates/agent-memory/src/rest/handlers.rs + - crates/agent-memory/src/mcp.rs + - README.md + +key-decisions: + - "Redaction lives at the MemoryService seam (not in the two transport mappers, which 04-REVIEW IN-01 suggested): one change covers MCP, REST including /health, and the CLI; the mappers are byte-identical and still match every MemoryError variant exhaustively" + - "The nine op strings are store, list, search, import, forget, link, unlink, update, sweep; SEARCH_STORAGE_FAILURE keeps its exact text and a unit test pins it to the template at op = \"search\" so Phase 4 goldens cannot drift" + - "LinkOutcome::Linked gained `created: bool` across all 7 sites (store impl, 2 transport arms, 4 relations.rs asserts) — storage-only, not a wire change: transports match `Linked { .. }` and emit byte-identical JSON" + - "ImportedMemory is a dedicated type + Store::insert_imported (zero of the 15 NewMemory construction sites touched), with no embedding parameter so a restored row can never land with a vector" + - "The two expiry predicates are named in the trait doc: the importer skips `expires_at <= now`, the sweep deletes `expires_at < now`" + - "Tracer feedback gate resolved by re-running the fully automated (all green) rather than returning a mid-plan human-verify checkpoint, because project config sets human_verify_mode: end-of-phase and the tracer's verify contains no human-judgment step" + +patterns-established: + - "redact_storage_error(op, e): Sqlite | Pool | Join | Migration -> tracing::error!(op, error = %e, \"storage error\") then Internal(\" failed due to a storage error (details are in the server log)\"); client tier and crate-authored text pass through; no catch-all arm" + - "Plans 02/03 add op = \"export\" and reuse op = \"import\" through the same helper and inner-fn shape" + +requirements-completed: [INTEROP-02] + +# Coverage metadata (#1602) +coverage: + - id: D1 + description: "A storage fault on any of the nine MemoryService paths reaches REST (incl. /health), MCP and the CLI only as ` failed due to a storage error (details are in the server log)`; no `sqlite` substring; SEARCH_STORAGE_FAILURE byte-identical" + requirement: INTEROP-02 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/redaction.rs (8 tests: store, list, import, forget, link, unlink, update, sweep)" + status: pass + - kind: unit + ref: "crates/agent-memory-core/src/service.rs#storage_error_redaction_search_text_equals_the_constant, #storage_error_redaction_passes_client_and_crate_text_through" + status: pass + - kind: integration + ref: "cargo test -p agent-memory --bin agent-memory storage_failure (health_storage_failure_is_500_without_sqlite_text, store_storage_failure_is_500_without_sqlite_text, memory_store_storage_failure_is_internal_error_without_sqlite_text + 2 existing search tests)" + status: pass + - kind: integration + ref: "cargo test -p agent-memory-core --test hybrid (SEARCH_STORAGE_FAILURE equality goldens unchanged)" + status: pass + human_judgment: false + - id: D2 + description: "Store::find_by_key returns Some(id) for the exact (source, mem_type, content) key with NULL-safe source equality, None otherwise; import dedups through it" + requirement: INTEROP-02 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/store.rs#find_by_key_returns_the_matching_id_null_safely" + status: pass + - kind: integration + ref: "cargo test -p agent-memory-core --test import (idempotent re-import unchanged)" + status: pass + human_judgment: false + - id: D3 + description: "Store::link reports Linked { created: true } on a fresh edge and Linked { created: false } on a duplicate or symmetric relates_to re-link; MCP/REST link JSON unchanged" + requirement: INTEROP-02 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/relations.rs#link_succeeds_and_a_duplicate_link_is_an_idempotent_no_op, #relates_to_linked_both_ways_is_one_relation_listed_once_from_each_side" + status: pass + - kind: integration + ref: "cargo test -p agent-memory --bin agent-memory link (15 tests, JSON assertions untouched)" + status: pass + human_judgment: false + - id: D4 + description: "Store::insert_imported lands a row with preserved created_at/access_count/expires_at, last_accessed = now, decay_score 1.0, base_weight from the type pin, embedding_status 0 and no vector row, visible via list and keyword search" + requirement: INTEROP-02 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/store.rs#insert_imported_preserves_provenance_and_lands_pending" + status: pass + - kind: integration + ref: "cargo test -p agent-memory-core --test store --test ttl --test decay --test update --test backfill_race (insert refactor behaviour-preserving)" + status: pass + human_judgment: false + - id: D5 + description: "README scopes storage-error redaction to every request the daemon serves and every CLI import/export, and lists it as behavioral change 4 for v0.1.0" + requirement: INTEROP-02 + verification: + - kind: other + ref: "grep -c 'failed due to a storage error (details are in the server log)' README.md -> 2; sed -n '/^## Behavioral changes/,/^## /p' README.md | grep -c '^4\\. ' -> 1" + status: pass + human_judgment: true + rationale: "Presence is grep-proven, but whether the release-note wording is clear enough for clients that parsed `sqlite error: …` strings is an editorial judgment" + +# Metrics +duration: 14min +completed: 2026-10-25 +status: complete +--- + +# Phase 5 Plan 01: Redact storage-engine text on every service path; teach the store the format-v1 contract Summary + +**`redact_storage_error(op, e)` on all nine `MemoryService` methods (closing 04-REVIEW IN-01 on REST, MCP and CLI with 13 in-process proofs), plus the store contract the export/import tracers build on: `Store::find_by_key -> Option`, `LinkOutcome::Linked { created }`, and `ImportedMemory` + `Store::insert_imported` behind one shared `insert_row`.** + +## Performance + +- **Duration:** 14 min +- **Started:** 2026-10-25T11:24:54Z +- **Completed:** 2026-10-25T11:39:13Z +- **Tasks:** 3 +- **Files modified:** 10 (1 created, 9 modified; +928 / −69 lines) + +## Accomplishments + +- **D-14 landed before any new write path exists.** `redact_search_error` became `redact_storage_error(op, e)`; every public `MemoryService` method — `store`, `list`, `search`, `import`, `forget`, `link`, `unlink`, `update`, `sweep` — now delegates to an `async fn _unredacted` body and maps its result with the op string, so no early `?` return can escape unredacted. The storage tier (`Sqlite | Pool | Join | Migration`) is logged at error level with full detail and replaced by `Internal(" failed due to a storage error (details are in the server log)")`; client tier and crate-authored text pass through. `SEARCH_STORAGE_FAILURE` keeps its exact text and a unit test pins it to the template at `op = "search"`. Both transport mappers are byte-identical. +- **Proven on every surface class, in process.** `tests/redaction.rs` takes `memories` (or `memory_links` for `link`/`unlink`, so the endpoint pre-check still passes) offline with the T-04-19 `ALTER TABLE … RENAME` recipe and asserts the exact redacted text and no `sqlite` substring for all eight non-search ops. REST `GET /health` and `POST /api/memories` return 500 with `list failed…` / `store failed…`; MCP `memory_store` returns `internal_error` with `store failed…`. All Phase 4 goldens (`hybrid.rs` equality on the constant, the two existing search redaction tests) are unchanged and green. +- **`Store::find_by_key` replaces `Store::exists`** (one caller, one impl): same NULL-safe `source IS ?1` three-column key (CR-01), now `SELECT id` so plan 03's importer can map an export-local key onto the existing row (D-07). `import()` dedups through `.is_some()`; `tests/import.rs` is untouched and green. +- **`LinkOutcome::Linked { created: bool }`** read from `tx.changes() == 1` before `commit`. Seven sites updated: the store impl, the MCP and REST arms (`Linked { .. }`, JSON byte-identical), and four `relations.rs` asserts that are now stronger idempotency proofs (`created: true` on the first insert, `created: false` on the D-02 duplicate and the symmetric `b→a` re-link). Not a wire change. +- **`ImportedMemory` + `Store::insert_imported(m, now)`** — a dedicated type (zero of the 15 `NewMemory` construction sites touched) carrying absolute `created_at`, `access_count`, `expires_at`; the store writes `last_accessed = now`, `decay_score = 1.0`, `base_weight` from the pin, `embedding_status = 0` and NO vector row (D-10, D-13). The single `memories` INSERT moved into a private `insert_row(tx, &MemoryRow)` helper used by both `insert` and `insert_imported`, with `access_count`/`created_at`/`last_accessed` bound separately (today's `?7` double-bind and literal `0` are gone). `grep -v '^\s*//' sqlite.rs | grep -c "INTO memories "` is exactly 1. The trait doc names both expiry predicates (importer `<= now` skip vs sweep `< now` delete). +- **README:** the search-only redaction sentence is widened to "any request the daemon serves — and any CLI `import`/`export`"; "Behavioral changes in v0.1.0" now lists four changes, item 4 telling clients that parsed `sqlite error: …` strings to stop. +- **No migration, no dependency change:** `sql/` diff empty, `LATEST_SCHEMA_VERSION = 3`, `Cargo.lock` byte-identical under `--locked`. + +## Task Commits + +Each task was committed atomically (fmt + clippy `-D warnings` + full workspace suite green before each): + +1. **Task 1 (tracer): A storage fault on any service path reaches REST, MCP and the CLI only as the redacted internal error** — `00d8080` (feat) +2. **Task 2: The dedup probe returns the matching row id, and a link insert reports whether it created the edge** — `9ee42ed` (feat) +3. **Task 3: The store accepts a restored memory with its provenance preserved and always pending embedding** — `5994d38` (feat) + +**Plan metadata:** see the `docs(05-01)` commit that follows this summary. + +## Files Created/Modified + +- `crates/agent-memory-core/src/service.rs` — `redact_storage_error(op, e)` (exhaustive, no catch-all), nine `*_unredacted` inner fns, `SEARCH_STORAGE_FAILURE` doc reworded, `#[cfg(test)] mod tests` with the two pinning tests; `import_unredacted` dedups via `find_by_key(..)?.is_some()` +- `crates/agent-memory-core/tests/redaction.rs` (new) — harness copied from relations.rs, `take_offline(db_path, table)`, `assert_redacted(err, op)`, eight `#[tokio::test]`s +- `crates/agent-memory-core/src/domain.rs` — `LinkOutcome::Linked { created: bool }`; `pub struct ImportedMemory` directly after `NewMemory` +- `crates/agent-memory-core/src/store/mod.rs` — `fn find_by_key(..) -> Result, _>` (CR-01 paragraph kept, D-07 sentence added); `fn insert_imported(m, now)` with the preserved/refreshed column list and both expiry predicates; `link` doc reworded to `created: false` +- `crates/agent-memory-core/src/store/sqlite.rs` — `struct MemoryRow<'a>` + `fn insert_row`; `insert` rebuilt on it (behaviour identical); `insert_imported`; `find_by_key` (`SELECT id`, same `source IS ?1`); `link` reads `tx.changes()` before commit +- `crates/agent-memory-core/tests/relations.rs` — four asserts upgraded to `Linked { created: true | false }` +- `crates/agent-memory-core/tests/store.rs` — `find_by_key_returns_the_matching_id_null_safely`, `insert_imported_preserves_provenance_and_lands_pending` (+ `RawRow`/`raw_row`/`vec_rows` raw-connection helpers) +- `crates/agent-memory/src/rest/handlers.rs` — `Linked { .. }` arm; tests `health_storage_failure_is_500_without_sqlite_text`, `store_storage_failure_is_500_without_sqlite_text` (+ `take_memories_offline` helper); `map_memory_error` unchanged +- `crates/agent-memory/src/mcp.rs` — `Linked { .. }` arm; test `memory_store_storage_failure_is_internal_error_without_sqlite_text`; `map_mcp_error` unchanged +- `README.md` — widened redaction sentence; "Four deliberate contract changes" + item 4 + +## Decisions Made + +- **Seam, not mappers.** 04-REVIEW IN-01 proposed redacting inside the two transport mappers; the plan (D-14) chose the service seam and that is what shipped: one helper, nine call sites, MCP + REST (`/health` included) + CLI covered together, mappers untouched. Widening/narrowing the redacted set later is a local edit. +- **The nine op strings** are exactly `store`, `list`, `search`, `import`, `forget`, `link`, `unlink`, `update`, `sweep`. Plans 02/03 add `export` and reuse `import`. +- **`insert_row` takes a struct, not positional args** — keeps the helper under clippy's `too_many_arguments` limit and makes a transposed column a compile error rather than a silent swap. +- **Tracer feedback gate:** auto mode is off, but `.planning/config.json` sets `human_verify_mode: "end-of-phase"` and the tracer's `` is wholly ``. I re-ran it end-to-end after the Task 1 commit (lib redaction tests, `--test redaction`, `--test hybrid`, `--bin agent-memory storage_failure`, workspace clippy — all green) and continued to the expansion tasks instead of returning a mid-plan `checkpoint:human-verify`. Recorded here so the orchestrator can route the end-of-phase UAT. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 3 - Blocking] Renamed the two service.rs unit tests so the acceptance grep pins exactly one helper** +- **Found during:** Task 1 (acceptance-criteria gate) +- **Issue:** The plan prescribes the test names `redact_storage_error_search_text_equals_the_constant` / `redact_storage_error_passes_client_and_crate_text_through` AND the criterion `grep -c "fn redact_storage_error" service.rs` → `1`. Both cannot hold: the test names begin with `fn redact_storage_error`, so the grep returned 3. +- **Fix:** Tests renamed to `storage_error_redaction_search_text_equals_the_constant` and `storage_error_redaction_passes_client_and_crate_text_through`; bodies exactly as specified in ``. +- **Files modified:** `crates/agent-memory-core/src/service.rs` +- **Verification:** grep → 1; `cargo test -p agent-memory-core --lib storage_error_redaction` 2 passed +- **Committed in:** `00d8080` + +**2. [Rule 3 - Blocking] Reflowed two README passages so the redaction template sits on one line** +- **Found during:** Task 1 (acceptance-criteria gate) +- **Issue:** The widened sentence and item 4 wrapped `… storage error\n(details are in the server log)`, so the one-line grep `failed due to a storage error (details are in the server log)` returned 0. +- **Fix:** Line breaks moved so the full template phrase is on a single line in both places (wording unchanged). +- **Files modified:** `README.md` +- **Verification:** grep → 2; `sed -n '/^## Behavioral changes/,/^## /p' README.md | grep -c '^4\. '` → 1 +- **Committed in:** `00d8080` + +### Process notes (not code deviations) + +- **TDD RED/GREEN without a separate RED commit.** All three tasks are `tdd="true"`. RED was executed and observed — Task 1's eight redaction tests were run before implementation and failed with the raw `Sqlite(… no such table …)` leak IN-01 describes; Tasks 2/3 introduce new API (`find_by_key`, `Linked { created }`, `insert_imported`) so their tests fail to compile against the old code by construction. The failing tests were NOT committed on their own because the project operating rules forbid committing failing tests and the plan calls for exactly one commit per task; each task's `feat(05-01)` commit carries test + implementation together. + +--- + +**Total deviations:** 2 auto-fixed (both Rule 3, both plan-internal criterion conflicts; no behaviour change) +**Impact on plan:** None on scope or behaviour. Both fixes exist only to make the plan's own acceptance greps pass as written. + +## Issues Encountered + +None. Every targeted `` and the full workspace suite passed on the first run after each implementation (post-RED); no fix-attempt budget was used. + +## Verification + +Plan-level `` re-run after the third commit: + +- `cargo fmt --all -- --check` — clean; `cargo clippy --workspace --all-targets --locked -- -D warnings` — clean; `cargo test --workspace --locked` — **19 suites, 232 passed, 0 failed, 2 ignored** (230 after Task 1, 231 after Task 2, 232 after Task 3). +- `cargo test -p agent-memory-core --test redaction` — 8 passed; `cargo test -p agent-memory --bin agent-memory storage_failure` — 5 passed (health, store on REST, memory_store on MCP, plus the two existing search tests). +- `cargo test -p agent-memory-core --test relations --test import --test store` — 27 + 2 + 7 passed with the stronger `created` asserts and the two new store tests. +- `git diff --stat -- crates/agent-memory-core/sql/` empty; `Cargo.lock` unchanged; `LATEST_SCHEMA_VERSION: i64 = 3`. +- Transport mappers: `git diff` of `handlers.rs` / `mcp.rs` across the plan is confined to the `#[cfg(test)]` modules plus the two `Linked { .. }` arms. + +## Known Stubs + +None — no placeholder values, TODO/FIXME markers, skipped tests, or unwired data paths were introduced. + +## Threat Flags + +None. No new network endpoint, auth path, file access or schema surface was added; T-05-01 (information disclosure via engine text) is mitigated as planned, T-05-15 (divergent INSERTs) is structurally closed by `insert_row`, T-05-16's two predicates are fixed in the trait doc for plan 03 to enforce. + +## User Setup Required + +None - no external service configuration required. + +## Next Phase Readiness + +- **Plan 05-02 (export tracer)** can add `MemoryService::export` as `export_unredacted` + `redact_storage_error("export", e)` and read `meta` through the store; the inner-fn shape and op-string convention are established. +- **Plan 05-03 (import tracer)** compiles against a tested contract: `find_by_key` for key→existing-id mapping (D-07), `insert_imported` for restored rows (D-10/D-13, caller owns the `expires_at <= now` skip), and `Linked { created: false }` to count `links_skipped_duplicates` (D-12). +- **Plan 05-04 (docs)** inherits a README that already states the every-surface redaction and behavioral change 4; it only needs the "Export & import" section. +- No blockers. `.planning/config.json` and the untracked harness files were left unstaged as instructed. + +--- +*Phase: 05-portable-export-import* +*Completed: 2026-10-25* + +## Self-Check: PASSED + +All 6 key files exist on disk; task commits 00d8080, 9ee42ed, 5994d38 are present in git history. diff --git a/.planning/phases/05-portable-export-import/05-02-PLAN.md b/.planning/phases/05-portable-export-import/05-02-PLAN.md new file mode 100644 index 0000000..621c3c1 --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-02-PLAN.md @@ -0,0 +1,339 @@ +--- +phase: 05-portable-export-import +plan: 02 +type: execute +wave: 2 +depends_on: [05-01] +files_modified: + - crates/agent-memory-core/src/export.rs + - crates/agent-memory-core/src/lib.rs + - crates/agent-memory-core/src/store/mod.rs + - crates/agent-memory-core/src/store/sqlite.rs + - crates/agent-memory-core/src/service.rs + - crates/agent-memory-core/tests/export_import.rs + - crates/agent-memory/src/main.rs + - crates/agent-memory/src/export_file.rs +autonomous: true +requirements: [DIST-04] + +estimate: + tokens: 50000 + raw_tokens: 50000 + tasks: 2 + confidence: low + +must_haves: + truths: + - "`agent-memory export` writes line 1 as the header record {record:'header', format:'agent-memory-export', version:1, exported_at, embedding_model, embedding_dim, schema_version, agent_memory_version, exported_rows, exported_links}; embedding_model and embedding_dim are read from the meta table of the source database, exported_at from the injected clock, agent_memory_version from the binary's CARGO_PKG_VERSION (D-03)" + - "Every memory record carries exactly record, key, type, content, tags, source, scope, created_at, last_accessed, access_count, expires_at in that order — key is the source row id used only as an export-local key; decay_score, base_weight, embedding_status and every vector are excluded as derived data (D-04)" + - "Every link record carries record, from, to, kind, created_at with endpoints exactly as stored (relates_to as (min,max), directed kinds as given) and is emitted after all memory records (D-05)" + - "Export writes every row present in the database — expired rows included, no scope or type filter — in ascending id order through one streaming read; nothing buffers the corpus as a single document (D-06, D-02)" + - "Export is a CLI subcommand only; no MCP tool and no REST route exposes it (D-01)" + - "Exporting an empty database writes exactly one header line with exported_rows: 0 and exported_links: 0 and reports 0 memories and 0 links" + - "Records are emitted in ascending id order, links after all memories ordered by (from_id, to_id, kind), and two exports of an unchanged database under the same clock are byte-identical" + - statement: "Memories and links are read on one pooled connection inside one read transaction, so a concurrent link cannot produce a link record whose key is absent from the file, and --output writes a same-directory temp file renamed into place so an interrupted export never leaves a partial target" + verification: backstop + - "`export --output PATH` creates `PATH.export-tmp-` in PATH's own directory with mode 0600 on Unix (create_new, never a shared tmp dir), renames it over PATH — replacing an existing file — only after the stream is flushed and synced, removes the temp on any failure, and stdout is the default sink (D-02)" + - "An export storage fault reaches the CLI only as `export failed due to a storage error (details are in the server log)`; a writer I/O failure is the crate-authored `export write failed: …` (D-14)" + artifacts: + - path: "crates/agent-memory-core/src/export.rs" + provides: "format v1 single definition: FORMAT_NAME, FORMAT_VERSION, Record enum (Header/Memory/Link), ExportSnapshot, ExportEvent, module-doc spec, golden tests" + contains: "pub enum Record" + - path: "crates/agent-memory-core/src/store/mod.rs" + provides: "Store::export_rows — one snapshot, visitor-driven streaming read" + contains: "fn export_rows" + - path: "crates/agent-memory-core/src/service.rs" + provides: "MemoryService::export(out, agent_memory_version) -> ExportReport, redacted with op export" + contains: "pub async fn export" + - path: "crates/agent-memory/src/main.rs" + provides: "Command::Export { output } and run_export" + contains: "fn run_export" + - path: "crates/agent-memory/src/export_file.rs" + provides: "PendingExport: private temp file (0600) + commit (sync + rename) + abort" + contains: "pub struct PendingExport" + - path: "crates/agent-memory-core/tests/export_import.rs" + provides: "export proofs: header from meta/clock, empty DB, ordering, excluded fields, expired rows exported, byte-stable" + contains: "exported_rows" + key_links: + - from: "crates/agent-memory/src/main.rs run_export" + to: "MemoryService::export" + via: "stdout or PendingExport writer boxed as Box, version from env!(\"CARGO_PKG_VERSION\")" + pattern: "service\\s*\\.export\\(" + - from: "crates/agent-memory-core/src/service.rs export_unredacted" + to: "Store::export_rows" + via: "one spawn_blocking hop; Snapshot event builds the header, Record events are written one line each" + pattern: "export_rows\\(" + - from: "crates/agent-memory-core/src/store/sqlite.rs export_rows" + to: "meta table + memories + memory_links" + via: "one read transaction: counts and pin first, then SELECT ... ORDER BY id, then SELECT ... ORDER BY from_id, to_id, kind" + pattern: "FROM meta WHERE key" + - from: "crates/agent-memory/src/main.rs run_export" + to: "export_file::PendingExport::commit / abort" + via: "commit after Ok(report), abort on Err" + pattern: "pending\\.(commit|abort)\\(" + prohibitions: + - statement: "Embedding vectors must never travel in the export file — no vec_memories read, no embedding key in any record" + status: resolved + verification: test + - statement: "Export must never be exposed over MCP or REST in this phase — no tool, no route; the corpus never crosses the JSON-RPC stdout channel" + status: resolved + verification: test + - statement: "Export must never silently filter or skip rows — no scope/type filter and no expiry skipping at export time; every row in the table is written" + status: resolved + verification: test +--- + +# Plan 05-02: `agent-memory export` — format v1 JSONL, streamed, atomic and private on disk + +## Phase Goal + +**As a** user, **I want to** export all my memories to a portable file and import them on another machine, **so that** my agent's memory moves with me — across machines, with no cloud sync. + +This plan is the first half of that story: the user can run `agent-memory export` and get a versioned JSONL file — header, one record per memory, one record per link, no vectors — on stdout or atomically written to a private file. + + +1. **Tracer:** define format v1 once (`export.rs`: `Record` enum + consts + module-doc spec, golden-tested), add `Store::export_rows` (ONE pooled connection, ONE read transaction: meta pin + counts, then memories ascending id, then links), add `MemoryService::export(out, version)` streaming one line per record through a boxed writer in one `spawn_blocking` hop, and wire `agent-memory export` to stdout. The end-to-end check is the real binary printing a valid header line on an empty database. +2. **Expansion:** `--output PATH` through `export_file::PendingExport` — `create_new` + mode 0600 temp in the target's directory, sync + rename after the stream is flushed, temp removed on failure. + +Purpose: the on-disk contract (D-03..D-06) is frozen exactly once, in one Rust type, written by a path that cannot buffer the corpus or straddle two snapshots, and the file never exists in a world-readable or half-written state. +Output: `export.rs`, `Store::export_rows`, `ExportReport` + `MemoryService::export`, `Command::Export { output }`, `export_file.rs`, `tests/export_import.rs` (export half). + + + +@$HOME/.claude/gsd-core/workflows/execute-plan.md +@$HOME/.claude/gsd-core/templates/summary.md + + + +@.planning/STATE.md +@.planning/phases/05-portable-export-import/05-CONTEXT.md +@.planning/phases/05-portable-export-import/05-RESEARCH.md +@.planning/phases/05-portable-export-import/05-PATTERNS.md +@.planning/phases/05-portable-export-import/05-01-SUMMARY.md +@crates/agent-memory-core/src/service.rs +@crates/agent-memory-core/src/store/sqlite.rs +@crates/agent-memory/src/main.rs + + +## Execution notes for every task + +- Gate before EVERY commit: `cargo fmt --all`, `cargo clippy --workspace --all-targets --locked -- -D warnings`, `cargo test --workspace --locked` — run clippy and the full suite in the background with a log file and poll (watchdog ~600 s). +- One commit per task; stage explicit paths only. NEVER stage `CLAUDE.md`, `.claude/`, `.mcp.json`, `.gitignore`, `.gsd/`, `docs/`, `10-agent-memory.md`, `claude-code-harness-engineering-guide-v2.md`, `.planning/research/.cache/`, `.planning/config.json`, `.planning/milestone.lock`. +- No `unwrap()`/`expect()` in production code; `anyhow::Context` at the binary edges. No new crate; `Cargo.lock` unchanged. +- Plan 05-01 already landed `redact_storage_error(op, e)` and the `*_unredacted` shape — the new `export` method MUST use it with `op = "export"`. +- Research correction: `std::io::StdoutLock` is `!Send`, so it cannot live in a `Box`. The stdout sink is `BufWriter::new(std::io::stdout())` (the `Stdout` handle is `Send + Sync`; `BufWriter` keeps per-call locking cheap). +- Never change the CI runner or add a workflow. Never push, tag or release. + +## Interface contracts (established by this plan) + +**`crates/agent-memory-core/src/export.rs`** (new; `pub mod export;` in lib.rs between `embed` and `import`) + +```text +pub const FORMAT_NAME: &str = "agent-memory-export"; +pub const FORMAT_VERSION: u32 = 1; + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(tag = "record", rename_all = "lowercase", deny_unknown_fields)] +pub enum Record { + Header { format: String, version: u32, exported_at: i64, embedding_model: String, embedding_dim: u32, + schema_version: i64, agent_memory_version: String, exported_rows: u64, exported_links: u64 }, + Memory { key: i64, #[serde(rename = "type")] mem_type: String, content: String, tags: Vec, + source: Option, scope: Option, created_at: i64, last_accessed: i64, + access_count: i64, expires_at: Option }, + Link { from: i64, to: i64, kind: String, created_at: i64 }, +} + +pub struct ExportSnapshot { pub embedding_model: String, pub embedding_dim: u32, pub memories: u64, pub links: u64 } +pub enum ExportEvent { Snapshot(ExportSnapshot), Record(Record) } +``` +- Field order on the wire = declaration order with `record` first (serde-verified in RESEARCH Pattern 1). `type`/`kind` are `String` so conversion runs through `MemoryType::try_from` / `LinkKind::try_from` in the importer (D-09). +- `exported_rows`/`exported_links` are the orchestrator's resolution 1: a cheap truncation check, decided now so it is inside v1 (`deny_unknown_fields` would reject it later). +- `ExportSnapshot`/`ExportEvent` are the store→service streaming protocol, not wire types (no serde derives needed). + +**`Store::export_rows`** (`store/mod.rs`) + +```text +fn export_rows(&self, visit: &mut dyn FnMut(ExportEvent) -> Result<(), MemoryError>) -> Result<(), MemoryError>; +``` +Contract: one pooled read connection; one read transaction (`conn.unchecked_transaction()?`; fallback `execute_batch("BEGIN")` … `COMMIT`); emits exactly ONE `ExportEvent::Snapshot` first (meta pin + `count(*)` of `memories` and `memory_links`), then one `ExportEvent::Record(Record::Memory { .. })` per row of `SELECT id, mem_type, content, tags, source, scope, created_at, last_accessed, access_count, expires_at FROM memories ORDER BY id`, then one `ExportEvent::Record(Record::Link { .. })` per row of `SELECT from_id, to_id, kind, created_at FROM memory_links ORDER BY from_id, to_id, kind`. Rows are visited lazily from `query_map` — never collected into a `Vec`. The single snapshot is what makes a concurrent `link` unable to produce a dangling key (PITFALLS 5; the FK guarantees every endpoint is in the same snapshot). `embedding_dim` is TEXT in `meta` and is parsed to `u32`; a non-integer is `MemoryError::Internal("meta.embedding_dim is not an integer (…)")` — never a client error. `tags` decode with `serde_json::from_str::>(..).unwrap_or_default()` (the `row_to_view` convention). `mem_type` and `kind` are passed through as the stored strings. + +**`MemoryService::export`** (`service.rs`) + +```text +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Serialize)] +pub struct ExportReport { pub memories: u64, pub links: u64 } + +pub async fn export(&self, out: Box, agent_memory_version: String) -> Result +``` +`export` = `export_unredacted(..).await.map_err(|e| redact_storage_error("export", e))`. Body: `let now = self.clock.now();` then ONE `spawn_blocking` that moves `out`, `store`, `version`: on `Snapshot(s)` build `Record::Header { format: FORMAT_NAME.into(), version: FORMAT_VERSION, exported_at: now, embedding_model: s.embedding_model, embedding_dim: s.embedding_dim, schema_version: LATEST_SCHEMA_VERSION, agent_memory_version: version.clone(), exported_rows: s.memories, exported_links: s.links }`, record the counts, write it; on `Record(r)` write it; after `export_rows` returns, `out.flush()`. `write_line` = `serde_json::to_writer(&mut *out, &rec)` then `out.write_all(b"\n")`; any `serde_json::Error`/`io::Error` → `MemoryError::Internal(format!("export write failed: {e}"))` (crate-authored, passes redaction unchanged). `LATEST_SCHEMA_VERSION` is `crate::store::migrations::LATEST_SCHEMA_VERSION`. + +**CLI** (`main.rs`) + +- `Command::Export { #[arg(long)] output: Option }` — doc: "Export every memory and link as format-v1 JSONL (DIST-04): a header line, one record per memory, one record per link; embedding vectors are never included. Writes to stdout by default." `--output` doc: "Write to PATH instead of stdout. The file is created private (0600 on Unix) as a temp file in PATH's directory and renamed into place; an existing PATH is replaced; an interrupted export never leaves a partial PATH." +- `async fn run_export(db_flag: Option, ollama_url: String, output: Option) -> anyhow::Result<()>`: resolve DB path, open store, build the service exactly like `run_import` (the embedder is constructed but never called by export), pick the sink, call `service.export(sink, env!("CARGO_PKG_VERSION").to_string())`, then `eprintln!("exported {} memories and {} links", report.memories, report.links)` — ALWAYS stderr, because stdout may be the stream. + +**`crates/agent-memory/src/export_file.rs`** (new; `mod export_file;` in main.rs) + +```text +pub struct PendingExport { temp: PathBuf, target: PathBuf, file: std::fs::File } +pub fn export_temp_path(target: &Path) -> PathBuf // target.with_file_name("{name}.export-tmp-{pid}") +pub fn begin(target: &Path) -> std::io::Result // create_private(&temp) +impl PendingExport { + pub fn writer(&self) -> std::io::Result // self.file.try_clone() + pub fn commit(self) -> std::io::Result<()> // sync_all, then rename(temp, target); on rename error remove temp (best-effort) and return the error + pub fn abort(self) // best-effort remove_file(temp); warn via tracing on a non-NotFound error +} +#[cfg(unix)] fn create_private(path) -> io::Result // OpenOptions write(true).create_new(true).mode(0o600) via std::os::unix::fs::OpenOptionsExt +#[cfg(not(unix))] fn create_private(path) -> io::Result // same without mode; Windows ACL inheritance is the Phase 6 seam (DIST-03), mirroring config.rs::create_data_dir +``` + +## Artifacts this phase produces (this plan's share) + +- `crates/agent-memory-core/src/export.rs` (new): `FORMAT_NAME`, `FORMAT_VERSION`, `enum Record { Header, Memory, Link }`, `struct ExportSnapshot`, `enum ExportEvent`, module-doc format spec, golden tests. +- `crates/agent-memory-core/src/lib.rs`: `pub mod export;`. +- `store/mod.rs`: `fn export_rows(...)`; `sqlite.rs`: its implementation. +- `service.rs`: `pub struct ExportReport`, `pub async fn export(...)`, `async fn export_unredacted(...)`, private `fn write_record_line(...)`. +- `crates/agent-memory/src/main.rs`: `Command::Export { output }`, `async fn run_export(...)`, `mod export_file;`. +- `crates/agent-memory/src/export_file.rs` (new): `PendingExport`, `export_temp_path`, `begin`, `create_private`, unit tests. +- `crates/agent-memory-core/tests/export_import.rs` (new): harness + `export_to_string` helper + export tests (plan 03 appends the import tests). +- New CLI surface: `agent-memory export [--output PATH]`. + + + + + Task 1 (tracer): `agent-memory export` streams a valid format-v1 JSONL document to stdout — one path, every layer + crates/agent-memory-core/src/export.rs, crates/agent-memory-core/src/lib.rs, crates/agent-memory-core/src/store/mod.rs, crates/agent-memory-core/src/store/sqlite.rs, crates/agent-memory-core/src/service.rs, crates/agent-memory/src/main.rs, crates/agent-memory-core/tests/export_import.rs + + - .planning/phases/05-portable-export-import/05-RESEARCH.md: Pattern 1 (verified serde behaviour of the tagged enum), Pattern 4 (one-snapshot streaming), Pitfalls 8 and 9, Code Examples 1, 4, 5. + - .planning/phases/05-portable-export-import/05-PATTERNS.md: sections for `export.rs`, `service.rs`, `store/mod.rs`, `sqlite.rs`, `main.rs`, `tests/export_import.rs`. + - crates/agent-memory/src/rest/handlers.rs lines 104-117 (`UpdateBody` — the `deny_unknown_fields` DTO doc style) and crates/agent-memory-core/src/domain.rs lines 186-195 (`LinkDirection` — `rename_all = "lowercase"` derive style). + - crates/agent-memory-core/src/store/mod.rs lines 75-96 (`pending_embeddings` + the now-`find_by_key` doc) and sqlite.rs lines 735-749 (`query_map` streaming shape), 443-470 (`row_to_view` column conventions), 774-803 (`list` SELECT column list to start from). + - crates/agent-memory-core/src/store/migrations.rs line 17 (`pub const LATEST_SCHEMA_VERSION: i64 = 3`). + - crates/agent-memory-core/sql/0002_embeddings.sql lines 20-26 (`meta` is `(key TEXT PRIMARY KEY, value TEXT NOT NULL)` seeded with `embedding_model` / `embedding_dim`). + - crates/agent-memory-core/src/service.rs: the `search`/`search_unredacted` split and `redact_storage_error` (plan 05-01), `ImportReport` derive style, the `spawn_blocking … .map_err(MemoryError::Join)??` hop idiom. + - crates/agent-memory/src/main.rs lines 60-127 (`Command` enum + dispatch) and 221-258 (`run_import` — copy its store/service wiring). + - crates/agent-memory-core/tests/relations.rs lines 20-104 (harness to copy into the new test file). + + + - **export.rs unit tests (`#[cfg(test)]`):** `Record::Memory { key: 7, mem_type: "DECISION", content: "c", tags: ["a"], source: None, scope: Some("s"), created_at: 1, last_accessed: 2, access_count: 3, expires_at: None }` serialises to exactly `{"record":"memory","key":7,"type":"DECISION","content":"c","tags":["a"],"source":null,"scope":"s","created_at":1,"last_accessed":2,"access_count":3,"expires_at":null}`; a header and a link record likewise serialise in declaration order with `record` first; deserialising a memory line with an extra `"bogus":1` fails with a message containing `unknown field`; `{"record":"memo","key":1}` fails with `unknown variant`; a memory line missing `source`/`scope`/`expires_at` parses with those as `None`; `"key":"1"` fails; `"version":1.5` fails on the header. + - **tests/export_import.rs (new, in-process, failing `FakeEmbedder`, `TestClock::new(1_000)`):** + - `export_of_an_empty_database_is_exactly_one_header_line`: output has exactly 1 line; parsed as `Record::Header` it has `format == "agent-memory-export"`, `version == 1`, `exported_at == 1_000`, `embedding_model == "nomic-embed-text"`, `embedding_dim == 768`, `schema_version == 3`, `agent_memory_version == "test-0.0.0"` (the literal passed in), `exported_rows == 0`, `exported_links == 0`; the returned `ExportReport == { memories: 0, links: 0 }`. + - `export_writes_every_memory_ascending_then_every_link_as_stored`: store three memories advancing the clock between them (one with tags + scope + source + a far-future ttl, one Pattern type, one with no tags); `link(a, b, Supersedes)` and `link(c, b, RelatesTo)`; export → line 1 header with `exported_rows 3`, `exported_links 2`; lines 2-4 are `Record::Memory` in ascending `key` == stored id with `type`, `content`, `tags`, `source`, `scope`, `created_at`, `last_accessed`, `access_count`, `expires_at` (== created_at + ttl where set, else null) matching the rows read through `list`; lines 5-6 are `Record::Link` ordered by `(from, to, kind)` — the `relates_to` edge has `from == min(b, c)` and `to == max(b, c)`, the `supersedes` edge has `from == a, to == b`; every link `created_at` equals the clock value at link time; `ExportReport == { 3, 2 }`. + - `memory_records_carry_exactly_the_documented_keys`: parsing each memory line as `serde_json::Value`, its object has exactly the 11 keys `record,key,type,content,tags,source,scope,created_at,last_accessed,access_count,expires_at` — no `decay_score`, `base_weight`, `embedding_status`, `embedding`. + - `expired_rows_are_still_exported`: a memory with `ttl_secs: Some(10)`, clock advanced by 100 with NO sweep → it appears in the export with its `expires_at` (D-06: the importer owns expiry policy). + - `two_exports_of_an_unchanged_database_are_byte_identical`: export twice without advancing the clock → identical strings. + - **End-to-end (the tracer's own check):** on a fresh database the real binary prints exactly one stdout line starting with `{"record":"header","format":"agent-memory-export","version":1,` and reports `exported 0 memories and 0 links` on stderr. + + + Implements D-03, D-04, D-05, D-06 and the stdout half of D-02 per RESEARCH Patterns 1 and 4; D-01 (CLI only); the orchestrator's resolution 1 (header counts). The format v1 contract is pre-approved as one-way in 05-CONTEXT.md (D-03) — record the rating, do not pause. + + 1. **export.rs (new).** Write the module doc as the format v1 specification: line 1 is the mandatory header; every later line is one JSON object with a `record` tag; a table per record type listing the fields in wire order with types and meaning; `key` is the source row id used ONLY as an export-local key and never written as an id on import; excluded derived data (`decay_score`, `base_weight`, `embedding_status`, vectors) and why (re-computed on read / re-embedded locally against the local model pin); `relates_to` endpoints appear as stored `(min, max)`; links come after all memories; optional fields are always written explicitly as `null` by the exporter but may be omitted by a hand-written file; the reader is strict (`deny_unknown_fields`, closed `record` set) and version `> 1` is refused with an upgrade message — point at `import::jsonl` for the reader rules and at the README "Export & import" section. Then the consts, `Record`, `ExportSnapshot`, `ExportEvent` per "Interface contracts", with `///` docs on every public item, and the unit tests in `` (follow `gsd_state.rs:188-200` for the in-module test style). + 2. **lib.rs:** `pub mod export;` between `embed` and `import`. + 3. **store/mod.rs:** add `fn export_rows(...)` after `pending_embeddings` with the full contract from "Interface contracts" in its doc, including the one-snapshot rationale (PITFALLS 5) and the `embedding_dim` parse rule. + 4. **sqlite.rs:** implement `export_rows`: `let conn = self.reads.get().map_err(MemoryError::Pool)?; let tx = conn.unchecked_transaction()?;` (if `unchecked_transaction` is unavailable on the pooled connection type, `conn.execute_batch("BEGIN")?` … `COMMIT`); read `SELECT value FROM meta WHERE key = 'embedding_model'` and `… = 'embedding_dim'` (parse `u32`, map failure to the `Internal` text above); `SELECT count(*) FROM memories` and `SELECT count(*) FROM memory_links` as `i64` → `u64`; `visit(ExportEvent::Snapshot(..))?`; then `prepare` + `query_map` over the memories SELECT, mapping each row to `Record::Memory { key: id, mem_type: , .. }` and calling `visit(ExportEvent::Record(..))?` per row inside the `for row in rows` loop; then the links SELECT the same way; `tx.commit()?` (a read-only tx; `finish` is equivalent). All SQL is static text with no interpolation. Add a comment naming the memories column list as the D-04 contract (no `base_weight`/`decay_score`/`embedding_status`). + 5. **service.rs:** add `ExportReport`, `pub async fn export(...)` + `async fn export_unredacted(...)` + a private `fn write_record_line(out: &mut dyn Write, rec: &Record) -> Result<(), MemoryError>` per "Interface contracts". The header is built from the `Snapshot` event — the counts come from the same read transaction as the rows. `out.flush()` happens INSIDE the blocking task before `Ok(report)` (Pitfall 8: a `BufWriter` dropped without flush loses the tail). Doc the method: streams one line per record, never buffers the corpus (D-02); `agent_memory_version` is a parameter because the binary and core crates have separate versions (Pitfall 9); errors tiered by `redact_storage_error("export", _)` (D-14). + 6. **main.rs:** add the `Command::Export` variant WITHOUT fields in this task (`Export,`) with the doc from "Interface contracts" minus the `--output` sentence, dispatch it to `run_export(cli.db, cli.ollama_url)`, and implement `run_export` with the stdout sink `Box::new(std::io::BufWriter::new(std::io::stdout()))`, the version from `env!("CARGO_PKG_VERSION")`, and the stderr report line. Do NOT add `mod export_file;` or the `output` field here — both land in Task 2 (adding a flag later is additive, not a stub). `anyhow::Context` on every edge (`"resolving database path"`, `"opening database at …"`, `"exporting"`). + 7. **tests/export_import.rs (new).** Module doc: DIST-04 / INTEROP-02 proof file. Copy the relations.rs harness (`Harness`, `harness`, `harness_with`, `new_memory`, `store`, `raw_conn`, `edge_count`, `edges_touching`). Add `async fn export_to_string(h: &Harness) -> (String, ExportReport)` that exports into a temp file under `h._dir` through `Box::new(std::fs::File::create(&path)…)` (or an `Arc>>` `Write` wrapper) and reads it back; pass `"test-0.0.0"` as the version. Add a `fn lines(text: &str) -> Vec` helper that `serde_json::from_str`s each line. Then the five tests in ``. + 8. `cargo fmt --all`; clippy; full tests (background + poll); commit with explicit paths: `feat(05-02): agent-memory export streams format v1 JSONL to stdout (D-03..D-06)`. + + The moment a user writes a file, format v1 is a compatibility contract: any field removal or meaning change needs `version: 2` plus a reader that still accepts 1. Pre-approved by the user in 05-CONTEXT.md (D-03) and by the orchestrator's unattended-run instruction, so no `checkpoint:decision` is emitted — the rating is recorded here for the plan reader. + + cargo test -p agent-memory-core --lib export:: && cargo test -p agent-memory-core --test export_import && cargo clippy --workspace --all-targets --locked -- -D warnings && T="$(mktemp -d)" && cargo run -q -p agent-memory -- export --db "$T/memory.db" 2>"$T/err" | head -1 | grep -c '^{"record":"header","format":"agent-memory-export","version":1,' && grep -c 'exported 0 memories and 0 links' "$T/err" + + + - `grep -c "pub enum Record" crates/agent-memory-core/src/export.rs` outputs 1; `grep -c 'tag = "record"' crates/agent-memory-core/src/export.rs` outputs 1; `grep -c "deny_unknown_fields" crates/agent-memory-core/src/export.rs` outputs at least 1; `grep -c "exported_rows" crates/agent-memory-core/src/export.rs` outputs at least 1. + - `grep -c "pub mod export;" crates/agent-memory-core/src/lib.rs` outputs 1. + - `grep -c "fn export_rows" crates/agent-memory-core/src/store/mod.rs` outputs 1 and the same grep on sqlite.rs outputs 1; `grep -c "FROM meta WHERE key" crates/agent-memory-core/src/store/sqlite.rs` outputs 2; `grep -c "ORDER BY from_id, to_id, kind" crates/agent-memory-core/src/store/sqlite.rs` outputs 1. + - `grep -c "pub async fn export" crates/agent-memory-core/src/service.rs` outputs 1 and `grep -c 'redact_storage_error("export"' crates/agent-memory-core/src/service.rs` outputs 1; `grep -c "pub struct ExportReport" crates/agent-memory-core/src/service.rs` outputs 1. + - `grep -c "fn run_export" crates/agent-memory/src/main.rs` outputs 1 and `grep -c 'env!("CARGO_PKG_VERSION")' crates/agent-memory/src/main.rs` outputs at least 1; `grep -c "eprintln!(\"exported" crates/agent-memory/src/main.rs` outputs 1. + - `cargo test -p agent-memory-core --lib export::` is green (golden order + the strictness probes); `cargo test -p agent-memory-core --test export_import` runs the five export tests green. + - The end-to-end command in `` prints `1` twice (one header line on stdout; the stderr report present). + - `grep -rn "memory_export\|/api/export" crates/agent-memory/src/mcp.rs crates/agent-memory/src/rest/` outputs nothing (D-01: no MCP tool, no REST route). + - `Cargo.lock` unchanged (`--locked` passes); fmt, clippy `-D warnings` and `cargo test --workspace --locked` green. + + A user can run `agent-memory export > memories.jsonl` and get a header line plus every memory and link in a frozen, versioned, vector-free format, streamed from one snapshot, verified by the real binary. + + + + Task 2: `export --output PATH` writes a private temp file in PATH's directory and renames it into place + crates/agent-memory/src/export_file.rs, crates/agent-memory/src/main.rs + + - crates/agent-memory/src/config.rs (whole file — the `cfg(unix)` / `cfg(not(unix))` split, the module-doc "ONE place that touches platform paths" sentence, and the `tempfile::tempdir()` unit-test style to mirror). + - crates/agent-memory-core/src/store/sqlite.rs lines 136-160 (`backup_temp_path`, `discard_backup_temp`) and 265-293 (verify-then-rename + remove-temp-on-failure discipline) — reuse the SHAPE, not the code (those helpers are private, SQLite-sidecar-specific). + - crates/agent-memory/src/main.rs: `Command::Export` and `run_export` from Task 1. + - .planning/phases/05-portable-export-import/05-RESEARCH.md: Pattern 7, Pitfall 7, Pitfall 12 (unit-test the helper in-process — spawned-binary tests contribute zero coverage). + + + Unit tests in `export_file.rs` (`#[cfg(test)]`, `tempfile::tempdir()`): + - `begin_creates_a_private_temp_beside_the_target`: `begin(&dir.join("out.jsonl"))` → the temp is `dir.join("out.jsonl.export-tmp-{pid}")`, it exists, the target does not exist yet; under `#[cfg(unix)]` its `metadata().permissions().mode() & 0o777 == 0o600`. + - `commit_replaces_an_existing_target_and_leaves_no_temp`: write "old" to the target first; `begin`, write "new\n" through `writer()`, `commit()` → target content is "new\n"; no directory entry contains `.export-tmp-`. + - `abort_removes_the_temp_and_keeps_the_target`: target holds "old"; `begin`, write, `abort()` → target still "old"; no `.export-tmp-` entry. + - `begin_refuses_to_clobber_an_existing_temp`: pre-create the exact temp path, then `begin` → `Err` (the `create_new` guard, kind `AlreadyExists`). + - `export_temp_path_stays_in_the_target_directory`: for `/a/b/out.jsonl` the parent of the temp path is `/a/b` and the file name starts with `out.jsonl.export-tmp-`. + End-to-end: `agent-memory export --db /memory.db --output /out.jsonl` on a fresh database exits 0, leaves exactly one line in `out.jsonl`, leaves no `*.export-tmp-*` entry, and `exported 0 memories and 0 links` is on stderr. + + + Implements the `--output` half of D-02 (atomic same-directory temp + rename, mode 0600, `cfg(unix)` split) per RESEARCH Pattern 7, resolving the temp-naming discretion as `{target}.export-tmp-{pid}`. + + 1. **export_file.rs (new).** Module doc modelled on config.rs: this is the ONE place that touches export-file permissions and atomic replacement; why 0600 (the corpus is T-01-03-sensitive — never looser than the DB directory's 0700); why the temp lives beside the target (`std::fs::rename` does not cross mount points, and a shared tmp dir would be a different trust domain); why `create_new` (a stale temp from a crashed run is an explicit error, never a silent truncate); Windows ACL inheritance is the Phase 6 seam (DIST-03). Implement `PendingExport`, `export_temp_path`, `begin`, `create_private` (both cfg variants), `writer`, `commit`, `abort` per "Interface contracts". `abort` and the `commit` failure path remove the temp best-effort and `tracing::warn!` only on a non-`NotFound` error, mirroring `discard_backup_temp`. + 2. **main.rs.** `mod export_file;`. Give `Command::Export` the field `#[arg(long)] output: Option` with the doc from "Interface contracts"; extend `run_export` to take `output`. Sink selection: `None` → the Task 1 stdout sink; `Some(path)` → `let pending = export_file::begin(&path).with_context(|| format!("creating export file beside {}", path.display()))?; let sink: Box = Box::new(BufWriter::new(pending.writer().context("duplicating the export file handle")?));`. Then `match service.export(sink, version).await { Ok(report) => { pending.commit().with_context(|| format!("moving the export into place at {}", path.display()))?; report } Err(e) => { pending.abort(); return Err(e).context("exporting"); } }`. The service already flushed the `BufWriter` inside the blocking task, so `commit`'s `sync_all` sees every byte. Keep the stderr report line. + 3. Add the five unit tests from `` to `export_file.rs`. + 4. `cargo fmt --all`; clippy; full tests (background + poll); commit with explicit paths: `feat(05-02): export --output writes a private temp file and renames it into place (D-02)`. + + Sink selection is a `match` in one CLI function over a self-contained helper module; swapping the atomic-write strategy later touches nothing in core or the file format. + + cargo test -p agent-memory --bin agent-memory export_file:: && cargo clippy -p agent-memory --all-targets --locked -- -D warnings && T="$(mktemp -d)" && cargo run -q -p agent-memory -- export --db "$T/memory.db" --output "$T/out.jsonl" 2>"$T/err" && test "$(wc -l < "$T/out.jsonl" | tr -d ' ')" = 1 && test "$(find "$T" -name '*.export-tmp-*' | wc -l | tr -d ' ')" = 0 && grep -c 'exported 0 memories and 0 links' "$T/err" + + + - `grep -c "pub struct PendingExport" crates/agent-memory/src/export_file.rs` outputs 1; `grep -c "create_new(true)" crates/agent-memory/src/export_file.rs` outputs at least 1; `grep -c "mode(0o600)" crates/agent-memory/src/export_file.rs` outputs 1; `grep -c "#\[cfg(unix)\]" crates/agent-memory/src/export_file.rs` outputs at least 2 (the `create_private` variant and the permission test); `grep -c "#\[cfg(not(unix))\]" crates/agent-memory/src/export_file.rs` outputs 1. + - `grep -c "mod export_file;" crates/agent-memory/src/main.rs` outputs 1; `grep -c "output: Option" crates/agent-memory/src/main.rs` outputs at least 1; `grep -c "pending.commit()" crates/agent-memory/src/main.rs` outputs 1 and `grep -c "pending.abort()" crates/agent-memory/src/main.rs` outputs 1. + - `cargo test -p agent-memory --bin agent-memory export_file::` runs 5 tests green, including the `#[cfg(unix)]` 0600 assertion. + - The end-to-end command in `` succeeds: exactly one line in `out.jsonl`, zero `*.export-tmp-*` entries, the stderr report present. + - `cargo run -q -p agent-memory -- export --help | grep -c -- '--output'` outputs at least 1. + - fmt, clippy `-D warnings`, `cargo test --workspace --locked` green; `Cargo.lock` unchanged. + + A user can write the export straight to a private file that is either complete or absent — never partial, never world-readable, never in a shared temp directory. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| database (0700 dir, owner-only) → export file on disk | the corpus leaves the DB's protection; the file must be no looser than the DB | +| store read pool ↔ concurrent writers (MCP server, REST daemon) | WAL readers see a snapshot per transaction; two separate statements may straddle a write | +| service → CLI stdout/stderr | stdout carries the data stream; stderr carries the report and any error text | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan | +|-----------|----------|-----------|----------|-------------|-----------------| +| T-05-02 | Information Disclosure | export file created world-readable (0644 via umask) or in a shared `/tmp` | high | mitigate | `OpenOptions::create_new(true).mode(0o600)` at creation — no 0644 window; the temp lives beside the target (the DB directory's trust domain), never in a shared tmp dir; `#[cfg(unix)]` unit test asserts `mode & 0o777 == 0o600` (Task 2) | +| T-05-03 | Denial of Service (data loss) | half-written target after a crash, a full disk or a failed rename | medium | mitigate | write to `PATH.export-tmp-`, `flush` inside the service, `sync_all` + `rename` in `commit`; temp removed on every failure path; `commit`/`abort` unit tests (Task 2) | +| T-05-04 | Tampering | embedding vectors travelling in the file and later poisoning a differently-pinned `vec_memories` | high | mitigate | no vector field exists in `Record`; `export_rows` never reads `vec_memories`; `memory_records_carry_exactly_the_documented_keys` asserts the 11-key set (Task 1) | +| T-05-11 | Elevation of Privilege / Information Disclosure | export as an MCP tool or REST route pushing the corpus through the daemon's JSON-RPC stdout or an unauthenticated HTTP response | high | mitigate (by design) | D-01: CLI subcommand only; acceptance grep for `memory_export` / `/api/export` in the transports is empty (Task 1) | +| T-05-12 | Information Disclosure | stdout export landing in terminal scrollback or a shell log | low | accept | user-directed; `--output` is the private alternative and the README (plan 04) says so | +| T-05-19 | Tampering (integrity) | memories and links read on two connections straddling a concurrent `link`, producing a link record whose key is absent from the file (which the strict importer then rejects) | medium | mitigate | one pooled connection, one read transaction (WAL snapshot) spanning the counts, the memories SELECT and the links SELECT; the FK guarantees every endpoint is in the same snapshot; documented in the `export_rows` trait doc and recorded as the backstop truth (Task 1) | +| T-05-20 | Information Disclosure | storage-engine text on an export fault | medium | mitigate | `redact_storage_error("export", _)` from plan 05-01; writer I/O failures are the crate-authored `export write failed: …` (Task 1) | +| T-05-SC | Tampering | cargo installs | low | accept | zero new dependencies (std `OpenOptionsExt`, `rename`, `BufWriter`; serde/serde_json already pinned); `--locked` | + +No `high` severity threat is left open (T-05-02, T-05-04, T-05-11 are mitigated). ASVS L1, block_on: high. + + + +- `cargo fmt --all --check`, `cargo clippy --workspace --all-targets --locked -- -D warnings`, `cargo test --workspace --locked` green after both commits. +- `cargo test -p agent-memory-core --lib export::` (golden order + strictness probes) and `cargo test -p agent-memory-core --test export_import` (five export tests) green; `cargo test -p agent-memory --bin agent-memory export_file::` (five helper tests) green. +- Real binary: `export` on a fresh DB prints one header line to stdout and `exported 0 memories and 0 links` to stderr; `export --output` leaves one line, no temp sibling. +- `grep -rn "memory_export\|/api/export" crates/agent-memory/src/mcp.rs crates/agent-memory/src/rest/` empty; `Cargo.lock` unchanged; `git diff --stat -- crates/agent-memory-core/sql/` empty. + + + +- `agent-memory export [--output PATH]` exists; format v1 is defined once in `export.rs` with golden tests; the header is read from `meta` + clock; every row and link streams from one snapshot; `--output` is private and atomic. +- Two commits on `main`, each with fmt + clippy + full tests green; nothing pushed. + + + +Create `.planning/phases/05-portable-export-import/05-02-SUMMARY.md` when done. Record: the `Record` field order as shipped, whether `unchecked_transaction` or `BEGIN`/`COMMIT` was used for the snapshot, the `StdoutLock`-is-not-`Send` correction, and the full-suite run count. + \ No newline at end of file diff --git a/.planning/phases/05-portable-export-import/05-02-SUMMARY.md b/.planning/phases/05-portable-export-import/05-02-SUMMARY.md new file mode 100644 index 0000000..0d0d14e --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-02-SUMMARY.md @@ -0,0 +1,262 @@ +--- +phase: 05-portable-export-import +plan: 02 +subsystem: core-service, store, cli +tags: [rust, serde, jsonl, sqlite, rusqlite, export, streaming, atomic-write, file-permissions] + +# Dependency graph +requires: + - phase: 05-portable-export-import + provides: "plan 05-01: redact_storage_error(op, e) + the *_unredacted inner-fn shape (D-14); LinkOutcome::Linked { created }; the one-INSERT store contract" + - phase: 04-memory-update-relations-hybrid-search + provides: "memory_links schema (0003) with relates_to stored canonically as (min, max); LinkKind wire strings; the backup temp-file + rename discipline this plan mirrors" + - phase: 02-semantic-search-import-release + provides: "meta table (0002) pinning embedding_model / embedding_dim — the header's source of truth (D-03)" +provides: + - "export.rs: format v1 defined ONCE — FORMAT_NAME, FORMAT_VERSION, `Record { Header, Memory, Link }` (tag = record, lowercase, deny_unknown_fields), ExportSnapshot / ExportEvent store→service stream protocol, module-doc spec, 9 golden + strictness tests" + - "Store::export_rows(visit): one pooled connection, one read transaction (unchecked_transaction) spanning the meta pin, both counts, memories ORDER BY id and links ORDER BY from_id, to_id, kind; rows visited lazily, never collected" + - "MemoryService::export(out: Box, agent_memory_version) -> ExportReport { memories, links }: one spawn_blocking hop, header built from the snapshot, one JSON line per record, flush inside the task, redact_storage_error(\"export\", _)" + - "CLI `agent-memory export [--output PATH]`: stdout BufWriter by default; --output goes through export_file::PendingExport (create_new + 0600 temp beside the target, sync_all + rename on Ok, best-effort remove on Err); report line on stderr" + - "tests/export_import.rs (export half): harness + export_to_string + lines helpers; 5 proofs (empty-DB header from meta/clock, ascending memories then links as stored, exactly 11 memory keys, expired rows exported, byte-identical re-export) — plan 03 appends the import tests" +affects: [05-03 import tracer (reads Record via import::jsonl, uses tests/export_import.rs harness), 05-04 README Export & import section, phase-6 release notes] + +# Actuals (#2632) — same estimateTokens scale as the plan's `estimate` (chars/4 over the realized diff) +actuals: + tokens: 13300 + tasks: 2 + commits: 2 + +# Tech tracking +tech-stack: + added: [] + patterns: + - "Wire contract as one serde enum: a single internally-tagged, deny_unknown_fields `Record` enum is the only definition of the file format; wire field order == declaration order with the tag first, pinned by exact-string golden tests" + - "Visitor-driven streaming read: `Store::export_rows(&mut dyn FnMut(ExportEvent) -> Result)` on ONE pooled connection + ONE read transaction (`conn.unchecked_transaction()`), stepping `query_map` lazily — the first read path in the repo that never builds a Vec" + - "Boxed sink through spawn_blocking: `Box` moved into the blocking task, written with `serde_json::to_writer` + `b\"\\n\"`, flushed INSIDE the task before Ok; stdout sink is `BufWriter::new(std::io::stdout())` because `StdoutLock` is !Send" + - "Private atomic file publish (binary crate): `{target}.export-tmp-{pid}` beside the target, `OpenOptions::create_new(true).mode(0o600)` under cfg(unix), `sync_all` then `rename`, temp removed best-effort on every failure path; one `async {}` block collects every fallible step between begin and commit so exactly one abort covers them" + +key-files: + created: + - crates/agent-memory-core/src/export.rs + - crates/agent-memory-core/tests/export_import.rs + - crates/agent-memory/src/export_file.rs + modified: + - crates/agent-memory-core/src/lib.rs + - crates/agent-memory-core/src/store/mod.rs + - crates/agent-memory-core/src/store/sqlite.rs + - crates/agent-memory-core/src/service.rs + - crates/agent-memory/src/main.rs + +key-decisions: + - "Format v1 frozen (D-03, rated one-way, pre-approved): wire order is exactly `record, format, version, exported_at, embedding_model, embedding_dim, schema_version, agent_memory_version, exported_rows, exported_links` for the header; `record, key, type, content, tags, source, scope, created_at, last_accessed, access_count, expires_at` for a memory; `record, from, to, kind, created_at` for a link. The header carries `exported_rows` / `exported_links` (orchestrator resolution 1, decided now so it is inside v1)" + - "The one-snapshot read uses `rusqlite::Connection::unchecked_transaction()` on the pooled `&Connection` (RESEARCH assumption A1 confirmed: present in rusqlite 0.39); the `BEGIN`/`COMMIT` execute_batch fallback was not needed" + - "Research correction applied: the stdout sink is `BufWriter::new(std::io::stdout())`, NOT `stdout().lock()` — `StdoutLock` is `!Send` and cannot live in the `Box` that crosses into the blocking task; `Stdout` is `Send + Sync` and `BufWriter` keeps per-call locking cheap" + - "Links are ordered by `(from_id, to_id, kind)` (D-05 fixes only 'after all memories'), making two exports of one database byte-identical and giving plan 03's round-trip diff a stable baseline" + - "`run_export` wraps `pending.writer()` and `service.export` in one `async {}` result so a `try_clone` failure aborts the temp exactly like an export failure does — one `pending.commit()`, one `pending.abort()`, no temp can outlive an error" + - "Tracer feedback gate resolved by re-running the fully automated on the committed tree (goldens, 5 export tests, workspace clippy, real-binary header + stderr report) rather than returning a mid-plan human-verify checkpoint: auto mode is off, but config sets human_verify_mode: end-of-phase and the tracer's verify has no human-judgment step (same resolution as 05-01)" + +patterns-established: + - "redact_storage_error op strings now number ten: store, list, search, import, forget, link, unlink, update, sweep, export" + - "Crate-authored writer failures: `MemoryError::Internal(\"export write failed: {e}\")` for serde_json / io errors from the sink — passes redaction unchanged, never names the storage engine" + - "export_file.rs is the ONE place that touches export-file permissions and atomic replacement (mirrors config.rs for the data dir); Windows ACL inheritance is the Phase 6 seam (DIST-03)" + +requirements-completed: [DIST-04] + +# Coverage metadata (#1602) +coverage: + - id: D1 + description: "`agent-memory export` writes line 1 as the format-v1 header: format/version constants, exported_at from the clock, embedding_model/embedding_dim from the source DB's meta table, schema_version 3, agent_memory_version as passed by the binary (CARGO_PKG_VERSION), exported_rows/exported_links from the snapshot" + requirement: DIST-04 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/export_import.rs#export_of_an_empty_database_is_exactly_one_header_line" + status: pass + - kind: unit + ref: "crates/agent-memory-core/src/export.rs#header_record_serializes_in_documented_order_with_record_first" + status: pass + - kind: e2e + ref: "cargo run -q -p agent-memory -- export --db /memory.db | head -1 | grep -c '^{\"record\":\"header\",\"format\":\"agent-memory-export\",\"version\":1,' -> 1; stderr has 'exported 0 memories and 0 links'" + status: pass + human_judgment: false + - id: D2 + description: "Every memory record carries exactly the 11 documented keys in declaration order (record, key, type, content, tags, source, scope, created_at, last_accessed, access_count, expires_at); decay_score, base_weight, embedding_status and vectors never travel; optional fields are written explicitly as null" + requirement: DIST-04 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/export_import.rs#memory_records_carry_exactly_the_documented_keys" + status: pass + - kind: unit + ref: "crates/agent-memory-core/src/export.rs#memory_record_serializes_in_documented_order_with_record_first" + status: pass + human_judgment: false + - id: D3 + description: "Memories stream in ascending id order with values equal to the stored rows (type, tags, source, scope, created_at, last_accessed, access_count, expires_at = created_at + ttl or null); links follow all memories ordered by (from, to, kind) with endpoints exactly as stored (relates_to as (min, max), supersedes as given) and their original created_at; ExportReport equals the header counts" + requirement: DIST-04 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/export_import.rs#export_writes_every_memory_ascending_then_every_link_as_stored" + status: pass + human_judgment: false + - id: D4 + description: "Export writes every row present — an expired row (ttl passed, no sweep) is exported with its expires_at (D-06) — and two exports of an unchanged database under the same clock are byte-identical" + requirement: DIST-04 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/export_import.rs#expired_rows_are_still_exported, #two_exports_of_an_unchanged_database_are_byte_identical" + status: pass + human_judgment: false + - id: D5 + description: "The reader side of the contract is strict: an unknown field inside a record, an unknown record tag, a string key and a fractional header version are all rejected; omitted optional fields read as None" + requirement: DIST-04 + verification: + - kind: unit + ref: "crates/agent-memory-core/src/export.rs#an_unknown_field_inside_a_memory_record_is_rejected, #an_unknown_record_tag_is_rejected, #a_string_key_is_rejected, #a_fractional_header_version_is_rejected, #missing_optional_fields_read_as_none, #every_record_round_trips_through_its_own_json" + status: pass + human_judgment: false + - id: D6 + description: "`export --output PATH` creates `PATH.export-tmp-` in PATH's directory with mode 0600 (create_new — a stale temp is an AlreadyExists error, never truncated), renames it over PATH after sync, replaces an existing PATH, removes the temp on abort, and the real binary leaves exactly one line in out.jsonl, no temp sibling, and the stderr report" + requirement: DIST-04 + verification: + - kind: unit + ref: "cargo test -p agent-memory --bin agent-memory export_file:: (5 tests: begin_creates_a_private_temp_beside_the_target incl. the cfg(unix) 0600 assertion, commit_replaces_an_existing_target_and_leaves_no_temp, abort_removes_the_temp_and_keeps_the_target, begin_refuses_to_clobber_an_existing_temp, export_temp_path_stays_in_the_target_directory)" + status: pass + - kind: e2e + ref: "cargo run -q -p agent-memory -- export --db /memory.db --output /out.jsonl -> exit 0, 1 line, 0 *.export-tmp-* entries, stat mode 600, stderr 'exported 0 memories and 0 links'; `export --help | grep -c -- --output` -> 1" + status: pass + human_judgment: false + - id: D7 + description: "Export is a CLI subcommand only — no MCP tool and no REST route (D-01); no vector table is read; a storage fault reaches the CLI only as the redacted `export failed due to a storage error (details are in the server log)`" + requirement: DIST-04 + verification: + - kind: other + ref: "grep -rn 'memory_export\\|/api/export' crates/agent-memory/src/mcp.rs crates/agent-memory/src/rest/ -> empty; grep -c 'redact_storage_error(\"export\"' service.rs -> 1; export_rows SELECTs name only meta, memories, memory_links" + status: pass + human_judgment: true + rationale: "The D-01 absence and the redaction wiring are grep-proven, but no fault-injection test exercises the export path end to end (plan 05-01's redaction.rs covers the nine prior ops; adding `export` there is a cheap follow-up for plan 03/04). A reviewer should confirm the op string wiring reads correctly." + +# Metrics +duration: 13min +completed: 2026-10-25 +status: complete +--- + +# Phase 5 Plan 02: `agent-memory export` — format v1 JSONL, streamed, atomic and private on disk Summary + +**`agent-memory export [--output PATH]` ships: format v1 is defined once in `export.rs` as a strict tagged `Record` enum with golden-tested wire order, every memory and link streams one line at a time from ONE SQLite read snapshot through a boxed writer, and `--output` publishes the file via a 0600 `create_new` temp beside the target that is synced and renamed into place only after the stream has flushed.** + +## Performance + +- **Duration:** 13 min +- **Started:** 2026-10-25T11:44:48Z +- **Completed:** 2026-10-25T11:57:52Z +- **Tasks:** 2 +- **Files modified:** 8 (3 created, 5 modified; +1306 / −0 lines) + +## Accomplishments + +- **Format v1 frozen in one Rust type (D-03/D-04/D-05).** `crates/agent-memory-core/src/export.rs` holds `FORMAT_NAME = "agent-memory-export"`, `FORMAT_VERSION = 1`, and `enum Record { Header, Memory, Link }` with `#[serde(tag = "record", rename_all = "lowercase", deny_unknown_fields)]`. The module doc is the format specification (a table per record type, the derived-data exclusion and why, the `key`-is-export-local rule, link placement and canonical endpoints, the canonical-output rule, the strict reader rules). Nine in-module tests pin the exact serialized strings for all three records (tag first, declaration order), round-trip each, and probe strictness: unknown field → `unknown field`, unknown tag → `unknown variant`, `"key":"1"` and `"version":1.5` rejected, omitted optionals → `None`. `type`/`kind` stay `String` so the importer converts through `MemoryType::try_from` / `LinkKind::try_from` (D-09). The header carries `exported_rows`/`exported_links` (orchestrator resolution 1), decided now so it lives inside v1. +- **One snapshot, zero buffering (D-02/D-06, T-05-19).** `Store::export_rows(visit)` takes one pooled read connection and opens one read transaction with `conn.unchecked_transaction()` (RESEARCH assumption A1 confirmed against rusqlite 0.39 — the `BEGIN`/`COMMIT` fallback was not needed). Inside it, in order: the `meta` pin (`embedding_model`, `embedding_dim` parsed from TEXT to `u32`, a non-integer being `Internal("meta.embedding_dim is not an integer (…)")`), `count(*)` of both tables → one `ExportEvent::Snapshot`; then `SELECT id, mem_type, content, tags, source, scope, created_at, last_accessed, access_count, expires_at FROM memories ORDER BY id` and `SELECT from_id, to_id, kind, created_at FROM memory_links ORDER BY from_id, to_id, kind`, each stepped lazily through `query_map` and handed to the visitor per row. No `base_weight`, `decay_score`, `embedding_status`, no `vec_memories` read. The FK guarantees every link endpoint is a memory in the same snapshot, so a concurrent `link` can never produce a dangling key. +- **`MemoryService::export(out, agent_memory_version) -> ExportReport` (D-14).** One `spawn_blocking` hop moves the `Box`, the store and the version; the `Snapshot` event builds `Record::Header` (clock → `exported_at`, `LATEST_SCHEMA_VERSION` → `schema_version`), every `Record` event is one `serde_json::to_writer` + `\n`, and `out.flush()` runs inside the task before `Ok(report)` so a `BufWriter` never loses its tail. Sink failures are the crate-authored `Internal("export write failed: …")`; storage faults go through `redact_storage_error("export", e)` — the tenth op string. +- **CLI `agent-memory export`.** `Command::Export { --output }` and `run_export` wire the store/service exactly like `run_import` (the embedder is constructed, never called). Default sink: `Box::new(BufWriter::new(std::io::stdout()))` — the research's `stdout().lock()` was corrected because `StdoutLock` is `!Send`. The version is the binary's `env!("CARGO_PKG_VERSION")` (Pitfall 9). The report `exported N memories and M links` is ALWAYS on stderr, since stdout may be the file. Real-binary check on a fresh DB: exactly one stdout line starting `{"record":"header","format":"agent-memory-export","version":1,` and the stderr report. +- **`--output` is private and atomic (D-02, T-05-02/T-05-03).** `crates/agent-memory/src/export_file.rs` is the one place touching export-file permissions: `export_temp_path` → `{target}.export-tmp-{pid}` in the target's own directory (rename never crosses a mount point; a shared tmp dir is a different trust domain); `begin` → `OpenOptions::write(true).create_new(true).mode(0o600)` under `cfg(unix)` (umask can only remove bits; a stale temp is an explicit `AlreadyExists`, never a truncate), the `cfg(not(unix))` arm being the Phase 6 Windows seam; `PendingExport::writer` (`try_clone`), `commit` (`sync_all` then `rename`, temp removed on either failure), `abort` (best-effort remove, `tracing::warn!` only on non-`NotFound`). Five unit tests cover temp placement + 0600 mode, replace-and-no-temp, abort-keeps-target, create_new refusal, and path derivation. Real binary: `--output` exits 0, leaves exactly one line, zero `*.export-tmp-*` entries, file mode 600. +- **Proof file for the phase.** `tests/export_import.rs` carries the relations.rs-style harness plus `export_to_string` (real file sink under the temp dir) and `lines` helpers, and the five export proofs listed in the coverage block. Plan 03 appends the import half. +- **Nothing else moved:** `sql/` diff empty, `LATEST_SCHEMA_VERSION = 3`, `Cargo.lock` byte-identical under `--locked`, no workflow touched, no new dependency; `grep -rn "memory_export\|/api/export"` over `mcp.rs` and `rest/` is empty (D-01). + +## Task Commits + +Each task was committed atomically (fmt + clippy `-D warnings` + full workspace suite green before each): + +1. **Task 1 (tracer): `agent-memory export` streams a valid format-v1 JSONL document to stdout — one path, every layer** — `6524cf1` (feat) +2. **Task 2: `export --output PATH` writes a private temp file in PATH's directory and renames it into place** — `cffe7a2` (feat) + +**Plan metadata:** see the `docs(05-02)` commit that follows this summary. + +## Files Created/Modified + +- `crates/agent-memory-core/src/export.rs` (new) — format v1 module-doc spec; `FORMAT_NAME`, `FORMAT_VERSION`; `Record { Header, Memory, Link }`; `ExportSnapshot`, `ExportEvent`; 9 golden/strictness tests +- `crates/agent-memory-core/src/lib.rs` — `pub mod export;` between `embed` and `import` +- `crates/agent-memory-core/src/store/mod.rs` — `fn export_rows(&self, visit: &mut dyn FnMut(ExportEvent) -> Result<(), MemoryError>)` with the one-snapshot contract, column list, ordering and the `embedding_dim` parse rule in its doc +- `crates/agent-memory-core/src/store/sqlite.rs` — `export_rows` impl: `unchecked_transaction`, two `FROM meta WHERE key` reads, two counts, memories scan `ORDER BY id`, links scan `ORDER BY from_id, to_id, kind`, lazy visitor calls, `tx.commit()` +- `crates/agent-memory-core/src/service.rs` — `pub struct ExportReport { memories, links }`, private `write_record_line`, `pub async fn export` + `async fn export_unredacted`; new imports (`std::io::Write`, `crate::export::*`, `LATEST_SCHEMA_VERSION`) +- `crates/agent-memory-core/tests/export_import.rs` (new) — harness, `export_to_string`, `lines`, five `#[tokio::test]` export proofs +- `crates/agent-memory/src/export_file.rs` (new) — `PendingExport`, `export_temp_path`, `begin`, `writer`, `commit`, `abort`, `discard_temp`, `create_private` (both cfg arms), 5 unit tests +- `crates/agent-memory/src/main.rs` — `mod export_file;`, `Command::Export { output: Option }`, dispatch arm, `run_export(db, ollama_url, output)` with the stdout / pending-file sink match and the stderr report + +## Decisions Made + +- **Record field order as shipped** (tag first, then declaration order — serde-verified by the goldens): header `record, format, version, exported_at, embedding_model, embedding_dim, schema_version, agent_memory_version, exported_rows, exported_links`; memory `record, key, type, content, tags, source, scope, created_at, last_accessed, access_count, expires_at`; link `record, from, to, kind, created_at`. The D-03 one-way rating is recorded here as pre-approved (05-CONTEXT.md + the orchestrator's unattended-run instruction); no `checkpoint:decision` was emitted. +- **`unchecked_transaction`, not `BEGIN`/`COMMIT`.** `rusqlite::Connection::unchecked_transaction(&self)` exists in the pinned 0.39 and works on the pooled `&Connection`; the read-only `tx.commit()` simply ends the snapshot. +- **`StdoutLock` is `!Send`** — the research's `BufWriter::new(stdout().lock())` cannot be boxed as `dyn Write + Send`; shipped `BufWriter::new(std::io::stdout())` (the `Stdout` handle is `Send + Sync`). +- **Deterministic link order `(from_id, to_id, kind)`** beyond D-05's "after all memories", so re-exports are byte-identical and the plan 03 round-trip diff has a stable baseline. +- **One abort covers every failure between `begin` and `commit`.** The plan wrote `pending.writer().context(..)?` directly, which would leak a 0-byte temp if `try_clone` failed; `run_export` instead runs `writer()` + `service.export` inside one `async {}` result and matches on it, keeping exactly one `pending.commit()` and one `pending.abort()` (the acceptance greps) while making a leaked temp impossible. +- **Tracer feedback gate:** auto mode off, `human_verify_mode: end-of-phase`, verify fully automated → re-ran it on the committed tree (lib `export::` 9/9, `--test export_import` 5/5, workspace clippy clean, real binary header line + stderr report) and continued to Task 2 rather than returning a mid-plan `checkpoint:human-verify`. Recorded so the orchestrator can route end-of-phase UAT. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 3 - Blocking] Destructured the report counts so the stderr `eprintln!` stays on one line** +- **Found during:** Task 1 (acceptance-criteria gate) +- **Issue:** The criterion `grep -c 'eprintln!("exported' main.rs` → `1` requires the string literal on the macro's line, but rustfmt's `fn_call_width` (60) wraps `eprintln!("exported {} memories and {} links", report.memories, report.links)` onto four lines (the args exceed 60 columns), and the hook re-applies rustfmt on every write. +- **Fix:** `let (memories, links) = (report.memories, report.links); eprintln!("exported {memories} memories and {links} links");` — same output, fits rustfmt's width. +- **Files modified:** `crates/agent-memory/src/main.rs` +- **Verification:** grep → 1; `cargo fmt --all -- --check` clean; e2e stderr grep → 1 +- **Committed in:** `6524cf1` + +**2. [Rule 1 - Bug] `PendingExport::temp_path` was dead code in the non-test build** +- **Found during:** Task 2 (clippy gate, `-D warnings` → `-D dead-code`) +- **Issue:** I added a `temp_path()` accessor (not in the plan's contract) so the `begin_creates_a_private_temp_beside_the_target` test can assert the exact temp path; production code never calls it, so clippy failed the bin with `method temp_path is never used`. +- **Fix:** Gated the accessor with `#[cfg(test)]` and documented it as test-only. +- **Files modified:** `crates/agent-memory/src/export_file.rs` +- **Verification:** `cargo clippy --workspace --all-targets --locked -- -D warnings` exit 0; `export_file::` 5/5 +- **Committed in:** `cffe7a2` + +### Process notes (not code deviations) + +- **TDD RED/GREEN without a separate RED commit.** Both tasks are `tdd="true"`. RED was executed and observed: Task 1's `tests/export_import.rs` failed to compile against the pre-implementation tree (`unresolved import ExportReport`, `no method named export`), and Task 2's `export_file` test module was written first and failed with `cannot find function begin / export_temp_path`; each task's `feat(05-02)` commit then carries tests + implementation together, because the project operating rules forbid committing failing tests and the plan calls for one commit per task. The `export.rs` goldens were authored alongside the type they pin (a type and its serialization goldens are one unit) and passed on first run. +- **Pre-bash safety hook.** The repo's `.claude/hooks/pre-bash-rust.sh` blocked two gate invocations because `rm -f ` and a `memory.db` path appeared in the same command (its regex is `rm -[rfRF]+ .*\.db`); re-issued with a `: > log` truncation and no temp-dir removal. No code impact. + +--- + +**Total deviations:** 2 auto-fixed (1 Rule 3 plan-internal criterion conflict, 1 Rule 1 clippy dead-code on a test-only accessor); no scope or behaviour change. +**Impact on plan:** None on scope. Both fixes exist only to make the plan's own gates (acceptance grep, `-D warnings`) pass as written. + +## Issues Encountered + +None beyond the two auto-fixes. Every targeted `` passed on the first post-GREEN run; the full suite passed on every run (246 after Task 1, 251 after Task 2); no fix-attempt budget was consumed on behaviour. + +## Verification + +Plan-level `` re-run after the second commit: + +- `cargo fmt --all -- --check` — clean; `cargo clippy --workspace --all-targets --locked -- -D warnings` — exit 0; `cargo test --workspace --locked` — **20 suites, 251 passed, 0 failed, 2 ignored** (232 before this plan → 246 after Task 1: +9 `export::` goldens, +5 `export_import` → 251 after Task 2: +5 `export_file::`). +- `cargo test -p agent-memory-core --lib export::` — 9 passed; `cargo test -p agent-memory-core --test export_import` — 5 passed; `cargo test -p agent-memory --bin agent-memory export_file::` — 5 passed (incl. the `cfg(unix)` `mode & 0o777 == 0o600` assertion). +- Real binary, fresh DB: `export` → 1 stdout line matching `^{"record":"header","format":"agent-memory-export","version":1,`, stderr `exported 0 memories and 0 links`; `export --output out.jsonl` → exit 0, 1 line, 0 `*.export-tmp-*` entries, `stat` mode `600`, same stderr line; `export --help | grep -c -- --output` → 1. +- `grep -rn "memory_export\|/api/export" crates/agent-memory/src/mcp.rs crates/agent-memory/src/rest/` — empty (D-01); `git diff --stat -- crates/agent-memory-core/sql/ Cargo.lock .github/` — empty; `LATEST_SCHEMA_VERSION: i64 = 3`. +- No `unwrap()`/`expect()` outside `#[cfg(test)]` in any touched production file. + +## Known Stubs + +None — no placeholder values, TODO/FIXME markers, skipped tests, or unwired data paths were introduced. + +## Threat Flags + +None new. The plan's register is implemented as mitigated: T-05-02 (0600 at create, temp beside the target, unit-tested), T-05-03 (flush → sync_all → rename, temp removed on every failure, unit-tested), T-05-04 (no vector field exists in `Record`; `export_rows` never reads `vec_memories`; 11-key test), T-05-11 (CLI only; transport grep empty), T-05-19 (one read transaction spans counts and both scans), T-05-20 (`redact_storage_error("export", _)`; writer faults are crate-authored text), T-05-SC (zero new deps, `--locked`). T-05-12 (stdout in scrollback) stays accepted; the README sentence pointing at `--output` is plan 04's. + +## User Setup Required + +None - no external service configuration required. + +## Next Phase Readiness + +- **Plan 05-03 (import tracer)** compiles against a frozen `Record` enum with `deny_unknown_fields` and pinned goldens; `import::jsonl` can `serde_json::from_str::` each line, convert `type`/`kind` through the Phase 3 seam, and map `key → id` via `find_by_key` / `insert_imported` (plan 01). `tests/export_import.rs` already has the harness, `export_to_string` and `lines` helpers for the round-trip kill-test; the header's `exported_rows`/`exported_links` are available as the truncation check. +- **Plan 05-04 (docs + CLI realism)** needs the README "Export & import" section (point it at the `export.rs` module doc, which is the spec), the T-05-12 `--output` recommendation, and may add `export` to `tests/redaction.rs` for an end-to-end fault-injection proof of the tenth op string. +- No blockers. `.planning/config.json` and the untracked harness files were left unstaged as instructed. + +--- +*Phase: 05-portable-export-import* +*Completed: 2026-10-25* + +## Self-Check: PASSED + +All 8 key files exist on disk; task commits 6524cf1 and cffe7a2 are present in git history. diff --git a/.planning/phases/05-portable-export-import/05-03-PLAN.md b/.planning/phases/05-portable-export-import/05-03-PLAN.md new file mode 100644 index 0000000..93a70ba --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-03-PLAN.md @@ -0,0 +1,326 @@ +--- +phase: 05-portable-export-import +plan: 03 +type: execute +wave: 3 +depends_on: [05-01, 05-02] +files_modified: + - crates/agent-memory-core/src/import/jsonl.rs + - crates/agent-memory-core/src/import/mod.rs + - crates/agent-memory-core/src/service.rs + - crates/agent-memory-core/tests/export_import.rs + - crates/agent-memory-core/tests/import.rs + - crates/agent-memory/src/main.rs +autonomous: true +requirements: [INTEROP-02, DIST-04] + +estimate: + tokens: 65000 + raw_tokens: 65000 + tasks: 2 + confidence: low + +must_haves: + truths: + - "`agent-memory import --from jsonl PATH` recreates every memory and every link from a format-v1 file into an empty OR a non-empty database, mapping export-local keys to the target's own ids in a two-pass validate-before-write import; no pre-existing row ever gains a phantom edge (D-08, PITFALLS 18)" + - "Re-running the same import reports imported 0, skipped_duplicates N, links_imported 0, links_skipped_duplicates M and leaves row and edge counts unchanged — dedup runs on the unchanged (source, mem_type, content) key through Store::find_by_key, source is preserved verbatim, and two source rows identical on that key collapse to one (D-07)" + - "Imported rows land embedding_status = 0 with no vector row; the JSONL importer never calls the embedder; after one sweep they are hybrid-searchable — exactly one re-embed mechanism (D-13)" + - "created_at, access_count and expires_at are preserved; last_accessed is refreshed to import-time now; a row with expires_at <= now is skipped, counted as skipped_expired and never inserted; a link whose endpoint was skipped is counted as links_dropped; link created_at is preserved; a header exported_at more than 3600 s ahead of the local clock logs a tracing warning and the import continues (D-10)" + - "Every wire type and kind string parses through MemoryType::try_from / LinkKind::try_from, empty or whitespace content and self-links are InvalidArgument, tags must be a JSON array of strings, expires_at must be an integer or null — the shared Phase 3 seam, no bypass (D-09)" + - "Every malformed file is rejected as MemoryError::InvalidArgument naming the 1-based line — missing/invalid header, format mismatch, version < 1 or non-integer, version > 1 (the 'upgrade agent-memory' text), non-object line, unknown record, unknown field, wrong field type, invalid type/kind, empty content, self-link, link key absent from the file, duplicate key, declared-count mismatch — and nothing is written: parsing completes before the database is even opened; the gsd-state importer's tolerance rules are unchanged (D-11)" + - "ImportReport carries imported, skipped_duplicates, skipped_expired, links_imported, links_skipped_duplicates, links_dropped; the gsd-state path reports zeros for the four new fields; the CLI prints all six counts on one line plus a line stating how many rows are pending embedding (D-12)" + - "Import is a CLI subcommand only — the file read happens in main.rs; core exposes parse_jsonl + import_jsonl with the InvalidArgument contract so any future transport inherits 400 / invalid_params structurally (D-01)" + - "Two source rows identical on (source, type, content) are both written by export and collapse to one row on import, counted as one skipped_duplicate" + - "Importing the header-only file exported from an empty database is a no-op that reports all-zero counts" + - "A link whose endpoints both dedup to existing target rows attaches to those existing rows — never to an unrelated pre-existing id" + - "A file with only a valid header imports nothing and reports all-zero counts; a zero-byte file is rejected as InvalidArgument at line 1" + - "Links are applied after all memories regardless of their position in the file, and importing the same file always yields the same counts" + artifacts: + - path: "crates/agent-memory-core/src/import/jsonl.rs" + provides: "strict pure parser: parse_jsonl(text) -> Result with the two-step header probe and the importer's own 1-based line counter" + contains: "pub fn parse_jsonl" + - path: "crates/agent-memory-core/src/import/mod.rs" + provides: "pub mod jsonl; invariants reworded — tolerance is per importer (gsd-state tolerant, jsonl strict)" + contains: "pub mod jsonl;" + - path: "crates/agent-memory-core/src/service.rs" + provides: "ImportReport +4 fields; import_jsonl(file) -> ImportReport (passes 2 and 3, no embedder); clock_skew_message" + contains: "pub async fn import_jsonl" + - path: "crates/agent-memory/src/main.rs" + provides: "--from jsonl arm: parse before open, six-count line, pending-embedding line, --scope rejected for jsonl" + contains: "\"jsonl\" =>" + - path: "crates/agent-memory-core/tests/export_import.rs" + provides: "round trip, idempotency, overlapping-rowid kill-test, dedup-to-existing adjacency, in-file duplicates, expiry fixture, pending→sweep→hybrid, header-only/zero-byte, file-order independence" + contains: "phantom" + key_links: + - from: "crates/agent-memory/src/main.rs run_import (jsonl arm)" + to: "import::jsonl::parse_jsonl then MemoryService::import_jsonl" + via: "parse_jsonl(&text) runs BEFORE SqliteStore::open so a malformed file never touches the database" + pattern: "parse_jsonl\\(&text\\)" + - from: "crates/agent-memory-core/src/service.rs import_jsonl_unredacted" + to: "Store::find_by_key / Store::insert_imported / Store::link" + via: "pass 2 builds HashMap; pass 3 links through the map with the record's created_at as now" + pattern: "insert_imported\\(" + - from: "crates/agent-memory-core/src/import/jsonl.rs" + to: "crate::export::Record + MemoryType::try_from + LinkKind::try_from + validate_link_endpoints" + via: "wire strings converted at the seam; every rejection wrapped as InvalidArgument(\"line N: …\")" + pattern: "MemoryType::try_from" + prohibitions: + - statement: "Import must never write anything when validation fails — pass 1 completes over the whole file before the database is opened or any store call runs" + status: resolved + verification: test + - statement: "The JSONL importer must never call the embedder or write a vector — rows land embedding_status = 0 and only the sweep backfill re-embeds them" + status: resolved + verification: test + - statement: "The existing gsd-state importer's tolerance rules must not change — gsd_state.rs is byte-identical and its tests are unchanged" + status: resolved + verification: test + - statement: "Import must never rewrite provenance — source and scope are stored exactly as the file carries them (no jsonl stamp), so a gsd-state row carried in a JSONL file dedups against an earlier direct GSD import" + status: resolved + verification: test + - statement: "Import must never be exposed over MCP or REST in this phase — a file-path read on an unauthenticated daemon is an arbitrary-file-read primitive" + status: resolved + verification: test +--- + +# Plan 05-03: `agent-memory import --from jsonl` — strict two-pass restore with key→id remapping + +## Phase Goal + +**As a** user, **I want to** export all my memories to a portable file and import them on another machine, **so that** my agent's memory moves with me — across machines, with no cloud sync. + +This plan is the second half: the file written by plan 02 is read back on another machine — into an empty or an already-populated database — and every memory and link comes back with the right identity, idempotently, with expired rows skipped and counted, re-embedded locally by the sweep, and with every defect in the file refused as a client error before anything is written. + + +1. **Tracer:** a pure, strict parser (`import/jsonl.rs::parse_jsonl`) that validates the whole file into typed records with 1-based line numbers; `MemoryService::import_jsonl` running the memories pass (expiry skip → `find_by_key` dedup → `insert_imported`, building `key → id`) and the links pass (`Store::link` through the map, counting created/duplicate/dropped); the `--from jsonl` CLI arm that parses before it opens the database and prints the six-count report plus the pending-embedding line. End-to-end check: the real binary exports an empty database and imports the file into a second database with an all-zero report. +2. **Expansion:** the kill-tests and the malformed-file table — overlapping rowids never attach phantom edges (PITFALLS 18), dedup-to-existing adjacency, in-file duplicates collapse, the expiry fixture with `links_dropped` (PITFALLS 19), pending → sweep → hybrid-searchable (criterion 3), header-only/zero-byte/file-order edges, hostile-input non-panic probes, and every D-11 rejection class naming its line. + +Purpose: INTEROP-02 holds literally — idempotent, `embedding_status = 0`, expired skipped and counted, malformed rejected as client errors through the Phase 3 seam — and DIST-04's "export-local key remapping" is proven against a target whose rowids collide with the file's keys. +Output: `import/jsonl.rs`, `import/mod.rs` reword, `ImportReport` +4, `import_jsonl`, `clock_skew_message`, the `jsonl` CLI arm, the import half of `tests/export_import.rs`, one assertion in `tests/import.rs`. + + + +@$HOME/.claude/gsd-core/workflows/execute-plan.md +@$HOME/.claude/gsd-core/templates/summary.md + + + +@.planning/STATE.md +@.planning/phases/05-portable-export-import/05-CONTEXT.md +@.planning/phases/05-portable-export-import/05-RESEARCH.md +@.planning/phases/05-portable-export-import/05-PATTERNS.md +@.planning/phases/05-portable-export-import/05-01-SUMMARY.md +@.planning/phases/05-portable-export-import/05-02-SUMMARY.md +@crates/agent-memory-core/src/export.rs +@crates/agent-memory-core/src/import/gsd_state.rs +@crates/agent-memory-core/src/service.rs +@crates/agent-memory/src/main.rs + + +## Execution notes for every task + +- Gate before EVERY commit: `cargo fmt --all`, `cargo clippy --workspace --all-targets --locked -- -D warnings`, `cargo test --workspace --locked` — clippy and the full suite in the background with a log file, then poll (watchdog ~600 s). +- One commit per task; stage explicit paths only. NEVER stage `CLAUDE.md`, `.claude/`, `.mcp.json`, `.gitignore`, `.gsd/`, `docs/`, `10-agent-memory.md`, `claude-code-harness-engineering-guide-v2.md`, `.planning/research/.cache/`, `.planning/config.json`, `.planning/milestone.lock`. +- No `unwrap()`/`expect()` in production code. No new crate; `Cargo.lock` unchanged. +- `gsd_state.rs` is NOT touched by this plan — `git diff --stat -- crates/agent-memory-core/src/import/gsd_state.rs` must be empty at every commit. +- Never surface `serde_json::Error::line()` in a message: it is `1` for every per-line parse and `0` for in-variant data errors (RESEARCH Pattern 3). The importer's own counter is the only line number. +- Never change the CI runner or add a workflow. Never push, tag or release. + + +- **Noun:** import source format (the phase introduces a SECOND importer, `jsonl`, beside `gsd-state`, and a second insert shape, `ImportedMemory`, beside `NewMemory`). +- **Decision:** `add-alongside`. +- **Rationale:** `gsd-state` is a lossy scrape of a human-edited text file with tolerant skip-count semantics and relative TTLs; `jsonl` is agent-memory's own strict contract with absolute timestamps. A shared `Importer` trait would force one tolerance policy and one draft shape onto both, which is exactly what D-11 forbids. The two importers share the seam (`MemoryService`), the dedup key (`find_by_key`) and the writer lane, not a type. +- **Accepted debt:** two `--from` arms in `main.rs`, two parser modules, two insert shapes. **What would force a promote:** a third import format — at that point introduce an `ImportSource` enum / trait over the parse step and fold both dispatch arms into it. + + +## Interface contracts (established by this plan) + +**`crates/agent-memory-core/src/import/jsonl.rs`** (new; pure, no I/O) + +```text +pub struct JsonlHeader { pub exported_at: i64, pub embedding_model: String, pub embedding_dim: u32, + pub schema_version: i64, pub agent_memory_version: String, + pub exported_rows: u64, pub exported_links: u64 } +pub struct JsonlMemory { pub key: i64, pub memory: ImportedMemory } // last_accessed is validated, then dropped (D-10 refresh) +pub struct JsonlLink { pub from: i64, pub to: i64, pub kind: LinkKind, pub created_at: i64 } +pub struct JsonlFile { pub header: JsonlHeader, pub memories: Vec, pub links: Vec } + +pub fn parse_jsonl(text: &str) -> Result +``` +Every rejection is `MemoryError::InvalidArgument(format!("line {n}: {msg}"))` (Display → `invalid argument: line 42: …`, the D-11 shape), built by a private `fn invalid_line(line_no: usize, msg: impl Display) -> MemoryError`. + +Line 1 — two-step header (RESEARCH Pattern 2): a private lenient `#[derive(Deserialize)] struct HeaderProbe { record: String, format: String, version: serde_json::Value }` WITHOUT `deny_unknown_fields`, checked in this order with these messages — parse failure → `missing or invalid header ({serde message})`; `record != "header"` → `first record must be the header`; `format != FORMAT_NAME` → `format must be 'agent-memory-export' (got '{format}')`; `version` not an unsigned integer → `version must be a positive integer (got {value})`; `version == 0` → `version must be >= 1 (got 0)`; `version > FORMAT_VERSION` → `file is format version {v} but this binary reads up to version {FORMAT_VERSION} — upgrade agent-memory`; only then the strict `serde_json::from_str::(line)` which must be `Record::Header { .. }` → `JsonlHeader`. + +Lines 2.. — `text.lines().enumerate()`, `line_no = idx + 1`: empty/whitespace-only or first non-whitespace byte not `{` → `expected a JSON object`; `from_str::` error → its message verbatim under the line prefix; `Record::Header` → `duplicate header`; `Record::Memory` → `MemoryType::try_from(mem_type.as_str())` (its message re-wrapped under the line prefix), `content.trim().is_empty()` → `content must be a non-empty string`, duplicate `key` → `duplicate memory key {key}`, else push `JsonlMemory { key, memory: ImportedMemory { content, mem_type, tags, source, scope, created_at, access_count, expires_at } }`; `Record::Link` → `LinkKind::try_from(kind.as_str())` (already `InvalidArgument` naming the allowed set — re-wrap under the line prefix), `validate_link_endpoints(from, to)` (self-link → `from_id and to_id must differ`), push `JsonlLink`. After the loop: each link's `from`/`to` must be a memory key seen anywhere in the file (order-independent) else `link key {k} does not appear in the file` at that link's line; `memories.len() != exported_rows` → `line {last}: file declares {exported_rows} memory records but contains {n}`; likewise for links. A zero-byte or header-less file fails at line 1. + +**`MemoryService`** (`service.rs`) + +```text +pub struct ImportReport { imported, skipped_duplicates, // existing + skipped_expired, links_imported, links_skipped_duplicates, links_dropped } // new, all usize +pub const CLOCK_SKEW_TOLERANCE_SECS: i64 = 3600; +pub(crate) fn clock_skew_message(exported_at: i64, now: i64) -> Option // Some(..) iff exported_at > now + CLOCK_SKEW_TOLERANCE_SECS +pub async fn import_jsonl(&self, file: JsonlFile) -> Result // = import_jsonl_unredacted(..).map_err(|e| redact_storage_error("import", e)) +``` +`import_jsonl_unredacted`: `let now = self.clock.now();` `if let Some(msg) = clock_skew_message(file.header.exported_at, now) { tracing::warn!("{msg}"); }` — NO embedder call anywhere. ONE `spawn_blocking` hop over `store`: `let mut map: HashMap`; **pass 2** for each `JsonlMemory { key, memory }`: `if memory.expires_at.is_some_and(|t| t <= now) { skipped_expired += 1; continue; }` (no map entry — the D-10 import rule, `<=`); `match store.find_by_key(memory.source.as_deref(), memory.mem_type, &memory.content)? { Some(id) => { skipped_duplicates += 1; map.insert(key, id); } None => { let id = store.insert_imported(memory, now)?; imported += 1; map.insert(key, id); } }`; **pass 3** for each `JsonlLink`: `match (map.get(&from), map.get(&to)) { (Some(a), Some(b)) => match store.link(*a, *b, kind, created_at)? { LinkOutcome::Linked { created: true } => links_imported += 1, LinkOutcome::Linked { created: false } => links_skipped_duplicates += 1, LinkOutcome::NotFound { .. } => links_dropped += 1 }, _ => links_dropped += 1 }`. Per-record transactions (the `insert`/`link` shape): a storage fault mid-pass leaves a partial import that a re-run completes idempotently — validation (pass 1, in `parse_jsonl`) is the all-or-nothing gate. Skew message text: `export header exported_at {exported_at} is {delta}s ahead of this machine's clock ({now}); check both clocks — expiry and freshness of imported rows are computed against this machine`. + +**CLI** (`main.rs`, `run_import`) + +`match from.as_str()`: `"gsd-state"` → the existing body unchanged (incl. `default_scope_for`); `"jsonl"` → if `scope.is_some()` bail `--scope applies only to --from gsd-state (a jsonl file carries each memory's own scope)`; `let file = agent_memory_core::import::jsonl::parse_jsonl(&text).with_context(|| format!("validating {}", path.display()))?;` BEFORE `resolve_db_path`/`SqliteStore::open`; then open store + service like today; `let report = service.import_jsonl(file).await.context("importing")?;` `println!("imported: {}, skipped duplicates: {}, skipped expired: {}, links imported: {}, links skipped duplicates: {}, links dropped: {}", …)` and `println!("{} rows pending embedding — the sweep backfills them on the next serve/serve-rest start (128 per tick)", report.imported)`; any other value → `unsupported --from value {from:?} (supported formats: gsd-state, jsonl)`. The `from` flag doc becomes "Source format. Supported: gsd-state (a GSD .planning/STATE.md file), jsonl (an agent-memory export)."; the `Import` variant doc mentions both and the dedup key. + +## Artifacts this phase produces (this plan's share) + +- `crates/agent-memory-core/src/import/jsonl.rs` (new): `JsonlHeader`, `JsonlMemory`, `JsonlLink`, `JsonlFile`, `pub fn parse_jsonl`, private `invalid_line`, `HeaderProbe`, `check_header`; unit tests incl. the malformed table. +- `crates/agent-memory-core/src/import/mod.rs`: `pub mod jsonl;`; invariants reworded. +- `service.rs`: `ImportReport::{skipped_expired, links_imported, links_skipped_duplicates, links_dropped}`, `pub const CLOCK_SKEW_TOLERANCE_SECS`, `pub(crate) fn clock_skew_message`, `pub async fn import_jsonl`, `async fn import_jsonl_unredacted`. +- `crates/agent-memory/src/main.rs`: the `"jsonl"` arm in `run_import`; updated `--from` doc; `--scope` rejection for jsonl. +- `tests/export_import.rs`: import tests (round trip, idempotent, phantom-edge kill-test, dedup-to-existing adjacency, in-file duplicates, expiry fixture, pending→sweep, header-only, zero-byte, file-order, hostile input). +- `tests/import.rs`: zeros assertion for the gsd-state path. +- New CLI surface: `agent-memory import --from jsonl PATH`. + + + + + Task 1 (tracer): A format-v1 file imports into another database with memories, links and provenance intact — and importing it again is a no-op + crates/agent-memory-core/src/import/jsonl.rs, crates/agent-memory-core/src/import/mod.rs, crates/agent-memory-core/src/service.rs, crates/agent-memory/src/main.rs, crates/agent-memory-core/tests/export_import.rs, crates/agent-memory-core/tests/import.rs + + - .planning/phases/05-portable-export-import/05-RESEARCH.md: Patterns 2, 3, 5, 6 (+6b), Pitfalls 1-4, 6, 9-11, Code Examples 2, 3; Open Questions 1-3 as resolved by the orchestrator (header counts, skew warning in core via `tracing::warn!`, `Linked { created }`). + - .planning/phases/05-portable-export-import/05-PATTERNS.md: sections for `jsonl.rs` (the `gsd_state.rs` shape to copy and invert), `import/mod.rs`, `service.rs` (`import()` structure the sibling copies), `main.rs`. + - crates/agent-memory-core/src/import/gsd_state.rs lines 1-45 and 188-200 (module doc, result struct, in-module test style) and crates/agent-memory-core/src/import/mod.rs (the "Tolerant" invariant to reword). + - crates/agent-memory-core/src/export.rs (plan 05-02: `Record`, `FORMAT_NAME`, `FORMAT_VERSION`). + - crates/agent-memory-core/src/domain.rs: `ImportedMemory` (plan 05-01), `LinkOutcome::Linked { created }`, `MemoryType::try_from` (lines 55-70), `LinkKind::try_from` (158-171), `validate_link_endpoints` (490-497, `pub(crate)`), `validate_patch`'s "content must be a non-empty string" wording (475-478), `MemoryError` Display strings. + - crates/agent-memory-core/src/store/mod.rs: `find_by_key`, `insert_imported`, `link` (all from plan 05-01). + - crates/agent-memory-core/src/service.rs: `ImportReport` (both construction sites inside `import_unredacted`), `import_unredacted` (the structure to mirror), `redact_storage_error`, `export_unredacted` (plan 05-02 — the hop + clock idioms). + - crates/agent-memory/src/main.rs: `Command::Import`, `run_import`, `default_scope_for`, and `run_export` (plan 05-02). + - crates/agent-memory-core/tests/export_import.rs (plan 05-02: harness + `export_to_string` + `lines`) and tests/import.rs lines 34-99 (the idempotency proof shape; the assertion you add). + - crates/agent-memory-core/tests/backfill_race.rs lines 195-213 (`vec_rows`/`status` raw helpers to copy). + + + - **jsonl.rs unit tests (basic set; Task 2 adds the full table):** the exporter's own output for a 2-memory/1-link database parses to `exported_rows == 2`, `memories.len() == 2`, `links.len() == 1`, `links[0].kind == LinkKind::RelatesTo`; a header with `"version":2,"later":1` is rejected with a message containing `upgrade agent-memory` (the probe runs before `deny_unknown_fields`); `"format":"x"` → `line 1: format must be`; `"version":0` → `line 1: version`; a link to a key absent from the file → `does not appear in the file`; `parse_jsonl("")` → `InvalidArgument` containing `line 1:`. + - **tests/export_import.rs — `round_trip_into_an_empty_database_recreates_memories_and_links`:** source harness (clock 1_000) stores A (tags `["x","y"]`, scope `Some("proj")`, source `Some("gsd-state")`, ttl `Some(1_000_000)`), B (Pattern, no tags, source `None`), C (Decision, source `Some("manual")`), advancing the clock between stores; `link(a, b, Supersedes)` and `link(c, b, RelatesTo)`; export via `export_to_string`. Target harness with its clock SET to 5_000 (so refresh is observable): `import_jsonl(parse_jsonl(&text)?)` → `ImportReport { imported: 3, skipped_duplicates: 0, skipped_expired: 0, links_imported: 2, links_skipped_duplicates: 0, links_dropped: 0 }`. Target `list` shows three rows whose `content`, `mem_type`, `tags`, `source`, `scope` equal the source's. Through a raw connection: each imported row's `created_at` and `access_count` equal the source row's, `expires_at` equals the source's (A: created + 1_000_000; B, C: NULL), `last_accessed == 5_000` for all three, `embedding_status == 0`, and `vec_memories` has zero rows for them; `memory_links` has two rows whose `created_at` values equal the source's link times and whose endpoints are the TARGET ids of A/B/C (`edges_touching(target_b) == 2`). Re-exporting the target and comparing line by line to the source export after normalising `key`, `last_accessed`, link `from`/`to`, and the header's `exported_at`/`agent_memory_version` yields equality (the remap itself is proven separately by `edges_touching(target_b) == 2` and the Task 2 phantom-edge kill-test). + - **`re_importing_the_same_file_is_idempotent`:** a second `import_jsonl(parse_jsonl(&text)?)` on the target → `{ imported: 0, skipped_duplicates: 3, skipped_expired: 0, links_imported: 0, links_skipped_duplicates: 2, links_dropped: 0 }`; `edge_count` and `SELECT count(*) FROM memories` unchanged. + - **`a_gsd_state_import_reports_zeros_for_the_jsonl_only_counts`** (tests/import.rs, extend the existing first test): `first.skipped_expired == 0 && first.links_imported == 0 && first.links_skipped_duplicates == 0 && first.links_dropped == 0`. + - **service.rs unit:** `clock_skew_message(now + 3600, now)` is `None`; `clock_skew_message(now + 3601, now)` is `Some(m)` with `m` containing `ahead of this machine's clock`. + - **End-to-end (tracer check):** `agent-memory export --db A --output x.jsonl` on a fresh database, then `agent-memory import --from jsonl x.jsonl --db B` exits 0 and prints `imported: 0, skipped duplicates: 0, skipped expired: 0, links imported: 0, links skipped duplicates: 0, links dropped: 0` followed by `0 rows pending embedding — …`. + + + Implements D-07, D-08, D-09, D-10, D-11 (parser), D-12, D-13 and the import half of D-01 per RESEARCH Patterns 2, 3, 5, 6, 6b. + + 1. **jsonl.rs (new).** Module doc: the strict reader for format v1 — deliberately the opposite tolerance policy from `gsd_state` (D-11: the file is agent-memory's own contract, so a defect means corruption or a hand edit, and a half-restored corpus is worse than a clear refusal); pure function, no I/O (the CLI reads the file); every rejection is `InvalidArgument` naming the 1-based line, so the CLI prints it and any future transport maps it to 400 / `invalid_params` (D-01); line numbers come from the importer's own counter, never from serde (Pitfall 1); the header is probed leniently before the strict parse so a v2 file reaches the upgrade message (Pitfall 2); timestamp handling is the importer's (`last_accessed` validated and discarded — D-10). Implement the types, `parse_jsonl`, `invalid_line`, `HeaderProbe` and a private `check_header(line: &str) -> Result<(), MemoryError>` exactly per "Interface contracts". Imports: `crate::domain::{ImportedMemory, LinkKind, MemoryError, MemoryType, validate_link_endpoints}`, `crate::export::{Record, FORMAT_NAME, FORMAT_VERSION}`. Add the basic unit tests from `` in a `#[cfg(test)] mod tests` (Task 2 completes the table). + 2. **import/mod.rs:** add `pub mod jsonl;` after `pub mod gsd_state;` and reword the first invariant from a blanket "Tolerant" to **tolerance is per importer**: `gsd_state` scrapes a human-edited text file and degrades messy input to skip-counts; `jsonl` reads agent-memory's own format-v1 contract and is strict — any defect rejects the whole file as `MemoryError::InvalidArgument` naming the line, and nothing is written (D-11); both must survive any text without panicking. In the Idempotent bullet name `Store::find_by_key` and the `(source, mem_type, content)` key; keep Data-only as is. + 3. **service.rs:** extend `ImportReport` with the four fields and `///` docs (what each counts; `links_dropped` = an endpoint skipped as expired or vanished between passes); set them to `0` at both construction sites in `import_unredacted` (gsd-state path). Add `CLOCK_SKEW_TOLERANCE_SECS`, `clock_skew_message` (+ its two unit tests), `import_jsonl` and `import_jsonl_unredacted` per "Interface contracts". Doc `import_jsonl` with the three passes, the per-record transaction note, the `<=` boundary vs the sweep's `<`, and the explicit "never embeds inline; the sweep backfill (`BACKFILL_BATCH` per tick) re-embeds against the local pin" sentence (D-13, PITFALLS 17). + 4. **main.rs:** rewrite `run_import`'s dispatch as the three-arm `match` per "Interface contracts". Parsing MUST precede `resolve_db_path`/`SqliteStore::open` in the jsonl arm so a malformed file never creates or touches a database. Keep the gsd-state arm's behaviour and output line byte-identical. Update the `--from` and `Import` docs. + 5. **tests/export_import.rs:** append `use agent_memory_core::import::jsonl::parse_jsonl;` plus raw helpers `status`, `vec_rows`, `row_count`, `column_i64(db, id, col)` as needed; add the two tests from ``. For the "normalised re-export equality" step parse each line as `serde_json::Value`, null out `key`, `last_accessed`, link `from`/`to`, `exported_at`, `agent_memory_version`, and compare the vectors. + 6. **tests/import.rs:** add the zeros assertion to `import_loads_typed_searchable_memories_idempotently` right after the existing `first.*` asserts. + 7. `cargo fmt --all`; clippy; full tests (background + poll); commit with explicit paths: `feat(05-03): import --from jsonl recreates memories and links idempotently (D-07..D-13)`. + + The parser and `import_jsonl` are additive beside the gsd-state path; the only shared surface is `ImportReport`'s additive fields. The strictness policy can be loosened later by editing one parser without touching the file format. + + cargo test -p agent-memory-core --lib import::jsonl && cargo test -p agent-memory-core --lib clock_skew && cargo test -p agent-memory-core --test export_import round_trip && cargo test -p agent-memory-core --test export_import idempotent && cargo test -p agent-memory-core --test import && cargo clippy --workspace --all-targets --locked -- -D warnings && T="$(mktemp -d)" && cargo run -q -p agent-memory -- export --db "$T/a.db" --output "$T/x.jsonl" 2>/dev/null && cargo run -q -p agent-memory -- import --from jsonl "$T/x.jsonl" --db "$T/b.db" | grep -c '^imported: 0, skipped duplicates: 0, skipped expired: 0, links imported: 0, links skipped duplicates: 0, links dropped: 0$' + + + - `grep -c "pub fn parse_jsonl" crates/agent-memory-core/src/import/jsonl.rs` outputs 1; `grep -c "upgrade agent-memory" crates/agent-memory-core/src/import/jsonl.rs` outputs at least 1; `grep -c "MemoryType::try_from" crates/agent-memory-core/src/import/jsonl.rs` outputs at least 1 and `grep -c "LinkKind::try_from" …/jsonl.rs` outputs at least 1; `grep -c "validate_link_endpoints" …/jsonl.rs` outputs at least 1; `grep -c "does not appear in the file" …/jsonl.rs` outputs at least 1. + - `grep -c "pub mod jsonl;" crates/agent-memory-core/src/import/mod.rs` outputs 1 and `grep -c "per importer" crates/agent-memory-core/src/import/mod.rs` outputs at least 1. + - `grep -c "pub async fn import_jsonl" crates/agent-memory-core/src/service.rs` outputs 1; `grep -c 'redact_storage_error("import"' …/service.rs` outputs 2 (gsd path + jsonl path); `grep -c "links_skipped_duplicates" …/service.rs` outputs at least 3; `grep -c "CLOCK_SKEW_TOLERANCE_SECS" …/service.rs` outputs at least 2. + - In `crates/agent-memory/src/main.rs`, inside `run_import`, `parse_jsonl(&text)` appears on a lower line number than the jsonl arm's `SqliteStore::open` (check: `grep -n "parse_jsonl(&text)\|SqliteStore::open" crates/agent-memory/src/main.rs`); `grep -c '"jsonl" =>' crates/agent-memory/src/main.rs` outputs 1; `grep -c "supported formats: gsd-state, jsonl" crates/agent-memory/src/main.rs` outputs 1; `grep -c "rows pending embedding" crates/agent-memory/src/main.rs` outputs 1. + - `cargo test -p agent-memory-core --test export_import round_trip` and `… idempotent` pass with the exact six-field reports in ``; `cargo test -p agent-memory-core --test import` is green including the zeros assertion. + - The end-to-end command in `` prints `1`. + - `git diff --stat -- crates/agent-memory-core/src/import/gsd_state.rs` is empty; `grep -c "embedder" crates/agent-memory-core/src/service.rs` is unchanged from before this task except inside doc comments (the jsonl path never references `self.embedder`): `sed -n '/async fn import_jsonl_unredacted/,/^ }/p' crates/agent-memory-core/src/service.rs | grep -c "self.embedder"` outputs 0. + - fmt, clippy `-D warnings`, `cargo test --workspace --locked` green; `Cargo.lock` unchanged. + + A user can move a memory file to another machine, run `agent-memory import --from jsonl`, and get every memory and link back with provenance intact, pending re-embedding, and a second run that changes nothing. + + + + Task 2: Overlapping rowids, duplicates, expiry, pending re-embed and every malformed-file class behave exactly as decided + crates/agent-memory-core/tests/export_import.rs, crates/agent-memory-core/src/import/jsonl.rs + + - crates/agent-memory-core/tests/export_import.rs (Task 1 state) and tests/relations.rs lines 85-104 (`edges_touching`, `orphan_edges`). + - crates/agent-memory-core/src/import/jsonl.rs (Task 1 state — every message string you assert on). + - .planning/phases/05-portable-export-import/05-RESEARCH.md: Code Examples 2, 3, 6; Pitfalls 3, 4, 6, 10, 11; Assumptions A3, A4; Security Domain rows (parser never panics on hostile input). + - .planning/phases/05-portable-export-import/05-CONTEXT.md `` (the three kill-tests the user named). + - crates/agent-memory-core/src/embed/mod.rs lines 92-110 (`FakeEmbedder::with_vectors` — an UP embedder for the sweep proof) and service.rs `sweep`/`BACKFILL_BATCH`. + + + All in `tests/export_import.rs` unless stated; `HEADER`/`MEM1` string consts as in RESEARCH Code Example 2 (with `exported_rows`/`exported_links` added to the header const and adjusted per case). + - `importing_into_a_non_empty_db_with_overlapping_rowids_never_attaches_phantom_edges` (PITFALLS 18): source: 3 memories (ids 1..=3) + `link(1, 2, RelatesTo)`; target pre-seeded with 3 unrelated memories so ids 1..=3 are taken; import → `{ imported: 3, skipped_duplicates: 0, skipped_expired: 0, links_imported: 1, links_skipped_duplicates: 0, links_dropped: 0 }`; for every pre-existing id `edges_touching(id) == 0`; `edges_touching(new_a) == 1` and `edges_touching(new_b) == 1`; `orphan_edges == 0`; target row count 6. + - `link_endpoints_that_dedup_to_existing_rows_attach_to_those_rows` (INTEROP-02 adjacency): target already holds a row identical on `(source, mem_type, content)` to source memory A (stored BEFORE the import, so it has a different id than A's key); import a file with A, B and `A→B relates_to` → `imported: 1, skipped_duplicates: 1, links_imported: 1`; `edges_touching(existing_a_id) == 1`; row count grew by exactly 1 (no second A). + - `identical_rows_collapse_to_one_on_import_and_count_a_duplicate` (DIST-04 adjacency): source stores the SAME `NewMemory` twice (no uniqueness constraint), export → header `exported_rows == 2` and two memory records; import into a fresh target → `imported: 1, skipped_duplicates: 1`; target row count 1. + - `expired_rows_are_skipped_and_counted_and_their_links_dropped` (PITFALLS 19): source at clock 1_000 stores A (`ttl 50` → `expires_at 1_050`), B (`ttl 10_000`), C (no ttl); `link(a, c, CausedBy)`, `link(b, c, Supersedes)`; export at 1_000. Target clock SET to exactly 1_050 (boundary: `<=` skips) → `{ imported: 2, skipped_duplicates: 0, skipped_expired: 1, links_imported: 1, links_skipped_duplicates: 0, links_dropped: 1 }`; target `list` shows B and C only; no row with A's content. Also: a source row whose `last_accessed` is 400 days older than its export (store at 1_000, advance the source clock 400 days, export) imports into a target at clock T with `last_accessed == T` and is returned by `list` (decay never deletes; PITFALLS 19 refresh). + - `imported_rows_land_pending_and_the_sweep_backfills_them` (criterion 3): target harness with `FakeEmbedder::with_vectors(HashMap::new())` (embedder UP); import 2 rows → both `status == 0`, `vec_rows == 0`; `sweep(now)` → both `status == 1`, `vec_rows == 1`; `search(SearchArgs { query: , .. })` returns `search_mode == SearchMode::Hybrid` and contains that row. + - `a_header_only_file_imports_nothing_and_reports_zeros` (INTEROP-02 empty + DIST-04 empty import half): export an empty source → one header line; `import_jsonl(parse_jsonl(..)?)` → every field 0; target row count unchanged. `a_zero_byte_file_is_rejected_at_line_one`: `parse_jsonl("")` and `parse_jsonl("\n")` → `InvalidArgument` with `line 1:`. + - `links_are_applied_after_all_memories_regardless_of_file_order` (INTEROP-02 ordering): hand-build HEADER (`exported_rows 2, exported_links 1`) + the LINK line + two MEM lines → parses, imports with `links_imported: 1`; importing a canonical-order file with the same records into a second fresh target yields an identical report. + - `malformed_files_are_rejected_naming_the_line` (jsonl.rs `#[cfg(test)]`, table-driven over `(name, text, needle)`): no header (`MEM1` first) → `line 1:`; bad format → `line 1: format must be`; `"version":0` → `line 1: version must be >= 1`; `"version":"1"` → `line 1: version must be a positive integer`; `"version":2` + unknown field → `upgrade agent-memory`; `[1,2]` on line 2 → `line 2: expected a JSON object`; blank line 2 → `line 2: expected a JSON object`; `{"record":"memo","key":1}` → `line 2:` and `unknown variant`; MEM1 + `"bogus":1` → `line 2:` and `unknown field`; `"key":"1"` → `line 2:`; `"type":"MEMO"` → `line 2:` and `MEMO`; `"content":" "` → `content must be a non-empty string`; link `"kind":"loves"` → `kind must be one of`; self-link → `must differ`; dangling key → `does not appear in the file`; two MEM lines with the same key → `duplicate memory key`; header `exported_rows 5` with one MEM → `declares 5 memory records but contains 1`; a second header line → `duplicate header`. Every case: `matches!(err, MemoryError::InvalidArgument(_))` and `err.to_string().contains(needle)`. + - `hostile_input_never_panics` (jsonl.rs): a memory line whose `tags` value is 300 nested arrays → `Err` (serde_json's recursion limit), not a panic; a 4 MiB `content` string parses `Ok`; a line of 1 MiB of `{` characters → `Err`; `"\u0000"` in content → `Ok` (stored as data). No `catch_unwind` — the test simply must not abort. + - `nothing_is_written_when_validation_fails`: a file whose line 2 is valid MEM1 and line 3 is `{"record":"memo"}` → `parse_jsonl` is `Err`, so `import_jsonl` is never reached; assert the target row count is unchanged after the failed parse (the structural guarantee: the only entry point to writes takes a fully-parsed `JsonlFile`). + + + Proves D-07 (collapse), D-08 (two-pass, drop vs reject), D-10 (boundary, refresh), D-11 (the full rejection table), D-13 (pending → sweep → hybrid) and the four edge probes assigned to this plan; fixes any parser or pass gap the tests surface — without touching `gsd_state.rs` or the file format. + + 1. Add the raw helpers you still need to `tests/export_import.rs` (`status`, `vec_rows`, `row_count`, `orphan_edges` copied from relations.rs/backfill_race.rs) and a `fn file_from(lines: &[&str]) -> String` joiner. + 2. Write the nine `tests/export_import.rs` tests from ``. For the expiry fixture use `h.clock.set(..)` on the target harness; for the "ancient row" use `h.clock.advance(400 * 86_400)` on the source before exporting. + 3. Write `malformed_files_are_rejected_naming_the_line` and `hostile_input_never_panics` in `jsonl.rs`'s test module per ``. Build the hostile strings with `"[".repeat(300) + &"]".repeat(300)` and `"x".repeat(4 << 20)`. + 4. If a case fails because the parser's message differs from the table, fix the PARSER to match the "Interface contracts" wording (the table is the contract), unless the contract text itself is ambiguous — then prefer the more specific message and record the choice in SUMMARY.md. + 5. `cargo fmt --all`; clippy; full tests (background + poll); commit with explicit paths: `test(05-03): JSONL import kill-tests — phantom edges, expiry, pending→sweep, malformed table`. + + + cargo test -p agent-memory-core --test export_import && cargo test -p agent-memory-core --lib import::jsonl && cargo clippy -p agent-memory-core --all-targets --locked -- -D warnings + + + - `cargo test -p agent-memory-core --test export_import` runs at least 16 tests (the plan 02 five, the Task 1 two, the nine here) all green; the test names `importing_into_a_non_empty_db_with_overlapping_rowids_never_attaches_phantom_edges`, `link_endpoints_that_dedup_to_existing_rows_attach_to_those_rows`, `identical_rows_collapse_to_one_on_import_and_count_a_duplicate`, `expired_rows_are_skipped_and_counted_and_their_links_dropped`, `imported_rows_land_pending_and_the_sweep_backfills_them`, `a_header_only_file_imports_nothing_and_reports_zeros`, `a_zero_byte_file_is_rejected_at_line_one`, `links_are_applied_after_all_memories_regardless_of_file_order`, `nothing_is_written_when_validation_fails` each appear once in the file (`grep -c "fn "` → 1). + - `cargo test -p agent-memory-core --lib import::jsonl` is green and the table test covers at least 18 cases (`grep -c '^\s*("' crates/agent-memory-core/src/import/jsonl.rs` ≥ 18 inside the test module). + - The phantom-edge test asserts `edges_touching(pre_existing) == 0` for all three pre-seeded ids AND `orphan_edges == 0`. + - The expiry test asserts the exact report `{2, 0, 1, 1, 0, 1}` at target clock 1_050 (the `<=` boundary). + - The sweep test asserts `embedding_status` 0 → 1 across `sweep` and `search_mode == SearchMode::Hybrid` afterwards. + - `git diff --stat -- crates/agent-memory-core/src/import/gsd_state.rs crates/agent-memory-core/src/export.rs` is empty (the format and the gsd importer did not move). + - fmt, clippy `-D warnings`, `cargo test --workspace --locked` green. + + Every INTEROP-02 criterion and every edge the user named is a green test: no phantom edges on a non-empty target, duplicates collapse, expired rows are skipped and counted, re-embedding happens only through the sweep, and every malformed-file class is refused by line with nothing written. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| user-supplied file (CLI path) → parser | untrusted bytes: hand-edited, truncated, hostile, or from another machine's clock | +| parsed records → store writer lane | typed data; `type`/`kind` already closed-enum, content stored strictly as data | +| file-local keys ↔ target database ids | two id spaces that must never be confused | +| local clock ↔ file timestamps | absolute timestamps from a possibly skewed source machine | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan | +|-----------|----------|-----------|----------|-------------|-----------------| +| T-05-04 | Tampering | model-mismatched vectors poisoning `vec_memories` on import | high | mitigate | no vector field exists in format v1 (`deny_unknown_fields` rejects one); `insert_imported` always writes `embedding_status = 0`; the sweep re-embeds against the local `meta` pin (Task 1; sweep proof Task 2) | +| T-05-05 | Tampering (integrity) | file `key`/`from`/`to` treated as target ids → phantom edges on unrelated rows | high | mitigate | keys only index the pass-2 `HashMap`; pass 3 looks both ends up; links to unmapped keys are dropped and counted; overlapping-rowid kill-test with `edges_touching(pre_existing) == 0` (Task 2) | +| T-05-06 | Tampering | hand-edited `type`/`kind`/`tags` reaching SQL | medium | mitigate | `MemoryType::try_from` / `LinkKind::try_from` at the seam; `tags: Vec` enforced by serde; every store statement bound with `params![]` (Task 1) | +| T-05-07 | Denial of Service | parser panic/abort on hostile input (deep nesting, huge line, control chars) | medium | mitigate | serde_json's recursion limit returns `Err`; no `unwrap`/index arithmetic on untrusted data; `hostile_input_never_panics` (Task 2); invalid UTF-8 is refused by `read_to_string` in the CLI with an `anyhow` context (plan 04 proves) | +| T-05-08 | Tampering / DoS | half-restored corpus on a malformed file | high | mitigate | pass 1 validates the entire file before the database is opened; `import_jsonl` only accepts a fully parsed `JsonlFile`; `nothing_is_written_when_validation_fails` (Tasks 1-2) | +| T-05-09 | Repudiation (UX) | expired rows reported "imported" then deleted by the sweep | low | mitigate | `expires_at <= now` → `skipped_expired`, never inserted; dependent links → `links_dropped`; boundary test at exactly `now` (Task 2) | +| T-05-10 | Denial of Service | memory exhaustion: pass 1 holds every parsed record of a huge file in RAM | low | accept | Inherent to validate-before-write (D-08); a 4 MiB content line is tested to parse; documented in the README (plan 04); content size is bounded only by what the store accepts | +| T-05-11 | Elevation of Privilege / Information Disclosure | import as a REST/MCP endpoint = arbitrary-file-read primitive on an unauthenticated daemon | high | mitigate (by design) | D-01: no tool, no route; the path is read only in `main.rs`; acceptance grep for `memory_import`/`/api/import` in the transports is empty (plan 04 re-checks) | +| T-05-13 | Tampering | source-machine clock skew making fresh rows look expired or stale | low | mitigate | absolute timestamps evaluated against the local clock; `exported_at > now + 3600` logs a warning naming both clocks and continues (Task 1) | +| T-05-17 | Information Disclosure | storage-engine text on an import fault | medium | mitigate | `import_jsonl` wrapped by `redact_storage_error("import", _)` (plan 05-01 seam) | +| T-05-SC | Tampering | cargo installs | low | accept | zero new dependencies; `--locked` | + +No `high` severity threat is left open (T-05-04, T-05-05, T-05-08, T-05-11 are mitigated). ASVS L1, block_on: high. + + + +- `cargo fmt --all --check`, `cargo clippy --workspace --all-targets --locked -- -D warnings`, `cargo test --workspace --locked` green after both commits. +- `cargo test -p agent-memory-core --test export_import` ≥ 16 green; `cargo test -p agent-memory-core --lib import::jsonl` green with the ≥ 18-case table; `cargo test -p agent-memory-core --test import` green with the zeros assertion. +- The real binary round-trips an empty database between two `--db` paths with the all-zero six-count line. +- `git diff --stat -- crates/agent-memory-core/src/import/gsd_state.rs crates/agent-memory-core/sql/` empty; `Cargo.lock` unchanged. + + + +- `agent-memory import --from jsonl PATH` exists, parses before opening the database, and prints the six counts plus the pending line. +- Round trip, idempotency, overlapping-rowid, dedup-adjacency, in-file duplicate, expiry, pending→sweep→hybrid, header-only, zero-byte, file-order and nothing-written proofs are green; the malformed table covers every D-11 class. +- Two commits on `main`, fmt + clippy + full tests green each; nothing pushed. + + + +Create `.planning/phases/05-portable-export-import/05-03-SUMMARY.md` when done. Record: the final rejection-message table (one line per class), the pass structure (one blocking hop, per-record transactions), the `add-alongside` assumption-delta decision, and any message wording you changed from the interface contract. + diff --git a/.planning/phases/05-portable-export-import/05-03-SUMMARY.md b/.planning/phases/05-portable-export-import/05-03-SUMMARY.md new file mode 100644 index 0000000..3ba084c --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-03-SUMMARY.md @@ -0,0 +1,286 @@ +--- +phase: 05-portable-export-import +plan: 03 +subsystem: core-service, import, cli +tags: [rust, serde, jsonl, sqlite, import, idempotency, validation, two-pass, key-remap] + +# Dependency graph +requires: + - phase: 05-portable-export-import + provides: "plan 05-01: Store::find_by_key -> Option (D-07), LinkOutcome::Linked { created } (D-12), ImportedMemory + Store::insert_imported (D-10/D-13), redact_storage_error(op, e) + the *_unredacted inner-fn shape (D-14)" + - phase: 05-portable-export-import + provides: "plan 05-02: export.rs Record enum (deny_unknown_fields, tag = record) + FORMAT_NAME / FORMAT_VERSION, header exported_rows / exported_links, Store::export_rows + MemoryService::export, CLI export / --output, tests/export_import.rs harness + export_to_string" + - phase: 03-api-hardening-toolchain-spikes + provides: "MemoryError::InvalidArgument client tier at the MemoryService seam; MemoryType::try_from / LinkKind::try_from / validate_link_endpoints as the one validation seam (D-04/D-06, D-09)" + - phase: 04-memory-update-relations-hybrid-search + provides: "Store::link with endpoint pre-check + INSERT OR IGNORE on (from_id, to_id, kind); relates_to canonical (min, max); sweep backfill (BACKFILL_BATCH = 128) that re-embeds pending rows" +provides: + - "import/jsonl.rs: pure strict parser parse_jsonl(text) -> Result — lenient header probe before the strict Record parse, the importer's own 1-based line counter, every rejection InvalidArgument(\"line N: …\"), link keys resolved order-independently, declared counts enforced; 25-row malformed table + hostile-input probes" + - "service.rs: ImportReport + skipped_expired / links_imported / links_skipped_duplicates / links_dropped (Default-derived; gsd path reports zeros); CLOCK_SKEW_TOLERANCE_SECS + clock_skew_message; MemoryService::import_jsonl(file) running pass 2 (expiry skip -> find_by_key dedup -> insert_imported, building key -> id) and pass 3 (Store::link through the map with the record's created_at) in ONE spawn_blocking hop, never touching the embedder, redacted as op = \"import\"" + - "CLI `agent-memory import --from jsonl PATH`: parses and validates the whole file BEFORE the database is resolved or opened, prints the six-count line plus the pending-embedding line, rejects --scope; gsd-state arm output byte-identical; shared open_service / read_import_file helpers" + - "tests/export_import.rs import half (11 tests): round trip into an empty target, idempotent re-import, overlapping-rowid phantom-edge kill-test, dedup-to-existing adjacency, in-file duplicates collapse, expiry boundary fixture + ancient-row refresh, pending -> sweep -> hybrid, header-only no-op, zero-byte rejection, file-order independence, nothing-written-on-failure" +affects: [05-04 README Export & import section + spawned-binary CLI round trip, phase-6 release notes] + +# Actuals (#2632) — same estimateTokens scale as the plan's `estimate` (chars/4 over the realized diff) +actuals: + tokens: 20000 + tasks: 2 + commits: 2 + +# Tech tracking +tech-stack: + added: [] + patterns: + - "Two-pass validate-before-write import: a pure parser returns a fully typed file or an InvalidArgument naming the line; the service method's ONLY input type is that parsed file, so a malformed file structurally cannot reach a write" + - "Lenient-probe-then-strict header parse: a serde struct WITHOUT deny_unknown_fields reads record/format/version first so a newer-version file reaches the upgrade message instead of failing on an unknown field" + - "Importer-owned line numbers: `text.lines().enumerate()` is the only line counter; serde_json's ` at line 1 column N` suffix is stripped from every re-wrapped message because serde sees one line at a time" + - "Seam re-wrap without double prefix: an InvalidArgument from the shared validators is lifted out of its variant before `line N:` is prepended; other variants keep their full Display text" + - "Export-local keys live only in a HashMap built during the memories pass; the links pass looks both endpoints up and drops-and-counts a miss — a file key is never used as a target id" + - "Table-driven contract tests under #[rustfmt::skip]: one (name, input, expected-prefix) tuple per line so the rejection table reads as the spec" + +key-files: + created: + - crates/agent-memory-core/src/import/jsonl.rs + modified: + - crates/agent-memory-core/src/import/mod.rs + - crates/agent-memory-core/src/service.rs + - crates/agent-memory/src/main.rs + - crates/agent-memory-core/tests/export_import.rs + - crates/agent-memory-core/tests/import.rs + +key-decisions: + - "add-alongside (assumption delta): `jsonl` is a second importer beside `gsd-state` with its own parser module and the dedicated ImportedMemory insert shape; no shared Importer trait, because the two tolerance policies (tolerant scrape vs strict contract) are the point of D-11. A third import format is what would force a promote to an ImportSource enum/trait" + - "HeaderProbe reads `format` and `version` as Option so a file whose first line is a memory record is refused as `first record must be the header` (the contract's intended message) rather than `missing field format`; a header lacking them says `(missing)`" + - "serde_json's ` at line 1 column N` suffix is stripped from re-wrapped messages — surfacing it would contradict the rule that the importer's counter is the only line number" + - "An InvalidArgument from MemoryType/LinkKind/validate_link_endpoints is unwrapped before re-wrapping, so `kind must be one of …` reads `invalid argument: line 3: kind must be one of …` and not `invalid argument: line 3: invalid argument: kind …`" + - "Per-record writer transactions (the insert / link shape) in one blocking hop: a storage fault mid-pass leaves a partial import that a re-run completes idempotently; parse_jsonl is the all-or-nothing gate" + - "main.rs factors open_service / read_import_file so both --from arms share the wiring; run_export is untouched" + - "Tracer feedback gate resolved by re-running the fully automated after the Task 1 commit (same resolution as 05-01 / 05-02: auto mode off, config human_verify_mode: end-of-phase, no human-judgment step in the verify)" + +patterns-established: + - "redact_storage_error op = \"import\" now covers two service methods (import, import_jsonl); the acceptance grep for `redact_storage_error(\"import\"` is 2" + - "Import CLI output contract: `imported: N, skipped duplicates: N, skipped expired: N, links imported: N, links skipped duplicates: N, links dropped: N` then `N rows pending embedding — the sweep backfills them on the next serve/serve-rest start (128 per tick)`" + +requirements-completed: [INTEROP-02, DIST-04] + +# Coverage metadata (#1602) +coverage: + - id: D1 + description: "A format-v1 file imports into an empty OR a non-empty target with every memory and link recreated on the target's own ids; export-local keys are remapped through the pass-2 map and no pre-existing row ever gains a phantom edge (D-08, PITFALLS 18)" + requirement: DIST-04 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/export_import.rs#round_trip_into_an_empty_database_recreates_memories_and_links, #importing_into_a_non_empty_db_with_overlapping_rowids_never_attaches_phantom_edges, #link_endpoints_that_dedup_to_existing_rows_attach_to_those_rows" + status: pass + - kind: e2e + ref: "cargo run -q -p agent-memory -- export --db A --output x.jsonl && import --from jsonl x.jsonl --db B -> exit 0, `imported: 0, … links dropped: 0`, `0 rows pending embedding — …`" + status: pass + human_judgment: false + - id: D2 + description: "Re-running the same import reports imported 0 / skipped_duplicates N / links_imported 0 / links_skipped_duplicates M with row and edge counts unchanged; two source rows identical on (source, mem_type, content) collapse to one; source is preserved verbatim (D-07)" + requirement: INTEROP-02 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/export_import.rs#re_importing_the_same_file_is_idempotent, #identical_rows_collapse_to_one_on_import_and_count_a_duplicate (+ the second import inside the phantom-edge test)" + status: pass + human_judgment: false + - id: D3 + description: "Imported rows land embedding_status = 0 with no vector row; the importer never calls the embedder; one sweep backfills them and they are hybrid-searchable (D-13)" + requirement: INTEROP-02 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/export_import.rs#imported_rows_land_pending_and_the_sweep_backfills_them" + status: pass + - kind: other + ref: "sed -n '/async fn import_jsonl_unredacted/,/^ }/p' service.rs | grep -c self.embedder -> 0" + status: pass + human_judgment: false + - id: D4 + description: "created_at / access_count / expires_at preserved, last_accessed refreshed to import-time now, expires_at <= now skipped and counted (boundary exact), dependent links dropped and counted, link created_at preserved, exported_at > now + 3600 warns and continues (D-10)" + requirement: INTEROP-02 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/export_import.rs#expired_rows_are_skipped_and_counted_and_their_links_dropped (target clock 1_050 -> {2,0,1,1,0,1}; 1_049 -> {3,0,0,2,0,0}; 400-day-old row -> last_accessed == 7_777 through the skew-warning path), #round_trip_into_an_empty_database_recreates_memories_and_links (provenance columns)" + status: pass + - kind: unit + ref: "crates/agent-memory-core/src/service.rs#clock_skew_message_fires_only_beyond_the_tolerance" + status: pass + human_judgment: false + - id: D5 + description: "Every malformed-file class is rejected as InvalidArgument naming the 1-based line and nothing is written; wire type/kind strings go through MemoryType::try_from / LinkKind::try_from; hostile input never panics (D-09, D-11)" + requirement: INTEROP-02 + verification: + - kind: unit + ref: "crates/agent-memory-core/src/import/jsonl.rs#malformed_files_are_rejected_naming_the_line (25 cases), #hostile_input_never_panics, #a_newer_version_with_extra_fields_reaches_the_upgrade_message, #serde_position_suffix_is_stripped_from_messages (+ 7 more)" + status: pass + - kind: integration + ref: "crates/agent-memory-core/tests/export_import.rs#nothing_is_written_when_validation_fails, #a_zero_byte_file_is_rejected_at_line_one, #a_header_only_file_imports_nothing_and_reports_zeros, #links_are_applied_after_all_memories_regardless_of_file_order" + status: pass + - kind: e2e + ref: "real binary: a file with `{\"record\":\"memo\"}` on line 2 exits non-zero with `invalid argument: line 2: unknown variant `memo`…` and the --db path is never created" + status: pass + human_judgment: false + - id: D6 + description: "ImportReport carries six counts; the gsd-state path reports zeros for the four new ones; the CLI prints all six on one line plus the pending-embedding line; --scope is rejected for jsonl; unsupported --from names both formats (D-12)" + requirement: INTEROP-02 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/import.rs#import_loads_typed_searchable_memories_idempotently (four zeros assertion)" + status: pass + - kind: e2e + ref: "real binary: six-count line + pending line; `--scope s` -> `--scope applies only to --from gsd-state …`; `--from yaml` -> `unsupported --from value \"yaml\" (supported formats: gsd-state, jsonl)`" + status: pass + human_judgment: true + rationale: "Counts and exit codes are machine-checked; whether the two CLI lines read clearly to a user restoring a corpus is editorial — plan 04's spawned-binary round trip and README section are where a reviewer sees them in context" + +# Metrics +duration: 16min +completed: 2026-10-25 +status: complete +--- + +# Phase 5 Plan 03: `agent-memory import --from jsonl` — strict two-pass restore with key→id remapping Summary + +**A format-v1 file now restores on another machine: `parse_jsonl` validates the whole file into typed records (every defect refused as `InvalidArgument` naming the 1-based line, nothing written), `MemoryService::import_jsonl` runs the memories pass (expiry skip → `find_by_key` dedup → `insert_imported`, building `key → id`) and the links pass through that map in one blocking hop without ever touching the embedder, and the CLI arm parses before it opens the database and prints the six-count report plus the pending-embedding line.** + +## Performance + +- **Duration:** 16 min +- **Started:** 2026-10-25T12:04:27Z +- **Completed:** 2026-10-25T12:20:00Z +- **Tasks:** 2 +- **Files modified:** 6 (1 created, 5 modified; +1681 / −53 lines) + +## Accomplishments + +- **Strict parser, one entry point to writes (D-08 pass 1, D-11).** `crates/agent-memory-core/src/import/jsonl.rs` exposes `parse_jsonl(text) -> Result` with `JsonlHeader`, `JsonlMemory { key, memory: ImportedMemory }`, `JsonlLink { from, to, kind: LinkKind, created_at }`. Line 1 is probed leniently (`HeaderProbe` without `deny_unknown_fields`: record → format → version-is-unsigned → `>= 1` → `<= FORMAT_VERSION`) before the strict `Record::Header` parse, so a `"version":2,"later":1` header reaches `file is format version 2 but this binary reads up to version 1 — upgrade agent-memory`. Every later line must start with `{`, parse as a `Record`, and pass the Phase 3 seam: `MemoryType::try_from`, non-empty content, unique key; `LinkKind::try_from`, `validate_link_endpoints`. Link endpoints are resolved against the complete key set after the loop (order-independent), and the record counts must equal the header's `exported_rows` / `exported_links`. `last_accessed` is validated as an integer and dropped (D-10 refresh). The module is pure — the CLI reads the file (D-01). +- **`import_jsonl` — passes 2 and 3, one hop, no embedder (D-07, D-08, D-10, D-12, D-13, D-14).** `ImportReport` gained `skipped_expired`, `links_imported`, `links_skipped_duplicates`, `links_dropped` (and `Default`); both gsd-state construction sites fill them with `..ImportReport::default()`. `import_jsonl_unredacted` reads the clock once, logs `clock_skew_message` when `exported_at > now + CLOCK_SKEW_TOLERANCE_SECS` (3600), then in ONE `spawn_blocking`: for each memory, `expires_at <= now` → `skipped_expired` (no map entry); else `find_by_key(source, mem_type, content)` → `Some(id)` maps the key to the EXISTING row and counts a duplicate, `None` → `insert_imported(memory, now)` and maps to the new id. For each link, both endpoints are looked up in the map: `Linked { created: true }` → `links_imported`, `Linked { created: false }` → `links_skipped_duplicates`, `NotFound` or an unmapped key → `links_dropped`; the edge time is the record's own `created_at`. `sed -n '/async fn import_jsonl_unredacted/,/^ }/p' service.rs | grep -c self.embedder` → 0. Errors leave through `redact_storage_error("import", _)` — the grep count is now 2. +- **CLI arm (D-01, D-12).** `run_import` is a three-arm `match from.as_str()`: `"gsd-state"` (behaviour and output line byte-identical), `"jsonl"` (rejects `--scope` with `--scope applies only to --from gsd-state (a jsonl file carries each memory's own scope)`; `parse_jsonl(&text)` at main.rs:282 runs BEFORE `open_service` → `SqliteStore::open` at :320, so a malformed file never creates a database; prints `imported: …, skipped duplicates: …, skipped expired: …, links imported: …, links skipped duplicates: …, links dropped: …` then `N rows pending embedding — the sweep backfills them on the next serve/serve-rest start (128 per tick)`), and `other` → `unsupported --from value {other:?} (supported formats: gsd-state, jsonl)`. The two arms share new `read_import_file` / `open_service` helpers. `--from` and `Import` docs name both formats and the dedup key. +- **Kill-tests, all green on first run.** `tests/export_import.rs` grew from 5 to 16 tests. Overlapping rowids: source keys 1..=3 vs a target whose ids 1..=3 are unrelated rows → `{3,0,0,1,0,0}`, `edges_touching(pre) == 0` for all three, `orphan_edges == 0`, 6 rows, and a second import of the same file → `{0,3,0,0,1,0}`. Dedup adjacency: a row identical to A already at id 2 while A's key is 1 → the edge attaches to id 2, never to id 1, row count +1. In-file twins → `exported_rows == 2`, import `{1,1,0,0,0,0}`, one row. Expiry: target clock exactly 1_050 → `{2,0,1,1,0,1}` with B and C listed and A absent; 1_049 → `{3,0,0,2,0,0}`; a row 400 days stale imports with `last_accessed == 7_777`, `created_at == 1_000`, and is listed (through the skew-warning path). Pending → sweep: with an UP `FakeEmbedder`, both rows land `status 0 / vec_rows 0`, one `sweep` makes them `1 / 1`, and `search("orchid")` returns `Hybrid` containing the row. Header-only → all zeros, counts unchanged; `""` and `"\n"` → `line 1:`; LINK-before-MEM parses and imports identically to canonical order with the edge carrying `created_at 5`; a valid line 2 followed by `{"record":"memo"}` writes nothing. +- **The D-11 table.** 25 `(name, input, expected-prefix)` rows in `jsonl.rs` under `#[rustfmt::skip]`, every one asserted `InvalidArgument` + `starts_with("invalid argument: line ")` + the needle; plus `hostile_input_never_panics` (300 nested arrays → `Err`; 4 MiB content → `Ok`; 1 MiB of `{` → `Err`; `\u0000` → `Ok`, stored as data). +- **Nothing else moved:** `git diff --stat` for `gsd_state.rs`, `export.rs`, `sql/`, `Cargo.lock`, `.github/` is empty; no new dependency; `--locked` throughout. + +## Task Commits + +Each task was committed atomically (fmt + clippy `-D warnings` + full workspace suite green before each): + +1. **Task 1 (tracer): A format-v1 file imports into another database with memories, links and provenance intact — and importing it again is a no-op** — `9d937f4` (feat) +2. **Task 2: Overlapping rowids, duplicates, expiry, pending re-embed and every malformed-file class behave exactly as decided** — `c7bc811` (test) + +**Plan metadata:** see the `docs(05-03)` commit that follows this summary. + +## Files Created/Modified + +- `crates/agent-memory-core/src/import/jsonl.rs` (new, 634 lines) — module doc (strict policy, error contract, header probe, timestamp handling); `JsonlHeader`, `JsonlMemory`, `JsonlLink`, `JsonlFile`; `HeaderProbe`; `invalid_line`, `rewrap`, `serde_message`, `check_header`, `parse_record`, `pub fn parse_jsonl`; 11 unit tests incl. the 25-row table and the hostile probes +- `crates/agent-memory-core/src/import/mod.rs` — `pub mod jsonl;`; invariants reworded: tolerance is per importer, idempotency names `Store::find_by_key` and both service entry points +- `crates/agent-memory-core/src/service.rs` — `ImportReport` +4 fields + `Default`; `CLOCK_SKEW_TOLERANCE_SECS`, `clock_skew_message`; `pub async fn import_jsonl` + `async fn import_jsonl_unredacted`; `clock_skew_message_fires_only_beyond_the_tolerance` test; `use crate::import::jsonl::{JsonlFile, JsonlLink, JsonlMemory}` +- `crates/agent-memory/src/main.rs` — `Import` / `--from` / `--scope` docs; `run_import` three-arm match; `read_import_file`; `open_service` +- `crates/agent-memory-core/tests/export_import.rs` — module doc extended; `db_path` exposed; raw helpers (`raw_conn`, `row_count`, `edge_count`, `edges_touching`, `orphan_edges`, `status`, `vec_rows`, `id_by_content`, `Provenance`/`provenance`, `edges`), `import`, `normalised`, `six`, `header`, `MEM1`/`MEM2`/`LINK12`, `file_from`, `seed_round_trip_source`; 11 import tests +- `crates/agent-memory-core/tests/import.rs` — four-zeros assertion on the gsd-state report + +## Decisions Made + +- **`add-alongside`** (the plan's assumption-delta decision, confirmed in code): `jsonl` sits beside `gsd_state` as a second parser module with its own policy; both share the seam (`MemoryService`), the dedup probe (`find_by_key`) and the writer lane, not a type. Accepted debt: two `--from` arms, two parser modules, two insert shapes. A third format forces an `ImportSource` enum/trait. +- **Pass structure:** pass 1 (validation) is the all-or-nothing gate and lives entirely in `parse_jsonl`; passes 2 and 3 share one `spawn_blocking` hop with per-record writer transactions (the shape of `insert` / `link`), so a storage fault mid-pass leaves a partial import that an idempotent re-run completes. +- **Message wording changed from the interface contract (recorded per the plan's Task 2 step 4):** + 1. `HeaderProbe.format` / `.version` are `Option` — a first line that is not a header now says `line 1: first record must be the header` (the contract's intended message; with `String` fields it would have been `missing field format`), and a header lacking them says `format must be 'agent-memory-export' (missing)` / `version must be a positive integer (missing)`. + 2. serde_json's ` at line 1 column N` suffix is stripped from every re-wrapped message (`serde_message`) — the contract said "verbatim", but the suffix IS serde's line number, which the same contract forbids surfacing. + 3. Seam errors are re-wrapped through `rewrap`, which lifts an `InvalidArgument` payload out of its variant: `invalid argument: line 3: kind must be one of …`, not `invalid argument: line 3: invalid argument: kind …`. `InvalidType` keeps its full text (`invalid memory type 'MEMO' (expected one of …)`). +- **Tracer feedback gate:** auto mode is off, but `.planning/config.json` sets `human_verify_mode: "end-of-phase"` and the tracer's `` is wholly ``. I re-ran it end-to-end on the committed tree (lib `import::jsonl` 11/11, `clock_skew` 1/1, `--test export_import` round_trip + idempotent, `--test import` 2/2, workspace clippy, the real-binary e2e printing `1`) and continued to Task 2 rather than returning a mid-plan `checkpoint:human-verify` — the same resolution as 05-01 and 05-02, recorded so the orchestrator can route end-of-phase UAT. + +## Final rejection-message table (one line per class) + +| Class | Message (after `invalid argument: `) | +|---|---| +| zero-byte file | `line 1: missing or invalid header (empty file)` | +| line 1 not JSON | `line 1: missing or invalid header ()` | +| line 1 not a header record | `line 1: first record must be the header` | +| wrong format | `line 1: format must be 'agent-memory-export' (got '')` / `(missing)` | +| non-integer version | `line 1: version must be a positive integer (got )` / `(missing)` | +| version 0 | `line 1: version must be >= 1 (got 0)` | +| version > 1 | `line 1: file is format version but this binary reads up to version 1 — upgrade agent-memory` | +| header missing a strict field | `line 1: missing field \`\`` | +| non-object line | `line N: expected a JSON object` | +| unknown record tag | `line N: unknown variant \`\`, expected one of \`header\`, \`memory\`, \`link\`` | +| unknown field | `line N: unknown field \`\`, expected one of …` | +| wrong field type | `line N: invalid type: , expected ` | +| invalid type | `line N: invalid memory type '' (expected one of DECISION, PATTERN, ERROR, TODO, ARCHITECTURE, CONSTRAINT)` | +| empty content | `line N: content must be a non-empty string` | +| duplicate memory key | `line N: duplicate memory key ` | +| invalid kind | `line N: kind must be one of relates_to, supersedes, caused_by (got '')` | +| self-link | `line N: from_id and to_id must differ (got )` | +| link key absent from file | `line N: link key does not appear in the file` (N = the link's line) | +| memory count mismatch | `line : file declares memory records but contains ` | +| link count mismatch | `line : file declares link records but contains ` | +| second header | `line N: duplicate header` | + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] Double `invalid argument:` prefix on seam errors** +- **Found during:** Task 1 (the malformed table's `bad kind` case, run before the parser was finished — the RED observation) +- **Issue:** `LinkKind::try_from` and `validate_link_endpoints` already return `InvalidArgument`, whose Display is `invalid argument: …`; re-wrapping them through `Display` produced `invalid argument: line 3: invalid argument: kind must be one of …`. +- **Fix:** `rewrap(line_no, e)` lifts an `InvalidArgument` payload out before prefixing; other variants keep their Display text. +- **Files modified:** `crates/agent-memory-core/src/import/jsonl.rs` +- **Verification:** `cargo test -p agent-memory-core --lib import::jsonl` 11/11; the table asserts the exact `line 3: kind must be one of` prefix +- **Committed in:** `9d937f4` + +**2. [Rule 3 - Blocking] rustfmt broke the table's one-tuple-per-line layout the acceptance grep counts** +- **Found during:** Task 2 (acceptance-criteria gate: `grep -c '^\s*("' jsonl.rs` ≥ 18 returned 13, then 2, for a 25-row table) +- **Issue:** rustfmt's 60-column short-argument width wraps any tuple wider than that onto four lines, so the literal grep undercounts however the inputs are factored. +- **Fix:** `#[rustfmt::skip]` on the `let cases` statement (with a comment saying why), the standard idiom for table-driven tests; the in-test `assert!(cases.len() >= 18)` guards the count independently of layout. +- **Files modified:** `crates/agent-memory-core/src/import/jsonl.rs` +- **Verification:** grep → 25; `cargo fmt --all -- --check` clean; clippy clean +- **Committed in:** `c7bc811` + +### Process notes (not code deviations) + +- **TDD RED/GREEN without a separate RED commit.** Both tasks are `tdd="true"`. RED was observed: Task 1's `tests/export_import.rs` and `tests/import.rs` additions reference `import_jsonl`, `parse_jsonl` and the four new `ImportReport` fields, so they failed to compile against the pre-task tree by construction, and the parser's table test failed on the double-prefix bug above before the parser was corrected. Each task's commit carries tests + implementation together because the project operating rules forbid committing failing tests and the plan calls for one commit per task. +- **Task 2's `jsonl.rs` tests landed in Task 1.** The malformed table and hostile probes were authored alongside the parser (a parser and its rejection table are one unit, like 05-02's goldens), so the Task 1 commit already contains them; the Task 2 commit reflows the table to one case per line. The nine `tests/export_import.rs` tests are Task 2's as planned. +- **Pre-bash safety hook.** Never tripped: logs were truncated with `: > file`, databases lived in `mktemp -d` dirs and were never removed by the shell. + +--- + +**Total deviations:** 2 auto-fixed (1 Rule 1 message bug caught by the table before it shipped, 1 Rule 3 plan-internal criterion vs rustfmt); no scope or behaviour change beyond the three recorded wording choices. +**Impact on plan:** None on scope. The wording choices make the D-11 messages read as the contract intended; the formatting attribute exists only to make the plan's own grep pass as written. + +## Issues Encountered + +None beyond the two auto-fixes. Every targeted `` and the full workspace suite passed on the first post-GREEN run (265 after Task 1, 274 after Task 2); no fix-attempt budget was consumed on behaviour. + +## Verification + +Plan-level `` re-run after the second commit: + +- `cargo fmt --all -- --check` — clean; `cargo clippy --workspace --all-targets --locked -- -D warnings` — exit 0; `cargo test --workspace --locked` — **274 passed, 0 failed, 2 ignored** (251 before this plan → 265 after Task 1: +11 `import::jsonl`, +1 `clock_skew`, +2 `export_import` → 274 after Task 2: +9 `export_import`). +- `cargo test -p agent-memory-core --test export_import` — 16 passed (≥ 16 required); `cargo test -p agent-memory-core --lib import::jsonl` — 11 passed with the 25-row table (≥ 18 required, grep → 25); `cargo test -p agent-memory-core --test import` — 2 passed incl. the zeros assertion. +- Real binary: `export --db A --output x.jsonl` then `import --from jsonl x.jsonl --db B` → exit 0, exactly one line matching `^imported: 0, skipped duplicates: 0, skipped expired: 0, links imported: 0, links skipped duplicates: 0, links dropped: 0$` (grep → `1`) followed by `0 rows pending embedding — …`; a malformed file → non-zero exit, `invalid argument: line 2: unknown variant \`memo\`…`, and the `--db` file is never created; `--scope` with jsonl and `--from yaml` print their respective messages. +- Acceptance greps: `pub fn parse_jsonl` 1; `upgrade agent-memory` 4; `MemoryType::try_from` 3; `LinkKind::try_from` 4; `validate_link_endpoints` 4; `does not appear in the file` 3; `pub mod jsonl;` 1; `per importer` 1; `pub async fn import_jsonl` 1; `redact_storage_error("import"` 2; `links_skipped_duplicates` ≥ 3; `CLOCK_SKEW_TOLERANCE_SECS` 5; `"jsonl" =>` 1; `supported formats: gsd-state, jsonl` 1; `rows pending embedding` 1; `parse_jsonl(&text)` (main.rs:282) precedes the arm's `SqliteStore::open` (:320); `self.embedder` inside `import_jsonl_unredacted` 0. +- `git diff --stat -- crates/agent-memory-core/src/import/gsd_state.rs crates/agent-memory-core/src/export.rs crates/agent-memory-core/sql/ Cargo.lock .github/` — empty. +- No `unwrap()` / `expect()` outside `#[cfg(test)]` in any touched production file (`jsonl.rs` test module starts at line 393; `service.rs` at 1116). + +## Known Stubs + +None — no placeholder values, TODO/FIXME markers, skipped tests, or unwired data paths were introduced. + +## Threat Flags + +None new. The plan's register is implemented as mitigated: T-05-04 (no vector field in `Record`; `insert_imported` always `embedding_status = 0`; sweep proof), T-05-05 (keys only index the pass-2 map; overlapping-rowid kill-test), T-05-06 (seam conversion; `tags: Vec`; `params![]` in the store), T-05-07 (`hostile_input_never_panics`; no `unwrap`), T-05-08 (`parse_jsonl` completes before `open_service`; `nothing_is_written_when_validation_fails`), T-05-09 (`<=` boundary test), T-05-11 (CLI only — no `memory_import` / `/api/import` added), T-05-13 (`clock_skew_message` + unit test, exercised by the ancient-row fixture), T-05-17 (`redact_storage_error("import", _)`), T-05-SC (zero new deps, `--locked`). T-05-10 (pass 1 holds the file in RAM) stays accepted; the 4 MiB content probe parses. + +## User Setup Required + +None - no external service configuration required. + +## Next Phase Readiness + +- **Plan 05-04 (docs + CLI realism)** has everything it documents: the format spec in `export.rs`'s module doc, the rejection table above (copy it into the README "Export & import" section), the six-count + pending-line CLI contract, the `--scope`-is-gsd-only rule, and the strict-vs-tolerant importer note from `import/mod.rs`. Its spawned-binary round trip can reuse the real-binary commands in Verification. Adding `export` and `import_jsonl` to `tests/redaction.rs` remains the cheap follow-up noted in 05-02. +- No blockers. `.planning/config.json` and the untracked harness files were left unstaged as instructed. + +--- +*Phase: 05-portable-export-import* +*Completed: 2026-10-25* + +## Self-Check: PASSED + +All 7 key files exist on disk; task commits 9d937f4 and c7bc811 are present in git history. diff --git a/.planning/phases/05-portable-export-import/05-04-PLAN.md b/.planning/phases/05-portable-export-import/05-04-PLAN.md new file mode 100644 index 0000000..23443ed --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-04-PLAN.md @@ -0,0 +1,203 @@ +--- +phase: 05-portable-export-import +plan: 04 +type: execute +wave: 4 +depends_on: [05-02, 05-03] +files_modified: + - README.md + - crates/agent-memory/tests/cli.rs +autonomous: true +requirements: [DIST-04, INTEROP-02] + +estimate: + tokens: 30000 + raw_tokens: 30000 + tasks: 2 + confidence: low + +must_haves: + truths: + - "The README has an 'Export & import' section that documents the CLI usage, the format v1 contract (header, memory and link field tables in wire order), what is excluded and why, the strict-reader rules including the version policy, the dedup/idempotency rule and non-empty-target safety, the per-column timestamp policy, the pending-embedding behaviour with the sweep cadence, the six-count report, the private atomic --output, the CLI-only security rationale, and a same-machine backup tip contrasting a SQLite backup with a portable export (D-01, D-02, D-03, D-04, D-05, D-06, D-07, D-08, D-10, D-11, D-12, D-13)" + - "The real binary round-trips: a seed JSONL imports with the documented six-count line and pending line, exports to a private file whose content re-imports into a second database, and a re-import reports only duplicates (D-01, D-02, D-12)" + - "A malformed file makes `agent-memory import --from jsonl` exit non-zero with stderr naming the line, and the target database file is not created (D-11); `--from bogus` and `--scope` with jsonl are refused with the documented messages; invalid UTF-8 is refused without a panic" + artifacts: + - path: "README.md" + provides: "## Export & import section; Lifecycle cross-reference; Run example" + contains: "## Export & import" + - path: "crates/agent-memory/tests/cli.rs" + provides: "spawned-binary realism tests for export/import" + contains: "CARGO_BIN_EXE_agent-memory" + key_links: + - from: "README.md Export & import" + to: "crates/agent-memory-core/src/export.rs module doc" + via: "the README field tables restate the Record declaration order exactly" + pattern: "agent-memory-export" + - from: "crates/agent-memory/tests/cli.rs" + to: "crates/agent-memory/src/main.rs run_export / run_import" + via: "Command::new(env!(\"CARGO_BIN_EXE_agent-memory\")) with --db on a tempdir" + pattern: "\"--from\", \"jsonl\"" + prohibitions: + - statement: "Export and import must never be exposed over MCP or REST in this phase — the README states the CLI-only rule and the transports carry no such tool or route" + status: resolved + verification: test +--- + +# Plan 05-04: Document the contract and prove it over the real binary + +## Phase Goal + +**As a** user, **I want to** export all my memories to a portable file and import them on another machine, **so that** my agent's memory moves with me — across machines, with no cloud sync. + +Plans 02 and 03 built both halves; this plan makes the contract legible to the user and exercises the shipped binary end-to-end exactly the way a user will. + + +1. Add the README "Export & import" section — format v1 spec tables, strict-vs-tolerant note, timestamp policy, pending-embedding note, the six-count report, `--output` privacy/atomicity, the CLI-only rationale, and the backup-vs-export tip — plus a Lifecycle cross-reference and a Run example. +2. Add `crates/agent-memory/tests/cli.rs`: realism-only spawned-binary tests (zero coverage by design, like `tests/rest.rs`) for the export → import → re-import round trip, the malformed-file exit path, `--from bogus`, `--scope` rejection and invalid UTF-8. + +Purpose: a user can read how to move their memory and what every count means, and the exact binary that ships is proven to do it. +Output: README section; `tests/cli.rs`. + + + +@$HOME/.claude/gsd-core/workflows/execute-plan.md +@$HOME/.claude/gsd-core/templates/summary.md + + + +@.planning/STATE.md +@.planning/phases/05-portable-export-import/05-CONTEXT.md +@.planning/phases/05-portable-export-import/05-02-SUMMARY.md +@.planning/phases/05-portable-export-import/05-03-SUMMARY.md +@crates/agent-memory-core/src/export.rs +@crates/agent-memory-core/src/import/jsonl.rs +@crates/agent-memory/src/main.rs +@README.md + + +## Execution notes for every task + +- Gate before EVERY commit: `cargo fmt --all`, `cargo clippy --workspace --all-targets --locked -- -D warnings`, `cargo test --workspace --locked` (background + poll). +- One commit per task; stage explicit paths only. NEVER stage `CLAUDE.md`, `.claude/`, `.mcp.json`, `.gitignore`, `.gsd/`, `docs/`, `10-agent-memory.md`, `claude-code-harness-engineering-guide-v2.md`, `.planning/research/.cache/`, `.planning/config.json`, `.planning/milestone.lock`. +- Never change the CI runner or add a workflow. Never push, tag or release. + +## Artifacts this phase produces (this plan's share) + +- README.md: `## Export & import` (new section between "Import from GSD" and "Lifecycle: TTL & decay"); one sentence under Lifecycle item 3; one `agent-memory export --output memories.jsonl` line in "Run". +- `crates/agent-memory/tests/cli.rs` (new): `export_import_round_trip_over_the_real_binary`, `a_malformed_file_exits_non_zero_naming_the_line_and_creates_no_database`, `an_unsupported_from_value_still_bails`, `scope_is_rejected_for_jsonl`, `invalid_utf8_is_rejected_cleanly`. + + + + + Task 1: The README tells a user how to move their memory and what every line of the report means + README.md + + - README.md: lines 70-83 ("Run"), 414-470 ("REST API" — the redaction sentence plan 01 widened), 486-512 ("Behavioral changes in v0.1.0" — item 4 from plan 01), 513-533 ("Import from GSD" — the section yours follows), 534-556 ("Lifecycle: TTL & decay" — item 3 gets the cross-reference). + - crates/agent-memory-core/src/export.rs module doc (the authoritative field tables — restate, do not contradict) and crates/agent-memory-core/src/import/jsonl.rs module doc (the reader rules and message shapes). + - crates/agent-memory/src/main.rs: the `Export`/`Import` clap docs and the exact `println!`/`eprintln!` report strings. + - .planning/phases/05-portable-export-import/05-CONTEXT.md: D-01..D-13 and the Claude's-Discretion bullet on the `VACUUM INTO` backup tip; .planning/research/FEATURES.md §3 (backup vs portable export anti-pattern row). + + + Documents D-01..D-13 for users. Insert `## Export & import` immediately after the "Import from GSD" section and before "## Lifecycle: TTL & decay", containing in this order: + + 1. **Usage** — a bash block with `agent-memory export > memories.jsonl`, `agent-memory export --output memories.jsonl`, and `agent-memory import --from jsonl memories.jsonl`; one sentence: both are CLI subcommands only (no MCP tool, no REST route — a file-path read on an unauthenticated daemon would be an arbitrary-file-read primitive, and the corpus must not cross the JSON-RPC stdout channel) (D-01). + 2. **What travels** — a header line, one record per memory, one record per link; **what does not** — embedding vectors, `decay_score`, `base_weight`, `embedding_status` (derived; recomputed on read or re-embedded locally against the local model pin) (D-04, D-13). + 3. **Format v1** — three tables restating `export.rs` in wire order: header (`record`, `format` = `agent-memory-export`, `version` = 1, `exported_at`, `embedding_model`, `embedding_dim`, `schema_version`, `agent_memory_version`, `exported_rows`, `exported_links`), memory (`record`, `key`, `type`, `content`, `tags`, `source`, `scope`, `created_at`, `last_accessed`, `access_count`, `expires_at`), link (`record`, `from`, `to`, `kind`, `created_at`); notes: `key` is the source row id used only as an export-local key and never becomes an id on import; `relates_to` endpoints appear as stored (smaller id first); links follow all memories; every row is exported — no filters, expired rows included (the importer decides) (D-03, D-05, D-06). + 4. **The reader is strict** — in contrast to the tolerant `gsd-state` importer: any defect (bad header, unknown `record`, unknown field, wrong type, invalid `type`/`kind`, empty content, self-link, a link to a key not in the file, a declared-count mismatch) rejects the whole file with `invalid argument: line N: …` and nothing is written; a file with `version` greater than 1 is refused with `file is format version N but this binary reads up to version 1 — upgrade agent-memory` (D-11). + 5. **Idempotent and safe on a non-empty database** — dedup on the exact `(source, type, content)` key with `source` kept verbatim; re-running the same import is a no-op reported as skipped duplicates; two identical source rows collapse to one; links are remapped through the file's keys, so pre-existing rows never gain edges (D-07, D-08). + 6. **Timestamps** — a table: `created_at` preserved; `expires_at` preserved, and a row already expired at import time (`expires_at <= now`) is skipped and counted, never inserted; `last_accessed` refreshed to import time so restored memories start decay-fresh; `access_count` preserved (provenance only — it does not affect ranking); link `created_at` preserved; a header `exported_at` more than an hour ahead of the local clock prints a warning and continues (D-10). + 7. **Embeddings** — imported rows land pending; the existing lifecycle sweep re-embeds them, 128 per tick, first tick at `serve`/`serve-rest` start; a large restore is keyword-only until the backfill catches up; nothing contacts Ollama during the import itself (D-13). + 8. **The report** — the one-line six counts (`imported`, `skipped duplicates`, `skipped expired`, `links imported`, `links skipped duplicates`, `links dropped` — the last being links whose endpoint was skipped as expired) and the pending-embedding line (D-12). + 9. **`--output` is private and atomic** — created `0600` on Unix as `PATH.export-tmp-` beside PATH and renamed into place; an existing PATH is replaced; an interrupted export never leaves a partial PATH; Windows permissions follow the directory (Phase 6) (D-02). + 10. **Backup vs export** — for a same-machine, full-fidelity copy (vectors included) stop every agent-memory process and use SQLite's own backup, e.g. `sqlite3 memory.db "VACUUM INTO 'memory-backup.db'"`, or keep the automatic pre-migration backup described under Upgrading; use `export` when the memory must move machines or survive a model change. + Also: in "Lifecycle: TTL & decay" item 3 append "Rows restored by `agent-memory import --from jsonl` take this path too." In "Run", add `agent-memory export --output memories.jsonl # portable JSONL export (see Export & import)` beside the existing examples. Keep the "Import from GSD" section as is. Do not restate the REST redaction sentence (plan 01 widened it) — link to it only if natural. + Commit with explicit paths: `docs(05-04): README Export & import section (format v1, timestamp policy, pending embeddings)`. + + + test "$(grep -c '^## Export & import' README.md)" = 1 && test "$(grep -c 'agent-memory-export' README.md)" -ge 1 && test "$(grep -c 'upgrade agent-memory' README.md)" -ge 1 && test "$(grep -c 'VACUUM INTO' README.md)" -ge 1 && test "$(grep -c '0600' README.md)" -ge 1 && test "$(grep -c 'links dropped' README.md)" -ge 1 && test "$(grep -c 'export-tmp-' README.md)" -ge 1 && sed -n '/^## Import from GSD/,/^## Lifecycle/p' README.md | grep -c '^## Export & import' + + + - `grep -c '^## Export & import' README.md` outputs 1 and the section sits between "## Import from GSD" and "## Lifecycle: TTL & decay". + - The section contains all ten topics in ``: check `grep -c 'exported_rows' README.md` ≥ 1, `grep -c 'skipped expired' README.md` ≥ 1, `grep -c 'last_accessed' README.md` ≥ 1, `grep -c '128 per tick\|128 rows per tick\|128 at a time' README.md` ≥ 1, `grep -c 'no MCP tool\|CLI subcommands only\|CLI-only' README.md` ≥ 1, `grep -c 'VACUUM INTO' README.md` ≥ 1. + - The three field tables list the fields in the same order as `Record`'s declaration in `crates/agent-memory-core/src/export.rs` (compare by eye against the enum; every field name in the enum appears in README: `for f in exported_rows exported_links embedding_model embedding_dim schema_version agent_memory_version access_count expires_at last_accessed created_at; do grep -q "$f" README.md || echo MISSING $f; done` prints nothing). + - `sed -n '/^## Lifecycle/,/^## Development/p' README.md | grep -c 'import --from jsonl'` outputs 1; `sed -n '/^## Run/,/^## Upgrading/p' README.md | grep -c 'agent-memory export'` outputs at least 1. + - `cargo test --workspace --locked` still green (docs-only change; the gate is run anyway). + + A user who has never read the code can export, move, and import their memory, understand every count in the report, and knows why vectors are re-created locally and why the commands are CLI-only. + + + + Task 2: The shipped binary round-trips a memory file between two databases and refuses bad input cleanly + crates/agent-memory/tests/cli.rs + + - crates/agent-memory/tests/tools.rs lines 1-60 (`binary_path()` via `env!("CARGO_BIN_EXE_agent-memory")`, `--db` on a tempdir, the dead-Ollama env var `AGENT_MEMORY_OLLAMA_URL=http://127.0.0.1:9`). + - crates/agent-memory/tests/rest.rs lines 1-50 (second spawn precedent) and crates/agent-memory/src/rest/handlers.rs lines 456-460 (why spawned-binary tests are realism-only: a child never flushes LLVM profile data). + - crates/agent-memory/src/main.rs `run_export`/`run_import` (the exact stdout/stderr strings and exit paths) and crates/agent-memory/src/export_file.rs (temp naming). + - crates/agent-memory-core/src/export.rs (`Record` field order — to hand-write a valid seed file). + + + Each test spawns `Command::new(binary_path())` with `.env("AGENT_MEMORY_OLLAMA_URL", "http://127.0.0.1:9")`, `--db` under a fresh `tempfile::tempdir()`, and captures `Output { status, stdout, stderr }`. + - `export_import_round_trip_over_the_real_binary`: write `seed.jsonl` by hand — header (`exported_rows 2`, `exported_links 1`, `embedding_model "nomic-embed-text"`, `embedding_dim 768`, `schema_version 3`, any `agent_memory_version`) + two memory records (keys 10 and 20, so they cannot coincide with target ids) + one link `10 → 20 relates_to`; `import --from jsonl seed.jsonl --db A` → exit 0, stdout line 1 == `imported: 2, skipped duplicates: 0, skipped expired: 0, links imported: 1, links skipped duplicates: 0, links dropped: 0`, line 2 starts with `2 rows pending embedding`; `export --db A --output out.jsonl` → exit 0, stderr contains `exported 2 memories and 1 links`, `out.jsonl` has 4 lines, line 1 starts `{"record":"header"`, under `#[cfg(unix)]` its mode `& 0o777 == 0o600`, and no sibling named `*.export-tmp-*` exists; `export --db A` to stdout → exit 0 and its lines 2-4 equal `out.jsonl` lines 2-4 (the header's `exported_at` may differ by wall clock); `import --from jsonl out.jsonl --db B` → `imported: 2, … links imported: 1, …`; the same again → `imported: 0, skipped duplicates: 2, skipped expired: 0, links imported: 0, links skipped duplicates: 1, links dropped: 0`. + - `a_malformed_file_exits_non_zero_naming_the_line_and_creates_no_database`: file = valid header + `{"record":"memo"}`; `import --from jsonl bad.jsonl --db /never.db` → non-zero exit, stderr contains `line 2:`, and `/never.db` does not exist afterwards. + - `an_unsupported_from_value_still_bails`: `--from bogus` → non-zero, stderr contains `supported formats: gsd-state, jsonl`. + - `scope_is_rejected_for_jsonl`: `--from jsonl seed.jsonl --scope x` → non-zero, stderr contains `--scope applies only to --from gsd-state`. + - `invalid_utf8_is_rejected_cleanly`: a file containing bytes `[0xff, 0xfe, 0x0a]` → non-zero, stderr contains `reading`, stderr does NOT contain `PANIC`. + + + Realism proof for D-01, D-02, D-11, D-12 over the exact binary that ships (coverage-bearing proofs live in core and the in-process handler tests — this file is intentionally spawned-only, mirroring `tests/rest.rs`). + + 1. Create `crates/agent-memory/tests/cli.rs` with a module doc saying exactly that. Helpers: `fn binary_path() -> String`, `fn run(args: &[&str]) -> std::process::Output` (sets the dead-Ollama env, inherits nothing else, `.output()`), `fn stdout(o: &Output) -> String` / `fn stderr(o: &Output) -> String` (lossy UTF-8), `fn seed_jsonl() -> String` building the header + two memories + one link with `serde_json::json!` so field order follows the `Record` declaration (serialise via `serde_json::to_string` of hand-built `Value`s is fine — the parser does not care about key order; the exporter does). + 2. Write the five tests from ``. For the file-permission check use `std::os::unix::fs::PermissionsExt` under `#[cfg(unix)]`. For the "no temp sibling" check, iterate `std::fs::read_dir` and assert no file name contains `.export-tmp-`. + 3. `cargo fmt --all`; clippy; full tests (background + poll); commit with explicit paths: `test(05-04): CLI realism round trip for export/import over the spawned binary`. + + + cargo test -p agent-memory --test cli && cargo clippy -p agent-memory --all-targets --locked -- -D warnings + + + - `cargo test -p agent-memory --test cli` runs 5 tests, all green: `grep -c "^async fn \|^fn \|#\[test\]" crates/agent-memory/tests/cli.rs` shows 5 `#[test]` attributes. + - The round-trip test asserts the exact first-line report strings for the first import (`imported: 2, … links imported: 1, …`) and the re-import (`imported: 0, skipped duplicates: 2, … links skipped duplicates: 1, …`), the `exported 2 memories and 1 links` stderr line, the 4-line file, and (unix) mode `0600`. + - The malformed test asserts both the non-zero exit with `line 2:` on stderr AND that the `--db` path was never created. + - `grep -rn "memory_export\|memory_import\|/api/export\|/api/import" crates/agent-memory/src/mcp.rs crates/agent-memory/src/rest/` outputs nothing (D-01 re-checked at phase end). + - fmt, clippy `-D warnings`, `cargo test --workspace --locked` green. + + The binary a user installs moves a memory file between two databases with the documented output, refuses malformed input before touching the target database, and never panics on garbage. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| documentation → user expectations | the README is what users rely on for the privacy and integrity promises | +| CLI arguments and files → the shipped binary | realism tests exercise the real argument parsing, file reading and exit paths | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan | +|-----------|----------|-----------|----------|-------------|-----------------| +| T-05-11 | Elevation / Information Disclosure | export/import exposed as a daemon endpoint | high | mitigate (by design) | README states the CLI-only rule and its reason (D-01); Task 2's acceptance grep re-checks that no MCP tool or REST route exists | +| T-05-10 | Denial of Service | huge import held in memory during validation | low | accept | documented behaviour (validate-before-write); no cap added this phase | +| T-05-12 | Information Disclosure | export to stdout landing in a terminal scrollback or log | low | accept | user-directed; README shows `--output` as the private alternative and documents 0600 | +| T-05-18 | Repudiation (UX) | users misreading the six counts (e.g. treating `links dropped` as an error) | low | mitigate | README defines every count and the pending line (Task 1); CLI test pins the exact wording (Task 2) | +| T-05-SC | Tampering | cargo installs | low | accept | zero new dependencies; `--locked` | + +No `high` severity threat is left open (T-05-11 is mitigated by design and re-verified). ASVS L1, block_on: high. + + + +- `cargo test -p agent-memory --test cli` — 5 green against the built binary. +- README section present with all ten topics; Lifecycle and Run cross-references present. +- Phase-level: `cargo fmt --all --check`, `cargo clippy --workspace --all-targets --locked -- -D warnings`, `cargo test --workspace --locked` green; `cargo llvm-cov --workspace --fail-under-lines 80` green (the CI gate — run once at phase end if `cargo-llvm-cov` is installed locally; it is per RESEARCH Environment Availability). +- `git diff --stat -- crates/agent-memory-core/sql/` empty across the phase; `Cargo.lock` unchanged across the phase. + + + +- README documents export/import completely; the spawned-binary round trip and the four refusal paths are green. +- Two commits on `main`; nothing pushed, tagged or released. + + + +Create `.planning/phases/05-portable-export-import/05-04-SUMMARY.md` when done. Record the coverage-gate result if run, and confirm the D-01 transport grep is empty. + diff --git a/.planning/phases/05-portable-export-import/05-04-SUMMARY.md b/.planning/phases/05-portable-export-import/05-04-SUMMARY.md new file mode 100644 index 0000000..a257006 --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-04-SUMMARY.md @@ -0,0 +1,208 @@ +--- +phase: 05-portable-export-import +plan: 04 +subsystem: docs, cli, testing +tags: [rust, readme, jsonl, export, import, spawned-binary, realism-test, format-v1] + +# Dependency graph +requires: + - phase: 05-portable-export-import + provides: "plan 05-02: export.rs module-doc field tables (the authoritative format-v1 spec), `agent-memory export [--output]`, export_file::PendingExport (0600 temp beside the target + rename), the stderr `exported N memories and M links` report" + - phase: 05-portable-export-import + provides: "plan 05-03: import::jsonl strict parser + rejection table, MemoryService::import_jsonl, the three-arm `run_import` with its six-count line, pending-embedding line, `--scope` refusal and `unsupported --from value` message" + - phase: 05-portable-export-import + provides: "plan 05-01: README redaction sentence widened to every surface (not restated here); ImportedMemory / insert_imported timestamp semantics the Timestamps table documents" +provides: + - "README `## Export & import` (between 'Import from GSD' and 'Lifecycle: TTL & decay'): usage, CLI-only rationale, what travels / what is derived, the three format-v1 field tables in wire order, strict-reader rules + version policy, dedup and non-empty-target safety, per-column timestamp table, pending embeddings + 128-per-tick hourly sweep, the six-count report with a meaning table, private atomic --output, backup-vs-export tip (D-01..D-13)" + - "README Lifecycle item 3 cross-reference for rows restored by `import --from jsonl`; Run section `agent-memory export --output memories.jsonl` example" + - "crates/agent-memory/tests/cli.rs: five realism-only spawned-binary tests (zero coverage by design, like tests/rest.rs) — export → import → re-import round trip across two databases with the exact report strings, 0600 file mode, no temp sibling; malformed file names `line 2:` and never creates the --db; `--from bogus`, `--scope` with jsonl and invalid UTF-8 refused with the documented messages and no PANIC" + - "Phase-end gates recorded: D-01 transport grep empty; `cargo llvm-cov --workspace --fail-under-lines 80` → 95.61% lines; sql/, Cargo.lock and .github/ unchanged across all phase-5 commits" +affects: [phase-6 release notes and Homebrew/Windows distribution, end-of-phase UAT for phase 5, any future transport for export/import (must update the README CLI-only paragraph)] + +# Actuals (#2632) — same estimateTokens scale as the plan's `estimate` (chars/4 over the realized diff) +actuals: + tokens: 5300 + tasks: 2 + commits: 2 + +# Tech tracking +tech-stack: + added: [] + patterns: + - "Spawned-binary realism tests are realism-only: `Command::new(env!(\"CARGO_BIN_EXE_agent-memory\"))`, `--db` under a fresh tempdir, `AGENT_MEMORY_OLLAMA_URL=http://127.0.0.1:9`, `.output()` captured; the module doc says they contribute zero coverage (a child never flushes LLVM profile data) and names where the coverage-bearing proofs live" + - "Hand-written seed files use export-local keys that cannot coincide with target ids (10 and 20, not 1 and 2), so a remapping bug would surface as a wrong count rather than pass by coincidence" + - "README restates the `Record` declaration order verbatim in three tables and points at `export.rs` as the authoritative definition — the doc never contradicts the code because it copies the code's order" + - "A new integration test file is drafted outside the crate until the previous task's gate has run and committed, so an in-flight `cargo test --workspace` never compiles an uncommitted test target" + +key-files: + created: + - crates/agent-memory/tests/cli.rs + modified: + - README.md + +key-decisions: + - "The README section is one `##` with ten `###` subsections in the plan's order (Usage → What travels → Format v1 → The reader is strict → Idempotent → Timestamps → Embeddings → The report → --output → Backup vs export), so each topic is linkable and the field tables are not buried in prose" + - "The sweep cadence is documented as '128 rows per tick, the first tick at serve/serve-rest start and then hourly' — `SWEEP_INTERVAL` is 3600 s in main.rs, so 'hourly' is the shipped number, not an approximation" + - "The usage paragraph states the `--scope` refusal for jsonl (each memory carries its own scope) — not in the plan's ten topics, but it is the one CLI flag a gsd-state user would reach for and the message is pinned by the CLI test" + - "The refusal tests additionally assert the `--db` path is never created for `--from bogus`, `--scope` and invalid UTF-8 (the plan required this only for the malformed-file test) — all three bail or fail before `open_service`, and the extra assertion costs nothing while pinning that order" + - "Coverage gate run once at phase end as the plan prescribed: `cargo llvm-cov --workspace --fail-under-lines 80` passes at 95.61% lines (regions 93.28%, functions 88.81%); tests/cli.rs runs under it but, as designed, adds no coverage" + +patterns-established: + - "README field tables mirror the serde declaration order of the wire enum and say so" + - "Realism tests pin exact user-visible strings (report lines, error substrings) rather than parsing them, so a wording change is a deliberate README + test change together" + +requirements-completed: [DIST-04, INTEROP-02] + +# Coverage metadata (#1602) +coverage: + - id: D1 + description: "README `## Export & import` section documents CLI usage, the CLI-only rule and its reason, what travels and what is derived, the three format-v1 field tables in wire order, the strict-reader rules and version policy, dedup/idempotency and non-empty-target safety, the per-column timestamp policy, the pending-embedding behaviour and sweep cadence, the six-count report, the private atomic --output, and the backup-vs-export tip; Lifecycle item 3 and Run cross-references present" + requirement: DIST-04 + verification: + - kind: other + ref: "grep -c '^## Export & import' README.md → 1; section sits between '## Import from GSD' (l.527) and '## Lifecycle: TTL & decay' (l.742); every Record field name present; 'skipped expired', 'last_accessed', '128 per tick', 'no MCP tool', 'VACUUM INTO', '0600', 'export-tmp-', 'links dropped', 'upgrade agent-memory' all ≥ 1; Lifecycle 'import --from jsonl' → 1; Run 'agent-memory export' → 1" + status: pass + human_judgment: true + rationale: "The greps prove every topic is present; whether a user who has never read the code can follow the section and understand every count is a legibility judgment no grep makes" + - id: D2 + description: "The shipped binary round-trips a hand-written seed through two databases: first import prints the exact six-count line and `2 rows pending embedding`; `--output` writes a 4-line 0600 file with no `.export-tmp-` sibling and the stderr report; the stdout export carries identical records; the file restores into a second database and a re-import reports only duplicates" + requirement: INTEROP-02 + verification: + - kind: e2e + ref: "cargo test -p agent-memory --test cli export_import_round_trip_over_the_real_binary" + status: pass + human_judgment: false + - id: D3 + description: "The binary refuses bad input cleanly: a malformed file exits non-zero with `line 2:` on stderr and the --db path is never created; `--from bogus` lists the supported formats; `--scope` with jsonl prints the documented message; invalid UTF-8 is refused via the `reading` step with no PANIC" + requirement: INTEROP-02 + verification: + - kind: e2e + ref: "cargo test -p agent-memory --test cli a_malformed_file_exits_non_zero_naming_the_line_and_creates_no_database" + status: pass + - kind: e2e + ref: "cargo test -p agent-memory --test cli an_unsupported_from_value_still_bails" + status: pass + - kind: e2e + ref: "cargo test -p agent-memory --test cli scope_is_rejected_for_jsonl" + status: pass + - kind: e2e + ref: "cargo test -p agent-memory --test cli invalid_utf8_is_rejected_cleanly" + status: pass + human_judgment: false + - id: D4 + description: "Export and import are CLI-only — no MCP tool and no REST route exist (D-01 prohibition, re-checked at phase end)" + requirement: DIST-04 + verification: + - kind: other + ref: "grep -rn 'memory_export\\|memory_import\\|/api/export\\|/api/import' crates/agent-memory/src/mcp.rs crates/agent-memory/src/rest/ → no output (exit 1)" + status: pass + human_judgment: false + - id: D5 + description: "Phase-level gates: fmt, clippy -D warnings, full workspace suite, llvm-cov ≥ 80% lines; sql/, Cargo.lock and .github/ unchanged across the phase" + verification: + - kind: other + ref: "cargo fmt --all -- --check; cargo clippy --workspace --all-targets --locked -- -D warnings; cargo test --workspace --locked → 279 passed / 0 failed / 2 ignored; cargo llvm-cov --workspace --fail-under-lines 80 → TOTAL 95.61% lines; git diff --stat 00d8080^..HEAD -- crates/agent-memory-core/sql/ Cargo.lock .github/ → empty" + status: pass + human_judgment: false + +# Metrics +duration: 9 min +completed: 2026-10-25 +status: complete +--- + +# Phase 5 Plan 04: Document the contract and prove it over the real binary Summary + +**The README now carries a complete "Export & import" section — format v1 field tables in wire order, the strict-reader and version rules, the per-column timestamp policy, the pending-embedding sweep, every count of the six-count report, the private atomic `--output`, the CLI-only rationale and a `VACUUM INTO` backup tip — and five spawned-binary realism tests prove the exact shipped `agent-memory` round-trips a memory file between two databases with that documented output and refuses malformed, mis-flagged and non-UTF-8 input before touching the target database.** + +## Performance + +- **Duration:** 9 min +- **Started:** 2026-10-25T12:25:28Z +- **Completed:** 2026-10-25T12:34:30Z +- **Tasks:** 2 +- **Files modified:** 2 (1 created, 1 modified; README +199 / −1, cli.rs 326 lines) + +## Accomplishments + +- **README `## Export & import` (D-01..D-13 for users).** Inserted between "Import from GSD" and "Lifecycle: TTL & decay", ten `###` subsections in the plan's order. *Usage* shows the three commands and states the CLI-only rule with both reasons (arbitrary-file-read primitive on the unauthenticated daemon; the corpus must not cross the JSON-RPC stdout channel) plus the `--scope` refusal. *What travels / what does not* names the four derived fields and why vectors are re-created locally. *Format v1* restates `export.rs`'s header (10 fields), memory (11) and link (5) tables in the exact `Record` declaration order, with the ordering/canonical-output paragraph (`relates_to` smaller key first, explicit `null`s, byte-identical re-exports). *The reader is strict* contrasts the tolerant `gsd-state` importer, lists every rejection class, shows the `invalid argument: line 42: …` shape and the `upgrade agent-memory` version policy. *Idempotent, and safe on a non-empty database* documents the `(source, type, content)` key, verbatim `source`, twin collapse and key remapping. *Timestamps* is the five-row per-column table plus the one-hour clock-skew warning. *Embeddings* states 128 per tick, first tick at `serve`/`serve-rest` start, then hourly, keyword-only until backfill. *The report* shows both output lines and a six-row meaning table (`links dropped` is "not an error"). *`--output` is private and atomic* names `PATH.export-tmp-`, `0600` at creation, sync + rename, replace-existing, never-partial, and the Windows seam. *Backup vs export* gives `sqlite3 memory.db "VACUUM INTO 'memory-backup.db'"` with the stop-every-process caveat and points at the pre-migration backup under Upgrading. +- **Cross-references.** Lifecycle item 3 ends with "Rows restored by `agent-memory import --from jsonl` take this path too."; Run gains `agent-memory export --output memories.jsonl # portable JSONL export (see Export & import)`. The "Import from GSD" section and the plan-01 redaction sentence are untouched. +- **`crates/agent-memory/tests/cli.rs` (realism-only, five tests, all green first run).** Helpers `binary_path`, `run` (dead-Ollama env, `Stdio::null()` stdin, `.output()`), `stdout`/`stderr` (lossy), `s` (path → `&str`), `header(rows, links)` and `seed_jsonl()` built with `serde_json::json!` (keys 10 and 20 so they can never coincide with target ids), `export_temp_entries`. `export_import_round_trip_over_the_real_binary`: import seed → A prints exactly `imported: 2, skipped duplicates: 0, skipped expired: 0, links imported: 1, links skipped duplicates: 0, links dropped: 0` then `2 rows pending embedding…`; `export --output out.jsonl` → stderr `exported 2 memories and 1 links`, 4 lines, line 1 starts `{"record":"header"`, Unix mode `& 0o777 == 0o600`, zero `.export-tmp-` siblings; stdout export lines 2–4 equal the file's; import into B → the same first line; import again → `imported: 0, skipped duplicates: 2, skipped expired: 0, links imported: 0, links skipped duplicates: 1, links dropped: 0` and `0 rows pending embedding…`. `a_malformed_file_exits_non_zero_naming_the_line_and_creates_no_database`: header + `{"record":"memo"}` → non-zero, stderr contains `line 2:`, `never.db` absent. `an_unsupported_from_value_still_bails` (`supported formats: gsd-state, jsonl`), `scope_is_rejected_for_jsonl` (`--scope applies only to --from gsd-state`), `invalid_utf8_is_rejected_cleanly` (`[0xff, 0xfe, 0x0a]` → stderr contains `reading`, not `PANIC`) — each also asserts the `--db` path was never created. +- **Phase-end gates (plan ``).** D-01 transport grep over `mcp.rs` and `rest/` is empty. `cargo llvm-cov --workspace --fail-under-lines 80` → **COV OK, TOTAL 95.61% lines** (regions 93.28%, functions 88.81%; `export.rs` 99.02%, `import/jsonl.rs` 98.57%, `export_file.rs` 93.55%). `git diff --stat 00d8080^..HEAD -- crates/agent-memory-core/sql/ Cargo.lock .github/` is empty across all eleven phase-5 commits. No new dependency; `--locked` throughout. + +## Task Commits + +Each task was committed atomically (fmt + clippy `-D warnings` + full workspace suite green before each): + +1. **Task 1: The README tells a user how to move their memory and what every line of the report means** — `5a15d13` (docs) +2. **Task 2: The shipped binary round-trips a memory file between two databases and refuses bad input cleanly** — `6eb354a` (test) + +**Plan metadata:** see the `docs(05-04)` commit that follows this summary. + +## Files Created/Modified + +- `README.md` — `## Export & import` (+~190 lines, l.548–741) with ten `###` subsections and five tables; Lifecycle item 3 sentence; Run example line +- `crates/agent-memory/tests/cli.rs` (new, 326 lines) — module doc (realism-only rationale, where coverage lives, dead-Ollama env), `FIRST_IMPORT` / `REIMPORT` constants, seven helpers, five `#[test]`s + +## Decisions Made + +- **Subsections, not one block.** The plan said "containing in this order"; the section is one `##` with ten `###` headings in exactly that order so each topic is linkable from issues and release notes and the three field tables stand on their own. +- **"Then hourly."** `SWEEP_INTERVAL` is `Duration::from_secs(3600)` in `main.rs`, so the README states the shipped cadence instead of "on the sweep interval". +- **The `--scope` refusal is documented in Usage** (one sentence) although it is not one of the plan's ten topics — it is the flag a `gsd-state` user would reach for, and the CLI test pins the message. +- **Extra "database never created" assertions** on the three refusal tests beyond the malformed-file one the plan required — `--from bogus` bails before any I/O, `--scope` bails before reading, and an unreadable file fails in `read_import_file`, all ahead of `open_service`; pinning that order is free. +- **Draft-outside-the-crate staging.** `cli.rs` was written to `/tmp` while Task 1's `cargo test --workspace` gate ran and copied into `tests/` only after `5a15d13`, so the Task 1 gate could not compile (or fail on) an uncommitted test target. + +## Deviations from Plan + +None - plan executed exactly as written. Both commits carry only the planned files; no code outside `README.md` and `tests/cli.rs` changed. + +### Process notes (not code deviations) + +- **TDD on an already-shipped behaviour.** Task 2 is `tdd="true"`, but it pins behaviour plans 05-02 and 05-03 already shipped, so a RED run cannot fail by construction — the five tests passed on their first run (`5 passed; 0 failed`), which is the expected outcome for a realism proof, not the "passes unexpectedly" fail-fast case (the feature is known to exist; the tests exist to pin its user-visible contract). The single `test(05-04)` commit is the plan's one-commit-per-task shape; `MVP_MODE=true` with `TDD_MODE=false` means the MVP+TDD RED-commit gate did not apply. +- **Pre-bash safety hook** never tripped: logs were truncated with `: > file`, databases lived in `tempfile::tempdir()` and were never removed by the shell. + +**Total deviations:** 0. +**Impact on plan:** None. + +## Issues Encountered + +None. Every grep in both tasks' acceptance criteria passed on the first check; both gates were green on the first run (274 → 279 tests). + +## Verification + +Plan-level `` re-run after the second commit: + +- `cargo test -p agent-memory --test cli` — 5 passed (`export_import_round_trip_over_the_real_binary`, `a_malformed_file_exits_non_zero_naming_the_line_and_creates_no_database`, `an_unsupported_from_value_still_bails`, `scope_is_rejected_for_jsonl`, `invalid_utf8_is_rejected_cleanly`). +- README: `grep -c '^## Export & import'` → 1, between `## Import from GSD` (l.527) and `## Lifecycle: TTL & decay` (l.742); `exported_rows` 2, `skipped expired` 2, `last_accessed` 4, `128 per tick` 2, `no MCP tool` 1, `VACUUM INTO` 1, `0600` ≥ 1, `export-tmp-` ≥ 1, `links dropped` ≥ 1, `upgrade agent-memory` ≥ 1; every `Record` field name present; Lifecycle `import --from jsonl` → 1; Run `agent-memory export` → 1. +- `cargo fmt --all -- --check` clean; `cargo clippy --workspace --all-targets --locked -- -D warnings` exit 0; `cargo test --workspace --locked` — **279 passed, 0 failed, 2 ignored** (274 after 05-03 + 5 CLI tests). +- `cargo llvm-cov --workspace --fail-under-lines 80` — **COV OK**, TOTAL 95.61% lines / 93.28% regions / 88.81% functions. +- D-01: `grep -rn "memory_export\|memory_import\|/api/export\|/api/import" crates/agent-memory/src/mcp.rs crates/agent-memory/src/rest/` → empty. +- `git diff --stat 00d8080^..HEAD -- crates/agent-memory-core/sql/ Cargo.lock .github/` → empty; `git diff --diff-filter=D` on both task commits → no deletions. +- `grep -c '#\[test\]' crates/agent-memory/tests/cli.rs` → 5; `CARGO_BIN_EXE_agent-memory` → 1; `"--from", "jsonl"` → 5. + +## Known Stubs + +None — no placeholder values, TODO/FIXME markers, skipped tests, or unwired data paths were introduced. + +## Threat Flags + +None new. The plan's register is implemented as mitigated or accepted as recorded: T-05-11 (CLI-only — README states the rule and both reasons; the transport grep is empty), T-05-18 (every count defined in the README table, `links dropped` explicitly "not an error"; the CLI test pins the exact wording), T-05-12 (README shows `--output` as the private alternative and warns that stdout may land in scrollback), T-05-10 (validate-before-write documented; no cap added), T-05-SC (zero new dependencies, `--locked`). + +## User Setup Required + +None - no external service configuration required. + +## Next Phase Readiness + +- **Phase 5 is fully executed (4/4 plans).** The orchestrator runs phase verification next; `human_verify_mode: end-of-phase` means the three tracer gates (05-01, 05-02, 05-03) and D1 above route to end-of-phase UAT together. The phase is not marked complete here. +- **For Phase 6 (release):** the README section is the user-facing changelog source for DIST-04 / INTEROP-02; the "Behavioral changes in v0.1.0" list may want one line pointing at it. The Windows `--output` permission sentence ("follows its directory's permissions") is the one README statement Phase 6's DIST-03 work must revisit. +- **Cheap follow-up carried from 05-02/05-03:** adding `export` and `import_jsonl` to `tests/redaction.rs` remains open. +- No blockers. `.planning/config.json` and the untracked harness files were left unstaged as instructed. + +--- +*Phase: 05-portable-export-import* +*Completed: 2026-10-25* + +## Self-Check: PASSED + +Both key files and the SUMMARY exist on disk; task commits 5a15d13 and 6eb354a are present in git history; the D-01 transport grep is empty; the llvm-cov gate passed at 95.61% lines. diff --git a/.planning/phases/05-portable-export-import/05-05-PLAN.md b/.planning/phases/05-portable-export-import/05-05-PLAN.md new file mode 100644 index 0000000..d48b062 --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-05-PLAN.md @@ -0,0 +1,271 @@ +--- +phase: 05-portable-export-import +plan: 05 +type: execute +wave: 5 +depends_on: [05-02, 05-03] +files_modified: + - crates/agent-memory-core/tests/export_import.rs + - .planning/phases/05-portable-export-import/05-VALIDATION.md +autonomous: true +gap_closure: true +requirements: [DIST-04, INTEROP-02] + +estimate: + tokens: 9000 + raw_tokens: 9000 + tasks: 2 + confidence: low + +must_haves: + truths: + - "Memories and links are read on ONE pooled connection inside ONE read transaction, so a memory and a link committed by a concurrent writer while `Store::export_rows` is mid-stream are invisible to that export — the raced output parses with `parse_jsonl`, its header `exported_rows`/`exported_links` equal the emitted record counts, and no link key is absent from the file — and both appear in the next export. Proven by `a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key` in `crates/agent-memory-core/tests/export_import.rs`; this restates the 05-02 backstop truth as explicit evidence (DIST-04, D-06, T-05-19)" + - "The concurrent writer COMMITS while the export snapshot is still open: the write never blocks behind the reader (WAL + busy_timeout on every connection, writer mutex vs read pool), and a fresh connection already sees the new row and edge before the export has finished streaming — the test fails by bounded timeout, never by hanging, if either side blocks" + - "On this host with Ollama + nomic-embed-text live, a corpus exported from database A and imported into database B lands pending (`imported: 3, …` six-count line and `3 rows pending embedding …`), the first `serve-rest` sweep tick backfills it (`embedded pending memories on sweep` with `backfilled=3`), and `POST /api/search` with a paraphrase of one memory answers `\"search_mode\":\"hybrid\"` with that imported memory in `results` — captured verbatim in 05-05-SUMMARY.md under `## Live Ollama realism evidence` and in 05-VALIDATION.md (INTEROP-02, D-13)" + artifacts: + - path: "crates/agent-memory-core/tests/export_import.rs" + provides: "held-out concurrency proof for the one-snapshot export; Harness exposes `store: Arc` so the writer thread and the raced `export_rows` call share one store" + contains: "fn a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key" + - path: ".planning/phases/05-portable-export-import/05-05-realism.sh" + provides: "the executable live-Ollama realism flow (precondition probe, build, gsd-state seed of A, export, jsonl import into B, serve-rest with ephemeral port, bounded wait for the first sweep backfill, /health, hybrid paraphrase search, cleanup, evidence block)" + contains: "embedded pending memories on sweep" + - path: ".planning/phases/05-portable-export-import/05-VALIDATION.md" + provides: "05-05-T1 / 05-05-T2 rows in the per-task map; the concurrency Manual-Only row retired in favour of the automated test; the live-Ollama realism row filled with the run's status and evidence pointer" + contains: "05-05-T1" + key_links: + - from: "crates/agent-memory-core/tests/export_import.rs" + to: "crates/agent-memory-core/src/store/sqlite.rs" + via: "the test drives `Store::export_rows` directly with a visitor closure that pauses on the first memory record" + pattern: "export_rows\\(" + - from: "crates/agent-memory-core/tests/export_import.rs" + to: "crates/agent-memory-core/src/import/jsonl.rs" + via: "the raced output (header mirrored from the snapshot + one line per record) is accepted by the strict reader" + pattern: "parse_jsonl\\(" + - from: "crates/agent-memory-core/tests/export_import.rs" + to: "crates/agent-memory-core/src/store/sqlite.rs" + via: "a std::thread writer holding an `Arc` calls `Store::insert` then `Store::link` while the exporter's read transaction is open; ordering by mpsc channels with `recv_timeout` on both sides" + pattern: "thread::spawn" + - from: ".planning/phases/05-portable-export-import/05-05-realism.sh" + to: "crates/agent-memory/src/main.rs" + via: "the real binary: `import --from gsd-state`, `export --output`, `import --from jsonl`, `serve-rest --addr 127.0.0.1:0` (port parsed from the `REST listening on` stderr line), then `POST /api/search`" + pattern: "REST listening on" + prohibitions: + - statement: "No production file changes in this plan — `crates/agent-memory-core/src/` and `crates/agent-memory/src/` stay byte-identical; the gap is closed with evidence, not with code changes" + status: resolved + verification: test + - statement: "The race test must not let a timed wait decide ordering — both hand-offs are channel-driven and every wait is bounded by `recv_timeout`" + status: resolved + verification: test + - statement: "No new crate, no CI/workflow change, no committed database file — `Cargo.lock` and `.github/` unchanged; the realism run's databases live in a `mktemp -d` directory that is never removed by the script" + status: resolved + verification: test +--- + +# Plan 05-05 (gap closure): explicit evidence for the one-snapshot export and the live re-embed after a cross-database move + +## Phase Goal + +**As a** user, **I want to** export all my memories to a portable file and import them on another machine, **so that** my agent's memory moves with me — across machines, with no cloud sync. + +This plan closes the two `human_verification` items in 05-VERIFICATION.md (status `human_needed`, 32/33) so the re-verifier can mark the phase `passed` without a human: (1) the 05-02 backstop truth about one WAL snapshot gets a held-out race test, and (2) the "on the other machine" step of the user story is actually run against the live Ollama on this host and recorded. + + +1. **Gap 1 (automatable):** add `a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key` to `crates/agent-memory-core/tests/export_import.rs`. A writer thread stores a NEW memory and links it to an existing one while `Store::export_rows` is paused inside its visitor on the first memory record; the test proves the write commits (visible to a fresh connection) while the export's snapshot does not see it, that the raced file is accepted by `parse_jsonl` with header counts equal to record counts, and that the next export contains both the new memory and the new edge. +2. **Gap 2 (live-Ollama realism):** run `.planning/phases/05-portable-export-import/05-05-realism.sh` — the real binary moves a three-memory corpus from database A to database B through a JSONL file, `serve-rest` on B backfills the pending rows on its first sweep tick against the local `nomic-embed-text`, and a paraphrase search returns `search_mode: "hybrid"` with the imported memory — and record every command and captured output in the SUMMARY and in 05-VALIDATION.md. + +Purpose: the verifier's honest-evidence rule does not accept "the `unchecked_transaction` is present and wired" for a backstop truth, and MVP mode routes the real cross-machine/live-embedder flow to a human. Both items are convertible into explicit, re-runnable evidence on this host. +Output: one new test (plus a `store` field on the test harness), one recorded realism run, 05-VALIDATION.md rows filled, 05-05-SUMMARY.md with the evidence block. No production code changes. + + + +@$HOME/.claude/gsd-core/workflows/execute-plan.md +@$HOME/.claude/gsd-core/templates/summary.md + + + +@.planning/STATE.md +@.planning/phases/05-portable-export-import/05-CONTEXT.md +@.planning/phases/05-portable-export-import/05-VERIFICATION.md +@.planning/phases/05-portable-export-import/05-02-SUMMARY.md +@.planning/phases/05-portable-export-import/05-03-SUMMARY.md +@crates/agent-memory-core/tests/export_import.rs +@crates/agent-memory-core/src/store/sqlite.rs + + +## Execution notes for every task + +- Gate before EVERY commit: `cargo fmt --all`, `cargo clippy --workspace --all-targets --locked -- -D warnings`, `cargo test --workspace --locked` — run clippy and the full suite in the background with a log file and poll (watchdog ~600 s). Baseline at `f3d1568`: 286 passed / 0 failed / 2 ignored (05-REVIEW fixes applied); `--test export_import` runs 19 tests. +- One commit per task; stage explicit paths only. NEVER stage `CLAUDE.md`, `.claude/`, `.mcp.json`, `.gitignore`, `.gsd/`, `docs/`, `10-agent-memory.md`, `claude-code-harness-engineering-guide-v2.md`, `.planning/research/.cache/`, `.planning/config.json`, `.planning/milestone.lock`. +- No production code changes. `unwrap()`/`expect()` are fine inside the test file (tests may). No new crate; `Cargo.lock` unchanged under `--locked`. Never change the CI runner or add a workflow. Never push, tag or release. +- The repo's `.claude/hooks/pre-bash-rust.sh` blocks shell text matching `rm -[rfRF]+ … .db` and `drop table`: never remove the temp databases from the shell, never spell a DROP statement — the realism script leaves its `mktemp -d` directory in place on purpose. +- This host has no `timeout`/`gtimeout` binary (checked at planning time): every wait in the realism script is a counter-bounded loop; the race test bounds its waits with `std::sync::mpsc::Receiver::recv_timeout`. +- Leave `05-02-PLAN.md` untouched (its `verification: backstop` marker stays); this plan's `must_haves.truths` restates the truth as a plain string with the new test as evidence and the re-verifier reconciles. +- The planner smoke-ran `05-05-realism.sh` once on 2026-10-25 before committing it (not recorded as phase evidence — Task 2 must produce its own run): exit 0, `backfilled=3` logged ~50 ms after `REST listening on`, `/health` `"embedder":"ready"`, `"search_mode":"hybrid"`, paraphrase target at rank 0 (id 1 in B). Expect the whole flow to take well under a minute once the binary is built. + +## Interface contracts (what already exists — nothing new is introduced) + +```text +// crates/agent-memory-core/src/store/mod.rs (pub trait Store, line 21) +fn export_rows(&self, visit: &mut dyn FnMut(ExportEvent) -> Result<(), MemoryError>) -> Result<(), MemoryError>; +fn insert(&self, new: NewMemory, embedding: Option>, now: i64) -> Result; +fn link(&self, from_id: i64, to_id: i64, kind: LinkKind, now: i64) -> Result; + +// crates/agent-memory-core/src/export.rs +pub enum ExportEvent { Snapshot(ExportSnapshot), Record(Record) } +pub struct ExportSnapshot { pub embedding_model: String, pub embedding_dim: u32, pub memories: u64, pub links: u64 } +pub enum Record { Header { format, version, exported_at, embedding_model, embedding_dim, schema_version, + agent_memory_version, exported_rows, exported_links }, + Memory { key, mem_type, content, tags, source, scope, created_at, last_accessed, access_count, expires_at }, + Link { from, to, kind, created_at } } + +// crates/agent-memory-core/src/import/jsonl.rs +pub fn parse_jsonl(text: &str) -> Result; // JsonlFile { header, memories: Vec, links: Vec } + +// crates/agent-memory-core/src/store/sqlite.rs +// export_rows: self.reads.get() → conn.unchecked_transaction() → meta pin → count(*) ×2 → Snapshot +// → SELECT … FROM memories ORDER BY id (visit per row) → SELECT … FROM memory_links ORDER BY … → tx.commit() +// apply_pragmas (every connection): busy_timeout 5000, journal_mode WAL, synchronous NORMAL, foreign_keys ON +// SqliteStore { writer: Mutex, reads: Pool } + +// crates/agent-memory/src/main.rs (real binary, stderr logging at `info` by default) +// serve-rest --addr 127.0.0.1:0 → "REST listening on 127.0.0.1:" logged AFTER bind +// spawn_sweep_task: tokio::time::interval(SWEEP_INTERVAL) — the FIRST tick fires immediately +// sweep log lines: "lifecycle sweep complete expired=.. rescored=.." then +// "embedded pending memories on sweep backfilled= skipped_stale=" (service.rs ~1112) +// health probe log: "semantic search ready (nomic-embed-text)" +// GET /health → {"status":"ok","embedder":"ready"|"model_missing"|"unreachable"} +// POST /api/search {"query": "...", "limit": n} → {"search_mode":"hybrid"|"keyword","results":[MemoryView{id, content, ...}]} +``` + +## Artifacts this phase produces (this plan's share) + +- `crates/agent-memory-core/tests/export_import.rs`: `Harness` gains `store: Arc`; new test fn `a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key`; new imports `agent_memory_core::store::Store`, `std::sync::mpsc`, `std::time::Duration`, `agent_memory_core::export::{ExportEvent, ExportSnapshot}`. +- `.planning/phases/05-portable-export-import/05-05-realism.sh` (committed with this plan by the planner; executed by Task 2). +- `.planning/phases/05-portable-export-import/05-VALIDATION.md`: rows `05-05-T1`, `05-05-T2`; Manual-Only table updated. +- `.planning/phases/05-portable-export-import/05-05-SUMMARY.md` with a `## Live Ollama realism evidence` section. + + + + + Task 1: Held-out race test — a memory and link committed mid-export are invisible to that snapshot and present in the next + crates/agent-memory-core/tests/export_import.rs + + - crates/agent-memory-core/src/store/sqlite.rs lines 832-924 (`export_rows`: `self.reads.get()`, `conn.unchecked_transaction()`, meta pin, both counts, memories scan, links scan, `tx.commit()`; the T-05-19 / PITFALLS 5 comment is the claim under test) and lines 83-96 (`apply_pragmas`: `busy_timeout 5000` + WAL on every connection) and 347-351 (`writer: Mutex`, `reads: Pool`). + - crates/agent-memory-core/src/store/mod.rs lines 120-145 (`export_rows` trait doc — "The header counts come from the same snapshot, so they always equal the records that follow"), line 28 (`insert`), line 270 (`link`). + - crates/agent-memory-core/tests/export_import.rs lines 33-128 (imports, `Harness`, `harness_with`, `new_memory`, `store`, `link`, `export_to_string`, `lines`), 425-445 (`raw_conn`, `row_count`, `edge_count`), 549-557 (`import` — the `parse_jsonl` call shape), 605-648 (`seed_round_trip_source` — the 3-memory / 2-link fixture to reuse; returns `((a, b, c), (supersedes_at, relates_at))`). + - crates/agent-memory-core/tests/backfill_race.rs lines 1-30 (module doc: ordering decided by signals, never by timing) and 120-150 (a `Harness` that keeps `store: Arc` beside the service — the field shape to add here). + - crates/agent-memory-core/src/service.rs lines 746-790 (`export_unredacted`: how `Record::Header` is built from `ExportSnapshot` — `format: FORMAT_NAME`, `version: FORMAT_VERSION`, `exported_at: now`, `schema_version: LATEST_SCHEMA_VERSION`, `exported_rows: snapshot.memories`, `exported_links: snapshot.links`; the test mirrors this to build the raced file's header). + - .planning/phases/05-portable-export-import/05-VERIFICATION.md truth 2.8 and "Human Verification Required" item 1 (the verifier names exactly this test as sufficient). + + + New `#[tokio::test] async fn a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key()` in `tests/export_import.rs`: + - Seed with `seed_round_trip_source(&h)` → ids `(a, b, c)`, 3 memories, 2 links; `row_count == 3`, `edge_count == 2`. + - Two `std::sync::mpsc` channels: `go` (exporter → writer, unit) and `done` (writer → exporter, the new row id). A writer `std::thread::spawn` owning `h.store.clone()` and `h.clock.now()`: waits on `go_rx.recv_timeout(Duration::from_secs(10))` (panic message: the exporter never reached its first memory record), then `store.insert(new_memory("delta arrived mid-export", MemoryType::Pattern), None, now)` → `new_id`, then `store.link(new_id, a, LinkKind::CausedBy, now)` → `LinkOutcome::Linked { created: true }` (panic message if the write errs: a concurrent write must not block behind the open read snapshot), then `done_tx.send(new_id)`. + - Exporter: `h.store.export_rows(&mut |event| { … })` on the test thread, collecting `ExportEvent::Snapshot` into an `Option` and every `Record` into a `Vec`. On the FIRST `Record::Memory` only (a `fired` flag): `go_tx.send(())`, then `new_id = done_rx.recv_timeout(Duration::from_secs(10))` (panic message: the writer did not commit within 10 s while the export snapshot was open — a timeout here means the write blocked, i.e. a deadlock, which must fail the test rather than hang it), then assert `row_count(&h.db_path) == 4` and `edge_count(&h.db_path) == 3` — a FRESH raw connection already sees the commit while this export's read transaction is still open. The closure returns `Ok(())`; `export_rows` must return `Ok`. + - After `export_rows`: snapshot is `Some` with `memories == 3`, `links == 2`; exactly 3 `Record::Memory` and 2 `Record::Link` were visited; no memory record has `key == new_id`; no link record has `from == new_id || to == new_id` (snapshot isolation — the writer's commit is invisible to this export). Build the raced file text: a `Record::Header` mirrored from the snapshot (`FORMAT_NAME`, `FORMAT_VERSION`, `exported_at: h.clock.now()`, `embedding_model`/`embedding_dim` from the snapshot, `LATEST_SCHEMA_VERSION`, `TEST_VERSION`, `exported_rows: snapshot.memories`, `exported_links: snapshot.links`) followed by every visited record, one `serde_json::to_string` line each, newline-terminated. `parse_jsonl(&text)` is `Ok` (no `does not appear in the file`, no count mismatch) and the parsed file has `memories.len() == 3`, `links.len() == 2`. `writer.join()` is `Ok`. + - Second export through the service: `export_to_string(&h, "after-race.jsonl")` → `ExportReport { memories: 4, links: 3 }`; `lines(&text)` contains `Record::Memory { key: new_id, content: "delta arrived mid-export", .. }` and `Record::Link { from: new_id, to: a, kind: "caused_by", .. }`; `parse_jsonl` of that text is `Ok` with `memories.len() == 4`, `links.len() == 3` (the writer's commit is visible to the NEXT snapshot). + - RED first: the test references `h.store`, which does not exist on `Harness` yet, so it fails to compile against the current tree; after the harness change it goes GREEN on first run (the behaviour already ships — this is a held-out proof, the same situation as 05-04 Task 2). + + + Closes 05-VERIFICATION.md Human Verification item 1 (truth 2.8, `insufficient_spec`) with the exact test the verifier named. The claim under test is D-06 / D-02 as implemented by 05-02: every row streams from ONE read transaction on ONE pooled connection, so the header counts (D-03 `exported_rows`/`exported_links`) and the link records (D-05) can never straddle a concurrent write (T-05-19, PITFALLS 5). + + 1. **Harness.** Add `store: Arc` to `struct Harness`. In `harness_with`, build `let store = Arc::new(SqliteStore::open(&db_path).expect("open store"));`, pass `store.clone()` as the first argument of `MemoryService::new`, and keep `store` in the struct literal. Nothing else in the file changes (all existing tests use `h.service`, `h.clock`, `h.db_path`, `h.dir`). Add the imports `use agent_memory_core::store::Store;` (trait methods on the `Arc`), `use agent_memory_core::export::{ExportEvent, ExportSnapshot};` (extend the existing `export::{Record, FORMAT_NAME, FORMAT_VERSION}` line), `use std::sync::mpsc;`, `use std::time::Duration;`. Mention in the module doc's "Harness" paragraph that the store is also exposed so the concurrency proof can drive `Store::export_rows` and a writer thread against one store. + 2. **Test.** Append the test from `` at the end of the export half (after `two_exports_of_an_unchanged_database_are_byte_identical`, before the import-half helpers), under a comment banner "Export half — gap closure 05-05: one-snapshot concurrency proof". Doc-comment it: this is the held-out evidence for the 05-02 backstop truth; the writer thread is a real second connection lane (the writer mutex) racing the exporter's read-pool connection, exactly the MCP-server-plus-REST-daemon situation; ordering is decided only by the two channel hand-offs; every wait is bounded by `recv_timeout` so a blocked write (deadlock) fails fast instead of hanging; WAL snapshot isolation is what makes the committed row invisible to the open export while a fresh connection sees it. + Structure the visitor as a single `&mut |event: ExportEvent| -> Result<(), MemoryError>` closure capturing `fired: bool`, `snapshot: Option`, `records: Vec`, `go_tx`, `done_rx`, `new_id_seen: Option` (or a `std::cell::Cell`/plain locals — the closure is `FnMut`, so plain mutable captures work), `&h.db_path`. Match exhaustively on `ExportEvent::Snapshot(s)` / `ExportEvent::Record(r)`; inside the `Record` arm detect `Record::Memory { .. }` with `matches!`. Do not add a `_ =>` arm. + Build the raced file with a small local `fn record_lines(header: Record, records: &[Record]) -> String` (or inline iterator) — `serde_json::to_string(rec).expect("record serialises")` + `'\n'` per record, header first. Use `parse_jsonl(&text).expect("the raced export must parse as format v1 — a dangling link key or a count mismatch here means two snapshots were straddled")`. + The writer thread's `now` is `h.clock.now()` read on the test thread before spawning (the `TestClock` is `Send + Sync`, but a plain `i64` is simpler). Use `LinkKind::CausedBy` so the stored edge is directed `(new_id, a)` exactly as given (D-05) and the `from == new_id` assertion is literal. + 3. **Timing discipline.** No timed waits anywhere in the test: the exporter blocks on `done_rx.recv_timeout`, the writer blocks on `go_rx.recv_timeout`; the 10 s bounds are generous for a local SQLite write and only exist so a regression fails with a message. Do not spawn the writer with tokio — `Store` methods are synchronous and `#[tokio::test]` is a current-thread runtime, so a `std::thread` keeps the writer independent of the runtime the exporter is blocking. + 4. Run `cargo test -p agent-memory-core --test export_import a_concurrent_link_committed_mid_export` until green, then the whole file (20 tests), then five consecutive runs of the new test to prove it is deterministic; `cargo fmt --all`; clippy; full suite (background + poll). Commit with the explicit path `crates/agent-memory-core/tests/export_import.rs`: `test(05-05): held-out race proof — a link committed mid-export is invisible to that snapshot and present in the next (DIST-04 backstop truth)`. + + A test-only change; removing it restores the prior tree byte-for-byte. No production code, no schema, no dependency is touched. + + cargo test -p agent-memory-core --test export_import a_concurrent_link_committed_mid_export && ok=1 && for i in 1 2 3 4 5; do cargo test -q -p agent-memory-core --test export_import a_concurrent_link_committed_mid_export >/dev/null 2>&1 || ok=0; done && test "$ok" = 1 && echo stable-5x && cargo test -p agent-memory-core --test export_import 2>&1 | grep -c 'test result: ok. 20 passed' && cargo clippy --workspace --all-targets --locked -- -D warnings && git diff --stat HEAD -- crates/agent-memory-core/src crates/agent-memory/src | wc -l | tr -d ' ' + + + - `grep -c "fn a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key" crates/agent-memory-core/tests/export_import.rs` outputs 1. + - `grep -c "store: Arc" crates/agent-memory-core/tests/export_import.rs` outputs 1; `grep -c "use agent_memory_core::store::Store;" crates/agent-memory-core/tests/export_import.rs` outputs 1. + - `grep -c "recv_timeout" crates/agent-memory-core/tests/export_import.rs` outputs at least 2 (both hand-offs bounded); `grep -c "thread::spawn" crates/agent-memory-core/tests/export_import.rs` outputs at least 1; `grep -c "export_rows(" crates/agent-memory-core/tests/export_import.rs` outputs at least 1; `grep -c "parse_jsonl(" crates/agent-memory-core/tests/export_import.rs` outputs at least 3 (the `import` helper plus the two new calls). + - `grep -v '^\s*//' crates/agent-memory-core/tests/export_import.rs | grep -c "thread::sleep"` outputs 0 (ordering is channel-driven, never timed). + - `cargo test -p agent-memory-core --test export_import` reports `20 passed` (19 before this task + 1); the new test passes 5 consecutive runs. + - `git diff --stat HEAD -- crates/agent-memory-core/src crates/agent-memory/src` is empty (no production change); `git diff --stat HEAD -- Cargo.lock .github/ crates/agent-memory-core/sql/` is empty. + - `cargo fmt --all -- --check` clean; `cargo clippy --workspace --all-targets --locked -- -D warnings` exit 0; `cargo test --workspace --locked` green (287 passed, 0 failed, 2 ignored expected). + + The 05-02 backstop truth has explicit, deterministic evidence: a writer committing a memory and a link while the export streams never reaches that export's file (which still parses strictly with exact header counts), and does reach the next one — so the re-verifier can mark truth 2.8 VERIFIED from a test rather than from presence + wiring. + + + + Task 2: Live-Ollama realism run — export from A, import into B, first-sweep backfill, hybrid paraphrase search (recorded evidence) + .planning/phases/05-portable-export-import/05-VALIDATION.md + + - .planning/phases/05-portable-export-import/05-05-realism.sh (whole file — the exact flow this task runs; read it before running it so the SUMMARY can explain each check). + - crates/agent-memory/src/main.rs lines 36-60 (`--db` global flag / `AGENT_MEMORY_DB`; `--ollama-url` / `AGENT_MEMORY_OLLAMA_URL` default `http://localhost:11434`), 116-130 (tracing to stderr, `info` when `RUST_LOG` is unset), 190-240 (`serve_rest`: health probe, sweep task, `REST listening on {addr}` logged after bind so `--addr 127.0.0.1:0` reports the real ephemeral port), 448-470 (`spawn_sweep_task`: `tokio::time::interval` — the first tick fires immediately — logging `lifecycle sweep complete`). + - crates/agent-memory-core/src/service.rs lines 1065-1130 (`sweep_unredacted`: `pending_embeddings(BACKFILL_BATCH)` → one batch embed → guarded `insert_embedding` → `tracing::info!(backfilled = written, skipped_stale = skipped, "embedded pending memories on sweep")`). + - crates/agent-memory-core/src/import/gsd_state.rs lines 95-105 (`### Decisions` bullets become `MemoryType::Decision`) — the seed of database A. + - crates/agent-memory/src/rest/mod.rs lines 71-86 (routes) and crates/agent-memory/src/rest/handlers.rs `SearchRequest` (`query`, optional `limit`) and `health_handler` (`{"status":"ok","embedder":"ready"}` when Ollama + model are up). + - crates/agent-memory/tests/cli.rs lines 20-30 (the exact six-count and re-import line formats) and README.md lines 685-705 (sweep cadence sentence + the pending line the import prints). + - .planning/phases/05-portable-export-import/05-VERIFICATION.md "Human Verification Required" item 2 (what must be shown) and 05-VALIDATION.md "Manual-Only Verifications" (the rows this task fills). + + + Closes 05-VERIFICATION.md Human Verification item 2 by performing the "on the other machine" step of the user story against the live embedder on this host and recording it. This is NOT a cargo test and must not become one (Ollama is not available in CI; D-13's proof `imported_rows_land_pending_and_the_sweep_backfills_them` already covers the mechanism with a fake embedder) — it is realism evidence for INTEROP-02 criterion 3 with the real `nomic-embed-text`. + + 1. **Precondition.** `curl -s --max-time 5 http://localhost:11434/api/tags | grep -c '"name":"nomic-embed-text'` must print at least 1 (at planning time: Ollama 0.35.0, `nomic-embed-text:latest` present, `/api/embeddings` answered). If Ollama is down or the model is missing, STOP and report `## CHECKPOINT REACHED` naming the missing precondition — never fabricate the evidence and never substitute a fake embedder here. + 2. **Run the flow.** From the repo root: `bash .planning/phases/05-portable-export-import/05-05-realism.sh`. Run it in the background with a log file and poll (it builds the binary with `cargo build -p agent-memory --locked`, so the first run can take a minute; every wait inside is counter-bounded — 30 s for `REST listening on`, 120 s for the first sweep backfill — and the whole flow finishes well inside the ~600 s watchdog). The script: writes a three-bullet `### Decisions` fixture; imports it into `$T/a.db` with `--from gsd-state --scope realism`; exports `$T/a.db` to `$T/move.jsonl` (expects `exported 3 memories and 0 links`, 4 lines, `"exported_rows":3` in the header, D-03/D-12); imports the file into `$T/b.db` with `--from jsonl` (expects the exact six-count line `imported: 3, skipped duplicates: 0, skipped expired: 0, links imported: 0, links skipped duplicates: 0, links dropped: 0` and `3 rows pending embedding …`, D-12/D-13); starts `serve-rest --db $T/b.db --addr 127.0.0.1:0` with stderr to `$T/daemon.log`, parses the port from the `REST listening on` line, waits for `embedded pending memories on sweep` and asserts `backfilled=3`; checks `GET /health` reports `"embedder":"ready"`; posts `{"query":"which database did we choose so that nothing needs a hosted service","limit":3}` to `/api/search`, asserts `"search_mode":"hybrid"` and that the SQLite/storage-engine memory (the paraphrase target) is in `results`, recording its rank and id; kills the daemon via a trap-protected `kill`; prints and saves an evidence block to `$T/EVIDENCE.md`. The temp directory is deliberately kept (the pre-bash hook forbids removing database files from the shell). + If the harness refuses the Bash command, run `bash .planning/phases/05-portable-export-import/05-05-realism.sh > /tmp/05-05-realism.log 2>&1` in the background and read the log; do not edit the script's checks to make them pass — a failing check is a finding to report, not to paper over. + 3. **Record the evidence.** In `05-05-SUMMARY.md` add a section `## Live Ollama realism evidence` containing the script's evidence block verbatim: host + Ollama version + model tag line, each command, the A-import line, the export stderr line, the header line of `move.jsonl`, the B-import six-count line and pending line, the daemon log lines (`semantic search ready (nomic-embed-text)`, `REST listening on 127.0.0.1:`, `lifecycle sweep complete …`, `embedded pending memories on sweep backfilled=3 skipped_stale=0`), the `/health` body, the full `search.json` response, and the sentence "paraphrase target rank (0 = first): N; imported memory id in b.db: M". The expected rank is 0; if it is not 0, record the actual rank and the three result contents verbatim — the phase truth is `search_mode: "hybrid"` + the imported memory present (INTEROP-02 criterion 3), so a non-zero rank is recorded, not hidden. + 4. **05-VALIDATION.md.** Set the `05-05-T2` row's Status cell to ✅ green and append the run date; in "Manual-Only Verifications", fill the live-Ollama row's "Test Instructions" cell's trailing `Result:` with `PASS — search_mode hybrid, target rank N, id M; evidence: 05-05-SUMMARY.md § Live Ollama realism evidence`. Also set `05-05-T1`'s Status cell to ✅ (Task 1 has run by now). Use the Edit tool with scoped replacements; never rewrite the file. + 5. Commit with the explicit path `.planning/phases/05-portable-export-import/05-VALIDATION.md`: `docs(05-05): live-Ollama realism run recorded — export → import → sweep backfill → hybrid paraphrase search (INTEROP-02)`. (The SUMMARY is committed by the standard `docs(05-05)` metadata commit that follows.) + + Nothing in the product changes; the run writes only to a temp directory and to planning docs. + + grep -c '^## Live Ollama realism evidence' .planning/phases/05-portable-export-import/05-05-SUMMARY.md && grep -c '"search_mode":"hybrid"' .planning/phases/05-portable-export-import/05-05-SUMMARY.md && grep -c 'backfilled=3' .planning/phases/05-portable-export-import/05-05-SUMMARY.md && grep -c 'Result: PASS' .planning/phases/05-portable-export-import/05-VALIDATION.md + The live script runs ONCE, in the action (step 2); verify checks the recorded artefacts instead of re-running the build + live embed a second time (plan-checker WR-2: keeps feedback latency under 30 s and avoids a second kept temp directory). + + + - The script exits 0 and its last line starts with `PASS: live Ollama realism run`; its evidence block contains `"search_mode":"hybrid"`, `backfilled=3`, `REST listening on 127.0.0.1:`, `"embedder":"ready"`, `exported 3 memories and 0 links`, and the exact line `imported: 3, skipped duplicates: 0, skipped expired: 0, links imported: 0, links skipped duplicates: 0, links dropped: 0`. + - `grep -c '^## Live Ollama realism evidence' .planning/phases/05-portable-export-import/05-05-SUMMARY.md` outputs 1; `grep -c '"search_mode":"hybrid"' .planning/phases/05-portable-export-import/05-05-SUMMARY.md` outputs at least 1; `grep -c 'backfilled=3' .planning/phases/05-portable-export-import/05-05-SUMMARY.md` outputs at least 1; `grep -c '^imported: 3, skipped duplicates: 0, skipped expired: 0, links imported: 0, links skipped duplicates: 0, links dropped: 0' .planning/phases/05-portable-export-import/05-05-SUMMARY.md` outputs at least 1. + - `grep -c '^| 05-05-T[12] .*⬜ pending' .planning/phases/05-portable-export-import/05-VALIDATION.md` outputs 0 (both Status cells flipped off pending) and `grep -c '^| 05-05-T2 .*| ✅ [^|]*2026-[0-9-]* |$' .planning/phases/05-portable-export-import/05-VALIDATION.md` outputs 1; `grep -c 'Result: PASS' .planning/phases/05-portable-export-import/05-VALIDATION.md` outputs at least 1. + - `git diff --stat HEAD -- crates/ Cargo.lock .github/` is empty after this task (evidence only, no code). + - No database file and no `EVIDENCE.md` is staged: `git status --short | grep -c '\.db\|EVIDENCE'` outputs 0. + + The user-story step "import on the other machine, start the server, search with a paraphrase" has been performed with the real binary against the live local `nomic-embed-text`, the first sweep tick re-embedded all three imported rows, the search answered `hybrid` with the imported memory, and every command and output is recorded where the re-verifier reads it. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| exporter read-pool connection ↔ concurrent writer (writer mutex) | the test deliberately drives a write across the open read snapshot — the boundary the 05-02 backstop truth is about | +| realism run → local Ollama (`http://localhost:11434`) | synthetic fixture text leaves the process to the local embedder, exactly as in normal use | +| realism run → loopback REST daemon on an ephemeral port | an unauthenticated daemon is bound for ~1 minute on `127.0.0.1` over a throwaway database | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan | +|-----------|----------|-----------|----------|-------------|-----------------| +| T-05-19 | Tampering (integrity) | `Store::export_rows` straddling a concurrent `link` → dangling link key in the file | medium | mitigate (already shipped in 05-02; this plan adds the proof) | Task 1's race test: writer commits mid-stream, raced file parses with exact header counts, new rows appear only in the next export | +| T-05-21 | Denial of Service (test hang) | race test deadlocking the suite if a write ever blocked behind the open reader | low | mitigate | both hand-offs use `recv_timeout(10 s)`; a blocked write fails the test with a named message instead of hanging the workspace run (Task 1) | +| T-05-22 | Elevation of Privilege / Information Disclosure | realism run binds the unauthenticated REST daemon | low | mitigate | loopback only (`127.0.0.1:0`, `ensure_bind_allowed` refuses non-loopback without `--allow-remote`), ephemeral port, throwaway `mktemp -d` database, daemon killed by an EXIT trap even on failure (Task 2) | +| T-05-23 | Information Disclosure | fixture text sent to the local Ollama and left in a temp directory | low | accept | the three fixture sentences are synthetic engineering notes written for this run; nothing user-private is exported, embedded or kept; the temp dir is OS-managed | +| T-05-24 | Repudiation (evidence integrity) | a realism run whose checks were loosened until they passed | low | mitigate | the script's checks are fixed in the committed `05-05-realism.sh`; Task 2 forbids editing them to pass and requires the raw outputs verbatim in the SUMMARY; the re-verifier can re-run the same script (Task 2) | +| T-05-SC | Tampering | cargo installs | low | accept | zero new dependencies; `--locked` throughout; `Cargo.lock` unchanged | + +No `high` severity threat is introduced. ASVS L1, block_on: high. + + + +- `cargo test -p agent-memory-core --test export_import` → 20 passed; the new race test passes 5 consecutive runs; `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets --locked -- -D warnings`, `cargo test --workspace --locked` green (287 passed expected). +- `bash .planning/phases/05-portable-export-import/05-05-realism.sh` exits 0 with the `PASS:` line; evidence block present in 05-05-SUMMARY.md and referenced from 05-VALIDATION.md. +- `git diff --stat HEAD~2 -- crates/agent-memory-core/src crates/agent-memory/src Cargo.lock .github/ crates/agent-memory-core/sql/` empty across both task commits (test + docs only). + + + +- Both 05-VERIFICATION.md `human_verification` items have explicit evidence: truth 2.8 via `a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key`; the live cross-database re-embed via the recorded realism run. +- Two commits on `main` (one test, one docs), each gated by fmt + clippy + full suite; nothing pushed; no production code touched. + + + +Create `.planning/phases/05-portable-export-import/05-05-SUMMARY.md` when done. It MUST contain the `## Live Ollama realism evidence` section (verbatim script evidence block), the new test's name and the 5-run stability result, the full-suite count, and a one-line note that 05-02-PLAN.md was left untouched (the backstop marker is reconciled by the re-verifier against this plan's plain-string truth). + diff --git a/.planning/phases/05-portable-export-import/05-05-SUMMARY.md b/.planning/phases/05-portable-export-import/05-05-SUMMARY.md new file mode 100644 index 0000000..57cdb8d --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-05-SUMMARY.md @@ -0,0 +1,182 @@ +--- +phase: 05-portable-export-import +plan: 05 +subsystem: testing, validation-evidence +tags: [rust, sqlite, wal, snapshot-isolation, mpsc, export, import, ollama, nomic-embed-text, hybrid-search, gap-closure] + +# Dependency graph +requires: + - phase: 05-portable-export-import + provides: "plan 05-02: Store::export_rows on ONE pooled connection + ONE read transaction (unchecked_transaction), ExportEvent/ExportSnapshot stream, tests/export_import.rs harness; the backstop truth 2.8 this plan turns into evidence" + - phase: 05-portable-export-import + provides: "plan 05-03: parse_jsonl strict reader, MemoryService::import_jsonl, `import --from jsonl` six-count + pending lines; D-13 rows land pending and the sweep re-embeds them" + - phase: 05-portable-export-import + provides: "plan 05-04: real-binary CLI round trip + README; 05-VERIFICATION.md (human_needed, 32/33) naming the two items closed here" +provides: + - "tests/export_import.rs: `a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key` — the held-out, channel-ordered race proof for the one-snapshot export (DIST-04 backstop truth 2.8); Harness exposes `store: Arc`" + - "Recorded live-Ollama realism run (05-05-realism.sh, exit 0): A → move.jsonl → B lands 3 pending, first serve-rest sweep tick `backfilled=3`, `/health` embedder ready, paraphrase `POST /api/search` → `search_mode: hybrid` with the imported memory at rank 0 (INTEROP-02 criterion 3)" + - "05-VALIDATION.md: 05-05-T1 / 05-05-T2 green; Manual-Only live re-embed row `Result: PASS`; concurrency row retired as manual" +affects: [05 re-verification (both human_verification items now have explicit evidence), phase-6 release notes] + +# Actuals (#2632) — same estimateTokens scale as the plan's `estimate` (chars/4 over the realized diff) +actuals: + tokens: 8100 + tasks: 2 + commits: 2 + +# Tech tracking +tech-stack: + added: [] + patterns: + - "Channel-ordered race test: a std::thread writer on the store's writer lane and the exporter's read-pool snapshot hand off through two mpsc channels (go: exporter→writer on the first memory record; done: writer→exporter with the new id); every wait is `recv_timeout(10 s)` so a blocked write fails with a named message instead of hanging the suite — no timed waits decide ordering" + - "Realism run as a committed script, not a cargo test: live-service evidence (Ollama) lives in `.planning/phases/.../05-05-realism.sh` with fixed checks, counter-bounded waits, a trap-killed daemon on an ephemeral loopback port, and an evidence block pasted verbatim into the SUMMARY" + +key-files: + created: [] + modified: + - crates/agent-memory-core/tests/export_import.rs + - .planning/phases/05-portable-export-import/05-VALIDATION.md + +key-decisions: + - "Gap 1 closed with a test, not a code change: WAL snapshot isolation was already the mechanism; the race test drives `Store::export_rows` directly with a pausing visitor and proves the committed row is visible to a fresh connection yet absent from the open snapshot, the raced file parses strictly with exact header counts, and the next export carries both the memory and the `caused_by` edge" + - "Gap 2 closed by running the real binary against the real `nomic-embed-text` on this host and recording the raw outputs — never a fake embedder, never a loosened check; the fake-embedder mechanism proof (`imported_rows_land_pending_and_the_sweep_backfills_them`) stays as the CI gate" + - "05-02-PLAN.md left untouched (its `verification: backstop` marker stays); this plan's plain-string truth plus the new test is what the re-verifier reconciles truth 2.8 against" + - "TDD RED observed as a compile failure (`no field store on type Harness` + four missing imports) before the harness change; GREEN on first run; test and harness change committed together because the project rule forbids committing failing tests" + +patterns-established: + - "Held-out proof for a `verification: backstop` truth: restate the truth as a plain string in the gap-closure plan, add the exact test the verifier named, leave the original plan's marker in place" + +requirements-completed: [DIST-04, INTEROP-02] + +# Coverage metadata (#1602) +coverage: + - id: D1 + description: "A memory and a link committed by a concurrent writer while `Store::export_rows` is mid-stream are invisible to that export (header counts == record counts, no dangling link key, raced file accepted by `parse_jsonl`) and present in the next export; the write commits while the snapshot is open" + requirement: DIST-04 + verification: + - kind: integration + ref: "crates/agent-memory-core/tests/export_import.rs#a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key (5 consecutive runs green; file 20/20)" + status: pass + human_judgment: false + - id: D2 + description: "Cross-database move with the live embedder: export from A, import into B lands pending, first serve-rest sweep tick re-embeds all three rows against the local nomic-embed-text, paraphrase search answers search_mode hybrid with the imported memory in results" + requirement: INTEROP-02 + verification: + - kind: other + ref: "bash .planning/phases/05-portable-export-import/05-05-realism.sh (2026-10-25T13:39:00Z, exit 0, `PASS: live Ollama realism run — search_mode hybrid, imported memory id 1 at rank 0`)" + status: pass + human_judgment: false + +# Metrics +duration: 6 min +completed: 2026-10-25 +status: complete +--- + +# Phase 5 Plan 05: Gap closure — one-snapshot race proof + live-Ollama cross-database re-embed Summary + +**A channel-ordered race test proves the export's WAL snapshot cannot straddle a concurrent memory+link commit, and a recorded run of the real binary against the live `nomic-embed-text` moves three memories A → JSONL → B, backfills them on the first sweep tick and answers a paraphrase search in hybrid mode — closing both `human_verification` items of 05-VERIFICATION.md without touching production code.** + +## Performance + +- **Duration:** 6 min +- **Started:** 2026-10-25T13:35:24Z +- **Completed:** 2026-10-25T13:42:16Z +- **Tasks:** 2 +- **Files modified:** 2 (one test file, one planning doc) + +## Accomplishments + +- `a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key` (tests/export_import.rs): a `std::thread` writer owning `Arc` inserts "delta arrived mid-export" and links it `caused_by` → `a` while the exporter is paused inside the visitor on the first memory record. A fresh raw connection sees 4 rows / 3 edges while the export's read transaction is still open; the snapshot still reports 3 / 2 and no record carries the new id; the raced file (header mirrored from the snapshot as the service does, plus every visited record) is accepted by `parse_jsonl` with `memories.len() == 3`, `links.len() == 2`; the next `export_to_string` reports `{ memories: 4, links: 3 }` and carries both the new memory and the `caused_by` edge. Ordering is decided only by two `mpsc` hand-offs, every wait bounded by `recv_timeout(10 s)`; no `thread::sleep`. Stable over 5 consecutive runs; `--test export_import` is 20/20. +- `Harness` gained `store: Arc` (the one argument now shared with `MemoryService::new`); module doc updated; nothing else in the 19 existing tests changed. +- Live realism run (`05-05-realism.sh`, exit 0, 2026-10-25T13:39:00Z) recorded verbatim below: six-count import line, `3 rows pending embedding`, `embedded pending memories on sweep backfilled=3 skipped_stale=0` about 50 ms after `REST listening on 127.0.0.1:57875`, `/health` `"embedder":"ready"`, `"search_mode":"hybrid"` with the SQLite-engine memory (id 1) at rank 0 for a paraphrase that shares almost no keyword with it. +- 05-VALIDATION.md: `05-05-T1` and `05-05-T2` flipped to green with the run date; the Manual-Only live re-embed row carries `Result: PASS …`; the concurrency Manual-Only row was already retired in favour of the automated test. +- Gates per commit: `cargo fmt --all -- --check` clean, `cargo clippy --workspace --all-targets --locked -- -D warnings` exit 0, `cargo test --workspace --locked` **287 passed / 0 failed / 2 ignored** (baseline 286 + the new test). + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Held-out race test — a memory and link committed mid-export are invisible to that snapshot and present in the next** - `6cb8893` (test) +2. **Task 2: Live-Ollama realism run — export from A, import into B, first-sweep backfill, hybrid paraphrase search (recorded evidence)** - `d52c1ff` (docs) + +**Plan metadata:** see the final `docs(05-05)` commit (this SUMMARY + STATE/ROADMAP/REQUIREMENTS). + +_Task 1 was `tdd="true"`: RED was observed as a compile failure (`error[E0609]: no field `store` on type `Harness`` plus the four planned missing imports) before the harness change; GREEN on the first run after it. Test and harness change were committed together because the project rule forbids committing failing tests._ + +## Files Created/Modified + +- `crates/agent-memory-core/tests/export_import.rs` - `store: Arc` on `Harness`; imports `Store`, `ExportEvent`, `ExportSnapshot`, `mpsc`, `Duration`; `record_lines` helper; the new race test under the banner "Export half — gap closure 05-05: one-snapshot concurrency proof" +- `.planning/phases/05-portable-export-import/05-VALIDATION.md` - 05-05-T1 / 05-05-T2 Status cells green; Manual-Only live re-embed `Result: PASS` with the evidence pointer + +No production file changed: `git diff --stat 30085a2 HEAD -- crates/agent-memory-core/src crates/agent-memory/src Cargo.lock .github/ crates/agent-memory-core/sql/` is empty. `05-02-PLAN.md` was left untouched — its `verification: backstop` marker is reconciled by the re-verifier against this plan's plain-string truth and the new test. + +## Live Ollama realism evidence + +Command run from the repo root (background + log, as the plan directs): `bash .planning/phases/05-portable-export-import/05-05-realism.sh > /tmp/p0505/realism.log 2>&1` — exit 0; last line `PASS: live Ollama realism run — search_mode hybrid, imported memory id 1 at rank 0 (evidence: /var/folders/jr/lwzz65cx5pndqqdgzhnym1pc0000gn/T/tmp.Ws0nq17M6o/EVIDENCE.md)`. Precondition probe before the run: `curl -s --max-time 5 http://localhost:11434/api/tags | grep -c '"name":"nomic-embed-text'` → `1`; `/api/version` → `{"version":"0.35.0"}`. The script's checks were not edited. The evidence block below is the script's output verbatim (`EVIDENCE.md`): + +**Live Ollama realism evidence (2026-10-25T13:39:00Z)** — the script's own heading line, demoted to bold so the section heading above stays the only `## Live Ollama realism evidence` heading + +- host: Darwin 27.0.0 arm64; Ollama at http://localhost:11434, version {"version":"0.35.0"}; model: {"name":"nomic-embed-text:latest" +- binary: target/debug/agent-memory (agent-memory 0.0.1); workdir: /var/folders/jr/lwzz65cx5pndqqdgzhnym1pc0000gn/T/tmp.Ws0nq17M6o (kept, not removed) + +```text +$ target/debug/agent-memory import --from gsd-state --scope realism --db /var/folders/jr/lwzz65cx5pndqqdgzhnym1pc0000gn/T/tmp.Ws0nq17M6o/a.db /var/folders/jr/lwzz65cx5pndqqdgzhnym1pc0000gn/T/tmp.Ws0nq17M6o/STATE.md +imported: 3, skipped duplicates: 0, malformed lines skipped: 0 +$ target/debug/agent-memory export --db /var/folders/jr/lwzz65cx5pndqqdgzhnym1pc0000gn/T/tmp.Ws0nq17M6o/a.db --output /var/folders/jr/lwzz65cx5pndqqdgzhnym1pc0000gn/T/tmp.Ws0nq17M6o/move.jsonl +exported 3 memories and 0 links +move.jsonl line 1: {"record":"header","format":"agent-memory-export","version":1,"exported_at":1792935539,"embedding_model":"nomic-embed-text","embedding_dim":768,"schema_version":3,"agent_memory_version":"0.0.1","exported_rows":3,"exported_links":0} +$ target/debug/agent-memory import --from jsonl --db /var/folders/jr/lwzz65cx5pndqqdgzhnym1pc0000gn/T/tmp.Ws0nq17M6o/b.db /var/folders/jr/lwzz65cx5pndqqdgzhnym1pc0000gn/T/tmp.Ws0nq17M6o/move.jsonl +imported: 3, skipped duplicates: 0, skipped expired: 0, links imported: 0, links skipped duplicates: 0, links dropped: 0 +3 rows pending embedding — the sweep backfills them on the next serve/serve-rest start (128 per tick) +$ RUST_LOG=info target/debug/agent-memory serve-rest --db /var/folders/jr/lwzz65cx5pndqqdgzhnym1pc0000gn/T/tmp.Ws0nq17M6o/b.db --addr 127.0.0.1:0 (pid 99888, stderr → daemon.log) +2026-10-25T13:38:59.490867Z INFO agent_memory: REST listening on 127.0.0.1:57875 +2026-10-25T13:38:59.495966Z INFO agent_memory: semantic search ready (nomic-embed-text) +2026-10-25T13:38:59.539602Z INFO agent_memory_core::service: embedded pending memories on sweep backfilled=3 skipped_stale=0 +2026-10-25T13:38:59.539644Z INFO agent_memory: lifecycle sweep complete expired=0 rescored=3 +$ curl http://127.0.0.1:57875/health +{"embedder":"ready","status":"ok"} +$ curl -X POST http://127.0.0.1:57875/api/search -d '{ "query": "which database did we choose so that nothing needs a hosted service", "limit": 3}' +{"search_mode":"hybrid","results":[{"id":1,"content":"SQLite is the embedded storage engine so the memory daemon runs with no cloud account and no server process","type":"DECISION","tags":["gsd"],"scope":"realism","decay_score":0.999999955430352,"created_at":1792935539,"last_accessed":1792935539},{"id":2,"content":"Release tarballs carry a signed SLSA provenance attestation built on GitHub-hosted runners","type":"DECISION","tags":["gsd"],"scope":"realism","decay_score":0.999999955430352,"created_at":1792935539,"last_accessed":1792935539},{"id":3,"content":"Clippy must pass with -D warnings and cargo fmt must be clean before every commit","type":"DECISION","tags":["gsd"],"scope":"realism","decay_score":0.999999955430352,"created_at":1792935539,"last_accessed":1792935539}]} +paraphrase target rank (0 = first): 0; imported memory id in b.db: 1 +``` + +Reading: the gsd-state seed of A reports the gsd-path three-count line (zeros for the link counters are implicit there — D-12's additive report prints the six-count form only on the JSONL path); the export header is format v1 with `exported_rows: 3` / `exported_links: 0`; B's import prints the exact six-count line and the pending line; the daemon's first sweep tick embedded all three rows (`backfilled=3 skipped_stale=0`) 49 ms after bind, before `lifecycle sweep complete expired=0 rescored=3`; `/health` reports the embedder ready; the paraphrase ("which database did we choose so that nothing needs a hosted service") shares essentially no keyword with the target ("SQLite is the embedded storage engine so the memory daemon runs with no cloud account and no server process"), so the KNN leg carried rank 0 — `search_mode: "hybrid"` with the imported memory present is INTEROP-02 criterion 3 on the live embedder. The temp directory was kept (the repo's pre-bash hook forbids removing `.db` files from the shell). + +## Decisions Made + +- The gap was closed with evidence only (one test, one recorded run, planning rows) — no production, schema, lock or workflow change, as the plan's prohibitions require. +- The writer in the race test is a `std::thread`, not a tokio task: `Store` methods are synchronous and `#[tokio::test]` is a current-thread runtime that the exporter blocks while waiting on `done_rx`. +- The realism run stays a script and is not promoted to a cargo test (Ollama is absent in CI; the fake-embedder proof remains the CI gate). + +## Deviations from Plan + +None - plan executed exactly as written. + +## Issues Encountered + +None. The `echo ====` separators in early read commands were rewritten by the local RTK hook into failing commands; switched to `printf` separators — no effect on the work. + +## TDD Gate Compliance + +Task 1 (`tdd="true"`): RED observed as a compile failure against the unchanged harness (`no field store on type Harness`), GREEN on the first run after adding `store: Arc`. Per the project rule "never commit failing tests", the RED state was not committed; a single `test(05-05)` commit (`6cb8893`) carries the test and the harness change. No refactor commit was needed. + +## User Setup Required + +None - no external service configuration required (the run used the already-running local Ollama). + +## Next Phase Readiness + +- Both 05-VERIFICATION.md `human_verification` items now have explicit, re-runnable evidence on this host: truth 2.8 via the race test (CI-gated), the cross-database live re-embed via the recorded script run. The orchestrator re-runs phase verification next; this plan does not mark the phase complete. +- Phase 5 plan progress after this plan: 5/5 plans executed. +- Workspace gate at `d52c1ff`: fmt clean, clippy clean, 287 passed / 0 failed / 2 ignored. + +--- +*Phase: 05-portable-export-import* +*Completed: 2026-10-25* + +## Self-Check: PASSED + +- files present: 05-05-SUMMARY.md, 05-VALIDATION.md, tests/export_import.rs, 05-05-realism.sh +- commits present: 6cb8893 (Task 1), d52c1ff (Task 2) +- Task 2 acceptance greps on this file: section heading 1, `"search_mode":"hybrid"` ≥1, `backfilled=3` ≥1, six-count line ≥1; VALIDATION `Result: PASS` ≥1 diff --git a/.planning/phases/05-portable-export-import/05-05-realism.sh b/.planning/phases/05-portable-export-import/05-05-realism.sh new file mode 100755 index 0000000..e06a50d --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-05-realism.sh @@ -0,0 +1,166 @@ +#!/usr/bin/env bash +# Phase 5 gap closure, plan 05-05 Task 2: live-Ollama realism run. +# +# Performs the "on the other machine" step of the Phase 5 user story with the +# REAL binary against the REAL local embedder (INTEROP-02 criterion 3, D-13): +# +# 1. seed database A from a three-bullet GSD STATE.md fixture +# 2. `agent-memory export --output move.jsonl` (A → file) +# 3. `agent-memory import --from jsonl` into database B (file → B, rows pending) +# 4. `agent-memory serve-rest --db B --addr 127.0.0.1:0`; wait for the FIRST +# sweep tick to backfill the pending rows against nomic-embed-text +# 5. `POST /api/search` with a PARAPHRASE of one memory → search_mode hybrid, +# imported memory present in results +# 6. stop the daemon, print the evidence block +# +# This is NOT a cargo test and must not become one: it needs a local Ollama with +# nomic-embed-text, which CI does not have. The mechanism is covered with a fake +# embedder by `imported_rows_land_pending_and_the_sweep_backfills_them`; this run +# is the realism evidence 05-VERIFICATION.md routed to a human. +# +# Every wait is a counter-bounded loop (this host has no `timeout` binary). The +# temp directory is kept on purpose: the repo's pre-bash hook refuses to remove +# database files from the shell, and the kept files ARE the evidence. +set -u + +OLLAMA_URL="${AGENT_MEMORY_OLLAMA_URL:-http://localhost:11434}" +STEP_SECS=0.5 + +fail() { + echo "FAIL: $*" >&2 + exit 1 +} + +# wait_for (STEP_SECS per iteration) +wait_for() { + local pattern="$1" file="$2" max="$3" i=0 + until grep -q -- "$pattern" "$file" 2>/dev/null; do + if [ "$i" -ge "$max" ]; then + return 1 + fi + sleep "$STEP_SECS" + i=$((i + 1)) + done + return 0 +} + +# --- 0. precondition: live Ollama with the pinned model ----------------------- +TAGS="$(curl -s --max-time 5 "$OLLAMA_URL/api/tags")" \ + || fail "Ollama is not reachable at $OLLAMA_URL — this run needs the live daemon" +MODEL_LINE="$(printf '%s' "$TAGS" | tr ',' '\n' | grep '"name":"nomic-embed-text' | head -1)" +[ -n "$MODEL_LINE" ] \ + || fail "Ollama at $OLLAMA_URL does not list nomic-embed-text — run: ollama pull nomic-embed-text" +OLLAMA_VERSION="$(curl -s --max-time 5 "$OLLAMA_URL/api/version")" + +# --- 1. build the real binary ------------------------------------------------- +cargo build -q -p agent-memory --locked || fail "cargo build -p agent-memory --locked" +B="target/debug/agent-memory" +[ -x "$B" ] || fail "binary not found at $B" +T="$(mktemp -d)" +echo "workdir: $T (kept)" + +DPID="" +DPID_RECORDED="" +cleanup() { + if [ -n "$DPID" ] && kill -0 "$DPID" 2>/dev/null; then + kill "$DPID" 2>/dev/null + wait "$DPID" 2>/dev/null + fi +} +trap cleanup EXIT + +M1="SQLite is the embedded storage engine so the memory daemon runs with no cloud account and no server process" +M2="Release tarballs carry a signed SLSA provenance attestation built on GitHub-hosted runners" +M3="Clippy must pass with -D warnings and cargo fmt must be clean before every commit" +# A paraphrase of M1 that shares (almost) no keyword with it — the semantic leg +# has to do the work. +QUERY="which database did we choose so that nothing needs a hosted service" + +# --- 2. fixture: a GSD STATE.md with three Decisions bullets ------------------ +printf '### Decisions\n\n- %s\n- %s\n- %s\n' "$M1" "$M2" "$M3" > "$T/STATE.md" + +# --- 3. database A via the gsd-state importer --------------------------------- +"$B" import --from gsd-state --scope realism --db "$T/a.db" "$T/STATE.md" \ + > "$T/a-import.out" 2> "$T/a-import.err" || fail "gsd-state import into a.db: $(cat "$T/a-import.err")" +grep -q '^imported: 3, skipped duplicates: 0' "$T/a-import.out" \ + || fail "a.db import line unexpected: $(cat "$T/a-import.out")" + +# --- 4. export A → move.jsonl ------------------------------------------------- +"$B" export --db "$T/a.db" --output "$T/move.jsonl" 2> "$T/export.err" \ + || fail "export: $(cat "$T/export.err")" +grep -q 'exported 3 memories and 0 links' "$T/export.err" \ + || fail "export report unexpected: $(cat "$T/export.err")" +LINES="$(wc -l < "$T/move.jsonl" | tr -d ' ')" +[ "$LINES" = 4 ] || fail "move.jsonl must have 4 lines (header + 3 memories), got $LINES" +head -1 "$T/move.jsonl" | grep -q '"exported_rows":3' || fail "header lacks exported_rows 3" +head -1 "$T/move.jsonl" | grep -q '"exported_links":0' || fail "header lacks exported_links 0" + +# --- 5. import move.jsonl into database B ------------------------------------- +"$B" import --from jsonl --db "$T/b.db" "$T/move.jsonl" \ + > "$T/b-import.out" 2> "$T/b-import.err" || fail "jsonl import into b.db: $(cat "$T/b-import.err")" +grep -q '^imported: 3, skipped duplicates: 0, skipped expired: 0, links imported: 0, links skipped duplicates: 0, links dropped: 0$' "$T/b-import.out" \ + || fail "b.db six-count line unexpected: $(cat "$T/b-import.out")" +grep -q '^3 rows pending embedding' "$T/b-import.out" \ + || fail "pending line unexpected: $(cat "$T/b-import.out")" + +# --- 6. serve-rest on B; first sweep tick backfills the three pending rows ---- +RUST_LOG="${RUST_LOG:-info}" "$B" serve-rest --db "$T/b.db" --addr 127.0.0.1:0 \ + > /dev/null 2> "$T/daemon.log" & +DPID=$! +DPID_RECORDED="$DPID" +wait_for 'REST listening on' "$T/daemon.log" 60 \ + || fail "daemon did not log 'REST listening on' within 30 s: $(cat "$T/daemon.log")" +PORT="$(grep -o 'REST listening on 127.0.0.1:[0-9]*' "$T/daemon.log" | head -1 | sed 's/.*://')" +[ -n "$PORT" ] || fail "could not parse the ephemeral port from: $(grep 'REST listening on' "$T/daemon.log")" +wait_for 'embedded pending memories on sweep' "$T/daemon.log" 240 \ + || fail "first sweep did not backfill within 120 s: $(cat "$T/daemon.log")" +grep -q 'backfilled=3' "$T/daemon.log" \ + || fail "expected backfilled=3, got: $(grep 'embedded pending memories on sweep' "$T/daemon.log")" + +# --- 7. /health then the paraphrase search ------------------------------------ +curl -s --max-time 10 "http://127.0.0.1:$PORT/health" > "$T/health.json" || fail "GET /health" +grep -q '"embedder":"ready"' "$T/health.json" || fail "embedder not ready: $(cat "$T/health.json")" + +jq -n --arg q "$QUERY" '{query: $q, limit: 3}' > "$T/search-request.json" +curl -s --max-time 60 -X POST "http://127.0.0.1:$PORT/api/search" \ + -H 'content-type: application/json' --data-binary @"$T/search-request.json" \ + > "$T/search.json" || fail "POST /api/search" +grep -q '"search_mode":"hybrid"' "$T/search.json" \ + || fail "search_mode is not hybrid: $(cat "$T/search.json")" +RANK="$(jq -r --arg m "$M1" '[.results[].content] | index($m)' "$T/search.json")" +[ -n "$RANK" ] && [ "$RANK" != "null" ] \ + || fail "the imported memory is not in results: $(cat "$T/search.json")" +ID="$(jq -r --arg m "$M1" '.results[] | select(.content == $m) | .id' "$T/search.json" | head -1)" + +# --- 8. stop the daemon -------------------------------------------------------- +cleanup +DPID="" + +# --- 9. evidence block ----------------------------------------------------------- +{ + echo "## Live Ollama realism evidence ($(date -u +%Y-%m-%dT%H:%M:%SZ))" + echo + echo "- host: $(uname -srm); Ollama at $OLLAMA_URL, version $OLLAMA_VERSION; model: $MODEL_LINE" + echo "- binary: $B ($("$B" --version 2>/dev/null | head -1)); workdir: $T (kept, not removed)" + echo + echo '```text' + echo "\$ $B import --from gsd-state --scope realism --db $T/a.db $T/STATE.md" + cat "$T/a-import.out" + echo "\$ $B export --db $T/a.db --output $T/move.jsonl" + cat "$T/export.err" + echo "move.jsonl line 1: $(head -1 "$T/move.jsonl")" + echo "\$ $B import --from jsonl --db $T/b.db $T/move.jsonl" + cat "$T/b-import.out" + echo "\$ RUST_LOG=info $B serve-rest --db $T/b.db --addr 127.0.0.1:0 (pid $DPID_RECORDED, stderr → daemon.log)" + grep -E 'semantic search ready|REST listening on|lifecycle sweep complete|embedded pending memories on sweep|Ollama' "$T/daemon.log" + echo "\$ curl http://127.0.0.1:$PORT/health" + cat "$T/health.json" + echo + echo "\$ curl -X POST http://127.0.0.1:$PORT/api/search -d '$(cat "$T/search-request.json" | tr -d '\n')'" + cat "$T/search.json" + echo + echo "paraphrase target rank (0 = first): $RANK; imported memory id in b.db: $ID" + echo '```' +} | tee "$T/EVIDENCE.md" + +echo "PASS: live Ollama realism run — search_mode hybrid, imported memory id $ID at rank $RANK (evidence: $T/EVIDENCE.md)" diff --git a/.planning/phases/05-portable-export-import/05-CONTEXT.md b/.planning/phases/05-portable-export-import/05-CONTEXT.md new file mode 100644 index 0000000..b30f71c --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-CONTEXT.md @@ -0,0 +1,146 @@ +# Phase 5: Portable Export & Import - Context + +**Gathered:** 2026-10-25 +**Status:** Ready for planning +**Mode:** auto (user delegated: resolve every gray area by recommendation; every choice below is the recommended option, logged for audit). ROADMAP success criteria are settled and were not re-discussed. + + +## Phase Boundary + +A user can move an agent's whole memory between machines with no cloud: `agent-memory export` writes a versioned JSONL file (header with format version + embedding model/dim, one record per memory, one record per relation edge keyed by export-local keys, no embedding vectors), and `agent-memory import --from jsonl` recreates memories and edges on another machine — into an empty or a non-empty database — idempotently, with imported rows landing `embedding_status = 0` for the existing sweep backfill, expired rows skipped and counted, and malformed files rejected as client errors through the Phase 3 seam. Requirements: DIST-04, INTEROP-02. + +**User-approved scope addition (folds the Phase 4 deferral, 04-REVIEW IN-01):** every non-search service path — `store`, `list` (incl. `expand_links`), `update`, `link`/`unlink`, `forget`, `import`, the new `export`, and `/health` — still lets raw SQLite text (`sqlite error: …`, table/column names) reach the client on a storage fault. Import adds another write path through the same mapping, so this phase redacts storage-engine text on EVERY service path, on both transports, the way Phase 4 did for search. + +Not in this phase: embedding vectors in the file, export filters, MCP/REST export or import endpoints, cross-tool import dialects, Windows/musl binaries and the v0.1.0 release (Phase 6). + + + + +## Implementation Decisions + +### Surface & CLI shape (DIST-04 / INTEROP-02) +- **D-01:** Export and import are **CLI subcommands only** in v0.1.0: `agent-memory export [--output PATH]` and `agent-memory import --from jsonl PATH` (the second value of the existing `--from` dispatch beside `gsd-state`). No MCP tool and no REST endpoint for either. Rationale: an MCP export would route the corpus through the JSON-RPC stdout channel (MCP-05 purity), a REST import is an arbitrary-file-read primitive on an unauthenticated daemon (PITFALLS security row), and both roadmap criteria describe a file written and read by the user. The **error contract still lives in core**: every malformed-file rejection is `MemoryError::InvalidArgument` (Phase 3 D-04/D-06), so the two-tier taxonomy holds and any later transport inherits 400/`invalid_params` structurally; the CLI exits non-zero printing that message. — **Reversibility:** reversible — adding a transport later is additive. +- **D-02:** `export` writes to **stdout by default**; `--output PATH` writes **temp-file-in-the-same-directory + rename** (atomic; the target is replaced if it exists), **mode 0600 on Unix** (the corpus is T-01-03-sensitive; never looser than the DB), using the same `cfg(unix)` / `cfg(not(unix))` split `config.rs` already has (Windows is one seam, Phase 6). stdout is sanctioned for CLI subcommands (the `import` precedent); MCP-05 applies only to `serve`. Export streams one line at a time — never buffers the corpus as one JSON document. + +### JSONL format v1 — the frozen contract (DIST-04) +- **D-03:** **Line 1 is a header record**, mandatory: `{"record":"header","format":"agent-memory-export","version":1,"exported_at":,"embedding_model":"nomic-embed-text","embedding_dim":768,"schema_version":3,"agent_memory_version":""}`. `embedding_model`/`embedding_dim` are **read from the `meta` table** (the 0002 pin), not from constants, so the header always states what the source DB actually pinned. — **Reversibility:** one-way — the moment one user writes a file, format v1 is a compatibility contract; any field removal or meaning change needs `version: 2` plus a reader that still accepts 1. +- **D-04:** **Memory record:** `{"record":"memory","key":,"type":"DECISION","content":"…","tags":[…],"source":,"scope":,"created_at":,"last_accessed":,"access_count":,"expires_at":}`. `key` is the source database's id used **only as an export-local key** — it is never written as an id on import (PITFALLS 18). **Excluded as derived data:** `decay_score` (recomputed on read), `base_weight` (a function of `type`), `embedding_status`, and every vector (PITFALLS 17; ARCHITECTURE Anti-Pattern 4). Field order is the declaration order above (serde struct), golden-tested like the v1.0 envelopes. +- **D-05:** **Link record:** `{"record":"link","from":,"to":,"kind":"relates_to|supersedes|caused_by","created_at":}`, emitted **after all memory records**, endpoints exactly as stored (04-06 canonical `(min,max)` for `relates_to`; request order for directed kinds). `created_at` travels because D-07 of Phase 4 orders expansion by edge recency. +- **D-06:** Export writes **every row present in the database** (no `--scope`/`--type` filters, no expiry skipping at export time — the importer owns expiry policy) in ascending id order, through one streaming read-pool query per record type. Filters are deferred (additive). + +### Import identity, dedup & edge remapping (INTEROP-02) +- **D-07:** The idempotency key stays the existing INTEROP-01 key **`(source, mem_type, content)`** — no new identity column, **no migration 0004**. `Store::exists` is generalised to return the matching row's id (`Option`, NULL-safe `source IS ?1` kept, CR-01) so a duplicate maps its export key to the **existing** row. Consequence, accepted and documented: two source rows identical on that key collapse to one on import (counted as a duplicate). `source` is **preserved verbatim** (no `jsonl` provenance stamp), so the round trip is faithful and a `gsd-state` row carried in a JSONL file dedups against an earlier direct GSD import. +- **D-08:** **Two-pass, validate-before-write import:** (1) parse and validate the whole file into typed records (header, memories, links) — any error aborts with nothing written; (2) memories pass: per record, skip-and-count if expired (D-10), else dedup-or-insert, building `key → row id`; (3) links pass through the map via the existing `Store::link` (idempotent `INSERT OR IGNORE` + endpoint pre-check), with the record's `created_at` as `now`. A link whose endpoint was **skipped as expired is dropped and counted** (`links_dropped`); a link whose key **does not appear in the file at all is malformed** (whole-file rejection, D-11). All writes go through the existing writer lane; batching into transactions is the planner's call. +- **D-09:** Imported rows go through the shared Phase 3 seam like every insert path: the `type` and `kind` wire strings parse via `MemoryType::try_from` / `LinkKind::try_from`, empty/whitespace `content` is `InvalidArgument`, self-links are `InvalidArgument` (Phase 4 D-02), tags must be a JSON array of strings. `expires_at` is absolute, so `validate_ttl` does not apply; a non-null `expires_at` must be an integer. + +### Timestamp & lifecycle policy on import (INTEROP-02; PITFALLS 19) +- **D-10:** Per column, written into the format doc: **`created_at` preserved** (provenance); **`expires_at` preserved**, and a row with `expires_at <= import-time now` is **skipped and counted (`skipped_expired`) — never inserted** for the sweep to delete later; **`last_accessed` refreshed to import-time now** so imported memories start decay-fresh (PITFALLS 19: a preserved months-old `last_accessed` would make a restored corpus unfindable by rank); **`access_count` preserved** (provenance only — `decay_score` depends solely on `last_accessed`, so this changes no ranking); link `created_at` preserved (D-05). The decay-never-deletes invariant is untouched. If the header's `exported_at` is more than one hour in the future, the CLI prints a clock-skew **warning** to stderr and continues. + +### Malformed-file policy & the import report (INTEROP-02 criterion 4) +- **D-11:** **Strict, all-or-nothing validation — the opposite of the tolerant `gsd-state` scraper, deliberately:** JSONL is agent-memory's own contract, so a defect means a corrupted or hand-edited file and a half-restored corpus is worse than a clear refusal. Rejected as `InvalidArgument` naming the 1-based line: missing/invalid header, `format` ≠ `agent-memory-export`, non-integer or `< 1` `version`, `version > 1` (message says to upgrade agent-memory), a line that is not a JSON object, an unknown `record` value, an unknown field (`deny_unknown_fields`, the 04-07 precedent), a wrong field type, an invalid `type`/`kind`, empty content, a self-link, a link key absent from the file. Message shape follows Phase 3 D-05: `invalid argument: line 42: unknown record type 'memo'`. **Nothing is written when validation fails.** Expired rows are **not** malformed (D-10). The `gsd-state` importer's tolerance rules are unchanged. +- **D-12:** `ImportReport` grows **additively**: `imported`, `skipped_duplicates` (existing), plus `skipped_expired`, `links_imported`, `links_skipped_duplicates`, `links_dropped`. The `gsd-state` path reports zeros for the new fields. The CLI prints every count on one line plus, for JSONL, a line stating how many rows are pending embedding (D-13). + +### Embeddings on import (INTEROP-02 criterion 3) +- **D-13:** **JSONL import never embeds inline.** Every imported row lands `embedding_status = 0` and is re-embedded by the **existing sweep backfill** (128 per tick; the first tick fires on every `serve`/`serve-rest` start) — exactly one re-embed mechanism, zero mixing risk (PITFALLS 17), the CLI never blocks on Ollama, and the criterion holds literally regardless of embedder state. Cost, accepted and documented in the README: a large restore is keyword-only until backfill catches up (128 rows per hourly tick after the first). The `gsd-state` importer keeps its existing one-shot best-effort embed. A faster drain is deferred (see Deferred Ideas). + +### Write-path storage-error redaction (user-approved scope addition; 04-REVIEW IN-01) +- **D-14:** Generalise `redact_search_error` into a **service-seam `redact_storage_error(op, e)`** applied to the result of **every public `MemoryService` method** (`store`, `list`, `search`, `update`, `link`, `unlink`, `forget`, `import`, the new `export`, `sweep`). Storage-engine variants (`Sqlite | Pool | Join | Migration`) are logged at `error` level with full detail, then replaced by `MemoryError::Internal(" failed due to a storage error (details are in the server log)")`; client tier (`InvalidType | InvalidQuery | InvalidArgument`) and crate-authored text (`Internal`, `SchemaTooNew`) pass through unchanged. `SEARCH_STORAGE_FAILURE` keeps its exact current text (it already fits the pattern), so Phase 4's golden assertions still hold. The two transport mappers stay as they are (their exhaustive matches still compile), which means **MCP, REST — including `/health` — and the CLI are covered by one change at one seam**, mirroring Phase 3 D-06 and Phase 4 T-04-19. The README sentence that scopes redaction to search is widened to every surface. — **Reversibility:** reversible. + +### Claude's Discretion +- How the import insert carries preserved `created_at`/`access_count` into the store: extend `NewMemory` with optional overrides (ARCHITECTURE §4) or add a dedicated `ImportedMemory` + `Store::insert_imported` — pick whichever touches fewer existing call sites while keeping `insert`'s single-writer-transaction shape. +- Transaction batching for the import passes (per row vs chunked) and the export read (one statement vs paged) — streaming on both sides is the requirement, the chunk size is not. +- Temp-file naming for `--output`; whether to reuse the backup temp-path helper from `sqlite.rs`. +- Whether the header also carries `exported_rows`/`exported_links` counts (nice for a quick integrity check; additive within v1 only if decided now, since D-11 rejects unknown fields later). +- Exact rustdoc placement of the format spec (`crates/agent-memory-core/src/export.rs` module doc + a README "Export & import" section is the minimum). +- Zero-code README tip documenting `VACUUM INTO` as a same-machine *backup* (FEATURES anti-pattern row) — include if cheap. + + + + +## Canonical References + +**Downstream agents MUST read these before planning or implementing.** + +### Milestone research (decides the patterns this phase implements) +- `.planning/research/SUMMARY.md` — "Phase 3: Export/Import (JSONL)" section (header, no vectors, key→id remap, timestamp policy, rich report) and the pin-triangle do-not-touch list +- `.planning/research/ARCHITECTURE.md` §4 "Export / Import (DIST-04)" — streaming JSONL, `Store::export_rows`, `import/jsonl.rs` beside `gsd_state.rs`, same `MemoryService::import` dedup path, timestamp-preservation note; Anti-Pattern 4 (never export vectors) +- `.planning/research/PITFALLS.md` — Pitfall 17 (model-pinned vectors), 18 (rowid collisions + edge remap, two-pass import), 19 (absolute timestamps: preserve/skip/refresh), Security rows (export 0600; import file-path only on the CLI), Performance row (stream, never through MCP stdout) +- `.planning/research/FEATURES.md` §3 "Portable export/import formats" — JSONL precedent, header fields, embeddings-excluded-by-default, round-trip idempotency + +### Prior phase contracts (locked, must be honored) +- `.planning/phases/03-api-hardening-toolchain-spikes/03-CONTEXT.md` — D-04/D-05/D-06: `InvalidArgument` at the `MemoryService` seam, message shape `field + allowed set + offending value`, transports stay thin +- `.planning/phases/04-memory-update-relations-hybrid-search/04-CONTEXT.md` — D-01..D-03 (closed `LinkKind`, duplicate link = idempotent, directed storage), D-07 (edge recency ordering → link `created_at` must travel), D-11/D-12 (backup mechanism this phase inherits for free — no migration here) +- `.planning/phases/04-memory-update-relations-hybrid-search/04-REVIEW.md` — IN-01 (the raw-SQLite-text deferral this phase closes, with the mapper-vs-seam options), WR-02 (`deny_unknown_fields` precedent for new DTOs), IN-05 (SchemaTooNew re-probe note — do not regress) +- `.planning/STATE.md` — accumulated decisions: [03-01] `import()` validates per draft through the shared helper (the seam this phase extends); [Phase 04] 04-06 `relates_to` canonical storage; 04-07 `deny_unknown_fields` on new v0.1.0 DTOs only + +### Code the phase extends +- `crates/agent-memory-core/src/service.rs` — `import()` (dedup partition → best-effort embed → insert), `ImportReport`, `redact_search_error` + `SEARCH_STORAGE_FAILURE` (the redaction helper D-14 generalises), `BACKFILL_BATCH` +- `crates/agent-memory-core/src/import/mod.rs` + `import/gsd_state.rs` — the importer module the `jsonl.rs` parser joins (keep the gsd tolerance rules intact) +- `crates/agent-memory-core/src/store/mod.rs` + `store/sqlite.rs` — `Store` trait (`insert`, `exists`, `link`, `pending_embeddings`), `exists`'s NULL-safe probe, the writer-mutex single-writer lane, the `backup_temp_path`/rename helpers (reusable shape for `--output`) +- `crates/agent-memory-core/src/domain.rs` — `MemoryError`, `NewMemory`, `MemoryType`/`LinkKind::try_from`, `validate_*` helpers +- `crates/agent-memory-core/sql/0002_embeddings.sql` (the `meta` table the header reads) and `sql/0003_relations.sql` (the `memory_links` shape link records mirror) — both FROZEN; this phase ships **no migration** +- `crates/agent-memory/src/main.rs` — `Command::Import { from, path, scope }` + `run_import` (the `--from` dispatch gains `jsonl`; `Export` is a new variant); `crates/agent-memory/src/config.rs` — the `cfg(unix)` 0700 pattern D-02 mirrors for 0600 files +- `crates/agent-memory/src/mcp.rs` `map_mcp_error` + `crates/agent-memory/src/rest/handlers.rs` `map_memory_error` — unchanged by D-14 but must keep compiling exhaustively; `health_handler` is one of the surfaces D-14 covers +- `crates/agent-memory-core/tests/import.rs` + `tests/fixtures/STATE.md` — the idempotent-import proof shape to mirror for JSONL (non-empty-DB and overlapping-rowid kill-tests per PITFALLS 18) +- `README.md` — "Import from GSD", "REST API" (the search-only redaction sentence to widen), "Behavioral changes in v0.1.0", "Lifecycle" (backfill cadence the D-13 note points at) + + + + +## Existing Code Insights + +### Reusable Assets +- `MemoryService::import(drafts)` already does validate → dedup (`Store::exists`) → insert; the JSONL importer feeds a sibling path that adds key→id mapping, expiry skipping, and a links pass — same seam, same writer lane. +- `Store::exists(source: Option<&str>, mem_type, content)` with NULL-safe `source IS ?1` (CR-01) is the dedup probe to generalise into an id-returning lookup (D-07). +- `Store::link(from, to, kind, now)` already pre-checks endpoints and uses `INSERT OR IGNORE` on the `(from_id, to_id, kind)` primary key — link-record import is idempotent for free, and `now` lets the record's `created_at` be preserved. +- `redact_search_error` / `SEARCH_STORAGE_FAILURE` (service.rs:33-60): exhaustive-match redaction helper; D-14 parameterises it by operation. +- `backup_temp_path` + verify-then-rename in `sqlite.rs` (04-07): the atomic-write shape `--output` reuses. +- `config.rs` `cfg(unix)`/`cfg(not(unix))` `create_data_dir` (0700): the one-seam platform split to mirror for 0600 export files. +- `tests/import.rs` + `FakeEmbedder::failing()` harness: proves import succeeds and is keyword-searchable with Ollama dead — the exact fresh-machine story D-13 relies on. +- `meta` table (`embedding_model`, `embedding_dim`) — the header's source of truth (D-03). + +### Established Patterns +- Bad client input never surfaces as 500/`internal_error`; validation lives once at the `MemoryService` seam, transports are thin (Phase 3 D-06). Malformed-file errors are `InvalidArgument` in core even though the CLI is the only caller (D-01). +- Exhaustive `match` on `MemoryError` in both transport mappers and in the redaction helper — new variants fail to compile until classified. D-14 adds no variant. +- All SQL parameterised; export reads and import writes keep bound parameters; `kind`/`type` wire strings never reach SQL unparsed. +- Shipped migrations are frozen (fixture divergence test) and `LATEST_SCHEMA_VERSION = 3` — this phase adds no schema, so neither moves. +- Coverage-bearing tests run in-process (core + handler tests); spawned-binary tests are realism-only. The CLI `export`/`import` glue in `main.rs` should stay thin so the format logic is unit-tested in core. +- `cargo fmt` + `clippy --workspace --all-targets --locked -- -D warnings` + `cargo test --workspace --locked` green before every commit; no `unwrap()` in production code. + +### Integration Points +- `main.rs` `Command` enum: new `Export { output: Option }` variant; `Import` gains the `jsonl` branch in `run_import`'s `--from` match (the `gsd-state`-only bail becomes a two-arm match). +- `service.rs`: new `export()` (streaming read) and `import_jsonl()`-style entry (or an extended `import` taking typed records) beside `import()`; every public method's error path routed through `redact_storage_error`. +- `store/mod.rs` trait: `export_memories`/`export_links` streaming reads, id-returning dedup lookup, an insert that accepts preserved `created_at`/`access_count`, and a `meta` read for the header. +- `import/jsonl.rs` (new) beside `gsd_state.rs`; `export.rs` (new) in core holding the record structs (serde `Serialize + Deserialize`, `deny_unknown_fields`) that are the single definition of format v1. +- README: new "Export & import" section (format v1 spec, timestamp policy table, pending-embedding note, strict-vs-tolerant importer note), REST section redaction sentence widened, lifecycle section cross-reference. + + + + +## Specific Ideas + +- Error strings stay greppable and stable-prefixed (Phase 3 D-05): tests assert on `line 42:`-style prefixes and the field name, not on full equality. +- The `version > 1` refusal message names both versions and the fix, in the spirit of Phase 4 D-13: `file is format version 2 but this binary reads up to version 1 — upgrade agent-memory`. +- The round-trip kill-test (PITFALLS 18): export → import into a NON-empty DB whose rowids overlap the file's keys → assert no pre-existing memory gained a phantom edge, then export again and diff modulo `key`/`last_accessed`. +- The expiry kill-test (PITFALLS 19): a fixture with one expired, one ancient (retrievable via `list`, low decay) and one fresh row → report shows `skipped_expired: 1`, `imported: 2`. +- The redaction proof (D-14): one in-process test per surface class (a write path and `/health`) against a store whose underlying file is made unreadable or whose table is dropped, asserting the client body contains no `sqlite` substring while the log carries it — the T-04-19 pattern. + + + + +## Deferred Ideas + +- `--include-embeddings` export flag with enforced model/dim header check on import (FEATURES P2) — only when a same-model offline-restore use case appears. +- Export filters (`--scope`, `--type`) — additive; D-06 exports everything. +- MCP tool / REST endpoint for export or import — would need an in-body payload design and a stdout-purity story; D-01 keeps both CLI-only. +- `import --embed-now` or a one-shot `backfill` subcommand that drains pending embeddings immediately after a large restore — mitigates the D-13 latency (128 rows per hourly tick) without a second re-embed mechanism. +- Cross-tool import adapters (mem0 / Chroma JSONL dialects) — no standard exists; wait for demand (FEATURES). +- A stable per-memory identity column (UUID) for dedup that survives content edits — would need migration 0004 and changes the idempotency contract; D-07 keeps the content key. +- Cross-process exclusive lock on the pre-migration backup (04-REVIEW WR-01 residual) and the 04-REVIEW IN-02/IN-04/IN-05/IN-06 items — unchanged, still deferred; IN-05's `SchemaTooNew` re-probe fix is cheap if a planner wants to ride it along, but it is not Phase 5 scope. + + + +--- + +*Phase: 5-Portable Export & Import* +*Context gathered: 2026-10-25* diff --git a/.planning/phases/05-portable-export-import/05-DISCUSSION-LOG.md b/.planning/phases/05-portable-export-import/05-DISCUSSION-LOG.md new file mode 100644 index 0000000..648e6c5 --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-DISCUSSION-LOG.md @@ -0,0 +1,127 @@ +# Phase 5: Portable Export & Import - Discussion Log + +> **Audit trail only.** Do not use as input to planning, research, or execution agents. +> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered. + +**Date:** 2026-10-25 +**Phase:** 5-Portable Export & Import +**Mode:** `--auto` — the user delegated every gray area to the recommended option. No AskUserQuestion was issued; each row marked ✓ is the recommended default, chosen by Claude and logged here for audit. +**Areas discussed:** Surface & CLI shape, JSONL format v1, Import identity & edge remapping, Timestamp & lifecycle policy, Malformed-file policy & report, Embeddings on import, Write-path storage-error redaction + +--- + +## Surface & CLI shape + +| Option | Description | Selected | +|--------|-------------|----------| +| CLI-only (`export` / `import --from jsonl`) | File in, file out; core raises `InvalidArgument` so the taxonomy holds for any later transport | ✓ | +| CLI + MCP tools | `memory_export`/`memory_import` over stdio | | +| CLI + REST endpoints | `GET /api/export`, `POST /api/import` | | + +**Choice:** CLI-only (recommended). MCP export would push the corpus through the JSON-RPC stdout channel; REST import is an arbitrary-file-read primitive on an unauthenticated daemon (PITFALLS security row). +`[auto] Surface — Q: "Where do export/import live?" → Selected: "CLI-only" (recommended default)` + +| Option | Description | Selected | +|--------|-------------|----------| +| stdout default, `--output PATH` atomic + 0600 | Unix convention, pipe-friendly; file path gets temp+rename and owner-only mode | ✓ | +| `--output` required | Forces a file; refuses stdout | | +| Default path in the data dir | Guesses a location; risks a shared/tmp default | | + +`[auto] Surface — Q: "Export destination?" → Selected: "stdout default, --output atomic 0600" (recommended default)` + +## JSONL format v1 + +| Option | Description | Selected | +|--------|-------------|----------| +| Header record + `record`-tagged lines; model/dim from `meta`; derived fields excluded | `{"record":"header",…}` then memories then links; `key` = source id as export-local key | ✓ | +| No header, infer from first record | Smaller but unversioned — a shipped tool's export is a contract | | +| Include `decay_score`/`base_weight`/`embedding_status` | "Round-trip everything" | | + +`[auto] Format — Q: "Header and record shape?" → Selected: "versioned header, tagged records, derived fields excluded" (recommended default)` + +| Option | Description | Selected | +|--------|-------------|----------| +| Export everything present, no filters, importer owns expiry | One streaming query per record type | ✓ | +| `--scope`/`--type` filters in v1 | Partial exports | | +| Skip expired rows at export time | Policy split across both sides | | + +`[auto] Format — Q: "Export scope?" → Selected: "everything, no filters" (recommended default)` + +## Import identity & edge remapping + +| Option | Description | Selected | +|--------|-------------|----------| +| Reuse `(source, mem_type, content)` key; `exists` → id-returning lookup; no migration | Zero new schema; duplicate maps to existing row id | ✓ | +| New UUID identity column (migration 0004) | Survives content edits; changes the idempotency contract | | +| Preserve source ids | Collides in a non-empty DB (PITFALLS 18) | | + +`[auto] Identity — Q: "Idempotency key for JSONL rows?" → Selected: "existing content key" (recommended default)` + +| Option | Description | Selected | +|--------|-------------|----------| +| Two-pass validate-then-write: memories build key→id map, links through the map; expired-endpoint links dropped+counted; unknown key = malformed | PITFALLS 18 shape | ✓ | +| Single pass, edges inline | Edges can precede the dedup result of their endpoints | | +| Drop unknown-key links silently | Hides corruption | | + +`[auto] Identity — Q: "How are edges remapped?" → Selected: "two-pass through key→id map" (recommended default)` + +## Timestamp & lifecycle policy + +| Option | Description | Selected | +|--------|-------------|----------| +| `created_at` preserve; `expires_at` preserve + skip-and-count already-expired; `last_accessed` refresh to now; `access_count` preserve; link `created_at` preserve; skew → warning | PITFALLS 19 recommendation | ✓ | +| Preserve everything verbatim | Restored corpus arrives fully decayed and unfindable by rank; expired rows "eaten" by the sweep later | | +| Refresh everything to now | Loses provenance | | + +`[auto] Lifecycle — Q: "Per-column timestamp policy?" → Selected: "preserve created/expires, refresh last_accessed, skip expired" (recommended default)` + +## Malformed-file policy & report + +| Option | Description | Selected | +|--------|-------------|----------| +| Strict all-or-nothing `InvalidArgument` naming the line; `deny_unknown_fields`; nothing written on failure | Our own contract — half-restores are worse than refusals | ✓ | +| Tolerant skip-and-count (gsd-state style) | Silent data loss on a corrupted backup | | +| Best-effort: import valid rows, report bad ones | Partial state, non-idempotent recovery | | + +`[auto] Malformed — Q: "Reject or tolerate bad rows?" → Selected: "strict, all-or-nothing" (recommended default)` + +| Option | Description | Selected | +|--------|-------------|----------| +| Extend `ImportReport` additively (`skipped_expired`, `links_imported`, `links_skipped_duplicates`, `links_dropped`) | One report type; gsd path reports zeros | ✓ | +| Separate `JsonlImportReport` | Two shapes for one concept | | + +`[auto] Report — Q: "Report shape?" → Selected: "extend ImportReport" (recommended default)` + +## Embeddings on import + +| Option | Description | Selected | +|--------|-------------|----------| +| Never embed inline — rows land `embedding_status = 0`, sweep backfill re-embeds | Criterion 3 literal; one mechanism; CLI never blocks on Ollama | ✓ | +| Best-effort inline batch embed (gsd-state style) | Faster semantic availability, but blocks the CLI and contradicts the criterion's stated landing state | | +| Import then drain backfill in the CLI | Immediate availability; extra scope — deferred as `--embed-now` | | + +`[auto] Embeddings — Q: "Embed on import?" → Selected: "never inline; sweep backfill" (recommended default)` + +## Write-path storage-error redaction (user-approved scope addition) + +| Option | Description | Selected | +|--------|-------------|----------| +| Service-seam `redact_storage_error(op)` on every public `MemoryService` method | One seam covers MCP, REST (incl. `/health`) and the CLI; mirrors Phase 4's search redaction | ✓ | +| Redact in the two transport mappers | Two files, CLI uncovered (04-REVIEW IN-01's suggestion) | | +| Leave deferred | Contradicts the user's scope addition | | + +`[auto] Redaction — Q: "Where does write-path redaction live?" → Selected: "service seam" (recommended default)` + +--- + +## Claude's Discretion + +- `NewMemory` override fields vs a dedicated `ImportedMemory` + `Store::insert_imported`. +- Transaction batching for import passes; paging for the export read. +- Temp-file naming for `--output`; reuse of the backup temp-path helper. +- Whether the header carries `exported_rows`/`exported_links` counts. +- Rustdoc/README placement of the format spec; the optional `VACUUM INTO` backup tip. + +## Deferred Ideas + +- `--include-embeddings` with model/dim check; export filters; MCP/REST export/import; `import --embed-now` / `backfill` subcommand; cross-tool import dialects; UUID identity column (migration 0004); 04-REVIEW residuals (WR-01 cross-process lock, IN-02/IN-04/IN-05/IN-06). diff --git a/.planning/phases/05-portable-export-import/05-PATTERNS.md b/.planning/phases/05-portable-export-import/05-PATTERNS.md new file mode 100644 index 0000000..2dc91ea --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-PATTERNS.md @@ -0,0 +1,915 @@ +# Phase 5: Portable Export & Import - Pattern Map + +**Mapped:** 2026-10-25 +**Files analyzed:** 16 (7 new, 9 modified) +**Analogs found:** 15 / 16 (the one gap — a streaming boxed-writer export — has a partial analog only) + +All line numbers below were read from the working tree on 2026-10-25 and match the `[VERIFIED]` anchors in 05-RESEARCH.md. Every excerpt is quoted verbatim from the analog; nothing here is invented code. Where RESEARCH already prescribes the new shape (Patterns 1–8, Code Examples 1–7) this document says *where to copy the surrounding conventions from*, not what the new code is. + +## File Classification + +| New/Modified File | Role | Data Flow | Closest Analog | Match Quality | +|-------------------|------|-----------|----------------|---------------| +| `crates/agent-memory-core/src/export.rs` (NEW) | model (format v1 record structs + consts, module-doc spec) | transform (serde wire ↔ domain) | `crates/agent-memory/src/rest/handlers.rs:104-117` (`UpdateBody`, `deny_unknown_fields` DTO) + `domain.rs:176-183` (`LinkOutcome` derive style) | role-match | +| `crates/agent-memory-core/src/import/jsonl.rs` (NEW) | utility (pure parser/validator, no I/O) | transform (text → typed records) | `crates/agent-memory-core/src/import/gsd_state.rs` (whole file) | exact (same role, opposite tolerance policy by design) | +| `crates/agent-memory-core/src/import/mod.rs` (MOD) | module index | — | itself (`import/mod.rs:1-16`) | exact | +| `crates/agent-memory-core/src/service.rs` (MOD: `redact_storage_error`, `export`, `import_jsonl`, `ImportReport` +4) | service | request-response + batch (two writer hops) | itself: `redact_search_error` 46-60, `search`/`search_unredacted` 299-303, `import` 394-463, `sweep` 688-744 | exact | +| `crates/agent-memory-core/src/store/mod.rs` (MOD: `find_by_key`, `insert_imported`, `export_rows`, `embedding_pin`) | model (trait) | CRUD | itself: `exists` doc 81-91, `link` doc 174-194 | exact | +| `crates/agent-memory-core/src/store/sqlite.rs` (MOD: impls) | model (store impl) | CRUD + streaming read | itself: `insert` 596-644, `exists` 751-772, `pending_embeddings` 735-749, `list` 774-803, `link` 945-990, `row_to_view` 443-470 | exact | +| `crates/agent-memory-core/src/domain.rs` (MOD: `ImportedMemory`, `LinkOutcome::Linked { created }`) | model | — | itself: `NewMemory` 240-253, `LinkOutcome` 173-183, `validate_link_endpoints` 486-497 | exact | +| `crates/agent-memory/src/main.rs` (MOD: `Command::Export`, `jsonl` arm) | controller (CLI) | file-I/O + request-response | itself: `Command::Import` 80-97, dispatch 118-126, `run_import` 225-258 | exact | +| `crates/agent-memory/src/export_file.rs` (NEW, or in `config.rs`) | utility (atomic 0600 file write) | file-I/O | `crates/agent-memory/src/config.rs:37-54` (cfg split) + `sqlite.rs:140-159, 265-291` (temp+rename discipline) | role-match | +| `crates/agent-memory-core/tests/export_import.rs` (NEW) | test (integration, in-process) | batch | `tests/relations.rs:33-102` (harness + raw helpers) + `tests/import.rs:34-99` (idempotency proof shape) | exact | +| `crates/agent-memory-core/tests/redaction.rs` (NEW) | test (integration, fault-injection) | request-response | `tests/hybrid.rs:712-736` | exact | +| `crates/agent-memory-core/src/export.rs` `#[cfg(test)]` goldens | test (unit) | transform | `gsd_state.rs:188-278` (in-module test style) | exact | +| `crates/agent-memory/src/rest/handlers.rs` test module (MOD: `/health` + store redaction tests) | test (in-process handler) | request-response | itself `handlers.rs:851-889` | exact | +| `crates/agent-memory/src/mcp.rs` test module (MOD: `memory_store` redaction test) + `LinkOutcome` match 437-444 | test / transport | request-response | itself `mcp.rs:1309-1344`, `mcp.rs:437-444` | exact | +| `crates/agent-memory/tests/cli.rs` (NEW, realism-only) | test (spawned binary) | file-I/O | `tests/tools.rs:12-44` + `tests/rest.rs:18-50` | role-match (those spawn long-running servers; CLI test runs one-shot subcommands) | +| `README.md` (MOD: "Export & import", widen redaction sentence 460-462) | docs | — | README 513-532 ("Import from GSD") | exact | + +## Pattern Assignments + +### `crates/agent-memory-core/src/import/jsonl.rs` (utility, transform) — the parser + +**Analog:** `crates/agent-memory-core/src/import/gsd_state.rs` — the sibling importer. Copy its *shape* (module doc, pure function, `text.lines()` loop, result struct, in-module tests); invert its *policy* (strict `Result<_, MemoryError>` instead of skip-counts, per D-11). + +**Module-doc pattern** (`gsd_state.rs:1-16`) — state the contract and the tolerance rule up front; the jsonl module doc should say the opposite explicitly: +```rust +//! Tolerant line-scanner parser for GSD `STATE.md` files (INTEROP-01). +//! ... +//! Tolerance rules (T-02-20/T-02-22): headings inside fenced code blocks never +//! change the current section; unknown sections are ignored; malformed bullets +//! and oversized items are skipped and counted; a garbage file yields zero +//! drafts and never panics. Pure function, no I/O — the CLI reads the file. +``` + +**Imports pattern** (`gsd_state.rs:18`) — core-internal `crate::domain` paths, nothing else: +```rust +use crate::domain::{MemoryType, NewMemory}; +``` +For jsonl add `MemoryError`, `LinkKind`, `validate_link_endpoints` (it is `pub(crate)`, `domain.rs:490`) and `crate::export::{Record, FORMAT_NAME, FORMAT_VERSION}`. + +**Result-struct pattern** (`gsd_state.rs:33-41`) — a public `#[derive(Debug)]` struct with doc'd pub fields, consumed by the service: +```rust +/// Result of parsing one STATE.md: the typed drafts plus the count of lines +/// that were recognized as items but skipped (malformed or oversized). +#[derive(Debug)] +pub struct ParsedStateFile { + /// Drafts ready for [`crate::service::MemoryService::import`]. + pub drafts: Vec, + /// Malformed or oversized items skipped by the tolerance rules. + pub skipped: usize, +} +``` +jsonl equivalent: `pub struct JsonlFile { pub header: ..., pub memories: Vec<...>, pub links: Vec<...> }` (RESEARCH Pattern 3). No `skipped` counter — strictness means parse errors abort. + +**Core line-loop pattern** (`gsd_state.rs:61-69, 161-164`) — single pass over `text.lines()`; jsonl adds `.enumerate()` for the 1-based counter (RESEARCH Pitfall 1): +```rust +pub fn parse_gsd_state(text: &str, scope: Option) -> ParsedStateFile { + let mut drafts = Vec::new(); + let mut skipped = 0usize; + ... + for line in text.lines() { + let trimmed = line.trim(); +``` + +**Validation-at-the-seam pattern** — the wire strings must go through the existing `try_from`s, whose messages are the established client-tier text (`domain.rs:158-171`): +```rust +impl std::convert::TryFrom<&str> for LinkKind { + type Error = MemoryError; + + fn try_from(s: &str) -> Result { + match s { + "relates_to" => Ok(LinkKind::RelatesTo), + "supersedes" => Ok(LinkKind::Supersedes), + "caused_by" => Ok(LinkKind::CausedBy), + other => Err(MemoryError::InvalidArgument(format!( + "kind must be one of relates_to, supersedes, caused_by (got '{other}')" + ))), + } + } +} +``` +and the self-link + empty-content wording to reuse (`domain.rs:490-497`, `domain.rs:475-478`): +```rust +pub(crate) fn validate_link_endpoints(from_id: i64, to_id: i64) -> Result<(), MemoryError> { + if from_id == to_id { + return Err(MemoryError::InvalidArgument(format!( + "from_id and to_id must differ (got {from_id})" + ))); + } + Ok(()) +} +``` +```rust + return Err(MemoryError::InvalidArgument( + "content must be a non-empty string".to_string(), + )); +``` +Wrap each as `MemoryError::InvalidArgument(format!("line {line_no}: {inner}"))` — `InvalidArgument`'s Display is `"invalid argument: {0}"` (`domain.rs:389-390`), giving D-11's `invalid argument: line 42: …`. + +**In-module test pattern** (`gsd_state.rs:188-200`) — `#[cfg(test)] mod tests { use super::*; fn parse(...) }` with one behaviour per `#[test]`, descriptive snake_case names: +```rust +#[cfg(test)] +mod tests { + use super::*; + + fn parse(text: &str) -> ParsedStateFile { + parse_gsd_state(text, Some("test-scope".to_string())) + } + + #[test] + fn decisions_bullets_map_to_decision_drafts_with_provenance() { + let parsed = parse("### Decisions\n\n- first decision\n- second decision\n"); + assert_eq!(parsed.drafts.len(), 2); +``` + +--- + +### `crates/agent-memory-core/src/import/mod.rs` (module index) + +**Analog:** itself (`import/mod.rs:1-16`). Add `pub mod jsonl;` after `pub mod gsd_state;` and amend the module doc: the first invariant currently reads "**Tolerant**" for *every* importer — it must become "gsd-state is tolerant; jsonl is strict (D-11)" or the doc contradicts the new module: +```rust +//! Invariants every importer guards: +//! +//! - **Tolerant**: messy input degrades to skip-counts — ... +//! - **Idempotent**: importers produce [`crate::domain::NewMemory`] drafts that +//! [`crate::service::MemoryService::import`] dedups against the store on the +//! exact `(source, mem_type, content)` key, so re-running an import never +//! duplicates. +//! - **Data-only**: imported content is stored strictly as data. ... + +pub mod gsd_state; +``` + +--- + +### `crates/agent-memory-core/src/export.rs` (model, transform) — format v1 definition + +**Analog (DTO strictness):** `crates/agent-memory/src/rest/handlers.rs:104-110` — the 04-07 `deny_unknown_fields` precedent, including the doc sentence explaining *why* unknown keys are rejected: +```rust +/// Unknown keys are rejected (`deny_unknown_fields`, REVIEW WR-02), exactly as +/// on the MCP DTO: the immutable `type`/`source` and typos such as `ttl_sec` +/// make the whole PATCH a 400 naming the field, and nothing is written. A +/// silently dropped merge-patch member would be a silent partial write. +#[derive(Debug, serde::Deserialize)] +#[serde(deny_unknown_fields)] +pub struct UpdateBody { +``` + +**Analog (enum derive + lowercase wire tags):** `domain.rs:188-193`: +```rust +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +#[serde(rename_all = "lowercase")] +pub enum LinkDirection { + Out, + In, +} +``` +The `Record` enum shape itself is RESEARCH Pattern 1 (`#[serde(tag = "record", rename_all = "lowercase", deny_unknown_fields)]`, `String` for `type`/`kind`, `#[serde(rename = "type")] mem_type`). Keep `pub const FORMAT_NAME` / `FORMAT_VERSION` beside the enum the way `gsd_state.rs:22` keeps `pub const MAX_IMPORT_CONTENT_BYTES` beside its parser. + +**Registration:** add `pub mod export;` to `crates/agent-memory-core/src/lib.rs:7-13` (alphabetical: after `embed`, before `import`). + +**Golden test:** RESEARCH Code Example 1, in a `#[cfg(test)] mod tests` following `gsd_state.rs:188-194`. + +--- + +### `crates/agent-memory-core/src/service.rs` (service) — `redact_storage_error`, `export`, `import_jsonl`, `ImportReport` + +**Analog:** itself. + +**D-14 redaction helper to generalise** (`service.rs:33-60`) — keep the constant, add an `op: &str` parameter, keep the exhaustive match with no catch-all: +```rust +pub const SEARCH_STORAGE_FAILURE: &str = + "search failed due to a storage error (details are in the server log)"; + +/// Tier every search-path error for the wire — the ONE place this happens. +/// ... +/// No catch-all arm: a new [`MemoryError`] variant must be tiered here at +/// compile time. +fn redact_search_error(e: MemoryError) -> MemoryError { + match e { + MemoryError::InvalidType(_) + | MemoryError::InvalidQuery(_) + | MemoryError::InvalidArgument(_) => e, + MemoryError::Internal(_) | MemoryError::SchemaTooNew { .. } => e, + MemoryError::Sqlite(_) + | MemoryError::Pool(_) + | MemoryError::Join(_) + | MemoryError::Migration(_) => { + tracing::error!(error = %e, "search failed with a storage error"); + MemoryError::Internal(SEARCH_STORAGE_FAILURE.to_string()) + } + } +} +``` +New body: `MemoryError::Internal(format!("{op} failed due to a storage error (details are in the server log)"))` — byte-identical to the constant when `op == "search"` (RESEARCH Pattern 8). Add a unit test pinning that equality. + +**Public/unredacted split to apply to every method** (`service.rs:299-303`) — the inner-fn shape, because the bodies have multiple `?` exits: +```rust + pub async fn search(&self, args: SearchArgs) -> Result { + self.search_unredacted(args) + .await + .map_err(redact_search_error) + } +``` +Apply to `store` 209-232, `list` 244-265, `import` 394-463, `forget` 573-578, `link` 592-604, `unlink` 610-620, `update` 634-666, `sweep` 688-744, and the two new methods. + +**Current `spawn_blocking` hop idiom** (`service.rs:228-231`, every method uses it; `?` after `map_err(Join)` is doubled when the closure returns `Result`): +```rust + let store = self.store.clone(); + tokio::task::spawn_blocking(move || store.insert(new, embedding, now)) + .await + .map_err(MemoryError::Join)? +``` + +**`import()` — the sibling `import_jsonl` copies this structure** (`service.rs:394-463`): one clock read, hop 1 dedup on the read pool, hop 3 inserts on the writer lane, counts assembled in one `ImportReport`: +```rust + pub async fn import(&self, drafts: Vec) -> Result { + for draft in &drafts { + crate::domain::validate_ttl(draft.ttl_secs)?; + } + let now = self.clock.now(); + + // (1) Partition new vs duplicate in one blocking hop (read pool). + let store = self.store.clone(); + let (fresh, skipped_duplicates) = + tokio::task::spawn_blocking(move || -> Result<(Vec, usize), MemoryError> { + let mut fresh = Vec::new(); + let mut duplicates = 0usize; + for draft in drafts { + // Pass the source through as-is: a `None` must probe for + // NULL, exactly what the insert stores (CR-01). + if store.exists(draft.source.as_deref(), draft.mem_type, &draft.content)? { + duplicates += 1; + } else { + fresh.push(draft); + } + } + Ok((fresh, duplicates)) + }) + .await + .map_err(MemoryError::Join)??; + ... + // (3) Insert every new draft through the standard store path. + let store = self.store.clone(); + let imported = tokio::task::spawn_blocking(move || -> Result { + let mut imported = 0usize; + for (i, draft) in fresh.into_iter().enumerate() { + let embedding = embeddings.as_ref().map(|v| v[i].clone()); + store.insert(draft, embedding, now)?; + imported += 1; + } + Ok(imported) + }) + .await + .map_err(MemoryError::Join)??; + + Ok(ImportReport { + imported, + skipped_duplicates, + }) + } +``` +Differences for `import_jsonl` (D-08/D-13): **no step (2) embed call at all**; dedup and insert happen per record in the same pass so the `key → id` map is built in order (`find_by_key` → `Some(id)` maps, else `insert_imported`); a third hop walks links through `store.link(a, b, kind, record.created_at)`. Line 413 (`store.exists(...)`) becomes `.find_by_key(...)?.is_some()` here. + +**`ImportReport` to extend additively** (`service.rs:159-168`) — `Copy + serde::Serialize`, doc comment per field; the two constructors at 425-428 and 459-462 are the only literals (RESEARCH Pitfall 11), so adding four `usize` fields with explicit `0` there breaks no test: +```rust +#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize)] +pub struct ImportReport { + /// Drafts inserted as new memories. + pub imported: usize, + /// Drafts skipped because an identical `(source, mem_type, content)` row + /// already exists — the idempotency guarantee. + pub skipped_duplicates: usize, +} +``` + +**`sweep` backfill — the D-13 mechanism the import relies on, untouched** (`service.rs:698-701, 747-749`): +```rust + let store = self.store.clone(); + let pending = tokio::task::spawn_blocking(move || store.pending_embeddings(BACKFILL_BATCH)) + .await + .map_err(MemoryError::Join)??; +``` +```rust +const BACKFILL_BATCH: usize = 128; +``` + +**Clock-skew warning + crate log idiom** (`service.rs:224`, `218-220`): `tracing::warn!("embedding failed ({e}); storing with embedding_status = 0");` — use the same `tracing::warn!` for the `exported_at > now + 3600` case (RESEARCH Open Question 2 recommends emitting it in core). + +--- + +### `crates/agent-memory-core/src/store/mod.rs` (trait) — `find_by_key`, `insert_imported`, `export_rows`, `embedding_pin` + +**Analog:** itself. Trait-method doc style: explain the invariant, name the review finding, state the NULL/edge semantics. + +**`exists` doc to carry over to `find_by_key`** (`store/mod.rs:81-91`) — keep the CR-01 paragraph verbatim, change the return type sentence: +```rust + /// The INTEROP-01 idempotency probe: does a row with exactly this + /// `(source, mem_type, content)` key already exist? `MemoryService::import` + /// checks it before every insert so re-importing the same file is a no-op. + /// `source` is `Option` and compared NULL-safely: a `None` source matches a + /// stored NULL source, never an empty string (CR-01). + fn exists( + &self, + source: Option<&str>, + mem_type: MemoryType, + content: &str, + ) -> Result; +``` + +**`insert` doc + signature to mirror for `insert_imported`** (`store/mod.rs:21-32`): +```rust + /// Insert a new memory, returning its assigned id. `now` is the injected + /// timestamp used for `created_at`/`last_accessed`. When `embedding` is + /// `Some`, the memories INSERT, the `vec_memories` vector INSERT, and + /// `embedding_status = 1` all happen in ONE writer transaction; when `None`, + /// the row lands with `embedding_status = 0` (pending — backfilled by the + /// sweep once the embedder recovers, SEARCH-03). + fn insert( + &self, + new: NewMemory, + embedding: Option>, + now: i64, + ) -> Result; +``` +`insert_imported(&self, m: ImportedMemory, now: i64) -> Result` takes no embedding parameter — document "ALWAYS lands `embedding_status = 0` (D-13)". + +**`link` doc — the idempotent `INSERT OR IGNORE` contract the links pass relies on** (`store/mod.rs:174-194`); its sentence "A duplicate of the stored row is an idempotent no-op that still returns `LinkOutcome::Linked`" must be reworded for `Linked { created: false }`: +```rust + /// The row is stored under [`LinkKind::canonical_endpoints`]: `relates_to` + /// as (smaller id, larger id), so linking b→a after a→b is the same + /// symmetric relation (REVIEW WR-03); `supersedes` / `caused_by` as given + /// (D-03 directed identity). A duplicate of the stored row is an idempotent + /// no-op that still returns [`LinkOutcome::Linked`]. ... + fn link( + &self, + from_id: i64, + to_id: i64, + kind: LinkKind, + now: i64, + ) -> Result; +``` + +**Streaming read-pool method precedent** (`store/mod.rs:75-79`, `pending_embeddings`): returns a `Vec` today; `export_rows` should instead take a visitor closure (RESEARCH Pattern 4) — document "one pooled connection, one read transaction, memories ascending id then links" in the trait doc as Pitfall 5 asks. + +**Expiry predicate to document the boundary against** (`store/mod.rs:219`): the sweep deletes `expires_at < now` (strict); import skips `expires_at <= now` (D-10). State both in `insert_imported`'s doc. + +--- + +### `crates/agent-memory-core/src/store/sqlite.rs` (store impl) — `find_by_key`, `insert_imported`, `export_rows`, `embedding_pin`, `link { created }` + +**Analog:** itself. + +**Imports convention (already present at top of file):** `params!`, `OptionalExtension` (`.optional()` is used at 770 and 966), `Connection`, `Backup`. Nothing new needed. + +**Writer-lane transaction shape** (`sqlite.rs:607-614, 642-643`) — every write starts exactly like this; `insert_imported` copies it: +```rust + // ONE writer transaction covers the memories row, the vector row, and + // the status flag, so `vec_memories` can never hold a vector for a row + // that failed to insert (T-02-05). + let mut conn = self + .writer + .lock() + .map_err(|_| MemoryError::Internal("writer lock poisoned".into()))?; + let tx = conn.transaction()?; + ... + tx.commit()?; + Ok(id) +``` + +**The INSERT to share via a private `insert_row` helper** (`sqlite.rs:602-605, 615-632`) — note `?7` is bound once for both `created_at` and `last_accessed`; `insert_imported` binds them separately and binds `access_count` instead of literal `0`: +```rust + let tags_json = serde_json::to_string(&new.tags).unwrap_or_else(|_| "[]".to_string()); + let base_weight = if new.mem_type.is_pinned() { 2.0 } else { 1.0 }; + let expires_at = new.ttl_secs.map(|ttl| now + ttl); + let embedding_status: i64 = i64::from(embedding.is_some()); + ... + tx.execute( + "INSERT INTO memories \ + (mem_type, content, tags, source, scope, base_weight, decay_score, \ + access_count, created_at, last_accessed, expires_at, embedding_status) \ + VALUES (?1, ?2, ?3, ?4, ?5, ?6, 1.0, 0, ?7, ?7, ?8, ?9)", + params![ + new.mem_type.as_wire_str(), + new.content, + tags_json, + new.source, + new.scope, + base_weight, + now, + expires_at, + embedding_status, + ], + )?; + let id = tx.last_insert_rowid(); +``` + +**`exists` → `find_by_key`** (`sqlite.rs:751-772`) — keep the comment and `IS ?1`; change `SELECT 1` to `SELECT id`, the row mapper to `|row| row.get::<_, i64>(0)`, and return `Ok(found)`: +```rust + let conn = self.reads.get().map_err(MemoryError::Pool)?; + // The INTEROP-01 dedup probe: all three key values bound, never + // formatted into the SQL (T-02-20). `source IS ?1` is SQLite's + // NULL-safe equality: a `None` source binds NULL and matches the NULL + // the insert stored (`source = ?1` would be NULL = NULL → never true, + // which made every source-less re-import a duplicate — CR-01). + let found = conn + .query_row( + "SELECT 1 FROM memories \ + WHERE source IS ?1 AND mem_type = ?2 AND content = ?3 LIMIT 1", + params![source, mem_type.as_wire_str(), content], + |_| Ok(()), + ) + .optional()?; + Ok(found.is_some()) +``` + +**Read-pool `query_map` streaming shape for `export_rows`** (`sqlite.rs:735-749`) — same `prepare` + `query_map` + `for row in rows`, but call the visitor instead of `out.push`: +```rust + fn pending_embeddings(&self, limit: usize) -> Result, MemoryError> { + let conn = self.reads.get().map_err(MemoryError::Pool)?; + let mut stmt = conn.prepare( + "SELECT id, content FROM memories \ + WHERE embedding_status = 0 ORDER BY id LIMIT ?1", + )?; + let rows = stmt.query_map(params![limit as i64], |row| { + Ok((row.get::<_, i64>(0)?, row.get::<_, String>(1)?)) + })?; + let mut out = Vec::new(); + for row in rows { + out.push(row?); + } + Ok(out) + } +``` +The memories SELECT column list to start from is `list`'s (`sqlite.rs:782-784`); export needs `id, mem_type, content, tags, source, scope, created_at, last_accessed, access_count, expires_at` (no `base_weight`/`decay_score`, D-04). Row mapping convention: `row_to_view` (`sqlite.rs:443-470`) — named columns via `row.get("col")`, tags via `serde_json::from_str(&tags_json).unwrap_or_default()`, and an invalid stored `mem_type` reported as `rusqlite::Error::FromSqlConversionFailure` (internal tier, never client). Export can write `mem_type` as the stored string directly since the record field is `String`. + +**`link` — add `created`** (`sqlite.rs:982-989`): read `tx.changes()` *before* `commit`: +```rust + let (stored_from, stored_to) = kind.canonical_endpoints(from_id, to_id); + tx.execute( + "INSERT OR IGNORE INTO memory_links (from_id, to_id, kind, created_at) \ + VALUES (?1, ?2, ?3, ?4)", + params![stored_from, stored_to, kind.as_wire_str(), now], + )?; + tx.commit()?; + Ok(LinkOutcome::Linked) +``` +The links SELECT for export mirrors this column list: `SELECT from_id, to_id, kind, created_at FROM memory_links ORDER BY from_id, to_id, kind`. + +**`meta` pin read for `embedding_pin`:** RESEARCH Code Example 5 (`conn.query_row("SELECT value FROM meta WHERE key = 'embedding_model'", [], |r| r.get(0))?`), on the read pool like `pending_embeddings`. + +--- + +### `crates/agent-memory-core/src/domain.rs` (model) — `ImportedMemory`, `LinkOutcome::Linked { created }` + +**Analog:** itself. + +**`NewMemory` — the struct `ImportedMemory` sits beside and mirrors** (`domain.rs:240-253`); do NOT extend it (15 construction sites, RESEARCH Pattern 5): +```rust +/// Input shape for storing a new memory (D-05). +/// +/// Only `content` and `mem_type` are conceptually required at the service boundary; +/// the remaining fields carry sensible defaults supplied by the caller +/// (`tags = []`, `source`/`scope` = `None`, `ttl_secs` = `None`). +#[derive(Debug, Clone)] +pub struct NewMemory { + pub content: String, + pub mem_type: MemoryType, + pub tags: Vec, + pub source: Option, + pub scope: Option, + pub ttl_secs: Option, +} +``` +`ImportedMemory` replaces `ttl_secs` with absolute `created_at: i64, access_count: i64, expires_at: Option` (RESEARCH Pattern 5 shape). + +**`LinkOutcome` to change** (`domain.rs:173-183`) — `Linked` becomes `Linked { created: bool }`; keep `Copy`: +```rust +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum LinkOutcome { + /// The edge exists now — freshly created, or already present (a duplicate + /// link is an idempotent no-op success). + Linked, + /// No memory with `missing_id` exists; nothing was written. + NotFound { missing_id: i64 }, +} +``` +Sites to update (grep-verified in RESEARCH 6b): `sqlite.rs:989`, `mcp.rs:438`, `handlers.rs:370`, `tests/relations.rs:139,149,791,794`. Transport matches become `LinkOutcome::Linked { .. } =>` — the wire JSON at `mcp.rs:438-443` is unchanged: +```rust + let status = match outcome { + LinkOutcome::Linked => serde_json::json!({ + "from_id": args.from_id, + "to_id": args.to_id, + "kind": kind, + "linked": true + }), +``` + +--- + +### `crates/agent-memory/src/main.rs` (CLI controller) — `Command::Export`, `jsonl` arm + +**Analog:** itself. + +**Subcommand declaration pattern** (`main.rs:80-97`) — clap derive, doc comments become `--help` text, `#[arg(long)]` for flags, positional `PathBuf`: +```rust + /// Import memories from an external file (INTEROP-01). Re-running the same + /// import is a no-op: duplicates are skipped on the exact + /// (source, type, content) key. + Import { + /// Source format. Supported: gsd-state (a GSD .planning/STATE.md file). + #[arg(long)] + from: String, + + /// Path to the file to import. + path: PathBuf, + ... + }, +``` +Add `Export { #[arg(long)] output: Option }` and update the `from` doc to "Supported: gsd-state, jsonl". + +**Dispatch pattern** (`main.rs:118-126`): +```rust + match cli.command { + Command::Serve => serve(cli.db, cli.ollama_url).await, + Command::ServeRest { addr, allow_remote } => { + serve_rest(cli.db, cli.ollama_url, addr, allow_remote).await + } + Command::Import { from, path, scope } => { + run_import(cli.db, cli.ollama_url, from, path, scope).await + } + } +``` + +**One-shot subcommand body — `run_export` copies this wiring** (`main.rs:225-258`): `anyhow::Context` on every edge, `read_to_string`, open store, build service, print report to **stdout** (sanctioned for CLI subcommands per the doc at 221-224): +```rust +async fn run_import( + db_flag: Option, + ollama_url: String, + from: String, + path: PathBuf, + scope: Option, +) -> anyhow::Result<()> { + if from != "gsd-state" { + anyhow::bail!("unsupported --from value {from:?} (supported formats: gsd-state)"); + } + + let text = + std::fs::read_to_string(&path).with_context(|| format!("reading {}", path.display()))?; + let scope = scope.unwrap_or_else(|| default_scope_for(&path)); + let parsed = agent_memory_core::import::gsd_state::parse_gsd_state(&text, Some(scope)); + + let db_path = resolve_db_path(db_flag).context("resolving database path")?; + let store = SqliteStore::open(&db_path) + .with_context(|| format!("opening database at {}", db_path.display()))?; + let embedder: Arc = Arc::new(OllamaClient::new(ollama_url)); + let service = MemoryService::new( + Arc::new(store), + Arc::new(SystemClock), + embedder, + DecayConfig::default(), + ); + + let report = service.import(parsed.drafts).await.context("importing")?; + println!( + "imported: {}, skipped duplicates: {}, malformed lines skipped: {}", + report.imported, report.skipped_duplicates, parsed.skipped + ); + Ok(()) +} +``` +The `if from != "gsd-state" { bail }` at 232-234 becomes a `match from.as_str() { "gsd-state" => ..., "jsonl" => ..., other => bail!(... "supported formats: gsd-state, jsonl") }`. **For `export` the report line must go to `eprintln!`** — stdout may be the file (RESEARCH Code Example 4). Keep the binary's `env!("CARGO_PKG_VERSION")` as the header version source (Pitfall 9). + +--- + +### `crates/agent-memory/src/export_file.rs` (utility, file-I/O) — atomic 0600 write + +**Analog A (platform split):** `crates/agent-memory/src/config.rs:37-54` — the one-seam `cfg(unix)` / `cfg(not(unix))` pair with the Windows-deferral comment; copy the structure for `create_private`: +```rust +/// Create `dir` (and parents) if missing, restricting it to the owner on Unix. +#[cfg(unix)] +fn create_data_dir(dir: &std::path::Path) -> std::io::Result<()> { + use std::os::unix::fs::DirBuilderExt; + if dir.exists() { + return Ok(()); + } + std::fs::DirBuilder::new() + .recursive(true) + .mode(0o700) + .create(dir) +} + +/// Non-Unix fallback (Windows support is deferred to v2 — DIST-03). +#[cfg(not(unix))] +fn create_data_dir(dir: &std::path::Path) -> std::io::Result<()> { + std::fs::create_dir_all(dir) +} +``` +Also copy `config.rs:1-5`'s module-doc sentence pattern ("This is the ONE place that touches platform paths … Windows support is a single-file change"). + +**Analog B (temp naming + rename + cleanup discipline):** `crates/agent-memory-core/src/store/sqlite.rs:136-159, 265-291` — pid-qualified temp in the *same directory*, best-effort removal that never masks the original error, rename as the last step: +```rust +/// The in-flight temp file for this process's backup of `file_name`: +/// `{db}.backup-tmp-{NNNN}-{pid}`. ... +fn backup_temp_path(dir: &Path, file_name: &str) -> PathBuf { + dir.join(format!( + "{file_name}.backup-tmp-{LATEST_SCHEMA_VERSION:04}-{}", + std::process::id() + )) +} + +/// Best-effort removal of a backup temp file and any SQLite sidecar it left. +/// Never fails: a leftover temp is harmless (T-04-32) and must not mask the +/// error that led here. +fn discard_backup_temp(temp: &Path) { + for suffix in ["", "-journal", "-wal", "-shm"] { + let path = PathBuf::from(format!("{}{suffix}", temp.display())); + if let Err(e) = std::fs::remove_file(&path) { + if e.kind() != std::io::ErrorKind::NotFound { + tracing::warn!(temp = %path.display(), error = %e, "could not remove backup temp file"); + } + } + } +} +``` +```rust + let copied_version = match copy_into_temp(src, &temp_path) { + Ok(version) => version, + Err(what) => { + discard_backup_temp(&temp_path); + return Err(fail(what)); + } + }; + ... + if let Err(e) = std::fs::rename(&temp_path, &final_path) { + discard_backup_temp(&temp_path); + return Err(fail(format!( + "cannot move {} into place as {}: {e}", + temp_path.display(), + final_path.display() + ))); + } +``` +These are private fns in core and specific to SQLite sidecars — **reuse the shape, not the code** (RESEARCH Pattern 7 verdict). The export helper needs only `""` (no sidecars) and `OpenOptions::new().write(true).create_new(true).mode(0o600)`. + +**Unit-test style for the helper** (`config.rs:56-67`) — plain `#[test]` with `tempfile::tempdir()`; add a `#[cfg(unix)]` test asserting `PermissionsExt::mode() & 0o777 == 0o600` (Pitfall 12: this is where the coverage for the file helper comes from, not the spawned CLI test): +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn flag_takes_precedence_over_env_and_default() { + let dir = tempfile::tempdir().expect("tempdir"); + let explicit = dir.path().join("explicit").join("memory.db"); + let resolved = resolve_db_path(Some(explicit.clone())).expect("resolve"); + assert_eq!(resolved, explicit); + assert!(explicit.parent().expect("parent").exists()); + } +``` + +--- + +### `crates/agent-memory-core/tests/export_import.rs` (integration test) + +**Analog A (harness + raw-connection helpers):** `crates/agent-memory-core/tests/relations.rs:20-102` — copy verbatim (`Harness`, `harness()`, `harness_with()`, `new_memory`, `store`, `raw_conn`, `edge_count`, `edges_touching`): +```rust +use agent_memory_core::clock::{Clock, TestClock}; +use agent_memory_core::decay::DecayConfig; +use agent_memory_core::domain::{ ... MemoryError, MemoryType, MemoryView, NewMemory, ... }; +use agent_memory_core::embed::{Embedder, FakeEmbedder}; +use agent_memory_core::service::{ListArgs, MemoryService, SearchArgs, SearchMode}; +use agent_memory_core::store::sqlite::{register_vec_extension, SqliteStore}; + +struct Harness { + service: MemoryService, + clock: Arc, + db_path: PathBuf, + _dir: tempfile::TempDir, +} + +fn harness() -> Harness { + harness_with(Arc::new(FakeEmbedder::failing())) +} + +fn harness_with(embedder: Arc) -> Harness { + let dir = tempfile::tempdir().expect("tempdir"); + let db_path = dir.path().join("memory.db"); + let store = SqliteStore::open(&db_path).expect("open store"); + let clock = Arc::new(TestClock::new(1_000)); + let service = MemoryService::new( + Arc::new(store), + clock.clone() as Arc, + embedder, + DecayConfig::default(), + ); + Harness { service, clock, db_path, _dir: dir } +} + +fn raw_conn(db_path: &Path) -> rusqlite::Connection { + register_vec_extension().expect("vec0 registers"); + rusqlite::Connection::open(db_path).expect("open raw connection") +} + +/// Edge rows touching `id` in either direction. +fn edges_touching(db_path: &Path, id: i64) -> i64 { + raw_conn(db_path) + .query_row( + "SELECT count(*) FROM memory_links WHERE from_id = ?1 OR to_id = ?1", + [id], + |row| row.get(0), + ) + .expect("count edges touching id") +} +``` +Two harnesses (source + target DB) are needed for the round-trip — `harness()` is already a factory, call it twice. `TestClock` drives `exported_at`, the expiry fixture and the D-10 `last_accessed` refresh. + +**Analog B (idempotent-import proof shape):** `tests/import.rs:34-99` — first import asserts counts, list-by-type counts, keyword search with a *failing* embedder proves fresh-machine recall, second import asserts `imported == 0` and `skipped_duplicates == first.imported`: +```rust + // First import: everything lands, nothing deduped. + let first = service.import(parsed.drafts).await.expect("first import"); + assert_eq!(first.imported, 9); + assert_eq!(first.skipped_duplicates, 0); + ... + assert_eq!(outcome.search_mode, SearchMode::Keyword); + ... + // Idempotency (phase success criterion 4): a re-parse + re-import of the + // SAME file imports nothing and reports every draft as a duplicate. + let reparsed = parse_gsd_state(FIXTURE, Some("fixture".to_string())); + let second = service.import(reparsed.drafts).await.expect("second import"); + assert_eq!(second.imported, 0, "re-run must import nothing"); + assert_eq!(second.skipped_duplicates, first.imported); +``` +Extend `tests/import.rs` itself with one assertion that the gsd path reports zeros for the four new fields (RESEARCH test map row). + +**Export-to-string helper for tests:** pass `Box::new(Vec::::new())` is not retrievable after the move — use a `Box` over an `Arc>>` wrapper, or export to a `tempfile` path and `read_to_string` it (simplest; `tempfile` is a dev-dep of core, `Cargo.toml:32`). + +--- + +### `crates/agent-memory-core/tests/redaction.rs` (fault-injection test) + +**Analog:** `crates/agent-memory-core/tests/hybrid.rs:712-736` — the T-04-19 recipe: rename a table through a raw side connection, call the service, assert the exact `Internal` text and no `sqlite` substring: +```rust +#[tokio::test] +async fn a_broken_fts_index_in_keyword_mode_is_the_generic_internal_error() { + let h = harness(Arc::new(FakeEmbedder::failing())); + store(&h, "ledger reconciliation runbook", MemoryType::Pattern).await; + raw_conn(&h.db_path) + .execute_batch("ALTER TABLE memories_fts RENAME TO memories_fts_offline;") + .expect("rename fts table"); + + let err = h + .service + .search(query("ledger", None)) + .await + .expect_err("a broken FTS index must fail keyword search"); + assert!( + matches!(err, MemoryError::Internal(ref m) if m == SEARCH_STORAGE_FAILURE), + "keyword mode must surface the generic internal error, got {err:?}" + ); + assert!( + !err.to_string().to_lowercase().contains("sqlite"), + "no SQLite text may reach the client: {err}" + ); +} +``` +For write paths rename `memories` (→ `store`, `list`, `update`, `forget`, `import`, `export`) or `memory_links` (→ `link`, `unlink`). **Use RENAME, never DROP** — `.claude/hooks/pre-bash-rust.sh` blocks shell text matching `drop table`. Note `hybrid.rs:729,760` pin `SEARCH_STORAGE_FAILURE` by equality — they must stay green after D-14. + +--- + +### `crates/agent-memory/src/rest/handlers.rs` + `crates/agent-memory/src/mcp.rs` test modules (in-process redaction tests) + +**Analog (REST):** `handlers.rs:851-889` — `test_state(&dir)` / `live_state(&dir)`, store one row, rename via raw connection on `dir.path().join("mem.db")`, call the handler fn directly, convert `Err` with `.into_response()`, assert 500 + body text: +```rust + #[tokio::test] + async fn search_storage_failure_is_500_without_sqlite_text() { + for live in [true, false] { + let dir = tempfile::tempdir().expect("tempdir"); + let state = if live { live_state(&dir) } else { test_state(&dir) }; + stored_id(&state, "ledger reconciliation runbook").await; + agent_memory_core::store::sqlite::register_vec_extension().expect("register vec0"); + rusqlite::Connection::open(dir.path().join("mem.db")) + .expect("open db") + .execute_batch("ALTER TABLE memories_fts RENAME TO memories_fts_offline;") + .expect("rename fts table"); + + let resp = match search_handler(State(state), Json(search_request("ledger"))).await { + Ok(_) => panic!("live={live}: a broken FTS index must fail the search"), + Err(err) => err.into_response(), + }; + assert_eq!(resp.status(), StatusCode::INTERNAL_SERVER_ERROR, "live={live}"); + let body = body_json(resp).await; + let error = body["error"].as_str().expect("500 body carries an error"); + assert!(error.contains("search failed due to a storage error"), "live={live}: {body}"); + assert!(!error.to_lowercase().contains("sqlite"), "live={live}: no SQLite text may reach the client: {body}"); + } + } +``` +New tests: `health_handler(State(state))` after `ALTER TABLE memories RENAME TO memories_offline` → 500 with `"list failed due to a storage error"`; `store_handler` → `"store failed …"`. `/health`'s store ping is `service.list(limit 1)` (`handlers.rs:437-445`), which is why the op string is `list`. + +**Analog (MCP):** `mcp.rs:1309-1344` — same recipe through `test_server(&dir)` / `live_server(&dir)`, asserting `err.code == McpError::internal_error("x", None).code` and `err.message` content. + +**Mappers stay untouched** (`handlers.rs:167-182`, `mcp.rs:35-50`) — both match every `MemoryError` variant explicitly with no catch-all; D-14 adds no variant so they keep compiling. + +--- + +### `crates/agent-memory/tests/cli.rs` (spawned-binary realism test) + +**Analog:** `crates/agent-memory/tests/tools.rs:12-36` and `tests/rest.rs:18-50` — `env!("CARGO_BIN_EXE_agent-memory")`, `--db` on a tempdir, dead Ollama URL on the discard port: +```rust +fn binary_path() -> String { + env!("CARGO_BIN_EXE_agent-memory").to_string() +} +... + let mut child = Command::new(binary_path()) + .args(["serve", "--db"]) + .arg(db_path) + // Deterministically dead Ollama URL: the binary is in keyword mode + // on EVERY machine regardless of a locally running daemon + // (SEARCH-03 — port 9 is the discard port, connection refused). + .env("AGENT_MEMORY_OLLAMA_URL", "http://127.0.0.1:9") + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::null()) + .spawn() + .expect("spawn agent-memory serve"); +``` +Difference: `export`/`import` are one-shot, so use `Command::output()` (no `Child` lifecycle, no `Drop` reaper) and assert on `status.success()`, `stdout` (the JSONL for `export`, the counts line for `import`) and `stderr` (the export report line). Also assert `--from bogus` exits non-zero. These tests contribute zero coverage (`handlers.rs:456-460` comment) — keep all logic assertions in core tests. + +--- + +### `README.md` (docs) + +**Analog:** README 513-532 ("Import from GSD") — short bash block, a table, one paragraph on provenance and idempotency: +```markdown +## Import from GSD + +Bootstrap your memory from an existing [GSD](...) project's state: + +```bash +agent-memory import --from gsd-state .planning/STATE.md +``` + +| STATE.md section | Imported as | +|------------------|-------------| +| `### Decisions` bullets | `DECISION` | +... +Every imported memory carries `source = "gsd-state"`, the `gsd` tag, and a +scope (...). Re-running the same import is a **no-op**: unchanged items are +deduplicated on the exact (source, type, content) key and reported as skipped. +``` +New "Export & import" section follows this shape with: the two commands, the format v1 header/memory/link spec, a D-10 timestamp-policy table, the pending-embedding note (cross-reference "Lifecycle: TTL & decay" at 534), the strict-vs-tolerant importer contrast, and the optional `VACUUM INTO` backup tip. **Widen README 460-462** from "A storage failure during search reports a generic internal error … `search failed due to a storage error`" to every service response (` failed due to a storage error`). + +## Shared Patterns + +### Error tiering at the service seam (D-14 / Phase 3 D-06) +**Source:** `crates/agent-memory-core/src/service.rs:46-60` (helper), `:299-303` (application), `crates/agent-memory-core/src/domain.rs:375-426` (the nine `MemoryError` variants). +**Apply to:** every public `MemoryService` method, both new methods, and the new parser (which only ever returns `InvalidArgument`). +Rule: client tier = `InvalidType | InvalidQuery | InvalidArgument` (pass through); crate text = `Internal | SchemaTooNew` (pass through); storage = `Sqlite | Pool | Join | Migration` (log at `error`, replace). Exhaustive match, no `_` arm. + +### `spawn_blocking` hop + `map_err(MemoryError::Join)` +**Source:** `service.rs:228-231` (single `?`), `service.rs:406-422` (closure returning `Result`, double `??`). +**Apply to:** `export`, `import_jsonl` (two or three hops), any new store call from the service. Embed calls (none in this phase's import) stay *outside* `spawn_blocking`. + +### Writer-lane transaction +**Source:** `sqlite.rs:610-614, 642` (`self.writer.lock().map_err(|_| Internal("writer lock poisoned"))`, `conn.transaction()?`, `tx.commit()?`). +**Apply to:** `insert_imported`; `link` already has it. Read-only work (`find_by_key`, `export_rows`, `embedding_pin`) uses `self.reads.get().map_err(MemoryError::Pool)?` (`sqlite.rs:736, 757, 775`). + +### Parameterised SQL only +**Source:** every `params![...]` in `sqlite.rs` (e.g. 620-630, 767, 986); comments cite T-02-01/T-02-20. +**Apply to:** all new statements. Wire strings reach SQL only as `as_wire_str()` of a parsed enum (`sqlite.rs:621, 986`). + +### Fault injection = RENAME, never DROP +**Source:** `tests/hybrid.rs:719-721`, `handlers.rs:863-867`, `mcp.rs:1322-1326`. +**Apply to:** `tests/redaction.rs`, new handler/MCP tests. Rename `memories` for store/list/update/forget/import/export/`/health`; rename `memory_links` for link/unlink. + +### `cfg(unix)` / `cfg(not(unix))` one-seam split +**Source:** `crates/agent-memory/src/config.rs:37-54`. +**Apply to:** `export_file.rs` `create_private` (0600). Comment the non-unix arm with the DIST-03 / Phase 6 deferral like line 50. + +### Temp-in-same-dir + rename + best-effort cleanup +**Source:** `sqlite.rs:140-145` (pid-suffixed name), `:150-159` (never-failing discard), `:265-291` (discard on every error path, rename last). +**Apply to:** `--output` handling in `main.rs` / `export_file.rs`. + +### Test harness conventions +**Source:** `tests/relations.rs:33-61` (`Harness` with `TestClock::new(1_000)`, `FakeEmbedder::failing()` default), `tests/relations.rs:81-102` (raw-connection counters), `tests/import.rs:21-32` (`service_for(path)`). +**Apply to:** `tests/export_import.rs`, `tests/redaction.rs`. Dev-deps already present: `tempfile`, `tokio`, self-dep with `test-clock` (`crates/agent-memory-core/Cargo.toml:31-37`); binary crate also has `serde_json` + `rusqlite` as dev-deps (`crates/agent-memory/Cargo.toml:27-33`). + +### Doc-comment conventions +**Source:** every trait method in `store/mod.rs` (invariant + review-finding tag, e.g. `(CR-01)`, `(D-02 / T-04-13)`); module docs in `gsd_state.rs:1-16`, `config.rs:1-5`. +**Apply to:** all new `pub` items (ecosystem rule: rustdoc on every public item). Cite the Phase 5 decision ids (D-03..D-14) the same way. + +## No Analog Found + +| File | Role | Data Flow | Reason / What to use instead | +|------|------|-----------|------------------------------| +| `MemoryService::export` + `Store::export_rows` (streaming boxed `Write` through `spawn_blocking`, visitor over one read snapshot) | service + store | streaming | Nothing in the repo streams rows to a writer; every read returns a `Vec` (`pending_embeddings` 735-749, `list` 774-803). Copy the read-pool `prepare`/`query_map` loop from `pending_embeddings` and the `spawn_blocking` idiom from `service.rs:228-231`; the `Box` sink, `serde_json::to_writer` + `b"\n"`, explicit `flush()`, and the single read transaction are RESEARCH Pattern 4 / Pitfall 8 (ASSUMED A1 on `unchecked_transaction`). | + +## Metadata + +**Analog search scope:** `crates/agent-memory-core/src/{service.rs, domain.rs, lib.rs, import/, store/}`, `crates/agent-memory-core/tests/{import.rs, relations.rs, hybrid.rs}`, `crates/agent-memory/src/{main.rs, config.rs, mcp.rs, rest/handlers.rs}`, `crates/agent-memory/tests/{tools.rs, rest.rs}`, `README.md`, both `Cargo.toml` dev-dep sections. +**Files scanned:** 17 (6 read whole; `service.rs`, `sqlite.rs`, `domain.rs`, `hybrid.rs`, `handlers.rs`, `mcp.rs`, `relations.rs`, `tools.rs`, `rest.rs`, `main.rs`, `README.md` read in targeted non-overlapping ranges). +**Pattern extraction date:** 2026-10-25 +**Verified-against:** 05-RESEARCH.md line anchors; no divergence found between RESEARCH's `[VERIFIED]` quotes and the working tree. diff --git a/.planning/phases/05-portable-export-import/05-RESEARCH.md b/.planning/phases/05-portable-export-import/05-RESEARCH.md new file mode 100644 index 0000000..2ab00d8 --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-RESEARCH.md @@ -0,0 +1,806 @@ +# Phase 5: Portable Export & Import - Research + +**Researched:** 2026-10-25 +**Domain:** Rust / serde JSONL interchange format + SQLite store extension + CLI atomic file write + service-seam error redaction (agent-memory workspace, no new crates) +**Confidence:** HIGH (every load-bearing claim is either read from this repo's source this session or verified by a scratch probe against the pinned serde 1.0.228 / serde_json 1.0.150) + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions + +#### Surface & CLI shape (DIST-04 / INTEROP-02) +- **D-01:** Export and import are **CLI subcommands only** in v0.1.0: `agent-memory export [--output PATH]` and `agent-memory import --from jsonl PATH` (the second value of the existing `--from` dispatch beside `gsd-state`). No MCP tool and no REST endpoint for either. Rationale: an MCP export would route the corpus through the JSON-RPC stdout channel (MCP-05 purity), a REST import is an arbitrary-file-read primitive on an unauthenticated daemon (PITFALLS security row), and both roadmap criteria describe a file written and read by the user. The **error contract still lives in core**: every malformed-file rejection is `MemoryError::InvalidArgument` (Phase 3 D-04/D-06), so the two-tier taxonomy holds and any later transport inherits 400/`invalid_params` structurally; the CLI exits non-zero printing that message. — **Reversibility:** reversible — adding a transport later is additive. +- **D-02:** `export` writes to **stdout by default**; `--output PATH` writes **temp-file-in-the-same-directory + rename** (atomic; the target is replaced if it exists), **mode 0600 on Unix** (the corpus is T-01-03-sensitive; never looser than the DB), using the same `cfg(unix)` / `cfg(not(unix))` split `config.rs` already has (Windows is one seam, Phase 6). stdout is sanctioned for CLI subcommands (the `import` precedent); MCP-05 applies only to `serve`. Export streams one line at a time — never buffers the corpus as one JSON document. + +#### JSONL format v1 — the frozen contract (DIST-04) +- **D-03:** **Line 1 is a header record**, mandatory: `{"record":"header","format":"agent-memory-export","version":1,"exported_at":,"embedding_model":"nomic-embed-text","embedding_dim":768,"schema_version":3,"agent_memory_version":""}`. `embedding_model`/`embedding_dim` are **read from the `meta` table** (the 0002 pin), not from constants, so the header always states what the source DB actually pinned. — **Reversibility:** one-way — the moment one user writes a file, format v1 is a compatibility contract; any field removal or meaning change needs `version: 2` plus a reader that still accepts 1. +- **D-04:** **Memory record:** `{"record":"memory","key":,"type":"DECISION","content":"…","tags":[…],"source":,"scope":,"created_at":,"last_accessed":,"access_count":,"expires_at":}`. `key` is the source database's id used **only as an export-local key** — it is never written as an id on import (PITFALLS 18). **Excluded as derived data:** `decay_score` (recomputed on read), `base_weight` (a function of `type`), `embedding_status`, and every vector (PITFALLS 17; ARCHITECTURE Anti-Pattern 4). Field order is the declaration order above (serde struct), golden-tested like the v1.0 envelopes. +- **D-05:** **Link record:** `{"record":"link","from":,"to":,"kind":"relates_to|supersedes|caused_by","created_at":}`, emitted **after all memory records**, endpoints exactly as stored (04-06 canonical `(min,max)` for `relates_to`; request order for directed kinds). `created_at` travels because D-07 of Phase 4 orders expansion by edge recency. +- **D-06:** Export writes **every row present in the database** (no `--scope`/`--type` filters, no expiry skipping at export time — the importer owns expiry policy) in ascending id order, through one streaming read-pool query per record type. Filters are deferred (additive). + +#### Import identity, dedup & edge remapping (INTEROP-02) +- **D-07:** The idempotency key stays the existing INTEROP-01 key **`(source, mem_type, content)`** — no new identity column, **no migration 0004**. `Store::exists` is generalised to return the matching row's id (`Option`, NULL-safe `source IS ?1` kept, CR-01) so a duplicate maps its export key to the **existing** row. Consequence, accepted and documented: two source rows identical on that key collapse to one on import (counted as a duplicate). `source` is **preserved verbatim** (no `jsonl` provenance stamp), so the round trip is faithful and a `gsd-state` row carried in a JSONL file dedups against an earlier direct GSD import. +- **D-08:** **Two-pass, validate-before-write import:** (1) parse and validate the whole file into typed records (header, memories, links) — any error aborts with nothing written; (2) memories pass: per record, skip-and-count if expired (D-10), else dedup-or-insert, building `key → row id`; (3) links pass through the map via the existing `Store::link` (idempotent `INSERT OR IGNORE` + endpoint pre-check), with the record's `created_at` as `now`. A link whose endpoint was **skipped as expired is dropped and counted** (`links_dropped`); a link whose key **does not appear in the file at all is malformed** (whole-file rejection, D-11). All writes go through the existing writer lane; batching into transactions is the planner's call. +- **D-09:** Imported rows go through the shared Phase 3 seam like every insert path: the `type` and `kind` wire strings parse via `MemoryType::try_from` / `LinkKind::try_from`, empty/whitespace `content` is `InvalidArgument`, self-links are `InvalidArgument` (Phase 4 D-02), tags must be a JSON array of strings. `expires_at` is absolute, so `validate_ttl` does not apply; a non-null `expires_at` must be an integer. + +#### Timestamp & lifecycle policy on import (INTEROP-02; PITFALLS 19) +- **D-10:** Per column, written into the format doc: **`created_at` preserved** (provenance); **`expires_at` preserved**, and a row with `expires_at <= import-time now` is **skipped and counted (`skipped_expired`) — never inserted** for the sweep to delete later; **`last_accessed` refreshed to import-time now** so imported memories start decay-fresh (PITFALLS 19: a preserved months-old `last_accessed` would make a restored corpus unfindable by rank); **`access_count` preserved** (provenance only — `decay_score` depends solely on `last_accessed`, so this changes no ranking); link `created_at` preserved (D-05). The decay-never-deletes invariant is untouched. If the header's `exported_at` is more than one hour in the future, the CLI prints a clock-skew **warning** to stderr and continues. + +#### Malformed-file policy & the import report (INTEROP-02 criterion 4) +- **D-11:** **Strict, all-or-nothing validation — the opposite of the tolerant `gsd-state` scraper, deliberately:** JSONL is agent-memory's own contract, so a defect means a corrupted or hand-edited file and a half-restored corpus is worse than a clear refusal. Rejected as `InvalidArgument` naming the 1-based line: missing/invalid header, `format` ≠ `agent-memory-export`, non-integer or `< 1` `version`, `version > 1` (message says to upgrade agent-memory), a line that is not a JSON object, an unknown `record` value, an unknown field (`deny_unknown_fields`, the 04-07 precedent), a wrong field type, an invalid `type`/`kind`, empty content, a self-link, a link key absent from the file. Message shape follows Phase 3 D-05: `invalid argument: line 42: unknown record type 'memo'`. **Nothing is written when validation fails.** Expired rows are **not** malformed (D-10). The `gsd-state` importer's tolerance rules are unchanged. +- **D-12:** `ImportReport` grows **additively**: `imported`, `skipped_duplicates` (existing), plus `skipped_expired`, `links_imported`, `links_skipped_duplicates`, `links_dropped`. The `gsd-state` path reports zeros for the new fields. The CLI prints every count on one line plus, for JSONL, a line stating how many rows are pending embedding (D-13). + +#### Embeddings on import (INTEROP-02 criterion 3) +- **D-13:** **JSONL import never embeds inline.** Every imported row lands `embedding_status = 0` and is re-embedded by the **existing sweep backfill** (128 per tick; the first tick fires on every `serve`/`serve-rest` start) — exactly one re-embed mechanism, zero mixing risk (PITFALLS 17), the CLI never blocks on Ollama, and the criterion holds literally regardless of embedder state. Cost, accepted and documented in the README: a large restore is keyword-only until backfill catches up (128 rows per hourly tick after the first). The `gsd-state` importer keeps its existing one-shot best-effort embed. A faster drain is deferred (see Deferred Ideas). + +#### Write-path storage-error redaction (user-approved scope addition; 04-REVIEW IN-01) +- **D-14:** Generalise `redact_search_error` into a **service-seam `redact_storage_error(op, e)`** applied to the result of **every public `MemoryService` method** (`store`, `list`, `search`, `update`, `link`, `unlink`, `forget`, `import`, the new `export`, `sweep`). Storage-engine variants (`Sqlite | Pool | Join | Migration`) are logged at `error` level with full detail, then replaced by `MemoryError::Internal(" failed due to a storage error (details are in the server log)")`; client tier (`InvalidType | InvalidQuery | InvalidArgument`) and crate-authored text (`Internal`, `SchemaTooNew`) pass through unchanged. `SEARCH_STORAGE_FAILURE` keeps its exact current text (it already fits the pattern), so Phase 4's golden assertions still hold. The two transport mappers stay as they are (their exhaustive matches still compile), which means **MCP, REST — including `/health` — and the CLI are covered by one change at one seam**, mirroring Phase 3 D-06 and Phase 4 T-04-19. The README sentence that scopes redaction to search is widened to every surface. — **Reversibility:** reversible. + +### Claude's Discretion +- How the import insert carries preserved `created_at`/`access_count` into the store: extend `NewMemory` with optional overrides (ARCHITECTURE §4) or add a dedicated `ImportedMemory` + `Store::insert_imported` — pick whichever touches fewer existing call sites while keeping `insert`'s single-writer-transaction shape. +- Transaction batching for the import passes (per row vs chunked) and the export read (one statement vs paged) — streaming on both sides is the requirement, the chunk size is not. +- Temp-file naming for `--output`; whether to reuse the backup temp-path helper from `sqlite.rs`. +- Whether the header also carries `exported_rows`/`exported_links` counts (nice for a quick integrity check; additive within v1 only if decided now, since D-11 rejects unknown fields later). +- Exact rustdoc placement of the format spec (`crates/agent-memory-core/src/export.rs` module doc + a README "Export & import" section is the minimum). +- Zero-code README tip documenting `VACUUM INTO` as a same-machine *backup* (FEATURES anti-pattern row) — include if cheap. + +### Deferred Ideas (OUT OF SCOPE) +- `--include-embeddings` export flag with enforced model/dim header check on import (FEATURES P2) — only when a same-model offline-restore use case appears. +- Export filters (`--scope`, `--type`) — additive; D-06 exports everything. +- MCP tool / REST endpoint for export or import — would need an in-body payload design and a stdout-purity story; D-01 keeps both CLI-only. +- `import --embed-now` or a one-shot `backfill` subcommand that drains pending embeddings immediately after a large restore — mitigates the D-13 latency (128 rows per hourly tick) without a second re-embed mechanism. +- Cross-tool import adapters (mem0 / Chroma JSONL dialects) — no standard exists; wait for demand (FEATURES). +- A stable per-memory identity column (UUID) for dedup that survives content edits — would need migration 0004 and changes the idempotency contract; D-07 keeps the content key. +- Cross-process exclusive lock on the pre-migration backup (04-REVIEW WR-01 residual) and the 04-REVIEW IN-02/IN-04/IN-05/IN-06 items — unchanged, still deferred; IN-05's `SchemaTooNew` re-probe fix is cheap if a planner wants to ride it along, but it is not Phase 5 scope. + + + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|------------------| +| DIST-04 | User can export all memories to a versioned JSONL file (header records format version + embedding model/dim; embeddings excluded as derived data; relation edges included with export-local key remapping) | Patterns 1 (record enum, verified field order), 4 (streaming boxed-writer export on one read snapshot), 7 (atomic 0600 `--output`), 5b (`meta` read for the header); Pitfalls 5, 7, 8; Code Examples 1, 4, 5 | +| INTEROP-02 | User can import that JSONL on another machine — idempotent re-run, imported rows land `embedding_status = 0` and re-embed via the existing sweep backfill, expired items skipped and counted | Patterns 2 (two-step header parse), 3 (per-line parse with own line counter), 5 (`ImportedMemory` + `insert_imported`), 6 (`find_by_key` dedup → key map), 6b (`LinkOutcome::Linked { created }` for `links_skipped_duplicates`); Pitfalls 1–4, 6, 9–11; Code Examples 2, 3, 6 | +| D-14 (scope addition, 04-REVIEW IN-01) | Redact raw SQLite text on every `MemoryService` path, both transports + CLI | Pattern 8 (`redact_storage_error(op, e)` with byte-identical `SEARCH_STORAGE_FAILURE`), the public-method inventory, the existing-assertion inventory, the RENAME-table fault-injection recipe; Code Example 7 | + + +## Summary + +Phase 5 is almost entirely in-repo engineering against code that already exists: the `MemoryService::import` → `Store::exists` → `Store::insert` seam, the `Store::link` idempotent edge insert, the `meta` pin table, the `redact_search_error` helper, the `config.rs` `cfg(unix)` split and the `sqlite.rs` temp-then-rename backup shape. No new crate is needed or recommended: serde 1.0.228 + serde_json 1.0.150 (already pinned in `Cargo.lock`) and `std` cover the record enum, per-line JSONL reading/writing, and the 0600 atomic file write. The one genuinely uncertain technical question — whether `#[serde(deny_unknown_fields)]` works on an internally tagged `#[serde(tag = "record")]` enum — was settled empirically this session with a scratch crate against the exact pinned versions: **it works** (the only documented serde limitation is `flatten`), but with one consequence the planner must design around: for errors detected inside a variant (unknown field, wrong type) serde_json reports `line 0, column 0`, so D-11's `line N:` prefix must come from the importer's own 1-based line counter, never from `serde_json::Error::line()`. + +Three design recommendations fall out of counting call sites rather than taste. (1) Carry preserved `created_at`/`access_count` with a new `ImportedMemory` struct + `Store::insert_imported` — extending `NewMemory` would touch 15 struct-literal construction sites (12 in core tests, `gsd_state.rs`, `mcp.rs`, `handlers.rs`), the dedicated type touches zero, and `Store` has exactly one implementor. (2) Generalise `Store::exists` into `find_by_key(...) -> Result>`: it has one caller (`service.rs:413`) and one impl; no test calls it directly. (3) D-12's `links_skipped_duplicates` cannot be counted with today's `LinkOutcome::Linked` (a duplicate and a fresh insert are indistinguishable); the least-churn honest fix is `LinkOutcome::Linked { created: bool }` — 7 sites (1 producer, 2 transport matches that become `Linked { .. }`, 4 test asserts that become *stronger*). The wire shape is unchanged because transports echo request ids, not the outcome. + +D-14 is a mechanical generalisation with a precise compatibility constraint: four existing tests assert on the exact `SEARCH_STORAGE_FAILURE` text (`hybrid.rs:729,760` by equality; `handlers.rs:881` and `mcp.rs:1336` by `contains`), and the new `format!("{op} failed due to a storage error (details are in the server log)")` with `op = "search"` is byte-identical to the current constant, so nothing moves. The fault-injection recipe for the new redaction tests is already proven in three layers (`ALTER TABLE … RENAME TO …_offline` on a raw side connection); for write paths and `/health` the lever is renaming `memories` (or `memory_links` for `link`/`unlink`). Note the repo's `pre-bash-rust.sh` hook blocks any Bash command whose text matches `drop table` — another reason to keep using RENAME, not DROP, in test SQL. + +**Primary recommendation:** Build format v1 as one `#[serde(tag = "record", deny_unknown_fields)]` enum in `crates/agent-memory-core/src/export.rs` with `String` wire fields for `type`/`kind` (converted through `MemoryType::try_from` / `LinkKind::try_from` in a pure `import/jsonl.rs` validator that owns the line counter); stream export through a `Box` moved into one `spawn_blocking` hop that reads memories then links on ONE read-pool connection inside one read transaction; add `ImportedMemory`/`insert_imported`, `find_by_key`, and `LinkOutcome::Linked { created }` to the store; wrap every public `MemoryService` method with `redact_storage_error(op, e)`; keep `main.rs` glue thin (file read, 0600 temp+rename, report printing). + +## Architectural Responsibility Map + +| Capability | Primary Tier | Secondary Tier | Rationale | +|------------|-------------|----------------|-----------| +| Format v1 record definitions (header/memory/link structs, serde attrs, golden field order) | Core library (`agent-memory-core::export`) | — | Single definition of the contract; unit-testable in-process; both the exporter and the importer compile against it | +| Export streaming read (memories asc id, then links) | Store (`SqliteStore`, read pool) | Core service (`MemoryService::export` orchestrates, maps io errors) | Only the store knows the SQL/snapshot semantics; the service owns the `spawn_blocking` + redaction seam | +| Header `embedding_model`/`embedding_dim` | Store (`meta` table read) | Core service (parses `'768'` → integer) | D-03: header must state what the source DB pinned, not a constant | +| `agent_memory_version` in the header | CLI binary (passes `env!("CARGO_PKG_VERSION")` in) | Core (accepts it as a parameter) | The binary's version is the user-facing one; core and binary are separate crates with separate `version =` lines (both `0.0.1` today) | +| stdout vs `--output` temp+rename+0600 | CLI binary (`main.rs` / a small file helper beside `config.rs`) | — | OS/filesystem concern; mirrors the `config.rs` `cfg(unix)` split; core stays I/O-agnostic like `gsd_state.rs` | +| JSONL parse + strict validation (D-08 pass 1, D-09, D-11) | Core library (`import/jsonl.rs`, pure, no I/O) | — | Mirrors `parse_gsd_state`: the CLI reads the file, core validates; errors are `MemoryError::InvalidArgument` so any future transport inherits 400/`invalid_params` | +| Memories pass (expiry skip, dedup via `find_by_key`, `insert_imported`, key→id map) | Core service (`MemoryService::import_jsonl`) | Store (writer lane) | Same seam as `import()`; the service owns the clock (`now`) and the report | +| Links pass (`Store::link` with preserved `created_at`, drop/duplicate counting) | Core service | Store (`link` on writer lane) | D-08 locks `Store::link`; the service maps keys → ids | +| Import report printing, pending-embedding line, clock-skew warning | CLI binary (`run_import`) | Core (`ImportReport` fields; `tracing::warn` for skew) | Presentation is the CLI's; counts come from core | +| Storage-error redaction on every path (D-14) | Core service seam | — (transport mappers unchanged) | One change covers MCP, REST incl. `/health`, and CLI — Phase 3 D-06 / Phase 4 T-04-19 pattern | +| Re-embedding imported rows | Existing sweep backfill (`MemoryService::sweep`, `BACKFILL_BATCH = 128`) | — | D-13: exactly one re-embed mechanism; import never calls the embedder | + +## Standard Stack + +### Core (all already in the workspace — no installs this phase) +| Library | Version (Cargo.lock) | Purpose | Why Standard | +|---------|---------|---------|--------------| +| `serde` (derive) | 1.0.228 `[VERIFIED: Cargo.lock]` | `Serialize`/`Deserialize` for the format v1 record enum; `deny_unknown_fields`; `rename_all` | Already a workspace dependency of both crates (`serde.workspace = true`) `[VERIFIED: crates/agent-memory-core/Cargo.toml, crates/agent-memory/Cargo.toml]` | +| `serde_json` | 1.0.150 `[VERIFIED: Cargo.lock]` | `to_writer` per record + `\n`; `from_str::` per line | Already a workspace dependency; the only JSON codec in the tree | +| `rusqlite` (bundled, functions, backup) | 0.39.0 `[VERIFIED: Cargo.lock]` | Streaming `query_map` reads, writer-lane transactions, `meta` read | Pin triangle must not move (REQUIREMENTS Out of Scope) | +| `std` (`std::fs::OpenOptions`, `std::os::unix::fs::OpenOptionsExt::mode`, `std::fs::rename`, `std::io::{BufWriter, Write}`, `std::io::Stdout::lock`) | rustc 1.94.1 `[VERIFIED: rustc --version]` | 0600 temp file, atomic replace, buffered streaming writer | No dependency needed; `config.rs` already uses the sibling `DirBuilderExt::mode` under `cfg(unix)` `[VERIFIED: crates/agent-memory/src/config.rs:38-48]` | +| `tracing` | workspace | `error!` log line carrying the redacted storage detail; `warn!` for clock skew | Existing logging channel (stderr only) | + +### Supporting (test-side, already present) +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| `tempfile` | 3.27.0 `[VERIFIED: Cargo.lock]` (dev-dependency of both crates) | Temp DB dirs and export files in tests | Tests only — it is NOT a runtime dependency of the binary, so `NamedTempFile::persist` is not available for `--output` without promoting it (rejected: std suffices) `[VERIFIED: crates/agent-memory/Cargo.toml dev-dependencies]` | +| `FakeEmbedder` / `TestClock` (`test-clock` feature) | in-repo | Deterministic `exported_at`, expiry fixtures, backfill proof | Every core test `[VERIFIED: crates/agent-memory-core/src/embed/mod.rs:92-110, src/clock.rs:29-44]` | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| One `#[serde(tag = "record")]` enum | Three separate structs + a `serde_json::Value` pre-read of `record` | Two parses per line; the enum gives the `unknown variant 'memo', expected one of header, memory, link` message for free and one definition of the format | +| `from_str::(line)` per line with own counter | `serde_json::Deserializer::from_reader(file).into_iter::()` (StreamDeserializer) | StreamDeserializer's `line()` is real for syntax errors but **0 for in-variant data errors** (probe); the per-line approach gives correct `line N:` for every rejection class and makes "a line that is not a JSON object" trivially checkable | +| std `OpenOptions` + `mode(0o600)` + `rename` | `tempfile::NamedTempFile::new_in(dir)` + `persist` | Would promote `tempfile` to a runtime dep for one helper; CONTEXT requires a reason for any dependency — none | +| `Box` moved into `spawn_blocking` | `tokio::task::block_in_place` with `&mut dyn Write` | `block_in_place` is unusual in this codebase; the boxed writer keeps the `spawn_blocking` convention every other service method uses | +| `LinkOutcome::Linked { created: bool }` | A second store method `link_imported` returning `bool` | D-08 locks "the existing `Store::link`"; the enum field touches 7 sites, a sibling method duplicates the endpoint pre-check | + +**Installation:** none — `cargo build --workspace --locked` must keep passing with an unchanged `Cargo.lock`. + +**Version verification:** `grep -A1 'name = "serde"' Cargo.lock` → `1.0.228`; `serde_json` → `1.0.150`; `rusqlite` → `0.39.0`; `tempfile` → `3.27.0` `[VERIFIED: Cargo.lock, 2026-10-25]`. + +## Package Legitimacy Audit + +Not applicable — **this phase installs no external packages**. Every dependency used (`serde`, `serde_json`, `rusqlite`, `tracing`, `std`) is already pinned in the committed `Cargo.lock`; `cargo test --workspace --locked` will fail if a plan accidentally adds one, which is the intended guard. + +**Packages removed due to [SLOP] verdict:** none +**Packages flagged as suspicious [SUS]:** none + +## Project Constraints (from CLAUDE.md) + +From `/Users/jirihermann/Documents/workspace-1-ideas/unity-in-flow-ai/10-agent-memory/CLAUDE.md` and the ecosystem `CLAUDE.md` one level up (both read this session): + +- Rust stable, edition 2021; `clap` derive for CLI; `serde` + `serde_json`; `tokio` for async; `anyhow` in the binary, `thiserror` in the library. +- `cargo fmt` before every commit; `cargo clippy --workspace --all-targets --locked -- -D warnings` must pass; `cargo test --workspace --locked` green before every commit (orchestrator's wording; CI runs `cargo clippy --workspace --all-targets -- -D warnings` and `cargo test --workspace` `[VERIFIED: .github/workflows/ci.yml:32,38]`). +- **No `unwrap()` in production code** — use `?` or handle the error. (`insert` uses `unwrap_or_else` for tags JSON, which is the accepted shape.) +- Pattern match exhaustively — no catch-all `_` unless truly needed (the error mappers and `redact_search_error` rely on this: a new `MemoryError` variant must fail to compile until tiered). +- Test coverage > 80 % on core logic; CI gate `cargo llvm-cov --workspace --fail-under-lines 80` `[VERIFIED: ci.yml:44]`. Coverage-bearing tests run in-process; spawned-binary tests contribute zero coverage (a SIGKILL'd child never flushes profile data — `handlers.rs:456-460` comment). +- Zero cloud dependency; no secrets committed. +- **CI is GitHub-hosted on purpose** (`runs-on: ubuntu-latest`, `permissions: contents: read`) — never change runners; `release.yml` is tag-triggered, SHA-pinned, no org secrets. +- Document public items with `///`; KDoc/JSDoc/rustdoc on all public APIs. +- No new crates without a reason (CONTEXT.md reinforces: prefer none). +- Shipped migrations 0001–0003 are FROZEN; `LATEST_SCHEMA_VERSION = 3`; this phase ships no migration. +- Bash safety hook `.claude/hooks/pre-bash-rust.sh` blocks: force-push, `cargo publish`, `rm -rf *.db`, and any command text matching `drop (table|database|schema)` (case-insensitive) `[VERIFIED: .claude/hooks/pre-bash-rust.sh]`. Planner note: test SQL must use `ALTER TABLE … RENAME` (the existing T-04-19 recipe), and executors must not `grep`/`echo` a literal DROP in a shell command. +- GSD workflow enforcement: edits happen through `/gsd-execute-phase` / `/gsd-quick`. + +## Architecture Patterns + +### System Architecture Diagram + +``` +EXPORT (agent-memory export [--output PATH]) +──────────────────────────────────────────────────────────────────────────────── + main.rs MemoryService::export(out, version) SqliteStore + ┌─────────────────────┐ ┌──────────────────────────────┐ ┌────────────────────┐ + │ choose sink: │ │ now = clock.now() │ │ ONE pooled read │ + │ stdout.lock() ──┐ │ │ spawn_blocking { │ │ conn, ONE read tx │ + │ OR temp file ──┼─┼──► │ store.export_rows(|rec| │ ──► │ SELECT meta pin │ + │ (0600,create_new)│ │ │ to_writer(out, rec)?; │ │ SELECT memories │ + │ BufWriter(Box) │ │ │ ) │ │ SELECT links │ + └──────────┬────────┘ │ │ out.flush() │ │ ORDER BY from,to│ + │ │ │ }.map_err(redact("export")) │ │ ,kind │ + ▼ │ └──────────────────────────────┘ └────────────────────┘ + Ok → sync_all, rename(temp → PATH) io::Error → MemoryError::Internal (crate text) + Err → remove temp, exit non-zero Sqlite/Pool/Join → redacted "export failed …" + +IMPORT (agent-memory import --from jsonl PATH) +──────────────────────────────────────────────────────────────────────────────── + main.rs import/jsonl.rs (pure) MemoryService::import_jsonl + ┌──────────────┐ ┌────────────────────────────┐ ┌──────────────────────────────────┐ + │ read_to_string│──►│ PASS 1 (no writes) │ │ now = clock.now() │ + │ (anyhow ctx) │ │ line 1: lenient probe │ │ skew warn if exported_at>now+3600│ + └──────────────┘ │ {record,format,version} │ │ PASS 2 (writer lane, 1 hop): │ + │ → version>1 "upgrade" msg │ │ for mem: expires_at<=now? │ + │ line 1: strict Record::Header│ │ skipped_expired++ , no map │ + │ lines 2..: first non-ws '{'?│ ──► │ else find_by_key → Some(id)? │ + │ from_str::(line) │ │ dup++, map[key]=id │ + │ type/kind via try_from │ │ else insert_imported → id │ + │ content.trim non-empty │ │ imported++, map[key]=id │ + │ from!=to; keys unique; │ │ PASS 3 (writer lane, 1 hop): │ + │ link keys ∈ memory keys │ │ for link: both keys in map? │ + │ any Err → InvalidArgument │ │ store.link(a,b,kind,created_at)│ + │ "line N: …" (own counter) │ │ Linked{created:true} → +1 │ + └────────────────────────────┘ │ Linked{created:false}→ dup+1 │ + │ else links_dropped++ │ + │ }.map_err(redact("import")) │ + └──────────────┬───────────────────┘ + ▼ + ImportReport{imported, skipped_duplicates, skipped_expired, + links_imported, links_skipped_duplicates, links_dropped} + main.rs prints one line of counts + "N rows pending embedding (backfilled by the sweep)" + rows land embedding_status = 0 → existing sweep backfill (128/tick) re-embeds locally +``` + +### Recommended Project Structure + +``` +crates/agent-memory-core/src/ +├── export.rs # NEW: format v1 — Record enum (Header/Memory/Link), FORMAT_NAME, +│ # FORMAT_VERSION = 1, module doc = the format spec; golden tests +├── import/ +│ ├── mod.rs # add `pub mod jsonl;` + a doc note that jsonl is STRICT (D-11) unlike gsd_state +│ ├── gsd_state.rs # unchanged +│ └── jsonl.rs # NEW: parse_jsonl(text) -> Result (pure, no I/O) +├── service.rs # redact_storage_error(op, e); export(); import_jsonl(); ImportReport +4 fields +├── domain.rs # ImportedMemory; LinkOutcome::Linked { created: bool } +└── store/ + ├── mod.rs # find_by_key (replaces exists); insert_imported; export_rows; embedding_pin + └── sqlite.rs # impls; insert/insert_imported share one private row-insert helper +crates/agent-memory-core/tests/ +├── export_import.rs # NEW: round trip, overlapping-rowid kill-test, expiry fixture, idempotency, +│ # malformed-file table, pending→sweep→searchable +├── redaction.rs # NEW (or extend hybrid.rs): one write path + list redacted, no "sqlite" text +└── import.rs # unchanged gsd-state proofs (still pass: new report fields are zeros) +crates/agent-memory/src/ +├── main.rs # Command::Export { output }, `jsonl` arm in run_import, report printing +├── export_file.rs # NEW (or in config.rs): create_private_temp (cfg split) + rename helper +└── rest/handlers.rs, mcp.rs # unchanged mappers; add in-process tests: /health + a write path redacted +crates/agent-memory/tests/ +└── cli.rs # NEW, realism-only: spawn `export --output` then `import --from jsonl` +README.md # "Export & import" section; widen the REST redaction sentence (line 462) +``` + +### Pattern 1: One internally tagged, strict record enum (format v1) + +**What:** A single `Record` enum with `#[serde(tag = "record", rename_all = "lowercase", deny_unknown_fields)]`, struct variants `Header`, `Memory`, `Link`. Wire `type`/`kind` are `String` so conversion runs through the Phase 3 seam (`MemoryType::try_from` / `LinkKind::try_from`) and produces the established messages. + +**When to use:** Always — it is the single definition of the contract for both directions. + +**Verified facts (scratch probe, serde 1.0.228 / serde_json 1.0.150, 2026-10-25):** `[VERIFIED: /tmp/serde-probe run output]` +- `deny_unknown_fields` on the tagged enum **rejects unknown fields inside every variant**: `unknown field 'extra', expected 'format' or 'version'` / `unknown field 'bogus', expected one of 'key', 'content', 'source'`. +- Unknown tag value → `unknown variant 'memo', expected one of 'header', 'memory', 'link' at line 1 column 16` (position kept). +- **In-variant errors report `line 0 column 0`** (unknown field, `invalid type: string "1", expected i64`, `invalid type: floating point 1.5, expected u32`, `invalid value: integer -1, expected u32`) — the internally-tagged path buffers the object into serde's `Content`, losing position. +- A missing `Option` field deserializes as `None` (so `source`/`scope`/`expires_at` may be absent in a hand-edited file and still parse — the exporter always writes them explicitly as `null`, so round-trip output is canonical). +- A non-object line `[1,2]` → `invalid type: integer 1, expected variant identifier` (opaque — pre-check the first non-whitespace byte is `{` to say `line N: expected a JSON object`). +- Trailing text after the object → `trailing characters at line 1 column 55`. +- Serialization emits the tag **first**, then fields in declaration order: `{"record":"memory","key":7,"content":"c","source":null}` — so D-04's documented order is exactly the struct's declaration order with `record` first. + +Serde's only documented `deny_unknown_fields` limitation is `flatten` (not used here): "this attribute is not supported in combination with `flatten`, neither on the outer struct nor on the flattened field" `[CITED: serde.rs/container-attrs.html via Context7 /websites/serde_rs]`. Internally tagged enums support struct variants, newtype variants containing structs/maps, and unit variants — not tuple variants `[CITED: serde.rs/enum-representations.html]`. + +**Example:** +```rust +// crates/agent-memory-core/src/export.rs (shape verified by the probe; field names per D-03/D-04/D-05) +use serde::{Deserialize, Serialize}; + +pub const FORMAT_NAME: &str = "agent-memory-export"; +pub const FORMAT_VERSION: u32 = 1; + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(tag = "record", rename_all = "lowercase", deny_unknown_fields)] +pub enum Record { + Header { + format: String, + version: u32, // u32 rejects -1 and 1.5 at serde level; version == 0 needs an explicit check + exported_at: i64, + embedding_model: String, + embedding_dim: u32, + schema_version: i64, + agent_memory_version: String, + }, + Memory { + key: i64, + #[serde(rename = "type")] + mem_type: String, // String on the wire → MemoryType::try_from in jsonl.rs (D-09) + content: String, + tags: Vec, // "must be a JSON array of strings" falls out of the type + source: Option, + scope: Option, + created_at: i64, + last_accessed: i64, + access_count: i64, + expires_at: Option, + }, + Link { + from: i64, + to: i64, + kind: String, // → LinkKind::try_from (D-09) + created_at: i64, + }, +} +``` + +### Pattern 2: Two-step header parse (lenient probe, then strict) + +**What:** Parse line 1 twice: first into a lenient `HeaderProbe { record: String, format: String, version: serde_json::Value }` **without** `deny_unknown_fields` (serde ignores unknown fields by default `[CITED: serde.rs/container-attrs.html]`), check `record == "header"`, `format == FORMAT_NAME`, `version` is an integer ≥ 1, then `version > FORMAT_VERSION` → the D-11 "upgrade" message; only then parse line 1 strictly as `Record::Header`. + +**Why:** With a single strict parse, a future v2 header that adds a field fails first with `unknown field …` and the user never sees `file is format version 2 but this binary reads up to version 1 — upgrade agent-memory`. The probe is what makes D-11's version message reachable. `version: 0` passes a `u32` field, so the `< 1` rule is an explicit check either way. + +**Example:** +```rust +#[derive(Deserialize)] +struct HeaderProbe { record: String, format: String, version: serde_json::Value } + +fn check_header_line(line: &str) -> Result<(), MemoryError> { + let probe: HeaderProbe = serde_json::from_str(line) + .map_err(|e| invalid_line(1, format!("missing or invalid header ({e})")))?; + if probe.record != "header" { return Err(invalid_line(1, "first record must be the header")); } + if probe.format != FORMAT_NAME { + return Err(invalid_line(1, format!("format must be '{FORMAT_NAME}' (got '{}')", probe.format))); + } + let version = probe.version.as_u64() + .ok_or_else(|| invalid_line(1, format!("version must be a positive integer (got {})", probe.version)))?; + if version < 1 { return Err(invalid_line(1, "version must be >= 1 (got 0)")); } + if version > u64::from(FORMAT_VERSION) { + return Err(invalid_line(1, format!( + "file is format version {version} but this binary reads up to version {FORMAT_VERSION} — upgrade agent-memory"))); + } + Ok(()) +} + +fn invalid_line(line_no: usize, msg: impl std::fmt::Display) -> MemoryError { + // Display of InvalidArgument is "invalid argument: {0}" → "invalid argument: line 42: …" (D-11 shape) + MemoryError::InvalidArgument(format!("line {line_no}: {msg}")) +} +``` +`MemoryError::InvalidArgument`'s Display is `"invalid argument: {0}"` `[VERIFIED: crates/agent-memory-core/src/domain.rs:389-390]` — quote: `#[error("invalid argument: {0}")]` / `InvalidArgument(String),`. + +### Pattern 3: Per-line deserialization with the importer's own 1-based counter + +**What:** `for (idx, line) in text.lines().enumerate() { let line_no = idx + 1; … serde_json::from_str::(line) … }`. Every rejection is wrapped as `line {line_no}: {serde message}`; because serde's own `line()` is always `1` (per-line input) or `0` (in-variant data errors), **never surface `e.line()`**. Optionally strip serde's trailing ` at line 1 column N` suffix (present only on syntax/variant errors) or keep it — tests assert on the `line N:` prefix and the field name, not full equality (CONTEXT ``). + +**Pass-1 checks per line (D-09/D-11), in this order:** empty/whitespace-only or first non-ws byte ≠ `{` → "expected a JSON object"; `from_str::` → serde message; `Record::Header` after line 1 → "duplicate header"; `Memory`: `MemoryType::try_from(&mem_type)` (its `InvalidType` text re-wrapped under `InvalidArgument("line N: …")`), `content.trim().is_empty()` → "content must be a non-empty string" (the `validate_patch` wording `[VERIFIED: domain.rs:476-478]`), duplicate `key` → malformed (ASSUMED-reasonable extension of D-11: an ambiguous key map is a corrupted file; see A3); `Link`: `LinkKind::try_from(&kind)` (already `InvalidArgument` naming the allowed set `[VERIFIED: domain.rs:166-168]`), `validate_link_endpoints(from, to)` for self-links `[VERIFIED: domain.rs:490-497]`, then after the loop: every link `from`/`to` must be in the memory-key set → "link key 7 does not appear in the file". Memory records after the first link record are allowed by the parser but the exporter never writes them that way (D-05 ordering is an exporter guarantee; the importer only needs the key set complete before the links pass). + +`serde_json::Error::line()` docs: "One-based line number at which the error was detected. Characters in the first line of the input (before the first newline character) are in line 1." `[CITED: docs.rs/serde_json error.rs via Context7 /websites/rs_serde_json]` — correct for syntax errors, but the probe shows `0` for data errors raised from buffered `Content`, hence the own counter. + +### Pattern 4: Streaming export through a boxed writer on ONE read snapshot + +**What:** `MemoryService::export(&self, out: Box, agent_memory_version: String) -> Result` runs one `spawn_blocking` hop that calls `store.export_rows(&mut |record| { serde_json::to_writer(&mut out, &record)?; out.write_all(b"\n") })` and finally `out.flush()`. `Store::export_rows` takes **one** pooled connection, opens **one** read transaction (`conn.unchecked_transaction()` — in WAL mode a reader sees a consistent snapshot), reads the `meta` pin, then `SELECT … FROM memories ORDER BY id`, then `SELECT from_id, to_id, kind, created_at FROM memory_links ORDER BY from_id, to_id, kind`, invoking the visitor per row (rusqlite `query_map` steps rows lazily — nothing is buffered). + +**Why one snapshot:** two statements on two pooled connections can straddle a concurrent `link` whose endpoint was inserted after the memories SELECT ran, producing a file whose link references a key absent from the file — which the strict importer then rejects (D-11). Inside one snapshot, the `memory_links` FK guarantees every endpoint is also exported. `[ASSUMED]` that `rusqlite::Connection::unchecked_transaction` is the right call on a pooled `&Connection` (the existing code uses `conn.transaction()` on the `&mut` writer `[VERIFIED: sqlite.rs:614,956]`); alternative: `execute_batch("BEGIN")`/`COMMIT`. + +**Errors:** `serde_json::Error`/`io::Error` from the writer → `MemoryError::Internal(format!("export write failed: {e}"))` — crate-authored text, passes redaction unchanged; `Sqlite`/`Pool`/`Join` → `redact_storage_error("export", e)`. + +**Sinks (main.rs):** stdout → `Box::new(BufWriter::new(std::io::stdout().lock()))` — `Stdout::lock` returns `StdoutLock<'static>` `[CITED: doc.rust-lang.org/std/io/struct.Stdout.html]`, so it is movable into the blocking task; `--output` → `Box::new(BufWriter::new(file.try_clone()?))` so `main.rs` keeps the original `File` for `sync_all()` and the rename after `Ok` (fsync is durability, not atomicity — optional; the backup helper does not fsync `[VERIFIED: sqlite.rs:284-291]`). + +**Ordering:** memories ascending `id` (D-06); links by `(from_id, to_id, kind)` — D-05 fixes only "after all memory records"; a deterministic link order makes the export→import→export diff test and golden files stable (planner's call, recommended). + +### Pattern 5: `ImportedMemory` + `Store::insert_imported` (recommended over extending `NewMemory`) + +**Call-site count (the discretion criterion):** `NewMemory { … }` struct-literal constructions: **15** — `tests/hybrid.rs:73`, `relations.rs:64`, `backfill_race.rs:166`, `store.rs:30` and `:106`, `fallback.rs:19`, `decay.rs:29`, `validation.rs:26`, `semantic.rs:21`, `update.rs:105`, `ttl.rs:31`, `import.rs:110` (12 in core tests), `src/import/gsd_state.rs:178`, `crates/agent-memory/src/mcp.rs:285`, `crates/agent-memory/src/rest/handlers.rs:203` `[VERIFIED: grep -rn --include='*.rs' "NewMemory {" crates, signatures excluded]`. `NewMemory` has no `Default` (and `MemoryType` has no sensible default) `[VERIFIED: domain.rs:245-253]` — quote: `#[derive(Debug, Clone)] pub struct NewMemory { pub content: String, pub mem_type: MemoryType, pub tags: Vec, pub source: Option, pub scope: Option, pub ttl_secs: Option, }` — so two new `Option` fields mean 15 edits. A dedicated type touches **0** existing sites. `Store` has exactly one implementor (`impl Store for SqliteStore`, `sqlite.rs:595`) and no test fakes `[VERIFIED: grep "impl Store for"]`, so a new trait method costs one impl. + +**Shape:** +```rust +// domain.rs +/// A memory restored from a format-v1 file (INTEROP-02). Unlike NewMemory it carries absolute +/// timestamps: created_at/access_count/expires_at are preserved, last_accessed is the import-time +/// `now` (D-10), and the row ALWAYS lands embedding_status = 0 (D-13). +#[derive(Debug, Clone, PartialEq)] +pub struct ImportedMemory { + pub content: String, pub mem_type: MemoryType, pub tags: Vec, + pub source: Option, pub scope: Option, + pub created_at: i64, pub access_count: i64, pub expires_at: Option, +} +// store/mod.rs +fn insert_imported(&self, m: ImportedMemory, now: i64) -> Result; +``` +`insert_imported` mirrors `insert`'s single writer transaction and its exact INSERT column list `[VERIFIED: sqlite.rs:615-631]` — quote of the current statement: `"INSERT INTO memories (mem_type, content, tags, source, scope, base_weight, decay_score, access_count, created_at, last_accessed, expires_at, embedding_status) VALUES (?1, ?2, ?3, ?4, ?5, ?6, 1.0, 0, ?7, ?7, ?8, ?9)"` — with `access_count = ?`, `created_at = ?`, `last_accessed = now`, `embedding_status = 0`, `base_weight` from `mem_type.is_pinned()` (2.0 / 1.0 `[VERIFIED: sqlite.rs:603]`), `decay_score = 1.0`. Refactor the body into one private `insert_row(tx, …)` helper so both inserts share one statement (avoids drift). The `memories_ai` FTS trigger fires on the insert `[CITED: sql/0001_init.sql]`, so imported rows are keyword-searchable immediately. + +### Pattern 6: `find_by_key` replaces `exists` (D-07) + +`Store::exists` has **one caller** (`service.rs:413` — `if store.exists(draft.source.as_deref(), draft.mem_type, &draft.content)?`) and **one impl** (`sqlite.rs:751-772`); no test calls it directly `[VERIFIED: grep "\.exists(" crates — the other hits are `Path::exists`]`. Change the signature to `fn find_by_key(&self, source: Option<&str>, mem_type: MemoryType, content: &str) -> Result, MemoryError>` with the SQL `SELECT id FROM memories WHERE source IS ?1 AND mem_type = ?2 AND content = ?3 LIMIT 1` (keep `IS` — CR-01 NULL-safety `[VERIFIED: sqlite.rs:765-767]`), `.optional()?`; `import()` becomes `.is_some()`. Keep the trait doc's CR-01 paragraph. Within the JSONL memories pass, a second file record identical on the key (or identical to a row inserted earlier in the same pass — the per-row tx has committed) maps to the existing id and counts as `skipped_duplicates` (D-07's accepted collapse). + +### Pattern 6b: `LinkOutcome::Linked { created: bool }` so `links_skipped_duplicates` is countable + +Today `Store::link` runs `INSERT OR IGNORE` and returns `Ok(LinkOutcome::Linked)` whether or not a row was inserted (`changes() == 0 is still Linked` `[VERIFIED: sqlite.rs:974-989]`). D-12 requires `links_skipped_duplicates`, D-08 requires using `Store::link`, so the outcome must carry `created: tx.changes() == 1` (read `changes()` before `commit`). `LinkOutcome` is `#[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum LinkOutcome { Linked, NotFound { missing_id: i64 }, }` `[VERIFIED: domain.rs:176-183]`. Sites to update: producer `sqlite.rs:989`; transport matches `mcp.rs:438` and `handlers.rs:370` (become `LinkOutcome::Linked { .. } =>`, wire JSON unchanged); test asserts `tests/relations.rs:139,149,791,794` (become `Linked { created: true }` on first link and `Linked { created: false }` on the D-02 duplicate — a stronger idempotency proof) `[VERIFIED: grep "LinkOutcome::Linked" crates]`. Not a wire change: `LinkOutcome` carries no stored ids and transports echo request ids. + +### Pattern 7: Atomic `--output` write with std only (binary crate) + +```rust +// crates/agent-memory/src/export_file.rs (or beside create_data_dir in config.rs — same cfg split) +use std::fs::{File, OpenOptions}; +use std::path::{Path, PathBuf}; + +/// `{target}.export-tmp-{pid}` in the SAME directory, so the final rename never crosses a filesystem +/// (std::fs::rename "will not work if the new name is on a different mount point"). +pub fn export_temp_path(target: &Path) -> PathBuf { + let name = target.file_name().and_then(|n| n.to_str()).unwrap_or("export"); + target.with_file_name(format!("{name}.export-tmp-{}", std::process::id())) +} + +#[cfg(unix)] +pub fn create_private(path: &Path) -> std::io::Result { + use std::os::unix::fs::OpenOptionsExt; + OpenOptions::new().write(true).create_new(true).mode(0o600).open(path) +} + +#[cfg(not(unix))] +pub fn create_private(path: &Path) -> std::io::Result { + // Windows ACL inheritance is the Phase 6 seam (DIST-03); same split as config.rs::create_data_dir. + OpenOptions::new().write(true).create_new(true).open(path) +} +// main.rs flow: temp = export_temp_path(&out); file = create_private(&temp)?; run export into +// BufWriter(file.try_clone()?); on Ok → file.sync_all()?; std::fs::rename(&temp, &out)?; +// on Err → let _ = std::fs::remove_file(&temp); propagate. +``` +`OpenOptionsExt::mode`: "Sets the mode bits that a new file will be created with. If a new file is created as part of an `OpenOptions::open` call then this specified `mode` will be used as the permission bits for the new file. If no `mode` is set, the default of `0o666` will be used. The operating system masks out bits with the system's `umask`" `[CITED: doc.rust-lang.org/std/os/unix/fs/trait.OpenOptionsExt.html]` — umask can only remove bits, so the result is never looser than 0600. `std::fs::rename`: "Renames a file or directory to a new name, replacing the original file if `to` already exists" … "corresponds to the `rename` function on Unix, and `MoveFileExW` with a fallback to `SetFileInformationByHandle` on Windows" … "This will not work if the new name is on a different mount point" `[CITED: doc.rust-lang.org/std/fs/fn.rename.html]`. `create_new` makes a stale temp from a crashed run an explicit error rather than a silent truncate; the pid suffix makes that rare. + +**Reuse verdict on the `sqlite.rs` backup helpers:** reuse the *shape*, not the code. `backup_temp_path` builds `{file_name}.backup-tmp-{LATEST_SCHEMA_VERSION:04}-{pid}` `[VERIFIED: sqlite.rs:140-145]`, `copy_into_temp` is the SQLite Online Backup API, `discard_backup_temp` removes SQLite sidecars, and all three are private `fn`s inside the core store `[VERIFIED: sqlite.rs:131-176]`. None applies to a plain text file written by the binary crate; the verify-then-rename + remove-temp-on-failure discipline (`sqlite.rs:265-291`) is what to mirror. + +### Pattern 8: `redact_storage_error(op, e)` at the service seam (D-14) + +**Current state** `[VERIFIED: crates/agent-memory-core/src/service.rs:33-60]` — quotes: `pub const SEARCH_STORAGE_FAILURE: &str = "search failed due to a storage error (details are in the server log)";` and the helper `fn redact_search_error(e: MemoryError) -> MemoryError { match e { MemoryError::InvalidType(_) | MemoryError::InvalidQuery(_) | MemoryError::InvalidArgument(_) => e, MemoryError::Internal(_) | MemoryError::SchemaTooNew { .. } => e, MemoryError::Sqlite(_) | MemoryError::Pool(_) | MemoryError::Join(_) | MemoryError::Migration(_) => { tracing::error!(error = %e, "search failed with a storage error"); MemoryError::Internal(SEARCH_STORAGE_FAILURE.to_string()) } } }`. It is applied in exactly one place: `search()` → `self.search_unredacted(args).await.map_err(redact_search_error)` `[VERIFIED: service.rs:299-303]`. + +**`MemoryError` variants (complete)** `[VERIFIED: domain.rs:375-426]`: `InvalidType(String)`, `InvalidQuery(String)`, `InvalidArgument(String)`, `Sqlite(#[from] rusqlite::Error)`, `Pool(#[from] r2d2::Error)`, `Join(#[from] tokio::task::JoinError)`, `Migration(#[from] rusqlite_migration::Error)`, `Internal(String)`, `SchemaTooNew { db_version: i64, binary_version: i64 }`. D-14 adds none. + +**Public `MemoryService` methods and today's error flow** `[VERIFIED: service.rs]`: + +| Method | Lines | Error path today | D-14 op string | +|--------|-------|------------------|----------------| +| `store(new)` | 209-232 | `validate_ttl?` then `spawn_blocking(insert).await.map_err(Join)?` — raw | `"store"` | +| `list(args)` | 244-265 | `validate_limit?`, WR-04 check, `spawn_blocking(list)`, optional `expand_related` — raw | `"list"` | +| `search(args)` | 299-303 | **already redacted** via `search_unredacted` | `"search"` (text must stay byte-identical) | +| `import(drafts)` | 394-463 | `validate_ttl` per draft, two `spawn_blocking` hops — raw | `"import"` | +| `forget(id)` | 573-578 | `spawn_blocking(forget)` — raw | `"forget"` | +| `link(from,to,kind)` | 592-604 | `validate_link_endpoints?`, `spawn_blocking(link)` — raw | `"link"` | +| `unlink(from,to,kind)` | 610-620 | raw | `"unlink"` | +| `update(id, patch)` | 634-666 | `validate_patch?`, embed, `spawn_blocking(update)` — raw | `"update"` | +| `sweep(now)` | 688-744 | three `spawn_blocking` hops — raw (callers only log) | `"sweep"` | +| `export(...)` / `import_jsonl(...)` | new | — | `"export"` / `"import"` | + +**Shape:** keep the body of each method in a private `*_unredacted` fn (the `search` precedent) and make the public fn `self.x_unredacted(..).await.map_err(|e| redact_storage_error("x", e))`. Methods have multiple `?` exits, so a tail `.map_err` on the last expression would miss early `Join` returns — the inner-fn shape is the safe one. Message: `format!("{op} failed due to a storage error (details are in the server log)")`; keep `pub const SEARCH_STORAGE_FAILURE` and add a unit test asserting `redact_storage_error("search", MemoryError::Internal-free storage variant).to_string() == format!("internal error: {SEARCH_STORAGE_FAILURE}")` so the constant and the template can never drift. Log line: `tracing::error!(op, error = %e, "storage error")`. + +**Existing assertions that must keep passing** `[VERIFIED: grep]`: `tests/hybrid.rs:729` and `:760` — `matches!(err, MemoryError::Internal(ref m) if m == SEARCH_STORAGE_FAILURE)` plus `!err.to_string().to_lowercase().contains("sqlite")`; `rest/handlers.rs:881` — `error.contains("search failed due to a storage error")`; `mcp.rs:1336` — `err.message.contains("search failed due to a storage error")`; `tests/rest.rs:219`, `tests/tools.rs:328`, `handlers.rs:767,827`, `mcp.rs:1206,1286` — `!…contains("sqlite error")` on search responses; `handlers.rs:1794` — `Internal("writer lock poisoned")` passes through the REST mapper (still true: `Internal` passes through the new seam unchanged, and the mapper is untouched). The transport mappers `map_mcp_error` `[CITED: mcp.rs:35-50]` and `map_memory_error` `[CITED: rest/handlers.rs:167-182]` match every variant explicitly and stay as they are. + +**Fault injection recipe (in-process, deterministic)** — the T-04-19 pattern `[CITED: tests/hybrid.rs:712-735; handlers.rs:851-889; mcp.rs:1309-1344]`: open a raw side connection (`register_vec_extension()` first, then `rusqlite::Connection::open(&db_path)`), run `execute_batch("ALTER TABLE memories RENAME TO memories_offline;")`, then call the service/handler. Effects: `store` → INSERT fails `no such table: memories` → `Sqlite` → redacted `store failed …`; `/health` → `service.list(limit 1)` `[CITED: handlers.rs:437-445]` fails the same way → 500 `{"error": "internal error: list failed due to a storage error (…)"}` with no `sqlite` substring; `link`/`unlink` → rename `memory_links` instead. Renaming `memories` is safe for the trigger set (modern SQLite rewrites trigger references on RENAME) and already-open pooled connections re-prepare on `SQLITE_SCHEMA` — the hybrid tests prove the same store instance keeps working after a rename. **Do not use DROP** (the repo hook blocks it in shell text; RENAME is the precedent). `chmod`/read-only tricks are unreliable because the writer connection already holds an open fd. + +### Anti-Patterns to Avoid +- **Surfacing `serde_json::Error::line()` in messages:** it is `1` for every per-line parse and `0` for in-variant data errors (probe). Use the importer's counter. +- **Strict header parse before the version check:** a v2 header fails on `unknown field` and the "upgrade agent-memory" message is unreachable (Pattern 2). +- **Two export SELECTs on two pooled connections:** a concurrent `link` can produce a dangling key that the strict importer rejects (Pattern 4). +- **Deserializing `type`/`kind` straight into `MemoryType`/`LinkKind`:** D-09 locks `try_from`; serde's `unknown variant` text would replace the established `kind must be one of relates_to, supersedes, caused_by (got '…')` message. +- **Writing `INSERT INTO memories (id, …)`** with the file's `key` — PITFALLS 18; `key` is a map key only. +- **Importing links before the memories pass completes** — endpoints that deduped to existing rows must map to the existing id (PITFALLS 18 interplay). +- **Calling the embedder anywhere in `import_jsonl`** — D-13; `insert_imported` hard-codes `embedding_status = 0`. +- **Applying `validate_ttl` to `expires_at`** — it is absolute (D-09); the only lifecycle rule is `expires_at <= now` → skip. +- **A tail `.map_err(redact…)` after a `?`-laden body** — early returns escape unredacted; use the `*_unredacted` inner-fn shape. +- **`unwrap()` / `expect()` in `main.rs`/core production paths** — CLAUDE.md; use `?` with `anyhow::Context` in the binary. +- **Buffering the corpus** (`Vec` then `to_string`) on export — D-02; stream per row. (Holding pass-1 records in memory on import is accepted by D-08's validate-before-write.) + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| Tagged-record JSON encoding/decoding | A manual `serde_json::Value` walker that reads `record` and dispatches | `#[serde(tag = "record", deny_unknown_fields)]` enum | Probe-verified messages for unknown variant/field/type; one definition for both directions | +| Unknown-field rejection | Field-name whitelists | `deny_unknown_fields` (04-07 precedent) | Already the project's pattern for new v0.1.0 DTOs | +| JSON array-of-strings validation for `tags` | Manual `Value::Array` checks | `tags: Vec` | serde enforces element types; the store already serializes tags with `serde_json::to_string(&new.tags)` `[VERIFIED: sqlite.rs:602]` | +| Edge idempotency | A pre-check SELECT per link | Existing `Store::link` `INSERT OR IGNORE` on PK `(from_id, to_id, kind)` + `changes()` | PK already enforces it `[VERIFIED: sql/0003_relations.sql:19]`; `relates_to` canonicalisation happens inside `link` `[VERIFIED: sqlite.rs:982]` | +| Dedup identity | A UUID/hash column (migration 0004) | `find_by_key` on `(source, mem_type, content)` | D-07 locked; deferred idea | +| Re-embedding on import | An inline batch embed | Existing sweep backfill (`BACKFILL_BATCH = 128` `[VERIFIED: service.rs:749]`, guarded `insert_embedding`) | D-13; CR-03 race safety already proven in `tests/backfill_race.rs` | +| Atomic replace | Write-then-copy or truncate-in-place | temp in same dir + `std::fs::rename` | POSIX rename atomicity; mirrors the backup helper | +| Private file creation | `set_permissions` after create | `OpenOptions::mode(0o600)` at creation | No window where the file is 0644 | +| Clock for `exported_at`/import `now` | `SystemTime::now()` in core | Injected `Clock` (`self.clock.now()`) | Deterministic tests with `TestClock` (`clock.rs:29-44`) | + +**Key insight:** every hard part of this phase (identity, idempotent edges, re-embedding, snapshot reads, the error taxonomy) already has a tested mechanism in the repo; the phase is wiring a format onto them, and every "shortcut" that bypasses one of those mechanisms re-opens a closed review finding (CR-01, CR-03, WR-03, T-04-13, T-04-19). + +## Common Pitfalls + +### Pitfall 1: Line numbers vanish inside tagged variants +**What goes wrong:** `invalid argument: line 0: unknown field 'bogus'` — or no line at all. +**Why it happens:** Internally tagged enums buffer the object into serde `Content`; the position is lost for data errors (probe-verified: `line 0 col 0`). +**How to avoid:** Own 1-based counter from `text.lines().enumerate()`; format `line {n}: {e}`; never print `e.line()`. +**Warning signs:** A test expecting `line 3:` seeing `line 1:`/`line 0:`. + +### Pitfall 2: The "upgrade agent-memory" message is unreachable +**What goes wrong:** A v2 file with a new header field reports `unknown field …` instead of D-11's version message. +**Why it happens:** `deny_unknown_fields` fires before any version comparison. +**How to avoid:** Pattern 2's lenient probe first; a test that feeds `{"record":"header","format":"agent-memory-export","version":2,"new_field":1}` and asserts the upgrade text. +**Warning signs:** Only `version: 2` without extra fields is tested. + +### Pitfall 3: `links_skipped_duplicates` always 0 (or always equals `links_imported`) +**What goes wrong:** Re-importing the same file reports every link as imported again, violating criterion 2's "idempotent" report. +**Why it happens:** `LinkOutcome::Linked` hides whether `INSERT OR IGNORE` inserted. +**How to avoid:** Pattern 6b (`created: bool` from `tx.changes()` before commit); test: second import → `links_imported == 0`, `links_skipped_duplicates == N`, `SELECT count(*) FROM memory_links` unchanged (raw side connection, `relations.rs::edge_count` shape). + +### Pitfall 4: Phantom edges in a non-empty target DB +**What goes wrong:** Link `(from: 3, to: 7)` attaches to whatever rows 3 and 7 are in the target. +**Why it happens:** Treating `key` as an id, or running the links pass with a stale/incomplete map. +**How to avoid:** Keys only ever index the `HashMap` built in pass 2; pass 3 looks up both ends and drops (`links_dropped`) when either is missing (expired endpoint); kill-test per CONTEXT ``: pre-seed the target with rows whose ids overlap the file's keys, import, assert `edges_touching(pre_existing_id) == 0`. + +### Pitfall 5: Dangling-key export from a split read +**What goes wrong:** The exporter's own output fails the importer with `link key N does not appear in the file`. +**Why it happens:** Memories and links SELECTed on different connections while another process writes. +**How to avoid:** Pattern 4 — one connection, one read transaction, memories then links. Test is hard to make deterministic; a code-review check plus the FK argument suffices (document in the trait doc). + +### Pitfall 6: Expired rows imported "successfully" then swept +**What goes wrong:** Report says `imported: 3`, an hour later `memory_list` shows 2 (PITFALLS 19a). +**Why it happens:** Preserving `expires_at` without the import-time check. +**How to avoid:** `expires_at.is_some_and(|t| t <= now)` → `skipped_expired += 1`, no insert, no map entry. Note the sweep's own predicate is `expires_at < now` (strict) `[VERIFIED: store/mod.rs:219]`; D-10's `<=` is the locked import rule — document both in the format spec so the boundary case is deliberate. + +### Pitfall 7: Export file briefly world-readable or left half-written +**What goes wrong:** A 0644 file for a moment, or a truncated `PATH` after a crash. +**Why it happens:** `File::create` (0666 & umask) then chmod; or writing directly to `PATH`. +**How to avoid:** Pattern 7: `create_new` + `mode(0o600)` on the temp, rename after flush; remove the temp on any error; test with `std::os::unix::fs::PermissionsExt::mode() & 0o777 == 0o600` under `#[cfg(unix)]`, and a test that an existing `PATH` is replaced. + +### Pitfall 8: `BufWriter` dropped without flush / stdout EPIPE +**What goes wrong:** Last records missing from the file; `export | head` panics or errors noisily. +**Why it happens:** `BufWriter` flushes on drop but swallows errors; a closed pipe returns `BrokenPipe`. +**How to avoid:** Explicit `out.flush()` inside the blocking task before returning `Ok`; map `io::Error` → `MemoryError::Internal("export write failed: …")`; in `main.rs`, treat `ErrorKind::BrokenPipe` as a quiet non-zero exit (ASSUMED-nice-to-have, A6). + +### Pitfall 9: `CARGO_PKG_VERSION` ambiguity +**What goes wrong:** The header says `0.0.1` from the core crate while the binary is `0.1.0` (or vice versa) after a partial bump. +**Why it happens:** Two crates, two `version =` lines (`crates/agent-memory-core/Cargo.toml:3`, `crates/agent-memory/Cargo.toml:3`, both `0.0.1` `[VERIFIED]`); `env!` resolves to whichever crate compiles it. +**How to avoid:** Pass the version string from `main.rs` (`env!("CARGO_PKG_VERSION")` of the binary) into `MemoryService::export`; tests pass a literal. Phase 6's release checklist bumps both crates together. + +### Pitfall 10: `deny_unknown_fields` strictness vs `Option` leniency +**What goes wrong:** A hand-edited file omitting `source` parses (as `None`) while the same file with `"sourc":null` is rejected — surprising but consistent. +**Why it happens:** serde treats missing `Option` as `None` (probe); `deny_unknown_fields` only governs extra keys. +**How to avoid:** Accept it (document: "optional fields may be omitted; the exporter always writes them"), or add `#[serde(deserialize_with = "…")]`/a required marker — not worth it. Golden tests pin the exporter's canonical output. + +### Pitfall 11: Report arithmetic drift +**What goes wrong:** `imported + skipped_duplicates + skipped_expired != memory records` or `links_* != link records`. +**Why it happens:** Counting in two places or skipping a branch. +**How to avoid:** Count in exactly one match per pass; test asserts the sums for every fixture, including the gsd-state path's zeros (`import()` must set the four new fields to 0 — `ImportReport` is constructed only at `service.rs:425` and `:459` `[VERIFIED: grep "ImportReport {"]`, so no test literal breaks when fields are added). + +### Pitfall 12: Coverage gate regression from CLI-only tests +**What goes wrong:** `cargo llvm-cov --fail-under-lines 80` drops because new `main.rs` glue and `export_file.rs` are exercised only by spawned-binary tests (zero profile data). +**Why it happens:** `handlers.rs:456-460` explains the SIGKILL/profile issue. +**How to avoid:** Keep `main.rs` arms thin; unit-test `export_temp_path`/`create_private` in the binary crate (plain `#[test]` with tempdir); put all format logic in core tests. + +## Code Examples + +### 1. Golden field-order test for a memory record (D-04) +```rust +// crates/agent-memory-core/src/export.rs #[cfg(test)] +#[test] +fn memory_record_serializes_in_the_documented_order() { + let rec = Record::Memory { key: 7, mem_type: "DECISION".into(), content: "c".into(), + tags: vec!["a".into()], source: None, scope: Some("s".into()), + created_at: 1, last_accessed: 2, access_count: 3, expires_at: None }; + assert_eq!(serde_json::to_string(&rec).expect("serialize"), + r#"{"record":"memory","key":7,"type":"DECISION","content":"c","tags":["a"],"source":null,"scope":"s","created_at":1,"last_accessed":2,"access_count":3,"expires_at":null}"#); +} +// Probe-verified: tag first, then declaration order; Option::None → null. +``` + +### 2. Malformed-file table test shape (D-11) +```rust +// crates/agent-memory-core/tests/export_import.rs +const HEADER: &str = r#"{"record":"header","format":"agent-memory-export","version":1,"exported_at":1000,"embedding_model":"nomic-embed-text","embedding_dim":768,"schema_version":3,"agent_memory_version":"0.0.1"}"#; +const MEM1: &str = r#"{"record":"memory","key":1,"type":"DECISION","content":"a","tags":[],"source":null,"scope":null,"created_at":1,"last_accessed":1,"access_count":0,"expires_at":null}"#; + +#[test] +fn malformed_files_are_rejected_naming_the_line() { + let cases: &[(&str, &str, &str)] = &[ + ("no header", &format!("{MEM1}\n"), "line 1:"), + ("bad format", "{\"record\":\"header\",\"format\":\"x\",\"version\":1}\n", "line 1: format"), + ("version 0", "{\"record\":\"header\",\"format\":\"agent-memory-export\",\"version\":0}\n", "line 1:"), + ("version 2 + field","{\"record\":\"header\",\"format\":\"agent-memory-export\",\"version\":2,\"later\":1}\n", "upgrade agent-memory"), + ("not an object", &format!("{HEADER}\n[1,2]\n"), "line 2: expected a JSON object"), + ("unknown record", &format!("{HEADER}\n{{\"record\":\"memo\",\"key\":1}}\n"), "line 2:"), + ("unknown field", &format!("{HEADER}\n{}\n", MEM1.replace("\"expires_at\":null", "\"expires_at\":null,\"bogus\":1")), "line 2: unknown field"), + ("bad type", &format!("{HEADER}\n{}\n", MEM1.replace("\"key\":1", "\"key\":\"1\"")), "line 2:"), + ("bad kind", &format!("{HEADER}\n{MEM1}\n{{\"record\":\"link\",\"from\":1,\"to\":1,\"kind\":\"loves\",\"created_at\":1}}\n"), "kind must be one of"), + ("self link", &format!("{HEADER}\n{MEM1}\n{{\"record\":\"link\",\"from\":1,\"to\":1,\"kind\":\"relates_to\",\"created_at\":1}}\n"), "must differ"), + ("dangling key", &format!("{HEADER}\n{MEM1}\n{{\"record\":\"link\",\"from\":1,\"to\":9,\"kind\":\"relates_to\",\"created_at\":1}}\n"), "does not appear"), + ]; + for (name, text, needle) in cases { + let err = parse_jsonl(text).expect_err(name); + assert!(matches!(err, MemoryError::InvalidArgument(_)), "{name}: {err:?}"); + assert!(err.to_string().contains(needle), "{name}: {err}"); + } +} +``` + +### 3. Non-empty-DB overlapping-rowid kill-test (PITFALLS 18) — shape +```rust +#[tokio::test] +async fn importing_into_a_non_empty_db_never_attaches_phantom_edges() { + // Source: 3 memories (ids 1..=3) + link 1→2. Target: pre-seed 3 unrelated memories so ids 1..=3 are taken. + let src = harness(); /* store a,b,c; link(a,b,RelatesTo) */ let file = export_to_string(&src).await; + let dst = harness(); let pre: Vec = /* store x,y,z */; + let report = dst.service.import_jsonl(parse_jsonl(&file).expect("valid")).await.expect("import"); + assert_eq!((report.imported, report.links_imported), (3, 1)); + for id in pre { assert_eq!(edges_touching(&dst.db_path, id), 0, "pre-existing row {id} gained an edge"); } + // Then: export dst, import again → imported 0, skipped_duplicates 3, links_imported 0, links_skipped_duplicates 1. +} +``` +(`edges_touching` is the raw-connection helper already in `tests/relations.rs:95-104` `[CITED]`; copy it or lift it into a shared test helper.) + +### 4. Export sink selection in `main.rs` +```rust +Command::Export { output } => { + let (sink, finish): (Box, Box anyhow::Result<()>>) = match output { + None => (Box::new(BufWriter::new(std::io::stdout().lock())), Box::new(|| Ok(()))), + Some(path) => { + let temp = export_file::export_temp_path(&path); + let file = export_file::create_private(&temp) + .with_context(|| format!("creating {}", temp.display()))?; + let writer = BufWriter::new(file.try_clone().context("duplicating export handle")?); + (Box::new(writer), Box::new(move || { file.sync_all()?; std::fs::rename(&temp, &path)?; Ok(()) })) + } + }; + let report = service.export(sink, env!("CARGO_PKG_VERSION").to_string()).await.context("exporting")?; + finish()?; // on Err above: remove_file(temp) best-effort before returning + eprintln!("exported {} memories and {} links", report.memories, report.links); // stderr: stdout may be the file +} +``` + +### 5. `meta` pin read for the header (D-03) +```rust +// store/mod.rs +/// The 0002 embedding pin: `(embedding_model, embedding_dim)` from `meta`. A missing or non-integer +/// value is a crate-authored `Internal` (never a client error) — the table is written only by 0002. +fn embedding_pin(&self) -> Result<(String, u32), MemoryError>; +// sqlite.rs +let model: String = conn.query_row("SELECT value FROM meta WHERE key = 'embedding_model'", [], |r| r.get(0))?; +let dim_text: String = conn.query_row("SELECT value FROM meta WHERE key = 'embedding_dim'", [], |r| r.get(0))?; +let dim = dim_text.parse::() + .map_err(|e| MemoryError::Internal(format!("meta.embedding_dim is not an integer ({dim_text:?}): {e}")))?; +``` +`meta` is `(key TEXT PRIMARY KEY, value TEXT NOT NULL)` seeded with `('embedding_model', 'nomic-embed-text'), ('embedding_dim', '768')` `[VERIFIED: crates/agent-memory-core/sql/0002_embeddings.sql:21-26]` — values are TEXT, hence the parse. `schema_version` in the header is `LATEST_SCHEMA_VERSION` (3). + +### 6. Pending → sweep → searchable proof (criterion 3) +```rust +#[tokio::test] +async fn imported_rows_land_pending_and_the_sweep_backfills_them() { + let h = harness_with(Arc::new(FakeEmbedder::with_vectors(HashMap::new()))); // embedder UP, still never called by import + h.service.import_jsonl(parsed).await.expect("import"); + assert_eq!(status(&h, id), 0); // raw: SELECT embedding_status (backfill_race.rs:205 shape) + assert_eq!(vec_rows(&h, id), 0); // no vector travelled or was written inline + h.service.sweep(h.clock.now()).await.expect("sweep"); // BACKFILL_BATCH = 128 per tick + assert_eq!(status(&h, id), 1); + let out = h.service.search(SearchArgs { query: "…".into(), ..Default::default() }).await.expect("search"); + assert_eq!(out.search_mode, SearchMode::Hybrid); +} +``` + +### 7. D-14 redaction test for a write path and `/health` +```rust +// core: tests/redaction.rs +#[tokio::test] +async fn a_storage_fault_on_store_is_redacted() { + let h = harness(); + raw_conn(&h.db_path).execute_batch("ALTER TABLE memories RENAME TO memories_offline;").expect("rename"); + let err = h.service.store(new_memory("x")).await.expect_err("insert must fail"); + assert!(matches!(err, MemoryError::Internal(ref m) if m == "store failed due to a storage error (details are in the server log)"), "{err:?}"); + assert!(!err.to_string().to_lowercase().contains("sqlite")); +} +// binary: rest/handlers.rs tests — health_handler(State(test_state(&dir))) after the same rename → 500, body["error"] has no "sqlite" +``` + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| `redact_search_error` on `search` only | `redact_storage_error(op, e)` on every public service method | This phase (D-14, closes 04-REVIEW IN-01) | `/health`, write paths and the CLI never leak `sqlite error: …`; README sentence widened | +| `Store::exists -> bool` | `Store::find_by_key -> Option` | This phase (D-07) | Dedup hits map export keys to existing ids | +| `LinkOutcome::Linked` (unit) | `LinkOutcome::Linked { created: bool }` | This phase (D-12 countability) | Transport JSON unchanged; D-02 idempotency test becomes stronger | +| Import = `gsd-state` only, tolerant | `--from jsonl` strict sibling | This phase (D-11) | Two importers with opposite tolerance philosophies, both documented | +| Backup = same-machine `.backup-pre-*` copy | Portable JSONL export (vectors excluded) | This phase (DIST-04) | README should contrast "backup" (`VACUUM INTO`/backup file) vs "portable export" (CONTEXT discretion) | + +**Deprecated/outdated:** none in scope. The 04-REVIEW IN-01 suggested fix location ("move the redaction into the two transport mappers") is superseded by D-14's service-seam placement (one change covers the CLI too). + +## Assumptions Log + +| # | Claim | Section | Risk if Wrong | +|---|-------|---------|---------------| +| A1 | `rusqlite::Connection::unchecked_transaction()` (or an explicit `BEGIN`) on a pooled read connection gives the single WAL snapshot Pattern 4 needs | Pattern 4 | Low — fallback is `execute_batch("BEGIN")`/`COMMIT` or a single JOIN-free statement pair on one connection without a tx (WAL readers are still snapshot-consistent per statement; the risk is only the straddle) | +| A2 | Modern bundled SQLite rewrites trigger bodies on `ALTER TABLE memories RENAME`, so the fault-injection rename does not itself error | Pattern 8 recipe | Low — the hybrid tests already rename `memories_fts`/`vec_memories`; if `memories` RENAME misbehaves, rename `memory_links` for link paths and use `PRAGMA writable_schema`-free alternatives such as renaming `memories_fts` for `/health` via a search-based probe | +| A3 | Rejecting a duplicate `key` within one file as malformed is within D-11's "strict" intent (not an enumerated case) | Pattern 3 | Low — alternative is last-wins; the exporter never produces duplicates (ids are unique) | +| A4 | Empty/whitespace-only lines are "not a JSON object" and therefore malformed (D-11), including a blank trailing line | Pattern 3 | Low — if the user prefers tolerance of blank lines, it is one `continue`; the exporter never emits them | +| A5 | Deterministic link ordering `(from_id, to_id, kind)` is acceptable under D-05 ("after all memory records" is the only locked ordering) | Pattern 4 | None — ordering is a planner discretion | +| A6 | Treating `BrokenPipe` on stdout as a quiet non-zero exit is desirable | Pitfall 8 | Cosmetic | +| A7 | Passing `agent_memory_version` from the binary (not `env!` in core) matches D-03's intent of "" | Pattern 4 / Pitfall 9 | None functionally while both crates are `0.0.1`; matters at the Phase 6 bump | +| A8 | `access_count` needs no extra bounds check (any `i64` accepted) | Pattern 3 | Negligible — provenance-only field (D-10) | + +## Open Questions (RESOLVED) + +> All four resolved by the orchestrator (auto mode, recommended options) and carried into the plans — see the inline `— RESOLVED:` markers. + +1. **Header `exported_rows`/`exported_links` counts (CONTEXT discretion)** — RESOLVED: included in format v1 (05-02 Task 1 `Record::Header { exported_rows, exported_links }`; 05-03 Task 1 parser count check) + - What we know: additive only if decided now (D-11 rejects unknown fields later); the exporter knows the counts only after streaming, but the header is line 1. + - What's unclear: whether a pre-count query (`SELECT count(*)`) inside the same read snapshot is worth two extra statements. + - Recommendation: include them — two `count(*)` on the same snapshot are cheap, and the importer can then reject a truncated file (`line N: file ends after 12 of 40 declared memory records`) — a real integrity check for a copied file. If omitted, it is gone for v1. + +2. **Where the clock-skew warning lives** — RESOLVED: `tracing::warn!` inside core `import_jsonl` (05-03 Task 1 `clock_skew_message`); reaches the CLI's stderr through the binary's subscriber + - What we know: D-10 says "the CLI prints … to stderr"; core has `tracing::warn!` (stderr in the binary) and the injected clock. + - Recommendation: emit it in `import_jsonl` via `tracing::warn!` (testable with the `TestClock`), which reaches stderr through the binary's subscriber — the CLI needs no extra code. + +3. **`LinkOutcome` change scope** — RESOLVED: `LinkOutcome::Linked { created: bool }` accepted as a core-API, non-wire change (05-01 Task 2) + - What we know: 7 sites; not a wire change. + - What's unclear: whether the user considers `LinkOutcome` part of the Phase 4 "closed" contract. + - Recommendation: proceed (it is `pub` in core, not on the wire); mention in the plan's summary. + +4. **`SqliteStore::open` errors on the CLI** still print raw `sqlite error:` text via anyhow (e.g. unreadable DB path). D-14 scopes redaction to `MemoryService` methods; `open` is outside. Recommendation: leave as is (local stderr, no transport). — RESOLVED: left as is, accepted as threat T-05-14 (05-01); the README scopes D-14 to every request the daemon serves and every CLI import/export. + +## Environment Availability + +| Dependency | Required By | Available | Version | Fallback | +|------------|------------|-----------|---------|----------| +| rustc / cargo | build + tests | ✓ | 1.94.1 (2026-03-25) `[VERIFIED]` | — | +| serde / serde_json (locked) | format v1 | ✓ | 1.0.228 / 1.0.150 `[VERIFIED: Cargo.lock]` | — | +| cargo-llvm-cov | coverage gate (CI + local) | ✓ | 0.8.7 `[VERIFIED]` | CI installs it with `cargo install cargo-llvm-cov --locked` | +| jq | `.claude/hooks/pre-bash-rust.sh` | ✓ | 1.6 | — | +| Ollama | none — D-13 forbids inline embedding; tests use `FakeEmbedder` | not required | — | — | +| Node/gsd-tools | planning tooling only | ✓ | — | — | + +**Missing dependencies with no fallback:** none. +**Missing dependencies with fallback:** none. + +## Validation Architecture + +### Test Framework +| Property | Value | +|----------|-------| +| Framework | cargo test (built-in) + `#[tokio::test]`; `FakeEmbedder`/`TestClock` via the `test-clock` feature self-dev-dependency `[VERIFIED: crates/agent-memory-core/Cargo.toml dev-dependencies]` | +| Config file | none — workspace `Cargo.toml` | +| Quick run command | `cargo test -p agent-memory-core --test export_import --test import --test redaction` | +| Full suite command | `cargo fmt --check && cargo clippy --workspace --all-targets --locked -- -D warnings && cargo test --workspace --locked` | +| Coverage gate | `cargo llvm-cov --workspace --fail-under-lines 80` (CI-enforced `[VERIFIED: .github/workflows/ci.yml:44]`) | + +### Phase Requirements → Test Map +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| DIST-04 | Header line carries format/version/`meta` model+dim/schema_version/agent_memory_version; `exported_at` from the injected clock | unit (core, in-process) | `cargo test -p agent-memory-core --test export_import header_` | ❌ Wave 0 | +| DIST-04 | Memory/link record golden field order (D-04/D-05); no `decay_score`/`base_weight`/`embedding_status`/vector fields | unit (core, `export.rs` `#[cfg(test)]`) | `cargo test -p agent-memory-core --lib export::` | ❌ Wave 0 | +| DIST-04 | Every row exported, ascending id; links after memories; `relates_to` endpoints as stored `(min,max)` | unit (core) | `cargo test -p agent-memory-core --test export_import export_` | ❌ Wave 0 | +| DIST-04 | `--output`: temp in same dir, 0600 on Unix, existing target replaced, temp removed on failure; stdout default | unit (binary crate, `#[cfg(unix)]` perms) + realism CLI | `cargo test -p agent-memory --bin agent-memory export_file::` ; `cargo test -p agent-memory --test cli` | ❌ Wave 0 | +| INTEROP-02 | Round trip into an EMPTY and a NON-EMPTY overlapping-rowid DB: counts, key→id remap, zero phantom edges; export→import→export diff modulo `key`/`last_accessed` | integration (core, real temp DB) | `cargo test -p agent-memory-core --test export_import round_trip_` | ❌ Wave 0 | +| INTEROP-02 | Idempotent re-import: `imported 0`, `skipped_duplicates N`, `links_imported 0`, `links_skipped_duplicates M`, edge count unchanged | integration (core) | `cargo test -p agent-memory-core --test export_import idempotent_` | ❌ Wave 0 | +| INTEROP-02 | Expiry fixture: one expired (skipped, counted, never inserted), one ancient (listable, low decay, `last_accessed` refreshed), one fresh; link to expired endpoint → `links_dropped` | integration (core, `TestClock`) | `cargo test -p agent-memory-core --test export_import expired_` | ❌ Wave 0 | +| INTEROP-02 | Imported rows `embedding_status = 0`, no vec rows; `sweep` backfills; then hybrid-searchable | integration (core) | `cargo test -p agent-memory-core --test export_import pending_` | ❌ Wave 0 | +| INTEROP-02 | Malformed-file table (header/format/version 0/version 2+field/non-object/unknown record/unknown field/bad type/bad kind/empty content/self-link/dangling key/duplicate key) → `InvalidArgument` with `line N:`; nothing written (row count unchanged) | unit (core, `jsonl.rs` + `export_import.rs`) | `cargo test -p agent-memory-core --test export_import malformed_` | ❌ Wave 0 | +| INTEROP-02 | gsd-state path unchanged and reports zeros for the four new fields | existing + 1 assertion | `cargo test -p agent-memory-core --test import` | ✅ `tests/import.rs` (extend) | +| INTEROP-02 | CLI: `import --from jsonl` prints all six counts + pending line; `--from bogus` still bails; exit non-zero on malformed file | realism (spawned binary) | `cargo test -p agent-memory --test cli` | ❌ Wave 0 | +| D-14 | `redact_storage_error("search", storage)` text == `SEARCH_STORAGE_FAILURE` (byte-identical); client/Internal/SchemaTooNew pass through | unit (core, `service.rs` `#[cfg(test)]`) | `cargo test -p agent-memory-core --lib redact_` | ❌ Wave 0 | +| D-14 | A write path (`store`) and `list` on a renamed `memories` table → `Internal(" failed …")`, no `sqlite` substring | integration (core) | `cargo test -p agent-memory-core --test redaction` | ❌ Wave 0 | +| D-14 | REST `/health` and `POST /api/memories` → 500 JSON `{"error"}` without `sqlite`; MCP `memory_store` → `internal_error` without `sqlite` | in-process handler/MCP unit tests | `cargo test -p agent-memory --bin agent-memory storage_failure` | partial (extend `handlers.rs`/`mcp.rs` test modules; T-04-19 search tests exist) | +| D-14 | Existing Phase 4 golden assertions still green (`hybrid.rs:729,760`, `handlers.rs:881`, `mcp.rs:1336`) | existing | `cargo test -p agent-memory-core --test hybrid && cargo test -p agent-memory --bin agent-memory` | ✅ | + +### Sampling Rate +- **Per task commit:** `cargo test -p agent-memory-core --test export_import --test import --test redaction` (< 30 s) — plus `cargo clippy -p agent-memory-core --all-targets -- -D warnings` +- **Per wave merge:** `cargo test --workspace --locked && cargo clippy --workspace --all-targets --locked -- -D warnings && cargo fmt --check` +- **Phase gate:** full suite + `cargo llvm-cov --workspace --fail-under-lines 80` green before `/gsd-verify-work` + +### Wave 0 Gaps +- [ ] `crates/agent-memory-core/tests/export_import.rs` — covers DIST-04 + INTEROP-02 (round trip, kill-tests, expiry fixture, idempotency, malformed table, pending→sweep); harness copied from `tests/relations.rs:33-61` (`Harness { service, clock, db_path, _dir }`) with `raw_conn`/`edge_count`/`edges_touching` helpers +- [ ] `crates/agent-memory-core/tests/redaction.rs` — covers D-14 on `store` and `list` (RENAME recipe) +- [ ] `crates/agent-memory-core/src/export.rs` `#[cfg(test)]` — golden serialization + `deny_unknown_fields` probes (the 11 probe cases from this research as the table) +- [ ] `crates/agent-memory/tests/cli.rs` — realism-only spawned-binary round trip (`CARGO_BIN_EXE_agent-memory`, `--db` on a tempdir — the `tests/tools.rs:12-13` pattern) +- [ ] In-process `/health` + `store` redaction tests appended to `rest/handlers.rs` and `mcp.rs` test modules (`test_state`/`test_server` harnesses already exist) +- [ ] Framework install: none — cargo test is built-in + +*Task-level TDD (Phase 3/4 precedent): each new test file lands RED inside the owning task before its implementation.* + +## Security Domain + +### Applicable ASVS Categories (Level 1) + +| ASVS Category | Applies | Standard Control | +|---------------|---------|-----------------| +| V2 Authentication | no | CLI-only surfaces on the user's own machine (D-01); no new network endpoint | +| V3 Session Management | no | — | +| V4 Access Control | yes | Export file `0600` at creation (`OpenOptionsExt::mode`), temp in the DB's own directory (never `/tmp`), never looser than the DB (T-01-03 twin); import path accepted only from the CLI argument — no REST/MCP file-path primitive (PITFALLS security row) | +| V5 Input Validation | yes | serde typed records + `deny_unknown_fields`; `MemoryType::try_from` / `LinkKind::try_from`; `validate_link_endpoints`; whole-file validate-before-write; all SQL parameterised (`params![]`) — `kind`/`type` never reach SQL unparsed | +| V6 Cryptography | no | none — no hashing/signing in scope (checksums are a Phase 6 release concern) | +| V7 Error Handling & Logging | yes | D-14: storage-engine text only in the stderr log at `error` level; clients see `" failed due to a storage error (details are in the server log)"`; client errors echo only client input and public bounds | +| V12 Files & Resources | yes | `create_new` (no clobbering a stranger's file through a temp-name race), same-directory rename (no cross-FS partial copy), `read_to_string` only on the user-supplied path; file content is stored strictly as data (import/mod.rs invariant) | + +### Known Threat Patterns for this stack + +| Pattern | STRIDE | Standard Mitigation | +|---------|--------|---------------------| +| World-readable export of a sensitive corpus | Information Disclosure | `mode(0o600)` at creation; README documents Windows ACL inheritance as Phase 6 | +| Hand-edited file injecting junk `type`/`kind` into SQL | Tampering | closed enums via `try_from`; parameterised INSERTs | +| Rowid collision / phantom edges in the target DB | Tampering (data integrity) | export-local keys + two-pass map; kill-test | +| Model-mismatched vectors poisoning `vec_memories` | Tampering | no vectors in the file; re-embed via sweep against the local pin | +| Raw SQLite text (table/column names) to clients | Information Disclosure | D-14 redaction at the service seam; tests assert no `sqlite` substring | +| Half-written export left on crash | Denial of Service (data loss) | temp + rename; remove temp on error | +| Half-restored corpus on a malformed file | Tampering / DoS | validate-before-write (D-08/D-11); row-count-unchanged assertion in tests | +| Resource exhaustion on import of a huge file | DoS | accepted: pass 1 holds parsed records in memory (D-08); content size bounded only by what `store` accepts — document | +| Sweep deleting "successfully imported" expired rows | Repudiation (UX) | import-time `expires_at <= now` skip + `skipped_expired` count | + +## Sources + +### Primary (HIGH confidence — read with the Read tool this session) +- `crates/agent-memory-core/src/service.rs` (redaction helper 33-60, public methods, `import()` 394-463, `ImportReport` 162-168, `BACKFILL_BATCH` 749) +- `crates/agent-memory-core/src/domain.rs` (`MemoryError` 375-426, `NewMemory` 245-253, `LinkOutcome` 176-183, `MemoryType`/`LinkKind` + `try_from`, `validate_*`) +- `crates/agent-memory-core/src/store/mod.rs` (trait, `exists` doc 81-91, `link` 174-194, `sweep_expired` 219-228) +- `crates/agent-memory-core/src/store/sqlite.rs` 120-445 (backup helpers, struct, `open`), 595-654 (`insert`), 735-804 (`pending_embeddings`, `exists`, `list`), 945-994 (`link`) +- `crates/agent-memory-core/src/import/{mod.rs,gsd_state.rs}`, `crates/agent-memory/src/{main.rs,config.rs}`, `crates/agent-memory-core/tests/import.rs`, `sql/0002_embeddings.sql`, `sql/0003_relations.sql`, `lib.rs` +- `.planning/phases/05-portable-export-import/05-CONTEXT.md`, `.planning/REQUIREMENTS.md`, `.planning/STATE.md` +- Scratch probe `/tmp/serde-probe` (serde `=1.0.228`, serde_json `=1.0.150`, `cargo run --offline`) — 11 deserialization cases + serialization order + `Deserializer::from_str` + trailing characters + +### Secondary (MEDIUM confidence — official docs via Context7, or repo files viewed via grep/sed) +- Context7 `/websites/serde_rs` — container-attrs (`deny_unknown_fields`, `flatten` limitation), enum-representations (internally tagged) +- Context7 `/websites/rs_serde_json` — `Error::line/column/classify`, `Deserializer::from_str/from_reader` +- Repo via grep/sed: `.planning/research/ARCHITECTURE.md` §4 + Anti-Pattern 4; `PITFALLS.md` 17–19 + Security/Performance/UX rows; `04-REVIEW.md` IN-01/WR-02; `tests/hybrid.rs:712-770`; `rest/handlers.rs:167-182,437-454,851-889`; `mcp.rs:35-50,1309-1344`; `rest/mod.rs:34-48`; `tests/relations.rs:1-104`; `embed/mod.rs:85-160`; `sql/0001_init.sql`; `.github/workflows/ci.yml`; `.claude/hooks/pre-bash-rust.sh`; `README.md` 414-535, 557-567; `04-RESEARCH.md`/`04-VALIDATION.md` Validation sections + +### Tertiary (LOW confidence per the classify-confidence seam — WebFetch of official std docs; content is authoritative, provider tier is not) +- https://doc.rust-lang.org/std/fs/fn.rename.html — replace semantics, syscalls, mount-point caveat +- https://doc.rust-lang.org/std/os/unix/fs/trait.OpenOptionsExt.html — `mode`, umask +- https://doc.rust-lang.org/std/io/struct.Stdout.html — `lock() -> StdoutLock<'static>` + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH — no new packages; versions read from `Cargo.lock`; serde behaviour probe-verified on the pinned versions +- Architecture: HIGH — every integration point read in source this session; call-site counts are grep-verified; the only assumptions (A1–A8) are low-risk and listed +- Pitfalls: HIGH for 1–6, 9, 11, 12 (source/probe-backed); MEDIUM for 7, 8, 10 (std docs + reasoning) +- D-14 compatibility: HIGH — the exact assertion sites and the byte-identical message template are verified + +**Research date:** 2026-10-25 +**Valid until:** 2026-11-24 (stable: no external moving parts; re-check only if the serde/serde_json pins change or a migration lands) diff --git a/.planning/phases/05-portable-export-import/05-REVIEW.md b/.planning/phases/05-portable-export-import/05-REVIEW.md new file mode 100644 index 0000000..1babcde --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-REVIEW.md @@ -0,0 +1,171 @@ +--- +phase: 05-portable-export-import +reviewed: 2026-10-25T12:43:01Z +depth: standard +review_kind: phase review (diff f96431b..HEAD, code commits 00d8080..033bd86) +files_reviewed: 19 +files_reviewed_list: + - crates/agent-memory-core/src/domain.rs + - crates/agent-memory-core/src/export.rs + - crates/agent-memory-core/src/import/jsonl.rs + - crates/agent-memory-core/src/import/mod.rs + - crates/agent-memory-core/src/lib.rs + - crates/agent-memory-core/src/service.rs + - crates/agent-memory-core/src/store/mod.rs + - crates/agent-memory-core/src/store/sqlite.rs + - crates/agent-memory-core/tests/export_import.rs + - crates/agent-memory-core/tests/import.rs + - crates/agent-memory-core/tests/redaction.rs + - crates/agent-memory-core/tests/relations.rs + - crates/agent-memory-core/tests/store.rs + - crates/agent-memory/src/export_file.rs + - crates/agent-memory/src/main.rs + - crates/agent-memory/src/mcp.rs + - crates/agent-memory/src/rest/handlers.rs + - crates/agent-memory/tests/cli.rs + - README.md +findings: + critical: 0 + warning: 3 + info: 3 + total: 6 +status: issues_found +--- + +# Phase 5: Code Review Report + +**Reviewed:** 2026-10-25T12:43:01Z +**Depth:** standard +**Files Reviewed:** 19 +**Status:** issues_found + +## Summary + +Reviewed the Phase 5 diff (`f96431b..HEAD`): the format-v1 wire types (`export.rs`), the strict JSONL reader (`import/jsonl.rs`), the `export` / `import_jsonl` service paths and the D-14 `redact_storage_error` seam (`service.rs`), the `insert_row` / `insert_imported` / `find_by_key` / `export_rows` / `Linked { created }` store changes (`store/mod.rs`, `store/sqlite.rs`), the atomic private `--output` writer (`export_file.rs`), the CLI arms (`main.rs`), the two transport pattern updates, the five test files and the README section. + +The locked decisions are implemented as written. Checked and found sound: the one-snapshot export (`unchecked_transaction` on one pooled connection, counts and both scans inside it, WAL); `PendingExport` (0600 at `create_new`, temp beside the target, `sync_all` then `rename`, temp removed on every failure path, `BufWriter` flushed inside the blocking task before `commit`); the lenient header probe before the strict parse; `rewrap` lifting `InvalidArgument` payloads under the line prefix; `serde_message` stripping serde's own position; the `--from jsonl` arm rejecting `--scope` and never opening the database on a parse failure; `clock_skew_message` with `saturating_sub`; `insert_row` as the single `memories` INSERT; `tx.changes()` read before commit for `Linked { created }`; every public `MemoryService` method wrapped and `SEARCH_STORAGE_FAILURE` text unchanged; no `unwrap()` / `expect()` outside tests; bound SQL only; no new crates; migrations untouched. Hostile-input probes (deep nesting, 1 MiB of `{`, NUL, i64 overflow in timestamps, negative / fractional / string `version`, duplicate keys, dangling link keys, self-links by file key) are all rejected or stored as data without panicking, and the CLI test proves invalid UTF-8 is refused at `read_to_string`. + +Three warnings remain, all in the import pass. The most important one is a legitimate-export case the kill-tests do not cover: two rows identical on the dedup key collapse to one on import (D-07, accepted), but a link *between* those two rows then becomes a self-link that bypasses the service's self-link seam and is silently swallowed by `INSERT OR IGNORE` (SQLite's `OR IGNORE` applies to `CHECK` violations), so it is reported as a skipped duplicate. The other two are an unbounded `access_count` that can poison the row so every later `export` of the database fails, and the dedup probe matching DB rows that are already expired-but-unswept, so a restore can attach imported edges to a row the next sweep deletes. + +No `` block was supplied, so there is no fallow section. The 04-REVIEW carry-forwards (IN-02, IN-04, IN-05, IN-06, WR-01) are unchanged by this phase and are not re-raised; IN-01 is closed by D-14. + +## Narrative Findings (AI reviewer) + +## Warnings + +### WR-01: A link between two rows that collapse on import becomes a self-link that `INSERT OR IGNORE` silently swallows and the report mis-counts + +**File:** `crates/agent-memory-core/src/service.rs:677-686` (pass 3 calls `store.link` directly), `crates/agent-memory-core/src/store/sqlite.rs:1161-1171` (`INSERT OR IGNORE` + `changes() == 1`), `crates/agent-memory-core/sql/0003_relations.sql:19` (`CHECK (from_id != to_id)`) + +**Issue:** The parser validates `from != to` on **file keys** (`jsonl.rs:338`), but D-07 then maps keys to ids through `find_by_key`, and two distinct keys can map to the **same** id. That is not exotic: `store` has no uniqueness constraint, so a user who stored the same `(source, type, content)` twice and linked the two rows produces a perfectly valid export (`identical_rows_collapse_to_one_on_import_and_count_a_duplicate` builds exactly this corpus, minus the link). On import both keys resolve to one id, and pass 3 calls `store.link(N, N, kind, ts)`: + +- the service-seam `validate_link_endpoints` is bypassed (the importer calls the store directly), so the only guard is the 0003 `CHECK`; +- `INSERT OR IGNORE` treats a `CHECK` violation as an ignorable conflict (SQLite: `OR IGNORE` applies to `UNIQUE`, `NOT NULL`, `CHECK` and `PRIMARY KEY`), so no error is raised and `changes()` is 0 — verified on this machine's SQLite: `INSERT OR IGNORE` into a `CHECK(f <> t)` table with `(1,1)` yields `changes() = 0` and no row; +- the importer therefore returns `Linked { created: false }` and counts the edge as `links_skipped_duplicates`, which the README defines as "edges that already existed between the same two rows". Nothing existed; the edge was lost, and the report claims idempotent success. + +Also note `relates_to`'s `canonical_endpoints` makes `(N, N)` in both orders, so this is independent of kind. + +**Fix:** Treat a post-remap self-pair as a drop (it is the link-level consequence of the D-07 collapse) and never reach the store with it: + +```rust +// service.rs, pass 3 +match (ids.get(&from), ids.get(&to)) { + (Some(a), Some(b)) if a == b => { + // Both endpoints collapsed onto one row (D-07): the edge has no + // second endpoint any more. Count it as dropped, like an expired + // endpoint, instead of handing the store a self-link that + // INSERT OR IGNORE would swallow as a "duplicate". + report.links_dropped += 1; + } + (Some(a), Some(b)) => match store.link(*a, *b, kind, created_at)? { /* unchanged */ }, + (None, _) | (_, None) => report.links_dropped += 1, +} +``` + +Add a kill-test next to `identical_rows_collapse_to_one_on_import_and_count_a_duplicate`: store the twin twice, `link(first, second, RelatesTo)`, export, import into an empty DB, assert `six(1, 1, 0, 0, 0, 1)` and `edge_count == 0`. Update the README `links dropped` row ("edges whose endpoint was skipped as expired **or collapsed into its other endpoint**"). + +### WR-02: An unbounded `access_count` in the file can poison a row so that every later `export` of the database fails + +**File:** `crates/agent-memory-core/src/import/jsonl.rs:293-330` (no range check on `access_count`), `crates/agent-memory-core/src/store/sqlite.rs:1386` (`access_count = access_count + 1`), `crates/agent-memory-core/src/store/sqlite.rs:891` (`row.get("access_count")` as `i64`) + +**Issue:** `access_count` is carried verbatim (D-10) with no bounds: `9223372036854775807` and negative values both parse and land in the row. The first search or update that bumps the row executes `access_count + 1`; SQLite does not raise on integer overflow in `+`, it converts the result to `REAL` (verified: `SELECT typeof(a) ... UPDATE t SET a = a + 1` on `i64::MAX` gives `real|9.22337203685478e+18`). rusqlite's `FromSql for i64` only accepts `Integer`, so the next `export_rows` fails on that row with `InvalidColumnType`, which `redact_storage_error` turns into `export failed due to a storage error (details are in the server log)` — for the **whole** database, persistently, until someone repairs the column by hand. `list`/`search` keep working because `row_to_view` does not read `access_count`, so the damage is invisible until the user tries to back up. One hand-edited line in a file the README calls "agent-memory's own contract" therefore breaks the tool's only portable backup path. `created_at` and `expires_at` are only compared or echoed, so they are safe; `access_count` is the one preserved column that is later used in arithmetic. + +**Fix:** Validate it in pass 1 like every other field, under the D-11 line prefix: + +```rust +// jsonl.rs, Memory arm, after the content check +if access_count < 0 { + return Err(invalid_line(line_no, format!("access_count must be >= 0 (got {access_count})"))); +} +``` + +and make the bump saturate so no path can ever push the column out of integer range: + +```sql +UPDATE memories SET last_accessed = ?1, + access_count = CASE WHEN access_count < 9223372036854775807 THEN access_count + 1 ELSE access_count END +WHERE id IN (...) +``` + +(the same expression at `sqlite.rs:1271` for `update`). Add the `i64::MAX` and `-1` cases to `malformed_files_are_rejected_naming_the_line` / `hostile_input_never_panics`. + +### WR-03: The dedup probe matches target rows that are already expired-but-unswept, so a restore can attach imported edges to a row the next sweep deletes + +**File:** `crates/agent-memory-core/src/service.rs:648-666` (expiry skip uses the **file** row only; `find_by_key` has no `now`), `crates/agent-memory-core/src/store/sqlite.rs:926-945` (`find_by_key` has no expiry predicate) + +**Issue:** D-10 skips a *file* row whose `expires_at <= now`, but the dedup probe that runs right after it is blind to the *target* row's expiry. A CLI import runs no sweep, so a target row whose TTL lapsed while no server was running is still present. If the file carries the same `(source, type, content)` with a later or null `expires_at`, the importer maps the key onto the expired row, counts `skipped_duplicates`, and pass 3 links every imported neighbour to it (`links_imported`). The next `serve` tick's `sweep_expired` then deletes that row and cascades those freshly imported edges away. The report said "skipped duplicate, N links imported"; the corpus ends with neither. This was latent in the gsd-state path (04 code), but Phase 5 makes it reachable: JSONL rows carry absolute `expires_at`, and links now ride on the mapping. + +**Fix:** Make the probe see the same world the file policy sees. Either give `find_by_key` the clock and exclude expired rows: + +```rust +fn find_by_key(&self, source: Option<&str>, mem_type: MemoryType, content: &str, now: i64) + -> Result, MemoryError>; +// SQL: ... AND content = ?3 AND (expires_at IS NULL OR expires_at > ?4) ORDER BY id LIMIT 1 +``` + +(then the file's live copy is inserted and the stale row is swept on its own), or, if the Phase 4 trait shape must stay, run `store.sweep_expired(now)?` as the first statement of the `import_jsonl` blocking hop so the target has no expired rows when the probe runs. Either way add a test: store with `ttl_secs = 1`, advance the `TestClock` past it without sweeping, import a file carrying that content with `expires_at: null` plus a link to it, sweep, and assert the memory and its edge survive. + +## Info + +### IN-01: `find_by_key` uses `LIMIT 1` with no `ORDER BY`, so the key-to-id mapping depends on scan order when duplicates exist + +**File:** `crates/agent-memory-core/src/store/sqlite.rs:926-945` + +**Issue:** The INTEROP-01 probe only ever answered "exists?", where any row would do. D-07 now reads its **id** and hangs edges on it. With duplicate rows (which `store` permits) the row chosen is whatever SQLite steps first; today that is rowid order because no index covers `(source, mem_type, content)`, but that is a planner accident, not a contract, and a future index would change which duplicate gets the imported edges between two runs. + +**Fix:** `... LIMIT 1` → `... ORDER BY id LIMIT 1`, and say "the lowest matching id" in the trait doc. + +### IN-02: No test proves `export` and `import_jsonl` are redacted (T-05-17 / T-05-20 rest on the wrapper alone) + +**File:** `crates/agent-memory-core/tests/redaction.rs` (covers store, list, import, forget, link, unlink, update, sweep), `crates/agent-memory-core/tests/export_import.rs` (no storage-fault case) + +**Issue:** The two new service paths are wrapped (`service.rs:716-724`, `622-626`), but neither has the `take_offline` proof the other eight paths have, so a future refactor that unwraps one would not turn anything red. `export` additionally has a second error family (`Internal("export write failed: …")`) whose pass-through is also untested at the service level. + +**Fix:** Add two tests to `redaction.rs`: rename `memories` away, call `service.export(Box::new(Vec::new()), "t".into())`, `assert_redacted(&err, "export")`; rename `memory_links` away, call `import_jsonl` on a parsed file with one link, `assert_redacted(&err, "import")`. + +### IN-03: `run_export` duplicates `open_service` and opens (or creates) the database before the `--output` path is validated + +**File:** `crates/agent-memory/src/main.rs:348-357` (duplicate of `open_service`, 318-329), `crates/agent-memory/src/main.rs:368` (`export_file::begin` after the open) + +**Issue:** The wiring block is copied rather than reused, and the order means `agent-memory export --output /nonexistent/dir/out.jsonl` first resolves, creates and migrates the default database and only then fails on `create_new`. The `import --from jsonl` arm deliberately validates before opening; `export` should match. + +**Fix:** Call `export_file::begin(&path)` before `open_service(db_flag, ollama_url)?` (abort the pending temp if the open fails), and replace lines 348-357 with `let service = open_service(db_flag, ollama_url)?;`. + +--- + +_Reviewed: 2026-10-25T12:43:01Z_ +_Reviewer: Claude (gsd-code-reviewer)_ +_Depth: standard_ + +## Fixes applied (2026-10-25) + +All six findings fixed on local `main`, one commit per finding (WR-03 and IN-01 share one, since both land in `find_by_key`). Every commit passed `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets --locked -- -D warnings` and `cargo test --workspace --locked` in the main checkout before it was made (final: 286 passed / 0 failed / 2 ignored). No migration, no new crate, no workflow change. + +| ID | Status | Commit | Change | +|---|---|---|---| +| WR-01 | fixed | `80e905b` | `import_jsonl` pass 3 counts a post-remap self-pair (both keys mapped to one id by D-07 dedup) as `links_dropped` and never calls `store.link` with it. Kill-test `a_link_between_two_rows_that_collapse_on_import_is_dropped_not_miscounted` (report `(1, 1, 0, 0, 0, 1)`, zero `memory_links` rows, re-import idempotent). README `links dropped` row widened to "…or whose two endpoints collapsed into one row". | +| WR-02 | fixed | `c28fc04` | Pass 1 rejects `access_count < 0` (`line N: access_count must be >= 0 (got -1)`); `bump_access` and `update`'s freshness bump saturate via `CASE WHEN access_count < 9223372036854775807 THEN access_count + 1 ELSE access_count END` (statement change only). Tests: `-1` and `2^63` rows in the malformed table, `i64::MAX` parses in the hostile-input test, direct `bump_access` saturation proof asserting `typeof(access_count) = integer`, and an import → `update` → `export` round trip of an `i64::MAX` row. README strict-list gains the rule. | +| WR-03 | fixed | `3c28202` | `Store::find_by_key(source, mem_type, content, now)` adds `AND (expires_at IS NULL OR expires_at > ?4)`, so an expired-but-unswept target row is not a dedup hit and imported edges never hang on a row the next sweep deletes. Both service callers (`import`, `import_jsonl`) pass their clock reading. Test: TTL-1 s twin stored, `TestClock` advanced without sweeping, file row with `expires_at: null` + a link imported → `(2, 0, 0, 1, 0, 0)`, no edge on the stale row; after `sweep` the memory and its edge survive. | +| IN-01 | fixed | `3c28202` | Same probe gains `ORDER BY id LIMIT 1`; trait doc states "the lowest matching live id". Store test asserts the lowest id wins with duplicate live rows. | +| IN-02 | fixed | `22cd8d7` | `tests/redaction.rs` gains `export_storage_failure_is_redacted` (rename `memories`, export into a `Vec` sink, op `export`) and `import_jsonl_storage_failure_is_redacted` (rename `memory_links`, import a parsed two-memory/one-link file, op `import`). | +| IN-03 | fixed | `621b29a` | `run_export` reuses `open_service`; with `--output`, `export_file::begin` runs before the open and the open sits inside the one result the existing `abort` covers. CLI test `an_unwritable_output_path_fails_before_the_database_is_created`: non-zero exit, stderr names the output step, `--db` file never created, no temp left (verified to fail against the pre-fix code). | diff --git a/.planning/phases/05-portable-export-import/05-VALIDATION.md b/.planning/phases/05-portable-export-import/05-VALIDATION.md new file mode 100644 index 0000000..2fc1554 --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-VALIDATION.md @@ -0,0 +1,97 @@ +--- +phase: 5 +slug: portable-export-import +# status lifecycle: draft (seeded by plan-phase) → planned (filled by plan-phase) → validated (set by validate-phase §6) +# audit-milestone §5.5 distinguishes NOT-VALIDATED (draft) from PARTIAL (validated + nyquist_compliant: false) (#2117) +status: planned +nyquist_compliant: true +wave_0_complete: false +created: 2026-10-25 +--- + +# Phase 5 — Validation Strategy + +> Per-phase validation contract for feedback sampling during execution. + +--- + +## Test Infrastructure + +| Property | Value | +|----------|-------| +| **Framework** | cargo test (built-in) + `#[tokio::test]`; `FakeEmbedder` / `TestClock` injected via the `test-clock` feature self-dev-dependency (`crates/agent-memory-core/Cargo.toml`) | +| **Config file** | none — workspace `Cargo.toml`; no framework install this phase | +| **Quick run command** | `cargo test -p agent-memory-core --test export_import --test import --test redaction` | +| **Full suite command** | `cargo fmt --all --check && cargo clippy --workspace --all-targets --locked -- -D warnings && cargo test --workspace --locked` | +| **Coverage gate** | `cargo llvm-cov --workspace --fail-under-lines 80` (CI-enforced, `.github/workflows/ci.yml:44`; `cargo-llvm-cov 0.8.7` is installed locally) | +| **Estimated runtime** | quick: ~30–60 s incremental; full: several minutes on a cold build (executors run clippy + full tests in the background with a log file and poll — watchdog ~600 s) | + +--- + +## Sampling Rate + +- **After every task commit:** Run `cargo test -p agent-memory-core --test export_import --test import --test redaction` plus `cargo clippy -p --all-targets --locked -- -D warnings` (every task's `` names its own narrower command too) +- **After every plan wave:** Run `cargo fmt --all --check && cargo clippy --workspace --all-targets --locked -- -D warnings && cargo test --workspace --locked` +- **Before `/gsd-verify-work`:** Full suite green AND `cargo llvm-cov --workspace --fail-under-lines 80` green +- **Max feedback latency:** ~60 s (quick command, incremental build) + +--- + +## Per-Task Verification Map + +> Wave 0 note: this phase uses task-level TDD (Phase 3/4 precedent) — each new test file is created RED by the owning task's `` block before its implementation lands. No separate Wave 0 plan. + +| Task ID | Plan | Wave | Requirement | Threat Ref | Secure Behavior | Test Type | Automated Command | File Exists | Status | +|---------|------|------|-------------|------------|-----------------|-----------|-------------------|-------------|--------| +| 05-01-T1 | 01 | 1 | INTEROP-02 (D-14, 04-REVIEW IN-01) | T-05-01, T-05-14 | storage faults on store/list/import/forget/link/unlink/update/sweep redacted to ` failed due to a storage error (…)`; `/health`, `POST /api/memories`, `memory_store` carry no `sqlite` substring; `SEARCH_STORAGE_FAILURE` byte-identical | unit (service.rs) + integration (core, RENAME fault injection) + in-process REST/MCP | `cargo test -p agent-memory-core --lib redact_storage_error && cargo test -p agent-memory-core --test redaction && cargo test -p agent-memory --bin agent-memory storage_failure` | ❌ `tests/redaction.rs` created RED by 05-01-T1; handlers.rs/mcp.rs test modules extended | ⬜ pending | +| 05-01-T2 | 01 | 1 | INTEROP-02 (D-07, D-12) | T-05-15 | NULL-safe id-returning dedup probe (`source IS ?1`); `link` reports `created` from `tx.changes()`; wire JSON unchanged | integration (core) + in-process transports | `cargo test -p agent-memory-core --test store find_by_key && cargo test -p agent-memory-core --test relations && cargo test -p agent-memory-core --test import && cargo test -p agent-memory --bin agent-memory link` | ✅ extends `tests/store.rs`, `tests/relations.rs` | ⬜ pending | +| 05-01-T3 | 01 | 1 | INTEROP-02 (D-10, D-13) | T-05-15, T-05-16 | `insert_imported`: `embedding_status = 0`, no vector row, `last_accessed = now`, provenance preserved; exactly one `memories` INSERT behind `insert_row` | integration (core, raw-connection reads) | `cargo test -p agent-memory-core --test store && cargo test -p agent-memory-core --test ttl && cargo test -p agent-memory-core --test backfill_race && cargo test -p agent-memory-core --test update` | ✅ extends `tests/store.rs` | ⬜ pending | +| 05-02-T1 | 02 | 2 | DIST-04 (D-01, D-03, D-04, D-05, D-06; edges: empty, ordering, concurrency-backstop) | T-05-04, T-05-11, T-05-19, T-05-20 | header from `meta` + injected clock; 11-key memory records, no vectors/derived fields; one read snapshot; export faults redacted; no MCP/REST route | unit (export.rs goldens) + integration (core) + real-binary e2e | `cargo test -p agent-memory-core --lib export:: && cargo test -p agent-memory-core --test export_import` + the `` stdout header check | ❌ `tests/export_import.rs` + `export.rs` tests created RED by 05-02-T1 | ⬜ pending | +| 05-02-T2 | 02 | 2 | DIST-04 (D-02) | T-05-02, T-05-03 | `create_new` + `mode(0o600)` temp beside the target; `sync_all` + `rename` after flush; temp removed on failure; existing target replaced | unit (binary crate, `#[cfg(unix)]` perms) + real-binary e2e | `cargo test -p agent-memory --bin agent-memory export_file::` + the `` `--output` check | ❌ `export_file.rs` tests created RED by 05-02-T2 | ⬜ pending | +| 05-03-T1 | 03 | 3 | INTEROP-02, DIST-04 (D-01, D-07, D-08, D-09, D-10, D-11, D-12, D-13) | T-05-05, T-05-06, T-05-08, T-05-13, T-05-17 | `parse_jsonl` validates the whole file before `SqliteStore::open`; keys index a map, never ids; no embedder call; provenance preserved, `last_accessed` refreshed; skew warning; import faults redacted | unit (jsonl.rs, service.rs) + integration (core round trip + idempotency) + real-binary e2e | `cargo test -p agent-memory-core --lib import::jsonl && cargo test -p agent-memory-core --lib clock_skew && cargo test -p agent-memory-core --test export_import round_trip && cargo test -p agent-memory-core --test export_import idempotent && cargo test -p agent-memory-core --test import` | ❌ `jsonl.rs` tests created RED by 05-03-T1; extends `tests/export_import.rs`, `tests/import.rs` | ⬜ pending | +| 05-03-T2 | 03 | 3 | INTEROP-02, DIST-04 (D-07, D-08, D-10, D-11, D-13; edges: both adjacency, both empty, INTEROP ordering) | T-05-04, T-05-05, T-05-07, T-05-08, T-05-09 | overlapping-rowid kill-test (`edges_touching(pre_existing) == 0`, `orphan_edges == 0`); dedup-to-existing adjacency; in-file duplicates collapse; `expires_at <= now` skipped + `links_dropped`; pending → sweep → hybrid; header-only / zero-byte; file-order independence; ≥ 18-case malformed table; hostile input never panics; nothing written on a failed parse | integration (core) + unit (jsonl.rs) | `cargo test -p agent-memory-core --test export_import && cargo test -p agent-memory-core --lib import::jsonl` | ✅ extends `tests/export_import.rs`, `jsonl.rs` tests | ⬜ pending | +| 05-04-T1 | 04 | 4 | DIST-04, INTEROP-02 (docs: D-01..D-13) | T-05-11, T-05-18 | README states CLI-only rationale, 0600/atomic `--output`, strict reader + upgrade message, timestamp policy, pending-embedding cadence, the six counts, backup-vs-export | docs (grep gates) | `test "$(grep -c '^## Export & import' README.md)" = 1 && test "$(grep -c 'VACUUM INTO' README.md)" -ge 1 && test "$(grep -c 'upgrade agent-memory' README.md)" -ge 1` | ✅ README exists | ⬜ pending | +| 05-04-T2 | 04 | 4 | DIST-04, INTEROP-02 (D-01, D-02, D-11, D-12) | T-05-07, T-05-11 | malformed file → non-zero exit naming the line AND the `--db` target is never created; invalid UTF-8 refused without `PANIC`; `--from bogus` / `--scope` with jsonl refused; D-01 transport grep empty | realism (spawned binary, zero coverage by design) | `cargo test -p agent-memory --test cli` | ❌ `tests/cli.rs` created by 05-04-T2 | ⬜ pending | +| 05-05-T1 | 05 (gap closure) | 5 | DIST-04 (D-02, D-03, D-05, D-06; 05-02 backstop truth 2.8 → explicit) | T-05-19, T-05-21 | a memory + link committed by a writer thread while `Store::export_rows` is paused on its first memory record are invisible to that export (raced file parses with `parse_jsonl`, header counts == record counts, no dangling key) and present in the next export; the write commits while the snapshot is open (fresh connection sees it); both hand-offs bounded by `recv_timeout` | integration (core; channel-ordered race, 5× stability run) | `cargo test -p agent-memory-core --test export_import a_concurrent_link_committed_mid_export` (+ 5 consecutive runs, see plan ``) | ✅ extends `tests/export_import.rs` (`Harness` gains `store: Arc`) | ✅ green 2026-10-25 — `export_import` 20/20, new test stable 5×, workspace 287 passed / 0 failed / 2 ignored, clippy clean (commit 6cb8893) | +| 05-05-T2 | 05 (gap closure) | 5 | INTEROP-02 (D-12, D-13; 05-VERIFICATION human item 2) | T-05-22, T-05-23, T-05-24 | real binary + live `nomic-embed-text`: A → `move.jsonl` → B lands `imported: 3` + `3 rows pending embedding`; first `serve-rest` sweep tick logs `embedded pending memories on sweep … backfilled=3`; `/health` `"embedder":"ready"`; `POST /api/search` with a paraphrase → `"search_mode":"hybrid"` with the imported memory in `results`; daemon stopped by trap | realism (scripted live-Ollama run; host-only, never CI; evidence recorded in 05-05-SUMMARY.md) | `bash .planning/phases/05-portable-export-import/05-05-realism.sh` (exit 0 + `PASS:` line) | ✅ `05-05-realism.sh` committed with the plan | ✅ green — realism run 2026-10-25 | + +*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky* + +--- + +## Wave 0 Requirements + +Satisfied structurally by task-level TDD (RED test files land inside the owning task, before implementation): + +- [x] `crates/agent-memory-core/tests/redaction.rs` — created RED by 05-01-T1 (D-14 on the eight non-search paths; harness copied from `tests/relations.rs:33-104`) +- [x] `crates/agent-memory-core/tests/export_import.rs` — created RED by 05-02-T1 (export half), extended by 05-03-T1/T2 (import half, kill-tests) +- [x] `crates/agent-memory-core/src/export.rs` `#[cfg(test)]` — created by 05-02-T1 (golden field order + `deny_unknown_fields` probes) +- [x] `crates/agent-memory/src/export_file.rs` `#[cfg(test)]` — created by 05-02-T2 (0600 / commit / abort / create_new / same-dir) +- [x] `crates/agent-memory-core/src/import/jsonl.rs` `#[cfg(test)]` — created by 05-03-T1 (basic set), completed by 05-03-T2 (≥ 18-case malformed table + hostile input) +- [x] `crates/agent-memory/tests/cli.rs` — created by 05-04-T2 (realism only) +- [x] In-process redaction tests appended to `rest/handlers.rs` and `mcp.rs` test modules — 05-01-T1 (`test_state` / `test_server` harnesses already exist) +- [x] Gap closure 05-05-T1 — extends the existing `tests/export_import.rs` (RED by construction: the test references `Harness.store`, which does not exist until the task adds it); 05-05-T2 runs the committed `05-05-realism.sh`, no test file +- [x] Framework install: none — cargo test is built-in + +--- + +## Manual-Only Verifications + +| Behavior | Requirement | Why Manual | Test Instructions | +|----------|-------------|------------|-------------------| +| ~~DIST-04 concurrency edge: memories and links are read inside ONE read transaction so a concurrent `link` cannot produce a dangling key (`must_haves.truths` backstop item in 05-02)~~ **RETIRED as manual — automated by gap closure 05-05-T1** | DIST-04 | Was: "a deterministic interleaving test would need a second process". 05-VERIFICATION.md (truth 2.8, `insufficient_spec`) did not accept the code-review reading as evidence for a backstop truth and named the in-process race test as sufficient | Now `cargo test -p agent-memory-core --test export_import a_concurrent_link_committed_mid_export`: a `std::thread` writer on the writer lane commits a memory + link while the exporter's read-pool snapshot is paused inside the visitor; the raced file parses strictly with exact header counts and excludes both; the next export includes both. The 05-02 code-review reading stands as background only. | +| INTEROP-02 live re-embed across databases (05-VERIFICATION.md human item 2, MVP user-flow step "on the other machine") | INTEROP-02 | Needs a live Ollama with `nomic-embed-text` — present on this host (Ollama 0.35.0, verified 2026-10-25), never in CI; a fake embedder already proves the mechanism (`imported_rows_land_pending_and_the_sweep_backfills_them`) | Run `bash .planning/phases/05-portable-export-import/05-05-realism.sh` from the repo root (builds the binary, seeds A via `--from gsd-state`, exports, imports into B, starts `serve-rest --addr 127.0.0.1:0`, waits for `embedded pending memories on sweep … backfilled=3`, posts a paraphrase to `/api/search`, kills the daemon, prints the evidence block). Expected: exit 0, last line `PASS: live Ollama realism run …`, `"search_mode":"hybrid"` with the SQLite-engine memory in `results` (expected rank 0; any other rank is recorded verbatim). Paste the evidence block into 05-05-SUMMARY.md § Live Ollama realism evidence. Result: PASS 2026-10-25 — search_mode hybrid, target rank 0, id 1; `backfilled=3` on the first sweep tick (~50 ms after `REST listening on 127.0.0.1:57875`), `/health` `"embedder":"ready"`; evidence: 05-05-SUMMARY.md § Live Ollama realism evidence | +| Coverage gate at phase end | all | CI runs it; locally it takes minutes | `cargo llvm-cov --workspace --fail-under-lines 80` once after plan 05-04 (record the figure in 05-04-SUMMARY.md) — done: 95.61% lines (05-04-SUMMARY.md); 05-05 adds one test and no production code, so the gate is unaffected | + +--- + +## Validation Sign-Off + +- [x] All tasks have `` verify or Wave 0 dependencies (11/11 tasks carry an `` command — 9 original + the two gap-closure tasks of 05-05; 05-05-T2's command is host-only realism, re-runnable by the verifier on this host, not a CI gate) +- [x] Sampling continuity: no 3 consecutive tasks without automated verify +- [x] Wave 0 covers all MISSING references (via task-level TDD RED phases) +- [x] No watch-mode flags +- [x] Feedback latency ≈ 60 s on the quick command (cargo incremental build; documented, accepted — matches the Phase 4 contract) +- [x] `nyquist_compliant: true` set in frontmatter + +**Approval:** planner sign-off 2026-10-25 (execution pending) diff --git a/.planning/phases/05-portable-export-import/05-VERIFICATION.md b/.planning/phases/05-portable-export-import/05-VERIFICATION.md new file mode 100644 index 0000000..4e1bf55 --- /dev/null +++ b/.planning/phases/05-portable-export-import/05-VERIFICATION.md @@ -0,0 +1,228 @@ +--- +phase: 05-portable-export-import +verified: 2026-10-25T13:46:52Z +status: passed +score: 35/35 must-haves verified +behavior_unverified: 0 +overrides_applied: 0 +mode: mvp +verified_at_commit: fe18db7 +requirements: + - id: DIST-04 + status: satisfied + - id: INTEROP-02 + status: satisfied +re_verification: + previous_status: human_needed + previous_score: 32/33 + previous_commit: efcc1dd + gaps_closed: + - "Truth 2.8 (05-02, `verification: backstop`): one pooled connection + one read transaction so a concurrent link cannot produce a dangling key. Now has explicit held-out evidence — `a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key` (tests/export_import.rs) run by the verifier 6 times at HEAD (1 in the 20/20 file run + 5 consecutive named runs), all green. A writer thread commits a memory + `caused_by` edge while `Store::export_rows` is paused on its first memory record; a fresh connection sees 4 rows / 3 edges while the snapshot still reports 3 / 2, no record carries the new id, the raced file parses with `parse_jsonl` (3 memories, 2 links), and the next export carries both (4 / 3). The atomic-rename half keeps its unit + CLI tests. Previous human item 1 resolved." + - "MVP user-flow step 'on the other machine' + live re-embed (previous human item 2): the verifier ran `05-05-realism.sh` first-hand at HEAD against the live Ollama 0.35.0 / nomic-embed-text on this host (2026-10-25T13:45:50Z): A seeded (3 rows) → `export --output move.jsonl` (header `exported_rows:3`, `exported_links:0`, model/dim pinned) → `import --from jsonl` into B prints the six-count line + `3 rows pending embedding` → `serve-rest` first sweep tick logs `embedded pending memories on sweep backfilled=3 skipped_stale=0` 52 ms after bind → `/health` `embedder: ready` → paraphrase `POST /api/search` answers `search_mode: hybrid` with the imported memory (id 1) at rank 0. Terminal `PASS:` line reached. Matches the recorded run in 05-05-SUMMARY.md byte-for-byte apart from timestamps/port/pid/workdir." + gaps_remaining: [] + regressions: [] +--- + +# Phase 5: Portable Export & Import — Verification Report + +**Phase Goal:** As a user, I want to export all my memories to a portable file and import them on another machine, so that my agent's memory moves with me — across machines, with no cloud sync. +**Verified:** 2026-10-25T13:46:52Z against HEAD `fe18db7` (gap-closure commits `6cb8893`, `d52c1ff`, `fe18db7` on top of the previously verified `efcc1dd`) +**Status:** passed +**Re-verification:** Yes — after gap-closure plan 05-05. Previous run: `human_needed`, 32/33 at `efcc1dd`, two `human_verification` items. +**Mode:** mvp (goal is a valid User Story: role = user; capability = export/import across machines; outcome = memory moves with me, no cloud sync) + +## Evidence basis for this re-verification + +- **Production code is unchanged since the previous verification.** `git diff --stat efcc1dd..HEAD -- crates/agent-memory-core/src crates/agent-memory/src crates/agent-memory-core/sql/ Cargo.lock .github/` is empty. The only code change in the gap wave is `crates/agent-memory-core/tests/export_import.rs` (+196/−4: `Harness.store: Arc`, `record_lines` helper, the new race test). The gap-wave commits touch exactly that file plus `05-VALIDATION.md`, `05-05-SUMMARY.md`, `ROADMAP.md`, `STATE.md` (`git show --stat`). No `.db` file was added (the only tracked database is the pre-existing Phase-4 fixture `tests/fixtures/v0.0.1.db`). +- **Consequently the 32 previously VERIFIED truths carry forward on two grounds:** (a) the code they were verified against is byte-identical, and (b) every scoped test binary that backed them was re-run by the verifier at HEAD with identical pass counts (table below). Their evidence columns are carried from the `efcc1dd` report with status `✓ VERIFIED (regression)`. +- **The two previously human-routed items were re-evaluated against new, directly observed evidence** (not SUMMARY claims): the verifier ran the race test itself (6 green runs) and ran the realism script itself against the live Ollama. + +### Scoped test runs performed by the verifier at HEAD (never `--workspace`) + +| Suite | Command | Previous (efcc1dd) | Now (fe18db7) | Status | +|---|---|---|---|---| +| Export/import integration proofs (incl. new race test) | `cargo test -p agent-memory-core --locked --test export_import` | 19 passed | **20 passed** / 0 failed | ✓ | +| Race test stability | `cargo test -q -p agent-memory-core --locked --test export_import a_concurrent_link_committed_mid_export` ×5 | — | 5/5 `1 passed`, each in 0.01 s | ✓ | +| Core unit tests (export goldens, jsonl table, clock skew, gsd-state) | `… --lib` | 63 passed | 63 passed | ✓ | +| Redaction fault injection | `… --test redaction` | 10 passed | 10 passed | ✓ | +| Store contract (`find_by_key`) | `… --test store` | 8 passed | 8 passed | ✓ | +| Migrations frozen | `… --test migration_hygiene` | 8 passed / 1 ignored | 8 passed / 1 ignored | ✓ | +| gsd-state importer | `… --test import` | (in lib run) | 2 passed | ✓ | +| Relations (`link` created flag consumers) | `… --test relations` | (not re-run) | 27 passed | ✓ | +| Transports + handler tests + `export_file` units | `cargo test -p agent-memory --locked --bin agent-memory` | 72 passed | 72 passed | ✓ | +| Real-binary CLI round trip, malformed, refusals, unwritable `--output` | `… --test cli` | 6 passed | 6 passed | ✓ | +| Live-Ollama realism flow | `bash .planning/phases/05-portable-export-import/05-05-realism.sh` | not run (routed to human) | **PASS** — `search_mode hybrid, imported memory id 1 at rank 0`, `backfilled=3` | ✓ | + +## User Flow Coverage + +User story: «As a user, I want to export all my memories to a portable file and import them on another machine, so that my agent's memory moves with me — across machines, with no cloud sync.» + +| Step | Expected | Evidence | Status | +|------|----------|----------|--------| +| Run `agent-memory export --output memories.jsonl` | Private JSONL: header, memories, links; `exported N memories and M links` on stderr | Carried: `main.rs:351-399`, `export_file.rs` (create_new + 0600, same-dir temp, sync + rename; target touched only by `rename` at line 94); CLI test asserts 4 lines, header first, mode 0600, no temp sibling. **Observed live:** `exported 3 memories and 0 links`; header `{"record":"header","format":"agent-memory-export","version":1,…,"embedding_model":"nomic-embed-text","embedding_dim":768,"schema_version":3,"agent_memory_version":"0.0.1","exported_rows":3,"exported_links":0}` | ✓ | +| Copy the file to another machine | Nothing machine-bound: no vectors, no row ids used as ids | Carried: `sqlite.rs:876-878` column list excludes `base_weight`/`decay_score`/`embedding_status`, never reads `vec_memories`; `memory_records_carry_exactly_the_documented_keys` PASS. The realism run moved the file between two independent databases (A → B) on this host; the file is plain text with no host-specific content, so a second physical host adds nothing the software could do differently | ✓ | +| Run `agent-memory import --from jsonl memories.jsonl` there | Memories/links recreated with target ids; six-count report + pending line; re-run reports only duplicates | Carried: `main.rs:273-300`, `service.rs:632-712`; tests `round_trip_into_an_empty_database_recreates_memories_and_links`, `importing_into_a_non_empty_db_with_overlapping_rowids_never_attaches_phantom_edges`, `re_importing_the_same_file_is_idempotent` PASS. **Observed live:** `imported: 3, skipped duplicates: 0, skipped expired: 0, links imported: 0, links skipped duplicates: 0, links dropped: 0` + `3 rows pending embedding — the sweep backfills them on the next serve/serve-rest start (128 per tick)` | ✓ | +| Start the server; memories become searchable again locally | Rows land pending, sweep re-embeds against the local model pin, hybrid search works | Carried: `insert_imported` hard-codes `embedding_status: 0`, no vec write; `imported_rows_land_pending_and_the_sweep_backfills_them` PASS (fake embedder). **Observed live (real nomic-embed-text):** `REST listening on 127.0.0.1:64759` → `semantic search ready (nomic-embed-text)` → `embedded pending memories on sweep backfilled=3 skipped_stale=0` (+52 ms) → `/health` `{"embedder":"ready","status":"ok"}` → paraphrase "which database did we choose so that nothing needs a hosted service" → `"search_mode":"hybrid"` with "SQLite is the embedded storage engine…" (id 1) at rank 0 | ✓ | +| Outcome: "memory moves with me, no cloud sync" | Whole corpus round-trips via a local file; no network call on either side except the local embedder | All of the above; `run_import`/`run_export` never call the Ollama client (CLI tests pass with `AGENT_MEMORY_OLLAMA_URL=http://127.0.0.1:9`); the realism run's only network peer is `http://localhost:11434` | ✓ | + +## Goal Achievement + +### Observable Truths + +**Roadmap contract (Success Criteria 1–4) + user-approved scope addition D-14** + +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| SC1 | `export` writes a versioned JSONL file whose header records format version and embedding model/dim; embeddings excluded; relation edges included with export-local keys | ✓ VERIFIED (regression) | `export.rs:117-180` `Record` enum (`deny_unknown_fields`), `FORMAT_VERSION=1`; `sqlite.rs:845-851` header pin from `meta`; goldens `header_record_serializes_in_documented_order_with_record_first`, `export_writes_every_memory_ascending_then_every_link_as_stored`, `memory_records_carry_exactly_the_documented_keys` PASS (lib 63 + export_import 20). Live header observed above | +| SC2 | Import on another machine — incl. non-empty DB — recreates memories and link edges with correct key→id remapping; re-run idempotent | ✓ VERIFIED (regression) | `service.rs:646-705` two-pass `HashMap`; `find_by_key` NULL-safe; `importing_into_a_non_empty_db_with_overlapping_rowids_never_attaches_phantom_edges`, `link_endpoints_that_dedup_to_existing_rows_attach_to_those_rows`, `re_importing_the_same_file_is_idempotent` PASS; CLI re-import `imported: 0, skipped duplicates: 2 … links skipped duplicates: 1` (cli 6/6) | +| SC3 | Imported rows land `embedding_status = 0`, become semantically searchable after sweep backfill; no vectors travel | ✓ VERIFIED | Behavior-dependent (state transition 0 → 1). `insert_imported` (`sqlite.rs:720`), `import_jsonl_unredacted` never references `self.embedder`; `imported_rows_land_pending_and_the_sweep_backfills_them` PASS (status 0 / vec_rows 0 → status 1 / vec_rows 1, `SearchMode::Hybrid`); **and observed live** `backfilled=3` + `search_mode: hybrid` on the real embedder | +| SC4 | Expired items skipped and counted; six-count report; malformed files rejected as `InvalidArgument`, never 500/crash | ✓ VERIFIED (regression) | `service.rs:648-652`; `ImportReport` six fields; `jsonl.rs` `invalid_line` → `InvalidArgument("line N: …")`; 27-row `malformed_files_are_rejected_naming_the_line` + `hostile_input_never_panics` PASS (lib); CLI `a_malformed_file_exits_non_zero_naming_the_line_and_creates_no_database`, `invalid_utf8_is_rejected_cleanly` PASS | +| D-14 | Raw SQLite text redacted on every `MemoryService` path, both transports and CLI; `SEARCH_STORAGE_FAILURE` unchanged | ✓ VERIFIED (regression) | `service.rs:64-80` `redact_storage_error` exhaustive; 11 op wrappers; `tests/redaction.rs` 10/10; REST/MCP storage-failure tests in bin run 72/72 | + +**Plan 05-01** + +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| 1.3 | `Store::find_by_key` NULL-safe id probe; `import` dedups through it | ✓ VERIFIED (regression) | `sqlite.rs:926-957` `source IS ?1 … ORDER BY id LIMIT 1`; `find_by_key_returns_the_matching_id_null_safely` PASS (store 8/8) | +| 1.4 | `Store::link` reports `Linked { created }`; wire JSON unchanged | ✓ VERIFIED (regression) | `sqlite.rs:1179-1181` `tx.changes()`; relations 27/27; MCP link round trip in bin 72/72; the new race test also asserts `Linked { created: true }` on the writer lane | +| 1.5 | `insert_imported` preserves created_at/access_count/expires_at, refreshes last_accessed, status 0, no vector row | ✓ VERIFIED (regression) | `sqlite.rs:693-727`; round-trip test asserts preserved columns | +| 1.6 | README scopes redaction to every surface, behavioral change 4 | ✓ VERIFIED (regression) | `README.md:466`, `:518-525` (unchanged since efcc1dd) | + +**Plan 05-02** + +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| 2.1 | Header fields in order; model/dim from `meta`, `exported_at` from clock, version from `CARGO_PKG_VERSION` | ✓ VERIFIED (regression) | `service.rs:759-773` builds header from the `ExportSnapshot` (re-read this run); `main.rs:357`; `export_of_an_empty_database_is_exactly_one_header_line` PASS; live header shows `agent_memory_version: "0.0.1"` | +| 2.2 | Memory record carries exactly the 11 documented fields in order | ✓ VERIFIED (regression) | goldens PASS (lib 63) | +| 2.3 | Link record `record, from, to, kind, created_at`, endpoints as stored, after all memories | ✓ VERIFIED (regression) | `sqlite.rs:903-906` second scan; goldens PASS | +| 2.4 | Export writes every row incl. expired, no filter, ascending id, one streaming read | ✓ VERIFIED (regression) | no WHERE clause, `ORDER BY id`, lazy `query_map` (re-read `sqlite.rs:832-924`); `expired_rows_are_still_exported` PASS | +| 2.5 | Export is CLI-only | ✓ VERIFIED (regression) | `grep -rn "memory_export\|memory_import\|/api/export\|/api/import" crates/agent-memory/src/mcp.rs crates/agent-memory/src/rest/` → empty at HEAD | +| 2.6 | Empty DB → exactly one header line, 0/0 | ✓ VERIFIED (regression) | `export_of_an_empty_database_is_exactly_one_header_line` PASS | +| 2.7 | Ascending ids, links by `(from_id,to_id,kind)`, byte-identical re-export | ✓ VERIFIED (regression) | `two_exports_of_an_unchanged_database_are_byte_identical` PASS | +| 2.8 | **(backstop)** One pooled connection, one read transaction, so a concurrent link cannot produce a dangling key; `--output` temp-beside-target renamed so an interrupted export never leaves a partial target | ✓ VERIFIED (**gap closed**) | **Snapshot half — held-out test run by the verifier 6×:** `a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key`. Writer `std::thread` on the writer mutex commits `insert` + `link(new, a, CausedBy)` while the exporter's read-pool `unchecked_transaction` (`sqlite.rs:843`) is paused inside the visitor on the first `Record::Memory`; asserts a fresh connection sees 4 rows / 3 edges while the open snapshot reports `memories: 3, links: 2`, no emitted record carries `new_id`, the raced file (header mirrored from the snapshot exactly as `service.rs:759-773` does) is accepted by `parse_jsonl` with 3 / 2, and the next `export_to_string` reports 4 / 3 and contains both the new memory and the `caused_by` edge. Discriminating: two transactions (or none) would emit 3 link records and `parse_jsonl` would reject the dangling key. **Atomic half (unchanged):** `export_file.rs` opens only the temp (`create_new` + `0o600`, lines 125-138), the target is touched solely by `std::fs::rename` after `sync_all` (lines 90-94), so no code path can leave PATH partial even under SIGKILL; `commit_replaces_an_existing_target_and_leaves_no_temp`, `abort_removes_the_temp_and_keeps_the_target`, `begin_refuses_to_clobber_an_existing_temp` PASS (bin 72/72); CLI `an_unwritable_output_path_fails_before_the_database_is_created` PASS | +| 2.9 | `--output` creates `PATH.export-tmp-` beside PATH, 0600, `create_new`, rename after flush+sync, temp removed on failure, stdout default | ✓ VERIFIED (regression) | `export_file.rs:50-75,96-118,125-142`; `main.rs:361-393`; unit + CLI tests PASS | +| 2.10 | Export storage fault → redacted; writer I/O → `export write failed: …` | ✓ VERIFIED (regression) | `service.rs:742,790-791`; `redaction.rs::export_storage_failure_is_redacted` PASS (10/10) | + +**Plan 05-03** + +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| 3.2 | Re-import reports 0 imported / N dup / 0 links / M link-dup; source verbatim; identical rows collapse | ✓ VERIFIED (regression) | `re_importing_the_same_file_is_idempotent`, `identical_rows_collapse_to_one_on_import_and_count_a_duplicate` PASS | +| 3.4 | Timestamps preserved, last_accessed refreshed, expired skipped + links dropped, skew > 3600 s warns and continues | ✓ VERIFIED (regression) | `service.rs:633-636`; `clock_skew_message_fires_only_beyond_the_tolerance` (lib), `expired_rows_are_skipped_and_counted_and_their_links_dropped` PASS | +| 3.5 | type/kind via `try_from`, empty content & self-links `InvalidArgument`, tags array, expires_at int-or-null | ✓ VERIFIED (regression) | `jsonl.rs:300-349`; table rows PASS | +| 3.6 | Every malformed class rejected naming the 1-based line; nothing written; parse before DB open; gsd-state unchanged | ✓ VERIFIED (regression) | 27-row table; `nothing_is_written_when_validation_fails`; `main.rs:282-285`; `gsd_state.rs` untouched (import 2/2); live run's gsd-state seed printed the unchanged three-count line | +| 3.7 | `ImportReport` six counts; gsd-state zeros; CLI one line + pending line | ✓ VERIFIED (regression) | `service.rs:184-200`; `main.rs:287-299`; CLI pins strings; observed live verbatim | +| 3.8 | Import CLI-only; core exposes `parse_jsonl` + `import_jsonl` | ✓ VERIFIED (regression) | `pub fn parse_jsonl` (`jsonl.rs:219`), `pub async fn import_jsonl` (`service.rs:624`); transport grep empty | +| 3.9 | Two identical source rows both exported, collapse to one, one duplicate counted | ✓ VERIFIED (regression) | `identical_rows_collapse_to_one_on_import_and_count_a_duplicate`, `a_link_between_two_rows_that_collapse_on_import_is_dropped_not_miscounted` PASS | +| 3.10 | Header-only file imports as all-zero no-op | ✓ VERIFIED (regression) | `a_header_only_file_imports_nothing_and_reports_zeros` PASS | +| 3.11 | Link whose endpoints dedup to existing rows attaches to those rows | ✓ VERIFIED (regression) | `link_endpoints_that_dedup_to_existing_rows_attach_to_those_rows` PASS | +| 3.12 | Zero-byte / empty → `InvalidArgument` line 1 | ✓ VERIFIED (regression) | `a_zero_byte_file_is_rejected_at_line_one`, `an_empty_file_is_rejected_at_line_one` PASS | +| 3.13 | Links applied after all memories regardless of file order | ✓ VERIFIED (regression) | `links_are_applied_after_all_memories_regardless_of_file_order`, `links_may_precede_the_memories_they_name` PASS | + +**Plan 05-04** + +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| 4.1 | README "Export & import" documents usage, format v1, strict rules, dedup, timestamps, pending embeddings, six-count report, atomic `--output`, CLI-only rationale, backup tip | ✓ VERIFIED (regression) | `README.md:548-741` unchanged since efcc1dd | +| 4.2 | Real binary round-trips with six-count + pending line; re-import only duplicates | ✓ VERIFIED (regression) | `tests/cli.rs::export_import_round_trip_over_the_real_binary` PASS (6/6) | +| 4.3 | Malformed → non-zero exit naming line, DB not created; `--from bogus`/`--scope` refused; invalid UTF-8 no panic | ✓ VERIFIED (regression) | cli 6/6 | + +**Plan 05-05 (gap closure)** + +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| 5.1 | Memory + link committed mid-`export_rows` are invisible to that export (raced output parses, header counts == record counts, no absent link key) and appear in the next export; proven by `a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key` | ✓ VERIFIED — same truth as 2.8 (restated as explicit evidence; counted once, under 2.8) | see 2.8 | +| 5.2 | The writer COMMITS while the snapshot is open: never blocks behind the reader (WAL + `busy_timeout` on every connection, writer mutex vs read pool); a fresh connection sees row + edge before the export finishes; test fails by bounded timeout, never hangs | ✓ VERIFIED | Behavior-dependent (ordering invariant) — behavioral test passed 6×. `sqlite.rs:90-92` `busy_timeout 5000` then `journal_mode WAL` on every connection (production `SqliteStore::open`, which the test harness uses — not a fixture-only precondition); test asserts `row_count == 4` / `edge_count == 3` on a fresh raw connection inside the visitor while the tx is open; 3× `recv_timeout(Duration::from_secs(10))`, zero `sleep` in the file (grep). Ordering decided by the two `mpsc` hand-offs — code-enforced, not incidental | +| 5.3 | On this host with live Ollama + nomic-embed-text: A → B lands pending (`imported: 3, …` + `3 rows pending embedding …`), first `serve-rest` sweep tick logs `backfilled=3`, paraphrase `POST /api/search` answers `search_mode: hybrid` with the imported memory in `results`; captured in 05-05-SUMMARY.md and 05-VALIDATION.md | ✓ VERIFIED (directly observed) | Verifier ran `bash 05-05-realism.sh` at HEAD: every check passed and the terminal `PASS: live Ollama realism run — search_mode hybrid, imported memory id 1 at rank 0` line was reached (the script `exit 1`s on any failed check before that line). Output matches the block recorded in 05-05-SUMMARY.md "## Live Ollama realism evidence" (grep: heading ×1, `"search_mode":"hybrid"`, `backfilled=3`, six-count line all present) and 05-VALIDATION.md Manual-Only row `Result: PASS` | + +**Score:** 35/35 truths verified (33 previous + 05-05 truths 5.2 and 5.3; 5.1 dedups into 2.8). 0 present-behavior-unverified. 0 overrides. + +### Prohibitions (must-NOT, all `verification: test`) + +| Prohibition | Plans | Status | Enforcement evidence (direct observation at HEAD) | +|---|---|---|---| +| No migration / `LATEST_SCHEMA_VERSION` change | 01 | ✓ VERIFIED | `git diff --stat efcc1dd..HEAD -- crates/agent-memory-core/sql/` empty (and `00d8080^..efcc1dd` empty previously); `migration_hygiene` 8 passed / 1 ignored | +| `(source, mem_type, content)` identity contract unchanged | 01 | ✓ VERIFIED | `sqlite.rs` unchanged; `find_by_key_returns_the_matching_id_null_safely` PASS | +| gsd-state importer tolerance unchanged | 01, 03 | ✓ VERIFIED | `gsd_state.rs` unchanged; `--test import` 2/2; live run seed line `imported: 3, skipped duplicates: 0, malformed lines skipped: 0` | +| No embedding vectors in the file | 02 | ✓ VERIFIED | export SQL never touches `vec_memories` (re-read); goldens PASS; live header/records carry no vector field | +| Export/import never exposed over MCP or REST | 02, 03, 04 | ✓ VERIFIED (direct observation) | transport grep empty at HEAD; route table / tool list unchanged. *Info (carried):* no negative test pins the tool/route list | +| Export never filters or skips rows | 02 | ✓ VERIFIED | no WHERE clause; `expired_rows_are_still_exported` PASS | +| Import writes nothing when validation fails | 03 | ✓ VERIFIED | `nothing_is_written_when_validation_fails` PASS; CLI malformed test asserts `--db` never created | +| JSONL importer never calls the embedder / writes a vector | 03 | ✓ VERIFIED | `import_jsonl_unredacted` has no embedder reference; `imported_rows_land_pending_and_the_sweep_backfills_them` PASS; live run shows 3 rows pending until the sweep | +| Import never rewrites provenance (`source`, `scope` verbatim) | 03 | ✓ VERIFIED | round trip asserts; live run: `scope: "realism"` survived A → B | +| **05-05:** No production file changes — `crates/*/src` byte-identical | 05 | ✓ VERIFIED | `git diff --stat 30085a2..HEAD -- crates/*/src` empty; `git show --stat 6cb8893 d52c1ff fe18db7` lists only the test file + planning docs | +| **05-05:** No timed wait decides ordering — channel hand-offs, every wait `recv_timeout`-bounded | 05 | ✓ VERIFIED | `grep -n "thread::sleep\|sleep("` on the test file → empty; `recv_timeout` ×3 (go, done, and the writer's go wait) | +| **05-05:** No new crate, no CI/workflow change, no committed database file; realism DBs live in a kept `mktemp -d` dir | 05 | ✓ VERIFIED | `git diff --stat efcc1dd..HEAD -- Cargo.lock .github/` empty; `git ls-files | grep '\.db$'` → only the pre-existing Phase-4 fixture `tests/fixtures/v0.0.1.db`; realism workdir `/var/folders/…/tmp.o6tHWy5cR5` kept outside the repo | + +### Required Artifacts + +| Artifact | Expected | Status | Details | +|---|---|---|---| +| `crates/agent-memory-core/src/export.rs` | format v1 single definition, `Record`, consts, goldens | ✓ VERIFIED (regression) | unchanged; lib 63 PASS | +| `crates/agent-memory-core/src/import/jsonl.rs` | `pub fn parse_jsonl`, strict reader | ✓ VERIFIED (regression) | unchanged; consumed by `main.rs` and by the new race test | +| `crates/agent-memory-core/src/service.rs` | `redact_storage_error`, `export`, `import_jsonl`, `ImportReport`, `clock_skew_message` | ✓ VERIFIED (regression) | unchanged; header built from `ExportSnapshot` (re-read lines 750-795) | +| `crates/agent-memory-core/src/domain.rs`, `store/mod.rs`, `store/sqlite.rs` | `ImportedMemory`, `Linked { created }`, `find_by_key` / `insert_imported` / `export_rows`, one-snapshot `unchecked_transaction` | ✓ VERIFIED (regression) | unchanged; `sqlite.rs:843` tx spans pin, counts, both scans (re-read) | +| `crates/agent-memory/src/main.rs`, `export_file.rs` | `run_export`/`run_import`, `PendingExport` begin/commit/abort | ✓ VERIFIED (regression) | unchanged; `main.rs:231` emits `REST listening on` (the line the realism script parses) | +| `crates/agent-memory-core/tests/export_import.rs` | export + import proofs incl. phantom kill-test **+ `fn a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key`** | ✓ VERIFIED | 20 tests PASS; `contains` pattern present; `Harness.store: Arc` shared with `MemoryService::new` | +| `crates/agent-memory-core/tests/redaction.rs`, `crates/agent-memory/tests/cli.rs` | fault injection; spawned-binary realism | ✓ VERIFIED (regression) | 10/10, 6/6 | +| `.planning/phases/05-portable-export-import/05-05-realism.sh` | executable live-Ollama flow; contains `embedded pending memories on sweep` | ✓ VERIFIED | present, pattern present (wait_for + evidence grep); **executed by the verifier, PASS** | +| `.planning/phases/05-portable-export-import/05-VALIDATION.md` | 05-05-T1/T2 rows; concurrency Manual-Only row retired; live re-embed row filled | ✓ VERIFIED | rows `05-05-T1` (green, 20/20, 5× stable) and `05-05-T2` (green) present; Manual-Only concurrency row struck through "RETIRED as manual — automated by gap closure 05-05-T1"; live re-embed row `Result: PASS 2026-10-25 …` | +| `README.md` | `## Export & import`, redaction sentence, behavioral change 4 | ✓ VERIFIED (regression) | unchanged | + +### Key Link Verification + +Previous 13 links carry forward unchanged (production code identical). New 05-05 links, checked by hand against the patterns the plan declares: + +| From | To | Via | Pattern | Status | +|---|---|---|---|---| +| `tests/export_import.rs` | `store/sqlite.rs` `export_rows` | test drives `Store::export_rows` directly with a pausing visitor | `export_rows\(` | WIRED — `h.store.export_rows(&mut |event| …)` in the race test | +| `tests/export_import.rs` | `import/jsonl.rs` | raced output accepted by the strict reader | `parse_jsonl\(` | WIRED — `parse_jsonl(&raced)` asserts 3 / 2, then `parse_jsonl(&text)` asserts 4 / 3 | +| `tests/export_import.rs` | `store/sqlite.rs` writer lane | `std::thread` writer holding `Arc` calls `insert` then `link`; mpsc with `recv_timeout` both sides | `thread::spawn` | WIRED — `std::thread::spawn(move || { go_rx.recv_timeout … writer_store.insert … writer_store.link … done_tx.send })` | +| `05-05-realism.sh` | `crates/agent-memory/src/main.rs` | real binary: `import --from gsd-state`, `export --output`, `import --from jsonl`, `serve-rest --addr 127.0.0.1:0`, `POST /api/search` | `REST listening on` | WIRED — script `wait_for 'REST listening on'` + port parse; `main.rs:231` emits it; observed `REST listening on 127.0.0.1:64759` | + +### Data-Flow Trace (Level 4) + +Carried forward (unchanged code): export header/records flow from `meta` + `memories` + `memory_links` inside one snapshot tx; import report counters increment per store outcome; CLI pending line reads `report.imported`. **Live confirmation this run:** header `exported_rows:3` equals the 3 memory lines; B's `imported: 3` equals the pending line's `3 rows`; the sweep's `backfilled=3` equals the pending count; search returns the 3 imported rows with `scope: "realism"` and real `decay_score` values — every rendered number traces to real data, nothing static. + +### Behavioral Spot-Checks + +See "Scoped test runs performed by the verifier at HEAD" above — 10 scoped suites + the realism script, 0 failures. The race test was additionally run 5 consecutive times as a named single test (stability). Realism script exit-code capture via `${PIPESTATUS[0]}` printed empty under the zsh harness; the script's terminal `PASS:` line is only reachable after every guarded check (each failure path is `fail()` → `exit 1`), so the pass is unambiguous. + +### Probe Execution + +No `scripts/*/tests/probe-*.sh` exist and no plan declares a probe. Step 7c: SKIPPED (no probes declared). The committed `05-05-realism.sh` was executed as the equivalent runnable check (above). + +### Requirements Coverage + +| Requirement | Source Plan(s) | Description | Status | Evidence | +|---|---|---|---|---| +| DIST-04 | 05-02, 05-03, 05-04, 05-05 | Export all memories to a versioned JSONL file (header with format version + model/dim; embeddings excluded; edges with export-local key remapping) | ✓ SATISFIED | SC1 + truths 2.1–2.10 (2.8 now with held-out evidence) + 5.2 | +| INTEROP-02 | 05-01, 05-03, 05-04, 05-05 | Import on another machine — idempotent, rows land `embedding_status = 0` and re-embed via sweep, expired skipped and counted | ✓ SATISFIED | SC2–SC4 + truths 3.2–3.13 + 5.3 (live re-embed observed) | + +REQUIREMENTS.md lines 32-33 mark both `[x]`; lines 72-73 map exactly DIST-04 and INTEROP-02 to Phase 5 as Complete. No ORPHANED requirements. + +### Anti-Patterns Found + +Only one code file changed in the gap wave (`tests/export_import.rs`); production files were re-scanned by diff (none changed). + +| File | Line | Pattern | Severity | Impact | +|---|---|---|---|---| +| `crates/agent-memory-core/tests/export_import.rs` | — | No `TBD`/`FIXME`/`XXX`/`TODO`/`HACK`/`PLACEHOLDER`; no `thread::sleep`/`sleep(` (all waits channel-bounded) | — | — | +| `crates/agent-memory/src/mcp.rs`, `rest/mod.rs` | — | D-01 prohibition still has no negative regression test pinning the tool/route list (carried from previous report) | ℹ️ Info | A future `memory_export` tool would not turn anything red; grep + review remain the guards | +| `crates/agent-memory/src/export_file.rs` | 125-138 | After a SIGKILL mid-export a stale `PATH.export-tmp-` can remain (no cleanup can run); the target is never partial and a later run uses a different pid so does not collide | ℹ️ Info | Matches truth 2.8 as written ("never leaves a partial target"); the module doc (line 18) already documents stale temps as an expected artefact | + +### Human Verification Required + +None. Both items from the previous report are closed with explicit, re-runnable evidence that the verifier executed itself at HEAD: + +1. **One-snapshot export under concurrent writes (truth 2.8)** — closed by `a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key` (6/6 green runs). The previous item's additional kill-9 expectation "no `.export-tmp-*` sibling is left" over-reached the truth's actual statement (no partial target); the partial-target guarantee is structural (target touched only by `rename` after `sync_all`) and covered by the `export_file` unit tests. Not a gap. +2. **Real cross-machine move with live Ollama** — closed by running `05-05-realism.sh` first-hand against Ollama 0.35.0 / `nomic-embed-text`: `backfilled=3` on the first sweep tick, `/health` embedder ready, paraphrase search `search_mode: hybrid` with the imported memory at rank 0. The only un-exercised literal is copying the text file between two physical hosts, which is outside the software (the file carries no host-specific content — verified by the column list and the goldens). + +### Gaps Summary + +No gaps. Every roadmap success criterion, the D-14 scope addition, all five plans' truths (including the formerly abstained 05-02 backstop truth), all artifacts, all key links, all twelve prohibitions and the six review fixes are confirmed in the code at `fe18db7`. Production code is byte-identical to the previously verified `efcc1dd`; the gap wave added one discriminating held-out concurrency test and a recorded-and-now-re-observed live-Ollama realism run. 10 scoped test suites (226 tests) plus 5 stability re-runs of the race test and the live realism script all pass with 0 failures. No regressions against the previous verification. Phase 5 goal achieved; ready for Phase 6. + +--- + +_Verified: 2026-10-25T13:46:52Z_ +_Verifier: Claude (gsd-verifier) — re-verification after gap-closure plan 05-05_ diff --git a/.planning/phases/05-portable-export-import/COVERAGE.md b/.planning/phases/05-portable-export-import/COVERAGE.md new file mode 100644 index 0000000..f502c5b --- /dev/null +++ b/.planning/phases/05-portable-export-import/COVERAGE.md @@ -0,0 +1,18 @@ +# Phase 5 — API Coverage Declaration + +**Phase:** 05-portable-export-import +**Declared:** 2026-10-25 (planner, at plan-phase time) + +## Declaration + +No external API integration: Phase 5 adds file-based JSONL export/import over the existing embedded SQLite store and reuses the existing Ollama path untouched. + +## Reasoning + +- **Surfaces added:** two CLI subcommands (`agent-memory export [--output PATH]`, `agent-memory import --from jsonl PATH`). Both are local file I/O plus SQLite reads/writes through the existing `Store` trait and `MemoryService` seam (05-CONTEXT.md D-01). +- **No new SDK, HTTP client, or network endpoint.** The only network-capable component in the workspace is the Ollama embedder, and the JSONL importer never calls it (D-13): imported rows land `embedding_status = 0` and are re-embedded by the pre-existing sweep backfill, whose Ollama usage is unchanged. +- **No new MCP tool and no new REST route** (D-01), so the two transports' API surface is unchanged; the only transport-visible change is the storage-error redaction text widened to every path (D-14), which alters error message bodies, not endpoints or schemas. +- **No new crate dependency** (RESEARCH "Standard Stack" / "Package Legitimacy Audit": not applicable — serde, serde_json, rusqlite, tracing and std only; `Cargo.lock` unchanged). +- **No schema migration** (0001–0003 frozen, `LATEST_SCHEMA_VERSION = 3`). + +The API-coverage gate therefore has nothing external to cover for this phase; the format v1 file contract is documented in `crates/agent-memory-core/src/export.rs` (module doc) and the README "Export & import" section (plan 05-04). diff --git a/README.md b/README.md index 37a052c..3124d53 100644 --- a/README.md +++ b/README.md @@ -76,6 +76,9 @@ agent-memory serve # Choose the database file (default: your OS data dir; honors $AGENT_MEMORY_DB). agent-memory --db /path/to/memory.db serve AGENT_MEMORY_DB=/path/to/memory.db agent-memory serve + +# Move the memory elsewhere. +agent-memory export --output memories.jsonl # portable JSONL export (see Export & import) ``` The server speaks JSON-RPC over **stdout**; all logs go to **stderr**, so stdout @@ -457,9 +460,12 @@ axum's default body rejection: a **plain-text** (not JSON) response with 422 for a missing or wrongly typed field, 400 for malformed JSON, and 415 for a missing `Content-Type`. A body that does deserialize is validated as above, so an out-of-bounds `limit`/`ttl_secs` or an unknown `type` on those routes is -still a 400 `{"error"}`. A storage failure during search reports a generic -internal error (HTTP 500 / MCP `internal_error`, `search failed due to a -storage error`); the detail goes only to the server's stderr log. +still a 400 `{"error"}`. A storage failure on any request the daemon serves — +and on any CLI `import`/`export` — reports a generic internal error (HTTP 500 +/ MCP `internal_error`) shaped +` failed due to a storage error (details are in the server log)`; +the engine detail (SQLite error text, table and column names) goes only to +the server's stderr log. ```bash curl -X PATCH localhost:7437/api/memories/42 \ @@ -485,7 +491,7 @@ VPN/SSH tunnel or an authenticating reverse proxy. ## Behavioral changes in v0.1.0 -Three deliberate contract changes from v0.0.1 (all apply identically to MCP +Four deliberate contract changes from v0.0.1 (all apply identically to MCP and REST): 1. **Tag filtering is now an exact, case-sensitive match** instead of a @@ -509,6 +515,14 @@ and REST): if FTS5 cannot parse it is it retried once as its words OR-ed together. - Exact keyword hits now surface alongside paraphrase matches, so result order can differ from v0.0.1's embedding-only ranking. +4. **Storage faults on every surface now report a generic message** — + ` failed due to a storage error (details are in the server log)` + — instead of echoing the storage engine's text. v0.0.1 returned + `sqlite error: …` strings (with table and column names) on non-search 500s + / `internal_error`s; v0.1.0 redacts `store`, `list` (including `/health`), + `update`, `link`/`unlink`, `forget`, `import` and the sweep exactly as it + already did for `search`. Clients that parsed those strings must stop — the + detail is in the server's stderr log. ## Import from GSD @@ -531,6 +545,201 @@ scope (the project directory containing `.planning` by default; override with `--scope`). Re-running the same import is a **no-op**: unchanged items are deduplicated on the exact (source, type, content) key and reported as skipped. +## Export & import + +Move an agent's whole memory to another machine — or keep a copy that outlives +a change of embedding model — with one portable JSONL file. No cloud, no +account: + +```bash +agent-memory export > memories.jsonl # to stdout +agent-memory export --output memories.jsonl # to a private file (see below) +agent-memory import --from jsonl memories.jsonl # on the other machine +``` + +Both are **CLI subcommands only** — there is no MCP tool and no REST route for +either. A file-path read on the unauthenticated REST daemon would be an +arbitrary-file-read primitive, and the whole corpus must never cross the MCP +server's JSON-RPC stdout channel. The validation and error contract still live +in the core library, so any later transport would inherit them unchanged. +`--scope` is not accepted with `--from jsonl`: every memory in the file carries +its own scope. + +### What travels + +A header line, then one record per memory, then one record per link. Every +row in the database is exported — no scope or type filter, and expired rows are +included (the importer decides what to skip). + +**What does not travel:** embedding vectors, `decay_score`, `base_weight` and +`embedding_status`. They are derived data: the score is recomputed on read, the +weight is a function of `type`, and vectors are pinned to the exporting +machine's model — so they are re-created locally on import, against the +importing database's own model pin (see *Embeddings* below). + +### Format v1 + +The file is JSON Lines: one JSON object per line, UTF-8, `\n`-terminated. Each +object carries a `record` tag, and the tag set is closed (`header`, `memory`, +`link`). Fields appear in the wire order below, so two exports of an unchanged +database under the same clock are byte-identical. The authoritative definition +is the `export` module of `agent-memory-core`; this section restates it. + +**Header** (line 1, mandatory): + +| field | type | meaning | +|---|---|---| +| `record` | `"header"` | tag | +| `format` | string | always `agent-memory-export` | +| `version` | integer | `1` for files this version writes | +| `exported_at` | integer | unix seconds at export time | +| `embedding_model` | string | the source database's embedding-model pin | +| `embedding_dim` | integer | the source database's embedding-dimension pin | +| `schema_version` | integer | the source database's schema version | +| `agent_memory_version` | string | the exporting binary's version | +| `exported_rows` | integer | number of `memory` records that follow | +| `exported_links` | integer | number of `link` records that follow | + +**Memory** (one per row): + +| field | type | meaning | +|---|---|---| +| `record` | `"memory"` | tag | +| `key` | integer | the source row id, used **only** as an export-local key that `link` records refer to — it never becomes an id on import | +| `type` | string | `DECISION`, `PATTERN`, `ERROR`, `TODO`, `ARCHITECTURE` or `CONSTRAINT` | +| `content` | string | the memory text, verbatim | +| `tags` | array of strings | verbatim | +| `source` | string or `null` | provenance, verbatim | +| `scope` | string or `null` | verbatim | +| `created_at` | integer | unix seconds; preserved on import | +| `last_accessed` | integer | unix seconds; refreshed to import time on import | +| `access_count` | integer | preserved on import (provenance only) | +| `expires_at` | integer or `null` | absolute unix seconds; preserved on import | + +**Link** (one per relation edge, after all memories): + +| field | type | meaning | +|---|---|---| +| `record` | `"link"` | tag | +| `from` | integer | `key` of the source memory | +| `to` | integer | `key` of the target memory | +| `kind` | string | `relates_to`, `supersedes` or `caused_by` | +| `created_at` | integer | unix seconds; preserved on import | + +Memories are written in ascending `key` order and links ordered by +`(from, to, kind)`. `relates_to` endpoints appear exactly as stored — smaller +key first; directed kinds keep the direction they were given. Optional fields +(`source`, `scope`, `expires_at`) are always written explicitly as `null`. + +### The reader is strict + +The `gsd-state` importer is tolerant, because a `STATE.md` is a human-edited +file. A JSONL export is agent-memory's **own** contract, so any defect means a +corrupted or hand-edited file — and a half-restored corpus is worse than a +clear refusal. The whole file is validated first; **any** defect rejects **all** +of it, and nothing is written (the target database is not even opened): + +- a missing or invalid header, a `format` other than `agent-memory-export`, a + `version` that is not a positive integer, a second header; +- a line that is not a JSON object, an unknown `record` tag, an unknown field + (a typo, or a field from a newer format), a wrong field type (`"1"` is not an + integer); +- an invalid `type` or `kind`, empty content, a negative `access_count`, a + duplicate `key`, a self-link, a link to a `key` that does not appear in the + file; +- a record count that does not match the header's `exported_rows` / + `exported_links`. + +The message names the 1-based line, for example +`invalid argument: line 42: unknown variant `memo`, expected one of `header`, `memory`, `link``. +A hand-written file may omit an optional field (`source`, `scope`, +`expires_at`), which reads as `null`; expired rows are not malformed (see +*Timestamps*). + +**Version policy.** A file whose `version` is greater than 1 is refused with +`file is format version N but this binary reads up to version 1 — upgrade agent-memory`. +An older binary never silently reads a newer format. + +### Idempotent, and safe on a non-empty database + +Import works into an empty or an already populated database. Each memory is +deduplicated on the exact `(source, type, content)` key, with `source` kept +verbatim — so re-running the same import is a **no-op** reported as skipped +duplicates, and two source rows identical on that key collapse into one +(counted as a duplicate). Links are remapped through the file's own keys to +the target database's ids: a pre-existing row never gains an edge it did not +have, even when its id happens to equal a `key` in the file. + +### Timestamps + +| column | on import | +|---|---| +| `created_at` | preserved (provenance) | +| `expires_at` | preserved; a row already expired at import time (`expires_at <= now`) is **skipped and counted**, never inserted | +| `last_accessed` | **refreshed to import time**, so restored memories start decay-fresh instead of ranking as months-stale | +| `access_count` | preserved (provenance only — it does not affect ranking) | +| link `created_at` | preserved (edge recency orders relation expansion) | + +If the header's `exported_at` is more than an hour ahead of the local clock, a +clock-skew warning is logged and the import continues. + +### Embeddings + +The import never contacts Ollama. Every imported row lands **pending +embedding**, and the existing lifecycle sweep re-embeds it locally against the +importing database's model pin — 128 rows per tick, the first tick at +`serve`/`serve-rest` start and then hourly. Until that backfill catches up, a +large restore is keyword-searchable only; nothing is lost, and vectors from two +different models are never mixed in one database. + +### The report + +A successful import prints two lines on stdout: + +```text +imported: 42, skipped duplicates: 3, skipped expired: 1, links imported: 17, links skipped duplicates: 0, links dropped: 2 +42 rows pending embedding — the sweep backfills them on the next serve/serve-rest start (128 per tick) +``` + +| count | meaning | +|---|---| +| `imported` | memories inserted | +| `skipped duplicates` | memories already present on the `(source, type, content)` key | +| `skipped expired` | memories whose `expires_at` had already passed | +| `links imported` | edges created | +| `links skipped duplicates` | edges that already existed between the same two rows | +| `links dropped` | edges whose endpoint was skipped as expired or whose two endpoints collapsed into one row — not an error | + +`export` reports `exported N memories and M links` on **stderr**, because +stdout may be the file. + +### `--output` is private and atomic + +`export --output PATH` never writes PATH directly. It creates +`PATH.export-tmp-` in PATH's own directory with mode `0600` on Unix — the +corpus is sensitive, and the mode is applied at creation, so there is no +world-readable instant — streams into it, syncs, and renames it over PATH in +one step. An existing PATH is replaced; an interrupted export never leaves a +partial PATH behind. On Windows the file follows its directory's permissions +(the Windows ACL story ships with the Windows binaries). + +Without `--output` the file goes to stdout — handy for piping, but a terminal +scrollback or a shell log may keep a copy. + +### Backup vs export + +For a same-machine, full-fidelity copy — vectors included — stop every +agent-memory process and use SQLite's own backup: + +```bash +sqlite3 memory.db "VACUUM INTO 'memory-backup.db'" +``` + +or keep the automatic pre-migration backup described under *Upgrading*. A raw +database copy pins the schema version and the embedding model, so it is a +backup, not an interchange format. Use `export` when the memory must move +machines or survive a change of embedding model. + ## Lifecycle: TTL & decay A background task sweeps on an interval while the server runs: @@ -543,7 +752,8 @@ A background task sweeps on an interval while the server runs: embedded in batches once it comes back. A memory that is edited or deleted while its backfill is in flight is never overwritten with a vector of its old text: that write is skipped, and the memory's current content is - embedded on a later sweep (or by the edit itself, when Ollama is up). + embedded on a later sweep (or by the edit itself, when Ollama is up). Rows + restored by `agent-memory import --from jsonl` take this path too. **Decay never deletes.** A memory with no TTL stays retrievable forever, however low its decay score falls — it simply ranks lower. Removal happens only via TTL or diff --git a/crates/agent-memory-core/src/domain.rs b/crates/agent-memory-core/src/domain.rs index 59ecc4e..1768094 100644 --- a/crates/agent-memory-core/src/domain.rs +++ b/crates/agent-memory-core/src/domain.rs @@ -175,9 +175,12 @@ impl std::convert::TryFrom<&str> for LinkKind { /// before `to_id`), so a client never sees a foreign-key failure (T-04-13). #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum LinkOutcome { - /// The edge exists now — freshly created, or already present (a duplicate - /// link is an idempotent no-op success). - Linked, + /// The edge exists now. `created` is `true` when the row was inserted by + /// this call and `false` when the stored edge already existed — the D-02 + /// idempotent no-op, still a success. The distinction is storage-only + /// (transports emit the same JSON either way); the JSONL importer counts + /// the `false` case as `links_skipped_duplicates` (Phase 5 D-12). + Linked { created: bool }, /// No memory with `missing_id` exists; nothing was written. NotFound { missing_id: i64 }, } @@ -252,6 +255,38 @@ pub struct NewMemory { pub ttl_secs: Option, } +/// A memory restored from a portable format-v1 file (INTEROP-02), the input +/// of [`crate::store::Store::insert_imported`]. +/// +/// Unlike [`NewMemory`] — which is stamped with the clock and carries a +/// relative `ttl_secs` — this type carries ABSOLUTE timestamps copied from the +/// file, because a restored row keeps its provenance (Phase 5 D-10): +/// `created_at`, `access_count` and `expires_at` are preserved exactly. +/// `last_accessed` is deliberately NOT a field: it is refreshed to the +/// import-time `now` so restored memories start decay-fresh instead of +/// arriving months stale and unfindable by rank (PITFALLS 19). The row always +/// lands with `embedding_status = 0` for the sweep backfill (D-13) — there is +/// no embedding to carry. +/// +/// A dedicated type (rather than optional overrides on `NewMemory`) keeps the +/// fifteen existing `NewMemory` construction sites untouched. +#[derive(Debug, Clone, PartialEq)] +pub struct ImportedMemory { + pub content: String, + pub mem_type: MemoryType, + pub tags: Vec, + pub source: Option, + pub scope: Option, + /// Preserved from the file: when the memory was originally stored. + pub created_at: i64, + /// Preserved from the file: provenance only — `decay_score` depends solely + /// on `last_accessed`, so this changes no ranking. + pub access_count: i64, + /// Preserved from the file, absolute. The importer skips a row whose + /// `expires_at <= now` before it ever reaches the store (D-10). + pub expires_at: Option, +} + /// The full stored record as persisted in the `memories` table. #[derive(Debug, Clone, PartialEq)] pub struct Memory { diff --git a/crates/agent-memory-core/src/export.rs b/crates/agent-memory-core/src/export.rs new file mode 100644 index 0000000..b937aa0 --- /dev/null +++ b/crates/agent-memory-core/src/export.rs @@ -0,0 +1,347 @@ +//! Portable export format **v1** — the single definition of the on-disk +//! contract behind `agent-memory export` and `agent-memory import --from jsonl` +//! (DIST-04 / INTEROP-02). +//! +//! An export is a JSON Lines document: one JSON object per line, UTF-8, `\n` +//! terminated. **Line 1 is the mandatory header record**; every later line is +//! one memory or one link. Every object carries a `record` tag naming its +//! type; the set of tags is closed (`header`, `memory`, `link`). +//! +//! The moment one user writes a file this format is a compatibility contract +//! (Phase 5 D-03, rated one-way): removing a field or changing a meaning needs +//! `version: 2` plus a reader that still accepts version 1. Adding a field is +//! likewise a new version, because the reader is strict (below). +//! +//! # Header record (line 1) +//! +//! | field | type | meaning | +//! |---|---|---| +//! | `record` | `"header"` | tag | +//! | `format` | string | always [`FORMAT_NAME`] | +//! | `version` | integer | always [`FORMAT_VERSION`] for files this crate writes | +//! | `exported_at` | integer | unix seconds, from the service clock | +//! | `embedding_model` | string | the source database's `meta.embedding_model` pin | +//! | `embedding_dim` | integer | the source database's `meta.embedding_dim` pin | +//! | `schema_version` | integer | the source database's schema (`LATEST_SCHEMA_VERSION` at export time) | +//! | `agent_memory_version` | string | the exporting binary's `CARGO_PKG_VERSION` | +//! | `exported_rows` | integer | number of `memory` records that follow | +//! | `exported_links` | integer | number of `link` records that follow | +//! +//! `embedding_model` / `embedding_dim` are read from the source database's +//! `meta` table, never from constants, so the header states what the source +//! actually pinned. `exported_rows` / `exported_links` are a cheap truncation +//! check for a reader — they are counted inside the same read snapshot as the +//! rows themselves. +//! +//! # Memory record +//! +//! | field | type | meaning | +//! |---|---|---| +//! | `record` | `"memory"` | tag | +//! | `key` | integer | the source row id, used ONLY as an export-local key for `link` records; never written as an id on import (PITFALLS 18) | +//! | `type` | string | memory type wire string (`DECISION`, `PATTERN`, …); parsed through `MemoryType::try_from` on import (D-09) | +//! | `content` | string | the memory text, verbatim | +//! | `tags` | array of strings | tags, verbatim | +//! | `source` | string or null | provenance, verbatim | +//! | `scope` | string or null | scope, verbatim | +//! | `created_at` | integer | unix seconds, preserved on import | +//! | `last_accessed` | integer | unix seconds; the importer REFRESHES this to import time (D-10) | +//! | `access_count` | integer | preserved on import (provenance only) | +//! | `expires_at` | integer or null | absolute unix seconds; preserved on import, and a row already expired at import time is skipped and counted (D-10) | +//! +//! **Excluded as derived data (D-04):** `decay_score` (recomputed on read from +//! `last_accessed`), `base_weight` (a function of `type`), `embedding_status` +//! and every embedding vector. Vectors are pinned to the source machine's +//! model; the importer lands every row pending and the sweep re-embeds it +//! locally against the local pin (D-13), so a vector in the file could only +//! ever poison a differently-pinned corpus (PITFALLS 17). +//! +//! # Link record +//! +//! | field | type | meaning | +//! |---|---|---| +//! | `record` | `"link"` | tag | +//! | `from` | integer | the `key` of the source memory | +//! | `to` | integer | the `key` of the target memory | +//! | `kind` | string | `relates_to`, `supersedes` or `caused_by`; parsed through `LinkKind::try_from` on import (D-09) | +//! | `created_at` | integer | unix seconds, preserved on import (edge recency orders expansion — Phase 4 D-07) | +//! +//! Links are emitted **after all memory records** (D-05). Endpoints appear +//! exactly as stored: `relates_to` as `(min key, max key)` (the Phase 4 04-06 +//! canonical form), directed kinds in the direction they were given. +//! +//! # Ordering and canonical output +//! +//! The exporter writes memories in ascending `key` order and links ordered by +//! `(from, to, kind)`, every row present in the database — expired rows +//! included, no scope or type filter (D-06; expiry is the importer's policy). +//! Optional fields (`source`, `scope`, `expires_at`) are always written +//! explicitly as `null`. Two exports of an unchanged database under the same +//! clock are therefore byte-identical. +//! +//! # Reader rules +//! +//! A hand-written file may omit an optional field (it reads as `null`), but is +//! otherwise read **strictly**: unknown fields are rejected +//! (`deny_unknown_fields`, the 04-07 precedent), the `record` set is closed, +//! field types are exact (`"1"` is not an integer, `1.5` is not a version), +//! and a `version` above [`FORMAT_VERSION`] is refused with an upgrade +//! message. The full per-line validation and error wording live in +//! `import::jsonl`; the user-facing summary is the README's "Export & import" +//! section. + +use serde::{Deserialize, Serialize}; + +/// The `format` value every header carries. A file whose header says anything +/// else is not an agent-memory export, whatever its `version`. +pub const FORMAT_NAME: &str = "agent-memory-export"; + +/// The format version this crate writes and the highest it reads. +pub const FORMAT_VERSION: u32 = 1; + +/// One line of a format-v1 export file — the single definition of the wire +/// contract in both directions. +/// +/// Serialised internally tagged on `record` (lowercase variant name first, +/// then the fields in declaration order), so the documented field order IS +/// the declaration order below. `deny_unknown_fields` applies inside every +/// variant: a typo or a field from a newer version is a parse error, never a +/// silently dropped value (D-11). `type` / `kind` stay `String` here so the +/// closed-enum conversion runs through `MemoryType::try_from` / +/// `LinkKind::try_from` in the importer and produces the established +/// client-tier messages (D-09). +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(tag = "record", rename_all = "lowercase", deny_unknown_fields)] +pub enum Record { + /// Line 1 of every file (D-03). + Header { + /// Always [`FORMAT_NAME`]. + format: String, + /// The format version of this file. + version: u32, + /// Unix seconds at export time, from the service clock. + exported_at: i64, + /// The source database's `meta.embedding_model` pin. + embedding_model: String, + /// The source database's `meta.embedding_dim` pin. + embedding_dim: u32, + /// The source database's schema version. + schema_version: i64, + /// The exporting binary's `CARGO_PKG_VERSION`. + agent_memory_version: String, + /// Number of `memory` records in the file. + exported_rows: u64, + /// Number of `link` records in the file. + exported_links: u64, + }, + /// One memory row (D-04). + Memory { + /// Source row id — an export-local key only, never an id on import. + key: i64, + /// Memory type wire string. + #[serde(rename = "type")] + mem_type: String, + /// The memory text. + content: String, + /// Tags, verbatim. + tags: Vec, + /// Provenance, verbatim. + source: Option, + /// Scope, verbatim. + scope: Option, + /// Unix seconds; preserved on import. + created_at: i64, + /// Unix seconds; refreshed to import time on import (D-10). + last_accessed: i64, + /// Preserved on import (provenance only). + access_count: i64, + /// Absolute unix seconds or `null`; preserved on import. + expires_at: Option, + }, + /// One relation edge, emitted after all memories (D-05). + Link { + /// `key` of the source memory. + from: i64, + /// `key` of the target memory. + to: i64, + /// Link kind wire string. + kind: String, + /// Unix seconds; preserved on import. + created_at: i64, + }, +} + +/// What the store pins at the start of an export, before any row is read — +/// the header's inputs. Emitted exactly once per +/// [`crate::store::Store::export_rows`] call, as the first event, from the +/// same read snapshot the rows come from. +/// +/// A store-to-service streaming message, not a wire type. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ExportSnapshot { + /// `meta.embedding_model`. + pub embedding_model: String, + /// `meta.embedding_dim`, parsed. + pub embedding_dim: u32, + /// `count(*)` of `memories` in the snapshot. + pub memories: u64, + /// `count(*)` of `memory_links` in the snapshot. + pub links: u64, +} + +/// One step of the store-to-service export stream: the single leading +/// [`ExportEvent::Snapshot`], then one [`ExportEvent::Record`] per memory (in +/// ascending id order) and per link (after all memories). +/// +/// A store-to-service streaming message, not a wire type. +#[derive(Debug, Clone, PartialEq)] +pub enum ExportEvent { + /// The pin and counts for the header; always first, exactly once. + Snapshot(ExportSnapshot), + /// One memory or link record, in file order. + Record(Record), +} + +#[cfg(test)] +mod tests { + use super::*; + + fn memory_record() -> Record { + Record::Memory { + key: 7, + mem_type: "DECISION".to_string(), + content: "c".to_string(), + tags: vec!["a".to_string()], + source: None, + scope: Some("s".to_string()), + created_at: 1, + last_accessed: 2, + access_count: 3, + expires_at: None, + } + } + + fn to_json(rec: &Record) -> String { + serde_json::to_string(rec).expect("serialize record") + } + + fn parse(line: &str) -> Result { + serde_json::from_str(line) + } + + #[test] + fn memory_record_serializes_in_documented_order_with_record_first() { + assert_eq!( + to_json(&memory_record()), + r#"{"record":"memory","key":7,"type":"DECISION","content":"c","tags":["a"],"source":null,"scope":"s","created_at":1,"last_accessed":2,"access_count":3,"expires_at":null}"# + ); + } + + #[test] + fn header_record_serializes_in_documented_order_with_record_first() { + let header = Record::Header { + format: FORMAT_NAME.to_string(), + version: FORMAT_VERSION, + exported_at: 1_000, + embedding_model: "nomic-embed-text".to_string(), + embedding_dim: 768, + schema_version: 3, + agent_memory_version: "0.0.1".to_string(), + exported_rows: 2, + exported_links: 1, + }; + assert_eq!( + to_json(&header), + r#"{"record":"header","format":"agent-memory-export","version":1,"exported_at":1000,"embedding_model":"nomic-embed-text","embedding_dim":768,"schema_version":3,"agent_memory_version":"0.0.1","exported_rows":2,"exported_links":1}"# + ); + } + + #[test] + fn link_record_serializes_in_documented_order_with_record_first() { + let link = Record::Link { + from: 1, + to: 2, + kind: "supersedes".to_string(), + created_at: 5, + }; + assert_eq!( + to_json(&link), + r#"{"record":"link","from":1,"to":2,"kind":"supersedes","created_at":5}"# + ); + } + + #[test] + fn every_record_round_trips_through_its_own_json() { + let header = Record::Header { + format: FORMAT_NAME.to_string(), + version: FORMAT_VERSION, + exported_at: 1, + embedding_model: "m".to_string(), + embedding_dim: 8, + schema_version: 3, + agent_memory_version: "v".to_string(), + exported_rows: 0, + exported_links: 0, + }; + let link = Record::Link { + from: 1, + to: 2, + kind: "relates_to".to_string(), + created_at: 9, + }; + for rec in [header, memory_record(), link] { + let back = parse(&to_json(&rec)).expect("round trip parses"); + assert_eq!(back, rec); + } + } + + #[test] + fn an_unknown_field_inside_a_memory_record_is_rejected() { + let line = r#"{"record":"memory","key":7,"type":"DECISION","content":"c","tags":[],"source":null,"scope":null,"created_at":1,"last_accessed":2,"access_count":3,"expires_at":null,"bogus":1}"#; + let err = parse(line).expect_err("unknown field must fail"); + assert!( + err.to_string().contains("unknown field"), + "expected an unknown-field error, got: {err}" + ); + } + + #[test] + fn an_unknown_record_tag_is_rejected() { + let err = parse(r#"{"record":"memo","key":1}"#).expect_err("unknown tag must fail"); + assert!( + err.to_string().contains("unknown variant"), + "expected an unknown-variant error, got: {err}" + ); + } + + #[test] + fn missing_optional_fields_read_as_none() { + let line = r#"{"record":"memory","key":7,"type":"DECISION","content":"c","tags":["a"],"created_at":1,"last_accessed":2,"access_count":3}"#; + let rec = parse(line).expect("optional fields may be omitted"); + match rec { + Record::Memory { + source, + scope, + expires_at, + .. + } => { + assert_eq!(source, None); + assert_eq!(scope, None); + assert_eq!(expires_at, None); + } + other => panic!("expected a memory record, got {other:?}"), + } + } + + #[test] + fn a_string_key_is_rejected() { + let line = r#"{"record":"memory","key":"1","type":"DECISION","content":"c","tags":[],"source":null,"scope":null,"created_at":1,"last_accessed":2,"access_count":3,"expires_at":null}"#; + assert!(parse(line).is_err(), "a string key must not parse as i64"); + } + + #[test] + fn a_fractional_header_version_is_rejected() { + let line = r#"{"record":"header","format":"agent-memory-export","version":1.5,"exported_at":1,"embedding_model":"m","embedding_dim":8,"schema_version":3,"agent_memory_version":"v","exported_rows":0,"exported_links":0}"#; + assert!(parse(line).is_err(), "1.5 must not parse as a u32 version"); + } +} diff --git a/crates/agent-memory-core/src/import/jsonl.rs b/crates/agent-memory-core/src/import/jsonl.rs new file mode 100644 index 0000000..971873d --- /dev/null +++ b/crates/agent-memory-core/src/import/jsonl.rs @@ -0,0 +1,658 @@ +//! Strict reader for format-v1 export files — the import half of +//! `agent-memory export` / `agent-memory import --from jsonl` (INTEROP-02). +//! +//! This module is deliberately the **opposite tolerance policy** from +//! [`super::gsd_state`] (Phase 5 D-11). A `STATE.md` is a human-edited text +//! file, so the gsd-state scanner degrades messy input to skip-counts. A JSONL +//! export is agent-memory's **own contract** ([`crate::export`]), so any defect +//! means corruption or a hand edit — and a half-restored corpus is worse than a +//! clear refusal. Every defect therefore rejects the **whole file**, and +//! nothing is written: [`parse_jsonl`] is the single entry point to the +//! importer's writes, and it only ever returns a fully validated +//! [`JsonlFile`]. +//! +//! Pure function, no I/O — the CLI reads the file (D-01: import is a CLI +//! subcommand only; a file-path read on an unauthenticated daemon would be an +//! arbitrary-file-read primitive). +//! +//! # Error contract +//! +//! Every rejection is [`MemoryError::InvalidArgument`] whose text starts with +//! the **1-based line number** — `invalid argument: line 42: …` — so the CLI +//! prints a locatable message and any future transport maps it to REST 400 / +//! MCP `invalid_params` structurally (Phase 3 D-04/D-06). Line numbers come +//! from this module's own counter over `text.lines()`, **never** from +//! `serde_json::Error::line()`: serde sees one line at a time, so its own +//! number is always `1` (or `0` for data errors raised from a buffered +//! variant) — RESEARCH Pitfall 1. +//! +//! # Header handling +//! +//! Line 1 is probed **leniently** before the strict parse (RESEARCH Pattern 2): +//! a probe struct reads only `record`, `format` and `version`, ignoring unknown +//! fields, so a version-2 file that adds a field still reaches the +//! `upgrade agent-memory` message instead of failing first on +//! `unknown field …`. Only after the probe passes is line 1 parsed strictly as +//! [`Record::Header`]. +//! +//! # Timestamp handling +//! +//! The parser validates every memory record's `last_accessed` (it must be an +//! integer) and then **drops** it: the importer refreshes `last_accessed` to +//! import-time `now` (D-10), so [`JsonlMemory`] carries an +//! [`ImportedMemory`] that has no such field. `created_at`, `access_count` +//! and `expires_at` are carried verbatim; whether a row is expired is decided +//! by the importer against the local clock, not here (expired rows are not +//! malformed). `access_count` must be `>= 0` — it is the one preserved column +//! the store later increments, and the store's bump saturates at `i64::MAX`, +//! so no file value can push the column out of integer range. + +use std::collections::HashSet; +use std::fmt::Display; + +use serde::Deserialize; + +use crate::domain::{validate_link_endpoints, ImportedMemory, LinkKind, MemoryError, MemoryType}; +use crate::export::{Record, FORMAT_NAME, FORMAT_VERSION}; + +/// The validated header of a format-v1 file (line 1). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct JsonlHeader { + /// Unix seconds at export time, from the exporting machine's clock. + pub exported_at: i64, + /// The source database's `meta.embedding_model` pin. + pub embedding_model: String, + /// The source database's `meta.embedding_dim` pin. + pub embedding_dim: u32, + /// The source database's schema version. + pub schema_version: i64, + /// The exporting binary's `CARGO_PKG_VERSION`. + pub agent_memory_version: String, + /// Number of `memory` records the file declares (and, once parsed, + /// contains — a mismatch is rejected). + pub exported_rows: u64, + /// Number of `link` records the file declares (and contains). + pub exported_links: u64, +} + +/// One validated memory record: the export-local `key` the file's links +/// refer to, plus the row to restore. `key` is never written as an id — the +/// importer maps it to the target database's own id (PITFALLS 18). +#[derive(Debug, Clone, PartialEq)] +pub struct JsonlMemory { + /// The export-local key (the source row id), used only to resolve links. + pub key: i64, + /// The row as the store will receive it (`last_accessed` already dropped). + pub memory: ImportedMemory, +} + +/// One validated link record, endpoints still expressed as export-local keys. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct JsonlLink { + /// `key` of the source memory. + pub from: i64, + /// `key` of the target memory. + pub to: i64, + /// The closed relation kind, already parsed through [`LinkKind::try_from`]. + pub kind: LinkKind, + /// Unix seconds; preserved on import (edge recency orders expansion). + pub created_at: i64, +} + +/// A fully validated format-v1 file — the only input +/// [`crate::service::MemoryService::import_jsonl`] accepts. Every link's +/// endpoints name a key in `memories`; every type and kind is a closed enum; +/// no content is empty; the declared counts match. +#[derive(Debug, Clone, PartialEq)] +pub struct JsonlFile { + /// Line 1. + pub header: JsonlHeader, + /// Every memory record, in file order. + pub memories: Vec, + /// Every link record, in file order. + pub links: Vec, +} + +/// The lenient line-1 probe (RESEARCH Pattern 2): no `deny_unknown_fields`, +/// so a newer version's extra header fields are ignored here and the version +/// check below gets to speak first. `format` and `version` are optional so +/// that a file whose first line is not a header at all is reported as such, +/// rather than as a missing field. +#[derive(Deserialize)] +struct HeaderProbe { + record: String, + #[serde(default)] + format: Option, + #[serde(default)] + version: Option, +} + +/// Build the D-11 rejection: `InvalidArgument("line {n}: {msg}")`, whose +/// Display is `invalid argument: line {n}: {msg}`. +fn invalid_line(line_no: usize, msg: impl Display) -> MemoryError { + MemoryError::InvalidArgument(format!("line {line_no}: {msg}")) +} + +/// Re-wrap a seam error (`MemoryType::try_from`, `LinkKind::try_from`, +/// `validate_link_endpoints`) under the line prefix. An `InvalidArgument` +/// already reads `invalid argument: …`, so its payload is lifted out rather +/// than nested (`invalid argument: line 3: invalid argument: kind …`); every +/// other variant keeps its full Display text. +fn rewrap(line_no: usize, e: MemoryError) -> MemoryError { + match e { + MemoryError::InvalidArgument(msg) => invalid_line(line_no, msg), + other => invalid_line(line_no, other), + } +} + +/// serde_json appends ` at line 1 column N` to syntax and data errors — its +/// own per-line position, which is meaningless (and misleading) next to the +/// importer's line counter (Pitfall 1). Strip it; keep the message itself. +fn serde_message(e: &serde_json::Error) -> String { + let text = e.to_string(); + let suffix = format!(" at line {} column {}", e.line(), e.column()); + match text.strip_suffix(&suffix) { + Some(stripped) => stripped.to_string(), + None => text, + } +} + +/// The lenient probe of line 1, in contract order: parseable → `record` is +/// `header` → `format` is [`FORMAT_NAME`] → `version` is an unsigned integer +/// → `>= 1` → `<= FORMAT_VERSION`. Returns `Ok(())` when the strict parse may +/// proceed. +fn check_header(line: &str) -> Result<(), MemoryError> { + let probe: HeaderProbe = serde_json::from_str(line).map_err(|e| { + invalid_line( + 1, + format!("missing or invalid header ({})", serde_message(&e)), + ) + })?; + if probe.record != "header" { + return Err(invalid_line(1, "first record must be the header")); + } + match probe.format.as_deref() { + Some(format) if format == FORMAT_NAME => {} + Some(format) => { + return Err(invalid_line( + 1, + format!("format must be '{FORMAT_NAME}' (got '{format}')"), + )); + } + None => { + return Err(invalid_line( + 1, + format!("format must be '{FORMAT_NAME}' (missing)"), + )); + } + } + let version = match probe.version { + Some(value) => value.as_u64().ok_or_else(|| { + invalid_line( + 1, + format!("version must be a positive integer (got {value})"), + ) + })?, + None => { + return Err(invalid_line( + 1, + "version must be a positive integer (missing)", + )); + } + }; + if version == 0 { + return Err(invalid_line(1, "version must be >= 1 (got 0)")); + } + if version > u64::from(FORMAT_VERSION) { + return Err(invalid_line( + 1, + format!( + "file is format version {version} but this binary reads up to version \ + {FORMAT_VERSION} — upgrade agent-memory" + ), + )); + } + Ok(()) +} + +/// Parse one strict [`Record`] from a line that has already passed the +/// object-shape check, wrapping serde's message under the line prefix. +fn parse_record(line_no: usize, line: &str) -> Result { + serde_json::from_str::(line).map_err(|e| invalid_line(line_no, serde_message(&e))) +} + +/// Parse and validate a whole format-v1 export file (D-08 pass 1, D-09, D-11). +/// +/// Strict and all-or-nothing: the first defect is returned as +/// [`MemoryError::InvalidArgument`] naming the 1-based line, and no +/// [`JsonlFile`] exists for the importer to write from. Checks, in order: +/// +/// 1. **Line 1** — the lenient header probe ([`check_header`]), then the strict +/// [`Record::Header`] parse. +/// 2. **Every later line** — must start with `{` (`expected a JSON object`); +/// must parse as a [`Record`] (unknown record tags, unknown fields and wrong +/// field types are serde errors); a second `header` is `duplicate header`; +/// a `memory` must have a valid `type` ([`MemoryType::try_from`]), non-empty +/// content and a `key` not seen before; a `link` must have a valid `kind` +/// ([`LinkKind::try_from`]) and distinct endpoints +/// ([`validate_link_endpoints`]). +/// 3. **After the loop** — every link endpoint must be a memory key that +/// appears *anywhere* in the file (order-independent, so a hand-ordered file +/// still parses), and the record counts must equal the header's +/// `exported_rows` / `exported_links` (a truncated file is refused). +/// +/// A zero-byte or header-less file fails at line 1. Trailing blank lines are +/// NOT tolerated — the exporter never writes them, and a blank line is where a +/// truncated or concatenated file shows first. +pub fn parse_jsonl(text: &str) -> Result { + let mut lines = text.lines().enumerate(); + + let Some((_, first)) = lines.next() else { + return Err(invalid_line(1, "missing or invalid header (empty file)")); + }; + check_header(first)?; + let header = match parse_record(1, first)? { + Record::Header { + exported_at, + embedding_model, + embedding_dim, + schema_version, + agent_memory_version, + exported_rows, + exported_links, + // `format` / `version` were validated by the probe. + format: _, + version: _, + } => JsonlHeader { + exported_at, + embedding_model, + embedding_dim, + schema_version, + agent_memory_version, + exported_rows, + exported_links, + }, + Record::Memory { .. } | Record::Link { .. } => { + return Err(invalid_line(1, "first record must be the header")); + } + }; + + let mut memories: Vec = Vec::new(); + let mut links: Vec<(usize, JsonlLink)> = Vec::new(); + let mut keys: HashSet = HashSet::new(); + let mut last_line = 1usize; + + for (idx, line) in lines { + let line_no = idx + 1; + last_line = line_no; + if !line.trim_start().starts_with('{') { + return Err(invalid_line(line_no, "expected a JSON object")); + } + match parse_record(line_no, line)? { + Record::Header { .. } => { + return Err(invalid_line(line_no, "duplicate header")); + } + Record::Memory { + key, + mem_type, + content, + tags, + source, + scope, + created_at, + // Validated as an integer by serde, then dropped: the importer + // refreshes it to import-time now (D-10). + last_accessed: _, + access_count, + expires_at, + } => { + // The shared Phase 3 seam: the wire string becomes the closed + // enum here or nowhere (D-09). + let mem_type = + MemoryType::try_from(mem_type.as_str()).map_err(|e| rewrap(line_no, e))?; + if content.trim().is_empty() { + return Err(invalid_line(line_no, "content must be a non-empty string")); + } + // `access_count` is the one preserved column later used in + // arithmetic (the recency bump), so it is bounded here like + // every other field (05-REVIEW WR-02). The upper bound is the + // column's own i64 range, which serde already enforces. + if access_count < 0 { + return Err(invalid_line( + line_no, + format!("access_count must be >= 0 (got {access_count})"), + )); + } + if !keys.insert(key) { + return Err(invalid_line(line_no, format!("duplicate memory key {key}"))); + } + memories.push(JsonlMemory { + key, + memory: ImportedMemory { + content, + mem_type, + tags, + source, + scope, + created_at, + access_count, + expires_at, + }, + }); + } + Record::Link { + from, + to, + kind, + created_at, + } => { + let kind = LinkKind::try_from(kind.as_str()).map_err(|e| rewrap(line_no, e))?; + validate_link_endpoints(from, to).map_err(|e| rewrap(line_no, e))?; + links.push(( + line_no, + JsonlLink { + from, + to, + kind, + created_at, + }, + )); + } + } + } + + // Links may name a key defined later in the file (order-independent), so + // the key set is only complete once the loop has finished. + for (line_no, link) in &links { + for endpoint in [link.from, link.to] { + if !keys.contains(&endpoint) { + return Err(invalid_line( + *line_no, + format!("link key {endpoint} does not appear in the file"), + )); + } + } + } + + let memory_count = memories.len() as u64; + if memory_count != header.exported_rows { + return Err(invalid_line( + last_line, + format!( + "file declares {} memory records but contains {memory_count}", + header.exported_rows + ), + )); + } + let link_count = links.len() as u64; + if link_count != header.exported_links { + return Err(invalid_line( + last_line, + format!( + "file declares {} link records but contains {link_count}", + header.exported_links + ), + )); + } + + Ok(JsonlFile { + header, + memories, + links: links.into_iter().map(|(_, link)| link).collect(), + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A valid v1 header declaring `rows` memories and `links` links. + fn header(rows: u64, links: u64) -> String { + format!( + r#"{{"record":"header","format":"agent-memory-export","version":1,"exported_at":1000,"embedding_model":"nomic-embed-text","embedding_dim":768,"schema_version":3,"agent_memory_version":"0.0.1","exported_rows":{rows},"exported_links":{links}}}"# + ) + } + + const MEM1: &str = r#"{"record":"memory","key":1,"type":"DECISION","content":"a","tags":[],"source":null,"scope":null,"created_at":1,"last_accessed":1,"access_count":0,"expires_at":null}"#; + const MEM2: &str = r#"{"record":"memory","key":2,"type":"PATTERN","content":"b","tags":["t"],"source":"s","scope":"sc","created_at":2,"last_accessed":2,"access_count":3,"expires_at":99}"#; + const LINK12: &str = r#"{"record":"link","from":1,"to":2,"kind":"relates_to","created_at":5}"#; + + fn file(lines: &[&str]) -> String { + let mut text = lines.join("\n"); + text.push('\n'); + text + } + + fn expect_invalid(name: &str, text: &str, needle: &str) { + let err = parse_jsonl(text).expect_err(name); + assert!( + matches!(err, MemoryError::InvalidArgument(_)), + "{name}: expected InvalidArgument, got {err:?}" + ); + let shown = err.to_string(); + assert!( + shown.contains(needle), + "{name}: expected {needle:?} in {shown:?}" + ); + assert!( + shown.starts_with("invalid argument: line "), + "{name}: D-11 shape, got {shown:?}" + ); + } + + #[test] + fn a_two_memory_one_link_file_parses_into_typed_records() { + let text = file(&[&header(2, 1), MEM1, MEM2, LINK12]); + let parsed = parse_jsonl(&text).expect("valid file parses"); + + assert_eq!(parsed.header.exported_rows, 2); + assert_eq!(parsed.header.exported_links, 1); + assert_eq!(parsed.header.exported_at, 1_000); + assert_eq!(parsed.header.embedding_model, "nomic-embed-text"); + assert_eq!(parsed.header.embedding_dim, 768); + assert_eq!(parsed.header.schema_version, 3); + assert_eq!(parsed.header.agent_memory_version, "0.0.1"); + + assert_eq!(parsed.memories.len(), 2); + assert_eq!(parsed.memories[0].key, 1); + assert_eq!(parsed.memories[0].memory.mem_type, MemoryType::Decision); + assert_eq!(parsed.memories[0].memory.content, "a"); + assert_eq!(parsed.memories[1].key, 2); + assert_eq!( + parsed.memories[1].memory, + ImportedMemory { + content: "b".to_string(), + mem_type: MemoryType::Pattern, + tags: vec!["t".to_string()], + source: Some("s".to_string()), + scope: Some("sc".to_string()), + created_at: 2, + access_count: 3, + expires_at: Some(99), + } + ); + + assert_eq!(parsed.links.len(), 1); + assert_eq!( + parsed.links[0], + JsonlLink { + from: 1, + to: 2, + kind: LinkKind::RelatesTo, + created_at: 5, + } + ); + } + + #[test] + fn a_newer_version_with_extra_fields_reaches_the_upgrade_message() { + let text = "{\"record\":\"header\",\"format\":\"agent-memory-export\",\"version\":2,\"later\":1}\n"; + expect_invalid("version 2", text, "upgrade agent-memory"); + expect_invalid( + "version 2 names both versions", + text, + "file is format version 2 but this binary reads up to version 1", + ); + } + + #[test] + fn a_wrong_format_is_rejected_at_line_one() { + expect_invalid( + "bad format", + "{\"record\":\"header\",\"format\":\"x\",\"version\":1}\n", + "line 1: format must be 'agent-memory-export' (got 'x')", + ); + } + + #[test] + fn version_zero_is_rejected_at_line_one() { + expect_invalid( + "version 0", + "{\"record\":\"header\",\"format\":\"agent-memory-export\",\"version\":0}\n", + "line 1: version must be >= 1 (got 0)", + ); + } + + #[test] + fn a_link_to_a_key_absent_from_the_file_is_rejected() { + let dangling = r#"{"record":"link","from":1,"to":9,"kind":"relates_to","created_at":1}"#; + expect_invalid( + "dangling key", + &file(&[&header(1, 1), MEM1, dangling]), + "line 3: link key 9 does not appear in the file", + ); + } + + #[test] + fn an_empty_file_is_rejected_at_line_one() { + expect_invalid("zero bytes", "", "line 1:"); + expect_invalid("one newline", "\n", "line 1:"); + } + + #[test] + fn a_header_only_file_parses_to_no_records() { + let parsed = parse_jsonl(&file(&[&header(0, 0)])).expect("header only parses"); + assert!(parsed.memories.is_empty()); + assert!(parsed.links.is_empty()); + } + + #[test] + fn links_may_precede_the_memories_they_name() { + let parsed = parse_jsonl(&file(&[&header(2, 1), LINK12, MEM1, MEM2])) + .expect("order-independent key resolution"); + assert_eq!(parsed.memories.len(), 2); + assert_eq!(parsed.links.len(), 1); + } + + #[test] + fn serde_position_suffix_is_stripped_from_messages() { + let err = parse_jsonl(&file(&[&header(1, 0), r#"{"record":"memo","key":1}"#])) + .expect_err("unknown record"); + let shown = err.to_string(); + assert!(shown.contains("unknown variant"), "{shown}"); + assert!( + !shown.contains(" at line 1 column "), + "serde's own position must never surface: {shown}" + ); + } + + #[test] + fn malformed_files_are_rejected_naming_the_line() { + let h10 = header(1, 0); + let h11 = header(1, 1); + let h20 = header(2, 0); + let h50 = header(5, 0); + let h00 = header(0, 0); + let unknown_field = MEM1.replace("\"expires_at\":null", "\"expires_at\":null,\"bogus\":1"); + let string_key = MEM1.replace("\"key\":1", "\"key\":\"1\""); + let bad_type = MEM1.replace("\"type\":\"DECISION\"", "\"type\":\"MEMO\""); + let blank_content = MEM1.replace("\"content\":\"a\"", "\"content\":\" \""); + let bad_kind = r#"{"record":"link","from":1,"to":2,"kind":"loves","created_at":1}"#; + let self_link = r#"{"record":"link","from":1,"to":1,"kind":"relates_to","created_at":1}"#; + let dangling = r#"{"record":"link","from":1,"to":9,"kind":"relates_to","created_at":1}"#; + let string_tags = MEM1.replace("\"tags\":[]", "\"tags\":\"x\""); + let float_expiry = MEM1.replace("\"expires_at\":null", "\"expires_at\":1.5"); + let negative_access = MEM1.replace("\"access_count\":0", "\"access_count\":-1"); + let huge_access = + MEM1.replace("\"access_count\":0", "\"access_count\":9223372036854775808"); + + let hdr = |body: &str| format!("{{\"record\":\"header\",{body}}}\n"); + let fmt_x = hdr(r#""format":"x","version":1"#); + let no_fmt = hdr(r#""version":1"#); + let v0 = hdr(r#""format":"agent-memory-export","version":0"#); + let v_str = hdr(r#""format":"agent-memory-export","version":"1""#); + let v_frac = hdr(r#""format":"agent-memory-export","version":1.5"#); + let v2 = hdr(r#""format":"agent-memory-export","version":2,"later":1"#); + let v1_short = hdr(r#""format":"agent-memory-export","version":1"#); + let memo = r#"{"record":"memo","key":1}"#; + + // One case per line on purpose: the table IS the D-11 contract, and a + // reader should be able to scan name / input / expected prefix across. + #[rustfmt::skip] + let cases: Vec<(&str, String, &str)> = vec![ + ("no header", file(&[MEM1]), "line 1: first record must be the header"), + ("not json at all", "garbage\n".to_string(), "line 1: missing or invalid header"), + ("bad format", fmt_x, "line 1: format must be 'agent-memory-export' (got 'x')"), + ("missing format", no_fmt, "line 1: format must be 'agent-memory-export' (missing)"), + ("version 0", v0, "line 1: version must be >= 1 (got 0)"), + ("string version", v_str, "line 1: version must be a positive integer (got \"1\")"), + ("fractional version", v_frac, "line 1: version must be a positive integer (got 1.5)"), + ("version 2 + unknown field", v2, "upgrade agent-memory"), + ("header missing a required field", v1_short, "line 1: missing field"), + ("not an object", file(&[&h10, "[1,2]"]), "line 2: expected a JSON object"), + ("blank line", file(&[&h10, ""]), "line 2: expected a JSON object"), + ("unknown record", file(&[&h10, memo]), "line 2: unknown variant `memo`"), + ("unknown field", file(&[&h10, &unknown_field]), "line 2: unknown field `bogus`"), + ("string key", file(&[&h10, &string_key]), "line 2: invalid type: string \"1\""), + ("bad type", file(&[&h10, &bad_type]), "line 2: invalid memory type 'MEMO'"), + ("blank content", file(&[&h10, &blank_content]), "line 2: content must be a non-empty string"), + ("tags not an array", file(&[&h10, &string_tags]), "line 2: invalid type"), + ("fractional expires_at", file(&[&h10, &float_expiry]), "line 2: invalid type"), + ("negative access_count", file(&[&h10, &negative_access]), "line 2: access_count must be >= 0 (got -1)"), + ("access_count above i64", file(&[&h10, &huge_access]), "line 2: invalid value"), + ("bad kind", file(&[&h11, MEM1, bad_kind]), "line 3: kind must be one of"), + ("self link", file(&[&h11, MEM1, self_link]), "line 3: from_id and to_id must differ"), + ("dangling key", file(&[&h11, MEM1, dangling]), "line 3: link key 9 does not appear in the file"), + ("duplicate key", file(&[&h20, MEM1, MEM1]), "line 3: duplicate memory key 1"), + ("count mismatch", file(&[&h50, MEM1]), "line 2: file declares 5 memory records but contains 1"), + ("link count mismatch", file(&[&h11, MEM1]), "line 2: file declares 1 link records but contains 0"), + ("duplicate header", file(&[&h00, &h00]), "line 2: duplicate header"), + ]; + assert!(cases.len() >= 18, "the D-11 table must cover every class"); + for (name, text, needle) in &cases { + expect_invalid(name, text, needle); + } + } + + #[test] + fn hostile_input_never_panics() { + // 300 nested arrays as `tags`: serde_json's recursion limit (128) + // returns an error long before the stack is at risk. + let nested = format!("{}{}", "[".repeat(300), "]".repeat(300)); + let deep_tags = MEM1.replace("\"tags\":[]", &format!("\"tags\":{nested}")); + assert!(parse_jsonl(&file(&[&header(1, 0), &deep_tags])).is_err()); + + // A 4 MiB content string is data, not an attack: it parses. + let big = "x".repeat(4 << 20); + let big_content = MEM1.replace("\"content\":\"a\"", &format!("\"content\":\"{big}\"")); + let parsed = + parse_jsonl(&file(&[&header(1, 0), &big_content])).expect("4 MiB content parses"); + assert_eq!(parsed.memories[0].memory.content.len(), 4 << 20); + + // 1 MiB of `{`: unbalanced and too deep — an error, not an abort. + let braces = "{".repeat(1 << 20); + assert!(parse_jsonl(&file(&[&header(1, 0), &braces])).is_err()); + + // A NUL escape inside content is stored as data. + let nul = MEM1.replace("\"content\":\"a\"", "\"content\":\"a\\u0000b\""); + let parsed = parse_jsonl(&file(&[&header(1, 0), &nul])).expect("NUL is data"); + assert_eq!(parsed.memories[0].memory.content, "a\u{0}b"); + + // `access_count` at exactly i64::MAX is in range and carried verbatim + // (05-REVIEW WR-02): the store's bump saturates, so this row can never + // overflow the column into REAL and poison a later export. + let max_access = MEM1.replace("\"access_count\":0", "\"access_count\":9223372036854775807"); + let parsed = parse_jsonl(&file(&[&header(1, 0), &max_access])).expect("i64::MAX parses"); + assert_eq!(parsed.memories[0].memory.access_count, i64::MAX); + } +} diff --git a/crates/agent-memory-core/src/import/mod.rs b/crates/agent-memory-core/src/import/mod.rs index 0ace3c4..4ad2823 100644 --- a/crates/agent-memory-core/src/import/mod.rs +++ b/crates/agent-memory-core/src/import/mod.rs @@ -1,16 +1,24 @@ -//! Importers for external memory sources (INTEROP-01). +//! Importers for external memory sources (INTEROP-01, INTEROP-02). //! //! Invariants every importer guards: //! -//! - **Tolerant**: messy input degrades to skip-counts — malformed lines, -//! unknown sections, and oversized items are skipped and counted, never a -//! hard error. An importer must survive any text file without panicking. -//! - **Idempotent**: importers produce [`crate::domain::NewMemory`] drafts that -//! [`crate::service::MemoryService::import`] dedups against the store on the -//! exact `(source, mem_type, content)` key, so re-running an import never -//! duplicates. +//! - **Tolerance is per importer.** [`gsd_state`] scrapes a human-edited text +//! file, so messy input degrades to skip-counts — malformed lines, unknown +//! sections and oversized items are skipped and counted, never a hard +//! error. [`jsonl`] reads agent-memory's own format-v1 contract +//! ([`crate::export`]) and is **strict**: any defect rejects the whole file +//! as [`crate::domain::MemoryError::InvalidArgument`] naming the 1-based +//! line, and nothing is written (Phase 5 D-11). Both must survive any text +//! without panicking. +//! - **Idempotent**: every importer dedups against the store through +//! [`crate::store::Store::find_by_key`] on the exact +//! `(source, mem_type, content)` key, so re-running an import never +//! duplicates — [`crate::service::MemoryService::import`] for gsd-state +//! drafts, [`crate::service::MemoryService::import_jsonl`] for a parsed +//! export file. //! - **Data-only**: imported content is stored strictly as data. It is never //! executed or interpreted by agent-memory (T-02-20/T-02-21 — downstream //! prompt-injection screening composes via the ecosystem injection-scanner). pub mod gsd_state; +pub mod jsonl; diff --git a/crates/agent-memory-core/src/lib.rs b/crates/agent-memory-core/src/lib.rs index 5022f89..64ef2d4 100644 --- a/crates/agent-memory-core/src/lib.rs +++ b/crates/agent-memory-core/src/lib.rs @@ -8,6 +8,7 @@ pub mod clock; pub mod decay; pub mod domain; pub mod embed; +pub mod export; pub mod import; pub mod service; pub mod store; diff --git a/crates/agent-memory-core/src/service.rs b/crates/agent-memory-core/src/service.rs index 662319b..2c14780 100644 --- a/crates/agent-memory-core/src/service.rs +++ b/crates/agent-memory-core/src/service.rs @@ -13,6 +13,7 @@ //! hybrid result set is a valid `Hybrid` answer). use std::collections::HashMap; +use std::io::Write; use std::sync::Arc; use crate::clock::Clock; @@ -22,28 +23,45 @@ use crate::domain::{ UpdatedMemory, MAX_EXPANDED_LINKS, MAX_LIMIT, MIN_LIMIT, }; use crate::embed::Embedder; +use crate::export::{ExportEvent, Record, FORMAT_NAME, FORMAT_VERSION}; +use crate::import::jsonl::{JsonlFile, JsonlLink, JsonlMemory}; +use crate::store::migrations::LATEST_SCHEMA_VERSION; use crate::store::sqlite::DEFAULT_SEARCH_LIMIT; use crate::store::Store; /// The only message a client ever sees for a storage-tier search failure -/// (T-04-19). The underlying error goes to the stderr log, never to a response -/// body, so no SQLite text (table/column names, engine detail) leaks out of -/// any search path. Both transports surface it on the internal tier (REST 500 -/// / MCP `internal_error`). +/// (T-04-19). It is exactly the [`redact_storage_error`] template at +/// `op = "search"` — a unit test pins the two together so they can never +/// drift — and is kept as a named constant because Phase 4's golden tests +/// assert equality against it. The underlying error goes to the stderr log, +/// never to a response body, so no SQLite text (table/column names, engine +/// detail) leaks out of any search path. Both transports surface it on the +/// internal tier (REST 500 / MCP `internal_error`). pub const SEARCH_STORAGE_FAILURE: &str = "search failed due to a storage error (details are in the server log)"; -/// Tier every search-path error for the wire — the ONE place this happens. +/// Tier every service-path error for the wire — the ONE place storage-engine +/// text is stripped (D-14, closing 04-REVIEW IN-01). /// /// - Client tier (`InvalidType`, `InvalidQuery`, `InvalidArgument`) passes /// through: its text echoes only client input and public bounds. /// - Crate-authored internal text (`Internal`, `SchemaTooNew`) passes through. /// - Storage-engine errors (`Sqlite`, `Pool`, `Join`, `Migration`) are logged -/// with full detail, then replaced by [`SEARCH_STORAGE_FAILURE`]. +/// at error level with full detail, then replaced by +/// `MemoryError::Internal(" failed due to a storage error (details are +/// in the server log)")`. +/// +/// Why the seam and not the transport mappers: every public +/// [`MemoryService`] method delegates to a `*_unredacted` body and maps its +/// result through this helper, so MCP, REST — including `/health`, which is a +/// one-row `list` — and the CLI (`import`) are covered by ONE change, the way +/// Phase 3 D-06 put validation here and Phase 4 T-04-19 put the search +/// redaction here. The two transport mappers stay as they are and keep +/// matching every variant exhaustively. /// /// No catch-all arm: a new [`MemoryError`] variant must be tiered here at /// compile time. -fn redact_search_error(e: MemoryError) -> MemoryError { +fn redact_storage_error(op: &str, e: MemoryError) -> MemoryError { match e { MemoryError::InvalidType(_) | MemoryError::InvalidQuery(_) @@ -53,8 +71,10 @@ fn redact_search_error(e: MemoryError) -> MemoryError { | MemoryError::Pool(_) | MemoryError::Join(_) | MemoryError::Migration(_) => { - tracing::error!(error = %e, "search failed with a storage error"); - MemoryError::Internal(SEARCH_STORAGE_FAILURE.to_string()) + tracing::error!(op, error = %e, "storage error"); + MemoryError::Internal(format!( + "{op} failed due to a storage error (details are in the server log)" + )) } } } @@ -156,15 +176,71 @@ pub struct SearchOutcome { pub results: Vec, } -/// Outcome of one [`MemoryService::import`] batch (INTEROP-01): how many drafts -/// landed as new rows and how many were dropped as exact duplicates. -#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize)] +/// Outcome of one import — [`MemoryService::import`] (INTEROP-01, gsd-state +/// drafts) or [`MemoryService::import_jsonl`] (INTEROP-02, a format-v1 file). +/// The gsd-state path only ever populates the first two counts and reports +/// zeros for the rest (Phase 5 D-12). +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Serialize)] pub struct ImportReport { - /// Drafts inserted as new memories. + /// Drafts (or memory records) inserted as new memories. pub imported: usize, - /// Drafts skipped because an identical `(source, mem_type, content)` row - /// already exists — the idempotency guarantee. + /// Drafts (or memory records) skipped because an identical + /// `(source, mem_type, content)` row already exists — the idempotency + /// guarantee. pub skipped_duplicates: usize, + /// Memory records whose `expires_at <= now` at import time: never inserted + /// (D-10). Always `0` for gsd-state drafts, which carry no expiry. + pub skipped_expired: usize, + /// Link records that created a new edge. + pub links_imported: usize, + /// Link records whose edge already existed (an idempotent re-import). + pub links_skipped_duplicates: usize, + /// Link records dropped because an endpoint was skipped as expired, or + /// vanished between the memories pass and the links pass. + pub links_dropped: usize, +} + +/// How far ahead of the local clock an export header's `exported_at` may sit +/// before the import logs a clock-skew warning (D-10): one hour. +pub const CLOCK_SKEW_TOLERANCE_SECS: i64 = 3600; + +/// `Some(warning)` iff `exported_at` is more than +/// [`CLOCK_SKEW_TOLERANCE_SECS`] ahead of `now`. Expiry and freshness of the +/// imported rows are evaluated against THIS machine's clock, so a source +/// machine running fast can make fresh rows look expired; the warning names +/// both clocks so the user can check them. Never an error — the import +/// continues. +pub(crate) fn clock_skew_message(exported_at: i64, now: i64) -> Option { + let delta = exported_at.saturating_sub(now); + if delta <= CLOCK_SKEW_TOLERANCE_SECS { + return None; + } + Some(format!( + "export header exported_at {exported_at} is {delta}s ahead of this machine's clock \ + ({now}); check both clocks — expiry and freshness of imported rows are computed \ + against this machine" + )) +} + +/// Outcome of one [`MemoryService::export`] (DIST-04): how many memory and +/// link records were written after the header. Equal to the header's +/// `exported_rows` / `exported_links`, which come from the same read snapshot. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Serialize)] +pub struct ExportReport { + /// `memory` records written. + pub memories: u64, + /// `link` records written. + pub links: u64, +} + +/// Write one record as one JSON line. Writer and serialiser failures are +/// crate-authored `Internal` text (they name no storage engine), so they pass +/// [`redact_storage_error`] unchanged. +fn write_record_line(out: &mut dyn Write, rec: &Record) -> Result<(), MemoryError> { + let fail = + |e: &dyn std::fmt::Display| MemoryError::Internal(format!("export write failed: {e}")); + serde_json::to_writer(&mut *out, rec).map_err(|e| fail(&e))?; + out.write_all(b"\n").map_err(|e| fail(&e)) } /// The core memory service: validates input, stamps timestamps from the injected @@ -206,7 +282,17 @@ impl MemoryService { /// Pitfall 5). Storing NEVER fails because Ollama is down: on embed failure /// the row lands with `embedding_status = 0` and exactly one loud warning is /// emitted (Pitfall 4 — loud, not silent; SEARCH-03). + /// + /// Errors are tiered for the wire by [`redact_storage_error`] (D-14): a + /// storage fault reaches the client only as the generic internal error. pub async fn store(&self, new: NewMemory) -> Result { + self.store_unredacted(new) + .await + .map_err(|e| redact_storage_error("store", e)) + } + + /// The body of [`Self::store`], before its errors are tiered for the wire. + async fn store_unredacted(&self, new: NewMemory) -> Result { // API-02 / D-02: reject a bad TTL FIRST — before the embed call, so a // bad request never costs an Ollama round-trip (D-06 seam). crate::domain::validate_ttl(new.ttl_secs)?; @@ -241,7 +327,17 @@ impl MemoryService { /// `200 × MAX_EXPANDED_LINKS` bound therefore covers every expanded /// response. A list without `expand_links` still returns every row when /// `limit` is omitted. + /// + /// Errors are tiered for the wire by [`redact_storage_error`] (D-14) — + /// this also covers REST `/health`, which is a one-row list. pub async fn list(&self, args: ListArgs) -> Result, MemoryError> { + self.list_unredacted(args) + .await + .map_err(|e| redact_storage_error("list", e)) + } + + /// The body of [`Self::list`], before its errors are tiered for the wire. + async fn list_unredacted(&self, args: ListArgs) -> Result, MemoryError> { // API-02 / D-01: a present limit must be 1..=200. An omitted limit // stays unlimited (v1.0 behavior, RESEARCH Pitfall 6) only without // `expand_links`; range validation runs first. @@ -293,17 +389,17 @@ impl MemoryService { /// attach the 1-hop expansion when `expand_links` is set. /// /// Every error, from either arm and from the expansion, leaves through - /// [`redact_search_error`]: a storage failure reaches the client only as - /// [`MemoryError::Internal`] carrying [`SEARCH_STORAGE_FAILURE`], with the - /// detail in the stderr log (T-04-19). + /// [`redact_storage_error`] (D-14): a storage failure reaches the client + /// only as [`MemoryError::Internal`] carrying [`SEARCH_STORAGE_FAILURE`], + /// with the detail in the stderr log (T-04-19). pub async fn search(&self, args: SearchArgs) -> Result { self.search_unredacted(args) .await - .map_err(redact_search_error) + .map_err(|e| redact_storage_error("search", e)) } /// The body of [`Self::search`], before its errors are tiered for the wire - /// by [`redact_search_error`]. + /// by [`redact_storage_error`]. async fn search_unredacted(&self, args: SearchArgs) -> Result { // API-02 / D-01: validate before any embed hop — an out-of-bounds // limit rejects with MemoryError::InvalidArgument; the old silent @@ -380,7 +476,7 @@ impl MemoryService { /// Import a batch of drafts idempotently (INTEROP-01). /// /// 1. **Dedup** — every draft is checked against the store on the exact - /// `(source, mem_type, content)` key via [`Store::exists`], in ONE + /// `(source, mem_type, content)` key via [`Store::find_by_key`], in ONE /// `spawn_blocking` hop over the whole batch. An edited bullet imports /// as a NEW memory by design: the key is exact content equality — the /// idempotency contract (phase success criterion 4) requires only that @@ -391,7 +487,17 @@ impl MemoryService { /// fails because Ollama is down (SEARCH-03; the sweep backfills later). /// 3. **Insert** — new drafts land via the standard [`Store::insert`] /// path, all stamped with one clock reading. + /// + /// Errors are tiered for the wire by [`redact_storage_error`] (D-14), so + /// the CLI importer prints the generic message, never engine text. pub async fn import(&self, drafts: Vec) -> Result { + self.import_unredacted(drafts) + .await + .map_err(|e| redact_storage_error("import", e)) + } + + /// The body of [`Self::import`], before its errors are tiered for the wire. + async fn import_unredacted(&self, drafts: Vec) -> Result { // API-02 / D-02 + RESEARCH Pitfall 3: import() builds rows via // store.insert, bypassing store() — validate every draft through the // SAME shared helper here, before the dedup hop, so future import @@ -410,7 +516,10 @@ impl MemoryService { for draft in drafts { // Pass the source through as-is: a `None` must probe for // NULL, exactly what the insert stores (CR-01). - if store.exists(draft.source.as_deref(), draft.mem_type, &draft.content)? { + if store + .find_by_key(draft.source.as_deref(), draft.mem_type, &draft.content, now)? + .is_some() + { duplicates += 1; } else { fresh.push(draft); @@ -425,6 +534,7 @@ impl MemoryService { return Ok(ImportReport { imported: 0, skipped_duplicates, + ..ImportReport::default() }); } @@ -459,9 +569,222 @@ impl MemoryService { Ok(ImportReport { imported, skipped_duplicates, + ..ImportReport::default() }) } + /// Restore a parsed format-v1 export file (INTEROP-02; D-07, D-08, D-10, + /// D-12, D-13) — passes 2 and 3 of the two-pass import. Pass 1, the + /// all-or-nothing validation, is [`crate::import::jsonl::parse_jsonl`]: + /// this method only accepts a [`JsonlFile`] that already parsed whole, so + /// a malformed file can never reach a write. + /// + /// Both passes run in ONE `spawn_blocking` hop over the store, with ONE + /// clock reading as `now`: + /// + /// 2. **Memories pass**, in file order. A record with + /// `expires_at <= now` is skipped and counted as `skipped_expired` — + /// never inserted for the sweep to delete later (note the `<=`: the + /// sweep deletes `expires_at < now`, so a row expiring exactly now is + /// skipped here and would not yet be swept). Otherwise the record is + /// deduped through [`Store::find_by_key`] on `(source, mem_type, + /// content)` against LIVE target rows only (an expired-but-unswept + /// row is not a hit) — a hit maps the export-local `key` to the + /// EXISTING row's id and counts `skipped_duplicates`; a miss inserts via + /// [`Store::insert_imported`] (preserved `created_at`, `access_count`, + /// `expires_at`; `last_accessed` refreshed to `now`) and maps the key + /// to the new id. The `key → id` map is the ONLY way a file key is ever + /// read; a key is never written as an id (PITFALLS 18). + /// 3. **Links pass**. Each link's endpoints are looked up in the map and + /// linked through [`Store::link`] with the record's own `created_at` + /// as the edge time (D-05): `Linked { created: true }` counts + /// `links_imported`, `Linked { created: false }` counts + /// `links_skipped_duplicates`, and a link whose endpoint is not in the + /// map (skipped as expired), no longer exists, or whose two endpoints + /// collapsed onto ONE row through D-07 dedup (a post-remap self-pair, + /// never handed to the store) counts `links_dropped`. Links are + /// applied after ALL memories regardless of their position in the file. + /// + /// Each insert and link is its own writer transaction (the shape of + /// `insert` / `link`), so a storage fault mid-pass leaves a partial + /// import — which a re-run completes idempotently, since every record + /// dedups. Validation, not writing, is the all-or-nothing gate. + /// + /// **This path never embeds inline** (D-13): every imported row lands + /// `embedding_status = 0` with no vector, and the sweep backfill + /// ([`BACKFILL_BATCH`] per tick) re-embeds it against the LOCAL `meta` + /// pin — exactly one re-embed mechanism, so a vector produced by the + /// source machine's model can never enter this corpus (PITFALLS 17). + /// + /// A header `exported_at` more than [`CLOCK_SKEW_TOLERANCE_SECS`] ahead + /// of `now` logs one `tracing` warning and the import continues (D-10). + /// + /// Errors are tiered for the wire by [`redact_storage_error`] with + /// `op = "import"` (D-14). + pub async fn import_jsonl(&self, file: JsonlFile) -> Result { + self.import_jsonl_unredacted(file) + .await + .map_err(|e| redact_storage_error("import", e)) + } + + /// The body of [`Self::import_jsonl`], before its errors are tiered for + /// the wire. + async fn import_jsonl_unredacted(&self, file: JsonlFile) -> Result { + let now = self.clock.now(); + if let Some(msg) = clock_skew_message(file.header.exported_at, now) { + tracing::warn!("{msg}"); + } + let JsonlFile { + memories, links, .. + } = file; + + let store = self.store.clone(); + tokio::task::spawn_blocking(move || -> Result { + let mut report = ImportReport::default(); + // Export-local key -> this database's id. Only keys that landed + // (inserted or deduped) are present; expired keys are absent. + let mut ids: HashMap = HashMap::new(); + + // Pass 2: memories. + for JsonlMemory { key, memory } in memories { + if memory.expires_at.is_some_and(|t| t <= now) { + report.skipped_expired += 1; + continue; + } + // The probe sees the same world the expiry skip above sees: + // a target row with `expires_at <= now` is not a dedup hit, so + // the file's live copy is inserted and the stale row is left + // for the sweep (05-REVIEW WR-03). + match store.find_by_key( + memory.source.as_deref(), + memory.mem_type, + &memory.content, + now, + )? { + Some(existing) => { + report.skipped_duplicates += 1; + ids.insert(key, existing); + } + None => { + let id = store.insert_imported(memory, now)?; + report.imported += 1; + ids.insert(key, id); + } + } + } + + // Pass 3: links, through the map only. + for JsonlLink { + from, + to, + kind, + created_at, + } in links + { + match (ids.get(&from), ids.get(&to)) { + (Some(a), Some(b)) if a == b => { + // Both endpoints collapsed onto one row (D-07: two + // file rows identical on the dedup key map to the + // same id), so the edge has no second endpoint any + // more. Count it as dropped, like an expired + // endpoint, instead of handing the store a self-link: + // `INSERT OR IGNORE` treats the 0003 + // `CHECK (from_id != to_id)` violation as an + // ignorable conflict and would report it as a + // "duplicate" that never existed (05-REVIEW WR-01). + report.links_dropped += 1; + } + (Some(a), Some(b)) => match store.link(*a, *b, kind, created_at)? { + LinkOutcome::Linked { created: true } => report.links_imported += 1, + LinkOutcome::Linked { created: false } => { + report.links_skipped_duplicates += 1 + } + LinkOutcome::NotFound { .. } => report.links_dropped += 1, + }, + (None, _) | (_, None) => report.links_dropped += 1, + } + } + + Ok(report) + }) + .await + .map_err(MemoryError::Join)? + } + + /// Write the whole corpus to `out` as a format-v1 JSONL export (DIST-04): + /// the header line, then one line per memory in ascending id order, then + /// one line per link (see [`crate::export`] for the contract). + /// + /// Streams one line per record through the boxed writer inside ONE + /// `spawn_blocking` hop — the corpus is never buffered as a document + /// (D-02) — and reads everything from ONE store snapshot via + /// [`Store::export_rows`], so the header counts always match the records + /// that follow and no link can name a key the file lacks (T-05-19). The + /// writer is flushed inside the blocking task before this returns, so a + /// `BufWriter` caller never loses its tail (Pitfall 8). The embedder is + /// never touched: vectors do not travel (D-04). + /// + /// `agent_memory_version` is a parameter because the binary and this core + /// crate have separate versions (Pitfall 9) — the CLI passes its own + /// `CARGO_PKG_VERSION`. `exported_at` is the injected clock. + /// + /// Errors are tiered for the wire by [`redact_storage_error`] with + /// `op = "export"` (D-14): a storage fault reaches the CLI only as the + /// generic message; a writer I/O failure is the crate-authored + /// `export write failed: …`. + pub async fn export( + &self, + out: Box, + agent_memory_version: String, + ) -> Result { + self.export_unredacted(out, agent_memory_version) + .await + .map_err(|e| redact_storage_error("export", e)) + } + + /// The body of [`Self::export`], before its errors are tiered for the wire. + async fn export_unredacted( + &self, + mut out: Box, + agent_memory_version: String, + ) -> Result { + let now = self.clock.now(); + let store = self.store.clone(); + tokio::task::spawn_blocking(move || -> Result { + let mut report = ExportReport::default(); + store.export_rows(&mut |event| match event { + ExportEvent::Snapshot(snapshot) => { + // The header is built from the SAME snapshot the rows come + // from, so its counts are exact for this file. + report = ExportReport { + memories: snapshot.memories, + links: snapshot.links, + }; + let header = Record::Header { + format: FORMAT_NAME.to_string(), + version: FORMAT_VERSION, + exported_at: now, + embedding_model: snapshot.embedding_model, + embedding_dim: snapshot.embedding_dim, + schema_version: LATEST_SCHEMA_VERSION, + agent_memory_version: agent_memory_version.clone(), + exported_rows: snapshot.memories, + exported_links: snapshot.links, + }; + write_record_line(out.as_mut(), &header) + } + ExportEvent::Record(record) => write_record_line(out.as_mut(), &record), + })?; + // Flush INSIDE the blocking task: a BufWriter dropped unflushed + // would silently lose the tail of the file (Pitfall 8). + out.flush() + .map_err(|e| MemoryError::Internal(format!("export write failed: {e}")))?; + Ok(report) + }) + .await + .map_err(MemoryError::Join)? + } + /// The hybrid ranking (SEARCH-04/05), before truncation: every fusion /// candidate, best first. /// @@ -570,7 +893,16 @@ impl MemoryService { /// Delete a memory by id (MCP-04 / D-06). Returns `true` if a row was deleted, /// `false` for an unknown id (clean not-found, never an error). + /// + /// Errors are tiered for the wire by [`redact_storage_error`] (D-14). pub async fn forget(&self, id: i64) -> Result { + self.forget_unredacted(id) + .await + .map_err(|e| redact_storage_error("forget", e)) + } + + /// The body of [`Self::forget`], before its errors are tiered for the wire. + async fn forget_unredacted(&self, id: i64) -> Result { let store = self.store.clone(); tokio::task::spawn_blocking(move || store.forget(id)) .await @@ -589,11 +921,25 @@ impl MemoryService { /// not-found data, never an error); linking an edge that already exists is /// an idempotent [`LinkOutcome::Linked`] (D-02). The edge is stamped with /// the injected clock, which orders 1-hop expansion (D-07). + /// + /// Errors are tiered for the wire by [`redact_storage_error`] (D-14). pub async fn link( &self, from_id: i64, to_id: i64, kind: LinkKind, + ) -> Result { + self.link_unredacted(from_id, to_id, kind) + .await + .map_err(|e| redact_storage_error("link", e)) + } + + /// The body of [`Self::link`], before its errors are tiered for the wire. + async fn link_unredacted( + &self, + from_id: i64, + to_id: i64, + kind: LinkKind, ) -> Result { crate::domain::validate_link_endpoints(from_id, to_id)?; let now = self.clock.now(); @@ -607,11 +953,25 @@ impl MemoryService { /// `true` if it was removed, `false` if no such edge existed — the clean /// not-found of [`MemoryService::forget`], never an error (D-02). A /// self-pair simply never matches an edge, so it is a not-found too. + /// + /// Errors are tiered for the wire by [`redact_storage_error`] (D-14). pub async fn unlink( &self, from_id: i64, to_id: i64, kind: LinkKind, + ) -> Result { + self.unlink_unredacted(from_id, to_id, kind) + .await + .map_err(|e| redact_storage_error("unlink", e)) + } + + /// The body of [`Self::unlink`], before its errors are tiered for the wire. + async fn unlink_unredacted( + &self, + from_id: i64, + to_id: i64, + kind: LinkKind, ) -> Result { let store = self.store.clone(); tokio::task::spawn_blocking(move || store.unlink(from_id, to_id, kind)) @@ -631,10 +991,23 @@ impl MemoryService { /// (`embedding_status = 0`) and the sweep backfills it (SEARCH-03). The /// store applies the patch, the vector swap and the freshness bump (D-09) /// in one writer transaction. + /// + /// Errors are tiered for the wire by [`redact_storage_error`] (D-14). pub async fn update( &self, id: i64, patch: MemoryPatch, + ) -> Result, MemoryError> { + self.update_unredacted(id, patch) + .await + .map_err(|e| redact_storage_error("update", e)) + } + + /// The body of [`Self::update`], before its errors are tiered for the wire. + async fn update_unredacted( + &self, + id: i64, + patch: MemoryPatch, ) -> Result, MemoryError> { crate::domain::validate_patch(&patch)?; let now = self.clock.now(); @@ -685,7 +1058,18 @@ impl MemoryService { /// both the background interval task and `tests/ttl.rs`. The blocking /// `DecayEngine::sweep` runs on a `spawn_blocking` thread (Pitfall 2); the /// embed call is async, OUTSIDE `spawn_blocking` (Pitfall 5). + /// + /// Errors are tiered for the wire by [`redact_storage_error`] (D-14). The + /// background sweep task only logs the result, and the engine detail has + /// already gone to the error log inside the helper, so nothing is lost. pub async fn sweep(&self, now: i64) -> Result { + self.sweep_unredacted(now) + .await + .map_err(|e| redact_storage_error("sweep", e)) + } + + /// The body of [`Self::sweep`], before its errors are tiered for the wire. + async fn sweep_unredacted(&self, now: i64) -> Result { let store = self.store.clone(); let engine = DecayEngine::new(self.decay_cfg); let report = tokio::task::spawn_blocking(move || engine.sweep(store.as_ref(), now)) @@ -747,3 +1131,77 @@ impl MemoryService { /// How many pending rows one sweep tick backfills (bounded so a huge corpus /// embeds incrementally instead of one giant request — T-02-03). const BACKFILL_BATCH: usize = 128; + +#[cfg(test)] +mod tests { + use super::*; + + /// A storage-tier error with no client-input cause — the variant the seam + /// must strip. + fn storage_error() -> MemoryError { + MemoryError::Sqlite(rusqlite::Error::QueryReturnedNoRows) + } + + #[test] + fn storage_error_redaction_search_text_equals_the_constant() { + // The constant Phase 4's goldens assert against IS the template at + // op = "search"; this pins them together so neither can drift. + match redact_storage_error("search", storage_error()) { + MemoryError::Internal(m) => assert_eq!(m, SEARCH_STORAGE_FAILURE), + other => panic!("expected Internal, got {other:?}"), + } + match redact_storage_error("store", storage_error()) { + MemoryError::Internal(m) => assert_eq!( + m, + "store failed due to a storage error (details are in the server log)" + ), + other => panic!("expected Internal, got {other:?}"), + } + } + + #[test] + fn clock_skew_message_fires_only_beyond_the_tolerance() { + let now = 1_700_000_000; + assert_eq!(clock_skew_message(now, now), None); + assert_eq!( + clock_skew_message(now - 86_400, now), + None, + "a past export is normal" + ); + assert_eq!( + clock_skew_message(now + CLOCK_SKEW_TOLERANCE_SECS, now), + None, + "exactly the tolerance is still fine" + ); + let msg = + clock_skew_message(now + 3_601, now).expect("one second past the tolerance warns"); + assert!(msg.contains("ahead of this machine's clock"), "{msg}"); + assert!(msg.contains("3601s"), "names the delta: {msg}"); + assert!( + msg.contains(&now.to_string()), + "names the local clock: {msg}" + ); + } + + #[test] + fn storage_error_redaction_passes_client_and_crate_text_through() { + let pass_through = [ + MemoryError::InvalidArgument("x".to_string()), + MemoryError::InvalidType("x".to_string()), + MemoryError::InvalidQuery("x".to_string()), + MemoryError::Internal("y".to_string()), + MemoryError::SchemaTooNew { + db_version: 9, + binary_version: 3, + }, + ]; + for err in pass_through { + let before = err.to_string(); + let after = redact_storage_error("store", err).to_string(); + assert_eq!( + after, before, + "client/crate text must pass through unchanged" + ); + } + } +} diff --git a/crates/agent-memory-core/src/store/mod.rs b/crates/agent-memory-core/src/store/mod.rs index 9db26b9..1c5df91 100644 --- a/crates/agent-memory-core/src/store/mod.rs +++ b/crates/agent-memory-core/src/store/mod.rs @@ -11,9 +11,10 @@ pub mod sqlite; use crate::decay::{DecayConfig, RankWeights}; use crate::domain::{ - LinkKind, LinkOutcome, MemoryError, MemoryPatch, MemoryType, MemoryView, NewMemory, - RelatedLink, UpdatedMemory, + ImportedMemory, LinkKind, LinkOutcome, MemoryError, MemoryPatch, MemoryType, MemoryView, + NewMemory, RelatedLink, UpdatedMemory, }; +use crate::export::ExportEvent; use crate::service::{ListArgs, SearchArgs}; /// Synchronous persistence interface, implemented by [`sqlite::SqliteStore`]. @@ -31,6 +32,30 @@ pub trait Store: Send + Sync { now: i64, ) -> Result; + /// Insert a memory restored from a portable file (INTEROP-02), returning + /// its assigned id. The same single writer transaction and the same + /// `memories` INSERT as [`Store::insert`] — one shared statement, so the + /// two paths cannot drift — but with the file's provenance preserved + /// (Phase 5 D-10): + /// + /// - `created_at`, `access_count` and `expires_at` are written exactly as + /// carried by `m`; + /// - `last_accessed = now` (refreshed, so the restored row starts + /// decay-fresh) and `decay_score = 1.0`; + /// - `base_weight` follows the type pin, as for any insert; + /// - there is NO embedding parameter: the row ALWAYS lands + /// `embedding_status = 0` and no vector row is written. Restored rows are + /// re-embedded only by the sweep backfill against the local model pin + /// (D-13) — exactly one re-embed mechanism, never a mixed corpus. + /// + /// Expiry is the caller's rule, not this method's: the JSONL importer + /// skips and counts a row whose `expires_at <= now` BEFORE calling this, + /// whereas [`Store::sweep_expired`] deletes `expires_at < now`. Both + /// predicates are named here so the boundary is deliberate: a row whose + /// `expires_at == now` is never inserted by an import, and a row inserted + /// with `expires_at > now` is deleted by the first sweep after it passes. + fn insert_imported(&self, m: ImportedMemory, now: i64) -> Result; + /// K-nearest-neighbor candidate selection over `vec_memories` (SEARCH-02). /// Returns `(view, cosine_distance)` pairs, nearest first, oversampled /// beyond `args.limit` so the caller can re-rank by the similarity×decay @@ -78,17 +103,73 @@ pub trait Store: Send + Sync { /// back to [`Store::insert_embedding`] as `expected_content`. fn pending_embeddings(&self, limit: usize) -> Result, MemoryError>; - /// The INTEROP-01 idempotency probe: does a row with exactly this - /// `(source, mem_type, content)` key already exist? `MemoryService::import` - /// checks it before every insert so re-importing the same file is a no-op. + /// Stream the whole corpus for a portable export (DIST-04), driving + /// `visit` with the format-v1 inputs in file order. Nothing is collected: + /// each row is visited lazily as it is stepped, so the corpus is never + /// buffered (Phase 5 D-02). + /// + /// Exactly ONE pooled read connection and ONE read transaction span the + /// entire call, and inside it, in order: + /// + /// 1. one [`ExportEvent::Snapshot`] — the `meta` pin (`embedding_model`, + /// `embedding_dim`) plus `count(*)` of `memories` and `memory_links`; + /// 2. one [`ExportEvent::Record`] holding a `Record::Memory` per row of + /// `SELECT id, mem_type, content, tags, source, scope, created_at, + /// last_accessed, access_count, expires_at FROM memories ORDER BY id` + /// — every row present, expired rows included, no filter (D-06); + /// `key` is the row id, `type` is the stored string as-is (D-04); + /// 3. one `Record::Link` per row of `SELECT from_id, to_id, kind, + /// created_at FROM memory_links ORDER BY from_id, to_id, kind`, + /// endpoints exactly as stored (D-05). + /// + /// Why one snapshot (PITFALLS 5 / T-05-19): two statements on two pooled + /// connections could straddle a concurrent `link` whose endpoint was + /// inserted after the memories SELECT ran, yielding a link record whose + /// key is absent from the file — which the strict importer then rejects. + /// In WAL mode a single read transaction sees one consistent snapshot, and + /// the `memory_links` foreign keys guarantee every endpoint in that + /// snapshot is also a memory in it. The header counts come from the same + /// snapshot, so they always equal the records that follow. + /// + /// `embedding_dim` is TEXT in `meta`; a value that does not parse as an + /// integer is corrupt data and surfaces as + /// `MemoryError::Internal("meta.embedding_dim is not an integer (…)")` — + /// never a client-tier error. Tags decode with the `row_to_view` + /// convention (an undecodable tags column reads as no tags). The first + /// `Err` returned by `visit` aborts the stream and is returned unchanged. + fn export_rows( + &self, + visit: &mut dyn FnMut(ExportEvent) -> Result<(), MemoryError>, + ) -> Result<(), MemoryError>; + + /// The INTEROP-01 idempotency probe: which row, if any, carries exactly + /// this `(source, mem_type, content)` key? `MemoryService::import` checks + /// it before every insert so re-importing the same file is a no-op. /// `source` is `Option` and compared NULL-safely: a `None` source matches a /// stored NULL source, never an empty string (CR-01). - fn exists( + /// + /// The matching row's id is returned so an importer can map an + /// export-local key onto the EXISTING row instead of inserting a second + /// copy (Phase 5 D-07 / INTEROP-02); `None` means no such row. The key + /// itself is unchanged — the exact three-column, NULL-safe match, no new + /// identity column. + /// + /// Only **live** rows match: a row whose `expires_at <= now` is invisible + /// to the probe, exactly the predicate the JSONL importer applies to FILE + /// rows (D-10). Without this, a CLI import (which runs no sweep) could map + /// a key onto an expired-but-unswept row and hang imported edges on it, + /// only for the next `sweep_expired` to cascade them away (05-REVIEW + /// WR-03). When several live rows share the key (the store has no + /// uniqueness constraint), the **lowest matching id** is returned, so the + /// mapping is a contract rather than a scan-order accident (05-REVIEW + /// IN-01). + fn find_by_key( &self, source: Option<&str>, mem_type: MemoryType, content: &str, - ) -> Result; + now: i64, + ) -> Result, MemoryError>; /// List stored memories newest-first, honoring optional type/tag/scope filters /// and an optional limit. `now` is reserved for on-read decay recomputation in @@ -181,7 +262,8 @@ pub trait Store: Send + Sync { /// as (smaller id, larger id), so linking b→a after a→b is the same /// symmetric relation (REVIEW WR-03); `supersedes` / `caused_by` as given /// (D-03 directed identity). A duplicate of the stored row is an idempotent - /// no-op that still returns [`LinkOutcome::Linked`]. Canonicalisation is + /// no-op that returns `LinkOutcome::Linked { created: false }`; a fresh + /// insert returns `created: true` (Phase 5 D-12). Canonicalisation is /// storage-only: callers echo the request's ids on the wire, and /// [`LinkOutcome`] carries no stored ids. The self-link check belongs to /// the service seam and has already run. @@ -213,7 +295,9 @@ pub trait Store: Send + Sync { fn related(&self, id: i64, cap: usize) -> Result, MemoryError>; /// Bump the recency of the given ids: set `last_accessed = now` and increment - /// `access_count`. Runs on the writer lane. A no-op for an empty id slice. + /// `access_count`, saturating at `i64::MAX` so the column can never leave + /// integer range (an imported row carries its count verbatim — D-10). + /// Runs on the writer lane. A no-op for an empty id slice. fn bump_access(&self, ids: &[i64], now: i64) -> Result<(), MemoryError>; /// Delete every TTL-expired row (`expires_at IS NOT NULL AND expires_at < now`) diff --git a/crates/agent-memory-core/src/store/sqlite.rs b/crates/agent-memory-core/src/store/sqlite.rs index 860b404..a23f912 100644 --- a/crates/agent-memory-core/src/store/sqlite.rs +++ b/crates/agent-memory-core/src/store/sqlite.rs @@ -21,9 +21,10 @@ use rusqlite_migration::Migrations; use crate::decay::{DecayConfig, RankWeights, PINNED_HALF_LIFE_MULTIPLIER}; use crate::domain::{ - snippet, LinkDirection, LinkKind, LinkOutcome, MemoryError, MemoryPatch, MemoryType, - MemoryView, NewMemory, RelatedLink, UpdatedMemory, + snippet, ImportedMemory, LinkDirection, LinkKind, LinkOutcome, MemoryError, MemoryPatch, + MemoryType, MemoryView, NewMemory, RelatedLink, UpdatedMemory, }; +use crate::export::{ExportEvent, ExportSnapshot, Record}; use crate::service::{ListArgs, SearchArgs}; use crate::store::migrations::{migrations, LATEST_SCHEMA_VERSION}; use crate::store::Store; @@ -592,6 +593,56 @@ fn match_with_sanitized_retry( } } +/// The column values of one `memories` row about to be inserted, as bound by +/// [`insert_row`]. A struct rather than positional arguments so the two +/// callers cannot transpose a column and the helper stays under clippy's +/// argument limit. +struct MemoryRow<'a> { + mem_type: MemoryType, + content: &'a str, + /// The tags array already serialised with `serde_json`. + tags_json: &'a str, + source: Option<&'a str>, + scope: Option<&'a str>, + access_count: i64, + created_at: i64, + last_accessed: i64, + expires_at: Option, + embedding_status: i64, +} + +/// THE one `memories` INSERT, shared by [`Store::insert`] (fresh rows: counts +/// zeroed, both timestamps `now`) and [`Store::insert_imported`] (restored +/// rows: provenance preserved, `last_accessed` refreshed, always pending). +/// Keeping a single statement is what stops the two paths drifting in column +/// order or defaults (Phase 5 T-05-15). `base_weight` follows the type pin and +/// `decay_score` starts at 1.0 for every row. Runs inside the caller's writer +/// transaction and returns the new rowid; the `memories_ai` trigger mirrors +/// the content into FTS5 on the same statement. +fn insert_row(tx: &rusqlite::Transaction<'_>, row: &MemoryRow<'_>) -> Result { + let base_weight = if row.mem_type.is_pinned() { 2.0 } else { 1.0 }; + tx.execute( + "INSERT INTO memories \ + (mem_type, content, tags, source, scope, base_weight, decay_score, \ + access_count, created_at, last_accessed, expires_at, embedding_status) \ + VALUES (?1, ?2, ?3, ?4, ?5, ?6, 1.0, ?7, ?8, ?9, ?10, ?11)", + params![ + row.mem_type.as_wire_str(), + row.content, + row.tags_json, + row.source, + row.scope, + base_weight, + row.access_count, + row.created_at, + row.last_accessed, + row.expires_at, + row.embedding_status, + ], + )?; + Ok(tx.last_insert_rowid()) +} + impl Store for SqliteStore { fn insert( &self, @@ -600,7 +651,6 @@ impl Store for SqliteStore { now: i64, ) -> Result { let tags_json = serde_json::to_string(&new.tags).unwrap_or_else(|_| "[]".to_string()); - let base_weight = if new.mem_type.is_pinned() { 2.0 } else { 1.0 }; let expires_at = new.ttl_secs.map(|ttl| now + ttl); let embedding_status: i64 = i64::from(embedding.is_some()); @@ -612,24 +662,21 @@ impl Store for SqliteStore { .lock() .map_err(|_| MemoryError::Internal("writer lock poisoned".into()))?; let tx = conn.transaction()?; - tx.execute( - "INSERT INTO memories \ - (mem_type, content, tags, source, scope, base_weight, decay_score, \ - access_count, created_at, last_accessed, expires_at, embedding_status) \ - VALUES (?1, ?2, ?3, ?4, ?5, ?6, 1.0, 0, ?7, ?7, ?8, ?9)", - params![ - new.mem_type.as_wire_str(), - new.content, - tags_json, - new.source, - new.scope, - base_weight, - now, + let id = insert_row( + &tx, + &MemoryRow { + mem_type: new.mem_type, + content: &new.content, + tags_json: &tags_json, + source: new.source.as_deref(), + scope: new.scope.as_deref(), + access_count: 0, + created_at: now, + last_accessed: now, expires_at, embedding_status, - ], + }, )?; - let id = tx.last_insert_rowid(); if let Some(vector) = embedding { // Vector blob bound via bytemuck (safe zero-copy f32→u8 cast), // never formatted into the SQL (T-02-01). @@ -643,6 +690,40 @@ impl Store for SqliteStore { Ok(id) } + fn insert_imported(&self, m: ImportedMemory, now: i64) -> Result { + let tags_json = serde_json::to_string(&m.tags).unwrap_or_else(|_| "[]".to_string()); + + // Same writer lane and single transaction as `insert`, through the same + // shared statement. The file's provenance columns are bound as-is; + // `last_accessed` is the import-time clock (D-10). There is no vector + // write and `embedding_status` is hard-coded pending: restored rows are + // re-embedded only by the sweep backfill against the LOCAL model pin + // (D-13, PITFALLS 17), so a file written beside another model can never + // mix vectors into this corpus. + let mut conn = self + .writer + .lock() + .map_err(|_| MemoryError::Internal("writer lock poisoned".into()))?; + let tx = conn.transaction()?; + let id = insert_row( + &tx, + &MemoryRow { + mem_type: m.mem_type, + content: &m.content, + tags_json: &tags_json, + source: m.source.as_deref(), + scope: m.scope.as_deref(), + access_count: m.access_count, + created_at: m.created_at, + last_accessed: now, + expires_at: m.expires_at, + embedding_status: 0, + }, + )?; + tx.commit()?; + Ok(id) + } + fn knn_search( &self, query: Vec, @@ -748,27 +829,132 @@ impl Store for SqliteStore { Ok(out) } - fn exists( + fn export_rows( + &self, + visit: &mut dyn FnMut(ExportEvent) -> Result<(), MemoryError>, + ) -> Result<(), MemoryError> { + let conn = self.reads.get().map_err(MemoryError::Pool)?; + // ONE read transaction on ONE pooled connection spans the pin, both + // counts, and both row scans, so everything below is one WAL snapshot: + // a link committed concurrently can never reference a memory this + // export did not also see (T-05-19 / PITFALLS 5). `unchecked` only + // because the pooled handle is shared (`&Connection`); nothing else + // runs on it while we hold it. + let tx = conn.unchecked_transaction()?; + + let embedding_model: String = tx.query_row( + "SELECT value FROM meta WHERE key = 'embedding_model'", + [], + |row| row.get(0), + )?; + let dim_text: String = tx.query_row( + "SELECT value FROM meta WHERE key = 'embedding_dim'", + [], + |row| row.get(0), + )?; + // `meta.value` is TEXT; a non-integer dim is corrupt data (internal + // tier), never something a client did. + let embedding_dim: u32 = dim_text.trim().parse().map_err(|_| { + MemoryError::Internal(format!( + "meta.embedding_dim is not an integer ({dim_text:?})" + )) + })?; + let memories: i64 = tx.query_row("SELECT count(*) FROM memories", [], |row| row.get(0))?; + let links: i64 = tx.query_row("SELECT count(*) FROM memory_links", [], |row| row.get(0))?; + visit(ExportEvent::Snapshot(ExportSnapshot { + embedding_model, + embedding_dim, + memories: u64::try_from(memories).unwrap_or(0), + links: u64::try_from(links).unwrap_or(0), + }))?; + + { + // The D-04 memory column list — the wire contract. Deliberately + // NOT `base_weight`, `decay_score` or `embedding_status`, and + // `vec_memories` is never read: derived data and vectors stay home. + let mut stmt = tx.prepare( + "SELECT id, mem_type, content, tags, source, scope, created_at, \ + last_accessed, access_count, expires_at \ + FROM memories ORDER BY id", + )?; + let rows = stmt.query_map([], |row| { + let tags_json: String = row.get("tags")?; + Ok(Record::Memory { + key: row.get("id")?, + mem_type: row.get("mem_type")?, + content: row.get("content")?, + tags: serde_json::from_str(&tags_json).unwrap_or_default(), + source: row.get("source")?, + scope: row.get("scope")?, + created_at: row.get("created_at")?, + last_accessed: row.get("last_accessed")?, + access_count: row.get("access_count")?, + expires_at: row.get("expires_at")?, + }) + })?; + for row in rows { + visit(ExportEvent::Record(row?))?; + } + } + + { + // Endpoints exactly as stored (relates_to canonical (min, max), + // directed kinds as given — D-05); deterministic order so two + // exports of one database are byte-identical. + let mut stmt = tx.prepare( + "SELECT from_id, to_id, kind, created_at FROM memory_links \ + ORDER BY from_id, to_id, kind", + )?; + let rows = stmt.query_map([], |row| { + Ok(Record::Link { + from: row.get("from_id")?, + to: row.get("to_id")?, + kind: row.get("kind")?, + created_at: row.get("created_at")?, + }) + })?; + for row in rows { + visit(ExportEvent::Record(row?))?; + } + } + + // Read-only: commit simply ends the snapshot. + tx.commit()?; + Ok(()) + } + + fn find_by_key( &self, source: Option<&str>, mem_type: MemoryType, content: &str, - ) -> Result { + now: i64, + ) -> Result, MemoryError> { let conn = self.reads.get().map_err(MemoryError::Pool)?; - // The INTEROP-01 dedup probe: all three key values bound, never - // formatted into the SQL (T-02-20). `source IS ?1` is SQLite's - // NULL-safe equality: a `None` source binds NULL and matches the NULL - // the insert stored (`source = ?1` would be NULL = NULL → never true, - // which made every source-less re-import a duplicate — CR-01). + // The INTEROP-01 dedup probe: all key values bound, never formatted + // into the SQL (T-02-20). `source IS ?1` is SQLite's NULL-safe + // equality: a `None` source binds NULL and matches the NULL the insert + // stored (`source = ?1` would be NULL = NULL → never true, which made + // every source-less re-import a duplicate — CR-01). The matching id + // comes back so an importer can map onto the existing row (Phase 5 + // D-07). + // + // Only live rows count: `expires_at <= now` is the importer's own + // "already expired" predicate for file rows (D-10), applied here to the + // target so an edge is never hung on a row the next sweep deletes + // (05-REVIEW WR-03). `ORDER BY id` pins which duplicate wins when the + // key is not unique — the lowest id, by contract (05-REVIEW IN-01). let found = conn .query_row( - "SELECT 1 FROM memories \ - WHERE source IS ?1 AND mem_type = ?2 AND content = ?3 LIMIT 1", - params![source, mem_type.as_wire_str(), content], - |_| Ok(()), + "SELECT id FROM memories \ + WHERE source IS ?1 AND mem_type = ?2 AND content = ?3 \ + AND (expires_at IS NULL OR expires_at > ?4) \ + ORDER BY id LIMIT 1", + params![source, mem_type.as_wire_str(), content, now], + |row| row.get::<_, i64>(0), ) .optional()?; - Ok(found.is_some()) + Ok(found) } fn list(&self, args: ListArgs, _now: i64) -> Result, MemoryError> { @@ -972,8 +1158,10 @@ impl Store for SqliteStore { } // The (from_id, to_id, kind) primary key makes a duplicate an - // idempotent no-op: changes() == 0 is still Linked (D-02). The kind - // binds only the closed enum's wire form (T-04-10). The endpoints are + // idempotent no-op: changes() == 0 is still a success (D-02), now + // reported as `Linked { created: false }` so an importer can count it + // (Phase 5 D-12). The kind binds only the closed enum's wire form + // (T-04-10). The endpoints are // the STORED pair: relates_to is canonicalised to (min, max), so b->a // after a->b hits the same key and stays one symmetric relation // (REVIEW WR-03); directed kinds bind the request order unchanged. @@ -985,8 +1173,12 @@ impl Store for SqliteStore { VALUES (?1, ?2, ?3, ?4)", params![stored_from, stored_to, kind.as_wire_str(), now], )?; + // Read before commit: changes() reports the most recent statement on + // this connection, and INSERT OR IGNORE changes exactly one row when it + // inserted and zero when the primary key already existed. + let created = tx.changes() == 1; tx.commit()?; - Ok(LinkOutcome::Linked) + Ok(LinkOutcome::Linked { created }) } fn unlink(&self, from_id: i64, to_id: i64, kind: LinkKind) -> Result { @@ -1083,9 +1275,17 @@ impl Store for SqliteStore { // // The memories_au trigger re-mirrors the FTS5 content row for free. // All values bound — never formatted into the SQL (T-04-06). + // + // The bump saturates at i64::MAX (same expression as bump_access): + // SQLite's `+` does not raise on integer overflow, it converts the + // result to REAL, and rusqlite's `FromSql for i64` then rejects the + // column on every later export_rows read (05-REVIEW WR-02). tx.execute( "UPDATE memories SET content = ?2, tags = ?3, scope = ?4, expires_at = ?5, \ - last_accessed = ?6, access_count = access_count + 1, decay_score = 1.0 \ + last_accessed = ?6, \ + access_count = CASE WHEN access_count < 9223372036854775807 \ + THEN access_count + 1 ELSE access_count END, \ + decay_score = 1.0 \ WHERE id = ?1", params![id, content, tags_json, scope, expires_at, now], )?; @@ -1199,8 +1399,15 @@ impl Store for SqliteStore { .map(|i| format!("?{}", i + 2)) .collect::>() .join(", "); + // The increment saturates at i64::MAX: SQLite's `+` silently converts + // an overflowing integer to REAL, which rusqlite's `i64` reader then + // rejects on every later `export_rows` — one poisoned row would break + // the whole backup path (05-REVIEW WR-02). An imported row may carry + // any non-negative count verbatim (D-10), so the guard lives here. let sql = format!( - "UPDATE memories SET last_accessed = ?1, access_count = access_count + 1 \ + "UPDATE memories SET last_accessed = ?1, \ + access_count = CASE WHEN access_count < 9223372036854775807 \ + THEN access_count + 1 ELSE access_count END \ WHERE id IN ({placeholders})" ); let mut params: Vec<&dyn rusqlite::ToSql> = Vec::with_capacity(ids.len() + 1); diff --git a/crates/agent-memory-core/tests/export_import.rs b/crates/agent-memory-core/tests/export_import.rs new file mode 100644 index 0000000..cd92b54 --- /dev/null +++ b/crates/agent-memory-core/tests/export_import.rs @@ -0,0 +1,1460 @@ +//! DIST-04 / INTEROP-02 proof file: the portable export (format v1 JSONL) and +//! its import. +//! +//! Export decisions proven here: D-03 (line 1 is the header, read from the +//! `meta` pin and the injected clock, with `agent_memory_version` supplied by +//! the caller), D-04 (a memory record carries exactly the eleven documented +//! keys — no derived data, no vector), D-05 (links after all memories, as +//! stored), D-06 (every row, expired rows included, ascending id) and the +//! canonical-output rule (two exports of an unchanged database are +//! byte-identical). +//! +//! Import decisions proven here: D-07 (dedup on the unchanged `(source, +//! mem_type, content)` key; identical rows collapse; `source` preserved +//! verbatim), D-08 (two-pass validate-before-write; export-local keys are +//! remapped through a `key → id` map so a non-empty target with overlapping +//! rowids never gains a phantom edge — PITFALLS 18; links whose endpoint was +//! skipped are dropped and counted), D-10 (`created_at` / `access_count` / +//! `expires_at` preserved, `last_accessed` refreshed, `expires_at <= now` +//! skipped at the boundary — PITFALLS 19), D-11 (nothing is written when +//! validation fails), D-12 (the six-count report) and D-13 (rows land pending +//! and only the sweep re-embeds them, after which they are hybrid-searchable). +//! +//! Harness: a REAL on-disk temp DB, an injected `TestClock`, and a failing +//! `FakeEmbedder` by default (stores land pending; neither the exporter nor +//! the jsonl importer touches the embedder). The export is written through a +//! real file sink under the harness's temp dir and read back, so the +//! boxed-writer path is exercised end to end. Round trips use TWO harnesses — +//! a source and a target — each with its own database and clock. Stored +//! columns and edge rows are read through a raw side connection so the +//! assertions see the tables themselves. The store is also exposed as an +//! `Arc` so the one-snapshot concurrency proof (gap closure +//! 05-05) can drive `Store::export_rows` and a writer thread against one +//! store. + +use std::collections::HashMap; +use std::path::{Path, PathBuf}; +use std::sync::mpsc; +use std::sync::Arc; +use std::time::Duration; + +use agent_memory_core::clock::{Clock, TestClock}; +use agent_memory_core::decay::DecayConfig; +use agent_memory_core::domain::{ + LinkKind, LinkOutcome, MemoryError, MemoryPatch, MemoryType, NewMemory, +}; +use agent_memory_core::embed::{Embedder, FakeEmbedder}; +use agent_memory_core::export::{ExportEvent, ExportSnapshot, Record, FORMAT_NAME, FORMAT_VERSION}; +use agent_memory_core::import::jsonl::{parse_jsonl, JsonlFile}; +use agent_memory_core::service::{ + ExportReport, ImportReport, ListArgs, MemoryService, SearchArgs, SearchMode, +}; +use agent_memory_core::store::migrations::LATEST_SCHEMA_VERSION; +use agent_memory_core::store::sqlite::{register_vec_extension, SqliteStore}; +use agent_memory_core::store::Store; + +/// The literal passed as `agent_memory_version` — the core crate has no say +/// in the binary's version (RESEARCH Pitfall 9), so the test pins what it +/// passed in. +const TEST_VERSION: &str = "test-0.0.0"; + +struct Harness { + service: MemoryService, + store: Arc, + clock: Arc, + db_path: PathBuf, + dir: tempfile::TempDir, +} + +fn harness() -> Harness { + harness_with(Arc::new(FakeEmbedder::failing())) +} + +fn harness_with(embedder: Arc) -> Harness { + let dir = tempfile::tempdir().expect("tempdir"); + let db_path = dir.path().join("memory.db"); + let store = Arc::new(SqliteStore::open(&db_path).expect("open store")); + let clock = Arc::new(TestClock::new(1_000)); + let service = MemoryService::new( + store.clone(), + clock.clone() as Arc, + embedder, + DecayConfig::default(), + ); + Harness { + service, + store, + clock, + db_path, + dir, + } +} + +fn new_memory(content: &str, mem_type: MemoryType) -> NewMemory { + NewMemory { + content: content.to_string(), + mem_type, + tags: Vec::new(), + source: None, + scope: None, + ttl_secs: None, + } +} + +async fn store(h: &Harness, new: NewMemory) -> i64 { + h.service.store(new).await.expect("store succeeds") +} + +async fn link(h: &Harness, from: i64, to: i64, kind: LinkKind) { + let outcome = h.service.link(from, to, kind).await.expect("link succeeds"); + assert_eq!(outcome, LinkOutcome::Linked { created: true }); +} + +/// Export through a real file sink under the harness dir and read it back. +async fn export_to_string(h: &Harness, name: &str) -> (String, ExportReport) { + let path = h.dir.path().join(name); + let file = std::fs::File::create(&path).expect("create export sink"); + let sink: Box = Box::new(std::io::BufWriter::new(file)); + let report = h + .service + .export(sink, TEST_VERSION.to_string()) + .await + .expect("export succeeds"); + let text = std::fs::read_to_string(&path).expect("read export back"); + (text, report) +} + +/// Parse every line of an export as a typed record. +fn lines(text: &str) -> Vec { + text.lines() + .enumerate() + .map(|(i, line)| { + serde_json::from_str(line) + .unwrap_or_else(|e| panic!("line {} is not a record: {e}\n{line}", i + 1)) + }) + .collect() +} + +#[tokio::test] +async fn export_of_an_empty_database_is_exactly_one_header_line() { + let h = harness(); + + let (text, report) = export_to_string(&h, "empty.jsonl").await; + + assert_eq!(text.lines().count(), 1, "exactly one line:\n{text}"); + assert!( + text.ends_with('\n'), + "the header line is newline-terminated" + ); + let records = lines(&text); + match &records[0] { + Record::Header { + format, + version, + exported_at, + embedding_model, + embedding_dim, + schema_version, + agent_memory_version, + exported_rows, + exported_links, + } => { + assert_eq!(format, FORMAT_NAME); + assert_eq!(format, "agent-memory-export"); + assert_eq!(*version, FORMAT_VERSION); + assert_eq!(*version, 1); + assert_eq!(*exported_at, 1_000, "exported_at comes from the clock"); + assert_eq!(embedding_model, "nomic-embed-text", "read from meta"); + assert_eq!(*embedding_dim, 768, "read from meta"); + assert_eq!(*schema_version, LATEST_SCHEMA_VERSION); + assert_eq!(*schema_version, 3); + assert_eq!(agent_memory_version, TEST_VERSION); + assert_eq!(*exported_rows, 0); + assert_eq!(*exported_links, 0); + } + other => panic!("line 1 must be the header, got {other:?}"), + } + assert_eq!( + report, + ExportReport { + memories: 0, + links: 0 + } + ); +} + +#[tokio::test] +async fn export_writes_every_memory_ascending_then_every_link_as_stored() { + let h = harness(); + + // Three memories at three clock values: one fully decorated with a + // far-future TTL, one of a different type, one bare. + let a = store( + &h, + NewMemory { + content: "decided to export JSONL".to_string(), + mem_type: MemoryType::Decision, + tags: vec!["export".to_string(), "format".to_string()], + source: Some("unit-test".to_string()), + scope: Some("phase-5".to_string()), + ttl_secs: Some(1_000_000), + }, + ) + .await; + h.clock.advance(10); + let b = store( + &h, + new_memory("stream one line at a time", MemoryType::Pattern), + ) + .await; + h.clock.advance(10); + let c = store(&h, new_memory("no tags here", MemoryType::Todo)).await; + assert!(a < b && b < c, "ids ascend with insertion order"); + + h.clock.advance(5); + let supersedes_at = h.clock.now(); + link(&h, a, b, LinkKind::Supersedes).await; + h.clock.advance(5); + let relates_at = h.clock.now(); + // Linked c -> b: stored canonically as (min, max) = (b, c). + link(&h, c, b, LinkKind::RelatesTo).await; + + let (text, report) = export_to_string(&h, "three.jsonl").await; + let records = lines(&text); + assert_eq!(records.len(), 6, "header + 3 memories + 2 links:\n{text}"); + + match &records[0] { + Record::Header { + exported_rows, + exported_links, + exported_at, + .. + } => { + assert_eq!(*exported_rows, 3); + assert_eq!(*exported_links, 2); + assert_eq!(*exported_at, h.clock.now()); + } + other => panic!("line 1 must be the header, got {other:?}"), + } + + // The rows as the service itself reads them, oldest first to match + // ascending id. + let mut views = h + .service + .list(ListArgs::default()) + .await + .expect("list succeeds"); + views.sort_by_key(|v| v.id); + assert_eq!(views.len(), 3); + + let expected_sources = [Some("unit-test".to_string()), None, None]; + let expected_expiry = [Some(1_000 + 1_000_000), None, None]; + for (i, rec) in records[1..4].iter().enumerate() { + let view = &views[i]; + match rec { + Record::Memory { + key, + mem_type, + content, + tags, + source, + scope, + created_at, + last_accessed, + access_count, + expires_at, + } => { + assert_eq!(*key, view.id, "key is the stored id"); + assert_eq!(mem_type, view.mem_type.as_wire_str()); + assert_eq!(content, &view.content); + assert_eq!(tags, &view.tags); + assert_eq!(source, &expected_sources[i]); + assert_eq!(scope, &view.scope); + assert_eq!(*created_at, view.created_at); + assert_eq!(*last_accessed, view.last_accessed); + assert_eq!(*access_count, 0, "fresh rows have never been accessed"); + assert_eq!(*expires_at, expected_expiry[i]); + } + other => panic!("line {} must be a memory, got {other:?}", i + 2), + } + } + // Ascending key order, strictly. + let keys: Vec = records[1..4] + .iter() + .map(|r| match r { + Record::Memory { key, .. } => *key, + other => panic!("expected memory, got {other:?}"), + }) + .collect(); + assert_eq!(keys, vec![a, b, c]); + + // Links after all memories, ordered by (from, to, kind), endpoints as + // stored: supersedes keeps request order, relates_to is (min, max). + assert_eq!( + records[4], + Record::Link { + from: a, + to: b, + kind: "supersedes".to_string(), + created_at: supersedes_at, + } + ); + assert_eq!( + records[5], + Record::Link { + from: b.min(c), + to: b.max(c), + kind: "relates_to".to_string(), + created_at: relates_at, + } + ); + + assert_eq!( + report, + ExportReport { + memories: 3, + links: 2 + } + ); +} + +#[tokio::test] +async fn memory_records_carry_exactly_the_documented_keys() { + let h = harness(); + store( + &h, + NewMemory { + content: "fully decorated".to_string(), + mem_type: MemoryType::Architecture, + tags: vec!["t".to_string()], + source: Some("s".to_string()), + scope: Some("sc".to_string()), + ttl_secs: Some(500), + }, + ) + .await; + store(&h, new_memory("bare", MemoryType::Error)).await; + + let (text, _) = export_to_string(&h, "keys.jsonl").await; + let memory_lines: Vec<&str> = text.lines().skip(1).collect(); + assert_eq!(memory_lines.len(), 2); + + let documented = [ + "record", + "key", + "type", + "content", + "tags", + "source", + "scope", + "created_at", + "last_accessed", + "access_count", + "expires_at", + ]; + for line in memory_lines { + let value: serde_json::Value = serde_json::from_str(line).expect("memory line is JSON"); + let object = value.as_object().expect("memory line is an object"); + let mut keys: Vec<&str> = object.keys().map(String::as_str).collect(); + keys.sort_unstable(); + let mut expected = documented.to_vec(); + expected.sort_unstable(); + assert_eq!(keys, expected, "exactly the 11 documented keys:\n{line}"); + for derived in [ + "decay_score", + "base_weight", + "embedding_status", + "embedding", + ] { + assert!( + !object.contains_key(derived), + "{derived} is derived data and must never travel:\n{line}" + ); + } + // Optional fields are written explicitly, never omitted. + assert!(object.contains_key("source") && object.contains_key("expires_at")); + } +} + +#[tokio::test] +async fn expired_rows_are_still_exported() { + let h = harness(); + let id = store( + &h, + NewMemory { + content: "short lived".to_string(), + mem_type: MemoryType::Todo, + tags: Vec::new(), + source: None, + scope: None, + ttl_secs: Some(10), + }, + ) + .await; + // Well past expiry, and NO sweep has run: the row is still in the table, + // and export writes what is in the table (D-06 — expiry is the importer's + // policy, not the exporter's). + h.clock.advance(100); + + let (text, report) = export_to_string(&h, "expired.jsonl").await; + let records = lines(&text); + assert_eq!(records.len(), 2, "header + the expired row:\n{text}"); + match &records[1] { + Record::Memory { + key, expires_at, .. + } => { + assert_eq!(*key, id); + assert_eq!(*expires_at, Some(1_000 + 10)); + } + other => panic!("expected the expired memory, got {other:?}"), + } + assert_eq!(report.memories, 1); +} + +#[tokio::test] +async fn two_exports_of_an_unchanged_database_are_byte_identical() { + let h = harness(); + let a = store(&h, new_memory("first", MemoryType::Decision)).await; + let b = store(&h, new_memory("second", MemoryType::Pattern)).await; + link(&h, a, b, LinkKind::CausedBy).await; + + let (first, first_report) = export_to_string(&h, "one.jsonl").await; + let (second, second_report) = export_to_string(&h, "two.jsonl").await; + + assert_eq!(first, second); + assert_eq!(first_report, second_report); + assert_eq!(first.lines().count(), 4); +} + +// --------------------------------------------------------------------------- +// Export half — gap closure 05-05: one-snapshot concurrency proof +// --------------------------------------------------------------------------- + +/// Serialise a header followed by the records that were streamed after it as +/// one format-v1 file: one `serde_json` line per record, newline-terminated. +fn record_lines(header: Record, records: &[Record]) -> String { + let mut text = String::new(); + for record in std::iter::once(&header).chain(records.iter()) { + text.push_str(&serde_json::to_string(record).expect("record serialises")); + text.push('\n'); + } + text +} + +/// Held-out evidence for the 05-02 backstop truth (D-06 / D-02, T-05-19, +/// PITFALLS 5): every row of an export streams from ONE read transaction on +/// ONE pooled connection, so the header counts and the link records can never +/// straddle a concurrent write. +/// +/// The writer thread is a real second connection lane — the store's writer +/// mutex — racing the exporter's read-pool connection, exactly the +/// MCP-server-plus-REST-daemon situation. Ordering is decided only by the two +/// channel hand-offs (`go`: exporter → writer, fired on the first memory +/// record; `done`: writer → exporter, carrying the new id); every wait is +/// bounded by `recv_timeout`, so a blocked write (a deadlock between the open +/// snapshot and the writer) fails fast with a named message instead of hanging +/// the suite. WAL snapshot isolation is what makes the committed row invisible +/// to the open export while a fresh connection already sees it. +#[tokio::test] +async fn a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key() { + let h = harness(); + let ((a, _b, _c), _) = seed_round_trip_source(&h).await; + assert_eq!(row_count(&h.db_path), 3); + assert_eq!(edge_count(&h.db_path), 2); + + let (go_tx, go_rx) = mpsc::channel::<()>(); + let (done_tx, done_rx) = mpsc::channel::(); + let writer_store = h.store.clone(); + let now = h.clock.now(); + // A std::thread, not a tokio task: `Store` methods are synchronous and the + // exporter below blocks the current-thread runtime while it waits. + let writer = std::thread::spawn(move || { + go_rx + .recv_timeout(Duration::from_secs(10)) + .expect("the exporter never reached its first memory record within 10 s"); + let new_id = writer_store + .insert( + new_memory("delta arrived mid-export", MemoryType::Pattern), + None, + now, + ) + .expect("a concurrent insert must not block behind the open read snapshot"); + let outcome = writer_store + .link(new_id, a, LinkKind::CausedBy, now) + .expect("a concurrent link must not block behind the open read snapshot"); + assert_eq!(outcome, LinkOutcome::Linked { created: true }); + done_tx + .send(new_id) + .expect("the exporter is waiting for the committed id"); + new_id + }); + + let mut fired = false; + let mut snapshot: Option = None; + let mut records: Vec = Vec::new(); + let mut new_id_seen: Option = None; + h.store + .export_rows(&mut |event: ExportEvent| -> Result<(), MemoryError> { + match event { + ExportEvent::Snapshot(s) => snapshot = Some(s), + ExportEvent::Record(record) => { + if !fired && matches!(record, Record::Memory { .. }) { + fired = true; + go_tx.send(()).expect("the writer thread is waiting on go"); + let new_id = done_rx.recv_timeout(Duration::from_secs(10)).expect( + "the writer did not commit within 10 s while the export snapshot \ + was open — the write blocked behind the reader (deadlock)", + ); + // A FRESH raw connection already sees the commit while + // this export's read transaction is still open. + assert_eq!( + row_count(&h.db_path), + 4, + "the writer's memory is committed while the snapshot is open" + ); + assert_eq!( + edge_count(&h.db_path), + 3, + "the writer's link is committed while the snapshot is open" + ); + new_id_seen = Some(new_id); + } + records.push(record); + } + } + Ok(()) + }) + .expect("export_rows succeeds with a writer racing it"); + + let new_id = writer.join().expect("the writer thread completed"); + assert_eq!(new_id_seen, Some(new_id), "the hand-off carried the new id"); + + // Snapshot isolation: the writer's commit is invisible to THIS export. + let snapshot = snapshot.expect("the snapshot event was emitted first"); + assert_eq!(snapshot.memories, 3); + assert_eq!(snapshot.links, 2); + let mut memory_records = 0; + let mut link_records = 0; + for record in &records { + match record { + Record::Header { .. } => panic!("the store never emits a header record"), + Record::Memory { key, .. } => { + memory_records += 1; + assert_ne!( + *key, new_id, + "the mid-export memory leaked into the snapshot" + ); + } + Record::Link { from, to, .. } => { + link_records += 1; + assert!( + *from != new_id && *to != new_id, + "the mid-export link leaked into the snapshot: {from} -> {to}" + ); + } + } + } + assert_eq!(memory_records, 3); + assert_eq!(link_records, 2); + + // The raced file — header mirrored from the snapshot (as the service does) + // plus every visited record — is accepted by the strict reader with exact + // counts and no dangling key. + let header = Record::Header { + format: FORMAT_NAME.to_string(), + version: FORMAT_VERSION, + exported_at: h.clock.now(), + embedding_model: snapshot.embedding_model.clone(), + embedding_dim: snapshot.embedding_dim, + schema_version: LATEST_SCHEMA_VERSION, + agent_memory_version: TEST_VERSION.to_string(), + exported_rows: snapshot.memories, + exported_links: snapshot.links, + }; + let raced = record_lines(header, &records); + let parsed = parse_jsonl(&raced).expect( + "the raced export must parse as format v1 — a dangling link key or a count \ + mismatch here means two snapshots were straddled", + ); + assert_eq!(parsed.memories.len(), 3); + assert_eq!(parsed.links.len(), 2); + + // The NEXT snapshot sees both the memory and the edge. + let (text, report) = export_to_string(&h, "after-race.jsonl").await; + assert_eq!( + report, + ExportReport { + memories: 4, + links: 3 + } + ); + let after = lines(&text); + assert!( + after.iter().any(|record| matches!( + record, + Record::Memory { key, content, .. } + if *key == new_id && content == "delta arrived mid-export" + )), + "the next export carries the mid-export memory:\n{text}" + ); + assert!( + after.iter().any(|record| matches!( + record, + Record::Link { from, to, kind, .. } + if *from == new_id && *to == a && kind == "caused_by" + )), + "the next export carries the mid-export link:\n{text}" + ); + let parsed = parse_jsonl(&text).expect("the next export parses as format v1"); + assert_eq!(parsed.memories.len(), 4); + assert_eq!(parsed.links.len(), 3); +} + +// --------------------------------------------------------------------------- +// Import half (plan 05-03) — helpers +// --------------------------------------------------------------------------- + +fn raw_conn(db_path: &Path) -> rusqlite::Connection { + register_vec_extension().expect("vec0 registers"); + rusqlite::Connection::open(db_path).expect("open raw connection") +} + +/// Total rows in `memories`. +fn row_count(db_path: &Path) -> i64 { + raw_conn(db_path) + .query_row("SELECT count(*) FROM memories", [], |row| row.get(0)) + .expect("count memories") +} + +/// Total number of edge rows in `memory_links`. +fn edge_count(db_path: &Path) -> i64 { + raw_conn(db_path) + .query_row("SELECT count(*) FROM memory_links", [], |row| row.get(0)) + .expect("count edges") +} + +/// Edge rows touching `id` in either direction. +fn edges_touching(db_path: &Path, id: i64) -> i64 { + raw_conn(db_path) + .query_row( + "SELECT count(*) FROM memory_links WHERE from_id = ?1 OR to_id = ?1", + [id], + |row| row.get(0), + ) + .expect("count edges touching id") +} + +/// Edge rows whose either endpoint no longer exists in `memories`. +fn orphan_edges(db_path: &Path) -> i64 { + raw_conn(db_path) + .query_row( + "SELECT count(*) FROM memory_links \ + WHERE from_id NOT IN (SELECT id FROM memories) \ + OR to_id NOT IN (SELECT id FROM memories)", + [], + |row| row.get(0), + ) + .expect("count orphan edges") +} + +/// `embedding_status` of one row. +fn status(db_path: &Path, id: i64) -> i64 { + raw_conn(db_path) + .query_row( + "SELECT embedding_status FROM memories WHERE id = ?1", + [id], + |row| row.get(0), + ) + .expect("read embedding_status") +} + +/// Vector rows for one memory id. +fn vec_rows(db_path: &Path, id: i64) -> i64 { + raw_conn(db_path) + .query_row( + "SELECT count(*) FROM vec_memories WHERE memory_id = ?1", + [id], + |row| row.get(0), + ) + .expect("count vector rows") +} + +/// The id of the (single) row with exactly this content. +fn id_by_content(db_path: &Path, content: &str) -> i64 { + raw_conn(db_path) + .query_row( + "SELECT id FROM memories WHERE content = ?1", + [content], + |row| row.get(0), + ) + .unwrap_or_else(|e| panic!("exactly one row with content {content:?}: {e}")) +} + +/// The provenance columns the importer must preserve or refresh (D-10), read +/// straight from the table. +#[derive(Debug, PartialEq, Eq)] +struct Provenance { + source: Option, + created_at: i64, + last_accessed: i64, + access_count: i64, + expires_at: Option, +} + +fn provenance(db_path: &Path, id: i64) -> Provenance { + raw_conn(db_path) + .query_row( + "SELECT source, created_at, last_accessed, access_count, expires_at \ + FROM memories WHERE id = ?1", + [id], + |row| { + Ok(Provenance { + source: row.get(0)?, + created_at: row.get(1)?, + last_accessed: row.get(2)?, + access_count: row.get(3)?, + expires_at: row.get(4)?, + }) + }, + ) + .expect("read provenance columns") +} + +/// `(from_id, to_id, kind, created_at)` of every edge, in storage order. +fn edges(db_path: &Path) -> Vec<(i64, i64, String, i64)> { + let conn = raw_conn(db_path); + let mut stmt = conn + .prepare( + "SELECT from_id, to_id, kind, created_at FROM memory_links \ + ORDER BY from_id, to_id, kind", + ) + .expect("prepare edges"); + let rows = stmt + .query_map([], |row| { + Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?)) + }) + .expect("query edges"); + rows.map(|r| r.expect("edge row")).collect() +} + +/// Parse a file the strict way and import it into `h`. +async fn import(h: &Harness, text: &str) -> ImportReport { + let file: JsonlFile = parse_jsonl(text).expect("export parses as format v1"); + h.service + .import_jsonl(file) + .await + .expect("import_jsonl succeeds") +} + +/// Every line as a JSON value with the fields that legitimately differ between +/// a source export and a re-export of its import nulled out: `key` and link +/// `from`/`to` (the target assigns its own ids), `last_accessed` (refreshed on +/// import, D-10) and the header's `exported_at` / `agent_memory_version`. +fn normalised(text: &str) -> Vec { + text.lines() + .map(|line| { + let mut value: serde_json::Value = serde_json::from_str(line).expect("json line"); + if let Some(object) = value.as_object_mut() { + for field in [ + "key", + "last_accessed", + "from", + "to", + "exported_at", + "agent_memory_version", + ] { + if object.contains_key(field) { + object.insert(field.to_string(), serde_json::Value::Null); + } + } + } + value + }) + .collect() +} + +fn six( + imported: usize, + skipped_duplicates: usize, + skipped_expired: usize, + links_imported: usize, + links_skipped_duplicates: usize, + links_dropped: usize, +) -> ImportReport { + ImportReport { + imported, + skipped_duplicates, + skipped_expired, + links_imported, + links_skipped_duplicates, + links_dropped, + } +} + +// --------------------------------------------------------------------------- +// Import half — Task 1 (tracer): round trip + idempotency +// --------------------------------------------------------------------------- + +/// Source fixture shared by the round-trip proofs: three decorated memories +/// stored at distinct clock values, two links of two kinds. Returns the +/// source ids `(a, b, c)` and the two link times `(supersedes_at, relates_at)`. +async fn seed_round_trip_source(src: &Harness) -> ((i64, i64, i64), (i64, i64)) { + let a = store( + src, + NewMemory { + content: "alpha decided the export format".to_string(), + mem_type: MemoryType::Decision, + tags: vec!["x".to_string(), "y".to_string()], + source: Some("gsd-state".to_string()), + scope: Some("proj".to_string()), + ttl_secs: Some(1_000_000), + }, + ) + .await; + src.clock.advance(10); + let b = store( + src, + new_memory("beta streams one line at a time", MemoryType::Pattern), + ) + .await; + src.clock.advance(10); + let c = store( + src, + NewMemory { + content: "gamma supersedes nothing yet".to_string(), + mem_type: MemoryType::Decision, + tags: Vec::new(), + source: Some("manual".to_string()), + scope: None, + ttl_secs: None, + }, + ) + .await; + src.clock.advance(5); + let supersedes_at = src.clock.now(); + link(src, a, b, LinkKind::Supersedes).await; + src.clock.advance(5); + let relates_at = src.clock.now(); + link(src, c, b, LinkKind::RelatesTo).await; + ((a, b, c), (supersedes_at, relates_at)) +} + +#[tokio::test] +async fn round_trip_into_an_empty_database_recreates_memories_and_links() { + let src = harness(); + let ((a, b, c), (supersedes_at, relates_at)) = seed_round_trip_source(&src).await; + let (text, export_report) = export_to_string(&src, "source.jsonl").await; + assert_eq!( + export_report, + ExportReport { + memories: 3, + links: 2 + } + ); + + // The target is another machine: its own empty database and a clock far + // from the source's, so the D-10 refresh is observable. + let dst = harness(); + dst.clock.set(5_000); + let report = import(&dst, &text).await; + assert_eq!(report, six(3, 0, 0, 2, 0, 0)); + + // Every row is back with content, type, tags and scope intact. + let mut src_views = src + .service + .list(ListArgs::default()) + .await + .expect("source list"); + let mut dst_views = dst + .service + .list(ListArgs::default()) + .await + .expect("target list"); + src_views.sort_by_key(|v| v.created_at); + dst_views.sort_by_key(|v| v.created_at); + assert_eq!(dst_views.len(), 3); + for (s, d) in src_views.iter().zip(&dst_views) { + assert_eq!(d.content, s.content); + assert_eq!(d.mem_type, s.mem_type); + assert_eq!(d.tags, s.tags); + assert_eq!(d.scope, s.scope); + } + + // Provenance columns: preserved, except last_accessed, which is refreshed + // to the TARGET clock; rows land pending with no vector (D-10, D-13). + let target_ids: Vec = [a, b, c] + .iter() + .map(|id| id_by_content(&dst.db_path, &provenance_content(&src.db_path, *id))) + .collect(); + for (src_id, dst_id) in [a, b, c].iter().zip(&target_ids) { + let s = provenance(&src.db_path, *src_id); + let d = provenance(&dst.db_path, *dst_id); + assert_eq!(d.source, s.source, "source is preserved verbatim (D-07)"); + assert_eq!(d.created_at, s.created_at, "created_at preserved"); + assert_eq!(d.access_count, s.access_count, "access_count preserved"); + assert_eq!(d.expires_at, s.expires_at, "expires_at preserved"); + assert_eq!( + d.last_accessed, 5_000, + "last_accessed refreshed to import time" + ); + assert_eq!(status(&dst.db_path, *dst_id), 0, "lands pending"); + assert_eq!(vec_rows(&dst.db_path, *dst_id), 0, "no vector travelled"); + } + assert_eq!( + provenance(&dst.db_path, target_ids[0]).expires_at, + Some(1_000 + 1_000_000) + ); + assert_eq!(provenance(&dst.db_path, target_ids[1]).expires_at, None); + assert_eq!(provenance(&dst.db_path, target_ids[2]).expires_at, None); + + // Edges attach to the TARGET ids with their original times (D-05, D-08). + let (ta, tb, tc) = (target_ids[0], target_ids[1], target_ids[2]); + let mut expected = vec![ + (ta, tb, "supersedes".to_string(), supersedes_at), + (tb.min(tc), tb.max(tc), "relates_to".to_string(), relates_at), + ]; + expected.sort(); + assert_eq!(edges(&dst.db_path), expected); + assert_eq!(edges_touching(&dst.db_path, tb), 2); + assert_eq!(orphan_edges(&dst.db_path), 0); + + // A re-export of the target equals the source export modulo the fields + // that legitimately differ (ids, refreshed last_accessed, header clock). + let (again, _) = export_to_string(&dst, "target.jsonl").await; + assert_eq!(normalised(&again), normalised(&text)); +} + +/// Content of one source row, to find its restored twin in the target. +fn provenance_content(db_path: &Path, id: i64) -> String { + raw_conn(db_path) + .query_row("SELECT content FROM memories WHERE id = ?1", [id], |row| { + row.get(0) + }) + .expect("read content") +} + +#[tokio::test] +async fn re_importing_the_same_file_is_idempotent() { + let src = harness(); + seed_round_trip_source(&src).await; + let (text, _) = export_to_string(&src, "source.jsonl").await; + + let dst = harness(); + dst.clock.set(5_000); + let first = import(&dst, &text).await; + assert_eq!(first, six(3, 0, 0, 2, 0, 0)); + let rows_after_first = row_count(&dst.db_path); + let edges_after_first = edge_count(&dst.db_path); + assert_eq!((rows_after_first, edges_after_first), (3, 2)); + + dst.clock.advance(100); + let second = import(&dst, &text).await; + assert_eq!(second, six(0, 3, 0, 0, 2, 0)); + assert_eq!(row_count(&dst.db_path), rows_after_first); + assert_eq!(edge_count(&dst.db_path), edges_after_first); +} + +// --------------------------------------------------------------------------- +// Import half — Task 2: kill-tests and edge probes +// --------------------------------------------------------------------------- + +/// A valid v1 header declaring `rows` memories and `links` links, for +/// hand-built files. +fn header(rows: u64, links: u64) -> String { + format!( + r#"{{"record":"header","format":"agent-memory-export","version":1,"exported_at":1000,"embedding_model":"nomic-embed-text","embedding_dim":768,"schema_version":3,"agent_memory_version":"0.0.1","exported_rows":{rows},"exported_links":{links}}}"# + ) +} + +const MEM1: &str = r#"{"record":"memory","key":1,"type":"DECISION","content":"hand-built first","tags":[],"source":null,"scope":null,"created_at":1,"last_accessed":1,"access_count":0,"expires_at":null}"#; +const MEM2: &str = r#"{"record":"memory","key":2,"type":"PATTERN","content":"hand-built second","tags":["t"],"source":"s","scope":"sc","created_at":2,"last_accessed":2,"access_count":3,"expires_at":null}"#; +const LINK12: &str = r#"{"record":"link","from":1,"to":2,"kind":"relates_to","created_at":5}"#; + +/// Join lines into a newline-terminated file. +fn file_from(lines: &[&str]) -> String { + let mut text = lines.join("\n"); + text.push('\n'); + text +} + +#[tokio::test] +async fn importing_into_a_non_empty_db_with_overlapping_rowids_never_attaches_phantom_edges() { + // Source: three memories whose ids are exactly 1..=3 (a fresh database), + // and one edge 1 -> 2. + let src = harness(); + let a = store(&src, new_memory("source one", MemoryType::Decision)).await; + let b = store(&src, new_memory("source two", MemoryType::Pattern)).await; + let c = store(&src, new_memory("source three", MemoryType::Todo)).await; + assert_eq!((a, b, c), (1, 2, 3), "the file's keys are 1..=3"); + link(&src, a, b, LinkKind::RelatesTo).await; + let (text, _) = export_to_string(&src, "overlap.jsonl").await; + + // Target: three UNRELATED rows already hold ids 1..=3 — exactly the keys + // the file's link record names (PITFALLS 18). + let dst = harness(); + let pre: Vec = vec![ + store(&dst, new_memory("target x", MemoryType::Error)).await, + store(&dst, new_memory("target y", MemoryType::Architecture)).await, + store(&dst, new_memory("target z", MemoryType::Constraint)).await, + ]; + assert_eq!( + pre, + vec![1, 2, 3], + "the target's rowids overlap the file's keys" + ); + + let report = import(&dst, &text).await; + assert_eq!(report, six(3, 0, 0, 1, 0, 0)); + + for id in &pre { + assert_eq!( + edges_touching(&dst.db_path, *id), + 0, + "pre-existing row {id} gained a phantom edge" + ); + } + let new_a = id_by_content(&dst.db_path, "source one"); + let new_b = id_by_content(&dst.db_path, "source two"); + assert!(new_a > 3 && new_b > 3, "restored rows get fresh target ids"); + assert_eq!(edges_touching(&dst.db_path, new_a), 1); + assert_eq!(edges_touching(&dst.db_path, new_b), 1); + assert_eq!(orphan_edges(&dst.db_path), 0); + assert_eq!(row_count(&dst.db_path), 6); + assert_eq!(edge_count(&dst.db_path), 1); + + // Importing the same file again on top of the now-populated target maps + // every key onto the restored rows and changes nothing. + let again = import(&dst, &text).await; + assert_eq!(again, six(0, 3, 0, 0, 1, 0)); + assert_eq!(row_count(&dst.db_path), 6); + assert_eq!(edge_count(&dst.db_path), 1); + for id in &pre { + assert_eq!(edges_touching(&dst.db_path, *id), 0); + } +} + +#[tokio::test] +async fn link_endpoints_that_dedup_to_existing_rows_attach_to_those_rows() { + let shared = || NewMemory { + content: "shared decision already on the target".to_string(), + mem_type: MemoryType::Decision, + tags: vec!["any".to_string()], + source: Some("shared-src".to_string()), + scope: Some("other-scope".to_string()), + ttl_secs: None, + }; + + // Source: A (key 1) and B (key 2), A -> B. + let src = harness(); + let a = store(&src, shared()).await; + let b = store(&src, new_memory("only in the file", MemoryType::Pattern)).await; + link(&src, a, b, LinkKind::RelatesTo).await; + let (text, _) = export_to_string(&src, "dedup.jsonl").await; + + // Target: a filler row takes id 1, then a row identical to A on the + // (source, mem_type, content) key takes id 2 — so A's key (1) and the + // existing row's id (2) differ. + let dst = harness(); + let filler = store(&dst, new_memory("unrelated filler row", MemoryType::Error)).await; + let existing_a = store(&dst, shared()).await; + assert_eq!((filler, existing_a), (1, 2)); + assert_ne!(existing_a, a, "the existing row's id differs from A's key"); + + let report = import(&dst, &text).await; + assert_eq!(report.imported, 1, "only B is new"); + assert_eq!(report.skipped_duplicates, 1, "A dedups to the existing row"); + assert_eq!(report.links_imported, 1); + assert_eq!(report, six(1, 1, 0, 1, 0, 0)); + + assert_eq!( + edges_touching(&dst.db_path, existing_a), + 1, + "the edge attaches to the EXISTING row A dedups to" + ); + assert_eq!( + edges_touching(&dst.db_path, filler), + 0, + "never to the row at A's key" + ); + assert_eq!( + row_count(&dst.db_path), + 3, + "grew by exactly one (no second A)" + ); + assert_eq!( + id_by_content(&dst.db_path, "shared decision already on the target"), + existing_a + ); +} + +#[tokio::test] +async fn identical_rows_collapse_to_one_on_import_and_count_a_duplicate() { + let twin = || new_memory("the same row stored twice", MemoryType::Architecture); + let src = harness(); + let first = store(&src, twin()).await; + let second = store(&src, twin()).await; + assert_ne!(first, second, "no uniqueness constraint: two rows exist"); + + let (text, report) = export_to_string(&src, "twins.jsonl").await; + assert_eq!(report.memories, 2); + match &lines(&text)[0] { + Record::Header { exported_rows, .. } => assert_eq!(*exported_rows, 2), + other => panic!("expected header, got {other:?}"), + } + assert_eq!(text.lines().count(), 3, "header + two memory records"); + + let dst = harness(); + assert_eq!(import(&dst, &text).await, six(1, 1, 0, 0, 0, 0)); + assert_eq!(row_count(&dst.db_path), 1, "identical rows collapse (D-07)"); +} + +#[tokio::test] +async fn a_link_between_two_rows_that_collapse_on_import_is_dropped_not_miscounted() { + // 05-REVIEW WR-01: the parser only rejects self-links on FILE keys. Two + // distinct keys that dedup to the same target id would reach the store as + // `link(N, N)`, which `INSERT OR IGNORE` swallows under the 0003 CHECK as + // a phantom "duplicate". The importer must count it as dropped instead. + let twin = || new_memory("the same row stored twice and linked", MemoryType::Decision); + let src = harness(); + let first = store(&src, twin()).await; + let second = store(&src, twin()).await; + link(&src, first, second, LinkKind::RelatesTo).await; + + let (text, report) = export_to_string(&src, "linked-twins.jsonl").await; + assert_eq!((report.memories, report.links), (2, 1)); + assert_eq!(text.lines().count(), 4, "header + two memories + one link"); + + let dst = harness(); + assert_eq!( + import(&dst, &text).await, + six(1, 1, 0, 0, 0, 1), + "the collapsed edge is dropped, never a 'skipped duplicate'" + ); + assert_eq!(row_count(&dst.db_path), 1); + assert_eq!( + edge_count(&dst.db_path), + 0, + "no self-link ever reaches the store" + ); + + // Re-importing is still idempotent and still reports the drop. + assert_eq!(import(&dst, &text).await, six(0, 2, 0, 0, 0, 1)); + assert_eq!(edge_count(&dst.db_path), 0); +} + +#[tokio::test] +async fn an_access_count_at_i64_max_imports_survives_a_bump_and_still_exports() { + // 05-REVIEW WR-02: access_count is carried verbatim (D-10) and is the one + // preserved column later used in arithmetic. Before the saturating bump, + // one `update` of this row overflowed the column to REAL and every later + // `export` of the WHOLE database failed with the redacted storage error. + let maxed = MEM1.replace("\"access_count\":0", "\"access_count\":9223372036854775807"); + let text = file_from(&[&header(1, 0), &maxed]); + + let dst = harness(); + assert_eq!(import(&dst, &text).await, six(1, 0, 0, 0, 0, 0)); + let id = id_by_content(&dst.db_path, "hand-built first"); + assert_eq!(provenance(&dst.db_path, id).access_count, i64::MAX); + + // The freshness bump that rides on `update` (D-09) must saturate. + dst.clock.set(5_000); + let updated = dst + .service + .update( + id, + MemoryPatch { + tags: Some(Some(vec!["touched".to_string()])), + ..MemoryPatch::default() + }, + ) + .await + .expect("update succeeds") + .expect("the row exists"); + assert_eq!(updated.view.id, id); + let after = provenance(&dst.db_path, id); + assert_eq!(after.access_count, i64::MAX, "saturated, not wrapped"); + assert_eq!(after.last_accessed, 5_000, "recency still refreshed"); + + // The backup path still works, and the record carries the exact value. + let (exported, report) = export_to_string(&dst, "maxed.jsonl").await; + assert_eq!(report.memories, 1); + match &lines(&exported)[1] { + Record::Memory { access_count, .. } => assert_eq!(*access_count, i64::MAX), + other => panic!("expected a memory record, got {other:?}"), + } +} + +#[tokio::test] +async fn an_expired_but_unswept_target_row_is_not_a_dedup_hit_so_imported_edges_survive_the_sweep() +{ + // 05-REVIEW WR-03: a CLI import runs no sweep. Before the fix the probe + // mapped MEM1's key onto the stale row, counted a duplicate, hung the + // imported edge on it, and the next sweep cascaded that edge away — the + // report said "1 link imported", the corpus ended with none. + let dst = harness(); + // Same (source = NULL, DECISION, content) key as MEM1, TTL 1 s at 1_000. + let stale = store( + &dst, + NewMemory { + ttl_secs: Some(1), + ..new_memory("hand-built first", MemoryType::Decision) + }, + ) + .await; + assert_eq!(provenance(&dst.db_path, stale).expires_at, Some(1_001)); + + // Well past expiry, and NO sweep has run: the row is still in the table. + dst.clock.set(2_000); + assert_eq!(row_count(&dst.db_path), 1); + + // MEM1 carries expires_at: null and a link to MEM2. + let text = file_from(&[&header(2, 1), MEM1, MEM2, LINK12]); + let report = import(&dst, &text).await; + assert_eq!( + report, + six(2, 0, 0, 1, 0, 0), + "the file's live copy is imported, not deduped onto the stale row" + ); + assert_eq!(row_count(&dst.db_path), 3, "stale + two imported"); + assert_eq!( + edges_touching(&dst.db_path, stale), + 0, + "no imported edge is hung on the expired row" + ); + + // The sweep deletes only the stale row; the import survives intact. + let swept = dst.service.sweep(dst.clock.now()).await.expect("sweep"); + assert_eq!(swept.expired, 1); + assert_eq!(row_count(&dst.db_path), 2); + let restored = id_by_content(&dst.db_path, "hand-built first"); + assert_ne!(restored, stale); + assert_eq!(provenance(&dst.db_path, restored).expires_at, None); + assert_eq!(edges_touching(&dst.db_path, restored), 1); + assert_eq!(edge_count(&dst.db_path), 1); + assert_eq!(orphan_edges(&dst.db_path), 0); +} + +#[tokio::test] +async fn expired_rows_are_skipped_and_counted_and_their_links_dropped() { + // Source at 1_000: A expires at 1_050, B at 11_000, C never. + let src = harness(); + let a = store( + &src, + NewMemory { + ttl_secs: Some(50), + ..new_memory("alpha expires soon", MemoryType::Todo) + }, + ) + .await; + let b = store( + &src, + NewMemory { + ttl_secs: Some(10_000), + ..new_memory("beta lives long", MemoryType::Decision) + }, + ) + .await; + let c = store(&src, new_memory("gamma is permanent", MemoryType::Pattern)).await; + link(&src, a, c, LinkKind::CausedBy).await; + link(&src, b, c, LinkKind::Supersedes).await; + let (text, _) = export_to_string(&src, "expiry.jsonl").await; + + // Target clock EXACTLY at A's expiry: `expires_at <= now` skips (D-10), + // and the edge that needed A is dropped and counted (PITFALLS 19). + let dst = harness(); + dst.clock.set(1_050); + let report = import(&dst, &text).await; + assert_eq!(report, six(2, 0, 1, 1, 0, 1)); + let contents: Vec = dst + .service + .list(ListArgs::default()) + .await + .expect("list") + .into_iter() + .map(|v| v.content) + .collect(); + assert_eq!(contents.len(), 2); + assert!(contents.iter().any(|c| c == "beta lives long")); + assert!(contents.iter().any(|c| c == "gamma is permanent")); + assert!( + !contents.iter().any(|c| c == "alpha expires soon"), + "the expired row was never inserted" + ); + assert_eq!(row_count(&dst.db_path), 2); + assert_eq!(edge_count(&dst.db_path), 1); + assert_eq!(orphan_edges(&dst.db_path), 0); + + // One second earlier the same file restores everything: the boundary is + // `<=`, not `<`. + let earlier = harness(); + earlier.clock.set(1_049); + assert_eq!(import(&earlier, &text).await, six(3, 0, 0, 2, 0, 0)); + + // An ancient row — last touched 400 days before its export — is imported + // decay-fresh: last_accessed is refreshed to the target clock and the row + // is listed (decay never deletes; PITFALLS 19). + let old = harness(); + let ancient = store( + &old, + new_memory("ancient wisdom about caches", MemoryType::Pattern), + ) + .await; + old.clock.advance(400 * 86_400); + let (old_text, _) = export_to_string(&old, "ancient.jsonl").await; + assert_eq!(provenance(&old.db_path, ancient).last_accessed, 1_000); + + let fresh = harness(); + fresh.clock.set(7_777); + // Note: the header's exported_at (400 days ahead of 7_777) also exercises + // the clock-skew warning path — the import must continue regardless. + assert_eq!(import(&fresh, &old_text).await, six(1, 0, 0, 0, 0, 0)); + let restored = id_by_content(&fresh.db_path, "ancient wisdom about caches"); + assert_eq!(provenance(&fresh.db_path, restored).last_accessed, 7_777); + assert_eq!( + provenance(&fresh.db_path, restored).created_at, + 1_000, + "created_at preserved" + ); + let listed = fresh.service.list(ListArgs::default()).await.expect("list"); + assert_eq!(listed.len(), 1); + assert_eq!(listed[0].content, "ancient wisdom about caches"); +} + +#[tokio::test] +async fn imported_rows_land_pending_and_the_sweep_backfills_them() { + let src = harness(); + store( + &src, + new_memory("orchid pollination schedule", MemoryType::Pattern), + ) + .await; + store( + &src, + new_memory("granite countertop sealing", MemoryType::Todo), + ) + .await; + let (text, _) = export_to_string(&src, "pending.jsonl").await; + + // The target's embedder is UP — and the importer still never calls it. + let dst = harness_with(Arc::new(FakeEmbedder::with_vectors(HashMap::new()))); + assert_eq!(import(&dst, &text).await, six(2, 0, 0, 0, 0, 0)); + let orchid = id_by_content(&dst.db_path, "orchid pollination schedule"); + let granite = id_by_content(&dst.db_path, "granite countertop sealing"); + for id in [orchid, granite] { + assert_eq!( + status(&dst.db_path, id), + 0, + "imported rows land pending (D-13)" + ); + assert_eq!( + vec_rows(&dst.db_path, id), + 0, + "no vector was written inline" + ); + } + + // Exactly one re-embed mechanism: the sweep backfill. + dst.service + .sweep(dst.clock.now()) + .await + .expect("sweep succeeds"); + for id in [orchid, granite] { + assert_eq!(status(&dst.db_path, id), 1, "the sweep embedded the row"); + assert_eq!(vec_rows(&dst.db_path, id), 1); + } + + let outcome = dst + .service + .search(SearchArgs { + query: "orchid".to_string(), + ..SearchArgs::default() + }) + .await + .expect("search"); + assert_eq!(outcome.search_mode, SearchMode::Hybrid); + assert!( + outcome.results.iter().any(|v| v.id == orchid), + "the backfilled row is hybrid-searchable" + ); +} + +#[tokio::test] +async fn a_header_only_file_imports_nothing_and_reports_zeros() { + let src = harness(); + let (text, _) = export_to_string(&src, "empty.jsonl").await; + assert_eq!(text.lines().count(), 1); + + let dst = harness(); + store(&dst, new_memory("already here", MemoryType::Decision)).await; + let before = row_count(&dst.db_path); + assert_eq!(import(&dst, &text).await, ImportReport::default()); + assert_eq!(import(&dst, &text).await, six(0, 0, 0, 0, 0, 0)); + assert_eq!(row_count(&dst.db_path), before); + assert_eq!(edge_count(&dst.db_path), 0); +} + +#[test] +fn a_zero_byte_file_is_rejected_at_line_one() { + for text in ["", "\n"] { + let err = parse_jsonl(text).expect_err("no header"); + assert!(matches!(err, MemoryError::InvalidArgument(_)), "{err:?}"); + assert!( + err.to_string().starts_with("invalid argument: line 1:"), + "{err}" + ); + } +} + +#[tokio::test] +async fn links_are_applied_after_all_memories_regardless_of_file_order() { + let link_first = file_from(&[&header(2, 1), LINK12, MEM1, MEM2]); + let canonical = file_from(&[&header(2, 1), MEM1, MEM2, LINK12]); + + let shuffled = harness(); + let report = import(&shuffled, &link_first).await; + assert_eq!(report, six(2, 0, 0, 1, 0, 0)); + let first = id_by_content(&shuffled.db_path, "hand-built first"); + let second = id_by_content(&shuffled.db_path, "hand-built second"); + assert_eq!( + edges(&shuffled.db_path), + vec![( + first.min(second), + first.max(second), + "relates_to".to_string(), + 5 + )], + "the edge carries the record's own created_at" + ); + + let ordered = harness(); + assert_eq!(import(&ordered, &canonical).await, report); + assert_eq!(row_count(&ordered.db_path), row_count(&shuffled.db_path)); + assert_eq!(edge_count(&ordered.db_path), edge_count(&shuffled.db_path)); +} + +#[tokio::test] +async fn nothing_is_written_when_validation_fails() { + let dst = harness(); + store(&dst, new_memory("pre-existing row", MemoryType::Decision)).await; + let before = row_count(&dst.db_path); + + // Line 2 is a perfectly valid memory; line 3 is not. The whole file is + // refused, so `import_jsonl` is never reached — the only entry point to + // the importer's writes takes a fully parsed JsonlFile. + let text = file_from(&[&header(2, 0), MEM1, r#"{"record":"memo"}"#]); + let err = parse_jsonl(&text).expect_err("line 3 is malformed"); + assert!(matches!(err, MemoryError::InvalidArgument(_)), "{err:?}"); + assert!(err.to_string().contains("line 3:"), "{err}"); + + assert_eq!(row_count(&dst.db_path), before, "nothing was written"); + assert_eq!(edge_count(&dst.db_path), 0); + assert!( + dst.service + .list(ListArgs::default()) + .await + .expect("list") + .iter() + .all(|v| v.content == "pre-existing row"), + "the valid line 2 record was NOT imported" + ); +} diff --git a/crates/agent-memory-core/tests/import.rs b/crates/agent-memory-core/tests/import.rs index 022eedf..40a2d60 100644 --- a/crates/agent-memory-core/tests/import.rs +++ b/crates/agent-memory-core/tests/import.rs @@ -46,6 +46,12 @@ async fn import_loads_typed_searchable_memories_idempotently() { let first = service.import(parsed.drafts).await.expect("first import"); assert_eq!(first.imported, 9); assert_eq!(first.skipped_duplicates, 0); + // The gsd-state path has no expiry and no links: the four jsonl-only + // counts are always zero here (Phase 5 D-12). + assert_eq!(first.skipped_expired, 0); + assert_eq!(first.links_imported, 0); + assert_eq!(first.links_skipped_duplicates, 0); + assert_eq!(first.links_dropped, 0); // Per-type counts through the normal list path. let count = |mem_type: Option, tag: Option<&str>| { diff --git a/crates/agent-memory-core/tests/redaction.rs b/crates/agent-memory-core/tests/redaction.rs new file mode 100644 index 0000000..bb7e871 --- /dev/null +++ b/crates/agent-memory-core/tests/redaction.rs @@ -0,0 +1,273 @@ +//! D-14 proof: a storage fault on EVERY non-search `MemoryService` path is +//! redacted at the service seam — the T-04-19 pattern generalised from +//! `search` to `store`, `list`, `import`, `forget`, `link`, `unlink`, `update`, +//! `sweep`, and the Phase 5 `export` / `import_jsonl` paths (04-REVIEW IN-01 +//! closed; 05-REVIEW IN-02). +//! +//! Each test takes the table the operation writes or reads offline (an +//! `ALTER TABLE … RENAME` on a raw side connection, exactly the hybrid.rs +//! recipe), calls the operation, and asserts the caller sees ONLY +//! `MemoryError::Internal(" failed due to a storage error (details are +//! in the server log)")` — never the engine's `sqlite error: …` text with its +//! table and column names. The detail goes to the `tracing::error!` log. +//! +//! Harness: a REAL on-disk temp DB, an injected `TestClock`, and a failing +//! `FakeEmbedder` (stores land pending; nothing here depends on Ollama). + +use std::path::{Path, PathBuf}; +use std::sync::Arc; + +use agent_memory_core::clock::{Clock, TestClock}; +use agent_memory_core::decay::DecayConfig; +use agent_memory_core::domain::{LinkKind, MemoryError, MemoryPatch, MemoryType, NewMemory}; +use agent_memory_core::embed::{Embedder, FakeEmbedder}; +use agent_memory_core::import::jsonl::parse_jsonl; +use agent_memory_core::service::{ListArgs, MemoryService}; +use agent_memory_core::store::sqlite::{register_vec_extension, SqliteStore}; + +struct Harness { + service: MemoryService, + clock: Arc, + db_path: PathBuf, + _dir: tempfile::TempDir, +} + +fn harness() -> Harness { + harness_with(Arc::new(FakeEmbedder::failing())) +} + +fn harness_with(embedder: Arc) -> Harness { + let dir = tempfile::tempdir().expect("tempdir"); + let db_path = dir.path().join("memory.db"); + let store = SqliteStore::open(&db_path).expect("open store"); + let clock = Arc::new(TestClock::new(1_000)); + let service = MemoryService::new( + Arc::new(store), + clock.clone() as Arc, + embedder, + DecayConfig::default(), + ); + Harness { + service, + clock, + db_path, + _dir: dir, + } +} + +fn new_memory(content: &str) -> NewMemory { + NewMemory { + content: content.to_string(), + mem_type: MemoryType::Decision, + tags: vec!["redaction".to_string()], + source: Some("redaction-test".to_string()), + scope: None, + ttl_secs: None, + } +} + +async fn store(h: &Harness, content: &str) -> i64 { + h.service + .store(new_memory(content)) + .await + .expect("store succeeds while the table is online") +} + +fn raw_conn(db_path: &Path) -> rusqlite::Connection { + register_vec_extension().expect("vec0 registers"); + rusqlite::Connection::open(db_path).expect("open raw connection") +} + +/// Take `table` offline underneath the live store by renaming it away. The +/// store's next statement against it fails at prepare time with the engine's +/// `no such table` error — the storage-tier fault the seam must redact. +fn take_offline(db_path: &Path, table: &str) { + raw_conn(db_path) + .execute_batch(&format!("ALTER TABLE {table} RENAME TO {table}_offline;")) + .expect("rename table offline"); +} + +/// Both D-14 assertions: the exact redacted template for `op`, and no +/// storage-engine text anywhere in the rendered error. +fn assert_redacted(err: &MemoryError, op: &str) { + let expected = format!("{op} failed due to a storage error (details are in the server log)"); + assert!( + matches!(err, MemoryError::Internal(ref m) if m == &expected), + "{op}: expected Internal({expected:?}), got {err:?}" + ); + assert!( + !err.to_string().to_lowercase().contains("sqlite"), + "{op}: no SQLite text may reach the client: {err}" + ); +} + +#[tokio::test] +async fn store_storage_failure_is_redacted() { + let h = harness(); + take_offline(&h.db_path, "memories"); + + let err = h + .service + .store(new_memory("will not land")) + .await + .expect_err("store must fail with the table offline"); + assert_redacted(&err, "store"); +} + +#[tokio::test] +async fn list_storage_failure_is_redacted() { + let h = harness(); + store(&h, "listed memory").await; + take_offline(&h.db_path, "memories"); + + let err = h + .service + .list(ListArgs::default()) + .await + .expect_err("list must fail with the table offline"); + assert_redacted(&err, "list"); +} + +#[tokio::test] +async fn import_storage_failure_is_redacted() { + let h = harness(); + take_offline(&h.db_path, "memories"); + + let err = h + .service + .import(vec![new_memory("imported draft")]) + .await + .expect_err("import must fail with the table offline"); + assert_redacted(&err, "import"); +} + +#[tokio::test] +async fn forget_storage_failure_is_redacted() { + let h = harness(); + let id = store(&h, "to be forgotten").await; + take_offline(&h.db_path, "memories"); + + let err = h + .service + .forget(id) + .await + .expect_err("forget must fail with the table offline"); + assert_redacted(&err, "forget"); +} + +#[tokio::test] +async fn link_storage_failure_is_redacted() { + let h = harness(); + let a = store(&h, "link endpoint a").await; + let b = store(&h, "link endpoint b").await; + // The endpoint pre-check reads `memories` and must still pass; it is the + // edge INSERT into `memory_links` that fails. + take_offline(&h.db_path, "memory_links"); + + let err = h + .service + .link(a, b, LinkKind::RelatesTo) + .await + .expect_err("link must fail with the edge table offline"); + assert_redacted(&err, "link"); +} + +#[tokio::test] +async fn unlink_storage_failure_is_redacted() { + let h = harness(); + let a = store(&h, "unlink endpoint a").await; + let b = store(&h, "unlink endpoint b").await; + take_offline(&h.db_path, "memory_links"); + + let err = h + .service + .unlink(a, b, LinkKind::RelatesTo) + .await + .expect_err("unlink must fail with the edge table offline"); + assert_redacted(&err, "unlink"); +} + +#[tokio::test] +async fn update_storage_failure_is_redacted() { + let h = harness(); + let id = store(&h, "original wording").await; + take_offline(&h.db_path, "memories"); + + let patch = MemoryPatch { + content: Some(Some("new wording".to_string())), + ..MemoryPatch::default() + }; + let err = h + .service + .update(id, patch) + .await + .expect_err("update must fail with the table offline"); + assert_redacted(&err, "update"); +} + +#[tokio::test] +async fn sweep_storage_failure_is_redacted() { + let h = harness(); + store(&h, "swept memory").await; + take_offline(&h.db_path, "memories"); + + let err = h + .service + .sweep(h.clock.now()) + .await + .expect_err("sweep must fail with the table offline"); + assert_redacted(&err, "sweep"); +} + +// --------------------------------------------------------------------------- +// Phase 5 paths (05-REVIEW IN-02): `export` and `import_jsonl` get the same +// take_offline proof as the eight paths above, so a refactor that unwraps +// either one turns red here instead of leaking engine text to the CLI. +// --------------------------------------------------------------------------- + +/// A valid v1 header declaring `rows` memories and `links` links. +fn jsonl_header(rows: u64, links: u64) -> String { + format!( + r#"{{"record":"header","format":"agent-memory-export","version":1,"exported_at":1000,"embedding_model":"nomic-embed-text","embedding_dim":768,"schema_version":3,"agent_memory_version":"0.0.1","exported_rows":{rows},"exported_links":{links}}}"# + ) +} + +const JSONL_MEM1: &str = r#"{"record":"memory","key":1,"type":"DECISION","content":"redaction first","tags":[],"source":null,"scope":null,"created_at":1,"last_accessed":1,"access_count":0,"expires_at":null}"#; +const JSONL_MEM2: &str = r#"{"record":"memory","key":2,"type":"PATTERN","content":"redaction second","tags":[],"source":null,"scope":null,"created_at":2,"last_accessed":2,"access_count":0,"expires_at":null}"#; +const JSONL_LINK12: &str = + r#"{"record":"link","from":1,"to":2,"kind":"relates_to","created_at":5}"#; + +#[tokio::test] +async fn export_storage_failure_is_redacted() { + let h = harness(); + store(&h, "exported memory").await; + take_offline(&h.db_path, "memories"); + + let sink: Box = Box::new(Vec::new()); + let err = h + .service + .export(sink, "test-0.0.0".to_string()) + .await + .expect_err("export must fail with the table offline"); + assert_redacted(&err, "export"); +} + +#[tokio::test] +async fn import_jsonl_storage_failure_is_redacted() { + let h = harness(); + // `memories` stays online so pass 2 inserts both rows; pass 3's + // `store.link` is the first statement to touch the renamed table. + take_offline(&h.db_path, "memory_links"); + + let text = format!( + "{}\n{JSONL_MEM1}\n{JSONL_MEM2}\n{JSONL_LINK12}\n", + jsonl_header(2, 1) + ); + let file = parse_jsonl(&text).expect("the file itself is valid"); + let err = h + .service + .import_jsonl(file) + .await + .expect_err("import_jsonl must fail with the links table offline"); + assert_redacted(&err, "import"); +} diff --git a/crates/agent-memory-core/tests/relations.rs b/crates/agent-memory-core/tests/relations.rs index 4e3a802..7aa582d 100644 --- a/crates/agent-memory-core/tests/relations.rs +++ b/crates/agent-memory-core/tests/relations.rs @@ -136,17 +136,18 @@ async fn link_succeeds_and_a_duplicate_link_is_an_idempotent_no_op() { .link(a, b, LinkKind::RelatesTo) .await .expect("link succeeds"); - assert_eq!(first, LinkOutcome::Linked); + assert_eq!(first, LinkOutcome::Linked { created: true }); assert_eq!(edge_count(&h.db_path), 1); // D-02: the identical (from, to, kind) again is a success, not an error, - // and still exactly one row. + // and still exactly one row — reported as `created: false` so an importer + // can count the duplicate (Phase 5 D-12). let again = h .service .link(a, b, LinkKind::RelatesTo) .await .expect("duplicate link is not an error"); - assert_eq!(again, LinkOutcome::Linked); + assert_eq!(again, LinkOutcome::Linked { created: false }); assert_eq!(edge_count(&h.db_path), 1, "duplicate link adds no row"); // A different kind between the same pair is a distinct edge. @@ -788,11 +789,11 @@ async fn relates_to_linked_both_ways_is_one_relation_listed_once_from_each_side( .link(b, a, LinkKind::RelatesTo) .await .expect("b -> a"); - assert_eq!(forward, LinkOutcome::Linked); + assert_eq!(forward, LinkOutcome::Linked { created: true }); assert_eq!( backward, - LinkOutcome::Linked, - "the reverse is the same relation: a D-02 no-op success" + LinkOutcome::Linked { created: false }, + "the reverse is the same relation: a D-02 no-op success that inserted nothing" ); assert_eq!(edge_count(&h.db_path), 1, "one symmetric relation, one row"); diff --git a/crates/agent-memory-core/tests/store.rs b/crates/agent-memory-core/tests/store.rs index f290bc3..a4d98b9 100644 --- a/crates/agent-memory-core/tests/store.rs +++ b/crates/agent-memory-core/tests/store.rs @@ -12,10 +12,11 @@ use std::sync::Arc; use agent_memory_core::clock::{Clock, TestClock}; use agent_memory_core::decay::DecayConfig; -use agent_memory_core::domain::{MemoryType, NewMemory}; +use agent_memory_core::domain::{ImportedMemory, MemoryType, NewMemory}; use agent_memory_core::embed::FakeEmbedder; -use agent_memory_core::service::{ListArgs, MemoryService}; -use agent_memory_core::store::sqlite::SqliteStore; +use agent_memory_core::service::{ListArgs, MemoryService, SearchArgs}; +use agent_memory_core::store::sqlite::{register_vec_extension, SqliteStore}; +use agent_memory_core::store::Store; const ALL_TYPES: [&str; 6] = [ "DECISION", @@ -229,3 +230,323 @@ async fn limit_caps_count_and_order_is_newest_first() { assert_eq!(limited[0].content, "row 4"); assert_eq!(limited[1].content, "row 3"); } + +// --------------------------------------------------------------------------- +// Phase 5 store contract (D-07): the dedup probe returns the matching row id +// --------------------------------------------------------------------------- + +#[test] +fn find_by_key_returns_the_matching_id_null_safely() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = SqliteStore::open(&dir.path().join("memory.db")).expect("open store"); + + // Row A has no source (stored NULL); row B has source "s". + let id_a = store + .insert(new_memory(MemoryType::Decision, "alpha"), None, 1_000) + .expect("insert a"); + let mut b = new_memory(MemoryType::Decision, "beta"); + b.source = Some("s".to_string()); + let id_b = store.insert(b, None, 1_000).expect("insert b"); + + assert_eq!( + store + .find_by_key(None, MemoryType::Decision, "alpha", 1_000) + .expect("probe a"), + Some(id_a), + "a None source matches the stored NULL (CR-01)" + ); + assert_eq!( + store + .find_by_key(Some("s"), MemoryType::Decision, "beta", 1_000) + .expect("probe b"), + Some(id_b) + ); + assert_eq!( + store + .find_by_key(Some(""), MemoryType::Decision, "alpha", 1_000) + .expect("probe empty source"), + None, + "NULL is not the empty string (CR-01)" + ); + assert_eq!( + store + .find_by_key(None, MemoryType::Pattern, "alpha", 1_000) + .expect("probe other type"), + None, + "mem_type is part of the key" + ); + assert_eq!( + store + .find_by_key(None, MemoryType::Decision, "gamma", 1_000) + .expect("probe other content"), + None, + "content is part of the key" + ); + + // 05-REVIEW WR-03: only LIVE rows match. A row whose TTL lapsed with no + // sweep run is still in the table but must be invisible to the probe, on + // the importer's own `expires_at <= now` boundary. + let mut stale = new_memory(MemoryType::Decision, "delta"); + stale.ttl_secs = Some(500); // expires_at = 1_500 + let id_stale = store.insert(stale, None, 1_000).expect("insert stale"); + assert_eq!( + store + .find_by_key(None, MemoryType::Decision, "delta", 1_499) + .expect("probe before expiry"), + Some(id_stale), + "live until expires_at" + ); + assert_eq!( + store + .find_by_key(None, MemoryType::Decision, "delta", 1_500) + .expect("probe at expiry"), + None, + "expires_at == now is already expired (the importer's `<=`)" + ); + assert_eq!( + store + .find_by_key(None, MemoryType::Decision, "delta", 9_000) + .expect("probe long after expiry, unswept"), + None, + "an expired-but-unswept row is never a dedup hit" + ); + + // 05-REVIEW IN-01: with duplicate live rows the LOWEST id wins, by + // contract, not by scan order. + let id_a2 = store + .insert(new_memory(MemoryType::Decision, "alpha"), None, 1_000) + .expect("insert second alpha"); + assert!(id_a2 > id_a); + assert_eq!( + store + .find_by_key(None, MemoryType::Decision, "alpha", 1_000) + .expect("probe duplicated key"), + Some(id_a), + "the lowest matching live id is returned" + ); +} + +// --------------------------------------------------------------------------- +// Phase 5 store contract (D-10 / D-13): a restored row keeps its provenance, +// starts decay-fresh, and always lands pending for the sweep backfill +// --------------------------------------------------------------------------- + +/// Every stored column of one `memories` row, read through a raw connection +/// so the assertions see the table itself, not a view of it. +#[derive(Debug)] +struct RawRow { + mem_type: String, + content: String, + tags: String, + source: Option, + scope: Option, + base_weight: f64, + decay_score: f64, + access_count: i64, + created_at: i64, + last_accessed: i64, + expires_at: Option, + embedding_status: i64, +} + +fn raw_row(conn: &rusqlite::Connection, id: i64) -> RawRow { + conn.query_row( + "SELECT mem_type, content, tags, source, scope, base_weight, decay_score, \ + access_count, created_at, last_accessed, expires_at, embedding_status \ + FROM memories WHERE id = ?1", + [id], + |r| { + Ok(RawRow { + mem_type: r.get(0)?, + content: r.get(1)?, + tags: r.get(2)?, + source: r.get(3)?, + scope: r.get(4)?, + base_weight: r.get(5)?, + decay_score: r.get(6)?, + access_count: r.get(7)?, + created_at: r.get(8)?, + last_accessed: r.get(9)?, + expires_at: r.get(10)?, + embedding_status: r.get(11)?, + }) + }, + ) + .expect("read raw row") +} + +fn vec_rows(conn: &rusqlite::Connection, id: i64) -> i64 { + conn.query_row( + "SELECT count(*) FROM vec_memories WHERE memory_id = ?1", + [id], + |r| r.get(0), + ) + .expect("count vector rows") +} + +#[tokio::test] +async fn insert_imported_preserves_provenance_and_lands_pending() { + let dir = tempfile::tempdir().expect("tempdir"); + let db_path = dir.path().join("memory.db"); + let store = Arc::new(SqliteStore::open(&db_path).expect("open store")); + let now = 1_000; + + let id = store + .insert_imported( + ImportedMemory { + content: "restored decision".to_string(), + mem_type: MemoryType::Decision, + tags: vec!["a".to_string(), "b".to_string()], + source: Some("gsd-state".to_string()), + scope: Some("proj".to_string()), + created_at: 500, + access_count: 7, + expires_at: Some(9_000), + }, + now, + ) + .expect("insert imported row"); + + register_vec_extension().expect("vec0 registers"); + let conn = rusqlite::Connection::open(&db_path).expect("open raw connection"); + let row = raw_row(&conn, id); + assert_eq!(row.created_at, 500, "created_at is preserved (D-10)"); + assert_eq!( + row.last_accessed, now, + "last_accessed is refreshed to now (D-10)" + ); + assert_eq!(row.access_count, 7, "access_count is preserved (D-10)"); + assert_eq!( + row.expires_at, + Some(9_000), + "expires_at is preserved (D-10)" + ); + assert_eq!( + row.embedding_status, 0, + "always pending for the sweep (D-13)" + ); + assert_eq!(row.decay_score, 1.0); + assert_eq!(row.base_weight, 2.0, "DECISION is a pinned type"); + assert_eq!(row.tags, "[\"a\",\"b\"]"); + assert_eq!(row.source.as_deref(), Some("gsd-state")); + assert_eq!(row.scope.as_deref(), Some("proj")); + assert_eq!(row.mem_type, "DECISION"); + assert_eq!(row.content, "restored decision"); + assert_eq!( + vec_rows(&conn, id), + 0, + "no vector row is ever written (D-13)" + ); + + // An unpinned type with every optional column absent. + let id2 = store + .insert_imported( + ImportedMemory { + content: "restored pattern".to_string(), + mem_type: MemoryType::Pattern, + tags: vec![], + source: None, + scope: None, + created_at: 600, + access_count: 0, + expires_at: None, + }, + now, + ) + .expect("insert second imported row"); + let row2 = raw_row(&conn, id2); + assert_eq!(row2.base_weight, 1.0, "PATTERN is not pinned"); + assert_eq!(row2.source, None); + assert_eq!(row2.scope, None); + assert_eq!(row2.expires_at, None); + assert_eq!(row2.embedding_status, 0); + + // Visible through the normal read path: the FTS trigger fired on the + // shared INSERT, so a keyword search finds the restored row at once. + let clock = Arc::new(TestClock::new(now)); + let service = MemoryService::new( + store, + clock, + Arc::new(FakeEmbedder::failing()), + DecayConfig::default(), + ); + let listed = service.list(ListArgs::default()).await.expect("list"); + assert!( + listed + .iter() + .any(|v| v.id == id && v.content == "restored decision"), + "the restored row is listed: {listed:?}" + ); + let found = service + .search(SearchArgs { + query: "restored".to_string(), + ..SearchArgs::default() + }) + .await + .expect("keyword search"); + assert!( + found.results.iter().any(|v| v.id == id), + "the restored row is keyword-searchable: {:?}", + found.results + ); +} + +#[test] +fn bump_access_saturates_at_i64_max_and_keeps_the_column_an_integer() { + // 05-REVIEW WR-02: an imported row carries its access_count verbatim + // (D-10), so i64::MAX is a legal stored value. SQLite's `+` would overflow + // it to REAL (no error), and every later export_rows read of the column + // would then fail with InvalidColumnType. The bump must saturate instead. + let dir = tempfile::tempdir().expect("tempdir"); + let db_path = dir.path().join("memory.db"); + let store = SqliteStore::open(&db_path).expect("open store"); + + let maxed = store + .insert_imported( + ImportedMemory { + content: "counted to the brim".to_string(), + mem_type: MemoryType::Pattern, + tags: vec![], + source: None, + scope: None, + created_at: 500, + access_count: i64::MAX, + expires_at: None, + }, + 1_000, + ) + .expect("insert maxed row"); + let ordinary = store + .insert( + new_memory(MemoryType::Pattern, "counted normally"), + None, + 1_000, + ) + .expect("insert ordinary row"); + + store + .bump_access(&[maxed, ordinary], 2_000) + .expect("bump both rows"); + + register_vec_extension().expect("vec0 registers"); + let conn = rusqlite::Connection::open(&db_path).expect("open raw connection"); + let (maxed_count, maxed_type): (i64, String) = conn + .query_row( + "SELECT access_count, typeof(access_count) FROM memories WHERE id = ?1", + [maxed], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .expect("read maxed row"); + assert_eq!(maxed_count, i64::MAX, "the bump saturates, never wraps"); + assert_eq!(maxed_type, "integer", "the column never degrades to REAL"); + assert_eq!( + raw_row(&conn, maxed).last_accessed, + 2_000, + "recency still bumps" + ); + assert_eq!( + raw_row(&conn, ordinary).access_count, + 1, + "an ordinary row still increments" + ); +} diff --git a/crates/agent-memory/src/export_file.rs b/crates/agent-memory/src/export_file.rs new file mode 100644 index 0000000..9dc01a9 --- /dev/null +++ b/crates/agent-memory/src/export_file.rs @@ -0,0 +1,243 @@ +//! Private, atomic export-file writing for `agent-memory export --output`. +//! +//! This is the ONE place that touches export-file permissions and atomic +//! replacement (Phase 5 D-02) — keep it here so the Windows ACL story +//! (deferred to Phase 6, DIST-03) is a single-file change, exactly like +//! `config.rs` for the data directory. +//! +//! Three rules, each with a reason: +//! +//! - **Mode `0600` from the first byte.** The export is the whole memory +//! corpus — T-01-03-sensitive, and never to be looser than the `0700` +//! database directory. The mode is set at `create`, so there is no +//! `0644`-then-`chmod` window (T-05-02); umask can only remove bits. +//! - **The temp file lives beside the target.** `std::fs::rename` does not +//! cross mount points, and a shared temp directory is a different trust +//! domain (other users, cleaners, predictable names). The temp is +//! `{target}.export-tmp-{pid}` in the target's own directory. +//! - **`create_new`, never truncate.** A stale temp from a crashed run is an +//! explicit `AlreadyExists` error the user sees, never a silent overwrite of +//! whatever that file is — the pid suffix makes a collision rare, not +//! impossible. +//! +//! The write is committed only after the stream is flushed (the service does +//! that inside its blocking task), `sync_all`-ed and renamed over the target +//! in one step, so the target is always either the previous complete file or +//! the new complete file, never a partial one (T-05-03). Every failure path +//! removes the temp best-effort and never masks the original error. + +use std::ffi::OsString; +use std::fs::File; +use std::path::{Path, PathBuf}; + +/// An export in flight: a private temp file beside `target`, to be +/// [`committed`](PendingExport::commit) into place or +/// [`aborted`](PendingExport::abort). Dropping it without either leaves the +/// temp on disk — callers must do one or the other. +#[derive(Debug)] +pub struct PendingExport { + temp: PathBuf, + target: PathBuf, + file: File, +} + +/// The in-flight temp path for `target`: `{file name}.export-tmp-{pid}` in +/// the SAME directory, so the final rename never crosses a filesystem. A +/// target without a usable file name (e.g. `/`) falls back to the name +/// `export`. +pub fn export_temp_path(target: &Path) -> PathBuf { + let mut name: OsString = target + .file_name() + .map(OsString::from) + .unwrap_or_else(|| OsString::from("export")); + name.push(format!(".export-tmp-{}", std::process::id())); + target.with_file_name(name) +} + +/// Start an export to `target`: create its private temp file (mode `0600` on +/// Unix, `create_new`) in the target's directory. The target itself is not +/// touched until [`PendingExport::commit`]. +pub fn begin(target: &Path) -> std::io::Result { + let temp = export_temp_path(target); + let file = create_private(&temp)?; + Ok(PendingExport { + temp, + target: target.to_path_buf(), + file, + }) +} + +impl PendingExport { + /// The temp file this export is being written to (test-only accessor; + /// production callers never need the path — `commit`/`abort` own it). + #[cfg(test)] + pub fn temp_path(&self) -> &Path { + &self.temp + } + + /// A second handle on the temp file for the writer, so this struct keeps + /// the original for `sync_all` at commit. Both handles share one file + /// position and one set of bytes. + pub fn writer(&self) -> std::io::Result { + self.file.try_clone() + } + + /// Durably publish the export: `sync_all` the temp, then rename it over + /// `target` (replacing an existing file). The caller must have flushed its + /// writer first. On any failure the temp is removed (best-effort) and the + /// error is returned unchanged; the previous target, if any, is untouched. + pub fn commit(self) -> std::io::Result<()> { + if let Err(e) = self.file.sync_all() { + discard_temp(&self.temp); + return Err(e); + } + if let Err(e) = std::fs::rename(&self.temp, &self.target) { + discard_temp(&self.temp); + return Err(e); + } + Ok(()) + } + + /// Give up: remove the temp (best-effort) and leave `target` exactly as it + /// was. Never fails — this runs on error paths and must not mask them. + pub fn abort(self) { + discard_temp(&self.temp); + } +} + +/// Best-effort removal of an export temp file. A leftover is harmless but +/// noisy, so a non-`NotFound` failure is logged at warn level and otherwise +/// ignored — mirroring the store's backup-temp discipline. +fn discard_temp(temp: &Path) { + if let Err(e) = std::fs::remove_file(temp) { + if e.kind() != std::io::ErrorKind::NotFound { + tracing::warn!(temp = %temp.display(), error = %e, "could not remove export temp file"); + } + } +} + +/// Create `path` for writing, owner-only (`0600`), refusing to open an +/// existing file. The mode is applied at creation, so the file is never +/// readable by others for even an instant. +#[cfg(unix)] +fn create_private(path: &Path) -> std::io::Result { + use std::os::unix::fs::OpenOptionsExt; + std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .mode(0o600) + .open(path) +} + +/// Non-Unix fallback: `create_new` without a mode. Windows ACL inheritance is +/// the Phase 6 seam (DIST-03), the same split as `config.rs::create_data_dir`. +#[cfg(not(unix))] +fn create_private(path: &Path) -> std::io::Result { + std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(path) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::Write; + + fn temp_entries(dir: &std::path::Path) -> Vec { + std::fs::read_dir(dir) + .expect("read dir") + .map(|e| e.expect("entry").file_name().to_string_lossy().into_owned()) + .filter(|name| name.contains(".export-tmp-")) + .collect() + } + + #[test] + fn begin_creates_a_private_temp_beside_the_target() { + let dir = tempfile::tempdir().expect("tempdir"); + let target = dir.path().join("out.jsonl"); + + let pending = begin(&target).expect("begin"); + + let expected_temp = dir + .path() + .join(format!("out.jsonl.export-tmp-{}", std::process::id())); + assert_eq!(pending.temp_path(), expected_temp.as_path()); + assert!(expected_temp.exists(), "the temp file exists"); + assert!(!target.exists(), "the target does not exist until commit"); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let mode = std::fs::metadata(&expected_temp) + .expect("temp metadata") + .permissions() + .mode(); + assert_eq!(mode & 0o777, 0o600, "temp is owner-only (T-05-02)"); + } + pending.abort(); + } + + #[test] + fn commit_replaces_an_existing_target_and_leaves_no_temp() { + let dir = tempfile::tempdir().expect("tempdir"); + let target = dir.path().join("out.jsonl"); + std::fs::write(&target, "old").expect("seed target"); + + let pending = begin(&target).expect("begin"); + { + let mut w = pending.writer().expect("writer"); + w.write_all(b"new\n").expect("write"); + w.flush().expect("flush"); + } + pending.commit().expect("commit"); + + assert_eq!(std::fs::read_to_string(&target).expect("read"), "new\n"); + assert!(temp_entries(dir.path()).is_empty(), "no temp left behind"); + } + + #[test] + fn abort_removes_the_temp_and_keeps_the_target() { + let dir = tempfile::tempdir().expect("tempdir"); + let target = dir.path().join("out.jsonl"); + std::fs::write(&target, "old").expect("seed target"); + + let pending = begin(&target).expect("begin"); + pending + .writer() + .expect("writer") + .write_all(b"partial") + .expect("write"); + pending.abort(); + + assert_eq!(std::fs::read_to_string(&target).expect("read"), "old"); + assert!(temp_entries(dir.path()).is_empty(), "no temp left behind"); + } + + #[test] + fn begin_refuses_to_clobber_an_existing_temp() { + let dir = tempfile::tempdir().expect("tempdir"); + let target = dir.path().join("out.jsonl"); + let stale = export_temp_path(&target); + std::fs::write(&stale, "stale run").expect("seed stale temp"); + + let err = begin(&target).expect_err("create_new must refuse an existing temp"); + assert_eq!(err.kind(), std::io::ErrorKind::AlreadyExists); + assert_eq!( + std::fs::read_to_string(&stale).expect("read stale"), + "stale run", + "the stale temp is never truncated" + ); + } + + #[test] + fn export_temp_path_stays_in_the_target_directory() { + let target = std::path::Path::new("/a/b/out.jsonl"); + let temp = export_temp_path(target); + assert_eq!(temp.parent(), Some(std::path::Path::new("/a/b"))); + let name = temp.file_name().expect("file name").to_string_lossy(); + assert!( + name.starts_with("out.jsonl.export-tmp-"), + "temp name is derived from the target: {name}" + ); + } +} diff --git a/crates/agent-memory/src/main.rs b/crates/agent-memory/src/main.rs index 049c25b..2d1c897 100644 --- a/crates/agent-memory/src/main.rs +++ b/crates/agent-memory/src/main.rs @@ -5,6 +5,7 @@ //! `anyhow` is used at these binary edges; the core library uses `thiserror`. mod config; +mod export_file; mod mcp; mod rest; mod serde_util; @@ -77,24 +78,40 @@ enum Command { #[arg(long)] allow_remote: bool, }, - /// Import memories from an external file (INTEROP-01). Re-running the same - /// import is a no-op: duplicates are skipped on the exact - /// (source, type, content) key. + /// Import memories from an external file: a GSD STATE.md (INTEROP-01) or + /// an agent-memory export (INTEROP-02). Re-running the same import is a + /// no-op: duplicates are skipped on the exact (source, type, content) key. + /// A jsonl import also recreates every link, lands every row pending + /// embedding (the sweep re-embeds locally), skips rows already expired, + /// and rejects a malformed file by line before anything is written. Import { - /// Source format. Supported: gsd-state (a GSD .planning/STATE.md file). + /// Source format. Supported: gsd-state (a GSD .planning/STATE.md + /// file), jsonl (an agent-memory export). #[arg(long)] from: String, /// Path to the file to import. path: PathBuf, - /// Scope stamped on every imported memory. Default: the GSD project - /// directory name — the parent of the nearest `.planning` component in - /// the (canonicalized) path — or the current directory's name when the - /// path has no `.planning` ancestor. + /// gsd-state only: scope stamped on every imported memory. Default: + /// the GSD project directory name — the parent of the nearest + /// `.planning` component in the (canonicalized) path — or the current + /// directory's name when the path has no `.planning` ancestor. A + /// jsonl file carries each memory's own scope, so --scope is rejected. #[arg(long)] scope: Option, }, + /// Export every memory and link as format-v1 JSONL (DIST-04): a header + /// line, one record per memory, one record per link; embedding vectors are + /// never included. Writes to stdout by default. + Export { + /// Write to PATH instead of stdout. The file is created private (0600 + /// on Unix) as a temp file in PATH's directory and renamed into place; + /// an existing PATH is replaced; an interrupted export never leaves a + /// partial PATH. + #[arg(long)] + output: Option, + }, } #[tokio::main] @@ -123,6 +140,7 @@ async fn main() -> anyhow::Result<()> { Command::Import { from, path, scope } => { run_import(cli.db, cli.ollama_url, from, path, scope).await } + Command::Export { output } => run_export(cli.db, cli.ollama_url, output).await, } } @@ -218,10 +236,20 @@ async fn serve_rest( Ok(()) } -/// Run a one-shot import (INTEROP-01): parse the file into typed drafts, wire -/// the store/embedder exactly like [`serve`], import idempotently, and print -/// the report to STDOUT. stdout is sanctioned here — this is a CLI subcommand, -/// not the MCP transport (MCP-05 applies only to `serve`). +/// Run a one-shot import: parse the file into typed records, wire the +/// store/embedder exactly like [`serve`], import idempotently, and print the +/// report to STDOUT. stdout is sanctioned here — this is a CLI subcommand, not +/// the MCP transport (MCP-05 applies only to `serve`). +/// +/// Two formats, two tolerance policies (Phase 5 D-11): +/// +/// - `gsd-state` (INTEROP-01): the tolerant STATE.md scraper; malformed +/// bullets are skip-counted and reported. +/// - `jsonl` (INTEROP-02): the strict format-v1 reader. The WHOLE file is +/// parsed and validated BEFORE the database is resolved or opened, so a +/// malformed file never creates or touches a database and the error names +/// the offending line. The embedder is constructed but never called: every +/// imported row lands pending and the sweep re-embeds it locally (D-13). async fn run_import( db_flag: Option, ollama_url: String, @@ -229,31 +257,144 @@ async fn run_import( path: PathBuf, scope: Option, ) -> anyhow::Result<()> { - if from != "gsd-state" { - anyhow::bail!("unsupported --from value {from:?} (supported formats: gsd-state)"); + match from.as_str() { + "gsd-state" => { + let text = read_import_file(&path)?; + let scope = scope.unwrap_or_else(|| default_scope_for(&path)); + let parsed = agent_memory_core::import::gsd_state::parse_gsd_state(&text, Some(scope)); + + let service = open_service(db_flag, ollama_url)?; + let report = service.import(parsed.drafts).await.context("importing")?; + println!( + "imported: {}, skipped duplicates: {}, malformed lines skipped: {}", + report.imported, report.skipped_duplicates, parsed.skipped + ); + } + "jsonl" => { + if scope.is_some() { + anyhow::bail!( + "--scope applies only to --from gsd-state (a jsonl file carries each memory's own scope)" + ); + } + let text = read_import_file(&path)?; + // Validate the whole file first (D-08 pass 1): a defect is + // reported by line and the database is never even opened. + let file = agent_memory_core::import::jsonl::parse_jsonl(&text) + .with_context(|| format!("validating {}", path.display()))?; + + let service = open_service(db_flag, ollama_url)?; + let report = service.import_jsonl(file).await.context("importing")?; + println!( + "imported: {}, skipped duplicates: {}, skipped expired: {}, links imported: {}, links skipped duplicates: {}, links dropped: {}", + report.imported, + report.skipped_duplicates, + report.skipped_expired, + report.links_imported, + report.links_skipped_duplicates, + report.links_dropped + ); + let pending = report.imported; + println!( + "{pending} rows pending embedding — the sweep backfills them on the next serve/serve-rest start (128 per tick)" + ); + } + other => { + anyhow::bail!( + "unsupported --from value {other:?} (supported formats: gsd-state, jsonl)" + ); + } } + Ok(()) +} - let text = - std::fs::read_to_string(&path).with_context(|| format!("reading {}", path.display()))?; - let scope = scope.unwrap_or_else(|| default_scope_for(&path)); - let parsed = agent_memory_core::import::gsd_state::parse_gsd_state(&text, Some(scope)); +/// Read an import file to a string with the path in the error context. +fn read_import_file(path: &std::path::Path) -> anyhow::Result { + std::fs::read_to_string(path).with_context(|| format!("reading {}", path.display())) +} +/// Resolve the DB path, open the store and build the service with the real +/// clock and the Ollama embedder — the one-shot subcommand wiring shared by +/// both import formats. +fn open_service(db_flag: Option, ollama_url: String) -> anyhow::Result { let db_path = resolve_db_path(db_flag).context("resolving database path")?; let store = SqliteStore::open(&db_path) .with_context(|| format!("opening database at {}", db_path.display()))?; let embedder: Arc = Arc::new(OllamaClient::new(ollama_url)); - let service = MemoryService::new( + Ok(MemoryService::new( Arc::new(store), Arc::new(SystemClock), embedder, DecayConfig::default(), - ); + )) +} - let report = service.import(parsed.drafts).await.context("importing")?; - println!( - "imported: {}, skipped duplicates: {}, malformed lines skipped: {}", - report.imported, report.skipped_duplicates, parsed.skipped - ); +/// Run a one-shot export (DIST-04): wire the store/embedder through +/// [`open_service`], the same block [`run_import`] uses (the embedder is +/// constructed but never called — vectors do not travel), then stream the +/// format-v1 JSONL document to STDOUT or, with `--output`, into a private +/// temp file beside the target that is renamed into place only after the +/// whole stream is flushed and synced (D-02). The report line goes to +/// STDERR, always: stdout may BE the file here, so the convention is the +/// reverse of `import`'s. +/// +/// With `--output`, the target path is validated (`export_file::begin`) +/// BEFORE the database is opened, matching `import --from jsonl`, which +/// validates the file before opening: an unwritable `--output` must never +/// resolve, create and migrate a database first (05-REVIEW IN-03). If the +/// open fails after `begin`, the pending temp is aborted like any other +/// failure between `begin` and `commit`. +/// +/// The stdout sink is `BufWriter::new(std::io::stdout())`: the `Stdout` +/// handle is `Send`, whereas `StdoutLock` is not and cannot cross into the +/// service's blocking task. The service flushes the writer before returning, +/// so `commit`'s `sync_all` sees every byte. +async fn run_export( + db_flag: Option, + ollama_url: String, + output: Option, +) -> anyhow::Result<()> { + // The binary's own version, not the core crate's (Pitfall 9). + let version = env!("CARGO_PKG_VERSION").to_string(); + + let report = match output { + None => { + let service = open_service(db_flag, ollama_url)?; + let sink: Box = + Box::new(std::io::BufWriter::new(std::io::stdout())); + service.export(sink, version).await.context("exporting")? + } + Some(path) => { + let pending = export_file::begin(&path) + .with_context(|| format!("creating export file beside {}", path.display()))?; + // Everything that can fail between `begin` and `commit` — opening + // the database included — runs under one result, so exactly one + // abort covers every failure and the temp never outlives an error. + let streamed: anyhow::Result<_> = async { + let service = open_service(db_flag, ollama_url)?; + let file = pending + .writer() + .context("duplicating the export file handle")?; + let sink: Box = Box::new(std::io::BufWriter::new(file)); + service.export(sink, version).await.context("exporting") + } + .await; + match streamed { + Ok(report) => { + pending.commit().with_context(|| { + format!("moving the export into place at {}", path.display()) + })?; + report + } + Err(e) => { + pending.abort(); + return Err(e); + } + } + } + }; + + let (memories, links) = (report.memories, report.links); + eprintln!("exported {memories} memories and {links} links"); Ok(()) } diff --git a/crates/agent-memory/src/mcp.rs b/crates/agent-memory/src/mcp.rs index b4e2d16..073e255 100644 --- a/crates/agent-memory/src/mcp.rs +++ b/crates/agent-memory/src/mcp.rs @@ -435,7 +435,7 @@ impl MemoryMcp { // A missing endpoint is a clean not-found, NOT an error — the // memory_forget shape, plus the id that was missing (D-02). let status = match outcome { - LinkOutcome::Linked => serde_json::json!({ + LinkOutcome::Linked { .. } => serde_json::json!({ "from_id": args.from_id, "to_id": args.to_id, "kind": kind, @@ -1343,6 +1343,40 @@ mod tests { } } + #[tokio::test] + async fn memory_store_storage_failure_is_internal_error_without_sqlite_text() { + // D-14 (04-REVIEW IN-01): a write-path storage fault — the `memories` + // table renamed away underneath the server — reaches the agent only + // as the generic internal error, like search since T-04-19. + let internal_error_code = McpError::internal_error("x", None).code; + let dir = tempfile::tempdir().expect("tempdir"); + let server = test_server(&dir); + agent_memory_core::store::sqlite::register_vec_extension().expect("register vec0"); + rusqlite::Connection::open(dir.path().join("mem.db")) + .expect("open db") + .execute_batch("ALTER TABLE memories RENAME TO memories_offline;") + .expect("rename memories table"); + + let args: StoreArgs = serde_json::from_value(serde_json::json!({ + "content": "will not land", + "type": "DECISION" + })) + .expect("store args deserialize"); + let err = server + .memory_store(Parameters(args)) + .await + .expect_err("a missing memories table must fail the store"); + assert_eq!(err.code, internal_error_code, "{err:?}"); + assert!( + err.message.contains("store failed due to a storage error"), + "{err:?}" + ); + assert!( + !err.message.to_lowercase().contains("sqlite"), + "no SQLite text may reach the agent: {err:?}" + ); + } + #[test] fn map_mcp_error_splits_client_and_internal_tiers() { let invalid_params_code = McpError::invalid_params("x", None).code; diff --git a/crates/agent-memory/src/rest/handlers.rs b/crates/agent-memory/src/rest/handlers.rs index 17d0600..6f55244 100644 --- a/crates/agent-memory/src/rest/handlers.rs +++ b/crates/agent-memory/src/rest/handlers.rs @@ -367,7 +367,7 @@ pub async fn link_handler( .map_err(map_memory_error)?; match outcome { - LinkOutcome::Linked => Ok(Json(serde_json::json!({ + LinkOutcome::Linked { .. } => Ok(Json(serde_json::json!({ "from_id": body.from_id, "to_id": body.to_id, "kind": kind, @@ -888,6 +888,68 @@ mod tests { } } + /// Take the `memories` table offline underneath a live state (the T-04-19 + /// RENAME recipe, now against the main table): every store/list statement + /// fails at prepare time with the engine's `no such table` text, which + /// must never reach the REST client (D-14). + fn take_memories_offline(dir: &tempfile::TempDir) { + agent_memory_core::store::sqlite::register_vec_extension().expect("register vec0"); + rusqlite::Connection::open(dir.path().join("mem.db")) + .expect("open db") + .execute_batch("ALTER TABLE memories RENAME TO memories_offline;") + .expect("rename memories table"); + } + + #[tokio::test] + async fn health_storage_failure_is_500_without_sqlite_text() { + // D-14 (04-REVIEW IN-01): /health is a one-row list through the + // service, so its storage fault is redacted at the same seam. + let dir = tempfile::tempdir().expect("tempdir"); + let state = test_state(&dir); + take_memories_offline(&dir); + + let resp = match health_handler(State(state)).await { + Ok(_) => panic!("a missing memories table must fail the health check"), + Err(err) => err.into_response(), + }; + assert_eq!(resp.status(), StatusCode::INTERNAL_SERVER_ERROR); + let body = body_json(resp).await; + let error = body["error"].as_str().expect("500 body carries an error"); + assert!( + error.contains("list failed due to a storage error"), + "{body}" + ); + assert!( + !error.to_lowercase().contains("sqlite"), + "no SQLite text may reach the client: {body}" + ); + } + + #[tokio::test] + async fn store_storage_failure_is_500_without_sqlite_text() { + // D-14 (04-REVIEW IN-01): a write-path storage fault is redacted too — + // v0.0.1 echoed `sqlite error: no such table: memories` here. + let dir = tempfile::tempdir().expect("tempdir"); + let state = test_state(&dir); + take_memories_offline(&dir); + + let resp = match store_handler(State(state), Json(store_request("x", "DECISION"))).await { + Ok(_) => panic!("a missing memories table must fail the store"), + Err(err) => err.into_response(), + }; + assert_eq!(resp.status(), StatusCode::INTERNAL_SERVER_ERROR); + let body = body_json(resp).await; + let error = body["error"].as_str().expect("500 body carries an error"); + assert!( + error.contains("store failed due to a storage error"), + "{body}" + ); + assert!( + !error.to_lowercase().contains("sqlite"), + "no SQLite text may reach the client: {body}" + ); + } + #[tokio::test] async fn search_with_zero_limit_returns_400_never_500() { // API-02 / D-01: an out-of-bounds limit is CLIENT input — 400 with the diff --git a/crates/agent-memory/tests/cli.rs b/crates/agent-memory/tests/cli.rs new file mode 100644 index 0000000..e44d1e4 --- /dev/null +++ b/crates/agent-memory/tests/cli.rs @@ -0,0 +1,355 @@ +//! Spawned-binary realism tests for `agent-memory export` and +//! `agent-memory import --from jsonl` (Phase 5 D-01, D-02, D-11, D-12). +//! +//! These tests run the exact binary that ships, the way a user does: a fresh +//! `--db` under a tempdir, a hand-written seed file, real stdout/stderr and +//! real exit codes. They are **realism-only and contribute zero coverage by +//! design**, exactly like `tests/rest.rs`: a child process never flushes LLVM +//! profile data, so the coverage-bearing proofs of the same behaviour live in +//! `agent-memory-core` (`export::`, `import::jsonl`, `tests/export_import.rs`) +//! and in the in-process handler tests. What this file adds is the one thing +//! those cannot: `main.rs`'s argument parsing, file I/O, report strings and +//! exit paths, end to end over two databases. +//! +//! Every spawn gets `AGENT_MEMORY_OLLAMA_URL=http://127.0.0.1:9` (the discard +//! port — connection refused) so the outcome is deterministic on any machine; +//! neither subcommand contacts the embedder anyway (D-13). + +use std::path::Path; +use std::process::{Command, Output, Stdio}; + +/// The first-line report after the seed (2 memories, 1 link) lands in an +/// empty database — and again after `out.jsonl` lands in a second one. +const FIRST_IMPORT: &str = "imported: 2, skipped duplicates: 0, skipped expired: 0, \ + links imported: 1, links skipped duplicates: 0, links dropped: 0"; + +/// The first-line report when the same file is imported a second time: every +/// memory and the one edge dedup, nothing is expired, nothing is dropped. +const REIMPORT: &str = "imported: 0, skipped duplicates: 2, skipped expired: 0, \ + links imported: 0, links skipped duplicates: 1, links dropped: 0"; + +fn binary_path() -> String { + // CARGO_BIN_EXE_ is set by cargo for integration tests of a binary crate. + env!("CARGO_BIN_EXE_agent-memory").to_string() +} + +/// Run the binary with `args` and capture everything. Stdin is closed and +/// nothing but the environment is inherited from the test process. +fn run(args: &[&str]) -> Output { + Command::new(binary_path()) + .args(args) + // Deterministically dead Ollama URL: the same outcome on every + // machine, with or without a local daemon. + .env("AGENT_MEMORY_OLLAMA_URL", "http://127.0.0.1:9") + .stdin(Stdio::null()) + .output() + .expect("spawn agent-memory") +} + +fn stdout(o: &Output) -> String { + String::from_utf8_lossy(&o.stdout).into_owned() +} + +fn stderr(o: &Output) -> String { + String::from_utf8_lossy(&o.stderr).into_owned() +} + +/// Tempdir paths are UTF-8 on every platform the suite runs on. +fn s(path: &Path) -> &str { + path.to_str().expect("utf-8 path") +} + +/// A format-v1 header declaring `rows` memories and `links` links. Built with +/// `serde_json::json!` — the parser does not care about key order (only the +/// exporter's output order is pinned, by the goldens in `export.rs`). +fn header(rows: u64, links: u64) -> serde_json::Value { + serde_json::json!({ + "record": "header", + "format": "agent-memory-export", + "version": 1, + "exported_at": 1_700_000_000, + "embedding_model": "nomic-embed-text", + "embedding_dim": 768, + "schema_version": 3, + "agent_memory_version": "0.0.1-seed", + "exported_rows": rows, + "exported_links": links, + }) +} + +/// Two memories under keys 10 and 20 — deliberately not 1 and 2, so a key can +/// never coincide with the id the target database hands out — and one +/// `relates_to` edge between them. +fn seed_jsonl() -> String { + let records = [ + header(2, 1), + serde_json::json!({ + "record": "memory", + "key": 10, + "type": "DECISION", + "content": "Use reciprocal-rank fusion for hybrid search", + "tags": ["search", "ranking"], + "source": "cli-realism", + "scope": null, + "created_at": 1_690_000_000, + "last_accessed": 1_690_000_000, + "access_count": 3, + "expires_at": null, + }), + serde_json::json!({ + "record": "memory", + "key": 20, + "type": "PATTERN", + "content": "Exports stream one JSON line per record", + "tags": [], + "source": null, + "scope": "project-a", + "created_at": 1_690_000_100, + "last_accessed": 1_690_000_100, + "access_count": 0, + "expires_at": null, + }), + serde_json::json!({ + "record": "link", + "from": 10, + "to": 20, + "kind": "relates_to", + "created_at": 1_690_000_200, + }), + ]; + let mut text = String::new(); + for record in &records { + text.push_str(&serde_json::to_string(record).expect("serialise seed record")); + text.push('\n'); + } + text +} + +/// Names of entries in `dir` that look like an in-flight export temp file. +fn export_temp_entries(dir: &Path) -> Vec { + std::fs::read_dir(dir) + .expect("read dir") + .map(|e| e.expect("entry").file_name().to_string_lossy().into_owned()) + .filter(|name| name.contains(".export-tmp-")) + .collect() +} + +#[test] +fn export_import_round_trip_over_the_real_binary() { + let dir = tempfile::tempdir().expect("tempdir"); + let seed = dir.path().join("seed.jsonl"); + std::fs::write(&seed, seed_jsonl()).expect("write seed"); + let db_a = dir.path().join("a.db"); + let db_b = dir.path().join("b.db"); + let out = dir.path().join("out.jsonl"); + + // 1. The seed lands in A with the documented six counts and pending line. + let o = run(&["import", "--from", "jsonl", s(&seed), "--db", s(&db_a)]); + assert!(o.status.success(), "import into A failed: {}", stderr(&o)); + let text = stdout(&o); + let lines: Vec<&str> = text.lines().collect(); + assert_eq!(lines.first().copied(), Some(FIRST_IMPORT), "stdout: {text}"); + assert!( + lines + .get(1) + .is_some_and(|l| l.starts_with("2 rows pending embedding")), + "pending line: {text}" + ); + + // 2. `--output` writes a private, complete file beside no temp sibling. + let o = run(&["export", "--db", s(&db_a), "--output", s(&out)]); + assert!(o.status.success(), "export --output failed: {}", stderr(&o)); + assert!( + stderr(&o).contains("exported 2 memories and 1 links"), + "stderr report: {}", + stderr(&o) + ); + let file_text = std::fs::read_to_string(&out).expect("read out.jsonl"); + let file_lines: Vec<&str> = file_text.lines().collect(); + assert_eq!( + file_lines.len(), + 4, + "header + 2 memories + 1 link: {file_text}" + ); + assert!( + file_lines[0].starts_with("{\"record\":\"header\""), + "line 1: {}", + file_lines[0] + ); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let mode = std::fs::metadata(&out) + .expect("out.jsonl metadata") + .permissions() + .mode(); + assert_eq!(mode & 0o777, 0o600, "the export file is owner-only (D-02)"); + } + assert!( + export_temp_entries(dir.path()).is_empty(), + "no `.export-tmp-` sibling may survive a successful export" + ); + + // 3. The stdout export carries the same records (the header's + // `exported_at` may differ by wall clock). + let o = run(&["export", "--db", s(&db_a)]); + assert!( + o.status.success(), + "export to stdout failed: {}", + stderr(&o) + ); + let stdout_text = stdout(&o); + let stdout_lines: Vec<&str> = stdout_text.lines().collect(); + assert_eq!(stdout_lines.len(), 4, "stdout export: {stdout_text}"); + assert_eq!( + &stdout_lines[1..], + &file_lines[1..], + "records differ between sinks" + ); + + // 4. The exported file restores into an empty B ... + let o = run(&["import", "--from", "jsonl", s(&out), "--db", s(&db_b)]); + assert!(o.status.success(), "import into B failed: {}", stderr(&o)); + let text = stdout(&o); + assert_eq!(text.lines().next(), Some(FIRST_IMPORT), "stdout: {text}"); + + // 5. ... and importing it again is a no-op reported as duplicates (D-07). + let o = run(&["import", "--from", "jsonl", s(&out), "--db", s(&db_b)]); + assert!( + o.status.success(), + "re-import into B failed: {}", + stderr(&o) + ); + let text = stdout(&o); + assert_eq!(text.lines().next(), Some(REIMPORT), "stdout: {text}"); + assert!( + text.lines() + .nth(1) + .is_some_and(|l| l.starts_with("0 rows pending embedding")), + "pending line after re-import: {text}" + ); +} + +#[test] +fn a_malformed_file_exits_non_zero_naming_the_line_and_creates_no_database() { + let dir = tempfile::tempdir().expect("tempdir"); + let bad = dir.path().join("bad.jsonl"); + let header_line = serde_json::to_string(&header(0, 0)).expect("serialise header"); + std::fs::write(&bad, format!("{header_line}\n{{\"record\":\"memo\"}}\n")).expect("write bad"); + let never = dir.path().join("never.db"); + + let o = run(&["import", "--from", "jsonl", s(&bad), "--db", s(&never)]); + + assert!(!o.status.success(), "a malformed file must not import"); + let err = stderr(&o); + assert!( + err.contains("line 2:"), + "stderr names the offending line: {err}" + ); + assert!( + !never.exists(), + "validation runs before the database is opened (D-08 pass 1 / D-11)" + ); +} + +#[test] +fn an_unsupported_from_value_still_bails() { + let dir = tempfile::tempdir().expect("tempdir"); + let seed = dir.path().join("seed.jsonl"); + std::fs::write(&seed, seed_jsonl()).expect("write seed"); + let db = dir.path().join("never.db"); + + let o = run(&["import", "--from", "bogus", s(&seed), "--db", s(&db)]); + + assert!(!o.status.success(), "an unknown --from must fail"); + let err = stderr(&o); + assert!( + err.contains("supported formats: gsd-state, jsonl"), + "stderr lists the supported formats: {err}" + ); + assert!( + !db.exists(), + "the database is never created for a bad --from" + ); +} + +#[test] +fn scope_is_rejected_for_jsonl() { + let dir = tempfile::tempdir().expect("tempdir"); + let seed = dir.path().join("seed.jsonl"); + std::fs::write(&seed, seed_jsonl()).expect("write seed"); + let db = dir.path().join("never.db"); + + let o = run(&[ + "import", + "--from", + "jsonl", + s(&seed), + "--scope", + "x", + "--db", + s(&db), + ]); + + assert!(!o.status.success(), "--scope with jsonl must fail"); + let err = stderr(&o); + assert!( + err.contains("--scope applies only to --from gsd-state"), + "stderr explains the rejection: {err}" + ); + assert!( + !db.exists(), + "the database is never created when --scope is rejected" + ); +} + +#[test] +fn an_unwritable_output_path_fails_before_the_database_is_created() { + // 05-REVIEW IN-03: `--output` is validated BEFORE the database is opened, + // the way `import --from jsonl` validates the file first. Before the fix + // this command resolved, created and migrated the database and only then + // failed on `create_new` in the missing directory. + let dir = tempfile::tempdir().expect("tempdir"); + let out = dir.path().join("nonexistent").join("out.jsonl"); + let never = dir.path().join("never.db"); + + let o = run(&["export", "--db", s(&never), "--output", s(&out)]); + + assert!(!o.status.success(), "a missing output directory must fail"); + let err = stderr(&o); + assert!( + err.contains("creating export file beside"), + "stderr names the output step: {err}" + ); + assert!( + !never.exists(), + "the database is never created when --output is unwritable" + ); + assert!(!out.exists()); + assert!( + export_temp_entries(dir.path()).is_empty(), + "no temp is left anywhere under the tempdir" + ); +} + +#[test] +fn invalid_utf8_is_rejected_cleanly() { + let dir = tempfile::tempdir().expect("tempdir"); + let garbage = dir.path().join("garbage.jsonl"); + std::fs::write(&garbage, [0xffu8, 0xfe, 0x0a]).expect("write garbage"); + let db = dir.path().join("never.db"); + + let o = run(&["import", "--from", "jsonl", s(&garbage), "--db", s(&db)]); + + assert!(!o.status.success(), "invalid UTF-8 must fail"); + let err = stderr(&o); + assert!(err.contains("reading"), "the read step names itself: {err}"); + assert!( + !err.contains("PANIC"), + "garbage must never panic the binary: {err}" + ); + assert!( + !db.exists(), + "the database is never created for unreadable input" + ); +}