feat: Phase 5 — portable JSONL export and import by hermanngeorge15 · Pull Request #17 · UnityInFlow/agent-memory

hermanngeorge15 · 2026-10-25T13:55:48Z

Closes #16.

Milestone v1.1 (ships as product v0.1.0), Phase 5. 34 commits, one per task; I intend to rebase-merge it.

What changes

agent-memory export writes versioned JSONL. Line 1 is a header (format version, embedding model/dim, schema and binary version, counts). Memories follow in id order, then link edges with keys local to the file. Embeddings and derived fields are excluded. Output goes to stdout by default; --output writes a temp file beside the target, syncs it and renames it over the target, with mode 0600 on Unix.
agent-memory import reads that format:
- It validates the whole file first (line N: errors, all-or-nothing, nothing written on failure).
- It then imports memories, building a key→id map, and imports links through that map.
- It is idempotent on (source, type, content); duplicates map to the existing row. It works into a non-empty DB.
- Expired rows, and links to expired rows, are skipped and counted.
- Rows land with embedding_status = 0, so the existing sweep backfill embeds them. No vectors travel in the file.
Redaction across every error path (closes the Phase 4 deferral, 04-REVIEW IN-01). Every public service method returns errors through one exhaustive redact_storage_error. SQLite, pool, join and migration errors are logged in full and returned as a generic Internal message. This covers MCP, REST and the CLI. Search error text is unchanged byte for byte.
ImportReport gains six counts. The change is additive.

No new dependencies, and no changes to Cargo.lock, .github/ or the SQL migrations.

Verification

cargo fmt --all -- --check: clean
cargo clippy --workspace --all-targets --locked -- -D warnings: clean
cargo test --workspace --locked: 287 passed, 0 failed, 2 ignored. I re-ran this locally at the PR head.
GSD verification: passed 35/35. The first pass was 32/33. Gap plan 05-05 added a single-snapshot race test and a live run: export → import → sweep → hybrid search against nomic-embed-text, where the imported memory came back at rank 0.
Code review: 0 critical, 3 warnings and 3 info, all fixed with tests:
- after id remapping, a link from a memory to itself was silently dropped by INSERT OR IGNORE;
- access_count could overflow to REAL and break every later export;
- the duplicate check matched rows that had expired but not yet been swept.
No unwrap() added to production code.
Docs: README export/import section; rustdoc.
ADR: none. Decisions D-01 to D-14 are in .planning/phases/05-portable-export-import/05-CONTEXT.md.

Not in this PR

/gsd-secure-phase 5, the threat-model audit, has not run yet. I'll run it before Phase 6.
Deferred ideas recorded in CONTEXT: --include-embeddings, export filters, MCP/REST export, import --embed-now.

🤖 Generated with Claude Code

…t, export tracer, import tracer, docs + CLI realism) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

- 05-RESEARCH.md: mark the four open questions RESOLVED with the plan/task that carries each - 05-02-PLAN.md: single-imperative Task 1 step 6 for Command::Export; artifact anchor on fn run_export - 05-03-PLAN.md: re-export equality also normalises link from/to - STATE.md: Phase 5 planned (4 plans); ROADMAP wave annotations Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

- generalise redact_search_error into redact_storage_error(op, e): the storage tier (Sqlite | Pool | Join | Migration) is logged at error level and replaced by Internal("<op> failed due to a storage error (details are in the server log)"); client and crate-authored text passes through - every public MemoryService method (store, list, search, import, forget, link, unlink, update, sweep) delegates to a *_unredacted body so early returns cannot escape unredacted; SEARCH_STORAGE_FAILURE keeps its exact text and a unit test pins it to the template at op = "search" - tests/redaction.rs: eight fault-injection proofs (ALTER TABLE ... RENAME on a raw side connection) for the non-search paths - REST /health + POST /api/memories and MCP memory_store proven to return 500 / internal_error with no "sqlite" substring; transport mappers unchanged (closes 04-REVIEW IN-01) - README: redaction sentence widened to every surface; behavioral change 4 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…, D-12) - Store::exists -> Store::find_by_key returning Option<i64>: same NULL-safe `source IS ?1` three-column key (CR-01), now `SELECT id` so an importer can map an export-local key onto the existing row; import() dedups through `.is_some()` - LinkOutcome::Linked gains `created: bool`, read from tx.changes() before commit: true on a fresh edge, false on the D-02 duplicate / symmetric relates_to re-link, so links_skipped_duplicates is countable through the existing Store::link - seven Linked sites updated (store impl, MCP + REST arms as `{ .. }` with byte-identical JSON, four relations.rs asserts upgraded to the stronger created: true/false idempotency proofs) - tests/store.rs: find_by_key_returns_the_matching_id_null_safely Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…s (D-10, D-13) - domain.rs: ImportedMemory beside NewMemory — absolute created_at, access_count and expires_at preserved from a format-v1 file; no last_accessed field because it is refreshed to import-time now - store/mod.rs: Store::insert_imported(m, now) — same single writer transaction as insert, no embedding parameter (row always lands embedding_status = 0 for the sweep backfill); trait doc names both expiry predicates (importer skips expires_at <= now, sweep deletes < now) - sqlite.rs: the one memories INSERT moves into a private insert_row(tx, &MemoryRow) helper behind both insert and insert_imported, with access_count / created_at / last_accessed bound separately; insert behaviour is unchanged, no vector write on the imported path - tests/store.rs: insert_imported_preserves_provenance_and_lands_pending (raw-connection column asserts, zero vec_memories rows, visible via list and keyword search through the FTS trigger) - no migration: sql/ untouched, LATEST_SCHEMA_VERSION stays 3 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…-03..D-06) - export.rs: format v1 defined once — FORMAT_NAME/FORMAT_VERSION, tagged strict `Record { Header, Memory, Link }` (record first, declaration order, deny_unknown_fields), ExportSnapshot/ExportEvent stream protocol, module-doc spec, 9 golden/strictness tests - Store::export_rows: one pooled connection, one read transaction (unchecked_transaction) — meta pin + counts, memories ORDER BY id, links ORDER BY from_id, to_id, kind; rows visited lazily, never collected - MemoryService::export(out, agent_memory_version) -> ExportReport: one spawn_blocking hop, header built from the snapshot, one line per record, flush inside the task; redact_storage_error("export", _) (D-14) - CLI `agent-memory export`: BufWriter over Stdout (StdoutLock is !Send), CARGO_PKG_VERSION as the header version, report on stderr - tests/export_import.rs: empty DB header, ascending memories then links as stored, exactly 11 memory keys, expired rows exported, byte-identical re-export Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…t into place (D-02) - export_file.rs (new): PendingExport — `{target}.export-tmp-{pid}` beside the target, OpenOptions create_new + mode 0600 on Unix (cfg split like config.rs; Windows ACLs are the Phase 6 seam), writer() via try_clone, commit() = sync_all then rename over the target, abort() = best-effort remove; temp removed on every failure path; 5 unit tests incl. the 0600 mode - main.rs: `Command::Export { --output PATH }`; run_export picks the stdout BufWriter or the pending file, commits only after the service has flushed and returned Ok, aborts on any error (one abort covers writer() + export) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ently (D-07..D-13) - import/jsonl.rs: strict pure parser parse_jsonl -> JsonlFile; lenient header probe before the strict Record parse so a v2 file reaches the 'upgrade agent-memory' message; every rejection is InvalidArgument naming the importer's own 1-based line (serde's position suffix is stripped); type/kind/self-link go through the Phase 3 seam; link keys resolved order-independently; declared counts enforced - import/mod.rs: pub mod jsonl; tolerance reworded as per-importer - service.rs: ImportReport +skipped_expired/links_imported/ links_skipped_duplicates/links_dropped (gsd path reports zeros); CLOCK_SKEW_TOLERANCE_SECS + clock_skew_message; import_jsonl runs the memories pass (expiry skip -> find_by_key dedup -> insert_imported, building key->id) and the links pass through the map in ONE blocking hop, never touching the embedder; redact_storage_error("import") - main.rs: three-arm --from match; jsonl parses BEFORE the database is resolved or opened, prints the six-count line + pending-embedding line, rejects --scope; gsd-state output byte-identical - tests: round trip into an empty target (provenance preserved, last_accessed refreshed, pending, edges on target ids, normalised re-export equality), idempotent re-import, gsd-state zeros assertion Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…→sweep, malformed table - overlapping rowids on a non-empty target: no pre-existing row gains an edge, orphan_edges == 0, re-import maps onto the restored rows - link endpoints that dedup to an existing row attach to THAT row, never to the row sitting at the file's key - identical rows collapse to one and count a duplicate (D-07) - expiry fixture at the exact <= boundary {2,0,1,1,0,1}, one second earlier restores everything; a 400-day-old row imports decay-fresh (last_accessed refreshed, created_at preserved) through the clock-skew warning path - rows land pending with no vector; the sweep (embedder UP) backfills them and they become hybrid-searchable (D-13) - header-only import is an all-zero no-op; zero-byte / newline-only rejected at line 1; links before memories import identically to the canonical order; a file with one bad line writes nothing - jsonl.rs: the D-11 malformed table reflowed to one case per line (#[rustfmt::skip]) — 25 classes, every rejection asserted by line Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…icy, pending embeddings) - New '## Export & import' between 'Import from GSD' and 'Lifecycle': usage, CLI-only rationale (D-01), what travels / what is derived (D-04, D-13), the three format-v1 field tables in wire order (D-03, D-05, D-06), the strict-reader rules and version policy (D-11), dedup and non-empty-target safety (D-07, D-08), the per-column timestamp table (D-10), pending embeddings and the 128-per-tick sweep (D-13), the six-count report (D-12), private atomic --output (D-02), and the VACUUM INTO backup-vs-export tip - Lifecycle item 3 cross-references rows restored by import --from jsonl - Run section gains an 'agent-memory export --output' example Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…d binary - New crates/agent-memory/tests/cli.rs: realism-only spawned-binary tests (zero coverage by design, like tests/rest.rs) over the exact binary that ships, each with --db under a fresh tempdir and a dead Ollama URL - export_import_round_trip_over_the_real_binary: hand-written seed (keys 10 and 20, one relates_to edge) imports with the exact six-count line and the pending-embedding line; --output writes a 4-line 0600 file with no .export-tmp- sibling; the stdout export carries identical records; the file restores into a second database and a re-import reports only duplicates (D-01, D-02, D-07, D-12) - a_malformed_file_exits_non_zero_naming_the_line_and_creates_no_database: 'line 2:' on stderr and the --db path never created (D-08 pass 1, D-11) - an_unsupported_from_value_still_bails, scope_is_rejected_for_jsonl, invalid_utf8_is_rejected_cleanly: documented messages, no PANIC, no database created Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…it (REVIEW WR-01) The JSONL parser rejects self-links on file keys, but D-07 dedup can map two distinct keys onto the same target row. Pass 3 then called store.link(N, N), which INSERT OR IGNORE swallowed under the 0003 CHECK and reported as links_skipped_duplicates — an edge that never existed. Guard the collapsed pair in pass 3: count it as links_dropped (the link-level consequence of the D-07 collapse) and never reach the store. Kill-test: twin rows linked in the source, imported into an empty DB, report (1, 1, 0, 0, 0, 1) and zero memory_links rows; re-import still idempotent. README "links dropped" row widened accordingly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…EVIEW WR-02) access_count travels verbatim (D-10) with no bounds, and it is the one preserved column later used in arithmetic. SQLite's `+` does not raise on integer overflow — it converts the result to REAL — so one imported row at i64::MAX, bumped once by a search or update, made every later export of the whole database fail with the redacted storage error until the column was repaired by hand. - jsonl.rs pass 1 rejects `access_count < 0` under the D-11 line prefix (`line N: access_count must be >= 0 (got -1)`); the upper bound is the i64 range serde already enforces. - bump_access and update's freshness bump saturate at i64::MAX via `CASE WHEN access_count < 9223372036854775807 THEN access_count + 1 ELSE access_count END` — a statement change, no schema change. - Tests: -1 and 2^63 rows in the malformed table, i64::MAX parses in the hostile-input test, a direct bump_access saturation proof that checks typeof(access_count) stays integer, and an import -> update -> export round trip of a maxed row in export_import.rs. README strict-list gains the negative access_count rule. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…WR-03, IN-01) Store::find_by_key gains a `now` parameter and matches only live rows: `AND (expires_at IS NULL OR expires_at > ?4)` — the same `<= now` boundary the JSONL importer applies to FILE rows (D-10). A CLI import runs no sweep, so before this a key could map onto a target row whose TTL had lapsed, count a duplicate, and hang every imported edge on it; the next sweep then cascaded those edges away while the report claimed "N links imported". Also `ORDER BY id LIMIT 1` (IN-01): with duplicate live rows the lowest id is the contract, not whatever the planner steps first. Both service callers (gsd-state `import` and `import_jsonl`) pass their one clock reading. Tests: the direct store probe test covers live-until-expiry, the `==` now boundary, long-unswept, and lowest-id-wins; export_import.rs stores a TTL-1s twin of MEM1, advances the TestClock without sweeping, imports MEM1 (expires_at null) + MEM2 + a link, asserts (2, 0, 0, 1, 0, 0) and no edge on the stale row, then sweeps and asserts the memory and its edge survive. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The two Phase 5 service paths were wrapped by redact_storage_error but had no take_offline proof, so a refactor that unwrapped either would not have turned anything red. Rename `memories` away and call export into a Vec sink: the caller sees only the redacted "export" template. Rename `memory_links` away and import_jsonl a parsed two-memory, one-link file: pass 2 inserts, pass 3's store.link hits the missing table, and the caller sees only the redacted "import" template. Both assert no "sqlite" text. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…efore opening the DB (REVIEW IN-03) The wiring block in run_export was a copy of open_service, and the order meant `export --output <unwritable path>` first resolved, created and migrated the database and only then failed on create_new. Replace the copy with open_service and move the open inside the per-arm flow: on stdout it runs first; with --output it runs after export_file::begin, inside the one result that the existing abort covers, so a failed open never leaves the pending temp behind. Matches `import --from jsonl`, which validates before opening. CLI realism test: export --output into a missing directory with a fresh --db exits non-zero, names the output step on stderr, never creates the database, and leaves no temp under the tempdir. Verified to fail against the pre-fix run_export on the "never created" assertion. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Append the "Fixes applied (2026-10-25)" table: one row per finding with status, commit and the change made, plus the gate result the fixes were committed under. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…stop + live re-embed)

…ma realism run Plans the closure of both human_verification items in 05-VERIFICATION.md (status human_needed, 32/33) so the re-verifier can mark Phase 5 passed: - Task 1: held-out race test a_concurrent_link_committed_mid_export_cannot_ produce_a_dangling_key in tests/export_import.rs — a writer thread commits a memory + link while Store::export_rows is paused on its first memory record; the raced file parses with parse_jsonl (header counts == records, no dangling key), excludes both, and the next export includes both. Converts the 05-02 backstop truth 2.8 into explicit evidence (DIST-04). - Task 2: recorded live-Ollama realism run via the committed 05-05-realism.sh (gsd-state seed → export → jsonl import → serve-rest first-sweep backfill → hybrid paraphrase search), evidence into the SUMMARY and 05-VALIDATION.md (INTEROP-02). Smoke-run green on this host. ROADMAP: Phase 5 gains a Gap closure entry (4/5 executed); VALIDATION: rows 05-05-T1/T2, concurrency manual row retired in favour of the test, live re-embed row added. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

- acceptance grep anchors on the VALIDATION Status cell, not the Test-File-Exists column - <automated> verify checks the recorded SUMMARY/VALIDATION evidence instead of re-running the live Ollama flow Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…isible to that snapshot and present in the next (DIST-04 backstop truth) - tests/export_import.rs: Harness exposes `store: Arc<SqliteStore>` so a writer thread and the raced `Store::export_rows` call share one store - new `a_concurrent_link_committed_mid_export_cannot_produce_a_dangling_key`: a std::thread writer on the writer lane inserts a memory and links it while the exporter's read-pool snapshot is paused inside the visitor on the first memory record; ordering is decided only by two mpsc hand-offs, every wait bounded by `recv_timeout(10 s)` so a blocked write fails instead of hanging - proves: the write commits while the snapshot is open (fresh connection sees 4 rows / 3 edges), the snapshot still reports 3 / 2 with no leaked key, the raced file parses with `parse_jsonl` (exact header counts), and the next export carries both the memory and the `caused_by` edge - closes 05-VERIFICATION.md human item 1 (truth 2.8); no production change Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ep backfill → hybrid paraphrase search (INTEROP-02) - 05-VALIDATION.md: 05-05-T1 and 05-05-T2 rows flipped to green with the run date; the Manual-Only live re-embed row now carries `Result: PASS` (search_mode hybrid, target rank 0, id 1, backfilled=3 on the first tick) - run: bash .planning/phases/05-portable-export-import/05-05-realism.sh (exit 0, `PASS: live Ollama realism run`) against Ollama 0.35.0 with nomic-embed-text:latest on this host; script checks untouched - closes 05-VERIFICATION.md human item 2; no code change Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…lism evidence recorded

…hases 4–5, current state, key decisions Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

hermanngeorge15 and others added 30 commits October 25, 2026 11:11

docs(05): capture phase context

2544dd9

docs(state): record phase 5 context session

d46d83c

docs(05): research portable export/import phase domain

ae657d7

docs(phase-5): add validation strategy

b90e833

docs(05): create phase plan — 4 plans (D-14 redaction + store contrac…

e45113c

…t, export tracer, import tracer, docs + CLI realism) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs(05-01): complete redaction + store-contract plan

3ca3138

docs(05-02): complete export tracer + atomic --output plan

2dcf27d

docs(05-03): complete import --from jsonl plan

dc7822c

docs(05-04): complete contract docs + CLI realism plan

033bd86

docs(05): add code review report

5acea04

docs(05): record review fixes in 05-REVIEW.md

efcc1dd

Append the "Fixes applied (2026-10-25)" table: one row per finding with status, commit and the change made, plus the gate result the fixes were committed under. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs(05): verification report — 32/33, human_needed (concurrency back…

f3d1568

…stop + live re-embed)

hermanngeorge15 and others added 4 commits October 25, 2026 14:39

docs(05-05): complete gap-closure plan — race proof + live-Ollama rea…

fe18db7

…lism evidence recorded

docs(phase-5): complete phase execution — verification passed 35/35

3bf0f09

docs(project): evolve PROJECT.md after Phase 5 — validated reqs for P…

7785ad1

…hases 4–5, current state, key decisions Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

hermanngeorge15 merged commit b1053c5 into main Oct 25, 2026
1 check passed

hermanngeorge15 deleted the feat/phase-05-export-import branch October 25, 2026 14:00

hermanngeorge15 mentioned this pull request Oct 25, 2026

Launch v0.1.0: release, rename crate, directories (2026-10-24 plan) #7

Open

6 tasks

Navigation Menu

Repository navigation

feat: Phase 5 — portable JSONL export and import - #17

feat: Phase 5 — portable JSONL export and import#17
hermanngeorge15 merged 34 commits into
mainfrom
feat/phase-05-export-import

hermanngeorge15 commented Oct 25, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

hermanngeorge15 commented Oct 25, 2026

What changes

Verification

Not in this PR

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant