feat: Phase 4 — memory_update, typed relations, hybrid RRF search

.planning/REQUIREMENTS.md

@@ -10,13 +10,13 @@ Each maps to a roadmap phase. IDs continue from the v1.0 numbering (archived in
 
 ### Search (SEARCH)
 
-- [ ] **SEARCH-04**: With Ollama available, `memory_search` fuses the keyword and semantic legs via reciprocal-rank fusion (rank-based, k=60, duplicate ids sum contributions from both legs); results carry `search_mode: "hybrid"`. With Ollama unreachable, search degrades to keyword exactly as in v1.0 — the envelope shape is unchanged and existing `search_mode == "keyword"` checks keep working
-- [ ] **SEARCH-05**: Hybrid ranking remains decay-aware — a post-fusion decay blend so a fresh memory outranks a stale one at equal fused rank, and access bumps apply only to post-truncation returned ids
+- [x] **SEARCH-04**: With Ollama available, `memory_search` fuses the keyword and semantic legs via reciprocal-rank fusion (rank-based, k=60, duplicate ids sum contributions from both legs); results carry `search_mode: "hybrid"`. With Ollama unreachable, search degrades to keyword exactly as in v1.0 — the envelope shape is unchanged and existing `search_mode == "keyword"` checks keep working
+- [x] **SEARCH-05**: Hybrid ranking remains decay-aware — a post-fusion decay blend so a fresh memory outranks a stale one at equal fused rank, and access bumps apply only to post-truncation returned ids
 
 ### MCP Interface (MCP)
 
-- [ ] **MCP-06**: Agent can call `memory_update` to patch an existing memory (content, tags, scope, TTL) by id; a content update re-mirrors FTS and invalidates/re-embeds the vector **in the same writer transaction** (embed-outage falls back to `embedding_status = 0` + existing sweep backfill); unknown id returns the clean not-found shape
-- [ ] **MCP-07**: Agent can link two memories with a flat typed relation (`memory_link` / `memory_unlink`: from_id, to_id, kind) and search/list results can expand 1-hop related memories; forgetting a memory cascades its links (no orphan edges, no graph/entity model)
+- [x] **MCP-06**: Agent can call `memory_update` to patch an existing memory (content, tags, scope, TTL) by id; a content update re-mirrors FTS and invalidates/re-embeds the vector **in the same writer transaction** (embed-outage falls back to `embedding_status = 0` + existing sweep backfill); unknown id returns the clean not-found shape
+- [x] **MCP-07**: Agent can link two memories with a flat typed relation (`memory_link` / `memory_unlink`: from_id, to_id, kind) and search/list results can expand 1-hop related memories; forgetting a memory cascades its links (no orphan edges, no graph/entity model)
 
 ### REST & Input Hardening (API)
 
@@ -25,7 +25,7 @@ Each maps to a roadmap phase. IDs continue from the v1.0 numbering (archived in
 
 ### Storage & Migration Hygiene (STORE)
 
-- [ ] **STORE-05**: Schema migration 0003 (relations) ships with migration hygiene: shipped 0001/0002 SQL frozen (fixture divergence test against a v0.0.1 database), pre-migration backup, and a friendly error when a newer DB meets an older binary
+- [x] **STORE-05**: Schema migration 0003 (relations) ships with migration hygiene: shipped 0001/0002 SQL frozen (fixture divergence test against a v0.0.1 database), pre-migration backup, and a friendly error when a newer DB meets an older binary
 
 ### Portability (DIST / INTEROP)
 
@@ -64,11 +64,11 @@ Mapped by roadmap creation 2026-07-12. Coverage: 12/12 v1.1 requirements mapped
 |-------------|-------|--------|
 | API-02 | Phase 3 | Complete |
 | API-03 | Phase 3 | Complete |
-| MCP-06 | Phase 4 | Pending |
-| MCP-07 | Phase 4 | Pending |
-| STORE-05 | Phase 4 | Pending |
-| SEARCH-04 | Phase 4 | Pending |
-| SEARCH-05 | Phase 4 | Pending |
+| MCP-06 | Phase 4 | Complete |
+| MCP-07 | Phase 4 | Complete |
+| STORE-05 | Phase 4 | Complete |
+| SEARCH-04 | Phase 4 | Complete |
+| SEARCH-05 | Phase 4 | Complete |
 | DIST-04 | Phase 5 | Pending |
 | INTEROP-02 | Phase 5 | Pending |
 | DIST-05 | Phase 6 | Pending |

.planning/ROADMAP.md

@@ -27,7 +27,7 @@ Full phase details: [milestones/v1.0-ROADMAP.md](milestones/v1.0-ROADMAP.md)
 - Decimal phases (3.1, 3.2): Urgent insertions (marked INSERTED)
 
 - [x] **Phase 3: API Hardening & Toolchain Spikes** - Out-of-bounds limit/ttl rejected as client errors and exact tag matching at the shared service seam, plus early Windows/musl cross-compile spikes answering the milestone's only feasibility unknown (completed 2026-07-12)
-- [ ] **Phase 4: Memory Update, Relations & Hybrid Search** - `memory_update` with same-transaction re-embed, flat typed links with cascade, migration 0003 hygiene, and decay-aware hybrid RRF as a parallel track
+- [x] **Phase 4: Memory Update, Relations & Hybrid Search** - `memory_update` with same-transaction re-embed, flat typed links with cascade, migration 0003 hygiene, and decay-aware hybrid RRF as a parallel track (completed 2026-10-23)
 - [ ] **Phase 5: Portable Export & Import** - Versioned JSONL export (link edges included, embeddings excluded) and idempotent cross-machine import with sweep-backfill re-embedding
 - [ ] **Phase 6: Distribution & v0.1.0 Release** - musl + Windows binaries on the spike-chosen toolchains and the checksummed v0.1.0 release with Homebrew update and behavioral-change release notes
 
@@ -67,23 +67,29 @@ Plans:
   4. Hybrid ranking stays decay-aware — a fresh memory outranks a stale one at equal fused rank — and access bumps apply only to post-truncation returned ids.
   5. Upgrading a real v0.0.1 database applies migration 0003 only after a pre-migration backup; shipped 0001/0002 SQL is provably unchanged (fixture divergence test against a v0.0.1 database), and a newer-schema DB met by an older binary yields a friendly error, not a crash.
 
-**Plans**: 4 plans
+**Plans**: 7/7 plans executed (4 executed + 3 gap closure)
 Plans:
 **Wave 1**
 
-- [ ] 04-01-PLAN.md — Error foundation (Internal tier, NotFound removal, CR-01 dedup, MAX_KNN_K decouple) + migration 0003 with backup/fixture-freeze/too-new hygiene (STORE-05)
+- [x] 04-01-PLAN.md — Error foundation (Internal tier, NotFound removal, CR-01 dedup, MAX_KNN_K decouple) + migration 0003 with backup/fixture-freeze/too-new hygiene (STORE-05)
 
 **Wave 2** *(blocked on Wave 1 completion)*
 
-- [ ] 04-02-PLAN.md — memory_update end-to-end: merge-patch semantics, same-tx FTS+vector re-embed, PATCH /api/memories/{id} (MCP-06)
+- [x] 04-02-PLAN.md — memory_update end-to-end: merge-patch semantics, same-tx FTS+vector re-embed, PATCH /api/memories/{id} (MCP-06)
 
 **Wave 3** *(blocked on Wave 2 completion)*
 
-- [ ] 04-03-PLAN.md — memory_link/memory_unlink with cascade + opt-in 1-hop expansion on search/list (MCP-07)
+- [x] 04-03-PLAN.md — memory_link/memory_unlink with cascade + opt-in 1-hop expansion on search/list (MCP-07)
 
 **Wave 4** *(blocked on Wave 3 completion)*
 
-- [ ] 04-04-PLAN.md — Decay-aware hybrid RRF fusion with keyword degrade, error taxonomy, and wire/docs (SEARCH-04, SEARCH-05)
+- [x] 04-04-PLAN.md — Decay-aware hybrid RRF fusion with keyword degrade, error taxonomy, and wire/docs (SEARCH-04, SEARCH-05)
+
+**Gap closure** *(from 04-VERIFICATION.md + 04-REVIEW.md; gap waves run in order)*
+
+- [x] 04-05-PLAN.md — Gap wave 1: forgiving keyword leg per user decision option-a (raw MATCH first, one sanitized OR-of-quoted-tokens retry, tokenless = empty leg, identical in both modes) + search-path error redaction (CR-01, CR-02, truths #14/#15, IN-04) (SEARCH-04)
+- [x] 04-06-PLAN.md — Gap wave 2: conditional sweep backfill closing the update/forget/sweep races + symmetric relates_to + IN-05/IN-06 docs (CR-03, WR-03) (MCP-06, MCP-07)
+- [x] 04-07-PLAN.md — Gap wave 3: deny_unknown_fields on update DTOs, uniform REST rejection bodies, race-safe verify-then-rename pre-migration backup + Upgrading docs (WR-02, IN-02, WR-01, IN-03, IN-01) (MCP-06, MCP-07, STORE-05)
 
 ### Phase 5: Portable Export & Import
 
@@ -124,6 +130,6 @@ Phases execute in numeric order: 3 → 4 → 5 → 6
 | 1. Core Memory Foundation | v1.0 | 3/3 | Complete | 2026-06-25 |
 | 2. Semantic Search, Interop & Release | v1.0 | 5/5 | Complete | 2026-07-12 |
 | 3. API Hardening & Toolchain Spikes | v1.1 | 2/2 | Complete    | 2026-07-12 |
-| 4. Memory Update, Relations & Hybrid Search | v1.1 | 0/? | Not started | - |
+| 4. Memory Update, Relations & Hybrid Search | v1.1 | 7/7 | Complete    | 2026-10-23 |
 | 5. Portable Export & Import | v1.1 | 0/? | Not started | - |
 | 6. Distribution & v0.1.0 Release | v1.1 | 0/? | Not started | - |

.planning/STATE.md

@@ -2,16 +2,19 @@
 gsd_state_version: 1.0
 milestone: v1.1
 milestone_name: Hardening & Interop
-status: executing
-stopped_at: Phase 4 context gathered
-last_updated: "2026-07-13T13:42:43.486Z"
-last_activity: 2026-10-09 -- Completed quick task 261009-hc1 (hosted release pipeline, closes #1)
+current_phase: 5
+current_phase_name: Portable Export & Import
+status: planning
+stopped_at: Phase 04 complete, ready to plan Phase 5
+last_updated: "2026-10-23T20:25:00.455Z"
+last_activity: 2026-10-23
+last_activity_desc: Phase 04 complete, transitioned to Phase 5
+state_head: 4441bdafab5d420a1fbcf3ba348c0589d72bdc11
 progress:
   total_phases: 4
-  completed_phases: 1
-  total_plans: 2
-  completed_plans: 2
-  percent: 25
+  completed_phases: 2
+  total_plans: 9
+  completed_plans: 9
 ---
 
 # Project State
@@ -21,22 +24,22 @@ progress:
 See: .planning/PROJECT.md (updated 2026-07-12)
 
 **Core value:** An agent can persist a structured memory and retrieve the right one later — across sessions and tools — over a standard MCP interface, with no cloud.
-**Current focus:** Phase 04 — memory-update-relations-hybrid-search
+**Current focus:** Phase 04 — Memory Update, Relations & Hybrid Search
 
 ## Current Position
 
-Phase: 4
+Phase: 5 — Portable Export & Import
 Plan: Not started
-Status: Ready to execute
-Last activity: 2026-10-09 - Completed quick task 261009-hc1: hosted release pipeline with SLSA provenance (closes #1)
+Status: Ready to plan
+Last activity: 2026-10-23 - Completed quick task 261023-v6w: WR-04: reject memory_list expand_links without limit
 
 Progress: [░░░░░░░░░░] 0% (0/4 v1.1 phases)
 
 ## Performance Metrics
 
 **Velocity:**
 
-- Total plans completed: 13
+- Total plans completed: 17
 - Average duration: ~28 min
 - Total execution time: ~1.4 hours
 
@@ -47,6 +50,7 @@ Progress: [░░░░░░░░░░] 0% (0/4 v1.1 phases)
 | 1 | 3 | - | - |
 | 02 | 5 | - | - |
 | 03 | 2 | - | - |
+| 04 | 7 | - | - |
 
 **Recent Trend:**
 
@@ -61,6 +65,17 @@ Progress: [░░░░░░░░░░] 0% (0/4 v1.1 phases)
 | Phase 02 P05 | 8 min | 2 tasks | 6 files |
 | Phase 03 P01 | 12 min | 3 tasks | 11 files |
 | Phase 03 P02 | 30 min | 3 tasks | 2 files |
+**Per-Plan Metrics:**
+
+| Plan | Duration | Tasks | Files |
+|------|----------|-------|-------|
+| Phase 04 P01 | 9 min | 2 tasks | 13 files |
+| Phase 04 P02 | 10min | 2 tasks | 11 files |
+| Phase 04 P03 | 10 min | 2 tasks | 9 files |
+| Phase 04 P04 | 11 min | 2 tasks | 11 files |
+| Phase 04 P05 | 10 min | 3 tasks | 12 files |
+| Phase 04 P06 | 7 min | 3 tasks | 9 files |
+| Phase 04 P07 | 12 min | 3 tasks | 7 files |
 
 ## Accumulated Context
 
@@ -84,6 +99,30 @@ Recent decisions affecting current work:
 - [03-01]: import() validates ttl per draft through the shared helper — the store() bypass is closed structurally for Phase 5 JSONL import
 - [03-02]: Phase 6 toolchain verdict (D-11): all 4 spike legs GREEN in one dispatch — Windows = cargo-xwin 0.23.0 / x86_64-pc-windows-msvc (gnu mingw-w64 proven fallback); musl = zigbuild 0.23.0 + target-suffixed CFLAGS -Du_int*_t shim (both legs green, no DIST-05 blocker)
 - [03-02]: CI revived on GitHub-hosted ubuntu-latest, secretless contents: read (D-12/D-10, ecosystem D-02 exception) — supersedes inner CLAUDE.md 'never ubuntu-latest' for this public repo; release.yml runner decision stays with Phase 6
+- [Phase 04]: [04-01]: MemoryError::Internal(String) is the 500/internal_error tier for server faults (poisoned writer lock); MemoryError::NotFound and ApiError::NotFound deleted — not-found is data (Ok(false)/Ok(None)) only (WR-02)
+- [Phase 04]: [04-01]: Store::exists takes Option<&str> with NULL-safe 'source IS ?1' — source:None re-imports are idempotent (CR-01)
+- [Phase 04]: [04-01]: MAX_KNN_K = MAX_LIMIT * 4 (800) — candidate oversample cap, distinct from RRF k=60 (WR-01 partial; full filter-aware fix deferred)
+- [Phase 04]: [04-01]: SqliteStore::open order is probe user_version (before any PRAGMA, so a refused DB is byte-untouched) -> SchemaTooNew refusal -> pragmas -> sibling online backup {db}.backup-pre-{NNNN}-{YYYYMMDD} (one per target version, failure aborts) -> to_latest; LATEST_SCHEMA_VERSION=3 must be bumped with every new migration
+- [Phase 04]: [04-01]: tests/fixtures/v0.0.1.db is a committed binary fixture generated once from tag-identical 0001+0002 — never regenerate it; it is gitignored locally by *.db so stage with git add -f
+- [Phase 04]: 04-02: memory_update returns a dedicated UpdatedMemory (flattened MemoryView + embedding_status); MemoryView untouched so list/search envelopes stay byte-identical (D-10)
+- [Phase 04]: 04-02: an all-fields-absent patch is InvalidArgument, not a no-op; content null/empty rejected at the seam; TTL patches reuse validate_ttl (D-08)
+- [Phase 04]: 04-02: update is UPDATE-in-place (never delete+insert), folds the D-09 freshness bump and decay_score = 1.0 into the same statement, and swaps the vector in the same writer tx
+- [Phase 04]: 04-02: patch DTO fields carry skip_serializing_if so the MCP tool schema never advertises default: null (null means clear in a merge-patch)
+- [Phase 04]: 04-02: PATCH /api/memories/{id} maps every Json extractor rejection to 400 {error}; other REST handlers keep axum defaults
+- [Phase 04]: 04-03: Relations are flat directed edges (LinkKind relates_to|supersedes|caused_by, closed enum); duplicate link = idempotent Linked, self-link = InvalidArgument at the seam, missing endpoint = LinkOutcome::NotFound{missing_id} data (never an FK 500)
+- [Phase 04]: 04-03: forget() and sweep_expired() each carry an explicit orphan-edge DELETE beside the FK ON DELETE CASCADE, proven by pragma-less-connection tests
+- [Phase 04]: 04-03: expand_links attaches MemoryView.related (Option, skip_serializing_if) after truncation and after spawn_bump; MAX_EXPANDED_LINKS = 10 newest-first, snippet 160 chars; neighbors never bumped; flag-less envelope golden-tested byte-identical
+- [Phase 04]: 04-03: Incoming edges read natural reverse labels (superseded_by, caused); RelatedLink.direction is out|in; REST POST /api/links + DELETE /api/links/{from_id}/{to_id}/{kind} with 200/404 and 400 on any bad body/kind
+- [Phase 04]: 04-04: Hybrid search = RRF (k=60, 1-based ranks, duplicate ids sum, id tie-break) over two pure-relevance legs (KNN distance, bm25 ASC), each oversampled limit*4 capped at MAX_KNN_K; decay applied once, multiplicatively, post-fusion (SEARCH-04/05)
+- [Phase 04]: 04-04: SearchMode::Semantic removed; embedder-up searches report search_mode "hybrid", embedder-down stays "keyword" byte-identical to v1.0 (README behavioral change 3)
+- [Phase 04]: 04-04: Keyword-leg errors propagate first, so a malformed MATCH (incl. bare ?/,/' punctuation) is InvalidQuery/400 in hybrid as in keyword mode (T-04-15); follow-up candidate: sanitize natural-language queries for the keyword leg
+- [Phase 04]: 04-05: T-04-15 revised by user decision option-a (2026-10-23) — search runs the raw FTS5 MATCH first, then one sanitized retry (word tokens quoted and OR-joined) on any FTS5 query-parse error, including the step-time `no such column` class from hyphens/colons; a tokenless query is an empty keyword leg (hybrid answers semantic-only, keyword mode returns empty); identical in both modes, so search never returns InvalidQuery; every search-path internal error is redacted to SEARCH_STORAGE_FAILURE (T-04-19). Supersedes the 04-04 decision that a malformed MATCH is InvalidQuery/400 in hybrid mode.
+- [Phase 04]: 04-06: sweep backfill writes a vector only through a guarded claim (embedding_status = 0 AND content = embedded text) run first in the writer tx; Store::insert_embedding returns bool. Closes REVIEW CR-03, across processes too.
+- [Phase 04]: 04-06: relates_to stored canonically as (min id, max id) and unlinked from either end; supersedes/caused_by stay directed (D-03). Storage-only: MCP/REST responses echo request ids; NotFound names the request's from_id first. No migration (REVIEW WR-03).
+- [Phase 04]: 04-06: WR-04 resolved by user decision option-a (reject, 2026-10-23, quick task 261023-v6w) — memory_list with expand_links and no limit is a client error (MemoryError::InvalidArgument → REST 400 {error} / MCP invalid_params) whose message tells the caller to pass a limit; expand_links with a limit, a list without expand_links, and search (already bounded) are unchanged, so the 200 x 10 expansion bound now covers every expanded response. Supersedes the 04-06 doc-only deferral; closed, nothing left for Phase 5/6.
+- [Phase 04]: 04-07: deny_unknown_fields added to UpdateArgs/UpdateBody only (new v0.1.0 surfaces, REVIEW WR-02); the [03-01] decision stands for the v0.0.1 DTOs, so the v0.1.0 release-notes delta for existing surfaces is unchanged. Over MCP, rmcp 1.8 reports the rejection as a tool error result (isError: true, "failed to deserialize parameters: unknown field ..."), not a JSON-RPC -32602; REST is a 400 {"error"}.
+- [Phase 04]: 04-07: backup temp is pid-qualified and a process-wide BACKUP_LOCK serializes backups within one process, so same-process openers never share a temp; prune tolerates NotFound from a concurrent opener
+- [Phase 04]: 04-07: WR-01 closed with verify-then-rename + tolerate-by-re-probe (no file lock, no new dep); cross-process exclusive lock stays deferred
 
 ### Pending Todos
 
@@ -108,6 +147,7 @@ Watch items:
 | # | Description | Date | Commit | Directory |
 |---|-------------|------|--------|-----------|
 | 261009-hc1 | Release pipeline on GitHub-hosted runners with SLSA provenance, semver-only trigger, all actions SHA-pinned (closes #1); resolves the release-runner decision Phase 3 D-12 deferred to Phase 6 | 2026-10-09 | 9810e31 | [261009-hc1-move-release-yml-to-github-hosted-runner](./quick/261009-hc1-move-release-yml-to-github-hosted-runner/) |
+| 261023-v6w | WR-04: reject `memory_list` with `expand_links` and no `limit` as a client error (MCP `invalid_params`, REST 400); user decision option (a) | 2026-10-23 | 969e891 | [261023-v6w-wr-04-reject-memory-list-expand-links-wi](./quick/261023-v6w-wr-04-reject-memory-list-expand-links-wi/) |
 
 ## Deferred Items
 
@@ -124,9 +164,9 @@ Items acknowledged and carried forward:
 
 ## Session Continuity
 
-Last session: 2026-07-12T20:49:51.528Z
-Stopped at: Phase 4 context gathered
-Resume file: .planning/phases/04-memory-update-relations-hybrid-search/04-CONTEXT.md
+Last session: 2026-10-23T20:12:50.720Z
+Stopped at: Phase 04 complete, ready to plan Phase 5
+Resume file: None
 
 ## Operator Next Steps