diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 8dd6efc..6d82cad 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -11,7 +11,7 @@ "displayName": "Agent Stack", "source": "./plugins/agent-stack", "description": "Two skills: agent-orchestrator — tool-calling loops, multi-stage pipelines with checkpoints, provider routing with fallback, four-layer memory, context engineering, plus the wallet side of reselling LLM access; and agent-evals — run/trace/thread evals, judges, and fixtures grown from production.", - "version": "0.15.0", + "version": "0.16.0", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/CHANGELOG.md b/CHANGELOG.md index 42214e9..5108c80 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,45 @@ # Changelog +## v0.16.0 — the whole survey, not just its taxonomy + +v0.15.0 took the taxonomy and the named failure modes. This takes the rest: the write +path and the landscape, as two references under the spine that already exists. + +- **`memory-lifecycle.md` — how an entry is made, changed and thrown away.** Five formation + operations with the cost of each stated rather than implied: semantic summarization is + *lossy by design* and wrong for evidence-critical tasks; structured construction buys + multi-hop and pays schema rigidity; latent is a black box; parametric cannot be precisely + removed. Summarization's two shapes fail differently — incremental drifts because each + summary is built from the last, partitioned loses cross-partition dependencies — and + **summarizing by fixed window is partitioned summarization with the worst partition rule**. +- **Updating is not consolidation.** One resolves conflict, the other abstracts, and a + system needs both. The field's own trajectory is worth copying rather than rediscovering: + early systems deleted the superseded entry and broke temporal continuity; the better + pattern is **temporal annotation** — mark a validity window instead — which also makes the + stability–plasticity decision reversible, and it has no general answer. +- **Consolidation has a cost this pack did not state**: it risks information smoothing, and + the outlier it smooths away is often the entry worth keeping. +- **`memory-landscape.md` — build or adopt, measure, and what is not practice yet.** Around + twenty-five open-source frameworks give you an index and leave you the judgement: what + becomes a memory, when to retrieve, what to abstain on, what to demote. **Adopt for the + index, not for the judgement.** Frameworks are compared by the axes that separate them + rather than by a list that expires. +- Benchmarks are split into memory-oriented and long-horizon-that-stresses-memory, and the + first question is neither: it is `agent-evals`' question — **what fails if memory is + silently disabled?** A memory never queried and a memory that is empty score identically + on every benchmark; only the retrieval log separates them. +- Frontiers are marked as frontiers, with one exception acted on: **expose memory operations + as tools the agent calls.** It makes every memory decision legible in the trace, including + the decision NOT to retrieve — which is the silent failure this whole subject is about, + and which a background memory module cannot show you. +- Three things the pack deliberately does NOT implement are named as absent rather than + quietly added: frequency-based forgetting, temporal annotation, dual-phase updating. Each + is a real change to a mechanism in production, and a reference's job is to say what the + options are, not to rewrite `patterns.md` from a survey. +- The citation now has **one home**. The member's own validator refused three files each + repeating the pinned source — "a fact with two homes disagrees with itself on the first + edit" — so the spine holds it and the siblings name it. + ## v0.15.0 — memory architecture, and the axes the layer table does not have - **`agent-orchestrator/references/memory-architecture.md`** — form, function and dynamics diff --git a/package.json b/package.json index ac6b380..f5c4c56 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@ssheleg/agent-stack", - "version": "0.15.0", + "version": "0.16.0", "scripts": { "test": "python3 test/validate.py && python3 test/plant_guard_test.py" }, diff --git a/plugins/agent-stack/.claude-plugin/plugin.json b/plugins/agent-stack/.claude-plugin/plugin.json index 93bb4a3..4f32453 100644 --- a/plugins/agent-stack/.claude-plugin/plugin.json +++ b/plugins/agent-stack/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "agent-stack", "displayName": "Agent Stack", "description": "Two skills: agent-orchestrator — tool-calling loops, multi-stage pipelines with checkpoints, provider routing with fallback, four-layer memory, context engineering, plus the wallet side of reselling LLM access; and agent-evals — run/trace/thread evals, judges, and fixtures grown from production.", - "version": "0.15.0", + "version": "0.16.0", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/plugins/agent-stack/skills/agent-orchestrator/SKILL.md b/plugins/agent-stack/skills/agent-orchestrator/SKILL.md index 120f85b..400f901 100644 --- a/plugins/agent-stack/skills/agent-orchestrator/SKILL.md +++ b/plugins/agent-stack/skills/agent-orchestrator/SKILL.md @@ -244,7 +244,10 @@ answer built from nothing, in no error log. **Design a memory layer from [`references/memory-architecture.md`](references/memory-architecture.md)**, not from this -table. It also carries the context-budget trap, layer 0 carryover and workspace scale. +table — it also carries the context-budget trap, layer 0 carryover and workspace scale. +The write path is [`references/memory-lifecycle.md`](references/memory-lifecycle.md); +what to build on and measure with is +[`references/memory-landscape.md`](references/memory-landscape.md). ## 8. Self-Learning Feedback Loops Three cycles feed the memory layers, and they differ by what supplies the signal: a failed diff --git a/plugins/agent-stack/skills/agent-orchestrator/references/memory-architecture.md b/plugins/agent-stack/skills/agent-orchestrator/references/memory-architecture.md index d2f5c53..1e7abb5 100644 --- a/plugins/agent-stack/skills/agent-orchestrator/references/memory-architecture.md +++ b/plugins/agent-stack/skills/agent-orchestrator/references/memory-architecture.md @@ -30,6 +30,7 @@ the diversity and dynamics of contemporary agent memory systems."* - [8. Trustworthy memory](#8-trustworthy-memory) - [9. What this pack does NOT claim](#9-what-this-pack-does-not-claim) - [10. The checklist](#10-the-checklist) +- [11. The rest of it](#11-the-rest-of-it) ## The source, and what this file adds @@ -306,3 +307,20 @@ made by accident: 9. **Can a person see, correct and delete what is held about them?** (§8) 10. **What eval fails if memory is silently disabled?** If none, the memory layer is unmeasured and its value is a belief. (`agent-evals`) + +## 11. The rest of it + +This file is the decision spine and it deliberately stops at decisions. Two siblings carry +the depth, both pinned to the same survey and the same read date: + +- [`memory-lifecycle.md`](memory-lifecycle.md) — the **write path**. Five ways to turn + experience into an entry and how each one fails, consolidation at three granularities, + why temporal annotation beats deletion when a fact is superseded, dual-phase updating, + and the three forgetting policies. It ends with a table of what this pack already + implements and three things it deliberately does not. +- [`memory-landscape.md`](memory-landscape.md) — **build or adopt**, the axes that actually + separate the twenty-odd open-source memory frameworks, which benchmarks measure what, and + which frontier ideas are directions rather than practice. Its last section is the one + frontier worth acting on now: exposing memory operations as tools the agent calls, which + makes every memory decision visible in the trace — including the decision not to + retrieve. diff --git a/plugins/agent-stack/skills/agent-orchestrator/references/memory-landscape.md b/plugins/agent-stack/skills/agent-orchestrator/references/memory-landscape.md new file mode 100644 index 0000000..2c70fac --- /dev/null +++ b/plugins/agent-stack/skills/agent-orchestrator/references/memory-landscape.md @@ -0,0 +1,128 @@ +# Memory landscape — what exists to build on, what to measure with, where it is going + +**Load this when** deciding whether to build a memory layer or adopt one, choosing what to +evaluate it against, or judging whether a technique somebody is proposing is settled +practice or a research direction. + +**Source:** the survey pinned in [`memory-architecture.md`](memory-architecture.md) +— §6 resources, §7 positions and frontiers. That file is its one home; repeating the citation here would give it two. + +**Read the dates on everything here.** This is the fastest-moving part of the subject, and +a landscape file is stale the day after it is written. What does not go stale is the +*shape*: which axes the frameworks differ on, and which of the frontier ideas is a +direction rather than a practice. + +## Contents + +- [1. Build or adopt](#1-build-or-adopt) +- [2. Frameworks, by what actually separates them](#2-frameworks-by-what-actually-separates-them) +- [3. Evaluating a memory layer](#3-evaluating-a-memory-layer) +- [4. Frontiers — and which of them is not practice yet](#4-frontiers--and-which-of-them-is-not-practice-yet) +- [5. The one that changes how you build today](#5-the-one-that-changes-how-you-build-today) + +## 1. Build or adopt + +An ecosystem of open-source memory frameworks exists — the survey tabulates around +twenty-five, including MemGPT, Mem0, Memobase, MemoryOS, MemOS, Zep, LangMem, Cognee, +Memary, MIRIX, and vector stores used as memory (Pinecone, Chroma, Weaviate). + +**What most of them give you** is storage plus retrieval: a vector or graph database, an +API, a short/long-term split. **What most of them leave to you** is the part that decides +whether the memory works — *"they often leave agent behavior and evaluation protocols to +the application."* + +So the honest framing of the choice: + +| | Adopting a framework | Building on your own store | +|---|---|---| +| You get | schema, indexing, retrieval, a short/long split | nothing you did not write | +| You still own | formation policy, what enters at all, forgetting policy, retrieval timing, **evals** | the same list | +| Argues for it | not writing an index; graph or temporal support you would not build | your entries are few and structured, and a table plus a query is genuinely enough | + +**The decision is usually smaller than it looks.** The parts that go wrong — deciding what +becomes a memory, when to retrieve, what to abstain on, what to demote — are yours in both +columns. Adopt for the index, not for the judgement. + +## 2. Frameworks, by what actually separates them + +Rather than a list that expires, the axes that distinguish them — check a candidate on +these: + +- **Does it model experiential memory, or only factual?** Several store facts well and have + no notion of a strategy learned from a trajectory. If the agent is supposed to improve, + this is the axis that matters. +- **What is the structure?** Hierarchical short/long-term, graph, temporal knowledge graph, + flat vectors, profile-based. This decides which queries are cheap. (§3 of + `memory-architecture.md`.) +- **Is temporal validity first-class?** A store that can express *"this was true until + March"* supports soft updating; one that cannot will make you delete history to stay + correct. +- **Does it report results on memory benchmarks at all?** Most do not. Absence is not + failure, but a framework with published LoCoMo or LongMemEval numbers has at least been + measured by someone. +- **Multimodal?** Most are text-only. The survey is explicit that **no system provides + truly omnimodal support** yet. + +## 3. Evaluating a memory layer + +The survey groups benchmarks two ways, and the split is the useful part. + +**Memory-oriented** — built to test memory directly. `LoCoMo` and `LongMemEval` are the two +most frequently reported and the closest thing to a common yardstick. `PersonaMem`, +`PrefEval`, `MemoryBank`, `PerLTQA`, `MPR` stress user modelling and preference tracking. +`StreamBench`, `LifelongAgentBench`, `MemoryAgentBench`, `Evo-Memory` test lifelong and +self-evolving behaviour — new information arriving while old information becomes obsolete +or conflicting. `HaluMem` targets memory-induced hallucination specifically. + +**Long-horizon agent benchmarks that stress memory implicitly** — `SWE-Bench Verified`, +`GAIA`, `WebArena`, `ToolBench`, `ALFWorld`. Memory is not the measured target, but +performance depends on it. + +**Which to use is not the first question.** The first question is the one `agent-evals` +asks: **what fails if memory is silently disabled?** A public benchmark measures a system +against a distribution that is not yours. A fixture built from your own traces, where the +answer is obtainable *only* from memory, catches the failure mode this whole subject is +about — the agent that skips retrieval and answers confidently from nothing +(`memory-architecture.md` §5.1). Build that fixture first; reach for a public benchmark +when you need to compare against somebody else's system. + +**Instrument the negatives.** A memory never queried and a memory that is empty score +identically on every benchmark. Only the retrieval log separates them. + +## 4. Frontiers — and which of them is not practice yet + +Marked plainly, because the risk with a survey's frontier section is building on a +direction as though it were a technique. + +| Frontier | State | What it would change | +|---|---|---| +| **Memory generation** (over retrieval) | direction | memory synthesized on demand for the current task rather than fetched and concatenated. Two shapes: *retrieve-then-generate*, which is buildable now and is essentially §5.4 post-retrieval taken seriously; and *direct generation* with no retrieval step, which is research | +| **Automated memory management** | early | the agent reasons about its own memory through **explicit tool calls** — add/update/delete/retrieve as actions in its loop rather than a module beside it. See §5 | +| **RL-driven memory** | research | the progression is RL-free → RL for selected operations (reranking, the write policy) → fully learned. Most production systems are and will remain RL-free | +| **Multimodal memory** | partial | vision is furthest along, audio underexplored, and **no omnimodal system exists** | +| **Shared memory for multi-agent** | early | from isolated stores with message passing, through naive global stores, toward **role- and trust-aware** access. `memory-architecture.md` §7 | +| **Trustworthy memory** | **requirement, not frontier** | privacy, explainability, hallucination robustness. `memory-architecture.md` §8 | +| **Offline consolidation** ("sleep") | direction | a dedicated interval to reorganize, prune and replay, away from latency constraints. The buildable half of it is the **dual-phase update** in `memory-lifecycle.md` §4 | + +## 5. The one that changes how you build today + +Of everything above, **automated memory management via explicit tool calls** is the item +worth acting on now, and it costs little. + +Instead of a memory module that runs beside the agent — summarizing on a timer, evicting on +a policy, retrieving on every turn — expose memory operations as **tools the agent calls**: +`memory.search`, `memory.write`, `memory.update`, `memory.forget`. Three consequences, and +they are what make it worth doing rather than an aesthetic preference: + +1. **The decisions become legible.** Every memory operation is a tool call in the trace, + which means `agent-evals` can judge them and the retrieval log from + `memory-architecture.md` §5.1 exists for free — including the negatives, because a turn + with no `memory.search` call is visibly a turn that chose not to retrieve. +2. **The agent can reason about them** — retrieve, find it insufficient, decompose and + retrieve again, which is the fast–slow pattern that the timing stage needs anyway. +3. **They inherit everything the tool layer already has** — descriptions the model can act + on (`agent-harness`), error hierarchies, budgets and loop guards (`SKILL.md` §2). + +**The cost, so it is a choice:** more model calls, and a model that can decline to use +memory at all. That second one is the silent failure mode again, which is why point 1 +matters — with tool calls you can *see* it happen, and with a background module you cannot. diff --git a/plugins/agent-stack/skills/agent-orchestrator/references/memory-lifecycle.md b/plugins/agent-stack/skills/agent-orchestrator/references/memory-lifecycle.md new file mode 100644 index 0000000..255b43b --- /dev/null +++ b/plugins/agent-stack/skills/agent-orchestrator/references/memory-lifecycle.md @@ -0,0 +1,154 @@ +# Memory lifecycle — how an entry is made, changed and thrown away + +**Load this when** writing the code that decides what becomes a memory, how a new entry +meets the ones already there, and what leaves. `memory-architecture.md` decides *what kind* +of memory is being built and owns retrieval; this file is the **write path**. + +**Source:** the survey pinned in [`memory-architecture.md`](memory-architecture.md) +— §5.1 formation, §5.2 evolution. That file is its one home; repeating the citation here would give it two. + +The three operators run at **different frequencies**, and that is the design. Formation can +run per turn, evolution per task boundary or offline, retrieval per call. Short- and +long-term behaviour is a consequence of those frequencies, not of separate boxes. + +## Contents + +- [1. Formation — five ways to turn experience into an entry](#1-formation--five-ways-to-turn-experience-into-an-entry) +- [2. Choosing among them](#2-choosing-among-them) +- [3. Evolution — consolidation](#3-evolution--consolidation) +- [4. Evolution — updating, and the stability–plasticity dilemma](#4-evolution--updating-and-the-stabilityplasticity-dilemma) +- [5. Evolution — forgetting](#5-evolution--forgetting) +- [6. What this pack already implements](#6-what-this-pack-already-implements) + +## 1. Formation — five ways to turn experience into an entry + +*"Instead of passively logging all interaction history, the memory system selectively +identifies information with long-term utility."* Five operations, and they compose — one +system commonly runs several. + +| Operation | What it produces | Strength | The cost, stated | +|---|---|---|---| +| **Semantic summarization** | a compact narrative of a long stream | drastically shorter context; ideal for long dialogue | **lossy by design** — specific details and subtle cues get smoothed out, so it is wrong for evidence-critical tasks | +| **Knowledge distillation** | discrete reusable facts or strategies | fine-grained, function-specific | produces flat units with no relation between them | +| **Structured construction** | a graph or tree — entities, relations, hierarchy | explainability and multi-hop queries | **schema rigidity**; extraction and maintenance cost is high, and nuance that does not fit the schema is lost | +| **Latent representation** | vectors or KV states | high density, cross-modal, no decoding loss | **a black box** — cannot be inspected, edited or verified by a person | +| **Parametric internalization** | changed weights | zero retrieval cost, "instinctive" access | catastrophic forgetting, high update cost, and **cannot be precisely removed** | + +### Summarization has two shapes and they fail differently + +- **Incremental** — fuse each new chunk into the running summary. Supports streaming and + avoids reprocessing the whole history. Fails by **semantic drift**: errors compound + across iterations, because each summary is built from the last one. +- **Partitioned** — summarize segments independently, then aggregate. Finer-grained and + parallelizable. Fails by **losing cross-partition dependencies**, and by cutting at + arbitrary boundaries unless segments are chosen semantically rather than by length. + +If you summarize by fixed window size, you have chosen partitioned summarization with the +worst possible partition rule. + +### Distillation splits by what it is distilling + +- **Factual** — dialogue turns into stated facts, user intent, environment state. Watch for + *goal drift*: separate confirmed constraints from unresolved intents, or the agent starts + treating a floated idea as a decision. +- **Experiential** — strategies from trajectories. **From contrast, not from success.** The + survey's own split confirms what §8 of `SKILL.md` already requires: success-only + distillation learns the agent's habits, and the systems that work compare successful and + failed runs, or reflect against ground truth. + +## 2. Choosing among them + +The question is not which is best. It is **what the memory will be asked for later**: + +| If the later question is… | Form it now with | +|---|---| +| "what happened, roughly" | semantic summarization | +| "what is true about X" | knowledge distillation, factual | +| "how do I do this kind of task" | knowledge distillation, experiential | +| "what connects to what" / multi-hop | structured construction | +| "match this to anything similar" at volume | latent | +| "behave this way, always, with no lookup" | parametric — and read §2 of `memory-architecture.md` on why this is rarely the answer | + +**A schema chosen before this question has been asked is a guess.** The commonest and most +expensive version of that guess is a knowledge graph built because graphs sound thorough, +then maintained for multi-hop queries nobody runs. + +## 3. Evolution — consolidation + +Merging new entries with existing ones so learning is cumulative rather than a growing pile +of near-duplicates. Three granularities: + +- **Local** — a new entry retrieves its top-K nearest and a model decides whether to merge. + Cheapest, and the one to build first. +- **Cluster-level** — align a new cluster with similar existing clusters and fuse. Captures + regularities across instances that local merging cannot see. +- **Global integration** — periodically distil system-level insight from the whole store. + +**The cost of consolidation, which is the reason not to run it aggressively:** +*"it risks information smoothing, where outlier events or unique exceptions are lost during +the abstraction process."* The exception is often the entry worth keeping — see §5. + +## 4. Evolution — updating, and the stability–plasticity dilemma + +Updating resolves *conflict*; consolidation performs *abstraction*. They are different +operations and a system needs both. + +**The trajectory the field took, and it is worth copying rather than rediscovering:** +early systems detected a conflict and **replaced or deleted** the old entry — destructive, +and it erased historical context and broke temporal continuity. The better pattern is +**temporal annotation**: mark the superseded fact with a validity window instead of +deleting it. Soft, time-aware updating keeps both semantic consistency and history. + +**Dual-phase updating** is the shape that survives real load: a soft online update for +responsiveness, then an offline reflective pass that merges similar entries and resolves +conflicts properly. Eventual consistency, deliberately — because doing full reflective +consolidation inline puts a model call on the write path of every interaction. + +**The dilemma has no general answer:** *"determining when to overwrite existing knowledge +versus when to treat new information as noise. Incorrect updates can overwrite critical +information."* What a system can do is make the decision **reversible** — which is the +argument for annotation over deletion, again. + +## 5. Evolution — forgetting + +Three policies on three different signals — creation time, access frequency, judged +importance. They are orthogonal and most systems need more than one. + +| Policy | Signal | Watch for | +|---|---|---| +| **Time-based** | age | evicting on age alone drops stable facts that were simply written early | +| **Frequency-based** | reads | **the long-tail trap** — see below | +| **Importance-driven** | a composite, increasingly a model's judgement of salience | the judge becomes a dependency, and an unaudited judge silently sets policy | + +**The long-tail trap, stated because frequency-based eviction is the easy one to reach +for:** *"heuristic forgetting mechanisms like LRU may eliminate long-tail knowledge, which +is seldom accessed but essential for correct decision-making."* The entry read twice a year +is often the incident, the exception, the one customer whose setup differs — precisely the +entry that prevents an expensive mistake. + +**The rule:** where storage is not the binding constraint, **demote rather than delete**. +Move it out of the default retrieval path, keep it reachable by explicit query. The survey +reports this is what many systems do in practice: *"when storage cost is not a critical +constraint, many memory systems avoid directly deleting certain memories."* + +Deletion remains a **correctness and privacy** operation — a person asking to be forgotten +is not a capacity decision, and `memory-architecture.md` §8 covers it. + +## 6. What this pack already implements + +Stated so this file is read as an extension and not as a replacement: + +| Survey concept | Where it already lives here | +|---|---| +| Formation, experiential, from contrast | `SKILL.md` §8 · `patterns.md` → Learning Extraction Heuristics | +| Consolidation, local | `patterns.md` → Fuzzy Deduplication | +| Updating, conflict resolution | `patterns.md` → Conflict Resolution | +| Forgetting, time-based | `patterns.md` → Confidence Management | +| Global integration, cross-scope | `patterns.md` → Cross-Resource Learning Transfer | +| **Frequency-based forgetting** | **nowhere — and the long-tail trap above is why that is a deliberate omission rather than a gap to close carelessly** | +| **Temporal annotation instead of deletion** | **nowhere** — Conflict Resolution currently resolves rather than annotates | +| **Dual-phase updating** | **nowhere** — the pack updates inline | + +The last three are named as absent rather than quietly added: each is a real change to a +mechanism that is in production, and this file's job is to say what the options are, not to +change `patterns.md` from a survey.