From 57b55ed1f5b953c8da9b3804a91e1a354337ae66 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:21:51 +0000 Subject: [PATCH 01/23] feat: recall memory pages from both scopes beside notes Memory pages join the one recall the prompt router and `rules --prompt` already run for notes (spc-2610111627213138 step 3). A page whose frontmatter declares a `recall:` list is an item keyed page:, read from the project's .abcd/memory/ through the store handle and, read only, from ~/.abcd.noindex/memory/ through the home scope's guarded open, each machine page held to the guards the machine memory.json gets. A page in the project's store suppresses the machine's page of the same name, and pages follow notes in the id order until the ranking lands. The recall still opens only its four declared places: a test records every open through inotify, with a host memory folder of matching text under the home, and finds none there. A page or page store that cannot be read is named on stderr and in the diagnostic's unreadable list, and the rules and every other item still come back. Refs: itd-2610091918433290, itd-39 Assisted-by: Claude:claude-opus-5-5 --- commands/ahoy.md | 18 +- docs/reference/cli/commands.md | 7 +- internal/core/memory/ancestor_symlink_test.go | 13 ++ internal/core/memory/recallpages.go | 140 +++++++++++++ internal/core/memory/recallpages_test.go | 171 +++++++++++++++ internal/core/recall/items.go | 40 ++-- internal/core/recall/opens_linux_test.go | 194 ++++++++++++++++++ internal/core/recall/pages.go | 67 ++++++ internal/core/recall/pages_test.go | 169 +++++++++++++++ internal/core/recall/recall.go | 2 +- internal/core/recall/recall_test.go | 2 +- internal/surface/cli/cli.go | 24 ++- internal/surface/cli/memory_recall_test.go | 127 ++++++++++++ internal/surface/cli/rules_prompt.go | 69 +++++-- 14 files changed, 991 insertions(+), 52 deletions(-) create mode 100644 internal/core/memory/recallpages.go create mode 100644 internal/core/memory/recallpages_test.go create mode 100644 internal/core/recall/opens_linux_test.go create mode 100644 internal/core/recall/pages.go create mode 100644 internal/core/recall/pages_test.go diff --git a/commands/ahoy.md b/commands/ahoy.md index ef6eeddf00..4657831f9b 100644 --- a/commands/ahoy.md +++ b/commands/ahoy.md @@ -1213,12 +1213,18 @@ prompt in a fresh session, and writes nothing. `domains` lists each rule domain the prompt matches, with the recall words that hit (`hits`, or `*NAME` for a star activation); `notes` lists each memory note it recalls from `~/.abcd.noindex/memory.json` or the project's `.abcd/memory.json`, with -its `layer` (`machine` or `project`) and the recall words that hit. Each -carries a `form`: `full`, or `over-budget` when it would not fit the -per-turn budget and would come back on a later prompt instead. `disabled` -is true under the kill switch, and then nothing comes back; `unreadable` -names a `memory.json` that could not be read, and why. Use it when someone -asks why a rule or a note did, or did not, come back on a prompt. A +its `layer` (`machine` or `project`) and the recall words that hit; `pages` +lists each memory page it recalls from the project's `.abcd/memory/` or +`~/.abcd.noindex/memory/`, by its `key` (`page:`, the name the +hook's envelope gives it), with its `filename`, `title`, `layer` and the +recall words that hit. A page comes back only when its frontmatter declares +a `recall:` list, and a page in the project's store suppresses the machine's +page of the same name. Each carries a `form`: `full`, or `over-budget` when +it would not fit the per-turn budget and would come back on a later prompt +instead. `disabled` is true under the kill switch, and then nothing comes +back; `unreadable` names a `memory.json`, page store or page that could not +be read, why, and its `kind` (`note` or `page`). Use it when someone asks why +a rule, a note or a page did, or did not, come back on a prompt. A positional DOMAIN with `--prompt` is refused, exit 2: `rules ` renders one domain's rules instead. diff --git a/docs/reference/cli/commands.md b/docs/reference/cli/commands.md index b499cedf4d..ceb1c2313d 100644 --- a/docs/reference/cli/commands.md +++ b/docs/reference/cli/commands.md @@ -3656,15 +3656,16 @@ its .abcd/rules.json. --prompt "" shows what that one prompt would bring back to a fresh session: each rule domain it matches, with the recall words that hit, and each memory note it recalls from ~/.abcd.noindex/memory.json and .abcd/memory.json, -with its layer and the recall words that hit, each in full or as not fitting -the per-turn budget. It reads against an empty ledger and writes nothing. +and each memory page from .abcd/memory/ and ~/.abcd.noindex/memory/, with its +layer and the recall words that hit, each in full or as not fitting the +per-turn budget. It reads against an empty ledger and writes nothing. --prompt and a DOMAIN are two forms and are refused together, exit 2. Read-only. **Flags:** ``` - --prompt string show what this one prompt would bring back to a fresh session: the domains it matches and the memory notes it recalls + --prompt string show what this one prompt would bring back to a fresh session: the domains it matches and the memory notes and pages it recalls ``` ### `abcd scribe` diff --git a/internal/core/memory/ancestor_symlink_test.go b/internal/core/memory/ancestor_symlink_test.go index d7effa543a..e142147311 100644 --- a/internal/core/memory/ancestor_symlink_test.go +++ b/internal/core/memory/ancestor_symlink_test.go @@ -77,4 +77,17 @@ func TestMemoryStoreDirSymlinkRefused(t *testing.T) { if _, err := Bare(repoRoot); err == nil { t.Error("Bare followed a symlinked .abcd/memory store; the directory symlink must be refused") } + + // (d) The prompt router's recall read (spc-2610111627213138 step 3) must + // refuse it too, and disclose nothing of the target. + pages, refused := ReadRecallPages(repoRoot) + for _, p := range pages { + if strings.Contains(p.Body, secret) { + t.Fatalf("ReadRecallPages disclosed an out-of-repo page through the store symlink: %q", p.Filename) + } + } + var unsafe *UnsafeStorePathError + if len(refused) != 1 || !errors.As(refused[0].Err, &unsafe) { + t.Errorf("ReadRecallPages followed a symlinked .abcd/memory store (refused %+v); the directory symlink must be refused", refused) + } } diff --git a/internal/core/memory/recallpages.go b/internal/core/memory/recallpages.go new file mode 100644 index 0000000000..09615f8fbf --- /dev/null +++ b/internal/core/memory/recallpages.go @@ -0,0 +1,140 @@ +package memory + +import ( + "errors" + "fmt" + "io/fs" + "os" + "path/filepath" + "strings" + "syscall" + + "github.com/intentdriven/abcd/internal/abcdhome" + "github.com/intentdriven/abcd/internal/fsutil" +) + +// recallpages.go — the read the prompt router's memory recall makes of the +// pages (spc-2610111627213138 step 3): each page's filename, its index line's +// summary, its body and its recall: list, from the workspace store and, read +// only, from the machine's ~/.abcd.noindex/memory/. Nothing here writes. + +// RecallPage is one memory page as the recall reads it. +type RecallPage struct { + // Filename is the page's __.md name in its store. + Filename string + // Summary is the page's index.md summary, derived from the page as + // RenderIndex derives it, so the recall reads no index.md. + Summary string + // Body is the page's text after its frontmatter. + Body string + // Recall is the page's recall: frontmatter list, empty when it has none. + Recall []string +} + +// RecallRefusal names a scope or a page the read refused, and why. File is +// the store-relative or tilde form, never an expanded home. +type RecallRefusal struct { + File string + Err error +} + +// machineStoreDisplay is the machine page scope as a message names it. +var machineStoreDisplay = abcdhome.Display("memory") + "/" + +// workspaceStoreDisplay is the workspace page scope as a message names it. +const workspaceStoreDisplay = ".abcd/memory/" + +// ReadRecallPages reads the workspace store under repoRoot through the store +// handle: a symlinked or swapped store is refused whole, each page is read +// through the handle's guarded read. An absent store yields nothing. Only +// pages the index lists whose names are __.md are read. +func ReadRecallPages(repoRoot string) ([]RecallPage, []RecallRefusal) { + store, err := openStore(repoRoot) + if err != nil { + return nil, []RecallRefusal{{File: workspaceStoreDisplay, Err: err}} + } + defer store.Close() + return recallPagesFrom(store, workspaceStoreDisplay, store.read) +} + +// ReadMachineRecallPages reads the machine scope, ~/.abcd.noindex/memory/ +// under home, read only: the directory is opened through +// fsutil.OpenHomeScope, so a symlinked ~/.abcd.noindex or memory/ is refused, +// and each page through the declaration's guarded read relative to that +// descriptor, as the machine memory.json is read: a regular file this account +// owns that no one else can write. An absent scope, no absolute home, or a +// home or ~/.abcd.noindex this account cannot search yields nothing. +func ReadMachineRecallPages(home string) ([]RecallPage, []RecallRefusal) { + if home == "" || !filepath.IsAbs(home) { + return nil, nil + } + root, err := fsutil.OpenHomeScope(home, abcdhome.Rel("memory")) + if err != nil { + if errors.Is(err, fs.ErrNotExist) || errors.Is(err, syscall.ENOTDIR) || errors.Is(err, fs.ErrPermission) { + return nil, nil + } + return nil, []RecallRefusal{{File: machineStoreDisplay, Err: err}} + } + store := &storeHandle{dir: abcdhome.Path(home, "memory"), root: root} + defer store.Close() + read := func(name string, limit int64) ([]byte, error) { + data, _, err := fsutil.ReadHomeDeclarationDenyingIn(root, home, abcdhome.Rel("memory", name), limit, 0) + return data, err + } + return recallPagesFrom(store, machineStoreDisplay, read) +} + +// recallPagesFrom reads, through read, every page the store lists whose name +// is __.md; a page read refuses is named, never followed. +func recallPagesFrom(store *storeHandle, shown string, read func(string, int64) ([]byte, error)) ([]RecallPage, []RecallRefusal) { + var pages []RecallPage + var refused []RecallRefusal + for _, name := range store.pageNames() { + if _, _, _, ok := ParsePageFilename(name); !ok { + continue + } + raw, err := read(name, maxMemoryPageBytes) + if err != nil { + refused = append(refused, RecallRefusal{File: shown + name, Err: recallReadError(err)}) + continue + } + p := parsePage(string(raw)) + info := pageInfoOf(name, p) + pages = append(pages, RecallPage{Filename: name, Summary: info.Summary, Body: p.body, Recall: recallList(p.fm)}) + } + return pages, refused +} + +// recallList is a page's recall: list, its non-blank string entries trimmed. +func recallList(fm map[string]any) []string { + raw, ok := fm["recall"].([]any) + if !ok { + return nil + } + var out []string + for _, v := range raw { + if s, ok := v.(string); ok && strings.TrimSpace(s) != "" { + out = append(out, strings.TrimSpace(s)) + } + } + return out +} + +// recallReadError words a refused page read without the expanded path. +func recallReadError(err error) error { + switch { + case errors.Is(err, fsutil.ErrTooBig): + return fmt.Errorf("it exceeds the %d-byte page cap", maxMemoryPageBytes) + case errors.Is(err, fsutil.ErrNotRegular): + return errors.New("it is not a regular file") + case errors.Is(err, fsutil.ErrDeclarationWritable): + return errors.New("it is writable by others, so its words are not necessarily yours (chmod go-w it)") + case errors.Is(err, fsutil.ErrDeclarationForeignOwner): + return errors.New("it is not owned by this session's uid") + } + var pe *os.PathError + if errors.As(err, &pe) { + return pe.Err + } + return err +} diff --git a/internal/core/memory/recallpages_test.go b/internal/core/memory/recallpages_test.go new file mode 100644 index 0000000000..2d381e11c2 --- /dev/null +++ b/internal/core/memory/recallpages_test.go @@ -0,0 +1,171 @@ +package memory + +import ( + "errors" + "os" + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/intentdriven/abcd/internal/abcdhome" + "github.com/intentdriven/abcd/internal/fsutil" +) + +// recallPageText is a typed page with a recall: list. +func recallPageText(summary string, recall ...string) string { + return "---\nrecall: [" + strings.Join(recall, ", ") + "]\nsource:\n class: session_memory\n ingested_at: 2026-10-01\n---\n# " + summary + "\n\nThe body.\n" +} + +func writeStoreFile(t *testing.T, dir, name, text string, mode os.FileMode) { + t.Helper() + if err := os.MkdirAll(dir, 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, name), []byte(text), mode); err != nil { + t.Fatal(err) + } + if err := os.Chmod(filepath.Join(dir, name), mode); err != nil { + t.Fatal(err) + } +} + +// TestRecallPagesReadsTheIndexedPages: the recall's read yields each page's +// filename, its index line's summary, its body and its recall: list, for +// pages named __.md at the store's top level only. +func TestRecallPagesReadsTheIndexedPages(t *testing.T) { + repo := t.TempDir() + mem := filepath.Join(repo, ".abcd", "memory") + writeStoreFile(t, mem, "fact_eng_auth-flow.md", recallPageText("Auth uses short-lived tokens", "auth", "login"), 0o644) + writeStoreFile(t, mem, "fact_eng_silent.md", "---\nsource:\n class: session_memory\n---\n# No recall list\n", 0o644) + writeStoreFile(t, mem, "index.md", recallPageText("not a page", "auth"), 0o644) + writeStoreFile(t, mem, "notes.md", recallPageText("not a typed name", "auth"), 0o644) + writeStoreFile(t, filepath.Join(mem, "sources"), "fact_eng_nested.md", recallPageText("nested", "auth"), 0o644) + + pages, refused := ReadRecallPages(repo) + if len(refused) != 0 { + t.Fatalf("refused = %+v", refused) + } + var names []string + for _, p := range pages { + names = append(names, p.Filename) + } + if !slices.Equal(names, []string{"fact_eng_auth-flow.md", "fact_eng_silent.md"}) { + t.Fatalf("pages = %v", names) + } + p := pages[0] + if p.Summary != "Auth uses short-lived tokens" || !slices.Equal(p.Recall, []string{"auth", "login"}) || + !strings.Contains(p.Body, "The body.") || strings.Contains(p.Body, "recall:") { + t.Fatalf("page = %+v", p) + } + if len(pages[1].Recall) != 0 { + t.Fatalf("a page with no recall list read one: %+v", pages[1]) + } + if got, refused := ReadRecallPages(t.TempDir()); len(got) != 0 || len(refused) != 0 { + t.Fatalf("an absent store read %v %v", got, refused) + } +} + +// TestRecallPagesRefuseASymlinkedPage: a page committed as a symlink to a +// file outside the store is never followed — it is not listed as a page, so +// its target never reaches the recall — in either scope. +func TestRecallPagesRefuseASymlinkedPage(t *testing.T) { + repo := t.TempDir() + home := t.TempDir() + outside := filepath.Join(t.TempDir(), "outside.md") + if err := os.WriteFile(outside, []byte(recallPageText("LEAKED outside text", "auth")), 0o600); err != nil { + t.Fatal(err) + } + for _, dir := range []string{filepath.Join(repo, ".abcd", "memory"), abcdhome.Path(home, "memory")} { + if err := os.MkdirAll(dir, 0o700); err != nil { + t.Fatal(err) + } + plantSymlink(t, outside, filepath.Join(dir, "fact_eng_auth-flow.md")) + } + workspace, _ := ReadRecallPages(repo) + machine, _ := ReadMachineRecallPages(home) + for _, p := range append(workspace, machine...) { + if strings.Contains(p.Summary+p.Body, "LEAKED") { + t.Fatalf("a symlinked page was followed: %+v", p) + } + } +} + +// TestRecallPagesRefuseASymlinkedStore: the workspace store committed as a +// directory symlink, and a symlinked ~/.abcd.noindex or memory/, are refused +// whole and named, never walked. +func TestRecallPagesRefuseASymlinkedStore(t *testing.T) { + target := t.TempDir() + writeStoreFile(t, target, "fact_eng_auth-flow.md", recallPageText("LEAKED through the link", "auth"), 0o600) + + repo := t.TempDir() + if err := os.MkdirAll(filepath.Join(repo, ".abcd"), 0o755); err != nil { + t.Fatal(err) + } + plantSymlink(t, target, filepath.Join(repo, ".abcd", "memory")) + pages, refused := ReadRecallPages(repo) + var unsafe *UnsafeStorePathError + if len(pages) != 0 || len(refused) != 1 || refused[0].File != ".abcd/memory/" || !errors.As(refused[0].Err, &unsafe) { + t.Fatalf("a symlinked workspace store was read: pages %+v refused %+v", pages, refused) + } + + for name, link := range map[string]func(home string){ + "memory": func(home string) { + if err := os.MkdirAll(abcdhome.Path(home), 0o700); err != nil { + t.Fatal(err) + } + plantSymlink(t, target, abcdhome.Path(home, "memory")) + }, + "abcd-home": func(home string) { + real := filepath.Join(t.TempDir(), "dotfiles") + if err := os.MkdirAll(filepath.Join(real, "memory"), 0o700); err != nil { + t.Fatal(err) + } + writeStoreFile(t, filepath.Join(real, "memory"), "fact_eng_auth-flow.md", recallPageText("LEAKED through the link", "auth"), 0o600) + plantSymlink(t, real, abcdhome.Path(home)) + }, + } { + t.Run(name, func(t *testing.T) { + home := t.TempDir() + link(home) + pages, refused := ReadMachineRecallPages(home) + if len(pages) != 0 || len(refused) != 1 || !errors.Is(refused[0].Err, fsutil.ErrHomeScopeSymlinked) { + t.Fatalf("a symlinked machine scope was read: pages %+v refused %+v", pages, refused) + } + if strings.Contains(refused[0].Err.Error(), home) { + t.Fatalf("the refusal names the expanded home: %v", refused[0].Err) + } + }) + } +} + +// TestMachineRecallPagesAreReadAsTheMachineFilesAre: a machine page is read +// as the machine memory.json is — a regular file this account owns that no +// one else can write — and one others can write is refused and named, while +// the scope's other pages are still read. An absent scope, or no home, reads +// nothing and creates nothing. +func TestMachineRecallPagesAreReadAsTheMachineFilesAre(t *testing.T) { + home := t.TempDir() + dir := abcdhome.Path(home, "memory") + writeStoreFile(t, dir, "fact_eng_auth-flow.md", recallPageText("Kept to this account", "auth"), 0o600) + writeStoreFile(t, dir, "fact_eng_auth-keys.md", recallPageText("Anyone can rewrite this", "auth"), 0o666) + + pages, refused := ReadMachineRecallPages(home) + if len(pages) != 1 || pages[0].Filename != "fact_eng_auth-flow.md" { + t.Fatalf("pages = %+v", pages) + } + if len(refused) != 1 || refused[0].File != abcdhome.Display("memory", "fact_eng_auth-keys.md") || + !strings.Contains(refused[0].Err.Error(), "writable by others") { + t.Fatalf("refused = %+v", refused) + } + + empty := t.TempDir() + for _, h := range []string{empty, "", "relative/home"} { + if pages, refused := ReadMachineRecallPages(h); len(pages) != 0 || len(refused) != 0 { + t.Fatalf("home %q read %+v %+v", h, pages, refused) + } + } + if _, err := os.Lstat(abcdhome.Path(empty)); !os.IsNotExist(err) { + t.Fatalf("the read created %s: %v", abcdhome.Path(empty), err) + } +} diff --git a/internal/core/recall/items.go b/internal/core/recall/items.go index 6da844301c..051d93fd15 100644 --- a/internal/core/recall/items.go +++ b/internal/core/recall/items.go @@ -11,8 +11,9 @@ // loader's one per-turn budget (rules.InjectionBudgetBytes). Until the // relevance ranking lands (step 4), the candidates are taken in id order. // -// Step 2 recalls notes; pages join the same path in step 3. Nothing here -// writes to stdout or knows a transport. +// Notes come from both memory.json layers (step 2) and pages from both page +// scopes (step 3, pages.go), through one path. Nothing here writes to stdout +// or knows a transport. package recall import ( @@ -24,29 +25,40 @@ import ( "github.com/intentdriven/abcd/internal/termsafe" ) -// The layers an item comes from, as the diagnostic names them. +// The layers an item comes from, as the diagnostic names them: a note's +// memory.json, or a page's scope (the project's .abcd/memory/ or the +// machine's ~/.abcd.noindex/memory/). const ( LayerProject = "project" LayerMachine = "machine" ) -// Item is one thing the recall can bring back: a note now, a page from step -// 3. Key is its ledger key and its place in the id order — a note's id -// (mem-…), and a page's "page:" after every note. +// The kinds of item. +const ( + KindNote = "note" + KindPage = "page" +) + +// Item is one thing the recall can bring back: a note or a page. Key is its +// ledger key and its place in the id order — a note's id (mem-…), and a +// page's "page:" after every note. type Item struct { Key string + Kind string Layer string Title string Body string Recall []string } -// Refusal names a layer the recall could not read, and why, in words safe to -// print: the file in tilde or repository-relative form, the reason -// home-redacted and terminal-safe. +// Refusal names a layer, page scope or page the recall could not read, and +// why, in words safe to print: the file in tilde or repository-relative form, +// the reason home-redacted and terminal-safe. Kind is what it would have +// brought back: KindNote for a memory.json, KindPage for a page scope or page. type Refusal struct { File string `json:"file"` Reason string `json:"reason"` + Kind string `json:"kind"` } // noteItems turns both note layers into items, in id order. A personal note @@ -77,28 +89,28 @@ func noteItems(machine, project []memnote.Note, root string) []Item { } func noteItem(n memnote.Note, layer string) Item { - return Item{Key: n.ID, Layer: layer, Title: n.Title, Body: n.Body, Recall: append([]string(nil), n.Recall...)} + return Item{Key: n.ID, Kind: KindNote, Layer: layer, Title: n.Title, Body: n.Body, Recall: append([]string(nil), n.Recall...)} } // rootCommit is the repository's root commit; a seam for the tests. var rootCommit = gitutil.RootCommit -// ReadNotes reads both note layers through memnote's layer readers — the +// readNotes reads both note layers through memnote's layer readers — the // machine's under home, the project's under repoRoot, the root the rules // loader resolved — and returns their items. A layer that cannot be read is // a Refusal and yields no items, never an error: the hook still injects the // rules, and recalls what the other layer holds. The repository's root // commit is asked of git only when a personal note is tied to one. -func ReadNotes(home, repoRoot string) ([]Item, []Refusal) { +func readNotes(home, repoRoot string) ([]Item, []Refusal) { var refused []Refusal var machine, project []memnote.Note if f, _, err := memnote.ReadMachine(home); err != nil { - refused = append(refused, Refusal{File: memnote.MachineDisplayPath, Reason: reason(err)}) + refused = append(refused, Refusal{File: memnote.MachineDisplayPath, Reason: reason(err), Kind: KindNote}) } else { machine = f.Notes } if f, _, err := memnote.ReadRepo(repoRoot); err != nil { - refused = append(refused, Refusal{File: memnote.RepoRelPath, Reason: reason(err)}) + refused = append(refused, Refusal{File: memnote.RepoRelPath, Reason: reason(err), Kind: KindNote}) } else { project = f.Notes } diff --git a/internal/core/recall/opens_linux_test.go b/internal/core/recall/opens_linux_test.go new file mode 100644 index 0000000000..cd3d43ff4d --- /dev/null +++ b/internal/core/recall/opens_linux_test.go @@ -0,0 +1,194 @@ +//go:build linux + +package recall + +import ( + "encoding/binary" + "errors" + "io/fs" + "os" + "path/filepath" + "slices" + "strings" + "syscall" + "testing" + + "github.com/intentdriven/abcd/internal/abcdhome" + "github.com/intentdriven/abcd/internal/core/rules" + "github.com/intentdriven/abcd/internal/gittest" +) + +// openRecorder is the filesystem seam that records every open under the +// trees it watches: an inotify instance with an IN_OPEN watch on every +// directory below each root, so a file or a directory opened by anything in +// the process — a primitive, a library, a git child — is recorded, whatever +// path the code took to reach it. +type openRecorder struct { + fd int + dirs map[int32]string +} + +func recordOpens(t *testing.T, roots ...string) *openRecorder { + t.Helper() + fd, err := syscall.InotifyInit1(syscall.IN_CLOEXEC | syscall.IN_NONBLOCK) + if err != nil { + t.Skipf("inotify unavailable: %v", err) + } + t.Cleanup(func() { syscall.Close(fd) }) + r := &openRecorder{fd: fd, dirs: map[int32]string{}} + for _, root := range roots { + err := filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error { + if err != nil || !d.IsDir() { + return err + } + wd, err := syscall.InotifyAddWatch(fd, p, syscall.IN_OPEN|syscall.IN_ONLYDIR|syscall.IN_DONT_FOLLOW) + if err != nil { + return err + } + r.dirs[int32(wd)] = p + return nil + }) + if err != nil { + t.Fatalf("watching %s: %v", root, err) + } + } + // The walk that placed the watches opened each directory it listed; + // those opens are the recorder's own, so they are drained unread. + r.opened(t) + return r +} + +// opened drains the events queued so far and returns each path opened, once. +func (r *openRecorder) opened(t *testing.T) []string { + t.Helper() + seen := map[string]bool{} + buf := make([]byte, 64<<10) + for { + n, err := syscall.Read(r.fd, buf) + if errors.Is(err, syscall.EAGAIN) { + break + } + if err != nil { + t.Fatalf("reading inotify events: %v", err) + } + for off := 0; off+syscall.SizeofInotifyEvent <= n; { + wd := int32(binary.NativeEndian.Uint32(buf[off:])) + mask := binary.NativeEndian.Uint32(buf[off+4:]) + nameLen := int(binary.NativeEndian.Uint32(buf[off+12:])) + name := strings.TrimRight(string(buf[off+syscall.SizeofInotifyEvent:off+syscall.SizeofInotifyEvent+nameLen]), "\x00") + off += syscall.SizeofInotifyEvent + nameLen + if mask&syscall.IN_Q_OVERFLOW != 0 { + t.Fatal("the inotify queue overflowed, so opens went unrecorded") + } + dir, ok := r.dirs[wd] + if !ok || mask&syscall.IN_OPEN == 0 { + continue + } + p := dir + if name != "" { + p = filepath.Join(dir, name) + } + seen[p] = true + } + } + var out []string + for p := range seen { + out = append(out, p) + } + slices.Sort(out) + return out +} + +// TestRecallOpensOnlyTheDeclaredPlaces (P5): with a host memory directory +// full of matching text under the home, the recall — every note and page in +// reach read, then matched — opens only the four places it declares (the two +// memory.json layers and the two page scopes) and the directories on the way +// to them, plus the repository's own git data when a personal note asks git +// for the root commit. Nothing under the host's memory directory is opened. +func TestRecallOpensOnlyTheDeclaredPlaces(t *testing.T) { + home := t.TempDir() + repo := gittest.NewRepo(t) + repo.Write("README.md", "a repository\n") + repo.Commit("root") + root := repo.Root() + tied := strings.TrimSpace(repo.Git("rev-list", "--max-parents=0", "HEAD")) + + machineNotes := abcdhome.Path(home, "memory.json") + machinePages := abcdhome.Path(home, "memory") + projectNotes := filepath.Join(root, ".abcd", "memory.json") + projectPages := filepath.Join(root, ".abcd", "memory") + for _, dir := range []string{machinePages, projectPages} { + if err := os.MkdirAll(dir, 0o700); err != nil { + t.Fatal(err) + } + } + writeFile(t, machineNotes, `{"schema_version": 1, "notes": [`+ + noteJSON("mem-2610111200000001", "Auth tokens expire hourly", "Refresh before the hour.", "auth")+`, `+ + tiedNoteJSON("mem-2610111200000002", "This repo's auth lives in one package", "Look there first.", tied, "auth")+`]}`) + writeFile(t, projectNotes, `{"schema_version": 1, "notes": [`+projectNoteJSON("mem-2610111200000003", "Auth is reviewed twice", "Ask for a second reviewer.", "auth")+`]}`) + writePage(t, machinePages, "fact_eng_auth-keys.md", "# Auth keys live in the keychain\n", "auth") + writePage(t, projectPages, "fact_eng_auth-flow.md", "# Auth uses short-lived tokens\n", "auth") + + // The host's own memory directory, full of text that matches: a reader + // that scanned the home, or followed the host's layout, would recall it. + host := filepath.Join(home, ".claude", "projects", "-repo", "memory") + if err := os.MkdirAll(host, 0o700); err != nil { + t.Fatal(err) + } + writeFile(t, filepath.Join(host, "MEMORY.md"), "- auth: the host's own lesson about auth\n") + writePage(t, host, "fact_eng_auth-host.md", "# The host's auth page\n", "auth") + writeFile(t, filepath.Join(home, ".claude", "memory.json"), `{"schema_version": 1, "notes": []}`) + + rec := recordOpens(t, home, root) + items, refused := ReadItems(home, root) + res := Recall(Request{Items: items, Prompt: "the auth login is broken", Budget: rules.InjectionBudgetBytes}) + opened := rec.opened(t) + + declared := []string{machineNotes, machinePages, projectNotes, projectPages} + allowed := func(p string) bool { + for _, d := range declared { + if p == d || filepath.Dir(p) == d && (d == machinePages || d == projectPages) || strings.HasPrefix(d, p+string(filepath.Separator)) { + return true + } + } + // A personal note tied to this repository asks git for its root + // commit, and git reads the repository's own data to answer. + return strings.HasPrefix(p, filepath.Join(root, ".git")+string(filepath.Separator)) || p == filepath.Join(root, ".git") + } + if len(opened) == 0 { + t.Fatal("the recorder saw no open at all, so it proves nothing") + } + for _, p := range opened { + if p == host || strings.HasPrefix(p, filepath.Join(home, ".claude")) { + t.Errorf("the recall opened %s, under the host's own files", p) + continue + } + if !allowed(p) { + t.Errorf("the recall opened %s, which is not one of the places it declares", p) + } + } + for _, d := range declared { + if !slices.Contains(opened, d) { + t.Errorf("the recorder did not see %s opened, so it is not watching what the recall reads", d) + } + } + + // And the recall read what the four places hold, and nothing of the host's. + if len(refused) != 0 { + t.Errorf("refused = %+v", refused) + } + want := []string{"mem-2610111200000001", "mem-2610111200000002", "mem-2610111200000003", "page:fact_eng_auth-flow.md", "page:fact_eng_auth-keys.md"} + if got := slices.Sorted(slices.Values(res.Recalled)); !slices.Equal(got, want) { + t.Errorf("recalled %v, want %v", got, want) + } + if strings.Contains(res.Text, "host's") { + t.Errorf("the host's memory came back:\n%s", res.Text) + } +} + +func writeFile(t *testing.T, p, content string) { + t.Helper() + if err := os.WriteFile(p, []byte(content), 0o600); err != nil { + t.Fatal(err) + } +} diff --git a/internal/core/recall/pages.go b/internal/core/recall/pages.go new file mode 100644 index 0000000000..ffb59501d7 --- /dev/null +++ b/internal/core/recall/pages.go @@ -0,0 +1,67 @@ +package recall + +import ( + "sort" + + "github.com/intentdriven/abcd/internal/core/memory" +) + +// pages.go — memory pages join the recall (spc-2610111627213138 step 3). A +// page is an item when its frontmatter carries a recall: list; its key is +// "page:", after every note in the id order. + +// PageKeyPrefix begins a page item's key; the rest is its filename. +const PageKeyPrefix = "page:" + +// pageItems turns both page scopes into items, in key order. A page of the +// same __ name in both scopes is the workspace's: the +// narrower scope's page is the page of that name, and the machine's is +// suppressed, whether or not the workspace page carries a recall list. A page +// with no recall: list is never an item, so it is never a candidate. +func pageItems(workspace, machine []memory.RecallPage) []Item { + shadowed := map[string]bool{} + var out []Item + for _, p := range workspace { + shadowed[p.Filename] = true + if len(p.Recall) > 0 { + out = append(out, pageItem(p, LayerProject)) + } + } + for _, p := range machine { + if shadowed[p.Filename] || len(p.Recall) == 0 { + continue + } + out = append(out, pageItem(p, LayerMachine)) + } + sort.SliceStable(out, func(i, j int) bool { return out[i].Key < out[j].Key }) + return out +} + +func pageItem(p memory.RecallPage, layer string) Item { + return Item{Key: PageKeyPrefix + p.Filename, Kind: KindPage, Layer: layer, Title: p.Summary, Body: p.Body, + Recall: append([]string(nil), p.Recall...)} +} + +// readPages reads both page scopes through the memory store's recall read — +// the workspace's .abcd/memory/ under repoRoot, the root the rules loader +// resolved, and the machine's ~/.abcd.noindex/memory/ under home, read only — +// and returns their items. A scope or a page that cannot be read is a +// Refusal and yields nothing, never an error. +func readPages(home, repoRoot string) ([]Item, []Refusal) { + workspace, wr := memory.ReadRecallPages(repoRoot) + machine, mr := memory.ReadMachineRecallPages(home) + var refused []Refusal + for _, r := range append(wr, mr...) { + refused = append(refused, Refusal{File: r.File, Reason: reason(r.Err), Kind: KindPage}) + } + return pageItems(workspace, machine), refused +} + +// ReadItems is every note and page in reach, in id order — notes (mem-…) +// before pages (page:…) — and every layer, scope or page that could not be +// read. It is what the prompt router and `rules --prompt` hand the recall. +func ReadItems(home, repoRoot string) ([]Item, []Refusal) { + notes, refused := readNotes(home, repoRoot) + pages, pr := readPages(home, repoRoot) + return append(notes, pages...), append(refused, pr...) +} diff --git a/internal/core/recall/pages_test.go b/internal/core/recall/pages_test.go new file mode 100644 index 0000000000..ac972b1997 --- /dev/null +++ b/internal/core/recall/pages_test.go @@ -0,0 +1,169 @@ +package recall + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/intentdriven/abcd/internal/abcdhome" + "github.com/intentdriven/abcd/internal/core/rules" +) + +// pagesFixture is a home and a repository with an empty page store in each +// scope: the workspace's .abcd/memory/ and the machine's ~/.abcd.noindex/memory/. +func pagesFixture(t *testing.T) (home, repo, workspace, machine string) { + t.Helper() + home = t.TempDir() + repo = t.TempDir() + workspace = filepath.Join(repo, ".abcd", "memory") + machine = abcdhome.Path(home, "memory") + for _, dir := range []string{workspace, machine} { + if err := os.MkdirAll(dir, 0o700); err != nil { + t.Fatal(err) + } + } + return home, repo, workspace, machine +} + +// writePage writes one memory page into dir: a recall: list when recall is +// given, a source block, then body. +func writePage(t *testing.T, dir, name, body string, recall ...string) { + t.Helper() + fm := "---\n" + if len(recall) > 0 { + fm += "recall: [" + strings.Join(recall, ", ") + "]\n" + } + fm += "source:\n class: session_memory\n---\n" + if err := os.WriteFile(filepath.Join(dir, name), []byte(fm+body), 0o600); err != nil { + t.Fatal(err) + } +} + +// noteJSON is a machine-layer note recalled everywhere (root null). +func noteJSON(id, title, body string, recall ...string) string { + return fmt.Sprintf(`{"id": %q, "title": %q, "body": %q, "recall": %s, "root": null, "origin": "agent", + "written_by": "Example:model-1", "written_at": "2026-10-11T12:00:00Z", "session": null}`, id, title, body, jsonList(recall)) +} + +// tiedNoteJSON is a machine-layer note tied to the repository whose root +// commit is root. +func tiedNoteJSON(id, title, body, root string, recall ...string) string { + return fmt.Sprintf(`{"id": %q, "title": %q, "body": %q, "recall": %s, "root": %q, "origin": "agent", + "written_by": "Example:model-1", "written_at": "2026-10-11T12:00:00Z", "session": null}`, id, title, body, jsonList(recall), root) +} + +// projectNoteJSON is a note in the project's file, which carries no root. +func projectNoteJSON(id, title, body string, recall ...string) string { + return fmt.Sprintf(`{"id": %q, "title": %q, "body": %q, "recall": %s, "origin": "agent", + "written_by": "Example:model-1", "written_at": "2026-10-11T12:00:00Z", "session": null}`, id, title, body, jsonList(recall)) +} + +func jsonList(words []string) string { + b, err := json.Marshal(words) + if err != nil { + panic(err) + } + return string(b) +} + +func readAndRecall(t *testing.T, home, repo, prompt string, ledger map[string]string) Result { + t.Helper() + items, refused := ReadItems(home, repo) + if len(refused) != 0 { + t.Fatalf("refused = %+v", refused) + } + return Recall(Request{Items: items, Prompt: prompt, Ledger: ledger, Budget: rules.InjectionBudgetBytes}) +} + +// TestAPageDeclaringAuthComesBackAndNoOther (P1): a page declaring +// recall: [auth, login] comes back on a prompt containing "auth", and no +// other page does. +func TestAPageDeclaringAuthComesBackAndNoOther(t *testing.T) { + home, repo, workspace, machine := pagesFixture(t) + writePage(t, workspace, "fact_eng_auth-flow.md", "# Auth uses short-lived tokens\n\nRefresh them before the hour is out.\n", "auth", "login") + writePage(t, workspace, "fact_eng_billing.md", "# Billing runs nightly\n\nThe auth of the billing job is separate.\n", "billing", "invoice") + writePage(t, machine, "pref_eng_editor.md", "# Prefer short commit subjects\n", "commit") + + res := readAndRecall(t, home, repo, "why does auth keep failing?", nil) + if !slices.Equal(res.Recalled, []string{"page:fact_eng_auth-flow.md"}) { + t.Fatalf("recalled %v, want only the auth page\n%s", res.Recalled, res.Text) + } + for _, want := range []string{"## MEMORY", "page:fact_eng_auth-flow.md", "Auth uses short-lived tokens", "Refresh them before the hour is out."} { + if !strings.Contains(res.Text, want) { + t.Fatalf("the block does not carry %q:\n%s", want, res.Text) + } + } + for _, not := range []string{"Billing runs nightly", "Prefer short commit subjects"} { + if strings.Contains(res.Text, not) { + t.Fatalf("a page that does not declare auth came back (%q):\n%s", not, res.Text) + } + } +} + +// TestRepeatedPromptInjectsMemoryOnce (P3): the same prompt repeated within +// a session injects the matched page once; the loader's backstop prompt, +// which clears both ledgers, brings it back. +func TestRepeatedPromptInjectsMemoryOnce(t *testing.T) { + home, repo, workspace, _ := pagesFixture(t) + writePage(t, workspace, "fact_eng_auth-flow.md", "# Auth uses short-lived tokens\n", "auth") + rs := rules.RuleSet{SchemaVersion: 1, Domains: map[string]rules.Domain{}} + const prompt = "fix the auth bug" + const backstop = 3 + + var state rules.SessionState + var sent []int + for turn := 1; turn <= backstop; turn++ { + inj := rules.Inject(rs, prompt, state, backstop) + res := readAndRecall(t, home, repo, prompt, inj.State.Recalled) + inj.State.Recalled = res.Ledger + state = inj.State + if strings.Contains(res.Text, "Auth uses short-lived tokens") { + sent = append(sent, turn) + } + } + if !slices.Equal(sent, []int{1, backstop}) { + t.Fatalf("the page came back on turns %v, want the first and the backstop's %v", sent, []int{1, backstop}) + } +} + +// TestWorkspacePageSuppressesTheMachinePage (P4): a recall word that hits a +// page of the same name in both scopes brings the workspace page back and +// suppresses the machine page. +func TestWorkspacePageSuppressesTheMachinePage(t *testing.T) { + home, repo, workspace, machine := pagesFixture(t) + writePage(t, workspace, "fact_eng_auth-flow.md", "# The project's auth page\n", "auth") + writePage(t, machine, "fact_eng_auth-flow.md", "# The person's own auth page\n", "auth") + writePage(t, machine, "fact_eng_auth-keys.md", "# Where the person keeps auth keys\n", "auth") + + res := readAndRecall(t, home, repo, "rotate the auth tokens", nil) + want := []string{"page:fact_eng_auth-flow.md", "page:fact_eng_auth-keys.md"} + if !slices.Equal(res.Recalled, want) { + t.Fatalf("recalled %v, want %v", res.Recalled, want) + } + if !strings.Contains(res.Text, "The project's auth page") || strings.Contains(res.Text, "The person's own auth page") { + t.Fatalf("the workspace page did not win over the machine page of the same name:\n%s", res.Text) + } + if !strings.Contains(res.Text, "Where the person keeps auth keys") { + t.Fatalf("a machine page with no workspace twin was suppressed:\n%s", res.Text) + } +} + +// TestPageWithoutRecallIsNeverACandidate: a page with no recall: list is +// silent whatever the prompt says, even its own words. +func TestPageWithoutRecallIsNeverACandidate(t *testing.T) { + home, repo, workspace, machine := pagesFixture(t) + writePage(t, workspace, "fact_eng_auth-flow.md", "# Auth uses short-lived tokens\n") + writePage(t, machine, "fact_eng_auth-keys.md", "# Auth keys live in the keychain\n") + + items, _ := ReadItems(home, repo) + if len(items) != 0 { + t.Fatalf("a page with no recall list is an item: %+v", items) + } + if res := readAndRecall(t, home, repo, "auth tokens short-lived keychain", nil); res.Text != "" { + t.Fatalf("a page with no recall list came back:\n%s", res.Text) + } +} diff --git a/internal/core/recall/recall.go b/internal/core/recall/recall.go index 53eb49471f..532c18fe82 100644 --- a/internal/core/recall/recall.go +++ b/internal/core/recall/recall.go @@ -8,7 +8,7 @@ import ( // Request is one prompt's recall. type Request struct { - // Items are what the memory layers yield (noteItems, ReadNotes). + // Items are every note and page in reach (ReadItems). Items []Item // Prompt is the prompt the hook was handed. Prompt string diff --git a/internal/core/recall/recall_test.go b/internal/core/recall/recall_test.go index 667b945ffd..c31257bdff 100644 --- a/internal/core/recall/recall_test.go +++ b/internal/core/recall/recall_test.go @@ -76,7 +76,7 @@ func TestANoteInTheProjectsFileIsRecalled(t *testing.T) { if err := os.WriteFile(filepath.Join(repo, ".abcd", "memory.json"), []byte(project), 0o644); err != nil { t.Fatal(err) } - items, refused := ReadNotes(t.TempDir(), repo) + items, refused := readNotes(t.TempDir(), repo) if len(refused) != 0 { t.Fatalf("refused = %+v", refused) } diff --git a/internal/surface/cli/cli.go b/internal/surface/cli/cli.go index 4178eae01f..ae17ec1b95 100644 --- a/internal/surface/cli/cli.go +++ b/internal/surface/cli/cli.go @@ -1679,20 +1679,21 @@ func newHookCommand(asJSON *bool) *cobra.Command { // prompt-router — UserPromptSubmit: recall-match, dedup, inject. hookCmd.AddCommand(&cobra.Command{ Use: "prompt-router", - Short: "UserPromptSubmit: inject the rules and memory notes matching the prompt", + Short: "UserPromptSubmit: inject the rules, memory notes and pages matching the prompt", Long: `Plain, stdout is the rendered rule block the host injects, followed by a -"## MEMORY" block of the memory notes the prompt recalls, and nothing on a -turn with nothing new. With --json, stdout is one envelope in place of the +"## MEMORY" block of the memory notes and pages the prompt recalls, and nothing +on a turn with nothing new. With --json, stdout is one envelope in place of the generic machine-reader shape: {"text": ..., "injected": [...], "recalled": [...], "active": [...], "error": ...} text is byte for byte what the plain form writes; injected names the domains -text carries, and recalled the notes; active is the full set of domains in +text carries, and recalled the notes and pages (a note by its id, a page as +page:); active is the full set of domains in force this turn, so a name missing from it has stopped. error is present only when the prompt could not be evaluated, and then active is absent: the set is -unknown. A memory.json that cannot be read is named on stderr and recalls -nothing, and the rules still inject. Every outcome, an error included, +unknown. A memory.json, page store or page that cannot be read is named on +stderr and recalls nothing, and the rules still inject. Every outcome, an error included, exits 0, so the hook can never wedge a session.`, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { @@ -1765,9 +1766,9 @@ exits 0, so the hook can never wedge a session.`, fmt.Fprintf(cmd.ErrOrStderr(), "abcd rules: turn %d, injected %d domain(s) %v, %d bytes\n", res.State.Count, len(res.Injected), res.Labels(), len(res.Text)) if len(mem.Recalled) > 0 || len(mem.Omitted) > 0 { - fmt.Fprintf(cmd.ErrOrStderr(), "abcd memory: recalled %d note(s) %v, %d bytes", len(mem.Recalled), mem.Recalled, len(mem.Text)) + fmt.Fprintf(cmd.ErrOrStderr(), "abcd memory: recalled %d item(s) %v, %d bytes", len(mem.Recalled), mem.Recalled, len(mem.Text)) if len(mem.Omitted) > 0 { - fmt.Fprintf(cmd.ErrOrStderr(), "; %d matched note(s) %v did not fit this turn's budget and are tried again on the next", len(mem.Omitted), mem.Omitted) + fmt.Fprintf(cmd.ErrOrStderr(), "; %d matched item(s) %v did not fit this turn's budget and are tried again on the next", len(mem.Omitted), mem.Omitted) } fmt.Fprintln(cmd.ErrOrStderr()) } @@ -2442,8 +2443,9 @@ its .abcd/rules.json. --prompt "" shows what that one prompt would bring back to a fresh session: each rule domain it matches, with the recall words that hit, and each memory note it recalls from ` + abcdhome.Display("memory.json") + ` and .abcd/memory.json, -with its layer and the recall words that hit, each in full or as not fitting -the per-turn budget. It reads against an empty ledger and writes nothing. +and each memory page from .abcd/memory/ and ` + abcdhome.Display("memory") + `/, with its +layer and the recall words that hit, each in full or as not fitting the +per-turn budget. It reads against an empty ledger and writes nothing. --prompt and a DOMAIN are two forms and are refused together, exit 2. Read-only.`, Args: cobra.MaximumNArgs(1), @@ -2500,7 +2502,7 @@ Read-only.`, }) }, } - cmd.Flags().StringVar(&prompt, "prompt", "", "show what this one prompt would bring back to a fresh session: the domains it matches and the memory notes it recalls") + cmd.Flags().StringVar(&prompt, "prompt", "", "show what this one prompt would bring back to a fresh session: the domains it matches and the memory notes and pages it recalls") return cmd } diff --git a/internal/surface/cli/memory_recall_test.go b/internal/surface/cli/memory_recall_test.go index 0fcd9def26..7f4297c3cc 100644 --- a/internal/surface/cli/memory_recall_test.go +++ b/internal/surface/cli/memory_recall_test.go @@ -295,3 +295,130 @@ func TestRulesPromptRefusesAPositionalDomain(t *testing.T) { t.Fatalf("a refused form rendered:\n%s", out) } } + +// writeMemoryPage writes one memory page into dir, with a recall: list when +// recall is given. +func writeRecallPage(t *testing.T, dir, name, body string, mode os.FileMode, recall ...string) { + t.Helper() + if err := os.MkdirAll(dir, 0o700); err != nil { + t.Fatal(err) + } + fm := "---\n" + if len(recall) > 0 { + fm += "recall: [" + strings.Join(recall, ", ") + "]\n" + } + fm += "source:\n class: session_memory\n---\n" + p := filepath.Join(dir, name) + if err := os.WriteFile(p, []byte(fm+body), mode); err != nil { + t.Fatal(err) + } + if err := os.Chmod(p, mode); err != nil { + t.Fatal(err) + } +} + +// TestANoteAndAPageComeBackTogether (N2): a note and a page that both match +// a prompt about tests come back together, in one block, through the one +// recall the prompt router runs. +func TestANoteAndAPageComeBackTogether(t *testing.T) { + t.Setenv("ABCD_RULES_STATE_DIR", t.TempDir()) + _, repo := memoryFixture(t) + id := addRaceNote(t, "session-one") + writeRecallPage(t, filepath.Join(repo, ".abcd", "memory"), "fact_eng_test-layout.md", + "# The tests live beside their package\n\nTable tests, one file per verb.\n", 0o644, "test") + + env, _ := recallRouterJSON(t, "session-two", repo, "why are the tests failing?") + if env.Recalled == nil || !slices.Equal(slices.Sorted(slices.Values(*env.Recalled)), []string{id, "page:fact_eng_test-layout.md"}) { + t.Fatalf("recalled = %v, want the note and the page\n%s", env.Recalled, env.Text) + } + if strings.Count(env.Text, "## MEMORY") != 1 { + t.Fatalf("the note and the page did not come back in one block:\n%s", env.Text) + } + for _, want := range []string{"The tests need the race flag", "The tests live beside their package", "Table tests, one file per verb."} { + if !strings.Contains(env.Text, want) { + t.Fatalf("the block does not carry %q:\n%s", want, env.Text) + } + } +} + +// TestRulesPromptShowsMatchedPages: `rules --prompt` lists each page the +// prompt recalls, from either scope, with its key, layer and hitting words, +// and names no page the prompt does not match. +func TestRulesPromptShowsMatchedPages(t *testing.T) { + t.Setenv("ABCD_RULES_STATE_DIR", t.TempDir()) + home, repo := memoryFixture(t) + workspace := filepath.Join(repo, ".abcd", "memory") + writeRecallPage(t, workspace, "fact_eng_auth-flow.md", "# Auth uses short-lived tokens\n\nThe page's own words.\n", 0o644, "auth", "login") + writeRecallPage(t, workspace, "fact_eng_billing.md", "# Billing runs nightly\n", 0o644, "billing") + writeRecallPage(t, abcdhome.Path(home, "memory"), "fact_eng_auth-keys.md", "# Auth keys live in the keychain\n", 0o600, "auth") + + out, stderr, err := runCLISplit(t, "rules", "--prompt", "the auth login is broken") + if err != nil { + t.Fatalf("rules --prompt: %v\n%s", err, stderr) + } + for _, want := range []string{"Pages:", "page:fact_eng_auth-flow.md Auth uses short-lived tokens (project layer) — hit by auth, login", + "page:fact_eng_auth-keys.md Auth keys live in the keychain (machine layer) — hit by auth"} { + if !strings.Contains(out, want) { + t.Fatalf("rules --prompt does not show %q:\n%s", want, out) + } + } + for _, not := range []string{"Billing", "The page's own words."} { + if strings.Contains(out, not) { + t.Fatalf("rules --prompt shows %q:\n%s", not, out) + } + } + + jout, stderr, err := runCLISplit(t, "rules", "--prompt", "the auth login is broken", "--json") + if err != nil { + t.Fatalf("rules --prompt --json: %v\n%s", err, stderr) + } + var env struct { + Pages []struct { + Key string `json:"key"` + Filename string `json:"filename"` + Title string `json:"title"` + Layer string `json:"layer"` + Hits []string `json:"hits"` + Form string `json:"form"` + } `json:"pages"` + } + if err := json.Unmarshal([]byte(jout), &env); err != nil { + t.Fatalf("rules --prompt --json is not one document: %v\n%s", err, jout) + } + if len(env.Pages) != 2 { + t.Fatalf("pages = %+v", env.Pages) + } + p := env.Pages[0] + if p.Key != "page:fact_eng_auth-flow.md" || p.Filename != "fact_eng_auth-flow.md" || p.Title != "Auth uses short-lived tokens" || + p.Layer != "project" || !slices.Equal(p.Hits, []string{"auth", "login"}) || p.Form != "full" { + t.Fatalf("pages[0] = %+v", p) + } + if env.Pages[1].Key != "page:fact_eng_auth-keys.md" || env.Pages[1].Layer != "machine" { + t.Fatalf("pages[1] = %+v", env.Pages[1]) + } +} + +// TestARefusedPageIsNamedAndTheRestComeBack: a machine page others can write +// is refused and named on stderr, in tilde form; the scope's other pages and +// the rules still come back. +func TestARefusedPageIsNamedAndTheRestComeBack(t *testing.T) { + t.Setenv("ABCD_RULES_STATE_DIR", t.TempDir()) + home, repo := memoryFixture(t) + machine := abcdhome.Path(home, "memory") + writeRecallPage(t, machine, "fact_eng_auth-flow.md", "# Kept to this account\n", 0o600, "commit") + writeRecallPage(t, machine, "fact_eng_auth-keys.md", "# Anyone can rewrite this\n", 0o666, "commit") + + env, errlog := recallRouterJSON(t, "refused-page", repo, "commit the fix") + if !slices.Contains(env.Injected, "COMMITTING") || env.Recalled == nil || !slices.Equal(*env.Recalled, []string{"page:fact_eng_auth-flow.md"}) { + t.Fatalf("injected %v recalled %v:\n%s\nstderr:\n%s", env.Injected, env.Recalled, env.Text, errlog) + } + if strings.Contains(env.Text, "Anyone can rewrite this") { + t.Fatalf("a page others can write came back:\n%s", env.Text) + } + if !strings.Contains(errlog, abcdhome.Display("memory", "fact_eng_auth-keys.md")) || !strings.Contains(errlog, "recalling nothing from it") { + t.Fatalf("stderr does not name the refused page:\n%s", errlog) + } + if strings.Contains(errlog, home) { + t.Fatalf("stderr carries the expanded home:\n%s", errlog) + } +} diff --git a/internal/surface/cli/rules_prompt.go b/internal/surface/cli/rules_prompt.go index 56bd132033..84b7f5aca8 100644 --- a/internal/surface/cli/rules_prompt.go +++ b/internal/surface/cli/rules_prompt.go @@ -11,25 +11,26 @@ import ( ) // rules_prompt.go — the memory recall's two front doors -// (spc-2610111627213138 step 2): the prompt router's recall after the rules, -// and the `rules --prompt ""` diagnostic that shows what one prompt -// brings back. +// (spc-2610111627213138 steps 2 and 3): the prompt router's recall after the +// rules, and the `rules --prompt ""` diagnostic that shows what one +// prompt brings back. // recallMemory is the prompt router's recall: the notes both memory layers -// hold, matched against prompt and taken within budget, what the rules left -// of the loader's per-turn budget. The machine layer is read under the home -// the rules user layer was (rules.UserHome), the project's under root, the -// root the loader resolved. A layer that cannot be read is named on stderr -// and recalls nothing, while the rules, already rendered, still inject. The -// loader's kill switch silences the recall: nothing is read, nothing comes -// back, and the ledger is kept as it was. +// hold and the pages both page scopes hold, matched against prompt and taken +// within budget, what the rules left of the loader's per-turn budget. The +// machine's are read under the home the rules user layer was +// (rules.UserHome), the project's under root, the root the loader resolved. A +// layer, scope or page that cannot be read is named on stderr and recalls +// nothing, while the rules, already rendered, still inject. The loader's kill +// switch silences the recall: nothing is read, nothing comes back, and the +// ledger is kept as it was. func recallMemory(w io.Writer, rs rules.RuleSet, root, prompt string, ledger map[string]string, budget int) recall.Result { if rs.Disabled { return recall.Result{Ledger: ledger} } - items, refused := recall.ReadNotes(rules.UserHome(), root) + items, refused := recall.ReadItems(rules.UserHome(), root) for _, r := range refused { - diagnosticLine(w, "abcd memory: %s; recalling no notes from it", refusalText(r)) + diagnosticLine(w, "abcd memory: %s; recalling %s from it", refusalText(r), refusalLoss(r)) } return recall.Recall(recall.Request{Items: items, Prompt: prompt, Ledger: ledger, Budget: budget}) } @@ -42,7 +43,16 @@ func refusalText(r recall.Refusal) string { return r.File + ": " + why } -// The forms the diagnostic gives a matched domain or note. +// refusalLoss is what a refused file costs the recall: a memory.json's notes, +// or whatever a page store or a page would have brought back. +func refusalLoss(r recall.Refusal) string { + if r.Kind == recall.KindPage { + return "nothing" + } + return "no notes" +} + +// The forms the diagnostic gives a matched domain, note or page. const ( promptFormFull = "full" promptFormOverBudget = "over-budget" @@ -66,11 +76,23 @@ type rulesPromptNote struct { Form string `json:"form"` } +// rulesPromptPage is one matched page in the diagnostic: its key is the one +// the router's envelope names it by (page:). +type rulesPromptPage struct { + Key string `json:"key"` + Filename string `json:"filename"` + Title string `json:"title"` + Layer string `json:"layer"` + Hits []string `json:"hits"` + Form string `json:"form"` +} + // rulesPromptView is `rules --prompt`'s --json envelope. type rulesPromptView struct { Disabled bool `json:"disabled"` Domains []rulesPromptDomain `json:"domains"` Notes []rulesPromptNote `json:"notes"` + Pages []rulesPromptPage `json:"pages"` Unreadable []recall.Refusal `json:"unreadable"` } @@ -79,7 +101,8 @@ type rulesPromptView struct { // empty ledger, then the recall within what the rules left — and writes // nothing: no session state is loaded or saved. func rulesPrompt(rs rules.RuleSet, root, prompt string) rulesPromptView { - v := rulesPromptView{Disabled: rs.Disabled, Domains: []rulesPromptDomain{}, Notes: []rulesPromptNote{}, Unreadable: []recall.Refusal{}} + v := rulesPromptView{Disabled: rs.Disabled, Domains: []rulesPromptDomain{}, Notes: []rulesPromptNote{}, Pages: []rulesPromptPage{}, + Unreadable: []recall.Refusal{}} if rs.Disabled { return v } @@ -96,7 +119,7 @@ func rulesPrompt(rs rules.RuleSet, root, prompt string) rulesPromptView { v.Domains = append(v.Domains, rulesPromptDomain{Name: d.Name, Label: rules.Label(d.Name, d.Source), Source: d.Source, Hits: rules.Hits(d, prompt), Form: form}) } - items, refused := recall.ReadNotes(rules.UserHome(), root) + items, refused := recall.ReadItems(rules.UserHome(), root) v.Unreadable = append(v.Unreadable, refused...) mem := recall.Recall(recall.Request{Items: items, Prompt: prompt, Budget: rules.InjectionBudgetBytes - len(res.Text)}) recalled := map[string]bool{} @@ -108,6 +131,11 @@ func rulesPrompt(rs rules.RuleSet, root, prompt string) rulesPromptView { if recalled[c.Item.Key] { form = promptFormFull } + if c.Item.Kind == recall.KindPage { + v.Pages = append(v.Pages, rulesPromptPage{Key: c.Item.Key, Filename: strings.TrimPrefix(c.Item.Key, recall.PageKeyPrefix), + Title: c.Item.Title, Layer: c.Item.Layer, Hits: c.Hits, Form: form}) + continue + } v.Notes = append(v.Notes, rulesPromptNote{ID: c.Item.Key, Title: c.Item.Title, Layer: c.Item.Layer, Hits: c.Hits, Form: form}) } return v @@ -137,8 +165,17 @@ func renderRulesPrompt(w io.Writer, rs rules.RuleSet, v rulesPromptView) { termsafe.Sanitize(strings.Join(n.Hits, ", ")), promptFormWords(n.Form)) } } + if len(v.Pages) == 0 { + fmt.Fprintln(w, "Pages: none matched") + } else { + fmt.Fprintln(w, "Pages:") + for _, p := range v.Pages { + fmt.Fprintf(w, " %s %s (%s layer) — hit by %s — %s\n", p.Key, termsafe.Sanitize(p.Title), p.Layer, + termsafe.Sanitize(strings.Join(p.Hits, ", ")), promptFormWords(p.Form)) + } + } for _, u := range v.Unreadable { - fmt.Fprintf(w, "unreadable: %s (no notes recalled from it)\n", refusalText(u)) + fmt.Fprintf(w, "unreadable: %s (%s recalled from it)\n", termsafe.Sanitize(refusalText(u)), refusalLoss(u)) } } From c5071b9cca7a5b318628c09f7f05150ec6595945 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:21:59 +0000 Subject: [PATCH 02/23] docs: record the choices step 3 made where the memory-recall spec is silent A Settled here section under the spec's Decisions carries each choice step 3 made, dated and marked as an agent's choice for review: which pages the recall reads and what a page's title is, a workspace page shadowing the machine's page of the same name, the guards a machine page is read with, and how a page that cannot be read is named. Refs: itd-2610091918433290, itd-39 Assisted-by: Claude:claude-opus-5-5 --- .../spc-2610111627213138-memory-recall.md | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md b/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md index 4346e72a1f..632277054f 100644 --- a/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md +++ b/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md @@ -728,6 +728,41 @@ each open to the facilitator's veto at review: bounds what a note can carry or how it is named without a ruling of its own, and is open to the facilitator's veto at review like the choices above. +### Settled here + +Choices an agent made while building steps 3 and 4 where this spec is silent, +each the narrowest reading the agent found that meets the criteria, and each +an agent's choice for review, open to the facilitator's veto like the design +choices above. + +- **Which pages the recall reads** (2026-10-12, step 3, agent's choice for + review). In each scope, the pages `index.md` lists, the page files at the + store's top level, whose names are `__.md`. A page's + title is its index line's summary, derived from the page the way the index + derives it, so the recall reads no `index.md` and opens only the four + places it declares. Grounds: P4 compares pages by that name, and the name's + characters keep a page's key (`page:`) safe to print in the + hook's stderr line and envelope. +- **A workspace page shadows the machine page of the same name, whether or + not it carries a recall list** (2026-10-12, step 3, agent's choice for + review). Grounds: the narrower scope's page is the page of that name, as a + repository layer's field replaces the machine's and a note in the project's + file suppresses the machine's copy of its id; it also keeps one + `page:` key per name in the ledger and the envelope. +- **A machine page is read with the guards the machine `memory.json` gets** + (2026-10-12, step 3, agent's choice for review): the directory through + `fsutil.OpenHomeScope`, each page a regular file this account owns that no + one else can write (`fsutil.ReadHomeDeclarationDenyingIn`). A page failing + them is named on stderr and brings nothing back, while the scope's other + pages still come back. Grounds: a machine page injects text into every + session as a machine note does; the spec names only the directory's open, + and reading it alone would trust a page another account could write while + the notes file beside it is refused for the same thing. +- **A page store or a page that cannot be read is named** (2026-10-12, + step 3, agent's choice for review): on the hook's stderr with "recalling + nothing from it", and in `rules --prompt`'s `unreadable` list, which gains a + `kind` (`note` or `page`); a refused `memory.json` keeps step 2's words. + ## Open Questions None. The nine questions first left here, and the three the spec review From f752318ffbae4d6092e343b546a2da7c001d3091 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:29:26 +0000 Subject: [PATCH 03/23] feat: rank recalled notes and pages by BM25F, with its reasons When only some matched notes and pages fit the per-turn budget, a relevance ranking now chooses which, in place of the id order (spc-2610111627213138 step 4). The score is BM25F in the standard library: a term counts three times in an item's recall words, twice in its title and once in its body, weighted frequencies summed before saturation, with k1 1.2, b 0.75 and idf over every item in reach, so a rare matching word outranks words every item shares. Ties break on the narrower scope, then the more recent note or page, then the key. The query terms come from the loader's own prompt index, and an item's words are stemmed by the same rule, so the ranking weighs the words the matcher matched. `rules --prompt` shows each candidate's rank and score and, under it, every matched term with the fields it was found in and what it added. A golden file pins the scores of a small fixed corpus. Refs: itd-2610091918433290, itd-39 Assisted-by: Claude:claude-opus-5-5 --- commands/ahoy.md | 6 +- docs/reference/cli/commands.md | 6 +- internal/core/memory/recallpages.go | 33 ++- internal/core/memory/recallpages_test.go | 28 +++ internal/core/recall/items.go | 15 +- internal/core/recall/match.go | 8 +- internal/core/recall/pages.go | 4 +- internal/core/recall/pages_test.go | 16 +- internal/core/recall/ranking.go | 165 ++++++++++++ internal/core/recall/ranking_test.go | 252 +++++++++++++++++++ internal/core/recall/recall.go | 13 +- internal/core/recall/recall_test.go | 37 +-- internal/core/recall/testdata/ranking.golden | 44 ++++ internal/core/rules/rules.go | 53 +++- internal/surface/cli/cli.go | 6 +- internal/surface/cli/memory_recall_test.go | 63 +++++ internal/surface/cli/rules_prompt.go | 77 ++++-- 17 files changed, 747 insertions(+), 79 deletions(-) create mode 100644 internal/core/recall/ranking.go create mode 100644 internal/core/recall/ranking_test.go create mode 100644 internal/core/recall/testdata/ranking.golden diff --git a/commands/ahoy.md b/commands/ahoy.md index 4657831f9b..e4d080a663 100644 --- a/commands/ahoy.md +++ b/commands/ahoy.md @@ -1219,7 +1219,11 @@ lists each memory page it recalls from the project's `.abcd/memory/` or hook's envelope gives it), with its `filename`, `title`, `layer` and the recall words that hit. A page comes back only when its frontmatter declares a `recall:` list, and a page in the project's store suppresses the machine's -page of the same name. Each carries a `form`: `full`, or `over-budget` when +page of the same name. Notes and pages share one ranked order: each carries +its `rank` (1 first), its BM25F relevance `score`, and `terms`, each matched +term with how often it was found in the item's `recall` words, `title` and +`body` and its `contribution` to the score; the ranking decides which items +fit the budget. Each carries a `form`: `full`, or `over-budget` when it would not fit the per-turn budget and would come back on a later prompt instead. `disabled` is true under the kill switch, and then nothing comes back; `unreadable` names a `memory.json`, page store or page that could not diff --git a/docs/reference/cli/commands.md b/docs/reference/cli/commands.md index ceb1c2313d..b700804914 100644 --- a/docs/reference/cli/commands.md +++ b/docs/reference/cli/commands.md @@ -3658,7 +3658,11 @@ session: each rule domain it matches, with the recall words that hit, and each memory note it recalls from ~/.abcd.noindex/memory.json and .abcd/memory.json, and each memory page from .abcd/memory/ and ~/.abcd.noindex/memory/, with its layer and the recall words that hit, each in full or as not fitting the -per-turn budget. It reads against an empty ledger and writes nothing. +per-turn budget. Notes and pages share one ranked order, a BM25F relevance +score that decides which of them fit the budget: each shows its rank and +score and, under it, every matched term with the fields it was found in +(recall words, title, body) and what it added to the score. It reads +against an empty ledger and writes nothing. --prompt and a DOMAIN are two forms and are refused together, exit 2. Read-only. diff --git a/internal/core/memory/recallpages.go b/internal/core/memory/recallpages.go index 09615f8fbf..9317c1f860 100644 --- a/internal/core/memory/recallpages.go +++ b/internal/core/memory/recallpages.go @@ -8,6 +8,7 @@ import ( "path/filepath" "strings" "syscall" + "time" "github.com/intentdriven/abcd/internal/abcdhome" "github.com/intentdriven/abcd/internal/fsutil" @@ -29,6 +30,11 @@ type RecallPage struct { Body string // Recall is the page's recall: frontmatter list, empty when it has none. Recall []string + // IngestedAt is the latest ingested_at the page's source block records + // (one source's, or the latest of several), the zero time when it records + // none. It is the page's own record of when it was written, which travels + // with it; the file's modification time is whenever a checkout wrote it. + IngestedAt time.Time } // RecallRefusal names a scope or a page the read refused, and why. File is @@ -100,7 +106,8 @@ func recallPagesFrom(store *storeHandle, shown string, read func(string, int64) } p := parsePage(string(raw)) info := pageInfoOf(name, p) - pages = append(pages, RecallPage{Filename: name, Summary: info.Summary, Body: p.body, Recall: recallList(p.fm)}) + pages = append(pages, RecallPage{Filename: name, Summary: info.Summary, Body: p.body, Recall: recallList(p.fm), + IngestedAt: latestIngest(p.source())}) } return pages, refused } @@ -120,6 +127,30 @@ func recallList(fm map[string]any) []string { return out } +// latestIngest is the latest ingested_at date a source block records, in +// its single-source or multi-source shape; the zero time when none parses. +func latestIngest(source map[string]any) time.Time { + entries := []any{source} + if list, ok := source["sources"].([]any); ok { + entries = append(entries, list...) + } + var latest time.Time + for _, e := range entries { + m, ok := e.(map[string]any) + if !ok { + continue + } + s, ok := m["ingested_at"].(string) + if !ok { + continue + } + if at, err := time.Parse(time.DateOnly, s); err == nil && at.After(latest) { + latest = at + } + } + return latest +} + // recallReadError words a refused page read without the expanded path. func recallReadError(err error) error { switch { diff --git a/internal/core/memory/recallpages_test.go b/internal/core/memory/recallpages_test.go index 2d381e11c2..6ae5421296 100644 --- a/internal/core/memory/recallpages_test.go +++ b/internal/core/memory/recallpages_test.go @@ -169,3 +169,31 @@ func TestMachineRecallPagesAreReadAsTheMachineFilesAre(t *testing.T) { t.Fatalf("the read created %s: %v", abcdhome.Path(empty), err) } } + +// TestRecallPagesCarryTheirLatestIngestDate: a page's recency, the +// ranking's tie-break, is the latest ingested_at its source block records — +// one source's, or the latest of several — and the zero time when it records +// none, never the file's modification time, which a checkout sets. +func TestRecallPagesCarryTheirLatestIngestDate(t *testing.T) { + repo := t.TempDir() + mem := filepath.Join(repo, ".abcd", "memory") + writeStoreFile(t, mem, "fact_eng_one.md", recallPageText("One source", "auth"), 0o644) + writeStoreFile(t, mem, "fact_eng_many.md", "---\nrecall: [auth]\nsource:\n classes: [session_memory]\n sources:\n"+ + " - {class: session_memory, ingested_at: 2026-09-01}\n - {class: session_memory, ingested_at: 2026-10-05}\n---\n# Many sources\n", 0o644) + writeStoreFile(t, mem, "fact_eng_none.md", "---\nrecall: [auth]\n---\n# No source block\n", 0o644) + + at := map[string]string{} + pages, refused := ReadRecallPages(repo) + if len(refused) != 0 { + t.Fatalf("refused = %+v", refused) + } + for _, p := range pages { + at[p.Filename] = p.IngestedAt.Format("2006-01-02") + if p.Filename == "fact_eng_none.md" && !p.IngestedAt.IsZero() { + t.Fatalf("a page with no source block has a date: %v", p.IngestedAt) + } + } + if at["fact_eng_one.md"] != "2026-10-01" || at["fact_eng_many.md"] != "2026-10-05" { + t.Fatalf("ingest dates = %v", at) + } +} diff --git a/internal/core/recall/items.go b/internal/core/recall/items.go index 051d93fd15..f6630774b0 100644 --- a/internal/core/recall/items.go +++ b/internal/core/recall/items.go @@ -8,8 +8,8 @@ // and a rule domain's recall word hit on exactly the same prompts. The // per-session ledger is SessionState.Recalled, keyed on each item's content, // cleared by the loader's resets. The budget is what the rules leave of the -// loader's one per-turn budget (rules.InjectionBudgetBytes). Until the -// relevance ranking lands (step 4), the candidates are taken in id order. +// loader's one per-turn budget (rules.InjectionBudgetBytes), and the +// relevance ranking (ranking.go, BM25F) decides which candidates fit it. // // Notes come from both memory.json layers (step 2) and pages from both page // scopes (step 3, pages.go), through one path. Nothing here writes to stdout @@ -18,6 +18,7 @@ package recall import ( "sort" + "time" "github.com/intentdriven/abcd/internal/core/memnote" "github.com/intentdriven/abcd/internal/fsutil" @@ -40,8 +41,10 @@ const ( ) // Item is one thing the recall can bring back: a note or a page. Key is its -// ledger key and its place in the id order — a note's id (mem-…), and a -// page's "page:" after every note. +// ledger key and the ranking's last tie-break — a note's id (mem-…), and a +// page's "page:". At is when it was written, the ranking's +// recency tie-break: a note's written_at, a page's latest ingested_at, the +// zero time when it records none. type Item struct { Key string Kind string @@ -49,6 +52,7 @@ type Item struct { Title string Body string Recall []string + At time.Time } // Refusal names a layer, page scope or page the recall could not read, and @@ -89,7 +93,8 @@ func noteItems(machine, project []memnote.Note, root string) []Item { } func noteItem(n memnote.Note, layer string) Item { - return Item{Key: n.ID, Kind: KindNote, Layer: layer, Title: n.Title, Body: n.Body, Recall: append([]string(nil), n.Recall...)} + at, _ := time.Parse(time.RFC3339, n.WrittenAt) + return Item{Key: n.ID, Kind: KindNote, Layer: layer, Title: n.Title, Body: n.Body, Recall: append([]string(nil), n.Recall...), At: at} } // rootCommit is the repository's root commit; a seam for the tests. diff --git a/internal/core/recall/match.go b/internal/core/recall/match.go index 2d7817c65e..4c7c588fe0 100644 --- a/internal/core/recall/match.go +++ b/internal/core/recall/match.go @@ -8,10 +8,10 @@ type Candidate struct { Hits []string } -// Candidates returns the items any of whose recall words hits prompt under -// the loader's term rule (rules.TermMatcher), in the items' order — id order -// from noteItems. The order chooses nothing yet; Recall takes them in it. -func Candidates(items []Item, prompt string) []Candidate { +// candidates returns the items any of whose recall words hits prompt under +// the loader's term rule (rules.TermMatcher), in the items' order. The order +// chooses nothing: Rank orders the candidates, and Recall takes them in it. +func candidates(items []Item, prompt string) []Candidate { if len(items) == 0 { return nil } diff --git a/internal/core/recall/pages.go b/internal/core/recall/pages.go index ffb59501d7..280b2290fe 100644 --- a/internal/core/recall/pages.go +++ b/internal/core/recall/pages.go @@ -8,7 +8,7 @@ import ( // pages.go — memory pages join the recall (spc-2610111627213138 step 3). A // page is an item when its frontmatter carries a recall: list; its key is -// "page:", after every note in the id order. +// "page:". // PageKeyPrefix begins a page item's key; the rest is its filename. const PageKeyPrefix = "page:" @@ -39,7 +39,7 @@ func pageItems(workspace, machine []memory.RecallPage) []Item { func pageItem(p memory.RecallPage, layer string) Item { return Item{Key: PageKeyPrefix + p.Filename, Kind: KindPage, Layer: layer, Title: p.Summary, Body: p.Body, - Recall: append([]string(nil), p.Recall...)} + Recall: append([]string(nil), p.Recall...), At: p.IngestedAt} } // readPages reads both page scopes through the memory store's recall read — diff --git a/internal/core/recall/pages_test.go b/internal/core/recall/pages_test.go index ac972b1997..5f530bf74f 100644 --- a/internal/core/recall/pages_test.go +++ b/internal/core/recall/pages_test.go @@ -141,7 +141,7 @@ func TestWorkspacePageSuppressesTheMachinePage(t *testing.T) { res := readAndRecall(t, home, repo, "rotate the auth tokens", nil) want := []string{"page:fact_eng_auth-flow.md", "page:fact_eng_auth-keys.md"} - if !slices.Equal(res.Recalled, want) { + if !slices.Equal(slices.Sorted(slices.Values(res.Recalled)), want) { t.Fatalf("recalled %v, want %v", res.Recalled, want) } if !strings.Contains(res.Text, "The project's auth page") || strings.Contains(res.Text, "The person's own auth page") { @@ -167,3 +167,17 @@ func TestPageWithoutRecallIsNeverACandidate(t *testing.T) { t.Fatalf("a page with no recall list came back:\n%s", res.Text) } } + +// TestPageItemsCarryTheirIngestDate: a page item's At, the ranking's recency +// tie-break, is the page's latest ingested_at. +func TestPageItemsCarryTheirIngestDate(t *testing.T) { + home, repo, workspace, _ := pagesFixture(t) + if err := os.WriteFile(filepath.Join(workspace, "fact_eng_auth-flow.md"), + []byte("---\nrecall: [auth]\nsource:\n class: session_memory\n ingested_at: 2026-10-01\n---\n# Auth\n"), 0o600); err != nil { + t.Fatal(err) + } + items, _ := ReadItems(home, repo) + if len(items) != 1 || items[0].At.Format("2006-01-02") != "2026-10-01" { + t.Fatalf("items = %+v", items) + } +} diff --git a/internal/core/recall/ranking.go b/internal/core/recall/ranking.go new file mode 100644 index 0000000000..fdc831401b --- /dev/null +++ b/internal/core/recall/ranking.go @@ -0,0 +1,165 @@ +package recall + +import ( + "math" + "sort" + "strings" + + "github.com/intentdriven/abcd/internal/core/rules" +) + +// ranking.go — the relevance ranking (spc-2610111627213138 step 4): BM25 with +// per-field weights (BM25F; Robertson, Zaragoza and Taylor, "Simple BM25 +// Extension to Multiple Weighted Fields", 2004), over every note and page in +// reach. The query terms are the prompt's stemmed tokens from the loader's own +// prompt index (rules.PromptTerms), and an item's fields are stemmed by the +// same rule (rules.TextStems), so the ranking weighs exactly the words the +// matcher matches. The ranking orders candidates and never adds one; the order +// decides which of them fit the budget. + +// The BM25 parameters, the usual defaults. +const ( + bm25K1 = 1.2 + bm25B = 0.75 +) + +// The fields an item's words are found in, as the diagnostic names them. +const ( + FieldRecall = "recall" + FieldTitle = "title" + FieldBody = "body" +) + +// fieldWeights is each field's weight, in the order fieldsOf returns them: a +// term found in an item's recall words counts three times, in its title +// twice, in its body once. The weighted frequencies are summed per term +// before saturation, and an item's length is its weighted length, as BM25F's +// one pseudo-document of repeated fields prescribes. +var fieldWeights = [3]float64{3, 2, 1} + +// TermScore is one query term's part in an item's score: how many times it +// was found in each field, and what it contributed. +type TermScore struct { + Term string `json:"term"` + Recall int `json:"recall"` + Title int `json:"title"` + Body int `json:"body"` + Contribution float64 `json:"contribution"` +} + +// Ranked is a candidate with its score and the terms behind it. +type Ranked struct { + Candidate + Score float64 + Terms []TermScore +} + +// itemStats is one item's stemmed fields and weighted length. +type itemStats struct { + fields [3]map[string]int + length float64 +} + +func statsOf(it Item) itemStats { + var st itemStats + for i, text := range [3]string{strings.Join(it.Recall, " "), it.Title, it.Body} { + counts := map[string]int{} + stems := rules.TextStems(text) + for _, s := range stems { + counts[s]++ + } + st.fields[i] = counts + st.length += fieldWeights[i] * float64(len(stems)) + } + return st +} + +// found counts a query term in one field: every token whose stem is among +// the roots the term rule matches the prompt token by. +func found(counts map[string]int, t rules.PromptTerm) int { + n := 0 + for _, root := range t.Roots { + n += counts[root] + } + return n +} + +// Rank matches items against prompt and returns the candidates in ranked +// order, each with its score and the terms behind it. The score is BM25F: +// for each query term t found in the item, +// +// idf(t) · tf_w / (tf_w + k1 · (1 − b + b · len / avg_len)) +// +// summed over the terms, with idf(t) = ln(1 + (N − n_t + 0.5) / (n_t + 0.5)) +// over every item in reach (N items, n_t of them holding t), so a word every +// item shares counts for little and a rare word that matches counts for a +// lot. Ties break on the narrower scope (the project's before the +// machine's), then the more recent At, then the key, so the order is the +// same on every machine and every run. +func Rank(items []Item, prompt string) []Ranked { + cands := candidates(items, prompt) + if len(cands) == 0 { + return nil + } + terms := rules.PromptTerms(prompt) + stats := make([]itemStats, len(items)) + stat := map[string]int{} + total := 0.0 + for i, it := range items { + stats[i] = statsOf(it) + stat[it.Key] = i + total += stats[i].length + } + avg := total / float64(len(items)) + idf := make([]float64, len(terms)) + for j, t := range terms { + holding := 0 + for _, st := range stats { + for _, counts := range st.fields { + if found(counts, t) > 0 { + holding++ + break + } + } + } + n, nt := float64(len(items)), float64(holding) + idf[j] = math.Log(1 + (n-nt+0.5)/(nt+0.5)) + } + out := make([]Ranked, 0, len(cands)) + for _, c := range cands { + st := stats[stat[c.Item.Key]] + norm := 1.0 + if avg > 0 { + norm = 1 - bm25B + bm25B*st.length/avg + } + r := Ranked{Candidate: c} + for j, t := range terms { + ts := TermScore{Term: t.Stem, Recall: found(st.fields[0], t), Title: found(st.fields[1], t), Body: found(st.fields[2], t)} + tf := fieldWeights[0]*float64(ts.Recall) + fieldWeights[1]*float64(ts.Title) + fieldWeights[2]*float64(ts.Body) + if tf == 0 { + continue + } + ts.Contribution = idf[j] * tf / (tf + bm25K1*norm) + r.Score += ts.Contribution + r.Terms = append(r.Terms, ts) + } + out = append(out, r) + } + sort.SliceStable(out, func(i, j int) bool { return rankedBefore(out[i], out[j]) }) + return out +} + +// rankedBefore is the ranked order: the higher score, then the narrower +// scope, then the more recent At, then the key. +func rankedBefore(a, b Ranked) bool { + if a.Score != b.Score { + return a.Score > b.Score + } + if ap, bp := a.Item.Layer == LayerProject, b.Item.Layer == LayerProject; ap != bp { + return ap + } + if !a.Item.At.Equal(b.Item.At) { + return a.Item.At.After(b.Item.At) + } + return a.Item.Key < b.Item.Key +} diff --git a/internal/core/recall/ranking_test.go b/internal/core/recall/ranking_test.go new file mode 100644 index 0000000000..f9918d6d83 --- /dev/null +++ b/internal/core/recall/ranking_test.go @@ -0,0 +1,252 @@ +package recall + +import ( + "flag" + "fmt" + "math" + "math/rand/v2" + "os" + "path/filepath" + "slices" + "strings" + "testing" + "time" + + "github.com/intentdriven/abcd/internal/core/rules" +) + +var update = flag.Bool("update", false, "rewrite testdata/ranking.golden from this run's scores") + +// item is a test item: a note-shaped key in the project's layer, written at +// the zero time, unless the caller changes it. +func item(key string, recall []string, title, body string) Item { + return Item{Key: key, Kind: KindNote, Layer: LayerProject, Title: title, Body: body, Recall: recall} +} + +func rankedKeys(rs []Ranked) []string { + var out []string + for _, r := range rs { + out = append(out, r.Item.Key) + } + return out +} + +// TestRareMatchingWordOutranksACommonOne: the case a raw count of matching +// recall words gets wrong. Every item shares "deploy" and "service"; one +// alone carries "kerberos". The prompt names all three, so a count puts the +// two-word matches first, while the ranking puts the rare word's item first. +func TestRareMatchingWordOutranksACommonOne(t *testing.T) { + items := []Item{ + item("mem-2610111200000001", []string{"deploy", "service"}, "Deploys roll one service at a time", "Wait for health checks."), + item("mem-2610111200000002", []string{"deploy", "service"}, "A service deploy needs a ticket", "File it first."), + item("mem-2610111200000003", []string{"deploy", "service"}, "Deploy windows are mornings", "Never on a Friday."), + item("mem-2610111200000004", []string{"deploy", "service"}, "Service logs rotate daily", "Grab them early."), + item("mem-2610111200000005", []string{"deploy", "service"}, "Each service owns its deploy script", "Keep it in the repo."), + item("mem-2610111200000006", []string{"kerberos"}, "Kerberos tickets expire in ten hours", "Renew before the long job."), + } + const prompt = "deploy the service behind kerberos" + + // The fixture is the case a count gets wrong: by matching words, the + // kerberos item is last. + count := slices.Clone(candidates(items, prompt)) + slices.SortStableFunc(count, func(a, b Candidate) int { return len(b.Hits) - len(a.Hits) }) + if count[0].Item.Key == "mem-2610111200000006" { + t.Fatal("the fixture does not make a raw count put the common words first") + } + + ranked := Rank(items, prompt) + if len(ranked) != len(items) || ranked[0].Item.Key != "mem-2610111200000006" { + t.Fatalf("ranked %v, want the rare word's item first", rankedKeys(ranked)) + } + if ranked[0].Score <= ranked[1].Score { + t.Fatalf("the rare word's item does not outscore the rest: %v vs %v", ranked[0].Score, ranked[1].Score) + } +} + +// TestRecallFieldOutweighsBody: a term found in an item's recall words +// counts three times, in its title twice, in its body once. Three items of +// the same weighted length differ only in where "auth" is found. +func TestRecallFieldOutweighsBody(t *testing.T) { + items := []Item{ + item("mem-2610111200000001", []string{"token"}, "", "auth filler filler filler"), // body: 3 + 4 = 7 + item("mem-2610111200000002", []string{"token"}, "auth", "filler filler"), // title: 3 + 2 + 2 = 7 + item("mem-2610111200000003", []string{"token", "auth"}, "", "filler"), // recall: 6 + 1 = 7 + } + ranked := Rank(items, "the auth token") + want := []string{"mem-2610111200000003", "mem-2610111200000002", "mem-2610111200000001"} + if got := rankedKeys(ranked); !slices.Equal(got, want) { + t.Fatalf("ranked %v, want recall over title over body %v", got, want) + } + if !(ranked[0].Score > ranked[1].Score && ranked[1].Score > ranked[2].Score) { + t.Fatalf("scores do not fall with the field's weight: %v %v %v", ranked[0].Score, ranked[1].Score, ranked[2].Score) + } +} + +// TestTiesBreakOnScopeThenRecencyThenID: items of equal score come in the +// narrower scope first, then the more recent, then by key. +func TestTiesBreakOnScopeThenRecencyThenID(t *testing.T) { + older := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC) + newer := time.Date(2026, 10, 11, 12, 0, 0, 0, time.UTC) + mk := func(key, layer string, at time.Time) Item { + it := item(key, []string{"auth"}, "The same lesson", "The same words.") + it.Layer, it.At = layer, at + return it + } + items := []Item{ + mk("mem-2610111200000001", LayerMachine, newer), + mk("mem-2610111200000004", LayerProject, older), + mk("mem-2610111200000003", LayerProject, newer), + mk("mem-2610111200000002", LayerProject, newer), + mk("mem-2610111200000005", LayerProject, time.Time{}), + } + ranked := Rank(items, "auth") + for _, r := range ranked[1:] { + if r.Score != ranked[0].Score { + t.Fatalf("the fixture's scores differ, so it tests no tie: %v", ranked) + } + } + want := []string{"mem-2610111200000002", "mem-2610111200000003", "mem-2610111200000004", "mem-2610111200000005", "mem-2610111200000001"} + if got := rankedKeys(ranked); !slices.Equal(got, want) { + t.Fatalf("ranked %v, want %v", got, want) + } +} + +// TestRankingIsDeterministicAcrossRuns: the same items in any order give the +// same ranking, scores and reasons included. +func TestRankingIsDeterministicAcrossRuns(t *testing.T) { + items := goldenCorpus() + const prompt = "the auth tokens expire before the deploy" + want := fmt.Sprint(Rank(items, prompt)) + r := rand.New(rand.NewPCG(1, 2)) + for i := 0; i < 50; i++ { + shuffled := slices.Clone(items) + r.Shuffle(len(shuffled), func(a, b int) { shuffled[a], shuffled[b] = shuffled[b], shuffled[a] }) + if got := fmt.Sprint(Rank(shuffled, prompt)); got != want { + t.Fatalf("run %d ranked differently:\n%s\n---\n%s", i, got, want) + } + } +} + +// TestRankingDecidesWhichItemsFitTheBudget: when only one of two candidates +// fits, the higher-ranked survives, not the one first in id order. +func TestRankingDecidesWhichItemsFitTheBudget(t *testing.T) { + filler := strings.TrimSpace(strings.Repeat("words ", 150)) + items := []Item{ + item("mem-2610111200000001", []string{"deploy"}, "A deploy lesson", filler), + item("mem-2610111200000002", []string{"deploy", "kerberos"}, "A kerberos deploy lesson", filler), + } + budget := len(headerLine) + len(renderItem(items[1])) + 10 + res := Recall(Request{Items: items, Prompt: "deploy with kerberos", Budget: budget}) + if !slices.Equal(res.Recalled, []string{"mem-2610111200000002"}) || !slices.Equal(res.Omitted, []string{"mem-2610111200000001"}) { + t.Fatalf("recalled %v omitted %v, want the higher-ranked item to fit", res.Recalled, res.Omitted) + } + // With room for both, they come back in ranked order. + all := Recall(Request{Items: items, Prompt: "deploy with kerberos", Budget: rules.InjectionBudgetBytes}) + if !slices.Equal(all.Recalled, []string{"mem-2610111200000002", "mem-2610111200000001"}) { + t.Fatalf("recalled %v, want ranked order", all.Recalled) + } +} + +// TestRankReasonsAddUpToTheScore: each matched term names the fields it was +// found in, and the contributions sum to the score. +func TestRankReasonsAddUpToTheScore(t *testing.T) { + for _, r := range Rank(goldenCorpus(), "the auth tokens expire before the deploy") { + sum := 0.0 + for _, ts := range r.Terms { + if ts.Recall+ts.Title+ts.Body == 0 || ts.Contribution <= 0 { + t.Fatalf("%s: a term with no field or no contribution: %+v", r.Item.Key, ts) + } + sum += ts.Contribution + } + if math.Abs(sum-r.Score) > 1e-12 { + t.Fatalf("%s: contributions sum to %v, score %v", r.Item.Key, sum, r.Score) + } + } +} + +// goldenCorpus is a small fixed corpus of notes and pages. +func goldenCorpus() []Item { + at := func(s string) time.Time { + tm, err := time.Parse(time.RFC3339, s) + if err != nil { + panic(err) + } + return tm + } + return []Item{ + {Key: "mem-2610111200000001", Kind: KindNote, Layer: LayerMachine, At: at("2026-10-11T12:00:00Z"), + Title: "Auth tokens expire hourly", Body: "Refresh the token before the hour is out.", Recall: []string{"auth", "token"}}, + {Key: "mem-2610111200000002", Kind: KindNote, Layer: LayerProject, At: at("2026-10-10T09:30:00Z"), + Title: "Deploys need the dry-run flag", Body: "Pass --dry-run to the deploy script first.", Recall: []string{"deploy"}}, + {Key: "mem-2610111200000003", Kind: KindNote, Layer: LayerMachine, At: at("2026-10-09T08:00:00Z"), + Title: "The tests need the race flag", Body: "Run the race-enabled tests before saying they pass.", Recall: []string{"test", "race"}}, + {Key: "page:fact_eng_auth-flow.md", Kind: KindPage, Layer: LayerProject, At: at("2026-10-01T00:00:00Z"), + Title: "Auth uses short-lived tokens", Body: "# Auth uses short-lived tokens\n\nTokens expire after an hour; the deploy rotates the signing key.", Recall: []string{"auth", "login"}}, + {Key: "page:fact_eng_deploy-window.md", Kind: KindPage, Layer: LayerMachine, At: at("2026-09-20T00:00:00Z"), + Title: "Deploy windows", Body: "# Deploy windows\n\nDeploy in the morning, never before a holiday.", Recall: []string{"deploy", "release"}}, + } +} + +// TestRankingGoldenScores: the scores of a small fixed corpus, so a change to +// the formula, its weights or its parameters is a visible diff of +// testdata/ranking.golden (go test -run TestRankingGoldenScores -update). +func TestRankingGoldenScores(t *testing.T) { + var b strings.Builder + for _, prompt := range []string{ + "the auth tokens expire before the deploy", + "deploy the release", + "run the tests with the race detector", + "login fails after the token refresh", + } { + fmt.Fprintf(&b, "prompt: %s\n", prompt) + for _, r := range Rank(goldenCorpus(), prompt) { + fmt.Fprintf(&b, " %s %.6f\n", r.Item.Key, r.Score) + for _, ts := range r.Terms { + fmt.Fprintf(&b, " %s recall=%d title=%d body=%d %.6f\n", ts.Term, ts.Recall, ts.Title, ts.Body, ts.Contribution) + } + } + } + golden := filepath.Join("testdata", "ranking.golden") + if *update { + if err := os.MkdirAll("testdata", 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(golden, []byte(b.String()), 0o644); err != nil { + t.Fatal(err) + } + } + want, err := os.ReadFile(golden) + if err != nil { + t.Fatalf("%v (run with -update to write it)", err) + } + if b.String() != string(want) { + t.Fatalf("the ranking's scores changed; if the change is meant, run with -update and review the diff:\n%s", b.String()) + } +} + +// TestRankingFindsATermWhereTheMatcherHits: a single-word recall word is +// found among the prompt's query terms exactly when the loader's term rule +// hits it, so the ranking weighs the words the matcher matched and no others. +func TestRankingFindsATermWhereTheMatcherHits(t *testing.T) { + words := []string{"test", "tests", "merge", "commit", "attest", "docs", "push", "release"} + prompts := []string{ + "the tests keep failing", "merging the branch", "an attestation", "committing the fix", + "pushes to the remote", "update the docs", "testing the release", "paint a landscape", + } + for _, p := range prompts { + hit := rules.TermMatcher(p) + terms := rules.PromptTerms(p) + for _, w := range words { + stems := rules.TextStems(w) + found := false + for _, term := range terms { + if len(stems) == 1 && slices.Contains(term.Roots, stems[0]) { + found = true + } + } + if found != hit(w) { + t.Errorf("%q in %q: found among the query terms = %v, the term rule hits = %v", w, p, found, hit(w)) + } + } + } +} diff --git a/internal/core/recall/recall.go b/internal/core/recall/recall.go index 532c18fe82..ff55432c1e 100644 --- a/internal/core/recall/recall.go +++ b/internal/core/recall/recall.go @@ -34,11 +34,12 @@ type Result struct { Ledger map[string]string } -// Recall matches the items against the prompt, drops every candidate already -// sent this session with unchanged content, and takes the rest in id order -// while they fit the budget: the first that does not fit stops the take, so -// an item never comes back ahead of one before it in the order. It performs -// no I/O and never reflects prompt bytes into the block. +// Recall matches the items against the prompt, ranks the candidates (Rank), +// drops every one already sent this session with unchanged content, and +// takes the rest in ranked order while they fit the budget: the first that +// does not fit stops the take, so an item never comes back ahead of one +// ranked above it. It performs no I/O and never reflects prompt bytes into +// the block. func Recall(req Request) Result { ledger := map[string]string{} for k, v := range req.Ledger { @@ -49,7 +50,7 @@ func Recall(req Request) Result { return res } var fresh []Item - for _, c := range Candidates(req.Items, req.Prompt) { + for _, c := range Rank(req.Items, req.Prompt) { if ledger[c.Item.Key] == signature(c.Item) { continue } diff --git a/internal/core/recall/recall_test.go b/internal/core/recall/recall_test.go index c31257bdff..f89a9cd165 100644 --- a/internal/core/recall/recall_test.go +++ b/internal/core/recall/recall_test.go @@ -49,13 +49,13 @@ func TestNoteRecallUsesTheRuleTermMatch(t *testing.T) { for _, p := range prompts { rs := rules.RuleSet{SchemaVersion: 1, Domains: map[string]rules.Domain{"PROBE": {Recall: []string{w}, Rules: []string{"r"}}}} domain := len(rs.Match(p)) == 1 - got := len(Candidates(items, p)) == 1 + got := len(candidates(items, p)) == 1 if got != domain { t.Errorf("recall word %q on %q: note candidate = %v, domain match = %v", w, p, got, domain) } } } - c := Candidates(noteItems([]memnote.Note{note("mem-2610111200000001", "A lesson", "Body.", "", "test", "race", "pull request")}, nil, thisRoot), + c := candidates(noteItems([]memnote.Note{note("mem-2610111200000001", "A lesson", "Body.", "", "test", "race", "pull request")}, nil, thisRoot), "run the tests before the pull requests") if len(c) != 1 || !slices.Equal(c[0].Hits, []string{"test", "pull request"}) { t.Fatalf("candidate hits = %+v", c) @@ -149,39 +149,6 @@ func TestMemoryTakesOnlyWhatTheRulesLeaveOfTheBudget(t *testing.T) { } } -// TestCandidatesThatFitAreTakenInIDOrder: until step 4's ranking, the -// candidates are taken in id order, and when only some fit, the survivors are -// the first in that order — the first that does not fit stops the take, so a -// later item never comes back ahead of an earlier one. -func TestCandidatesThatFitAreTakenInIDOrder(t *testing.T) { - body := strings.TrimSpace(strings.Repeat("word ", 150)) // about 750 bytes - notes := []memnote.Note{ - note("mem-2610111200000003", "Third", body, "", "test"), - note("mem-2610111200000001", "First", body, "", "test"), - note("mem-2610111200000004", "Fourth", "Short.", "", "test"), - note("mem-2610111200000002", "Second", body, "", "test"), - } - items := noteItems(notes, nil, thisRoot) - all := recallText(items, "the tests") - want := []string{"mem-2610111200000001", "mem-2610111200000002", "mem-2610111200000003", "mem-2610111200000004"} - if !slices.Equal(all.Recalled, want) { - t.Fatalf("recalled %v, want id order %v", all.Recalled, want) - } - if i, j := strings.Index(all.Text, "First"), strings.Index(all.Text, "Second"); i < 0 || j < i { - t.Fatalf("the block is not in id order:\n%s", all.Text) - } - // Room for two of the long ones: the first two in id order survive, and - // the short fourth, which would fit, does not jump the queue. - budget := len(headerLine) + 2*len(renderItem(items[0])) + 10 - some := Recall(Request{Items: items, Prompt: "the tests", Budget: budget}) - if !slices.Equal(some.Recalled, want[:2]) || !slices.Equal(some.Omitted, want[2:]) { - t.Fatalf("recalled %v omitted %v", some.Recalled, some.Omitted) - } - if len(some.Text) > budget { - t.Fatalf("the block is over its budget: %d > %d", len(some.Text), budget) - } -} - // TestProjectNoteSuppressesTheMachineCopy: an id in both layers (copied by // hand) is the project's. func TestProjectNoteSuppressesTheMachineCopy(t *testing.T) { diff --git a/internal/core/recall/testdata/ranking.golden b/internal/core/recall/testdata/ranking.golden new file mode 100644 index 0000000000..df994ddcfd --- /dev/null +++ b/internal/core/recall/testdata/ranking.golden @@ -0,0 +1,44 @@ +prompt: the auth tokens expire before the deploy + mem-2610111200000001 2.335928 + the recall=0 title=0 body=2 0.056166 + auth recall=1 title=1 body=0 0.717787 + token recall=1 title=1 body=1 0.740001 + expire recall=0 title=1 body=0 0.565113 + before recall=0 title=0 body=1 0.256862 + page:fact_eng_auth-flow.md 1.970187 + the recall=0 title=0 body=2 0.050277 + auth recall=1 title=1 body=1 0.704009 + token recall=0 title=1 body=2 0.641218 + expire recall=0 title=0 body=1 0.355695 + deploy recall=0 title=0 body=1 0.218989 + page:fact_eng_deploy-window.md 0.779122 + the recall=0 title=0 body=1 0.042951 + before recall=0 title=0 body=1 0.266065 + deploy recall=1 title=1 body=2 0.470105 + mem-2610111200000002 0.516409 + the recall=0 title=1 body=1 0.063133 + deploy recall=1 title=1 body=1 0.453276 +prompt: deploy the release + page:fact_eng_deploy-window.md 1.546113 + deploy recall=1 title=1 body=2 0.470105 + the recall=0 title=0 body=1 0.042951 + release recall=1 title=0 body=0 1.033056 + mem-2610111200000002 0.516409 + deploy recall=1 title=1 body=1 0.453276 + the recall=0 title=1 body=1 0.063133 +prompt: run the tests with the race detector + mem-2610111200000003 2.738434 + run recall=0 title=0 body=1 0.384005 + the recall=0 title=2 body=1 0.069278 + test recall=1 title=1 body=1 1.142576 + race recall=1 title=1 body=1 1.142576 +prompt: login fails after the token refresh + page:fact_eng_auth-flow.md 2.186949 + login recall=1 title=0 body=0 0.932215 + after recall=0 title=0 body=1 0.563239 + the recall=0 title=0 body=2 0.050277 + token recall=0 title=1 body=2 0.641218 + mem-2610111200000001 1.456814 + the recall=0 title=0 body=2 0.056166 + token recall=1 title=1 body=1 0.740001 + refresh recall=0 title=0 body=1 0.660648 diff --git a/internal/core/rules/rules.go b/internal/core/rules/rules.go index 007d8a6e6c..c647be2015 100644 --- a/internal/core/rules/rules.go +++ b/internal/core/rules/rules.go @@ -30,6 +30,7 @@ import ( "io" "os" "regexp" + "slices" "sort" "strings" @@ -707,6 +708,17 @@ type promptIndex struct { padded string // " tok tok " for boundary/phrase matching stemmedPadded string // " stem stem " for stemmed phrase matching stems map[string]bool // candidate stems of the single tokens + terms []PromptTerm // each distinct stem, with its candidate roots +} + +// PromptTerm is one distinct stem of a prompt's tokens, as the memory +// recall's ranking weighs it (spc-2610111627213138 step 4): Stem is the root +// stem gives the token, and Roots every candidate root the term rule matches +// the token by (stemVariants), so a word whose stem is among them is the term +// found exactly where termHit would hit it. +type PromptTerm struct { + Stem string + Roots []string } // indexPrompt lowercases, collapses non-alphanumeric runs to single spaces, and @@ -717,11 +729,25 @@ func indexPrompt(s string) promptIndex { idx := promptIndex{padded: " " + collapsed + " ", stems: map[string]bool{}} tokens := strings.Fields(collapsed) stemmed := make([]string, 0, len(tokens)) + at := map[string]int{} for _, tok := range tokens { - for _, v := range stemVariants(tok) { + variants := stemVariants(tok) + for _, v := range variants { idx.stems[v] = true } - stemmed = append(stemmed, stem(tok)) + s := stem(tok) + stemmed = append(stemmed, s) + i, seen := at[s] + if !seen { + i = len(idx.terms) + at[s] = i + idx.terms = append(idx.terms, PromptTerm{Stem: s}) + } + for _, v := range variants { + if !slices.Contains(idx.terms[i].Roots, v) { + idx.terms[i].Roots = append(idx.terms[i].Roots, v) + } + } } idx.stemmedPadded = " " + strings.Join(stemmed, " ") + " " return idx @@ -1023,6 +1049,29 @@ func TermMatcher(prompt string) func(term string) bool { return idx.termHit } +// PromptTerms is the prompt index's query terms for the memory recall's +// ranking: each distinct stem of the prompt's tokens, in the order the prompt +// first uses it, with its candidate roots. It reads the bytes TermMatcher +// reads, from the same index. +func PromptTerms(prompt string) []PromptTerm { + if len(prompt) > maxPromptBytes { + prompt = prompt[:maxPromptBytes] + } + return indexPrompt(prompt).terms +} + +// TextStems is the stem of each token of text, in order — lower-cased, split +// on every run of characters that is not a letter or a digit, each token +// stemmed as termHit stems a recall keyword's token — so the ranking counts a +// note's or page's words in the form the term rule compares them in. +func TextStems(text string) []string { + tokens := strings.Fields(nonAlnumRe.ReplaceAllString(strings.ToLower(text), " ")) + for i, tok := range tokens { + tokens[i] = stem(tok) + } + return tokens +} + // Hits names what made d match prompt, for the `rules --prompt` diagnostic: // "*NAME" when a star-command activated it, otherwise each recall word and // alias that hits, in the order the domain declares them. It says nothing of diff --git a/internal/surface/cli/cli.go b/internal/surface/cli/cli.go index ae17ec1b95..2010ac247d 100644 --- a/internal/surface/cli/cli.go +++ b/internal/surface/cli/cli.go @@ -2445,7 +2445,11 @@ session: each rule domain it matches, with the recall words that hit, and each memory note it recalls from ` + abcdhome.Display("memory.json") + ` and .abcd/memory.json, and each memory page from .abcd/memory/ and ` + abcdhome.Display("memory") + `/, with its layer and the recall words that hit, each in full or as not fitting the -per-turn budget. It reads against an empty ledger and writes nothing. +per-turn budget. Notes and pages share one ranked order, a BM25F relevance +score that decides which of them fit the budget: each shows its rank and +score and, under it, every matched term with the fields it was found in +(recall words, title, body) and what it added to the score. It reads +against an empty ledger and writes nothing. --prompt and a DOMAIN are two forms and are refused together, exit 2. Read-only.`, Args: cobra.MaximumNArgs(1), diff --git a/internal/surface/cli/memory_recall_test.go b/internal/surface/cli/memory_recall_test.go index 7f4297c3cc..05125edbd9 100644 --- a/internal/surface/cli/memory_recall_test.go +++ b/internal/surface/cli/memory_recall_test.go @@ -422,3 +422,66 @@ func TestARefusedPageIsNamedAndTheRestComeBack(t *testing.T) { t.Fatalf("stderr carries the expanded home:\n%s", errlog) } } + +// TestRulesPromptShowsEachTermsContribution: `rules --prompt` ranks the +// notes and pages the prompt recalls, one order across both, and shows each +// one's score and, for each matched term, the fields it was found in and +// what it contributed; --json carries the same, the contributions summing to +// the score. +func TestRulesPromptShowsEachTermsContribution(t *testing.T) { + t.Setenv("ABCD_RULES_STATE_DIR", t.TempDir()) + _, repo := memoryFixture(t) + id := addRaceNote(t, "session-one") + writeRecallPage(t, filepath.Join(repo, ".abcd", "memory"), "fact_eng_test-layout.md", + "# Where the tests live\n\nBeside their package.\n", 0o644, "test") + + out, stderr, err := runCLISplit(t, "rules", "--prompt", "run the race tests") + if err != nil { + t.Fatalf("rules --prompt: %v\n%s", err, stderr) + } + for _, want := range []string{"#1 " + id, "#2 page:fact_eng_test-layout.md", "score ", + "race — recall 1, title 1", "test — recall 1, title 1, body 1 — "} { + if !strings.Contains(out, want) { + t.Fatalf("rules --prompt does not show %q:\n%s", want, out) + } + } + + jout, stderr, err := runCLISplit(t, "rules", "--prompt", "run the race tests", "--json") + if err != nil { + t.Fatalf("rules --prompt --json: %v\n%s", err, stderr) + } + type ranked struct { + Rank int `json:"rank"` + Score float64 `json:"score"` + Terms []struct { + Term string `json:"term"` + Recall int `json:"recall"` + Title int `json:"title"` + Body int `json:"body"` + Contribution float64 `json:"contribution"` + } `json:"terms"` + } + var env struct { + Notes []ranked `json:"notes"` + Pages []ranked `json:"pages"` + } + if err := json.Unmarshal([]byte(jout), &env); err != nil { + t.Fatalf("rules --prompt --json is not one document: %v\n%s", err, jout) + } + if len(env.Notes) != 1 || len(env.Pages) != 1 || env.Notes[0].Rank != 1 || env.Pages[0].Rank != 2 || + env.Notes[0].Score <= env.Pages[0].Score { + t.Fatalf("notes %+v pages %+v", env.Notes, env.Pages) + } + for _, r := range append(env.Notes, env.Pages...) { + sum := 0.0 + for _, term := range r.Terms { + if term.Recall+term.Title+term.Body == 0 || term.Contribution <= 0 { + t.Fatalf("a term with no field or no contribution: %+v", term) + } + sum += term.Contribution + } + if len(r.Terms) == 0 || sum < r.Score-1e-9 || sum > r.Score+1e-9 { + t.Fatalf("the terms %+v do not add up to the score %v", r.Terms, r.Score) + } + } +} diff --git a/internal/surface/cli/rules_prompt.go b/internal/surface/cli/rules_prompt.go index 84b7f5aca8..ed2c3853b4 100644 --- a/internal/surface/cli/rules_prompt.go +++ b/internal/surface/cli/rules_prompt.go @@ -67,24 +67,34 @@ type rulesPromptDomain struct { Form string `json:"form"` } -// rulesPromptNote is one matched note in the diagnostic. +// rulesPromptNote is one matched note in the diagnostic. Rank is its place in +// the one ranked order notes and pages share (1 first), Score its BM25F +// score, and Terms each matched term with the fields it was found in and its +// contribution (recall.Rank). type rulesPromptNote struct { - ID string `json:"id"` - Title string `json:"title"` - Layer string `json:"layer"` - Hits []string `json:"hits"` - Form string `json:"form"` + ID string `json:"id"` + Title string `json:"title"` + Layer string `json:"layer"` + Hits []string `json:"hits"` + Rank int `json:"rank"` + Score float64 `json:"score"` + Terms []recall.TermScore `json:"terms"` + Form string `json:"form"` } // rulesPromptPage is one matched page in the diagnostic: its key is the one -// the router's envelope names it by (page:). +// the router's envelope names it by (page:); Rank, Score and Terms +// are a note's. type rulesPromptPage struct { - Key string `json:"key"` - Filename string `json:"filename"` - Title string `json:"title"` - Layer string `json:"layer"` - Hits []string `json:"hits"` - Form string `json:"form"` + Key string `json:"key"` + Filename string `json:"filename"` + Title string `json:"title"` + Layer string `json:"layer"` + Hits []string `json:"hits"` + Rank int `json:"rank"` + Score float64 `json:"score"` + Terms []recall.TermScore `json:"terms"` + Form string `json:"form"` } // rulesPromptView is `rules --prompt`'s --json envelope. @@ -126,17 +136,22 @@ func rulesPrompt(rs rules.RuleSet, root, prompt string) rulesPromptView { for _, key := range mem.Recalled { recalled[key] = true } - for _, c := range recall.Candidates(items, prompt) { + for i, c := range recall.Rank(items, prompt) { form := promptFormOverBudget if recalled[c.Item.Key] { form = promptFormFull } + terms := c.Terms + if terms == nil { + terms = []recall.TermScore{} + } if c.Item.Kind == recall.KindPage { v.Pages = append(v.Pages, rulesPromptPage{Key: c.Item.Key, Filename: strings.TrimPrefix(c.Item.Key, recall.PageKeyPrefix), - Title: c.Item.Title, Layer: c.Item.Layer, Hits: c.Hits, Form: form}) + Title: c.Item.Title, Layer: c.Item.Layer, Hits: c.Hits, Rank: i + 1, Score: c.Score, Terms: terms, Form: form}) continue } - v.Notes = append(v.Notes, rulesPromptNote{ID: c.Item.Key, Title: c.Item.Title, Layer: c.Item.Layer, Hits: c.Hits, Form: form}) + v.Notes = append(v.Notes, rulesPromptNote{ID: c.Item.Key, Title: c.Item.Title, Layer: c.Item.Layer, Hits: c.Hits, + Rank: i + 1, Score: c.Score, Terms: terms, Form: form}) } return v } @@ -148,6 +163,9 @@ func renderRulesPrompt(w io.Writer, rs rules.RuleSet, v rulesPromptView) { return } fmt.Fprintln(w, "abcd rules --prompt — what this prompt brings back to a fresh session (nothing written)") + if len(v.Notes)+len(v.Pages) > 0 { + fmt.Fprintln(w, "Notes and pages share one ranked order (#1 first), by relevance score (BM25F); under each, every matched term, the fields it was found in and what it added.") + } if len(v.Domains) == 0 { fmt.Fprintln(w, "Domains: none matched") } else { @@ -161,8 +179,9 @@ func renderRulesPrompt(w io.Writer, rs rules.RuleSet, v rulesPromptView) { } else { fmt.Fprintln(w, "Notes:") for _, n := range v.Notes { - fmt.Fprintf(w, " %s %s (%s layer) — hit by %s — %s\n", n.ID, termsafe.Sanitize(n.Title), n.Layer, - termsafe.Sanitize(strings.Join(n.Hits, ", ")), promptFormWords(n.Form)) + fmt.Fprintf(w, " #%d %s %s (%s layer) — hit by %s — score %.4f — %s\n", n.Rank, n.ID, termsafe.Sanitize(n.Title), n.Layer, + termsafe.Sanitize(strings.Join(n.Hits, ", ")), n.Score, promptFormWords(n.Form)) + renderTermScores(w, n.Terms) } } if len(v.Pages) == 0 { @@ -170,8 +189,9 @@ func renderRulesPrompt(w io.Writer, rs rules.RuleSet, v rulesPromptView) { } else { fmt.Fprintln(w, "Pages:") for _, p := range v.Pages { - fmt.Fprintf(w, " %s %s (%s layer) — hit by %s — %s\n", p.Key, termsafe.Sanitize(p.Title), p.Layer, - termsafe.Sanitize(strings.Join(p.Hits, ", ")), promptFormWords(p.Form)) + fmt.Fprintf(w, " #%d %s %s (%s layer) — hit by %s — score %.4f — %s\n", p.Rank, p.Key, termsafe.Sanitize(p.Title), p.Layer, + termsafe.Sanitize(strings.Join(p.Hits, ", ")), p.Score, promptFormWords(p.Form)) + renderTermScores(w, p.Terms) } } for _, u := range v.Unreadable { @@ -179,6 +199,23 @@ func renderRulesPrompt(w io.Writer, rs rules.RuleSet, v rulesPromptView) { } } +// renderTermScores writes, under a ranked note or page, each matched term +// with the fields it was found in and what it contributed to the score. +func renderTermScores(w io.Writer, terms []recall.TermScore) { + for _, ts := range terms { + var found []string + for _, f := range []struct { + name string + n int + }{{recall.FieldRecall, ts.Recall}, {recall.FieldTitle, ts.Title}, {recall.FieldBody, ts.Body}} { + if f.n > 0 { + found = append(found, fmt.Sprintf("%s %d", f.name, f.n)) + } + } + fmt.Fprintf(w, " %s — %s — %.4f\n", ts.Term, strings.Join(found, ", "), ts.Contribution) + } +} + // promptFormWords is a form as the plain diagnostic says it. func promptFormWords(form string) string { if form == promptFormFull { From 98656c49de1aab666c3b4165411882a26a06ceaa Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:29:31 +0000 Subject: [PATCH 04/23] docs: record the choices step 4 made where the memory-recall spec is silent The spec's Settled here section gains step 4's choices, dated and marked as an agent's choice for review: a page's recency is its latest ingested_at, how the ranking counts terms and lengths (with the two rules exports it needed), the ranked take keeping step 2's stop at the first item that does not fit, and the diagnostic ranking notes and pages in one order. Refs: itd-2610091918433290, itd-39 Assisted-by: Claude:claude-opus-5-5 --- .../spc-2610111627213138-memory-recall.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md b/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md index 632277054f..f6f82879c7 100644 --- a/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md +++ b/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md @@ -762,6 +762,35 @@ choices above. step 3, agent's choice for review): on the hook's stderr with "recalling nothing from it", and in `rules --prompt`'s `unreadable` list, which gains a `kind` (`note` or `page`); a refused `memory.json` keeps step 2's words. +- **What a page's recency is** (2026-10-12, step 4, agent's choice for + review). The tie-break's "page modification" is the latest `ingested_at` + the page's source block records, a date, and a page that records none + sorts as the oldest. Grounds: a checkout stamps each file with the time it + wrote it out, so a file's modification time differs between clones and the + order would not be the same on every machine, as Ranking requires; + `ingested_at` is in the page and travels with it. +- **How the ranking counts terms and lengths** (2026-10-12, step 4, agent's + choice for review). Each distinct stem of the prompt's tokens is one query + term, however often the prompt repeats it, and a word in an item counts as + that term when its stem is among the roots the loader's term rule matches + the prompt's token by, so the ranking finds a term exactly where the + matcher hits. That takes two small exports from `internal/core/rules` + beside `TermMatcher` (`PromptTerms`, read from the same prompt index, and + `TextStems`), though step 4's packages line names only the recall and the + CLI. An item's length is its weighted length, recall words three times, + title twice, body once, BM25F's one pseudo-document of repeated fields; + `avg_len` and N run over every item in reach after the root filter and the + narrower-scope suppression, so a page without a recall list, never an + item, counts in neither. +- **The ranked order keeps step 2's take** (2026-10-12, step 4, agent's + choice for review): candidates are taken in ranked order while they fit, + and the first that does not fit stops the take, so nothing comes back + ahead of an item ranked above it. +- **The diagnostic ranks notes and pages together** (2026-10-12, step 4, + agent's choice for review): `rules --prompt` keeps its `notes` and `pages` + lists, and each entry carries its `rank` in the one order they share, its + `score`, and its `terms`, each with its counts in `recall`, `title` and + `body` and its `contribution`. ## Open Questions From d63f67554a76b7e3720a62d1716dd770d236a82f Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:42:07 +0000 Subject: [PATCH 05/23] fix: refuse a machine page directory another account can change The machine page scope was opened through the home scope's walk, which refuses a symlinked directory but judges nothing about who can write one, so a ~/.abcd.noindex every account could write had its notes file refused and its pages beside it read and injected. The scope is now opened through fsutil.OpenHomeDeclarationScope, the walk with the directory judgement ReadHomeDeclaration applies, and absence is decided first, as it decides it, so an exposed home with no page scope still costs nothing. Refs: itd-2610091918433290, itd-39 Assisted-by: Claude:claude-opus-5-5 --- internal/core/memory/recallpages.go | 25 ++++++++++++----- internal/core/memory/recallpages_test.go | 34 ++++++++++++++++++++++++ internal/fsutil/home.go | 12 +++++++++ 3 files changed, 64 insertions(+), 7 deletions(-) diff --git a/internal/core/memory/recallpages.go b/internal/core/memory/recallpages.go index 9317c1f860..9433b355e5 100644 --- a/internal/core/memory/recallpages.go +++ b/internal/core/memory/recallpages.go @@ -64,17 +64,28 @@ func ReadRecallPages(repoRoot string) ([]RecallPage, []RecallRefusal) { } // ReadMachineRecallPages reads the machine scope, ~/.abcd.noindex/memory/ -// under home, read only: the directory is opened through -// fsutil.OpenHomeScope, so a symlinked ~/.abcd.noindex or memory/ is refused, -// and each page through the declaration's guarded read relative to that -// descriptor, as the machine memory.json is read: a regular file this account -// owns that no one else can write. An absent scope, no absolute home, or a -// home or ~/.abcd.noindex this account cannot search yields nothing. +// under home, read only, with the guards the machine memory.json is read +// with: the directory is opened through fsutil.OpenHomeDeclarationScope, so a +// symlinked ~/.abcd.noindex or memory/ is refused, and so is one another +// account can change; each page is read through the declaration's guarded +// read relative to that descriptor, a regular file this account owns that no +// one else can write. An absent scope, no absolute home, or a home or +// ~/.abcd.noindex this account cannot search yields nothing, whatever the +// directories above it are. func ReadMachineRecallPages(home string) ([]RecallPage, []RecallRefusal) { if home == "" || !filepath.IsAbs(home) { return nil, nil } - root, err := fsutil.OpenHomeScope(home, abcdhome.Rel("memory")) + // Absence is decided before any directory is judged, as + // ReadHomeDeclaration decides it, so an exposed ~/.abcd.noindex that + // holds no page scope costs nothing and says nothing. + if _, err := os.Lstat(abcdhome.Path(home, "memory")); err != nil { + if errors.Is(err, fs.ErrNotExist) || errors.Is(err, syscall.ENOTDIR) || errors.Is(err, fs.ErrPermission) { + return nil, nil + } + return nil, []RecallRefusal{{File: machineStoreDisplay, Err: recallReadError(err)}} + } + root, err := fsutil.OpenHomeDeclarationScope(home, abcdhome.Rel("memory")) if err != nil { if errors.Is(err, fs.ErrNotExist) || errors.Is(err, syscall.ENOTDIR) || errors.Is(err, fs.ErrPermission) { return nil, nil diff --git a/internal/core/memory/recallpages_test.go b/internal/core/memory/recallpages_test.go index 6ae5421296..2ef55aa93d 100644 --- a/internal/core/memory/recallpages_test.go +++ b/internal/core/memory/recallpages_test.go @@ -197,3 +197,37 @@ func TestRecallPagesCarryTheirLatestIngestDate(t *testing.T) { t.Fatalf("ingest dates = %v", at) } } + +// TestMachineRecallPagesRefuseADirectoryOthersCanChange: the machine scope's +// directories are judged as the machine memory.json's are — one every +// account can write is refused, its pages unread, and named in tilde form — +// while an exposed ~/.abcd.noindex holding no memory/ still reads as absent. +func TestMachineRecallPagesRefuseADirectoryOthersCanChange(t *testing.T) { + for name, exposed := range map[string][]string{"abcd-home": nil, "memory": {"memory"}} { + t.Run(name, func(t *testing.T) { + home := t.TempDir() + dir := abcdhome.Path(home, "memory") + writeStoreFile(t, dir, "fact_eng_auth-flow.md", recallPageText("Anyone could have put this here", "auth"), 0o600) + if err := os.Chmod(abcdhome.Path(home, exposed...), 0o777); err != nil { + t.Fatal(err) + } + pages, refused := ReadMachineRecallPages(home) + if len(pages) != 0 || len(refused) != 1 || !errors.Is(refused[0].Err, fsutil.ErrHomeScopeExposed) { + t.Fatalf("an exposed directory's pages were read: pages %+v refused %+v", pages, refused) + } + if msg := refused[0].Err.Error(); strings.Contains(msg, home) || !strings.Contains(msg, abcdhome.Display(exposed...)+" can be written by every account") { + t.Fatalf("the refusal does not name the directory in tilde form: %s", msg) + } + }) + } + home := t.TempDir() + if err := os.MkdirAll(abcdhome.Path(home), 0o700); err != nil { + t.Fatal(err) + } + if err := os.Chmod(abcdhome.Path(home), 0o777); err != nil { + t.Fatal(err) + } + if pages, refused := ReadMachineRecallPages(home); len(pages) != 0 || len(refused) != 0 { + t.Fatalf("an exposed home with no page scope read %+v %+v", pages, refused) + } +} diff --git a/internal/fsutil/home.go b/internal/fsutil/home.go index f32b4cbb2a..744eeab85c 100644 --- a/internal/fsutil/home.go +++ b/internal/fsutil/home.go @@ -220,6 +220,18 @@ func OpenHomeScope(home, dir string) (*os.Root, error) { return openHomeScope(home, dir, false, 0, nil) } +// OpenHomeDeclarationScope is OpenHomeScope for a reader that takes every +// file below dir as the caller's word, as ReadHomeDeclaration takes one: +// each directory level below home is also judged on its own descriptor, and +// one every account can write, or one owned by an account that is neither +// this uid nor root, is refused with a *HomeScopeExposedError naming it +// (ErrHomeScopeExposed), because whoever can change the directory can put a +// file of the caller's own shape in it. The files read through the root +// still need their own guards (ReadHomeDeclarationDenyingIn). +func OpenHomeDeclarationScope(home, dir string) (*os.Root, error) { + return openHomeScope(home, dir, false, 0, vetDeclarationDir) +} + // EnsureHomeScope is OpenHomeScope for a writer: each missing level is created // at perm (a single mkdir relative to the level above, which never follows a // link at the name it creates) before it is judged and opened, so the walk From 3f47c5408c9b2dc2b2ba5f96c8fe81cb958d3800 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:44:13 +0000 Subject: [PATCH 06/23] fix: count and send a page's title once A page's title is the first line of its body, the line its index summary comes from, and the body the recall weighed and rendered still carried it, so a title word counted in the title and again in the body (as much as a recall word) and the injected block repeated it. The page's body is now the rest of the page after that line. The golden corpus models pages the same way, and its scores are regenerated. Refs: itd-2610091918433290, itd-39 Assisted-by: Claude:claude-opus-5-5 --- internal/core/memory/recallpages.go | 15 +++- internal/core/memory/recallpages_test.go | 16 ++++ internal/core/recall/ranking_test.go | 4 +- internal/core/recall/testdata/ranking.golden | 80 ++++++++++---------- 4 files changed, 71 insertions(+), 44 deletions(-) diff --git a/internal/core/memory/recallpages.go b/internal/core/memory/recallpages.go index 9433b355e5..09aae0dc70 100644 --- a/internal/core/memory/recallpages.go +++ b/internal/core/memory/recallpages.go @@ -26,7 +26,8 @@ type RecallPage struct { // Summary is the page's index.md summary, derived from the page as // RenderIndex derives it, so the recall reads no index.md. Summary string - // Body is the page's text after its frontmatter. + // Body is the page's text after its frontmatter, less the line Summary + // comes from, trimmed: the title is not counted, or sent, twice. Body string // Recall is the page's recall: frontmatter list, empty when it has none. Recall []string @@ -117,12 +118,22 @@ func recallPagesFrom(store *storeHandle, shown string, read func(string, int64) } p := parsePage(string(raw)) info := pageInfoOf(name, p) - pages = append(pages, RecallPage{Filename: name, Summary: info.Summary, Body: p.body, Recall: recallList(p.fm), + pages = append(pages, RecallPage{Filename: name, Summary: info.Summary, Body: bodyAfterSummary(p.body), Recall: recallList(p.fm), IngestedAt: latestIngest(p.source())}) } return pages, refused } +// bodyAfterSummary is body less its first non-blank line, the one +// pageInfoOf takes the summary from, trimmed. +func bodyAfterSummary(body string) string { + rest := strings.TrimLeft(body, "\r\n\t ") + if i := strings.IndexByte(rest, '\n'); i >= 0 { + return strings.TrimSpace(rest[i+1:]) + } + return "" +} + // recallList is a page's recall: list, its non-blank string entries trimmed. func recallList(fm map[string]any) []string { raw, ok := fm["recall"].([]any) diff --git a/internal/core/memory/recallpages_test.go b/internal/core/memory/recallpages_test.go index 2ef55aa93d..bb0ba594d7 100644 --- a/internal/core/memory/recallpages_test.go +++ b/internal/core/memory/recallpages_test.go @@ -231,3 +231,19 @@ func TestMachineRecallPagesRefuseADirectoryOthersCanChange(t *testing.T) { t.Fatalf("an exposed home with no page scope read %+v %+v", pages, refused) } } + +// TestRecallPageBodyDoesNotRepeatItsTitle: a page's title is the line its +// index summary comes from, so the body the recall weighs and renders is the +// rest of the page; the title's words are not counted, or sent, twice. +func TestRecallPageBodyDoesNotRepeatItsTitle(t *testing.T) { + repo := t.TempDir() + writeStoreFile(t, filepath.Join(repo, ".abcd", "memory"), "fact_eng_auth-flow.md", + "---\nrecall: [auth]\n---\n\n# Auth uses short-lived tokens\n\nRefresh them hourly.\n# A later heading stays\n", 0o644) + pages, _ := ReadRecallPages(repo) + if len(pages) != 1 { + t.Fatalf("pages = %+v", pages) + } + if p := pages[0]; p.Summary != "Auth uses short-lived tokens" || p.Body != "Refresh them hourly.\n# A later heading stays" { + t.Fatalf("summary %q body %q", p.Summary, p.Body) + } +} diff --git a/internal/core/recall/ranking_test.go b/internal/core/recall/ranking_test.go index f9918d6d83..8a30d0ed9a 100644 --- a/internal/core/recall/ranking_test.go +++ b/internal/core/recall/ranking_test.go @@ -181,9 +181,9 @@ func goldenCorpus() []Item { {Key: "mem-2610111200000003", Kind: KindNote, Layer: LayerMachine, At: at("2026-10-09T08:00:00Z"), Title: "The tests need the race flag", Body: "Run the race-enabled tests before saying they pass.", Recall: []string{"test", "race"}}, {Key: "page:fact_eng_auth-flow.md", Kind: KindPage, Layer: LayerProject, At: at("2026-10-01T00:00:00Z"), - Title: "Auth uses short-lived tokens", Body: "# Auth uses short-lived tokens\n\nTokens expire after an hour; the deploy rotates the signing key.", Recall: []string{"auth", "login"}}, + Title: "Auth uses short-lived tokens", Body: "Tokens expire after an hour; the deploy rotates the signing key.", Recall: []string{"auth", "login"}}, {Key: "page:fact_eng_deploy-window.md", Kind: KindPage, Layer: LayerMachine, At: at("2026-09-20T00:00:00Z"), - Title: "Deploy windows", Body: "# Deploy windows\n\nDeploy in the morning, never before a holiday.", Recall: []string{"deploy", "release"}}, + Title: "Deploy windows", Body: "Deploy in the morning, never before a holiday.", Recall: []string{"deploy", "release"}}, } } diff --git a/internal/core/recall/testdata/ranking.golden b/internal/core/recall/testdata/ranking.golden index df994ddcfd..b52ab00421 100644 --- a/internal/core/recall/testdata/ranking.golden +++ b/internal/core/recall/testdata/ranking.golden @@ -1,44 +1,44 @@ prompt: the auth tokens expire before the deploy - mem-2610111200000001 2.335928 - the recall=0 title=0 body=2 0.056166 - auth recall=1 title=1 body=0 0.717787 - token recall=1 title=1 body=1 0.740001 - expire recall=0 title=1 body=0 0.565113 - before recall=0 title=0 body=1 0.256862 - page:fact_eng_auth-flow.md 1.970187 - the recall=0 title=0 body=2 0.050277 - auth recall=1 title=1 body=1 0.704009 - token recall=0 title=1 body=2 0.641218 - expire recall=0 title=0 body=1 0.355695 - deploy recall=0 title=0 body=1 0.218989 - page:fact_eng_deploy-window.md 0.779122 - the recall=0 title=0 body=1 0.042951 - before recall=0 title=0 body=1 0.266065 - deploy recall=1 title=1 body=2 0.470105 - mem-2610111200000002 0.516409 - the recall=0 title=1 body=1 0.063133 - deploy recall=1 title=1 body=1 0.453276 + mem-2610111200000001 2.310254 + the recall=0 title=0 body=2 0.055313 + auth recall=1 title=1 body=0 0.712209 + token recall=1 title=1 body=1 0.735054 + expire recall=0 title=1 body=0 0.556533 + before recall=0 title=0 body=1 0.251145 + page:fact_eng_auth-flow.md 1.952974 + the recall=0 title=0 body=2 0.052127 + auth recall=1 title=1 body=0 0.690600 + token recall=0 title=1 body=1 0.605377 + expire recall=0 title=0 body=1 0.374378 + deploy recall=0 title=0 body=1 0.230492 + page:fact_eng_deploy-window.md 0.776718 + the recall=0 title=0 body=1 0.043674 + before recall=0 title=0 body=1 0.270539 + deploy recall=1 title=1 body=1 0.462505 + mem-2610111200000002 0.512505 + the recall=0 title=1 body=1 0.062379 + deploy recall=1 title=1 body=1 0.450126 prompt: deploy the release - page:fact_eng_deploy-window.md 1.546113 - deploy recall=1 title=1 body=2 0.470105 - the recall=0 title=0 body=1 0.042951 - release recall=1 title=0 body=0 1.033056 - mem-2610111200000002 0.516409 - deploy recall=1 title=1 body=1 0.453276 - the recall=0 title=1 body=1 0.063133 + page:fact_eng_deploy-window.md 1.547903 + deploy recall=1 title=1 body=1 0.462505 + the recall=0 title=0 body=1 0.043674 + release recall=1 title=0 body=0 1.041724 + mem-2610111200000002 0.512505 + deploy recall=1 title=1 body=1 0.450126 + the recall=0 title=1 body=1 0.062379 prompt: run the tests with the race detector - mem-2610111200000003 2.738434 - run recall=0 title=0 body=1 0.384005 - the recall=0 title=2 body=1 0.069278 - test recall=1 title=1 body=1 1.142576 - race recall=1 title=1 body=1 1.142576 + mem-2610111200000003 2.709912 + run recall=0 title=0 body=1 0.374378 + the recall=0 title=2 body=1 0.068638 + test recall=1 title=1 body=1 1.133448 + race recall=1 title=1 body=1 1.133448 prompt: login fails after the token refresh - page:fact_eng_auth-flow.md 2.186949 - login recall=1 title=0 body=0 0.932215 - after recall=0 title=0 body=1 0.563239 - the recall=0 title=0 body=2 0.050277 - token recall=0 title=1 body=2 0.641218 - mem-2610111200000001 1.456814 - the recall=0 title=0 body=2 0.056166 - token recall=1 title=1 body=1 0.740001 - refresh recall=0 title=0 body=1 0.660648 + page:fact_eng_auth-flow.md 2.208935 + login recall=1 title=0 body=0 0.958608 + after recall=0 title=0 body=1 0.592823 + the recall=0 title=0 body=2 0.052127 + token recall=0 title=1 body=1 0.605377 + mem-2610111200000001 1.436311 + the recall=0 title=0 body=2 0.055313 + token recall=1 title=1 body=1 0.735054 + refresh recall=0 title=0 body=1 0.645944 From 4e606a148cc5ab8031b0a6248d21fcd63a17f03a Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:46:56 +0000 Subject: [PATCH 07/23] fix: set aside a recalled item larger than the whole budget A page can be as large as 4 MiB while a turn carries 64 KiB, and the take stopped at the first candidate that did not fit, so one oversized page ranked first held back every item below it on every turn, each reported as "tried again on the next" though no turn could change it. An item whose block alone is larger than the whole per-turn budget is now set aside as too large, never recorded, without stopping the take: the hook names it on stderr, and `rules --prompt` gives it the form too-large. An item that only misses this turn's remainder still stops the take, as before. Refs: itd-2610091918433290, itd-39 Assisted-by: Claude:claude-opus-5-5 --- commands/ahoy.md | 5 ++- docs/reference/cli/commands.md | 3 +- internal/core/recall/ranking_test.go | 34 ++++++++++++++++ internal/core/recall/recall.go | 39 +++++++++++------- internal/surface/cli/cli.go | 8 +++- internal/surface/cli/memory_recall_test.go | 46 ++++++++++++++++++++++ internal/surface/cli/rules_prompt.go | 17 ++++++-- 7 files changed, 130 insertions(+), 22 deletions(-) diff --git a/commands/ahoy.md b/commands/ahoy.md index e4d080a663..7736db093e 100644 --- a/commands/ahoy.md +++ b/commands/ahoy.md @@ -1223,9 +1223,10 @@ page of the same name. Notes and pages share one ranked order: each carries its `rank` (1 first), its BM25F relevance `score`, and `terms`, each matched term with how often it was found in the item's `recall` words, `title` and `body` and its `contribution` to the score; the ranking decides which items -fit the budget. Each carries a `form`: `full`, or `over-budget` when +fit the budget. Each carries a `form`: `full`, `over-budget` when it would not fit the per-turn budget and would come back on a later prompt -instead. `disabled` is true under the kill switch, and then nothing comes +instead, or `too-large` when it is larger than the whole per-turn budget and +no prompt brings it back in full. `disabled` is true under the kill switch, and then nothing comes back; `unreadable` names a `memory.json`, page store or page that could not be read, why, and its `kind` (`note` or `page`). Use it when someone asks why a rule, a note or a page did, or did not, come back on a prompt. A diff --git a/docs/reference/cli/commands.md b/docs/reference/cli/commands.md index b700804914..bf7c9f4c13 100644 --- a/docs/reference/cli/commands.md +++ b/docs/reference/cli/commands.md @@ -3658,7 +3658,8 @@ session: each rule domain it matches, with the recall words that hit, and each memory note it recalls from ~/.abcd.noindex/memory.json and .abcd/memory.json, and each memory page from .abcd/memory/ and ~/.abcd.noindex/memory/, with its layer and the recall words that hit, each in full or as not fitting the -per-turn budget. Notes and pages share one ranked order, a BM25F relevance +per-turn budget, or as larger than the whole budget, which no turn can +carry in full. Notes and pages share one ranked order, a BM25F relevance score that decides which of them fit the budget: each shows its rank and score and, under it, every matched term with the fields it was found in (recall words, title, body) and what it added to the score. It reads diff --git a/internal/core/recall/ranking_test.go b/internal/core/recall/ranking_test.go index 8a30d0ed9a..8ce548a56b 100644 --- a/internal/core/recall/ranking_test.go +++ b/internal/core/recall/ranking_test.go @@ -250,3 +250,37 @@ func TestRankingFindsATermWhereTheMatcherHits(t *testing.T) { } } } + +// TestAnItemLargerThanTheWholeBudgetDoesNotBlockTheRest: an item whose block +// alone is larger than the whole per-turn budget can never come back in +// full, so it is set aside as too large, never recorded, and the items +// ranked below it still come back; one that only misses this turn's +// remainder still stops the take, as before. +func TestAnItemLargerThanTheWholeBudgetDoesNotBlockTheRest(t *testing.T) { + huge := strings.TrimSpace(strings.Repeat("kerberos deploy notes ", rules.InjectionBudgetBytes/20)) + items := []Item{ + item("page:fact_eng_huge.md", []string{"deploy", "kerberos"}, "Every kerberos deploy ever", huge), + item("mem-2610111200000002", []string{"deploy"}, "Deploys need the dry-run flag", "Pass it first."), + } + ranked := Rank(items, "deploy with kerberos") + if ranked[0].Item.Key != "page:fact_eng_huge.md" { + t.Fatalf("the fixture does not rank the huge page first: %v", rankedKeys(ranked)) + } + res := Recall(Request{Items: items, Prompt: "deploy with kerberos", Budget: rules.InjectionBudgetBytes}) + if !slices.Equal(res.Recalled, []string{"mem-2610111200000002"}) || !slices.Equal(res.TooLarge, []string{"page:fact_eng_huge.md"}) || + len(res.Omitted) != 0 { + t.Fatalf("recalled %v too large %v omitted %v", res.Recalled, res.TooLarge, res.Omitted) + } + if _, ok := res.Ledger["page:fact_eng_huge.md"]; ok { + t.Fatal("an item that never came back was recorded as sent") + } + if len(res.Text) > rules.InjectionBudgetBytes { + t.Fatalf("the block is over the budget: %d", len(res.Text)) + } + // With the rules leaving too little for the note this turn, the note is + // omitted and tried again, and the huge page is still only too large. + tight := Recall(Request{Items: items, Prompt: "deploy with kerberos", Budget: 10}) + if len(tight.Recalled) != 0 || !slices.Equal(tight.Omitted, []string{"mem-2610111200000002"}) || !slices.Equal(tight.TooLarge, []string{"page:fact_eng_huge.md"}) { + t.Fatalf("recalled %v too large %v omitted %v", tight.Recalled, tight.TooLarge, tight.Omitted) + } +} diff --git a/internal/core/recall/recall.go b/internal/core/recall/recall.go index ff55432c1e..db436ba1dd 100644 --- a/internal/core/recall/recall.go +++ b/internal/core/recall/recall.go @@ -4,6 +4,8 @@ import ( "fmt" "hash/fnv" "strings" + + "github.com/intentdriven/abcd/internal/core/rules" ) // Request is one prompt's recall. @@ -25,12 +27,16 @@ type Request struct { // Result is one prompt's recall. Text is empty when nothing new comes back. // Recalled names each item Text carries, in order; Omitted each new -// candidate that did not fit the budget, not recorded and so retried on the -// next turn. Ledger is the updated recalled ledger, a copy. +// candidate that did not fit this turn's budget, not recorded and so retried +// on the next turn; TooLarge each new candidate whose block alone is larger +// than the whole per-turn budget (rules.InjectionBudgetBytes), which no turn +// can carry, so it is set aside without stopping the take and never +// recorded. Ledger is the updated recalled ledger, a copy. type Result struct { Text string Recalled []string Omitted []string + TooLarge []string Ledger map[string]string } @@ -38,8 +44,10 @@ type Result struct { // drops every one already sent this session with unchanged content, and // takes the rest in ranked order while they fit the budget: the first that // does not fit stops the take, so an item never comes back ahead of one -// ranked above it. It performs no I/O and never reflects prompt bytes into -// the block. +// ranked above it that a later turn can still carry. An item larger than the +// whole per-turn budget is set aside as TooLarge instead, so it cannot hold +// back every item below it on every turn. It performs no I/O and never +// reflects prompt bytes into the block. func Recall(req Request) Result { ledger := map[string]string{} for k, v := range req.Ledger { @@ -61,18 +69,21 @@ func Recall(req Request) Result { } var b strings.Builder size := len(headerLine) - for i, it := range fresh { + stopped := false + for _, it := range fresh { block := renderItem(it) - if size+len(block) > req.Budget { - for _, rest := range fresh[i:] { - res.Omitted = append(res.Omitted, rest.Key) - } - break + switch { + case len(headerLine)+len(block) > rules.InjectionBudgetBytes: + res.TooLarge = append(res.TooLarge, it.Key) + case stopped || size+len(block) > req.Budget: + stopped = true + res.Omitted = append(res.Omitted, it.Key) + default: + b.WriteString(block) + size += len(block) + ledger[it.Key] = signature(it) + res.Recalled = append(res.Recalled, it.Key) } - b.WriteString(block) - size += len(block) - ledger[it.Key] = signature(it) - res.Recalled = append(res.Recalled, it.Key) } if len(res.Recalled) > 0 { res.Text = headerLine + b.String() diff --git a/internal/surface/cli/cli.go b/internal/surface/cli/cli.go index 2010ac247d..60efdbc2b8 100644 --- a/internal/surface/cli/cli.go +++ b/internal/surface/cli/cli.go @@ -1765,11 +1765,14 @@ exits 0, so the hook can never wedge a session.`, // whose words went into the context (GHSA-22f8-qf5r-gjgq). fmt.Fprintf(cmd.ErrOrStderr(), "abcd rules: turn %d, injected %d domain(s) %v, %d bytes\n", res.State.Count, len(res.Injected), res.Labels(), len(res.Text)) - if len(mem.Recalled) > 0 || len(mem.Omitted) > 0 { + if len(mem.Recalled) > 0 || len(mem.Omitted) > 0 || len(mem.TooLarge) > 0 { fmt.Fprintf(cmd.ErrOrStderr(), "abcd memory: recalled %d item(s) %v, %d bytes", len(mem.Recalled), mem.Recalled, len(mem.Text)) if len(mem.Omitted) > 0 { fmt.Fprintf(cmd.ErrOrStderr(), "; %d matched item(s) %v did not fit this turn's budget and are tried again on the next", len(mem.Omitted), mem.Omitted) } + if len(mem.TooLarge) > 0 { + fmt.Fprintf(cmd.ErrOrStderr(), "; %d matched item(s) %v are larger than the whole per-turn budget and do not come back in full", len(mem.TooLarge), mem.TooLarge) + } fmt.Fprintln(cmd.ErrOrStderr()) } text := res.Text + mem.Text @@ -2445,7 +2448,8 @@ session: each rule domain it matches, with the recall words that hit, and each memory note it recalls from ` + abcdhome.Display("memory.json") + ` and .abcd/memory.json, and each memory page from .abcd/memory/ and ` + abcdhome.Display("memory") + `/, with its layer and the recall words that hit, each in full or as not fitting the -per-turn budget. Notes and pages share one ranked order, a BM25F relevance +per-turn budget, or as larger than the whole budget, which no turn can +carry in full. Notes and pages share one ranked order, a BM25F relevance score that decides which of them fit the budget: each shows its rank and score and, under it, every matched term with the fields it was found in (recall words, title, body) and what it added to the score. It reads diff --git a/internal/surface/cli/memory_recall_test.go b/internal/surface/cli/memory_recall_test.go index 05125edbd9..2de9d5d2cc 100644 --- a/internal/surface/cli/memory_recall_test.go +++ b/internal/surface/cli/memory_recall_test.go @@ -485,3 +485,49 @@ func TestRulesPromptShowsEachTermsContribution(t *testing.T) { } } } + +// TestAPageLargerThanTheBudgetIsNamedAndTheRestComeBack: a page whose block +// is larger than the whole per-turn budget is named on stderr as never +// coming back in full, and does not hold back the page ranked below it; +// `rules --prompt` gives it the form too-large. +func TestAPageLargerThanTheBudgetIsNamedAndTheRestComeBack(t *testing.T) { + t.Setenv("ABCD_RULES_STATE_DIR", t.TempDir()) + _, repo := memoryFixture(t) + workspace := filepath.Join(repo, ".abcd", "memory") + huge := strings.Repeat("Every kerberos deploy ever written down.\n", 2000) // about 80 KiB + writeRecallPage(t, workspace, "fact_eng_huge.md", "# Every kerberos deploy\n\n"+huge, 0o644, "deploy", "kerberos") + writeRecallPage(t, workspace, "fact_eng_dry-run.md", "# Deploys need the dry-run flag\n\nPass it first.\n", 0o644, "deploy") + + env, errlog := recallRouterJSON(t, "too-large", repo, "deploy with kerberos") + if env.Recalled == nil || !slices.Equal(*env.Recalled, []string{"page:fact_eng_dry-run.md"}) { + t.Fatalf("recalled %v:\n%s\nstderr:\n%s", env.Recalled, env.Text, errlog) + } + if !strings.Contains(errlog, "[page:fact_eng_huge.md] are larger than the whole per-turn budget") { + t.Fatalf("stderr does not name the page too large to come back:\n%s", errlog) + } + + jout, _, err := runCLISplit(t, "rules", "--prompt", "deploy with kerberos", "--json") + if err != nil { + t.Fatalf("rules --prompt --json: %v", err) + } + var v struct { + Pages []struct { + Key string `json:"key"` + Form string `json:"form"` + } `json:"pages"` + } + if err := json.Unmarshal([]byte(jout), &v); err != nil { + t.Fatal(err) + } + forms := map[string]string{} + for _, p := range v.Pages { + forms[p.Key] = p.Form + } + if forms["page:fact_eng_huge.md"] != "too-large" || forms["page:fact_eng_dry-run.md"] != "full" { + t.Fatalf("forms = %v", forms) + } + out, _, _ := runCLISplit(t, "rules", "--prompt", "deploy with kerberos") + if !strings.Contains(out, "larger than the whole per-turn budget") { + t.Fatalf("rules --prompt does not say the page is too large:\n%s", out) + } +} diff --git a/internal/surface/cli/rules_prompt.go b/internal/surface/cli/rules_prompt.go index ed2c3853b4..9df3a552df 100644 --- a/internal/surface/cli/rules_prompt.go +++ b/internal/surface/cli/rules_prompt.go @@ -11,7 +11,7 @@ import ( ) // rules_prompt.go — the memory recall's two front doors -// (spc-2610111627213138 steps 2 and 3): the prompt router's recall after the +// (spc-2610111627213138 steps 2 to 4): the prompt router's recall after the // rules, and the `rules --prompt ""` diagnostic that shows what one // prompt brings back. @@ -56,6 +56,7 @@ func refusalLoss(r recall.Refusal) string { const ( promptFormFull = "full" promptFormOverBudget = "over-budget" + promptFormTooLarge = "too-large" ) // rulesPromptDomain is one matched domain in the diagnostic. @@ -136,10 +137,17 @@ func rulesPrompt(rs rules.RuleSet, root, prompt string) rulesPromptView { for _, key := range mem.Recalled { recalled[key] = true } + tooLarge := map[string]bool{} + for _, key := range mem.TooLarge { + tooLarge[key] = true + } for i, c := range recall.Rank(items, prompt) { form := promptFormOverBudget - if recalled[c.Item.Key] { + switch { + case recalled[c.Item.Key]: form = promptFormFull + case tooLarge[c.Item.Key]: + form = promptFormTooLarge } terms := c.Terms if terms == nil { @@ -218,8 +226,11 @@ func renderTermScores(w io.Writer, terms []recall.TermScore) { // promptFormWords is a form as the plain diagnostic says it. func promptFormWords(form string) string { - if form == promptFormFull { + switch form { + case promptFormFull: return "in full" + case promptFormTooLarge: + return "larger than the whole per-turn budget, so it does not come back in full" } return "does not fit this turn's budget" } From ddcd9a94f98e02cd3ef9d892a9898329516170a5 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:48:21 +0000 Subject: [PATCH 08/23] fix: count one word in two inflections as one ranking term The ranking's query terms were keyed on each token's stem, but the term rule matches a token by several candidate roots, so "merge before merging" made two terms that both claimed an item's "merge" and scored it twice, each with its own idf. Tokens whose roots overlap are now one term, so each root belongs to one term and an item's word is found under one term at most. The terms are built with a root-to-term map, so a long prompt costs no more than its tokens. Refs: itd-2610091918433290, itd-39 Assisted-by: Claude:claude-opus-5-5 --- internal/core/recall/ranking_test.go | 32 +++++++++++ internal/core/rules/rules.go | 84 +++++++++++++++++++++------- 2 files changed, 96 insertions(+), 20 deletions(-) diff --git a/internal/core/recall/ranking_test.go b/internal/core/recall/ranking_test.go index 8ce548a56b..ba73418974 100644 --- a/internal/core/recall/ranking_test.go +++ b/internal/core/recall/ranking_test.go @@ -284,3 +284,35 @@ func TestAnItemLargerThanTheWholeBudgetDoesNotBlockTheRest(t *testing.T) { t.Fatalf("recalled %v too large %v omitted %v", tight.Recalled, tight.TooLarge, tight.Omitted) } } + +// TestOneWordInTwoInflectionsIsOneTerm: a prompt that says a word twice in +// two inflections ("merge before merging") asks for one term, so an item's +// word is found, and counted, once — never under two terms, each with its +// own idf. +func TestOneWordInTwoInflectionsIsOneTerm(t *testing.T) { + for _, prompt := range []string{"merge before merging", "merging before the merge", "commit after committing the fix"} { + terms := rules.PromptTerms(prompt) + owner := map[string]string{} + for _, term := range terms { + for _, root := range term.Roots { + if other, ok := owner[root]; ok { + t.Fatalf("%q: root %q belongs to two terms, %q and %q", prompt, root, other, term.Stem) + } + owner[root] = term.Stem + } + } + } + ranked := Rank([]Item{item("mem-2610111200000001", []string{"merge"}, "Merges need a review", "Ask first.")}, "merge before merging") + if len(ranked) != 1 { + t.Fatalf("ranked = %+v", ranked) + } + var found []string + for _, ts := range ranked[0].Terms { + if ts.Recall > 0 { + found = append(found, ts.Term) + } + } + if len(found) != 1 { + t.Fatalf("the recall word was found under %d terms %v: %+v", len(found), found, ranked[0].Terms) + } +} diff --git a/internal/core/rules/rules.go b/internal/core/rules/rules.go index c647be2015..d76d80e982 100644 --- a/internal/core/rules/rules.go +++ b/internal/core/rules/rules.go @@ -30,7 +30,6 @@ import ( "io" "os" "regexp" - "slices" "sort" "strings" @@ -708,14 +707,16 @@ type promptIndex struct { padded string // " tok tok " for boundary/phrase matching stemmedPadded string // " stem stem " for stemmed phrase matching stems map[string]bool // candidate stems of the single tokens - terms []PromptTerm // each distinct stem, with its candidate roots + terms []PromptTerm // each distinct word, with its candidate roots } -// PromptTerm is one distinct stem of a prompt's tokens, as the memory -// recall's ranking weighs it (spc-2610111627213138 step 4): Stem is the root -// stem gives the token, and Roots every candidate root the term rule matches -// the token by (stemVariants), so a word whose stem is among them is the term -// found exactly where termHit would hit it. +// PromptTerm is one distinct word of a prompt, as the memory recall's +// ranking weighs it (spc-2610111627213138 step 4): Stem is the root stem +// gives the token that first used it, and Roots every candidate root the +// term rule matches its tokens by (stemVariants), so a word whose stem is +// among them is the term found exactly where termHit would hit it. Tokens +// whose roots overlap are one term ("merge" and "merging"), so no root +// belongs to two terms and a word in an item is found under one term at most. type PromptTerm struct { Stem string Roots []string @@ -729,7 +730,7 @@ func indexPrompt(s string) promptIndex { idx := promptIndex{padded: " " + collapsed + " ", stems: map[string]bool{}} tokens := strings.Fields(collapsed) stemmed := make([]string, 0, len(tokens)) - at := map[string]int{} + var terms promptTerms for _, tok := range tokens { variants := stemVariants(tok) for _, v := range variants { @@ -737,20 +738,63 @@ func indexPrompt(s string) promptIndex { } s := stem(tok) stemmed = append(stemmed, s) - i, seen := at[s] - if !seen { - i = len(idx.terms) - at[s] = i - idx.terms = append(idx.terms, PromptTerm{Stem: s}) + terms.add(s, variants) + } + idx.terms = terms.list() + idx.stemmedPadded = " " + strings.Join(stemmed, " ") + " " + return idx +} + +// promptTerms collects a prompt's terms, keeping each root in one term. +type promptTerms struct { + terms []PromptTerm + owner map[string]int // root -> index of the term it belongs to +} + +// add adds one token, its stem s and its candidate roots: into the earliest +// term any of its roots already belongs to, merging in every other term +// those roots belong to, or as a new term when none does. Each root is +// looked up once, so a long prompt costs no more than its tokens. +func (pt *promptTerms) add(s string, roots []string) { + if pt.owner == nil { + pt.owner = map[string]int{} + } + target := -1 + for _, r := range roots { + if i, ok := pt.owner[r]; ok && (target < 0 || i < target) { + target = i } - for _, v := range variants { - if !slices.Contains(idx.terms[i].Roots, v) { - idx.terms[i].Roots = append(idx.terms[i].Roots, v) + } + if target < 0 { + target = len(pt.terms) + pt.terms = append(pt.terms, PromptTerm{Stem: s}) + } + for _, r := range roots { + i, ok := pt.owner[r] + switch { + case !ok: + pt.owner[r] = target + pt.terms[target].Roots = append(pt.terms[target].Roots, r) + case i != target: + for _, moved := range pt.terms[i].Roots { + pt.owner[moved] = target } + pt.terms[target].Roots = append(pt.terms[target].Roots, pt.terms[i].Roots...) + pt.terms[i].Roots = nil } } - idx.stemmedPadded = " " + strings.Join(stemmed, " ") + " " - return idx +} + +// list is the terms in the order the prompt first used them, less every +// term merged into an earlier one. +func (pt *promptTerms) list() []PromptTerm { + var out []PromptTerm + for _, t := range pt.terms { + if len(t.Roots) > 0 { + out = append(out, t) + } + } + return out } // hit reports whether any of a domain's recall keywords or aliases match. @@ -1050,8 +1094,8 @@ func TermMatcher(prompt string) func(term string) bool { } // PromptTerms is the prompt index's query terms for the memory recall's -// ranking: each distinct stem of the prompt's tokens, in the order the prompt -// first uses it, with its candidate roots. It reads the bytes TermMatcher +// ranking: each distinct word of the prompt, in the order the prompt first +// uses it, with its candidate roots. It reads the bytes TermMatcher // reads, from the same index. func PromptTerms(prompt string) []PromptTerm { if len(prompt) > maxPromptBytes { From c8db9064b446f2905a8ec0fb4e9c2c4746c03ce9 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:51:08 +0000 Subject: [PATCH 09/23] fix: a refused workspace page still shadows the machine page of its name The project's page of a name suppressed the machine's page of that name only when the project's page was read, so one the read refused (over the page cap, say) let the machine's page come back in its place, reversing the narrower-scope rule. A refused workspace page now shadows as a read one does, and a test holds both that and a silent workspace page (no recall list) shadowing. The read's comment now says the pages it reads are those the index is rendered from, and that it never reads index.md. Refs: itd-2610091918433290, itd-39 Assisted-by: Claude:claude-opus-5-5 --- internal/core/memory/recallpages.go | 16 ++++++++++------ internal/core/recall/pages.go | 18 ++++++++++++++---- internal/core/recall/pages_test.go | 23 +++++++++++++++++++++++ 3 files changed, 47 insertions(+), 10 deletions(-) diff --git a/internal/core/memory/recallpages.go b/internal/core/memory/recallpages.go index 09aae0dc70..618807823b 100644 --- a/internal/core/memory/recallpages.go +++ b/internal/core/memory/recallpages.go @@ -39,10 +39,12 @@ type RecallPage struct { } // RecallRefusal names a scope or a page the read refused, and why. File is -// the store-relative or tilde form, never an expanded home. +// the store-relative or tilde form, never an expanded home; Filename is the +// refused page's own name, "" when the whole scope was refused. type RecallRefusal struct { - File string - Err error + File string + Filename string + Err error } // machineStoreDisplay is the machine page scope as a message names it. @@ -53,8 +55,10 @@ const workspaceStoreDisplay = ".abcd/memory/" // ReadRecallPages reads the workspace store under repoRoot through the store // handle: a symlinked or swapped store is refused whole, each page is read -// through the handle's guarded read. An absent store yields nothing. Only -// pages the index lists whose names are __.md are read. +// through the handle's guarded read. An absent store yields nothing. The +// pages read are those index.md is rendered from (the store's top-level page +// files, storeHandle.pageNames) whose names are __.md; +// index.md itself is never read. func ReadRecallPages(repoRoot string) ([]RecallPage, []RecallRefusal) { store, err := openStore(repoRoot) if err != nil { @@ -113,7 +117,7 @@ func recallPagesFrom(store *storeHandle, shown string, read func(string, int64) } raw, err := read(name, maxMemoryPageBytes) if err != nil { - refused = append(refused, RecallRefusal{File: shown + name, Err: recallReadError(err)}) + refused = append(refused, RecallRefusal{File: shown + name, Filename: name, Err: recallReadError(err)}) continue } p := parsePage(string(raw)) diff --git a/internal/core/recall/pages.go b/internal/core/recall/pages.go index 280b2290fe..527caf97db 100644 --- a/internal/core/recall/pages.go +++ b/internal/core/recall/pages.go @@ -16,10 +16,14 @@ const PageKeyPrefix = "page:" // pageItems turns both page scopes into items, in key order. A page of the // same __ name in both scopes is the workspace's: the // narrower scope's page is the page of that name, and the machine's is -// suppressed, whether or not the workspace page carries a recall list. A page -// with no recall: list is never an item, so it is never a candidate. -func pageItems(workspace, machine []memory.RecallPage) []Item { +// suppressed, whether or not the workspace page carries a recall list, and +// when the workspace page could not be read (refusedHere names those). A +// page with no recall: list is never an item, so it is never a candidate. +func pageItems(workspace, machine []memory.RecallPage, refusedHere ...string) []Item { shadowed := map[string]bool{} + for _, name := range refusedHere { + shadowed[name] = true + } var out []Item for _, p := range workspace { shadowed[p.Filename] = true @@ -51,10 +55,16 @@ func readPages(home, repoRoot string) ([]Item, []Refusal) { workspace, wr := memory.ReadRecallPages(repoRoot) machine, mr := memory.ReadMachineRecallPages(home) var refused []Refusal + var refusedHere []string + for _, r := range wr { + if r.Filename != "" { + refusedHere = append(refusedHere, r.Filename) + } + } for _, r := range append(wr, mr...) { refused = append(refused, Refusal{File: r.File, Reason: reason(r.Err), Kind: KindPage}) } - return pageItems(workspace, machine), refused + return pageItems(workspace, machine, refusedHere...), refused } // ReadItems is every note and page in reach, in id order — notes (mem-…) diff --git a/internal/core/recall/pages_test.go b/internal/core/recall/pages_test.go index 5f530bf74f..6abe257a9a 100644 --- a/internal/core/recall/pages_test.go +++ b/internal/core/recall/pages_test.go @@ -181,3 +181,26 @@ func TestPageItemsCarryTheirIngestDate(t *testing.T) { t.Fatalf("items = %+v", items) } } + +// TestAWorkspacePageShadowsTheMachinePageEvenWhenItIsSilent: the project's +// page of a name is the page of that name whether or not it can come back — +// one with no recall list, and one the read refuses (over the page cap) — +// so the machine's page of the same name never stands in for it. +func TestAWorkspacePageShadowsTheMachinePageEvenWhenItIsSilent(t *testing.T) { + home, repo, workspace, machine := pagesFixture(t) + writePage(t, workspace, "fact_eng_silent.md", "# The project's page, with no recall list\n") + writePage(t, machine, "fact_eng_silent.md", "# The person's page of the same name\n", "auth") + big := filepath.Join(workspace, "fact_eng_too-big.md") + if err := os.WriteFile(big, []byte("---\nrecall: [auth]\n---\n# Big\n"+strings.Repeat("x", 4<<20)), 0o600); err != nil { + t.Fatal(err) + } + writePage(t, machine, "fact_eng_too-big.md", "# The person's page named like the refused one\n", "auth") + + items, refused := ReadItems(home, repo) + if len(items) != 0 { + t.Fatalf("a machine page stood in for the project's page of its name: %+v", items) + } + if len(refused) != 1 || refused[0].File != ".abcd/memory/fact_eng_too-big.md" { + t.Fatalf("refused = %+v", refused) + } +} From 446e4eb035daf950065bf01d55bba8933d513017 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:51:19 +0000 Subject: [PATCH 10/23] fix: break ranking ties on scores equal to a billionth The ranked order compared scores exactly, so two scores equal in arithmetic but for the last bits a machine's float operations may leave (a fused multiply-add on one architecture and not another) could order differently between machines. Scores are now compared at a billionth before the ties break on scope, recency and key. Refs: itd-2610091918433290, itd-39 Assisted-by: Claude:claude-opus-5-5 --- internal/core/recall/ranking.go | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/internal/core/recall/ranking.go b/internal/core/recall/ranking.go index fdc831401b..5e3033cde3 100644 --- a/internal/core/recall/ranking.go +++ b/internal/core/recall/ranking.go @@ -30,6 +30,10 @@ const ( FieldBody = "body" ) +// scoreGrain is the finest difference between two scores the ranked order +// tells apart. +const scoreGrain = 1e-9 + // fieldWeights is each field's weight, in the order fieldsOf returns them: a // term found in an item's recall words counts three times, in its title // twice, in its body once. The weighted frequencies are summed per term @@ -150,10 +154,12 @@ func Rank(items []Item, prompt string) []Ranked { } // rankedBefore is the ranked order: the higher score, then the narrower -// scope, then the more recent At, then the key. +// scope, then the more recent At, then the key. Scores are compared at a +// billionth (scoreGrain), so two scores equal but for the last bits a +// different machine's float arithmetic may leave tie on every machine. func rankedBefore(a, b Ranked) bool { - if a.Score != b.Score { - return a.Score > b.Score + if sa, sb := math.Round(a.Score/scoreGrain), math.Round(b.Score/scoreGrain); sa != sb { + return sa > sb } if ap, bp := a.Item.Layer == LayerProject, b.Item.Layer == LayerProject; ap != bp { return ap From 1f2a6d0bb4fcb04976d017614b0ec144b340474a Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:51:41 +0000 Subject: [PATCH 11/23] docs: bring the memory-recall spec's settled choices up to the review fixes The Settled here bullets now say what the review fixes changed: a page's body leaves out its title line, a refused workspace page still shadows, the machine page scope's directories are judged as the machine notes file's are, one word in two inflections is one ranking term, an item larger than the whole budget is set aside without stopping the take, and scores tie at a billionth. Refs: itd-2610091918433290, itd-39 Assisted-by: Claude:claude-opus-5-5 --- .../spc-2610111627213138-memory-recall.md | 37 +++++++++++++------ 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md b/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md index f6f82879c7..18cec43fe9 100644 --- a/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md +++ b/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md @@ -740,19 +740,23 @@ choices above. store's top level, whose names are `__.md`. A page's title is its index line's summary, derived from the page the way the index derives it, so the recall reads no `index.md` and opens only the four - places it declares. Grounds: P4 compares pages by that name, and the name's + places it declares; its body is the rest of the page after the line the + title comes from, so the title is counted and sent once. Grounds: P4 compares pages by that name, and the name's characters keep a page's key (`page:`) safe to print in the hook's stderr line and envelope. - **A workspace page shadows the machine page of the same name, whether or - not it carries a recall list** (2026-10-12, step 3, agent's choice for - review). Grounds: the narrower scope's page is the page of that name, as a + not it carries a recall list, and when the read refuses it** (2026-10-12, + step 3, agent's choice for review). Grounds: the narrower scope's page is the page of that name, as a repository layer's field replaces the machine's and a note in the project's file suppresses the machine's copy of its id; it also keeps one `page:` key per name in the ledger and the envelope. - **A machine page is read with the guards the machine `memory.json` gets** (2026-10-12, step 3, agent's choice for review): the directory through - `fsutil.OpenHomeScope`, each page a regular file this account owns that no - one else can write (`fsutil.ReadHomeDeclarationDenyingIn`). A page failing + `fsutil.OpenHomeDeclarationScope`, the home scope's walk with the + directory judgement `fsutil.ReadHomeDeclaration` makes (a symlinked + directory, or one another account can change, refused), and each page a + regular file this account owns that no one else can write + (`fsutil.ReadHomeDeclarationDenyingIn`). A page failing them is named on stderr and brings nothing back, while the scope's other pages still come back. Grounds: a machine page injects text into every session as a machine note does; the spec names only the directory's open, @@ -770,11 +774,12 @@ choices above. order would not be the same on every machine, as Ranking requires; `ingested_at` is in the page and travels with it. - **How the ranking counts terms and lengths** (2026-10-12, step 4, agent's - choice for review). Each distinct stem of the prompt's tokens is one query - term, however often the prompt repeats it, and a word in an item counts as - that term when its stem is among the roots the loader's term rule matches - the prompt's token by, so the ranking finds a term exactly where the - matcher hits. That takes two small exports from `internal/core/rules` + choice for review). Each distinct word of the prompt is one query term, + however often or in whatever inflection the prompt repeats it (tokens whose + candidate roots overlap, "merge" and "merging", are one term), and a word + in an item counts as that term when its stem is among the roots the + loader's term rule matches the prompt's tokens by, so the ranking finds a + term exactly where the matcher hits, and under one term at most. That takes two small exports from `internal/core/rules` beside `TermMatcher` (`PromptTerms`, read from the same prompt index, and `TextStems`), though step 4's packages line names only the recall and the CLI. An item's length is its weighted length, recall words three times, @@ -785,7 +790,17 @@ choices above. - **The ranked order keeps step 2's take** (2026-10-12, step 4, agent's choice for review): candidates are taken in ranked order while they fit, and the first that does not fit stops the take, so nothing comes back - ahead of an item ranked above it. + ahead of an item ranked above it. The one exception is an item whose block + alone is larger than the whole per-turn budget, which no turn can carry: it + is set aside as too large, never recorded, named on the hook's stderr and + given the form `too-large` in `rules --prompt`, and it does not stop the + take. Grounds: a page can be as large as 4 MiB, and one that ranks first + would otherwise hold back every item below it on every turn. +- **Scores tie at a billionth** (2026-10-12, step 4, agent's choice for + review): two scores within a billionth of each other are a tie, broken by + scope, recency and key. Grounds: arithmetic equal on paper can differ in + its last bits between machines, and the order must be the same on every + machine. - **The diagnostic ranks notes and pages together** (2026-10-12, step 4, agent's choice for review): `rules --prompt` keeps its `notes` and `pages` lists, and each entry carries its `rank` in the one order they share, its From dd49a32b28a492d6999a712a33a7930aeccd1301 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 17 Oct 2026 07:24:11 +0100 Subject: [PATCH 12/23] fix: label a recalled page as a page of its scope, never as a note Main's provenance labels named every machine item "personal note, agent-written" and every project item "project file", so a page from the machine's memory store read as an agent's note. A page's head now says "project page, committed in this repository" or "machine page, this account's memory store". Assisted-by: Claude:claude-opus-5-5 --- internal/core/recall/pages_test.go | 22 ++++++++++++++++++++ internal/core/recall/render.go | 33 ++++++++++++++++++++---------- 2 files changed, 44 insertions(+), 11 deletions(-) diff --git a/internal/core/recall/pages_test.go b/internal/core/recall/pages_test.go index 6abe257a9a..97c522e534 100644 --- a/internal/core/recall/pages_test.go +++ b/internal/core/recall/pages_test.go @@ -204,3 +204,25 @@ func TestAWorkspacePageShadowsTheMachinePageEvenWhenItIsSilent(t *testing.T) { t.Fatalf("refused = %+v", refused) } } + +// TestARecalledPageIsLabelledAsAPage: a page's head names it as a page of +// its scope, never as a note — the machine's label "personal note, +// agent-written" would misstate whose words a page is +// (GHSA-22f8-qf5r-gjgq's provenance rule). +func TestARecalledPageIsLabelledAsAPage(t *testing.T) { + home, repo, workspace, machine := pagesFixture(t) + writePage(t, workspace, "lesson_auth_tokens.md", "# Auth tokens\n\nRotate them.\n", "auth") + writePage(t, machine, "lesson_auth_sessions.md", "# Auth sessions\n\nExpire them.\n", "auth") + res := readAndRecall(t, home, repo, "the auth flow", nil) + for _, want := range []string{ + "- page:lesson_auth_tokens.md (project page, committed in this repository): ", + "- page:lesson_auth_sessions.md (machine page, this account's memory store): ", + } { + if !strings.Contains(res.Text, want) { + t.Errorf("block lacks %q:\n%s", want, res.Text) + } + } + if strings.Contains(res.Text, "note") { + t.Errorf("a page is labelled as a note:\n%s", res.Text) + } +} diff --git a/internal/core/recall/render.go b/internal/core/recall/render.go index 80b707ff46..a556f718d0 100644 --- a/internal/core/recall/render.go +++ b/internal/core/recall/render.go @@ -15,20 +15,31 @@ const headerLine = "## MEMORY\n" // the repository's authors committed, so a cloned or compromised repository // controls it; a note in the machine's file was written by an agent on this // account through the write gate. +// A page is labelled as a page of its scope: the project's .abcd/memory/ is +// committed like the project's file, and the machine's ~/.abcd.noindex/memory/ +// is this account's store, which no agent writes through the note gate. const ( - labelProject = "project file, committed in this repository" - labelMachine = "personal note, agent-written" + labelProject = "project file, committed in this repository" + labelMachine = "personal note, agent-written" + labelPageProject = "project page, committed in this repository" + labelPageMachine = "machine page, this account's memory store" ) -// layerLabel is the provenance an item's head carries: "project file, -// committed in this repository" for the project's file, "personal note, -// agent-written" for the machine's. An unknown layer is named as itself, so -// a new layer is never rendered as one of these by default. -func layerLabel(layer string) string { - switch layer { - case LayerProject: +// layerLabel is the provenance an item's head carries: for a note, "project +// file, committed in this repository" for the project's file and "personal +// note, agent-written" for the machine's; for a page, "project page, +// committed in this repository" or "machine page, this account's memory +// store". An unknown layer is named as itself, so a new layer is never +// rendered as one of these by default. +func layerLabel(kind, layer string) string { + switch { + case kind == KindPage && layer == LayerProject: + return labelPageProject + case kind == KindPage && layer == LayerMachine: + return labelPageMachine + case layer == LayerProject: return labelProject - case LayerMachine: + case layer == LayerMachine: return labelMachine } return layer @@ -44,7 +55,7 @@ func layerLabel(layer string) string { // every line sits at the bullet's content column. The head line, a one-line // title, takes the rule body's mask (rules.SanitizeRuleBody). func renderItem(it Item) string { - head := rules.SanitizeRuleBody(it.Key + " (" + layerLabel(it.Layer) + "): " + it.Title) + head := rules.SanitizeRuleBody(it.Key + " (" + layerLabel(it.Kind, it.Layer) + "): " + it.Title) var b strings.Builder b.WriteString("- ") b.WriteString(head) From 5193e17013f12a24c64837622582be49e0d5013e Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 17 Oct 2026 07:27:50 +0100 Subject: [PATCH 13/23] fix: hand the Linux open test a root lookup, as the router does Since main's step 2 review the recall judges a tied note against the RootFunc it is handed, and a nil one answers no root, so the test's note tied to this repository was left out and the want list failed on Linux. The test now asks git itself, which also keeps git's own opens among those recorded. Assisted-by: Claude:claude-opus-5-5 --- internal/core/recall/opens_linux_test.go | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/internal/core/recall/opens_linux_test.go b/internal/core/recall/opens_linux_test.go index cd3d43ff4d..54e9f964d2 100644 --- a/internal/core/recall/opens_linux_test.go +++ b/internal/core/recall/opens_linux_test.go @@ -3,6 +3,7 @@ package recall import ( + "context" "encoding/binary" "errors" "io/fs" @@ -16,6 +17,7 @@ import ( "github.com/intentdriven/abcd/internal/abcdhome" "github.com/intentdriven/abcd/internal/core/rules" "github.com/intentdriven/abcd/internal/gittest" + "github.com/intentdriven/abcd/internal/gitutil" ) // openRecorder is the filesystem seam that records every open under the @@ -141,7 +143,10 @@ func TestRecallOpensOnlyTheDeclaredPlaces(t *testing.T) { rec := recordOpens(t, home, root) items, refused := ReadItems(home, root) - res := Recall(Request{Items: items, Prompt: "the auth login is broken", Budget: rules.InjectionBudgetBytes}) + // The root is asked of git itself, as the router asks it, so the opens + // git makes to answer are among those recorded. + here := func() (string, error) { return gitutil.RootCommitContext(context.Background(), root) } + res := Recall(Request{Items: items, Prompt: "the auth login is broken", Budget: rules.InjectionBudgetBytes, Root: here}) opened := rec.opened(t) declared := []string{machineNotes, machinePages, projectNotes, projectPages} From c3cd1cdc08b092a96adfa5b9bf3560ec595bb310 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 17 Oct 2026 07:29:21 +0100 Subject: [PATCH 14/23] refactor: cap the ranking's prompt terms through the one prompt cap PromptTerms cut the prompt by hand; main's step 2 review made rules.CapPrompt the one cap every matcher input goes through. Assisted-by: Claude:claude-opus-5-5 --- internal/core/rules/rules.go | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/internal/core/rules/rules.go b/internal/core/rules/rules.go index b33b40696e..f3edd2df82 100644 --- a/internal/core/rules/rules.go +++ b/internal/core/rules/rules.go @@ -1184,10 +1184,7 @@ func TermMatcher(prompt string) func(term string) bool { // uses it, with its candidate roots. It reads the bytes TermMatcher // reads, from the same index. func PromptTerms(prompt string) []PromptTerm { - if len(prompt) > maxPromptBytes { - prompt = prompt[:maxPromptBytes] - } - return indexPrompt(prompt).terms + return indexPrompt(CapPrompt(prompt)).terms } // TextStems is the stem of each token of text, in order — lower-cased, split From 923a2f007272913c0c735cfc9a7b27b90fc1cda0 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 17 Oct 2026 07:55:03 +0100 Subject: [PATCH 15/23] fix: unexport the recall's matching and note read, which only it calls After the merge the ranking calls the tie-judging match and ReadItems calls the note read, so nothing outside the recall package reaches either and the exported-reach audit refused them. Assisted-by: Claude:claude-opus-5-5 --- .../specs/open/spc-2610111627213138-memory-recall.md | 2 +- internal/core/recall/items.go | 6 +++--- internal/core/recall/match.go | 6 +++--- internal/core/recall/pages.go | 2 +- internal/core/recall/ranking.go | 4 ++-- internal/core/recall/recall.go | 4 ++-- internal/core/recall/recall_test.go | 8 ++++---- 7 files changed, 16 insertions(+), 16 deletions(-) diff --git a/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md b/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md index 1355fd6b09..38096d2800 100644 --- a/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md +++ b/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md @@ -809,7 +809,7 @@ choices above. neither. A personal note tied to another repository counts in both (amended 2026-10-17 on merging main, agent's choice for review): step 2's review made the tie a lazy judgement of the matched candidates alone - (`recall.Matching`, which asks git only when a tied note matched), so + (the recall package's `matching`, which asks git only when a tied note matched), so the corpus is fixed before git is asked and never depends on whether it was. - **The ranked order keeps step 2's take** (2026-10-12, step 4, agent's diff --git a/internal/core/recall/items.go b/internal/core/recall/items.go index cb348a5fe3..dbfca7986c 100644 --- a/internal/core/recall/items.go +++ b/internal/core/recall/items.go @@ -78,7 +78,7 @@ type Refusal struct { // noteItems turns both note layers into items, in id order. A personal note // carries the root it is tied to (Item.Tie), and is recalled only in that // repository, or everywhere when its root is null; the tie is judged after -// matching (Matching), so reading the layers asks git nothing. Every note in +// matching (see matching), so reading the layers asks git nothing. Every note in // the project's file is an item. A note in both layers (an id is minted once, // so only a hand copy does that) is the project's: the machine's copy is // suppressed. @@ -139,9 +139,9 @@ func ReadLayers(home, repoRoot string) (Layers, []Refusal) { return l, refused } -// ReadNotes is ReadLayers turned into the recall's items. A personal note's +// readNotes is ReadLayers turned into the recall's items. A personal note's // tie is judged after matching, against the RootFunc the recall is handed. -func ReadNotes(home, repoRoot string) ([]Item, []Refusal) { +func readNotes(home, repoRoot string) ([]Item, []Refusal) { l, refused := ReadLayers(home, repoRoot) return noteItems(l.Machine, l.Project), refused } diff --git a/internal/core/recall/match.go b/internal/core/recall/match.go index 3b67b103b6..d1fd708e39 100644 --- a/internal/core/recall/match.go +++ b/internal/core/recall/match.go @@ -10,7 +10,7 @@ type Candidate struct { // candidates returns the items any of whose recall words hits prompt under // the loader's term rule (rules.TermMatcher), in the items' order. The order -// chooses nothing: Rank orders the candidates (after Matching judges the +// chooses nothing: Rank orders the candidates (after matching judges the // ties), and Recall takes them in it. func candidates(items []Item, prompt string) []Candidate { if len(items) == 0 { @@ -32,13 +32,13 @@ func candidates(items []Item, prompt string) []Candidate { return out } -// Matching is candidates with each personal note's tie judged: a candidate +// matching is candidates with each personal note's tie judged: a candidate // tied to a root is kept only when root answers that same root. root is // asked only when a tied item matched, and then once, so a prompt that // matches no tied item costs no git call; a nil root answers "". When root // errs, every tied candidate is left out and the error is returned beside // the rest. -func Matching(items []Item, prompt string, root RootFunc) ([]Candidate, error) { +func matching(items []Item, prompt string, root RootFunc) ([]Candidate, error) { cands := candidates(items, prompt) tied := false for _, c := range cands { diff --git a/internal/core/recall/pages.go b/internal/core/recall/pages.go index 00b9563e0e..35f681eb3a 100644 --- a/internal/core/recall/pages.go +++ b/internal/core/recall/pages.go @@ -71,7 +71,7 @@ func readPages(home, repoRoot string) ([]Item, []Refusal) { // before pages (page:…) — and every layer, scope or page that could not be // read. It is what the prompt router and `rules --prompt` hand the recall. func ReadItems(home, repoRoot string) ([]Item, []Refusal) { - notes, refused := ReadNotes(home, repoRoot) + notes, refused := readNotes(home, repoRoot) pages, pr := readPages(home, repoRoot) return append(notes, pages...), append(refused, pr...) } diff --git a/internal/core/recall/ranking.go b/internal/core/recall/ranking.go index c016abf77e..c49450d730 100644 --- a/internal/core/recall/ranking.go +++ b/internal/core/recall/ranking.go @@ -89,7 +89,7 @@ func found(counts map[string]int, t rules.PromptTerm) int { } // Rank matches items against prompt, judges each tied note's root -// (Matching, which asks root only when a tied item matched), and returns the +// (matching, which asks root only when a tied item matched), and returns the // candidates in ranked order, each with its score and the terms behind it, // with the root lookup's error beside them. The score is BM25F: // for each query term t found in the item, @@ -105,7 +105,7 @@ func found(counts map[string]int, t rules.PromptTerm) int { // machine's), then the more recent At, then the key, so the order is the // same on every machine and every run. func Rank(items []Item, prompt string, root RootFunc) ([]Ranked, error) { - cands, rootErr := Matching(items, prompt, root) + cands, rootErr := matching(items, prompt, root) if len(cands) == 0 { return nil, rootErr } diff --git a/internal/core/recall/recall.go b/internal/core/recall/recall.go index 6e59431dbf..31c52ace18 100644 --- a/internal/core/recall/recall.go +++ b/internal/core/recall/recall.go @@ -24,7 +24,7 @@ type Request struct { // switch silences it too. Disabled bool // Root answers the repository's root commit, for the items tied to one; - // asked only once a tied item matched (Matching). + // asked only once a tied item matched (matching). Root RootFunc } @@ -66,7 +66,7 @@ func (r Result) Labels(keys []string) []string { } // Recall matches the items against the prompt, judges each tied note's root -// (Matching), ranks the candidates (Rank), drops every one already sent this +// (matching), ranks the candidates (Rank), drops every one already sent this // session with unchanged content, and takes the rest in ranked order while // they fit the budget: the first that does not fit stops the take, so an // item never comes back ahead of one ranked above it that a later turn can diff --git a/internal/core/recall/recall_test.go b/internal/core/recall/recall_test.go index 9fba3ae0ca..764d5066ee 100644 --- a/internal/core/recall/recall_test.go +++ b/internal/core/recall/recall_test.go @@ -83,7 +83,7 @@ func TestANoteInTheProjectsFileIsRecalled(t *testing.T) { if err := os.WriteFile(filepath.Join(repo, ".abcd", "memory.json"), []byte(project), 0o644); err != nil { t.Fatal(err) } - items, refused := ReadNotes(t.TempDir(), repo) + items, refused := readNotes(t.TempDir(), repo) if len(refused) != 0 { t.Fatalf("refused = %+v", refused) } @@ -326,7 +326,7 @@ func TestRootIsAskedOnlyForAMatchedTiedNote(t *testing.T) { // TestReadLayersIsTheOneReader (F3): ReadLayers reads both note layers as // they stand — the notes in full, for the memory verb's list — and names a // layer it could not read, home-redacted, while still reading the other. -// ReadNotes is the same read turned into items. +// readNotes is the same read turned into items. func TestReadLayersIsTheOneReader(t *testing.T) { home, repo := t.TempDir(), t.TempDir() if err := os.MkdirAll(filepath.Join(repo, ".abcd"), 0o755); err != nil { @@ -352,8 +352,8 @@ func TestReadLayersIsTheOneReader(t *testing.T) { if len(refused) != 1 || refused[0].File != memnote.MachineDisplayPath || strings.Contains(refused[0].Reason, home) { t.Fatalf("refused = %+v", refused) } - items, again := ReadNotes(home, repo) + items, again := readNotes(home, repo) if len(items) != 1 || items[0].Key != "mem-2610111200004321" || !slices.Equal(again, refused) { - t.Fatalf("ReadNotes = %+v, %+v", items, again) + t.Fatalf("readNotes = %+v, %+v", items, again) } } From a6cd370f4db2b034ffd336740b44b7d511be9c11 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 17 Oct 2026 08:09:07 +0100 Subject: [PATCH 16/23] fix: expect the too-large page named with its layer on the hook's stderr The merged stderr line labels each item with its layer, as main's step 2 does for notes. Assisted-by: Claude:claude-opus-5-5 --- internal/surface/cli/memory_recall_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/surface/cli/memory_recall_test.go b/internal/surface/cli/memory_recall_test.go index 2de9d5d2cc..cd76a20884 100644 --- a/internal/surface/cli/memory_recall_test.go +++ b/internal/surface/cli/memory_recall_test.go @@ -502,7 +502,7 @@ func TestAPageLargerThanTheBudgetIsNamedAndTheRestComeBack(t *testing.T) { if env.Recalled == nil || !slices.Equal(*env.Recalled, []string{"page:fact_eng_dry-run.md"}) { t.Fatalf("recalled %v:\n%s\nstderr:\n%s", env.Recalled, env.Text, errlog) } - if !strings.Contains(errlog, "[page:fact_eng_huge.md] are larger than the whole per-turn budget") { + if !strings.Contains(errlog, "[page:fact_eng_huge.md (project)] are larger than the whole per-turn budget") { t.Fatalf("stderr does not name the page too large to come back:\n%s", errlog) } From f3cdc73251bc12245f55365aa2e630e106dbfd57 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 17 Oct 2026 22:40:36 +0100 Subject: [PATCH 17/23] docs: mark the memory-recall steps that have landed Assisted-by: Claude:claude-opus-5-5 --- .../specs/open/spc-2610111627213138-memory-recall.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md b/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md index 38096d2800..a62994841f 100644 --- a/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md +++ b/.abcd/development/specs/open/spc-2610111627213138-memory-recall.md @@ -575,15 +575,19 @@ Steps 7 and 8 change no code. - the memory store's redactor (`storeRedactor` in internal/core/memory) is exported for the gate rather than a second one written (one-canonical-primitive) - the store and its verb land together, so the store has a caller from its first commit and no unwired code reaches main; a note written here is listed and removed at once, and is recalled from step 2 - a live trial in a host session checks that the host substitutes its session id into the command page, as the verb split's trial checked hiding; the capture goes under this spec's `## Live captures`. If the first host offers no substitution, the page leaves `--session` out and every note on that host is recorded as session unknown, which is the ruled fallback, not a reopened question + - landed: c8d1ab0df, 9ae13776b 2. One recall: notes come back through the prompt hook, and `rules --prompt` shows what a prompt brings back - packages: internal/core/recall (new: items, matching, rendering, the `recalled` ledger keyed on content; candidates that do not all fit the budget are taken in id order until step 4), internal/core/rules (exported prompt-index and term-match function; SessionState gains `recalled`; the reset clears it), internal/surface/cli (the prompt router calls recall after rules.Inject; the `--json` envelope gains `recalled`; `rules --prompt` listing matched domains with their hitting words, and matched notes), commands/ahoy.md (documents `rules --prompt`), .abcd/development/release/surface.json and docs/reference/cli/commands.md (the new flag) - tests: TestANoteComesBackOnAPromptAboutTests over two simulated sessions (N1), watched fail first; TestNoteRecallUsesTheRuleTermMatch (stems and phrases hit as a domain's recall words do); TestANoteInTheProjectsFileIsRecalled (N5); TestNoteTiedToAnotherRootIsNotRecalled and TestEverywhereNoteIsRecalledInAnyRepository; TestRecalledLedgerSurvivesTheRulesPrune; TestResetClearsBothLedgers; TestKillSwitchSilencesRecall; TestARefusedMachineLayerStillInjectsRules; TestMemoryTakesOnlyWhatTheRulesLeaveOfTheBudget; TestCandidatesThatFitAreTakenInIDOrder (replaced by step 4's ranking tests); TestRulesPromptShowsMatchedDomainsAndNotes and TestRulesPromptWritesNothing, watched fail first; TestRulesPromptRefusesAPositionalDomain; the existing inject and hook tests pass unchanged when no note matches (a healthy no-match still renders zero tokens) + - landed: e6d2a7df7, 59b69f398 3. Pages join the same recall, from both scopes - packages: internal/core/memory (an exported read of each page's filename, index line, body and `recall:` list, through the existing store handle; a read-only open of `~/.abcd.noindex/memory/`), internal/core/recall (page items, narrower scope wins; pages join the id order as `page:`, after notes, until step 4), internal/surface/cli (the router and `rules --prompt` hand pages to recall) - tests: TestAPageDeclaringAuthComesBackAndNoOther (P1); TestANoteAndAPageComeBackTogether (N2); TestRepeatedPromptInjectsMemoryOnce (P3); TestWorkspacePageSuppressesTheMachinePage (P4); TestRecallOpensOnlyTheDeclaredPlaces with a host memory directory of matching text under the test home (P5), each watched fail first; TestPageWithoutRecallIsNeverACandidate; TestRulesPromptShowsMatchedPages; the memory store's symlink and guarded-read tests pass against the new read + - landed: 57b55ed1f 4. The relevance ranking and its reasons - packages: internal/core/recall (ranking.go: BM25F as designed, ties; the ranked order replaces the id order in deciding which items fit the budget), internal/surface/cli (`rules --prompt` prints each candidate's score and each matched term's field and contribution) - tests: TestRareMatchingWordOutranksACommonOne, the case a raw count gets wrong, watched fail against a count-based order; TestRecallFieldOutweighsBody; TestTiesBreakOnScopeThenRecencyThenID; TestRankingIsDeterministicAcrossRuns; TestRankingDecidesWhichItemsFitTheBudget, watched fail against step 2's id order; TestRulesPromptShowsEachTermsContribution; golden scores for a small fixed corpus so a change to the formula is a visible diff + - landed: f752318ff 5. How full the session is: the status line's fill file, the transcript's tail, and unknown - packages: internal/core/statusline (payload.go parses the payload's `session_id`, never rendered or echoed), internal/surface/cli (statusline.go writes the fill file through the one fill writer, only on the path that renders abcd's own row; its help text and the doc comment above newStatuslineCommand say it writes only that file and where; cli.go's comment on `hookInput.TranscriptPath` names the prompt router as a second reader; the prompt router reads the fill and reports it in the `--json` envelope's `fill`; the session-start reset creates `~/.abcd.noindex/fill/` and the resets delete the session's file), internal/core/rules (the fill writer and reader, the file name under `~/.abcd.noindex/fill/`, the per-model window table, the stale sweep extended to the fill directory), internal/core/history (the token-count shape exported for the fill reader, no behaviour change) - tests: TestStatuslineRecordsTheFillForItsSession, watched fail first; TestStatuslineWritesOnlyTheFillFile (the narrowed TestStatuslineWritesNothing: one changed entry, the session's fill file under `~/.abcd.noindex/fill/` holding only `used_percentage` and `at`, and every other case, the pass-through path among them, unchanged); the narrowed TestStatusVerbReachesNoWritePrimitive (exactly one fill-writer call from one function in statusline.go; any other write primitive or second call site refused by name), with a fixture of a second write watched fail; TestStatuslineCreatesNoDirectory; TestStatuslineWritesNothingWithoutASessionID; TestStatuslineRowRendersWhenTheFillWriteFails; TestFillFileIsRefusedBehindASymlinkedHome; TestFillPrefersTheStatusLineFile; TestFillFromTheTranscriptTail, watched fail first; TestTranscriptReadIsBoundedToTheTail; TestRefusedTranscriptYieldsUnknownFill (symlink, foreign owner); TestUnnamedModelYieldsUnknownFill; TestAmbiguousWindowTakesTheSmaller; TestResetDeletesTheFillFile; TestResetCreatesTheFillDirectoryAndRefusesASymlinkedHome; TestAbsentFillDirectoryIsASilentSkip; TestStaleSweepReachesTheFillDirectory; TestHookEnvelopeNamesTheFillAndItsSource; TestRecallOpensOnlyTheDeclaredPlaces extended to the fill reader's two (P5); the statusline help snapshot updated From 0ce85c327becf09ba7edab59d0ababa376fee845 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Fri, 23 Oct 2026 07:00:23 +0100 Subject: [PATCH 18/23] docs: settle the memory-recall intent's wording-only open question Its one bullet said there is no open question for the product thinker, which the readiness check read as a question; it becomes the italic line the convention uses. Assisted-by: Claude:claude-opus-5-5 --- .../itd-2610091918433290-an-agent-working-in-an-abcd-managed.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.abcd/development/intents/planned/itd-2610091918433290-an-agent-working-in-an-abcd-managed.md b/.abcd/development/intents/planned/itd-2610091918433290-an-agent-working-in-an-abcd-managed.md index 18483d1c8e..a4bb758dad 100644 --- a/.abcd/development/intents/planned/itd-2610091918433290-an-agent-working-in-an-abcd-managed.md +++ b/.abcd/development/intents/planned/itd-2610091918433290-an-agent-working-in-an-abcd-managed.md @@ -47,7 +47,7 @@ We expect fewer repeated mistakes, because a note comes back exactly when a prom ## Open Questions -- None for the product thinker. The four questions first left here are settled under Decisions (2026-10-11); the technical questions the plan leaves open are in the bundle's spec. +_None for the product thinker. The four questions first left here are settled under Decisions (2026-10-11); the technical questions the plan leaves open are in the bundle's spec._ ## Audit Notes From 08431965ad9397b76e8e814b2af83d417b272d1d Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Wed, 28 Oct 2026 12:19:16 +0000 Subject: [PATCH 19/23] fix: bound what one prompt's recall reads of a page store The prompt hook read every page in both scopes in full on every prompt, up to the 4 MiB page cap each and with no limit on their number, then tokenised every body. A recall now reads no page larger than it could send (128 KiB, twice the injection budget), and no more than 256 pages or 1 MiB of one scope per prompt, in name order. A page past the size cap is refused with that reason, and a scope past its bound is named once. The existing too-large path, for a page whose block exceeds the budget, is unchanged. Assisted-by: Claude:claude-opus-5-5 --- internal/core/memory/recallpages.go | 37 +++++++++++++-- internal/core/memory/recallpages_test.go | 57 ++++++++++++++++++++++++ 2 files changed, 90 insertions(+), 4 deletions(-) diff --git a/internal/core/memory/recallpages.go b/internal/core/memory/recallpages.go index 618807823b..2611521ef8 100644 --- a/internal/core/memory/recallpages.go +++ b/internal/core/memory/recallpages.go @@ -106,20 +106,49 @@ func ReadMachineRecallPages(home string) ([]RecallPage, []RecallRefusal) { return recallPagesFrom(store, machineStoreDisplay, read) } -// recallPagesFrom reads, through read, every page the store lists whose name -// is __.md; a page read refuses is named, never followed. +// The recall runs on every prompt, over a store of any size, +// so what one prompt reads of a scope is bounded: no page larger than a +// recall could send (a block over the injection budget is set aside whole), +// and no more pages, or bytes, than the bounds below. The bounds keep the +// hook's worst case within its budget; real stores sit far below them. +const ( + // maxRecallPageBytes is twice the injection budget (64 KiB): a page past + // it can never come back, so its body is never read for ranking. + maxRecallPageBytes = 128 << 10 + // maxRecallPagesPerScope and maxRecallBytesPerScope bound one scope's + // pages per prompt, read in name order. + maxRecallPagesPerScope = 256 + maxRecallBytesPerScope = 1 << 20 +) + +// recallPagesFrom reads, through read, the pages the store lists whose name +// is __.md, in name order and within the bounds above; a +// page read refuses is named, never followed, and pages past a bound are +// named once. func recallPagesFrom(store *storeHandle, shown string, read func(string, int64) ([]byte, error)) ([]RecallPage, []RecallRefusal) { var pages []RecallPage var refused []RecallRefusal + var bytesRead int for _, name := range store.pageNames() { if _, _, _, ok := ParsePageFilename(name); !ok { continue } - raw, err := read(name, maxMemoryPageBytes) + if len(pages) >= maxRecallPagesPerScope || bytesRead >= maxRecallBytesPerScope { + refused = append(refused, RecallRefusal{File: shown, Err: fmt.Errorf("it holds more pages than one prompt's recall reads (%d pages or %d bytes); from %s on, its pages are not recalled", + maxRecallPagesPerScope, maxRecallBytesPerScope, name)}) + break + } + raw, err := read(name, maxRecallPageBytes) if err != nil { refused = append(refused, RecallRefusal{File: shown + name, Filename: name, Err: recallReadError(err)}) continue } + if bytesRead+len(raw) > maxRecallBytesPerScope { + refused = append(refused, RecallRefusal{File: shown, Err: fmt.Errorf("it holds more pages than one prompt's recall reads (%d pages or %d bytes); from %s on, its pages are not recalled", + maxRecallPagesPerScope, maxRecallBytesPerScope, name)}) + break + } + bytesRead += len(raw) p := parsePage(string(raw)) info := pageInfoOf(name, p) pages = append(pages, RecallPage{Filename: name, Summary: info.Summary, Body: bodyAfterSummary(p.body), Recall: recallList(p.fm), @@ -181,7 +210,7 @@ func latestIngest(source map[string]any) time.Time { func recallReadError(err error) error { switch { case errors.Is(err, fsutil.ErrTooBig): - return fmt.Errorf("it exceeds the %d-byte page cap", maxMemoryPageBytes) + return fmt.Errorf("it is larger than the %d bytes a recall reads, so it is never recalled", maxRecallPageBytes) case errors.Is(err, fsutil.ErrNotRegular): return errors.New("it is not a regular file") case errors.Is(err, fsutil.ErrDeclarationWritable): diff --git a/internal/core/memory/recallpages_test.go b/internal/core/memory/recallpages_test.go index bb0ba594d7..579f722038 100644 --- a/internal/core/memory/recallpages_test.go +++ b/internal/core/memory/recallpages_test.go @@ -2,6 +2,7 @@ package memory import ( "errors" + "fmt" "os" "path/filepath" "slices" @@ -247,3 +248,59 @@ func TestRecallPageBodyDoesNotRepeatItsTitle(t *testing.T) { t.Fatalf("summary %q body %q", p.Summary, p.Body) } } + +// TestRecallPagesReadsNoPageLargerThanARecallCanSend: a page larger than the +// recall's read cap is refused with that reason and its body is never read +// into the recall, so one oversized page cannot slow every prompt. +func TestRecallPagesReadsNoPageLargerThanARecallCanSend(t *testing.T) { + repo := t.TempDir() + mem := filepath.Join(repo, ".abcd", "memory") + big := recallPageText("A big page", "auth") + strings.Repeat("word ", maxRecallPageBytes/5+1) + writeStoreFile(t, mem, "fact_eng_big.md", big, 0o644) + writeStoreFile(t, mem, "fact_eng_small.md", recallPageText("A small page", "auth"), 0o644) + pages, refused := ReadRecallPages(repo) + if len(pages) != 1 || pages[0].Filename != "fact_eng_small.md" { + t.Fatalf("pages = %+v", pages) + } + if len(refused) != 1 || refused[0].Filename != "fact_eng_big.md" || !strings.Contains(refused[0].Err.Error(), "never recalled") { + t.Fatalf("refused = %+v", refused) + } +} + +// TestRecallPagesBoundTheScope: a scope holding more pages, or more bytes, +// than one prompt's recall reads yields the first by name up to the bound, +// and one refusal naming the bound, so no store makes every +// prompt read without limit. +func TestRecallPagesBoundTheScope(t *testing.T) { + repo := t.TempDir() + mem := filepath.Join(repo, ".abcd", "memory") + for i := 0; i < maxRecallPagesPerScope+5; i++ { + writeStoreFile(t, mem, fmt.Sprintf("fact_eng_p%04d.md", i), recallPageText("Page", "auth"), 0o644) + } + pages, refused := ReadRecallPages(repo) + if len(pages) != maxRecallPagesPerScope { + t.Fatalf("read %d pages, want the bound %d", len(pages), maxRecallPagesPerScope) + } + if len(refused) != 1 || !strings.Contains(refused[0].Err.Error(), "more pages") { + t.Fatalf("refused = %+v", refused) + } + + heavy := t.TempDir() + hmem := filepath.Join(heavy, ".abcd", "memory") + body := strings.Repeat("word ", (maxRecallPageBytes-1024)/5) + n := maxRecallBytesPerScope/(maxRecallPageBytes-1024) + 3 + for i := 0; i < n; i++ { + writeStoreFile(t, hmem, fmt.Sprintf("fact_eng_h%02d.md", i), recallPageText("Heavy", "auth")+body, 0o644) + } + hpages, hrefused := ReadRecallPages(heavy) + total := 0 + for _, p := range hpages { + total += len(p.Body) + len(p.Summary) + } + if total > maxRecallBytesPerScope || len(hpages) >= n { + t.Fatalf("read %d of %d heavy pages, %d bytes, past the %d-byte bound", len(hpages), n, total, maxRecallBytesPerScope) + } + if len(hrefused) != 1 || !strings.Contains(hrefused[0].Err.Error(), "more pages") { + t.Fatalf("refused = %+v", hrefused) + } +} From 4461bd962fe86c24622f15c0e8545c5af338a4fd Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Fri, 23 Oct 2026 07:24:16 +0100 Subject: [PATCH 20/23] fix: count a refused page toward the recall's scope bound From the Sonnet review of steps 3 and 4: a page the recall refused, an oversized one, did not count toward the scope's 256-page bound, so a store of such pages still had every prompt open each of them and name each on stderr. Every page tried now counts. The review's minor notes stand as choices: a page of 64 to 128 KiB is read and set aside whole, the margin the read cap leaves for its frontmatter; pages past the bound are dropped in name order, said once; common words weigh little through BM25F's own term weighting. Assisted-by: Claude:claude-opus-5-5 --- internal/core/memory/recallpages.go | 7 +++++-- internal/core/memory/recallpages_test.go | 19 +++++++++++++++++++ 2 files changed, 24 insertions(+), 2 deletions(-) diff --git a/internal/core/memory/recallpages.go b/internal/core/memory/recallpages.go index 2611521ef8..6aebabd8e9 100644 --- a/internal/core/memory/recallpages.go +++ b/internal/core/memory/recallpages.go @@ -128,16 +128,19 @@ const ( func recallPagesFrom(store *storeHandle, shown string, read func(string, int64) ([]byte, error)) ([]RecallPage, []RecallRefusal) { var pages []RecallPage var refused []RecallRefusal - var bytesRead int + var bytesRead, tried int for _, name := range store.pageNames() { if _, _, _, ok := ParsePageFilename(name); !ok { continue } - if len(pages) >= maxRecallPagesPerScope || bytesRead >= maxRecallBytesPerScope { + // Every page tried counts, a refused one included, so a store of + // pages the recall refuses cannot make it open every one of them. + if tried >= maxRecallPagesPerScope || bytesRead >= maxRecallBytesPerScope { refused = append(refused, RecallRefusal{File: shown, Err: fmt.Errorf("it holds more pages than one prompt's recall reads (%d pages or %d bytes); from %s on, its pages are not recalled", maxRecallPagesPerScope, maxRecallBytesPerScope, name)}) break } + tried++ raw, err := read(name, maxRecallPageBytes) if err != nil { refused = append(refused, RecallRefusal{File: shown + name, Filename: name, Err: recallReadError(err)}) diff --git a/internal/core/memory/recallpages_test.go b/internal/core/memory/recallpages_test.go index 579f722038..7493b14566 100644 --- a/internal/core/memory/recallpages_test.go +++ b/internal/core/memory/recallpages_test.go @@ -304,3 +304,22 @@ func TestRecallPagesBoundTheScope(t *testing.T) { t.Fatalf("refused = %+v", hrefused) } } + +// TestRecallPagesCountRefusedPagesTowardTheBound: a page the recall refuses +// still counts toward the scope's bound, so a store of oversized pages cannot +// make every prompt open every one of them, or name each on stderr. +func TestRecallPagesCountRefusedPagesTowardTheBound(t *testing.T) { + repo := t.TempDir() + mem := filepath.Join(repo, ".abcd", "memory") + big := recallPageText("Too big", "auth") + strings.Repeat("x", maxRecallPageBytes) + for i := 0; i < maxRecallPagesPerScope+5; i++ { + writeStoreFile(t, mem, fmt.Sprintf("fact_eng_b%04d.md", i), big, 0o644) + } + pages, refused := ReadRecallPages(repo) + if len(pages) != 0 { + t.Fatalf("an oversized page was read: %d", len(pages)) + } + if len(refused) > maxRecallPagesPerScope+1 { + t.Fatalf("%d refusals for a bound of %d pages", len(refused), maxRecallPagesPerScope) + } +} From d4cbe09e9640d9c68de84e08f02b0c2f51d556ac Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Fri, 23 Oct 2026 07:28:56 +0100 Subject: [PATCH 21/23] docs: say that the recall reads memory pages, now steps 2 to 4 have shipped The docs-fidelity review of 332aa6a2d (Opus, with a Sonnet second check) found three brief sentences still describing note and page recall, and the machine page store's reader, as later work. The replacements are applied through `abcd docs fidelity --apply` and flagged for the product thinker, one reworded so its prose names no flag; the same claim in the configuration internals chapter is corrected by hand. Assisted-by: Claude:claude-opus-5-5 --- .../development/brief/04-surfaces/01-ahoy.md | 2 +- .../brief/04-surfaces/07-library.md | 2 +- .../brief/04-surfaces/37-memory.md | 2 +- .../brief/05-internals/03-configuration.md | 2 +- .abcd/work/brief-review-flags.json | 24 +++++++++++++++++++ 5 files changed, 28 insertions(+), 4 deletions(-) diff --git a/.abcd/development/brief/04-surfaces/01-ahoy.md b/.abcd/development/brief/04-surfaces/01-ahoy.md index d5574cb84a..c755bc2399 100644 --- a/.abcd/development/brief/04-surfaces/01-ahoy.md +++ b/.abcd/development/brief/04-surfaces/01-ahoy.md @@ -468,7 +468,7 @@ earlier version wrote wider is narrowed when abcd next writes it a person's reviewed commit, whose note wins an id both hold memory/ curated cross-project memory pages (a later phase; - nothing resolves it today) + the prompt router's recall reads them, read only) sources/ the local sources corpus the library verb maintains and /abcd:library reads. Created only by its explicit init: absent means every other corpus verb and the diff --git a/.abcd/development/brief/04-surfaces/07-library.md b/.abcd/development/brief/04-surfaces/07-library.md index afd9295aaf..873f0488ff 100644 --- a/.abcd/development/brief/04-surfaces/07-library.md +++ b/.abcd/development/brief/04-surfaces/07-library.md @@ -412,7 +412,7 @@ by the bare render; orphan and stale-claim audits are deferred. Not built: a curator role on the `principle-distiller` agent (the agent and the disembark sub-verb it feeds ship; this role does not), ingest and ask run -reports, user-scope memory outside the repo, and the automatic extraction of a +reports, any verb that writes user-scope memory outside the repo (the prompt recall only reads `~/.abcd.noindex/memory/`), and the automatic extraction of a spec's modification grammar into the store at spec completion. ### Composition with adjacent surfaces diff --git a/.abcd/development/brief/04-surfaces/37-memory.md b/.abcd/development/brief/04-surfaces/37-memory.md index 321f03b252..7542d5776e 100644 --- a/.abcd/development/brief/04-surfaces/37-memory.md +++ b/.abcd/development/brief/04-surfaces/37-memory.md @@ -4,7 +4,7 @@ session does not learn them again (itd-2610091918433290, spc-2610111627213138). It is an agents' command: its page is hidden from a person's menu, and the binary's help lists the verb with the agents. Recall of a note through the prompt hook is the same -spec's later step; this chapter covers the store and the verb. +spec's steps 2 to 4, shipped: a matching note or page comes back under `## MEMORY`, ranked by BM25F, and the rules verb can show why for a given prompt; this chapter covers the store and the verb. ## Sub-verbs diff --git a/.abcd/development/brief/05-internals/03-configuration.md b/.abcd/development/brief/05-internals/03-configuration.md index 22f20b95e4..dcbba86133 100644 --- a/.abcd/development/brief/05-internals/03-configuration.md +++ b/.abcd/development/brief/05-internals/03-configuration.md @@ -679,7 +679,7 @@ with `--everywhere`; [the prompt router](#the-prompt-routers-output) recalls the beside the repository's `.abcd/memory.json`, whose notes a person commits in a reviewed change and the verb never writes, as one set in which a project note shadows a personal note with the same id), curated cross-project memory pages -under `memory/` (a later phase: nothing resolves it today), and the `sources/` corpus +under `memory/` (the prompt router's recall reads them, read only), and the `sources/` corpus `/abcd:library` reads and writes (abcd never creates that one unasked: only its explicit init does, and every other corpus verb and the page say so and stop when it is absent). It also holds the caller-controlled declarations: the owned diff --git a/.abcd/work/brief-review-flags.json b/.abcd/work/brief-review-flags.json index 671f92922a..23d685d7f8 100644 --- a/.abcd/work/brief-review-flags.json +++ b/.abcd/work/brief-review-flags.json @@ -208,6 +208,30 @@ "evidence": ".abcd/work/decisions/ (538 records) against .abcd/development/decisions/adrs/ (92 files)", "commit": "2a692195fd82fec4229e064a2fb0f15b34c80172", "applied": "2026-10-16T05:32:44Z" + }, + { + "chapter": "01-ahoy.md", + "sentence": "nothing resolves it today)", + "replacement": "the prompt router's recall reads them, read only)", + "evidence": "internal/core/memory/recallpages.go:80", + "commit": "332aa6a2dfd2bdd4a4631188b528cd1002a8729f", + "applied": "2026-10-23T06:27:37Z" + }, + { + "chapter": "07-library.md", + "sentence": "reports, user-scope memory outside the repo, and the automatic extraction of a", + "replacement": "reports, any verb that writes user-scope memory outside the repo (the prompt recall only reads `~/.abcd.noindex/memory/`), and the automatic extraction of a", + "evidence": "internal/core/recall/pages.go:56", + "commit": "332aa6a2dfd2bdd4a4631188b528cd1002a8729f", + "applied": "2026-10-23T06:27:37Z" + }, + { + "chapter": "37-memory.md", + "sentence": "spec's later step; this chapter covers the store and the verb.", + "replacement": "spec's steps 2 to 4, shipped: a matching note or page comes back under `## MEMORY`, ranked by BM25F, and the rules verb can show why for a given prompt; this chapter covers the store and the verb.", + "evidence": "internal/surface/cli/rules_prompt.go:51", + "commit": "332aa6a2dfd2bdd4a4631188b528cd1002a8729f", + "applied": "2026-10-23T06:27:37Z" } ] } From 7263d1b9d57dcf16d17db179f6478f6351a25456 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Fri, 23 Oct 2026 07:30:50 +0100 Subject: [PATCH 22/23] chore: record that the memory-recall spec stays open past steps 3 and 4 Assisted-by: Claude:claude-opus-5-5 --- ...10230630506726-the-memory-recall-spec-stays-open-when.md | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 .abcd/work/decisions/dcn-2610230630506726-the-memory-recall-spec-stays-open-when.md diff --git a/.abcd/work/decisions/dcn-2610230630506726-the-memory-recall-spec-stays-open-when.md b/.abcd/work/decisions/dcn-2610230630506726-the-memory-recall-spec-stays-open-when.md new file mode 100644 index 0000000000..ead61b488d --- /dev/null +++ b/.abcd/work/decisions/dcn-2610230630506726-the-memory-recall-spec-stays-open-when.md @@ -0,0 +1,6 @@ +--- +id: dcn-2610230630506726 +slug: the-memory-recall-spec-stays-open-when +date: 2026-10-23 +--- +2026-10-23 — The memory-recall spec stays open when steps 3 and 4 land, with no close and no remainder (facilitator, on abcd-ae's question of 2026-10-23; spc-2610111627213138, itd-2610091918433290): PR #982 delivers steps 3 and 4, so steps 1 to 4 have landed, but steps 5 to 8 (how full the session is, context brackets, the docs and brief, and the live session with the doc-fidelity review and the close) are not built, so the spec is not closed and the intent stays planned; a --remainder close would only move those four steps into a new spec under the same intent, which adds a record and changes nothing, so the open spec keeps them and the change carries no Delivers trailer (grounds: AGENTS.md definition of done, a change that delivers part of an intent carries no trailer; commands on spec close --remainder) From 8c0cafe84c1ad294a3e6b33412e9ec7b4e4d31da Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Fri, 23 Oct 2026 07:32:42 +0100 Subject: [PATCH 23/23] docs: say how recalled pages come back, and what rules --prompt shows The docs-fidelity review left two gaps: the memory command page said how a note comes back but not a page, and the brief's rules section never named the prompt diagnostic. The page now gives each store's label, the shared ranking and the too-large case; the brief's rules section gains the --prompt form, read from rules_prompt.go. Assisted-by: Claude:claude-opus-5-5 --- .abcd/development/brief/04-surfaces/README.md | 6 ++++++ commands/memory.md | 15 +++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/.abcd/development/brief/04-surfaces/README.md b/.abcd/development/brief/04-surfaces/README.md index 1c00d4274a..fe4ccf2bc4 100644 --- a/.abcd/development/brief/04-surfaces/README.md +++ b/.abcd/development/brief/04-surfaces/README.md @@ -294,6 +294,12 @@ the checkout from the root the loader resolves for the working directory. - **`abcd rules `** renders that one domain, matched case-insensitively, whatever its state and whatever the kill switch, so a dormant domain stays inspectable. An unknown name exits 2. +- **`abcd rules --prompt ""`** shows what that one prompt would bring back + to a fresh session, running the prompt router's own path: the domains its + words match, then the memory notes and pages it would recall, ranked together + with each one's rank, score and the terms that scored it, and the form each + would take (`full`, `over-budget` or `too-large`), with any layer or page that + could not be read. It writes nothing: no session state is loaded or saved. - **Diagnostics go to stderr, never stdout**, so the JSON form stays one document: a refused root, a domain skipped for carrying no rules, and each bundled entry of a guardrail domain an override's list leaves out. diff --git a/commands/memory.md b/commands/memory.md index e2b0884ecb..c0aec584bc 100644 --- a/commands/memory.md +++ b/commands/memory.md @@ -85,6 +85,21 @@ repository's words, not the person's, because anyone who can commit to the repository can write one. A heading or a rule written inside a fenced body is part of the note. It is not a rule the loader injected. +Memory pages come back the same way, under the same heading, when a prompt +matches one of a page's `recall:` words. Notes and pages are ranked together, +the best match first, until the turn's budget is spent. A page's first line +names it `page:` and says which store it came from: + +- `(project page, committed in this repository)` is a page from the project's + `.abcd/memory/`. Read it as the repository's words, as a project note. +- `(machine page, this account's memory store)` is a page from this machine's + `~/.abcd.noindex/memory/`. + +A page too large to come back in full is named on stderr rather than sent. To +see what one prompt would bring back, and why each note or page ranks where it +does, run `"${CLAUDE_PLUGIN_ROOT}/abcd" rules --prompt ""`; it +writes nothing. + ## Move a lesson from the host's own memory When the person points at a lesson in the host's own memory files, or pastes