feat: memory pages join the recall, ranked by BM25F (spc-2610111627213138 steps 3 and 4)

.abcd/development/specs/open/spc-2610111627213138-memory-recall.md

@@ -575,15 +575,19 @@ Steps 7 and 8 change no code.
    - the memory store's redactor (`storeRedactor` in internal/core/memory) is exported for the gate rather than a second one written (one-canonical-primitive)
    - the store and its verb land together, so the store has a caller from its first commit and no unwired code reaches main; a note written here is listed and removed at once, and is recalled from step 2
    - a live trial in a host session checks that the host substitutes its session id into the command page, as the verb split's trial checked hiding; the capture goes under this spec's `## Live captures`. If the first host offers no substitution, the page leaves `--session` out and every note on that host is recorded as session unknown, which is the ruled fallback, not a reopened question
+   - landed: c8d1ab0df, 9ae13776b
 2. One recall: notes come back through the prompt hook, and `rules --prompt` shows what a prompt brings back
    - packages: internal/core/recall (new: items, matching, rendering, the `recalled` ledger keyed on content; candidates that do not all fit the budget are taken in id order until step 4), internal/core/rules (exported prompt-index and term-match function; SessionState gains `recalled`; the reset clears it), internal/surface/cli (the prompt router calls recall after rules.Inject; the `--json` envelope gains `recalled`; `rules --prompt` listing matched domains with their hitting words, and matched notes), commands/ahoy.md (documents `rules --prompt`), .abcd/development/release/surface.json and docs/reference/cli/commands.md (the new flag)
    - tests: TestANoteComesBackOnAPromptAboutTests over two simulated sessions (N1), watched fail first; TestNoteRecallUsesTheRuleTermMatch (stems and phrases hit as a domain's recall words do); TestANoteInTheProjectsFileIsRecalled (N5); TestNoteTiedToAnotherRootIsNotRecalled and TestEverywhereNoteIsRecalledInAnyRepository; TestRecalledLedgerSurvivesTheRulesPrune; TestResetClearsBothLedgers; TestKillSwitchSilencesRecall; TestARefusedMachineLayerStillInjectsRules; TestMemoryTakesOnlyWhatTheRulesLeaveOfTheBudget; TestCandidatesThatFitAreTakenInIDOrder (replaced by step 4's ranking tests); TestRulesPromptShowsMatchedDomainsAndNotes and TestRulesPromptWritesNothing, watched fail first; TestRulesPromptRefusesAPositionalDomain; the existing inject and hook tests pass unchanged when no note matches (a healthy no-match still renders zero tokens)
+   - landed: e6d2a7df7, 59b69f398
 3. Pages join the same recall, from both scopes
    - packages: internal/core/memory (an exported read of each page's filename, index line, body and `recall:` list, through the existing store handle; a read-only open of `~/.abcd.noindex/memory/`), internal/core/recall (page items, narrower scope wins; pages join the id order as `page:<filename>`, after notes, until step 4), internal/surface/cli (the router and `rules --prompt` hand pages to recall)
    - tests: TestAPageDeclaringAuthComesBackAndNoOther (P1); TestANoteAndAPageComeBackTogether (N2); TestRepeatedPromptInjectsMemoryOnce (P3); TestWorkspacePageSuppressesTheMachinePage (P4); TestRecallOpensOnlyTheDeclaredPlaces with a host memory directory of matching text under the test home (P5), each watched fail first; TestPageWithoutRecallIsNeverACandidate; TestRulesPromptShowsMatchedPages; the memory store's symlink and guarded-read tests pass against the new read
+   - landed: 57b55ed1f
 4. The relevance ranking and its reasons
    - packages: internal/core/recall (ranking.go: BM25F as designed, ties; the ranked order replaces the id order in deciding which items fit the budget), internal/surface/cli (`rules --prompt` prints each candidate's score and each matched term's field and contribution)
    - tests: TestRareMatchingWordOutranksACommonOne, the case a raw count gets wrong, watched fail against a count-based order; TestRecallFieldOutweighsBody; TestTiesBreakOnScopeThenRecencyThenID; TestRankingIsDeterministicAcrossRuns; TestRankingDecidesWhichItemsFitTheBudget, watched fail against step 2's id order; TestRulesPromptShowsEachTermsContribution; golden scores for a small fixed corpus so a change to the formula is a visible diff
+   - landed: f752318ff
 5. How full the session is: the status line's fill file, the transcript's tail, and unknown
    - packages: internal/core/statusline (payload.go parses the payload's `session_id`, never rendered or echoed), internal/surface/cli (statusline.go writes the fill file through the one fill writer, only on the path that renders abcd's own row; its help text and the doc comment above newStatuslineCommand say it writes only that file and where; cli.go's comment on `hookInput.TranscriptPath` names the prompt router as a second reader; the prompt router reads the fill and reports it in the `--json` envelope's `fill`; the session-start reset creates `~/.abcd.noindex/fill/` and the resets delete the session's file), internal/core/rules (the fill writer and reader, the file name under `~/.abcd.noindex/fill/`, the per-model window table, the stale sweep extended to the fill directory), internal/core/history (the token-count shape exported for the fill reader, no behaviour change)
    - tests: TestStatuslineRecordsTheFillForItsSession, watched fail first; TestStatuslineWritesOnlyTheFillFile (the narrowed TestStatuslineWritesNothing: one changed entry, the session's fill file under `~/.abcd.noindex/fill/` holding only `used_percentage` and `at`, and every other case, the pass-through path among them, unchanged); the narrowed TestStatusVerbReachesNoWritePrimitive (exactly one fill-writer call from one function in statusline.go; any other write primitive or second call site refused by name), with a fixture of a second write watched fail; TestStatuslineCreatesNoDirectory; TestStatuslineWritesNothingWithoutASessionID; TestStatuslineRowRendersWhenTheFillWriteFails; TestFillFileIsRefusedBehindASymlinkedHome; TestFillPrefersTheStatusLineFile; TestFillFromTheTranscriptTail, watched fail first; TestTranscriptReadIsBoundedToTheTail; TestRefusedTranscriptYieldsUnknownFill (symlink, foreign owner); TestUnnamedModelYieldsUnknownFill; TestAmbiguousWindowTakesTheSmaller; TestResetDeletesTheFillFile; TestResetCreatesTheFillDirectoryAndRefusesASymlinkedHome; TestAbsentFillDirectoryIsASilentSkip; TestStaleSweepReachesTheFillDirectory; TestHookEnvelopeNamesTheFillAndItsSource; TestRecallOpensOnlyTheDeclaredPlaces extended to the fill reader's two (P5); the statusline help snapshot updated
@@ -748,6 +752,90 @@ each open to the facilitator's veto at review:
   bounds what a note can carry or how it is named without a ruling of its own,
   and is open to the facilitator's veto at review like the choices above.
 
+### Settled here
+
+Choices an agent made while building steps 3 and 4 where this spec is silent,
+each the narrowest reading the agent found that meets the criteria, and each
+an agent's choice for review, open to the facilitator's veto like the design
+choices above.
+
+- **Which pages the recall reads** (2026-10-12, step 3, agent's choice for
+  review). In each scope, the pages `index.md` lists, the page files at the
+  store's top level, whose names are `<type>_<domain>_<slug>.md`. A page's
+  title is its index line's summary, derived from the page the way the index
+  derives it, so the recall reads no `index.md` and opens only the four
+  places it declares; its body is the rest of the page after the line the
+  title comes from, so the title is counted and sent once. Grounds: P4 compares pages by that name, and the name's
+  characters keep a page's key (`page:<filename>`) safe to print in the
+  hook's stderr line and envelope.
+- **A workspace page shadows the machine page of the same name, whether or
+  not it carries a recall list, and when the read refuses it** (2026-10-12,
+  step 3, agent's choice for review). Grounds: the narrower scope's page is the page of that name, as a
+  repository layer's field replaces the machine's and a note in the project's
+  file suppresses the machine's copy of its id; it also keeps one
+  `page:<filename>` key per name in the ledger and the envelope.
+- **A machine page is read with the guards the machine `memory.json` gets**
+  (2026-10-12, step 3, agent's choice for review): the directory through
+  `fsutil.OpenHomeDeclarationScope`, the home scope's walk with the
+  directory judgement `fsutil.ReadHomeDeclaration` makes (a symlinked
+  directory, or one another account can change, refused), and each page a
+  regular file this account owns that no one else can write
+  (`fsutil.ReadHomeDeclarationDenyingIn`). A page failing
+  them is named on stderr and brings nothing back, while the scope's other
+  pages still come back. Grounds: a machine page injects text into every
+  session as a machine note does; the spec names only the directory's open,
+  and reading it alone would trust a page another account could write while
+  the notes file beside it is refused for the same thing.
+- **A page store or a page that cannot be read is named** (2026-10-12,
+  step 3, agent's choice for review): on the hook's stderr with "recalling
+  nothing from it", and in `rules --prompt`'s `unreadable` list, which gains a
+  `kind` (`note` or `page`); a refused `memory.json` keeps step 2's words.
+- **What a page's recency is** (2026-10-12, step 4, agent's choice for
+  review). The tie-break's "page modification" is the latest `ingested_at`
+  the page's source block records, a date, and a page that records none
+  sorts as the oldest. Grounds: a checkout stamps each file with the time it
+  wrote it out, so a file's modification time differs between clones and the
+  order would not be the same on every machine, as Ranking requires;
+  `ingested_at` is in the page and travels with it.
+- **How the ranking counts terms and lengths** (2026-10-12, step 4, agent's
+  choice for review). Each distinct word of the prompt is one query term,
+  however often or in whatever inflection the prompt repeats it (tokens whose
+  candidate roots overlap, "merge" and "merging", are one term), and a word
+  in an item counts as that term when its stem is among the roots the
+  loader's term rule matches the prompt's tokens by, so the ranking finds a
+  term exactly where the matcher hits, and under one term at most. That takes two small exports from `internal/core/rules`
+  beside `TermMatcher` (`PromptTerms`, read from the same prompt index, and
+  `TextStems`), though step 4's packages line names only the recall and the
+  CLI. An item's length is its weighted length, recall words three times,
+  title twice, body once, BM25F's one pseudo-document of repeated fields;
+  `avg_len` and N run over every item read after the narrower-scope
+  suppression, so a page without a recall list, never an item, counts in
+  neither. A personal note tied to another repository counts in both
+  (amended 2026-10-17 on merging main, agent's choice for review): step 2's
+  review made the tie a lazy judgement of the matched candidates alone
+  (the recall package's `matching`, which asks git only when a tied note matched), so
+  the corpus is fixed before git is asked and never depends on whether it
+  was.
+- **The ranked order keeps step 2's take** (2026-10-12, step 4, agent's
+  choice for review): candidates are taken in ranked order while they fit,
+  and the first that does not fit stops the take, so nothing comes back
+  ahead of an item ranked above it. The one exception is an item whose block
+  alone is larger than the whole per-turn budget, which no turn can carry: it
+  is set aside as too large, never recorded, named on the hook's stderr and
+  given the form `too-large` in `rules --prompt`, and it does not stop the
+  take. Grounds: a page can be as large as 4 MiB, and one that ranks first
+  would otherwise hold back every item below it on every turn.
+- **Scores tie at a billionth** (2026-10-12, step 4, agent's choice for
+  review): two scores within a billionth of each other are a tie, broken by
+  scope, recency and key. Grounds: arithmetic equal on paper can differ in
+  its last bits between machines, and the order must be the same on every
+  machine.
+- **The diagnostic ranks notes and pages together** (2026-10-12, step 4,
+  agent's choice for review): `rules --prompt` keeps its `notes` and `pages`
+  lists, and each entry carries its `rank` in the one order they share, its
+  `score`, and its `terms`, each with its counts in `recall`, `title` and
+  `body` and its `contribution`.
+
 ## Open Questions
 
 None. The nine questions first left here, and the three the spec review

commands/ahoy.md

@@ -1327,12 +1327,23 @@ prompt in a fresh session, and writes nothing. `domains` lists each rule
 domain the prompt matches, with the recall words that hit (`hits`, or
 `*NAME` for a star activation); `notes` lists each memory note it recalls
 from `~/.abcd.noindex/memory.json` or the project's `.abcd/memory.json`, with
-its `layer` (`machine` or `project`) and the recall words that hit. Each
-carries a `form`: `full`, or `over-budget` when it would not fit the
-per-turn budget and would come back on a later prompt instead. `disabled`
-is true under the kill switch, and then nothing comes back; `unreadable`
-names a `memory.json` that could not be read, and why. Use it when someone
-asks why a rule or a note did, or did not, come back on a prompt. A
+its `layer` (`machine` or `project`) and the recall words that hit; `pages`
+lists each memory page it recalls from the project's `.abcd/memory/` or
+`~/.abcd.noindex/memory/`, by its `key` (`page:<filename>`, the name the
+hook's envelope gives it), with its `filename`, `title`, `layer` and the
+recall words that hit. A page comes back only when its frontmatter declares
+a `recall:` list, and a page in the project's store suppresses the machine's
+page of the same name. Notes and pages share one ranked order: each carries
+its `rank` (1 first), its BM25F relevance `score`, and `terms`, each matched
+term with how often it was found in the item's `recall` words, `title` and
+`body` and its `contribution` to the score; the ranking decides which items
+fit the budget. Each carries a `form`: `full`, `over-budget` when
+it would not fit the per-turn budget and would come back on a later prompt
+instead, or `too-large` when it is larger than the whole per-turn budget and
+no prompt brings it back in full. `disabled` is true under the kill switch, and then nothing comes
+back; `unreadable` names a `memory.json`, page store or page that could not
+be read, why, and its `kind` (`note` or `page`). Use it when someone asks why
+a rule, a note or a page did, or did not, come back on a prompt. A
 positional DOMAIN with `--prompt` is refused, exit 2: `rules <DOMAIN>`
 renders one domain's rules instead.
 

internal/core/memory/ancestor_symlink_test.go

@@ -77,4 +77,17 @@ func TestMemoryStoreDirSymlinkRefused(t *testing.T) {
 	if _, err := Bare(repoRoot); err == nil {
 		t.Error("Bare followed a symlinked .abcd/memory store; the directory symlink must be refused")
 	}
+
+	// (d) The prompt router's recall read (spc-2610111627213138 step 3) must
+	// refuse it too, and disclose nothing of the target.
+	pages, refused := ReadRecallPages(repoRoot)
+	for _, p := range pages {
+		if strings.Contains(p.Body, secret) {
+			t.Fatalf("ReadRecallPages disclosed an out-of-repo page through the store symlink: %q", p.Filename)
+		}
+	}
+	var unsafe *UnsafeStorePathError
+	if len(refused) != 1 || !errors.As(refused[0].Err, &unsafe) {
+		t.Errorf("ReadRecallPages followed a symlinked .abcd/memory store (refused %+v); the directory symlink must be refused", refused)
+	}
 }