From 1f186941c65e78f031118bf2f6a631845138de73 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 09:51:56 -0700 Subject: [PATCH 01/18] docs: design for indexing docs page content in search Search matches titles, not documentation -- searchableText() only reads title/description/slug/section/library out of docs-config, never a page body. That mattered less when search was a power-user shortcut; #986 made it the front door in two places. Adds a dynamic route that indexes prose at heading granularity and returns anchor deep links with highlighted snippets, while keeping the existing client-side title matcher as the instant layer so the fast path stays fast and a failed request degrades to today's behaviour. Records two traps found while exploring: outputFileTracingIncludes does not cover content/docs (api/markdown only works because it is statically generated), and extract-headings hand-rolls slugification, so the index must reuse it rather than introduce a second slugger. Co-Authored-By: Claude Opus 5 --- ...-09-03-docs-search-content-index-design.md | 129 ++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-03-docs-search-content-index-design.md diff --git a/docs/superpowers/specs/2026-09-03-docs-search-content-index-design.md b/docs/superpowers/specs/2026-09-03-docs-search-content-index-design.md new file mode 100644 index 000000000..dc1a136b1 --- /dev/null +++ b/docs/superpowers/specs/2026-09-03-docs-search-content-index-design.md @@ -0,0 +1,129 @@ +# Docs search: index page content — design + +Date: 2026-09-03 +Status: approved, ready for planning + +Docs search matches titles, not documentation. This makes page body text searchable, returns results at heading granularity with anchor deep links, and shows the matching text. + +## Why now + +PR #986 promoted search to the primary navigation affordance in two places: it leads the control-plane pane on every docs page, and it closes every content page. Before that change search was a `⌘K` shortcut for people who already knew the docs; now it is the front door. + +But `DocsSearch` does not search the docs. Its `searchableText()` concatenates `title`, `description`, `slug`, `section` and `libraryTitle` — all of it drawn from `docs-config.ts`. No page body is ever read. Someone searching `checkpointer`, `toAgent`, or an error string pasted from a stack trace gets "No results found" unless the term happens to sit in a page title. + +So the front door is a title matcher. That is the gap this closes. + +## Constraints found in the codebase + +**Content size.** 122 `.mdx` files, ~847KB of raw source. Shipping that to the browser is the thing the delivery decision has to answer. + +**`outputFileTracingIncludes` does not cover `content/docs`.** `apps/website/next.config.ts` traces `cockpit/**` md/py/ts, the Mastra deployment mjs, and `nx.json` — nothing under `apps/website/content`. The existing `api/markdown/[library]/[section]/[slug]/route.ts` reads MDX and works anyway *because it declares `generateStaticParams()`*: the files are read at build time and the route ships as static output. A search route cannot be statically generated — the query space is unbounded — so it reads at request time and needs the content traced into the deployed function. This works in `next dev` and fails in production, which makes it the highest-risk item in the change. + +**Heading anchors come from an approximation.** `lib/extract-headings.ts` hand-rolls slugification under a comment claiming it matches `rehype-slug`. Real `github-slugger` (which `rehype-slug` uses) de-duplicates repeated headings by appending `-1`, `-2`; this implementation does not. An existing e2e test (`every rail link resolves to a heading in the article`) proves the approximation resolves for today's content, so it is consistent with what actually renders. The index therefore **reuses `extractHeadings`** rather than writing a second slugger — one approximation is a known quantity, two that drift apart is a bug generator. + +**No search dependency is installed**, and the existing matcher is hand-rolled token-AND. The design stays in that idiom. + +## Decisions + +| Question | Decision | +| --- | --- | +| How the index reaches the browser | It does not. A dynamic API route answers queries. | +| Result granularity | Heading-level, deep-linked to the section anchor. | +| What is indexed | Prose and inline code. Fenced code blocks stripped. | +| Result rows | Show a snippet with the matched terms highlighted. | + +## Architecture + +Three units with clean boundaries. + +### 1. `apps/website/src/lib/docs-search-index.ts` — pure + +Turns MDX source into section records. No I/O, no framework imports, trivially unit-testable. + +```ts +export interface DocSection { + library: string; + section: string; + slug: string; + /** Page title, for ranking and the result's second line. */ + title: string; + /** Heading text, or null for content above the first heading. */ + heading: string | null; + /** `#id` fragment, or null when the record covers the page preamble. */ + anchor: string | null; + /** Searchable prose: fenced code removed, inline code unwrapped. */ + text: string; +} +``` + +Sectioning rule: split the body at each `##`/`###`, using `extractHeadings` for both the heading text and its id so anchors match the rendered page by construction. Content before the first heading becomes one record with `heading: null` and `anchor: null`, which is where a page's opening paragraphs live — often the best summary of what the page is about. + +Text normalisation, in order: strip frontmatter (reuse `stripFrontmatter` from `lib/docs.ts`), remove fenced code blocks, unwrap inline code so `` `provideAgent` `` indexes as `provideAgent`, handle MDX components, and collapse whitespace. + +MDX components need an explicit rule, because the content tree uses them heavily. Drop the tags themselves and keep any text between them — a ``'s body is prose and should be searchable. For a self-closing component, index the value of a `caption` or `title` attribute if present (`` is real prose a reader might search) and discard every other attribute, which is markup configuration rather than content. + +Fenced code is dropped deliberately. It is roughly half the bytes and mostly noise for ranking. The cost is explicit: pasting an error string that appears only inside a code sample will not match. Inline code is kept because that is where API names appear in sentences. + +### 2. `apps/website/src/app/api/docs-search/route.ts` + +Builds the index once at module scope — once per lambda instance, not per request — by walking `getAllDocSlugs()` and `getDocBySlug()` and running each body through the indexer. + +Query handling: + +- Fewer than 2 characters after trimming: return `{ results: [] }` without scanning. +- Tokenise with the same rules the client already uses. `searchTokens` and `SEARCH_STOP_WORDS` currently live inside `DocsSearch.tsx`, a client component; move both to a shared module (`lib/docs-search-tokens.ts`) that the component and the route import. Sharing the literal function — rather than reimplementing it server-side — is what keeps client and server agreeing on what a query means; two copies would drift the moment a stop word is added. +- Every token must appear, matching today's AND semantics. +- Score by field: page title 3, heading text 2, body prose 1. Sum across tokens; ties break toward the shorter text, which favours a precise heading over a long prose blob. +- Cap at 8, matching the existing result cap. +- `Cache-Control: public, max-age=300` so repeated queries are served by the CDN. The corpus only changes on deploy. + +Response shape: + +```ts +interface DocsSearchHit { + href: string; // /docs//
/[#anchor] + title: string; // page title + heading: string | null; + libraryTitle: string; + snippet: string; + /** [start, end) offsets into `snippet`, for highlighting. */ + marks: [number, number][]; +} +``` + +Snippets are returned as text plus offsets, never as HTML. The client wraps the ranges itself, so no server-built markup is ever rendered into the page. + +### 3. `apps/website/src/components/docs/DocsSearch.tsx` + +The existing client-side matcher stays exactly as it is and keeps rendering immediately as the user types. Server hits merge in beneath it when they arrive, under a divider, each row showing its heading and snippet. + +This refines the "server per keystroke" decision rather than implementing it literally, and the reason is concrete: today's results are instant. A purely server-driven search would put a network round-trip in front of every result, and the first search of a session would additionally wait on a cold lambda. Keeping the instant layer means the fast path stays fast, and a slow, failed or offline request degrades to precisely today's behaviour instead of an empty box. + +Request handling: debounce 150ms, `AbortController` to cancel superseded requests, ignore any response whose query no longer matches the current input. Errors are swallowed — search silently shows the instant results only. A failed fetch must never surface an error state in the dialog. + +De-duplication: if the server returns a hit for a page the client matcher already listed, keep the client row and drop the server one unless the server hit carries an anchor, in which case the deeper link wins. + +## The production trap + +Add `content/docs/**/*.mdx` to `outputFileTracingIncludes` in `apps/website/next.config.ts`. + +The exact glob is relative to the app directory — existing entries reach the repo root with `../../` — and **must be verified against a real `nx build website` followed by a production-mode request to the route**, not assumed correct. The failure mode is silent in development: `next dev` reads from the working tree and never exercises tracing. + +## Testing + +| Suite | Assertion | +| --- | --- | +| `docs-search-index.spec.ts` | sectioning splits on `##`/`###`; preamble becomes an anchor-less record; fenced code removed; inline code retained unwrapped; frontmatter stripped | +| anchor-parity test | every anchor the indexer emits for a file is one `extractHeadings` produces for that same file — search deep links cannot drift from the TOC | +| `route.spec.ts` | a term present only in body prose returns a hit; ranking puts a title match above a prose match; queries under 2 characters return empty; response carries `Cache-Control` | +| `DocsSearch.spec.tsx` | a rejected fetch still renders instant results; a superseded response is discarded; snippet offsets render as marks | +| e2e | search a term that appears only in body prose, land on the correct `#anchor` | + +The anchor-parity test is the load-bearing one. It runs over real content, so a heading the indexer sections differently from the TOC fails the build rather than shipping a deep link to nowhere. + +## Out of scope + +- **Fenced code content.** Excluded by decision above. +- **Real `github-slugger` in `extract-headings.ts`.** It would fix duplicate-heading anchors for both the TOC and search, but it changes rendered TOC behaviour and belongs in its own change. The limitation is accepted here: two `## Overview` headings on one page share an anchor, so a hit may land on the first. +- **Fuzzy matching, stemming, typo tolerance.** The current matcher is exact-substring AND; this keeps that contract. Adding a scoring library is a separate decision. +- **Search analytics beyond what exists.** Noted while reading: `DocsSearch`'s `track()` calls map only `langgraph`/`render`/`chat` to a library name, so `ag-ui`, `a2ui`, `middleware`, `runtimes` and `deep-agents` all record as `unknown`. That is a pre-existing data-quality bug, unrelated to this change, and worth its own fix. From c2ac76e435d35315921908d46651931e4b929a88 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 09:58:44 -0700 Subject: [PATCH 02/18] docs: implementation plan for docs content search Eight tasks, 41 TDD steps. Ordered so the shared tokenizer lands before the route and query module that import it. Two refinements from the spec, both recorded in the plan: the wire types get a dependency-free module (a client component needs DocsSearchHit, whose natural home transitively imports fs), and DocSection drops the per-page fields the spec repeated on every record. Co-Authored-By: Claude Opus 5 --- .../2026-09-03-docs-search-content-index.md | 1259 +++++++++++++++++ 1 file changed, 1259 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-03-docs-search-content-index.md diff --git a/docs/superpowers/plans/2026-09-03-docs-search-content-index.md b/docs/superpowers/plans/2026-09-03-docs-search-content-index.md new file mode 100644 index 000000000..a624c6ba5 --- /dev/null +++ b/docs/superpowers/plans/2026-09-03-docs-search-content-index.md @@ -0,0 +1,1259 @@ +# Docs Search Content Index Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make docs page body text searchable, returning results at heading granularity with anchor deep links and highlighted snippets. + +**Architecture:** A pure indexer turns MDX source into per-heading section records. A dynamic API route builds that index once per lambda instance and answers queries. The existing client-side title matcher stays as an instant layer; server content hits merge in beneath it. + +**Tech Stack:** Next.js App Router (route handlers, RSC), TypeScript, Nx, Vitest + Testing Library, Playwright. + +**Spec:** `docs/superpowers/specs/2026-09-03-docs-search-content-index-design.md` + +--- + +## Before you start + +Read the spec. Then note these repo facts, which are not guessable: + +1. **Website tests need an env var.** `GROWTH_FORM_POLICY=growth_v1` or the app throws at import. Every website command below includes it. +2. **Targeted test runs must start from the project directory.** `cd apps/website` first. From the repo root vitest reports "No test files found" and exits 1 — a wrong CWD, not a broken command. +3. **Unused imports are ESLint ERRORS here,** not warnings. +4. **Route specs run in the node environment.** They start with `// @vitest-environment node`. Component specs use `// @vitest-environment jsdom`. Getting this wrong produces confusing failures. +5. **`next dev` does not exercise output file tracing.** Task 5 exists because of that: a route reading `content/docs` at request time works in dev and fails in a real build unless the config is updated. + +### Existing signatures you will use + +```ts +// apps/website/src/lib/docs.ts +export function getAllDocSlugs(): { library: string; section: string; slug: string }[] +export function getDocBySlug(library: string, section: string, slug: string): ResolvedDoc | null +export function stripFrontmatter(source: string): string +interface ResolvedDoc { page: DocsPage; content: string; body: string; title: string } + +// apps/website/src/lib/extract-headings.ts +export function extractHeadings(source: string): DocHeading[] +interface DocHeading { id: string; text: string; level: number } + +// apps/website/src/lib/docs-config.ts +export function getLibraryConfig(libraryId: string): DocsLibrary | undefined // .title is the display name +``` + +### File structure + +| File | Responsibility | Task | +| --- | --- | --- | +| `apps/website/src/lib/docs-search-tokens.ts` | `searchTokens` + `SEARCH_STOP_WORDS`, shared by client and route | 1 | +| `apps/website/src/lib/docs-search-types.ts` | The wire types (`DocSection`, `DocsSearchHit`). **Imports nothing.** | 2 | +| `apps/website/src/lib/docs-search-index.ts` | Pure: MDX source → `DocSection[]`. No I/O. | 2 | +| `apps/website/src/lib/docs-search-query.ts` | Pure: `DocSection[]` + query → ranked `DocsSearchHit[]` with snippets | 3 | +| `apps/website/src/app/api/docs-search/route.ts` | Builds the index once per instance; HTTP concerns only | 4 | +| `apps/website/next.config.ts` | Traces `content/docs` into the deployed function | 5 | +| `apps/website/src/components/docs/DocsSearch.tsx` | Debounced fetch merged under the instant layer | 6 | +| `apps/website/src/styles/docs.css` | Result rows: heading line, snippet, mark | 6 | +| `apps/website/e2e/docs.spec.ts` | Prose-only term lands on the right anchor | 7 | + +Splitting indexing (Task 2) from querying (Task 3) is deliberate: both are pure and independently testable, and the route becomes thin enough to reason about. + +**Why the types get their own module.** `DocsSearch.tsx` is a client component and needs `DocsSearchHit`. That type would naturally live beside `searchIndexedDocs` — but `docs-search-query.ts` imports `docs-search-index.ts`, which imports `lib/docs.ts`, which imports `fs`. A type-only import is erased at build time so it would work; the moment someone drops the `type` keyword, Node built-ins get pulled into the client bundle. Putting the shared types in a dependency-free module removes the trap instead of documenting it. + +**One refinement from the spec.** The spec sketched `DocSection` carrying `library`, `section`, `slug` and `title` on every record. Those are per-page, not per-section, so repeating them on each of a page's records is duplication. The plan keeps `DocSection` to `{heading, anchor, text}` and lifts the page-level fields into `IndexedDoc`, which owns a `sections` array. Same information, no repetition. + +--- + +### Task 1: Share the tokenizer between client and server + +`searchTokens` and `SEARCH_STOP_WORDS` live inside `DocsSearch.tsx`, a client component. The route needs identical tokenisation. Two copies would drift the first time someone adds a stop word, and the symptom — a query behaving differently in the instant layer than in server results — would be baffling. + +**Files:** +- Create: `apps/website/src/lib/docs-search-tokens.ts` +- Create: `apps/website/src/lib/docs-search-tokens.spec.ts` +- Modify: `apps/website/src/components/docs/DocsSearch.tsx` + +- [ ] **Step 1: Write the failing test** + +Create `apps/website/src/lib/docs-search-tokens.spec.ts`: + +```ts +import { describe, expect, it } from 'vitest'; +import { searchTokens } from './docs-search-tokens'; + +describe('searchTokens', () => { + it('lowercases and splits on non-token characters', () => { + expect(searchTokens('Streaming Tool Calls')).toEqual(['streaming', 'tool', 'calls']); + }); + + it('keeps the characters that appear in package and API names', () => { + // @, . and - are token characters so `@threadplane/ag-ui` survives usefully. + expect(searchTokens('@threadplane/ag-ui')).toEqual(['@threadplane', 'ag-ui']); + }); + + it('drops stop words so "the agent" searches for "agent"', () => { + expect(searchTokens('the agent')).toEqual(['agent']); + }); + + it('returns nothing for a query that is only stop words', () => { + expect(searchTokens('of the')).toEqual([]); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +```bash +cd apps/website && GROWTH_FORM_POLICY=growth_v1 npx vitest run --config vite.config.mts src/lib/docs-search-tokens.spec.ts +``` + +Expected: FAIL — cannot resolve `./docs-search-tokens`. + +- [ ] **Step 3: Create the shared module** + +Create `apps/website/src/lib/docs-search-tokens.ts` by moving the two declarations out of `DocsSearch.tsx` verbatim — do not change their behavior in this task: + +```ts +/** + * Query tokenisation, shared by the client-side instant matcher and the + * server search route. + * + * It lives here rather than in the component because both sides must agree on + * what a query means. Two copies would drift the first time a stop word is + * added, and the symptom — the same query behaving differently in the instant + * results than in the server results — is very hard to read. + */ +export const SEARCH_STOP_WORDS = new Set([ + 'a', 'an', 'and', 'for', 'in', 'of', 'on', 'or', 'the', 'to', 'with', +]); + +export function searchTokens(value: string): string[] { + return value + .toLowerCase() + .split(/[^a-z0-9@.-]+/) + .filter((token) => token.length > 0 && !SEARCH_STOP_WORDS.has(token)); +} +``` + +- [ ] **Step 4: Import it in the component** + +In `apps/website/src/components/docs/DocsSearch.tsx`, delete the local `SEARCH_STOP_WORDS` and `searchTokens` declarations and add to the imports: + +```ts +import { searchTokens } from '../../lib/docs-search-tokens'; +``` + +Leave `searchableText` and `matchesQuery` exactly where they are — they are client-only concerns and stay in the component. + +- [ ] **Step 5: Run the tests to verify they pass** + +```bash +cd apps/website && GROWTH_FORM_POLICY=growth_v1 npx vitest run --config vite.config.mts src/lib/docs-search-tokens.spec.ts src/components/docs +``` + +Expected: PASS. This is a pure move, so every existing `DocsSearch` test must still pass untouched. If any fails, the move changed behavior — fix the move, do not adjust the test. + +- [ ] **Step 6: Commit** + +```bash +git add apps/website/src/lib/docs-search-tokens.ts apps/website/src/lib/docs-search-tokens.spec.ts apps/website/src/components/docs/DocsSearch.tsx +git commit -m "refactor(docs): share the search tokenizer with the server + +The route needs identical tokenisation, and two copies would drift the +first time a stop word is added. + +Co-Authored-By: Claude Opus 5 " +``` + +--- + +### Task 2: The pure indexer + +Turns MDX source into per-heading section records. No I/O, no framework imports. + +**Files:** +- Create: `apps/website/src/lib/docs-search-types.ts` +- Create: `apps/website/src/lib/docs-search-index.ts` +- Create: `apps/website/src/lib/docs-search-index.spec.ts` + +- [ ] **Step 1: Write the failing test** + +Create `apps/website/src/lib/docs-search-index.spec.ts`: + +```ts +import { describe, expect, it } from 'vitest'; +import { indexDocSections } from './docs-search-index'; +import { extractHeadings } from './extract-headings'; +import { getAllDocSlugs, getDocBySlug } from './docs'; + +const SOURCE = `--- +description: Frontmatter must not be indexed. +--- + +# Streaming + +Intro prose above the first heading. + +## Token deltas + +The adapter merges \`TEXT_MESSAGE_CONTENT\` deltas. + +\`\`\`ts +const secretCodeToken = 'should-not-be-indexed'; +\`\`\` + + +Callout body prose is searchable. + + + +`; + +describe('indexDocSections', () => { + const sections = indexDocSections(SOURCE); + const preamble = sections.find((s) => s.heading === null); + const deltas = sections.find((s) => s.heading === 'Token deltas'); + + it('emits an anchor-less record for content above the first heading', () => { + expect(preamble?.anchor).toBeNull(); + expect(preamble?.text).toContain('Intro prose above the first heading'); + }); + + it('splits at each heading and anchors it', () => { + expect(deltas?.anchor).toBe('token-deltas'); + expect(deltas?.text).toContain('The adapter merges'); + }); + + it('never indexes frontmatter', () => { + expect(sections.map((s) => s.text).join(' ')).not.toContain('must not be indexed'); + }); + + it('drops fenced code blocks', () => { + expect(sections.map((s) => s.text).join(' ')).not.toContain('secretCodeToken'); + }); + + it('unwraps inline code so API names are searchable as words', () => { + expect(deltas?.text).toContain('TEXT_MESSAGE_CONTENT'); + expect(deltas?.text).not.toContain('`'); + }); + + it('keeps component body prose and caption attributes, not markup', () => { + const all = sections.map((s) => s.text).join(' '); + expect(all).toContain('Callout body prose is searchable'); + expect(all).toContain('Backend speaks AG-UI over SSE'); + expect(all).not.toContain('StackDiagram'); + expect(all).not.toContain('data-tone'); + }); +}); + +describe('anchor parity with the rendered table of contents', () => { + // The load-bearing test. extract-headings hand-rolls slugification, so the + // only guarantee that a search deep link resolves is that the index and the + // TOC derive anchors from the same function over the same source. Running it + // across real content means a heading either tool sections differently fails + // here rather than shipping a link to nowhere. + it('emits only anchors the TOC also produces, for every real doc', () => { + const mismatches: string[] = []; + + for (const { library, section, slug } of getAllDocSlugs()) { + const doc = getDocBySlug(library, section, slug); + if (!doc) continue; + const tocIds = new Set(extractHeadings(doc.body).map((h) => h.id)); + for (const record of indexDocSections(doc.body)) { + if (record.anchor && !tocIds.has(record.anchor)) { + mismatches.push(`${library}/${section}/${slug}#${record.anchor}`); + } + } + } + + expect(mismatches).toEqual([]); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +```bash +cd apps/website && GROWTH_FORM_POLICY=growth_v1 npx vitest run --config vite.config.mts src/lib/docs-search-index.spec.ts +``` + +Expected: FAIL — cannot resolve `./docs-search-index`. + +- [ ] **Step 3: Create the dependency-free types module** + +Create `apps/website/src/lib/docs-search-types.ts`: + +```ts +/** + * Wire types shared by the search route and the client dialog. + * + * This module imports nothing on purpose. `DocsSearchHit` would naturally sit + * beside `searchIndexedDocs`, but that file's dependency chain reaches + * `lib/docs.ts` and therefore `fs`. A type-only import is erased at build + * time, so it would work — right up until someone drops the `type` keyword + * and pulls Node built-ins into the client bundle. Keeping the types here + * removes that trap rather than commenting on it. + */ + +export interface DocSection { + /** Heading text, or null for content above the first heading. */ + heading: string | null; + /** Heading id for a deep link, or null for the page preamble. */ + anchor: string | null; + /** Searchable prose: fenced code removed, inline code unwrapped. */ + text: string; +} + +export interface DocsSearchHit { + href: string; + title: string; + heading: string | null; + libraryTitle: string; + snippet: string; + /** [start, end) offsets into `snippet`. The client renders the marks. */ + marks: [number, number][]; +} +``` + +- [ ] **Step 4: Write the indexer** + +Create `apps/website/src/lib/docs-search-index.ts`: + +```ts +import { stripFrontmatter } from './docs'; +import { extractHeadings } from './extract-headings'; +import type { DocSection } from './docs-search-types'; + +export type { DocSection }; + +/** + * Fenced code is dropped deliberately: it is roughly half the corpus and + * mostly noise for ranking. The cost is explicit — an error string that only + * ever appears inside a code sample will not match. Inline code is kept, + * because that is where API names appear in sentences. + */ +function toSearchableText(source: string): string { + return source + .replace(/```[\s\S]*?```/g, ' ') + // Self-closing components: keep prose-bearing attributes, drop the rest. + .replace(/<[A-Z][\w.]*\s[^>]*?\/>/g, (tag) => { + const prose = [...tag.matchAll(/\b(?:caption|title)="([^"]*)"/g)].map((m) => m[1]); + return ` ${prose.join(' ')} `; + }) + // Paired component tags: drop the tags, keep the children between them. + .replace(/<\/?[A-Z][\w.]*(?:\s[^>]*)?>/g, ' ') + .replace(/`([^`]+)`/g, '$1') + .replace(/\s+/g, ' ') + .trim(); +} + +/** + * Split a doc body into one record per heading, plus one for the preamble. + * + * Anchors come from `extractHeadings` rather than a second slugifier. That + * file hand-rolls GitHub-style slugification, and having one approximation is + * a known quantity while two that drift apart is a bug generator. + */ +export function indexDocSections(source: string): DocSection[] { + const body = stripFrontmatter(source); + const headings = extractHeadings(body); + const lines = body.split('\n'); + + const sections: DocSection[] = []; + let current: { heading: string | null; anchor: string | null; lines: string[] } = { + heading: null, + anchor: null, + lines: [], + }; + let headingIndex = 0; + let inCodeBlock = false; + + const flush = () => { + const text = toSearchableText(current.lines.join('\n')); + if (text.length > 0) { + sections.push({ heading: current.heading, anchor: current.anchor, text }); + } + }; + + for (const line of lines) { + // Track fences so a `## ` inside a code sample never starts a section. + if (line.trim().startsWith('```')) inCodeBlock = !inCodeBlock; + + const match = inCodeBlock ? null : line.match(/^#{2,3}\s+(.+)$/); + if (match) { + flush(); + const heading = headings[headingIndex]; + headingIndex += 1; + current = { + heading: heading?.text ?? match[1].replace(/`/g, ''), + anchor: heading?.id ?? null, + lines: [], + }; + continue; + } + current.lines.push(line); + } + flush(); + + return sections; +} +``` + +Note the heading walk consumes `extractHeadings` output positionally. Both functions skip fenced blocks and match the same `^#{2,3}\s+` shape, which is what keeps them aligned — and the anchor-parity test in Step 1 is what proves it over real content rather than by assertion. + +- [ ] **Step 5: Run the test to verify it passes** + +```bash +cd apps/website && GROWTH_FORM_POLICY=growth_v1 npx vitest run --config vite.config.mts src/lib/docs-search-index.spec.ts +``` + +Expected: PASS, including the anchor-parity test across all 122 docs. + +If anchor parity fails, do NOT relax the assertion — it is the only thing standing between this feature and deep links that 404 to nowhere. Report which files mismatch and why. + +- [ ] **Step 6: Commit** + +```bash +git add apps/website/src/lib/docs-search-types.ts apps/website/src/lib/docs-search-index.ts apps/website/src/lib/docs-search-index.spec.ts +git commit -m "feat(docs): index doc bodies into per-heading sections + +Pure MDX-to-sections indexer: frontmatter and fenced code stripped, +inline code unwrapped, component prose kept. Anchors come from +extract-headings so search deep links cannot drift from the TOC, which +an anchor-parity test pins across all real content. + +Co-Authored-By: Claude Opus 5 " +``` + +--- + +### Task 3: Ranking and snippets + +**Files:** +- Create: `apps/website/src/lib/docs-search-query.ts` +- Create: `apps/website/src/lib/docs-search-query.spec.ts` + +- [ ] **Step 1: Write the failing test** + +Create `apps/website/src/lib/docs-search-query.spec.ts`: + +```ts +import { describe, expect, it } from 'vitest'; +import { searchIndexedDocs, type IndexedDoc } from './docs-search-query'; + +const DOCS: IndexedDoc[] = [ + { + library: 'langgraph', + libraryTitle: 'LangGraph', + section: 'guides', + slug: 'persistence', + title: 'Persistence', + sections: [ + { heading: null, anchor: null, text: 'How threads survive a restart.' }, + { + heading: 'Production checkpointers', + anchor: 'production-checkpointers', + text: 'Use a Postgres checkpointer in production rather than memory.', + }, + ], + }, + { + library: 'chat', + libraryTitle: 'Chat', + section: 'guides', + slug: 'checkpointer', + title: 'Checkpointer', + sections: [{ heading: null, anchor: null, text: 'Unrelated prose.' }], + }, +]; + +describe('searchIndexedDocs', () => { + it('finds a term that appears only in body prose', () => { + const hits = searchIndexedDocs(DOCS, 'postgres'); + expect(hits).toHaveLength(1); + expect(hits[0].href).toBe('/docs/langgraph/guides/persistence#production-checkpointers'); + expect(hits[0].heading).toBe('Production checkpointers'); + }); + + it('ranks a title match above a body match', () => { + const hits = searchIndexedDocs(DOCS, 'checkpointer'); + expect(hits[0].title).toBe('Checkpointer'); + }); + + it('links to the page top when the match is in the preamble', () => { + const hits = searchIndexedDocs(DOCS, 'restart'); + expect(hits[0].href).toBe('/docs/langgraph/guides/persistence'); + }); + + it('requires every token, matching the instant layer', () => { + expect(searchIndexedDocs(DOCS, 'postgres nonexistent')).toEqual([]); + }); + + it('returns nothing for a query of only stop words', () => { + expect(searchIndexedDocs(DOCS, 'of the')).toEqual([]); + }); + + it('returns a snippet with offsets covering the matched term', () => { + const [hit] = searchIndexedDocs(DOCS, 'postgres'); + expect(hit.snippet).toContain('Postgres'); + expect(hit.marks.length).toBeGreaterThan(0); + const [start, end] = hit.marks[0]; + expect(hit.snippet.slice(start, end).toLowerCase()).toBe('postgres'); + }); + + it('caps results at eight, matching the existing result list', () => { + const many: IndexedDoc[] = Array.from({ length: 12 }, (_, i) => ({ + library: 'chat', + libraryTitle: 'Chat', + section: 'guides', + slug: `page-${i}`, + title: `Page ${i}`, + sections: [{ heading: null, anchor: null, text: 'streaming prose' }], + })); + expect(searchIndexedDocs(many, 'streaming')).toHaveLength(8); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +```bash +cd apps/website && GROWTH_FORM_POLICY=growth_v1 npx vitest run --config vite.config.mts src/lib/docs-search-query.spec.ts +``` + +Expected: FAIL — cannot resolve `./docs-search-query`. + +- [ ] **Step 3: Write the query module** + +Create `apps/website/src/lib/docs-search-query.ts`: + +```ts +import type { DocSection, DocsSearchHit } from './docs-search-types'; +import { searchTokens } from './docs-search-tokens'; + +export type { DocsSearchHit }; + +export interface IndexedDoc { + library: string; + libraryTitle: string; + section: string; + slug: string; + title: string; + sections: DocSection[]; +} + +const MAX_RESULTS = 8; +const SNIPPET_RADIUS = 80; + +/** Title matches beat heading matches, which beat body prose. */ +const TITLE_WEIGHT = 3; +const HEADING_WEIGHT = 2; +const TEXT_WEIGHT = 1; + +function countWeighted(haystack: string, token: string, weight: number): number { + return haystack.toLowerCase().includes(token) ? weight : 0; +} + +/** + * A window of `text` around the first token match. + * + * Offsets are returned rather than HTML so the client wraps the ranges + * itself — nothing server-built is ever rendered into the page. + */ +function buildSnippet(text: string, tokens: string[]): { snippet: string; marks: [number, number][] } { + const lower = text.toLowerCase(); + const first = tokens + .map((token) => lower.indexOf(token)) + .filter((index) => index >= 0) + .sort((a, b) => a - b)[0] ?? 0; + + const start = Math.max(0, first - SNIPPET_RADIUS); + const end = Math.min(text.length, first + SNIPPET_RADIUS); + const prefix = start > 0 ? '…' : ''; + const suffix = end < text.length ? '…' : ''; + const snippet = `${prefix}${text.slice(start, end)}${suffix}`; + + const snippetLower = snippet.toLowerCase(); + const marks: [number, number][] = []; + for (const token of tokens) { + let at = snippetLower.indexOf(token); + while (at >= 0) { + marks.push([at, at + token.length]); + at = snippetLower.indexOf(token, at + token.length); + } + } + marks.sort((a, b) => a[0] - b[0]); + return { snippet, marks }; +} + +export function searchIndexedDocs(docs: IndexedDoc[], query: string): DocsSearchHit[] { + const tokens = searchTokens(query); + if (tokens.length === 0) return []; + + const scored: { score: number; length: number; hit: DocsSearchHit }[] = []; + + for (const doc of docs) { + for (const section of doc.sections) { + const haystack = `${doc.title} ${section.heading ?? ''} ${section.text}`.toLowerCase(); + // AND semantics, matching the instant client matcher. + if (!tokens.every((token) => haystack.includes(token))) continue; + + const score = tokens.reduce( + (total, token) => + total + + countWeighted(doc.title, token, TITLE_WEIGHT) + + countWeighted(section.heading ?? '', token, HEADING_WEIGHT) + + countWeighted(section.text, token, TEXT_WEIGHT), + 0 + ); + + const { snippet, marks } = buildSnippet(section.text, tokens); + scored.push({ + score, + length: section.text.length, + hit: { + href: `/docs/${doc.library}/${doc.section}/${doc.slug}${section.anchor ? `#${section.anchor}` : ''}`, + title: doc.title, + heading: section.heading, + libraryTitle: doc.libraryTitle, + snippet, + marks, + }, + }); + } + } + + // Higher score first; ties go to the shorter section, which favours a + // precise heading over a long prose blob. + scored.sort((a, b) => b.score - a.score || a.length - b.length); + return scored.slice(0, MAX_RESULTS).map((entry) => entry.hit); +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +```bash +cd apps/website && GROWTH_FORM_POLICY=growth_v1 npx vitest run --config vite.config.mts src/lib/docs-search-query.spec.ts +``` + +Expected: PASS, 7 tests. + +- [ ] **Step 5: Commit** + +```bash +git add apps/website/src/lib/docs-search-query.ts apps/website/src/lib/docs-search-query.spec.ts +git commit -m "feat(docs): rank indexed doc sections and build snippets + +Weighted AND matching over title, heading and prose, capped at eight to +match the existing result list. Snippets return offsets rather than +HTML so the client renders the marks itself. + +Co-Authored-By: Claude Opus 5 " +``` + +--- + +### Task 4: The search route + +Thin: build the index once per instance, delegate to the query module, set headers. + +**Files:** +- Create: `apps/website/src/app/api/docs-search/route.ts` +- Create: `apps/website/src/app/api/docs-search/route.spec.ts` + +- [ ] **Step 1: Write the failing test** + +Create `apps/website/src/app/api/docs-search/route.spec.ts`: + +```ts +// @vitest-environment node +import { describe, expect, it } from 'vitest'; +import { GET } from './route'; + +const call = (q: string) => + GET(new Request(`http://localhost/api/docs-search?q=${encodeURIComponent(q)}`)); + +describe('GET /api/docs-search', () => { + it('finds a page by a term that appears only in its body prose', async () => { + // "checkpointer" is prose in the LangGraph persistence guide, and is in no + // page title — exactly the query the old title-only search could not serve. + const res = await call('checkpointer'); + expect(res.status).toBe(200); + const { results } = await res.json(); + expect(results.length).toBeGreaterThan(0); + expect(results.some((r: { href: string }) => r.href.includes('/docs/langgraph/'))).toBe(true); + }); + + it('returns hits that carry a snippet and marks', async () => { + const { results } = await (await call('checkpointer')).json(); + expect(typeof results[0].snippet).toBe('string'); + expect(Array.isArray(results[0].marks)).toBe(true); + }); + + it('returns empty without scanning for a query under two characters', async () => { + const { results } = await (await call('a')).json(); + expect(results).toEqual([]); + }); + + it('returns empty for a missing query parameter', async () => { + const res = await GET(new Request('http://localhost/api/docs-search')); + const { results } = await res.json(); + expect(results).toEqual([]); + }); + + it('is cacheable, because the corpus only changes on deploy', async () => { + const res = await call('streaming'); + expect(res.headers.get('cache-control')).toContain('max-age='); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +```bash +cd apps/website && GROWTH_FORM_POLICY=growth_v1 npx vitest run --config vite.config.mts src/app/api/docs-search/route.spec.ts +``` + +Expected: FAIL — cannot resolve `./route`. + +- [ ] **Step 3: Write the route** + +Create `apps/website/src/app/api/docs-search/route.ts`: + +```ts +import { NextResponse } from 'next/server'; +import { getAllDocSlugs, getDocBySlug } from '../../../lib/docs'; +import { getLibraryConfig } from '../../../lib/docs-config'; +import { indexDocSections } from '../../../lib/docs-search-index'; +import { searchIndexedDocs, type IndexedDoc } from '../../../lib/docs-search-query'; + +const MIN_QUERY_LENGTH = 2; + +/** + * Built once per instance, not per request. + * + * This reads MDX from disk at request time, which is why + * `content/docs/**` has to be traced into the deployed function — see + * `outputFileTracingIncludes` in next.config.ts. The route cannot be + * statically generated the way `api/markdown` is, because the query space is + * unbounded. + */ +let index: IndexedDoc[] | null = null; + +function getIndex(): IndexedDoc[] { + if (index) return index; + + index = getAllDocSlugs().flatMap(({ library, section, slug }) => { + const doc = getDocBySlug(library, section, slug); + if (!doc) return []; + return [ + { + library, + libraryTitle: getLibraryConfig(library)?.title ?? library, + section, + slug, + title: doc.title, + sections: indexDocSections(doc.body), + }, + ]; + }); + + return index; +} + +export async function GET(request: Request): Promise { + const query = new URL(request.url).searchParams.get('q')?.trim() ?? ''; + + if (query.length < MIN_QUERY_LENGTH) { + return NextResponse.json({ results: [] }); + } + + return NextResponse.json( + { results: searchIndexedDocs(getIndex(), query) }, + { + headers: { + // The corpus only changes on deploy, so repeated queries are served + // by the CDN rather than waking this function. + 'Cache-Control': 'public, max-age=300', + }, + } + ); +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +```bash +cd apps/website && GROWTH_FORM_POLICY=growth_v1 npx vitest run --config vite.config.mts src/app/api/docs-search/route.spec.ts +``` + +Expected: PASS, 5 tests. + +If the "checkpointer" test finds nothing, do NOT change the search term to something easier. Confirm the word really is in that guide's prose (`grep -rn "checkpointer" apps/website/content/docs/langgraph/guides/persistence.mdx`) and fix the pipeline. A test tuned until it passes proves nothing. + +- [ ] **Step 5: Commit** + +```bash +git add apps/website/src/app/api/docs-search/route.ts apps/website/src/app/api/docs-search/route.spec.ts +git commit -m "feat(docs): add the docs content search route + +Builds the section index once per instance and answers queries from it. +Short queries return empty without scanning, and responses are +cacheable because the corpus only changes on deploy. + +Co-Authored-By: Claude Opus 5 " +``` + +--- + +### Task 5: Trace the content into the deployed function + +**This is the task that fails in production if skipped, and passes every local test if it is.** `next dev` reads from the working tree, so nothing before this point exercises file tracing. + +`apps/website/next.config.ts` traces `cockpit/**` md/py/ts, a Mastra mjs, and `nx.json` — nothing under `apps/website/content`. `api/markdown` reads MDX and deploys fine only because `generateStaticParams()` makes it build-time output. The search route reads at request time. + +**Files:** +- Modify: `apps/website/next.config.ts` + +- [ ] **Step 1: Add the include** + +In `outputFileTracingIncludes`, extend the `'/*'` array with the docs content. Existing entries reach the repo root with `../../`; the content lives inside the app, so it needs no prefix: + +```ts + outputFileTracingIncludes: { + '/*': [ + '../../cockpit/**/*.md', + '../../cockpit/**/*.py', + '../../cockpit/**/*.ts', + '../../deployments/ag-ui-mastra/*.mjs', + '../../nx.json', + // The docs search route reads these at request time. Unlike + // api/markdown it cannot be statically generated, so without this the + // route deploys with no corpus and returns empty for every query — + // silently, and only in production. + 'content/docs/**/*.mdx', + ], + }, +``` + +- [ ] **Step 2: Build** + +```bash +cd /Users/blove/repos/angular-agent-framework/.claude/worktrees/gallant-clarke-963ed0 && GROWTH_FORM_POLICY=growth_v1 NX_DAEMON=false npx nx build website --outputStyle=static 2>&1 | tail -20 +``` + +Expected: success, with `/api/docs-search` listed as a dynamic (`ƒ`) route rather than static. + +- [ ] **Step 3: Verify the trace actually captured the content** + +A successful build does not prove the files were traced. Inspect the trace output: + +```bash +cd /Users/blove/repos/angular-agent-framework/.claude/worktrees/gallant-clarke-963ed0 && find apps/website/.next -name '*.nft.json' -path '*docs-search*' -exec sh -c 'echo "== $1"; grep -o "content/docs/[^\"]*\.mdx" "$1" | head -3; grep -c "content/docs/" "$1"' _ {} \; +``` + +Expected: a non-zero count and sample `.mdx` paths. If the count is 0, the glob is wrong — try `./content/docs/**/*.mdx` or an absolute-from-tracing-root form, rebuild, and re-check. **Do not proceed on a successful build alone.** + +- [ ] **Step 4: Prove the route works in a production server** + +```bash +cd /Users/blove/repos/angular-agent-framework/.claude/worktrees/gallant-clarke-963ed0 && lsof -ti:3100 | xargs -r kill -9 +GROWTH_FORM_POLICY=growth_v1 npx next start apps/website --port 3100 & +until curl -sf "http://localhost:3100/api/docs-search?q=checkpointer" >/dev/null 2>&1; do sleep 1; done +curl -s "http://localhost:3100/api/docs-search?q=checkpointer" | head -c 400 +``` + +Expected: JSON with a non-empty `results` array. An empty array here — while the unit test passes — is exactly the tracing failure this task exists to prevent. + +Kill the server afterwards: `lsof -ti:3100 | xargs -r kill -9`. + +- [ ] **Step 5: Commit** + +```bash +git add apps/website/next.config.ts +git commit -m "build(website): trace docs content for the search route + +The route reads MDX at request time and cannot be statically generated +the way api/markdown is, so without this it deploys with no corpus and +returns empty for every query -- silently, and only in production. + +Co-Authored-By: Claude Opus 5 " +``` + +--- + +### Task 6: Merge server hits into the search dialog + +The existing client matcher keeps rendering instantly. Server hits arrive underneath. + +**Files:** +- Modify: `apps/website/src/components/docs/DocsSearch.tsx` +- Modify: `apps/website/src/styles/docs.css` +- Modify: `apps/website/src/components/docs/DocsSearch.spec.tsx` (create if absent) + +- [ ] **Step 1: Write the failing test** + +Create or extend `apps/website/src/components/docs/DocsSearch.spec.tsx`: + +```tsx +// @vitest-environment jsdom +import React from 'react'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/react'; +import { DocsSearch } from './DocsSearch'; + +vi.mock('next/navigation', () => ({ useRouter: () => ({ push: vi.fn() }) })); +vi.mock('../../lib/analytics/client', () => ({ track: vi.fn() })); + +const HIT = { + href: '/docs/langgraph/guides/persistence#production-checkpointers', + title: 'Persistence', + heading: 'Production checkpointers', + libraryTitle: 'LangGraph', + snippet: 'Use a Postgres checkpointer in production.', + marks: [[8, 16]] as [number, number][], +}; + +function openSearch() { + render(); + fireEvent.keyDown(document, { key: 'k', metaKey: true }); +} + +beforeEach(() => vi.useFakeTimers({ shouldAdvanceTime: true })); +afterEach(() => { + vi.useRealTimers(); + vi.unstubAllGlobals(); +}); + +describe('DocsSearch content results', () => { + it('renders server hits with their heading and a highlighted snippet', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue({ ok: true, json: async () => ({ results: [HIT] }) }) + ); + openSearch(); + fireEvent.change(screen.getByRole('combobox'), { target: { value: 'checkpointer' } }); + + await waitFor(() => expect(screen.getByText('Production checkpointers')).toBeTruthy()); + // The mark is rendered from offsets, never from server HTML. + expect(screen.getByText('Postgres').tagName).toBe('MARK'); + }); + + it('still shows instant title results when the request fails', async () => { + vi.stubGlobal('fetch', vi.fn().mockRejectedValue(new Error('offline'))); + openSearch(); + // "quickstart" matches page titles in the client-side index. + fireEvent.change(screen.getByRole('combobox'), { target: { value: 'quickstart' } }); + + await waitFor(() => expect(screen.getAllByRole('option').length).toBeGreaterThan(0)); + // A failed search must never surface an error state in the dialog. + expect(screen.queryByText(/error/i)).toBeNull(); + }); + + it('does not request for a query under two characters', async () => { + const fetchMock = vi.fn().mockResolvedValue({ ok: true, json: async () => ({ results: [] }) }); + vi.stubGlobal('fetch', fetchMock); + openSearch(); + fireEvent.change(screen.getByRole('combobox'), { target: { value: 'a' } }); + + await vi.advanceTimersByTimeAsync(400); + expect(fetchMock).not.toHaveBeenCalled(); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +```bash +cd apps/website && GROWTH_FORM_POLICY=growth_v1 npx vitest run --config vite.config.mts src/components/docs/DocsSearch.spec.tsx +``` + +Expected: FAIL — no server results are rendered. + +- [ ] **Step 3: Add the fetch and merge** + +In `apps/website/src/components/docs/DocsSearch.tsx`, add the imports and state: + +```tsx +import type { DocsSearchHit } from '../../lib/docs-search-types'; +``` + +Inside the component, after the existing `results` computation: + +```tsx + const [contentHits, setContentHits] = useState([]); + + // The instant client matcher above renders as you type. These arrive after + // a round trip and merge in below it, so the fast path stays fast and a + // slow, failed or offline request degrades to exactly today's behaviour. + useEffect(() => { + const trimmed = query.trim(); + if (trimmed.length < 2) { + setContentHits([]); + return undefined; + } + + const controller = new AbortController(); + const timer = window.setTimeout(() => { + fetch(`/api/docs-search?q=${encodeURIComponent(trimmed)}`, { + signal: controller.signal, + }) + .then((response) => (response.ok ? response.json() : { results: [] })) + .then((payload: { results?: DocsSearchHit[] }) => { + setContentHits(payload.results ?? []); + }) + .catch(() => { + // An aborted request means a newer one is already in flight, so + // clearing here would blank results the new request is about to + // replace. Only a genuine failure falls back to the instant layer, + // and it does so silently — search never shows an error state. + if (!controller.signal.aborted) setContentHits([]); + }); + }, 150); + + return () => { + controller.abort(); + window.clearTimeout(timer); + }; + }, [query]); +``` + +Add the de-duplication and rendering. Place this immediately after the existing `results.map(...)` block, inside the listbox: + +```tsx + {(() => { + const titleHrefs = new Set(results.map((page) => page.href)); + // A deeper link wins over a page-level one the instant layer + // already listed; an identical page-level hit is dropped. + const merged = contentHits.filter( + (hit) => hit.href.includes('#') || !titleHrefs.has(hit.href) + ); + if (merged.length === 0) return null; + return ( + <> +
+ In page content +
+ {merged.map((hit, i) => ( + + ))} + + ); + })()} +``` + +Add this helper at module scope in the same file: + +```tsx +/** + * Wrap the matched ranges from server-supplied offsets. + * + * The server sends text plus offsets rather than HTML, so nothing it produces + * is ever rendered as markup. + */ +function renderSnippet(snippet: string, marks: [number, number][]) { + const parts: React.ReactNode[] = []; + let cursor = 0; + for (const [start, end] of marks) { + if (start < cursor) continue; + if (start > cursor) parts.push(snippet.slice(cursor, start)); + parts.push({snippet.slice(start, end)}); + cursor = end; + } + parts.push(snippet.slice(cursor)); + return parts; +} +``` + +Also change the empty state so it only shows when BOTH lists are empty: + +```tsx + {results.length === 0 && contentHits.length === 0 && ( +
+ No results found +
+ )} +``` + +- [ ] **Step 4: Style the new rows** + +Append to `apps/website/src/styles/docs.css`, after the existing `.docs-search-result` rules (find them by name): + +```css +/* Content hits: a section heading, its page, and why it matched. */ +.docs-search-group-label { + font-family: var(--font-inter); + font-size: 11px; + font-weight: 600; + letter-spacing: 0.04em; + text-transform: uppercase; + color: var(--color-text-muted); + padding: 10px 14px 4px; +} +.docs-search-result-snippet { + display: block; + font-family: var(--font-inter); + font-size: 12px; + line-height: 1.5; + color: var(--color-text-secondary); + margin-top: 2px; + overflow: hidden; + display: -webkit-box; + -webkit-line-clamp: 2; + -webkit-box-orient: vertical; +} +.docs-search-result-snippet mark { + background: var(--color-accent-surface); + color: var(--color-accent); + border-radius: 3px; + padding: 0 2px; +} +``` + +- [ ] **Step 5: Run the tests to verify they pass** + +```bash +cd apps/website && GROWTH_FORM_POLICY=growth_v1 npx vitest run --config vite.config.mts src/components/docs +``` + +Expected: PASS, including every pre-existing `DocsSearch` test. + +- [ ] **Step 6: Commit** + +```bash +git add apps/website/src/components/docs/DocsSearch.tsx apps/website/src/components/docs/DocsSearch.spec.tsx apps/website/src/styles/docs.css +git commit -m "feat(docs): show page-content hits in docs search + +Debounced, abortable requests merge server hits beneath the instant +title matches, each showing its section heading and a snippet with the +match highlighted. A failed request falls back to the instant results +rather than surfacing an error. + +Co-Authored-By: Claude Opus 5 " +``` + +--- + +### Task 7: End-to-end proof + +**Files:** +- Modify: `apps/website/e2e/docs.spec.ts` + +- [ ] **Step 1: Write the test** + +Append inside the `Docs slug page` describe in `apps/website/e2e/docs.spec.ts`: + +```ts + test('finds a term that appears only in body prose and lands on its section', async ({ page }) => { + await page.goto(route); + await page.keyboard.press('Meta+k'); + + const dialog = page.getByRole('dialog', { name: 'Search documentation' }); + await expect(dialog).toBeVisible(); + + // "checkpointer" is prose inside the persistence guide and is in no page + // title, so a title-only search returns nothing for it. + await dialog.getByRole('combobox').fill('checkpointer'); + + const hit = dialog.getByRole('option').filter({ hasText: /checkpointer/i }).first(); + await expect(hit).toBeVisible({ timeout: 10000 }); + await hit.click(); + + // The deep link must land on a section, not the page top. + await expect(page).toHaveURL(/\/docs\/langgraph\/.*#.+/); + }); +``` + +- [ ] **Step 2: Run it** + +Free the port first — a stale dev server will either fight Playwright's web server or silently serve an old bundle: + +```bash +lsof -ti:3000 | xargs -r kill -9; sleep 1 +cd /Users/blove/repos/angular-agent-framework/.claude/worktrees/gallant-clarke-963ed0 && GROWTH_FORM_POLICY=growth_v1 NX_DAEMON=false npx nx e2e website --outputStyle=static --grep "docs" 2>&1 | tail -25 +``` + +Expected: all pass. + +If the new test fails on timing, do NOT extend the timeout past 10s to force it green — that hides a real latency problem in the route. Investigate why the response is slow. + +- [ ] **Step 3: Commit** + +```bash +git add apps/website/e2e/docs.spec.ts +git commit -m "test(website): prove prose-only search reaches its section + +Searches a term that exists only in body text and asserts the result +lands on a section anchor rather than the page top. + +Co-Authored-By: Claude Opus 5 " +``` + +--- + +### Task 8: Full verification + +- [ ] **Step 1: Full website suite** + +```bash +cd /Users/blove/repos/angular-agent-framework/.claude/worktrees/gallant-clarke-963ed0 && GROWTH_FORM_POLICY=growth_v1 NX_DAEMON=false npx nx test website --outputStyle=static 2>&1 | tail -30 +``` + +Expected: PASS. + +- [ ] **Step 2: Lint** + +```bash +cd /Users/blove/repos/angular-agent-framework/.claude/worktrees/gallant-clarke-963ed0 && GROWTH_FORM_POLICY=growth_v1 NX_DAEMON=false npx nx lint website --outputStyle=static 2>&1 | sed 's/\x1b\[[0-9;]*m//g' | grep -E "error|problems" | head +``` + +Strip ANSI before grepping or colored output silently defeats the match. Errors must be zero; pre-existing warnings are fine. + +- [ ] **Step 3: Build** + +```bash +cd /Users/blove/repos/angular-agent-framework/.claude/worktrees/gallant-clarke-963ed0 && GROWTH_FORM_POLICY=growth_v1 NX_DAEMON=false npx nx build website --outputStyle=static 2>&1 | tail -20 +``` + +Expected: success. If Turbopack panics about the workspace root, a stale dev directory is the cause: `rm -rf apps/website/.next` and re-run. + +- [ ] **Step 4: Full e2e** + +```bash +lsof -ti:3000 | xargs -r kill -9; sleep 1 +cd /Users/blove/repos/angular-agent-framework/.claude/worktrees/gallant-clarke-963ed0 && GROWTH_FORM_POLICY=growth_v1 NX_DAEMON=false npx nx e2e website --outputStyle=static 2>&1 | tail -25 +``` + +Expected: PASS. + +- [ ] **Step 5: Confirm the diff** + +```bash +cd /Users/blove/repos/angular-agent-framework/.claude/worktrees/gallant-clarke-963ed0 && git status --short && git diff --stat origin/main...HEAD | tail -5 +``` + +`apps/website/.env.local` must NOT appear (it is gitignored). No `.next` or `test-results` artifacts staged. + +--- + +## Notes for the implementer + +- **Task order matters once:** Task 1 must land before Tasks 3 and 4, which import the shared tokenizer. Tasks 2 and 3 are independent of each other. +- **Do not weaken the anchor-parity test in Task 2.** It is the only thing preventing deep links to anchors that do not exist. +- **Task 5 cannot be verified by a passing build alone.** Inspect the trace file and hit a production server, as its steps specify. +- **The Browser pane suspends `requestAnimationFrame` and scroll events while hidden**, so a `computer` scroll can time out. Prefer `read_page`, `get_page_text` and `javascript_tool` for verification. From ef0cd93caba00d4aaae8acba6569db1dd6eec75a Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:09:12 -0700 Subject: [PATCH 03/18] refactor(docs): share the search tokenizer with the server The route needs identical tokenisation, and two copies would drift the first time a stop word is added. Co-Authored-By: Claude Opus 5 --- .../src/components/docs/DocsSearch.tsx | 10 +-------- .../src/lib/docs-search-tokens.spec.ts | 21 +++++++++++++++++++ apps/website/src/lib/docs-search-tokens.ts | 19 +++++++++++++++++ 3 files changed, 41 insertions(+), 9 deletions(-) create mode 100644 apps/website/src/lib/docs-search-tokens.spec.ts create mode 100644 apps/website/src/lib/docs-search-tokens.ts diff --git a/apps/website/src/components/docs/DocsSearch.tsx b/apps/website/src/components/docs/DocsSearch.tsx index 2a34a51a2..9963d2263 100644 --- a/apps/website/src/components/docs/DocsSearch.tsx +++ b/apps/website/src/components/docs/DocsSearch.tsx @@ -4,6 +4,7 @@ import { useRouter } from 'next/navigation'; import { docsConfig, specialDocsPages, type LibraryId } from '../../lib/docs-config'; import { analyticsEvents } from '../../lib/analytics/events'; import { track } from '../../lib/analytics/client'; +import { searchTokens } from '../../lib/docs-search-tokens'; interface SearchablePage { title: string; @@ -15,15 +16,6 @@ interface SearchablePage { libraryTitle: string; } -const SEARCH_STOP_WORDS = new Set(['a', 'an', 'and', 'for', 'in', 'of', 'on', 'or', 'the', 'to', 'with']); - -function searchTokens(value: string): string[] { - return value - .toLowerCase() - .split(/[^a-z0-9@.-]+/) - .filter((token) => token.length > 0 && !SEARCH_STOP_WORDS.has(token)); -} - function searchableText(page: SearchablePage): string { return [page.title, page.description, page.slug, page.section, page.libraryTitle].filter(Boolean).join(' '); } diff --git a/apps/website/src/lib/docs-search-tokens.spec.ts b/apps/website/src/lib/docs-search-tokens.spec.ts new file mode 100644 index 000000000..551ce6bd6 --- /dev/null +++ b/apps/website/src/lib/docs-search-tokens.spec.ts @@ -0,0 +1,21 @@ +import { describe, expect, it } from 'vitest'; +import { searchTokens } from './docs-search-tokens'; + +describe('searchTokens', () => { + it('lowercases and splits on non-token characters', () => { + expect(searchTokens('Streaming Tool Calls')).toEqual(['streaming', 'tool', 'calls']); + }); + + it('keeps the characters that appear in package and API names', () => { + // @, . and - are token characters so `@threadplane/ag-ui` survives usefully. + expect(searchTokens('@threadplane/ag-ui')).toEqual(['@threadplane', 'ag-ui']); + }); + + it('drops stop words so "the agent" searches for "agent"', () => { + expect(searchTokens('the agent')).toEqual(['agent']); + }); + + it('returns nothing for a query that is only stop words', () => { + expect(searchTokens('of the')).toEqual([]); + }); +}); diff --git a/apps/website/src/lib/docs-search-tokens.ts b/apps/website/src/lib/docs-search-tokens.ts new file mode 100644 index 000000000..6e09d082c --- /dev/null +++ b/apps/website/src/lib/docs-search-tokens.ts @@ -0,0 +1,19 @@ +/** + * Query tokenisation, shared by the client-side instant matcher and the + * server search route. + * + * It lives here rather than in the component because both sides must agree on + * what a query means. Two copies would drift the first time a stop word is + * added, and the symptom — the same query behaving differently in the instant + * results than in the server results — is very hard to read. + */ +export const SEARCH_STOP_WORDS = new Set([ + 'a', 'an', 'and', 'for', 'in', 'of', 'on', 'or', 'the', 'to', 'with', +]); + +export function searchTokens(value: string): string[] { + return value + .toLowerCase() + .split(/[^a-z0-9@.-]+/) + .filter((token) => token.length > 0 && !SEARCH_STOP_WORDS.has(token)); +} From 85e040da415135b24a2a64a167bacd42c8013d2c Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:11:48 -0700 Subject: [PATCH 04/18] test(docs): pin the search stop-word list The tokenizer is shared so the client matcher and the server route agree on what a query means. Adding or removing a stop word changes every query on both sides identically, so no behavioural test elsewhere would notice. This is the only thing that catches it. Co-Authored-By: Claude Opus 5 --- apps/website/src/lib/docs-search-tokens.spec.ts | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/apps/website/src/lib/docs-search-tokens.spec.ts b/apps/website/src/lib/docs-search-tokens.spec.ts index 551ce6bd6..aa743333b 100644 --- a/apps/website/src/lib/docs-search-tokens.spec.ts +++ b/apps/website/src/lib/docs-search-tokens.spec.ts @@ -1,5 +1,17 @@ import { describe, expect, it } from 'vitest'; -import { searchTokens } from './docs-search-tokens'; +import { SEARCH_STOP_WORDS, searchTokens } from './docs-search-tokens'; + +describe('SEARCH_STOP_WORDS', () => { + it('pins the exact list, because both sides of search depend on it', () => { + // The tokenizer is shared so the client's instant matcher and the server + // route agree on what a query means. Quietly adding or removing a word + // changes every query on both sides identically, so no behavioural test + // elsewhere would fail — this is the only thing that catches it. + expect([...SEARCH_STOP_WORDS].sort()).toEqual([ + 'a', 'an', 'and', 'for', 'in', 'of', 'on', 'or', 'the', 'to', 'with', + ]); + }); +}); describe('searchTokens', () => { it('lowercases and splits on non-token characters', () => { From 9ea1cf43d5329ef98aca1e319515f9129dfe010b Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:12:23 -0700 Subject: [PATCH 05/18] docs: record why "checkpointer" is the search test fixture Verified against real content: 16 occurrences in the persistence guide, 7 outside fenced code so it survives the indexer's code stripping, and it heads two sections so a deep link has somewhere to land. Critically it is in no page title, which is why today's title-only search cannot find it -- the test proves the feature rather than passing trivially. So a failing checkpointer assertion means the pipeline is broken, not that the term was a bad guess. Do not swap in an easier one. Co-Authored-By: Claude Opus 5 --- .../plans/2026-09-03-docs-search-content-index.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/superpowers/plans/2026-09-03-docs-search-content-index.md b/docs/superpowers/plans/2026-09-03-docs-search-content-index.md index a624c6ba5..4f42bfdcd 100644 --- a/docs/superpowers/plans/2026-09-03-docs-search-content-index.md +++ b/docs/superpowers/plans/2026-09-03-docs-search-content-index.md @@ -22,6 +22,16 @@ Read the spec. Then note these repo facts, which are not guessable: 4. **Route specs run in the node environment.** They start with `// @vitest-environment node`. Component specs use `// @vitest-environment jsdom`. Getting this wrong produces confusing failures. 5. **`next dev` does not exercise output file tracing.** Task 5 exists because of that: a route reading `content/docs` at request time works in dev and fails in a real build unless the config is updated. +### The `checkpointer` test fixture, verified + +Tasks 4 and 7 both search for `checkpointer` and assert a hit. That choice was checked against real content before the plan was written, so a failing test there means the pipeline is broken, not that the term was a bad guess: + +- It appears **16 times** in `apps/website/content/docs/langgraph/guides/persistence.mdx`, **7 of them outside fenced code** — so it survives the indexer's code stripping. +- It appears in the headings `Python: Checkpointer Setup` and `Checkpoint Recovery`, so there is a real anchor for a deep link to land on. +- It is in **no page title anywhere** in `docs-config.ts` — which is exactly why today's title-only search cannot find it, and why the test proves the feature rather than passing trivially. + +If a `checkpointer` assertion fails, fix the pipeline. Do not swap in an easier search term. + ### Existing signatures you will use ```ts From 9369ba505c71c4f00ec329c95d5fe85a92dabf15 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:12:36 -0700 Subject: [PATCH 06/18] feat(docs): index doc bodies into per-heading sections Pure MDX-to-sections indexer: frontmatter and fenced code stripped, inline code unwrapped, component prose kept. Anchors come from extract-headings so search deep links cannot drift from the TOC, which an anchor-parity test pins across all real content. Co-Authored-By: Claude Opus 5 --- .../website/src/lib/docs-search-index.spec.ts | 171 ++++++++++++++++++ apps/website/src/lib/docs-search-index.ts | 80 ++++++++ apps/website/src/lib/docs-search-types.ts | 29 +++ 3 files changed, 280 insertions(+) create mode 100644 apps/website/src/lib/docs-search-index.spec.ts create mode 100644 apps/website/src/lib/docs-search-index.ts create mode 100644 apps/website/src/lib/docs-search-types.ts diff --git a/apps/website/src/lib/docs-search-index.spec.ts b/apps/website/src/lib/docs-search-index.spec.ts new file mode 100644 index 000000000..d7591766a --- /dev/null +++ b/apps/website/src/lib/docs-search-index.spec.ts @@ -0,0 +1,171 @@ +import { describe, expect, it } from 'vitest'; +import { indexDocSections } from './docs-search-index'; +import { extractHeadings } from './extract-headings'; +import { getAllDocSlugs, getDocBySlug } from './docs'; + +const SOURCE = `--- +description: Frontmatter must not be indexed. +--- + +# Streaming + +Intro prose above the first heading. + +## Token deltas + +The adapter merges \`TEXT_MESSAGE_CONTENT\` deltas. + +\`\`\`ts +const secretCodeToken = 'should-not-be-indexed'; +\`\`\` + + +Callout body prose is searchable. + + + +`; + +describe('indexDocSections', () => { + const sections = indexDocSections(SOURCE); + const preamble = sections.find((s) => s.heading === null); + const deltas = sections.find((s) => s.heading === 'Token deltas'); + + it('emits an anchor-less record for content above the first heading', () => { + expect(preamble?.anchor).toBeNull(); + expect(preamble?.text).toContain('Intro prose above the first heading'); + }); + + it('splits at each heading and anchors it', () => { + expect(deltas?.anchor).toBe('token-deltas'); + expect(deltas?.text).toContain('The adapter merges'); + }); + + it('never indexes frontmatter', () => { + expect(sections.map((s) => s.text).join(' ')).not.toContain('must not be indexed'); + }); + + it('drops fenced code blocks', () => { + expect(sections.map((s) => s.text).join(' ')).not.toContain('secretCodeToken'); + }); + + it('unwraps inline code so API names are searchable as words', () => { + expect(deltas?.text).toContain('TEXT_MESSAGE_CONTENT'); + expect(deltas?.text).not.toContain('`'); + }); + + it('keeps component body prose and caption attributes, not markup', () => { + const all = sections.map((s) => s.text).join(' '); + expect(all).toContain('Callout body prose is searchable'); + expect(all).toContain('Backend speaks AG-UI over SSE'); + expect(all).not.toContain('StackDiagram'); + expect(all).not.toContain('data-tone'); + }); +}); + +describe('indexDocSections edge cases', () => { + it('returns no sections for an empty document', () => { + expect(indexDocSections('')).toEqual([]); + }); + + it('handles a doc with no headings at all as a single preamble section', () => { + const sections = indexDocSections('---\ndescription: x\n---\n\nJust prose, no headings anywhere.\n'); + expect(sections).toHaveLength(1); + expect(sections[0].heading).toBeNull(); + expect(sections[0].anchor).toBeNull(); + expect(sections[0].text).toContain('Just prose, no headings anywhere'); + }); + + it('starts a section at ### when it is the first heading', () => { + const sections = indexDocSections('Preamble.\n\n### Deep heading\n\nBody text.\n'); + const heading = sections.find((s) => s.heading === 'Deep heading'); + expect(heading?.anchor).toBe('deep-heading'); + expect(heading?.text).toContain('Body text'); + }); + + it('does not treat a ## inside a fenced code block as a heading', () => { + const source = [ + '# Title', + '', + '## Real heading', + '', + '```md', + '## not a real heading', + '```', + '', + 'Trailing prose.', + ].join('\n'); + const sections = indexDocSections(source); + expect(sections.find((s) => s.heading === 'not a real heading')).toBeUndefined(); + const real = sections.find((s) => s.heading === 'Real heading'); + expect(real?.text).toContain('Trailing prose'); + }); + + it('tolerates frontmatter with no closing fence', () => { + const source = '---\ndescription: unterminated\n\n## Heading\n\nBody.\n'; + expect(() => indexDocSections(source)).not.toThrow(); + }); + + it('keeps anchor alignment when a heading contains backticks', () => { + const source = '# Title\n\n## The `useAgent` hook\n\nBody about the hook.\n'; + const sections = indexDocSections(source); + const headings = extractHeadings(source); + const heading = sections.find((s) => s.heading === headings[0].text); + expect(heading?.anchor).toBe(headings[0].id); + expect(heading?.text).toContain('Body about the hook'); + }); + + it('keeps the title attribute of a paired component tag, not just self-closing ones', () => { + // Real docs wrap steps as `...` — a paired + // tag whose title is the exact phrase a reader would search for. Only extracting + // caption/title from self-closing tags would silently drop it. + const source = [ + '# Quick start', + '', + '', + '', + '', + 'Run the installer.', + '', + '', + '', + ].join('\n'); + const sections = indexDocSections(source); + const all = sections.map((s) => s.text).join(' '); + expect(all).toContain('Install the package'); + expect(all).toContain('Run the installer'); + expect(all).not.toContain('Step'); + expect(all).not.toContain('title='); + }); + + it('drops empty sections produced by a heading with no body text', () => { + const source = '# Title\n\n## Empty section\n\n## Next section\n\nSome text.\n'; + const sections = indexDocSections(source); + expect(sections.find((s) => s.heading === 'Empty section')).toBeUndefined(); + expect(sections.find((s) => s.heading === 'Next section')?.text).toContain('Some text'); + }); +}); + +describe('anchor parity with the rendered table of contents', () => { + // The load-bearing test. extract-headings hand-rolls slugification, so the + // only guarantee that a search deep link resolves is that the index and the + // TOC derive anchors from the same function over the same source. Running it + // across real content means a heading either tool sections differently fails + // here rather than shipping a link to nowhere. + it('emits only anchors the TOC also produces, for every real doc', () => { + const mismatches: string[] = []; + + for (const { library, section, slug } of getAllDocSlugs()) { + const doc = getDocBySlug(library, section, slug); + if (!doc) continue; + const tocIds = new Set(extractHeadings(doc.body).map((h) => h.id)); + for (const record of indexDocSections(doc.body)) { + if (record.anchor && !tocIds.has(record.anchor)) { + mismatches.push(`${library}/${section}/${slug}#${record.anchor}`); + } + } + } + + expect(mismatches).toEqual([]); + }); +}); diff --git a/apps/website/src/lib/docs-search-index.ts b/apps/website/src/lib/docs-search-index.ts new file mode 100644 index 000000000..85c99c85b --- /dev/null +++ b/apps/website/src/lib/docs-search-index.ts @@ -0,0 +1,80 @@ +import { stripFrontmatter } from './docs'; +import { extractHeadings } from './extract-headings'; +import type { DocSection } from './docs-search-types'; + +export type { DocSection }; + +/** + * Fenced code is dropped deliberately: it is roughly half the corpus and + * mostly noise for ranking. The cost is explicit — an error string that only + * ever appears inside a code sample will not match. Inline code is kept, + * because that is where API names appear in sentences. + */ +function toSearchableText(source: string): string { + return source + .replace(/```[\s\S]*?```/g, ' ') + // Opening component tags, self-closing or paired (e.g. ``, + // ``): keep prose-bearing attributes, drop the tag. + // Step/Callout titles carry real search content ("Install the package") that would + // otherwise vanish, since only the children of a paired tag survive below. + .replace(/<[A-Z][\w.]*(?:\s[^>]*)?\/?>/g, (tag) => { + const prose = [...tag.matchAll(/\b(?:caption|title)="([^"]*)"/g)].map((m) => m[1]); + return ` ${prose.join(' ')} `; + }) + // Closing component tags: drop. + .replace(/<\/[A-Z][\w.]*>/g, ' ') + .replace(/`([^`]+)`/g, '$1') + .replace(/\s+/g, ' ') + .trim(); +} + +/** + * Split a doc body into one record per heading, plus one for the preamble. + * + * Anchors come from `extractHeadings` rather than a second slugifier. That + * file hand-rolls GitHub-style slugification, and having one approximation is + * a known quantity while two that drift apart is a bug generator. + */ +export function indexDocSections(source: string): DocSection[] { + const body = stripFrontmatter(source); + const headings = extractHeadings(body); + const lines = body.split('\n'); + + const sections: DocSection[] = []; + let current: { heading: string | null; anchor: string | null; lines: string[] } = { + heading: null, + anchor: null, + lines: [], + }; + let headingIndex = 0; + let inCodeBlock = false; + + const flush = () => { + const text = toSearchableText(current.lines.join('\n')); + if (text.length > 0) { + sections.push({ heading: current.heading, anchor: current.anchor, text }); + } + }; + + for (const line of lines) { + // Track fences so a `## ` inside a code sample never starts a section. + if (line.trim().startsWith('```')) inCodeBlock = !inCodeBlock; + + const match = inCodeBlock ? null : line.match(/^#{2,3}\s+(.+)$/); + if (match) { + flush(); + const heading = headings[headingIndex]; + headingIndex += 1; + current = { + heading: heading?.text ?? match[1].replace(/`/g, ''), + anchor: heading?.id ?? null, + lines: [], + }; + continue; + } + current.lines.push(line); + } + flush(); + + return sections; +} diff --git a/apps/website/src/lib/docs-search-types.ts b/apps/website/src/lib/docs-search-types.ts new file mode 100644 index 000000000..8e1e30b25 --- /dev/null +++ b/apps/website/src/lib/docs-search-types.ts @@ -0,0 +1,29 @@ +/** + * Wire types shared by the search route and the client dialog. + * + * This module imports nothing on purpose. `DocsSearchHit` would naturally sit + * beside `searchIndexedDocs`, but that file's dependency chain reaches + * `lib/docs.ts` and therefore `fs`. A type-only import is erased at build + * time, so it would work — right up until someone drops the `type` keyword + * and pulls Node built-ins into the client bundle. Keeping the types here + * removes that trap rather than commenting on it. + */ + +export interface DocSection { + /** Heading text, or null for content above the first heading. */ + heading: string | null; + /** Heading id for a deep link, or null for the page preamble. */ + anchor: string | null; + /** Searchable prose: fenced code removed, inline code unwrapped. */ + text: string; +} + +export interface DocsSearchHit { + href: string; + title: string; + heading: string | null; + libraryTitle: string; + snippet: string; + /** [start, end) offsets into `snippet`. The client renders the marks. */ + marks: [number, number][]; +} From 3fcebee116ce882eede38796250710e63887b31e Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:14:07 -0700 Subject: [PATCH 07/18] docs: note the spurious cockpit-retirement failure from apps/website The plan tells implementers to cd into apps/website for targeted vitest runs, which makes that spec double-join its WEBSITE_ROOT path and report 3 failures. Verified pre-existing: the file matches origin/main and nx test website from the repo root is green. Flagged so nobody chases it. Co-Authored-By: Claude Opus 5 --- docs/superpowers/plans/2026-09-03-docs-search-content-index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/superpowers/plans/2026-09-03-docs-search-content-index.md b/docs/superpowers/plans/2026-09-03-docs-search-content-index.md index 4f42bfdcd..604e8f0ab 100644 --- a/docs/superpowers/plans/2026-09-03-docs-search-content-index.md +++ b/docs/superpowers/plans/2026-09-03-docs-search-content-index.md @@ -19,6 +19,7 @@ Read the spec. Then note these repo facts, which are not guessable: 1. **Website tests need an env var.** `GROWTH_FORM_POLICY=growth_v1` or the app throws at import. Every website command below includes it. 2. **Targeted test runs must start from the project directory.** `cd apps/website` first. From the repo root vitest reports "No test files found" and exits 1 — a wrong CWD, not a broken command. 3. **Unused imports are ESLint ERRORS here,** not warnings. +3b. **`cockpit-retirement.spec.ts` fails spuriously when vitest runs from `apps/website`.** It computes `WEBSITE_ROOT = join(process.cwd(), 'apps/website')`, which double-joins when the CWD is already `apps/website`, producing 3 failures. This is pre-existing and unrelated to any task here — verified: the file is byte-identical to `origin/main`, and `nx test website` from the repo root passes. If you see it in a broad `src/lib` run, ignore it; do not "fix" it as part of this work. 4. **Route specs run in the node environment.** They start with `// @vitest-environment node`. Component specs use `// @vitest-environment jsdom`. Getting this wrong produces confusing failures. 5. **`next dev` does not exercise output file tracing.** Task 5 exists because of that: a route reading `content/docs` at request time works in dev and fails in a real build unless the config is updated. From 9a28c5c19d06caf40d6515493a74373c7a141b77 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:14:57 -0700 Subject: [PATCH 08/18] fix(docs): strip markdown syntax from indexed section text Link targets were searchable, so a query for 'github' or 'docs' matched every page that happened to link somewhere, and snippets would have rendered raw [text](url). Keeps link text, unwraps emphasis, and leaves underscores inside identifiers like TEXT_MESSAGE_CONTENT alone. Co-Authored-By: Claude Opus 5 --- .../website/src/lib/docs-search-index.spec.ts | 43 +++++++++++++++++++ apps/website/src/lib/docs-search-index.ts | 15 +++++++ 2 files changed, 58 insertions(+) diff --git a/apps/website/src/lib/docs-search-index.spec.ts b/apps/website/src/lib/docs-search-index.spec.ts index d7591766a..5e681c401 100644 --- a/apps/website/src/lib/docs-search-index.spec.ts +++ b/apps/website/src/lib/docs-search-index.spec.ts @@ -138,6 +138,49 @@ describe('indexDocSections edge cases', () => { expect(all).not.toContain('title='); }); + it('keeps underscores inside identifiers while unwrapping emphasis', () => { + const [section] = indexDocSections('# T\n\nUse **TEXT_MESSAGE_CONTENT** with _care_.\n'); + expect(section.text).toContain('TEXT_MESSAGE_CONTENT'); + expect(section.text).toContain('care'); + expect(section.text).not.toContain('**'); + expect(section.text).not.toContain('_care_'); + }); + + it('indexes link text but not link targets', () => { + const [section] = indexDocSections('# T\n\nSee [Installation](/docs/langgraph/install) first.\n'); + expect(section.text).toContain('Installation'); + expect(section.text).not.toContain('/docs/langgraph/install'); + }); + + it('indexes image alt text but not the image target', () => { + const [section] = indexDocSections('# T\n\n![Architecture diagram](/img/arch.png) shows the flow.\n'); + expect(section.text).toContain('Architecture diagram'); + expect(section.text).not.toContain('/img/arch.png'); + }); + + it('unwraps single- and double-asterisk emphasis', () => { + const [section] = indexDocSections('# T\n\nThis is *italic* and **bold** text.\n'); + expect(section.text).toContain('italic'); + expect(section.text).toContain('bold'); + expect(section.text).not.toContain('*'); + }); + + it('unwraps double-underscore emphasis', () => { + const [section] = indexDocSections('# T\n\nThis is __strongly__ stated.\n'); + expect(section.text).toContain('strongly'); + expect(section.text).not.toContain('__'); + }); + + it('leaves multiple links on one line without runaway matching', () => { + const [section] = indexDocSections( + '# T\n\nSee [Installation](/docs/a) and [Quickstart](/docs/b) both.\n' + ); + expect(section.text).toContain('Installation'); + expect(section.text).toContain('Quickstart'); + expect(section.text).not.toContain('/docs/a'); + expect(section.text).not.toContain('/docs/b'); + }); + it('drops empty sections produced by a heading with no body text', () => { const source = '# Title\n\n## Empty section\n\n## Next section\n\nSome text.\n'; const sections = indexDocSections(source); diff --git a/apps/website/src/lib/docs-search-index.ts b/apps/website/src/lib/docs-search-index.ts index 85c99c85b..40d15e8ed 100644 --- a/apps/website/src/lib/docs-search-index.ts +++ b/apps/website/src/lib/docs-search-index.ts @@ -23,6 +23,21 @@ function toSearchableText(source: string): string { }) // Closing component tags: drop. .replace(/<\/[A-Z][\w.]*>/g, ' ') + // Markdown links and images: keep the text/alt, drop the target. The target is a + // URL, not content — indexing it makes "github" or "docs" match nearly every page + // that happens to link somewhere, and a raw `[text](url)` reads as broken in a + // snippet. Non-greedy character classes (no nested `[`/`(`) keep this from running + // away on a line with several links. + .replace(/!?\[([^[\]]*)\]\([^()]*\)/g, '$1') + // Emphasis: unwrap to inner text. Double markers first, so `**bold**` doesn't leave + // stray single markers behind for the single-marker passes to trip over. + .replace(/\*\*([^*]+?)\*\*/g, '$1') + .replace(/__([^_]+?)__/g, '$1') + .replace(/\*([^*\n]+?)\*/g, '$1') + // Single-underscore emphasis only opens/closes at a word boundary, same as + // CommonMark's intraword rule — so `_care_` unwraps but `TEXT_MESSAGE_CONTENT`, + // whose underscores sit between word characters, is left untouched. + .replace(/(? Date: Thu, 3 Sep 2026 11:18:35 -0700 Subject: [PATCH 09/18] feat(docs): rank indexed doc sections and build snippets Weighted AND matching over title, heading and prose, capped at eight to match the existing result list. Snippets return offsets rather than HTML so the client renders the marks itself; the window snaps to word boundaries and overlapping marks are merged rather than emitted raw. Co-Authored-By: Claude Opus 5 --- .../website/src/lib/docs-search-query.spec.ts | 150 ++++++++++++++++++ apps/website/src/lib/docs-search-query.ts | 143 +++++++++++++++++ 2 files changed, 293 insertions(+) create mode 100644 apps/website/src/lib/docs-search-query.spec.ts create mode 100644 apps/website/src/lib/docs-search-query.ts diff --git a/apps/website/src/lib/docs-search-query.spec.ts b/apps/website/src/lib/docs-search-query.spec.ts new file mode 100644 index 000000000..c92b6a26e --- /dev/null +++ b/apps/website/src/lib/docs-search-query.spec.ts @@ -0,0 +1,150 @@ +import { describe, expect, it } from 'vitest'; +import { searchIndexedDocs, type IndexedDoc } from './docs-search-query'; + +const DOCS: IndexedDoc[] = [ + { + library: 'langgraph', + libraryTitle: 'LangGraph', + section: 'guides', + slug: 'persistence', + title: 'Persistence', + sections: [ + { heading: null, anchor: null, text: 'How threads survive a restart.' }, + { + heading: 'Production checkpointers', + anchor: 'production-checkpointers', + text: 'Use a Postgres checkpointer in production rather than memory.', + }, + ], + }, + { + library: 'chat', + libraryTitle: 'Chat', + section: 'guides', + slug: 'checkpointer', + title: 'Checkpointer', + sections: [{ heading: null, anchor: null, text: 'Unrelated prose.' }], + }, +]; + +describe('searchIndexedDocs', () => { + it('finds a term that appears only in body prose', () => { + const hits = searchIndexedDocs(DOCS, 'postgres'); + expect(hits).toHaveLength(1); + expect(hits[0].href).toBe('/docs/langgraph/guides/persistence#production-checkpointers'); + expect(hits[0].heading).toBe('Production checkpointers'); + }); + + it('ranks a title match above a body match', () => { + const hits = searchIndexedDocs(DOCS, 'checkpointer'); + expect(hits[0].title).toBe('Checkpointer'); + }); + + it('links to the page top when the match is in the preamble', () => { + const hits = searchIndexedDocs(DOCS, 'restart'); + expect(hits[0].href).toBe('/docs/langgraph/guides/persistence'); + }); + + it('requires every token, matching the instant layer', () => { + expect(searchIndexedDocs(DOCS, 'postgres nonexistent')).toEqual([]); + }); + + it('returns nothing for a query of only stop words', () => { + expect(searchIndexedDocs(DOCS, 'of the')).toEqual([]); + }); + + it('returns a snippet with offsets covering the matched term', () => { + const [hit] = searchIndexedDocs(DOCS, 'postgres'); + expect(hit.snippet).toContain('Postgres'); + expect(hit.marks.length).toBeGreaterThan(0); + const [start, end] = hit.marks[0]; + expect(hit.snippet.slice(start, end).toLowerCase()).toBe('postgres'); + }); + + it('caps results at eight, matching the existing result list', () => { + const many: IndexedDoc[] = Array.from({ length: 12 }, (_, i) => ({ + library: 'chat', + libraryTitle: 'Chat', + section: 'guides', + slug: `page-${i}`, + title: `Page ${i}`, + sections: [{ heading: null, anchor: null, text: 'streaming prose' }], + })); + expect(searchIndexedDocs(many, 'streaming')).toHaveLength(8); + }); + + it('does not render a section whose text is shorter than the snippet window with stray ellipses', () => { + const short: IndexedDoc[] = [ + { + library: 'chat', + libraryTitle: 'Chat', + section: 'guides', + slug: 'short', + title: 'Short', + sections: [{ heading: null, anchor: null, text: 'A short bit of streaming prose.' }], + }, + ]; + const [hit] = searchIndexedDocs(short, 'streaming'); + expect(hit.snippet).toBe('A short bit of streaming prose.'); + expect(hit.snippet.startsWith('…')).toBe(false); + expect(hit.snippet.endsWith('…')).toBe(false); + }); + + it('finds a hit whose only match is the page title, not any section text', () => { + const titleOnly: IndexedDoc[] = [ + { + library: 'chat', + libraryTitle: 'Chat', + section: 'guides', + slug: 'zeta-widget', + title: 'Zeta Widget', + sections: [{ heading: null, anchor: null, text: 'Nothing relevant here.' }], + }, + ]; + const hits = searchIndexedDocs(titleOnly, 'zeta'); + expect(hits).toHaveLength(1); + expect(hits[0].title).toBe('Zeta Widget'); + }); + + it('keeps a deterministic order for hits tied on score and length', () => { + const tied: IndexedDoc[] = [ + { + library: 'chat', + libraryTitle: 'Chat', + section: 'guides', + slug: 'alpha', + title: 'Alpha', + sections: [{ heading: null, anchor: null, text: 'streaming prose here' }], + }, + { + library: 'chat', + libraryTitle: 'Chat', + section: 'guides', + slug: 'beta', + title: 'Beta', + sections: [{ heading: null, anchor: null, text: 'streaming prose here' }], + }, + ]; + const first = searchIndexedDocs(tied, 'streaming').map((h) => h.title); + const second = searchIndexedDocs(tied, 'streaming').map((h) => h.title); + expect(first).toEqual(second); + expect(first).toEqual(['Alpha', 'Beta']); + }); + + it('does not emit overlapping marks when one token is a substring of another', () => { + const overlap: IndexedDoc[] = [ + { + library: 'chat', + libraryTitle: 'Chat', + section: 'guides', + slug: 'agents', + title: 'Agents', + sections: [{ heading: null, anchor: null, text: 'Configure agents and their agent tools.' }], + }, + ]; + const [hit] = searchIndexedDocs(overlap, 'agent agents'); + for (let i = 1; i < hit.marks.length; i++) { + expect(hit.marks[i][0]).toBeGreaterThanOrEqual(hit.marks[i - 1][1]); + } + }); +}); diff --git a/apps/website/src/lib/docs-search-query.ts b/apps/website/src/lib/docs-search-query.ts new file mode 100644 index 000000000..b2df7dc8f --- /dev/null +++ b/apps/website/src/lib/docs-search-query.ts @@ -0,0 +1,143 @@ +import type { DocSection, DocsSearchHit } from './docs-search-types'; +import { searchTokens } from './docs-search-tokens'; + +export type { DocsSearchHit }; + +export interface IndexedDoc { + library: string; + libraryTitle: string; + section: string; + slug: string; + title: string; + sections: DocSection[]; +} + +const MAX_RESULTS = 8; +const SNIPPET_RADIUS = 80; + +/** Title matches beat heading matches, which beat body prose. */ +const TITLE_WEIGHT = 3; +const HEADING_WEIGHT = 2; +const TEXT_WEIGHT = 1; + +function countWeighted(haystack: string, token: string, weight: number): number { + return haystack.toLowerCase().includes(token) ? weight : 0; +} + +/** True for whitespace — the only boundary `toSearchableText` leaves behind. */ +function isBoundary(char: string | undefined): boolean { + return char === undefined || /\s/.test(char); +} + +/** Walk left from `index` to the nearest preceding word boundary. */ +function snapStart(text: string, index: number): number { + let i = index; + while (i > 0 && !isBoundary(text[i - 1])) i -= 1; + return i; +} + +/** Walk right from `index` to the nearest following word boundary. */ +function snapEnd(text: string, index: number): number { + let i = index; + while (i < text.length && !isBoundary(text[i])) i += 1; + return i; +} + +/** + * Merge or drop overlapping ranges so the client never has to reason about + * them. Ranges arrive sorted by start; a range that starts before the + * previous one ended either extends it (partial overlap) or is dropped + * entirely (fully contained) — the walk-in-order renderer never sees an + * overlap either way. + */ +function mergeRanges(ranges: [number, number][]): [number, number][] { + const merged: [number, number][] = []; + for (const [start, end] of ranges) { + const last = merged[merged.length - 1]; + if (last && start < last[1]) { + if (end > last[1]) last[1] = end; + continue; + } + merged.push([start, end]); + } + return merged; +} + +/** + * A window of `text` around the first token match, snapped to word + * boundaries so the snippet never opens or closes mid-word. + * + * Offsets are returned rather than HTML so the client wraps the ranges + * itself — nothing server-built is ever rendered into the page. + */ +function buildSnippet(text: string, tokens: string[]): { snippet: string; marks: [number, number][] } { + const lower = text.toLowerCase(); + const first = tokens + .map((token) => lower.indexOf(token)) + .filter((index) => index >= 0) + .sort((a, b) => a - b)[0] ?? 0; + + const rawStart = Math.max(0, first - SNIPPET_RADIUS); + const rawEnd = Math.min(text.length, first + SNIPPET_RADIUS); + const start = rawStart > 0 ? snapEnd(text, rawStart) : rawStart; + const end = rawEnd < text.length ? snapStart(text, rawEnd) : rawEnd; + const prefix = start > 0 ? '…' : ''; + const suffix = end < text.length ? '…' : ''; + const snippet = `${prefix}${text.slice(start, end)}${suffix}`; + + const snippetLower = snippet.toLowerCase(); + const rawMarks: [number, number][] = []; + for (const token of tokens) { + let at = snippetLower.indexOf(token); + while (at >= 0) { + rawMarks.push([at, at + token.length]); + at = snippetLower.indexOf(token, at + token.length); + } + } + rawMarks.sort((a, b) => a[0] - b[0]); + return { snippet, marks: mergeRanges(rawMarks) }; +} + +export function searchIndexedDocs(docs: IndexedDoc[], query: string): DocsSearchHit[] { + const tokens = searchTokens(query); + if (tokens.length === 0) return []; + + const scored: { score: number; length: number; hit: DocsSearchHit }[] = []; + + for (const doc of docs) { + for (const section of doc.sections) { + const haystack = `${doc.title} ${section.heading ?? ''} ${section.text}`.toLowerCase(); + // AND semantics, matching the instant client matcher. + if (!tokens.every((token) => haystack.includes(token))) continue; + + const score = tokens.reduce( + (total, token) => + total + + countWeighted(doc.title, token, TITLE_WEIGHT) + + countWeighted(section.heading ?? '', token, HEADING_WEIGHT) + + countWeighted(section.text, token, TEXT_WEIGHT), + 0 + ); + + const { snippet, marks } = buildSnippet(section.text, tokens); + scored.push({ + score, + length: section.text.length, + hit: { + href: `/docs/${doc.library}/${doc.section}/${doc.slug}${section.anchor ? `#${section.anchor}` : ''}`, + title: doc.title, + heading: section.heading, + libraryTitle: doc.libraryTitle, + snippet, + marks, + }, + }); + } + } + + // Higher score first; ties go to the shorter section, which favours a + // precise heading over a long prose blob. Array.prototype.sort is stable, + // so a tie on both keys keeps the order the docs were passed in. + scored.sort((a, b) => b.score - a.score || a.length - b.length); + return scored.slice(0, MAX_RESULTS).map((entry) => entry.hit); +} From 439e272abd6b6a92fffb1915eec074ae815a6dd0 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:24:38 -0700 Subject: [PATCH 10/18] fix(docs): stop a quoted attribute terminating a component tag The tag regex scanned to the first > even inside a quoted value, so title="a > b" leaked the tail of the tag into indexed text. No current doc trips it; the next one with an arrow in a caption would have, with nothing to catch it. Also drops the DocSection re-export, which offered a second import path through the module that reaches fs -- the exact thing the dependency-free types module exists to prevent. Co-Authored-By: Claude Opus 5 --- .../website/src/lib/docs-search-index.spec.ts | 11 +++ apps/website/src/lib/docs-search-index.ts | 88 ++++++++++++++++--- 2 files changed, 85 insertions(+), 14 deletions(-) diff --git a/apps/website/src/lib/docs-search-index.spec.ts b/apps/website/src/lib/docs-search-index.spec.ts index 5e681c401..8b13ba95c 100644 --- a/apps/website/src/lib/docs-search-index.spec.ts +++ b/apps/website/src/lib/docs-search-index.spec.ts @@ -181,6 +181,17 @@ describe('indexDocSections edge cases', () => { expect(section.text).not.toContain('/docs/b'); }); + it('does not let a > inside a quoted attribute value terminate the tag match', () => { + // A naive [^>]* attribute scan stops at the first >, even inside a quoted value, + // and leaks the rest of the tag into indexed text as raw markup. + const source = '# T\n\n\nBody prose.\n\n'; + const [section] = indexDocSections(source); + expect(section.text).toContain('a > b'); + expect(section.text).toContain('Body prose'); + expect(section.text).not.toContain('type="tip"'); + expect(section.text).not.toContain('"'); + }); + it('drops empty sections produced by a heading with no body text', () => { const source = '# Title\n\n## Empty section\n\n## Next section\n\nSome text.\n'; const sections = indexDocSections(source); diff --git a/apps/website/src/lib/docs-search-index.ts b/apps/website/src/lib/docs-search-index.ts index 40d15e8ed..d3493d130 100644 --- a/apps/website/src/lib/docs-search-index.ts +++ b/apps/website/src/lib/docs-search-index.ts @@ -2,7 +2,73 @@ import { stripFrontmatter } from './docs'; import { extractHeadings } from './extract-headings'; import type { DocSection } from './docs-search-types'; -export type { DocSection }; +/** + * Strip MDX component tags, keeping prose-bearing attributes and dropping the rest. + * + * `` and `` (self-closing + * or paired — a paired tag's own title carries real search content, e.g. a step name, that + * would otherwise vanish since only its children survive) both have their `caption`/`title` + * text kept; the markup itself is dropped. Closing tags are dropped outright. + * + * This is a hand-written scan, not a regex, on purpose: a regex whose attribute region is + * `[^>]*` stops at the first `>`, even inside a quoted value, and leaks the tag's tail into + * indexed text as raw markup for input like `title="a > b"`. The regex that instead matches + * attributes as discrete `name` / `name="..."` units — `(?:\s+[\w-]+(?:=(?:"[^"]*"|'[^']*'| + * [^\s>]+))?)*` — fixes that, but nests a `+`-quantified token inside a `*`-repeated group, + * the textbook catastrophic-backtracking shape: an unterminated tag with many attributes + * (verified with 5,000) hangs the process, since the engine tries every way to re-partition + * the attribute run before giving up. A linear left-to-right scan that tracks quote state + * has no such failure mode — worst case (many unterminated tags in a row) is polynomial, not + * exponential, and real MDX never produces that shape anyway. + */ +function stripComponentTags(text: string): string { + const isUpper = (ch: string | undefined) => ch !== undefined && /[A-Z]/.test(ch); + let result = ''; + let i = 0; + + while (i < text.length) { + const closing = text[i] === '<' && text[i + 1] === '/' && isUpper(text[i + 2]); + const opening = text[i] === '<' && isUpper(text[i + 1]); + + if (closing || opening) { + let j = i + (closing ? 2 : 1); + while (j < text.length && /[\w.]/.test(text[j])) j += 1; + + let quote: string | null = null; + let end = -1; + for (let k = j; k < text.length; k += 1) { + const ch = text[k]; + if (quote) { + if (ch === quote) quote = null; + } else if (ch === '"' || ch === "'") { + quote = ch; + } else if (ch === '>') { + end = k; + break; + } + } + + if (end !== -1) { + if (closing) { + result += ' '; + } else { + const tag = text.slice(i, end + 1); + const prose = [...tag.matchAll(/\b(?:caption|title)="([^"]*)"/g)].map((m) => m[1]); + result += ` ${prose.join(' ')} `; + } + i = end + 1; + continue; + } + // No closing '>' found before the end of the string: not a real tag. Fall through + // and copy the '<' literally rather than consuming the rest of the document. + } + + result += text[i]; + i += 1; + } + + return result; +} /** * Fenced code is dropped deliberately: it is roughly half the corpus and @@ -11,23 +77,17 @@ export type { DocSection }; * because that is where API names appear in sentences. */ function toSearchableText(source: string): string { - return source - .replace(/```[\s\S]*?```/g, ' ') - // Opening component tags, self-closing or paired (e.g. ``, - // ``): keep prose-bearing attributes, drop the tag. - // Step/Callout titles carry real search content ("Install the package") that would - // otherwise vanish, since only the children of a paired tag survive below. - .replace(/<[A-Z][\w.]*(?:\s[^>]*)?\/?>/g, (tag) => { - const prose = [...tag.matchAll(/\b(?:caption|title)="([^"]*)"/g)].map((m) => m[1]); - return ` ${prose.join(' ')} `; - }) - // Closing component tags: drop. - .replace(/<\/[A-Z][\w.]*>/g, ' ') + const withoutFencedCode = source.replace(/```[\s\S]*?```/g, ' '); + const withoutTags = stripComponentTags(withoutFencedCode); + + return withoutTags // Markdown links and images: keep the text/alt, drop the target. The target is a // URL, not content — indexing it makes "github" or "docs" match nearly every page // that happens to link somewhere, and a raw `[text](url)` reads as broken in a // snippet. Non-greedy character classes (no nested `[`/`(`) keep this from running - // away on a line with several links. + // away on a line with several links. Known limitation, not present in the corpus: + // link text containing nested brackets (`[A [nested] B](url)`) will not match, and + // the raw markdown including the URL survives instead of being stripped. .replace(/!?\[([^[\]]*)\]\([^()]*\)/g, '$1') // Emphasis: unwrap to inner text. Double markers first, so `**bold**` doesn't leave // stray single markers behind for the single-marker passes to trip over. From b4e216bc8593bffdbe676b5920cf399217a327d4 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:25:46 -0700 Subject: [PATCH 11/18] perf(docs): hoist lowercasing out of the search token loop Each field was re-lowercased once per query token per section, so a multi-token query allocated the same strings hundreds of times per request. Behaviour is unchanged; a mixed-case test now guards against a half-converted refactor making one field case-sensitive. Co-Authored-By: Claude Opus 5 --- .../website/src/lib/docs-search-query.spec.ts | 30 +++++++++++++++++ apps/website/src/lib/docs-search-query.ts | 33 ++++++++++++++----- 2 files changed, 54 insertions(+), 9 deletions(-) diff --git a/apps/website/src/lib/docs-search-query.spec.ts b/apps/website/src/lib/docs-search-query.spec.ts index c92b6a26e..6cad1e77c 100644 --- a/apps/website/src/lib/docs-search-query.spec.ts +++ b/apps/website/src/lib/docs-search-query.spec.ts @@ -147,4 +147,34 @@ describe('searchIndexedDocs', () => { expect(hit.marks[i][0]).toBeGreaterThanOrEqual(hit.marks[i - 1][1]); } }); + + it('matches and scores case-insensitively across mixed-case title, heading and body', () => { + const mixedCase: IndexedDoc[] = [ + { + library: 'chat', + libraryTitle: 'Chat', + section: 'guides', + slug: 'mixed-case', + title: 'MIXEDcase Widget', + sections: [ + { + heading: 'ConFig Section', + anchor: 'config-section', + text: 'Some Prose mentioning WIDGET behavior in a Sentence.', + }, + ], + }, + ]; + const hits = searchIndexedDocs(mixedCase, 'widget'); + expect(hits).toHaveLength(1); + expect(hits[0].title).toBe('MIXEDcase Widget'); + // Matches in all three fields (title, heading via a second query below, + // and body) must all be found regardless of the query's or the source + // text's casing -- a half-converted lower-casing refactor would make + // exactly one of these fields silently case-sensitive. + const headingHits = searchIndexedDocs(mixedCase, 'CONFIG'); + expect(headingHits).toHaveLength(1); + const bodyHits = searchIndexedDocs(mixedCase, 'SENTENCE'); + expect(bodyHits).toHaveLength(1); + }); }); diff --git a/apps/website/src/lib/docs-search-query.ts b/apps/website/src/lib/docs-search-query.ts index b2df7dc8f..4093faaba 100644 --- a/apps/website/src/lib/docs-search-query.ts +++ b/apps/website/src/lib/docs-search-query.ts @@ -20,8 +20,9 @@ const TITLE_WEIGHT = 3; const HEADING_WEIGHT = 2; const TEXT_WEIGHT = 1; -function countWeighted(haystack: string, token: string, weight: number): number { - return haystack.toLowerCase().includes(token) ? weight : 0; +/** `haystack` must already be lower-cased — callers hoist that conversion so it happens once per field per section, not once per token. */ +function countWeighted(lowerHaystack: string, token: string, weight: number): number { + return lowerHaystack.includes(token) ? weight : 0; } /** True for whitespace — the only boundary `toSearchableText` leaves behind. */ @@ -72,10 +73,16 @@ function mergeRanges(ranges: [number, number][]): [number, number][] { */ function buildSnippet(text: string, tokens: string[]): { snippet: string; marks: [number, number][] } { const lower = text.toLowerCase(); - const first = tokens + const firstMatch = tokens .map((token) => lower.indexOf(token)) .filter((index) => index >= 0) - .sort((a, b) => a - b)[0] ?? 0; + .sort((a, b) => a - b)[0]; + // No token appears in this section's text at all when the query matched + // only the page title (or only the heading) — the AND-check in + // searchIndexedDocs still passed because those fields carried the match. + // There is nothing to center a window on, so start the snippet at 0 + // rather than leaving `first` undefined. + const first = firstMatch ?? 0; const rawStart = Math.max(0, first - SNIPPET_RADIUS); const rawEnd = Math.min(text.length, first + SNIPPET_RADIUS); @@ -85,6 +92,10 @@ function buildSnippet(text: string, tokens: string[]): { snippet: string; marks: const suffix = end < text.length ? '…' : ''; const snippet = `${prefix}${text.slice(start, end)}${suffix}`; + // Marks are re-derived by scanning the rendered snippet rather than reused + // from the haystack scan above: they describe what is visible to the + // reader, so a token occurring in `text` outside this window correctly + // produces no mark. const snippetLower = snippet.toLowerCase(); const rawMarks: [number, number][] = []; for (const token of tokens) { @@ -105,17 +116,21 @@ export function searchIndexedDocs(docs: IndexedDoc[], query: string): DocsSearch const scored: { score: number; length: number; hit: DocsSearchHit }[] = []; for (const doc of docs) { + const lowerTitle = doc.title.toLowerCase(); for (const section of doc.sections) { - const haystack = `${doc.title} ${section.heading ?? ''} ${section.text}`.toLowerCase(); - // AND semantics, matching the instant client matcher. + const lowerHeading = (section.heading ?? '').toLowerCase(); + const lowerText = section.text.toLowerCase(); + // AND semantics, matching the instant client matcher. Each field is + // lower-cased once above rather than once per token below. + const haystack = `${lowerTitle} ${lowerHeading} ${lowerText}`; if (!tokens.every((token) => haystack.includes(token))) continue; const score = tokens.reduce( (total, token) => total + - countWeighted(doc.title, token, TITLE_WEIGHT) + - countWeighted(section.heading ?? '', token, HEADING_WEIGHT) + - countWeighted(section.text, token, TEXT_WEIGHT), + countWeighted(lowerTitle, token, TITLE_WEIGHT) + + countWeighted(lowerHeading, token, HEADING_WEIGHT) + + countWeighted(lowerText, token, TEXT_WEIGHT), 0 ); From 79cbafac7b56d0d05e746dd59b26f552173bbb3e Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:29:19 -0700 Subject: [PATCH 12/18] feat(docs): add the docs content search route Builds the section index once per instance and answers queries from it. Short queries return empty without scanning, and responses are cacheable because the corpus only changes on deploy. A per-document try/catch guards index construction so one malformed doc cannot take down search for the whole instance. Co-Authored-By: Claude Opus 5 --- .../src/app/api/docs-search/route.spec.ts | 84 ++++++++++++++ apps/website/src/app/api/docs-search/route.ts | 105 ++++++++++++++++++ 2 files changed, 189 insertions(+) create mode 100644 apps/website/src/app/api/docs-search/route.spec.ts create mode 100644 apps/website/src/app/api/docs-search/route.ts diff --git a/apps/website/src/app/api/docs-search/route.spec.ts b/apps/website/src/app/api/docs-search/route.spec.ts new file mode 100644 index 000000000..3d10f4be2 --- /dev/null +++ b/apps/website/src/app/api/docs-search/route.spec.ts @@ -0,0 +1,84 @@ +// @vitest-environment node +import { describe, expect, it } from 'vitest'; +import { buildIndex, GET } from './route'; + +const call = (q: string) => + GET(new Request(`http://localhost/api/docs-search?q=${encodeURIComponent(q)}`)); + +describe('GET /api/docs-search', () => { + it('finds a page by a term that appears only in its body prose', async () => { + // "checkpointer" is prose in the LangGraph persistence guide, and is in no + // page title — exactly the query the old title-only search could not serve. + const res = await call('checkpointer'); + expect(res.status).toBe(200); + const { results } = await res.json(); + expect(results.length).toBeGreaterThan(0); + expect(results.some((r: { href: string }) => r.href.includes('/docs/langgraph/'))).toBe(true); + }); + + it('returns hits that carry a snippet and marks', async () => { + const { results } = await (await call('checkpointer')).json(); + expect(typeof results[0].snippet).toBe('string'); + expect(Array.isArray(results[0].marks)).toBe(true); + }); + + it('returns a hit shaped exactly as the wire contract, no extra fields', async () => { + const { results } = await (await call('checkpointer')).json(); + expect(Object.keys(results[0]).sort()).toEqual( + ['href', 'title', 'heading', 'libraryTitle', 'snippet', 'marks'].sort() + ); + }); + + it('returns empty without scanning for a query under two characters', async () => { + const { results } = await (await call('a')).json(); + expect(results).toEqual([]); + }); + + it('searches at exactly the two-character boundary rather than short-circuiting', async () => { + // "ag" is a real substring in this corpus (ag-ui) — proves the >= 2 path + // actually reaches searchIndexedDocs instead of also being swallowed by + // the short-query guard. + const { results } = await (await call('ag')).json(); + expect(results.length).toBeGreaterThan(0); + }); + + it('returns empty for a missing query parameter', async () => { + const res = await GET(new Request('http://localhost/api/docs-search')); + const { results } = await res.json(); + expect(results).toEqual([]); + }); + + it('treats a whitespace-only query the same as an empty one', async () => { + const { results } = await (await call(' ')).json(); + expect(results).toEqual([]); + }); + + it('is cacheable, because the corpus only changes on deploy', async () => { + const res = await call('streaming'); + expect(res.headers.get('cache-control')).toContain('max-age='); + }); + + it('sets Cache-Control on the empty-result response too, not only populated ones', async () => { + const res = await call('a'); + expect(res.headers.get('cache-control')).toContain('max-age='); + }); +}); + +describe('buildIndex', () => { + it('skips a single document that throws without breaking the rest of the corpus', () => { + const entries = [ + { library: 'lib', section: 'guides', slug: 'good' }, + { library: 'lib', section: 'guides', slug: 'bad' }, + ]; + + const result = buildIndex(entries, (entry) => { + if (entry.slug === 'bad') { + throw new Error('simulated malformed doc'); + } + return { title: 'Good Doc', body: '# Good Doc\n\nSome findable prose here.' }; + }); + + expect(result).toHaveLength(1); + expect(result[0].slug).toBe('good'); + }); +}); diff --git a/apps/website/src/app/api/docs-search/route.ts b/apps/website/src/app/api/docs-search/route.ts new file mode 100644 index 000000000..e89a8cd16 --- /dev/null +++ b/apps/website/src/app/api/docs-search/route.ts @@ -0,0 +1,105 @@ +import { NextResponse } from 'next/server'; +import { getAllDocSlugs, getDocBySlug } from '../../../lib/docs'; +import { getLibraryConfig } from '../../../lib/docs-config'; +import { indexDocSections } from '../../../lib/docs-search-index'; +import { searchIndexedDocs, type IndexedDoc } from '../../../lib/docs-search-query'; + +const MIN_QUERY_LENGTH = 2; + +export interface DocSlugEntry { + library: string; + section: string; + slug: string; +} + +/** The slice of `ResolvedDoc` the indexer actually needs. */ +interface ResolvableDoc { + title: string; + body: string; +} + +function resolveDoc(entry: DocSlugEntry): ResolvableDoc | null { + return getDocBySlug(entry.library, entry.section, entry.slug); +} + +/** + * Build the search index from a list of doc slugs. + * + * `resolveDoc` is a parameter — not just a closure over `lib/docs` — so a + * test can inject a resolver that throws for one entry without touching the + * filesystem, proving the per-document guard below actually guards. + * + * One malformed doc must not take down search for the whole instance: the + * index is built once at module scope (see `getIndex`), so an uncaught throw + * here would fail every request that instance ever serves, not just the one + * request that happened to trigger the (re)build. + */ +export function buildIndex( + entries: DocSlugEntry[], + resolve: (entry: DocSlugEntry) => ResolvableDoc | null = resolveDoc +): IndexedDoc[] { + return entries.flatMap((entry) => { + try { + const doc = resolve(entry); + if (!doc) return []; + return [ + { + library: entry.library, + libraryTitle: getLibraryConfig(entry.library)?.title ?? entry.library, + section: entry.section, + slug: entry.slug, + title: doc.title, + sections: indexDocSections(doc.body), + }, + ]; + } catch { + return []; + } + }); +} + +/** + * Built once per instance, not per request. + * + * This reads MDX from disk at request time, which is why + * `content/docs/**` has to be traced into the deployed function — see + * `outputFileTracingIncludes` in next.config.ts. The route cannot be + * statically generated the way `api/markdown` is, because the query space is + * unbounded. + */ +let index: IndexedDoc[] | null = null; + +function getIndex(): IndexedDoc[] { + if (!index) { + index = buildIndex(getAllDocSlugs()); + } + return index; +} + +export async function GET(request: Request): Promise { + const query = new URL(request.url).searchParams.get('q')?.trim() ?? ''; + + if (query.length < MIN_QUERY_LENGTH) { + return NextResponse.json( + { results: [] }, + { + headers: { + // Same cacheability as the populated response — a short/empty + // query is just as deterministic a function of the corpus. + 'Cache-Control': 'public, max-age=300', + }, + } + ); + } + + return NextResponse.json( + { results: searchIndexedDocs(getIndex(), query) }, + { + headers: { + // The corpus only changes on deploy, so repeated queries are served + // by the CDN rather than waking this function. + 'Cache-Control': 'public, max-age=300', + }, + } + ); +} From d0a97f5e622307396c556f64290f03abf2dec8fa Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:41:27 -0700 Subject: [PATCH 13/18] build(website): trace docs content for the search route The route reads MDX at request time and cannot be statically generated the way api/markdown is, so without this it deploys with no corpus and returns empty for every query -- silently, and only in production. Co-Authored-By: Claude Opus 5 --- apps/website/next.config.ts | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/apps/website/next.config.ts b/apps/website/next.config.ts index aad1ff014..a667249c9 100644 --- a/apps/website/next.config.ts +++ b/apps/website/next.config.ts @@ -17,6 +17,11 @@ export const nextConfig: WithNxOptions = { '../../cockpit/**/*.ts', '../../deployments/ag-ui-mastra/*.mjs', '../../nx.json', + // The docs search route reads these at request time. Unlike + // api/markdown it cannot be statically generated, so without this the + // route deploys with no corpus and returns empty for every query — + // silently, and only in production. + 'content/docs/**/*.mdx', ], }, skipTrailingSlashRedirect: true, From e3e87f673ae88c1b9c47867bced6b32ee014eed9 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:42:48 -0700 Subject: [PATCH 14/18] docs: correct Task 5's premise and its verification recipe Executing it disproved two things I asserted. @vercel/nft already traces content/docs by statically resolving the fs reads in lib/docs.ts -- 122 mdx paths in the route trace both before and after the include, so the include is belt-and-braces rather than the load-bearing fix. More seriously, the verification I wrote does not verify anything: next start never consumes .nft.json (only Vercel's builder or an output:'standalone' build do), and nx build emits to dist/apps/website rather than the path the recipe used. It would have passed whether or not tracing worked, which is worse than no check. Co-Authored-By: Claude Opus 5 --- .../plans/2026-09-03-docs-search-content-index.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-09-03-docs-search-content-index.md b/docs/superpowers/plans/2026-09-03-docs-search-content-index.md index 604e8f0ab..4d7840e17 100644 --- a/docs/superpowers/plans/2026-09-03-docs-search-content-index.md +++ b/docs/superpowers/plans/2026-09-03-docs-search-content-index.md @@ -816,7 +816,14 @@ Co-Authored-By: Claude Opus 5 " ### Task 5: Trace the content into the deployed function -**This is the task that fails in production if skipped, and passes every local test if it is.** `next dev` reads from the working tree, so nothing before this point exercises file tracing. +> **Findings from executing this task — the premise below was partly wrong.** +> +> 1. **`@vercel/nft` already traces the content without the include.** It statically resolves the `fs.readFileSync` + `path.join(process.cwd(), …)` pattern in `lib/docs.ts` and walks the real directory at build time. Measured: **122 `.mdx` paths in the route's `.nft.json` both before and after** adding the include, and an unmodified-config `output: 'standalone'` build served non-empty `checkpointer` results. The include was kept anyway — it is cheap, it is documented in `route.ts`, and it removes reliance on an nft heuristic that is not guaranteed to hold under Vercel's builder, which is not bit-identical to a local build. But it is belt-and-braces, not the load-bearing fix this section claimed. +> 2. **The Step 4 verification below does not verify tracing.** `next start` does not consume `.nft.json` at all — only Vercel's builder or an `output: 'standalone'` build do. And `nx build website` emits to `dist/apps/website`, not `apps/website/.next`. So running Step 4 as written returns results whether or not tracing is correct. To actually test tracing locally you must build with `output: 'standalone'` and run the standalone server. **A green Step 4 is not evidence.** +> +> Net: the change is correct and worth keeping, the risk was lower than stated, and the verification recipe needed fixing. Anyone adding a future runtime-read asset should check a real Vercel deploy log rather than trusting a local `next start`. + +**This was written as the task that fails in production if skipped, and passes every local test if it is.** `next dev` reads from the working tree, so nothing before this point exercises file tracing. `apps/website/next.config.ts` traces `cockpit/**` md/py/ts, a Mastra mjs, and `nx.json` — nothing under `apps/website/content`. `api/markdown` reads MDX and deploys fine only because `generateStaticParams()` makes it build-time output. The search route reads at request time. From b46c0ab018ff80332ac0e70efc5d4b2a8ca55d26 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:47:08 -0700 Subject: [PATCH 15/18] docs: open the search dialog the way the existing e2e does Task 7 used Meta+k and an unnamed combobox. workspace-shell.spec.ts clicks the "Search docs" button and addresses the combobox by its accessible name "Search documentation..." -- more portable across runners, and the button is the affordance a real user has. Co-Authored-By: Claude Opus 5 --- .../plans/2026-09-03-docs-search-content-index.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-09-03-docs-search-content-index.md b/docs/superpowers/plans/2026-09-03-docs-search-content-index.md index 4d7840e17..c909d047d 100644 --- a/docs/superpowers/plans/2026-09-03-docs-search-content-index.md +++ b/docs/superpowers/plans/2026-09-03-docs-search-content-index.md @@ -1179,14 +1179,20 @@ Append inside the `Docs slug page` describe in `apps/website/e2e/docs.spec.ts`: ```ts test('finds a term that appears only in body prose and lands on its section', async ({ page }) => { await page.goto(route); - await page.keyboard.press('Meta+k'); + + // Open by clicking the trigger, matching workspace-shell.spec.ts. A + // keyboard shortcut would depend on how the runner maps Meta, and the + // button is the affordance a real user has anyway. + await page.getByRole('button', { name: 'Search docs' }).first().click(); const dialog = page.getByRole('dialog', { name: 'Search documentation' }); await expect(dialog).toBeVisible(); // "checkpointer" is prose inside the persistence guide and is in no page // title, so a title-only search returns nothing for it. - await dialog.getByRole('combobox').fill('checkpointer'); + await dialog + .getByRole('combobox', { name: 'Search documentation...' }) + .fill('checkpointer'); const hit = dialog.getByRole('option').filter({ hasText: /checkpointer/i }).first(); await expect(hit).toBeVisible({ timeout: 10000 }); From 2b629d2199b9d7924e2a125154254b7ed90e70af Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 11:49:09 -0700 Subject: [PATCH 16/18] feat(docs): show page-content hits in docs search Debounced, abortable requests merge server hits beneath the instant title matches, each showing its section heading and a snippet with the match highlighted. A failed request falls back to the instant results rather than surfacing an error. Title matches and content hits now share one continuous keyboard- navigable list instead of two disjoint ones: arrow keys and Enter operate over a combined array so every rendered option is reachable and selectable, aria-activedescendant and aria-selected stay in sync with it, and the selected index is clamped whenever the combined list shrinks (a narrower query, or a slow response landing after arrow navigation) so it can never dangle past the end. A response for a query that is no longer current is dropped even when the underlying fetch ignores the abort signal, so a slow, stale response can never clobber a newer query's results. Co-Authored-By: Claude Opus 5 --- .../src/components/docs/DocsSearch.spec.tsx | 238 ++++++++++++++++++ .../src/components/docs/DocsSearch.tsx | 149 ++++++++++- apps/website/src/styles/docs.css | 31 +++ 3 files changed, 412 insertions(+), 6 deletions(-) create mode 100644 apps/website/src/components/docs/DocsSearch.spec.tsx diff --git a/apps/website/src/components/docs/DocsSearch.spec.tsx b/apps/website/src/components/docs/DocsSearch.spec.tsx new file mode 100644 index 000000000..6a0d56af2 --- /dev/null +++ b/apps/website/src/components/docs/DocsSearch.spec.tsx @@ -0,0 +1,238 @@ +// @vitest-environment jsdom +import React from 'react'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/react'; +import { DocsSearch } from './DocsSearch'; + +vi.mock('next/navigation', () => ({ useRouter: () => ({ push: vi.fn() }) })); +vi.mock('../../lib/analytics/client', () => ({ track: vi.fn() })); + +const HIT = { + href: '/docs/langgraph/guides/persistence#production-checkpointers', + title: 'Persistence', + heading: 'Production checkpointers', + libraryTitle: 'LangGraph', + snippet: 'Use a Postgres checkpointer in production.', + marks: [[6, 14]] as [number, number][], +}; + +function openSearch() { + render(); + fireEvent.keyDown(document, { key: 'k', metaKey: true }); +} + +beforeEach(() => { + vi.useFakeTimers({ shouldAdvanceTime: true }); + // jsdom does not implement scrollIntoView; the "scroll the selected + // option into view" effect calls it unconditionally. + Element.prototype.scrollIntoView = vi.fn(); +}); +afterEach(() => { + vi.useRealTimers(); + vi.unstubAllGlobals(); +}); + +describe('DocsSearch content results', () => { + it('renders server hits with their heading and a highlighted snippet', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue({ ok: true, json: async () => ({ results: [HIT] }) }) + ); + openSearch(); + fireEvent.change(screen.getByRole('combobox'), { target: { value: 'checkpointer' } }); + + await waitFor(() => expect(screen.getByText('Production checkpointers')).toBeTruthy()); + // The mark is rendered from offsets, never from server HTML. + expect(screen.getByText('Postgres').tagName).toBe('MARK'); + }); + + it('still shows instant title results when the request fails', async () => { + vi.stubGlobal('fetch', vi.fn().mockRejectedValue(new Error('offline'))); + openSearch(); + // "quickstart" matches page titles in the client-side index. + fireEvent.change(screen.getByRole('combobox'), { target: { value: 'quickstart' } }); + + await waitFor(() => expect(screen.getAllByRole('option').length).toBeGreaterThan(0)); + // A failed search must never surface an error state in the dialog. + expect(screen.queryByText(/error/i)).toBeNull(); + }); + + it('does not request for a query under two characters', async () => { + const fetchMock = vi.fn().mockResolvedValue({ ok: true, json: async () => ({ results: [] }) }); + vi.stubGlobal('fetch', fetchMock); + openSearch(); + fireEvent.change(screen.getByRole('combobox'), { target: { value: 'a' } }); + + await vi.advanceTimersByTimeAsync(400); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it('reaches the first content hit by arrowing past the last title hit', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue({ ok: true, json: async () => ({ results: [HIT] }) }) + ); + openSearch(); + const input = screen.getByRole('combobox'); + fireEvent.change(input, { target: { value: 'checkpointer' } }); + + await waitFor(() => expect(screen.getByText('Production checkpointers')).toBeTruthy()); + + const options = screen.getAllByRole('option'); + // Arrow down once per option to walk off the end of the title group and + // into the content group. + for (let i = 0; i < options.length; i++) { + fireEvent.keyDown(input, { key: 'ArrowDown' }); + } + + const contentOption = screen.getByText('Production checkpointers').closest('[role="option"]'); + expect(contentOption?.getAttribute('aria-selected')).toBe('true'); + }); + + it('navigates to the content hit href (including the anchor) on Enter', async () => { + const push = vi.fn(); + const navModule = await import('next/navigation'); + vi.spyOn(navModule, 'useRouter').mockReturnValue({ push } as unknown as ReturnType); + + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue({ ok: true, json: async () => ({ results: [HIT] }) }) + ); + openSearch(); + const input = screen.getByRole('combobox'); + fireEvent.change(input, { target: { value: 'checkpointer' } }); + + await waitFor(() => expect(screen.getByText('Production checkpointers')).toBeTruthy()); + + const options = screen.getAllByRole('option'); + for (let i = 0; i < options.length; i++) { + fireEvent.keyDown(input, { key: 'ArrowDown' }); + } + fireEvent.keyDown(input, { key: 'Enter' }); + + expect(push).toHaveBeenCalledWith(HIT.href); + }); + + it('keeps exactly one option marked aria-selected at a time', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue({ ok: true, json: async () => ({ results: [HIT] }) }) + ); + openSearch(); + const input = screen.getByRole('combobox'); + fireEvent.change(input, { target: { value: 'checkpointer' } }); + + await waitFor(() => expect(screen.getByText('Production checkpointers')).toBeTruthy()); + + fireEvent.keyDown(input, { key: 'ArrowDown' }); + fireEvent.keyDown(input, { key: 'ArrowDown' }); + + const options = screen.getAllByRole('option'); + const selectedCount = options.filter((o) => o.getAttribute('aria-selected') === 'true').length; + expect(selectedCount).toBe(1); + }); + + it('clamps the selected index when a narrower query shrinks the results', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue({ ok: true, json: async () => ({ results: [HIT] }) }) + ); + openSearch(); + const input = screen.getByRole('combobox'); + fireEvent.change(input, { target: { value: 'checkpointer' } }); + await waitFor(() => expect(screen.getByText('Production checkpointers')).toBeTruthy()); + + const options = screen.getAllByRole('option'); + for (let i = 0; i < options.length; i++) { + fireEvent.keyDown(input, { key: 'ArrowDown' }); + } + + // Now narrow the query to something that matches nothing at all — the + // previously-selected index must not dangle past the new (empty) list. + vi.stubGlobal('fetch', vi.fn().mockResolvedValue({ ok: true, json: async () => ({ results: [] }) })); + fireEvent.change(input, { target: { value: 'zzzzzznomatch' } }); + + await waitFor(() => expect(screen.queryAllByRole('option').length).toBe(0)); + // No option is present, so nothing should throw when Enter is pressed + // and there is nothing to navigate to. + expect(() => fireEvent.keyDown(input, { key: 'Enter' })).not.toThrow(); + }); + + it('lets a newer query win when an older request resolves later', async () => { + let resolveFirst: (v: unknown) => void = () => undefined; + let resolveSecond: (v: unknown) => void = () => undefined; + const fetchMock = vi + .fn() + .mockImplementationOnce(() => new Promise((resolve) => { resolveFirst = resolve; })) + .mockImplementationOnce(() => new Promise((resolve) => { resolveSecond = resolve; })); + vi.stubGlobal('fetch', fetchMock); + + openSearch(); + const input = screen.getByRole('combobox'); + + fireEvent.change(input, { target: { value: 'first query' } }); + await vi.advanceTimersByTimeAsync(150); + + fireEvent.change(input, { target: { value: 'second query' } }); + await vi.advanceTimersByTimeAsync(150); + + expect(fetchMock).toHaveBeenCalledTimes(2); + + const secondHit = { ...HIT, heading: 'Second query heading' }; + const firstHit = { ...HIT, heading: 'First query heading', href: '/docs/first' }; + + // Resolve the newer (second) request first, then the stale first one. + resolveSecond({ ok: true, json: async () => ({ results: [secondHit] }) }); + await waitFor(() => expect(screen.getByText('Second query heading')).toBeTruthy()); + + resolveFirst({ ok: true, json: async () => ({ results: [firstHit] }) }); + // Give the stale promise a chance to resolve and (if buggy) clobber state. + await vi.advanceTimersByTimeAsync(0); + + expect(screen.queryByText('First query heading')).toBeNull(); + expect(screen.getByText('Second query heading')).toBeTruthy(); + }); +}); + +describe('DocsSearch snippet rendering', () => { + it('renders HTML-ish snippet text as plain text, never as markup', async () => { + const xssHit = { + ...HIT, + snippet: 'a b', + marks: [[2, 10]] as [number, number][], + }; + vi.stubGlobal( + 'fetch', + vi.fn().mockResolvedValue({ ok: true, json: async () => ({ results: [xssHit] }) }) + ); + openSearch(); + fireEvent.change(screen.getByRole('combobox'), { target: { value: 'checkpointer' } }); + + await waitFor(() => expect(screen.getByText('Production checkpointers')).toBeTruthy()); + + expect(document.querySelector('script')).toBeNull(); + const mark = document.querySelector('mark'); + expect(mark?.textContent).toBe('