Docs search: index page content, not just titles - #990
Merged
Conversation
Search matches titles, not documentation -- searchableText() only reads title/description/slug/section/library out of docs-config, never a page body. That mattered less when search was a power-user shortcut; #986 made it the front door in two places. Adds a dynamic route that indexes prose at heading granularity and returns anchor deep links with highlighted snippets, while keeping the existing client-side title matcher as the instant layer so the fast path stays fast and a failed request degrades to today's behaviour. Records two traps found while exploring: outputFileTracingIncludes does not cover content/docs (api/markdown only works because it is statically generated), and extract-headings hand-rolls slugification, so the index must reuse it rather than introduce a second slugger. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Eight tasks, 41 TDD steps. Ordered so the shared tokenizer lands before the route and query module that import it. Two refinements from the spec, both recorded in the plan: the wire types get a dependency-free module (a client component needs DocsSearchHit, whose natural home transitively imports fs), and DocSection drops the per-page fields the spec repeated on every record. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The route needs identical tokenisation, and two copies would drift the first time a stop word is added. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tokenizer is shared so the client matcher and the server route agree on what a query means. Adding or removing a stop word changes every query on both sides identically, so no behavioural test elsewhere would notice. This is the only thing that catches it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Verified against real content: 16 occurrences in the persistence guide, 7 outside fenced code so it survives the indexer's code stripping, and it heads two sections so a deep link has somewhere to land. Critically it is in no page title, which is why today's title-only search cannot find it -- the test proves the feature rather than passing trivially. So a failing checkpointer assertion means the pipeline is broken, not that the term was a bad guess. Do not swap in an easier one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pure MDX-to-sections indexer: frontmatter and fenced code stripped, inline code unwrapped, component prose kept. Anchors come from extract-headings so search deep links cannot drift from the TOC, which an anchor-parity test pins across all real content. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan tells implementers to cd into apps/website for targeted vitest runs, which makes that spec double-join its WEBSITE_ROOT path and report 3 failures. Verified pre-existing: the file matches origin/main and nx test website from the repo root is green. Flagged so nobody chases it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Link targets were searchable, so a query for 'github' or 'docs' matched every page that happened to link somewhere, and snippets would have rendered raw [text](url). Keeps link text, unwraps emphasis, and leaves underscores inside identifiers like TEXT_MESSAGE_CONTENT alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Weighted AND matching over title, heading and prose, capped at eight to match the existing result list. Snippets return offsets rather than HTML so the client renders the marks itself; the window snaps to word boundaries and overlapping marks are merged rather than emitted raw. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tag regex scanned to the first > even inside a quoted value, so title="a > b" leaked the tail of the tag into indexed text. No current doc trips it; the next one with an arrow in a caption would have, with nothing to catch it. Also drops the DocSection re-export, which offered a second import path through the module that reaches fs -- the exact thing the dependency-free types module exists to prevent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each field was re-lowercased once per query token per section, so a multi-token query allocated the same strings hundreds of times per request. Behaviour is unchanged; a mixed-case test now guards against a half-converted refactor making one field case-sensitive. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Builds the section index once per instance and answers queries from it. Short queries return empty without scanning, and responses are cacheable because the corpus only changes on deploy. A per-document try/catch guards index construction so one malformed doc cannot take down search for the whole instance. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The route reads MDX at request time and cannot be statically generated the way api/markdown is, so without this it deploys with no corpus and returns empty for every query -- silently, and only in production. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Executing it disproved two things I asserted. @vercel/nft already traces content/docs by statically resolving the fs reads in lib/docs.ts -- 122 mdx paths in the route trace both before and after the include, so the include is belt-and-braces rather than the load-bearing fix. More seriously, the verification I wrote does not verify anything: next start never consumes .nft.json (only Vercel's builder or an output:'standalone' build do), and nx build emits to dist/apps/website rather than the path the recipe used. It would have passed whether or not tracing worked, which is worse than no check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Task 7 used Meta+k and an unnamed combobox. workspace-shell.spec.ts clicks the "Search docs" button and addresses the combobox by its accessible name "Search documentation..." -- more portable across runners, and the button is the affordance a real user has. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Debounced, abortable requests merge server hits beneath the instant title matches, each showing its section heading and a snippet with the match highlighted. A failed request falls back to the instant results rather than surfacing an error. Title matches and content hits now share one continuous keyboard- navigable list instead of two disjoint ones: arrow keys and Enter operate over a combined array so every rendered option is reachable and selectable, aria-activedescendant and aria-selected stay in sync with it, and the selected index is clamped whenever the combined list shrinks (a narrower query, or a slow response landing after arrow navigation) so it can never dangle past the end. A response for a query that is no longer current is dropped even when the underlying fetch ignores the abort signal, so a slow, stale response can never clobber a newer query's results. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Searches "checkpointer" — a term that exists only in body prose in the persistence guide and appears in no page title — and asserts the hit comes from the content-search group (not the instant title matcher) and lands on a real section anchor, not just a well-formed fragment. Adds a negative-control test searching a term confirmed absent from all docs content, asserting the empty state, so a matcher that always returns something can't make the positive test pass by accident. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The indexer re-implemented extract-headings' fence tracking and heading regex, then paired the two scans by position. Copy-identical today, but a change to either would silently misassign anchors -- and the parity test could not catch it, since the anchors would all still be valid members of the same file's set. Only the assignment would be wrong, so a deep link would scroll to the wrong section and look like it worked. extractHeadings now reports each heading's source line, and indexDocSections slices the body between those lines instead of re-scanning for fences and `##`/`###`. There is exactly one place left that decides what a heading is. Also: clarify the section/sections field collision in IndexedDoc, and strip un-fenced import lines in toSearchableText so a doc that imports a component for real JSX use can't leak the import statement into a search snippet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Docs search matched titles, not documentation.
DocsSearch'ssearchableText()concatenatedtitle,description,slug,sectionandlibraryTitle— all fromdocs-config.ts. No page body was ever read, so searchingcheckpointer,toAgent, or an error string pasted from a stack trace returned "No results found" unless the term happened to sit in a page title.This makes body prose searchable at heading granularity, with anchor deep links and highlighted snippets.
lib/docs-search-index.ts) turns MDX into per-heading section records: frontmatter and fenced code stripped, inline code unwrapped, markdown link targets dropped (indexing them madegithubmatch every page linking to GitHub), component prose kept includingtitle/captionattributes.extract-headings.tsand nowhere else, so a search deep link always matches what the TOC produces. An anchor-parity test asserts this across all 121 real docs.lib/docs-search-query.ts): AND matching, weighted title 3 / heading 2 / prose 1, ties to the shorter section, capped at 8. Snippets return text plus[start,end)offsets, never HTML — the client wraps the ranges itself.api/docs-search): index built once per instance (measured 34ms for 121 docs / 983 sections),Cache-Control: public, max-age=300, queries under 2 characters return empty without scanning, one malformed document is skipped rather than poisoning the instance.Sequencing
Independent of #986 — this branch never touches the files that PR changes — but #986 is what makes search the primary nav affordance (leading the control-plane pane and closing every content page), which is what motivates indexing content at all. Landing #986 first reads better.
Test Plan
nx test websitegreen;nx lint website0 errors;nx build websitesucceeds with/api/docs-searchregistering as a dynamicƒroutenx e2e website— 119/119, including a new test that searches a term appearing only in body prose and asserts the result lands on an anchor that exists on the destination page, plus a negative control (a nonsense term returns the empty state) so a matcher returning everything can't make the positive test pass by accidentcheckpointer— that's the point), arrow keys traverse both groups, Enter lands on#whats-nextand the anchor existsNotes for review
role="option"elements while nav stayed bound to the title array — which would have left half the listbox mouse-only.outputFileTracingIncludesgainscontent/docs/**/*.mdx. Worth knowing:@vercel/nftalready traces those files by statically resolving thefsreads inlib/docs.ts(122 paths in the route trace with and without the include), so this is belt-and-braces rather than load-bearing. Alsonext startnever consumes.nft.json, so it cannot be used to verify tracing — only Vercel's builder or anoutput: 'standalone'build can.🤖 Generated with Claude Code