feat: serve markdown to agents via content negotiation - #28
Merged
Conversation
Vercel routing-layer rewrites (Docus-style) send Accept: text/markdown and curl requests to the /raw/** mirrors, ahead of the ISR cache. Adds raw mirrors for /tree and /blob previews, markdown 404 bodies with recovery links, an llms.txt when-to-use section, and an Organization JSON-LD node.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
commit: |
…arkdown # Conflicts: # playground/content/3.concepts/2.versioned-previews.md
Preview pages (/tree, /blob, /pr) serve HTML only: the raw mirrors and content negotiation now cover production pages exclusively. Previews are review surfaces, not agent-facing content, and mirroring them would have required re-implementing the preview authorization added in #31 on every raw route. - delete /raw/tree and /raw/blob routes - exclude /tree, /blob and /pr from the markdown rewrites - restore the prod-only guard on the Copy page / Open in Chat actions - align docs and the preview-versions skill
Documentation previews📚 Preview all documentation changes (follows new pushes)
Pinned to the current head: |
The llms.txt spec has no "when to use" section: the blockquote summary under the H1 is where key context belongs. Drop the invented docs.llms.whenToUse key and document putting the guidance in docs.llms.description instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improves the is-agentic score for comark.dev (currently 67/100) for every site built on this layer.
What changed
Markdown content negotiation (essential check, was failing)
modules/markdown-rewrite.tsinjects rewrites into.vercel/output/config.jsonafter Nitro compiles, same approach as Docus' markdown-rewrite.Accept: text/markdown(or acurl/*user-agent) on any production page URL serves its/raw/**mirror;/serves/llms.txt. Responses carrycontent-type: text/markdownandvary: Accept.llms.txt; this layer reads content at request time, so the rewrites use generic patterns instead (unit-tested intest/markdown-rewrite.test.ts)./tree,/blob,/pr) are excluded and serve HTML only: they are review surfaces, not agent-facing content, and mirroring them would mean re-implementing the preview authorization from feat: authorize commit previews and add /pr/:number preview routes #31 on every raw route. An earlier revision of this PR shipped preview mirrors; they were dropped after feat: authorize commit previews and add /pr/:number preview routes #31 landed.Agent-friendly 404s (essential check, was partial)
/raw/**misses return a real 404 with a short markdown body pointing at/llms.txt,/llms-full.txt,/raw/index.md, and the sitemap. Combined with the rewrites, an agent that guesses a wrong page URL gets a recoverable markdown 404.llms.txt agent guidance (recommended check)
docs.llms.description(the llms.txt blockquote summary, where the spec puts key context) is now documented as the place for "when to use" guidance. An earlier revision added adocs.llms.whenToUsekey and a custom section; dropped since the spec has no such section.Organization JSON-LD (recommended check)
docs.schemaOrg.organizationsub-key, emitted as a top-levelOrganizationnode on the landing page (forcontactPoint,address,sameAs).Docs
Verification
pnpm lint(no new warnings),pnpm test(146 passed),pnpm typecheckclean.NITRO_PRESET=vercel pnpm build→ 4 routes injected intoconfig.json(verified shape).EISDIRduring prerender of/.well-known/skills(from the skills module, unrelated to this PR).Out of scope (needs changes in the comark.dev content repo)
<h1 id="hello-world">which likely confuses the scanner./about,/contact,/privacy),docs.llms.description/organizationvalues, OpenAPI spec decision.