Skip to content

feat: support path-prefix canonical base URLs in --canonical-origin - #102

Merged
dacharyc merged 3 commits into
agent-ecosystem:mainfrom
philip:support-canonical-path-prefix
Aug 23, 2026
Merged

feat: support path-prefix canonical base URLs in --canonical-origin#102
dacharyc merged 3 commits into
agent-ecosystem:mainfrom
philip:support-canonical-path-prefix

Conversation

@philip

@philip philip commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

--canonical-origin previously accepted only an origin, so it worked when production and preview shared the same path structure. But when production docs live under a path prefix (e.g. https://example.com/docs) and the preview serves from a different base, an origin swap produces broken URLs.

This PR lets the flag accept an origin plus a path prefix. When a prefix is given, matching URLs in fetched content are rewritten to the full target base instead of just swapping origins.

Along the way this fixes two latent issues in the canonical-origin rewrite itself: a $ in the target URL was interpreted as a replacement pattern, and URLs ending at ?, #, ), or , (markdown links, prose) weren't matched. The canonical value is also normalized consistently now (trailing slashes, host case, default ports).

Verified end to end against a real deploy: a Vercel preview of neon.com/docs scored identically to production with --canonical-origin https://neon.com/docs.

🤖 Generated with Claude Code

philip and others added 3 commits August 14, 2026 16:51
A canonical like https://prod.com/docs// left a trailing slash in the
normalized value, so the rewrite regex silently matched nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rlap

A URL ending exactly at the canonical base wasn't rewritten when an XML
closing tag followed, so a sitemap's landing-page entry
(<loc>https://prod.example.com/docs</loc>) kept its production URL in
path-prefix mode. Add < to the boundary lookahead; a literal < can never
appear in a valid URL, so the match is unambiguous.

Also warn (without disabling the flag) when the canonical base is a
path-prefix of the target base, since URLs in fetched content that
already point at the target get re-prefixed by the rewrite.
@dacharyc

Copy link
Copy Markdown
Member

Hey, thanks for the PR Philip! This is a good enhancement, and I appreciate that it fixes a divergence we had between the CLI (which reduced the flag to .origin) and direct createContext() callers (which passed the raw value into the rewrite regex). It's nice that the CLI and API behave identically now and that was a miss on my part before this PR, so I appreciate the fix.

I did push one follow-up commit (9ebdac8) with two additions found while reviewing:

  1. Sitemap <loc> entries ending exactly at the canonical base didn't rewrite. The boundary lookahead didn't include <, so <loc>https://prod.example.com/docs</loc> (the docs landing page, no trailing slash) kept its production URL while /docs/ and /docs/guide rewrote fine. In path-prefix mode, almost every sitemap has such an entry, and the leaked URL either gets filtered from discovery as external or fetched against production. Since a literal < can never appear in a valid URL, adding it to the lookahead is unambiguous. Added a sitemap-shaped test alongside your boundary tests.

  2. A warning for prefix overlap. The new mode permits same-origin rewriting, and when the canonical base is a path-prefix of the target base itself (previews served under the production docs path), URLs in content that already point at the target get re-prefixed: /docs/preview-123/guide becomes /docs/preview-123/preview-123/guide. That ambiguity is inherent to prefix rewriting, so the CLI now warns without disabling the flag, and the docs mention the caveat.

@dacharyc
dacharyc merged commit 591d90a into agent-ecosystem:main Aug 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants