fix(docs): SEO improvements by jannikmaierhoefer · Pull Request #1858 · langfuse/langfuse-python

jannikmaierhoefer · 2026-09-04T08:53:23Z

Four of the errors in the September Ahrefs Site Audit of langfuse.com land on python.reference.langfuse.com (the crawl runs in subdomains mode). None of them come from the SDK — all four come from how pdoc's output is generated and served.

Important

This PR alone does not change the live site. The published build command lives outside this repo — docs/ is gitignored, and no workflow here references pdoc. Whoever owns the Cloudflare Pages project needs to point its build command at bash scripts/build_reference_docs.sh (output directory docs). Details in "Deploying this" below.

What's wrong

1. / is a stub with no title, no links and no canonical. With a single root module, pdoc makes langfuse.html the entry point and writes an index.html that is nothing but a redirect:

<!doctype html>
<html>
<head>
    <meta charset="utf-8">
    <meta http-equiv="refresh" content="0; url=./langfuse.html"/>
</head>
</html>

That accounts for three audit errors at once: Title tag missing or empty, Page has no outgoing links, and Duplicate pages without canonical (/ and /index.html are byte-identical with nothing marking either canonical).

2. Every unmatched path returns that stub with HTTP 200. The host has no 404.html in the output, so it falls back to the landing page for anything it cannot match:

$ for p in /nope-does-not-exist /langfuse/nope.html /langfuse/concepts/models.md /404.html; do
    curl -s -o /dev/null -w "%{http_code} %{size_download}b  $p\n" "https://python.reference.langfuse.com$p"; done
200 1077b  /nope-does-not-exist
200 1077b  /langfuse/nope.html
200 1077b  /langfuse/concepts/models.md
200 1077b  /404.html

Every stale or mistyped URL is therefore an indexable duplicate. That is how two paths that have never existed — …/langfuse/concepts/models.md and …/langfuse/api/concepts/models.md — ended up in the crawl as real pages. For comparison, js.reference.langfuse.com returns a proper 404 for the same request.

3. /langfuse is 2.7 MB. pdoc inlines the full source of every symbol. That is over Ahrefs' 2 MB crawl limit, so the single most important page on the reference site is not crawled at all — and it is slow for readers regardless of crawlers.

The fix

#	Change	Effect
1	`pdoc-templates/index.html.jinja2` sets `root_module_name` to false — the escape hatch pdoc's own template documents — and fills the main column, which pdoc's version leaves empty	`/` becomes a real landing page: title, meta description, canonical, and links to `langfuse`, `langfuse.experiment`, `langfuse.api` plus the docs, GitHub and PyPI
2	`pdoc-templates/module.html.jinja2` adds a self-referencing canonical to every module page	50/50 pages now carry one
3	`pdoc-templates/404.html` is copied into the output	unmatched paths get a real 404 instead of a 200
4	`--no-show-source`, with `--edit-url` adding a GitHub source link per module in its place	`langfuse.html` 2.7 MB → 484 KB
5	`scripts/build_reference_docs.sh` wraps all of the above	the site cannot be built incorrectly by accident

On (2): the .html suffix is deliberately dropped from the canonical. The site is served with clean URLs, and /langfuse.html 308-redirects to /langfuse — so a canonical pointing at the .html path would point at a redirect, which is its own audit error:

$ curl -so /dev/null -w "%{http_code} -> %{redirect_url}\n" https://python.reference.langfuse.com/langfuse.html
308 -> https://python.reference.langfuse.com/langfuse

On (4): losing the inline source is a real trade, which is why --edit-url is added alongside it — every module page now links to its source on GitHub (verified: /langfuse/__init__.py and /langfuse/experiment.py both 200). 2.7 MB of HTML on one page is worse for readers than one click through to GitHub.

PDOC_CANONICAL_BASE_URL overrides the canonical origin for local or preview builds.

Verification

bash scripts/build_reference_docs.sh <tmpdir> with pdoc 15.0.4, then served locally and driven in a browser.

Every page now has a title and a clean-URL canonical:

51 html pages, 404.html present: True
pages without a canonical:            none
canonicals ending in .html:           none
pages without a title:                none

index.html                title: Langfuse Python SDK API reference
                      canonical: https://python.reference.langfuse.com/
                          links: 49
langfuse.html             title: langfuse API documentation
                      canonical: https://python.reference.langfuse.com/langfuse
langfuse/experiment.html  title: langfuse.experiment API documentation
                      canonical: https://python.reference.langfuse.com/langfuse/experiment

Page sizes — nothing over 2 MB any more (was 2669 KB):

  484 KB  langfuse.html
  196 KB  langfuse/api.html
  120 KB  langfuse/experiment.html
    0     pages over 2 MB

Search still works, and pdoc's restore-on-clear behaviour keeps the new landing content intact — the content block writes into main.pdoc, which is exactly what pdoc's search captures as originalContent:

query "start_observation" → 10 results, first: "def langfuse.Langfuse.start_observation(…"
cleared query             → main restored to "Langfuse Python SDK API reference", 7 links

Rendered the landing page, a module page and the 404 page in a browser; all three look correct, logo and styling intact. Every link on the landing page checked: the three module targets exist in the output, and the three external links return 200.

uv run --frozen ruff check . passes, bash -n scripts/build_reference_docs.sh clean. Nothing here is touched by the pre-commit hooks or CI (they cover Python under langfuse/ only).

Deploying this

docs/ is gitignored and no workflow in this repo builds it, so the published site is built from outside — the host is Cloudflare Pages (it strips .html, 308-redirects trailing slashes, and falls back to the landing page when no 404.html exists; js.reference by contrast is on Vercel).

To make this PR take effect, the Pages project's build command needs to become:

bash scripts/build_reference_docs.sh

with output directory docs. If the site is instead deployed by hand, run that script before wrangler pages deploy docs.

Fix (3) — the 404.html — is the highest-value item, since it stops every stale URL from being indexed, not just the two the crawl happened to find.

🤖 Generated with Claude Code

Greptile Summary

This PR adds a standardized pdoc reference-site build that generates an indexable landing page, clean canonical metadata, a static 404 response, smaller module pages, and GitHub source links. It also updates contributor guidance to require the new build script.

Replaces pdoc’s redirect-only root page with navigable reference content and metadata.
Adds canonical links to generated module pages.
Copies a dedicated noindex 404 page into the deployment output.
Disables large inline source listings while retaining source links.
Documents the required reference-site build path.

Confidence Score: 4/5

The PR should not merge until alternate canonical origins are normalized, because the documented override can generate incorrect metadata across every module page.

The default production origin works, but supplying the documented override without a trailing slash concatenates the hostname and module name, breaking all generated module canonicals for that build.

Files Needing Attention: pdoc-templates/module.html.jinja2

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Script["build_reference_docs.sh"] --> Pdoc["pdoc 15.0.4"]
  Templates["pdoc-templates"] --> Pdoc
  Pdoc --> Index["docs/index.html"]
  Pdoc --> Modules["docs/module pages"]
  ErrorPage["pdoc-templates/404.html"] --> Copy["Copy step"]
  Copy --> NotFound["docs/404.html"]
  Index --> Pages["Cloudflare Pages output"]
  Modules --> Pages
  NotFound --> Pages

Prompt To Fix All With AI

### Issue 1
pdoc-templates/module.html.jinja2:14
**Canonical URLs Can Merge**

If `PDOC_CANONICAL_BASE_URL` is set to a normal origin without a trailing slash, this direct concatenation joins the hostname and module path. For example, `https://preview.example` produces `https://preview.examplelangfuse`, so every module page in that build receives an incorrect canonical URL.

```suggestion
          href="{{ canonical_base_url.rstrip('/') }}/{{ module.modulename.replace('.', '/') }}"/>
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "fix(docs): make the generated reference ..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

Four of the errors in the September Ahrefs Site Audit of langfuse.com land on python.reference.langfuse.com, and all four come from how pdoc's output is generated rather than from the SDK. pdoc writes an `index.html` that is only a `<meta http-equiv="refresh">` stub, because a single root module makes `langfuse.html` the entry point. That stub has no title, no links and no canonical URL, so `/` and `/index.html` read as duplicate, soft-redirecting pages. Worse, the host serves that same stub with a `200` for *any* unmatched path when the output contains no `404.html`, so every stale or mistyped URL becomes another indexable copy of it -- which is how two nonexistent `…/concepts/models.md` paths ended up in the crawl. Separately, pdoc inlines the full source of every symbol, which put `langfuse.html` at 2.7 MB, over the crawler's 2 MB limit and slow for readers. - `pdoc-templates/index.html.jinja2` sets `root_module_name` to false, the escape hatch pdoc's own template documents, and fills the main column, so `/` is a real landing page: title, description, canonical, and links to `langfuse`, `langfuse.experiment` and `langfuse.api`. - `pdoc-templates/module.html.jinja2` adds a self-referencing canonical to every module page, without the `.html` suffix -- the site is served with clean URLs, so `/langfuse.html` 308-redirects to `/langfuse` and a canonical pointing there would point at a redirect. - `pdoc-templates/404.html` is copied into the output so unmatched paths get a real 404 instead of a 200. - `--no-show-source` takes `langfuse.html` from 2.7 MB to 484 KB. `--edit-url` replaces the inline source with a link to the module on GitHub. `scripts/build_reference_docs.sh` wraps all of this, so the published site cannot be built correctly by accident; CONTRIBUTING.md and AGENTS.md now point at the script instead of the raw pdoc command. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

github-actions · 2026-09-04T08:53:32Z

@claude review

claude

Claude Code Review

No review was started: this request came from a bot account. Manual reviews can only be requested by someone with write access to this repository. Ask a maintainer to comment @claude review, or have your automation post the comment from a user account with write access.

_{Tip: disable this comment in your organization's Code Review settings.}

claude

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

…ailing slash Review follow-ups on the reference-site build: - `PDOC_CANONICAL_BASE_URL` was concatenated straight onto the module path, so an origin given without a trailing slash (`https://staging.example.com`, a perfectly valid form) produced `https://staging.example.comlangfuse` on every module page -- the same class of broken canonical this change exists to fix. The base URL is now normalized to exactly one trailing slash. - `OUT_DIR` was captured before the script cd'd to the repo root, so a relative output path resolved against the repo instead of the caller's directory. Relative paths are now made absolute first. - Recorded why the landing-page links keep their `.html` suffix, so the next reader does not mistake it for an oversight. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

jannikmaierhoefer · 2026-09-04T09:47:38Z

Addressed the review in aa1ba78 — two of the three suggestions taken, one declined with reasoning.

Fixed: canonical base URL could merge into the path. Both reviewers flagged this and they were right. PDOC_CANONICAL_BASE_URL was concatenated straight onto the module path, so an origin without a trailing slash produced a broken canonical on every module page — exactly the class of problem this PR exists to fix. Normalized to one trailing slash in both templates. Verified against the failing input:

$ PDOC_CANONICAL_BASE_URL="https://staging.example.com" bash scripts/build_reference_docs.sh out
index.html                 → https://staging.example.com/
langfuse.html              → https://staging.example.com/langfuse
langfuse/experiment.html   → https://staging.example.com/langfuse/experiment

Before the fix the last two would have been https://staging.example.comlangfuse….

Fixed: relative OUT_DIR resolved against the repo root. OUT_DIR was captured before the cd, so a relative path silently wrote inside the repo. Relative paths are now made absolute first. Verified:

$ cd /tmp/relcheck && bash /repo/scripts/build_reference_docs.sh out
landed in the caller's directory:  yes
leaked into <repo>/out:            no

Declined: landing-page links keeping the .html suffix. The observation is correct — those three links do take a 308 hop in production — but switching them to clean URLs costs more than it saves:

Extensionless paths 404 locally. Confirmed against the built output: /langfuse/experiment → 404, /langfuse/experiment.html → 200. The same applies to pdoc's dev server, which is the local workflow CONTRIBUTING.md documents.
pdoc generates its own 49 sidebar links as {{ submodule }}.html. Changing only the three links in my content block would make them inconsistent with every other link on the page while leaving the redirect-hop pattern in place everywhere else, so it does not actually remove the issue.

A 308 is cached and passes full signal, so one hop on an internal link is cheap; a canonical pointing at a redirect is not, which is why the canonicals do drop the suffix. Added a comment in the template recording that distinction so it does not read as an oversight.

Re-verified after the changes: 50/50 pages have a title and a clean-URL canonical, 404.html present, langfuse.html at 481 KB, ruff check and bash -n clean.

claude Bot reviewed Sep 4, 2026

View reviewed changes

jannikmaierhoefer mentioned this pull request Sep 4, 2026

fix(seo): resolve the in-repo Ahrefs September 2026 site-audit errors langfuse/langfuse-docs#3716

Merged

greptile-apps Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread pdoc-templates/module.html.jinja2

claude Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread pdoc-templates/module.html.jinja2 Outdated

Comment thread pdoc-templates/index.html.jinja2

Comment thread scripts/build_reference_docs.sh

jannikmaierhoefer changed the title ~~fix(docs): make the generated reference site indexable~~ fix(docs): SEO improvements Sep 4, 2026

Update CONTRIBUTING.md

31e42cb

jannikmaierhoefer enabled auto-merge (squash) September 4, 2026 11:31

jannikmaierhoefer merged commit 254c090 into main Sep 4, 2026
19 checks passed

jannikmaierhoefer deleted the seo/pdoc-reference-site branch September 4, 2026 11:34

Navigation Menu

Uh oh!

fix(docs): SEO improvements - #1858

fix(docs): SEO improvements#1858
jannikmaierhoefer merged 3 commits into
mainfrom
seo/pdoc-reference-site

jannikmaierhoefer commented Sep 4, 2026 •

edited by greptile-apps Bot

Loading

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

claude Bot left a comment

Uh oh!

Uh oh!

claude Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jannikmaierhoefer commented Sep 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jannikmaierhoefer commented Sep 4, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's wrong

The fix

Verification

Deploying this

Greptile Summary

Confidence Score: 4/5

Flowchart

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jannikmaierhoefer commented Sep 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jannikmaierhoefer commented Sep 4, 2026 •

edited by greptile-apps Bot

Loading