From 56082f117b8b2ca247334461b0041a0526c9affa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jannik=20Maierh=C3=B6fer?= Date: Fri, 4 Sep 2026 10:52:34 +0200 Subject: [PATCH 1/3] fix(docs): make the generated reference site indexable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four of the errors in the September Ahrefs Site Audit of langfuse.com land on python.reference.langfuse.com, and all four come from how pdoc's output is generated rather than from the SDK. pdoc writes an `index.html` that is only a `` stub, because a single root module makes `langfuse.html` the entry point. That stub has no title, no links and no canonical URL, so `/` and `/index.html` read as duplicate, soft-redirecting pages. Worse, the host serves that same stub with a `200` for *any* unmatched path when the output contains no `404.html`, so every stale or mistyped URL becomes another indexable copy of it -- which is how two nonexistent `…/concepts/models.md` paths ended up in the crawl. Separately, pdoc inlines the full source of every symbol, which put `langfuse.html` at 2.7 MB, over the crawler's 2 MB limit and slow for readers. - `pdoc-templates/index.html.jinja2` sets `root_module_name` to false, the escape hatch pdoc's own template documents, and fills the main column, so `/` is a real landing page: title, description, canonical, and links to `langfuse`, `langfuse.experiment` and `langfuse.api`. - `pdoc-templates/module.html.jinja2` adds a self-referencing canonical to every module page, without the `.html` suffix -- the site is served with clean URLs, so `/langfuse.html` 308-redirects to `/langfuse` and a canonical pointing there would point at a redirect. - `pdoc-templates/404.html` is copied into the output so unmatched paths get a real 404 instead of a 200. - `--no-show-source` takes `langfuse.html` from 2.7 MB to 484 KB. `--edit-url` replaces the inline source with a link to the module on GitHub. `scripts/build_reference_docs.sh` wraps all of this, so the published site cannot be built correctly by accident; CONTRIBUTING.md and AGENTS.md now point at the script instead of the raw pdoc command. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 4 +- CONTRIBUTING.md | 19 +++++++--- pdoc-templates/404.html | 63 +++++++++++++++++++++++++++++++ pdoc-templates/index.html.jinja2 | 62 ++++++++++++++++++++++++++++++ pdoc-templates/module.html.jinja2 | 15 ++++++++ scripts/build_reference_docs.sh | 28 ++++++++++++++ 6 files changed, 185 insertions(+), 6 deletions(-) create mode 100644 pdoc-templates/404.html create mode 100644 pdoc-templates/index.html.jinja2 create mode 100644 pdoc-templates/module.html.jinja2 create mode 100755 scripts/build_reference_docs.sh diff --git a/AGENTS.md b/AGENTS.md index 5b463527d..1d82ae5dd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -154,9 +154,11 @@ If you change CI bootstrap: ```bash uv build --no-sources -uv run --group docs pdoc -o docs/ --docformat google --logo "https://langfuse.com/langfuse_logo.svg" langfuse +bash scripts/build_reference_docs.sh ``` +Build the reference through that script, not by calling pdoc directly -- it applies the `pdoc-templates/` overrides and the 404 page that the hosted site needs. See "SDK Reference" in `CONTRIBUTING.md`. + Releases are handled by GitHub Actions. Do not build an ad hoc local release flow into repository instructions. ## External Docs diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 45f1ed55d..32bccca81 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -125,18 +125,27 @@ The workflow will automatically: Note: The generated SDK reference is currently work in progress. -The SDK reference is generated via pdoc. The docs dependency group is installed on demand when you run the documentation commands. +The SDK reference is generated via pdoc and published at [python.reference.langfuse.com](https://python.reference.langfuse.com). The docs dependency group is installed on demand when you run the documentation commands. -To update the reference, run the following command: +To build the reference into `docs/`, run: ```sh -uv run --group docs pdoc -o docs/ --docformat google --logo "https://langfuse.com/langfuse_logo.svg" langfuse +bash scripts/build_reference_docs.sh ``` -To run the reference locally, you can use the following command: +Build it through the script rather than calling pdoc directly. The script applies the template overrides in `pdoc-templates/` and copies the 404 page, both of which the hosted site needs: + +- `pdoc-templates/index.html.jinja2` replaces pdoc's default `index.html`, which is a bare `` stub with no title, no links and no canonical URL, with a real landing page. +- `pdoc-templates/module.html.jinja2` adds a self-referencing canonical URL to every module page. The `.html` suffix is dropped on purpose: the site is served with clean URLs, so `/langfuse.html` 308-redirects to `/langfuse`. +- `pdoc-templates/404.html` is copied into the output because the host serves the landing page with a `200` for any unmatched path when no `404.html` is present, which makes every stale URL an indexable duplicate. +- `--no-show-source` keeps `langfuse.html` at roughly 480 KB instead of 2.7 MB. `--edit-url` puts a link to the module's source on GitHub on each page instead. + +Set `PDOC_CANONICAL_BASE_URL` to build for an origin other than `https://python.reference.langfuse.com/`. + +To browse the reference locally with live reload, run pdoc's dev server: ```sh -uv run --group docs pdoc --docformat google --logo "https://langfuse.com/langfuse_logo.svg" langfuse +uv run --group docs pdoc --docformat google --logo "https://langfuse.com/langfuse_logo.svg" --template-directory pdoc-templates langfuse ``` ## Credits diff --git a/pdoc-templates/404.html b/pdoc-templates/404.html new file mode 100644 index 000000000..39192a777 --- /dev/null +++ b/pdoc-templates/404.html @@ -0,0 +1,63 @@ + + + + + + + Page not found – Langfuse Python SDK API reference + + + +
+ Langfuse +

Page not found

+

+ This page is not part of the Langfuse Python SDK API reference. The + symbol may have been renamed or removed in a later release. +

+

+ Browse the API reference · + Langfuse documentation +

+
+ + diff --git a/pdoc-templates/index.html.jinja2 b/pdoc-templates/index.html.jinja2 new file mode 100644 index 000000000..cf9a020f3 --- /dev/null +++ b/pdoc-templates/index.html.jinja2 @@ -0,0 +1,62 @@ +{# +Root page of https://python.reference.langfuse.com. + +pdoc's default index is a bare `` stub with no +title, no links and no canonical URL, because a single root module makes +`langfuse.html` the entry point. Search engines read that stub as a +soft-redirecting duplicate of `/index.html`. Setting `root_module_name` to +false -- the escape hatch the default template documents -- renders the real +module list instead, and the `content` block below gives that page something +to say, since pdoc's own version leaves the main column empty. +#} +{% set canonical_base_url = env.get("PDOC_CANONICAL_BASE_URL", "https://python.reference.langfuse.com/") %} +{% set root_module_name = false %} +{% extends "default/index.html.jinja2" %} + +{% block title %}Langfuse Python SDK API reference{% endblock %} + +{% block head %} + + +{% endblock %} + +{% block content %} +
+ {{ self.logo() }} + {% if search %} + + {% endif %} +
+
+
+

Langfuse Python SDK API reference

+

+ Generated reference for the langfuse package. It documents + every public symbol; the hand-written guides on + langfuse.com/docs are the better + starting point if you are setting Langfuse up for the first time. +

+

Start here

+
    +
  • langfuse — the + Langfuse client, tracing decorators and context helpers. + Most code only needs this module.
    • +
  • langfuse.experiment + — running experiments over datasets and scoring the results.
    • +
  • langfuse.api — the + generated low-level API client and its request and response models.
    • +
+

Elsewhere

+ +

Every module is listed in the sidebar, and the search box covers all of them.

+
+
+ {% if search %} + {% include "search.html.jinja2" %} + {% endif %} +{% endblock %} diff --git a/pdoc-templates/module.html.jinja2 b/pdoc-templates/module.html.jinja2 new file mode 100644 index 000000000..fce17db52 --- /dev/null +++ b/pdoc-templates/module.html.jinja2 @@ -0,0 +1,15 @@ +{# +Adds a self-referencing canonical URL to every module page. + +The `.html` suffix is deliberately dropped: Cloudflare Pages serves this site +with clean URLs and 308-redirects `/langfuse.html` to `/langfuse`, so a +canonical pointing at the `.html` path would point at a redirect. +#} +{% set canonical_base_url = env.get("PDOC_CANONICAL_BASE_URL", "https://python.reference.langfuse.com/") %} +{% extends "default/module.html.jinja2" %} + +{% block head %} + {{ super() }} + +{% endblock %} diff --git a/scripts/build_reference_docs.sh b/scripts/build_reference_docs.sh new file mode 100755 index 000000000..ece2209b5 --- /dev/null +++ b/scripts/build_reference_docs.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Builds the API reference published at https://python.reference.langfuse.com. +# +# Use this instead of calling pdoc directly: it applies the template overrides +# in pdoc-templates/ and ships the 404 page, both of which the hosted site +# needs. Set PDOC_CANONICAL_BASE_URL to build for a different origin. +set -euo pipefail + +OUT_DIR="${1:-docs}" + +cd "$(dirname "$0")/.." + +uv run --group docs pdoc \ + -o "$OUT_DIR" \ + --docformat google \ + --logo "https://langfuse.com/langfuse_logo.svg" \ + --logo-link "https://langfuse.com" \ + --template-directory pdoc-templates \ + --edit-url "langfuse=https://github.com/langfuse/langfuse-python/blob/main/langfuse/" \ + --no-show-source \ + langfuse + +# Cloudflare Pages serves the closest index.html with a 200 for any unmatched +# path unless the output contains a 404.html, which turns every stale or +# mistyped URL into an indexable duplicate of the landing page. +cp pdoc-templates/404.html "$OUT_DIR/404.html" + +echo "Reference docs written to $OUT_DIR/" From aa1ba78df314f20aaa25a5016eb91fe711eb669b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jannik=20Maierh=C3=B6fer?= Date: Fri, 4 Sep 2026 11:47:17 +0200 Subject: [PATCH 2/3] fix(docs): harden the reference build against a base URL without a trailing slash Review follow-ups on the reference-site build: - `PDOC_CANONICAL_BASE_URL` was concatenated straight onto the module path, so an origin given without a trailing slash (`https://staging.example.com`, a perfectly valid form) produced `https://staging.example.comlangfuse` on every module page -- the same class of broken canonical this change exists to fix. The base URL is now normalized to exactly one trailing slash. - `OUT_DIR` was captured before the script cd'd to the repo root, so a relative output path resolved against the repo instead of the caller's directory. Relative paths are now made absolute first. - Recorded why the landing-page links keep their `.html` suffix, so the next reader does not mistake it for an oversight. Co-Authored-By: Claude Opus 5 --- pdoc-templates/index.html.jinja2 | 10 +++++++++- pdoc-templates/module.html.jinja2 | 2 +- scripts/build_reference_docs.sh | 8 ++++++++ 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/pdoc-templates/index.html.jinja2 b/pdoc-templates/index.html.jinja2 index cf9a020f3..6000a8fbe 100644 --- a/pdoc-templates/index.html.jinja2 +++ b/pdoc-templates/index.html.jinja2 @@ -9,7 +9,7 @@ false -- the escape hatch the default template documents -- renders the real module list instead, and the `content` block below gives that page something to say, since pdoc's own version leaves the main column empty. #} -{% set canonical_base_url = env.get("PDOC_CANONICAL_BASE_URL", "https://python.reference.langfuse.com/") %} +{% set canonical_base_url = env.get("PDOC_CANONICAL_BASE_URL", "https://python.reference.langfuse.com/").rstrip("/") ~ "/" %} {% set root_module_name = false %} {% extends "default/index.html.jinja2" %} @@ -38,6 +38,14 @@ to say, since pdoc's own version leaves the main column empty. starting point if you are setting Langfuse up for the first time.

Start here

+ {# + These keep the `.html` suffix even though the hosted site serves + clean URLs. pdoc generates its own sidebar links that way, and + extensionless paths 404 both under pdoc's dev server and when the + built output is served locally, so the one 308 hop in production is + the cheaper trade. Canonical URLs are a different case -- those must + not point at a redirect, which is why they drop the suffix. + #}
  • langfuse — the Langfuse client, tracing decorators and context helpers. diff --git a/pdoc-templates/module.html.jinja2 b/pdoc-templates/module.html.jinja2 index fce17db52..83eef8bd4 100644 --- a/pdoc-templates/module.html.jinja2 +++ b/pdoc-templates/module.html.jinja2 @@ -5,7 +5,7 @@ The `.html` suffix is deliberately dropped: Cloudflare Pages serves this site with clean URLs and 308-redirects `/langfuse.html` to `/langfuse`, so a canonical pointing at the `.html` path would point at a redirect. #} -{% set canonical_base_url = env.get("PDOC_CANONICAL_BASE_URL", "https://python.reference.langfuse.com/") %} +{% set canonical_base_url = env.get("PDOC_CANONICAL_BASE_URL", "https://python.reference.langfuse.com/").rstrip("/") ~ "/" %} {% extends "default/module.html.jinja2" %} {% block head %} diff --git a/scripts/build_reference_docs.sh b/scripts/build_reference_docs.sh index ece2209b5..c94c46331 100755 --- a/scripts/build_reference_docs.sh +++ b/scripts/build_reference_docs.sh @@ -8,6 +8,14 @@ set -euo pipefail OUT_DIR="${1:-docs}" +# Resolve a relative output path against the caller's working directory before +# cd'ing to the repo root, so `build_reference_docs.sh out` from elsewhere does +# not silently write to /out. +case "$OUT_DIR" in + /*) ;; + *) OUT_DIR="$PWD/$OUT_DIR" ;; +esac + cd "$(dirname "$0")/.." uv run --group docs pdoc \ From 31e42cb2f01a282aa7ebb695ea093fd01c5502b6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jannik=20Maierh=C3=B6fer?= <48529566+jannikmaierhoefer@users.noreply.github.com> Date: Fri, 4 Sep 2026 13:31:09 +0200 Subject: [PATCH 3/3] Update CONTRIBUTING.md --- CONTRIBUTING.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 32bccca81..0aaf910ed 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -133,15 +133,6 @@ To build the reference into `docs/`, run: bash scripts/build_reference_docs.sh ``` -Build it through the script rather than calling pdoc directly. The script applies the template overrides in `pdoc-templates/` and copies the 404 page, both of which the hosted site needs: - -- `pdoc-templates/index.html.jinja2` replaces pdoc's default `index.html`, which is a bare `` stub with no title, no links and no canonical URL, with a real landing page. -- `pdoc-templates/module.html.jinja2` adds a self-referencing canonical URL to every module page. The `.html` suffix is dropped on purpose: the site is served with clean URLs, so `/langfuse.html` 308-redirects to `/langfuse`. -- `pdoc-templates/404.html` is copied into the output because the host serves the landing page with a `200` for any unmatched path when no `404.html` is present, which makes every stale URL an indexable duplicate. -- `--no-show-source` keeps `langfuse.html` at roughly 480 KB instead of 2.7 MB. `--edit-url` puts a link to the module's source on GitHub on each page instead. - -Set `PDOC_CANONICAL_BASE_URL` to build for an origin other than `https://python.reference.langfuse.com/`. - To browse the reference locally with live reload, run pdoc's dev server: ```sh