Skip to content

feat(test): docstring examples run as tests - #21

Merged
jmineau merged 1 commit into
mainfrom
feat/doctests
Oct 22, 2026
Merged

jmineau merged 1 commit into
mainfrom
feat/doctests

Conversation

@jmineau

@jmineau jmineau commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Runs the >>> examples in each package's docstrings as tests, through the template: [tool.pytest] gets --doctest-modules and src in testpaths. James agreed to this on his triage board on 2026-10-20 (card python-template-doctests).

Why. Docstring examples never ran, so they went stale unnoticed: in fips 1 of 45 was wrong (jmineau/fips#96; fixed in jmineau/fips#107). Putting it in the template means every package gets it by copier update, and each package fixes or skips the examples that need data CI lacks.

What changes

  • template/pyproject.toml.jinja: testpaths = ["tests", "src"] and addopts = ["-ra", "--doctest-modules"]. Nothing else in [tool.pytest] moves (strict markers, error warning filter, network/slow markers).
  • template/AGENTS.md.jinja, Tests section: one bullet saying the examples run as tests and that one needing data CI lacks gets # doctest: +SKIP on its >>> line.
  • template/CONTRIBUTING.md.jinja (the just test row), README.md (Tests row) and CONTRIBUTING.md (fixtures paragraph): one line each.
  • The template's own just test now also checks, in each bake, that pytest --collect-only lists the fixture's docstring example (src/testpackage/plotting.py::testpackage.plotting.square) and tests/test_pkg.py::test_version.

Checked (python-template just test on a compute node: both bakes, docs_notebooks and publish_to_pypi yes and no; in each, quality-check, build-docs, every pre-commit hook including zizmor, dist, the version checks)

  • just test passed: "All checks passed for both variants."
  • In the baked project, just test (parallel, 8 xdist workers) reports 2 passed where the template used to report 1: tests/test_pkg.py::test_version and the doctest src/testpackage/plotting.py::testpackage.plotting.square (-v shows both, in both bakes). Plain uv run pytest -v (serial) collects the same two.
  • just cov (what the Tests workflow runs) in a bake: 2 passed, plotting.py at 100% coverage from the doctest.
  • A module with a wrong example (>>> double(2) expecting 5) fails just test with the usual expected/got report and exit code 1; with the expected output corrected, and a second example marked # doctest: +SKIP, just test passes with 3 passed, 1 skipped.

Packages that will need their examples fixed or skipped when they update (copier update was not run on any; the examples were not run either). Counts of >>> lines under src/ on each package's origin/main today, read-only:

Package >>> lines in files
fips 100 6
PYSTILT 78 17
arl-met 70 10
uataq 12 3
slv 6 2
lair 0 0
pystiltctl 0 0

fips#96 is the known one. None of them has a +SKIP yet. A package with its own [tool.pytest] testpaths or addopts will get a conflict on pyproject.toml when it updates, as with any other change to that table.

Not checked. copier update on any package, and whether their examples pass. Pytest's ELLIPSIS and NORMALIZE_WHITESPACE flags are not turned on (an example that needs one writes # doctest: +ELLIPSIS); I kept to what the card said.

Other open python-template PRs: #20 edits template/AGENTS.md.jinja too, in the "Examples in the docs" section; this PR edits its Tests section. I merged the two branches in a scratch test merge: no conflict. The monthly uv lock --upgrade workflow PR (#22) touches README.md and template/CONTRIBUTING.md.jinja in other places: also no conflict.

pytest runs the `>>>` examples in the docstrings under src/ with
--doctest-modules, next to the tests folder (testpaths, addopts in
[tool.pytest]), so `just test`, `just cov` and CI check them. An example that
needs data CI lacks is skipped with `# doctest: +SKIP`.

`just test` (the template's own) checks that pytest collects the fixture's
docstring example and the tests folder in each bake. AGENTS.md, CONTRIBUTING.md
and the README say so.
@jmineau
jmineau merged commit ad10eb1 into main Oct 22, 2026
1 check passed
@jmineau
jmineau deleted the feat/doctests branch October 22, 2026 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant