Repository navigation
feat(test): docstring examples run as tests - #21
Merged
Merged
Conversation
pytest runs the `>>>` examples in the docstrings under src/ with --doctest-modules, next to the tests folder (testpaths, addopts in [tool.pytest]), so `just test`, `just cov` and CI check them. An example that needs data CI lacks is skipped with `# doctest: +SKIP`. `just test` (the template's own) checks that pytest collects the fixture's docstring example and the tests folder in each bake. AGENTS.md, CONTRIBUTING.md and the README say so.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs the
>>>examples in each package's docstrings as tests, through the template:[tool.pytest]gets--doctest-modulesandsrcintestpaths. James agreed to this on his triage board on 2026-10-20 (cardpython-template-doctests).Why. Docstring examples never ran, so they went stale unnoticed: in fips 1 of 45 was wrong (jmineau/fips#96; fixed in jmineau/fips#107). Putting it in the template means every package gets it by
copier update, and each package fixes or skips the examples that need data CI lacks.What changes
template/pyproject.toml.jinja:testpaths = ["tests", "src"]andaddopts = ["-ra", "--doctest-modules"]. Nothing else in[tool.pytest]moves (strict markers,errorwarning filter,network/slowmarkers).template/AGENTS.md.jinja, Tests section: one bullet saying the examples run as tests and that one needing data CI lacks gets# doctest: +SKIPon its>>>line.template/CONTRIBUTING.md.jinja(thejust testrow),README.md(Tests row) andCONTRIBUTING.md(fixtures paragraph): one line each.just testnow also checks, in each bake, thatpytest --collect-onlylists the fixture's docstring example (src/testpackage/plotting.py::testpackage.plotting.square) andtests/test_pkg.py::test_version.Checked (python-template
just teston a compute node: both bakes,docs_notebooksandpublish_to_pypiyes and no; in each,quality-check,build-docs, every pre-commit hook including zizmor,dist, the version checks)just testpassed: "All checks passed for both variants."just test(parallel, 8 xdist workers) reports2 passedwhere the template used to report 1:tests/test_pkg.py::test_versionand the doctestsrc/testpackage/plotting.py::testpackage.plotting.square(-vshows both, in both bakes). Plainuv run pytest -v(serial) collects the same two.just cov(what the Tests workflow runs) in a bake: 2 passed,plotting.pyat 100% coverage from the doctest.>>> double(2)expecting5) failsjust testwith the usual expected/got report and exit code 1; with the expected output corrected, and a second example marked# doctest: +SKIP,just testpasses with 3 passed, 1 skipped.Packages that will need their examples fixed or skipped when they update (
copier updatewas not run on any; the examples were not run either). Counts of>>>lines undersrc/on each package'sorigin/maintoday, read-only:>>>linesfips#96 is the known one. None of them has a
+SKIPyet. A package with its own[tool.pytest]testpathsoraddoptswill get a conflict onpyproject.tomlwhen it updates, as with any other change to that table.Not checked.
copier updateon any package, and whether their examples pass. Pytest'sELLIPSISandNORMALIZE_WHITESPACEflags are not turned on (an example that needs one writes# doctest: +ELLIPSIS); I kept to what the card said.Other open python-template PRs: #20 edits
template/AGENTS.md.jinjatoo, in the "Examples in the docs" section; this PR edits its Tests section. I merged the two branches in a scratch test merge: no conflict. The monthlyuv lock --upgradeworkflow PR (#22) touchesREADME.mdandtemplate/CONTRIBUTING.md.jinjain other places: also no conflict.