Add nbversion, which finds the cells that differ between 3.14 and 3.15

CONTRIBUTING.md

@@ -38,6 +38,25 @@ Write to one reader who knows Python and has never seen a struct. Say the thing,
 
 Numbers come from scripts, never from memory. If a paragraph says the small integer cache holds 1030 values, that number is interpolated from generated output rather than typed by a person.
 
+## The 3.14 and 3.15 problem
+
+Everything here is written against the pinned 3.15. Every reader who clicks a Colab badge is on 3.14, and so is every widget that runs in the browser, because Pyodide has not shipped 3.15 yet. That gap is not going away before the first milestone does, so lessons have to be honest about it rather than wait for it.
+
+`just versions` runs every lesson on both interpreters and compares the output of every cell. A cell whose output differs has to say so, and a cell that says so has to actually differ. Both halves are checked, because a note that has stopped being true is worse than no note: a reader who checks one against their own interpreter, finds it wrong, and decides the notes are decoration has been misled by the thing meant to help them.
+
+Declaring a difference is one keyword in the lesson's `build.py`:
+
+```python
+lesson.code(
+    source,
+    differs="On 3.14 the last instruction is LOAD_CONST rather than LOAD_COMMON_CONSTANT, and None is in co_consts.",
+)
+```
+
+Say what the reader is looking at and what the other version does instead. "This differs on 3.14" is not a note, it is an apology. Add `quiet=True` when a paragraph near the top of the lesson already explains a difference that then turns up in a dozen cells, so the same sentence is not repeated under every one of them.
+
+When the differing cell is the lesson's central observation, the note is not enough. Either the lesson gets a short section explaining both versions, because the difference is itself worth teaching, or the example changes to one that behaves the same on both. Which of the two depends on whether the difference is interesting. `LOAD_COMMON_CONSTANT` is interesting and gets explained. A line number inside `asyncio` is not, and the cell should stop printing it.
+
 ## Definition of done for a lesson
 
 No partial credit on any of these.
@@ -51,6 +70,7 @@ No partial credit on any of these.
 7. The blueprint fragment complete for its declared status
 8. A diagram or animation with alt text written by a person
 9. Three beginner testers have completed it
+10. Every cell whose output depends on the interpreter version declared, so `just versions` is green
 
 ## Filing things
 

README.md

@@ -54,6 +54,7 @@ Pinned to `v3.15.0rc1` today and moving to `v3.15.0` when it ships on 1 October
 | `nbcheck` | The rules a lesson notebook has to follow, checked before review rather than after: the Colab badge points at itself, the build banner runs before anything it could explain, no code cell appears without a sentence introducing it, and no outputs are committed | [tools/nbcheck](tools/nbcheck) |
 | `nbbuild` | Lessons are written as Python and generated into notebooks, because nobody should have to edit a `.ipynb` by hand or review a diff of one. The generated file is committed as well, and CI fails if it stops matching the code that produced it | [tools/nbbuild](tools/nbbuild) |
 | `nbdiagram` | Every picture in a lesson is an Excalidraw scene drawn from Python, written out as an editable `.excalidraw` and as the `.svg` GitHub and Colab display. Colours, type and spacing come from one shared theme, so the diagrams, the charts and the animations look like one project | [tools/nbdiagram](tools/nbdiagram) |
+| `nbversion` | The lessons are written against 3.15 and every reader in Colab or in a browser widget is on 3.14. This runs all of them on both, compares the output cell by cell, and fails when a cell that differs has no note saying so, or carries a note that stopped being true | [tools/nbversion](tools/nbversion) |
 | `bpcheck` | The shape a blueprint has to have before somebody can implement from it: the nine sections in order, the header block, the invariant numbering, and no fact deferred to a lesson | [tools/bpcheck](tools/bpcheck) |
 | `bpc` | The blueprint compiler. Where upstream ships the material in a form a program can read, the specification is generated from it rather than typed. It reads `Parser/Python.asdl` with CPython's own parser and writes the three sections of BP-AST that list all 113 node kinds, each one citing the line it is declared on | [tools/bpc](tools/bpc) |
 | `xraymanim` | The animations, and the fifteen shapes they are allowed to be made of. Each one is planned as a storyboard that is checked in milliseconds, so a mistake is caught before anybody pays for a render | [xraymanim](xraymanim) |

justfile

@@ -3,6 +3,7 @@
 # slowest possible feedback loop and the reason the two drift apart.
 
 pinned_tag := "v3.15.0rc1"
+pinned_version := "3.15"
 cpython_src := env("CPYTHON_SRC", "vendor/cpython")
 
 default:
@@ -101,6 +102,19 @@ notebooks:
     uv run nbcheck lint
     uv run nbcheck run
 
+# Run every lesson on both interpreters and check that the cells whose output differs are
+# the ones declared as differing. This is the slowest recipe here by a distance, because it
+# executes every notebook twice, so it is not part of `check`. CI does it in the two
+# notebook jobs it already runs and compares the results afterwards, which costs it nothing.
+versions:
+    #!/usr/bin/env bash
+    set -euo pipefail
+    rm -rf build/versions
+    uv run nbversion record --into build/versions
+    UV_PROJECT_ENVIRONMENT=/tmp/venv-314 uv run --python 3.14 --all-packages \
+        nbversion record --into build/versions
+    uv run nbversion compare build/versions/{{pinned_version}} build/versions/3.14
+
 # The structural checks on their own, with no kernel, for while you are still writing.
 notebooks-lint:
     uv run nbcheck lint

pyproject.toml

@@ -12,6 +12,7 @@ dependencies = [
     "nbbuild",
     "nbcheck",
     "nbdiagram",
+    "nbversion",
     "pyxray",
     "refcheck",
     "xraymanim",
@@ -42,6 +43,7 @@ members = [
     "tools/nbbuild",
     "tools/nbcheck",
     "tools/nbdiagram",
+    "tools/nbversion",
     "tools/refcheck",
     "xraymanim",
     "xraywidgets",
@@ -53,6 +55,7 @@ bpcheck = { workspace = true }
 nbbuild = { workspace = true }
 nbcheck = { workspace = true }
 nbdiagram = { workspace = true }
+nbversion = { workspace = true }
 pyxray = { workspace = true }
 refcheck = { workspace = true }
 xraymanim = { workspace = true }
@@ -66,6 +69,7 @@ testpaths = [
     "tools/nbbuild/tests",
     "tools/nbcheck/tests",
     "tools/nbdiagram/tests",
+    "tools/nbversion/tests",
     "tools/refcheck/tests",
     "xraymanim/tests",
     "xraywidgets/tests",
@@ -97,6 +101,7 @@ known-first-party = [
     "nbbuild",
     "nbcheck",
     "nbdiagram",
+    "nbversion",
     "pyxray",
     "refcheck",
     "xraymanim",

tools/nbbuild/pyproject.toml

@@ -4,7 +4,7 @@ version = "0.1.0"
 description = "Build the lesson notebooks from the Python files that define them"
 requires-python = ">=3.14"
 license = "MIT"
-dependencies = ["pyxray"]
+dependencies = ["nbversion", "pyxray"]
 
 [project.scripts]
 nbbuild = "nbbuild.cli:main"
@@ -17,4 +17,5 @@ build-backend = "hatchling.build"
 packages = ["src/nbbuild"]
 
 [tool.uv.sources]
+nbversion = { workspace = true }
 pyxray = { workspace = true }

tools/nbbuild/src/nbbuild/lesson.py

@@ -23,6 +23,7 @@
 from dataclasses import dataclass, field
 from pathlib import Path
 
+from nbversion.declare import KEY, NAMESPACE
 from pyxray.cite import markdown as cite_markdown
 from pyxray.glossary import link as glossary_link
 
@@ -35,6 +36,11 @@
 #: almost indistinguishable from a hyphen in a diff, which is exactly why they get in.
 BANNED = (("\u2014", "em dash"), ("\u2013", "en dash"))
 
+#: How a version note reads to somebody running the lesson. Short, and at the top of the
+#: cell's own output area rather than at the end of the section, because a reader who is
+#: comparing what they got against what the lesson says is looking right there.
+VERSION_NOTE = "> **Version note.** {text}"
+
 
 def repository_root(start: Path | None = None) -> Path:
     """The top of the checkout, found by looking for the workspace pyproject.
@@ -120,24 +126,50 @@ def _add(self, kind: str, text: str, extra: dict) -> None:
         # would mean opening a lesson in Jupyter and saving it reorders the whole file.
         self.cells.append(dict(sorted(cell.items())))
 
-    def md(self, text: str) -> None:
-        """A prose cell.
+    def _forbid(self, text: str) -> None:
+        """Two of the project's writing rules, checked here rather than in review.
 
-        Two of the project's writing rules are checked here rather than in review, because
-        both are invisible in a diff and neither has ever been caught by a human.
+        Both are invisible in a diff and neither has ever been caught by a human.
         """
         for character, name in BANNED:
             if character in text:
                 raise Malformed(f"cell {len(self.cells) + 1} contains an {name}")
+
+    def md(self, text: str) -> None:
+        """A prose cell."""
+        self._forbid(text)
         self._add("markdown", text, {})
 
-    def code(self, text: str) -> None:
+    def code(self, text: str, *, differs: str = "", quiet: bool = False) -> None:
         """A code cell, with no outputs and no execution count.
 
         Outputs are never committed. The only proof a cell works is CI executing it, and a
         stored output is a screenshot that goes stale without telling anybody.
+
+        `differs` is for the cells that print something different depending on which Python
+        is running. The lessons are written against the pinned 3.15 and a reader in Colab or
+        in a WASM widget is on 3.14, so a handful of cells show them something that is true
+        of neither the lesson nor their own interpreter unless somebody says so. Passing the
+        sentence here does two things: it goes in the cell's metadata, where `nbversion
+        compare` checks it against what the two interpreters actually printed, and it comes
+        out underneath the cell as a note the reader can see.
+
+        `quiet` turns off that second half, for the lessons where one paragraph near the top
+        already explains a difference that then shows up in a dozen cells. Repeating it
+        under every one of them would train the reader to skip the notes, which is the
+        opposite of what they are for.
         """
-        self._add("code", text, {"execution_count": None, "outputs": []})
+        extra = {"execution_count": None, "outputs": []}
+        if not differs:
+            self._add("code", text, extra)
+            return
+        # Checked before either cell is added, so a rejected note does not leave half of
+        # itself behind in a lesson somebody is building interactively.
+        self._forbid(differs)
+        extra["metadata"] = {NAMESPACE: {KEY: differs}}
+        self._add("code", text, extra)
+        if not quiet:
+            self.md(VERSION_NOTE.format(text=differs))
 
     def document(self) -> str:
         """The finished notebook as the exact text that belongs on disk."""

tools/nbbuild/tests/test_nbbuild_lesson.py

@@ -5,6 +5,7 @@
 import pytest
 
 from nbbuild import Lesson, Malformed
+from nbversion.declare import KEY, NAMESPACE
 
 
 @pytest.fixture
@@ -44,6 +45,43 @@ def test_a_code_cell_carries_no_output_and_no_execution_count(root):
     assert cell["execution_count"] is None
 
 
+def test_a_plain_code_cell_carries_no_version_note(root):
+    lesson = Lesson("t99-example", "t99", root=root)
+    lesson.code("print(1)")
+    assert json.loads(lesson.document())["cells"][0]["metadata"] == {}
+
+
+def test_a_version_note_goes_in_the_cells_own_metadata(root):
+    lesson = Lesson("t99-example", "t99", root=root)
+    lesson.code("print(1)", differs="On 3.14 this prints nothing.")
+    cell = json.loads(lesson.document())["cells"][0]
+    assert cell["metadata"] == {NAMESPACE: {KEY: "On 3.14 this prints nothing."}}
+
+
+def test_a_version_note_also_comes_out_as_something_the_reader_can_see(root):
+    """The metadata is for CI. A reader on Colab never opens it, so the note is said twice."""
+    lesson = Lesson("t99-example", "t99", root=root)
+    lesson.code("print(1)", differs="On 3.14 this prints nothing.")
+    cells = json.loads(lesson.document())["cells"]
+    assert [cell["cell_type"] for cell in cells] == ["code", "markdown"]
+    assert cells[1]["source"] == ["> **Version note.** On 3.14 this prints nothing."]
+
+
+def test_a_quiet_version_note_is_declared_without_a_cell_under_it(root):
+    """For the lessons where one paragraph up top covers a difference a dozen cells show."""
+    lesson = Lesson("t99-example", "t99", root=root)
+    lesson.code("print(1)", differs="Offsets are 2 lower on 3.14.", quiet=True)
+    cells = json.loads(lesson.document())["cells"]
+    assert [cell["cell_type"] for cell in cells] == ["code"]
+    assert cells[0]["metadata"] == {NAMESPACE: {KEY: "Offsets are 2 lower on 3.14."}}
+
+
+def test_a_version_note_goes_through_the_same_punctuation_check_as_the_prose(root):
+    lesson = Lesson("t99-example", "t99", root=root)
+    with pytest.raises(Malformed, match="em dash"):
+        lesson.code("print(1)", differs="On 3.14 \u2014 nothing.")
+
+
 def test_source_keeps_its_newlines_the_way_the_format_wants_them(root):
     lesson = Lesson("t99-example", "t99", root=root)
     lesson.code("one\ntwo")

tools/nbversion/README.md

@@ -0,0 +1,60 @@
+# nbversion
+
+Finds the lesson cells whose output depends on which Python is running, and checks that every one of them is declared. Run with `just versions`, which records the lessons twice and compares the two.
+
+```
+uv run nbversion record --into build/versions
+uv run nbversion compare build/versions/3.15 build/versions/3.14
+```
+
+## Why this exists
+
+The lessons are written against the pinned CPython 3.15. A reader who clicks the Colab badge is on whatever Google installed, which is 3.14, and so is every WASM widget, because Pyodide has not shipped 3.15 yet. Most cells do not notice. Some do, and those are the dangerous ones, because the cell still runs and still prints something that looks right.
+
+The one that started this is `LOAD_COMMON_CONSTANT`. On 3.15 a function that falls off the end loads `None` with that instruction and `co_consts` does not contain `None` at all. On 3.14 it is a `LOAD_CONST` and `None` is in the table. A lesson that says "look, `co_consts` is `(6,)`" is teaching the reader to read their own screen wrong, and nothing about the cell says so.
+
+So the lessons are executed on both interpreters and the outputs compared. Anything that differs has to carry a note.
+
+## The two halves
+
+`record` runs on one interpreter and writes a small JSON file per notebook: cell id to normalised output. It is not an executed notebook, because the diff of two executed notebooks is mostly metadata.
+
+`compare` reads two of those directories and produces one of four verdicts per cell.
+
+| verdict | what it means | fails |
+| --- | --- | --- |
+| `declared` | the cell differs and the notebook says so | no |
+| `undeclared` | the cell differs and nothing says so | yes |
+| `stale` | the cell carries a note and the two interpreters now agree | yes |
+| `missing` | the two runs saw different sets of cells | yes |
+
+`stale` is the half people forget. A note that has stopped being true is worse than no note: a reader who checks one against their own interpreter, finds it wrong, and concludes the notes are decoration has been misled by the thing that was supposed to help.
+
+## Declaring a difference
+
+In the lesson's `build.py`:
+
+```python
+lesson.code(
+    "import dis\ndis.dis(compile('answer = 6 * 7', '<lesson>', 'exec'))\n",
+    differs="On 3.14 the last instruction is LOAD_CONST rather than LOAD_COMMON_CONSTANT, and None is in co_consts.",
+)
+```
+
+That writes the sentence into the cell's metadata, where `compare` looks for it, and adds a markdown cell underneath so a reader on Colab sees it without opening the metadata. Pass `quiet=True` to skip the visible cell, for the lessons where one paragraph near the top already covers a difference that a dozen cells then show. Repeating it under every one of them teaches people to skip the notes.
+
+The metadata looks like this, and survives a round trip through Jupyter because it is on the cell rather than in a list somewhere else:
+
+```json
+"metadata": {"cpython_internals": {"differs": "On 3.14 ..."}}
+```
+
+## Normalising
+
+Every substitution in `normalise.py` throws away a real difference, and the differences worth finding are exactly the ones a careless normaliser sweeps up. So a pattern gets normalised only when it varies between two runs of the *same* interpreter, which makes it noise rather than a version difference. That is addresses, absolute paths, temporary file names and durations, and nothing else. An opcode name, a size, a byte count and an offset all survive, because those are the point.
+
+Errors keep the exception type and the message and lose the traceback. A lesson that raises on purpose cares about which exception it got, not about how many frames were on the stack.
+
+## What it does not do
+
+It does not decide what a lesson should say about a difference. It can tell you `co_consts` differs, and only a person can tell a reader why. It does not run the lessons on Pyodide, which is a separate question tracked in the issue this tool came from. And it does not replace `nbcheck run`, which is what fails when a cell raises by accident. `record` deliberately keeps going past an exception, so that one broken lesson cannot hide every version difference in the lessons after it.

tools/nbversion/pyproject.toml

@@ -0,0 +1,25 @@
+[project]
+name = "nbversion"
+version = "0.1.0"
+description = "Find the lesson cells whose output depends on which Python is running"
+requires-python = ">=3.14"
+license = "MIT"
+dependencies = [
+    "nbcheck",
+    "nbformat>=5.10",
+    "nbclient>=0.10",
+    "ipykernel>=6.29",
+]
+
+[project.scripts]
+nbversion = "nbversion.cli:main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/nbversion"]
+
+[tool.uv.sources]
+nbcheck = { workspace = true }

tools/nbversion/src/nbversion/__init__.py

@@ -0,0 +1,33 @@
+"""Finding the lesson cells whose output depends on which Python is running.
+
+The lessons are written against a pinned CPython 3.15, and a reader who opens one in Colab
+or clicks a WASM widget is on 3.14, because that is what those runtimes ship. Most cells
+do not care. A few do, and those are the dangerous ones: the cell runs, prints something
+plausible, and quietly teaches the reader a fact about the wrong interpreter.
+
+So the lessons are executed on both versions and the outputs compared. A cell that differs
+has to carry a note saying so, and a note has to correspond to a cell that really differs.
+Neither half is worth much without the other.
+"""
+
+from .compare import Finding, cells, notebooks, summary
+from .declare import KEY, NAMESPACE, note, notes
+from .normalise import outputs, text
+from .record import Recording, run, version
+
+__all__ = [
+    "KEY",
+    "NAMESPACE",
+    "Finding",
+    "Recording",
+    "cells",
+    "note",
+    "notebooks",
+    "notes",
+    "outputs",
+    "run",
+    "summary",
+    "text",
+    "version",
+]
+__version__ = "0.1.0"