Measure what actually works under Pyodide, and answer gate Q1

.github/workflows/ci.yml

@@ -112,6 +112,49 @@ jobs:
           path: build/versions
       - run: uv run nbversion compare build/versions/3.15 build/versions/3.14
 
+  # Everything in the browser tier rests on Pyodide keeping the surfaces the lessons poke
+  # at, and nobody notices when one of them goes away between releases. This job asks, on
+  # every pull request, rather than the once when somebody thought to check.
+  probe:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+      - uses: actions/setup-node@v5
+        with:
+          node-version: "22"
+          cache: npm
+          cache-dependency-path: tools/wasmprobe/package-lock.json
+      - run: uv sync --all-packages
+      # The committed pair first, which is cheap and catches the common failure: somebody
+      # edited a check and did not rerun `just build-probe`, so the report and the notebook
+      # next to it now describe something else.
+      - run: uv run wasmprobe check probes/pyodide
+      - run: uv run nbcheck run probes
+      # Then the real thing. `npm ci` installs the pinned Pyodide, and the two runs below
+      # answer the question again against whatever the world looks like today rather than
+      # against a recording from whenever somebody last looked.
+      - run: npm ci
+        working-directory: tools/wasmprobe
+      # 3.14 rather than the pin, because Pyodide ships 3.14. A native 3.15 control would
+      # report every version difference as a WebAssembly one.
+      - run: uv run --python 3.14 --all-packages wasmprobe native --into build/probe
+        env:
+          # Its own environment, or this overwrites the .venv the steps above are using and
+          # the browser run below finds a 3.14 interpreter it did not ask for.
+          UV_PROJECT_ENVIRONMENT: ${{ runner.temp }}/venv-314
+      - run: uv run wasmprobe browser --into build/probe
+      - run: uv run wasmprobe report build/probe --into build/probe/report.md
+      - run: uv run wasmprobe notebook --into build/probe/probe.ipynb
+      - run: uv run wasmprobe check build/probe
+      - uses: actions/upload-artifact@v4
+        with:
+          name: probe
+          path: build/probe
+          retention-days: 7
+
   blueprints:
     runs-on: ubuntu-latest
     steps:

.gitignore

@@ -218,4 +218,7 @@ __marimo__/
 .streamlit/secrets.toml
 vendor/
 .venv/
+
+# Node, which only the WebAssembly probe driver needs
+node_modules/
 citations.lock.json.bak

CONTRIBUTING.md

@@ -59,6 +59,14 @@ There is a second keyword for the other kind of cell. `varies=` is for output th
 
 When the differing cell is the lesson's central observation, the note is not enough. Either the lesson gets a short section explaining both versions, because the difference is itself worth teaching, or the example changes to one that behaves the same on both. Which of the two depends on whether the difference is interesting. `LOAD_COMMON_CONSTANT` is interesting and gets explained. A line number inside `asyncio` is not, and the cell should stop printing it.
 
+## What the browser can and cannot do
+
+A Tier 0 experiment has to run in a browser tab with nothing installed. Which surfaces survive that is measured, not assumed, and the answer lives in [probes/pyodide](probes/pyodide): a matrix, the two raw runs behind it, a notebook you can open in Colab to ask your own runtime the same questions, and a written decision.
+
+Read `decision.md` before writing an experiment that pokes at the interpreter. Three things are known to be different today. `optimize_cfg` cannot be handed the constants list `compiler_codegen` returns, because that build does not put one there. Handing `optimize_cfg` a constants list that is too short reads past the end of memory and kills the runtime, where a native interpreter raises a tidy `ValueError`, so anything that builds one has to build it correctly rather than catch the mistake. And a thread cannot be started.
+
+If you need a surface nobody has measured, add a check to `tools/wasmprobe/src/wasmprobe/checks.py` and run `just build-probe`. A check is a string of Python that leaves its answer in `result`, and it has to import everything it uses, because the check before it may have taken the runtime down. Mark it `TIER0` if a lesson would depend on it, and `just probe` will fail the build the day it stops working. If it already fails and you have decided what to do instead, write that decision in the check's `accepted` field rather than deleting the check, so the gap stays in the report and the next regression is still visible.
+
 ## Definition of done for a lesson
 
 No partial credit on any of these.

README.md

@@ -22,6 +22,8 @@ Every chapter therefore produces two things. The chapter teaches, in prose and p
 
 **Every chapter runs in a browser.** Pyodide 314 is CPython 3.14 compiled to WebAssembly, so real bytecode, refcount, dictionary layout and garbage collection experiments work on a locked down school laptop with nothing installed. Chapters that need a debug build and a debugger ship a recorded session you can step through instead, generated in CI against a real build rather than captured by hand.
 
+That claim was measured rather than assumed. Fifteen checks run on a native CPython and inside a real WebAssembly runtime, and twelve of them behave identically: `_testinternalcapi`, `ctypes` reading a live object header, `sys.monitoring`, `sys.settrace`, the cycle collector, and the whole front end. Threads cannot start, one call has to be made slightly differently, and one bad argument crashes the runtime instead of raising. The matrix, the raw runs and the decision that came out of them are in [probes/pyodide](probes/pyodide), and CI runs the same checks on every pull request so a Pyodide release that takes something away is noticed here rather than by a reader.
+
 **CPython already contains three machine readable specifications of itself, and almost nobody teaches this.** `Grammar/python.gram` is the grammar. `Parser/Python.asdl` is the AST. `Python/bytecodes.c` is the interpreter semantics, written in a DSL that `Tools/cases_generator` compiles into the tier 1 interpreter, the tier 2 interpreter, the optimizer cases and every metadata table. CPython does not hand write its front end or its interpreter, it generates them. So the right architecture for a reimplementation is not to port the C, it is to add a backend, which is what the capstone does.
 
 **The compiler is already exposed to Python.** `_testinternalcapi` exports `compiler_codegen`, `optimize_cfg` and `assemble_code_object` on a stock interpreter. You can run the CPython compiler one stage at a time, from a notebook, and diff the control flow graph before and after optimization, with no build. This is the best teaching hook in the codebase and no existing course uses it.
@@ -57,6 +59,7 @@ Pinned to `v3.15.0rc1` today and moving to `v3.15.0` when it ships on 1 October
 | `nbversion` | The lessons are written against 3.15 and every reader in Colab or in a browser widget is on 3.14. This runs all of them on both, compares the output cell by cell, and fails when a cell that differs has no note saying so, or carries a note that stopped being true | [tools/nbversion](tools/nbversion) |
 | `bpcheck` | The shape a blueprint has to have before somebody can implement from it: the nine sections in order, the header block, the invariant numbering, and no fact deferred to a lesson | [tools/bpcheck](tools/bpcheck) |
 | `bpc` | The blueprint compiler. Where upstream ships the material in a form a program can read, the specification is generated from it rather than typed. It reads `Parser/Python.asdl` with CPython's own parser and writes the three sections of BP-AST that list all 113 node kinds, each one citing the line it is declared on | [tools/bpc](tools/bpc) |
+| `wasmprobe` | Asks a browser Python which of the surfaces the lessons depend on actually work, runs the same questions on a native interpreter for comparison, and fails the build when one of them stops working in the browser without a written decision about it | [tools/wasmprobe](tools/wasmprobe) |
 | `xraymanim` | The animations, and the fifteen shapes they are allowed to be made of. Each one is planned as a storyboard that is checked in milliseconds, so a mistake is caught before anybody pays for a render | [xraymanim](xraymanim) |
 | `xraywidgets` | The parts of a lesson you can click: a disassembler that shows what `dis` hides, a pipeline explorer with six panes from source to code object, and a prediction gate that asks before it tells. Each one renders twice from one piece of code: plain HTML with nothing installed, and the same picture with working buttons when anywidget is there | [xraywidgets](xraywidgets) |
 

justfile

@@ -27,7 +27,7 @@ vendor:
     git -C "{{cpython_src}}" rev-parse HEAD
 
 # The full local check, in the order that fails fastest.
-check: lint test citations blueprints diagrams lessons notebooks animations
+check: lint test citations blueprints diagrams lessons notebooks probe animations
 
 lint:
     uv run ruff check .
@@ -130,6 +130,27 @@ animations:
 build-animations:
     uv run --extra anim xraymanim render
 
+# Read the committed probe results and fail when a surface the lessons need has stopped
+# working in the browser, or when the report and the notebook have fallen behind the checks.
+# Milliseconds: it reads two JSON files, it does not boot anything.
+probe:
+    uv run wasmprobe check probes/pyodide
+    uv run nbcheck run probes
+
+# Record the probe again on both runtimes and rewrite the report and the notebook. Needs
+# node and `npm install` in tools/wasmprobe, and a 3.14 to compare against, because Pyodide
+# ships 3.14 and a native 3.15 control would confuse a version difference for a build one.
+#
+# There is no `nbcheck lint probes` anywhere here on purpose. Those rules are written for a
+# lesson: install pyxray, print the version banner, and so on. The probe installs nothing,
+# which is the whole point of it, so it would fail two rules for doing its job properly.
+build-probe:
+    UV_PROJECT_ENVIRONMENT=/tmp/venv-314 uv run --python 3.14 --all-packages \
+        wasmprobe native --into probes/pyodide
+    uv run wasmprobe browser --into probes/pyodide
+    uv run wasmprobe report probes/pyodide --into probes/pyodide/report.md
+    uv run wasmprobe notebook --into probes/pyodide/probe.ipynb
+
 # Rewrite the citation lockfile after a human has read the diff. This is deliberately
 # not part of `check`, because a checker that silently repairs itself checks nothing.
 recheck:

probes/pyodide/decision.md

@@ -0,0 +1,61 @@
+# What stays in Tier 0 after measuring Pyodide
+
+This is the written half of gate Q1. The measured half is `report.md` next to it, and the raw runs are in `native.json` and `pyodide.json`. Reproduce them with `just build-probe`, or open `probe.ipynb` and run the same checks on whatever browser you are sitting in front of.
+
+## The short version
+
+Tier 0 survives. Fifteen checks, twelve behave the same in a browser as they do on a native CPython, and none of the three differences takes an experiment out of Tier 0.
+
+That is a better result than the issue expected. Rule 3 stands, and `pyxray.replay` stays a fallback in M2 rather than becoming a load bearing part of M0.
+
+## What was measured
+
+Pyodide 314.0.6 from npm, which is CPython 3.14.2 built for `emscripten-5.0.3-wasm32`, driven from Node. The control is a native CPython 3.14.7, deliberately 3.14 rather than the pinned 3.15, so a version difference does not get reported as a build difference.
+
+## The three differences
+
+**`optimize_cfg` cannot be called the way `pyxray` calls it.** `compiler_codegen` works and returns the same eight instructions in both places, but the metadata dictionary it hands back has no `consts` key in this build. It has `argcount`, `kwonlyargcount` and `posonlyargcount`, and that is all. Native 3.14 and 3.15 both include `consts`. Since `pyxray.compiler.stages` passes `metadata["consts"]` straight into `optimize_cfg`, that line raises `KeyError` in a browser.
+
+The optimizer itself is fine. Build a constants list of the right length from the instruction sequence and `optimize_cfg` runs and returns the same seven instructions it returns natively. So this is a missing key rather than a missing stage, and the fix is in our code, not in Pyodide. Filed as a bug. Until it lands, the compiler stage experiments in T05 stay in Tier 0 with that one line guarded.
+
+**A wrong constants list kills the runtime instead of raising.** Hand `optimize_cfg` a list that is too short and a native interpreter raises `ValueError: LOAD_CONST index 0 is out of range for consts (len=0)`. In WebAssembly the same call reads past the end of memory, the runtime does not come back, and in a notebook the kernel dies and the reader loses their work.
+
+This does not remove anything from Tier 0, but it does constrain how the pipeline widget is written. It has to build the constants list itself and never pass one it was given, because there is no way to catch this. The probe notebook runs that check last, on its own, with a paragraph warning the reader first.
+
+**A thread cannot be started.** `threading` imports and `threading.Thread(...)` constructs, and `start()` raises `RuntimeError: can't start new thread`. This is expected for a single threaded WebAssembly build without the pthread proxy, and it was already the assumption: the concurrency lessons are in M4 and were never Tier 0. Nothing moves.
+
+## What works, and is worth saying out loud
+
+`_testinternalcapi` imports, which was the check most likely to sink this. `compiler_codegen` and `optimize_cfg` are both there and both callable.
+
+`assemble_code_object` is present, and the probe only checks that it exists rather than calling it. That is not laziness. It asserts on its metadata instead of raising, and a failed assertion aborts the process, so calling it to see what happens would be the same class of mistake as the constants list above. `pyxray.compiler.assemble` refuses to call it for the same reason, tracked in issue 35.
+
+`ctypes` reads both fields in front of a live object: the reference count matches `sys.getrefcount`, and the type pointer one word further along really is `id(list)`. Object headers stay live in the browser rather than being shown from a recording.
+
+`sys.monitoring` registers a tool, sets a local event and fires the callback. The stepper stays in Tier 0.
+
+`sys.settrace` still reports call, line and return, so the fallback exists too.
+
+The cycle collector frees a two node cycle, has three generations, and is enabled.
+
+`dis`, `ast`, `symtable`, `tokenize`, `marshal`, `opcode` and `_opcode` all import, and `dis` gives the same five instructions, the same ten byte code object and the same 109 byte marshal blob as the native 3.14.
+
+## Where the two are genuinely different machines
+
+Four answers differ without anything being broken, and a lesson that asserts one of these numbers is teaching the build rather than the language.
+
+Pointers are four bytes rather than eight. `sysconfig.get_platform()` is `emscripten-5.0.3-wasm32`. The third garbage collector threshold is 0 rather than 10, so the oldest generation is never collected on a schedule. And the metadata key described above.
+
+The pointer size is the one to watch. Every diagram in the object lessons draws an eight byte word, and a reader in a browser who measures it gets four. T08 and T09 need a sentence about that, and it is a good sentence to have: it is the difference between memorising a number and knowing where the number comes from.
+
+## The phone question, answered partly
+
+The issue asks how long a cold boot takes on a mid range phone. This probe cannot answer that. It boots in about a second from a local disk under Node, which is a floor and not a promise.
+
+What it can measure is the part that dominates on a phone: 13.5 MB has to arrive before the first cell runs, which is the WebAssembly binary, the JavaScript glue, the standard library zip and the lock file. On a slow connection that is the wait, not the boot. Anything about tab memory, a service worker cache, or a real device is not answered here and is worth its own issue when the site actually exists.
+
+## What this changes
+
+Nothing moves from Tier 0 to Tier 1.
+
+Three pieces of work fall out of it. Guard the `metadata["consts"]` line in `pyxray.compiler.stages` so the compiler stages work in a browser. Make the pipeline widget build its own constants list rather than accepting one, because the alternative is a dead kernel. Add a sentence to the object lessons about the word size, and keep measuring it rather than asserting it, which is what those lessons already do for the small integer cache.

probes/pyodide/native.json

@@ -0,0 +1,160 @@
+{
+  "runtime": "native",
+  "python": "3.14.7",
+  "seconds": 0.0,
+  "payload_bytes": 0,
+  "outcomes": [
+    {
+      "key": "version",
+      "status": "ok",
+      "value": {
+        "python": "3.14.7",
+        "platform": "macosx-15.0-arm64",
+        "pointer_bytes": 8,
+        "free_threaded": false
+      }
+    },
+    {
+      "key": "internal_capi_import",
+      "status": "ok",
+      "value": {
+        "compiler_codegen": true,
+        "optimize_cfg": true,
+        "assemble_code_object": true
+      }
+    },
+    {
+      "key": "compiler_codegen",
+      "status": "ok",
+      "value": {
+        "instructions": 8,
+        "first": 128,
+        "metadata_keys": [
+          "argcount",
+          "consts",
+          "kwonlyargcount",
+          "posonlyargcount"
+        ]
+      }
+    },
+    {
+      "key": "optimize_cfg",
+      "status": "ok",
+      "value": {
+        "instructions": 5
+      }
+    },
+    {
+      "key": "optimize_cfg_direct",
+      "status": "ok",
+      "value": {
+        "slots": 3,
+        "instructions": 7
+      }
+    },
+    {
+      "key": "optimize_cfg_short_consts",
+      "status": "ok",
+      "value": {
+        "raised": "ValueError: LOAD_CONST index 0 is out of range for consts (len=0)"
+      }
+    },
+    {
+      "key": "ctypes_header",
+      "status": "ok",
+      "value": {
+        "refcount_field": 1,
+        "getrefcount": 2,
+        "type_pointer_matches": true,
+        "word_bytes": 8
+      }
+    },
+    {
+      "key": "monitoring",
+      "status": "ok",
+      "value": {
+        "fired": 1,
+        "event_names": 19
+      }
+    },
+    {
+      "key": "settrace",
+      "status": "ok",
+      "value": {
+        "events": [
+          "call",
+          "line",
+          "return"
+        ]
+      }
+    },
+    {
+      "key": "gc",
+      "status": "ok",
+      "value": {
+        "cycle_freed": true,
+        "thresholds": [
+          2000,
+          10,
+          10
+        ],
+        "enabled": true,
+        "generations": 3
+      }
+    },
+    {
+      "key": "debugmallocstats",
+      "status": "ok",
+      "value": {
+        "callable": true
+      }
+    },
+    {
+      "key": "front_end_modules",
+      "status": "ok",
+      "value": {
+        "dis": true,
+        "ast": true,
+        "symtable": true,
+        "tokenize": true,
+        "marshal": true,
+        "opcode": true,
+        "_opcode": true
+      }
+    },
+    {
+      "key": "disassembly",
+      "status": "ok",
+      "value": {
+        "opnames": [
+          "RESUME",
+          "LOAD_SMALL_INT",
+          "STORE_NAME",
+          "LOAD_CONST",
+          "RETURN_VALUE"
+        ],
+        "code_size": 10,
+        "consts": [
+          "6",
+          "None"
+        ],
+        "marshal_size": 109
+      }
+    },
+    {
+      "key": "small_integers",
+      "status": "ok",
+      "value": {
+        "top": 256
+      }
+    },
+    {
+      "key": "threading",
+      "status": "ok",
+      "value": {
+        "ran": true,
+        "active": 1
+      }
+    }
+  ]
+}