Skip to content

Measure what actually works under Pyodide, and answer gate Q1 - #76

Merged
tamnd merged 1 commit into
mainfrom
pyodide-probe
Aug 29, 2026
Merged

Measure what actually works under Pyodide, and answer gate Q1#76
tamnd merged 1 commit into
mainfrom
pyodide-probe

Conversation

@tamnd

@tamnd tamnd commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Closes #1.

Every Tier 0 experiment in this project is supposed to run in a browser tab with nothing installed. That rests on Pyodide keeping the introspection surfaces the lessons poke at, and nobody had checked. This checks, and keeps checking on every pull request.

The answer

Tier 0 survives. Fifteen questions asked twice, once on a native CPython 3.14.7 and once inside Pyodide 314.0.6, which is CPython 3.14.2 built for emscripten-5.0.3-wasm32. Twelve behave identically, and none of the three differences takes an experiment out of Tier 0.

_testinternalcapi imports, which was the one most likely to sink this. ctypes reads both fields in front of a live object and the type pointer really is id(list). sys.monitoring registers a tool and fires the callback, so the stepper stays live. sys.settrace still reports call, line and return. The cycle collector frees a two node cycle. dis, ast, symtable, tokenize, marshal, opcode and _opcode all import, and the same source disassembles to the same five instructions, the same ten byte code object and the same 109 byte marshal blob.

The three differences.

optimize_cfg cannot be called the way pyxray calls it. The metadata dictionary compiler_codegen returns in that build has argcount, kwonlyargcount and posonlyargcount and no consts. Native 3.14 and 3.15 both have it. So pyxray.compiler.stages raises KeyError in a browser. The optimizer itself is fine: build a constants list of the right length off the instruction sequence and it returns the same seven instructions it returns natively. Recorded as an accepted gap with the workaround written down, and filed as a bug.

A constants list that is too short kills the runtime. Native raises ValueError: LOAD_CONST index 0 is out of range for consts (len=0). In WebAssembly the same call reads past the end of memory, the runtime does not come back, and in a notebook the kernel dies. Nothing to catch. The pipeline widget has to build the list itself rather than validate one it was handed.

A thread cannot be started. RuntimeError: can't start new thread, which is expected for a single threaded build and was already the assumption. The concurrency lessons are in M4 and were never Tier 0.

Four more answers differ without anything being wrong: pointers are four bytes rather than eight, the platform string, the third GC threshold is 0 rather than 10, and the metadata key above. The pointer size is the one that matters, because every diagram in the object lessons draws an eight byte word and a reader in a browser who measures it gets four. That is a sentence T08 and T09 want anyway.

What is in the PR

tools/wasmprobe is a new workspace package. checks.py holds the fifteen questions, each one a string of Python rather than a function, because the same source has to run in two places and shipping the source is the only way to stop the two copies drifting. driver.mjs boots Pyodide under Node, runs the checks one at a time, and boots a fresh runtime after one takes the old one down, because otherwise every check after the crash looks broken. That is not theoretical, the short constants list does exactly that. native.py, browser.py, report.py, notebook.py and a CLI around them.

A check carries a weight. tier0 means a lesson depends on it. It can also carry an accepted sentence saying we already know it fails in the browser and what we do instead, which keeps a known gap in the report without leaving the build permanently red and hiding the next regression behind it.

probes/pyodide has the two raw runs, the rendered matrix, the written decision, and probe.ipynb, which carries the checks inside itself, installs nothing, and runs in Colab or JupyterLite so a reader can ask their own runtime rather than trust our recording. The check that can kill a kernel is last, on its own, with a paragraph warning them first.

How it is kept honest

just probe is in just check. It reads the committed pair, fails when a blocking check works natively and not in the browser, fails when the report or the notebook has fallen behind the checks, and then executes the notebook.

The probe job in CI does that and then records both runs again from scratch, so a Pyodide release that takes something away is noticed here rather than by a reader. just build-probe is the local version.

Checks

just check green: 1421 tests passed, 3 skipped, 12 lessons, 101 diagrams, 5 animations, 535 citations, and the probe notebook runs.

There is no nbcheck lint probes anywhere, on purpose. Those rules are written for a lesson, and they ask for a pip install pyxray cell and a version banner. The probe installs nothing, which is the whole point of it, so it would fail two rules for doing its job properly. It is still executed by nbcheck run.

Every Tier 0 experiment in this project is supposed to run in a browser tab
with nothing installed, which rests on Pyodide keeping the introspection
surfaces the lessons poke at. Nobody had checked. This checks, on every pull
request.

`wasmprobe` runs fifteen questions twice, once on a native CPython and once
inside a real WebAssembly runtime driven from Node, and puts the two answers
side by side. Twelve behave identically. `_testinternalcapi` imports,
`compiler_codegen` and `optimize_cfg` both run, `ctypes` reads the reference
count and the type pointer off a live object, `sys.monitoring` fires,
`sys.settrace` still reports call, line and return, the cycle collector frees a
cycle, and the whole front end imports and disassembles to the same bytes.

Three do not, and none of them takes an experiment out of Tier 0. The metadata
`compiler_codegen` returns in that build has no `consts` key, so the one line in
`pyxray.compiler.stages` that asks for it raises, while the optimizer itself is
fine once you build the list yourself. Handing `optimize_cfg` a list that is too
short reads past the end of memory and kills the runtime, where a native
interpreter raises a clean ValueError. And a thread cannot be started, which the
concurrency lessons already assumed.

The driver runs the checks one at a time and boots a fresh runtime after one
takes the old one down, because otherwise every check after the crash looks
broken. A Tier 0 check can carry an `accepted` sentence saying we know it fails
and what we do instead, so a known gap stays in the report without leaving the
build permanently red and hiding the next one.

`probes/pyodide` has the matrix, the two raw runs, the written decision, and a
notebook that asks your own browser the same questions rather than asking you to
trust a recording of ours.
@tamnd
tamnd merged commit 461048b into main Aug 29, 2026
10 checks passed
@tamnd
tamnd deleted the pyodide-probe branch August 29, 2026 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Q1: what actually works under Pyodide 314

1 participant