Measure what actually works under Pyodide, and answer gate Q1 - #76
Merged
Conversation
Every Tier 0 experiment in this project is supposed to run in a browser tab with nothing installed, which rests on Pyodide keeping the introspection surfaces the lessons poke at. Nobody had checked. This checks, on every pull request. `wasmprobe` runs fifteen questions twice, once on a native CPython and once inside a real WebAssembly runtime driven from Node, and puts the two answers side by side. Twelve behave identically. `_testinternalcapi` imports, `compiler_codegen` and `optimize_cfg` both run, `ctypes` reads the reference count and the type pointer off a live object, `sys.monitoring` fires, `sys.settrace` still reports call, line and return, the cycle collector frees a cycle, and the whole front end imports and disassembles to the same bytes. Three do not, and none of them takes an experiment out of Tier 0. The metadata `compiler_codegen` returns in that build has no `consts` key, so the one line in `pyxray.compiler.stages` that asks for it raises, while the optimizer itself is fine once you build the list yourself. Handing `optimize_cfg` a list that is too short reads past the end of memory and kills the runtime, where a native interpreter raises a clean ValueError. And a thread cannot be started, which the concurrency lessons already assumed. The driver runs the checks one at a time and boots a fresh runtime after one takes the old one down, because otherwise every check after the crash looks broken. A Tier 0 check can carry an `accepted` sentence saying we know it fails and what we do instead, so a known gap stays in the report without leaving the build permanently red and hiding the next one. `probes/pyodide` has the matrix, the two raw runs, the written decision, and a notebook that asks your own browser the same questions rather than asking you to trust a recording of ours.
This was referenced Aug 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1.
Every Tier 0 experiment in this project is supposed to run in a browser tab with nothing installed. That rests on Pyodide keeping the introspection surfaces the lessons poke at, and nobody had checked. This checks, and keeps checking on every pull request.
The answer
Tier 0 survives. Fifteen questions asked twice, once on a native CPython 3.14.7 and once inside Pyodide 314.0.6, which is CPython 3.14.2 built for
emscripten-5.0.3-wasm32. Twelve behave identically, and none of the three differences takes an experiment out of Tier 0._testinternalcapiimports, which was the one most likely to sink this.ctypesreads both fields in front of a live object and the type pointer really isid(list).sys.monitoringregisters a tool and fires the callback, so the stepper stays live.sys.settracestill reports call, line and return. The cycle collector frees a two node cycle.dis,ast,symtable,tokenize,marshal,opcodeand_opcodeall import, and the same source disassembles to the same five instructions, the same ten byte code object and the same 109 byte marshal blob.The three differences.
optimize_cfgcannot be called the waypyxraycalls it. The metadata dictionarycompiler_codegenreturns in that build hasargcount,kwonlyargcountandposonlyargcountand noconsts. Native 3.14 and 3.15 both have it. Sopyxray.compiler.stagesraisesKeyErrorin a browser. The optimizer itself is fine: build a constants list of the right length off the instruction sequence and it returns the same seven instructions it returns natively. Recorded as an accepted gap with the workaround written down, and filed as a bug.A constants list that is too short kills the runtime. Native raises
ValueError: LOAD_CONST index 0 is out of range for consts (len=0). In WebAssembly the same call reads past the end of memory, the runtime does not come back, and in a notebook the kernel dies. Nothing to catch. The pipeline widget has to build the list itself rather than validate one it was handed.A thread cannot be started.
RuntimeError: can't start new thread, which is expected for a single threaded build and was already the assumption. The concurrency lessons are in M4 and were never Tier 0.Four more answers differ without anything being wrong: pointers are four bytes rather than eight, the platform string, the third GC threshold is 0 rather than 10, and the metadata key above. The pointer size is the one that matters, because every diagram in the object lessons draws an eight byte word and a reader in a browser who measures it gets four. That is a sentence T08 and T09 want anyway.
What is in the PR
tools/wasmprobeis a new workspace package.checks.pyholds the fifteen questions, each one a string of Python rather than a function, because the same source has to run in two places and shipping the source is the only way to stop the two copies drifting.driver.mjsboots Pyodide under Node, runs the checks one at a time, and boots a fresh runtime after one takes the old one down, because otherwise every check after the crash looks broken. That is not theoretical, the short constants list does exactly that.native.py,browser.py,report.py,notebook.pyand a CLI around them.A check carries a weight.
tier0means a lesson depends on it. It can also carry anacceptedsentence saying we already know it fails in the browser and what we do instead, which keeps a known gap in the report without leaving the build permanently red and hiding the next regression behind it.probes/pyodidehas the two raw runs, the rendered matrix, the written decision, andprobe.ipynb, which carries the checks inside itself, installs nothing, and runs in Colab or JupyterLite so a reader can ask their own runtime rather than trust our recording. The check that can kill a kernel is last, on its own, with a paragraph warning them first.How it is kept honest
just probeis injust check. It reads the committed pair, fails when a blocking check works natively and not in the browser, fails when the report or the notebook has fallen behind the checks, and then executes the notebook.The
probejob in CI does that and then records both runs again from scratch, so a Pyodide release that takes something away is noticed here rather than by a reader.just build-probeis the local version.Checks
just checkgreen: 1421 tests passed, 3 skipped, 12 lessons, 101 diagrams, 5 animations, 535 citations, and the probe notebook runs.There is no
nbcheck lint probesanywhere, on purpose. Those rules are written for a lesson, and they ask for apip install pyxraycell and a version banner. The probe installs nothing, which is the whole point of it, so it would fail two rules for doing its job properly. It is still executed bynbcheck run.