Run the compiler stages in a browser, and build the constants list ourselves

CONTRIBUTING.md

@@ -63,7 +63,7 @@ When the differing cell is the lesson's central observation, the note is not eno
 
 A Tier 0 experiment has to run in a browser tab with nothing installed. Which surfaces survive that is measured, not assumed, and the answer lives in [probes/pyodide](probes/pyodide): a matrix, the two raw runs behind it, a notebook you can open in Colab to ask your own runtime the same questions, and a written decision.
 
-Read `decision.md` before writing an experiment that pokes at the interpreter. Three things are known to be different today. `optimize_cfg` cannot be handed the constants list `compiler_codegen` returns, because that build does not put one there. Handing `optimize_cfg` a constants list that is too short reads past the end of memory and kills the runtime, where a native interpreter raises a tidy `ValueError`, so anything that builds one has to build it correctly rather than catch the mistake. And a thread cannot be started.
+Read `decision.md` before writing an experiment that pokes at the interpreter. Three things are known to be different today. `compiler_codegen` returns no constants list there, so `pyxray.compiler.stages` builds one from the instruction sequence and tells you, through `constants_known`, that the optimizer was working without the real values: do not report a constant fold without checking it. Handing `optimize_cfg` a constants list that is too short reads past the end of memory and kills the runtime, where a native interpreter raises a tidy `ValueError`, so go through `pyxray.compiler` rather than building one yourself, and never catch this instead of preventing it. And a thread cannot be started.
 
 If you need a surface nobody has measured, add a check to `tools/wasmprobe/src/wasmprobe/checks.py` and run `just build-probe`. A check is a string of Python that leaves its answer in `result`, and it has to import everything it uses, because the check before it may have taken the runtime down. Mark it `TIER0` if a lesson would depend on it, and `just probe` will fail the build the day it stops working. If it already fails and you have decided what to do instead, write that decision in the check's `accepted` field rather than deleting the check, so the gap stays in the report and the next regression is still visible.
 

lessons/t05-the-tree-becomes-bytecode/build.py

@@ -126,11 +126,14 @@
 from pyxray import compiler
 
 print("stage by stage compiling available:", compiler.available())
+print("the compiler hands back its constants:", compiler.constants_available())
 """)
 
 
 lesson.md("""
-If that said False you are on a build without `_testinternalcapi`, which happens on some slimmed down distributions. Most cells below still work, and the two that run the stages separately will raise a clear error rather than a confusing one.
+If the first line said False you are on a build without `_testinternalcapi`, which happens on some slimmed down distributions. Most cells below still work, and the two that run the stages separately will raise a clear error rather than a confusing one.
+
+The second line is the one that catches people out in a browser. The code generator collects the constants it comes across and normally hands them back with everything else, and the optimizer needs those values to work out that 6 times 7 is 42. The WebAssembly build does not hand them back. Every cell in this lesson still runs there, because `pyxray` builds a list of placeholders of the right length rather than asking for one, but the folding does not happen and the two column output says so underneath. If that line said False and you want to watch a fold, run this lesson on an ordinary Python.
 
 ## The whole trip, counted
 

lessons/t05-the-tree-becomes-bytecode/t05.ipynb

@@ -127,15 +127,18 @@
    "source": [
     "from pyxray import compiler\n",
     "\n",
-    "print(\"stage by stage compiling available:\", compiler.available())"
+    "print(\"stage by stage compiling available:\", compiler.available())\n",
+    "print(\"the compiler hands back its constants:\", compiler.constants_available())"
    ]
   },
   {
    "cell_type": "markdown",
    "id": "t05-09",
    "metadata": {},
    "source": [
-    "If that said False you are on a build without `_testinternalcapi`, which happens on some slimmed down distributions. Most cells below still work, and the two that run the stages separately will raise a clear error rather than a confusing one.\n",
+    "If the first line said False you are on a build without `_testinternalcapi`, which happens on some slimmed down distributions. Most cells below still work, and the two that run the stages separately will raise a clear error rather than a confusing one.\n",
+    "\n",
+    "The second line is the one that catches people out in a browser. The code generator collects the constants it comes across and normally hands them back with everything else, and the optimizer needs those values to work out that 6 times 7 is 42. The WebAssembly build does not hand them back. Every cell in this lesson still runs there, because `pyxray` builds a list of placeholders of the right length rather than asking for one, but the folding does not happen and the two column output says so underneath. If that line said False and you want to watch a fold, run this lesson on an ordinary Python.\n",
     "\n",
     "## The whole trip, counted\n",
     "\n",

probes/pyodide/decision.md

@@ -14,9 +14,11 @@ Pyodide 314.0.6 from npm, which is CPython 3.14.2 built for `emscripten-5.0.3-wa
 
 ## The three differences
 
-**`optimize_cfg` cannot be called the way `pyxray` calls it.** `compiler_codegen` works and returns the same eight instructions in both places, but the metadata dictionary it hands back has no `consts` key in this build. It has `argcount`, `kwonlyargcount` and `posonlyargcount`, and that is all. Native 3.14 and 3.15 both include `consts`. Since `pyxray.compiler.stages` passes `metadata["consts"]` straight into `optimize_cfg`, that line raises `KeyError` in a browser.
+**`compiler_codegen` returns no constants list.** It works and returns the same eight instructions in both places, but the metadata dictionary it hands back has only `argcount`, `kwonlyargcount` and `posonlyargcount` in this build. Native 3.14 and 3.15 both include `consts` as well. `pyxray.compiler.stages` used to pass `metadata["consts"]` straight into `optimize_cfg`, so that line raised `KeyError` in a browser.
 
-The optimizer itself is fine. Build a constants list of the right length from the instruction sequence and `optimize_cfg` runs and returns the same seven instructions it returns natively. So this is a missing key rather than a missing stage, and the fix is in our code, not in Pyodide. Filed as a bug. Until it lands, the compiler stage experiments in T05 stay in Tier 0 with that one line guarded.
+The optimizer itself is fine. Build a constants list of the right length from the instruction sequence and `optimize_cfg` runs and returns the same seven instructions it returns natively. So this was a missing key rather than a missing stage, and the fix was in our code, not in Pyodide. It is fixed: `stages` builds that list itself now, from the sequence it is about to pass in, and records whether the values were real. Issue 77.
+
+One thing does not survive the fix, and it is worth being plain about. Without the values, the optimizer cannot fold `6 * 7` into `42`, and on some code it makes a different decision than it would with them. So the stage runs in a browser and its output is not what the source compiles to. The pipeline widget says so on that pane rather than letting the reader assume, and the last pane, which is the finished code object, is the real answer on every build.
 
 **A wrong constants list kills the runtime instead of raising.** Hand `optimize_cfg` a list that is too short and a native interpreter raises `ValueError: LOAD_CONST index 0 is out of range for consts (len=0)`. In WebAssembly the same call reads past the end of memory, the runtime does not come back, and in a notebook the kernel dies and the reader loses their work.
 
@@ -50,12 +52,12 @@ The pointer size is the one to watch. Every diagram in the object lessons draws
 
 ## The phone question, answered partly
 
-The issue asks how long a cold boot takes on a mid range phone. This probe cannot answer that. It boots in about a second from a local disk under Node, which is a floor and not a promise.
+The issue asks how long a cold boot takes on a mid range phone. This probe cannot answer that. It boots in a second or two from a local disk under Node, which is a floor and not a promise.
 
 What it can measure is the part that dominates on a phone: 13.5 MB has to arrive before the first cell runs, which is the WebAssembly binary, the JavaScript glue, the standard library zip and the lock file. On a slow connection that is the wait, not the boot. Anything about tab memory, a service worker cache, or a real device is not answered here and is worth its own issue when the site actually exists.
 
 ## What this changes
 
 Nothing moves from Tier 0 to Tier 1.
 
-Three pieces of work fall out of it. Guard the `metadata["consts"]` line in `pyxray.compiler.stages` so the compiler stages work in a browser. Make the pipeline widget build its own constants list rather than accepting one, because the alternative is a dead kernel. Add a sentence to the object lessons about the word size, and keep measuring it rather than asserting it, which is what those lessons already do for the small integer cache.
+Three pieces of work fell out of it. Two are done. `pyxray.compiler.stages` builds its own constants list, so the compiler stages run in a browser, and it says whether the values were real so nothing claims a fold that did not happen (issue 77). The pipeline widget reaches the optimizer only through that function and takes a list from nobody, because the alternative is a dead kernel (issue 78). Still open: a sentence in the object lessons about the word size, measured rather than asserted, which is what those lessons already do for the small integer cache (issue 79).

probes/pyodide/probe.ipynb

@@ -67,7 +67,7 @@
     "    \"metadata_keys\": sorted(metadata),\n",
     "}\n",
     "\"\"\",\n",
-    "    # Does optimize_cfg run over that sequence the way pyxray calls it\n",
+    "    # Does optimize_cfg run when it is handed the constants from the metadata\n",
     "    \"optimize_cfg\": \"\"\"\n",
     "import _testinternalcapi\n",
     "import ast\n",
@@ -80,12 +80,11 @@
     "    \"optimize_cfg_direct\": \"\"\"\n",
     "import _testinternalcapi\n",
     "import ast\n",
-    "import opcode\n",
+    "import dis\n",
     "\n",
     "sequence, metadata = _testinternalcapi.compiler_codegen(ast.parse(\"answer = 6 * 7\"), \"<probe>\", 0)\n",
-    "load = opcode.opmap[\"LOAD_CONST\"]\n",
-    "slots = [one[1] for one in sequence.get_instructions() if one[0] == load]\n",
-    "consts = [None] * (max(slots) + 1 if slots else 0)\n",
+    "slots = [one[1] for one in sequence.get_instructions() if one[0] in dis.hasconst]\n",
+    "consts = [object() for _ in range(max(slots, default=-1) + 1)]\n",
     "optimized = _testinternalcapi.optimize_cfg(sequence, consts, 0)\n",
     "result = {\"slots\": len(consts), \"instructions\": len(optimized.get_instructions())}\n",
     "\"\"\",\n",

probes/pyodide/pyodide.json

@@ -1,7 +1,7 @@
 {
   "runtime": "pyodide",
   "python": "3.14.2",
-  "seconds": 0.9,
+  "seconds": 2.52,
   "payload_bytes": 13508566,
   "outcomes": [
     {

probes/pyodide/report.md

@@ -1,6 +1,6 @@
 # What works under Pyodide
 
-Generated by `wasmprobe report`. Native run on CPython 3.14.7, WebAssembly run on Pyodide with CPython 3.14.2, which booted in 0.9 seconds off a local disk after 13.5 MB of runtime and standard library.
+Generated by `wasmprobe report`. Native run on CPython 3.14.7, WebAssembly run on Pyodide with CPython 3.14.2, which booted in 2.5 seconds off a local disk after 13.5 MB of runtime and standard library.
 
 15 checks: 12 works in both, 3 works natively, not in the browser.
 
@@ -9,7 +9,7 @@ Generated by `wasmprobe report`. Native run on CPython 3.14.7, WebAssembly run o
 | Which CPython is this, and what was it built for | info | yes | yes |
 | Is _testinternalcapi importable at all | tier0 | yes | yes |
 | Does compiler_codegen turn a tree into an instruction sequence | tier0 | yes | yes |
-| Does optimize_cfg run over that sequence the way pyxray calls it | tier0, known gap | yes | raises, KeyError: 'consts' |
+| Does optimize_cfg run when it is handed the constants from the metadata | info | yes | raises, KeyError: 'consts' |
 | Does optimize_cfg run at all, given a constants list built by hand | tier0 | yes | yes |
 | What happens when optimize_cfg is handed a constants list that is too short | info | yes | kills the runtime, memory access out of bounds |
 | Can ctypes read the two fields in front of every object | tier0 | yes | yes |
@@ -30,9 +30,9 @@ Generated by `wasmprobe report`. Native run on CPython 3.14.7, WebAssembly run o
 
 **Does compiler_codegen turn a tree into an instruction sequence.** The first compiler stage in T05 and in the pipeline explorer widget.
 
-**Does optimize_cfg run over that sequence the way pyxray calls it.** The middle stage. It is the one that makes 6 * 7 disappear, which is the single most convincing thing in the first part of the course.
+**Does optimize_cfg run when it is handed the constants from the metadata.** Nothing now. It cost the whole middle stage until this measurement, and the answer was to build the constants list off the instruction sequence rather than ask for it. Still worth asking every run: the day this passes is the day constant folding can be shown in a browser as well as here.
 
-**Does optimize_cfg run at all, given a constants list built by hand.** The same middle stage, asked in a way a missing metadata key cannot hide.
+**Does optimize_cfg run at all, given a constants list built by hand.** The middle stage. It is the one that makes 6 * 7 disappear, which is the single most convincing thing in the first part of the course.
 
 **What happens when optimize_cfg is handed a constants list that is too short.** Nothing, and it is worth knowing. A clean exception means the widget can show the reader their mistake. A dead runtime means it has to check first.
 
@@ -64,7 +64,3 @@ Both runtimes answered these, and answered them differently.
 | Does compiler_codegen turn a tree into an instruction sequence | `{'instructions': 8, 'first': 128, 'metadata_keys': ['argcount', 'consts', 'kwonlyargcount', 'posonlyargcount']}` | `{'instructions': 8, 'first': 128, 'metadata_keys': ['argcount', 'kwonlyargcount', 'posonlyargcount']}` |
 | Can ctypes read the two fields in front of every object | `{'refcount_field': 1, 'getrefcount': 2, 'type_pointer_matches': True, 'word_bytes': 8}` | `{'refcount_field': 1, 'getrefcount': 2, 'type_pointer_matches': True, 'word_bytes': 4}` |
 | Does the cycle collector behave the way T09 says it does | `{'cycle_freed': True, 'thresholds': [2000, 10, 10], 'enabled': True, 'generations': 3}` | `{'cycle_freed': True, 'thresholds': [2000, 10, 0], 'enabled': True, 'generations': 3}` |
-
-## Known gaps, and what we do instead
-
-**Does optimize_cfg run over that sequence the way pyxray calls it.** The metadata this build hands back has no consts key, so the call fails before it reaches the optimizer. The check below shows the optimizer itself is fine, so in the browser the pipeline builds that list from the instruction sequence instead of asking for it.