ENH: add a xeus-python kernel for the JupyterLite docs#14093
ENH: add a xeus-python kernel for the JupyterLite docs#14093natinew77-creator wants to merge 11 commits into
Conversation
dd530bc to
2a132cf
Compare
The setup cell probed /files/mne_data but never fetched anything, so every data-dependent notebook failed on the first read and the rest of the cells cascaded into NameErrors. The curated data is already served at the docs root via html_extra_path, so fetch it the same way the pyodide build does: a synchronous XMLHttpRequest into /tmp/mne_data, small files eagerly and the heavy ones lazily via the reader shims, so each notebook only downloads what it touches. The sync request may set responseType='arraybuffer' because the kernel runs in a web worker (the spec only forbids that on a Window), which means this needs neither cross-origin isolation nor a service worker -- static hosts such as the CircleCI artifact server provide neither. Also port the pyvista-js SourceEstimate.plot shim, which turned out to have no kernel-specific code, and add pyvista-js to the environment since the built kernel had no pyvista or vtk at all. Drop the threadpoolctl patch (Pyodide-only API) but keep the generic WASM ones.
natinew77-creator
changed the title
The xeus kernel's WASM heap is capped at 2 GiB by the emscripten build (bin/xpython.js requests WebAssembly.Memory with maximum=32768 pages), and the 3D shim runs at the end of a notebook that is still holding the raw, epochs and source estimates. Build the meshes with int32 face indices and float32 points, which are the types vtk.js uses anyway, and drop the KDTree query results and the full-resolution surface arrays as soon as they are dead. This is a small win (~14 MB retained, ~28 MB transient for the sample surfaces) and is NOT on its own enough to keep 10_overview under the cap -- that notebook still runs out of memory in the 3D cell.
SourceEstimate.plot defaults to hemi='lh' and 10_overview asks for hemi='rh', but the shim always built both hemispheres, so it drew a hemisphere the tutorial never asked for. That is expensive as well as wrong. pyvista-js serialises every mesh to JSON through .tolist(), which turns ~6.8 MB of surface arrays into ~55 MB of Python floats plus ~17 MB of text, and the browser then holds a UTF-16 copy of that text. Skipping the unwanted hemisphere removes about half of the scene. Also centre the surface when only one hemisphere is drawn, instead of applying the +/-60 mm split offset.
Experiment to check whether the 2 GiB cap is what kills 10_overview in ica.plot_properties. Revert once we know.
The wasm declares max=32768 pages on its env.memory import, so handing it a 4 GiB memory is a LinkError and the kernel never starts. Only an emscripten-forge rebuild can raise it.
|
Notes on where this is at. Same as #13925 but with the xeus-python kernel. Packages are pre-installed at build Working. Blocked. Can't be raised from here — patching the built 3D. pyvista-js shim carried over from #13925 unchanged, plus Kernel env is 87 MB / 82 packages, fetched on kernel start. |
1 participant
Draft. Exploring xeus-python as an alternative to the Pyodide kernel for the JupyterLite docs, since comparing the two is part of the GSoC evaluation (see #13925).
Scaffolding only so far:
doc/jupyterlite_environment.yml— the xeus kernel environment. The full MNE stack resolves foremscripten-wasm32(numpy, scipy, matplotlib, pandas, scikit-learn, h5py, nibabel, ...).doc/jupyterlite_xeus_setup_cell.py— first cut of the setup cell. It is much smaller than the Pyodide one because packages are pre-installed, sopipliteand the lzma/multiprocessing/requests patches are not needed.conf.pyyet, so the Pyodide build is untouched.Still to do:
conf.pywiring, serve the data as JupyterLite content so it mounts, port the pyvista-js 3D shim, then iterate on CI.