diff --git a/AGENTS.md b/AGENTS.md index 4fdf2e06a..09f92c878 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -64,10 +64,18 @@ Two orthogonal mixin axes: **braindata** (the array plus subject/transform ident Flow: dataviews → JSON + PNG mosaics (`data.py:Package`) + compressed CTM surface packs (`cortex/brainctm.py`, cached per subject) → Tornado template (`template.html`, extended by `static.html` etc., resolved via `FallbackLoader` so user template dirs can override) → Three.js app in `resources/js/`. - `view.py`: `show` (live Tornado server from `serve.py`, returns a `JSProxy` websocket RPC handle for driving JS from Python) and `make_static` (self-contained directory; `htmlembed.py` can inline everything into a single file). +- `serve.WebApp` binds its listening sockets in `__init__` through `bind_local_sockets`, one per address the names in `serve.LOCAL` resolve to (the loopback addresses and the machine's hostname), all on one port, unless a caller passes another `address`; `view.show` opens up to every interface only when the `domain_name` config option names a domain to reach the viewer under. +- Every handler is wrapped in `serve.TokenGuard` (by `serve.guarded`, in `WebApp.run`), which demands the server's `token`: `serve.WebApp.url` puts it in the address as `?token=`, the guard hands it back as a per-port cookie, and requests without either get a 403. Build URLs with `server.url(page)` rather than by hand, and pass `token=""` to turn the check off. +- `jsplot.Axes3D.draw` renders every entry of `this.views` as a viewport; a view may carry a `camera` of its own, a `prepare(width, height)` callback, an `overlay(camera)` one that draws over the view once the scene is in, and a `surf` index, which is how `mriview.Viewer.setSliceViews` puts the three slice planes beside the 3D view (each drawn down its own axis by an orthographic camera, the surface left to the 3D view, the crosshair laid over all four from a scene of its own). `setGrid` rebuilds the views on every `setData`, so the split is applied again after it. The scissor test is turned off once the views are drawn, because the passes that follow — the picker's among them — cover the whole canvas. `setOrthographic` swaps the camera the controls move for an orthographic one, where the radius sets the frustum rather than the distance; the lights are children of the camera and the renderer only finds the lights that are in the scene, so the camera is put into the scene with them. +- `surface..low poly` draws a surface from faces cut over groups of that many mm: `utils.get_lod` cuts them with `polyutils.decimate_faces` from the pack's own mesh, keyed on the pack's name in the subject cache dir, and `CTMHandler` cuts one the first time it is asked for, while `make_static` writes the widths in `utils.LOD_CELLS` and names them in the pack json as `lod_cells`. The coarse vertices are vertices of the surface, so only the index buffer and the normals change and everything held per vertex stays put; `mriview.Surface._applyLod` swaps them, keeping what the surface was loaded with to go back to. +- `surface..shading` picks how a face is colored and lit: the default interpolates a color across it and lights every pixel, while flat, gouraud and phong give it one color, read where its first corner is. One color to a face needs a vertex per corner, so `Surface._setFacets` writes the drawn faces out that way (`facepos`/`facewm` carry the read position, `FACECOLOR` in `shaderlib.surface_pixel`), which it does below `facet_limit` faces (a surface of its own detail would take hundreds of megabytes, and its faces are smaller than a pixel in any case). Data held a vertex at a time arrives for the surface as it is held, so `Surface.setDataAttribute` writes each frame out for the faces as it comes. `surface..outline` draws the surface again through `Shaders.outline`, inside out and grown by `outline width` of the screen (as many pixels up as across, from the `viewport` uniform `Surface.prerender` keeps current), which leaves a line around it; `outline corners` set to round adds a `THREE.PointCloud` of dots of the line's width at the vertices of the faces being drawn (`Surface._dotGeometry`) that face away from the eye, as the line is drawn from the faces that do, so the surface covers them except where they round off a corner of the silhouette; `Surface.prerender` keeps them out of the depth image the labels are hidden by. +- `surface..lighting.ambient occlusion` takes the sky a vertex misses out of the light on it. `polyutils.vertex_occlusion` bakes it a vertex at a time by following rays over the hemisphere of the normal through a voxel grid of the surface, and `utils.get_occlusion` works it out on each surface the viewer mixes between (the folded surface at depth 0 and 1, the pial and white matter surfaces, then the pack's `names`, zeros for the flat one), in the pack's numbering, writing a byte per vertex to `_ao.bin` beside it; `CTMHandler` makes the file the first time it is asked for, on a thread, and `make_static` writes it with the pack. The viewer fetches it on the first toggle and writes the values into spare components of attributes that are already there (the two depths as two bytes in the w of `wm`, mixed by `use_thickmix` like the position; each `mixSurfs`'s w; `auxdat.z` without a white matter surface), since the shaders are at WebGL's 16-slot limit; `mixfunc_occlusion` in the shader mixes them with the same weights as the positions, so what is dark follows the view and the depth, and `occlusion strength` (the `occlusionStrength` uniform) scales how much is taken out of the light. +- `Axes3D.getImage` draws into a `WebGLRenderTarget`, which carries one sample a pixel however the canvas was made (`antialias` is asked for only off a retina display, and a render target cannot be multisampled in webgl 1), so an image saved from the viewer has hard edges. `Save image.Antialias` renders it that many samples across a pixel and `mriview.getTexture` hands the buffer to `downsampleTexture`, which averages each block back down to a pixel with each sample's color weighted by its alpha, so the edge of the surface keeps the color of the surface and fades into the transparent background rather than into black. - The bundled Three.js is **r69** — very old; the shader pipeline depends on its conventions, do not casually upgrade. - All GLSL lives in `resources/js/shaderlib.js` as arrays of string lines assembled per-configuration — there are no `.glsl` files. - CTM packs reorder vertices: `cortex.utils.get_ctmmap` / `get_ctm2webgl_map` translate between CTM/WebGL ordering and the original surface ordering. Indexing viewer data with original vertex indices without remapping is a classic bug. - `cortex/export/headless.py` (`headless_viewer`) runs the viewer in headless Chromium via Playwright for screenshots/tests. +- `cortex/webgl/aligner.py` + `resources/js/aligner.js` (`aligner.html`, `resources/css/aligner.css`) is the browser-based manual aligner behind `cortex.align.webgl_manual`. It reuses the CTM pack, the mosaic PNG loader (`dataset.VolumeData`) and the `Shaders.aligner_*` builders in `shaderlib.js`, and works in a world frame that is the reference volume's voxel grid in mm permuted to RAS (`reference_frame`); the edited transform is the pycortex `coord` transform. Its `display` control picks one of three displays; the data view draws the surfaces in the anatomy's own frame with the morph targets of the CTM pack (`Shaders.aligner_volume({morphs: n})`), so they inflate and flatten there while the alignment only moves where the volume is sampled. `/save` writes to the filestore, so it takes only posts carrying the per-page `save_token` from the page config, and `JSAligner.save` polls the page's save state rather than returning while the request is still out. - Some JS resources use CRLF line endings (e.g. `dataset.js`); keep the existing endings when editing. - `setup.py` explicitly enumerates `cortex.webgl` `package_data` patterns — new resource subdirectories must be added there or they won't ship in wheels. diff --git a/cortex/align.py b/cortex/align.py index 1940221fd..4ad8ab7dd 100644 --- a/cortex/align.py +++ b/cortex/align.py @@ -14,90 +14,143 @@ from .xfm import Transform -def mayavi_manual(subject, xfmname, reference=None, **kwargs): - """Open GUI for manually aligning a functional volume to the cortical surface for `subject`. This - creates a new transform called `xfm`. The name of a nibabel-readable file (e.g. nii) should be - supplied as `reference`. This image will be copied into the database. - - To modify an existing functional-anatomical transform, `reference` can be left blank, and the - previously used reference will be loaded. - - <> - - When the GUI is closed, the transform will be saved into the pycortex database. The GUI requires - Mayavi support. +def webgl_manual( + subject: str, + xfmname: str, + reference: Optional[str] = None, + view_only: bool = False, + cmap: Optional[str] = None, + mesh_color: Optional[str] = None, + mesh_opacity: Optional[float] = None, + open_browser: Optional[bool] = None, + autoclose: Optional[bool] = None, + port: Optional[int] = None, + recache: bool = False, + types: tuple[str, ...] = ("inflated",), + title: Optional[str] = None, + display_url: bool = True, + token: Optional[str] = None, + template: str = "aligner.html", +): + """Open the browser-based aligner for manually aligning a functional volume + to the cortical surface of `subject`. + + The functional reference volume stays on its own voxel grid, so its slices + are shown without resampling, and the pial and white matter surfaces are + moved into its space; in each slice view the surfaces are cut off at the + displayed slice, so that their outline can be compared with the anatomy + in the image. Only rotations and translations are possible. + + The page shows the coronal, axial and sagittal slices and a 3D view of + the three slices. In a slice view, a left drag moves the cursor, which + sets the slices of the other views and is the pivot of rotations; the + wheel (or ``[`` and ``]``) changes the slice, ctrl + wheel zooms, and a + middle (or shift + left) drag pans. A right drag, the WASD keys or the + arrow keys translate the surfaces in the plane of the view under the + mouse, ctrl + right drag or ``q`` / ``e`` rotate them about the cursor; + shift makes the keyboard steps ten times smaller and ctrl + z undoes. The panel on + the right holds the Save button, the view mode (surface outlines on the + slices, or the volume painted on the surface, also toggled with ``m``), + the colormap with its range, brightness, contrast, gamma and flip, the + color and opacity of the surfaces, the slices and the keyboard steps. + + The ``transform`` field above the Save button holds the name the + alignment is saved under. It starts as `xfmname`; editing it saves the + alignment as a new transform and leaves the one it was loaded from + alone. An asterisk on the Save button and in the window title marks an + alignment that differs from the one last saved. + + When Save is pressed the transform is stored into the pycortex database, + as a 'coord' transform. A new transform requires `reference`, which is + copied into the database; an existing transform is loaded together with + its stored reference. + + Saving also deletes the masks cached for the transform it writes, since + they were cut out of the reference volume through the alignment being + replaced. The page warns about this when it opens a transform that has + masks, and the save message names the ones it deleted. Data already + masked with them has to be masked again from the volumes. Pass + ``view_only=True`` to inspect an alignment without saving. Parameters ---------- subject : str Subject identifier. xfmname : str - String identifying the transform to be created or loaded. + Name of the transform to create or modify. reference : str, optional - Path to a nibabel-readable image that will be used as the reference for this transform. - If given the default value of None, this function will attempt to load an existing reference - image from the database. - kwargs : dict - Passed to mayavi_aligner.get_aligner. + Path to a nibabel-readable functional volume, required for a new + transform. Must be None for an existing transform. + view_only : bool, optional + Open the aligner without the possibility to save, to inspect an + alignment. Default False. + cmap : str, optional + Initial colormap for the reference volume, one of the 1D pycortex + colormaps. Defaults to the `colormap` option of the `webgl_aligner` + section of the config file. + mesh_color : str, optional + Initial color of the surface outlines, as a matplotlib color. + Defaults to the `mesh_color` config option. + mesh_opacity : float, optional + Initial opacity of the whole surfaces in the 3D view (0 shows only + their outlines on the slices). Defaults to the `mesh_opacity` config + option. + open_browser : bool, optional + Open the aligner in the default browser. Defaults to the + `open_browser` option of the `webshow` config section. + autoclose : bool, optional + Stop the server when the last browser window disconnects. Defaults + to the `autoclose` option of the `webshow` config section. + port : int, optional + Port of the server; a free port is picked when None. + recache : bool, optional + Regenerate the cached surface (CTM) files. Default False. + types : tuple of str, optional + Surface types included in the CTM pack, to share the cache with the + viewer. Default ("inflated",). + title : str, optional + Title of the browser window. Defaults to the subject and the + transform. + display_url : bool, optional + When `open_browser` is False, display an IPython link to the + aligner. Default True. + token : str, optional + The session token the server demands, which the address it prints + carries and the page then keeps in a cookie. A new one is made for + each aligner; pass '' to take requests from anything that reaches + the port, which a script talking to the server itself may want. + template : str, optional + Name of the tornado template of the page. Default 'aligner.html'. Returns ------- - m : 2D ndarray, shape (4, 4) - Transformation matrix. + handle : cortex.webgl.aligner.JSAligner or cortex.webgl.serve.WebApp + A handle to the running aligner: ``handle.get_xfm()`` returns the + current transform, ``handle.save()`` saves it. When the aligner is + started with ``open_browser=False`` the tornado server is returned + instead; its ``get_client()`` returns the handle once a browser has + connected. """ - - warnings.warn("This is the old cortex.align.manual(), and has been " - "deprecated. Please use the new cortex.align.manual() " - "(previously cortex.align.fs_manual()), which uses " - "the `freeview` program in the freesurfer suite, to " - "perform manual alignment.", DeprecationWarning - ) - from .database import db - from .mayavi_aligner import get_aligner - def save_callback(aligner): - db.save_xfm(subject, xfmname, aligner.get_xfm("magnet"), xfmtype='magnet', reference=reference) - print("saved xfm") - - def view_callback(aligner): - print('view-only mode! ignoring changes') - - # Check whether transform w/ this xfmname already exists - view_only_mode = False - try: - db.get_xfm(subject, xfmname) - # Transform exists, make sure that reference is None - if reference is not None: - raise ValueError('Refusing to overwrite reference for existing transform %s, use reference=None to load stored reference' % xfmname) - - # if masks have been cached, quit! user must remove them by hand - from glob import glob - if len(glob(db.get_paths(subject)['masks'].format(xfmname=xfmname, type='*'))): - print('Refusing to overwrite existing transform %s because there are cached masks. Delete the masks manually if you want to modify the transform.' % xfmname) - checked = False - while not checked: - resp = input("Do you want to continue in view-only mode? (Y/N) ").lower().strip() - if resp in ["y", "yes", "n", "no"]: - checked = True - if resp in ["y", "yes"]: - view_only_mode = True - print("Continuing in view-only mode...") - else: - raise ValueError("Exiting...") - else: - print("Didn't get that, please try again..") - except IOError: - # Transform does not exist, make sure that reference exists - if reference is None or not os.path.exists(reference): - raise ValueError('Reference image file (%s) does not exist' % reference) - - - - - m = get_aligner(subject, xfmname, epifile=reference, **kwargs) - m.save_callback = view_callback if view_only_mode else save_callback - m.configure_traits() - - return m + from .webgl import aligner + + return aligner.show( + subject, + xfmname, + reference=reference, + view_only=view_only, + cmap=cmap, + mesh_color=mesh_color, + mesh_opacity=mesh_opacity, + open_browser=open_browser, + autoclose=autoclose, + port=port, + recache=recache, + types=types, + title=title, + display_url=display_url, + token=token, + template=template, + ) def fs_manual(subject, xfmname, **kwargs): @@ -136,8 +189,8 @@ def manual( ALSO: all the freesurfer environment stuff shouldn't be necessary, except that I don't know what vox2ras-tkr is doing. - Renamed from fs_manual() to manual(), since old manual() function was no longer - supported (or functional) for a while due to changes in mayavi. + Renamed from fs_manual() to manual(), since the old manual() function had + been unsupported for a while. Parameters diff --git a/cortex/dataset/view2D.py b/cortex/dataset/view2D.py index a6511140d..eb974aa31 100644 --- a/cortex/dataset/view2D.py +++ b/cortex/dataset/view2D.py @@ -7,7 +7,7 @@ import numpy.typing as npt from .. import options -from .views import Dataview, Volume, Vertex, VolumeRGB, VertexRGB +from .views import Dataview, Volume, Vertex, VolumeRGB, VertexRGB, json_attrs from .viewRGB import warn_alpha_range from .braindata import BrainData, VolumeData, VertexData @@ -146,7 +146,7 @@ def to_json(self, simple=False): """ sdict = dict(data=[[self.dim1.name, self.dim2.name]], state=self.state, - attrs=self.attrs, + attrs=json_attrs(self.attrs), desc=self.description, cmap=[self.cmap] ) diff --git a/cortex/dataset/views.py b/cortex/dataset/views.py index bd22fc217..1c683cd0d 100644 --- a/cortex/dataset/views.py +++ b/cortex/dataset/views.py @@ -4,6 +4,7 @@ import json import os import sys +import warnings from typing import Any, Optional, TypedDict, Union, cast, overload, Literal if sys.version_info < (3, 11): from typing_extensions import NotRequired @@ -191,6 +192,34 @@ class DataviewJSON(TypedDict): alpha: NotRequired[list[str]] # Volume2D/Vertex2D with alpha=: [name of the alpha brain] +def json_attrs(attrs: dict[str, Any]) -> dict[str, Any]: + """The attrs of a view as JSON can carry them. + + An attr holding an array is data, not something the page reads: the + ``alpha`` a 2D view takes is drawn into the colors of its ``raw`` view, + and the viewer colors the two dimensions itself from the data it is sent. + The json encoder refuses an array outright, so such an attr is left out, + with a warning. Numpy's own scalars, which it refuses as well, go in as + the python ones. + """ + carried, left_out = {}, [] + for name, value in attrs.items(): + if isinstance(value, np.generic): + value = value.item() + try: + json.dumps(value) + except TypeError: + left_out.append(name) + continue + carried[name] = value + if left_out: + warnings.warn( + "%s cannot be written as JSON, so %s left out of the data the " + "viewer is sent" % (", ".join(sorted(left_out)), + "they are" if len(left_out) > 1 else "it is")) + return carried + + class Dataview: _nan_mask: Optional[npt.NDArray[np.bool_]] @@ -261,7 +290,7 @@ def to_json(self, simple: bool=False) -> DataviewJSON: desc = self.description if isinstance(desc, bytes): desc = desc.decode() - sdict = dict(state=self.state, attrs=self.attrs.copy(), desc=desc) + sdict = dict(state=self.state, attrs=json_attrs(self.attrs), desc=desc) try: sdict.update( dict( diff --git a/cortex/defaults.cfg b/cortex/defaults.cfg index 79a5a4a32..da47dafbc 100644 --- a/cortex/defaults.cfg +++ b/cortex/defaults.cfg @@ -39,13 +39,14 @@ blender = blender slim = None meshlab = None -[mayavi_aligner] -line_width = 1 -point_size = 2 -outline_color = white -outline_rep = wireframe -opacity = 0 +[webgl_aligner] +# Initial settings of the browser-based aligner (cortex.align.webgl_manual): +# the colormap of the reference volume (a 1D pycortex colormap), the color of +# the surface outlines (a matplotlib color) and the opacity of the whole +# surfaces in the 3D view (0 shows only their outlines on the slices). colormap = gray +mesh_color = white +mesh_opacity = 0 [paths_default] stroke = white @@ -181,7 +182,8 @@ webgl_smooth = 0.0 [webgl] layers = rois, # When creating webgl viewers, this domain name will be appended to your computer name. Default is blank (no extra domain name) -domain_name = +# Blank also keeps the viewer on this computer's own names (localhost, 127.0.0.1 and its hostname); setting it serves the viewer on every interface, under that name. +domain_name = [webshow] autoclose = true open_browser = true diff --git a/cortex/export/headless.py b/cortex/export/headless.py index 3b1ccc563..a2c3396e4 100644 --- a/cortex/export/headless.py +++ b/cortex/export/headless.py @@ -486,7 +486,9 @@ def headless_viewer( display_url=False, **{k: v for k, v in viewer_params.items() if k not in ["port", "display_url"]}, ) - url = f"http://localhost:{server.port}/mixer.html" + # localhost rather than the machine's own name: this browser is on the + # same computer. The address carries the server's session token. + url = server.url("mixer.html", host="localhost") # Prevent the server from auto-stopping when the last WebSocket client # disconnects (ClientSocket.on_close calls server.stop() when diff --git a/cortex/mayavi_aligner.py b/cortex/mayavi_aligner.py deleted file mode 100644 index 5be332044..000000000 --- a/cortex/mayavi_aligner.py +++ /dev/null @@ -1,987 +0,0 @@ -import types -import nibabel -import numpy as np - -from traits.api import HasTraits, List, Instance, Array, Bool, Dict, Range, Float, Enum, Color, Int, Str, on_trait_change, Button, DelegatesTo, Any -from traitsui.api import View, Item, HGroup, Group, ImageEnumEditor, ColorEditor, TextEditor - -from tvtk.api import tvtk -from tvtk.pyface.scene import Scene - -from matplotlib.colors import ColorConverter -cc = ColorConverter() - -from mayavi import mlab -from mayavi.core.ui import lut_manager -from mayavi.core.api import PipelineBase, Source, Filter, Module -from mayavi.core.ui.api import SceneEditor, MlabSceneModel, MayaviScene - -from . import options -from . import polyutils -from . import volume - -legend = '''[, ]:\t\tNext, Prev slice -Ins:\t\tRotate view ccw -PgUp:\tRotate view cw -PgDn:\tScale view x up -Del:\t\tScale view x down -Home:\tScale view y up -End:\t\tScale view y down -arrows:\tMove view -i:\t\tShow 3D interactor -h:\t\tToggle outlines -Ctrl-z:\tUndo last -Shift:\tSmall increments -Scroll:\tZoom -Middle:\tPan -Left:\t\tMove slice -''' - -class RotationWidget(HasTraits): - radius = Float(value=1) - angle = Float(value=0) - pos = Array(value=[0,0,0]) - enabled = Bool(value=True) - constrain = Bool(value=False) - - def __init__(self, figure, callback, **traits): - self._t = np.linspace(0, 2*np.pi, 32) - - super().__init__(**traits) - self.callback = callback - self.figure = figure - - self.center = tvtk.HandleWidget() - self.center.set_representation(tvtk.SphereHandleRepresentation()) - self.center.representation.world_position = self.pos - self.center.priority = 1 - - self.edge = tvtk.HandleWidget() - self.edge.set_representation(tvtk.SphereHandleRepresentation()) - self.edge.representation.world_position = self.pos + (self.radius,0,0) - self.edge.priority = 1 - - self.circle = mlab.pipeline.line_source(*self._gen_circle(), figure=figure) - self.tube = mlab.pipeline.tube(self.circle, figure=figure) - self.surf = mlab.pipeline.surface(self.tube, figure=figure, color=(1,1,1)) - self.tube.filter.radius = 1 - - figure.scene.add_widgets([self.center, self.edge]) - def startmove(obj, evt): - self.startmove = (self.pos, self.angle, self.radius) - def endmove(obj, evt): - if hasattr(self.callback, "__call__"): - self.callback( self, self.pos - self.startmove[0], - self.angle - self.startmove[1], - self.radius / self.startmove[2]) - - self.edge.add_observer("StartInteractionEvent", startmove) - self.edge.add_observer("InteractionEvent", self._move_edge) - self.edge.add_observer("EndInteractionEvent", endmove) - self.center.add_observer("StartInteractionEvent", startmove) - self.center.add_observer("InteractionEvent", self._move_center) - self.center.add_observer("EndInteractionEvent", endmove) - - def move(self, pos=(0,0,0), angle=0, radius=1): - if hasattr(self.callback, "__call__"): - self.callback(self, pos, angle, radius) - self.set(pos=self.pos+pos, angle=self.angle+angle, radius=radius*self.radius) - - def _move_center(self, obj=None, evt=None): - self.pos = self.center.representation.world_position - - def _move_edge(self, obj=None, evt=None): - c = self.center.representation.world_position - r = self.edge.representation.world_position - - r -= c - - angle = np.arctan2(r[1], r[0]) - radius = np.sqrt(np.sum(r**2)) - - self.set(angle=angle, radius=radius) - - def _gen_circle(self): - t = self._t+self.angle - x = self.radius*np.cos(t) + self.pos[0] - y = self.radius*np.sin(t) + self.pos[1] - return x, y, np.repeat(self.pos[2], len(t)) - - @on_trait_change("pos,angle,radius") - def _set_circle(self): - if hasattr(self, "circle"): - self.center.representation.world_position = self.pos - rpos = [self.radius*f(self.angle) for f in [np.cos, np.sin]] - self.edge.representation.world_position = self.pos + (rpos+[0]) - - #self.circle.mlab_source.set(dict(zip(("x", "y", "z"), self._gen_circle()))) - self.circle.data.points = np.array(self._gen_circle()).T - - @on_trait_change("enabled") - def _enable(self): - if self.enabled: - self.center.on() - self.edge.on() - else: - self.center.off() - self.edge.off() - - self.circle.visible = self.enabled - -class ThreeDScene(MayaviScene): - #this will not work for qt4 windows, but its functionality is not important! - aligner = Instance("Align") - state = Array(value=[1,1,1]) - - def OnKeyDown(self, evt): - key = chr(evt.GetKeyCode() % 256) - focus = self.aligner.scene_3d.camera.focal_point - if key == "1": - self.aligner.scene_3d.camera.position = focus + [0, self.state[1]*500, 0] - self.state[1] = -self.state[1] - elif key == "3": - self.aligner.scene_3d.camera.position = focus + [self.state[0]*500, 0, 0] - self.state[0] = -self.state[0] - elif key == "5": - self.aligner.scene_3d.parallel_projection = not self.aligner.scene_3d.parallel_projection - elif key == "7": - self.aligner.scene_3d.camera.position = focus + [0, 1e-6, self.state[2]*500] - self.state[2] = -self.state[2] - elif key == "\x1a" and evt.CmdDown(): - self.aligner.undo() - else: - super().OnKeyDown(evt) - self.aligner.scene_3d.renderer.reset_camera_clipping_range() - self.aligner.scene_3d.render() - -#We need to override the mayavi default keys to avoid command overlap -try: - #This is a WX window - Scene.OnKeyDown - class FlatScene(Scene): - def OnKeyDown(self, event): - #emulate behavior in QT - lookup = {i: (chr(i), None) for i in range(256)} - lookup[315] = ('', 'Up') - lookup[314] = ('', 'Left') - lookup[316] = ('', 'Right') - lookup[317] = ('', 'Down') - lookup[322] = ('', 'Insert') - lookup[313] = ('', 'Home') - lookup[366] = ('', 'Prior') - lookup[127] = ('', 'Del') - lookup[312] = ('', 'End') - lookup[367] = ('', 'Next') - - event.Skip() - ctrl, shift = event.ControlDown(), event.ShiftDown() - key, keysym = lookup[event.GetKeyCode()] - - # wxPython 2.6.0.1 does not return a valid event.Get{X,Y}() - # for this event, so we use the cached position. - (x,y)= self._vtk_control._Iren.GetEventPosition() - self._vtk_control._Iren.SetEventInformation(x, y, - ctrl, shift, key, 0, - keysym) - - self._vtk_control._Iren.KeyPressEvent() - self._vtk_control._Iren.CharEvent() - -except AttributeError: - #This is a QT4 window - from tvtk.pyface.ui.qt4 import scene - class FlatInteractor(scene._VTKRenderWindowInteractor): - def keyPressEvent(self, e): - scene.QVTKRenderWindowInteractor.keyPressEvent(self, e) - - class FlatScene(Scene): - def _create_control(self, parent): - """ Create the toolkit-specific control that represents the widget. """ - - # Create the VTK widget. - self._vtk_control = window = FlatInteractor(self, parent, - stereo=self.stereo) - - # Switch the default interaction style to the trackball one. - window.GetInteractorStyle().SetCurrentStyleToTrackballCamera() - - # Grab the renderwindow. - renwin = self._renwin = tvtk.to_tvtk(window.GetRenderWindow()) - renwin.set(point_smoothing=self.point_smoothing, - line_smoothing=self.line_smoothing, - polygon_smoothing=self.polygon_smoothing) - # Create a renderer and add it to the renderwindow - self._renderer = tvtk.Renderer() - renwin.add_renderer(self._renderer) - # Save a reference to our camera so it is not GC'd -- needed for - # the sync_traits to work. - self._camera = self.camera - - # Sync various traits. - self._renderer.background = self.background - self.sync_trait('background', self._renderer) - self.renderer.on_trait_change(self.render, 'background') - renwin.off_screen_rendering = self.off_screen_rendering - self._camera.parallel_projection = self.parallel_projection - self.sync_trait('parallel_projection', self._camera) - self.sync_trait('off_screen_rendering', self._renwin) - self.render_window.on_trait_change(self.render, 'off_screen_rendering') - self.render_window.on_trait_change(self.render, 'stereo_render') - self.render_window.on_trait_change(self.render, 'stereo_type') - self.camera.on_trait_change(self.render, 'parallel_projection') - - self._interactor = tvtk.to_tvtk(window._Iren) - - return window - -################################################################################ - -class Axis(HasTraits): - axis = Int - parent = Instance('Align') - invert = Bool(value=False) - - ipw_3d = Instance(PipelineBase) - ipw = Instance(PipelineBase) - cursor = Instance(Module) - surf = Instance(PipelineBase) - outline = Instance(PipelineBase) - slab = Instance(tvtk.ClipPolyData) - handle = Instance(RotationWidget) - planes = List - - scene_3d = DelegatesTo('parent') - position = DelegatesTo('parent') - disable_render = DelegatesTo('parent') - xfm = DelegatesTo('parent') - - outline_color = DelegatesTo('parent') - outline_rep = DelegatesTo('parent') - line_width = DelegatesTo('parent') - point_size = DelegatesTo('parent') - - def __init__(self, **kwargs): - super().__init__(**kwargs) - self.slab - self.outline - self.ipw_3d - self.ipw - self._last = -1 - self._keytime = None - - spacing = list(np.abs(self.parent.spacing)) - shape = list(self.parent.epi.shape) - spacing.pop(self.axis) - shape.pop(self.axis) - if self.axis == 1: - shape = shape[::-1] - spacing = spacing[::-1] - shape.append(0) - spacing.append(1) - self.spacing = np.array(spacing) - self.shape = np.array(shape) - - self.handle - - def reset_view(self): - center = self.shape * self.spacing / 2. + (self.shape + 1) % 2 * self.spacing / 2. - width = (self.shape * self.spacing)[:2] - width = np.min(width) * 0.5 - - self.scene.scene.background = (0, 0, 0) - mlab.view(*([(0, 0), (90, 0), (0, 0)][self.axis]), - focalpoint=center, - figure=self.scene.mayavi_scene) - self.scene.scene.parallel_projection = True - self.scene.scene.camera.parallel_scale = width * 1.2 - self.scene.scene.interactor.interactor_style = tvtk.InteractorStyleImage() - - try: #WX window - self.scene.scene_editor.control.SetFocusFromKbd - def focusfunc(vtkobj, i): - self.scene.scene_editor.control.SetFocusFromKbd() - except AttributeError: #QT window - self.scene.scene_editor.control.setFocus - def focusfunc(vtkobj, i): - self.scene.scene_editor.control.setFocus() - - self.scene.interactor.add_observer("MouseMoveEvent", focusfunc) - self.scene.interactor.add_observer("KeyReleaseEvent", self.handle_keys) - self._outline_color_changed() - - def handle_keys(self, evt, name): - key, sym = evt.GetKeyCode(), evt.GetKeySym() - #print repr(key), repr(sym), evt.GetShiftKey(), evt.GetControlKey() - - if key in ('', chr(127)): - i = -1 if self.invert else 1 - mult = (2,.2)[evt.GetShiftKey()] - smult = (1.1, 1.01)[evt.GetShiftKey()] - - rotccw, rotcw = "Insert", "Prior" - moves = dict(Up=(0,1,0), Down=(0,-1,0), Left=(-1,0,0), Right=(1,0,0)) - if self.invert: - rotccw, rotcw = rotcw, rotccw - moves = dict(Up=(1,0,0), Down=(-1,0,0), Left=(0,1,0), Right=(0,-1,0)) - - if sym in moves: - self.handle.move(np.array(moves[sym])*mult) - elif sym == rotccw: #ins - self.handle.move(angle=np.pi / 120.*i*mult) - elif sym == rotcw: #pgup - self.handle.move(angle=-np.pi / 120.*i*mult) - elif sym == "Del": - self.transform(scale=(smult, 1)) - elif sym == "Home": - self.transform(scale=(1, smult)) - elif sym == "End": - self.transform(scale=(1, 1/smult)) - elif sym == "Next": - self.transform(scale=(1/smult, 1)) - elif key == ']': - self.next_slice() - elif key == '[': - self.prev_slice() - elif key == 'H': - self.parent.outlines_visible = not self.parent.outlines_visible - elif key == 'Z' and evt.GetControlKey() == 1: - self.parent.undo() - - #clear out key buffer, otherwise the ctrl release will have the wrong state - evt.SetKeyEventInformation(0,0,'',0,'') - - @on_trait_change("parent.scene_3d.activated") - def activate_3d(self): - self.ipw_3d.ipw.interaction = 0 - self.surf - - def _planes_default(self): - pos = [0, 0, 0] - vec = [0, 0, 0] - off = (self.parent.epi.shape[self.axis] + 1) % 2 * abs(self.parent.spacing[self.axis]) / 2. - vec[self.axis] = 1 - pos[self.axis] = off + abs(self.parent.spacing[self.axis]) / 2. - top = tvtk.Planes(normals=[vec[:]], points=[pos[:]]) - vec[self.axis] = -1 - pos[self.axis] = off - abs(self.parent.spacing[self.axis]) / 2. - bot = tvtk.Planes(normals=[vec[:]], points=[pos[:]]) - return [top, bot] - - def _slab_default(self): - top = tvtk.ClipPolyData(clip_function=self.planes[0], inside_out=1, - input=self.parent.surf.parent.parent.filter.output) - bot = tvtk.ClipPolyData(clip_function=self.planes[1], inside_out=1, - input=top.output) - bot.update() - return bot - - def _outline_default(self): - origin, spacing = self.parent.origin, self.parent.spacing - translate = origin * np.sign(spacing) - np.abs(spacing) / 2. - - mlab.figure(self.scene.mayavi_scene) - if self.slab.output.points is None or len(self.slab.output.points) < 3: - pts = np.array([[0, 0, 0], [0, 0, 0], [0,0,0]]) - polys = [[0, 1, 2]] - else: - pts = self.slab.output.points.to_array() - polys = self.slab.output.polys.to_array().reshape(-1, 4)[:,1:] - src = mlab.pipeline.triangular_mesh_source(pts[:,0], pts[:,1], pts[:,2], polys, - figure=self.scene.mayavi_scene) - xfm = mlab.pipeline.transform_data(src, figure=self.scene.mayavi_scene) - xfm.filter.transform.post_multiply() - xfm.filter.transform.translate(-translate) - xfm.widget.enabled = False - surf = mlab.pipeline.surface(xfm, - figure=self.scene.mayavi_scene, - color=(1,1,1), - representation=self.outline_rep) - surf.actor.property.line_width = self.line_width - surf.actor.property.point_size = self.point_size - return src - - def _surf_default(self): - if self.slab.output.points is None or len(self.slab.output.points) < 3: - pts = np.array([[0, 0, 0], [0, 0, 0], [0,0,0]]) - polys = [[0, 1, 2]] - else: - pts = self.slab.output.points.to_array() - polys = self.slab.output.polys.to_array().reshape(-1, 4)[:,1:] - src = mlab.pipeline.triangular_mesh_source(pts[:,0], pts[:,1], pts[:,2], polys, - figure=self.scene_3d.mayavi_scene) - surf = mlab.pipeline.surface(src, - color=(1,1,1), - figure=self.scene_3d.mayavi_scene, - representation=self.outline_rep) - surf.actor.property.line_width = self.line_width - surf.actor.property.point_size = self.point_size - return src - - def _ipw_3d_default(self): - spos = self.position + self.parent.origin * np.sign(self.parent.spacing) - space = list(np.abs(self.parent.spacing)) - shape = list(np.array(self.parent.epi.shape) / self.parent.padshape) - space.pop(self.axis) - shape.pop(self.axis) - if self.axis == 1: - space = space[::-1] - shape = shape[::-1] - - space.append(1) - shape = [(0,0), (shape[0],0), (0, shape[1]), (shape[0],shape[1])] - origin = [space[0] / 2., space[1] / 2., 0] - - self.ipw_space = (space, shape) - - ipw = mlab.pipeline.image_plane_widget(self.parent.epi_src, - figure=self.scene_3d.mayavi_scene, - plane_orientation='%s_axes' % 'xyz'[self.axis], - name='Cut %s' % self.axis) - ipw.ipw.color_map.output_format = 'rgb' - ipw.ipw.set(texture_interpolate=0, reslice_interpolate='nearest_neighbour', slice_position=spos[self.axis]) - ipw.ipw.reslice.set(output_spacing=space, output_origin=origin) - ipw.ipw.poly_data_algorithm.output.point_data.t_coords = shape - return ipw - - def _ipw_default(self): - extent = list(np.abs(self.parent.epi.shape * self.parent.spacing)) - extent.pop(self.axis) - if self.axis == 1: - extent = extent[::-1] - - side_src = self.ipw_3d.ipw.reslice_output - # Add a callback on the image plane widget interaction to - # move the others - def move_view(obj, evt): - # Disable rendering on all scene - cpos = obj.GetCurrentCursorPosition() - position = list(cpos*side_src.spacing)[:2] - position.insert(self.axis, self.position[self.axis]) - # We need to special case y, as the view has been rotated. - if self.axis == 1: - position = position[::-1] - - self.position = position - - ipw = mlab.pipeline.image_plane_widget( side_src, - plane_orientation='z_axes', - figure=self.scene.mayavi_scene, - name='Cut view %s' % self.axis) - ipw.ipw.plane_property.opacity = 0 - ipw.ipw.selected_plane_property.opacity = 0 - ipw.ipw.poly_data_algorithm.set(point1=[extent[0], 0, 0], point2=[0, extent[1], 0]) - ipw.ipw.set( - left_button_action=0, - middle_button_auto_modifier=2, - right_button_auto_modifier=2, - texture_interpolate=0, - reslice_interpolate='nearest_neighbour') - ipw.parent.scalar_lut_manager.set( - use_default_range=False, - default_data_range=[-1,1], - data_range=[-1,1]) - ipw.ipw.add_observer('InteractionEvent', move_view) - ipw.ipw.add_observer('StartInteractionEvent', move_view) - return ipw - - def _cursor_default(self): - return mlab.points3d(*self.position, mode='axes', color=(0, 0, 0), - scale_factor=2*max(self.parent.epi[0].shape), figure=self.scene.mayavi_scene, - name='Cursor view %s' % self.axis) - - def _handle_default(self): - center = self.shape * self.spacing / 2. + (self.shape + 1) % 2 * self.spacing / 2. - width = (self.shape * self.spacing)[:2] - width = np.min(width) * 0.5 - def handlemove(handle, pos, angle, radius): - self.transform(pos, angle, radius) - return RotationWidget(self.scene.scene.mayavi_scene, handlemove, radius=width, pos=center) - - def _disable_render_changed(self): - self.scene.scene.disable_render = self.disable_render - - def toggle_outline(self): - self.outline.children[0].children[0].visible = self.parent.outlines_visible - - def _outline_color_changed(self): - try: - if isinstance(self.outline_color, str): - color = cc.to_rgb(self.outline_color) - else: - color = tuple([c/255. for c in tuple(self.outline_color)]) - except TypeError: - color = self.outline_color.getRgbF()[:3] - self.surf.children[0].children[0].actor.property.color = color - self.outline.children[0].children[0].children[0].actor.property.color = color - - def _outline_rep_changed(self): - self.surf.children[0].children[0].actor.property.representation = self.outline_rep - self.outline.children[0].children[0].children[0].actor.property.representation = self.outline_rep - - def _line_width_changed(self): - self.surf.children[0].children[0].actor.property.line_width = self.line_width - self.outline.children[0].children[0].children[0].actor.property.line_width = self.line_width - - def _point_size_changed(self): - self.surf.children[0].children[0].actor.property.point_size = self.point_size - self.outline.children[0].children[0].children[0].actor.property.point_size = self.point_size - - def next_slice(self): - '''View the next slice''' - pos = list(self.position) - pos[self.axis] += np.abs(self.parent.spacing)[self.axis] - self.position = pos - - def prev_slice(self): - '''View the previous slice''' - pos = list(self.position) - pos[self.axis] -= np.abs(self.parent.spacing)[self.axis] - self.position = pos - - def transform(self, pos=(0,0), angle=0, scale=1): - '''In-plane transformation function. Update the 3D transform based on the 2D changes''' - center = self.shape * self.spacing / 2. + (self.shape + 1) % 2 * self.spacing / 2. - inv = self.xfm.transform.homogeneous_inverse - - wpos = self.handle.center.representation.world_position - wpos -= center - if not isinstance(scale, (tuple, list, np.ndarray)): - scale = [scale, scale] - - if self.axis == 1: - trans = np.insert(pos[:2][::-1], self.axis, 0) - wpos = np.insert(wpos[:2][::-1], self.axis, self.ipw_3d.ipw.slice_position) - #angle = -angle - else: - trans = np.insert(pos[:2], self.axis, 0) - wpos = np.insert(wpos[:2], self.axis, self.ipw_3d.ipw.slice_position) - scale = np.insert(scale, self.axis, 1) - - self.parent._undolist.append(self.xfm.transform.matrix.to_array()) - - self.xfm.transform.post_multiply() - self.xfm.transform.translate(-wpos) - self.xfm.transform.rotate_wxyz(np.degrees(angle), *self.ipw_3d.ipw.normal) - self.xfm.transform.scale(scale) - self.xfm.transform.translate(wpos) - self.xfm.transform.translate(trans) - self.xfm.transform.pre_multiply() - - self.xfm.widget.set_transform(self.xfm.filter.transform) - self.xfm.update_pipeline() - self.parent.update_slabs() - - np.save("/tmp/last_xfm.npy", self.parent.get_xfm()) - - def update_position(self): - """ Update the position of the cursors on each side view, as well - as the image_plane_widgets in the 3D view. - """ - - offset = np.abs(self.parent.spacing) / 2 - p = list(self.position + offset) - p.pop(self.axis) - if self.axis == 1: - p = p[::-1] - p.append(0) - self.cursor.parent.parent.data.points = [p] - - if self.position[self.axis] != self._last: - self._last = self.position[self.axis] - origin = self.parent.origin * np.sign(self.parent.spacing) - - space, shape = self.ipw_space - self.ipw_3d.ipw.slice_position = self.position[self.axis] + origin[self.axis] - self.ipw_3d.ipw.reslice.set(output_spacing=space, output_origin=[space[0] / 2., space[1] / 2., 0]) - self.ipw_3d.ipw.poly_data_algorithm.output.point_data.t_coords = shape - self.ipw.ipw.poly_data_algorithm.output.point_data.t_coords = shape - - origin, spacing = self.parent.origin, self.parent.spacing - origin = origin * np.sign(spacing) - np.abs(spacing) / 2. - - gap = abs(spacing[self.axis]) / 2. - pos = self.ipw_3d.ipw.slice_position - pts = [0, 0, 0] - pts[self.axis] = pos+gap - self.planes[0].points = [tuple(pts)] - pts[self.axis] = pos-gap - self.planes[1].points = [tuple(pts)] - self.update_slab() - - def update_slab(self): - self.slab.update() - self.outline.data.set(points=self.slab.output.points, polys=self.slab.output.polys) - self.surf.data.set(points=self.slab.output.points, polys=self.slab.output.polys) - -class XAxis(Axis): - axis = 0 - scene = DelegatesTo('parent', 'scene_x') - def _outline_default(self): - surf = super()._outline_default() - surf.children[0].filter.transform.rotate_x(-90) - surf.children[0].filter.transform.rotate_y(-90) - return surf - -class YAxis(Axis): - axis = 1 - invert = True - scene = DelegatesTo('parent', 'scene_y') - def _outline_default(self): - surf = super()._outline_default() - surf.children[0].filter.transform.rotate_y(90) - surf.children[0].filter.transform.rotate_x(90) - return surf - -class ZAxis(Axis): - axis = 2 - scene = DelegatesTo('parent', 'scene_z') - -outline_reps = set(('wireframe', 'points', 'surface')) -try: - default_rep = options.config.get("mayavi_aligner", "outline_rep") - outline_reps = outline_reps - set([default_rep]) - outline_reps = (default_rep,) + tuple(outline_reps) -except: - outline_reps = tuple(outline_reps) - -class Align(HasTraits): - # The position of the view - position = Array(shape=(3,)) - - brightness = Range(-2., 2., value=0.) - contrast = Range(0., 3., value=1.) - opacity = Range(0., 1., value=.1) - colormap = Enum(*lut_manager.lut_mode_list()) - fliplut = Bool - - outlines_visible = Bool(default_value=True) - outline_rep = Enum(outline_reps) - outline_color = Color(default=options.config.get("mayavi_aligner", "outline_color")) - line_width = Range(0.5, 10., value=float(options.config.get("mayavi_aligner", "line_width"))) - point_size = Range(0.5, 10., value=float(options.config.get("mayavi_aligner", "point_size"))) - - epi_filter = Enum(None, "median", "gradient") - filter_strength = Range(1, 20, value=3) - - scene_3d = Instance(MlabSceneModel, ()) - scene_x = Instance(MlabSceneModel, ()) - scene_y = Instance(MlabSceneModel, ()) - scene_z = Instance(MlabSceneModel, ()) - - # The data source - epi_src = Instance(Source) - surf_src = Instance(Source) - xfm = Instance(Filter) - surf = Instance(Module) - - disable_render = Bool - - flip_fb = Bool - flip_lr = Bool - flip_ud = Bool - - save_callback = Instance(types.FunctionType) - save_btn = Button(label="Save Transform") - - legend = Str(legend) - - #--------------------------------------------------------------------------- - # Object interface - #--------------------------------------------------------------------------- - def __init__(self, pts, polys, epi, xfm=None, xfmtype='magnet', **traits): - ''' - Parameters - ---------- - xfm : array_like, optional - The initial 4x4 rotation matrix into magnet space - (epi with slice affine) - ''' - self.load_epi(epi, xfm, xfmtype) - self.pts, self.polys = pts, polys - self._undolist = [] - self._redo = None - super().__init__(**traits) - - def load_epi(self, epifilename, xfm=None, xfmtype="magnet"): - """Loads the EPI image from the specified epifilename. - """ - nii = nibabel.load(epifilename) - self.epi_file = nii - epi = nii.get_fdata().astype(float).squeeze() - if epi.ndim>3: - epi = epi[:,:,:,0] - self.affine = nii.affine - base = nii.header.get_base_affine() - self.base = base - self.origin = base[:3, -1] - self.spacing = np.diag(base)[:3] - if xfm is None: - self.startxfm = np.dot(base, np.linalg.inv(self.affine)) - elif xfmtype == "magnet": - self.startxfm = np.dot(np.dot(base, np.linalg.inv(self.affine)), xfm) - else: - print("using xfmtype %s"%xfmtype) - self.startxfm = xfm - - self.center = self.spacing*nii.get_shape()[:3] / 2 + self.origin - - self.padshape = 2**(np.ceil(np.log2(np.array(epi.shape)))) - - epi = np.nan_to_num(epi) - self.epi_orig = epi - epi.min() - self.epi_orig /= self.epi_orig.max() - self.epi_orig *= 2 - self.epi_orig -= 1 - self.epi = self.epi_orig.copy() - - #--------------------------------------------------------------------------- - # Default values - #--------------------------------------------------------------------------- - def _position_default(self): - return np.abs(self.origin) + ((np.array(self.epi.shape)+1)%2) * np.abs(self.spacing) / 2 - - def _epi_src_default(self): - sf = mlab.pipeline.scalar_field(self.epi, - figure=self.scene_3d.mayavi_scene, - name='EPI') - sf.origin = self.origin - sf.spacing = self.spacing - return sf - - def _surf_src_default(self): - return mlab.pipeline.triangular_mesh_source( - self.pts[:,0], self.pts[:,1], self.pts[:,2], self.polys, - figure=self.scene_3d.mayavi_scene, - name='Cortex') - - def _surf_default(self): - smooth = mlab.pipeline.poly_data_normals(self.xfm, figure=self.scene_3d.mayavi_scene) - smooth.filter.splitting = False - surf = mlab.pipeline.surface(smooth, figure=self.scene_3d.mayavi_scene) - surf.actor.mapper.scalar_visibility = 0 - return surf - - def _xfm_default(self): - xfm = mlab.pipeline.transform_data(self.surf_src, figure=self.scene_3d.mayavi_scene) - def savexfm(info, evt): - self._undolist.append(xfm.transform.matrix.to_array()) - np.save("/tmp/last_xfm.npy", self.get_xfm()) - - xfm.widget.add_observer("EndInteractionEvent", savexfm) - xfm.widget.add_observer("EndInteractionEvent", self.update_slabs) - xfm.transform.set_matrix(self.startxfm.ravel()) - xfm.widget.set_transform(xfm.transform) - return xfm - - #--------------------------------------------------------------------------- - # Scene activation callbacks - #--------------------------------------------------------------------------- - @on_trait_change('scene_3d.activated') - def display_scene_3d(self): - self.scene_3d.mlab.view(40, 50) - self.scene_3d.scene.renderer.use_depth_peeling = True - self.scene_3d.scene.background = (0, 0, 0) - # Keep the view always pointing up - self.scene_3d.scene.interactor.interactor_style = tvtk.InteractorStyleTerrain() - self.scene_3d.scene_editor.aligner = self - - self.opacity = float(options.config.get("mayavi_aligner", "opacity")) - self.xfm.widget.enabled = False - self.colormap = options.config.get("mayavi_aligner", "colormap") - - self.disable_render = True - for ax in [self.x_axis, self.y_axis, self.z_axis]: - ax.update_position() - ax.reset_view() - self.disable_render = False - - @on_trait_change('scene_x.activated') - def display_scene_x(self): - self.x_axis = XAxis(parent=self) - - @on_trait_change('scene_y.activated') - def display_scene_y(self): - self.y_axis = YAxis(parent=self) - - @on_trait_change('scene_z.activated') - def display_scene_z(self): - self.z_axis = ZAxis(parent=self) - - #--------------------------------------------------------------------------- - # Traits callback - #--------------------------------------------------------------------------- - - def _save_btn_changed(self): - if self.save_callback is not None: - self.save_callback(self) - - def _disable_render_changed(self): - self.scene_3d.scene.disable_render = self.disable_render - - def _position_changed(self): - self.disable_render = True - for ax in [self.x_axis, self.y_axis, self.z_axis]: - ax.update_position() - self.disable_render = False - - def _outlines_visible_changed(self): - self.disable_render = True - for ax in [self.x_axis, self.y_axis, self.z_axis]: - ax.toggle_outline() - self.disable_render = False - - @on_trait_change("colormap, fliplut") - def update_colormap(self): - for ax in [self.x_axis, self.y_axis, self.z_axis]: - if ax.ipw_3d and ax.ipw: - ax.ipw_3d.parent.scalar_lut_manager.set(lut_mode=self.colormap, reverse_lut=self.fliplut) - ax.ipw.parent.scalar_lut_manager.set(lut_mode=self.colormap, reverse_lut=self.fliplut) - - def _opacity_changed(self): - self.surf.actor.property.opacity = self.opacity - - @on_trait_change("brightness,contrast") - def update_brightness(self): - self.epi_src.scalar_data = (self.epi*self.contrast)+self.brightness - - @on_trait_change("flip_ud") - def update_flipud(self): - #self.epi_src.scalar_data = self.epi_src.scalar_data[:,:,::-1] - flip = np.eye(4) - flip[2,2] = -1 - mat = self.xfm.transform.matrix.to_array() - self.set_xfm(np.dot(mat, flip), "base") - - @on_trait_change("flip_lr") - def update_fliplr(self): - #self.epi_src.scalar_data = self.epi_src.scalar_data[::-1] - flip = np.eye(4) - flip[0,0] = -1 - mat = self.xfm.transform.matrix.to_array() - self.set_xfm(np.dot(mat, flip), "base") - - @on_trait_change("flip_fb") - def update_flipfb(self): - #self.epi_src.scalar_data = self.epi_src.scalar_data[:,::-1] - flip = np.eye(4) - flip[1,1] = -1 - mat = self.xfm.transform.matrix.to_array() - self.set_xfm(np.dot(mat, flip), "base") - - @on_trait_change("epi_filter, filter_strength") - def update_epifilter(self): - if self.epi_filter is None: - self.epi = self.epi_orig.copy() - elif self.epi_filter == "median": - fstr = np.floor(self.filter_strength / 2)*2+1 - self.epi = volume.detrend_median(self.epi_orig.T, fstr).T - elif self.epi_filter == "gradient": - self.epi = volume.detrend_gradient(self.epi_orig.T, self.filter_strength).T - - self.update_brightness() - - def update_slabs(self, *args, **kwargs): - self.disable_render = True - for ax in [self.x_axis, self.y_axis, self.z_axis]: - ax.update_slab() - self.disable_render = False - - def get_xfm(self, xfmtype="magnet"): - if xfmtype in ["anat->epicoord", "coord"]: - ibase = np.linalg.inv(self.base) - xfm = self.xfm.transform.matrix.to_array() - return np.dot(ibase, xfm) - elif xfmtype in ["anat->epibase", "base"]: - return self.xfm.transform.matrix.to_array() - elif xfmtype in ['anat->magnet', "magnet"]: - ibase = np.linalg.inv(self.base) - xfm = self.xfm.transform.matrix.to_array() - return np.dot(self.affine, np.dot(ibase, xfm)) - - def set_xfm(self, matrix, xfmtype='magnet'): - assert xfmtype in "magnet coord base".split(), "Unknown transform type" - if xfmtype == "coord": - matrix = np.dot(self.base, matrix) - elif xfmtype == "magnet": - iaff = np.linalg.inv(self.affine) - matrix = np.dot(self.base, np.dot(iaff, matrix)) - - self.xfm.transform.set_matrix(matrix.ravel()) - self.xfm.widget.set_transform(self.xfm.transform) - self.xfm.update_pipeline() - self.update_slabs() - - def undo(self): - if len(self._undolist) > 0: - self.xfm.transform.set_matrix(self._undolist[-1].ravel()) - self.xfm.widget.set_transform(self.xfm.transform) - self.xfm.update_pipeline() - self.update_slabs() - self._redo = self._undolist.pop() - - #--------------------------------------------------------------------------- - # The layout of the dialog created - #--------------------------------------------------------------------------- - view = View(HGroup( - Group( - Item('scene_y', - editor=SceneEditor(scene_class=FlatScene)), - Item('scene_z', - editor=SceneEditor(scene_class=FlatScene)), - show_labels=False, - ), - Group( - Item('scene_x', - editor=SceneEditor(scene_class=FlatScene)), - Item('scene_3d', - editor=SceneEditor(scene_class=ThreeDScene)), - show_labels=False, - ), - Group( - Group(Item("save_btn", show_label=False, visible_when="save_callback is not None"), - "brightness", "contrast", "epi_filter", - Item('filter_strength', visible_when="epi_filter is not None"), - "_", "opacity", "_", - Item('colormap', - editor=ImageEnumEditor(values=lut_manager.lut_mode_list(), - cols=6, - path=lut_manager.lut_image_dir)), - "fliplut", - "_", "flip_ud", "flip_lr", "flip_fb", - "_", Item('outline_color', editor=ColorEditor()), 'outline_rep', 'line_width', 'point_size', - '_', - ), - Group( - Item('legend', editor=TextEditor(), style='readonly', show_label=False, emphasized=True, dock='vertical'), - show_labels=False, - ), - orientation='vertical' - ), - ), - resizable=True, - title='Aligner' - ) - -def get_aligner(subject, xfmname, epifile=None, xfm=None, xfmtype="magnet", decimate=True): - from .database import db - - dbxfm = None - try: - dbxfm = db.get_xfm(subject, xfmname, xfmtype='magnet') - epifile = dbxfm.reference.get_filename() - dbxfm = dbxfm.xfm - except IOError: - pass - - try: - wpts, wpolys = db.get_surf(subject, 'wm', merge=True, nudge=False) - ppts, ppolys = db.get_surf(subject, 'pia', merge=True, nudge=False) - pts = np.vstack([wpts, ppts]) - polys = np.vstack([wpolys, ppolys+len(wpts)]) - except IOError: - pts, polys = db.get_surf(subject, 'fiducial', merge=True, nudge=False) - - if decimate: - pts, polys = polyutils.decimate(pts, polys) - - return Align(pts, polys, epifile, xfm=dbxfm if xfm is None else xfm, xfmtype=xfmtype) diff --git a/cortex/polyutils/__init__.py b/cortex/polyutils/__init__.py index 8444ba9f0..dae7651bc 100644 --- a/cortex/polyutils/__init__.py +++ b/cortex/polyutils/__init__.py @@ -13,6 +13,8 @@ face_area, face_volume, decimate, + decimate_faces, + vertex_occlusion, inside_convex_poly, make_cube, boundary_edges, diff --git a/cortex/polyutils/misc.py b/cortex/polyutils/misc.py index 4660014c8..580aaf640 100644 --- a/cortex/polyutils/misc.py +++ b/cortex/polyutils/misc.py @@ -1,6 +1,7 @@ import io +from scipy import sparse from scipy.spatial import Delaunay import numpy as np import functools @@ -75,6 +76,183 @@ def decimate(pts, polys): dpolys = dec.output.polys.to_array().reshape(-1, 4)[:,1:] return dpts, dpolys +def decimate_faces(pts, polys, cell, split=None): + """Faces for drawing a surface coarsely, over a subset of its vertices. + + The vertices that fall in one box of a grid of `cell` mm over `pts`, and + that are joined to one another across the mesh, stand for one vertex of + the coarse surface: the one of them nearest the middle of the group. A box + holding two pieces of surface that run past each other, the two banks of a + sulcus for one, has them as separate groups, since they are joined to each + other only around the fold and so outside the box. + + The faces come back written in the numbering of `pts`, which is what lets + them be drawn from the arrays an existing surface is already held in: only + the triangles change, and every vertex of a triangle is a vertex of the + surface they were cut from. + + Parameters + ---------- + pts : array (n, 3) + The vertices of the surface the faces are cut from. + polys : array (m, 3) + Its triangles. + cell : float + The width of a box of the grid, in the units of `pts`. A surface is + left as it is when this is 0 or less. + split : array (n,), optional + A label per vertex that a group is never allowed to cross, such as + which side of a cut a vertex is on. Vertices of different labels are + put in different groups however close together they are. + + Returns + ------- + verts : array (k,) + The vertices of `pts` the coarse surface is drawn from. + faces : array (j, 3) + Its triangles, in the numbering of `pts`. + """ + pts = np.asarray(pts, dtype=float) + polys = np.asarray(polys) + if cell is None or cell <= 0: + return np.arange(len(pts)), polys.astype(np.uint32) + + grid = np.floor((pts - pts.min(0)) / cell).astype(np.int64) + span = grid.max(0) + 1 + if np.prod(span, dtype=float) < 2 ** 62: + #one number per box, which sorts far faster than the rows do + boxes = np.unique((grid * [1, span[0], span[0] * span[1]]).sum(1), + return_inverse=True)[1] + else: + boxes = np.unique(grid, axis=0, return_inverse=True)[1] + + #a box holds as many groups as it has pieces of surface in it + edges = np.vstack([polys[:, [0, 1]], polys[:, [1, 2]], polys[:, [0, 2]]]) + together = boxes[edges[:, 0]] == boxes[edges[:, 1]] + if split is not None: + split = np.asarray(split) + together &= split[edges[:, 0]] == split[edges[:, 1]] + joined = edges[together] + graph = sparse.coo_matrix( + (np.ones(len(joined), dtype=np.int8), (joined[:, 0], joined[:, 1])), + shape=(len(pts), len(pts))) + ngroups, groups = sparse.csgraph.connected_components(graph, directed=False) + + #the vertex nearest the middle of a group stands for it: writing the + #groups' vertices in from the farthest leaves the nearest one written last + middles = np.zeros((ngroups, 3)) + for axis in range(3): + middles[:, axis] = np.bincount(groups, weights=pts[:, axis], minlength=ngroups) + middles /= np.bincount(groups, minlength=ngroups)[:, None] + order = np.argsort(-((pts - middles[groups]) ** 2).sum(1)) + verts = np.zeros(ngroups, dtype=np.int64) + verts[groups[order]] = order + + faces = verts[groups[polys]] + #a face whose vertices are not three different groups has no area left, + #and several faces of the surface can come down to the same triangle + kept = ((faces[:, 0] != faces[:, 1]) & (faces[:, 1] != faces[:, 2]) & + (faces[:, 0] != faces[:, 2])) + faces = faces[kept] + first = np.unique(np.sort(faces, axis=1), axis=0, return_index=True)[1] + return np.sort(verts), faces[np.sort(first)].astype(np.uint32) + + +def vertex_occlusion(pts, polys, distance=12., voxel=1., rays=32): + """How much of the sky over each vertex the surface itself blocks. + + Ambient occlusion, baked a vertex at a time: the surface is set in a grid + of `voxel` mm cells, and from each vertex `rays` directions over the + hemisphere of its normal, more of them near the normal, are followed + through the grid for `distance` mm. The occlusion is the share of them + that meet the surface, 0 on a plane or over a gyral crown and near 1 at + the bottom of a deep sulcus. + + Parameters + ---------- + pts : (N, 3) array + The vertices, in mm. + polys : (M, 3) array + The faces, as indices into `pts`. The normals, and so which side of + the surface the sky is on, follow their winding. + distance : float, optional + How far a ray is followed, in mm: the surface beyond it does not + count. 12 by default, which takes in the far bank of a sulcus but + little of the gyri across from a crown. + voxel : float, optional + The width of a cell of the grid, in mm; 1 by default. + rays : int, optional + How many directions are followed from each vertex; 32 by default. + + Returns + ------- + occlusion : (N,) float32 array + The occlusion of each vertex, 0 to 1. A vertex in no face has no + normal to look out from and gets 0. + """ + pts = np.asarray(pts, dtype=np.float64) + polys = np.asarray(polys, dtype=np.int64) + corners = pts[polys] + + #the normal at a vertex, from the faces around it weighted by their area + fnorm = np.cross(corners[:, 1] - corners[:, 0], corners[:, 2] - corners[:, 0]) + normals = np.zeros_like(pts) + for c in range(3): + np.add.at(normals, polys[:, c], fnorm) + length = np.linalg.norm(normals, axis=1) + has = length > 0 + normals[has] /= length[has, None] + + #the surface in the grid: points over every face, close enough together + #that each cell a face passes through is marked + origin = pts.min(0) - 2 * voxel + shape = np.ceil((pts.max(0) - origin) / voxel).astype(int) + 3 + grid = np.zeros(int(np.prod(shape)), dtype=bool) + edges = np.linalg.norm(corners - np.roll(corners, 1, axis=1), axis=2) + across = int(np.ceil(edges.max() / voxel)) + 1 if len(edges) else 1 + for i in range(across + 1): + for j in range(across + 1 - i): + a, b = i / across, j / across + sample = corners[:, 0] * a + corners[:, 1] * b + corners[:, 2] * (1 - a - b) + cell = np.floor((sample - origin) / voxel).astype(int) + grid[np.ravel_multi_index(cell.T, shape)] = True + + #the directions, spread over the hemisphere with more of them near the + #normal (the sky nearer the normal lights a surface more), the same ones + #turned to the normal of each vertex + k = np.arange(rays) + 0.5 + radius = np.sqrt(k / rays) + angle = 2 * np.pi * ((k * 0.6180339887498949) % 1.) + local = np.stack([radius * np.cos(angle), radius * np.sin(angle), + np.sqrt(1 - k / rays)], axis=1) + helper = np.where(np.abs(normals[:, :1]) < 0.9, [[1., 0., 0.]], [[0., 1., 0.]]) + tangent = np.cross(normals, helper) + tlen = np.linalg.norm(tangent, axis=1) + tangent[has] /= tlen[has, None] + bitangent = np.cross(normals, tangent) + + #followed from a cell off the surface, so that the cells the surface + #around the vertex is in do not count as blocking it + steps = int(np.ceil(distance / voxel)) + start = pts + normals * voxel + occlusion = np.zeros(len(pts), dtype=np.float32) + chunk = 16384 + for lo in range(0, len(pts), chunk): + hi = min(lo + chunk, len(pts)) + dirs = (local[None, :, 0, None] * tangent[lo:hi, None, :] + + local[None, :, 1, None] * bitangent[lo:hi, None, :] + + local[None, :, 2, None] * normals[lo:hi, None, :]) + hit = np.zeros((hi - lo, rays), dtype=bool) + for step in range(1, steps + 1): + cell = np.floor((start[lo:hi, None, :] + dirs * (step * voxel) - origin) / voxel).astype(int) + inside = np.all((cell >= 0) & (cell < shape), axis=2) + flat = np.ravel_multi_index(np.clip(cell, 0, shape - 1).reshape(-1, 3).T, shape) + hit |= inside & grid[flat].reshape(hi - lo, rays) + occlusion[lo:hi] = hit.mean(axis=1) + occlusion[~has] = 0 + return occlusion + + def inside_convex_poly(pts): """Returns a function that checks if inputs are inside the convex hull of polyhedron defined by pts diff --git a/cortex/tests/test_nan_alpha.py b/cortex/tests/test_nan_alpha.py index a1f9d770b..2e900076d 100644 --- a/cortex/tests/test_nan_alpha.py +++ b/cortex/tests/test_nan_alpha.py @@ -281,7 +281,8 @@ def test_rgb_multiframe_nan_reaches_webgl(cls: str) -> None: try: def fetch(path: str) -> bytes: - url = "http://localhost:%d/%s" % (server.port, path) + #the address carries the session token the server demands + url = server.url(path, host="localhost") with urlopen(url, timeout=30) as response: return response.read() diff --git a/cortex/tests/test_utils.py b/cortex/tests/test_utils.py index 8fb07b4a7..f6c1caa3d 100644 --- a/cortex/tests/test_utils.py +++ b/cortex/tests/test_utils.py @@ -88,3 +88,170 @@ def test_get_roi_masks_missing_roi_does_not_fail_when_not_required(): ) assert "V1" in result assert "NotARealROI" not in result + + +def test_decimate_faces_draws_from_the_vertices_it_was_given(): + """The coarse faces are written over a subset of the original vertices, + which is what lets a viewer draw them from the arrays it already holds.""" + import numpy as np + + from cortex import polyutils + + pts, polys = cortex.db.get_surf("S1", "pia", merge=False)[0] + verts, faces = polyutils.decimate_faces(pts, polys, 8) + + assert len(faces) < len(polys) / 10 + assert faces.max() < len(pts), "a face points past the end of the surface" + assert set(np.unique(faces)) <= set(verts.tolist()), ( + "a face is drawn from a vertex that is not one of the ones kept") + #a face with a vertex twice over has no area left to draw + assert (faces[:, 0] != faces[:, 1]).all() + assert (faces[:, 1] != faces[:, 2]).all() + assert (faces[:, 0] != faces[:, 2]).all() + #and the same triangle is in there once + assert len(np.unique(np.sort(faces, axis=1), axis=0)) == len(faces) + + #a wider group leaves fewer faces + assert len(polyutils.decimate_faces(pts, polys, 16)[1]) < len(faces) + #and no group at all leaves the surface as it is + assert len(polyutils.decimate_faces(pts, polys, 0)[1]) == len(polys) + + +def test_decimate_faces_keeps_the_labels_apart(): + """A label a group may not cross, the medial wall for one, puts the + vertices on either side of it in groups of their own.""" + import numpy as np + + from cortex import polyutils + + #a strip of a surface, as one row of squares cut into triangles, with its + #two halves under different labels and narrow enough to be one group + x = np.arange(9.) + pts = np.vstack([np.repeat(x, 2), np.tile([0., 1.], len(x)), + np.zeros(2 * len(x))]).T + polys = np.array([[i, i + 1, i + 2] for i in range(2 * len(x) - 2)]) + split = pts[:, 0] >= 4 + + whole = polyutils.decimate_faces(pts, polys, 100)[0] + halves = polyutils.decimate_faces(pts, polys, 100, split=split)[0] + assert len(whole) == 1, "the strip is one group without a label" + assert len(halves) == 2, "the label did not divide it" + assert split[halves[0]] != split[halves[1]], ( + "the two groups stand for the same side of the label") + + +def test_get_lod_is_built_once_and_read_back(tmp_path): + """The coarse faces are cut when they are first asked for and kept.""" + import os + + import numpy as np + + from cortex import brainctm, utils + + ctmargs = dict(method="mg2", level=9) + path = utils.get_lod("S1", 16, **ctmargs) + assert os.path.exists(path) + written = os.path.getmtime(path) + assert utils.get_lod("S1", 16, **ctmargs) == path + assert os.path.getmtime(path) == written, "the file was cut a second time" + + raw = np.fromfile(path, dtype=" 0.6, "the floor of the shaft sees too much sky" + #the top edge of a wall looks half into the shaft and half out of it + assert 0.3 < walls[:, :, -1].mean() < 0.7 + assert floor.mean() > walls[:, :, -1].mean(), "the floor is no darker than the rim" + assert len(occlusion) == len(np.vstack(pts)) + assert occlusion.dtype == np.float32 + + +def test_get_occlusion_is_built_once_and_read_back(): + """The occlusion is worked out when it is first asked for and kept.""" + import json + import os + + import numpy as np + + from cortex import brainctm, utils + + ctmargs = dict(method="mg2", level=9) + path = utils.get_occlusion("S1", **ctmargs) + assert os.path.exists(path) + written = os.path.getmtime(path) + assert utils.get_occlusion("S1", **ctmargs) == path + assert os.path.getmtime(path) == written, "the file was built a second time" + + base = os.path.splitext(utils.get_ctmpack("S1", **ctmargs))[0] + with open(base + ".json") as fp: + pack = json.load(fp) + counts = [len(pts) for pts, _ in brainctm.read_pack(base + ".ctm")] + head = np.fromfile(path, dtype=" 0.05, "the two depths came out the same" + assert inflated.mean() < pial.mean() / 5, "inflation did not open the sulci" diff --git a/cortex/tests/test_webgl_aligner.py b/cortex/tests/test_webgl_aligner.py new file mode 100644 index 000000000..a7aa46fc4 --- /dev/null +++ b/cortex/tests/test_webgl_aligner.py @@ -0,0 +1,1123 @@ +"""Tests for the browser-based manual aligner (cortex.webgl.aligner). + +The pure-python tests cover the world frame the aligner works in and the +tornado endpoints of its server. The browser test drives the aligner in +headless Chromium and is skipped without playwright. +""" + +import base64 +import io +import json +import os +import time +import urllib.error +import urllib.parse +import urllib.request + +import numpy as np +import pytest + +import cortex +from cortex import align, database +from cortex.webgl import aligner +from cortex.tests.testing_utils import has_playwright + +subj = "S1" +xfmname = "fullhead" + + +def _reference(): + return database.db.get_xfm(subj, xfmname).reference_nifti + + +def _open(url, data=None, timeout=30): + """Fetch `url`; posting `data` (a dict) as a form when given.""" + body = None if data is None else urllib.parse.urlencode(data).encode() + with urllib.request.urlopen(url, data=body, timeout=timeout) as resp: + return resp.read() + + +def _url(srv, page="", host="localhost"): + """The address of a page of `srv`, carrying its session token.""" + return srv.url(page, host=host) + + +def _save_token(srv): + """The token the page is served with, which a save has to carry back.""" + return _page_config(_open(_url(srv, "aligner.html")).decode())["save_token"] + + +def _post_save(srv, **data): + """Post a save the way the page does, both tokens and all.""" + data.setdefault("save_token", _save_token(srv)) + return json.loads(_open(_url(srv, "save"), data).decode()) + + +class _SaveRecorder: + """Stands in for db.save_xfm, so tests never write into the filestore.""" + + def __init__(self): + self.calls = [] + + def __call__(self, subject, name, xfm, xfmtype="magnet", reference=None): + self.calls.append(dict(subject=subject, name=name, xfm=np.asarray(xfm, dtype=float), + xfmtype=xfmtype, reference=reference)) + + def wait(self, count=1, timeout=20): + deadline = time.monotonic() + timeout + while len(self.calls) < count and time.monotonic() < deadline: + time.sleep(0.1) + return len(self.calls) >= count + + +@pytest.fixture +def recorder(monkeypatch): + rec = _SaveRecorder() + monkeypatch.setattr(database.db, "save_xfm", rec) + return rec + + +@pytest.fixture(autouse=True) +def mask_dir(tmp_path, monkeypatch): + """Point the transform's mask paths at an empty directory of this test's. + + Saving deletes the masks cached for a transform, and `db.get_mask` writes + them on demand, so without this a test would delete or create masks in the + bundled filestore, and a mask another test file left there would decide + this one's results. Only the mask paths move; every other path stays as it + is. + """ + real_get_paths = database.db.get_paths + masks = tmp_path / "masks" + (masks / xfmname).mkdir(parents=True) + + def get_paths(subject): + paths = dict(real_get_paths(subject)) + # a directory per transform, as the filestore has + paths["masks"] = str(masks / "{xfmname}" / "mask_{type}.nii.gz") + return paths + + monkeypatch.setattr(database.db, "get_paths", get_paths) + return masks / xfmname + + +@pytest.fixture +def stale_masks(mask_dir): + """Two masks cached for the transform, as an earlier alignment left them.""" + names = ["thick", "thin"] + for name in names: + (mask_dir / ("mask_%s.nii.gz" % name)).write_bytes(b"a stale mask") + return mask_dir, names + + +def _page_config(html): + """The config object the page is generated with.""" + marker = 'viewer = figure.add(aligner.Aligner, "main", true, ' + start = html.index(marker) + len(marker) + return json.loads(html[start:html.index(");", start)]) + + +@pytest.fixture +def server(request): + """A running aligner server for the bundled transform, stopped at teardown.""" + kwargs = dict(open_browser=False, display_url=False) + kwargs.update(getattr(request, "param", {})) + srv = aligner.show(subj, xfmname, **kwargs) + yield srv + srv.stop() + + +# --------------------------------------------------------------------------- +# World frame and reference loading +# --------------------------------------------------------------------------- + + +def test_reference_frame_is_scaled_signed_permutation(): + """Each voxel axis maps to one world axis, scaled by its voxel size.""" + import nibabel + + nii = _reference() + world = aligner.reference_frame(nii) + zooms = np.asarray(nii.header.get_zooms()[:3]) + + assert world.shape == (4, 4) + assert np.allclose(world[3], [0, 0, 0, 1]) + assert np.allclose(world[:3, 3], 0) + linear = world[:3, :3] + # one nonzero entry per row and per column, of the voxel size + assert np.array_equal((linear != 0).sum(axis=0), [1, 1, 1]) + assert np.array_equal((linear != 0).sum(axis=1), [1, 1, 1]) + assert np.allclose(np.abs(linear).sum(axis=0), zooms) + + # the bundled reference is stored L, P, S: x and y flip, z does not + assert nibabel.aff2axcodes(nii.affine) == ("L", "P", "S") + assert world[0, 0] < 0 and world[1, 1] < 0 and world[2, 2] > 0 + + +def test_reference_frame_follows_axis_permutation(): + """A reference stored in a different axis order gets its axes permuted + so that world x, y, z point right, anterior and superior.""" + import nibabel + + # voxel axes are (superior, right, anterior) with 2, 3 and 4 mm voxels + affine = np.array([[0, 3.0, 0, 0], + [0, 0, 4.0, 0], + [2.0, 0, 0, 0], + [0, 0, 0, 1]]) + nii = nibabel.Nifti1Image(np.zeros((5, 6, 7), dtype=np.float32), affine) + world = aligner.reference_frame(nii) + expected = np.array([[0, 3.0, 0, 0], + [0, 0, 4.0, 0], + [2.0, 0, 0, 0], + [0, 0, 0, 1]]) + assert np.allclose(world, expected) + + +def test_load_reference_takes_first_volume_and_drops_nans(): + import nibabel + + data = np.random.RandomState(0).rand(4, 5, 6, 3) + data[0, 0, 0, 0] = np.nan + nii = nibabel.Nifti1Image(data, np.eye(4)) + epi = aligner.load_reference(nii) + assert epi.shape == (4, 5, 6) + assert epi.dtype == np.float32 + assert epi[0, 0, 0] == 0 + assert np.allclose(epi[1:], data[1:, :, :, 0]) + + +# --------------------------------------------------------------------------- +# Argument checks +# --------------------------------------------------------------------------- + + +def test_new_transform_requires_reference(): + with pytest.raises(ValueError, match="does not exist"): + aligner.show(subj, "aligner_test_missing_xfm", open_browser=False, display_url=False) + + +def test_existing_transform_refuses_new_reference(): + with pytest.raises(ValueError, match="Refusing to overwrite"): + aligner.show(subj, xfmname, reference=_reference().get_filename(), + open_browser=False, display_url=False) + + +def test_aligner_opens_with_cached_masks(stale_masks): + """Cached masks no longer keep a transform from being edited; the page + is told about them so it can warn that saving deletes them.""" + srv = aligner.show(subj, xfmname, open_browser=False, display_url=False) + try: + config = _page_config(_open(_url(srv)).decode()) + finally: + srv.stop() + assert config["view_only"] is False + assert config["masks"] == ["mask_thick.nii.gz", "mask_thin.nii.gz"] + + +def test_align_entry_point_names_every_option(monkeypatch): + """webgl_manual spells out every option the aligner takes, rather than + forwarding an opaque **kwargs, and passes them all on.""" + import inspect + + entry = inspect.signature(align.webgl_manual).parameters + shown = inspect.signature(aligner.show).parameters + assert not any(p.kind is inspect.Parameter.VAR_KEYWORD for p in entry.values()) + assert list(entry) == list(shown), "the entry point and the aligner disagree" + for name, param in shown.items(): + assert entry[name].default == param.default, "%s defaults differ" % name + + seen = {} + + def fake_show(subject, name, **kwargs): + seen.update(subject=subject, name=name, kwargs=kwargs) + return "handle" + + monkeypatch.setattr(aligner, "show", fake_show) + assert align.webgl_manual(subj, xfmname, view_only=True, port=1234) == "handle" + assert seen["subject"] == subj and seen["name"] == xfmname + assert seen["kwargs"]["view_only"] is True and seen["kwargs"]["port"] == 1234 + # everything after subject and xfmname is passed by name + assert set(seen["kwargs"]) == set(list(shown)[2:]) + + +# --------------------------------------------------------------------------- +# Stale masks +# --------------------------------------------------------------------------- + + +def test_cached_masks_are_listed_and_cleared(stale_masks): + mask_dir, names = stale_masks + paths = aligner.cached_masks(subj, xfmname) + assert [os.path.basename(p) for p in paths] == ["mask_thick.nii.gz", "mask_thin.nii.gz"] + # clear_masks reports the names db.get_mask takes, not the filenames + assert aligner.clear_masks(subj, xfmname) == names + assert aligner.cached_masks(subj, xfmname) == [] + assert sorted(mask_dir.glob("mask_*")) == [] + + +def test_clearing_a_transform_without_masks_does_nothing(mask_dir): + assert aligner.cached_masks(subj, xfmname) == [] + assert aligner.clear_masks(subj, xfmname) == [] + + +def test_save_deletes_the_stale_masks_first(stale_masks): + """Saving an edited alignment deletes the masks cut with the old one. + + They have to be gone before the transform is written: db.save_xfm + refuses to write over a transform that still has masks. + """ + mask_dir, names = stale_masks + seen = {} + + def save_xfm(subject, name, xfm, xfmtype="magnet", reference=None): + seen["masks"] = sorted(p.name for p in mask_dir.glob("mask_*")) + seen["xfm"] = np.asarray(xfm, dtype=float) + seen["xfmtype"] = xfmtype + + srv = aligner.show(subj, xfmname, open_browser=False, display_url=False) + xfm = np.arange(16, dtype=float).reshape(4, 4) + try: + token = _save_token(srv) + with pytest.MonkeyPatch.context() as patch: + patch.setattr(database.db, "save_xfm", save_xfm) + resp = _post_save(srv, xfm=json.dumps(xfm.tolist()), save_token=token) + finally: + srv.stop() + + assert resp["status"] == "ok" + assert resp["masks_deleted"] == names + assert "deleted 2 stale masks (thick, thin)" in resp["message"] + assert seen["masks"] == [], "the masks were still there when the transform was written" + assert seen["xfmtype"] == "coord" + assert np.allclose(seen["xfm"], xfm) + assert sorted(mask_dir.glob("mask_*")) == [] + + +@pytest.mark.parametrize("name", ["other-xfm", "other_xfm", "Other.2", "x"]) +def test_usable_transform_names(name): + assert aligner.check_xfm_name(name) == name + + +@pytest.mark.parametrize("name", ["", " ", "../escape", "a/b", "-leading", ".hidden", "with space", None]) +def test_unusable_transform_names(name): + with pytest.raises(ValueError, match="not a usable transform name"): + aligner.check_xfm_name(name) + + +def test_save_under_a_new_name_leaves_the_loaded_transform_alone(stale_masks, recorder): + """The name field saves the alignment as another transform, so the masks + of the one that was loaded are not touched.""" + mask_dir, names = stale_masks + srv = aligner.show(subj, xfmname, open_browser=False, display_url=False) + xfm = np.arange(16, dtype=float).reshape(4, 4) + try: + resp = _post_save(srv, xfm=json.dumps(xfm.tolist()), name="aligner-copy") + finally: + srv.stop() + + assert resp["status"] == "ok" + assert resp["name"] == "aligner-copy" + assert resp["masks_deleted"] == [] + assert len(recorder.calls) == 1 + call = recorder.calls[0] + assert call["name"] == "aligner-copy" + assert call["xfmtype"] == "coord" + assert call["reference"] is not None, "a new transform needs its reference copied" + assert np.allclose(call["xfm"], xfm) + # the masks belong to the transform that was loaded, which was not written + assert [p.name for p in sorted(mask_dir.glob("mask_*"))] == ["mask_thick.nii.gz", "mask_thin.nii.gz"] + + +def test_save_refuses_a_name_that_is_not_one(stale_masks, recorder): + """A name that would escape the transform directory never reaches the + filesystem, and nothing is saved or deleted.""" + mask_dir, names = stale_masks + srv = aligner.show(subj, xfmname, open_browser=False, display_url=False) + try: + resp = _post_save(srv, xfm=json.dumps(np.eye(4).tolist()), name="../elsewhere") + finally: + srv.stop() + assert resp["status"] == "error" + assert "not a usable transform name" in resp["message"] + assert recorder.calls == [] + assert len(sorted(mask_dir.glob("mask_*"))) == 2 + + +def test_a_refused_save_keeps_the_masks(stale_masks): + """A save that does not go through leaves the masks alone.""" + mask_dir, names = stale_masks + srv = aligner.show(subj, xfmname, open_browser=False, display_url=False) + try: + resp = _post_save(srv, xfm=json.dumps([1, 2, 3])) + finally: + srv.stop() + assert resp["status"] == "error" + assert [p.name for p in sorted(mask_dir.glob("mask_*"))] == ["mask_thick.nii.gz", "mask_thin.nii.gz"] + + +def test_view_only_keeps_the_masks(stale_masks): + mask_dir, names = stale_masks + srv = aligner.show(subj, xfmname, view_only=True, open_browser=False, display_url=False) + try: + resp = _post_save(srv, xfm=json.dumps(np.eye(4).tolist())) + finally: + srv.stop() + assert resp["status"] == "error" + assert [p.name for p in sorted(mask_dir.glob("mask_*"))] == ["mask_thick.nii.gz", "mask_thin.nii.gz"] + + +# --------------------------------------------------------------------------- +# Server endpoints (no browser) +# --------------------------------------------------------------------------- + + +def test_page_carries_config(server): + from PIL import Image + + html = _open(_url(server, "aligner.html")).decode() + assert "aligner.Aligner" in html + config = _page_config(html) + + xfm = database.db.get_xfm(subj, xfmname) + nii = xfm.reference_nifti + assert config["subject"] == subj + assert config["xfmname"] == xfmname + assert config["view_only"] is False + assert config["masks"] == [] + assert np.allclose(config["xfm"], xfm.xfm) + assert np.allclose(config["world"], aligner.reference_frame(nii)) + assert config["volume"]["shape"] == list(nii.shape[::-1]) + assert config["cmap"] == "gray" + assert config["mesh_color"] == "#ffffff" + assert config["vmin"] < config["vmax"] + + # the reference is served as the float mosaic the viewer expects + png = _open(_url(server, "data/reference.png")) + image = Image.open(io.BytesIO(png)) + nwide, ntall = config["volume"]["mosaic"] + assert image.size == (nwide * (nii.shape[0] + 1) + 1, ntall * (nii.shape[1] + 1) + 1) + + # the surfaces come from the viewer's CTM pack + ctm = json.loads(_open(_url(server, "ctm/%s/" % subj)).decode()) + assert len(ctm["offsets"]) == 2 + assert len(_open(_url(server, "ctm/%s/%s" % (subj, ctm["data"])))) > 0 + + +def test_save_endpoint_stores_coord_transform(server, recorder): + xfm = np.arange(16, dtype=float).reshape(4, 4) + resp = _post_save(server, xfm=json.dumps(xfm.tolist())) + assert resp["status"] == "ok" + assert len(recorder.calls) == 1 + call = recorder.calls[0] + assert call["subject"] == subj + assert call["name"] == xfmname + assert call["xfmtype"] == "coord" + assert np.allclose(call["xfm"], xfm) + + resp = _post_save(server, xfm=json.dumps([1, 2, 3])) + assert resp["status"] == "error" + assert len(recorder.calls) == 1 + + +@pytest.mark.parametrize("save_token", [None, "", "0" * 32]) +def test_save_needs_the_token_of_the_page(server, recorder, save_token): + """The save endpoint writes to the filestore, so it only answers the page + it served: a post from a site the browser is also on carries the session + cookie but no token of the page. + """ + data = dict(xfm=json.dumps(np.eye(4).tolist())) + if save_token is not None: + data["save_token"] = save_token + with pytest.raises(urllib.error.HTTPError) as caught: + _open(_url(server, "save"), data) + assert caught.value.code == 403 + assert "not the aligner's own page" in json.loads(caught.value.read().decode())["message"] + assert recorder.calls == [] + + # and the page's own token is taken + assert _post_save(server, xfm=json.dumps(np.eye(4).tolist()))["status"] == "ok" + assert len(recorder.calls) == 1 + + +@pytest.mark.parametrize("page", ["aligner.html", "", "ctm/%s/" % subj, "data/reference.png"]) +def test_every_page_needs_the_session_token(server, page): + """Nothing is served to a request that does not carry the token from the + address the aligner was opened at.""" + bare = "http://localhost:%d/%s" % (server.port, page) + with pytest.raises(urllib.error.HTTPError) as caught: + _open(bare) + assert caught.value.code == 403 + + with pytest.raises(urllib.error.HTTPError) as caught: + _open(bare + "?token=" + "0" * 32) + assert caught.value.code == 403 + + assert len(_open(_url(server, page))) > 0 + + +def test_the_token_comes_back_as_a_cookie(server): + """The page hands the token over once and is given a cookie for it, so + that the addresses it asks for afterwards do not have to carry it.""" + import http.cookiejar + + from cortex.webgl import serve + + jar = http.cookiejar.CookieJar() + browser = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(jar)) + with browser.open(_url(server, "aligner.html"), timeout=30) as resp: + assert "aligner.Aligner" in resp.read().decode() + cookie = {c.name: c.value for c in jar} + assert cookie == {serve.token_cookie(server.port): server.token} + + # the cookie is enough from here on + with browser.open("http://localhost:%d/ctm/%s/" % (server.port, subj), timeout=30) as resp: + assert len(resp.read()) > 0 + + +def test_a_server_without_a_token_answers_anything(recorder): + """The token can be turned off, for a script that talks to the server + itself rather than through the page.""" + srv = aligner.show(subj, xfmname, open_browser=False, display_url=False, token="") + try: + assert srv.token == "" + assert srv.url("aligner.html") == "http://%s:%d/aligner.html" % (srv.host, srv.port) + assert "aligner.Aligner" in _open( + "http://localhost:%d/aligner.html" % srv.port).decode() + finally: + srv.stop() + + +def test_server_answers_for_this_computer_only(server): + """The page opens as localhost, as 127.0.0.1 and under the name of the + machine, and the server listens for nothing else.""" + import socket + + from cortex.webgl import serve + + names = [serve.LOOPBACK, "127.0.0.1", socket.gethostname()] + resolved = set() + for name in names: + try: + resolved.update(info[4][0] for info in + socket.getaddrinfo(name, None, type=socket.SOCK_STREAM)) + except socket.gaierror: + continue + + bound = [sock.getsockname()[0] for sock in server._sockets] + assert len(bound) > 0 + assert set(bound) <= resolved, "listening on %s" % sorted(set(bound) - resolved) + assert len({sock.getsockname()[1] for sock in server._sockets}) == 1, "one port for all of them" + + for name in names: + try: + socket.getaddrinfo(name, None, type=socket.SOCK_STREAM) + except socket.gaierror: + continue # a machine whose own name does not resolve + html = _open(_url(server, "aligner.html", host=name)).decode() + assert "aligner.Aligner" in html, "the page did not open as %s" % name + + +def test_the_link_names_the_machine(server): + """The link names this computer, which is the name a port forward from + another machine is set up under, and falls back to the loopback name when + the server is not on an address that name leads to.""" + import socket + + from cortex.webgl import serve + + assert server.host == socket.gethostname() + assert "aligner.Aligner" in _open(_url(server, "aligner.html", host=server.host)).decode() + + try: + elsewhere = serve.WebApp([], 0, address="127.0.0.2") + except OSError: + return # the spare loopback address is not routed here + try: + assert elsewhere.host == serve.LOOPBACK + finally: + elsewhere.stop() + + +def test_every_name_of_the_computer_gets_a_socket_on_one_port(): + """Each name is listened for, one that another name already covers is + bound once, and one that does not resolve is passed over.""" + import socket + + from cortex.webgl import serve + + # 127.0.0.2 stands in for a machine whose hostname is not its loopback + # address, which is where two of these names would be the same one + spare = "127.0.0.2" + probe = socket.socket() + try: + probe.bind((spare, 0)) + usable = True + except OSError: + usable = False # the spare loopback address is not routed here + finally: + probe.close() + + sockets = serve.bind_local_sockets( + 0, ["localhost", "127.0.0.1", spare, "no-such-host.invalid"]) + try: + bound = [sock.getsockname()[0] for sock in sockets] + assert bound.count("127.0.0.1") == 1, "the same address was bound twice" + assert len({sock.getsockname()[1] for sock in sockets}) == 1, "one port for all of them" + if usable: + assert spare in bound + finally: + for sock in sockets: + sock.close() + + with pytest.raises(OSError, match="no address resolved"): + serve.bind_local_sockets(0, ["no-such-host.invalid"]) + + +@pytest.mark.parametrize("server", [dict(view_only=True)], indirect=True) +def test_view_only_never_saves(server, recorder): + resp = _post_save(server, xfm=json.dumps(np.eye(4).tolist())) + assert resp["status"] == "error" + assert "view only" in resp["message"] + assert recorder.calls == [] + assert '"view_only": true' in _open(_url(server)).decode() + + +# --------------------------------------------------------------------------- +# Headless browser +# --------------------------------------------------------------------------- + + +def _quadrants(png): + from PIL import Image + + rgb = np.asarray(Image.open(io.BytesIO(png)).convert("RGB")).astype(np.uint32) + h, w = rgb.shape[:2] + packed = (rgb[..., 0] << 16) | (rgb[..., 1] << 8) | rgb[..., 2] + return [packed[:h // 2, :w // 2], packed[h // 2:, :w // 2], + packed[:h // 2, w // 2:], packed[h // 2:, w // 2:]] + + +def _translation(vector): + mat = np.eye(4) + mat[:3, 3] = vector + return mat + + +@pytest.mark.skipif(not has_playwright, reason="playwright and chromium are required") +@pytest.mark.timeout(400) +def test_aligner_in_headless_browser(recorder): + """The aligner loads, edits the transform in world millimeters and saves it.""" + from cortex.export.headless import _PlaywrightThread, _wait_for_viewer_loaded, filter_webgl_failures + + server = aligner.show(subj, xfmname, open_browser=False, display_url=False) + server.disconnect_on_close = False + pw = _PlaywrightThread() + handle = None + try: + pw.start(_url(server, "aligner.html"), timeout=120) + handle = server.get_client() + # the handle skips replies to earlier requests by draining the + # server's queue, which it can only do with the server in hand + assert vars(handle).get("server") is server + _wait_for_viewer_loaded(handle, timeout=240) + # software rendering is slow: the first frame follows the load + assert handle.wait_for_frame(timeout=120) >= 1 + + nii = _reference() + world = aligner.reference_frame(nii) + coord0 = np.asarray(database.db.get_xfm(subj, xfmname).xfm) + assert np.allclose(handle.get_xfm(), coord0, atol=1e-3) + + # a translation in world millimeters is applied ahead of the transform + handle.translate([2.0, -3.0, 1.5]) + coord1 = handle.get_xfm() + expected = np.linalg.inv(world) @ _translation([2.0, -3.0, 1.5]) @ world @ coord0 + assert np.allclose(coord1, expected, atol=1e-3) + + # a rotation about the cursor keeps the cursor fixed + cursor_voxel = np.asarray(handle._call("getCursor"), dtype=float) + cursor_world = (world @ np.append(cursor_voxel, 1))[:3] + handle.rotate([0, 0, 1], 10) + coord2 = handle.get_xfm() + anat = np.linalg.inv(world @ coord2) @ np.append(cursor_world, 1) + assert np.allclose((world @ coord1 @ anat)[:3], cursor_world, atol=1e-2) + assert not np.allclose(coord2, coord1, atol=1e-4) + + handle.undo() + assert np.allclose(handle.get_xfm(), coord1, atol=1e-3) + + # every view drew something, and the fourth panel differs between the + # two four-panel displays + # (snapshot waits for the frame that shows the last change) + outline = handle.snapshot() + for quadrant in _quadrants(outline): + assert len(np.unique(quadrant)) > 10 + handle.set_control("display", aligner.DISPLAYS["brain"]) + projected = handle.snapshot() + for quadrant in _quadrants(projected): + assert len(np.unique(quadrant)) > 10 + assert outline != projected + + # a colormap change and a new mesh color reach the shaders + handle.set_control("display", aligner.DISPLAYS["slices"]) + handle.set_control("image.colormap", "hot") + handle.set_control("mesh.color", "#ff0000") + recolored = handle.snapshot() + assert recolored != outline + assert handle.get_control("mesh.color") == "#ff0000" + assert handle.get_control("image.colormap") == "hot" + assert handle.get_control("display") == aligner.DISPLAYS["slices"] + + # the history holds one entry per edit, and going back to one puts + # that alignment back + history = handle._call("getHistory") + assert [entry["kind"] for entry in history] == ["loaded", "translate", "rotate"] + # each edit is named in the terms it was made in + assert history[1]["label"] == "2.00 mm right, 3.00 mm posterior, 1.50 mm superior" + assert history[2]["label"] == "10.00° CW in axial" + handle._call("selectHistory", 0) + assert np.allclose(handle.get_xfm(), coord0, atol=1e-3) + handle._call("selectHistory", 1) + assert np.allclose(handle.get_xfm(), coord1, atol=1e-3) + # and an edit made from there drops the entries that followed + handle.translate([1.0, 0.0, 0.0]) + assert [entry["kind"] for entry in handle._call("getHistory")] == [ + "loaded", "translate", "translate"] + handle.undo() + assert np.allclose(handle.get_xfm(), coord1, atol=1e-3) + + # saving stores the current transform as a coord transform, and says + # so only once the transform has been written + message = handle.save() + assert len(recorder.calls) == 1, "save() came back before the save landed" + assert xfmname in message + call = recorder.calls[0] + assert call["subject"] == subj and call["name"] == xfmname + assert call["xfmtype"] == "coord" + assert np.allclose(call["xfm"], coord1, atol=1e-3) + assert not handle._call("isDirty"), "the saved alignment still counts as edited" + + errors = pw.browser_errors + assert not [e for e in errors if "[pageerror]" in e], errors + assert not filter_webgl_failures(errors), errors + finally: + pw.shutdown() + server.stop() + + +@pytest.mark.skipif(not has_playwright, reason="playwright and chromium are required") +@pytest.mark.timeout(400) +def test_displays_show_the_surface_and_follow_the_transform(): + """`display` puts the surface in the corner or over the whole window, + what it draws there follows the transform as it is edited, and the + surface of the data view inflates and flattens.""" + from playwright.sync_api import sync_playwright + + server = aligner.show(subj, xfmname, open_browser=False, display_url=False) + server.disconnect_on_close = False + try: + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True, args=[ + "--enable-webgl", "--use-gl=swiftshader", "--no-sandbox", "--disable-dev-shm-usage"]) + page = browser.new_page(viewport={"width": 1000, "height": 700}) + page.goto(_url(server, "aligner.html"), wait_until="load", timeout=120000) + page.wait_for_function("window.viewer && window.viewer.loaded.state() == 'resolved'", timeout=240000) + page.wait_for_function("window.viewer.nframes > 0", timeout=120000) + + def state(): + return page.evaluate("""() => { + var v = window.viewer; + return { + display: v.setDisplay(), + rects: v.viewlist.map(w => [w.name, w.rect.width, w.rect.height]), + modes: v.viewlist.map(w => [w.name, v._viewMode(w) == 0 ? "outline" : "surface"]), + canvas: [v.width, v.height], + shown: Array.from(document.querySelectorAll('.aligner-view')) + .filter(e => getComputedStyle(e).display != 'none').map(e => e.id), + }; + }""") + + def redraw(script): + """Run `script` and wait for the frame that shows its effect. + + The page draws on demand, so the frame count has to be read + before the change: read afterwards, it might already have + been drawn and the wait would never end. + """ + frames = page.evaluate("window.viewer.nframes") + page.evaluate(script) + page.wait_for_function("window.viewer.nframes > %d" % frames, timeout=120000) + + # the three displays the one control offers, and no other + options = page.evaluate( + "Array.from(window.viewer.ui._controls.display.__select.options).map(o => o.value)") + assert options == [aligner.DISPLAYS[k] for k in ("slices", "brain", "surface")] + + start = state() + assert start["display"] == aligner.DISPLAYS["slices"] + assert len(start["shown"]) == 4 + assert all(w > 0 for _, w, _ in start["rects"]) + assert all(m == "outline" for _, m in start["modes"]), "every view outlines the mesh" + + # the surface in the corner, with the slices still showing slices + redraw("window.viewer.ui.set('display', %r)" % aligner.DISPLAYS["brain"]) + split = state() + assert split["display"] == aligner.DISPLAYS["brain"] + assert len(split["shown"]) == 4, "the slice views stay" + assert dict(split["modes"])["3d"] == "surface", "the corner paints the data" + assert all(m == "outline" for name, m in split["modes"] if name != "3d") + + # what the corner draws follows an edit to the transform, unsaved + before = page.locator("#view-3d").screenshot() + redraw("window.viewer.translate([0, 0, 6])") + page.wait_for_timeout(300) + assert page.locator("#view-3d").screenshot() != before, ( + "the surface did not redraw when the transform moved") + assert page.evaluate("window.viewer.isDirty()") + redraw("window.viewer.undo()") + + # the corner unfolds too, by the keys the viewer uses and by the + # control in the panel, which are the same ones the data view has + def press(key): + frames = page.evaluate("window.viewer.nframes") + page.keyboard.press(key) + page.wait_for_function("window.viewer.nframes > %d" % frames, timeout=120000) + + folded = page.locator("#view-3d").screenshot() + press("i") + assert page.evaluate("window.viewer.setMix()") == 0.5, "i did not inflate the surface" + inflated = page.locator("#view-3d").screenshot() + assert inflated != folded, "the corner did not inflate" + press("f") + assert page.evaluate("window.viewer._flatness()") == 1, "f did not flatten the surface" + assert page.locator("#view-3d").screenshot() != inflated, "the corner did not flatten" + press("r") + assert page.evaluate("window.viewer.setMix()") == 0, "r did not fold the surface back" + + # the depth of the surface between pial and white matter is the + # panel's own control, and moves the one in the corner + redraw("window.viewer.ui.set('mesh.depth', 1)") + page.wait_for_timeout(300) + assert page.locator("#view-3d").screenshot() != folded, ( + "the corner did not follow the depth control") + redraw("window.viewer.ui.set('mesh.depth', 0.5)") + + # the unfolding carries over to the data view, which draws the + # same surface + redraw("window.viewer.ui.set('mesh.unfold', 0.5)") + redraw("window.viewer.ui.set('display', %r)" % aligner.DISPLAYS["surface"]) + assert page.evaluate("window.viewer.setMix()") == 0.5, ( + "the data view did not open on the unfolding the corner was left at") + assert page.evaluate("window.viewer.ui.get('mesh.unfold')") == 0.5 + redraw("window.viewer.ui.set('mesh.unfold', 0)") + + # and the surface on its own, over the whole window + redraw("window.viewer.ui.set('display', %r)" % aligner.DISPLAYS["surface"]) + single = state() + assert single["display"] == aligner.DISPLAYS["surface"] + assert single["shown"] == ["view-3d"], "only the surface is left" + assert dict((n, (w, h)) for n, w, h in single["rects"])["3d"] == tuple(single["canvas"]) + assert all(w == 0 for n, w, _ in single["rects"] if n != "3d") + assert all(m == "surface" for _, m in single["modes"]), "the data view paints the data" + + def filled(): + """Fraction of the window the surface covers.""" + return page.evaluate("""() => { + var c = document.querySelector('#aligner-canvas'); + var s = document.createElement('canvas'); + s.width = c.width; s.height = c.height; + s.getContext('2d').drawImage(c, 0, 0); + var d = s.getContext('2d').getImageData(0, 0, s.width, s.height).data; + var lit = 0; + for (var i = 0; i < d.length; i += 4) { if (d[i] + d[i+1] + d[i+2] > 45) lit++; } + return lit / (s.width * s.height); + }""") + + # the surface is framed for the window rather than left as it was + # drawn in a quarter of it + assert filled() > 0.2, "the surface covers only %.2f of the window" % filled() + + # it inflates and flattens, as the viewer's does: the flatmap + # takes the whole window and drops the medial wall + folded = page.locator("#view-3d").screenshot() + redraw("window.viewer.ui.set('mesh.unfold', 0.5)") + inflated = page.locator("#view-3d").screenshot() + assert inflated != folded, "the surface did not inflate" + assert page.evaluate("window.viewer._flatness()") == 0 + assert not page.evaluate("window.viewer._culled"), "the mesh is whole until it flattens" + + redraw("window.viewer.ui.set('mesh.unfold', 1)") + assert page.evaluate("window.viewer._flatness()") == 1 + assert page.evaluate("window.viewer._culled"), "the medial wall is still drawn" + assert page.evaluate("window.viewer.setPivot()") == 180 + assert page.locator("#view-3d").screenshot() != inflated, "the surface did not flatten" + assert filled() > 0.1, "the flatmap covers only %.2f of the window" % filled() + + # the surfaces of the other displays get the whole mesh back + redraw("window.viewer.ui.set('display', %r)" % aligner.DISPLAYS["slices"]) + assert not page.evaluate("window.viewer._culled") + + redraw("window.viewer.toggleDisplay()") + assert state()["display"] == aligner.DISPLAYS["brain"], "the toggle steps on" + browser.close() + finally: + server.stop() + + +@pytest.mark.skipif(not has_playwright, reason="playwright and chromium are required") +@pytest.mark.timeout(400) +def test_history_panel_lists_the_edits_and_goes_back_to_one(): + """The panel keeps a row per edit since the page opened, newest first, + each named in the terms the edit was made in, and clicking one puts that + alignment back. Closing the page with an edit on it asks the browser to + confirm, and a saved one does not. + """ + from playwright.sync_api import sync_playwright + + server = aligner.show(subj, xfmname, open_browser=False, display_url=False) + server.disconnect_on_close = False + try: + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True, args=[ + "--enable-webgl", "--use-gl=swiftshader", "--no-sandbox", "--disable-dev-shm-usage"]) + page = browser.new_page(viewport={"width": 1000, "height": 700}) + page.goto(_url(server, "aligner.html"), wait_until="load", timeout=120000) + page.wait_for_function("window.viewer && window.viewer.loaded.state() == 'resolved'", timeout=240000) + page.wait_for_function("window.viewer.nframes > 0", timeout=120000) + + def rows(): + return page.evaluate("""() => Array.from( + document.querySelectorAll('#aligner-history-list li')).map(li => [ + li.firstChild.textContent, li.lastChild.textContent, + li.className == 'current'])""") + + def xfm(): + return np.asarray(page.evaluate("window.viewer.getXfm()"), dtype=float) + + # the page opens on the alignment it was loaded with, and nothing else + assert rows() == [["loaded", "", True]] + loaded = xfm() + + page.evaluate("window.viewer.translate([4, 0, 0])") + page.evaluate("window.viewer.rotate([0, 0, 1], 5)") + listed = rows() + assert [row[0] for row in listed] == ["rotate", "translate", "loaded"], ( + "the newest edit is not at the top") + # each edit is named the way it was made: along the anatomical + # axes, or in the plane of the view it turned in + assert listed[1][1] == "4.00 mm right" + assert listed[0][1] == "5.00° CW in axial" + assert [row[2] for row in listed] == [True, False, False], "the newest edit is marked" + moved = xfm() + + # clicking a row puts that alignment back + page.click("#aligner-history-list li:last-child") + assert np.allclose(xfm(), loaded, atol=1e-3) + assert [row[2] for row in rows()] == [False, False, True] + page.click("#aligner-history-list li:first-child") + assert np.allclose(xfm(), moved, atol=1e-3) + + # a drag over a view is one entry, however many frames it takes + page.click("#aligner-history-list li:last-child") + box = page.locator("#view-y").bounding_box() + page.mouse.move(box["x"] + 100, box["y"] + 100) + page.mouse.down(button="right") + for step in range(1, 5): + page.mouse.move(box["x"] + 100 + 10 * step, box["y"] + 100) + page.mouse.up(button="right") + dragged = rows() + assert [row[0] for row in dragged] == ["translate", "loaded"], ( + "a drag left more than one entry, or dropped the ones it replaced") + # the coronal view is seen from the front, so dragging right on + # its screen moves the surfaces to the subject's left + assert dragged[0][1].endswith("mm left"), dragged[0][1] + + # a right click that moved nothing leaves no entry behind + page.mouse.down(button="right") + page.mouse.up(button="right") + assert len(rows()) == 2, "a click that moved nothing left an entry" + + # a history longer than its list scrolls inside it, leaving the + # panel where it stands + panel = "document.querySelector('#aligner-panel').scrollTop" + where = page.evaluate(panel) + for step in range(14): + page.evaluate("window.viewer.translate([1, 0, 0])") + assert page.evaluate(panel) == where, "the panel scrolled away from the controls" + assert page.evaluate("""() => { + var list = document.querySelector('#aligner-history-list'); + var row = list.querySelector('li.current'); + var r = row.getBoundingClientRect(), l = list.getBoundingClientRect(); + return r.top >= l.top - 1 && r.bottom <= l.bottom + 1; + }"""), "the entry being edited scrolled out of sight" + + # the browser is asked to confirm a close that would lose the edit + def asks(): + return page.evaluate("""() => { + var event = new Event('beforeunload', {cancelable: true}); + window.dispatchEvent(event); + return event.defaultPrevented; + }""") + + assert page.evaluate("window.viewer.isDirty()") + assert asks(), "closing an edited page asked nothing" + page.evaluate("window.viewer.selectHistory(0)") + assert not page.evaluate("window.viewer.isDirty()") + assert not asks(), "closing a page back at the saved alignment asked" + browser.close() + finally: + server.stop() + + +@pytest.mark.skipif(not has_playwright, reason="playwright and chromium are required") +@pytest.mark.timeout(400) +def test_keyboard_moves_the_mesh_and_colormaps_have_previews(): + """WASD moves the mesh like the arrows do, and every colormap in the + dropdown is drawn with a strip of itself. + + Drives the page directly rather than through the websocket handle, + because both are about what the browser does with real events. + """ + from playwright.sync_api import sync_playwright + + server = aligner.show(subj, xfmname, open_browser=False, display_url=False) + server.disconnect_on_close = False + try: + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True, args=[ + "--enable-webgl", "--use-gl=swiftshader", "--no-sandbox", "--disable-dev-shm-usage"]) + page = browser.new_page(viewport={"width": 1000, "height": 700}) + page.goto(_url(server, "aligner.html"), wait_until="load", timeout=120000) + page.wait_for_function("window.viewer && window.viewer.loaded.state() == 'resolved'", timeout=240000) + page.wait_for_function("window.viewer.nframes > 0", timeout=120000) + + def xfm(): + return np.asarray(page.evaluate("window.viewer.getXfm()"), dtype=float) + + def press(key, shift=False): + """Press `key` over the coronal view and return the transform + it produced, leaving the mesh where it was.""" + page.hover("#view-y") + page.keyboard.press(("Shift+" if shift else "") + key) + moved = xfm() + page.evaluate("window.viewer.undo()") + return moved + + page.evaluate("window.viewer.setTranslateStep(2)") + start = xfm() + for key, arrow in [("w", "ArrowUp"), ("a", "ArrowLeft"), + ("s", "ArrowDown"), ("d", "ArrowRight")]: + by_letter, by_arrow = press(key), press(arrow) + assert not np.allclose(by_letter, start), "%s did not move the mesh" % key + assert np.allclose(by_letter, by_arrow), "%s does not match %s" % (key, arrow) + + # shift is the fine step, and shift+w reaches the handler as "W" + full = press("w")[:3, 3] - start[:3, 3] + fine = press("w", shift=True)[:3, 3] - start[:3, 3] + assert np.linalg.norm(full) == pytest.approx(10 * np.linalg.norm(fine), rel=0.01) + + # a key pressed while typing in a control must not move the mesh + page.evaluate("() => document.querySelector('#figure_ui input').focus()") + page.keyboard.press("w") + assert np.allclose(xfm(), start), "typing in a control moved the mesh" + page.evaluate("() => document.activeElement.blur()") + + # every colormap in the dropdown carries a strip of itself + before = page.evaluate( + "document.querySelector('#figure_ui').getBoundingClientRect().left") + page.click(".select2-selection") + page.wait_for_selector(".select2-results__option .aligner-cmap img", timeout=30000) + + # the open list stays within the window: one that reaches past the + # right edge scrolls the page sideways, taking the views with it + assert page.evaluate( + "document.querySelector('#figure_ui').getBoundingClientRect().left" + ) == before, "opening the colormap list moved the page" + assert page.evaluate("document.documentElement.scrollLeft") == 0 + box = page.evaluate("""() => { + var r = document.querySelector('.select2-dropdown').getBoundingClientRect(); + return [r.left, r.right]; + }""") + assert box[0] >= 0 and box[1] <= page.evaluate("window.innerWidth") + + previews = page.evaluate("""() => { + var rows = document.querySelectorAll('.select2-results__option'); + var imgs = document.querySelectorAll('.select2-results__option .aligner-cmap img'); + var drawn = 0; + imgs.forEach(function(i) { if (i.complete && i.naturalWidth > 0) drawn++; }); + return {rows: rows.length, imgs: imgs.length, drawn: drawn, + options: window.viewer._cmapSelect.find('option').length}; + }""") + assert previews["options"] > 100, "the dropdown lists few colormaps" + assert previews["imgs"] == previews["rows"], "a colormap is shown without its strip" + assert previews["drawn"] == previews["imgs"], "a colormap strip did not load" + + # picking one from the list reaches the shader and the menu + page.fill(".select2-search__field", "hot") + page.click(".select2-results__option--highlighted") + page.wait_for_timeout(500) + picked = page.evaluate("""() => ({ + name: window.viewer.setColormap(), + menu: window.viewer.ui.get('image.colormap'), + shader: window.viewer.volUniforms.colormap.value === window.viewer.colormaps[window.viewer.setColormap()], + })""") + assert picked["name"] != "gray" and picked["name"] == picked["menu"] + assert picked["shader"], "the picked colormap did not reach the shader" + + # and a colormap set from the menu shows up in the closed control + page.evaluate("window.viewer.ui.set('image.colormap', 'viridis')") + page.wait_for_timeout(500) + assert page.evaluate( + "() => document.querySelector('.select2-selection__rendered .aligner-cmap-name').textContent" + ) == "viridis" + browser.close() + finally: + server.stop() + + +@pytest.mark.skipif(not has_playwright, reason="playwright and chromium are required") +@pytest.mark.timeout(400) +def test_the_hue_bar_of_the_color_picker_is_within_reach(): + """The mesh color picker shows only while the pointer is over it, so its + hue bar has to sit beside the saturation square and inside the picker. + Laid out below it, the pointer leaves the picker on the way and the + picker closes before the bar can be used.""" + from playwright.sync_api import sync_playwright + + server = aligner.show(subj, xfmname, open_browser=False, display_url=False) + server.disconnect_on_close = False + try: + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True, args=[ + "--enable-webgl", "--use-gl=swiftshader", "--no-sandbox", "--disable-dev-shm-usage"]) + page = browser.new_page(viewport={"width": 1000, "height": 700}) + page.goto(_url(server, "aligner.html"), wait_until="load", timeout=120000) + page.wait_for_function("window.viewer && window.viewer.loaded.state() == 'resolved'", timeout=240000) + page.wait_for_function("window.viewer.nframes > 0", timeout=120000) + + page.hover("#figure_ui .cr.color .c") + boxes = page.evaluate("""() => { + var picker = document.querySelector('#figure_ui .selector'); + var box = function(el) { + var r = el.getBoundingClientRect(); + return {left: r.left, right: r.right, top: r.top, bottom: r.bottom}; + }; + return {picker: box(picker), + square: box(picker.querySelector('.saturation-field')), + hue: box(picker.querySelector('.hue-field'))}; + }""") + picker, square, hue = boxes["picker"], boxes["square"], boxes["hue"] + assert hue["left"] >= square["right"] - 1, "the hue bar is not beside the square" + assert hue["top"] < square["bottom"], "the hue bar is below the square" + assert hue["right"] <= picker["right"] + 1 and hue["bottom"] <= picker["bottom"] + 1, ( + "the hue bar hangs out of the picker") + + # and the picker is still there once the pointer is on the bar + page.mouse.move((hue["left"] + hue["right"]) / 2, (hue["top"] + hue["bottom"]) / 2) + page.wait_for_timeout(200) + assert page.evaluate( + "() => getComputedStyle(document.querySelector('#figure_ui .selector')).display" + ) != "none", "the picker closed on the way to the hue bar" + browser.close() + finally: + server.stop() diff --git a/cortex/tests/test_webgl_data.py b/cortex/tests/test_webgl_data.py index 46977c256..d7123cc5d 100644 --- a/cortex/tests/test_webgl_data.py +++ b/cortex/tests/test_webgl_data.py @@ -1,5 +1,7 @@ """Tests for the WebGL serialization layer in cortex.webgl.data.""" +import json + import numpy as np import pytest @@ -228,3 +230,43 @@ def test_package_rejects_same_bytes_different_metadata() -> None: b.subject = "S2" # never reaches the database: the check comes first with pytest.raises(ValueError, match="identical bytes"): Package(cortex.Dataset(a=a, b=b)) + + +@pytest.mark.parametrize("name", ["volume", "volume2D", "vertex"]) +def test_a_view_with_an_array_attr_still_packages(name): + """A view carrying an array in its attrs has to reach the browser. + + ``alpha`` is the array that gets there: a view takes one to override the + alpha of its colormap, and it ends up among the attrs, which the page is + handed as JSON. json has no way to write an array, so the page the viewer + serves came back as a 500 and nothing was drawn. + """ + rng = np.random.default_rng(0) + alpha = (rng.uniform(0, 1, volshape) * 255).astype(np.uint8) + attr = "alpha" + if name == "volume": + view = cortex.Volume(rng.normal(size=volshape), subj, xfmname, alpha=alpha) + elif name == "volume2D": + # a 2D view draws the alpha it is given into the colors of its raw + # view, so the array left among its attrs is any other one + attr = "mask" + view = cortex.Volume2D(rng.normal(size=volshape), rng.normal(size=volshape), + subj, xfmname, mask=alpha) + else: + view = cortex.Vertex(rng.normal(size=nverts), subj, + alpha=rng.uniform(0, 1, nverts)) + + with pytest.warns(UserWarning, match=attr): + metadata = Package(dataset.Dataset(view=view)).metadata() + json.dumps(metadata) + assert attr not in metadata["views"][0]["attrs"] + + +def test_the_attrs_a_page_reads_survive_packaging(): + """Leaving the arrays out keeps everything the page does read, including + the numbers numpy hands back, which json cannot write either.""" + view = cortex.Volume(np.random.randn(*volshape), subj, xfmname, + priority=np.int64(3), rate=2.5, filter="trilinear") + attrs = json.loads(json.dumps(Package(dataset.Dataset(view=view)).metadata())) + attrs = attrs["views"][0]["attrs"] + assert attrs == {"priority": 3, "rate": 2.5, "filter": "trilinear"} diff --git a/cortex/tests/test_webgl_headless.py b/cortex/tests/test_webgl_headless.py index afb8764d1..881c54457 100644 --- a/cortex/tests/test_webgl_headless.py +++ b/cortex/tests/test_webgl_headless.py @@ -734,7 +734,7 @@ def _served_metadata(handle): ``addData`` merges into, so this is how we check that a reload of the viewer would show everything that has been added so far. """ - url = "http://localhost:%d/mixer.html" % handle.server.port + url = handle.server.url("mixer.html", host="localhost") with urllib.request.urlopen(url, timeout=30) as resp: page = resp.read().decode("utf-8") marker = "dataset.fromJSON(" @@ -744,7 +744,7 @@ def _served_metadata(handle): def _fetch(handle, path): """GET ``path`` from the viewer's tornado server, returning the body.""" - url = "http://localhost:%d%s" % (handle.server.port, path) + url = handle.server.url(path, host="localhost") with urllib.request.urlopen(url, timeout=30) as resp: return resp.read() @@ -1380,9 +1380,10 @@ def test_the_server_accepts_no_frames(): """ vol = cortex.Volume(np.random.randn(*volshape), subj, xfmname) with cortex.export.headless_viewer(vol, viewer_params={}) as handle: - url = f"http://localhost:{handle.server.port}/movie" - # Refused either way: nothing routes /movie any more, and the server's - # catch-all file handler (serve.py) only answers GET, hence 405. + # with the session token, so that the refusal is the route's own: + # nothing routes /movie any more, and the server's catch-all file + # handler (serve.py) only answers GET, hence 405. + url = handle.server.url("movie", host="localhost") assert _post(url, name="f", frame=0, png="x") in (404, 405) assert "movie_post" not in _js_attrs(handle, "window.viewopts") @@ -2805,3 +2806,858 @@ def type_size(page): _js_run(handle, "window.viewer._animPanel.render", []) assert flat_radius() == pytest.approx(fitted(700, 700), rel=1e-6) handle._pw_thread.wait_for_download(timeout=120) + +# --------------------------------------------------------------------------- +# Group 14: Three slice views beside the 3D one +# --------------------------------------------------------------------------- + + +@pytest.mark.timeout(400) +def test_ortho_views_split_the_canvas(): + """`ortho_views` draws each slice plane straight down its own axis in a + quarter of the canvas, leaving the 3D view the last quarter, and the keys + that move the planes keep working. + + Drives the page directly rather than through the websocket handle, + because what is being tested is what reaches the canvas. + """ + from playwright.sync_api import sync_playwright + + vol = cortex.Volume(np.random.randn(*volshape), subj, xfmname) + server = cortex.webgl.show(vol, open_browser=False, display_url=False, autoclose=False) + try: + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True, args=[ + "--enable-webgl", "--use-gl=swiftshader", "--no-sandbox", "--disable-dev-shm-usage"]) + page = browser.new_page(viewport={"width": 1000, "height": 700}) + errors = [] + page.on("pageerror", lambda exc: errors.append(str(exc))) + page.goto(server.url("mixer.html", host="localhost"), wait_until="load", timeout=120000) + page.wait_for_function( + "window.viewer && window.viewer.loaded.state() == 'resolved'", timeout=240000) + page.wait_for_timeout(3000) + + def quadrants(): + """How much of each quarter of the canvas was drawn on.""" + return page.evaluate("""() => { + var c = document.querySelector('#brain'); + var s = document.createElement('canvas'); + s.width = c.width; s.height = c.height; + s.getContext('2d').drawImage(c, 0, 0); + var d = s.getContext('2d').getImageData(0, 0, s.width, s.height).data; + var lit = [0, 0, 0, 0], seen = [0, 0, 0, 0]; + for (var y = 0; y < s.height; y++) { + for (var x = 0; x < s.width; x++) { + var q = (x < s.width / 2 ? 0 : 1) + (y < s.height / 2 ? 0 : 2); + var i = 4 * (y * s.width + x); + seen[q]++; + if (d[i] + d[i+1] + d[i+2] > 45) lit[q]++; + } + } + return lit.map(function(n, q) { return n / seen[q]; }); + }""") + + assert page.evaluate("window.viewer.setSliceViews()") is False + page.keyboard.press("v") + page.wait_for_timeout(2500) + assert page.evaluate("window.viewer.setSliceViews()") is True, ( + "the v key did not split the canvas") + assert page.evaluate("window.viewer.ui.sliceplanes['Show ortho views']") is True, ( + "the control did not follow the key") + + views = page.evaluate( + "window.viewer.views.map(v => [v.left, v.bottom, v.camera !== undefined])") + assert views == [[0, 0.5, True], [0, 0, True], [0.5, 0.5, True], [0.5, 0, False]], ( + "the canvas is not split between three slice views and the 3D one") + + filled = quadrants() + assert all(part > 0.05 for part in filled), ( + "a quarter of the canvas was left empty: %s" % filled) + + # the slice planes show in their own views whatever the checkboxes + # say, since that is what those views are for + assert page.evaluate( + "Object.keys(window.viewer.sliceplanes).every(k => !window.viewer.sliceplanes[k].setVisible())") + + # the keys that step through the slices keep working + before = page.evaluate("[window.viewer.sliceplanes.x.slice, " + "window.viewer.sliceplanes.y.slice, " + "window.viewer.sliceplanes.z.slice]") + top_left = page.locator("#brain").screenshot() + for key in ["q", "a", "z"]: + page.keyboard.press(key) + page.wait_for_timeout(1000) + after = page.evaluate("[window.viewer.sliceplanes.x.slice, " + "window.viewer.sliceplanes.y.slice, " + "window.viewer.sliceplanes.z.slice]") + assert [round(v) for v in after] == [round(v) + 1 for v in before], ( + "the slice keys did not move the planes") + assert page.locator("#brain").screenshot() != top_left, ( + "the slice views did not redraw when the planes moved") + + # a pick in the 3D view takes the slice views to that point and + # marks it in each of them + box = page.locator("#brain").bounding_box() + page.mouse.click(box["x"] + box["width"] * 0.76, box["y"] + box["height"] * 0.76) + page.wait_for_timeout(1500) + assert page.evaluate("window.viewer._cursorAt === true"), "nothing was picked" + picked = page.evaluate("[window.viewer.sliceplanes.x.slice, " + "window.viewer.sliceplanes.y.slice, " + "window.viewer.sliceplanes.z.slice]") + assert picked != after, "the slice views did not go to the picked point" + # the crosshair stands where the slices were taken to, which is + # the point that was picked + voxel = page.evaluate( + "() => { var xfm = window.viewer.active.uniforms.volxfm.value[0];" + " var p = window.viewer._cursor.position.clone().applyMatrix4(xfm);" + " return [p.x, p.y, p.z]; }") + assert [round(v) for v in voxel] == [round(v) for v in picked], ( + "the crosshair is not where the slices are") + assert page.evaluate("window.viewer._cursor.visible") is True, ( + "the crosshair is not drawn in the 3D view") + # and it is the only mark on the point: the picker's own marker, + # which sits on the surface and is in the 3D view alone, stands + # down while the crosshair is in every view + assert page.evaluate( + "window.viewer.surfs[0].surf.picker.markers.left.visible") is False, ( + "the picker's marker is up as well as the crosshair") + + # a click that lands on nothing leaves the crosshair where it is: + # it marks a place, and clicking beside the brain does not unmark it + where = page.evaluate("window.viewer._cursor.position.toArray()") + page.mouse.click(box["x"] + box["width"] * 0.97, box["y"] + box["height"] * 0.97) + page.wait_for_timeout(1000) + assert page.evaluate("window.viewer._cursorAt") is True, ( + "a click on nothing took the crosshair away") + assert page.evaluate("window.viewer._cursor.position.toArray()") == where + + # a click in a slice view puts the crosshair under the pointer and + # takes the other two views to it, leaving its own slice alone + page.mouse.click(box["x"] + box["width"] * 0.2, box["y"] + box["height"] * 0.2) + page.wait_for_timeout(1500) + assert page.evaluate("window.viewer._cursor.position.toArray()") != where, ( + "a click in the coronal view did not move the crosshair") + moved = page.evaluate("[window.viewer.sliceplanes.x.slice, " + "window.viewer.sliceplanes.y.slice, " + "window.viewer.sliceplanes.z.slice]") + voxel = page.evaluate( + "() => { var xfm = window.viewer.active.uniforms.volxfm.value[0];" + " var p = window.viewer._cursor.position.clone().applyMatrix4(xfm);" + " return [p.x, p.y, p.z]; }") + assert [round(v) for v in voxel] == [round(v) for v in moved], ( + "the crosshair is not on the slices the click took the views to") + assert round(moved[1]) == round(picked[1]), ( + "the coronal view moved the very slice the click was made on") + assert [round(v) for v in moved] != [round(v) for v in picked], ( + "the click in the coronal view left the other views where they were") + + # the help lists the key that splits the canvas + page.keyboard.press("h") + page.wait_for_timeout(500) + assert "Show ortho views along with 3D" in page.inner_text("#helpmenu"), ( + "the key is not in the help") + page.keyboard.press("h") + + # and the 3D view comes back on its own + page.evaluate("window.viewer.ui.set('sliceplanes.Show ortho views', false)") + page.wait_for_timeout(1500) + assert page.evaluate("window.viewer.views.length") == 1 + assert page.evaluate("window.viewer.root.visible") is True + # the point stays marked across the change of layout, by the + # picker's marker once the crosshair has no slices to be in + assert page.evaluate("window.viewer._cursorAt") is True + assert page.evaluate("window.viewer._cursor.visible") is False + assert page.evaluate( + "window.viewer.surfs[0].surf.picker.markers.left.visible") is True + + # a point picked with the 3D view on its own leaves the planes + # where they are, and the slice views open on it + was = page.evaluate("window.viewer._cursor.position.toArray()") + page.mouse.click(box["x"] + box["width"] * 0.45, box["y"] + box["height"] * 0.45) + page.wait_for_timeout(1500) + assert page.evaluate("window.viewer._cursor.position.toArray()") != was, ( + "the pick in the 3D view landed on nothing") + alone = page.evaluate("[window.viewer.sliceplanes.x.slice, " + "window.viewer.sliceplanes.y.slice, " + "window.viewer.sliceplanes.z.slice]") + assert [round(v) for v in alone] == [round(v) for v in moved], ( + "the 3D view on its own moved the slices") + page.keyboard.press("v") + page.wait_for_timeout(2000) + voxel = page.evaluate( + "() => { var xfm = window.viewer.active.uniforms.volxfm.value[0];" + " var p = window.viewer._cursor.position.clone().applyMatrix4(xfm);" + " return [p.x, p.y, p.z]; }") + opened = page.evaluate("[window.viewer.sliceplanes.x.slice, " + "window.viewer.sliceplanes.y.slice, " + "window.viewer.sliceplanes.z.slice]") + assert [round(v) for v in opened] == [round(v) for v in voxel], ( + "the slice views opened on slices the crosshair is not on") + assert [round(v) for v in opened] != [round(v) for v in alone], ( + "the planes were already there, so nothing was shown by this") + assert not errors, errors + browser.close() + finally: + server.stop() + + +# --------------------------------------------------------------------------- +# Group 8: The orthographic camera +# --------------------------------------------------------------------------- + + +@pytest.mark.timeout(400) +def test_the_camera_switches_to_orthographic(): + """`orthographic` swaps the camera the controls move for one with no + vanishing point, where the radius is the zoom rather than the distance, + and offers the isometric view it is wanted for. + + The camera carries the lights and stands in the scene, so the swap is + checked by what reaches the canvas as well as by the camera itself. + """ + from playwright.sync_api import sync_playwright + + vol = cortex.Volume(np.random.randn(*volshape), subj, xfmname) + server = cortex.webgl.show(vol, open_browser=False, display_url=False, autoclose=False) + try: + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True, args=[ + "--enable-webgl", "--use-gl=swiftshader", "--no-sandbox", "--disable-dev-shm-usage"]) + page = browser.new_page(viewport={"width": 1000, "height": 700}) + errors = [] + page.on("pageerror", lambda exc: errors.append(str(exc))) + page.goto(server.url("mixer.html", host="localhost"), wait_until="load", timeout=120000) + page.wait_for_function( + "window.viewer && window.viewer.loaded.state() == 'resolved'", timeout=240000) + page.wait_for_timeout(3000) + + def camera(): + return page.evaluate( + "() => { var c = window.viewer.camera;" + " return {type: c.type, top: c.top, right: c.right, aspect: c.aspect}; }") + + def lit(): + """How much of the canvas the brain is drawn on, lit.""" + return page.evaluate("""() => { + var c = document.querySelector('#brain'); + var s = document.createElement('canvas'); + s.width = c.width; s.height = c.height; + s.getContext('2d').drawImage(c, 0, 0); + var d = s.getContext('2d').getImageData(0, 0, s.width, s.height).data; + var n = 0; + for (var i = 0; i < d.length; i += 4) + if (d[i] + d[i+1] + d[i+2] > 150) n++; + return n / (s.width * s.height); + }""") + + def isometric_shown(): + return page.evaluate( + "window.viewer._cam_ui._controls.isometric.__li.style.display") != "none" + + assert page.evaluate("window.viewer.setOrthographic()") is False + assert camera()["type"] == "PerspectiveCamera" + assert not isometric_shown(), ( + "the isometric view is offered without the camera it is for") + perspective = lit() + assert perspective > 0.02, "the brain was not drawn to begin with" + + page.evaluate("window.viewer.ui.set('camera.orthographic', true)") + page.wait_for_timeout(1500) + ortho = camera() + assert ortho["type"] == "OrthographicCamera" + assert isometric_shown() + # the camera sees what the perspective one saw at that radius, so + # the brain is the same size on screen and the radius goes on + # being the zoom + half = 400 * np.tan(np.radians(45) / 2) + assert abs(ortho["top"] - half) < 1 + assert abs(ortho["right"] / ortho["top"] - 1000 / 700) < 0.01 + # the lights hang off the camera, which the renderer only finds + # through the scene: a camera swapped in outside it draws the + # brain in its emissive color alone + assert lit() > perspective / 2, "the brain went dark under the new camera" + + page.evaluate("window.viewer.ui.set('camera.radius', 200)") + page.wait_for_timeout(1500) + assert abs(camera()["top"] - half / 2) < 1, "the radius is not the zoom" + + # the isometric view turns each axis away from the eye by the same + # angle, so the eye lies along a diagonal of the three + page.evaluate("window.viewer.setIsometric()") + page.wait_for_timeout(2500) + eye = page.evaluate( + "window.viewer.camera.position.clone().sub(window.viewer.controls.target)" + ".normalize().toArray()") + assert [round(abs(v), 2) for v in eye] == [round(1 / np.sqrt(3), 2)] * 3, ( + "the isometric view is not down the diagonal: %s" % eye) + + page.evaluate("window.viewer.ui.set('camera.orthographic', false)") + page.wait_for_timeout(1500) + assert camera()["type"] == "PerspectiveCamera" + assert not isometric_shown() + assert lit() > perspective / 2 + assert not errors, errors + browser.close() + finally: + server.stop() + + +@pytest.mark.timeout(300) +def test_the_viewer_page_does_not_scroll(): + """The viewer is laid out from the edges of the window, with the ends of + the colorbar off it on purpose, so a page that scrolls shows a band of + nothing and moves the brain out from under the pointer. Both the page and + the body are pinned against it.""" + from playwright.sync_api import sync_playwright + + vol = cortex.Volume(np.random.randn(*volshape), subj, xfmname) + server = cortex.webgl.show(vol, open_browser=False, display_url=False, autoclose=False) + try: + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True, args=[ + "--enable-webgl", "--use-gl=swiftshader", "--no-sandbox", "--disable-dev-shm-usage"]) + page = browser.new_page(viewport={"width": 1200, "height": 760}) + errors = [] + page.on("pageerror", lambda exc: errors.append(str(exc))) + page.goto(server.url("mixer.html", host="localhost"), wait_until="load", timeout=120000) + page.wait_for_function( + "window.viewer && window.viewer.loaded.state() == 'resolved'", timeout=240000) + page.wait_for_timeout(3000) + + for width, height in [(1200, 760), (1920, 1080), (640, 480), (1600, 400)]: + page.set_viewport_size({"width": width, "height": height}) + page.wait_for_timeout(700) + over = page.evaluate( + "() => { var d = document.documentElement;" + " return [d.scrollWidth - d.clientWidth, d.scrollHeight - d.clientHeight]; }") + assert over == [0, 0], ( + "the page scrolls at %dx%d, by %s" % (width, height, over)) + assert page.evaluate( + "getComputedStyle(document.documentElement).overflowY") == "hidden" + assert not errors, errors + browser.close() + finally: + server.stop() + + +# --------------------------------------------------------------------------- +# Group 9: Drawing the surface coarsely +# --------------------------------------------------------------------------- + + +@pytest.mark.timeout(400) +def test_the_low_poly_slider_draws_fewer_faces(): + """`low poly` draws the surface from faces cut over groups of that many + mm, which the server cuts the first time they are asked for and keeps. + + The vertices of the coarse surface are vertices of the one it was cut + from, so only the triangles change: the surface goes back to what it was + at 0, down to the picture it draws. + """ + import glob + + from playwright.sync_api import sync_playwright + + #cut from nothing, which is what the first viewer on a subject does + for stale in glob.glob(os.path.join(cortex.db.get_cache(subj), "*_lod*.bin")): + os.remove(stale) + + vol = cortex.Volume(np.random.randn(*volshape), subj, xfmname) + server = cortex.webgl.show(vol, open_browser=False, display_url=False, autoclose=False) + try: + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True, args=[ + "--enable-webgl", "--use-gl=swiftshader", "--no-sandbox", "--disable-dev-shm-usage"]) + page = browser.new_page(viewport={"width": 1000, "height": 700}) + errors = [] + page.on("pageerror", lambda exc: errors.append(str(exc))) + page.goto(server.url("mixer.html", host="localhost"), wait_until="load", timeout=120000) + page.wait_for_function( + "window.viewer && window.viewer.loaded.state() == 'resolved'", timeout=240000) + page.wait_for_timeout(3000) + + def faces(): + return page.evaluate( + "() => { window.viewer.draw();" + " return window.viewer.renderer.info.render.faces; }") + + def lit(): + """How much of the canvas the surface is drawn on.""" + return page.evaluate("""() => { + var c = document.querySelector('#brain'); + var s = document.createElement('canvas'); + s.width = c.width; s.height = c.height; + s.getContext('2d').drawImage(c, 0, 0); + var d = s.getContext('2d').getImageData(0, 0, s.width, s.height).data; + var n = 0; + for (var i = 0; i < d.length; i += 4) + if (d[i] + d[i+1] + d[i+2] > 150) n++; + return n / (s.width * s.height); + }""") + + def wait_for_faces(fewer_than, seconds=60): + """The server cuts the faces when they are first asked for, + so a cold cache waits on the cut as well as the fetch.""" + deadline = time.time() + seconds + while time.time() < deadline: + if faces() < fewer_than: + return faces() + page.wait_for_timeout(500) + return faces() + + whole, drawn = faces(), lit() + opened = page.locator("#brain").screenshot() + assert whole > 100000, "the surface was not drawn to begin with" + assert drawn > 0.02 + + page.evaluate("window.viewer.ui.set('surface.%s.low poly', 8)" % subj) + coarse = wait_for_faces(whole / 10) + assert coarse < whole / 10, ( + "8 mm groups left %d of %d faces" % (coarse, whole)) + #the same brain, from fewer triangles: it covers the canvas as it + #did, rather than being left in pieces + assert abs(lit() - drawn) < drawn / 3, "the coarse surface is not the same shape" + + page.evaluate("window.viewer.ui.set('surface.%s.low poly', 32)" % subj) + assert wait_for_faces(coarse) < coarse, ( + "32 mm groups left as many faces as 8 mm ones") + + page.evaluate("window.viewer.ui.set('surface.%s.low poly', 0)" % subj) + page.wait_for_timeout(2500) + assert faces() == whole + assert page.locator("#brain").screenshot() == opened, ( + "the surface did not go back to what it was") + assert not errors, errors + browser.close() + finally: + server.stop() + + +@pytest.mark.timeout(400) +def test_shading_gives_a_face_one_color_and_an_outline_rings_the_surface(): + """`shading` colors a face at one of its corners and lights it from its + own normal, at its corners or at every pixel, which asks for a vertex per + corner of a face and so for a surface coarse enough to hold them. The + light is worked out on the face that is drawn, so a coarse surface drawn + flat is solid patches. `outline` draws the surface again, inside out and + grown, which leaves a line around it. + """ + from playwright.sync_api import sync_playwright + + data = cortex.Dataset( + volume=cortex.Volume(np.random.randn(*volshape), subj, xfmname), + vertex=cortex.Vertex(np.random.randn(nverts), subj)) + server = cortex.webgl.show(data, open_browser=False, display_url=False, autoclose=False) + try: + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True, args=[ + "--enable-webgl", "--use-gl=swiftshader", "--no-sandbox", "--disable-dev-shm-usage"]) + page = browser.new_page(viewport={"width": 800, "height": 560}) + errors = [] + page.on("pageerror", lambda exc: errors.append(str(exc))) + page.goto(server.url("mixer.html", host="localhost"), wait_until="load", timeout=120000) + page.wait_for_function( + "window.viewer && window.viewer.loaded.state() == 'resolved'", timeout=240000) + page.wait_for_timeout(3000) + + def surf(what): + return page.evaluate("window.viewer.surfs[0].surf." + what) + + def faces(): + return page.evaluate( + "() => { window.viewer.draw();" + " return window.viewer.renderer.info.render.faces; }") + + def shading(mode): + page.evaluate("window.viewer.ui.set('surface.%s.shading', %r)" % (subj, mode)) + page.wait_for_timeout(2500) + + def solid(): + """How much of the brain is drawn in patches of one color: a + face lit by its own normal is one shade all over, so the + pixels beside each other within it are the same.""" + return page.evaluate("""() => { + var c = document.querySelector('#brain'); + var s = document.createElement('canvas'); + s.width = c.width; s.height = c.height; + s.getContext('2d').drawImage(c, 0, 0); + var d = s.getContext('2d').getImageData(0, 0, s.width, s.height).data; + var same = 0, pairs = 0; + for (var y = 0; y < s.height; y++) { + for (var x = 0; x + 1 < s.width; x++) { + var i = 4 * (y * s.width + x), j = i + 4; + if (d[i] + d[i+1] + d[i+2] < 45) continue; + if (d[j] + d[j+1] + d[j+2] < 45) continue; + pairs++; + if (d[i] == d[j] && d[i+1] == d[j+1] && d[i+2] == d[j+2]) same++; + } + } + return pairs ? same / pairs : 0; + }""") + + whole = surf("hemis.left.attributes.position.array.length / 3") + + #a surface of its own detail is too many faces to give each one a + #vertex per corner, so it keeps the color it interpolates + shading("flat") + assert surf("_facets") is False + assert surf("hemis.left.attributes.position.array.length / 3") == whole + assert faces() > 100000, "the surface stopped being drawn" + shading("smooth") + + page.evaluate("window.viewer.ui.set('surface.%s.low poly', 8)" % subj) + deadline = time.time() + 60 + while time.time() < deadline and faces() > 100000: + page.wait_for_timeout(500) + coarse = faces() + assert coarse < 100000, "the coarse surface never arrived" + + for name in ["volume", "vertex"]: + page.evaluate("window.viewer.setData(%r)" % name) + page.wait_for_timeout(3000) + shading("smooth") + smooth = solid() + for mode in ["flat", "gouraud", "phong"]: + shading(mode) + assert surf("_facets") is True, ( + "%s on %s data did not take a vertex per corner" % (mode, name)) + assert surf("setShading()") == mode + assert faces() == coarse, ( + "%s changed how many faces are drawn" % mode) + patches = solid() + if mode == "flat": + #the light comes from the face that is drawn, so the + #coarse faces are solid patches + assert patches > 0.25, ( + "%s data drawn flat is not in patches of one color: %.2f" + % (name, patches)) + else: + assert patches < smooth + 0.1, ( + "%s is shading a face as flat does" % mode) + shading("smooth") + + assert surf("_facets") is False + assert surf("hemis.left.attributes.position.array.length / 3") == whole + + #the line is the surface drawn once more + page.evaluate("window.viewer.ui.set('surface.%s.outline', true)" % subj) + page.wait_for_timeout(2000) + assert surf("setOutline()") is True + assert faces() == 2 * coarse, "the outline is not drawn over the surface" + + #the line is black on a black page, so it is counted where the + #background is clear: in a saved image + def inked(black=False): + """How many pixels the surface and its line cover, or with + `black` how many of them are the line's own color.""" + return page.evaluate("""(black) => { + var c = window.viewer.getImage(256, 192); + var d = c.getContext('2d').getImageData(0, 0, c.width, c.height).data; + var on = 0; + for (var i = 0; i < d.length; i += 4) + if (d[i+3] > 0 && (!black || d[i] + d[i+1] + d[i+2] < 45)) on++; + return on; + }""", black) + + assert surf("setOutlineWidth()") == pytest.approx(0.004) + thin = inked() + assert thin > 1000, "the brain was not drawn" + page.evaluate( + "window.viewer.ui.set('surface.%s.outline width', 0.02)" % subj) + page.wait_for_timeout(1500) + assert surf("setOutlineWidth()") == pytest.approx(0.02) + wide = inked() + assert wide > thin * 1.02, "the wider line did not grow the surface" + + #round corners put a dot of the line's width at each vertex of + #the faces being drawn, which fills the corners the line comes + #to a point at + def points(): + return page.evaluate( + "() => { window.viewer.draw();" + " return window.viewer.renderer.info.render.points; }") + + #the labels are drawn as points of their own, so the dots are + #counted over them + assert surf("setOutlineCorners()") == "sharp" + labels = points() + pointed = inked(black=True) + page.evaluate( + "window.viewer.ui.set('surface.%s.outline corners', 'round')" % subj) + page.wait_for_timeout(1500) + assert surf("setOutlineCorners()") == "round" + drawn = points() - labels + assert 0 < drawn <= 2 * coarse * 3, "the dots are not on the vertices of the faces drawn" + assert inked(black=True) > pointed, "the round corners added nothing to the line" + page.evaluate( + "window.viewer.ui.set('surface.%s.outline corners', 'sharp')" % subj) + page.wait_for_timeout(1000) + assert points() == labels + page.evaluate( + "window.viewer.ui.set('surface.%s.outline width', 0.004)" % subj) + page.wait_for_timeout(1000) + + page.evaluate("window.viewer.ui.set('surface.%s.outline', false)" % subj) + page.wait_for_timeout(1500) + assert faces() == coarse + assert not errors, errors + browser.close() + finally: + server.stop() + + +@pytest.mark.timeout(400) +def test_the_saved_image_can_be_antialiased(): + """`Antialias` renders the image larger and averages it back down. + + A webgl render target carries one sample a pixel whatever the canvas was + made with, so an image read out of one has hard edges. Rendering it + several samples across a pixel and averaging those back down leaves an + edge pixel the color of the surface and an alpha from how much of it the + surface covers, so the edge fades into the transparent background rather + than into black. + """ + from playwright.sync_api import sync_playwright + + vol = cortex.Volume(np.random.randn(*volshape), subj, xfmname) + server = cortex.webgl.show(vol, open_browser=False, display_url=False, autoclose=False) + try: + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True, args=[ + "--enable-webgl", "--use-gl=swiftshader", "--no-sandbox", "--disable-dev-shm-usage"]) + page = browser.new_page(viewport={"width": 800, "height": 560}) + errors = [] + page.on("pageerror", lambda exc: errors.append(str(exc))) + page.goto(server.url("mixer.html", host="localhost"), wait_until="load", timeout=120000) + page.wait_for_function( + "window.viewer && window.viewer.loaded.state() == 'resolved'", timeout=240000) + page.wait_for_timeout(3000) + + #a buffer whose top left pixel is half on an opaque red surface and + #half off it, and whose bottom right one is all on a green surface + block = page.evaluate("""() => { + var w = 4, h = 4; + var raw = new Uint8Array(w * h * 4); + function put(x, y, r, g, b, a) { + var i = 4 * (y * w + x); + raw[i] = r; raw[i+1] = g; raw[i+2] = b; raw[i+3] = a; + } + //readPixels counts the rows from the bottom, so rows 2 and 3 + //are the top of the image and rows 0 and 1 the bottom of it + put(0, 3, 255, 0, 0, 255); + put(1, 2, 255, 0, 0, 255); + for (var y = 0; y < 2; y++) + for (var x = 2; x < 4; x++) + put(x, y, 0, 255, 0, 255); + var gl = {FRAMEBUFFER:0, RGBA:1, UNSIGNED_BYTE:2, + bindFramebuffer:function() {}, + readPixels:function(x, y, w, h, f, t, out) { out.set(raw); }}; + var canvas = mriview.downsampleTexture( + gl, {width:w, height:h, __webglFramebuffer:null}, 2); + var d = canvas.getContext('2d').getImageData(0, 0, 2, 2).data; + return [canvas.width, canvas.height].concat( + Array.prototype.slice.call(d)); + }""") + assert block[:2] == [2, 2], "the buffer was not averaged down" + #the half covered pixel keeps the color of the surface and takes + #its alpha from the half of it the surface is not on + assert block[2:5] == [255, 0, 0], "the edge was averaged into black" + assert abs(block[5] - 128) <= 2, "the edge did not take a part alpha" + assert block[6:10] == [0, 0, 0, 0] and block[10:14] == [0, 0, 0, 0] + #the covered pixel stays as it was, at the end it was read from + assert block[14:18] == [0, 255, 0, 255], "the image came back flipped" + + def saved(mode): + """Save an image and count how the surface meets the page. + + A pixel the surface does not cover is clear and one it covers + is solid; a hard edge is a clear pixel straight against a + solid one, which is the step antialiasing grades. Fractional + alpha on its own says nothing here, because the data layer + leaves some of the surface at an alpha of its own. + """ + page.evaluate( + "window.viewer.ui.set('camera.Save image.Antialias', %r)" % mode) + page.wait_for_timeout(500) + return page.evaluate("""() => { + var c = window.viewer.getImage(256, 192); + var d = c.getContext('2d').getImageData(0, 0, c.width, c.height).data; + var out = {width:c.width, height:c.height, solid:0, hard:0}; + for (var y = 0; y < c.height; y++) { + for (var x = 0; x < c.width; x++) { + var i = 4 * (y * c.width + x), j = i + 4; + if (d[i+3] === 255) out.solid++; + if (x + 1 === c.width) continue; + if ((d[i+3] === 0 && d[j+3] > 200) || + (d[j+3] === 0 && d[i+3] > 200)) out.hard++; + } + } + return out; + }""") + + one = saved("none") + assert (one["width"], one["height"]) == (256, 192) + assert one["solid"] > 1000, "the brain was not drawn" + #one sample a pixel, so the surface stops where a pixel does + assert one["hard"] > 40, ( + "the silhouette of %d pixels has only %d hard edges" + % (one["solid"], one["hard"])) + + many = saved("4x") + assert (many["width"], many["height"]) == (256, 192), ( + "antialiasing changed the size the image is saved at") + assert many["solid"] > 1000, "the brain was not drawn" + #averaged down from sixteen samples a pixel, the surface fades + #into the page instead of stopping against it + assert many["hard"] < one["hard"] / 2, ( + "%d of the %d hard edges are still there" + % (many["hard"], one["hard"])) + + #the buffer each image was read from is handed back, and the page + #goes on drawing + page.wait_for_timeout(1000) + assert page.evaluate( + "() => { window.viewer.draw();" + " return window.viewer.renderer.info.render.faces; }") > 100000 + assert not errors, errors + browser.close() + finally: + server.stop() + + +@pytest.mark.timeout(600) +def test_ambient_occlusion_darkens_the_folded_surface_and_not_the_flat_one(): + """`ambient occlusion` takes the sky a vertex misses out of the light on + it, worked out on each surface the viewer mixes between: the folded + surface is mostly sulcal wall and darkens, the inflated one has opened + its sulci and barely changes, and the flat one blocks none of its own sky + and does not change at all. + """ + from playwright.sync_api import sync_playwright + + data = cortex.Dataset( + volume=cortex.Volume(np.random.randn(*volshape), subj, xfmname), + vertex=cortex.Vertex(np.random.randn(nverts), subj)) + server = cortex.webgl.show(data, open_browser=False, display_url=False, autoclose=False) + try: + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True, args=[ + "--enable-webgl", "--use-gl=swiftshader", "--no-sandbox", "--disable-dev-shm-usage"]) + page = browser.new_page(viewport={"width": 800, "height": 560}) + errors = [] + page.on("pageerror", lambda exc: errors.append(str(exc))) + page.on("console", lambda msg: errors.append(msg.text) + if msg.type == "error" and "WebGL" in msg.text else None) + page.goto(server.url("mixer.html", host="localhost"), wait_until="load", timeout=120000) + page.wait_for_function( + "window.viewer && window.viewer.loaded.state() == 'resolved'", timeout=240000) + page.wait_for_timeout(3000) + + def brightness(width=320): + """The mean brightness of the pixels the surface covers in + a saved image, where the page is clear behind it.""" + return page.evaluate("""(width) => { + var c = window.viewer.getImage(width, 3 * width / 4); + var d = c.getContext('2d').getImageData(0, 0, c.width, c.height).data; + var sum = 0, n = 0; + for (var i = 0; i < d.length; i += 4) + if (d[i+3] > 200) { sum += d[i] + d[i+1] + d[i+2]; n++; } + return sum / (3 * n); + }""", width) + + def occlusion(on): + page.evaluate( + "window.viewer.ui.set('surface.%s.lighting.ambient occlusion', %s)" + % (subj, "true" if on else "false")) + if on: + #the values are fetched, and worked out by the server + #the first time, before the shaders switch over + page.wait_for_function( + "window.viewer.surfs[0].surf._occluded === true", timeout=300000) + page.wait_for_timeout(2000) + + def unfold(mix): + page.evaluate("window.viewer.ui.set('surface.%s.unfold', %s)" % (subj, mix)) + page.wait_for_timeout(1500) + + lit = {} + for mix, name in [(0, "folded"), (0.5, "inflated"), (1, "flat")]: + unfold(mix) + occlusion(False) + before = brightness() + occlusion(True) + lit[name] = (before, brightness()) + assert page.evaluate("window.viewer.surfs[0].surf.setOcclusion()") is True + + folded = 1 - lit["folded"][1] / lit["folded"][0] + inflated = 1 - lit["inflated"][1] / lit["inflated"][0] + flat = abs(1 - lit["flat"][1] / lit["flat"][0]) + #the data layer over it is noise, so the darkening of the picture + #as a whole is a fraction of what the curvature underlay shows + assert folded > 0.05, "the folded surface only darkened by %.0f%%" % (100 * folded) + assert inflated < folded / 4, ( + "the inflated surface darkened by %.0f%%, the folded one by %.0f%%" + % (100 * inflated, 100 * folded)) + assert flat < 0.005, "the flat surface changed by %.1f%%" % (100 * flat) + + #the folded surface is occluded at each depth by its own + #surface, the pial one at 0 and the white matter at 1: shown on + #the curvature alone, with the data layer out of the way + def depth(value): + page.evaluate("window.viewer.ui.set('surface.%s.depth', %s)" % (subj, value)) + page.wait_for_timeout(1500) + + def strength(value): + page.evaluate( + "window.viewer.ui.set('surface.%s.lighting.occlusion strength', %s)" + % (subj, value)) + page.wait_for_timeout(1500) + + #close up and in a larger image, so that the labels drawn over + #the surface, which the occlusion leaves alone, are few of its pixels + unfold(0) + page.evaluate("window.viewer.ui.set('surface.%s.opacity', 0)" % subj) + page.evaluate("window.viewer.ui.set('camera.radius', 250)") + darkening = {} + for value in (0, 1): + depth(value) + occlusion(False) + before = brightness(640) + occlusion(True) + darkening[value] = 1 - brightness(640) / before + assert darkening[value] > 0.05, ( + "at depth %d the surface only darkened by %.0f%%" + % (value, 100 * darkening[value])) + assert abs(darkening[0] - darkening[1]) > 0.03, ( + "the two depths darkened alike, %.0f%% and %.0f%%" + % (100 * darkening[0], 100 * darkening[1])) + + #the strength scales what is taken out: none at 0, more at 2 + depth(0) + occlusion(False) + unlit = brightness(640) + occlusion(True) + full = brightness(640) + strength(0) + none = brightness(640) + strength(2) + twice = brightness(640) + strength(1) + assert abs(none - unlit) < unlit * 0.02, ( + "at no strength the surface is lit as without occlusion: %.1f against %.1f" + % (none, unlit)) + assert twice < full < none, "the strength does not scale the darkening" + page.evaluate("window.viewer.ui.set('surface.%s.opacity', 1)" % subj) + page.evaluate("window.viewer.ui.set('camera.radius', 400)") + depth(0.5) + + #data held on the vertices is lit by the other shader + page.evaluate("window.viewer.setData('vertex')") + page.wait_for_timeout(3000) + with_ao = brightness() + occlusion(False) + assert brightness() > with_ao * 1.04, "the vertex shader did not darken" + assert not errors, errors + browser.close() + finally: + server.stop() diff --git a/cortex/utils.py b/cortex/utils.py index dcfe5ccb5..098b77c7a 100644 --- a/cortex/utils.py +++ b/cortex/utils.py @@ -138,6 +138,193 @@ def get_ctmpack(subject, types=("inflated",), method="raw", level=0, recache=Fal return ctmfile +#The widths, in mm, that a viewer served as files rather than by a server is +#given coarse surfaces for, and the widest one any viewer offers. +LOD_CELLS = (2, 4, 8, 16, 32) +LOD_MAX = 32 + + +def get_lod(subject, cell, recache=False, **kwargs): + """Faces for drawing this subject's surfaces coarsely, cached, built if missing. + + The faces are cut from the surfaces of the subject's ctm pack, in its + numbering and in groups of `cell` mm (see `polyutils.decimate_faces`), and + written beside it as one file holding both hemispheres: the number of + faces of each as two little-endian uint32, then their vertices, three + uint32 to a face, the left hemisphere's first. + + Parameters + ---------- + subject : str + Name of subject in the pycortex store. + cell : int + How wide a group of vertices is taken over, in mm. The surfaces are + left whole at 0. + recache : bool, optional + Whether to build the file again when one is already there. + **kwargs + Forwarded to `get_ctmpack`, which says which pack these are cut from. + + Returns + ------- + lodfile : str + Path to the file. + """ + from . import polyutils + + jsfile = get_ctmpack(subject, **kwargs) + base = os.path.splitext(jsfile)[0] + lodfile = "%s_lod%d.bin" % (base, int(cell)) + if os.path.exists(lodfile) and not recache: + return lodfile + + from . import brainctm + hemis = brainctm.read_pack(base + ".ctm") + #The medial wall, which the flatmap is cut along, keeps its vertices out + #of the groups around it: a group that took in vertices from both sides + #of a cut would be drawn across the flatmap by the face that bridges it, + #which the viewer leaves out by the wall it touches. + walls = _medial_wall(subject, base, [len(pts) for pts, _ in hemis]) + faces = [polyutils.decimate_faces(pts, polys, cell, split=wall)[1] + for (pts, polys), wall in zip(hemis, walls)] + + #written whole, so that a reader never gets half a file + handle, temp = tempfile.mkstemp(dir=os.path.split(lodfile)[0], suffix=".bin") + with os.fdopen(handle, "wb") as fp: + np.array([len(f) for f in faces], dtype=" +{% end %} +{% block css %} + +{% end %} +{% block jsinit %} + var viewer, figure, sock; + var viewopts = {}; +{% end %} +{% block onload %} + figure = new jsplot.W2Figure(); + viewer = figure.add(aligner.Aligner, "main", true, {{config}}); + {% if python_interface %} + sock = new Websock(); + {% end %} +{% end %} +{% block extrahtml %} + + +{% end %} diff --git a/cortex/webgl/aligner.py b/cortex/webgl/aligner.py new file mode 100644 index 000000000..b26aee8e8 --- /dev/null +++ b/cortex/webgl/aligner.py @@ -0,0 +1,645 @@ +"""Browser-based manual aligner. + +Moves the anatomical surfaces (pial and white matter) in the space of a +functional reference volume. The volume +stays on its own voxel grid, so its slices are displayed without +resampling, and the surfaces are cut off at the displayed slices, which +draws their outline on the anatomy in the image. A second view mode paints +the volume onto the surfaces instead. Rendering happens in the browser +through the WebGL viewer's machinery (``cortex/webgl/resources/js/aligner.js``); +the transform is served, edited and saved through a tornado server in this +process, like ``cortex.webgl.show``. + +Saving an edited alignment deletes the masks cached for the transform, +since they were cut through the alignment it replaces. + +The entry point for users is :func:`cortex.align.webgl_manual`. +""" +import base64 +import glob +import json +import mimetypes +import os +import queue +import re +import time +import uuid +import warnings +import webbrowser +from typing import Any, Optional, Union, cast + +import numpy as np +import numpy.typing as npt +from tornado import web + +from .. import options, utils, volume +from ..database import db +from . import serve +from .data import _pack_png +from .FallbackLoader import FallbackLoader +from .serve import P +from .view import colormaps + +#: Name under which the reference volume is served to the page +REFERENCE_NAME = "reference" + +#: What the page shows, as its `display` control names it +DISPLAYS = dict( + slices="3 ortho + 3D slices", + brain="3 ortho + 3D brain", + surface="data on the surface", +) + +#: A transform name has to serve as a directory name in the filestore +XFM_NAME = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$") + + +def check_xfm_name(name: str) -> str: + """Return `name` if it can be used as the name of a transform. + + The name becomes a directory in the filestore, so anything else is + refused here rather than reaching the filesystem. + + Parameters + ---------- + name : str + The name to check. + + Returns + ------- + name : str + The name, unchanged. + + Raises + ------ + ValueError + If the name is empty or holds anything but letters, digits, '.', + '_' and '-', or does not start with a letter or a digit. + """ + if not isinstance(name, str) or XFM_NAME.match(name) is None: + raise ValueError( + "%r is not a usable transform name: use letters, digits, '.', '_' and " + "'-', starting with a letter or a digit" % (name,) + ) + return name + + +def reference_frame(nii) -> npt.NDArray[np.float64]: + """The voxel-to-world matrix the aligner works in for a reference image. + + The world frame keeps the voxel grid of the reference image axis aligned, + scaled to millimeters by the voxel sizes, and permuted and flipped so that + its x, y and z axes point to the subject's right, anterior and superior. + Slices of the reference image are therefore drawn without resampling, and + the surfaces are moved by rigid transforms expressed in millimeters. + + Parameters + ---------- + nii : nibabel.Nifti1Image + The reference image. + + Returns + ------- + world : (4, 4) ndarray + Affine mapping voxel indices of `nii` to world coordinates. + """ + import nibabel + + zooms = np.asarray(nii.header.get_zooms()[:3], dtype=float) + ornt = nibabel.io_orientation(nii.affine) + world = np.zeros((4, 4)) + world[3, 3] = 1.0 + for voxel_axis, (world_axis, direction) in enumerate(ornt): + world[int(world_axis), voxel_axis] = float(direction) * zooms[voxel_axis] + return world + + +def load_reference(nii) -> npt.NDArray[np.float32]: + """The reference data as a 3D float32 array in (x, y, z) voxel order. + + A 4D image contributes its first volume; NaNs are replaced by zeros. + """ + data = np.asarray(nii.get_fdata()) + while data.ndim > 3: + data = data[..., 0] + if data.ndim != 3: + raise ValueError("The reference image must have three dimensions, got shape %s" % (data.shape,)) + return np.nan_to_num(data).astype(np.float32) + + +def _color_hex(color: str) -> str: + from matplotlib.colors import to_hex + + return to_hex(color) + + +def cached_masks(subject: str, xfmname: str) -> list[str]: + """The paths of the masks cached for a transform. + + Masks are cut out of the reference volume through the transform, so + editing the alignment makes every one of them wrong. + + Parameters + ---------- + subject : str + Subject identifier. + xfmname : str + Name of the transform. + + Returns + ------- + paths : list of str + Paths of the cached mask files, empty when the transform has none. + """ + pattern = db.get_paths(subject)["masks"].format(xfmname=xfmname, type="*") + return sorted(glob.glob(pattern)) + + +def clear_masks(subject: str, xfmname: str) -> list[str]: + """Delete the masks cached for a transform, and return their names. + + The aligner calls this when it saves an edited alignment: the masks it + deletes were cut with the previous alignment, and nothing else + invalidates them. ``db.save_xfm`` also refuses to write over a transform + that still has masks. + + Parameters + ---------- + subject : str + Subject identifier. + xfmname : str + Name of the transform. + + Returns + ------- + names : list of str + Names of the deleted masks, as `db.get_mask` takes them (the `thick` + of `mask_thick.nii.gz`). Empty when the transform had no masks. + """ + names = [] + for path in cached_masks(subject, xfmname): + name = os.path.split(path)[1] + if name.startswith("mask_") and name.endswith(".nii.gz"): + name = name[len("mask_"):-len(".nii.gz")] + os.unlink(path) + names.append(name) + return names + + +def _reply_of(message: Any) -> Any: + """What the page answered, out of the envelope the websocket wraps it in. + + Every answer carries the id of the request it belongs to (see + ``serve.WebApp.send``), which `send` itself unwraps. A reply read straight + off the server's queue, because it arrived after `send` stopped waiting, + still has it around. + """ + if isinstance(message, dict) and "id" in message and "result" in message: + return message["result"] + return message + + +class JSAligner(serve.JSProxy[P]): + """Handle to an aligner running in the browser. + + Besides the generic attribute access of :class:`cortex.webgl.serve.JSProxy`, + this exposes the transform being edited and the controls of the page. + Its methods tag their requests, so that replies delayed by a busy page + (while it parses the surfaces, or draws a slow frame) are matched to the + right request; the generic attribute access does not have this protection. + """ + + #: Seconds to wait for the tagged reply of a call + call_timeout = 120.0 + + def _call(self, name: str, *args: Any) -> Any: + token = uuid.uuid4().hex + resp = self.send(method="run", params=["window.viewer.call", [token, name, list(args)]]) + # plain attribute lookups on a JSProxy go to the page; read the + # bookkeeping kept on the python object from its dict instead + server = vars(self).get("server") + if len(resp) == 0: + return None + deadline = time.monotonic() + self.call_timeout + reply = resp[0] + while True: + if isinstance(reply, dict) and reply.get("token") == token: + # a redraw pending at the time of the reply lands in the next + # frame; wait_for_frame waits for it + if reply.get("scheduled", False): + target = int(cast(float, reply.get("frames", 0))) + 1 + object.__setattr__(self, "_frame_target", max(target, vars(self).get("_frame_target", 1))) + if "error" in reply: + raise RuntimeError("%s: %s" % (name, reply["error"])) + return reply.get("value") + if server is None: + # without the server's queue, stale replies cannot be skipped + if reply is None: + return None + raise RuntimeError("Unexpected reply to %s: %r" % (name, reply)) + if time.monotonic() > deadline: + raise TimeoutError("No reply to %s within %.0f s" % (name, self.call_timeout)) + try: + reply = _reply_of(json.loads(server.response.get(timeout=1))) + except queue.Empty: + reply = None + + def wait_for_frame(self, timeout: float = 120.0) -> int: + """Block until the page has drawn the effect of the last call. + + Returns the number of frames drawn so far. Drawing happens in the + page's own animation frames, which can be slow without a GPU, so a + snapshot taken right after a change may otherwise show the previous + state. + """ + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + # the reply to this call also records a redraw still pending + frames = self._call("getFrames") + target = max(vars(self).get("_frame_target", 1), 1) + if frames is not None and int(frames) >= target: + return int(frames) + time.sleep(0.2) + raise TimeoutError("The aligner did not draw a new frame within %.0f s" % timeout) + + def get_control(self, name: str) -> Any: + """The value of a control of the page by its dotted path, such as + ``"image.vmin"`` or ``"mesh.color"``.""" + return self._call("getControl", name) + + def set_control(self, name: str, value: Any) -> None: + """Set a control of the page by its dotted path, such as + ``"image.colormap"`` or ``"display"``.""" + self._call("setControl", name, value) + + def get_xfm(self) -> npt.NDArray[np.float64]: + """The current transform, as a (4, 4) pycortex 'coord' matrix + (anatomical coordinates to voxel indices of the reference).""" + return np.asarray(self._call("getXfm"), dtype=float) + + def set_xfm(self, xfm: npt.ArrayLike) -> None: + """Replace the current transform with a (4, 4) 'coord' matrix.""" + matrix = np.asarray(xfm, dtype=float) + if matrix.shape != (4, 4): + raise ValueError("The transform must be a 4x4 matrix") + self._call("setXfm", matrix.tolist()) + + def translate(self, vector: npt.ArrayLike) -> None: + """Move the surfaces by `vector`, in millimeters along the world axes + (right, anterior, superior).""" + self._call("translate", [float(v) for v in np.asarray(vector).ravel()]) + + def rotate(self, axis: npt.ArrayLike, angle: float) -> None: + """Rotate the surfaces by `angle` degrees about the world `axis` + through the cursor.""" + self._call("rotate", [float(v) for v in np.asarray(axis).ravel()], float(angle)) + + def undo(self) -> None: + """Undo the last change to the transform.""" + self._call("undo") + + def save(self, timeout: float = 60.0) -> str: + """Save the current transform into the database, as the Save button does. + + Returns the message the server answered with, once the transform is + written; the page posts the save, so this waits for that request to + come back rather than returning while it is still out. + + Parameters + ---------- + timeout : float, optional + Seconds to wait for the save to land. + + Raises + ------ + RuntimeError + If the server refused the save, with the reason it gave. + TimeoutError + If no answer arrived within `timeout`. + """ + self._call("save") + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + state = self._call("getSaveState") + if isinstance(state, dict) and state.get("status") != "saving": + message = str(state.get("message", "")) + if state.get("status") != "ok": + raise RuntimeError(message or "the transform was not saved") + return message + time.sleep(0.1) + raise TimeoutError("The aligner did not finish saving within %.0f s" % timeout) + + def snapshot(self, filename: Optional[str] = None) -> bytes: + """The current rendering of the four views as PNG bytes, also written + to `filename` when given. Waits for the frame showing the last change + first.""" + self.wait_for_frame() + data_url = self._call("snapshot") + png = base64.b64decode(data_url.split(",", 1)[1]) + if filename is not None: + with open(filename, "wb") as fp: + fp.write(png) + return png + + +def show( + subject: str, + xfmname: str, + reference: Optional[str] = None, + view_only: bool = False, + cmap: Optional[str] = None, + mesh_color: Optional[str] = None, + mesh_opacity: Optional[float] = None, + open_browser: Optional[bool] = None, + autoclose: Optional[bool] = None, + port: Optional[int] = None, + recache: bool = False, + types: tuple[str, ...] = ("inflated",), + title: Optional[str] = None, + display_url: bool = True, + token: Optional[str] = None, + template: str = "aligner.html", +) -> Union[JSAligner, serve.WebApp]: + """Open the browser-based aligner for a transform of `subject`. + + The functional reference volume is shown on its own voxel grid in three + slice views and a 3D view, with the pial and white matter surfaces of + `subject` cut to the displayed slices. Rotating and translating the + surfaces edits the transform; the Save button writes it into the + database as `xfmname`. See :func:`cortex.align.webgl_manual` for the + controls. + + Saving deletes the masks cached for `xfmname` (the page warns when it + opens a transform that has some): they were cut through the alignment + being replaced. Data already masked with them has to be masked again + from the volumes. + + Parameters + ---------- + subject : str + Subject identifier. + xfmname : str + Name of the transform to create or modify. + reference : str, optional + Path to a nibabel-readable functional volume, required when `xfmname` + does not exist yet. For an existing transform, leave it None: the + stored reference is loaded, and the transform is used as the starting + point. + view_only : bool, optional + Open the aligner without the possibility to save, to inspect an + alignment. + cmap : str, optional + Initial colormap for the reference volume, one of the 1D pycortex + colormaps. Defaults to the `colormap` option of the `webgl_aligner` + section of the config file. + mesh_color : str, optional + Initial color of the surface outlines, as a matplotlib color. Defaults + to the `mesh_color` config option. + mesh_opacity : float, optional + Initial opacity of the whole surfaces in the 3D view (0 shows only + their outlines on the slices). Defaults to the `mesh_opacity` config + option. + open_browser : bool, optional + Open the aligner in the default browser. Defaults to the + `open_browser` option of the `webshow` config section. + autoclose : bool, optional + Stop the server when the last browser window disconnects. Defaults to + the `autoclose` option of the `webshow` config section. + port : int, optional + Port of the server; a free port is picked when None. + recache : bool, optional + Regenerate the cached surface (CTM) files. + types : tuple of str, optional + Surface types included in the CTM pack, to share the cache with the + viewer. Default ("inflated",). + title : str, optional + Title of the browser window. + display_url : bool, optional + When `open_browser` is False, display an IPython link to the aligner. + token : str, optional + The session token the server demands, which the address it prints + carries and the page then keeps in a cookie. A new one is made for + each aligner; pass '' to take requests from anything that reaches + the port, which a script talking to the server itself may want. + template : str, optional + Name of the tornado template of the page. Default 'aligner.html'. + + Returns + ------- + handle : JSAligner or WebApp + With `open_browser`, a handle to the running aligner (its ``server`` + attribute is the tornado server); otherwise the server itself, whose + ``get_client()`` returns the handle once a browser has connected. + """ + import nibabel + + close_on_disconnect: bool = ( + options.config.get("webshow", "autoclose", fallback="true") == "true" + if autoclose is None + else autoclose + ) + if open_browser is None: + open_browser = options.config.get("webshow", "open_browser", fallback="true") == "true" + + # The transform to start from: the stored one, or the header alignment + # of a new reference (anatomical and functional scanner spaces coincide) + try: + dbxfm = db.get_xfm(subject, xfmname, xfmtype="coord") + except IOError: + dbxfm = None + + if dbxfm is not None: + if reference is not None: + raise ValueError( + "Refusing to overwrite the reference of the existing transform %s; " + "pass reference=None to load the stored reference" % xfmname + ) + nii = dbxfm.reference_nifti + reference = nii.get_filename() + coord = np.asarray(dbxfm.xfm, dtype=float) + else: + if reference is None or not os.path.exists(reference): + raise ValueError("Reference image file (%s) does not exist" % reference) + nii = cast("nibabel.Nifti1Image", nibabel.load(reference)) + coord = np.linalg.inv(nii.affine) + + epi = load_reference(nii) + world = reference_frame(nii) + percentiles = np.percentile(epi, [1, 99]) + vmin, vmax = float(percentiles[0]), float(percentiles[1]) + if vmin == vmax: + vmin, vmax = float(epi.min()), float(epi.max()) + + # The reference is served as the float mosaic PNG the viewer uses + mosaic, mosaic_shape = volume.mosaic(epi.T, show=False) + png = _pack_png(np.ascontiguousarray(mosaic, dtype=np.float32)) + + # The surfaces come from the CTM pack the viewer uses: base positions + # are the pial surface and the `wm` attribute is the white matter + ctmfile = utils.get_ctmpack(subject, types, method="mg2", level=9, recache=recache) + + cmap_names = [name for name, _ in cast(list[tuple[str, str]], colormaps)] + if cmap is None: + cmap = options.config.get("webgl_aligner", "colormap", fallback="gray") + if cmap not in cmap_names: + warnings.warn("Colormap %s is not available, using gray" % cmap) + cmap = "gray" + if mesh_color is None: + mesh_color = options.config.get("webgl_aligner", "mesh_color", fallback="white") + if mesh_opacity is None: + mesh_opacity = float(options.config.get("webgl_aligner", "mesh_opacity", fallback="0")) + if title is None: + title = "Aligner: %s %s" % (subject, xfmname) + + # Handed to the page and demanded back on a save. The session token says + # the request comes from this computer; this one says it comes from the + # page itself, which a site the browser is also on cannot read, so a form + # it posts carries the session cookie but no save token. + save_token = uuid.uuid4().hex + + config: dict[str, Any] = dict( + subject=subject, + xfmname=xfmname, + ctm="ctm/%s/" % subject, + view_only=view_only, + save_token=save_token, + # shown as a warning on the page: saving deletes these + masks=[os.path.split(path)[1] for path in cached_masks(subject, xfmname)], + volume=dict( + name=REFERENCE_NAME, + subject=subject, + raw=False, + min=float(epi.min()), + max=float(epi.max()), + mosaic=[int(v) for v in mosaic_shape], + shape=[int(v) for v in epi.T.shape], + ), + images={REFERENCE_NAME: ["data/%s.png" % REFERENCE_NAME]}, + world=world.tolist(), + xfm=coord.tolist(), + cmap=cmap, + vmin=vmin, + vmax=vmax, + mesh_color=_color_hex(mesh_color), + mesh_opacity=mesh_opacity, + ) + + html = FallbackLoader([os.path.split(os.path.abspath(template))[0], serve.cwd]).load(template) + + class CTMHandler(web.RequestHandler): + def get(self, path: str): + subj, path = path.split("/") + if subj != subject: + raise web.HTTPError(404) + if path == "": + self.set_header("Content-Type", "application/json") + with open(ctmfile) as fp: + self.write(fp.read()) + else: + fpath = os.path.join(os.path.split(ctmfile)[0], path) + mtype = mimetypes.guess_type(fpath)[0] + self.set_header("Content-Type", mtype if mtype is not None else "application/octet-stream") + with open(fpath, "rb") as fp: + self.write(fp.read()) + + class DataHandler(web.RequestHandler): + def get(self, path: str): + self.set_header("Content-Type", "image/png") + self.write(png) + + class AlignerHandler(web.RequestHandler): + def get(self): + self.set_header("Content-Type", "text/html") + self.write( + html.generate( + config=json.dumps(config), + colormaps=colormaps, + default_cmap=cmap, + python_interface=True, + leapmotion=False, + title=title, + ) + ) + + class SaveHandler(web.RequestHandler): + def post(self): + self.set_header("Content-Type", "application/json") + # This writes to the filestore, so it only answers the page it was + # served to. The save token is handed out in that page, which a + # site the browser is also on cannot read, so a form it posts + # carries the session cookie but not this. + if not serve.same_token(self.get_argument("save_token", None), save_token): + self.set_status(403) + self.write(json.dumps(dict( + status="error", message="not saved: this is not the aligner's own page"))) + return + if view_only: + self.write(json.dumps(dict(status="error", message="view only: the transform is not saved"))) + return + try: + xfm = np.asarray(json.loads(self.get_argument("xfm")), dtype=float) + if xfm.shape != (4, 4): + raise ValueError("expected a 4x4 matrix, got shape %s" % (xfm.shape,)) + # The page can save the alignment under another name, which + # creates a transform of that name rather than changing the + # one that was loaded. + name = check_xfm_name(self.get_argument("name", xfmname).strip()) + # The masks of the transform being written were cut with the + # alignment being replaced, so they are wrong from here on; + # db.save_xfm also refuses to write over a transform that + # still has them. + dropped = clear_masks(subject, name) + db.save_xfm(subject, name, xfm, xfmtype="coord", reference=reference) + except Exception as exc: + self.write(json.dumps(dict(status="error", message="not saved: %s" % exc))) + return + message = "saved transform %s for %s" % (name, subject) + if len(dropped) > 0: + message += "; deleted %d stale mask%s (%s)" % ( + len(dropped), "" if len(dropped) == 1 else "s", ", ".join(dropped)) + print(message) + self.write(json.dumps(dict(status="ok", message=message, name=name, masks_deleted=dropped))) + + class WebApp(serve.WebApp): + disconnect_on_close = close_on_disconnect + + def get_client(self): + self.connect.wait() + self.connect.clear() + client = JSAligner(self.send, "window.viewer") + # The handle matches replies to its own requests by draining the + # server's queue, and needs the server to do it. Without it a + # reply delayed past WebApp.send's two second wait -- which the + # page takes while it parses the surfaces or draws a slow frame -- + # is lost rather than waited for. + # (bypasses JSProxy.__setattr__, which would query the page) + object.__setattr__(client, "server", self) + return client + + server = WebApp( + [ + (r"/ctm/(.*)", CTMHandler), + (r"/data/(.*)", DataHandler), + (r"/save", SaveHandler), + (r"/aligner.html", AlignerHandler), + (r"/", AlignerHandler), + ], + 0 if port is None else port, + token=token, + ) + server.start() + print("Started aligner server on port %d" % server.port) + #: under the machine's own name, which is what a port forward from + #: another computer is set up under, and carrying the session token + url = server.url("aligner.html") + if open_browser: + webbrowser.open(url) + return server.get_client() + elif display_url: + try: + from IPython.display import HTML, display + + display(HTML('Open aligner: {0}'.format(url))) + except Exception: + print("Open the aligner at %s" % url) + return server diff --git a/cortex/webgl/resources/css/aligner.css b/cortex/webgl/resources/css/aligner.css new file mode 100644 index 000000000..451ba4734 --- /dev/null +++ b/cortex/webgl/resources/css/aligner.css @@ -0,0 +1,230 @@ +/************************************************* + * Manual aligner: a 2x2 grid of views on one canvas, + * with the controls in a panel on the right + *************************************************/ +.jsplot_axes { + position:relative; + width:100%; + height:100%; +} +#aligner { + position:relative; + width:100%; + height:100%; + overflow:hidden; + background:#000; +} +#aligner-canvas { + position:absolute; + left:0; + top:0; + z-index:2; + display:block; +} +/* Transparent overlays, one per view, that receive the mouse events */ +.aligner-view { + position:absolute; + width:50%; + height:50%; + z-index:3; + box-sizing:border-box; + border:1px solid #444; + cursor:crosshair; +} +#view-y { left:0; top:0; } +#view-z { left:0; top:50%; } +#view-x { left:50%; top:0; } +#view-3d { left:50%; top:50%; cursor:default; } +/* the surface on its own, the way the viewer shows one */ +#aligner.single .aligner-view { display:none; } +#aligner.single #view-3d { + display:block; + left:0; + top:0; + width:100%; + height:100%; +} +.aligner-label { + position:absolute; + left:8px; + top:6px; + color:#ddd; + font-size:10pt; + text-shadow:0px 1px 2px #000; + pointer-events:none; +} +#aligner-load { + left:50%; + top:50%; + right:auto; + width:200px; + margin:-20px 0px 0px -120px; +} +#aligner-colormaps { + display:none; +} + +/* Control panel */ +#aligner-panel { + height:100%; + overflow-y:auto; + background:#1a1a1a; + color:#ccc; + font-size:9pt; + text-align:left; +} +#aligner-panel #figure_ui { + position:static; + width:100% !important; + max-height:none; + overflow:visible; +} +#aligner-viewonly, #aligner-masks { + display:none; + padding:6px 8px; + color:#fc9; +} +#aligner-status { + min-height:1.5em; + padding:6px 8px; + color:#8fd; +} +#aligner-status.error { + color:#f88; +} +#aligner-legend { + padding:4px 8px 12px 8px; + line-height:1.4; +} +#aligner-legend h4, #aligner-history h4 { + margin:10px 0px 2px 0px; + color:#eee; + font-size:9pt; +} +/* Edit history: one row per edit, newest at the bottom */ +#aligner-history { + padding:0px 8px; +} +#aligner-history-list { + /* the rows are kept in view by their offset within this list */ + position:relative; + margin:0px; + padding:0px; + list-style:none; + max-height:160px; + overflow-y:auto; + border:1px solid #333; + background:#141414; +} +#aligner-history-list li { + display:flex; + justify-content:space-between; + align-items:baseline; + padding:1px 6px; + cursor:pointer; +} +.aligner-history-kind { + flex:none; +} +#aligner-history-list li:hover { + background:#262626; +} +#aligner-history-list li.current { + background:#0a6; + color:#fff; +} +.aligner-history-size { + color:#9ab; + padding-left:8px; + text-align:right; +} +#aligner-history-list li.current .aligner-history-size { + color:#dfe; +} +#aligner-legend table { + width:100%; +} +#aligner-legend td { + padding:1px 4px; + vertical-align:top; +} +#aligner-legend td.key { + white-space:nowrap; + color:#fc9; +} + +/* Colormap dropdown: a strip of the colormap beside its name. + mriview.css hides the select2 control and leaves only its dropdown + visible, which is how the viewer opens its picker from the colorbar. The + aligner uses the control itself, so this puts select2's own layout back + for it. Only for it: the open dropdown is attached beside the body, and + giving that one a place in the layout pushes the page sideways. */ +#figure_ui .select2-container { + visibility:visible; + position:relative; +} +/* The open dropdown is attached beside the body and placed over the control, + so a width of its own (mriview.css gives it 30vw for the viewer's picker) + reaches past the right edge of the window and scrolls the page sideways. + This one is as wide as its strips need and hangs from the right edge of + the control, which is a panel's width in from the edge of the window. */ +.select2-dropdown.aligner-cmap-dropdown { + width:240px !important; + margin-left:-80px; +} + +.aligner-cmap { + display:flex; + align-items:center; +} +.aligner-cmap img { + flex:none; + width:64px; + height:11px; + margin-right:8px; + border:1px solid #555; +} +.aligner-cmap-name { + overflow:hidden; + text-overflow:ellipsis; + white-space:nowrap; +} +/* the closed control sits in a dat.GUI cell, so its strip is smaller */ +.select2-selection__rendered .aligner-cmap img { + width:40px; + height:9px; + margin-right:6px; +} +.select2-container--default .select2-selection--single { + background:#303030; + border:1px solid #555; + border-radius:0; + height:20px; +} +.select2-container--default .select2-selection--single .select2-selection__rendered { + color:#eee; + line-height:18px; + padding-left:4px; +} +.select2-container--default .select2-selection--single .select2-selection__arrow { + height:18px; +} +.select2-dropdown { + background:#1a1a1a; + border-color:#555; + color:#ccc; +} +.select2-container--default .select2-results__option { + padding:2px 6px; +} +.select2-container--default .select2-results__option[aria-selected=true] { + background:#333; +} +.select2-container--default .select2-results__option--highlighted[aria-selected] { + background:#0a6; +} +.select2-search--dropdown .select2-search__field { + background:#303030; + border:1px solid #555; + color:#eee; +} diff --git a/cortex/webgl/resources/css/mriview.css b/cortex/webgl/resources/css/mriview.css index 2a58d6f8c..68da81944 100644 --- a/cortex/webgl/resources/css/mriview.css +++ b/cortex/webgl/resources/css/mriview.css @@ -1,3 +1,12 @@ +/* The viewer fills the window and is laid out from its edges: the ends of + the colorbar, for one, are placed off the window on purpose. Pinning the + page itself as well as the body keeps a scrollbar off it even where the + body's own overflow is overridden, by a stylesheet loaded after this one + or by the browser. */ +html { + height:100%; + overflow:hidden; +} body { background-color:#333; font-family:Helvetica Neue, sans-serif; @@ -837,3 +846,73 @@ button#twodbutton:disabled, button#twodbutton[disabled] { .pycortex-views .cr.function .c { display: none; } + +/* dat.GUI's color picker sizes the saturation square and the hue bar by + their content and lays them out to add up to the width of the picker. + The panel sits inside a w2ui one, whose reset makes every box count its + border and padding within its width instead, which leaves the hue bar + two pixels too wide to fit beside the square and puts it underneath, out + of the picker and out of reach: the picker only shows while the pointer + is over it. */ +.dg .selector, .dg .selector * { + box-sizing:content-box; +} + +/* A switch, for the settings that put the page into a state rather than + show one more thing. dat.GUI draws a checkbox for every boolean, so the + box itself is made over into the track and its marker into the knob. */ +.dg li.toggle .c input[type="checkbox"] { + -webkit-appearance:none; + -moz-appearance:none; + appearance:none; + position:relative; + width:34px; + height:16px; + margin:4px 0px 0px 0px; + border-radius:8px; + background:#555; + cursor:pointer; + transition:background 0.15s; +} +.dg li.toggle .c input[type="checkbox"]:after { + content:""; + position:absolute; + top:2px; + left:2px; + width:12px; + height:12px; + border-radius:6px; + background:#ddd; + transition:left 0.15s; +} +.dg li.toggle .c input[type="checkbox"]:checked { + background:#806787; +} +.dg li.toggle .c input[type="checkbox"]:checked:after { + left:20px; +} + +/************************************************* + * Slice views beside the 3D one + *************************************************/ +/* One of these sits over each slice view: it names the view and keeps a drag + there from turning the 3D one, which is not what the pointer is over. */ +.mriview-ortho { + display:none; + position:absolute; + width:50%; + height:50%; + z-index:3; + box-sizing:border-box; + border:1px solid #444; + cursor:default; +} +.mriview-ortho span { + position:absolute; + left:8px; + top:6px; + color:#ddd; + font-size:10pt; + text-shadow:0px 1px 2px #000; + pointer-events:none; +} diff --git a/cortex/webgl/resources/js/aligner.js b/cortex/webgl/resources/js/aligner.js new file mode 100644 index 000000000..6cc735695 --- /dev/null +++ b/cortex/webgl/resources/js/aligner.js @@ -0,0 +1,1932 @@ +//Manual aligner: moves the anatomical surfaces in the space of a functional +//reference volume, the way the old mayavi aligner did. The volume stays on +//its own voxel grid, so its slices are displayed without resampling, and the +//pial and white matter surfaces are cut off at the displayed slices, which +//draws their outline on top of the anatomy in the image. A second view mode +//paints the volume onto the surface instead, for judging the alignment from +//the pattern the data makes on the cortex. +// +//The scene lives in the "world" frame handed over by cortex/webgl/aligner.py: +//the voxel grid of the reference volume in millimeters, with the axes +//permuted and flipped so that x, y and z point right, anterior and superior. +//The transform being edited is the pycortex "coord" transform (anatomical +//millimeters to voxel indices); in the scene it appears as the world matrix +//of the surface object, world <- anatomical = config.world * coord. +var aligner = (function(module) { + + //One slice view per world axis. `look` is the viewing direction and `up` + //the screen-up direction. The coronal and axial views follow the + //radiological convention (the subject's right on the left of the screen, + //as in cortex.volume.mosaic); the sagittal view is seen from the right. + var VIEWS = { + x: {axis: 0, title: "sagittal", look: [-1, 0, 0], up: [0, 0, 1]}, + y: {axis: 1, title: "coronal", look: [ 0,-1, 0], up: [0, 0, 1]}, + z: {axis: 2, title: "axial", look: [ 0, 0, 1], up: [0, 1, 0]}, + }; + + //Placement of the four views in the 2x2 grid, matching the mayavi aligner + var LAYOUT = [ + {name: "y", col: 0, row: 0}, + {name: "z", col: 0, row: 1}, + {name: "x", col: 1, row: 0}, + {name: "3d", col: 1, row: 1}, + ]; + + //What the page shows. The first two keep the three slice views and differ + //in the fourth panel: the slice planes in space, or the surface with the + //reference data painted on it through the alignment being edited. The + //third gives that surface the whole window, where it inflates and + //flattens the way the viewer's does. + var DISPLAY = { + slices: "3 ortho + 3D slices", + brain: "3 ortho + 3D brain", + surface: "data on the surface", + }; + var DISPLAY_ORDER = [DISPLAY.slices, DISPLAY.brain, DISPLAY.surface]; + module.DISPLAY = DISPLAY; + + //What one view draws: the surfaces outlined on the slices, or the data + //on the surfaces in the anatomy's own frame, where they inflate and + //flatten + var OUTLINE = 0, MORPHED = 1; + + //Millimeters of the flatmap per unit of its 2D coordinates, as in + //mriview_surface.js, so that a flattened surface comes out brain-sized + var FLATSCALE = 0.3; + + //What a movement along each world axis is called, in the order the axis + //runs: world x, y and z point right, anterior and superior + var DIRECTIONS = [["left", "right"], ["posterior", "anterior"], ["inferior", "superior"]]; + + //Slice setters, one per world axis, so that the slice controls in the + //menu and the views stay in sync + var SLICE_SETTERS = ["setSagittal", "setCoronal", "setAxial"]; + + var UNDO_LIMIT = 200; + //Wheel movement (in normalized pixels) that steps one slice + var WHEEL_STEP = 30; + + //Nested 4x4 array in row-major order to a THREE.Matrix4, and back + module.matrixFromRows = function(rows) { + var m = new THREE.Matrix4(); + m.set(rows[0][0], rows[0][1], rows[0][2], rows[0][3], + rows[1][0], rows[1][1], rows[1][2], rows[1][3], + rows[2][0], rows[2][1], rows[2][2], rows[2][3], + rows[3][0], rows[3][1], rows[3][2], rows[3][3]); + return m; + }; + module.matrixToRows = function(m) { + var e = m.elements, rows = []; + for (var i = 0; i < 4; i++) { + var row = []; + for (var j = 0; j < 4; j++) + row.push(e[i + 4 * j]); + rows.push(row); + } + return rows; + }; + + //Wheel deltas are in pixels, lines or pages depending on the browser and + //the device; normalize them all to pixels (as in movement.js) + function wheelDelta(event) { + var delta = event.deltaY; + if (event.deltaMode == 1) + delta *= 18; + else if (event.deltaMode == 2) + delta *= 180; + return delta; + } + + //Resolves once every image has decoded, so that the colormap textures + //and the list of 1D colormaps can be built from them + function whenImagesLoaded(images) { + var waiting = []; + for (var i = 0; i < images.length; i++) { + var img = images[i]; + if (img.complete && img.naturalHeight > 0) + continue; + var deferred = $.Deferred(); + img.addEventListener("load", deferred.resolve); + img.addEventListener("error", deferred.resolve); + waiting.push(deferred); + } + return $.when.apply($, waiting); + } + + //Unique triangle edges of an indexed BufferGeometry, as a line index + //buffer with the same chunking (offsets) as the triangle index, so that + //the mesh can be drawn as a wireframe. three.js r69 draws wireframes of + //indexed BufferGeometries by reinterpreting the triangle index as line + //pairs, which does not give the triangle edges. + module.buildEdges = function(geometry) { + var index = geometry.attributes.index.array; + var offsets = geometry.offsets; + var edges = new Uint16Array(index.length * 2); + var edgeOffsets = []; + var total = 0; + for (var j = 0; j < offsets.length; j++) { + var start = offsets[j].start, count = offsets[j].count; + var seen = new Set(); + var chunkStart = total; + var add = function(a, b) { + var key = a < b ? a * 65536 + b : b * 65536 + a; + if (seen.has(key)) + return; + seen.add(key); + edges[total++] = a; + edges[total++] = b; + }; + for (var i = start, il = start + count; i < il; i += 3) { + add(index[i], index[i+1]); + add(index[i+1], index[i+2]); + add(index[i+2], index[i]); + } + edgeOffsets.push({start: chunkStart, count: total - chunkStart, index: offsets[j].index}); + } + var edgeGeom = new THREE.BufferGeometry(); + edgeGeom.addAttribute("index", new THREE.BufferAttribute(edges.subarray(0, total), 2)); + edgeGeom.addAttribute("position", geometry.attributes.position); + edgeGeom.addAttribute("wm", geometry.attributes.wm); + edgeGeom.offsets = edgeOffsets; + return edgeGeom; + }; + + module.Aligner = function(figure, config) { + jsplot.Axes.call(this, figure); + this.config = config; + this.loaded = $.Deferred(); + this.ready = false; + this._ready = {volume: false, mesh: false}; + this._scheduled = false; + this._draw = this.draw.bind(this); + this.nframes = 0; + + $(this.object).html($("#aligner_html").html()); + this.canvas = $(this.object).find("#aligner-canvas"); + + //Frames: voxel -> world, and anatomical -> world for the surfaces + this.world = module.matrixFromRows(config.world); + this.worldInv = new THREE.Matrix4().getInverse(this.world); + this.xfm = new THREE.Matrix4().multiplyMatrices(this.world, module.matrixFromRows(config.xfm)); + //Everything the alignment has been through since the page opened: the + //one it was loaded with, then an entry per edit holding what the edit + //was and the alignment it left behind. Picking one puts that + //alignment back. + this._history = [{kind: "loaded", xfm: this.xfm.clone(), label: ""}]; + this._historyIndex = 0; + this._historyRows = []; + this._restoring = false; + //the point whose movement measures an edit, until the surfaces load + this._meshCenter = new THREE.Vector3(); + + //Voxel dimensions in voxel axis order, and for each world axis the + //voxel axis it is drawn from and the voxel size along it + var shape = config.volume.shape; + this.dims = [shape[2], shape[1], shape[0]]; + this.vax = [0, 0, 0]; + this.vscale = [1, 1, 1]; + var e = this.world.elements; + for (var a = 0; a < 3; a++) { + var best = -1; + for (var j = 0; j < 3; j++) { + var v = Math.abs(e[a + 4 * j]); + if (v > best) { + best = v; + this.vax[a] = j; + } + } + this.vscale[a] = best; + } + //The cursor (voxel coordinates) picks the displayed slices and is the + //pivot for rotations + this.cursor = [(this.dims[0] - 1) / 2, (this.dims[1] - 1) / 2, (this.dims[2] - 1) / 2]; + this.planeCoord = [0, 0, 0]; + + this._display = DISPLAY.slices; + this._mix = 0; + this._pivot = 0; + this._culled = false; + this._framedFor = null; + //The transform the page saves to. It starts as the one being edited + //and can be changed, which saves the alignment as a new transform. + this._xfmName = config.xfmname; + this._savedXfm = this.xfm.clone(); + this._savedName = this._xfmName; + this._dirtyShown = false; + this._saveState = null; + this._title = document.title; + this._cmapName = config.cmap; + this._showSurf = [true, true]; + this._translateStep = 1; + this._rotateStep = 1; + this.hasWM = true; + this.hoverView = null; + this._drag = null; + this._wheelAcc = 0; + + //Renderer and scene. Objects are drawn in scene order (planes, then + //the surfaces, then the crosshairs): the slice views draw their plane + //without a depth test so that the surface outline always shows on top + this.renderer = new THREE.WebGLRenderer({ + canvas: this.canvas[0], + antialias: true, + preserveDrawingBuffer: true, + alpha: false, + }); + this.renderer.setClearColor(new THREE.Color(0x000000), 1); + this.renderer.sortObjects = false; + + this.scene = new THREE.Scene(); + this.light = new THREE.DirectionalLight(0xffffff, 1.0); + this.scene.add(this.light); + this.scene.add(this.light.target); + + this.planeGroup = new THREE.Object3D(); + this.brain = new THREE.Object3D(); + this.brain.matrixAutoUpdate = false; + this.brain.matrix.copy(this.xfm); + this.brain.matrixWorldNeedsUpdate = true; + this.crossGroup = new THREE.Object3D(); + //The data view holds the surfaces in the anatomy's own frame, so that + //they keep their place while the alignment is edited and while they + //inflate; the volume is sampled through the alignment instead. + this.surfGroup = new THREE.Object3D(); + this.scene.add(this.planeGroup); + this.scene.add(this.brain); + this.scene.add(this.crossGroup); + this.scene.add(this.surfGroup); + + this.camera3d = new THREE.PerspectiveCamera(45, 1, 1, 4000); + this.camera3d.up.set(0, 0, 1); + this.controls = new jsplot.LandscapeControls(); + this.controls.addEventListener("change", this.schedule.bind(this)); + + this.views = {}; + this.viewlist = []; + for (var i = 0; i < LAYOUT.length; i++) { + var name = LAYOUT[i].name; + var div = $(this.object).find("#view-" + name)[0]; + var view = { + name: name, + div: div, + label: $(div).find(".aligner-label")[0], + col: LAYOUT[i].col, + row: LAYOUT[i].row, + rect: {left: 0, top: 0, width: 1, height: 1}, + is2d: name != "3d", + }; + if (view.is2d) { + var spec = VIEWS[name]; + view.axis = spec.axis; + view.title = spec.title; + view.look = new THREE.Vector3().fromArray(spec.look); + view.up = new THREE.Vector3().fromArray(spec.up); + view.right = new THREE.Vector3().crossVectors(view.look, view.up); + view.camera = new THREE.OrthographicCamera(-1, 1, 1, -1, 1, 4000); + view.center = new THREE.Vector3(); + view.height = 200; + view.fitted = false; + } else { + view.camera = this.camera3d; + } + this.views[name] = view; + this.viewlist.push(view); + this._bindView(view); + //so that the keys move the mesh before the mouse has been over + //any view, rather than doing nothing for no visible reason + if (view.is2d && this.hoverView === null) + this.hoverView = view; + } + window.addEventListener("mousemove", this._onMouseMove.bind(this), false); + window.addEventListener("mouseup", this._onMouseUp.bind(this), false); + window.addEventListener("keydown", this._onKeyDown.bind(this), true); + window.addEventListener("beforeunload", this._onBeforeUnload.bind(this), false); + + //Uniforms shared by the planes and the painted surface + this.volUniforms = { + data: {type:'tv', value:[null, null, null, null]}, + mosaic: {type:'v2v', value:[new THREE.Vector2(1, 1), new THREE.Vector2(1, 1)]}, + dshape: {type:'v2v', value:[new THREE.Vector2(1, 1), new THREE.Vector2(1, 1)]}, + nslices: {type:'f', value:1}, + volxfm: {type:'m4', value:this.worldInv}, + colormap: {type:'t', value:null}, + vmin: {type:'f', value:config.vmin}, + vmax: {type:'f', value:config.vmax}, + brightness: {type:'f', value:0}, + contrast: {type:'f', value:1}, + gamma: {type:'f', value:1}, + flip: {type:'i', value:0}, + outside: {type:'v3', value:new THREE.Vector3(.25, .25, .25)}, + }; + //Uniforms shared by the surface outlines + this.meshUniforms = { + color: {type:'c', value:new THREE.Color(config.mesh_color)}, + opacity: {type:'f', value:config.mesh_opacity}, + slabLo: {type:'v3', value:new THREE.Vector3()}, + slabHi: {type:'v3', value:new THREE.Vector3()}, + slabMask: {type:'v3', value:new THREE.Vector3(1, 1, 1)}, + }; + this.planeMaterials = { + flat: this._makeVolumeMaterial({lights: false, depthTest: false}), + solid: this._makeVolumeMaterial({lights: false, depthTest: true}), + }; + //Where through the cortex the surface that carries the data is + //taken, from the pial surface (0) to the white matter (1) + this.depthUniform = {type:'f', value: 0.5}; + this.outlineMaterials = [this._makeMeshMaterial(0, true), this._makeMeshMaterial(1, true)]; + this.surfaceMaterials = [this._makeMeshMaterial(0, false), this._makeMeshMaterial(1, false)]; + + this.planes2d = []; + this.planes3d = []; + this.planeGeoms = []; + this.crosshairs = []; + this.hemis = []; + + //Control panel on the right, holding the menu, the save status and + //the list of controls + this.ui = new jsplot.Menu(); + this.ui.addEventListener("update", this.schedule.bind(this)); + this._setupPanel(figure); + + //Colormap textures, and the menu once the colormap images are ready + //(the 1D colormaps are told apart from the 2D ones by their height) + this.colormaps = {}; + var images = $(this.object).find(".cmap img").toArray(); + whenImagesLoaded(images).done(function() { + for (var i = 0; i < images.length; i++) { + var img = images[i]; + var tex = new THREE.Texture(img); + tex.minFilter = THREE.LinearFilter; + tex.magFilter = THREE.LinearFilter; + tex.flipY = true; + tex.needsUpdate = true; + this.colormaps[img.parentNode.id] = tex; + if (typeof(colormaps) !== "undefined") + colormaps[img.parentNode.id] = tex; + } + this._buildUI(); + }.bind(this)); + + //The reference volume, through the viewer's mosaic loader + this.volume = new dataset.VolumeData(config.volume, config.images); + this.volume.loaded.done(this._volumeReady.bind(this)); + + //The surfaces, from the CTM pack the viewer uses: the base positions + //are the pial surface and the `wm` attribute the white matter + var loader = new THREE.CTMLoader(false); + loader.loadParts(config.ctm, function(geometries, materials, json) { + this._meshReady(geometries, json); + }.bind(this), {useWorker: true}); + }; + module.Aligner.prototype = Object.create(jsplot.Axes.prototype); + module.Aligner.prototype.constructor = module.Aligner; + + module.Aligner.prototype._setupPanel = function(figure) { + var wrapper = document.createElement("div"); + wrapper.innerHTML = $("#aligner_panel_html").html(); + var panel = wrapper.querySelector("#aligner-panel"); + $(panel).find("#aligner-controls")[0].appendChild(figure.ui_element); + if (this.config.view_only) + $(panel).find("#aligner-viewonly").show(); + //The masks of this transform were cut through the alignment being + //edited, so saving deletes them + var masks = this.config.masks || []; + if (masks.length > 0 && !this.config.view_only) { + $(panel).find("#aligner-masks").text( + "Saving over " + this.config.xfmname + " deletes its " + masks.length + + " cached mask" + (masks.length == 1 ? "" : "s") + " (" + masks.join(", ") + + "). Data already masked with them has to be masked again from the " + + "volumes. Saving under another name leaves them alone.").show(); + } + this.statusElement = $(panel).find("#aligner-status"); + this.historyElement = $(panel).find("#aligner-history-list")[0]; + this._renderHistory(); + figure.gui.open(); + try { + figure.gui.width = 284; + } catch (e) {} + figure.w2obj.show("right", true); + figure.setSize("right", 300); + figure.w2obj.content("right", panel); + }; + + module.Aligner.prototype.showStatus = function(message, error) { + this.statusElement.text(message); + this.statusElement.toggleClass("error", !!error); + }; + + module.Aligner.prototype._makeVolumeMaterial = function(opts) { + var shaders = Shaders.aligner_volume({ + sampler: "nearest", + lights: opts.lights, + depthmix: opts.depthmix, + morphs: opts.morphs, + }); + var uniforms = THREE.UniformsUtils.merge([ + THREE.UniformsLib["lights"], + { + diffuse: {type:'v3', value:new THREE.Vector3(.7, .7, .7)}, + specular: {type:'v3', value:new THREE.Vector3(0, 0, 0)}, + emissive: {type:'v3', value:new THREE.Vector3(.35, .35, .35)}, + shininess: {type:'f', value:1}, + specularStrength: {type:'f', value:0}, + depth: {type:'f', value:0.5}, + } + ]); + for (var name in this.volUniforms) + uniforms[name] = this.volUniforms[name]; + var depth = opts.depthTest !== false; + return new THREE.ShaderMaterial({ + vertexShader: shaders.vertex, + fragmentShader: shaders.fragment, + uniforms: uniforms, + attributes: shaders.attrs, + lights: !!opts.lights, + side: opts.depthmix ? THREE.FrontSide : THREE.DoubleSide, + depthTest: depth, + depthWrite: depth, + }); + }; + + module.Aligner.prototype._makeMeshMaterial = function(depth, clip) { + var shaders = Shaders.aligner_mesh({}); + var uniforms = { + color: this.meshUniforms.color, + slabLo: this.meshUniforms.slabLo, + slabHi: this.meshUniforms.slabHi, + slabMask: this.meshUniforms.slabMask, + depth: {type:'f', value:depth}, + doClip: {type:'i', value:clip ? 1 : 0}, + opacity: clip ? {type:'f', value:1} : this.meshUniforms.opacity, + }; + return new THREE.ShaderMaterial({ + vertexShader: shaders.vertex, + fragmentShader: shaders.fragment, + uniforms: uniforms, + attributes: shaders.attrs, + transparent: !clip, + depthWrite: clip, + side: THREE.DoubleSide, + }); + }; + + //------------------------------------------------------------------------- + // Loading + //------------------------------------------------------------------------- + module.Aligner.prototype._volumeReady = function() { + var vol = this.volume; + this.volUniforms.data.value[0] = vol.textures[0]; + for (var i = 0; i < 2; i++) { + this.volUniforms.mosaic.value[i].set(vol.mosaic[0], vol.mosaic[1]); + this.volUniforms.dshape.value[i].set(vol.shape[0], vol.shape[1]); + } + this.volUniforms.nslices.value = vol.numslices; + + //World bounding box of the volume + var min = new THREE.Vector3(Infinity, Infinity, Infinity); + var max = new THREE.Vector3(-Infinity, -Infinity, -Infinity); + for (var c = 0; c < 8; c++) { + var corner = new THREE.Vector3( + (c & 1) ? this.dims[0] - 0.5 : -0.5, + (c & 2) ? this.dims[1] - 0.5 : -0.5, + (c & 4) ? this.dims[2] - 0.5 : -0.5).applyMatrix4(this.world); + min.min(corner); + max.max(corner); + } + this.bbox = {min: min, max: max}; + this.bboxCenter = min.clone().add(max).multiplyScalar(0.5); + + this._buildPlanes(); + this._updateSliceGeometry(); + + this.controls.setTarget(this.bboxCenter.toArray()); + this.controls.setRadius(1.1 * min.distanceTo(max)); + + this._ready.volume = true; + this._checkReady(); + }; + + module.Aligner.prototype._buildPlanes = function() { + var cross_material = new THREE.LineBasicMaterial({color: 0x44ccff, depthTest: false, depthWrite: false}); + var makeQuad = function() { + var geom = new THREE.Geometry(); + for (var i = 0; i < 4; i++) + geom.vertices.push(new THREE.Vector3()); + geom.faces.push(new THREE.Face3(0, 1, 2), new THREE.Face3(2, 1, 3)); + geom.dynamic = true; + return geom; + }; + for (var a = 0; a < 3; a++) { + //One quad for the slice views (drawn without depth test) and one + //for the 3D view (drawn with it), so that each mesh owns its + //geometry buffers + var geoms = [makeQuad(), makeQuad()]; + this.planeGeoms.push(geoms); + + var flat = new THREE.Mesh(geoms[0], this.planeMaterials.flat); + flat.frustumCulled = false; + this.planes2d.push(flat); + this.planeGroup.add(flat); + + var solid = new THREE.Mesh(geoms[1], this.planeMaterials.solid); + solid.frustumCulled = false; + this.planes3d.push(solid); + this.planeGroup.add(solid); + + var cross = new THREE.Geometry(); + for (var i = 0; i < 4; i++) + cross.vertices.push(new THREE.Vector3()); + cross.dynamic = true; + var lines = new THREE.Line(cross, cross_material, THREE.LinePieces); + lines.frustumCulled = false; + this.crosshairs.push(lines); + this.crossGroup.add(lines); + } + }; + + module.Aligner.prototype._meshReady = function(geometries, json) { + //The surfaces the data view morphs between: the anatomical surface, + //whatever the CTM pack carries (the inflated one) and the flatmap. + //Only a surface every hemisphere has can be morphed to. + var names = []; + var packed = (json && json.names) || []; + for (var k = 0; k < packed.length; k++) { + var everywhere = geometries.length > 0; + for (var i = 0; i < geometries.length; i++) + everywhere = everywhere && geometries[i].attributes[packed[k]] !== undefined; + if (everywhere) + names.push(packed[k]); + } + var hasFlat = !!(json && json.flatlims) && geometries.length > 1 && + geometries[0].attributes.uv !== undefined; + this.surfNames = ["anatomical"].concat(names); + if (hasFlat) + this.surfNames.push("flat"); + + for (var i = 0; i < geometries.length; i++) + geometries[i].computeBoundingBox(); + //The halves of the flatmap unfold from either side of this offset, + //as mriview lays them out + var offx = 0, offy = Infinity; + for (var i = 0; i < geometries.length; i++) { + var box = geometries[i].boundingBox; + offx = Math.max(offx, Math.abs(box.min.x), Math.abs(box.max.x)); + offy = Math.min(offy, box.min.y); + } + this.flatoff = [offx / 3, offy]; + this._buildMorphMaterial(this.surfNames.length); + + for (var i = 0; i < geometries.length; i++) { + var geom = geometries[i]; + if (geom.attributes.wm === undefined) { + //Only a fiducial surface: draw it once, as both surfaces + this.hasWM = false; + geom.addAttribute("wm", geom.attributes.position); + } + geom.addAttribute("wmnorm", mriview.computeNormal(geom.attributes.wm, geom.attributes.index, geom.offsets)); + var edges = module.buildEdges(geom); + + var hemi = {geometry: geom, edges: edges, bounds: geom.boundingBox, + outlines: [], surfaces: []}; + for (var s = 0; s < 2; s++) { + var line = new THREE.Line(edges, this.outlineMaterials[s], THREE.LinePieces); + line.frustumCulled = false; + hemi.outlines.push(line); + this.brain.add(line); + + var surf = new THREE.Mesh(geom, this.surfaceMaterials[s]); + surf.frustumCulled = false; + hemi.surfaces.push(surf); + this.brain.add(surf); + } + this._buildMorph(hemi, i, names, hasFlat); + this.hemis.push(hemi); + } + this._updateMorphXfm(); + //an edit is measured by how far this point travels + if (this.hemis.length > 0) { + var box = new THREE.Box3(); + for (var i = 0; i < this.hemis.length; i++) + box.union(this.hemis[i].bounds); + this._meshCenter = box.center(); + } + this._ready.mesh = true; + this._checkReady(); + }; + + //The material of the surface that carries the data, in the corner of the + //four panels and in the data view alike. It paints the volume on the + //surfaces where the alignment being edited puts them in it, while they + //are drawn in the anatomy's own frame, so that editing the alignment + //moves the colors over a surface that stays where it is. + module.Aligner.prototype._buildMorphMaterial = function(morphs) { + this.morphUniforms = {surfmix: {type:'f', value: 0}}; + var material = this._makeVolumeMaterial({lights: true, depthmix: true, morphs: morphs}); + material.uniforms.volxfm = {type:'m4', value: new THREE.Matrix4()}; + material.uniforms.surfmix = this.morphUniforms.surfmix; + material.uniforms.depth = this.depthUniform; + this.morphMaterial = material; + }; + + //The morphing copy of one hemisphere, hung from the pair of pivots that + //swing the flatmap open, as mriview does it + module.Aligner.prototype._buildMorph = function(hemi, index, names, hasFlat) { + var geom = hemi.geometry; + //the shader reads the surfaces as mixSurfs, and needs a normal + //for each of them + for (var k = 0; k < names.length; k++) { + geom.attributes["mixSurfs" + k] = geom.attributes[names[k]]; + geom.addAttribute("mixNorms" + k, mriview.computeNormal( + geom.attributes[names[k]], geom.attributes.index, geom.offsets)); + delete geom.attributes[names[k]]; + } + if (hasFlat) { + var flat = this._makeFlat(geom.attributes.uv.array, index == 1); + geom.addAttribute("mixSurfs" + names.length, new THREE.BufferAttribute(flat.pos, 4)); + geom.addAttribute("mixNorms" + names.length, new THREE.BufferAttribute(flat.norms, 3)); + //the medial wall is not part of the flatmap, so its polygons are + //dropped while the surface is flattened + var culled = mriview._cull_flatmap_vertices( + geom.attributes.index.array, geom.attributes.auxdat.array, geom.offsets); + hemi.culled = {index: new THREE.BufferAttribute(culled.indices, 3), offsets: culled.offsets}; + hemi.full = {index: geom.attributes.index, offsets: geom.offsets}; + } + + var box = hemi.bounds; + var pivots = {back: new THREE.Object3D(), front: new THREE.Object3D()}; + pivots.front.add(pivots.back); + pivots.back.position.y = box.min.y - box.max.y; + pivots.front.position.y = box.max.y - box.min.y + this.flatoff[1]; + var mesh = new THREE.Mesh(geom, this.morphMaterial); + mesh.position.y = -this.flatoff[1]; + mesh.frustumCulled = false; + mesh.visible = false; + pivots.back.add(mesh); + this.surfGroup.add(pivots.front); + hemi.pivots = pivots; + hemi.morph = mesh; + }; + + //The flatmap as a surface to morph to: its 2D coordinates laid in the + //plane that faces the camera when the halves have swung open + module.Aligner.prototype._makeFlat = function(uv, right) { + var count = uv.length / 2; + var flat = new Float32Array(count * 4); + var norms = new Float32Array(count * 3); + for (var i = 0; i < count; i++) { + flat[i*4+1] = FLATSCALE * (right ? uv[i*2] : -uv[i*2]) + this.flatoff[1]; + flat[i*4+2] = FLATSCALE * uv[i*2+1]; + norms[i*3] = right ? 1 : -1; + } + return {pos: flat, norms: norms}; + }; + + //The alignment as the data view uses it: anatomical millimeters to the + //voxels of the reference volume, which is the pycortex coord transform + module.Aligner.prototype._updateMorphXfm = function() { + if (this.morphMaterial === undefined) + return; + this.morphMaterial.uniforms.volxfm.value.multiplyMatrices(this.worldInv, this.xfm); + }; + + module.Aligner.prototype._checkReady = function() { + if (!this._ready.volume || !this._ready.mesh) + return; + $("#dataload").hide(); + $(this.object).find("#aligner-load").hide(); + this.ready = true; + this.resize(); + this.schedule(); + this.loaded.resolve(); + }; + + module.Aligner.prototype._buildUI = function() { + var names = []; + for (var name in this.colormaps) { + if (this.colormaps[name].image.height == 1) + names.push(name); + } + names.sort(); + if (names.indexOf(this._cmapName) < 0) + this._cmapName = names.indexOf("gray") >= 0 ? "gray" : names[0]; + this.volUniforms.colormap.value = this.colormaps[this._cmapName]; + + //the name field sits above the save button, so that the alignment can + //be saved as a new transform + var top = {}; + if (!this.config.view_only) { + top.transform = {action: [this, "setXfmName"]}; + top.save = {action: this.save.bind(this)}; + } + top.undo = {action: this.undo.bind(this)}; + top.display = {action: [this, "setDisplay", DISPLAY_ORDER]}; + this.ui.add(top); + this._updateDirty(); + + var vol = this.config.volume; + this.ui.addFolder("image", false).add({ + colormap: {action: [this, "setColormap", names]}, + flip: {action: [this, "setFlip"]}, + vmin: {action: [this, "setVmin", vol.min, vol.max]}, + vmax: {action: [this, "setVmax", vol.min, vol.max]}, + brightness: {action: [this, "setBrightness", -1, 1, 0.01]}, + contrast: {action: [this, "setContrast", 0, 4, 0.01]}, + gamma: {action: [this, "setGamma", 0.1, 4, 0.01]}, + }); + this.ui.addFolder("mesh", false).add({ + color: {action: [this, "setMeshColor"], color: true}, + opacity: {action: [this, "setMeshOpacity", 0, 1, 0.01]}, + pial: {action: [this, "setShowPial"]}, + white: {action: [this, "setShowWhite"]}, + depth: {action: [this, "setDepth", 0, 1, 0.01]}, + unfold: {action: [this, "setMix", 0, 1, 0.01]}, + pivot: {action: [this, "setPivot", -180, 180, 1]}, + }); + + var slices = {}; + var titles = ["sagittal", "coronal", "axial"]; + for (var a = 0; a < 3; a++) + slices[titles[a]] = {action: [this, SLICE_SETTERS[a], 0, this.dims[this.vax[a]] - 1, 1]}; + this.ui.addFolder("slices", false).add(slices); + + this.ui.addFolder("steps", true).add({ + "translate (mm)": {action: [this, "setTranslateStep", 0.05, 10, 0.05]}, + "rotate (deg)": {action: [this, "setRotateStep", 0.05, 10, 0.05]}, + }); + this._previewColormaps(); + this.schedule(); + }; + + //Draws a strip of each colormap beside its name in the colormap dropdown. + //dat.GUI renders a plain select, so select2 takes it over to draw the + //options, the same library the viewer's colormap picker uses. Picking one + //writes the value into the select and fires a jQuery event; dat.GUI + //listens for the native one, so the pick is passed on as such and the + //control goes on working the way the others do. + module.Aligner.prototype._previewColormaps = function() { + var folder = this.ui["image"]; + var control = folder === undefined ? undefined : folder._controls.colormap; + if (control === undefined || control.__select === undefined || $.fn.select2 === undefined) + return; + + var select = control.__select; + var colormaps = this.colormaps; + var draw = function(state) { + if (!state.id) + return state.text; + var texture = colormaps[state.id]; + var row = $(""); + if (texture !== undefined && texture.image !== undefined) + row.append($("").attr("src", texture.image.src)); + row.append($("").text(state.text)); + return row; + }; + + this._cmapSelect = $(select).select2({ + templateResult: draw, + templateSelection: draw, + width: "100%", + }); + //The open dropdown hangs off the body rather than off the control, + //and the width mriview.css gives it there reaches past the right + //edge of the window, which scrolls the page sideways. This marks it + //as the aligner's, for the width aligner.css gives it instead. + var instance = this._cmapSelect.data("select2"); + if (instance != null && instance.$dropdown !== undefined) { + //select2 hangs the dropdown in a container of its own when it + //attaches it to the body, so the list itself is a level down + var list = instance.$dropdown.hasClass("select2-dropdown") + ? instance.$dropdown : instance.$dropdown.find(".select2-dropdown"); + list.addClass("aligner-cmap-dropdown"); + } + this._cmapSelect.on("select2:select", function() { + select.dispatchEvent(new Event("change")); + }); + }; + + //Updates the value shown by a menu control without running its action + module.Aligner.prototype._syncControl = function(folder, name, value) { + var menu = this.ui[folder]; + if (menu === undefined || menu._controls[name] === undefined) + return; + menu[name] = value; + menu._controls[name].updateDisplay(); + }; + + //------------------------------------------------------------------------- + // Slices and cursor + //------------------------------------------------------------------------- + module.Aligner.prototype.getSlice = function(axis) { + var va = this.vax[axis]; + return Math.min(Math.max(Math.round(this.cursor[va]), 0), this.dims[va] - 1); + }; + module.Aligner.prototype._setSlice = function(axis, slice) { + var va = this.vax[axis]; + this.cursor[va] = Math.min(Math.max(Math.round(slice), 0), this.dims[va] - 1); + this._cursorChanged(); + }; + module.Aligner.prototype.setSagittal = function(slice) { + if (slice === undefined) + return this.getSlice(0); + this._setSlice(0, slice); + }; + module.Aligner.prototype.setCoronal = function(slice) { + if (slice === undefined) + return this.getSlice(1); + this._setSlice(1, slice); + }; + module.Aligner.prototype.setAxial = function(slice) { + if (slice === undefined) + return this.getSlice(2); + this._setSlice(2, slice); + }; + + //The cursor in voxel coordinates of the reference volume + module.Aligner.prototype.getCursor = function() { + return this.cursor.slice(); + }; + module.Aligner.prototype.setCursor = function(voxel) { + for (var i = 0; i < 3; i++) + this.cursor[i] = Math.min(Math.max(voxel[i], -0.5), this.dims[i] - 0.5); + this._cursorChanged(); + }; + //The cursor in world coordinates + module.Aligner.prototype.cursorWorld = function() { + return new THREE.Vector3().fromArray(this.cursor).applyMatrix4(this.world); + }; + + module.Aligner.prototype._cursorChanged = function() { + var titles = ["sagittal", "coronal", "axial"]; + for (var a = 0; a < 3; a++) + this._syncControl("slices", titles[a], this.getSlice(a)); + if (this._ready.volume) + this._updateSliceGeometry(); + this.schedule(); + }; + + //Moves the slice planes to the cursor's slices, the slabs that cut the + //surfaces to half a voxel around them, and the crosshairs to the cursor + module.Aligner.prototype._updateSliceGeometry = function() { + var e = this.world.elements; + var cw = this.cursorWorld(); + for (var a = 0; a < 3; a++) { + var va = this.vax[a]; + var slice = this.getSlice(a); + var b = (a + 1) % 3, c = (a + 2) % 3; + var vb = this.vax[b], vc = this.vax[c]; + + var lo = new THREE.Vector3(Infinity, Infinity, Infinity); + var hi = new THREE.Vector3(-Infinity, -Infinity, -Infinity); + for (var i = 0; i < 4; i++) { + var voxel = new THREE.Vector3(); + voxel.setComponent(va, slice); + voxel.setComponent(vb, (i & 1) ? this.dims[vb] - 0.5 : -0.5); + voxel.setComponent(vc, (i & 2) ? this.dims[vc] - 0.5 : -0.5); + voxel.applyMatrix4(this.world); + lo.min(voxel); + hi.max(voxel); + for (var g = 0; g < 2; g++) + this.planeGeoms[a][g].vertices[i].copy(voxel); + } + for (var g = 0; g < 2; g++) { + var geom = this.planeGeoms[a][g]; + geom.verticesNeedUpdate = true; + geom.computeFaceNormals(); + geom.computeVertexNormals(); + geom.normalsNeedUpdate = true; + } + + var coord = e[a + 4 * va] * slice + e[a + 12]; + this.planeCoord[a] = coord; + this.meshUniforms.slabLo.value.setComponent(a, coord - 0.5 * this.vscale[a]); + this.meshUniforms.slabHi.value.setComponent(a, coord + 0.5 * this.vscale[a]); + + //Crosshair: one line along each in-plane axis through the cursor + var cross = this.crosshairs[a].geometry; + for (var i = 0; i < 4; i++) { + cross.vertices[i].copy(cw); + cross.vertices[i].setComponent(a, coord); + } + cross.vertices[0].setComponent(b, lo.getComponent(b)); + cross.vertices[1].setComponent(b, hi.getComponent(b)); + cross.vertices[2].setComponent(c, lo.getComponent(c)); + cross.vertices[3].setComponent(c, hi.getComponent(c)); + cross.verticesNeedUpdate = true; + + var view = this.views["xyz"[a]]; + view.label.textContent = view.title + " " + (slice + 1) + " / " + this.dims[va]; + } + }; + + //------------------------------------------------------------------------- + // The transform + //------------------------------------------------------------------------- + //The pycortex coord transform (anatomical mm -> voxel indices) as a nested + //4x4 array, and its setter + module.Aligner.prototype.getXfm = function() { + var coord = new THREE.Matrix4().multiplyMatrices(this.worldInv, this.xfm); + return module.matrixToRows(coord); + }; + module.Aligner.prototype.setXfm = function(rows) { + this.beginEdit("set"); + this.xfm.multiplyMatrices(this.world, module.matrixFromRows(rows)); + this._xfmChanged(); + }; + + //Opens an entry in the history for the edit that is about to be made. A + //drag holds one open while it runs, so that the entry grows with it + //rather than leaving one behind per frame. An edit made after stepping + //back drops the entries that followed, as an editor does. + module.Aligner.prototype.beginEdit = function(kind) { + this._history.length = this._historyIndex + 1; + this._history.push({kind: kind || "edit", xfm: this.xfm.clone(), label: ""}); + //the alignment the page opened with stays as the first entry, so it + //is the oldest edit that gives way once the history is full + while (this._history.length > UNDO_LIMIT) + this._history.splice(1, 1); + this._historyIndex = this._history.length - 1; + this._describeEdit(this._historyIndex); + this._renderHistory(); + }; + module.Aligner.prototype.undo = function() { + if (this._historyIndex == 0) { + this.showStatus("nothing to undo"); + return; + } + this.selectHistory(this._historyIndex - 1); + }; + + //The history, as one entry per edit: what the edit was and how far it + //moved and turned the surfaces + module.Aligner.prototype.getHistory = function() { + var out = []; + for (var i = 0; i < this._history.length; i++) + out.push({kind: this._history[i].kind, label: this._history[i].label}); + return out; + }; + //Puts back the alignment as it stood at one entry of the history + module.Aligner.prototype.selectHistory = function(index) { + if (index === undefined) + return this._historyIndex; + index = Math.min(Math.max(Math.round(index), 0), this._history.length - 1); + this._historyIndex = index; + //the entries are what was, so going back writes none of them + this._restoring = true; + this.xfm.copy(this._history[index].xfm); + this._xfmChanged(); + this._restoring = false; + }; + + //A gesture that ended where it started left no edit, so it leaves no + //entry either + module.Aligner.prototype._dropEmptyEdit = function() { + var i = this._historyIndex; + if (i == 0 || i != this._history.length - 1) + return; + var now = this._history[i].xfm.elements, before = this._history[i - 1].xfm.elements; + for (var k = 0; k < 16; k++) { + if (now[k] !== before[k]) + return; + } + this._history.pop(); + this._historyIndex = i - 1; + this._renderHistory(); + }; + + //What one edit did, in the terms it was made in: where the surfaces went + //along the anatomical axes, and how far and which way round they turned + //in the plane of the view it was made in. + module.Aligner.prototype._describeEdit = function(index) { + var entry = this._history[index]; + if (index == 0) { + entry.label = ""; + return; + } + var from = this._history[index - 1].xfm, to = entry.xfm; + var center = this._meshCenter; + var moved = center.clone().applyMatrix4(to).sub(center.clone().applyMatrix4(from)); + var relative = new THREE.Matrix4().multiplyMatrices( + to, new THREE.Matrix4().getInverse(from)); + var parts = []; + //a rotation carries its pivot's own displacement, which is not + //anything the user asked for, so it is left out of that label + if (entry.kind != "rotate") + parts = parts.concat(this._describeMove(moved)); + if (entry.kind != "translate") { + var turn = this._describeTurn(relative); + if (turn !== null) + parts.push(turn); + } + entry.label = parts.join(", "); + }; + + //A movement along the world axes, as the directions they point in + module.Aligner.prototype._describeMove = function(vector) { + var parts = []; + for (var a = 0; a < 3; a++) { + var step = vector.getComponent(a); + if (Math.abs(step) >= 0.005) + parts.push(Math.abs(step).toFixed(2) + " mm " + DIRECTIONS[a][step > 0 ? 1 : 0]); + } + return parts; + }; + + //A rotation as it looks in the view of the plane it turns in: the view + //faces one end of the axis, and a turn that way round is counterclockwise + //on its screen. + module.Aligner.prototype._describeTurn = function(relative) { + var e = relative.elements; + var at = function(row, col) { return e[row + 4 * col]; }; + var trace = Math.min(Math.max((at(0, 0) + at(1, 1) + at(2, 2) - 1) / 2, -1), 1); + var angle = Math.acos(trace) * 180 / Math.PI; + if (angle < 0.005) + return null; + //the skew part of a rotation matrix is its axis, scaled by the sine + //of the angle + var axis = new THREE.Vector3(at(2, 1) - at(1, 2), + at(0, 2) - at(2, 0), + at(1, 0) - at(0, 1)); + if (axis.length() < 1e-9) + return angle.toFixed(2) + "°"; + var view = this.views["xyz"[this._majorAxis(axis)]]; + var facing = view.look.clone().negate(); + var way = axis.dot(facing) >= 0 ? "CCW" : "CW"; + return angle.toFixed(2) + "° " + way + " in " + view.title; + }; + + //Draws the history in the panel, newest first. Only the entry being + //edited changes while a drag runs, so the rows are built again only when + //there are different ones to show. + module.Aligner.prototype._renderHistory = function() { + var list = this.historyElement; + if (list === undefined || list === null) + return; + if (this._historyRows.length != this._history.length) { + list.innerHTML = ""; + this._historyRows = []; + for (var i = 0; i < this._history.length; i++) + this._historyRows.push(document.createElement("li")); + //the newest edit is the one to read first, so the list runs the + //other way round from the history it draws + for (var i = this._history.length - 1; i >= 0; i--) { + var row = this._historyRows[i]; + row.appendChild(document.createElement("span")).className = "aligner-history-kind"; + row.appendChild(document.createElement("span")).className = "aligner-history-size"; + row.addEventListener("click", this.selectHistory.bind(this, i), false); + list.appendChild(row); + } + } + for (var i = 0; i < this._history.length; i++) { + var row = this._historyRows[i]; + var entry = this._history[i]; + var kind = i == 0 ? "loaded" : entry.kind; + if (row.firstChild.textContent != kind) + row.firstChild.textContent = kind; + if (row.lastChild.textContent != entry.label) + row.lastChild.textContent = entry.label; + row.className = i == this._historyIndex ? "current" : ""; + } + //keep the entry in view inside the list, rather than scrolling the + //whole panel down to it and taking the controls off the screen + if (this._historyRows.length > 0) { + var row = this._historyRows[this._historyIndex]; + var top = row.offsetTop, bottom = top + row.offsetHeight; + if (top < list.scrollTop) + list.scrollTop = top; + else if (bottom > list.scrollTop + list.clientHeight) + list.scrollTop = bottom - list.clientHeight; + } + }; + + module.Aligner.prototype._xfmChanged = function() { + this.brain.matrix.copy(this.xfm); + this.brain.matrixWorldNeedsUpdate = true; + this._updateMorphXfm(); + //the open entry of the history follows the edit that is being made + if (!this._restoring && this._historyIndex > 0) { + this._history[this._historyIndex].xfm.copy(this.xfm); + this._describeEdit(this._historyIndex); + } + this._renderHistory(); + this._updateDirty(); + this.dispatchEvent({type: "xfm"}); + this.schedule(); + }; + + //The name of the transform the page saves to. Changing it saves the + //alignment as a new transform, leaving the one it was loaded from alone. + module.Aligner.prototype.setXfmName = function(name) { + if (name === undefined) + return this._xfmName; + this._xfmName = String(name).trim(); + this._updateDirty(); + }; + + //Whether the alignment on screen is the one that was last saved. An undo + //back to that alignment counts as saved again, which is why this compares + //the matrices rather than merely noting that something was moved. + module.Aligner.prototype.isDirty = function() { + if (this._xfmName !== this._savedName) + return true; + var now = this.xfm.elements, saved = this._savedXfm.elements; + for (var i = 0; i < 16; i++) { + if (now[i] !== saved[i]) + return true; + } + return false; + }; + + //Marks unsaved changes with an asterisk, on the save button and in the + //title, so that a page left open does not look like a saved alignment. + module.Aligner.prototype._updateDirty = function() { + var dirty = this.isDirty(); + if (dirty === this._dirtyShown) + return; + this._dirtyShown = dirty; + document.title = (dirty ? "* " : "") + this._title; + var button = this.ui._controls.save; + if (button !== undefined) + button.name(dirty ? "save *" : "save"); + }; + + //Translates the surfaces by a world vector (mm) + module.Aligner.prototype.translate = function(vector) { + this.beginEdit("translate"); + this._translate(new THREE.Vector3().fromArray(vector)); + }; + module.Aligner.prototype._translate = function(vector) { + var trans = new THREE.Matrix4().makeTranslation(vector.x, vector.y, vector.z); + this.xfm.multiplyMatrices(trans, this.xfm); + this._xfmChanged(); + }; + //Rotates the surfaces by `angle` degrees about a world axis through the + //cursor (or through `pivot`, a world point) + module.Aligner.prototype.rotate = function(axis, angle, pivot) { + this.beginEdit("rotate"); + this._rotate(new THREE.Vector3().fromArray(axis), angle * Math.PI / 180, + pivot === undefined ? this.cursorWorld() : new THREE.Vector3().fromArray(pivot)); + }; + module.Aligner.prototype._rotate = function(axis, radians, pivot) { + var rot = new THREE.Matrix4().makeRotationAxis(axis.clone().normalize(), radians); + var toPivot = new THREE.Matrix4().makeTranslation(pivot.x, pivot.y, pivot.z); + var fromPivot = new THREE.Matrix4().makeTranslation(-pivot.x, -pivot.y, -pivot.z); + var xfm = new THREE.Matrix4().multiplyMatrices(fromPivot, this.xfm); + xfm.multiplyMatrices(rot, xfm); + this.xfm.multiplyMatrices(toPivot, xfm); + this._xfmChanged(); + }; + + module.Aligner.prototype.save = function() { + if (this.config.view_only) { + var refused = "view only: the transform is not saved"; + this.showStatus(refused, true); + this._saveState = {status: "error", message: refused, name: this._xfmName}; + return "view only"; + } + //what is being saved, rather than what is on screen when the answer + //comes back: the alignment can be moved on while the request is out + var name = this._xfmName; + var saved = this.xfm.clone(); + this.showStatus("saving " + name + "..."); + //the state the python side reads to know whether the save landed, + //since the request is answered long after this returns + this._saveState = {status: "saving", message: "saving " + name + "...", name: name}; + $.ajax({ + type: "POST", + url: "save", + //the server only takes saves from the page it served, which this + //token is what makes it + data: {xfm: JSON.stringify(this.getXfm()), name: name, + save_token: this.config.save_token}, + dataType: "json", + }).done(function(resp) { + this.showStatus(resp.message, resp.status != "ok"); + this._saveState = {status: resp.status, message: resp.message, name: name}; + if (resp.status == "ok") { + this._savedXfm = saved; + this._savedName = name; + this._updateDirty(); + } + }.bind(this)).fail(function(xhr) { + //an error the server answered with says more than the request + //having failed does + var message = "saving failed: no answer from the server"; + if (xhr && xhr.responseJSON && xhr.responseJSON.message) + message = xhr.responseJSON.message; + this.showStatus(message, true); + this._saveState = {status: "error", message: message, name: name}; + }.bind(this)); + return "saving"; + }; + //What became of the last save: "saving" while the request is out, then + //"ok" or "error" with the message the server answered with + module.Aligner.prototype.getSaveState = function() { + return this._saveState; + }; + + //The canvas as drawn last, as a PNG data url. Drawing is left to the + //scheduled animation frames, so that a slow (software) render cannot hold + //up the reply to the python side; getFrames tells when a frame landed. + module.Aligner.prototype.snapshot = function() { + return this.renderer.domElement.toDataURL("image/png"); + }; + //Number of frames drawn so far + module.Aligner.prototype.getFrames = function() { + return this.nframes; + }; + //Runs a method for the python side and tags the result with the token + //of the request. The websocket gives up on a reply after two seconds, so + //one held up by a busy page (parsing the surfaces, a slow frame) is left + //in the server's queue; the token is how the python side finds it there. + //The reply also carries the number of frames drawn so far and whether a + //redraw is pending, so that the python side can wait for the frame that + //shows the effect of the call before taking a snapshot. + module.Aligner.prototype.call = function(token, name, args) { + var reply = {token: token, frames: this.nframes}; + if (!(this[name] instanceof Function)) { + reply.error = "no method " + name; + } else { + try { + reply.value = this[name].apply(this, args); + } catch (e) { + reply.error = e.message; + } + } + reply.scheduled = this._scheduled; + return reply; + }; + //A menu control by its dotted path, for instance image.vmin or mesh.color + module.Aligner.prototype.getControl = function(name) { + return this.ui.get(name); + }; + module.Aligner.prototype.setControl = function(name, value) { + this.ui.set(name, value); + }; + + //------------------------------------------------------------------------- + // Display settings + //------------------------------------------------------------------------- + module.Aligner.prototype.setColormap = function(name) { + if (name === undefined) + return this._cmapName; + if (this.colormaps[name] === undefined) + return; + this._cmapName = name; + this.volUniforms.colormap.value = this.colormaps[name]; + //keep the dropdown showing the colormap that is in effect when the + //change came from somewhere else, a menu set from python for instance + if (this._cmapSelect !== undefined && this._cmapSelect.val() != name) + this._cmapSelect.val(name).trigger("change.select2"); + this.schedule(); + }; + module.Aligner.prototype.setFlip = function(flip) { + if (flip === undefined) + return this.volUniforms.flip.value == 1; + this.volUniforms.flip.value = flip ? 1 : 0; + this.schedule(); + }; + module.Aligner.prototype.setVmin = function(value) { + if (value === undefined) + return this.volUniforms.vmin.value; + this.volUniforms.vmin.value = value; + this.schedule(); + }; + module.Aligner.prototype.setVmax = function(value) { + if (value === undefined) + return this.volUniforms.vmax.value; + this.volUniforms.vmax.value = value; + this.schedule(); + }; + module.Aligner.prototype.setBrightness = function(value) { + if (value === undefined) + return this.volUniforms.brightness.value; + this.volUniforms.brightness.value = value; + this.schedule(); + }; + module.Aligner.prototype.setContrast = function(value) { + if (value === undefined) + return this.volUniforms.contrast.value; + this.volUniforms.contrast.value = value; + this.schedule(); + }; + module.Aligner.prototype.setGamma = function(value) { + if (value === undefined) + return this.volUniforms.gamma.value; + this.volUniforms.gamma.value = value; + this.schedule(); + }; + module.Aligner.prototype.setMeshColor = function(color) { + if (color === undefined) + return "#" + this.meshUniforms.color.value.getHexString(); + this.meshUniforms.color.value.set(color); + this.schedule(); + }; + //Opacity of the whole surfaces in the 3D view (0 hides them, leaving + //the outlines on the slices) + module.Aligner.prototype.setMeshOpacity = function(value) { + if (value === undefined) + return this.meshUniforms.opacity.value; + this.meshUniforms.opacity.value = value; + this.schedule(); + }; + module.Aligner.prototype.setShowPial = function(show) { + if (show === undefined) + return this._showSurf[0]; + this._showSurf[0] = show; + this.schedule(); + }; + module.Aligner.prototype.setShowWhite = function(show) { + if (show === undefined) + return this._showSurf[1]; + this._showSurf[1] = show; + this.schedule(); + }; + //Where through the cortex the surface that carries the data is taken, + //from the pial surface (0) to the white matter (1). It moves that + //surface as well as the point the volume is read at, so the corner of + //the four panels and the data view show the same cortical depth. + module.Aligner.prototype.setDepth = function(value) { + if (value === undefined) + return this.depthUniform.value; + this.depthUniform.value = value; + this.schedule(); + }; + module.Aligner.prototype.setTranslateStep = function(value) { + if (value === undefined) + return this._translateStep; + this._translateStep = value; + }; + module.Aligner.prototype.setRotateStep = function(value) { + if (value === undefined) + return this._rotateStep; + this._rotateStep = value; + }; + + //------------------------------------------------------------------------- + // Drawing + //------------------------------------------------------------------------- + module.Aligner.prototype.resize = function() { + var w = $(this.object).width(), h = $(this.object).height(); + if (!w || !h) + return; + this.width = w; + this.height = h; + this.renderer.setSize(w, h); + var single = this._display == DISPLAY.surface; + for (var i = 0; i < this.viewlist.length; i++) { + var view = this.viewlist[i]; + if (single) { + //the 3D view takes the window; draw() skips the empty ones + view.rect = view.is2d ? {left: 0, top: 0, width: 0, height: 0} + : {left: 0, top: 0, width: w, height: h}; + } else { + view.rect = { + left: Math.floor(view.col * w / 2), + top: Math.floor(view.row * h / 2), + width: Math.floor(w / 2), + height: Math.floor(h / 2), + }; + } + } + this.schedule(); + }; + + //What the page shows: the three slice views with the planes in the fourth + //panel, the same with the surface there instead, or that surface on its + //own filling the window. The data view keeps every control, so the + //reference data can be looked at on the surface, through the alignment + //being edited, without saving it and opening the viewer. + module.Aligner.prototype.setDisplay = function(name) { + if (name === undefined) + return this._display; + if (DISPLAY_ORDER.indexOf(name) < 0) + return; + this._display = name; + var single = name == DISPLAY.surface; + $(this.object).find("#aligner").toggleClass("single", single); + //the surface stands in the anatomy's own frame, wherever it is drawn, + //and the slice planes in the volume's, so the camera is pointed at + //whichever of them the 3D view is showing + var surface = this._showsSurface(); + this.controls.setMix(surface ? this._flatness() : 0); + this._frameFor(surface ? "anatomical" : "world"); + this._applyIndex(); + this.resize(); + }; + module.Aligner.prototype.toggleDisplay = function() { + var next = (DISPLAY_ORDER.indexOf(this._display) + 1) % DISPLAY_ORDER.length; + this.setDisplay(DISPLAY_ORDER[next]); + }; + + //Whether the 3D view is showing the surface with the data on it, which + //is the one that unfolds. The corner of the four panels and the data + //view draw the same surface, so it carries the same state in both. + module.Aligner.prototype._showsSurface = function() { + return this._display == DISPLAY.brain || this._display == DISPLAY.surface; + }; + + //What one view draws + module.Aligner.prototype._viewMode = function(view) { + if (this._display == DISPLAY.surface) + return MORPHED; + if (this._display == DISPLAY.brain && !view.is2d) + return MORPHED; + return OUTLINE; + }; + + //Points the camera at the surfaces in the frame they are drawn in, and + //only when that frame changes, so that a view the user set up survives a + //trip through the other displays + module.Aligner.prototype._frameFor = function(frame) { + if (this._framedFor == frame) + return; + this._framedFor = frame; + this._frameSurface(frame == "world" ? this.xfm : null); + }; + + //Points the camera at the surface and sits it back far enough to see all + //of it, the way the viewer opens on a surface. The angle it is seen from + //is left as it was. + module.Aligner.prototype._frameSurface = function(matrix) { + if (this.hemis.length == 0) + return; + var min = new THREE.Vector3(Infinity, Infinity, Infinity); + var max = new THREE.Vector3(-Infinity, -Infinity, -Infinity); + for (var i = 0; i < this.hemis.length; i++) { + var box = this.hemis[i].bounds; + if (box === undefined) + continue; + for (var c = 0; c < 8; c++) { + var corner = new THREE.Vector3( + (c & 1) ? box.max.x : box.min.x, + (c & 2) ? box.max.y : box.min.y, + (c & 4) ? box.max.z : box.min.z); + if (matrix) + corner.applyMatrix4(matrix); + min.min(corner); + max.max(corner); + } + } + if (!isFinite(min.x)) + return; + //far enough for the widest side to fit the height of the view: the + //diagonal would be the safe distance for any angle, but a brain seen + //from any of them covers much less than its diagonal + var size = max.clone().sub(min); + var half = Math.max(size.x, size.y, size.z) / 2; + var fov = this.camera3d.fov * Math.PI / 360; + this.controls.setTarget(min.clone().add(max).multiplyScalar(0.5).toArray()); + this.controls.setRadius(1.1 * half / Math.sin(fov)); + }; + + //How far the surface has been unfolded, from the anatomical surface (0) + //through the inflated one to the flatmap (1), as the viewer's `unfold` + module.Aligner.prototype.setMix = function(mix) { + if (mix === undefined) + return this._mix; + this._mix = Math.min(Math.max(mix, 0), 1); + if (this.morphUniforms !== undefined) + this.morphUniforms.surfmix.value = this._mix; + var flat = this._flatness(); + this._applyIndex(); + this.setPivot(180 * flat); + for (var i = 0; i < this.hemis.length; i++) { + if (this.hemis[i].pivots !== undefined) + this.hemis[i].pivots.back.rotation.x = flat * -Math.PI / 2; + } + //A flatmap faces the light head on, which washes the data out, so the + //shading gives way to flat illumination as it opens; the viewer's + //lighting follows the flatmap the same way. + if (this.morphMaterial !== undefined) { + var uniforms = this.morphMaterial.uniforms; + var lit = 0.7 * (1 - flat); + uniforms.diffuse.value.set(lit, lit, lit); + var ambient = 0.35 + 0.65 * flat; + uniforms.emissive.value.set(ambient, ambient, ambient); + } + //the camera swings round to face the flatmap as it opens + if (this._showsSurface()) + this.controls.setMix(flat); + this.schedule(); + }; + + //The unfolding that lands on one of the surfaces the pack carries, or + //null when it has no such surface + module.Aligner.prototype._mixFor = function(name) { + var names = this.surfNames; + if (names === undefined || names.length < 2) + return null; + var index = names.indexOf(name); + return index < 0 ? null : index / (names.length - 1); + }; + + //How much of the flatmap is showing, which is what the medial wall and + //the camera follow: 1 once the surface is all the way unfolded + module.Aligner.prototype._flatness = function() { + if (this.surfNames === undefined || this.surfNames.indexOf("flat") < 0) + return 0; + var last = this.surfNames.length - 1; + return Math.min(Math.max(1 - Math.abs(this._mix * last - last), 0), 1); + }; + + //How far apart the two halves of the surface are swung, in degrees + module.Aligner.prototype.setPivot = function(value) { + if (value === undefined) + return this._pivot; + this._pivot = value; + var radians = value * Math.PI / 360; + for (var i = 0; i < this.hemis.length; i++) { + var pivots = this.hemis[i].pivots; + if (pivots === undefined) + continue; + var sign = i == 0 ? 1 : -1; + pivots.front.rotation.z = value > 0 ? 0 : radians * sign; + pivots.back.rotation.z = value > 0 ? radians * sign : 0; + } + this.schedule(); + }; + + //The medial wall has no place on the flatmap, so its polygons are dropped + //while one is showing. The surfaces of the other displays share these + //geometries, so they are given the whole mesh back on the way out. + module.Aligner.prototype._applyIndex = function() { + var cull = this._showsSurface() && this._flatness() > 0; + if (cull === this._culled) + return; + this._culled = cull; + for (var i = 0; i < this.hemis.length; i++) { + var hemi = this.hemis[i]; + if (hemi.culled === undefined) + continue; + var source = cull ? hemi.culled : hemi.full; + hemi.geometry.attributes.index = source.index; + hemi.geometry.offsets = source.offsets; + } + }; + + module.Aligner.prototype.schedule = function() { + if (!this._scheduled) { + this._scheduled = true; + requestAnimationFrame(this._draw); + } + }; + + module.Aligner.prototype.draw = function() { + this._scheduled = false; + if (!this.ready || !this.width || !this.height) + return; + this.controls.update(this.camera3d); + this.renderer.enableScissorTest(true); + for (var i = 0; i < this.viewlist.length; i++) { + var view = this.viewlist[i], r = view.rect; + if (r.width < 1 || r.height < 1) + continue; + var bottom = this.height - r.top - r.height; + this.renderer.setViewport(r.left, bottom, r.width, r.height); + this.renderer.setScissor(r.left, bottom, r.width, r.height); + this._prepareView(view); + if (view.is2d && this._viewMode(view) == OUTLINE) { + //Two passes: the slice first, then the outlines and the + //crosshair on top of it whatever their depth. Drawing them in + //one pass would leave the order to three.js, which draws its + //list of opaque objects back to front through the scene. + this._showLayer(view, "plane"); + this.renderer.render(this.scene, view.camera); + this._showLayer(view, "lines"); + this.renderer.autoClear = false; + this.renderer.render(this.scene, view.camera); + this.renderer.autoClear = true; + } else { + this._showLayer(view, "all"); + this.renderer.render(this.scene, view.camera); + } + } + this.renderer.enableScissorTest(false); + this.nframes++; + this.dispatchEvent({type: "draw"}); + }; + + //Sets the visibility of the objects for a view: a slice view shows its + //plane, the outline of the surfaces cut to that slice and the crosshair; + //the 3D view shows all three planes, the outlines on all of them and, + //when opaque enough, the whole surfaces. The other two modes show the + //surfaces colored by the volume instead, where the alignment puts them in + //it or in the anatomy's own frame. `layer` restricts the visible objects + //to the "plane" or the "lines" of a slice view. + module.Aligner.prototype._showLayer = function(view, layer) { + var mode = this._viewMode(view); + var outline = mode == OUTLINE; + var plane = layer != "lines", lines = layer != "plane"; + for (var a = 0; a < 3; a++) { + this.planes2d[a].visible = plane && outline && view.is2d && view.axis == a; + this.planes3d[a].visible = plane && outline && !view.is2d; + this.crosshairs[a].visible = lines && outline && view.is2d && view.axis == a; + } + var surfaces = outline && !view.is2d && this.meshUniforms.opacity.value > 0; + for (var i = 0; i < this.hemis.length; i++) { + var hemi = this.hemis[i]; + for (var s = 0; s < 2; s++) { + var shown = this._showSurf[s] && (s == 0 || this.hasWM); + hemi.outlines[s].visible = lines && outline && shown; + hemi.surfaces[s].visible = lines && surfaces && shown; + } + if (hemi.morph !== undefined) + hemi.morph.visible = lines && mode == MORPHED; + } + }; + + //Sets the slabs that cut the surfaces, the camera and the light of a view + module.Aligner.prototype._prepareView = function(view) { + var mask = this.meshUniforms.slabMask.value; + if (view.is2d) + mask.set(view.axis == 0 ? 1 : 0, view.axis == 1 ? 1 : 0, view.axis == 2 ? 1 : 0); + else + mask.set(1, 1, 1); + + if (view.is2d) { + if (!view.fitted) + this._fitView(view); + this._updateOrthoCamera(view); + this.light.position.copy(view.camera.position); + this.light.target.position.copy(view.center); + } else { + this.camera3d.aspect = view.rect.width / view.rect.height; + this.camera3d.updateProjectionMatrix(); + this.light.position.copy(this.camera3d.position); + this.light.target.position.copy(this.controls.target); + } + }; + + //Frames the whole volume in a slice view + module.Aligner.prototype._fitView = function(view) { + var size = this.bbox.max.clone().sub(this.bbox.min); + var upAxis = this._majorAxis(view.up), rightAxis = this._majorAxis(view.right); + var aspect = view.rect.width / view.rect.height; + view.height = 1.1 * Math.max(size.getComponent(upAxis), size.getComponent(rightAxis) / aspect); + view.center.copy(this.bboxCenter); + view.fitted = true; + }; + module.Aligner.prototype._majorAxis = function(vector) { + var arr = vector.toArray(), best = 0; + for (var i = 1; i < 3; i++) + if (Math.abs(arr[i]) > Math.abs(arr[best])) + best = i; + return best; + }; + + module.Aligner.prototype._updateOrthoCamera = function(view) { + var cam = view.camera; + var aspect = view.rect.width / view.rect.height; + var h = view.height, w = h * aspect; + cam.left = -w / 2; + cam.right = w / 2; + cam.top = h / 2; + cam.bottom = -h / 2; + cam.updateProjectionMatrix(); + cam.up.copy(view.up); + cam.position.copy(view.center).sub(view.look.clone().multiplyScalar(2000)); + cam.lookAt(view.center); + cam.updateMatrixWorld(); + }; + + //------------------------------------------------------------------------- + // Interaction + //------------------------------------------------------------------------- + module.Aligner.prototype._bindView = function(view) { + view.div.addEventListener("mouseenter", function() { + this.hoverView = view; + //A control in the panel keeps the keyboard once it has been + //used, and the browser would hand it the keys meant for the mesh. + //Pointing at a view is the aligner's way of saying which view the + //keys act on, so it takes them back here. + var focused = document.activeElement; + if (focused && focused.blur && $(focused).closest("#figure_ui").length > 0) + focused.blur(); + }.bind(this), false); + view.div.addEventListener("contextmenu", function(event) { + event.preventDefault(); + }, false); + if (!view.is2d) { + this.controls.bind(view.div); + return; + } + view.div.addEventListener("mousedown", this._onMouseDown.bind(this, view), false); + view.div.addEventListener("wheel", this._onWheel.bind(this, view), {passive: false}); + }; + + module.Aligner.prototype._localPos = function(view, event) { + var r = view.div.getBoundingClientRect(); + return {x: event.clientX - r.left, y: event.clientY - r.top}; + }; + + //The world point under the mouse in a slice view, on the slice plane + module.Aligner.prototype._mouseWorld = function(view, pos) { + this._updateOrthoCamera(view); + var nx = (pos.x / view.rect.width) * 2 - 1; + var ny = 1 - (pos.y / view.rect.height) * 2; + var point = new THREE.Vector3(nx, ny, 0).unproject(view.camera); + point.setComponent(view.axis, this.planeCoord[view.axis]); + return point; + }; + module.Aligner.prototype._worldToScreen = function(view, point) { + this._updateOrthoCamera(view); + var ndc = point.clone().project(view.camera); + return {x: (ndc.x + 1) / 2 * view.rect.width, y: (1 - ndc.y) / 2 * view.rect.height}; + }; + //Millimeters per screen pixel in a slice view + module.Aligner.prototype._mmPerPixel = function(view) { + return view.height / view.rect.height; + }; + + //Moves the cursor, in the plane of a slice view, to the mouse position + module.Aligner.prototype._setCursorFromMouse = function(view, pos) { + var voxel = this._mouseWorld(view, pos).applyMatrix4(this.worldInv); + for (var a = 0; a < 3; a++) { + if (a == view.axis) + continue; + var va = this.vax[a]; + this.cursor[va] = Math.min(Math.max(voxel.getComponent(va), -0.5), this.dims[va] - 0.5); + } + this._cursorChanged(); + }; + + module.Aligner.prototype._onMouseDown = function(view, event) { + event.preventDefault(); + this.hoverView = view; + var pos = this._localPos(view, event); + var state; + if (event.button === 0 && event.shiftKey) { + state = "pan"; + } else if (event.button === 0) { + state = "cursor"; + this._setCursorFromMouse(view, pos); + } else if (event.button === 1) { + state = "pan"; + } else if (event.button === 2) { + state = (event.ctrlKey || event.altKey || event.metaKey) ? "rotate" : "translate"; + //the whole drag is one edit, so its entry is opened here and + //grows until the button comes back up + this.beginEdit(state); + } else { + return; + } + this._drag = {view: view, state: state, last: pos}; + if (state == "rotate") { + this._drag.pivot = this._worldToScreen(view, this.cursorWorld()); + this._drag.angle = Math.atan2(-(pos.y - this._drag.pivot.y), pos.x - this._drag.pivot.x); + } + }; + + module.Aligner.prototype._onMouseMove = function(event) { + var drag = this._drag; + if (drag === null) + return; + event.preventDefault(); + var view = drag.view; + var pos = this._localPos(view, event); + var dx = pos.x - drag.last.x, dy = pos.y - drag.last.y; + var scale = this._mmPerPixel(view); + if (drag.state == "cursor") { + this._setCursorFromMouse(view, pos); + } else if (drag.state == "pan") { + view.center.sub(view.right.clone().multiplyScalar(dx * scale)); + view.center.add(view.up.clone().multiplyScalar(dy * scale)); + } else if (drag.state == "translate") { + var move = view.right.clone().multiplyScalar(dx * scale); + move.sub(view.up.clone().multiplyScalar(dy * scale)); + this._translate(move); + } else if (drag.state == "rotate") { + var angle = Math.atan2(-(pos.y - drag.pivot.y), pos.x - drag.pivot.x); + var delta = angle - drag.angle; + if (delta > Math.PI) + delta -= 2 * Math.PI; + else if (delta < -Math.PI) + delta += 2 * Math.PI; + drag.angle = angle; + //counterclockwise on the screen is a positive rotation about the + //axis pointing at the viewer + this._rotate(view.look.clone().negate(), delta, this.cursorWorld()); + } + drag.last = pos; + this.schedule(); + }; + + module.Aligner.prototype._onMouseUp = function(event) { + if (this._drag !== null && + (this._drag.state == "translate" || this._drag.state == "rotate")) + this._dropEmptyEdit(); + this._drag = null; + }; + + //Wheel: steps through the slices; with ctrl (or a trackpad pinch) zooms + //the view about the mouse position + module.Aligner.prototype._onWheel = function(view, event) { + event.preventDefault(); + var delta = wheelDelta(event); + if (event.ctrlKey || event.metaKey) { + var pos = this._localPos(view, event); + var before = this._mouseWorld(view, pos); + var factor = Math.exp(Math.min(Math.max(delta, -100), 100) * 0.005); + view.height *= factor; + //keep the point under the mouse where it is + view.center.sub(before).multiplyScalar(factor).add(before); + } else { + this._wheelAcc += delta; + while (Math.abs(this._wheelAcc) >= WHEEL_STEP) { + var step = this._wheelAcc > 0 ? 1 : -1; + this._setSlice(view.axis, this.getSlice(view.axis) + step); + this._wheelAcc -= step * WHEEL_STEP; + } + } + this.schedule(); + }; + + //A page closed with the alignment unsaved loses it, so the browser is + //asked to put the question to the user. Browsers word it themselves, and + //only ask at all once the page has been interacted with. + module.Aligner.prototype._onBeforeUnload = function(event) { + if (this.config.view_only || !this.isDirty()) + return; + event.preventDefault(); + //browsers from before the standard settled read the message off the + //event rather than from the return value + event.returnValue = "The alignment has changes that have not been saved."; + return event.returnValue; + }; + + module.Aligner.prototype._onKeyDown = function(event) { + var tag = event.target.tagName; + if (tag == "INPUT" || tag == "TEXTAREA" || tag == "SELECT") + return; + var key = event.key; + if ((event.ctrlKey || event.metaKey) && (key == "z" || key == "Z")) { + this.undo(); + event.preventDefault(); + return; + } + if (event.ctrlKey || event.metaKey || event.altKey) + return; + if (key == "m" || key == "M") { + this.toggleDisplay(); + event.preventDefault(); + return; + } + //The keys the viewer unfolds a surface with: the anatomical surface, + //the inflated one and the flatmap. They act wherever the surface is + //shown, so the corner of the four panels follows them too. + var unfold = {r: "anatomical", i: "inflated", f: "flat"}[key.toLowerCase()]; + if (unfold !== undefined) { + var mix = this._mixFor(unfold); + //the menu wraps this setter, so the control follows the key + if (mix !== null) + this.setMix(mix); + event.preventDefault(); + return; + } + + var view = this.hoverView; + if (view === null || !view.is2d) + return; + var fine = event.shiftKey ? 0.1 : 1; + var tstep = this._translateStep * fine; + var rstep = this._rotateStep * fine * Math.PI / 180; + var toViewer = view.look.clone().negate(); + var handled = true; + //WASD moves the mesh the same way the arrows do. Both upper and lower + //case, since shift is the fine-step modifier and shift+w arrives as W. + switch (key) { + case "ArrowLeft": case "a": case "A": + this.translate(view.right.clone().multiplyScalar(-tstep).toArray()); + break; + case "ArrowRight": case "d": case "D": + this.translate(view.right.clone().multiplyScalar(tstep).toArray()); + break; + case "ArrowUp": case "w": case "W": + this.translate(view.up.clone().multiplyScalar(tstep).toArray()); + break; + case "ArrowDown": case "s": case "S": + this.translate(view.up.clone().multiplyScalar(-tstep).toArray()); + break; + case "q": case "Q": + this.beginEdit("rotate"); + this._rotate(toViewer, rstep, this.cursorWorld()); + break; + case "e": case "E": + this.beginEdit("rotate"); + this._rotate(toViewer, -rstep, this.cursorWorld()); + break; + case "[": case "{": + this._setSlice(view.axis, this.getSlice(view.axis) - 1); + break; + case "]": case "}": + this._setSlice(view.axis, this.getSlice(view.axis) + 1); + break; + default: + handled = false; + } + if (handled) + event.preventDefault(); + }; + + return module; +}(aligner || {})); diff --git a/cortex/webgl/resources/js/axes3d.js b/cortex/webgl/resources/js/axes3d.js index e78684723..3cc66619f 100644 --- a/cortex/webgl/resources/js/axes3d.js +++ b/cortex/webgl/resources/js/axes3d.js @@ -2,6 +2,13 @@ var jsplot = (function (module) { var retina_scale = window.devicePixelRatio || 1; + //The angle the perspective camera takes in, and the slab an orthographic + //one sees: it has no vanishing point, so near and far only say what is + //clipped, and the slab is deep enough to hold the scene wherever the + //controls put the camera. + var camera_fov = 45; + var camera_slab = 1000; + //Fractional view options (top-left lighting, uniform illumination) used to //be plain booleans, so accept both spellings. module.parseFraction = function(val) { @@ -21,7 +28,7 @@ var jsplot = (function (module) { } // scene and camera - this.camera = new THREE.PerspectiveCamera( 45, this.canvas.width()/this.canvas.height(), 1., 1000. ); + this.camera = new THREE.PerspectiveCamera( camera_fov, this.canvas.width()/this.canvas.height(), 1., 1000. ); this.camera.up.set(0,0,1); this.camera.position.set(0, -500, 0); this.camera.lookAt(new THREE.Vector3(0,0,0)); @@ -122,8 +129,7 @@ var jsplot = (function (module) { this.renderer.domElement.style.width = w + 'px'; this.renderer.domElement.style.height = h + 'px'; - this.camera.aspect = aspect; - this.camera.updateProjectionMatrix(); + this.aimCamera(aspect); this.dispatchEvent({ type:"resize", width:w, height:h}); this.schedule(); @@ -142,6 +148,60 @@ var jsplot = (function (module) { if (this._schedule !== undefined) this.schedule(); }; + //Whether the viewer looks through an orthographic camera rather than the + //perspective one it opens with. Both are aimed by the same azimuth, + //altitude and radius, and the two show the brain at the same size at the + //radius it is switched over at, so the view carries across. + module.Axes3D.prototype.setOrthographic = function(val) { + if (val === undefined) + return this._orthographic === true; + if (this._orthographic === !!val) + return; + this._orthographic = !!val; + + var aspect = this.height ? this.width / this.height + : this.canvas.width() / this.canvas.height(); + var was = this.camera; + var camera = this._orthographic + ? new THREE.OrthographicCamera(-1, 1, 1, -1, -camera_slab, camera_slab) + : new THREE.PerspectiveCamera(camera_fov, aspect, 1., 1000.); + camera.up.copy(was.up); + camera.position.copy(was.position); + camera.quaternion.copy(was.quaternion); + //the lights are children of the camera and are aimed at it, so they + //move over to whichever one is now looking + for (var i = 0; i < this.lights.length; i++) { + was.remove(this.lights[i]); + this.lights[i].target = camera; + camera.add(this.lights[i]); + } + //and the camera itself stands in the scene, because the renderer + //lights a scene with the lights it finds in it + if (was.parent !== undefined && was.parent !== null) { + var scene = was.parent; + scene.remove(was); + scene.add(camera); + } + this.camera = camera; + this.aimCamera(aspect); + this.schedule(); + }; + //What the camera takes in across a viewport of this shape. A perspective + //camera takes the aspect alone; an orthographic one is given the rectangle + //the perspective one covers at the distance the controls hold it at, which + //is what leaves the radius as the zoom. + module.Axes3D.prototype.aimCamera = function(aspect) { + if (this._orthographic) { + var half = this.controls.radius * Math.tan(camera_fov * Math.PI / 360); + this.camera.top = half; + this.camera.bottom = -half; + this.camera.right = half * aspect; + this.camera.left = -half * aspect; + } else { + this.camera.aspect = aspect; + } + this.camera.updateProjectionMatrix(); + }; module.Axes3D.prototype.schedule = function() { if (!this._scheduled) { this._scheduled = true; @@ -160,8 +220,13 @@ var jsplot = (function (module) { } this.controls.update(this.camera); + //An orthographic camera sees everything at the size it is, however far + //off it stands, so the radius the controls hold is the zoom only once + //it is put into the frustum, which the controls have just moved. + if (this._orthographic) + this.aimCamera(this.width / this.height); - var view, left, bottom, width, height; + var view, left, bottom, width, height, camera; if (this.views.length > 1) { for (var i = 0; i < this.views.length; i++) { view = this.views[i]; @@ -174,21 +239,34 @@ var jsplot = (function (module) { this.renderer.setScissor( left, bottom, width, height ); this.renderer.enableScissorTest ( true ); - this.camera.aspect = width / height; - this.camera.updateProjectionMatrix(); - this.drawView(this.views[i].scene, i); + //A view can bring a camera of its own, and say what its + //scene is to show before it is drawn; the rest are drawn + //with the camera the controls move, one after another. + if (view.prepare !== undefined) + view.prepare(width, height); + camera = view.camera === undefined ? this.camera : view.camera; + if (camera === this.camera) + this.aimCamera(width / height); + this.drawView(view.scene, view.surf === undefined ? i : view.surf, camera); + //anything that belongs over the view rather than in it + if (view.overlay !== undefined) + view.overlay(camera); } + //Anything drawn after this, the picker's own passes among them, + //covers the whole canvas: a scissor left on would cut it down to + //the last view of the loop. + this.renderer.enableScissorTest(false); } else if (this.views.length > 0) { this.renderer.enableScissorTest(false); - this.drawView(this.views[0].scene, 0); + this.drawView(this.views[0].scene, 0, this.camera); } this._scheduled = false; this.dispatchEvent({type:"draw"}); //requestAnimationFrame( this._schedule ); }; - module.Axes3D.prototype.drawView = function(scene) { - this.renderer.render(scene, this.camera); + module.Axes3D.prototype.drawView = function(scene, idx, camera, target) { + this.renderer.render(scene, camera === undefined ? this.camera : camera, target); }; module.Axes3D.prototype.animate = function(animation) { var state = {}; @@ -325,7 +403,21 @@ var jsplot = (function (module) { height = width * this.canvas.height() / this.canvas.width(); console.log(width, height); - var renderbuf = new THREE.WebGLRenderTarget(width, height, { + + //A render target carries one sample a pixel, whatever the canvas was + //made with, so an image is antialiased by rendering it larger and + //averaging it back down to the size asked for. + width = Math.round(width); + height = Math.round(height); + var gl = this.renderer.context; + var maxsize = Math.min(gl.getParameter(gl.MAX_TEXTURE_SIZE), + gl.getParameter(gl.MAX_RENDERBUFFER_SIZE)); + var samples = parseInt(this.imageAntialias, 10) || 1; + while (samples > 1 && (width * samples > maxsize || height * samples > maxsize)) + samples--; + var bufwidth = width * samples, bufheight = height * samples; + + var renderbuf = new THREE.WebGLRenderTarget(bufwidth, bufheight, { minFilter: THREE.LinearFilter, magFilter: THREE.LinearFilter, format:THREE.RGBAFormat, @@ -335,17 +427,17 @@ var jsplot = (function (module) { var clearAlpha = this.renderer.getClearAlpha(); var clearColor = this.renderer.getClearColor(); var oldw = this.canvas.width(), oldh = this.canvas.height(); - this.camera.aspect = width / height; - this.camera.updateProjectionMatrix(); - this.renderer.setSize(width, height); + this.aimCamera(width / height); + this.renderer.setSize(bufwidth, bufheight); this.renderer.setClearColor(new THREE.Color(0,0,0), 0); - this.renderer.render(this.views[0].scene, this.camera, renderbuf); + //drawn as a view is, so that whatever a view does before it is drawn + //is done for the image as well + this.drawView(this.views[0].scene, 0, this.camera, renderbuf); this.renderer.setSize(oldw, oldh); this.renderer.setClearColor(new THREE.Color(0,0,0), 1); - this.camera.aspect = oldw / oldh; - this.camera.updateProjectionMatrix(); + this.aimCamera(oldw / oldh); - var img = mriview.getTexture(this.renderer.context, renderbuf) + var img = mriview.getTexture(this.renderer.context, renderbuf, samples); // Read back into `img` now, so free the target: left alone, every call // keeps its framebuffer and texture on the GPU, which over a rendered // movie adds up to a few megabytes a frame. diff --git a/cortex/webgl/resources/js/facepick.js b/cortex/webgl/resources/js/facepick.js index 0747ff576..8b947b0f3 100644 --- a/cortex/webgl/resources/js/facepick.js +++ b/cortex/webgl/resources/js/facepick.js @@ -155,6 +155,10 @@ PickPosition.prototype = { var clearColor = renderer.getClearColor().clone(); renderer.setClearColor(0x0, 0); + //the markers are not part of the surface, so they are kept out of + //the passes that read a vertex off it; whether they are shown at all + //is for the viewer to say, so it is put back afterwards + var shown = {left:this.markers.left.visible, right:this.markers.right.visible}; this.markers.left.visible = false; this.markers.right.visible = false; @@ -167,8 +171,8 @@ PickPosition.prototype = { this.scene.overrideMaterial = this.shade_z; renderer.render(this.scene, camera, this.z); - this.markers.left.visible = true; - this.markers.right.visible = true; + this.markers.left.visible = shown.left; + this.markers.right.visible = shown.right; renderer.setClearColor(clearColor, clearAlpha); }, diff --git a/cortex/webgl/resources/js/menu.js b/cortex/webgl/resources/js/menu.js index 4c4538d9b..14f906dd0 100644 --- a/cortex/webgl/resources/js/menu.js +++ b/cortex/webgl/resources/js/menu.js @@ -1,4 +1,14 @@ var jsplot = (function (module) { + //dat.GUI works out what a click on a checkbox means from the value it + //last set itself, so a change made anywhere else -- a key, a call from + //python -- has to be told to it as well, or the next click on the box + //asks for the value it already has and nothing happens. + function redraw(ctrl) { + ctrl.updateDisplay(); + if (ctrl.__prev !== undefined) + ctrl.__prev = ctrl.getValue(); + } + module.Menu = function(gui) { this._gui = gui; this._desc = {}; @@ -40,7 +50,7 @@ var jsplot = (function (module) { action[0][action[1]](value); } if (this._controls[n]) - this._controls[n].updateDisplay(); + redraw(this._controls[n]); this.dispatchEvent({type:"update"}); } module.Menu.prototype.get = function(name) { @@ -109,11 +119,13 @@ var jsplot = (function (module) { return folder; } module.Menu.prototype._add = function(gui, name, desc) { + var ctrl; if (desc.action instanceof Function) { //A button that runs a function (IE Reset) this[name] = desc.action; - if (!desc.hidden) - gui.add(desc, "action").name(name); + if (!desc.hidden) + //keep the controller, so that the button can be renamed later + ctrl = gui.add(desc, "action").name(name); } else if ( desc.action instanceof Array) { var obj = desc.action[0][desc.action[1]]; if (obj instanceof Function) { @@ -127,7 +139,8 @@ var jsplot = (function (module) { for (var i = 2; i < desc.action.length; i++) newargs.push(desc.action[i]); - var ctrl = gui.add.apply(gui, newargs); + //a color picker, for a method that gets and sets a css color string + ctrl = desc.color ? gui.addColor(this, name) : gui.add.apply(gui, newargs); ctrl.onChange(function(name) { parent[method](this[name]); this.dispatchEvent({type:"update"}); @@ -142,16 +155,31 @@ var jsplot = (function (module) { func(val); this[name] = val; - ctrl.updateDisplay(); + redraw(ctrl); }.bind(this, name) }; } else if (!desc.hidden) { - var ctrl = gui.add.apply(gui, desc.action).name(name); + ctrl = gui.add.apply(gui, desc.action).name(name); ctrl.onChange(function() { this.dispatchEvent({type:"update"}); }.bind(this)); } } + //a switch rather than a checkbox, for the settings that are a state + //of the page rather than one thing being shown or not + if (desc.toggle && ctrl !== undefined && ctrl.__li !== undefined) + ctrl.__li.className += " toggle"; + + //a slider that moves in steps rather than smoothly + if (desc.step !== undefined && ctrl !== undefined && ctrl.step !== undefined) + ctrl.step(desc.step); + + //a control that belongs to the menu only in some state of the page is + //built with the rest, so that it keeps its place among them, and hidden + //until whatever shows it does + if (desc.shown === false && ctrl !== undefined && ctrl.__li !== undefined) + ctrl.__li.style.display = "none"; + //setup keyboard shortcuts for commands if (desc.key) { var key = desc.key; diff --git a/cortex/webgl/resources/js/mriview.js b/cortex/webgl/resources/js/mriview.js index e0961b725..58bb8afa7 100644 --- a/cortex/webgl/resources/js/mriview.js +++ b/cortex/webgl/resources/js/mriview.js @@ -159,6 +159,9 @@ var mriview = (function(module) { this._clipx = false; this._clipy = false; this._clipz = false; + //whether the canvas is split between the three slice views and the + //3D one, rather than the 3D one alone + this._sliceviews = false; this.ui = new jsplot.Menu(); this.ui.addEventListener("update", this.schedule.bind(this)); @@ -174,16 +177,17 @@ var mriview = (function(module) { THREE.EventDispatcher.prototype.apply(module.Viewer.prototype); module.Viewer.prototype.constructor = module.Viewer; - module.Viewer.prototype.drawView = function(scene, idx) { - if (this.surfs[idx].prerender !== undefined) - this.surfs[idx].prerender(this.renderer, scene, this.camera); + module.Viewer.prototype.drawView = function(scene, idx, camera, target) { + camera = camera === undefined ? this.camera : camera; + if (this.surfs[idx] !== undefined && this.surfs[idx].prerender !== undefined) + this.surfs[idx].prerender(this.renderer, scene, camera); for (var i = 0; i < this.surfs.length; i++) this.surfs[i].apply(this.active); if (this.oculus) - this.oculus.render(scene, this.camera); + this.oculus.render(scene, camera); else - this.renderer.render(scene, this.camera); + this.renderer.render(scene, camera, target); } module.Viewer.prototype.setOculus = function() { @@ -751,6 +755,9 @@ var mriview = (function(module) { this.sliceplanes.x.setData(this.active); this.sliceplanes.y.setData(this.active); this.sliceplanes.z.setData(this.active); + //setGrid above builds the views again, so the slice views, which + //are views of its scene, are built again on top of it + this.setSliceViews(this._sliceviews); }.bind(this)); }; @@ -1130,10 +1137,21 @@ var mriview = (function(module) { // Cache last pick position so setData() can refresh the picked // indicator for the newly-active dataset at the same screen point. this._lastPickEvt = {x: evt.x, y: evt.y}; + var x = evt.x, y = evt.y; + if (this._sliceviews) { + // The picker draws the 3D view over the whole canvas, while the + // split shows it in the corner. Both have the canvas's aspect, so + // they are the same image at half the scale and a click in that + // corner lands here; a click in a slice view picks nothing. + if (x < this.width / 2 || y < this.height / 2) + return; + x = (x - this.width / 2) * 2; + y = (y - this.height / 2) * 2; + } let coords for (var i = 0; i < this.surfs.length; i++) { if (this.surfs[i].pick) - coords = this.surfs[i].pick(this.renderer, this.camera, evt.x, evt.y); + coords = this.surfs[i].pick(this.renderer, this.camera, x, y); } // On-demand timeseries fetch. if (coords && coords !== -1) { @@ -1141,6 +1159,7 @@ var mriview = (function(module) { if (this.tsplot && this.tsplot_visible) this.fetchTimeseries(coords); } + this.setCursor(coords); // set the picked value display // Length check first so we don't index data[0] on an empty array. // Skip RGB, then ensure all child buffers have populated. @@ -1441,6 +1460,8 @@ var mriview = (function(module) { this.canvas.resize(function() { this.resize(); }.bind(this)); var cam_ui = this.ui.addFolder("camera", true); + //kept, because the isometric button is shown with its own camera + this._cam_ui = cam_ui; cam_ui.add({ azimuth: {action:[this.controls, 'setAzimuth', 0, 360]}, altitude: {action:[this.controls, 'setAltitude', 0, 180]}, @@ -1450,6 +1471,8 @@ var mriview = (function(module) { // movement.js). target: {action:[this.controls, 'setFoldedTarget'], hidden:true}, flat_target: {action:[this.controls, 'setFlatTarget'], hidden:true}, + orthographic: {action:[this, "setOrthographic"], toggle:true}, + isometric: {action:this.setIsometric.bind(this), shown:false}, }); var fold_brain = function() { @@ -1479,6 +1502,9 @@ var mriview = (function(module) { // saves out current view as a png this.imageWidth = 2400; this.imageHeight = 1200; + //how many samples across a pixel the image is rendered at before it is + //averaged back down to the size asked for + this.imageAntialias = "none"; var saveImage = function() { var image = viewer.getImage(this.imageWidth.toFixed(), this.imageHeight.toFixed()); var a = document.createElement('a'); @@ -1500,7 +1526,8 @@ var mriview = (function(module) { imageFolder.add({ "Save": {action:saveImage, key:'S', modKeys: ['shiftKey'], help:'Save current view as a png'}, "Width": {action:[this, 'imageWidth', 500, 4000]}, - "Height": {action:[this, 'imageHeight', 500, 4000]} + "Height": {action:[this, 'imageHeight', 500, 4000]}, + "Antialias": {action:[this, 'imageAntialias', ["none", "2x", "3x", "4x"]]} }); // Saved views and the keyframe animation panel (resources/js/viewtools.js). @@ -1607,6 +1634,9 @@ var mriview = (function(module) { //add sliceplane gui var sliceplane_ui = this.ui.addFolder("sliceplanes", true) sliceplane_ui.add({ + "Show ortho views": {action:[this, "setSliceViews"], toggle:true}, + orthoToggle: {action: this.toggleSliceViews.bind(this), key: 'v', hidden: true, + help:'Show ortho views along with 3D'}, x: {action:[this.sliceplanes.x, "setVisible"]}, xToggle: {action: this.toggleXVis.bind(this), key: 'e', hidden: true, help:'Toggle X slice'}, y: {action:[this.sliceplanes.y, "setVisible"]}, @@ -1774,6 +1804,378 @@ var mriview = (function(module) { this.sliceplanes.z.setVisible(!this.sliceplanes.z._visible); viewer.schedule(); }; + module.Viewer.prototype.toggleSliceViews = function() { + this.setSliceViews(!this._sliceviews); + }; + //The view down the diagonal of the three axes, where each is turned away + //from the eye by the same angle: what an orthographic camera is usually + //wanted for, since it draws the three of them to one scale. The button + //for it is in the menu only while that camera is the one looking. + var isometric_altitude = 180 * Math.acos(1 / Math.sqrt(3)) / Math.PI; + module.Viewer.prototype.setIsometric = function() { + this.animate([ + {state:'camera.azimuth', idx:parseFloat(viewopts.anim_speed), value:45}, + {state:'camera.altitude', idx:parseFloat(viewopts.anim_speed), + value:isometric_altitude}, + ]); + }; + module.Viewer.prototype.setOrthographic = function(val) { + if (val === undefined) + return jsplot.Axes3D.prototype.setOrthographic.call(this); + + jsplot.Axes3D.prototype.setOrthographic.call(this, val); + this._showIsometric(this.setOrthographic()); + }; + module.Viewer.prototype._showIsometric = function(show) { + var ctrl = this._cam_ui === undefined ? undefined : this._cam_ui._controls.isometric; + if (ctrl !== undefined && ctrl.__li !== undefined) + ctrl.__li.style.display = show ? "" : "none"; + }; + //------------------------------------------------------------------------- + // Orthogonal slice views + //------------------------------------------------------------------------- + //Where each slice view sits in the canvas, in the fractions of it + //three.js takes (the origin is its bottom left corner). The 3D view + //keeps the last quarter. + var SLICE_VIEWS = [ + {plane: "y", axis: 1, left: 0, bottom: 0.5}, + {plane: "z", axis: 2, left: 0, bottom: 0}, + {plane: "x", axis: 0, left: 0.5, bottom: 0.5}, + ]; + + //Splits the canvas between the three slice planes, each drawn straight + //down its own axis in a quarter of it, and the 3D view, which keeps the + //last quarter along with the camera the controls move and the surface as + //it is always drawn. A slice view shows its plane alone: the surface + //would stand in front of the data the view is there to show. + module.Viewer.prototype.setSliceViews = function(val) { + if (val === undefined) + return this._sliceviews; + //A point picked in the 3D view on its own left the planes alone, so + //the slices are taken to it as the views open; once they are open a + //pick moves them, and the slice keys are free to move them again. + var opening = !!val && this._sliceviews !== true; + this._sliceviews = !!val; + if (this.views.length == 0) + return; + if (opening) + this._slicesToCursor(); + + var scene = this.views[0].scene; + this._cursorObject(); + var views = []; + if (this._sliceviews) { + for (var i = 0; i < SLICE_VIEWS.length; i++) { + var spec = SLICE_VIEWS[i]; + var view = { + left: spec.left, bottom: spec.bottom, width: 0.5, height: 0.5, + scene: scene, surf: 0, plane: this.sliceplanes[spec.plane], + axis: spec.axis, + camera: new THREE.OrthographicCamera(-1, 1, 1, -1, 1, 10000), + }; + view.prepare = this._prepareSlice.bind(this, view); + view.overlay = this._drawCursor.bind(this); + views.push(view); + } + var solid = {left: 0.5, bottom: 0, width: 0.5, height: 0.5, scene: scene, surf: 0}; + solid.prepare = this._showAllPlanes.bind(this); + solid.overlay = this._drawCursor.bind(this); + views.push(solid); + } else { + views.push({left: 0, bottom: 0, width: 1, height: 1, scene: scene, surf: 0}); + } + this.views = views; + this._orthoLabels(this._sliceviews); + this._showPickMarkers(!this._sliceviews); + //the surfaces and the planes are left as the 3D view wants them, so + //that a single view, a screenshot or a pick sees what it always did + this._showAllPlanes(); + this.resize(); + this.schedule(); + }; + + //The crosshair that marks the picked point in the slice views. It is + //not the picker's own marker, which rides with the surface as it + //unfolds: this one stays where the point is in the anatomy, which is + //where the slices cut it. + module.Viewer.prototype._cursorObject = function() { + if (this._cursor === undefined) { + var far = 500; + //Three lines along the axes of the view that draws them. In a + //slice view the third one runs down the line of sight, so it is + //seen end on and only the cross of the other two shows; in the + //3D view all three do, which is what marks a point in space. + var lines = new THREE.Geometry(); + lines.vertices.push(new THREE.Vector3(-far, 0, 0), new THREE.Vector3(far, 0, 0), + new THREE.Vector3(0, -far, 0), new THREE.Vector3(0, far, 0), + new THREE.Vector3(0, 0, -far), new THREE.Vector3(0, 0, far)); + var material = new THREE.LineBasicMaterial({ + color: 0x44ccff, depthTest: false, depthWrite: false}); + this._cursor = new THREE.Line(lines, material, THREE.LinePieces); + this._cursor.frustumCulled = false; + this._cursor.visible = false; + this._cursorAt = false; + //kept out of the scene the views draw, so that it can be laid + //over each of them once the rest of it has been drawn + this._cursorScene = new THREE.Scene(); + this._cursorScene.add(this._cursor); + } + return this._cursor; + }; + + //Puts the crosshair on a picked point and takes the slice views to the + //slices through it, so that the four views agree on where it is. Picking + //nothing leaves them where they are. + module.Viewer.prototype.setCursor = function(coords) { + var cursor = this._cursorObject(); + var voxel = (coords === undefined || coords === -1) ? undefined : coords.voxel; + var xfm = (this.active === null || this.active === undefined) + ? undefined : this.active.uniforms.volxfm; + //A click that picked nothing leaves the crosshair where it is: it + //marks a place, and clicking beside the brain is not a way of saying + //to forget it. + if (voxel === undefined) + return; + //Data on the vertices has no volume to point into, so there is no + //place for the crosshair to be. + if (!isFinite(voxel.x) || !isFinite(voxel.y) || !isFinite(voxel.z) || + xfm === undefined || xfm.value[0] === undefined) { + this._cursorAt = false; + cursor.visible = false; + this.schedule(); + return; + } + var toWorld = new THREE.Matrix4().getInverse(xfm.value[0]); + cursor.position.copy(voxel.clone().applyMatrix4(toWorld)); + this._cursorAt = true; + this._cursorVoxel = voxel.clone(); + + //The slice views take the slices the point is on, so that it is on + //screen in each of them. The 3D view on its own leaves its planes + //where they were put, and is given them when it is split. + if (this._sliceviews) + this._slicesToCursor(); + this.schedule(); + }; + module.Viewer.prototype._slicesToCursor = function() { + var voxel = this._cursorVoxel; + if (this._cursorAt !== true || voxel === undefined) + return; + var slices = {x: voxel.x, y: voxel.y, z: voxel.z}; + for (var name in this.sliceplanes) { + if (this.sliceplanes[name].mesh !== undefined) + this.sliceplanes[name].update(slices[name]); + } + }; + + //Draws one slice view: the plane of this view alone, seen from straight + //down its normal and framed on the slice it cuts. Data that has no + //volume to slice, such as data on the vertices, leaves the surface in + //view instead, seen down the axis this view stands for. + module.Viewer.prototype._prepareSlice = function(view, width, height) { + var plane = view.plane; + var sliced = plane.mesh !== undefined && plane.geometry !== undefined; + this.root.visible = !sliced; + //the 3D view has the picker's own marker, so this one is for these + var cursor = this._cursorObject(); + cursor.visible = this._cursorAt === true; + for (var name in this.sliceplanes) { + var mesh = this.sliceplanes[name].mesh; + if (mesh !== undefined) + mesh.visible = this.sliceplanes[name] === plane; + } + + var look = new THREE.Vector3(), up, right, center, halfUp = 0, halfRight = 0; + if (sliced) { + //the face normal rather than the clipping one, which the flip + //controls turn around + look.copy(plane.geometry.faces[0].normal).applyEuler(plane.mesh.rotation).normalize(); + //seen from the subject's right, front or top, whichever of those + //the plane faces, so that the slices are laid out as they are read + var axis = 0; + for (var a = 1; a < 3; a++) { + if (Math.abs(look.getComponent(a)) > Math.abs(look.getComponent(axis))) + axis = a; + } + if (look.getComponent(axis) < 0) + look.negate(); + + //Up is one of the plane's own edges, not a world axis: the volume + //sits at an angle to the anatomy, and a world axis would leave + //the slice tilted on screen. Of the two edges, the one that + //points most nearly superior is up, or most nearly anterior on + //the plane that superior is normal to. + var corners = plane.geometry.vertices; + var edges = [new THREE.Vector3().subVectors(corners[1], corners[0]).normalize(), + new THREE.Vector3().subVectors(corners[2], corners[0]).normalize()]; + var upAxis = axis == 2 ? 1 : 2; + up = Math.abs(edges[0].getComponent(upAxis)) >= Math.abs(edges[1].getComponent(upAxis)) + ? edges[0] : edges[1]; + if (up.getComponent(upAxis) < 0) + up.negate(); + right = new THREE.Vector3().crossVectors(up, look).normalize(); + up.crossVectors(look, right).normalize(); + + //the quad of the slice, which is centered on the object itself + for (var v = 0; v < corners.length; v++) { + halfUp = Math.max(halfUp, Math.abs(corners[v].dot(up))); + halfRight = Math.max(halfRight, Math.abs(corners[v].dot(right))); + } + center = plane.center; + } else { + var box = new THREE.Box3().setFromObject(this.root); + if (box.empty()) + return; + look.setComponent(view.axis, 1); + up = view.axis == 2 ? new THREE.Vector3(0, 1, 0) : new THREE.Vector3(0, 0, 1); + right = new THREE.Vector3().crossVectors(up, look).normalize(); + var size = box.size(); + halfUp = Math.abs(size.dot(up)) / 2; + halfRight = Math.abs(size.dot(right)) / 2; + center = box.center(); + } + this._aimOrtho(view.camera, center, look, up, halfRight, halfUp, width / height); + //where the view is aimed, which is what a click in it is read against + view.look = look.clone(); + view.center = center.clone(); + //the crosshair lies in the plane of the view that draws it, so its + //two lines cross at the point rather than running off at an angle + var basis = new THREE.Matrix4(); + basis.set(right.x, up.x, look.x, 0, + right.y, up.y, look.y, 0, + right.z, up.z, look.z, 0, + 0, 0, 0, 1); + cursor.quaternion.setFromRotationMatrix(basis); + + //The lights ride with the camera the controls move, so a view drawn + //with another one would be lit from wherever that camera happens to + //point. Turn them to face this view for as long as it is drawn; the + //3D view, which is drawn last, puts them back. + if (this._camQuat === undefined) + this._camQuat = this.camera.quaternion.clone(); + this.camera.quaternion.copy(view.camera.quaternion); + }; + + //Points an orthographic camera at `center` from along `look`, with `up` + //on the screen and wide enough to hold what is being looked at + module.Viewer.prototype._aimOrtho = function(camera, center, look, up, halfRight, halfUp, aspect) { + var half = Math.max(halfUp, halfRight / aspect) * 1.05; + camera.up.copy(up); + camera.position.copy(center).add(look.clone().multiplyScalar(1000)); + camera.lookAt(center); + camera.left = -half * aspect; + camera.right = half * aspect; + camera.top = half; + camera.bottom = -half; + camera.updateProjectionMatrix(); + camera.updateMatrixWorld(); + }; + + //The slice planes are transparent, so they are drawn over everything + //opaque, the crosshair included. It goes back on top in a pass of its + //own, over the view that was just drawn. + module.Viewer.prototype._drawCursor = function(camera) { + if (!this._cursorObject().visible) + return; + //the slices are transparent and the brain is solid, so either would + //cover the crosshair if it were drawn among them + this.renderer.autoClear = false; + this.renderer.render(this._cursorScene, camera); + this.renderer.autoClear = true; + }; + + //Puts the surfaces and the planes back the way the 3D view shows them + module.Viewer.prototype._showAllPlanes = function() { + if (this._camQuat !== undefined) { + this.camera.quaternion.copy(this._camQuat); + this._camQuat = undefined; + } + this.root.visible = true; + //the crosshair marks the same point in the 3D view as in the slices, + //as long as they are there to be marked in + this._cursorObject().visible = + this._sliceviews === true && this._cursorAt === true; + for (var name in this.sliceplanes) { + var plane = this.sliceplanes[name]; + if (plane.mesh !== undefined) + plane.mesh.visible = plane.setVisible(); + } + }; + + //The picker marks the vertex it picked on the surface itself. The + //crosshair marks the point in the volume, which is the same point and is + //in every view, so the two are never up at once. + module.Viewer.prototype._showPickMarkers = function(show) { + for (var i = 0; i < this.surfs.length; i++) { + var surf = this.surfs[i].surf; + if (surf === undefined || surf.picker === undefined) + continue; + surf.picker.markers.left.visible = show; + surf.picker.markers.right.visible = show; + } + this.schedule(); + }; + + //A click in a slice view puts the crosshair where it landed on that + //slice, and takes the other two views to the slices through it. + module.Viewer.prototype._pickSlice = function(index, event) { + var view = this.views[index]; + //the view aims itself as it is drawn, so its own frame says where in + //the volume the pointer is + if (view === undefined || view.camera === undefined || + view.look === undefined || view.center === undefined) + return; + //with data on the vertices these views show the surface instead, and + //there is no slice for a click to land on + if (view.plane === undefined || view.plane.mesh === undefined) + return; + var xfm = (this.active === null || this.active === undefined) + ? undefined : this.active.uniforms.volxfm; + if (xfm === undefined || xfm.value[0] === undefined) + return; + + var rect = event.currentTarget.getBoundingClientRect(); + var point = new THREE.Vector3( + ((event.clientX - rect.left) / rect.width) * 2 - 1, + 1 - ((event.clientY - rect.top) / rect.height) * 2, 0).unproject(view.camera); + //the camera looks down the normal of the slice, so the point under + //the pointer is that one carried along the line of sight onto it + var depth = view.look.dot(new THREE.Vector3().subVectors(view.center, point)); + point.add(view.look.clone().multiplyScalar(depth)); + + var voxel = point.applyMatrix4(xfm.value[0]); + this.setCursor({voxel: new THREE.Vector3( + Math.round(voxel.x), Math.round(voxel.y), Math.round(voxel.z))}); + }; + + //One div over each slice view, which names it in its corner and + //takes the clicks made in it + module.Viewer.prototype._orthoLabels = function(show) { + if (this._orthoDivs === undefined) { + this._orthoDivs = []; + for (var i = 0; i < SLICE_VIEWS.length; i++) { + var div = document.createElement("div"); + div.className = "mriview-ortho"; + div.style.left = (100 * SLICE_VIEWS[i].left) + "%"; + //the views count from the bottom of the canvas, CSS from the top + div.style.top = (100 * (0.5 - SLICE_VIEWS[i].bottom)) + "%"; + div.appendChild(document.createElement("span")).textContent = + SLICE_VIEWS[i].plane + " slice"; + //a drag here would otherwise turn the 3D view, which is not + //what the pointer is over + var swallow = function(event) { event.stopPropagation(); }; + div.addEventListener("mousedown", swallow, false); + div.addEventListener("wheel", swallow, false); + div.addEventListener("dblclick", swallow, false); + div.addEventListener("click", this._pickSlice.bind(this, i), false); + this.object.appendChild(div); + this._orthoDivs.push(div); + } + } + for (var i = 0; i < this._orthoDivs.length; i++) + this._orthoDivs[i].style.display = show ? "block" : "none"; + }; + module.Viewer.prototype.setClippingX = function(val) { if (val === undefined) return this._clipx; diff --git a/cortex/webgl/resources/js/mriview_surface.js b/cortex/webgl/resources/js/mriview_surface.js index ffb86e85d..2359dbf02 100644 --- a/cortex/webgl/resources/js/mriview_surface.js +++ b/cortex/webgl/resources/js/mriview_surface.js @@ -94,6 +94,9 @@ var mriview = (function(module) { hatchColor: { type:'v3', value:new THREE.Vector3( 0,0,0 )}, dataAlpha: { type:'f', value:1.}, + //how much of the sky a vertex misses is taken out of its + //light: 1 is the occlusion as worked out, 0 none of it + occlusionStrength: { type:'f', value:1.}, overlay: { type:'t', value:null }, brightness: { type:'f', value:parseFloat(viewopts.brightness)}, smoothness: { type:'f', value:parseFloat(viewopts.smoothness)}, @@ -116,6 +119,11 @@ var mriview = (function(module) { "pial surface": {action: this.to_pial_surface.bind(this), key: 'p', help: "Pial surface"}, "fiducial surface": {action: this.to_fiducial_surface.bind(this), key: 'u', help: "Fiducial surface"}, "WM surface": {action: this.to_white_matter_surface.bind(this), key: 'y', help: "White matter surface"}, + "low poly": {action:[this, "setLowPoly", 0, 32], step:2}, + shading: {action:[this, "setShading", ["smooth", "flat", "gouraud", "phong"]]}, + outline: {action:[this, "setOutline"], toggle:true}, + "outline width": {action:[this, "setOutlineWidth", 0, .02], step:.0005}, + "outline corners": {action:[this, "setOutlineCorners", ["sharp", "round"]]}, bumpy_flatmap: {action:[this, "setBumpyFlat"]}, //The slider starts wherever the config file put it. Its range goes //to 5x true scale, or to twice the configured value if that is @@ -145,6 +153,8 @@ var mriview = (function(module) { topleft_lighting: {action:[this, "setTopLeftLighting", 0, 1]}, uniform_illumination: {action:[this, "setUniformIllumination", 0, 1]}, specularity: {action:[this, "setSpecular", 0, 1]}, + "ambient occlusion": {action:[this, "setOcclusion"], toggle:true}, + "occlusion strength": {action:[this.uniforms.occlusionStrength, "value", 0, 2]}, }); this.ui.addFolder("curvature", true).add({ @@ -169,6 +179,20 @@ var mriview = (function(module) { )]; this.names = json.names; + //A coarse set of faces is named after the pack it is cut from, + //beside it, so the viewer asks for one by the name of the mesh. + //A viewer served as files has only the widths it was given, which + //it is told; one with a server behind it has any width it asks + //for, since the server cuts what it has not cut yet. + if (json.data !== undefined) { + this._lodurl = loader.extractUrlBase(ctminfo) + + json.data.replace(/\.ctm$/, "_lod{cell}.bin"); + this._lodcells = json.lod_cells; + this._lod = {}; + //and the ambient occlusion of its vertices, likewise + this._aourl = loader.extractUrlBase(ctminfo) + + json.data.replace(/\.ctm$/, "_ao.bin"); + } var gb0 = geometries[0].boundingBox, gb1 = geometries[1].boundingBox; var center = [ ((gb1.max.x - gb0.min.x) / 2) + gb0.min.x, @@ -403,6 +427,9 @@ var mriview = (function(module) { var shade_cls = Shaders.surface_pixel; } var shaders = dataview.getShader(shade_cls, this.uniforms, { + shading: this._shading, + faceted: this._facets === true, + occlusion: this._occlusion === true && this._occluded === true, hasflat: this.flatlims !== undefined, morphs: this.names.length, volume: this.volume, @@ -438,9 +465,47 @@ var mriview = (function(module) { } module.Surface.prototype.prerender = function(renderer, scene, camera) { if (this.svg !== undefined) { + //the labels are hidden behind the surface by a depth image of it, + //which the dots of the outline are kept out of + var dots = []; + for (var i = 0; this._dots !== undefined && i < this._dots.length; i++) { + dots.push(this._dots[i].visible); + this._dots[i].visible = false; + } this.svg.prerender(renderer, scene, camera); + for (var i = 0; i < dots.length; i++) + this._dots[i].visible = dots[i]; + } + if (this._outlineUniforms !== undefined) { + //the line is measured in pixels of whatever is being drawn to + this._outlineUniforms.viewport.value.set( + renderer.domElement.width, renderer.domElement.height); + //the dots follow the faces being drawn, which detail and + //flattening swap + for (var i = 0; this._dots !== undefined && i < this._dots.length; i++) { + var hemi = this.hemis[["left", "right"][i]]; + if (this._dots[i].visible && hemi !== undefined && + (hemi._dotgeom === undefined || hemi._dotgeom.index !== hemi.attributes.index)) + this._dots[i] = this._makeDots(hemi, this._dots[i]); + } } } + //A point cloud over the vertices the dots belong at, in place of the one + //there was: the renderer sets an object up for the geometry it is made + //with, so a new geometry gets a new object. + module.Surface.prototype._makeDots = function(hemi, before) { + var pivot = this.pivots[hemi === this.hemis.left ? "left" : "right"].back; + var was = hemi._dotgeom === undefined ? undefined : hemi._dotgeom.geometry; + var dots = new THREE.PointCloud(this._dotGeometry(hemi), before.material); + dots.position.y = -this.flatoff[1]; + dots.visible = before.visible; + pivot.remove(before); + pivot.add(dots); + //a set of vertices picked out for the dots is let go of with them + if (was !== undefined && was !== hemi && was !== dots.geometry) + was.dispose(); + return dots; + }; // var oldcolor, black = new THREE.Color(0,0,0); // module.Surface.prototype._prerender_halosurf = function(evt) { @@ -644,6 +709,560 @@ var mriview = (function(module) { //The lights themselves hang off the viewer's camera, not the surface this.dispatchEvent({type:'lighting', topleft:val}); }; + //Draws the surface from fewer of its vertices. The faces come from a + //coarse surface cut over groups of `cell` mm, whose vertices are vertices + //of this one, so only the triangles change: the data, the morph targets, + //the flatmap and everything else held per vertex stay as they are. 0 is + //the surface as it was loaded. + module.Surface.prototype.setLowPoly = function(cell) { + if (cell === undefined) + return this._lowpoly === undefined ? 0 : this._lowpoly; + + cell = this._lodCell(cell); + this._lowpoly = cell; + if (cell === 0) + return this._applyLod(undefined); + if (this._lod[cell] !== undefined) + return this._applyLod(this._lod[cell]); + + var request = new XMLHttpRequest(); + request.open("GET", this._lodurl.replace("{cell}", cell), true); + request.responseType = "arraybuffer"; + request.onload = function() { + if (request.status !== 200) + return; + this._lod[cell] = this._readLod(request.response); + //the control may have moved on while the request was out + if (this._lowpoly === cell) + this._applyLod(this._lod[cell]); + }.bind(this); + request.send(); + }; + //The width this one is to be read as: a viewer served as files has only + //the widths it was given, since nothing is there to cut another. + module.Surface.prototype._lodCell = function(cell) { + cell = Math.round(cell); + if (cell <= 0 || this._lodurl === undefined) + return 0; + if (this._lodcells === undefined) + return cell; + if (this._lodcells.length === 0) + return 0; + var nearest = this._lodcells[0]; + for (var i = 1; i < this._lodcells.length; i++) { + if (Math.abs(this._lodcells[i] - cell) < Math.abs(nearest - cell)) + nearest = this._lodcells[i]; + } + return nearest; + }; + //The faces of each hemisphere, as they are sent: the number of faces of + //the two as uint32, then their vertices, three uint32 to a face. + module.Surface.prototype._readLod = function(buffer) { + var raw = new Uint32Array(buffer); + var names = ["left", "right"], sets = {}, at = names.length; + for (var i = 0; i < names.length; i++) { + var hemi = this.hemis[names[i]]; + var map = this._lodMap(hemi); + var faces = new Uint32Array(3 * raw[i]); + for (var j = 0; j < faces.length; j++) + faces[j] = map[raw[at + j]]; + at += faces.length; + sets[names[i]] = { + full: {index: new THREE.BufferAttribute(faces, 3), offsets: []}, + culled: hemi.culled === undefined ? undefined : this._cullLod(hemi, faces), + }; + } + return sets; + }; + //The faces are cut in the numbering of the ctm pack, which the loader + //renumbers as it reads it so that the vertices of a face are within one + //block of 65535. Its own map says where each of them went. + module.Surface.prototype._lodMap = function(hemi) { + if (hemi._lodmap === undefined) { + if (hemi.indexMap === undefined) { + //nothing was renumbered, so the pack's numbering is this one + var n = hemi.attributes.position.array.length / 3; + hemi._lodmap = new Uint32Array(n); + for (var i = 0; i < n; i++) + hemi._lodmap[i] = i; + } else { + var keys = Object.keys(hemi.indexMap), last = 0; + for (var i = 0; i < keys.length; i++) + last = Math.max(last, keys[i] | 0); + hemi._lodmap = new Uint32Array(last + 1); + for (var i = 0; i < keys.length; i++) + hemi._lodmap[keys[i] | 0] = hemi.indexMap[keys[i]]; + } + } + return hemi._lodmap; + }; + //The same faces without the ones on the medial wall, which is what the + //flatmap is drawn from + module.Surface.prototype._cullLod = function(hemi, faces) { + var aux = hemi.attributes.auxdat.array; + var kept = new Uint32Array(faces.length), n = 0; + for (var i = 0; i < faces.length; i += 3) { + if (!aux[faces[i] * 4] && !aux[faces[i + 1] * 4] && !aux[faces[i + 2] * 4]) { + kept[n++] = faces[i]; + kept[n++] = faces[i + 1]; + kept[n++] = faces[i + 2]; + } + } + return {index: new THREE.BufferAttribute(kept.subarray(0, n), 3), offsets: []}; + }; + //How a face is colored and lit. The surface is drawn with one color + //interpolated across each face and the light worked out at every pixel of + //it ("smooth", which is what it opens with); the others give a face one + //color of its own, read where its first corner is, and light it from its + //own normal ("flat"), from its corners ("gouraud"), or at every pixel + //("phong"). One color to a face asks for a vertex per corner of a face, + //which is only built where the surface is coarse enough to hold them, and + //only for data in a volume: data on the vertices is itself held a vertex + //at a time. + //A line drawn around the surface and around the folds that turn away from + //the eye, in the way a comic is inked. It is the surface drawn once more, + //turned inside out and grown by a width of the screen, so what shows of + //it is a band where the surface ends. + module.Surface.prototype.setOutline = function(on) { + if (on === undefined) + return this._outline === true; + + this._outline = !!on; + if (this._inked === undefined && this._outline) + this._makeOutline(); + this._showOutline(); + this.dispatchEvent({type:"update"}); + }; + //Where the line turns a corner of the silhouette, a dot of its width at + //the vertex rounds it off, the way a round join does in a drawing + //program; without the dots the line comes to a point there. The dots are + //drawn at the vertices that face away, as the line is drawn from the + //faces that do, so the surface in front covers them except at its edge. + module.Surface.prototype.setOutlineCorners = function(mode) { + if (mode === undefined) + return this._corners === undefined ? "sharp" : this._corners; + + this._corners = mode; + this._showOutline(); + this.dispatchEvent({type:"update"}); + }; + module.Surface.prototype._showOutline = function() { + for (var i = 0; this._inked !== undefined && i < this._inked.length; i++) + this._inked[i].visible = this._outline === true; + for (var i = 0; this._dots !== undefined && i < this._dots.length; i++) + this._dots[i].visible = this._outline === true && this.setOutlineCorners() === "round"; + }; + //The vertices the dots are drawn at are the ones in the faces being + //drawn: a vertex left out at a coarser detail, or on the medial wall of + //the flatmap, would put a dot off the line. The surface's own geometry + //serves while every vertex is in them. + module.Surface.prototype._dotGeometry = function(hemi) { + var index = hemi.attributes.index; + if (hemi._dotgeom !== undefined && hemi._dotgeom.index === index) + return hemi._dotgeom.geometry; + var geometry = hemi; + if (hemi._welded !== undefined || hemi.fullind === undefined || index !== hemi.fullind.index || + (hemi._lodbase !== undefined && hemi.fullind !== hemi._lodbase.full)) { + var faces = this._faceList({index: index, offsets: hemi.offsets}); + var count = hemi.attributes.position.array.length / 3; + var used = new Uint8Array(count), n = 0; + for (var i = 0; i < faces.length; i++) + used[faces[i]] = 1; + for (var v = 0; v < count; v++) + n += used[v]; + var list = new Uint32Array(n); + for (var v = 0, j = 0; v < count; v++) + if (used[v]) list[j++] = v; + geometry = new THREE.BufferGeometry(); + for (var key in hemi.attributes) { + var attr = hemi.attributes[key]; + if (key === "index" || attr.array.length === 0) + continue; + var size = attr.itemSize; + var array = new attr.array.constructor(n * size); + for (var j = 0; j < n; j++) + for (var c = 0; c < size; c++) + array[j * size + c] = attr.array[list[j] * size + c]; + geometry.addAttribute(key, new THREE.BufferAttribute(array, size)); + } + } + hemi._dotgeom = {index: index, geometry: geometry}; + return geometry; + }; + //How wide the line is, across the screen rather than in the world: a + //width of a thousandth is about a pixel on a canvas a thousand across. + var default_outline_width = 0.004; + module.Surface.prototype.setOutlineWidth = function(width) { + if (width === undefined) + return this._outlineWidth === undefined ? default_outline_width : this._outlineWidth; + + this._outlineWidth = width; + if (this._outlineUniforms !== undefined) + this._outlineUniforms.outlineWidth.value = width; + this.dispatchEvent({type:"update"}); + }; + //Ambient occlusion: how much of the sky each vertex misses, worked out in + //python for each surface the viewer mixes between (utils.get_occlusion), + //the folded one at both depths, and fetched the first time it is asked + //for. It rides in a spare component of attributes that are already here, + //the w of each morph target's position and of the white matter position + //(auxdat.z for a subject with no white matter surface), because these + //shaders use every one of the 16 attribute slots WebGL guarantees. + module.Surface.prototype.setOcclusion = function(on) { + if (on === undefined) + return this._occlusion === true; + + this._occlusion = !!on; + if (this._occlusion && !this._occluded) { + //the shaders switch over once the values are in + if (this._aourl !== undefined && this._aorequest === undefined) { + var request = new XMLHttpRequest(); + request.open("GET", this._aourl, true); + request.responseType = "arraybuffer"; + request.onload = function() { + this._aorequest = undefined; + if (request.status !== 200) + return; + this._applyOcclusion(request.response); + this._occluded = true; + if (this._occlusion) + this.resetShaders(); + }.bind(this); + this._aorequest = request; + request.send(); + } + } else { + this.resetShaders(); + } + this.dispatchEvent({type:"update"}); + }; + //The file holds how many blocks it has and how many vertices each + //hemisphere has, as three uint32, then a byte per vertex a block at a + //time, the left hemisphere's before the right's: the folded surface at + //depth 0 (pial) and at depth 1 (white matter), then each surface the + //viewer mixes between after the folded one, the flat surface last. + module.Surface.prototype._applyOcclusion = function(buffer) { + var head = new Uint32Array(buffer, 0, 3); + var bytes = new Uint8Array(buffer, 12); + var counts = {left: head[1], right: head[2]}; + var names = ["left", "right"]; + for (var i = 0; i < names.length; i++) { + var hemi = this.hemis[names[i]], count = counts[names[i]]; + if (hemi === undefined) + continue; + //the blocks of this hemisphere + var blocks = []; + for (var block = 0; block < head[0]; block++) { + var at = block * (counts.left + counts.right) + (i == 0 ? 0 : counts.left); + blocks.push(bytes.subarray(at, at + count)); + } + //written into the surface as it is held, at the place the loader + //put each vertex of the pack; a surface drawn a vertex per corner + //reads copies, which are written out again + var attrs = hemi._welded === undefined ? hemi.attributes : hemi._welded.attributes; + var map = this._lodMap(hemi), n = Math.min(count, map.length); + var written = {}; + if (attrs.wm !== undefined) { + //both depths of the folded surface, as two bytes of one value + var wm = attrs.wm.array; + for (var v = 0; v < n; v++) + wm[map[v] * 4 + 3] = blocks[0][v] + 256 * blocks[1][v]; + written.wm = attrs.wm; + } else { + var aux = attrs.auxdat.array; + for (var v = 0; v < n; v++) + aux[map[v] * 4 + 2] = blocks[0][v] / 255; + written.auxdat = attrs.auxdat; + } + for (var block = 2; block < blocks.length; block++) { + var name = "mixSurfs" + (block - 2); + if (attrs[name] === undefined) + continue; + var array = attrs[name].array; + for (var v = 0; v < n; v++) + array[map[v] * 4 + 3] = blocks[block][v] / 255; + written[name] = attrs[name]; + } + for (var name in written) { + written[name].needsUpdate = true; + if (hemi._welded !== undefined) + hemi.addAttribute(name, this._facetAttribute(hemi, written[name], false)); + } + } + }; + module.Surface.prototype._makeOutline = function() { + //shared by the line and its dots, and with the surface, so that + //unfolding and depth move the line with it + this._outlineUniforms = { + outlineWidth: {type:'f', value:this.setOutlineWidth()}, + outlineColor: {type:'v3', value:new THREE.Vector3(0, 0, 0)}, + viewport: {type:'v2', value:new THREE.Vector2(1, 1)}, + }; + var opts = { + morphs: this.names.length, + volume: this.volume, + hasflat: this.flatlims !== undefined, + equivolume: this._equivolume, + }; + var material = function(shaders, side) { + var made = new THREE.ShaderMaterial({ + vertexShader: shaders.vertex, + fragmentShader: shaders.fragment, + attributes: shaders.attrs, + uniforms: THREE.UniformsUtils.merge([this.uniforms, this._outlineUniforms]), + side: side, + }); + //the uniforms themselves rather than copies of them + for (var name in this.uniforms) + made.uniforms[name] = this.uniforms[name]; + for (var name in this._outlineUniforms) + made.uniforms[name] = this._outlineUniforms[name]; + return made; + }.bind(this); + var inked = material(Shaders.outline(opts), THREE.BackSide); + var dotted = material(Shaders.outline(Object.assign({dots: true}, opts)), THREE.FrontSide); + + this._inked = []; + this._dots = []; + var names = ["left", "right"]; + for (var i = 0; i < names.length; i++) { + var hemi = this.hemis[names[i]]; + var mesh = this._makeMesh(hemi, inked); + mesh.visible = false; + this.pivots[names[i]].back.add(mesh); + this._inked.push(mesh); + + var dots = new THREE.PointCloud(this._dotGeometry(hemi), dotted); + dots.position.y = -this.flatoff[1]; + dots.visible = false; + this.pivots[names[i]].back.add(dots); + this._dots.push(dots); + } + }; + module.Surface.prototype.setShading = function(mode) { + if (mode === undefined) + return this._shading === undefined ? "smooth" : this._shading; + + this._shading = mode; + this._setFacets(mode !== "smooth"); + this.resetShaders(); + this.dispatchEvent({type:"update"}); + }; + //How many faces are worth taking a vertex per corner over + var facet_limit = 250000; + module.Surface.prototype._setFacets = function(on) { + var names = ["left", "right"], hemis = [], faces = 0; + for (var i = 0; i < names.length; i++) { + var hemi = this.hemis[names[i]]; + if (hemi === undefined || hemi.fullind === undefined) + continue; + if (hemi._welded !== undefined) { + //back to the surface as it is held, so that what is built + //here is always built from that + for (var key in hemi._welded.attributes) + hemi.addAttribute(key, hemi._welded.attributes[key]); + hemi.fullind = hemi._welded.fullind; + hemi.culled = hemi._welded.culled; + hemi._welded = undefined; + } + hemis.push(hemi); + faces += this._faceList(hemi.fullind).length / 3; + } + + //Every attribute of the surface is written out again for each corner + //of each face, so a surface that is not coarse would take hundreds of + //megabytes. Past the limit a face is lit the way it was asked for and + //keeps its interpolated color, which at that many faces to a pixel is + //what it looks like anyway. The two hemispheres go together, since + //one shader draws them both. + if (on && faces > facet_limit) { + console.log("pycortex: " + faces + " faces are too many to give " + + "each one a color of its own; the low poly control " + + "makes a coarser surface"); + on = false; + } + if (on) { + for (var i = 0; i < hemis.length; i++) + this._facetHemi(hemis[i]); + } + this._facets = on === true; + var now = this._flat > 0 && this.hemis.left.culled !== undefined; + for (var i = 0; i < names.length; i++) { + var hemi = this.hemis[names[i]]; + if (hemi === undefined || hemi.fullind === undefined) + continue; + var use = now ? hemi.culled : hemi.fullind; + hemi.attributes.index = use.index; + hemi.offsets = use.offsets; + } + }; + //The faces of one hemisphere written out with a vertex of their own per + //corner, each carrying what its corner carries, and the position its + //first corner reads the volume at. + module.Surface.prototype._facetHemi = function(hemi) { + var faces = this._faceList(hemi.fullind); + var welded = {attributes: {}, fullind: hemi.fullind, culled: hemi.culled}; + var wall = hemi.attributes.auxdat.array; + hemi._corners = faces; + for (var key in hemi.attributes) { + if (key == "index") + continue; + welded.attributes[key] = hemi.attributes[key]; + hemi.addAttribute(key, this._facetAttribute(hemi, hemi.attributes[key], + module.facet_data.test(key))); + } + + //the one position the whole face reads the volume at + var pos = welded.attributes.position, wm = welded.attributes.wm; + var facepos = new Float32Array(faces.length * 3); + var facewm = wm === undefined ? undefined : new Float32Array(faces.length * 4); + for (var v = 0; v < faces.length; v++) { + var first = faces[3 * Math.floor(v / 3)]; + for (var c = 0; c < 3; c++) + facepos[v * 3 + c] = pos.array[first * 3 + c]; + if (facewm !== undefined) { + for (var c = 0; c < 4; c++) + facewm[v * 4 + c] = wm.array[first * 4 + c]; + } + } + hemi.addAttribute("facepos", new THREE.BufferAttribute(facepos, 3)); + if (facewm !== undefined) + hemi.addAttribute("facewm", new THREE.BufferAttribute(facewm, 4)); + + //every corner is its own vertex now, so the faces are the vertices in + //the order they are written, and the flatmap leaves out the ones on + //the medial wall as it does at any detail + var index = new Uint32Array(faces.length); + var kept = new Uint32Array(faces.length), n = 0; + for (var v = 0; v < faces.length; v += 3) { + index[v] = v; + index[v + 1] = v + 1; + index[v + 2] = v + 2; + if (!wall[faces[v] * 4] && !wall[faces[v + 1] * 4] && !wall[faces[v + 2] * 4]) { + kept[n++] = v; + kept[n++] = v + 1; + kept[n++] = v + 2; + } + } + hemi._welded = welded; + hemi.fullind = {index: new THREE.BufferAttribute(index, 3), offsets: []}; + hemi.culled = welded.culled === undefined ? undefined : + {index: new THREE.BufferAttribute(kept.subarray(0, n), 3), offsets: []}; + }; + //The data a surface carries a vertex at a time, which is where the color + //of such a view comes from: a face takes what its first corner has, the + //way a face of a view of a volume takes what is read where its first + //corner is. + module.facet_data = /^(data[0-9]|nanmask)$/; + //One of a hemisphere's attributes written out for the corners of its + //faces, each corner taking what it carries, or all three taking what the + //first of them carries. + module.Surface.prototype._facetAttribute = function(hemi, attr, flatten) { + var corners = hemi._corners, size = attr.itemSize; + var array = new attr.array.constructor(corners.length * size); + for (var v = 0; v < corners.length; v++) { + var from = corners[flatten ? 3 * Math.floor(v / 3) : v]; + for (var c = 0; c < size; c++) + array[v * size + c] = attr.array[from * size + c]; + } + var out = new THREE.BufferAttribute(array, size); + out.needsUpdate = true; + return out; + }; + //A frame of data, which arrives for the surface as it is held rather than + //as it is drawn, so it is kept and written out for the faces. + module.Surface.prototype.setDataAttribute = function(name, left, right) { + var values = {left: left, right: right}; + for (var side in values) { + var hemi = this.hemis[side]; + if (hemi === undefined) + continue; + if (hemi._welded === undefined) { + hemi.addAttribute(name, values[side]); + } else { + hemi._welded.attributes[name] = values[side]; + hemi.addAttribute(name, this._facetAttribute( + hemi, values[side], module.facet_data.test(name))); + } + } + }; + //The vertices of a set of faces, counted from the start of the surface: + //the ones the surface came with are held in blocks of 65535, each face + //counted from the start of its own block. + module.Surface.prototype._faceList = function(entry) { + var indices = entry.index.array, offsets = entry.offsets; + if (offsets === undefined || offsets.length === 0) + return indices; + var out = new Uint32Array(indices.length), n = 0; + for (var j = 0; j < offsets.length; j++) { + var start = offsets[j].start, count = offsets[j].count, base = offsets[j].index; + for (var i = start; i < start + count; i++) + out[n++] = base + indices[i]; + } + return out.subarray(0, n); + }; + //Puts a set of faces in place of the ones being drawn, keeping the ones + //the surface came with to go back to. The flatmap draws the culled set, + //as it does at any detail, and unfolding goes on swapping between them. + module.Surface.prototype._applyLod = function(sets) { + var names = ["left", "right"]; + //the faces a vertex per corner is written out from are these ones, + //so they are put back first and written out again at the end + var facets = this._facets; + if (facets) + this._setFacets(false); + for (var i = 0; i < names.length; i++) { + var hemi = this.hemis[names[i]]; + if (hemi === undefined || hemi.fullind === undefined) + continue; + if (hemi._lodbase === undefined) { + hemi._lodbase = {full: hemi.fullind, culled: hemi.culled, normals: {}}; + var sources = this._lodNormals(hemi); + for (var attr in sources) + hemi._lodbase.normals[attr] = hemi.attributes[attr].array; + } + var use = sets === undefined ? hemi._lodbase : sets[names[i]]; + hemi.fullind = use.full; + hemi.culled = use.culled; + var now = this._flat > 0 && hemi.culled !== undefined ? hemi.culled : hemi.fullind; + hemi.attributes.index = now.index; + hemi.offsets = now.offsets; + + //A coarse surface shaded with the normals of the fine one it was + //cut from reads as the fine one, so the normals are taken from + //the faces being drawn. The flatmap's are the same at any + //detail, since they all point the same way. + var sources = this._lodNormals(hemi); + var span = [{start:0, count:now.index.array.length, index:0}]; + for (var attr in sources) { + hemi.attributes[attr].array = sets === undefined + ? hemi._lodbase.normals[attr] + : module.computeNormal(hemi.attributes[sources[attr]], + now.index, span).array; + hemi.attributes[attr].needsUpdate = true; + } + } + if (facets) + this._setFacets(true); + this.dispatchEvent({type:"update"}); + }; + //Which of a hemisphere's normals are worked out from which of its + //surfaces. The flatmap is left out: its normals are a constant. + module.Surface.prototype._lodNormals = function(hemi) { + if (hemi._lodnormals === undefined) { + var sources = {normal: "position"}; + if (hemi.attributes.wmnorm !== undefined) + sources.wmnorm = "wm"; + var last = 0; + while (hemi.attributes["mixNorms" + last] !== undefined) + last++; + //the last pair is the flatmap, where they are a constant + for (var i = 0; i < last - (this.flatlims !== undefined ? 1 : 0); i++) + sources["mixNorms" + i] = "mixSurfs" + i; + hemi._lodnormals = sources; + } + return hemi._lodnormals; + }; module.Surface.prototype.setBumpyFlat = function(val) { if (val === undefined) return this.uniforms.bumpyflat.value; @@ -918,10 +1537,9 @@ var mriview = (function(module) { } module.SurfDelegate.prototype.setAttribute = function(event) { var name = event.name, left = event.value[0], right = event.value[1]; - var hemis = this.surf.hemis; - this.surf.loaded.done(function() { - hemis.left.attributes[name] = left; - hemis.right.attributes[name] = right; + var surf = this.surf; + surf.loaded.done(function() { + surf.setDataAttribute(name, left, right); }); } module.SurfDelegate.prototype.pick = function(renderer, camera, x, y) { diff --git a/cortex/webgl/resources/js/mriview_utils.js b/cortex/webgl/resources/js/mriview_utils.js index fd5f5ae58..318b559f3 100644 --- a/cortex/webgl/resources/js/mriview_utils.js +++ b/cortex/webgl/resources/js/mriview_utils.js @@ -93,7 +93,10 @@ var mriview = (function(module) { var glcanvas = document.createElement("canvas"); var glctx = glcanvas.getContext("2d"); - module.getTexture = function(gl, renderbuf) { + module.getTexture = function(gl, renderbuf, samples) { + if (samples > 1) + return module.downsampleTexture(gl, renderbuf, samples); + glcanvas.width = renderbuf.width; glcanvas.height = renderbuf.height; var img = glctx.createImageData(renderbuf.width, renderbuf.height); @@ -113,6 +116,55 @@ var mriview = (function(module) { return canvas; } + //A buffer rendered samples times wider and taller than the image asked for, + //averaged a block of samples by samples at a time down to one pixel. The + //color of a sample is weighted by its alpha, so an edge where only some of + //the samples are on the surface keeps the color of the surface and takes + //the rest of its alpha from the transparent background. + module.downsampleTexture = function(gl, renderbuf, samples) { + var width = Math.floor(renderbuf.width / samples); + var height = Math.floor(renderbuf.height / samples); + var raw = new Uint8Array(renderbuf.width * renderbuf.height * 4); + gl.bindFramebuffer(gl.FRAMEBUFFER, renderbuf.__webglFramebuffer); + gl.readPixels(0, 0, renderbuf.width, renderbuf.height, gl.RGBA, gl.UNSIGNED_BYTE, raw); + gl.bindFramebuffer(gl.FRAMEBUFFER, null); + + var canvas = document.createElement("canvas"); + canvas.width = width; + canvas.height = height; + var ctx = canvas.getContext("2d"); + var img = ctx.createImageData(width, height); + var out = img.data; + var n = samples * samples; + for (var y = 0; y < height; y++) { + for (var x = 0; x < width; x++) { + var r = 0, g = 0, b = 0, a = 0; + for (var dy = 0; dy < samples; dy++) { + //readPixels hands back the rows from the bottom up, so a row + //of the image is read from the other end of the buffer + var row = (renderbuf.height - 1 - y * samples - dy) * renderbuf.width; + for (var dx = 0; dx < samples; dx++) { + var i = (row + x * samples + dx) * 4; + var alpha = raw[i+3]; + r += raw[i] * alpha; + g += raw[i+1] * alpha; + b += raw[i+2] * alpha; + a += alpha; + } + } + var o = (y * width + x) * 4; + if (a > 0) { + out[o] = Math.round(r / a); + out[o+1] = Math.round(g / a); + out[o+2] = Math.round(b / a); + } + out[o+3] = Math.round(a / n); + } + } + ctx.putImageData(img, 0, 0); + return canvas; + } + module._cull_flatmap_vertices = function(indices, auxdat, offsets) { var culled = new Uint16Array(indices.length), vA, vB, vC; var newOffsets = []; diff --git a/cortex/webgl/resources/js/shaderlib.js b/cortex/webgl/resources/js/shaderlib.js index 2cf3af0ae..3afe9e9d2 100644 --- a/cortex/webgl/resources/js/shaderlib.js +++ b/cortex/webgl/resources/js/shaderlib.js @@ -183,10 +183,56 @@ var Shaderlib = (function() { } glsl += [ "", "}", + //The ambient occlusion of a vertex, mixed between the surfaces as + //its position is: each surface's rides in the w of its position, + //the folded surface's in what the caller hands in. + "float mixfunc_occlusion(float baseocc) {", + "float smix = surfmix * "+(morphs-1)+".;", + "float factor = clamp(1. - smix, 0., 1.);", + "float occ = factor * baseocc;", + ].join("\n"); + for (var i = 0; i < morphs-1; i++) { + glsl += "factor = clamp( 1. - abs(smix - "+(i+1)+".) , 0., 1.);\n"; + glsl += "occ += factor * mixSurfs"+i+".w;\n"; + } + glsl += [ "", + "return occ;", + "}", ].join("\n"); return glsl; }, + //The occlusion of this vertex on the folded surface. It rides in a + //spare component of an attribute that is already here, since these + //shaders use every one of the 16 attribute slots WebGL guarantees. A + //subject with a white matter surface has two values, one on the pial + //surface and one on the white matter, mixed by the depth the sheet + //is drawn at as its position is; they ride as two bytes in the w of + //the white matter position, the pial one low. Without one there is + //a single value, in auxdat.z, where the equivolume areas it is meant + //for are never read. + occlusion_main: [ + "#ifdef OCCLUSION", + "#ifdef CORTSHEET", + "float pialocc = mod(wm.w, 256.) / 255.;", + "float wmocc = floor(wm.w / 256.) / 255.;", + "vOcclusion = mixfunc_occlusion(mix(pialocc, wmocc, use_thickmix));", + "#else", + "vOcclusion = mixfunc_occlusion(auxdat.z);", + "#endif", + "#endif", + ].join("\n"), + //Takes the sky a vertex misses out of the light that reaches it, once + //the fragment is lit. Squared, so that a crown with a little of its + //sky blocked by the gyri across from it stays nearly as bright as an + //open plane, while the floor of a sulcus goes dark; the strength + //scales how much is taken, past 1 for more than was worked out. + occlusion_fragment: [ + "#ifdef OCCLUSION", + "gl_FragColor.rgb *= clamp(1. - occlusionStrength * vOcclusion * vOcclusion, 0., 1.);", + "#endif", + ].join("\n"), + // thickmixer: header code that loads the uniforms needed to do // equivolume sampling, for vertex shaders. The white matter and pial // vertex areas it needs ride along in auxdat.zw, which mriview_surface @@ -401,6 +447,21 @@ var Shaderlib = (function() { header += "#define HASFLAT\n" if (opts.equivolume) header += "#define EQUIVOLUME\n" + //One color to a face rather than one interpolated across it: the + //volume is read at a position the whole face carries, which a + //surface with a vertex per corner of a face can hold. + if (opts.faceted) + header += "#define FACECOLOR\n"; + //and the light on a face is worked out from its own normal + //(flat) or at its corners (gouraud) rather than at every pixel + //of it, which is what the surface is drawn with otherwise + if (opts.shading == "flat") + header += "#define FLATSHADE\n"; + else if (opts.shading == "gouraud") + header += "#define GOURAUD\n"; + //and the sky each vertex misses is taken out of the light on it + if (opts.occlusion) + header += "#define OCCLUSION\n"; var vertShade = [ THREE.ShaderChunk[ "lights_phong_pars_vertex" ], @@ -417,6 +478,23 @@ var Shaderlib = (function() { utils.flatbump_attr, + "#ifdef FACECOLOR", + "attribute vec3 facepos;", + "attribute vec4 facewm;", + "#endif", + + "#ifdef GOURAUD", + "#if MAX_DIR_LIGHTS > 0", + "uniform vec3 directionalLightColor[ MAX_DIR_LIGHTS ];", + "uniform vec3 directionalLightDirection[ MAX_DIR_LIGHTS ];", + "#endif", + "uniform vec3 diffuse;", + "uniform vec3 specular;", + "uniform float shininess;", + "varying vec3 vLightDiffuse;", + "varying vec3 vLightSpecular;", + "#endif", + // "attribute float dropout;", "varying vec3 vViewPosition;", @@ -424,6 +502,10 @@ var Shaderlib = (function() { "varying vec2 vUv;", "varying float vCurv;", "varying float vMedial;", + "#ifdef OCCLUSION", + "varying float vOcclusion;", + "uniform float occlusionStrength;", + "#endif", "varying float vThickmix;", "varying vec3 vWorldPosition;", // "varying float vDrop;", @@ -443,14 +525,21 @@ var Shaderlib = (function() { "vViewPosition = -mvPosition.xyz;", //Find voxel positions with both transforms (2D colormap x and y datasets) - "vPos_x[0] = (volxfm[0]*vec4(position,1.)).xyz;", + "#ifdef FACECOLOR", + "vec3 readpos = facepos;", + "vec3 readwm = facewm.xyz;", + "#else", + "vec3 readpos = position;", + "vec3 readwm = wm.xyz;", + "#endif", + "vPos_x[0] = (volxfm[0]*vec4(readpos,1.)).xyz;", "#ifdef TWOD", - "vPos_y[0] = (volxfm[1]*vec4(position,1.)).xyz;", + "vPos_y[0] = (volxfm[1]*vec4(readpos,1.)).xyz;", "#endif", "#ifdef CORTSHEET", - "vPos_x[1] = (volxfm[0]*vec4(wm.xyz,1.)).xyz;", + "vPos_x[1] = (volxfm[0]*vec4(readwm,1.)).xyz;", "#ifdef TWOD", - "vPos_y[1] = (volxfm[1]*vec4(wm.xyz,1.)).xyz;", + "vPos_y[1] = (volxfm[1]*vec4(readwm,1.)).xyz;", "#endif", "#endif", @@ -470,6 +559,7 @@ var Shaderlib = (function() { "vec3 pos, norm;", "mixfunc(mpos, mnorm, pos, norm);", + utils.occlusion_main, "#ifdef CORTSHEET", // @@ -500,9 +590,45 @@ var Shaderlib = (function() { "gl_Position = projectionMatrix * modelViewMatrix * vec4( pos, 1.0 );", "vWorldPosition = pos;", + + "#ifdef GOURAUD", + //the directional lights worked out here rather than at every + //pixel, which is what tells gouraud from phong + "vec3 gnorm = normalize(vNormal);", + "vec3 gview = normalize(vViewPosition);", + "vLightDiffuse = vec3(0.);", + "vLightSpecular = vec3(0.);", + "#if MAX_DIR_LIGHTS > 0", + "for (int i = 0; i < MAX_DIR_LIGHTS; i++) {", + "vec4 lDirection = viewMatrix * vec4(directionalLightDirection[i], 0.);", + "vec3 dirVector = normalize(lDirection.xyz);", + "float dirDiffuseWeight = max(dot(gnorm, dirVector), 0.);", + "vLightDiffuse += diffuse * directionalLightColor[i] * dirDiffuseWeight;", + "vec3 dirHalfVector = normalize(dirVector + gview);", + "float dirDotNormalHalf = max(dot(gnorm, dirHalfVector), 0.);", + "float dirSpecularWeight = max(pow(dirDotNormalHalf, shininess), 0.);", + "float specularNormalization = (shininess + 2.) / 8.;", + "vec3 schlick = specular + vec3(1. - specular) * pow(max(1. - dot(dirVector, dirHalfVector), 0.), 5.);", + "vLightSpecular += schlick * directionalLightColor[i] * dirSpecularWeight * dirDiffuseWeight * specularNormalization;", + "}", + "#endif", + "#endif", "}" ].join("\n"); + //The light on a face: from its own normal, which the view + //position's change across the screen gives, so that the face has + //one shade of its own; or from what its corners were lit with. + var lighting = THREE.ShaderChunk[ "lights_phong_fragment" ]; + if (opts.shading == "flat") { + lighting = lighting.replace("vec3 normal = normalize( vNormal );", [ + "vec3 normal = normalize(cross(dFdx(vViewPosition), dFdy(vViewPosition)));", + "normal *= sign(dot(normal, normalize(vViewPosition)));"].join("\n")); + } else if (opts.shading == "gouraud") { + lighting = "gl_FragColor.xyz = gl_FragColor.xyz * (emissive + " + + "vLightDiffuse + ambientLightColor * ambient) + vLightSpecular;"; + } + var fragHead = [ "#extension GL_OES_standard_derivatives: enable", //"#extension GL_OES_texture_float: enable", @@ -559,9 +685,18 @@ var Shaderlib = (function() { "varying float vCurv;", "varying float vMedial;", + "#ifdef OCCLUSION", + "varying float vOcclusion;", + "uniform float occlusionStrength;", + "#endif", "varying float vThickmix;", "varying vec3 vWorldPosition;", // the x,y,z coordinates of this pixel - + + "#ifdef GOURAUD", + "varying vec3 vLightDiffuse;", + "varying vec3 vLightSpecular;", + "#endif", + utils.standard_frag_vars, utils.rand, utils.edge, @@ -772,7 +907,8 @@ var Shaderlib = (function() { "#ifdef EXTRATEX", "gl_FragColor = tColor + (1.-tColor.a)*gl_FragColor;", "#endif", - THREE.ShaderChunk[ "lights_phong_fragment" ], + lighting, + utils.occlusion_fragment, "#endif", "}" ].join("\n"); @@ -785,6 +921,10 @@ var Shaderlib = (function() { if (opts.hasflat) { attributes.flatbump = { type: 'v4', value:null }; } + if (opts.faceted) { + attributes.facepos = { type: 'v3', value:null }; + attributes.facewm = { type: 'v4', value:null }; + } for (var i = 0; i < morphs-1; i++) { attributes['mixSurfs'+i] = { type:'v4', value:null}; attributes['mixNorms'+i] = { type:'v3', value:null}; @@ -793,6 +933,114 @@ var Shaderlib = (function() { return {vertex:header+vertShade, fragment:header+fragHead+fragMid+fragTail, attrs:attributes}; }, + //A line around the surface, drawn as the surface again, turned inside + //out and grown by a width of the screen: what is left of it where the + //surface itself covers it is the band around its edge and around the + //folds that face away. It is moved by the same unfolding, depth and + //flattening as the surface, so it follows wherever that goes. + outline: function(opts) { + var header = ""; + var morphs = opts.morphs; + if (opts.volume > 0) + header += "#define CORTSHEET\n"; + if (opts.hasflat) + header += "#define HASFLAT\n"; + if (opts.equivolume) + header += "#define EQUIVOLUME\n"; + //a dot of the line's width at each vertex rather than the grown + //surface, which rounds the corners of the line + if (opts.dots) + header += "#define DOTS\n"; + + var vertShade = [ + utils.thickmixer, + "uniform int bumpyflat;", + "uniform float bumpyflat_scale;", + "float f_bumpyflat = float(bumpyflat);", + "uniform float outlineWidth;", + //the canvas in pixels, so that the width is the same count of + //them up and across + "uniform vec2 viewport;", + + "attribute vec4 wm;", + "attribute vec3 wmnorm;", + "attribute vec4 auxdat;", + + utils.flatbump_attr, + + utils.mixer(morphs), + + "void main() {", + utils.thickmixer_main, + "#ifdef CORTSHEET", + "vec3 mpos = mix(position, wm.xyz, use_thickmix);", + "vec3 mnorm = mix(normal, wmnorm, use_thickmix);", + "#else", + "vec3 mpos = position;", + "vec3 mnorm = normal;", + "#endif", + "vec3 pos, norm;", + "mixfunc(mpos, mnorm, pos, norm);", + "#ifdef CORTSHEET", + "#ifdef HASFLAT", + //the relief of the flatmap, the way the surface takes it + "vec3 bumpvector = vec3(flatbump.w * bumpyflat_scale, 0., 0.);", + "pos += clamp(surfmix*"+(morphs-1)+". - "+(morphs-2)+"., 0., 1.) * mix(1., 0., use_thickmix) * f_bumpyflat * bumpvector;", + "#else", + "pos += clamp(surfmix*"+(morphs-1)+"., 0., 1.) * normalize(norm) * .62 * distance(position, wm.xyz) * mix(1., 0., use_thickmix);", + "#endif", + "#endif", + + "#ifdef DOTS", + //only at the vertices that face away, as the line is only the + //faces that do: the surface in front covers such a dot except + //where it rounds off a corner of the silhouette + "vec4 eye = modelViewMatrix * vec4(pos, 1.0);", + "vec3 toEye = projectionMatrix[2][3] == 0. ? vec3(0., 0., 1.) : normalize(-eye.xyz);", + "float facing = dot(normalize(normalMatrix * normalize(norm)), toEye);", + "gl_Position = facing > 0. ? vec4(2., 2., 2., 1.) : projectionMatrix * eye;", + "gl_PointSize = facing > 0. ? 0. : outlineWidth * viewport.y;", + "#else", + //grown across the screen rather than in the world, so that + //the line is the same width however near the surface is, + //and by as many pixels up as across + "vec4 clip = projectionMatrix * modelViewMatrix * vec4(pos, 1.0);", + "vec3 vnorm = normalize(normalMatrix * normalize(norm));", + "vec2 across = (projectionMatrix * vec4(vnorm, 0.)).xy * viewport;", + "if (length(across) > 0.0001)", + "clip.xy += normalize(across) * outlineWidth * viewport.y / viewport * clip.w;", + "gl_Position = clip;", + "#endif", + "}" + ].join("\n"); + + var fragShade = [ + "uniform vec3 outlineColor;", + "void main() {", + "#ifdef DOTS", + "if (length(gl_PointCoord - 0.5) > 0.5) discard;", + "#endif", + "gl_FragColor = vec4(outlineColor, 1.);", + "}" + ].join("\n"); + + var attributes = { + wm: { type: 'v4', value:null }, + wmnorm: { type: 'v3', value:null }, + auxdat: { type: 'v4', value:null }, + wmarea: { type: 'f', value:null }, + pialarea: { type: 'f', value:null }, + }; + if (opts.hasflat) + attributes.flatbump = { type: 'v4', value:null }; + for (var i = 0; i < morphs-1; i++) { + attributes['mixSurfs'+i] = { type:'v4', value:null}; + attributes['mixNorms'+i] = { type:'v3', value:null}; + } + + return {vertex:header+vertShade, fragment:header+fragShade, attrs:attributes}; + }, + surface_vertex: function(opts) { var header = ""; if (opts.rgb) @@ -815,6 +1063,17 @@ var Shaderlib = (function() { header += "#define HASFLAT\n" if (opts.equivolume) header += "#define EQUIVOLUME\n" + //the light on a face worked out from its own normal (flat) or at + //its corners (gouraud) rather than at every pixel of it; the + //color is already one to a face where the surface is drawn with + //a vertex per corner, since it is read off the vertices + if (opts.shading == "flat") + header += "#define FLATSHADE\n"; + else if (opts.shading == "gouraud") + header += "#define GOURAUD\n"; + //and the sky each vertex misses is taken out of the light on it + if (opts.occlusion) + header += "#define OCCLUSION\n"; var vertShade = [ THREE.ShaderChunk[ "lights_phong_pars_vertex" ], @@ -856,8 +1115,24 @@ var Shaderlib = (function() { "varying vec2 vUv;", "varying float vCurv;", "varying float vMedial;", + "#ifdef OCCLUSION", + "varying float vOcclusion;", + "uniform float occlusionStrength;", + "#endif", // "varying float vDrop;", + "#ifdef GOURAUD", + "#if MAX_DIR_LIGHTS > 0", + "uniform vec3 directionalLightColor[ MAX_DIR_LIGHTS ];", + "uniform vec3 directionalLightDirection[ MAX_DIR_LIGHTS ];", + "#endif", + "uniform vec3 diffuse;", + "uniform vec3 specular;", + "uniform float shininess;", + "varying vec3 vLightDiffuse;", + "varying vec3 vLightSpecular;", + "#endif", + utils.mixer(morphs), "void main() {", @@ -902,6 +1177,7 @@ var Shaderlib = (function() { "vec3 pos, norm;", "mixfunc(mpos, mnorm, pos, norm);", + utils.occlusion_main, "#ifdef CORTSHEET", "#ifdef HASFLAT", @@ -930,9 +1206,41 @@ var Shaderlib = (function() { "gl_Position = projectionMatrix * modelViewMatrix * vec4( pos, 1.0 );", + "#ifdef GOURAUD", + "vec3 gnorm = normalize(vNormal);", + "vec3 gview = normalize(vViewPosition);", + "vLightDiffuse = vec3(0.);", + "vLightSpecular = vec3(0.);", + "#if MAX_DIR_LIGHTS > 0", + "for (int i = 0; i < MAX_DIR_LIGHTS; i++) {", + "vec4 lDirection = viewMatrix * vec4(directionalLightDirection[i], 0.);", + "vec3 dirVector = normalize(lDirection.xyz);", + "float dirDiffuseWeight = max(dot(gnorm, dirVector), 0.);", + "vLightDiffuse += diffuse * directionalLightColor[i] * dirDiffuseWeight;", + "vec3 dirHalfVector = normalize(dirVector + gview);", + "float dirDotNormalHalf = max(dot(gnorm, dirHalfVector), 0.);", + "float dirSpecularWeight = max(pow(dirDotNormalHalf, shininess), 0.);", + "float specularNormalization = (shininess + 2.) / 8.;", + "vec3 schlick = specular + vec3(1. - specular) * pow(max(1. - dot(dirVector, dirHalfVector), 0.), 5.);", + "vLightSpecular += schlick * directionalLightColor[i] * dirSpecularWeight * dirDiffuseWeight * specularNormalization;", + "}", + "#endif", + "#endif", "}" ].join("\n"); + //as in surface_pixel: the face's own normal, or the light its + //corners were given + var lighting = THREE.ShaderChunk[ "lights_phong_fragment" ]; + if (opts.shading == "flat") { + lighting = lighting.replace("vec3 normal = normalize( vNormal );", [ + "vec3 normal = normalize(cross(dFdx(vViewPosition), dFdy(vViewPosition)));", + "normal *= sign(dot(normal, normalize(vViewPosition)));"].join("\n")); + } else if (opts.shading == "gouraud") { + lighting = "gl_FragColor.xyz = gl_FragColor.xyz * (emissive + " + + "vLightDiffuse + ambientLightColor * ambient) + vLightSpecular;"; + } + var fragShade = [ "#extension GL_OES_standard_derivatives: enable", //"#extension GL_OES_texture_float: enable", @@ -958,7 +1266,16 @@ var Shaderlib = (function() { // "varying float vDrop;", "varying float vCurv;", "varying float vMedial;", + "#ifdef OCCLUSION", + "varying float vOcclusion;", + "uniform float occlusionStrength;", + "#endif", "uniform float thickmix;", + + "#ifdef GOURAUD", + "varying vec3 vLightDiffuse;", + "varying vec3 vLightSpecular;", + "#endif", // utils.thickmixer, utils.standard_frag_vars, @@ -1008,7 +1325,8 @@ var Shaderlib = (function() { "#ifdef EXTRATEX", "gl_FragColor = tColor + (1.-tColor.a)*gl_FragColor;", "#endif", - THREE.ShaderChunk[ "lights_phong_fragment" ], + lighting, + utils.occlusion_fragment, "}" ].join("\n"); @@ -1036,6 +1354,185 @@ var Shaderlib = (function() { return {vertex:header+vertShade, fragment:header+fragShade, attrs:attributes}; }, + aligner_volume: function(opts) { + //Colors each fragment with the functional volume sampled at the + //fragment's world position. The aligner uses it for its slice + //planes (unlit) and for painting the volume onto the surface (lit). + //The world frame is the functional voxel grid in millimeters, so + //volxfm is a plain scaling and the planes show unresampled voxels. + //The lookup adds brightness, contrast, gamma and a flip to the + //vmin/vmax range of the colormap. + //sampler: nearest or trilinear + //lights: whether to apply phong lighting (false for the planes) + //depthmix: mix the position between the pial (position) and white + // matter (wm) surfaces with the depth uniform + //morphs: number of surfaces to interpolate between (anatomical, + // inflated, flat), as in surface_pixel. With more than one + // the surface is drawn where the mixer puts it and stays + // put as the alignment is edited, so the volume is sampled + // at the anatomical position through volxfm alone rather + // than at the world position. + var sampler = opts.sampler || "nearest"; + var morphs = opts.morphs || 1; + var header = ""; + if (opts.lights !== undefined && !opts.lights) + header += "#define NOLIGHTS\n"; + if (opts.depthmix) + header += "#define DEPTHMIX\n"; + if (morphs > 1) + header += "#define MORPHS\n"; + + var vertShade = [ + "#ifndef NOLIGHTS", + THREE.ShaderChunk[ "lights_phong_pars_vertex" ], + "#endif", + "uniform mat4 volxfm;", + "#ifdef DEPTHMIX", + "uniform float depth;", + "attribute vec4 wm;", + "attribute vec3 wmnorm;", + "#endif", + "#ifdef MORPHS", + utils.mixer(morphs), + "#endif", + + "varying vec3 vViewPosition;", + "varying vec3 vNormal;", + "varying vec3 vPos;", + + "void main() {", + "vec3 mpos = position;", + "vec3 mnorm = normal;", + "#ifdef DEPTHMIX", + "mpos = mix(position, wm.xyz, depth);", + "mnorm = mix(normal, wmnorm, depth);", + "#endif", + "#ifdef MORPHS", + "vec3 dpos, dnorm;", + "mixfunc(mpos, mnorm, dpos, dnorm);", + "vec4 world = modelMatrix * vec4(dpos, 1.0);", + "vPos = (volxfm * vec4(mpos, 1.0)).xyz;", + "vNormal = normalMatrix * dnorm;", + "#else", + "vec4 world = modelMatrix * vec4(mpos, 1.0);", + "vPos = (volxfm * world).xyz;", + "vNormal = normalMatrix * mnorm;", + "#endif", + "vec4 mvPosition = viewMatrix * world;", + "vViewPosition = -mvPosition.xyz;", + "gl_Position = projectionMatrix * mvPosition;", + "}", + ].join("\n"); + + var fragShade = [ + "uniform sampler2D colormap;", + "uniform float vmin;", + "uniform float vmax;", + "uniform float brightness;", + "uniform float contrast;", + "uniform float gamma;", + "uniform int flip;", + "uniform vec2 mosaic[2];", + "uniform vec2 dshape[2];", + "uniform float nslices;", + "uniform sampler2D data[4];", + "uniform vec3 outside;", + + "varying vec3 vPos;", + "#ifndef NOLIGHTS", + THREE.ShaderChunk[ "lights_phong_pars_fragment" ], + "#endif", + + utils.standard_frag_vars, + utils.samplers, + + "void main() {", + //Fragments outside the volume, and the padding between the + //mosaic tiles (NaN), get a flat color instead of data + "vec3 lo = vec3(-0.5);", + "vec3 hi = vec3(dshape[0].x, dshape[0].y, nslices) - 0.5;", + "bool inside = all(greaterThanEqual(vPos, lo)) && all(lessThanEqual(vPos, hi));", + "float value = "+sampler+"_x(data[0], vPos).r;", + "bool valid = inside && (value <= 0. || 0. < value);", + + //the range can be closed up from the menu, and a zero span + //would leave every fragment undefined + "float span = vmax - vmin;", + "span = span < 0. ? min(span, -1e-6) : max(span, 1e-6);", + "float norm = (value - vmin) / span;", + "norm = clamp(norm * contrast + brightness, 0., 1.);", + "norm = pow(norm, gamma);", + "if (flip == 1) norm = 1. - norm;", + "vec4 vColor = texture2D(colormap, vec2(norm, 0.));", + + "gl_FragColor = valid ? vec4(vColor.rgb, 1.) : vec4(outside, 1.);", + "#ifndef NOLIGHTS", + THREE.ShaderChunk[ "lights_phong_fragment" ], + "#endif", + "}" + ].join("\n"); + + var attributes = {}; + if (opts.depthmix) { + attributes.wm = { type: 'v4', value: null }; + attributes.wmnorm = { type: 'v3', value: null }; + } + for (var i = 0; i < morphs-1; i++) { + attributes['mixSurfs'+i] = { type: 'v4', value: null }; + attributes['mixNorms'+i] = { type: 'v3', value: null }; + } + + return {vertex:header+vertShade, fragment:header+fragShade, attrs:attributes}; + }, + + aligner_mesh: function(opts) { + //Flat colored surface for the aligner. The depth uniform picks the + //surface between pial (0) and white matter (1). With doClip set, + //only fragments within the slabs (one per world axis, enabled by + //slabMask) survive, which draws the intersection of the surface + //with the displayed slices; drawn as lines this gives the outline + //of the cortex on each slice. + var vertShade = [ + "uniform float depth;", + "attribute vec4 wm;", + "varying vec3 vWorld;", + + "void main() {", + "vec3 mpos = mix(position, wm.xyz, depth);", + "vec4 world = modelMatrix * vec4(mpos, 1.0);", + "vWorld = world.xyz;", + "gl_Position = projectionMatrix * viewMatrix * world;", + "}", + ].join("\n"); + + var fragShade = [ + "uniform vec3 color;", + "uniform float opacity;", + "uniform int doClip;", + "uniform vec3 slabLo;", + "uniform vec3 slabHi;", + "uniform vec3 slabMask;", + "varying vec3 vWorld;", + + "void main() {", + "if (doClip == 1) {", + "bvec3 inside = bvec3(", + "slabMask.x > .5 && vWorld.x >= slabLo.x && vWorld.x <= slabHi.x,", + "slabMask.y > .5 && vWorld.y >= slabLo.y && vWorld.y <= slabHi.y,", + "slabMask.z > .5 && vWorld.z >= slabLo.z && vWorld.z <= slabHi.z);", + "if (!any(inside)) discard;", + "}", + "gl_FragColor = vec4(color, opacity);", + "}" + ].join("\n"); + + var attributes = { + wm: { type: 'v4', value: null }, + }; + + return {vertex:vertShade, fragment:fragShade, attrs:attributes}; + }, + cmap_quad: function() { //Colormaps the full-screen quad, used for stage 2 of volume integration var vertShade = [ diff --git a/cortex/webgl/serve.py b/cortex/webgl/serve.py index a04c3d5cc..c5ec307ee 100644 --- a/cortex/webgl/serve.py +++ b/cortex/webgl/serve.py @@ -12,11 +12,13 @@ import binascii import base64 import datetime +import hmac import mimetypes import functools import threading +import uuid -from typing import Callable, Literal, cast, Generic, Union, Any +from typing import Awaitable, Callable, Literal, Optional, Sequence, cast, Generic, Union, Any if sys.version_info < (3, 10): from typing_extensions import ParamSpec @@ -34,12 +36,82 @@ cwd = os.path.split(os.path.abspath(__file__))[0] hostname = socket.gethostname() +#: The host to reach the servers at. Both loopback addresses resolve from this +#: name, so a browser finds the server whether it asks for IPv4 or IPv6. +LOOPBACK = "localhost" + +#: The names the servers listen for by default: the loopback addresses and +#: whatever this computer's own hostname resolves to, so that the page opens +#: as localhost, as 127.0.0.1 or under the name of the machine. +LOCAL = (LOOPBACK, hostname) + # on Windows the mimetypes module reads the extension to type mapping out of the # registry, where .js is frequently registered as text/plain. browsers refuse to # execute scripts served with that type, so force the correct type here. mimetypes.add_type("text/javascript", ".js") +def bind_local_sockets( + port: Optional[int], addresses: Union[str, Sequence[str], None] = LOCAL +) -> list[socket.socket]: + """Listening sockets on `port`, one per address `addresses` resolves to. + + Parameters + ---------- + port : int or None + The port to listen on; 0 or None asks the operating system for a free + one. Every socket ends up on the same port, whichever way it is chosen. + addresses : str or sequence of str or None, optional + The names to listen for, `LOCAL` by default. None listens on every + interface. A name that does not resolve is passed over, as is one + that resolves to an address another name already covers, so that a + computer whose hostname is its loopback address still gets a server. + + Returns + ------- + sockets : list of socket.socket + The bound sockets, to hand to ``HTTPServer.add_sockets``. + """ + if addresses is None: + return bind_sockets(0 if port is None else port) + if isinstance(addresses, str): + addresses = [addresses] + + # Resolve first and bind afterwards: two of the names can stand for the + # same address (a hostname is often the loopback address), and binding + # that one twice fails rather than being ignored. + wanted: list[str] = [] + for name in addresses: + try: + found = socket.getaddrinfo(name, None, type=socket.SOCK_STREAM) + except socket.gaierror: + continue + for info in found: + host = info[4][0] + # the address of an IP socket is a string; anything else is a + # family this cannot listen on + if isinstance(host, str) and host not in wanted: + wanted.append(host) + + sockets: list[socket.socket] = [] + port = 0 if port is None else port + failures: list[str] = [] + for host in wanted: + try: + bound = bind_sockets(port, address=host) + except OSError as exc: + failures.append("%s: %s" % (host, exc)) + continue + sockets.extend(bound) + # the first address that binds decides the port, so the rest listen + # on that one rather than each on a free port of its own + port = bound[0].getsockname()[1] + if len(sockets) == 0: + raise OSError("Could not listen on %s (%s)" % ( + ", ".join(addresses), "; ".join(failures) if failures else "no address resolved")) + return sockets + + def make_base64(imgfile: str) -> str: with open(imgfile, "rb") as img: mtype = mimetypes.guess_type(imgfile)[0] @@ -272,6 +344,66 @@ def make_static_url(cls, settings, path): return static_url_prefix + path +def token_cookie(port: int) -> str: + """Name of the cookie a page keeps its session token in. + + Cookies are not kept apart by port, so the name carries it: two servers + on this computer would otherwise be handed each other's tokens and turn + each other's pages away. + """ + return "pycortex-token-%d" % port + + +def same_token(given: Optional[str], wanted: str) -> bool: + """Whether `given` is the token `wanted`, compared in constant time.""" + if given is None: + return False + return hmac.compare_digest(given.encode("utf-8", "replace"), wanted.encode("utf-8")) + + +class TokenGuard: + """Turns away a request that does not carry the server's session token. + + The token stands for whoever started the server: it is in the address + printed for them and nowhere else, so a request that carries it came + from that address. The first request hands it over as ``?token=`` and + gets it back as a cookie, which the requests the page makes afterwards + carry by themselves. + + Mixed in front of every handler by `guarded`, so that a server reachable + under the name of the machine does not hand its filestore to whoever + else can reach the port. + """ + + #: the token to demand, and the port whose cookie carries it; set on the + #: subclass that `guarded` builds + session_token: str = "" + session_port: int = 0 + + def prepare(self) -> Optional[Awaitable[None]]: + if self.session_token: + handler = cast(tornado.web.RequestHandler, self) + # the address carries it, so only the query is read: a form field + # of the same name would otherwise stand in for it + if same_token(handler.get_query_argument("token", None), self.session_token): + # the page keeps it from here on, so the addresses it asks + # for afterwards do not have to carry it + handler.set_cookie(token_cookie(self.session_port), self.session_token) + elif not same_token(handler.get_cookie(token_cookie(self.session_port)), + self.session_token): + raise HTTPError(403, "the session token is missing or wrong") + # the handler underneath may answer with a coroutine of its own, which + # tornado waits for only if it is handed back: the viewer's own + # handlers hold a request there until the data has been packaged + return super().prepare() # type: ignore[misc,no-any-return] + + +def guarded(handler: type, token: str, port: int) -> type: + """`handler` with the session token demanded in front of it.""" + return type("Guarded" + handler.__name__, (TokenGuard, handler), + dict(session_token=token, session_port=port)) + + class ClientSocket(websocket.WebSocketHandler): def initialize(self, parent: "WebApp"): self.parent = parent @@ -310,6 +442,8 @@ def __init__( ] ], port: int, + address: Union[str, Sequence[str], None] = LOCAL, + token: Optional[str] = None, ): super(WebApp, self).__init__() self.handlers = handlers + [ @@ -331,10 +465,20 @@ def __init__( # The socket is bound immediately, so the OS accepts and queues client # connections in the backlog even before the IOLoop starts serving -- # eliminating the connect race too. - self._sockets = bind_sockets(port if port is not None else 0) + # `address` names what to listen for: the loopback addresses and this + # computer's own hostname by default, so the page opens as localhost, + # as 127.0.0.1 or under the name of the machine. Pass None to listen + # on every interface. + self._sockets = bind_local_sockets(port, address) # When port==0 the OS assigns the port; read the real value back so # callers can build a correct URL. self.port = self._sockets[0].getsockname()[1] + self.bound = {cast(str, sock.getsockname()[0]) for sock in self._sockets} + # What the server takes as proof that a request comes from whoever + # started it: it is in the address printed for them, the page it + # opens keeps it in a cookie, and every handler demands it. An empty + # token turns that off and answers anything that reaches the port. + self.token = uuid.uuid4().hex if token is None else token self.response: Queue[Union[str, bytes]] = Queue() self._request_id = 0 self._request_lock = threading.Lock() @@ -346,6 +490,45 @@ def __init__( self._ready = threading.Event() self.sockets: list[websocket.WebSocketHandler] = [] + def url(self, page: str = "", host: Optional[str] = None) -> str: + """The address to open `page` of this server at. + + It carries the session token, which the page keeps in a cookie, so + that the addresses it asks for afterwards do not have to. + + Parameters + ---------- + page : str, optional + The page to open, such as ``"mixer.html"``. + host : str, optional + The name to reach the server under, `host` by default. + """ + where = self.host if host is None else host + address = "http://%s:%d/%s" % (where, self.port, page.lstrip("/")) + if not self.token: + return address + #after whatever query the page already carries + return address + ("&" if "?" in page else "?") + "token=" + self.token + + @property + def host(self) -> str: + """The name to build a link to this server from. + + This computer's own hostname, which is the name a port forward from + another machine is usually set up under. A machine whose name resolves + to nothing, or to an address this server is not listening on, gets the + loopback name instead, so that the link works either way. + """ + if self.bound & {"0.0.0.0", "::"}: + return hostname + try: + found = socket.getaddrinfo(hostname, None, type=socket.SOCK_STREAM) + except socket.gaierror: + return LOOPBACK + if any(info[4][0] in self.bound for info in found): + return hostname + return LOOPBACK + @property def n_clients(self) -> int: num = len(self.sockets) @@ -358,7 +541,11 @@ def run(self): ioloop.clear_current() ioloop.make_current() self.ioloop = ioloop - application = tornado.web.Application(self.handlers, gzip=True) + # every handler, the websocket and the static files included, answers + # only a request that carries the session token + handlers = [(spec[0], guarded(spec[1], self.token, self.port)) + tuple(spec[2:]) + for spec in self.handlers] + application = tornado.web.Application(handlers, gzip=True) # If tornado version is 5.0 or greater, io_loop arg does not exist if tornado.version_info[0] < 5: self.server = tornado.httpserver.HTTPServer(application, io_loop=ioloop) diff --git a/cortex/webgl/template.html b/cortex/webgl/template.html index a97328365..4d25faab8 100644 --- a/cortex/webgl/template.html +++ b/cortex/webgl/template.html @@ -135,6 +135,8 @@ +{% block css %} +{% end %}