diff --git a/mkdocs.yaml b/mkdocs.yaml index c3a57043..81c03ad2 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -55,6 +55,7 @@ nav: - Distributed Computing: tutorials/advanced/distributed.ipynb - Custom Codecs: tutorials/advanced/custom-codecs.ipynb - Instances: tutorials/advanced/instances.ipynb + - Three-Part make(): tutorials/advanced/three-part-make.ipynb - How-To: - how-to/index.md - Setup: diff --git a/src/how-to/run-computations.md b/src/how-to/run-computations.md index b925d116..f3a2fc09 100644 --- a/src/how-to/run-computations.md +++ b/src/how-to/run-computations.md @@ -207,15 +207,19 @@ class LongComputation(dj.Computed): data = (RawData & key).fetch1('data') return (data,) - def make_compute(self, key, fetched): - """Perform computation (outside transaction)""" - (data,) = fetched + def make_compute(self, key, data): + """Perform computation (outside transaction). + + `data` is make_fetch's returned tuple, unpacked positionally. + """ result = expensive_computation(data) return (result,) - def make_insert(self, key, fetched, computed): - """Insert results (inside brief transaction)""" - (result,) = computed + def make_insert(self, key, result): + """Insert results (inside a brief transaction). + + `result` is make_compute's returned tuple, unpacked positionally. + """ self.insert1({**key, 'result': result}) ``` diff --git a/src/tutorials/advanced/three-part-make.ipynb b/src/tutorials/advanced/three-part-make.ipynb new file mode 100644 index 00000000..4fe647c5 --- /dev/null +++ b/src/tutorials/advanced/three-part-make.ipynb @@ -0,0 +1,574 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "bcdb90f1", + "metadata": {}, + "source": [ + "# The Three-Part `make()` — Long Computations Without Long Transactions\n", + "\n", + "A standard `make()` runs fetch → compute → insert inside **one** database\n", + "transaction. That is fine for quick derivations, but when the compute step\n", + "takes minutes to hours, holding a transaction open that long blocks other\n", + "writers, risks transaction timeouts, and pins resources. The **three-part**\n", + "(tripartite) `make()` splits the work into `make_fetch`, `make_compute`, and\n", + "`make_insert` so the slow part runs with **no transaction held** — while\n", + "DataJoint still guarantees referential integrity by re-fetching and\n", + "hash-verifying the inputs before it inserts." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "d50ffc26", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-18T19:34:08.120075Z", + "iopub.status.busy": "2026-07-18T19:34:08.119789Z", + "iopub.status.idle": "2026-07-18T19:34:08.430970Z", + "shell.execute_reply": "2026-07-18T19:34:08.430543Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[2026-07-18 19:34:08] DataJoint 2.3.1 connected to root@mysql:3306\n" + ] + } + ], + "source": [ + "import datajoint as dj\n", + "import numpy as np\n", + "\n", + "schema = dj.Schema(\"tripartite_demo\")\n", + "schema.drop(prompt=False) # reset so the example is re-runnable\n", + "schema = dj.Schema(\"tripartite_demo\")" + ] + }, + { + "cell_type": "markdown", + "id": "3488c971", + "metadata": {}, + "source": [ + "## An upstream source table\n", + "\n", + "`Recording` is a `Manual` table holding one 1-D waveform per recording (an\n", + "in-store ``). It is the declared input — everything the computation\n", + "reads comes from here." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "c2fe46b4", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-18T19:34:08.432124Z", + "iopub.status.busy": "2026-07-18T19:34:08.431951Z", + "iopub.status.idle": "2026-07-18T19:34:08.479209Z", + "shell.execute_reply": "2026-07-18T19:34:08.478530Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
recording_idsampling_rate_hzsignal
01000.0<blob>
11000.0<blob>
21000.0<blob>
\n", + " \n", + "

Total: 3

\n", + " " + ], + "text/plain": [ + "*recording_id sampling_rate_ signal \n", + "+------------+ +------------+ +--------+\n", + "0 1000.0 \n", + "1 1000.0 \n", + "2 1000.0 \n", + " (Total: 3)" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "@schema\n", + "class Recording(dj.Manual):\n", + " definition = \"\"\"\n", + " recording_id : int32\n", + " ---\n", + " sampling_rate_hz : float64\n", + " signal : # 1-D waveform, float64 samples\n", + " \"\"\"\n", + "\n", + "rng = np.random.default_rng(0)\n", + "Recording.insert(\n", + " {\n", + " \"recording_id\": i,\n", + " \"sampling_rate_hz\": 1000.0,\n", + " \"signal\": (rng.standard_normal(5000) * (i + 1)).astype(np.float64),\n", + " }\n", + " for i in range(3)\n", + ")\n", + "Recording()" + ] + }, + { + "cell_type": "markdown", + "id": "79d265f8", + "metadata": {}, + "source": [ + "## The computed table, in three phases\n", + "\n", + "`RecordingStats` derives amplitude statistics for each recording and, in a\n", + "Part table, the RMS of four equal-length windows.\n", + "\n", + "- **`make_fetch(key)`** reads only the declared upstream input and returns it\n", + " as a tuple. It runs **outside** the transaction and is re-run **inside** it;\n", + " its output is hash-verified, so it must be reproducible and must not write.\n", + "- **`make_compute(key, signal)`** is the slow part. It touches **no database**\n", + " (no transaction is open) and depends only on what `make_fetch` returned.\n", + " Because its result is inserted once and never re-verified, it may even be\n", + " stochastic.\n", + "- **`make_insert(key, master_row, window_rows)`** runs **inside** a brief\n", + " transaction and writes only to `self` and its Part table, atomically." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "6aa4e0b7", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-18T19:34:08.480336Z", + "iopub.status.busy": "2026-07-18T19:34:08.480233Z", + "iopub.status.idle": "2026-07-18T19:34:08.519012Z", + "shell.execute_reply": "2026-07-18T19:34:08.518514Z" + } + }, + "outputs": [], + "source": [ + "@schema\n", + "class RecordingStats(dj.Computed):\n", + " definition = \"\"\"\n", + " -> Recording\n", + " ---\n", + " n_samples : int32\n", + " mean_amp : float64\n", + " rms_amp : float64\n", + " peak_amp : float64\n", + " \"\"\"\n", + "\n", + " class Window(dj.Part):\n", + " definition = \"\"\"\n", + " -> master\n", + " window_id : int32\n", + " ---\n", + " window_rms : float64\n", + " \"\"\"\n", + "\n", + " def make_fetch(self, key, **kwargs):\n", + " # Phase 1 — read the declared upstream input; return a tuple.\n", + " signal = (Recording & key).fetch1(\"signal\")\n", + " return (signal,)\n", + "\n", + " def make_compute(self, key, signal):\n", + " # Phase 2 — no DB access; a long computation would live here.\n", + " master_row = {\n", + " **key,\n", + " \"n_samples\": int(signal.size),\n", + " \"mean_amp\": float(signal.mean()),\n", + " \"rms_amp\": float(np.sqrt(np.mean(signal ** 2))),\n", + " \"peak_amp\": float(np.max(np.abs(signal))),\n", + " }\n", + " window_rows = [\n", + " {**key, \"window_id\": i, \"window_rms\": float(np.sqrt(np.mean(w ** 2)))}\n", + " for i, w in enumerate(np.array_split(signal, 4))\n", + " ]\n", + " return (master_row, window_rows) # tuple -> unpacked into make_insert\n", + "\n", + " def make_insert(self, key, master_row, window_rows):\n", + " # Phase 3 — atomic write to self + Part, inside the transaction.\n", + " self.insert1(master_row)\n", + " self.Window.insert(window_rows)" + ] + }, + { + "cell_type": "markdown", + "id": "2e1c5357", + "metadata": {}, + "source": [ + "## Run it\n", + "\n", + "`populate()` orchestrates all three phases per key: it calls `make_fetch` and\n", + "`make_compute` with no transaction open, then opens a transaction, re-runs\n", + "`make_fetch`, verifies the inputs are byte-identical, and finally calls\n", + "`make_insert` and commits. If an upstream row changed mid-computation, the\n", + "verification fails with a `DataJointError` and nothing is inserted." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "d5ae3a37", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-18T19:34:08.520481Z", + "iopub.status.busy": "2026-07-18T19:34:08.520280Z", + "iopub.status.idle": "2026-07-18T19:34:08.684980Z", + "shell.execute_reply": "2026-07-18T19:34:08.684478Z" + } + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\r", + "RecordingStats: 0%| | 0/3 [00:00\n", + " .Table{\n", + " border-collapse:collapse;\n", + " }\n", + " .Table th{\n", + " background: #A0A0A0; color: #ffffff; padding:2px 4px; border:#f0e0e0 1px solid;\n", + " font-weight: normal; font-family: monospace; font-size: 75%; text-align: center;\n", + " }\n", + " .Table th p{\n", + " margin: 0;\n", + " }\n", + " .Table td{\n", + " padding:2px 4px; border:#f0e0e0 1px solid; font-size: 75%;\n", + " }\n", + " .Table tr:nth-child(odd){\n", + " background: #ffffff;\n", + " color: #000000;\n", + " }\n", + " .Table tr:nth-child(even){\n", + " background: #f3f1ff;\n", + " color: #000000;\n", + " }\n", + " #primary {\n", + " font-weight: bold;\n", + " color: black;\n", + " }\n", + " #nonprimary {\n", + " font-weight: normal;\n", + " color: white;\n", + " }\n", + "\n", + " /* Dark mode support */\n", + " @media (prefers-color-scheme: dark) {\n", + " .Table th{\n", + " background: #4a4a4a; color: #ffffff; border:#555555 1px solid; text-align: center;\n", + " }\n", + " .Table td{\n", + " border:#555555 1px solid;\n", + " }\n", + " .Table tr:nth-child(odd){\n", + " background: #2d2d2d;\n", + " color: #e0e0e0;\n", + " }\n", + " .Table tr:nth-child(even){\n", + " background: #3d3d3d;\n", + " color: #e0e0e0;\n", + " }\n", + " #primary {\n", + " color: #bd93f9;\n", + " }\n", + " #nonprimary {\n", + " color: #e0e0e0;\n", + " }\n", + " }\n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
recording_idn_samplesmean_amprms_amppeak_amp
05000-0.0045322476216566860.99540482730545073.899421730054339
150000.034312043435177832.0015631460133976.963674475871197
250000.0294216825857780633.005659165647679712.069475942671023
\n", + " \n", + "

Total: 3

\n", + " " + ], + "text/plain": [ + "*recording_id n_samples mean_amp rms_amp peak_amp \n", + "+------------+ +-----------+ +------------+ +------------+ +------------+\n", + "0 5000 -0.00453224762 0.995404827305 3.899421730054\n", + "1 5000 0.034312043435 2.001563146013 6.963674475871\n", + "2 5000 0.029421682585 3.005659165647 12.06947594267\n", + " (Total: 3)" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "RecordingStats.populate(display_progress=True)\n", + "RecordingStats()" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "f14c4cfd", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-18T19:34:08.687707Z", + "iopub.status.busy": "2026-07-18T19:34:08.687486Z", + "iopub.status.idle": "2026-07-18T19:34:08.694140Z", + "shell.execute_reply": "2026-07-18T19:34:08.693474Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
recording_idwindow_idwindow_rms
000.9817028961918416
011.013580648563692
020.9919901565252676
030.9940786199867229
\n", + " \n", + "

Total: 4

\n", + " " + ], + "text/plain": [ + "*recording_id *window_id window_rms \n", + "+------------+ +-----------+ +------------+\n", + "0 0 0.981702896191\n", + "0 1 1.013580648563\n", + "0 2 0.991990156525\n", + "0 3 0.994078619986\n", + " (Total: 4)" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "RecordingStats.Window & \"recording_id = 0\"" + ] + }, + { + "cell_type": "markdown", + "id": "0c83cf71", + "metadata": {}, + "source": [ + "## Testing fetch and compute directly\n", + "\n", + "Because `make_fetch` performs no writes and `make_compute` touches no\n", + "database, you can call them directly to inspect intermediate results —\n", + "something you cannot do with a standard `make()`. The insert step is\n", + "exercised only through `populate()`." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}