From e4ad1e5b096bc93531597d01b92c98bcac3064b8 Mon Sep 17 00:00:00 2001
From: Dimitri Yatsenko <dimitri.yatsenko@gmail.com>
Date: Wed, 15 Jul 2026 08:57:23 -0500
Subject: [PATCH 01/11] docs(computation-model): correct transaction guidance;
 add references
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

make() runs inside a transaction that populate() opens per key, so its inserts
are already atomic. The old 'Best Practices' item recommended wrapping inserts
in an explicit `with dj.conn().transaction:` block — which is not just
redundant but raises an error, since DataJoint does not support nested
transactions. Replace it with an explanation of make()'s built-in atomicity and
a WRONG example showing the nested-transaction error.

Also add a 'See also' section linking the AutoPopulate/Cascade/Diagram specs,
the run-computations and distributed-computing how-tos, and related concept
pages.
---
 src/explanation/computation-model.md | 39 +++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)

diff --git a/src/explanation/computation-model.md b/src/explanation/computation-model.md
index f9a99335..65299b33 100644
--- a/src/explanation/computation-model.md
+++ b/src/explanation/computation-model.md
@@ -319,15 +319,30 @@ def make(self, key):
         process_chunk(row['data'])
 ```
 
-### 3. Use Transactions for Multi-Row Inserts
+### 3. Don't Open a Transaction Inside `make()`
+
+`make()` already runs inside a transaction: `populate()` opens one per key before
+calling `make()` and commits it only if `make()` returns without error. Everything
+a `make()` inserts — multiple rows, and inserts into Part tables — is therefore
+already atomic. It all commits together, or rolls back together if `make()` raises.
+No explicit transaction is needed:
 
 ```python
 def make(self, key):
     results = compute_multiple_results(key)
+    # Already atomic — make() runs inside a transaction managed by populate()
+    self.insert(results)
+```
+
+Do **not** open your own transaction inside `make()`. DataJoint does not support
+nested transactions, so starting one while `make()`'s transaction is already active
+raises an error:
 
-    # All-or-nothing insertion
+```python
+def make(self, key):
+    # WRONG — a transaction is already in progress, so this raises an error
     with dj.conn().transaction:
-        self.insert(results)
+        self.insert(compute_multiple_results(key))
 ```
 
 ### 4. Test with Single Keys First
@@ -349,3 +364,21 @@ Segmentation.populate()
 4. **Three-part make** — For long computations without long transactions
 5. **Cascade deletes** — Maintain workflow integrity
 6. **Error handling** — Robust retry mechanisms
+
+## See also
+
+**Specifications**
+
+- [AutoPopulate](../reference/specs/autopopulate.md) — normative spec for `key_source`, the [make() reproducibility contract](../reference/specs/autopopulate.md#43-the-make-reproducibility-contract), the tripartite pattern, and job reservation.
+- [Cascade](../reference/specs/cascade.md) — restriction propagation and master–part integrity for cascading deletes.
+- [Diagram](../reference/specs/diagram.md) — the dependency graph that `populate()` and `key_source` are computed from.
+
+**How-to guides**
+
+- [Run computations](../how-to/run-computations.md) — practical `populate()` usage, restrictions, and options.
+- [Distributed computing](../how-to/distributed-computing.md) — parallel and multi-worker populate with job reservation.
+
+**Related concepts**
+
+- [Relational workflow model](relational-workflow-model.md) — how computation fits DataJoint's data model.
+- [Data pipelines](data-pipelines.md) — the pipeline abstraction that auto-populated tables extend.

From 214ed3f3c73480769aefa11ff4f0432687b34091 Mon Sep 17 00:00:00 2001
From: Dimitri Yatsenko <dimitri.yatsenko@gmail.com>
Date: Wed, 15 Jul 2026 09:03:50 -0500
Subject: [PATCH 02/11] docs(computation-model): surface the full five-rule
 make() contract
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The 'make() Contract' section listed only the three mechanical steps
(receives/computes/inserts); the full five-rule reproducibility contract was
referenced only in the following prose, so readers saw 'three points'. Enumerate
the five rules (populate-only; one entity per call; read only the upstream cone;
write only to self and Parts; no other result-affecting input) inline and link
the AutoPopulate §4.3 spec where the full contract lives. Also de-duplicate the
now-redundant link in 'Why the contract matters'.
---
 src/explanation/computation-model.md | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/src/explanation/computation-model.md b/src/explanation/computation-model.md
index 65299b33..dc20924d 100644
--- a/src/explanation/computation-model.md
+++ b/src/explanation/computation-model.md
@@ -41,6 +41,19 @@ The `make(self, key)` method:
 2. **Computes** results for that entity
 3. **Inserts** results into the table
 
+Those three steps are the basic mechanics. Beyond them, a well-behaved `make()`
+observes the full **make() reproducibility contract** — five rules that keep every
+result reproducible and `populate()` safely parallel:
+
+1. **Populate-only** — rows are produced only by `make()` through `populate()`, never inserted directly.
+2. **One entity per call, in isolation** — a `make(key)` computes exactly the entity named by `key` (plus its Part rows) and shares no state across calls.
+3. **Read only the upstream cone** — fetch only declared ancestors, restricted to the current `key` (exposed as `self.upstream`).
+4. **Write only to `self` and its Parts** — atomically, as one unit; any fan-out write elsewhere must record the source identity.
+5. **No other result-affecting input** — anything that changes *what* is computed must enter through a declared upstream table.
+
+The full contract — with rationale and the enforcement model — is specified in the
+[AutoPopulate reference §4.3, "The make() reproducibility contract"](../reference/specs/autopopulate.md#43-the-make-reproducibility-contract).
+
 DataJoint guarantees:
 
 - `make()` is called once per upstream entity
@@ -50,8 +63,7 @@ DataJoint guarantees:
 ### Why the contract matters
 
 These guarantees hold because a well-behaved `make()` observes a small set of
-rules — the **make() reproducibility contract** (specified in full in the
-[AutoPopulate reference](../reference/specs/autopopulate.md#43-the-make-reproducibility-contract)).
+rules — the **make() reproducibility contract** listed above.
 The organizing idea is a single **read/write boundary**: a `make(key)` reads only
 from its declared upstream dependencies, restricted to the current `key`, and
 writes only to `self` and its Part tables. Because each call sees a fixed,

From f4beb0d33e209c747d6f5c44f477a4c1a883dfd7 Mon Sep 17 00:00:00 2001
From: Dimitri Yatsenko <dimitri.yatsenko@gmail.com>
Date: Wed, 15 Jul 2026 09:05:18 -0500
Subject: [PATCH 03/11] docs(computation-model): fix single-key testing
 antipattern

The 'Best Practices' section recommended testing by calling make() directly
(Segmentation().make(key)). That bypasses populate(): it runs outside the
per-key transaction (no rollback on partial/failed make()), skips job
reservation and error capture, and writes to the DB as an uncontrolled side
effect. Recommend populate(key, max_calls=1) instead, which exercises the real
machinery. Add a note that a direct make() call could become a safe dry-run
only if/when a no-insert test/debug mode is added (future, not current).
---
 src/explanation/computation-model.md | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)

diff --git a/src/explanation/computation-model.md b/src/explanation/computation-model.md
index dc20924d..2f6cb2fc 100644
--- a/src/explanation/computation-model.md
+++ b/src/explanation/computation-model.md
@@ -357,17 +357,30 @@ def make(self, key):
         self.insert(compute_multiple_results(key))
 ```
 
-### 4. Test with Single Keys First
+### 4. Test on One Entity with `populate()`, Not `make()` Directly
+
+To try a computation on a single entity, restrict `populate()` and cap the call
+count. This runs the entity through the real machinery — the per-key transaction,
+error handling, and (if enabled) job reservation:
 
 ```python
-# Test make() on one key
+# Compute just one pending entity, end-to-end
 key = (Scan - Segmentation).fetch1('KEY')
-Segmentation().make(key)
-
-# Then populate all
-Segmentation.populate()
+Segmentation.populate(key, max_calls=1, display_progress=True)
 ```
 
+Do **not** call `make()` directly (e.g. `Segmentation().make(key)`) to test. It
+bypasses `populate()`: it runs **outside** the per-key transaction, so a partial
+or failed `make()` is not rolled back and can leave the table inconsistent; it
+also skips job reservation and error capture, and writes to the database as an
+uncontrolled side effect rather than as a managed, atomic unit.
+
+!!! note "Future: a no-insert debug mode"
+    Calling `make()` directly could become a safe way to dry-run a computation
+    once DataJoint adds a dedicated test/debug mode that runs `make()` without
+    inserting. That is a planned future capability, not current behavior — today,
+    a direct `make()` call really does write to the database.
+
 ## Summary
 
 1. **`make(key)`** — Computes one entity at a time

From 99e4282e4841ecd278f57a2754e8f0eedd93f9ec Mon Sep 17 00:00:00 2001
From: Dimitri Yatsenko <dimitri.yatsenko@gmail.com>
Date: Wed, 15 Jul 2026 09:14:43 -0500
Subject: [PATCH 04/11] docs(computation-model): extend contract to tripartite
 make; enable safe testing
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Document the phase responsibilities of the three-part make as an extension of the
reproducibility contract: make_fetch reads only (no compute, no insert),
make_compute is pure (no reads, no writes), make_insert writes only. Because the
read and compute phases never touch the database as a side effect, they can be
called and tested directly and safely — unlike make()/make_insert, which write.
Add that guidance to the single-key testing best practice.
---
 src/explanation/computation-model.md | 34 ++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/src/explanation/computation-model.md b/src/explanation/computation-model.md
index 2f6cb2fc..e894bb26 100644
--- a/src/explanation/computation-model.md
+++ b/src/explanation/computation-model.md
@@ -289,6 +289,27 @@ referential integrity is preserved by re-fetching and verifying inputs before
 insertion. If upstream data changed during computation, the job is cancelled
 rather than inserting inconsistent results.
 
+### Phase responsibilities
+
+Splitting `make()` into three phases extends the reproducibility contract with a
+strict division of labor — each phase does exactly one thing:
+
+- **`make_fetch(key)` — reads only.** Fetches the entity's upstream cone and
+  returns it. It performs no computation and no writes, and it must be
+  idempotent (DataJoint calls it twice — the re-fetch above).
+- **`make_compute(key, fetched)` — pure.** Computes the result from the fetched
+  inputs alone. It neither reads the database nor writes to it, so its output
+  depends only on its arguments.
+- **`make_insert(key, fetched, computed)` — writes only.** Inserts the result
+  into `self` and its Part tables. It performs no fetching or computation.
+
+This keeps the make() reproducibility contract intact under the split: reads
+still come only from the upstream cone (in `make_fetch`), writes still go only to
+`self` and its Parts (in `make_insert`), and the computation between them is a
+deterministic function of the fetched inputs. It also makes the read and compute
+steps **safely testable in isolation** (see [Best Practices](#best-practices)),
+since neither touches the database as a side effect.
+
 ### Benefits
 
 | Aspect | Standard `make()` | Three-Part Pattern |
@@ -375,6 +396,19 @@ or failed `make()` is not rolled back and can leave the table inconsistent; it
 also skips job reservation and error capture, and writes to the database as an
 uncontrolled side effect rather than as a managed, atomic unit.
 
+If your table uses the [three-part make](#the-three-part-make-model), you *can*
+test the read and compute steps directly and safely: `make_fetch(key)` only reads
+and `make_compute(key, fetched)` is pure, so neither writes to the database.
+Reserve `populate()` for exercising the write step (`make_insert`):
+
+```python
+# Safe: neither call writes to the database
+fetched = MyTable().make_fetch(key)
+computed = MyTable().make_compute(key, fetched)
+# Then exercise the full path (including make_insert) through populate()
+MyTable.populate(key, max_calls=1)
+```
+
 !!! note "Future: a no-insert debug mode"
     Calling `make()` directly could become a safe way to dry-run a computation
     once DataJoint adds a dedicated test/debug mode that runs `make()` without

From b40a47a90a0f7d2fbbd4b0ad2e390bb4e2bf2dbf Mon Sep 17 00:00:00 2001
From: Dimitri Yatsenko <dimitri.yatsenko@gmail.com>
Date: Wed, 15 Jul 2026 09:16:14 -0500
Subject: [PATCH 05/11] docs(computation-model): correct tripartite phase
 constraints

Narrow the phase rules to what actually holds: make_fetch may compute (only must
not insert); make_insert always runs inside the transaction, so it may fetch and
compute in addition to inserting. The only hard restrictions are (a) make_fetch
must not insert and (b) make_compute must neither fetch nor insert, since
make_compute runs outside the transaction. Fetch/compute remain safely testable
because make_fetch does not write and make_compute is pure.
---
 src/explanation/computation-model.md | 46 +++++++++++++++-------------
 1 file changed, 25 insertions(+), 21 deletions(-)

diff --git a/src/explanation/computation-model.md b/src/explanation/computation-model.md
index e894bb26..12d344c5 100644
--- a/src/explanation/computation-model.md
+++ b/src/explanation/computation-model.md
@@ -291,24 +291,28 @@ rather than inserting inconsistent results.
 
 ### Phase responsibilities
 
-Splitting `make()` into three phases extends the reproducibility contract with a
-strict division of labor — each phase does exactly one thing:
-
-- **`make_fetch(key)` — reads only.** Fetches the entity's upstream cone and
-  returns it. It performs no computation and no writes, and it must be
-  idempotent (DataJoint calls it twice — the re-fetch above).
-- **`make_compute(key, fetched)` — pure.** Computes the result from the fetched
-  inputs alone. It neither reads the database nor writes to it, so its output
-  depends only on its arguments.
-- **`make_insert(key, fetched, computed)` — writes only.** Inserts the result
-  into `self` and its Part tables. It performs no fetching or computation.
-
-This keeps the make() reproducibility contract intact under the split: reads
-still come only from the upstream cone (in `make_fetch`), writes still go only to
-`self` and its Parts (in `make_insert`), and the computation between them is a
-deterministic function of the fetched inputs. It also makes the read and compute
-steps **safely testable in isolation** (see [Best Practices](#best-practices)),
-since neither touches the database as a side effect.
+The split constrains what each phase may do, but the essential rules are narrow:
+
+- **`make_fetch(key)` must not insert.** It fetches the entity's inputs — and may
+  do some computation — then returns them. It runs *outside* the transaction and
+  is re-run *inside* it to confirm the inputs have not changed, so it must be free
+  of write side effects.
+- **`make_compute(key, fetched)` must neither fetch nor insert.** It also runs
+  outside the transaction, so it must be a pure function of the values
+  `make_fetch` returned — reproducible from its arguments alone, with no database
+  access at all.
+- **`make_insert(key, fetched, computed)` inserts the result** into `self` and its
+  Part tables. It *always* runs inside the transaction, so it may additionally
+  fetch data or compute there — those reads and the write are covered by the same
+  transaction, so this does not break the model.
+
+So the only hard restrictions are **(a) `make_fetch` must not insert** and
+**(b) `make_compute` must neither fetch nor insert** (because it runs outside the
+transaction). The make() reproducibility contract still holds overall — reads
+come from the upstream cone and writes go only to `self` and its Parts. And
+because `make_fetch` performs no writes and `make_compute` touches no database,
+both can be called and tested **directly and safely** (see
+[Best Practices](#best-practices)).
 
 ### Benefits
 
@@ -397,9 +401,9 @@ also skips job reservation and error capture, and writes to the database as an
 uncontrolled side effect rather than as a managed, atomic unit.
 
 If your table uses the [three-part make](#the-three-part-make-model), you *can*
-test the read and compute steps directly and safely: `make_fetch(key)` only reads
-and `make_compute(key, fetched)` is pure, so neither writes to the database.
-Reserve `populate()` for exercising the write step (`make_insert`):
+test the fetch and compute steps directly and safely: `make_fetch(key)` performs
+no inserts and `make_compute(key, fetched)` is pure, so neither writes to the
+database. Reserve `populate()` for exercising the insert step (`make_insert`):
 
 ```python
 # Safe: neither call writes to the database

From ff8aecf47e43678bde7ba589ab274fd3a21f415e Mon Sep 17 00:00:00 2001
From: Dimitri Yatsenko <dimitri.yatsenko@gmail.com>
Date: Wed, 15 Jul 2026 09:17:23 -0500
Subject: [PATCH 06/11] docs(computation-model): lead tripartite section with
 the simple one-job-per-phase rule
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

State the easy convention users should follow up front — make_fetch only
fetches, make_compute only computes, make_insert only inserts — then present the
narrower technical boundaries as the reason it is safe.
---
 src/explanation/computation-model.md | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/src/explanation/computation-model.md b/src/explanation/computation-model.md
index 12d344c5..5d8f41de 100644
--- a/src/explanation/computation-model.md
+++ b/src/explanation/computation-model.md
@@ -291,7 +291,12 @@ rather than inserting inconsistent results.
 
 ### Phase responsibilities
 
-The split constrains what each phase may do, but the essential rules are narrow:
+**The simple rule to follow: `make_fetch` only fetches, `make_compute` only
+computes, and `make_insert` only inserts.** Keep each phase to its named job and
+your table is always within the contract — no further reasoning needed.
+
+The framework actually enforces less than that, and knowing the exact boundaries
+helps when a computation doesn't fit the clean split:
 
 - **`make_fetch(key)` must not insert.** It fetches the entity's inputs — and may
   do some computation — then returns them. It runs *outside* the transaction and

From 217c592d34f43c00366e6dcf2bab91dc08ee1c61 Mon Sep 17 00:00:00 2001
From: Dimitri Yatsenko <dimitri.yatsenko@gmail.com>
Date: Wed, 15 Jul 2026 09:18:52 -0500
Subject: [PATCH 07/11] docs(computation-model): frame tripartite rules as
 validated, not runtime-enforced

The framework does not enforce the phase rules at runtime; they are part of the
make() reproducibility contract that the author follows and against which a
pipeline should be validated (at review or deploy time). Reworded both the
intro and the closing so neither implies runtime enforcement.
---
 src/explanation/computation-model.md | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/src/explanation/computation-model.md b/src/explanation/computation-model.md
index 5d8f41de..7eb80b39 100644
--- a/src/explanation/computation-model.md
+++ b/src/explanation/computation-model.md
@@ -295,8 +295,11 @@ rather than inserting inconsistent results.
 computes, and `make_insert` only inserts.** Keep each phase to its named job and
 your table is always within the contract — no further reasoning needed.
 
-The framework actually enforces less than that, and knowing the exact boundaries
-helps when a computation doesn't fit the clean split:
+These are part of the make() reproducibility contract: the framework does **not**
+enforce them at runtime — they are rules the pipeline author must follow, and
+against which a pipeline should be validated (at review or deploy time). The
+precise requirements are narrower than the one-job-per-phase rule above, which
+matters when a computation doesn't fit the clean split:
 
 - **`make_fetch(key)` must not insert.** It fetches the entity's inputs — and may
   do some computation — then returns them. It runs *outside* the transaction and
@@ -311,10 +314,12 @@ helps when a computation doesn't fit the clean split:
   fetch data or compute there — those reads and the write are covered by the same
   transaction, so this does not break the model.
 
-So the only hard restrictions are **(a) `make_fetch` must not insert** and
-**(b) `make_compute` must neither fetch nor insert** (because it runs outside the
-transaction). The make() reproducibility contract still holds overall — reads
-come from the upstream cone and writes go only to `self` and its Parts. And
+So the two requirements the contract places on the split are **(a) `make_fetch`
+must not insert** and **(b) `make_compute` must neither fetch nor insert**
+(because it runs outside the transaction) — again, observed by the author and
+checked by validation, not by the runtime. The make() reproducibility contract
+still holds overall — reads come from the upstream cone and writes go only to
+`self` and its Parts. And
 because `make_fetch` performs no writes and `make_compute` touches no database,
 both can be called and tested **directly and safely** (see
 [Best Practices](#best-practices)).

From 1d0e460852d193f582759157dfe51e7f4c8e8614 Mon Sep 17 00:00:00 2001
From: Dimitri Yatsenko <dimitri.yatsenko@gmail.com>
Date: Wed, 15 Jul 2026 09:26:03 -0500
Subject: [PATCH 08/11] docs(autopopulate): document the tripartite phase
 contract in the spec
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Add the per-phase rules to §4.5 (make_fetch must not insert; make_compute must
neither fetch nor insert; make_insert always runs in the transaction and may
also fetch/compute) as the normative source, and describe the tripartite
extension from the canonical §4.3 contract anchor with a cross-link. Framed as
contract rules validated at review/deploy time, not runtime-enforced —
consistent with the explanation in computation-model.
---
 src/reference/specs/autopopulate.md | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/src/reference/specs/autopopulate.md b/src/reference/specs/autopopulate.md
index e8003dad..8b0b95e0 100644
--- a/src/reference/specs/autopopulate.md
+++ b/src/reference/specs/autopopulate.md
@@ -316,6 +316,8 @@ The value of an auto-populated table is that every one of its rows is a *reprodu
 
 In one line: **read from `self.upstream`, write to `self`.** The read/write boundary is what makes each computed result self-contained and reproducible — every row was produced by a specific `make()` call over a specific, declared, key-restricted set of upstream entities.
 
+**Tripartite extension.** When `make()` is written as the [tripartite pattern](#45-tripartite-make-pattern) (`make_fetch` / `make_compute` / `make_insert`), the contract extends to the phases: `make_fetch` must not insert, `make_compute` must neither fetch nor insert (it runs outside the transaction), and `make_insert` — which always runs inside the transaction — performs the write and may also fetch or compute. The simple convention is one job per phase; see [§4.5](#45-tripartite-make-pattern) for details.
+
 !!! note "How the contract is upheld"
     The contract has historically been a **convention** — observed by discipline, not enforced. DataJoint 2.3 adds an ergonomic read surface that makes it easy to follow and to inspect: `self.upstream` gives each `make()` its key-restricted upstream cone directly, and [`Diagram.trace`](diagram.md) walks the same ancestry for any row after the fact. The framework does not itself enforce the contract at runtime; observing it remains the pipeline author's responsibility.
 
@@ -354,6 +356,14 @@ DataJoint 2.3 introduced `self.upstream` (and the underlying [`Diagram.trace`](d
 
 For long-running computations, use the tripartite pattern to separate fetch, compute, and insert phases. This enables better transaction management for jobs that take minutes or hours.
 
+**Phase contract.** The simple convention is one job per phase: `make_fetch` only fetches, `make_compute` only computes, and `make_insert` only inserts. The precise requirements the [make() reproducibility contract](#43-the-make-reproducibility-contract) places on the split — observed by the author and checked by validation, not enforced at runtime — are:
+
+- **`make_fetch(key)` must not insert.** It fetches the entity's inputs (and may perform some computation), then returns them. It runs *outside* the transaction and is re-run *inside* it to confirm the inputs are unchanged, so it must have no write side effects.
+- **`make_compute(key, fetched)` must neither fetch nor insert.** It also runs outside the transaction, so it must be a pure function of the values `make_fetch` returned — reproducible from its arguments alone, with no database access.
+- **`make_insert(key, fetched, computed)` inserts the result** into `self` and its Part tables. It *always* runs inside the transaction, so it may additionally fetch or compute there without breaking the model.
+
+Because `make_fetch` performs no writes and `make_compute` touches no database, both can be called and tested directly and safely; only `make_insert` (and full `populate()`) writes.
+
 **Method-based tripartite:**
 
 ```python

From 50fcd41d0408963a7b9f22f250945b4f5c2bbd26 Mon Sep 17 00:00:00 2001
From: Dimitri Yatsenko <dimitri.yatsenko@gmail.com>
Date: Wed, 15 Jul 2026 09:29:21 -0500
Subject: [PATCH 09/11] docs: make_fetch must be bitwise reproducible;
 make_compute may be stochastic
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Add the reproducibility asymmetry to the tripartite phase contract in both the
spec (autopopulate §4.5) and the explanation (computation-model): make_fetch's
output is re-fetched and hash-verified inside the transaction, so it must be
bitwise reproducible; make_compute runs once and is never re-verified, so it may
use stochastic functions and need not be bitwise reproducible.
---
 src/explanation/computation-model.md | 19 +++++++++++--------
 src/reference/specs/autopopulate.md  |  4 ++--
 2 files changed, 13 insertions(+), 10 deletions(-)

diff --git a/src/explanation/computation-model.md b/src/explanation/computation-model.md
index 7eb80b39..44d81939 100644
--- a/src/explanation/computation-model.md
+++ b/src/explanation/computation-model.md
@@ -301,14 +301,17 @@ against which a pipeline should be validated (at review or deploy time). The
 precise requirements are narrower than the one-job-per-phase rule above, which
 matters when a computation doesn't fit the clean split:
 
-- **`make_fetch(key)` must not insert.** It fetches the entity's inputs — and may
-  do some computation — then returns them. It runs *outside* the transaction and
-  is re-run *inside* it to confirm the inputs have not changed, so it must be free
-  of write side effects.
-- **`make_compute(key, fetched)` must neither fetch nor insert.** It also runs
-  outside the transaction, so it must be a pure function of the values
-  `make_fetch` returned — reproducible from its arguments alone, with no database
-  access at all.
+- **`make_fetch(key)` must not insert, and must be bitwise reproducible.** It
+  fetches the entity's inputs (and may do *deterministic* computation), then
+  returns them. It runs *outside* the transaction and is re-run *inside* it, where
+  its output is **hash-verified** against the first call to catch inputs that
+  changed mid-computation — so the two calls must return byte-identical data, and
+  it must have no write side effects.
+- **`make_compute(key, fetched)` must neither fetch nor insert, but need not be
+  deterministic.** It runs outside the transaction and depends only on the values
+  `make_fetch` returned (no database access). Because its output is inserted once
+  and never re-verified, it *may* use stochastic functions (random initialization,
+  sampling) — unlike `make_fetch`, it need not be bitwise reproducible.
 - **`make_insert(key, fetched, computed)` inserts the result** into `self` and its
   Part tables. It *always* runs inside the transaction, so it may additionally
   fetch data or compute there — those reads and the write are covered by the same
diff --git a/src/reference/specs/autopopulate.md b/src/reference/specs/autopopulate.md
index 8b0b95e0..8372698c 100644
--- a/src/reference/specs/autopopulate.md
+++ b/src/reference/specs/autopopulate.md
@@ -358,8 +358,8 @@ For long-running computations, use the tripartite pattern to separate fetch, com
 
 **Phase contract.** The simple convention is one job per phase: `make_fetch` only fetches, `make_compute` only computes, and `make_insert` only inserts. The precise requirements the [make() reproducibility contract](#43-the-make-reproducibility-contract) places on the split — observed by the author and checked by validation, not enforced at runtime — are:
 
-- **`make_fetch(key)` must not insert.** It fetches the entity's inputs (and may perform some computation), then returns them. It runs *outside* the transaction and is re-run *inside* it to confirm the inputs are unchanged, so it must have no write side effects.
-- **`make_compute(key, fetched)` must neither fetch nor insert.** It also runs outside the transaction, so it must be a pure function of the values `make_fetch` returned — reproducible from its arguments alone, with no database access.
+- **`make_fetch(key)` must not insert, and must be bitwise reproducible.** It fetches the entity's inputs (and may perform *deterministic* computation), then returns them. It runs *outside* the transaction and is re-run *inside* it, where its output is **hash-verified** against the first call to detect inputs that changed mid-computation — so the two calls must return byte-identical data. It must also have no write side effects.
+- **`make_compute(key, fetched)` must neither fetch nor insert, but need not be deterministic.** It runs outside the transaction and depends only on the values `make_fetch` returned (no database access). Because its output is inserted once and never re-verified, it *may* use stochastic functions (random initialization, sampling, non-deterministic solvers) — unlike `make_fetch`, its result need not be bitwise reproducible.
 - **`make_insert(key, fetched, computed)` inserts the result** into `self` and its Part tables. It *always* runs inside the transaction, so it may additionally fetch or compute there without breaking the model.
 
 Because `make_fetch` performs no writes and `make_compute` touches no database, both can be called and tested directly and safely; only `make_insert` (and full `populate()`) writes.

From 32d6a26b08ae1ebc4bca5895ffdc4f75732e940e Mon Sep 17 00:00:00 2001
From: Dimitri Yatsenko <dimitri.yatsenko@gmail.com>
Date: Wed, 15 Jul 2026 09:38:10 -0500
Subject: [PATCH 10/11] docs: reproducibility = tracked derivation (lineage),
 drop 'provenance' wording

Per the framework's terminology (DataJoint 2.0 paper): reserve 'provenance' for
the industry/platform category and use 'traceable to declared inputs' /
'derivation' for the core model. Reframe the make() contract's reproducibility
as complete lineage/derivation from declared inputs (not bitwise determinism),
and remove the 'provenance' terms I had introduced. Stochastic make_compute
remains allowed; make_fetch must be bitwise reproducible.
---
 src/explanation/computation-model.md | 8 +++++---
 src/reference/specs/autopopulate.md  | 5 ++++-
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/src/explanation/computation-model.md b/src/explanation/computation-model.md
index 44d81939..694edf3f 100644
--- a/src/explanation/computation-model.md
+++ b/src/explanation/computation-model.md
@@ -75,9 +75,11 @@ of declared inputs — which is exactly what makes results reproducible and
 This boundary is why the auto-populated tiers split into two:
 
 - **Computed** tables derive entirely from other pipeline tables. Every input is
-  itself tracked under referential integrity, so a Computed result is
-  reproducible from within the pipeline alone — re-running `make()` on the same
-  upstream data yields the same result.
+  itself tracked under referential integrity, so a Computed result is fully
+  traceable within the pipeline — re-running `make()` derives it from the same
+  declared inputs. It is bitwise-identical only if the computation is
+  deterministic; stochastic computations are allowed (see the
+  [reproducibility contract](../reference/specs/autopopulate.md#43-the-make-reproducibility-contract)).
 - **Imported** tables read a source the pipeline does *not* track (a file, an
   instrument, an API). They cannot be reproduced from the pipeline alone, so an
   Imported `make()` is responsible for recording the source's identity (path,
diff --git a/src/reference/specs/autopopulate.md b/src/reference/specs/autopopulate.md
index 8372698c..498ce35e 100644
--- a/src/reference/specs/autopopulate.md
+++ b/src/reference/specs/autopopulate.md
@@ -302,7 +302,7 @@ These requirements are the surface of a broader rule set — the make() reproduc
 
 ### 4.3 The make() reproducibility contract
 
-The value of an auto-populated table is that every one of its rows is a *reproducible* result: re-running `make()` on the same upstream data yields the same output. That guarantee rests on a small set of rules — the **make() reproducibility contract** — that every `make()` should observe. The rules apply to both `dj.Computed` and `dj.Imported` tables.
+The value of an auto-populated table is that every one of its rows is a *reproducible* result — **fully traceable to its declared inputs**: the row is derived solely from its declared, key-restricted upstream inputs plus the recorded `make()` code, with nothing entering from outside the dependency graph. That guarantee rests on a small set of rules — the **make() reproducibility contract** — that every `make()` should observe. The rules apply to both `dj.Computed` and `dj.Imported` tables.
 
 1. **Populate-only.** Rows are produced only by `make()`, invoked through `populate()` — never inserted directly. Direct inserts into an auto-populated table are rejected at runtime.
 
@@ -321,6 +321,9 @@ In one line: **read from `self.upstream`, write to `self`.** The read/write boun
 !!! note "How the contract is upheld"
     The contract has historically been a **convention** — observed by discipline, not enforced. DataJoint 2.3 adds an ergonomic read surface that makes it easy to follow and to inspect: `self.upstream` gives each `make()` its key-restricted upstream cone directly, and [`Diagram.trace`](diagram.md) walks the same ancestry for any row after the fact. The framework does not itself enforce the contract at runtime; observing it remains the pipeline author's responsibility.
 
+!!! note "Reproducibility means tracked derivation, not bitwise determinism"
+    The contract guarantees that every row is **derived only from its declared, key-restricted upstream inputs** — its derivation is fully tracked — not that `make()` is bitwise deterministic. **Stochastic computations are fully allowed.** Optimization and machine-learning algorithms are frequently non-deterministic, and DataJoint's model does not oppose this: re-running such a `make()` produces an equally valid, equally-traceable result, just not a byte-identical one. Bitwise reproducibility holds *only if the computation itself is reproducible* — for that, the pipeline author must control the sources of nondeterminism (fix random seeds, pin library versions, and so on). In the [tripartite pattern](#45-tripartite-make-pattern) this is why nondeterministic work belongs in `make_compute` (run once, never re-verified) and never in `make_fetch` (re-run and hash-verified, so it must be bitwise reproducible).
+
 ### 4.4 Accessing Source Data
 
 ```python

From 146d28d679a3d84039e00275f2cc83ae2750f267 Mon Sep 17 00:00:00 2001
From: Dimitri Yatsenko <dimitri.yatsenko@gmail.com>
Date: Wed, 15 Jul 2026 11:04:13 -0500
Subject: [PATCH 11/11] docs: fix tripartite make examples to match impl
 calling convention
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Per review (#202): the worked example and phase-contract bullets contradicted the
implementation. make() invokes make_compute(key, *fetched_data) and
make_insert(key, *computed_result) (autopopulate.py:283-311) — the returned
tuples are unpacked into positional args, and make_insert receives only the
computed result (not fetched). Corrected the computation-model worked example,
the How-It-Works pseudocode, the safe-testing snippet, and the phase-contract
bullets in both computation-model.md and autopopulate.md §4.5; also wrapped the
§4.5 example returns in tuples and fixed its make_fetch docstring.
---
 src/explanation/computation-model.md | 35 +++++++++++++++-------------
 src/reference/specs/autopopulate.md  | 17 ++++++++------
 2 files changed, 29 insertions(+), 23 deletions(-)

diff --git a/src/explanation/computation-model.md b/src/explanation/computation-model.md
index 694edf3f..1f1dd1fd 100644
--- a/src/explanation/computation-model.md
+++ b/src/explanation/computation-model.md
@@ -257,15 +257,19 @@ class SignalAverage(dj.Computed):
         raw_signal = (RawSignal & key).fetch1("signal")
         return (raw_signal,)
 
-    def make_compute(self, key, fetched):
-        """Step 2: Perform computation (outside transaction)"""
-        (raw_signal,) = fetched
+    def make_compute(self, key, raw_signal):
+        """Step 2: Perform computation (outside transaction).
+
+        The tuple returned by make_fetch is unpacked into positional args here.
+        """
         avg = raw_signal.mean()
         return (avg,)
 
-    def make_insert(self, key, fetched, computed):
-        """Step 3: Insert results (inside brief transaction)"""
-        (avg,) = computed
+    def make_insert(self, key, avg):
+        """Step 3: Insert results (inside brief transaction).
+
+        The tuple returned by make_compute is unpacked into positional args here.
+        """
         self.insert1({**key, "avg_signal": avg})
 ```
 
@@ -274,15 +278,14 @@ class SignalAverage(dj.Computed):
 DataJoint executes the three parts with verification:
 
 ```
-fetched = make_fetch(key)              # Outside transaction
-computed = make_compute(key, fetched)  # Outside transaction
+fetched = make_fetch(key)              # a tuple; outside transaction
+computed = make_compute(key, *fetched) # tuple unpacked; outside transaction
 
 <begin transaction>
-fetched_again = make_fetch(key)        # Re-fetch to verify
-if fetched != fetched_again:
-    <rollback>                         # Inputs changed—abort
+if make_fetch(key) != fetched:         # re-fetch and hash-verify inputs
+    <rollback>                         # inputs changed—abort
 else:
-    make_insert(key, fetched, computed)
+    make_insert(key, *computed)        # computed tuple unpacked
     <commit>
 ```
 
@@ -309,12 +312,12 @@ matters when a computation doesn't fit the clean split:
   its output is **hash-verified** against the first call to catch inputs that
   changed mid-computation — so the two calls must return byte-identical data, and
   it must have no write side effects.
-- **`make_compute(key, fetched)` must neither fetch nor insert, but need not be
+- **`make_compute(key, *fetched)` must neither fetch nor insert, but need not be
   deterministic.** It runs outside the transaction and depends only on the values
   `make_fetch` returned (no database access). Because its output is inserted once
   and never re-verified, it *may* use stochastic functions (random initialization,
   sampling) — unlike `make_fetch`, it need not be bitwise reproducible.
-- **`make_insert(key, fetched, computed)` inserts the result** into `self` and its
+- **`make_insert(key, *computed)` inserts the result** into `self` and its
   Part tables. It *always* runs inside the transaction, so it may additionally
   fetch data or compute there — those reads and the write are covered by the same
   transaction, so this does not break the model.
@@ -417,13 +420,13 @@ uncontrolled side effect rather than as a managed, atomic unit.
 
 If your table uses the [three-part make](#the-three-part-make-model), you *can*
 test the fetch and compute steps directly and safely: `make_fetch(key)` performs
-no inserts and `make_compute(key, fetched)` is pure, so neither writes to the
+no inserts and `make_compute(key, *fetched)` is pure, so neither writes to the
 database. Reserve `populate()` for exercising the insert step (`make_insert`):
 
 ```python
 # Safe: neither call writes to the database
 fetched = MyTable().make_fetch(key)
-computed = MyTable().make_compute(key, fetched)
+computed = MyTable().make_compute(key, *fetched)
 # Then exercise the full path (including make_insert) through populate()
 MyTable.populate(key, max_calls=1)
 ```
diff --git a/src/reference/specs/autopopulate.md b/src/reference/specs/autopopulate.md
index 5689fb9b..23fcc7cd 100644
--- a/src/reference/specs/autopopulate.md
+++ b/src/reference/specs/autopopulate.md
@@ -369,8 +369,8 @@ For long-running computations, use the tripartite pattern to separate fetch, com
 **Phase contract.** The simple convention is one job per phase: `make_fetch` only fetches, `make_compute` only computes, and `make_insert` only inserts. The precise requirements the [make() reproducibility contract](#43-the-make-reproducibility-contract) places on the split — observed by the author and checked by validation, not enforced at runtime — are:
 
 - **`make_fetch(key)` must not insert, and must be bitwise reproducible.** It fetches the entity's inputs (and may perform *deterministic* computation), then returns them. It runs *outside* the transaction and is re-run *inside* it, where its output is **hash-verified** against the first call to detect inputs that changed mid-computation — so the two calls must return byte-identical data. It must also have no write side effects.
-- **`make_compute(key, fetched)` must neither fetch nor insert, but need not be deterministic.** It runs outside the transaction and depends only on the values `make_fetch` returned (no database access). Because its output is inserted once and never re-verified, it *may* use stochastic functions (random initialization, sampling, non-deterministic solvers) — unlike `make_fetch`, its result need not be bitwise reproducible.
-- **`make_insert(key, fetched, computed)` inserts the result** into `self` and its Part tables. It *always* runs inside the transaction, so it may additionally fetch or compute there without breaking the model.
+- **`make_compute(key, *fetched)` must neither fetch nor insert, but need not be deterministic.** It runs outside the transaction and depends only on the values `make_fetch` returned (no database access). Because its output is inserted once and never re-verified, it *may* use stochastic functions (random initialization, sampling, non-deterministic solvers) — unlike `make_fetch`, its result need not be bitwise reproducible.
+- **`make_insert(key, *computed)` inserts the result** into `self` and its Part tables. It *always* runs inside the transaction, so it may additionally fetch or compute there without breaking the model.
 
 Because `make_fetch` performs no writes and `make_compute` touches no database, both can be called and tested directly and safely; only `make_insert` (and full `populate()`) writes.
 
@@ -386,19 +386,22 @@ class HeavyComputation(dj.Computed):
     """
 
     def make_fetch(self, key, **kwargs):
-        """Fetch all required data (runs in transaction).
+        """Fetch all required data (runs outside the transaction; re-run and
+        hash-verified inside it).
 
         kwargs are passed from populate(make_kwargs={...}).
+        Return a tuple — it is unpacked into make_compute's positional args.
         """
-        return (Recording & key).fetch1('raw_data')
+        return ((Recording & key).fetch1('raw_data'),)
 
     def make_compute(self, key, data):
         """Perform computation (runs outside transaction)."""
-        # Long-running computation - no database locks held
-        return heavy_algorithm(data)
+        # Long-running computation - no database locks held.
+        # Return a tuple — unpacked into make_insert's positional args.
+        return (heavy_algorithm(data),)
 
     def make_insert(self, key, result):
-        """Insert results (runs in transaction)."""
+        """Insert results (runs inside the transaction)."""
         self.insert1({**key, 'result': result})
 ```