Skip to content

Add a complete, executed worked example of the tripartite make() #205

Description

@dimitri-yatsenko

Summary

Add a complete, executed worked example of the three-part (tripartite) makemake_fetch / make_compute / make_insert — as a tutorial. Today the pattern is documented in prose (Computation Model, autopopulate.md §4.5) but there is no end-to-end, runnable example.

Why

  • The calling convention is a footgun. The framework invokes make_compute(key, *fetched_data) and make_insert(key, *computed_result) (the returned tuples are unpacked; make_insert receives only the computed result, not fetched). This is easy to get wrong — the docs themselves shipped an incorrect example recently. A copy-pasteable, correct template is the best antidote.
  • The semantics are subtle. make_fetch runs outside the transaction and is re-run and hash-verified inside it (so it must be bitwise-reproducible); make_compute runs once and is never re-verified (so it may be stochastic); make_insert runs inside the transaction. These boundaries are hard to grasp abstractly and land better as concrete, running code.
  • Real need. It's the recommended pattern for long-running computations (minutes–hours) that can't hold a transaction open — a common ML / signal-processing case.

What it should show

  • All three phases with the correct signatures and tuple return/unpack convention.
  • The re-fetch/verify behavior, ideally including the failure mode: inputs change mid-compute → rollback, no insert.
  • A make_compute that is legitimately non-deterministic (e.g. a seeded vs. unseeded stochastic step), to make "reproducibility = tracked derivation, not bitwise determinism" concrete.
  • How populate(make_kwargs=...) forwards to make_fetch.

Recommended form

  • Executed, not prose — a tutorial notebook run in CI (or a doctest), so it stays correct as the API evolves. A non-executed snippet would drift out of sync again (exactly the failure we just fixed).
  • Placement: Tutorials → Advanced (e.g. "Long computations with the three-part make"), or How-To → Computation. Pair with, and cross-link, autopopulate.md §4.5 and the Computation Model.

Notes

  • Scope/reach is modest (most pipelines use plain make(); tripartite is advanced), but the value for its audience is high and the reliability win — a tested example that can't drift — is the main argument.
  • Simulating a genuinely "long" computation cheaply in CI is the one awkwardness; a trivial compute with a comment noting where the real work would go is acceptable.

Targeted for a future release, not v2.3.1.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions