Skip to content

Proposal: a toolkit template for test-only state fixtures (check / set / revert, rows found by stable key, every write one guarded transaction), for journey preconditions the app's own UI cannot reach, such as a scenario clock at an exact minute #566

Closed

Description

@AlvinnYeo

Related files: skills/fixture-seeding.md (seeding: derive, measure, seed via the right channel; "never seed from inside the harness"), skills/e2e-harness-base.md ("Seed-data bootstrap", idempotent seed SQL), bin/fixture-manifest.sh · Toolkit: 92331e4 (origin/master, 23 Oct) · mxcli: v0.27.0 · Mendix: 11.14.0

Summary

fixture-seeding.md covers getting data into the app: derive what journeys need, measure the gap, seed through
the right channel with approval. It does not cover a different need that came up twice on one app: putting existing
data into one exact state for a journey, and putting it back, when the app's own UI cannot produce that state.

Examples from our build:

  • A simulated clock at an exact minute. The app has an admin "scenario clock" with Jump to a few fixed moments.
    Journeys need the clock at 13:08 and 13:12 exactly (a seed checks "exactly 12 minutes after the 13:00 moment"), and
    on the next allowed day. Jump cannot reach either; running the clock at speed across midnight never moves the
    scenario date. Without a fixture those journeys can only ever be INVALID.
  • A record in an intermediate status for a screenshot or a "page looks like wireframe state 2" check, where
    getting there through the UI would consume the seed other journeys need.

Each time, a test agent wrote a small shell script against the local PostgreSQL database, and each time it needed
the same safety rules. Twice is enough to make it a template.

The pattern (what both scripts converged on)

fixtures/<name>.sh check            read-only; exit 0 = in the expected state, 1/2/… = which other state or "drifted"
fixtures/<name>.sh set <args>       one transaction; guards first, RAISE and roll back on any failure
fixtures/<name>.sh revert           back to the state saved before the first un-reverted set

Rules:

  1. Find rows by a stable key, never by object id. A base-data reload re-creates rows with new ids. Singletons are
    found as "the only row" and the script refuses when there is not exactly one.
  2. Every write is one transaction whose guards RAISE. Guards run inside the transaction before any UPDATE, and
    lock the rows they read. Typical guards: exactly one row; the record is in the expected start state; nothing that
    races the write is running (a clock is paused, no task queue item waiting, a background-job lock not held
    recently); the target is legal for the app's own rules (same day, forward only, …). A refused write leaves nothing
    behind.
  3. Write what the app would write. Mirror the microflow that normally makes the change, field for field (name it
    in the header), including derived fields kept elsewhere, so the app cannot tell the difference. List what the SQL
    does not do that the microflow would (events not played, no client refresh, no follow-on jobs), and the
    manual step for each, in the header.
  4. Save before, mark used. The pre-change state goes to an untracked file only after the write committed; revert
    marks it used; a second revert refuses. revert refuses once anything has run on the changed state that it
    cannot undo (later events processed, rows created after the saved moment); then the answer is the app's own reset.
  5. check is what journeys assert, never what they call. Journeys keep asserting the state through their seeds
    (null = INVALID). The harness never runs set itself, which keeps fixture-seeding.md's "a harness that repairs
    its own fixture cannot report it missing" intact. A human or the main session runs set, with approval, as a
    precondition step in a run sheet.
  6. Dry-run before first live use. In one psql session, create temp tables that shadow every table the script
    touches (a temp table wins over public on the search path, so the script must not name a schema; grep for it),
    paste the write's body, read the result, roll back. Then check must show the real state unchanged.
  7. Local, test-only, no secrets printed. Local dev database only, default local credentials never echoed, no
    model change, no mxcli exec.

Suggested deliverable

  1. project-tests/e2e/fixtures/state-fixture.template.sh: the skeleton above, with check / set / revert, the
    stable-key lookup, the guard block (DO $$ … RAISE EXCEPTION 'FIXTURE REFUSED: …' … $$), the saved-state file,
    and a header block with the "writes what / does not do / manual steps after" sections to fill in.
  2. A section in fixture-seeding.md (or a short state-fixtures.md): when a state fixture is allowed (the UI cannot
    reach the state, or reaching it would consume shared seed state), the seven rules, and where the approval sits.
  3. Optional: validate-journeys.js or the run sheet template names a fixture precondition per journey
    ("precondition": { "fixture": "<name>", "check": 0 }), so a run can say "fixture not in state" as INVALID
    before walking.
  4. Optional, links journey-runner.js --positive-control runs every mutant back to back on the same data: a journey whose action consumes its seed state leaves the later mutants INVALID at "seeds resolved", and the control row then says "BROKEN PRECONDITION WENT GREEN" #560: --positive-control could call a fixture's revert/set between mutants as its reset hook.

How to retest

Copy the template for a singleton settings row, set a value, check reads it, revert restores it, a second
revert refuses; a set with a failing guard writes nothing (check unchanged).

Activity

  1. MendixMau commented on Oct 23, 2026

    @MendixMau
    Owner

    Triage: DISCUSS track. A new template, a new skill section that adds a step allowed to write to the database, and an optional new journey field (precondition.fixture) that the validator and runner would read, verified (file references)
    Problem: Some journeys need an exact state the app's own UI cannot reach, such as a scenario clock at a set minute or a record in an intermediate status. Those journeys can only ever be INVALID. skills/fixture-seeding.md covers getting data in (channels table :132-138, approval :185), not setting and reverting a state. The issue cites bin/fixture-manifest.sh, but the file is at project-bin/fixture-manifest.sh.
    Value: Med. It came up twice on one app, it is generic (no source-specific code, nothing that reads a BRD), and it is the reset hook #560 needs.
    Options:

    1. Add skill text only: a "state fixtures" section in fixture-seeding.md with the seven rules and where approval sits. Adds no always-loaded words (the file is on-demand). No change to any gate.
    2. Option 1 plus project-tests/e2e/fixtures/state-fixture.template.sh. Under "Shipping an instrument" this needs a field run on a captured case (the reporter's two scripts are candidates), a portability check (psql on Git Bash; avoid bash-4-only features) and a cited project. The repo's own rules prefer a prose spec plus worked examples over scaffolds that stamp files out.
    3. Option 2 plus a precondition.fixture journey field, checked as INVALID before the walk, and reused as journey-runner.js --positive-control runs every mutant back to back on the same data: a journey whose action consumes its seed state leaves the later mutants INVALID at "seeds resolved", and the control row then says "BROKEN PRECONDITION WENT GREEN" #560's per-mutant reset. This changes the runner, the validator and the control loop, and adds a check call and a reset per mutant.

    Recommendation: Option 1 now, with the reporter's script attached as a worked example rather than a template. Decide option 3 together with #560's reset hook.
    Decision needed from you: Should state fixtures be a documented pattern only (option 1), ship a template (option 2), or become a journey precondition and reset hook (option 3)? Reply "go option N" to build it.


    Generated by Claude Code

  2. MendixMau commented on Oct 26, 2026

    @MendixMau
    Owner

    The template shipped in #578 (squash a088b4e): project-tests/e2e/fixtures/state-fixture.template.sh (check / dry-run / set / revert) and § State fixtures in fixture-seeding.md. It is:

    • local only;
    • refuses a NULL current value;
    • never splices values into SQL.

    24 checks pass on a real PostgreSQL 16. A run on your scenario clock is still owed, tracked on #579.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions