fix(trinet): a coordinator restart no longer pays the same (node, nonce) again

[dmitrii-f-t27]

Closes #906

Defect. At ea895b5, ledger.zig keeps the paid (node, nonce) set in memory and mesh.zig starts next_nonce at 1 on every start. A restarted coordinator pays the same (node, nonce) again. Reproduced on main: two coordinator runs, one honest emulated node, nonce 1 credited in both, 2 mTRI for one nonce. A receipt saved from an earlier run can also match a reissued nonce.

Fix. Spec first: gHashTag/t27#8426, PR gHashTag/t27#8427 (specs/trinet/node-work-credit.t27).

• src/trinet/journal.zig (new): the coordinator's durable record, JSON lines.
  • A nonce mark (blocks of 1024) is written before any nonce past it is issued. After a restart the coordinator resumes at the last mark, so no nonce is issued twice. The nonce space ends with NonceSpaceExhausted, never a wrap.
  • A credit line is written and fsynced before the credit counts. The paid set is rebuilt from the journal on start.
  • Reading fails closed: an unreadable file or a malformed line refuses to open. A torn last line (no newline) is skipped when reading. A run that will write refuses it with JournalTornTail (see e884a98 below).
  • Each credit carries its level: self_reported when the coordinator's operator is the node's owner or unknown, verified otherwise.
• ledger.zig: openJournal, write-before-credit in settle, new outcome not_recorded (not credited, not slashed: the fault is ours).
• mesh.zig: openJournal, a journal-backed freshNonce (now Error!u32), and a nonce this run did not issue is not_recorded.
• main.zig: TRINET_JOURNAL, TRINET_OWNER (default operator, as before) and TRINET_OPERATOR for fleet, demo and agent. Without TRINET_JOURNAL nothing changes.
• .github/workflows/conformance-frame-alignment.yml: the trinet-tests job runs zig test src/trinet/agent.zig -lc but did not trigger on src/trinet/**; it does now.

Checks run locally (Zig 0.16.0, macOS arm64):

• At 060ff68: zig test src/trinet/agent.zig -lc: 67/67 pass, 11 new (6 journal, 5 mesh: restart, owner level, forged answer, unwritable journal, unreadable journal).
• 9/9 code mutants fail a test: resume at 1, no mark written, any nonce credited, credit not written, write failure credits, owner vouches for self, restore forgets paid, malformed line ignored, unreadable read as empty.
• zig fmt --check on the four files: clean.
• Emulated test contour, no board (the board is in use by another session, so demo /nonexistent-no-board): run 1 credits 288 jobs; run 2 restores 288, resumes at nonce 1024, credits 288 more. Journal: 576 credits, 576 unique (node, nonce), 0 duplicates, marks 1024 and 2048, all self_reported, all physical:false.

Not in this PR. No payout path: a closed run still needs an attestation source the signers accept (the spec's open trust model). The tag still does not prove a board (DEVICE_BOUND_IDENTITY). fsync per credit costs throughput at UART rates; batching the fsync per batch is a follow-up.

Update at e884a98: two accounting defects fixed

Both defects broke the spec's "one credit per (node, nonce) across restarts". The fix is code only. The spec already has the rules: may_start_crediting, write-before-credit, and may_credit(..., !key_spent, journal_written). Before this commit the code did not meet them. No spec change is needed.

• Torn tail. A run that died mid-line left an unfinished last line. The next run opened the journal and appended with fopen("a"), so its first mark was glued onto the torn line. If the line was cut early, the glued line does not parse: the journal becomes unreadable and crediting stops for good. If it was cut later, the glued line parses as a credit and swallows the mark: the next start resumes at an older mark and reissues nonces. Now Ledger.openJournal returns JournalTornTail before any nonce or credit, and the file stays byte-identical. restore (read-only) still reports torn_tail. Nothing is truncated automatically. main.zig tells the operator what happened.
• Write before credit. The paid set grew after the credit line was fsynced. An allocation failure at that point left a durable credit that this run had not marked as paid. A second settle of the same (node, nonce) could then write a second line. Now the room is reserved before the write, and the insert after the write cannot fail. The spent key comes from journal.spentKey, the function restore uses.

Checks at e884a98, run locally (Zig 0.16.0, macOS arm64):

• zig test src/trinet/agent.zig -lc: 75/75 pass. The 7 new tests cover (68 at 060ff68):
  • a torn tail is refused twice and the file stays byte-identical;
  • the glued-line corruption is reproduced in both shapes;
  • a malformed line stops the coordinator;
  • unverifiable and tampered receipts never reach the journal;
  • a credit the journal refused is neither paid nor marked as paid;
  • no credit reaches disk without room in the paid set;
  • every credit line names the owner of its own node.
• zig fmt --check src/trinet/*.zig: clean.

Decision: self_reported records

Every accepted job is recorded at the settlement rate (1 mTRI) against the node owner's handle, self_reported included. This keeps today's behaviour, and nothing done on our own nodes is lost from the record. It does not open a payout or a double payment:

• Spec v1 (node-work-credit.t27) gates value on the level, not on the record. creditable(SELF_REPORTED) is false. may_request_attestation needs LEVEL_VERIFIED. mintable needs LEVEL_ATTESTED. Every line carries its level, so whatever reads the journal for a payout must apply those gates. A self_reported line is the run's tally, not a payable balance.
• One line per (node, nonce) is enforced by the journal across restarts (this PR). A line cannot be counted twice, at any level.
• This PR has no payout path (see "Not in this PR").

Dmitrii Fedorov (@dmitrii-f-t27)

[tri-claw]

Summary by t27-bees

This PR introduces a durable journal (journal.zig) so the coordinator remembers which (node, nonce) pairs it has already paid and which nonce block it has reserved across restarts. Previously the paid set lived only in memory and next_nonce reset to 1 on every start, causing double payment of the same nonce after a restart. The journal writes a mark line before issuing nonces in a new block and writes each credit line (fsynced) before counting it; on restart the paid set is rebuilt from the journal and nonces resume at the last mark.

Worth a look

• src/trinet/journal.zig:197 — The 512-byte stack buffer for bufPrint is large enough for current fields, but if tag_kind or level enum tag names grow, or if a future field is added, the buffer could overflow silently (bufPrint truncates with catch return Error.JournalWrite, but the line would be malformed on disk).
• src/trinet/journal.zig:260–290 — parse uses hand-rolled JSON extraction (indexOf for keys) rather than a JSON parser; it assumes the exact compact format written by writeCredit. If the file is ever edited by hand or written by a different tool with whitespace or field reordering, restore will fail with JournalMalformed and the coordinator will refuse to start.
• src/trinet/mesh.zig:148–160 — freshNonce writes the next mark when next_nonce >= nonce_mark. On a fresh journal (nonce_mark=0) this writes mark 1024 before issuing nonce 1, which is correct. However, if a crash occurs after the mark is fsynced but before the nonce is handed out, the mark is on disk but the nonce is never used; the next restart resumes at that mark, permanently skipping that block’s first nonce. At UART rates this is negligible, but it is a permanent gap in the nonce space.
• src/trinet/ledger.zig:347–350 — ensureUnusedCapacity(1) is called before the journal write, then putAssumeCapacity after. If the allocator fails on ensureUnusedCapacity, the credit is not written and OutOfMemory bubbles up; the node is not credited and not slashed. This is correct, but the error is not distinguished from other OutOfMemory paths, so an operator cannot tell whether the failure was in the paid-set map or elsewhere.
• src/trinet/main.zig:297–318 — openJournalFromEnv prints a detailed message on JournalTornTail and stops the run, leaving the file untouched. This is by design (automatic truncation is unsafe), but it means a single mid-write crash requires manual intervention before the coordinator can run again; there is no tooling in this PR to inspect or repair the torn tail.

---

_{Read: 5 files, +897 -7, the whole diff. Head 43aeda8. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.}

[github-actions]

X Brain Health Check

Score: 100.0/100
Status: 🟢 HEALTHY
Threshold: 80/100

X Brain is above merge threshold

[dmitrii-f-t27]

CI at head 5094fe4:

• trinet-tests (zig test src/trinet/agent.zig -lc, Ubuntu, Zig 0.16.0): pass on both the push run and the pull_request run. This is the job the workflow change now triggers for src/trinet/**.
• doc-refs and orphan-artefacts: red, and red on main at ea895b5 as well. The 9 new dangling references are in AGENTS.md, CLAUDE.md and research/*.md (files that moved to tri-net and t27); none is in a file this PR touches.

I am not merging while those two are red. Fixing them needs a decision on the research notes, which is not part of this PR.

Dmitrii Fedorov (@dmitrii-f-t27)

[github-actions]

🧪 Brain Stress Test Results

Score: not measured (no stress test prints a Score: line)
Status: 🟢 PASS
Threshold: 270/300

X Stress test passed

[dmitrii-f-t27]

Review at 5094fe4, answered in 060ff68 (the branch was rewritten upstream with identical trees; 5094fe4 is now e9c2486):

• journal.zig:148 / ledger.zig register, owner names. Fixed. With a journal open, register refuses an owner that is not a handle (InvalidOwner), and openJournal refuses an operator or an already-registered owner that is not one (InvalidHandle). A node can no longer join, work and never be credited. New test: "an owner name the journal cannot hold is refused at the door, not at payout"; removing any of the three checks fails it. zig test src/trinet/agent.zig -lc: 68/68.
• The other points (credit written before the paid set, mark written before the in-memory update, nonces outside [resumed_from, next_nonce), fail-closed open, no wrap at the end of the nonce space) describe the intended behaviour; no change.
• doc-refs and orphan-artefacts are red on main ea895b5 too and name no file of this PR (see the earlier comment). Not merging while they are red.

Dmitrii Fedorov (@dmitrii-f-t27)

[dmitrii-f-t27]

At head 060ff68ae2147991328581b314431e947052324e, run-syntax is red because of the gate's own
fixture self-test, not because of this PR's journal changes.

61 of 63 cases pass. Both failures, "main's build-matrix.yml before the fix" and "main's
iddr-golden-diff.yml before the fix", want rc 1 but get rc 2 with
cannot read: t.yml: no jobs mapping.

Those two cases load their input from the historical object e6eac090:<workflow path>. When that
read returns nothing, the code substitutes an empty string, so the gate parses empty YAML and
returns 2. I am not claiming a cause for the failed read.

Failing run:
https://github.com/gHashTag/trinity-fpga/actions/runs/38010467066/job/114089022932
Source:

trinity-fpga/tools/test_check_run_syntax.py

Line 114 in 060ff68

for name, path in (("main's build-matrix.yml before the fix", ".github/workflows/build-matrix.yml"),

Could you reconcile those historical inputs, by pinning or retaining their content or holding
them as immutable fixture data, after checking the related #893 and #894? Without weakening the
gate: no changed expected rc, no dropped cases, no disabling, no baseline bless.

Separately, doc-refs and orphan-artefacts remain red. A local full suite of 75 tests passed
on an unpublished working tree, which is secondary evidence only, not green CI.

Dmitrii Fedorov (@dmitrii-f-t27)