fix: race-proof command-turn lifecycle + non-wedging final outbox (production incident root causes)

[jerryfane]

Summary

This PR fixes the root causes of a multi-day production incident in the herdres↔tendwire bridge: completed agent replies were never delivered and one stuck inbound message blocked all Telegram ingress for 24h. A three-model research panel (Claude Fable, GPT-5.6-sol, Kimi K3) independently confirmed the same root causes with file:line receipts; the full verdict is reproduced below in condensed form.

Root causes fixed

1. Duplicate turn rows / immortal submission claims. Command submissions wrote their pending-turn row in the receipt's terminal transaction, after pane input was sent. The adapter (2s scheduler) could observe the real turn first; the two rows get different ids by construction (source_turn_id-hash vs origin_command_id-hash), linkage is frozen post-insert, and no sweeper/TTL existed — so the active submission row was immortal and, because the turns list orders by insertion sequence only, it permanently shadowed the completed row. Downstream consumers suppressed every real final as "historical" and ingress trackers waited forever.
2. Outbox head-of-line wedges. poll_connector_outbox blocked later final_ready rows behind ANY earlier row not in (delivered,superseded) — including dead_letter, awaiting_ack, leased, deferred — globally per host+connector. awaiting_ack could never be reclaimed (plan-commit cleared the lease fields the reclaimer scans). Attempt accounting couldn't distinguish real failures from ack-timeout reclaims, so slow-but-healthy plans dead-lettered.

Changes

Turns lifecycle (schema v15):

• Pending-turn claim is written in the send_started transaction, before pane I/O, with a pre-input-failure cleanup path (post-input failures stay uncertain by design).
• Bidirectional adopt-in-place: observation ingestion updates the matched command row (keeping turn_id + list_sequence); the submission upsert adopts a matching recent observed row. Exact-normalized-text matching, recency-bounded (60s window), tombstone-aware, ambiguity fails closed to previous behavior.
• Tombstone reconciliation (payload superseded_by_turn_id/superseded_at, list-filtered, never physically deleted) with grace/hard TTL sweeping (configurable TENDWIRE_TURN_CLAIM_HARD_TTL_SECONDS, default 24h) so downstream trackers always terminate. Rate-limited lazy sweep + scheduler-tick sweep.
• Accepted envelopes return the canonical turn_id (supersession chains followed, cycle-safe) and a live observed_turn_state.
• One-time migration heals existing stuck command rows (exactly-one matching, used_done tracking, ambiguity skipped, traversal generation bumped once).

Connector outbox (schema v16 + v17):

• dead_letter and awaiting_ack are non-blocking for the root FIFO; ordering is scoped per worker via a new backfilled ordering_key column (unresolvable rows get per-row orphan: partitions — never a shared sentinel). Parts inherit their root's enqueue-time key; source-less recovery plans keep their FIFO place.
• awaiting_ack rows carry deadlines, extended on part-ACK progress; expired ones are reclaimed via plan-recover, skipping jobs with live leases; ack-timeout reclaims do not consume the delivery-failure budget.
• Server-side turn-final lease cap (default 300s, configurable) + connector.renew / connector.release operations; eager reclaim on the daemon tick behind a read-only due-check.

Testing

Full suite green at each stage (2,811 passed / 2 skipped at head; the only recurring failure in our sandbox is a benchmark privacy-scan false positive on short PIDs, byte-identical file untouched). Review process: each PR went through multi-agent review with adversarial verification; every MUST-fix item carries a regression test that was mutation-verified (test fails against the pre-fix body). Known follow-ups (test-quality only) are noted in jerryfane#6's review comments.

Also merged: your four recent main fixes (52dd236, 46abd8c, d846741 taken as-is; decisions-calibration files resolved to your live-verified versions). One deliberate supersession: 9919e76 scoped final head-of-line blocking by payload stable_key — this PR's ordering_key column materializes the same owner identity at enqueue time (surviving worker re-keying, with per-row orphan partitions for unresolvable rows), so the payload-JSON clauses were dropped as subsumed. One semantic change relative to 9919e76: dead_letter no longer blocks even its own owner's later finals (a poisoned reply must never wedge its worker; recovery goes through retry/supersede) — your test_dead_letter_final_blocks_only_its_stable_owner was adapted accordingly (now test_dead_letter_final_blocks_no_later_finals). Happy to revisit if you want dead_letter to stay blocking within an owner.

Production notes

• The observed 900s prepare leases came from the client side (herdres env), not a repo default — herdres now requests 120s.
• Migration path v14→v17 tested end-to-end including interrupted/resumed/re-applied transitions.

🤖 Generated with Claude Code