feat: production-proven improvements — instant linking, delivery-lane hardening, turn-API consumer

[jerryfane]

Production-proven delivery and Herdr turn-API hardening

⚠️ Runtime requirement: the turn-API features need a Herdr build that has the turn API

The Herdr turn API consumed here — pane.turns (since-N replay), the
pane.turn_completed event, and per-pane turn counters/epochs — does not exist in
official Herdr 0.7.5. It is our addition, living in our Herdr fork:
jerryfane/herdr (design + adopted spec:
jerryfane/herdr#1), running in our
production deployment since 2026-07-23.

On stock Herdr 0.7.5 the turn API stays disabled and the connection stays healthy.
This is now verified against the real protocol response rather than asserted: the
capability probe sends pane.turns once per connection, stock Herdr answers with an
id-less invalid_request envelope whose message begins
invalid request: unknown variant, and that exact shape is tolerated for the probe
call only and classified as method-unsupported. Ordinary requests keep strict
correlation (an id-less error on a normal request is still rejected — asserted by
test). A regression built on the raw envelope proves the client remains usable
afterwards, that no pane.turn_completed subscription is requested, and that no turn
watermark rows are written.

(Earlier revisions of this PR claimed inertness without having validated it against a
live stock server; @plotarmordev's review caught that the probe would instead
protocol-degrade the event loop. Thank you — fixed in a3cab65.)

The other improvements in this PR are independent of the fork and take effect on
stock Herdr: the delivery-lane hardening (ack deadlines, attempt caps, drain wake,
retention floor) and the volatile-field fingerprint exclusion — the latter is worth
taking on its own merits, since it also protects any future adopter of a turn-style
API from the per-turn identity churn we hit in production.

We intend to propose the turn API upstream to
ogulcancelik/herdr once its soak record
matures; if it is accepted there, this consumer needs no changes. Until then, adopting
the turn-API half means running our Herdr fork.

Summary

This contribution curates the production-proven improvements developed after
upstream 6975c51 onto the upstream schema and Herdr 0.7.5 compatibility
model. It keeps new turn-API behavior capability-gated, hardens the connector
delivery lane against permanent head-of-line stalls, and removes volatile
turn counters at the shared identity-model choke point.

Instant submission linker (downstream PR #40)

Not ported in this branch. Upstream 6975c51 still uses the legacy
command-claim model and has no turn_submissions ledger, dual/observed turn
model, link-window graph, or submission-link sweep. PR #40 is a targeted change
to that later shadow-linker foundation; copying it here would require importing
the entire intervening submission-ledger series and would no longer be a clean
post-6975c51 contribution.

The upstream anchor behavior remains intentionally untouched:
test_submission_first_envelope_reflects_observation_during_send still proves
that an observation created during a single send is authoritative and is
returned by the accepted command envelope. In the downstream shadow-linker
deployment, the same single-open-component case moved from the 60-second
window boundary to the first sweep (2 seconds in the deterministic regression,
normally within the roughly 5-second poll cadence). Identical duplicate sends,
uncertain/stale sends, and multi-candidate components still wait for the full
window so ambiguity can surface before attribution.

Connector outbox delivery lane

• Persists a 60-second ACK deadline in both private state and
  next_attempt_at, so the one-second reclaim sweep wakes without a consumer
  poll.
• Reclaims expired or malformed awaiting_ack state, preserves delivered
  prefix evidence, and retries the source in-place with a backoff capped at 30
  seconds.
• Keeps the retrying source at the ordering-lane head; later finals cannot
  overtake it.
• Applies attempt caps and moves exhausted work to dead_letter.
• Reports due age, overdue ACKs, and a 30-second drain-starvation signal in
  daemon health.
• Bounds retention without pruning rows inside the fresh-row safety floor.

Production motivation was an eight-hour head-of-line zombie: one durable
awaiting_ack root had no actionable wake-up deadline and blocked later finals
on the same ordering key. Recovery retained the original source and delivery
identity, and the incident window closed with zero lost final deliveries. The
read-only autopsy also separated 658 cumulative dead-letter rows into failed
plan parts, retryable anchors, migration holds, and one generic row instead of
treating the count as a single new failure burst.

More detail is in docs/evidence/outbox-delivery-lane.md.

Turn-API volatile-field fingerprint exclusion

Herdr turn observations can carry turn, turn_epoch,
last_completed_turn, any turn_* field, outcome, and
state_change_seq. These values describe activity, not worker or space
identity. They are now stripped recursively in _status_and_meta, the shared
model construction choke point, so every current and future ingestion lane is
protected.

This addresses a measured production identity churn rate of roughly two new
fingerprints per second per worker. Regressions cover direct model
construction, nested metadata, and adapter observations while retaining
nonvolatile metadata.

Capability-gated Herdr turn-API consumer

• Probes pane.turns before advertising pane.turn_completed; Herdr 0.7.5
  remains on its existing subscription shape when the capability is absent.
• Stores per-pane epoch/turn watermarks and completion provenance in the
  upstream-adapted v18 -> v19 migration.
• Baselines a pane with no watermark without inferring historical completions.
• Replays missed completions after reconnect and performs a second replay after
  subscription to close the probe-to-subscribe race.
• Records truncation, epoch changes, gaps, and watermark-ahead states as
  completeness breaks instead of manufacturing a complete history.
• Quarantines malformed completion records during batch flush without
  poisoning valid events in the same batch.
• Treats non-terminal semantic refresh statuses as skip-with-diagnostic, then
  advances durable provenance. This removes the production crash loop while
  keeping the missing semantic attachment visible.
• Keeps capability probing and per-pane replay failure handling decoupled, so
  one unavailable pane cannot disable healthy panes.
• Retains completion provenance with both an age floor and a count bound; fresh
  rows are never deleted solely to satisfy the count limit.

The consumer preserves the existing zero-loss posture: reconnect replays from
the last durable watermark, live events never advance without a baseline, and
completeness breaks are explicit rather than silently skipped.

Verification

• python3 -m compileall -q src tests
• Focused outbox/config/daemon/release suite: 289 passed, 1 skipped.
• Focused Herdr turn consumer/fingerprint suite: 393 passed, 1 skipped after
  the upstream import adaptation.
• Schema/retention migration regressions, including v18 -> v19 and the fresh
  row floor, pass.
• Full upstream suite: pending final run.

Scope note

The instant shadow-linker itself is the only exclusion. Its required ledger and
turn-model foundation does not exist on upstream 6975c51; the other three
features are adapted directly to upstream conventions and schema numbering.

🤖 Generated with Claude Code

[plotarmordev]

Blocking compatibility issue found against the live stock Herdr 0.7.5 protocol. A pane.turns probe returns an id-less envelope with code invalid_request and message beginning invalid request: unknown variant pane.turns. HerdrSocketClient.request currently rejects that envelope as uncorrelated before _probe_turn_api can classify it, and _turn_api_method_unsupported does not classify this invalid_request shape anyway. As a result, deploying this PR before the Herdr fork would protocol-degrade/reconnect the event loop instead of safely disabling the turn API. Please add a probe-only narrow tolerance for this exact validated unknown-method response, keep ordinary requests strict, classify the observed invalid_request/unknown-variant response as unsupported, and add a regression using the actual raw envelope that proves the connection remains usable, no completion subscription occurs, and no turn watermarks are written. Validation otherwise looks good: full suite reached 2939 passed, 1 skipped; the two benchmark failures were environment-only and the benchmark file passes 7/7 when TENDWIRE_BENCHMARK_HERDRES_ROOT is set.

[jerryfane]

Thank you — this is exactly the finding we could not have made ourselves: our production runs the Herdr fork, so the stock-0.7.5 probe path was only ever exercised against simulated clients, never the real invalid request: unknown variant pane.turns envelope. That makes the "provably inert on stock Herdr" claim in the PR description wrong as written, and we appreciate you catching it against a live server rather than taking our word for it.

Fix in progress, matching your requirements exactly: probe-only narrow tolerance for that validated unknown-method shape (ordinary requests stay strict — the negotiation-scoped pattern from the 0.7.5 compat work), classification of invalid_request + unknown variant as method-unsupported, and a regression built on the raw envelope you reported asserting the connection stays usable afterward, no pane.turn_completed subscription is requested, and no turn watermarks are written — plus an assertion that ordinary id-less invalid_request responses are still rejected, so the tolerance cannot widen by accident. We are landing the same fix on our fork, since it carries the identical latent bug.

The PR description callout will be corrected to describe verified behavior once the regression proves it. Thanks also for confirming the benchmark failures as environment-only.

[jerryfane]

Pushed a3cab65 addressing the blocking finding.

What changed

• Probe-only tolerance. HerdrSocketClient.request gained an allow_uncorrelated keyword that is passed only from the pane.turns capability probe. Ordinary requests are unchanged and still reject id-less error envelopes — asserted by a new test (test_ordinary_request_rejects_idless_unknown_variant_error), so the tolerance cannot widen by accident.
• Unsupported classification. The observed response is matched on code == "invalid_request" and a invalid request: unknown variant message prefix (not a loose substring), and classified as method-unsupported so the turn API disables cleanly.
• Raw-envelope regression (test_turn_api_idless_unknown_variant_probe_is_inert), built on the exact envelope from your report, asserting all three of your requirements: an ordinary request succeeds on the same client after the probe (connection remains usable), the subscription request contains no pane.turn_completed entry, and herdr_turn_watermarks is empty. A companion test pins the classifier to the exact code + prefix.

PR description corrected. The top callout previously claimed inertness on stock Herdr as if verified — it was not, and your review was right to catch it against a live server. It now describes the validated behavior and notes what was fixed.

Local run on the rebased branch: 2941 passed, 2 skipped; the two benchmark failures reproduce on untouched upstream main here and pass with TENDWIRE_BENCHMARK_HERDRES_ROOT set, matching your validation. Branch is rebased on current main, one commit, mergeable.

Re-review welcome whenever convenient.

[plotarmordev]

Re-review against the live stock Herdr 0.7.5 server found one remaining exact-shape mismatch. The live response is {"id":"","error":{"code":"invalid_request","message":"invalid request: unknown variant pane.turns, expected one of ..."}}. This head accepts only "id" not in envelope in both validate_response and _read_response, while the new raw_probe_error regression omits id. Therefore the actual empty-string-id response is still rejected before capability classification. Please make the probe-only method tolerance require envelope.get("id") == "" (or narrowly accept both missing and empty if needed), update the regression to the exact live empty-string envelope, and keep ordinary requests strict. Not merged pending that correction.

[jerryfane]

Fixed in a8bd1f5. You were right on both counts, and the fixture was the root of it — our regression asserted a shape we had inferred rather than the one your server actually sends, so it passed while production reality would still have failed.

Corrected

• Probe-only tolerance now accepts the id being absent or empty string, matching the live {"id":"", "error":{...}} envelope. Ordinary requests remain strictly correlated (unchanged assertion).
• While updating the fixture we hit a second mismatch in the same predicate: your live message backticks the method (invalid request: unknown variant \pane.turns`, expected one of ...), while our prefix check expected the bare name — it would have failed the next round for a different reason. The classifier now matches on the invalid request: unknown variant` prefix plus the method name, independent of quoting.
• Both regressions (test_turn_api_idless_unknown_variant_probe_is_inert and the ordinary-request strictness test) now use the exact live envelope you reported, empty id and backticks included.

Verified by mutation: reverting the predicate to the previous missing-id-only form makes the regression fail; restoring it passes. Suite: 2941 passed, 2 skipped; the two benchmark failures are the environment-only ones you already identified. CI green on this head.

Thanks for testing against a live server twice — both findings were things our own tests were structurally unable to catch.

[plotarmordev]

Confirmed on head a8bd1f5. Complete Herdr socket/event files pass locally (221 passed, 1 skipped). More importantly, the updated client was exercised against the actual backed-up stock Herdr 0.7.5 binary: it accepted the live empty-string-id/backticked pane.turns unknown-variant response as an uncorrelated probe error, then workspace.list succeeded on the same connection. Ordinary requests remain strict in regression coverage. Approved.