Skip to content

fix(afk): prevent accepted-but-unconfirmed escalation replay #85

Description

@pranaypratyush

Away-mode repeated digest incident — replay report

Reproduction

This isolated command uses a temporary FM_HOME only, with no Herdr or live-fleet interaction.
It sources the real daemon implementation, enables away mode, seeds the five incident-shaped buffered rows, and replaces only backend seams.
The fake target records each payload as accepted by the UI while returning unknown to simulate lost submit acknowledgement.
The command exits nonzero when the unchanged replay is observed.

export ROOT="$PWD"; bash -s <<'SH'
set -u
ROOT=${ROOT:?}
SCRATCH=$(mktemp -d)
STATE="$SCRATCH/state"
SENT="$SCRATCH/accepted-but-unconfirmed.log"
mkdir -p "$STATE"
: > "$SENT"
export FM_HOME="$SCRATCH" FM_STATE_OVERRIDE="$STATE"
export FM_SUPERVISOR_BACKEND=tmux FM_SUPERVISOR_TARGET=default:w1:p1 FM_SUPERVISOR_HARNESS=omp
export FM_INJECT_CONFIRM_RETRIES=3 FM_INJECT_CONFIRM_SLEEP=0
printf '%s\n' '2026-08-31T12:08:00Z' > "$STATE/.afk"
. "$ROOT/bin/fm-supervise-daemon.sh"
fm_backend_target_exists() { return 0; }
pane_is_busy() { return 1; }
fm_backend_composer_state() { printf 'empty'; }
fm_backend_send_text_submit() { printf '%s\n' "$3" >> "$SENT"; printf 'unknown'; }
escalate_add "$STATE" 'paused/held 81925s (awaiting external recovery, recheck whether the wait still holds): default:w1:p3'
escalate_add "$STATE" 'firstmate-prs-local.status: done: PR https://github.com/dnth/firstmate/pull/84 (catch-all scan)'
escalate_add "$STATE" 'fm-herdr-away-admission-clean-replacement.status: done: PR https://github.com/dnth/firstmate/pull/84 (catch-all scan)'
escalate_add "$STATE" 'fm-pr69-root-bad-decision.status: done: PR 69 (catch-all scan)'
escalate_add "$STATE" 'fm-remote-omp-delivery-regression.status: failed: worker input remained inert; PR https://github.com/dnth/firstmate/pull/81 (catch-all scan)'
for attempt in 1 2 3; do
  rc=0
  escalate_flush "$STATE" || rc=$?
  printf 'attempt=%s rc=%s buffer_lines=%s\n' "$attempt" "$rc" "$(wc -l < "$STATE/.subsuper-escalations")"
done
count=$(wc -l < "$SENT")
distinct=$(awk 'seen[$0]++ == 0 { n++ } END { print n + 0 }' "$SENT")
printf 'accepted_digest_count=%s\n' "$count"
printf 'distinct_accepted_digests=%s\n' "$distinct"
printf 'buffer_survives=%s\n' "$(test -s "$STATE/.subsuper-escalations" && printf yes || printf no)"
printf 'accepted_digest_sha256=%s\n' "$(sha256sum "$SENT" | cut -d" " -f1)"
if [ "$count" -ge 2 ] && [ "$distinct" = 1 ] && [ -s "$STATE/.subsuper-escalations" ]; then
  printf 'RED: accepted-but-unconfirmed digest replayed unchanged\n' >&2
  exit 1
fi
SH

The exact published command block above was extracted from this report and executed verbatim in a clean bash -lc shell from the repository root.

Recorded result of that invocation:

returncode=1
attempt=1 rc=1 buffer_lines=5
attempt=2 rc=1 buffer_lines=5
attempt=3 rc=1 buffer_lines=5
accepted_digest_count=3
distinct_accepted_digests=1
buffer_survives=yes
accepted_digest_sha256=6391425b8f02f27f22066035f82ae45effde881ae9e05a1b44f4fdc1adfcab2a
RED: accepted-but-unconfirmed digest replayed unchanged

The recorded payload was:

[U+2063]FIRSTMATE_OP: v1 away-supervisor: Supervisor escalate (5 event(s)): paused/held 81925s (awaiting external recovery, recheck whether the wait still holds): default:w1:p3 | firstmate-prs-local.status: done: PR https://github.com/dnth/firstmate/pull/84 (catch-all scan) | fm-herdr-away-admission-clean-replacement.status: done: PR https://github.com/dnth/firstmate/pull/84 (catch-all scan) | fm-pr69-root-bad-decision.status: done: PR 69 (catch-all scan) | fm-remote-omp-delivery-regression.status: failed: worker input remained inert; PR https://github.com/dnth/firstmate/pull/81 (catch-all scan) (pre-read; re-arm not needed — watcher daemon-managed)

[U+2063] denotes the leading U+2063 invisible separator emitted by fm_operational_input_encode.
The three recorded payloads had one distinct SHA-256 value, proving byte-identical replay rather than changing digest generation.

Live evidence

The live daemon log at /home/pranay/wd/firstmate/state/.supervise-daemon.log:9099-9122 shows the same causal sequence.

  • Line 9099: daemon starts with afk=on.
  • Lines 9105, 9109, and 9115: inject failed: submit unconfirmed after 3 retries (verdict=unknown, text may be in composer).
  • Lines 9107, 9111, 9113, 9117, 9119, and 9121: injection is deferred because composer state is unknown or pending.
  • Line 9122: daemon shuts down.

The captain observed identical digests reaching main despite those unknown results.
That is consistent with the fake target's accepted-pane effect followed by retry of the retained buffer.

PR81's status log at /home/pranay/wd/firstmate/state/fm-remote-omp-delivery-regression.status:8-9 ends with a failed event, then a blocked note that PR 81 checks are green while the active run continues monitoring after the pipeline advanced its head.
bin/fm-crew-state.sh:2-14 explicitly defines state/<id>.status as an append-only event log whose tail is not necessarily current state.
bin/fm-crew-state.sh:32-42 defines authoritative run-step/pane reconciliation before status-log fallback.
Therefore the reported current validation state and the latest failed event are not contradictory, and current-state inspection cannot establish whether a pane accepted one payload or received repeated submissions.
That requires delivery history and identity at the submit boundary.

Source evidence and causal separation

  • bin/fm-supervise-daemon.sh:875-890: escalate_flush joins all rows in .subsuper-escalations into one digest, clears the buffer only in the inject_msg success branch, and returns failure otherwise.
  • bin/fm-supervise-daemon.sh:1396-1405: injection is presence-gated by state/.afk.
  • bin/fm-supervise-daemon.sh:1429-1440: injection proceeds only for an affirmatively empty composer.
  • bin/fm-supervise-daemon.sh:1441-1466: text is typed once, submit is verified through the backend, and unconfirmed results preserve the buffer.
  • bin/fm-supervise-daemon.sh:1469-1477: only backend verdict empty is success; unknown logs unconfirmed and returns 1.
  • bin/fm-supervise-daemon.sh:1259-1319: pause re-surface is a separate aged-marker path that appends a new item only after PAUSE_RESURFACE_SECS and while the pause remains valid.
  • bin/fm-supervise-daemon.sh:1344-1355: heartbeat catch-all compares the current status to .subsuper-seen-status-<task>, skips equal content, and records the marker after a new append.

The documented contract agrees with the code.
.agents/skills/afk/SKILL.md:90-103 says pending or unknown composer results defer and preserve the buffer.
.agents/skills/afk/SKILL.md:110-128 says the digest is typed once, Enter is retried, and the daemon clears the buffer only on backend success.
.agents/skills/afk/SKILL.md:132-137 distinguishes generation-scoped current recovery projections from the unrelated asynchronous escalation buffer.
.agents/skills/afk/SKILL.md:166-179 defines one-line buffered digests and preserves the buffer for every non-empty composer result.
docs/tmux-backend.md:91-98 says only proven empty composer state is positive delivery acknowledgement.

Trigger, masking, and symptom

Trigger: an escalation is already buffered and the target accepts the submit while backend acknowledgement is indeterminate (unknown).

Mask: strict acknowledgement couples buffer retirement to a positive empty verdict, but there is no identity distinguishing an accepted payload from an unaccepted attempt.
The retry loop and housekeeping keep attempting delivery while the unchanged buffer remains non-empty.
Composer deferrals also preserve the same buffer, although defer-before-typing alone cannot explain accepted identical payloads.

Symptom: every retry reconstructs and resubmits the same digest, each call returns rc=1, and all five source rows remain buffered.
The fixed 81925s pause age is retained prose, not a newly computed pause age.

Competing causes and disconfirmation

  1. Retained-buffer replay after accepted-but-unconfirmed submit — confirmed and highest confidence.
    The isolated run produces three accepted payloads, three failures, one distinct byte sequence, and the same five buffered rows.
    The live log supplies the matching repeated unknown verdicts.

  2. Fresh pause rescan — not supported as the primary cause.
    The pause path at bin/fm-supervise-daemon.sh:1304-1339 requires an aged pause marker, appends a new recheck, and resets the marker.
    A fresh append would change the buffer contents or event count and cannot produce three byte-identical complete payloads with a fixed embedded age.

  3. Unchanged heartbeat catch-all re-emission — disconfirmed for unchanged content.
    tests/fm-daemon.test.sh:992-1004 asserts that an unchanged catch-all status does not re-escalate after its seen marker is recorded.
    tests/fm-daemon.test.sh:1067-1074 asserts that a per-wake escalation writes the marker and prevents catch-all refiring.
    tests/fm-daemon.test.sh:1324-1340 covers signal deduplication against that marker.
    A real status revision can add a new row, but an unchanged status does not explain the repeated identical flushes.

  4. Durable queue acknowledgement or recovery projection — not supported as the direct cause.
    Queue rows are sequence/generation keyed event history and can legitimately remain eligible until acknowledged.
    handle_durable_wakes rebuilds a temporary current projection at bin/fm-supervise-daemon.sh:1593-1695, acknowledges at lines 1710-1717, and commits seen markers only after acknowledgement.
    Queue acknowledgement through 6545 can coexist with an uncleared asynchronous escalation buffer because those are separate durability paths.
    It does not explain an already accepted identical pane payload while the unrelated buffer remains unchanged.

  5. Reentrant valid-row fallback delivery — related but distinct.
    Issue #82 covers repeated operational notifications for newer valid rows while main is already handling claimed rows.
    That can preempt handling, but it is not needed for this reproduction and cannot account for an unchanged digest without retained-buffer replay.

Related work coverage

  • Issue #74: stale-advisory freshness.
    It covers opaque supervision prose crossing a delayed delivery boundary without task/decision/source freshness identity.
    It is the correct broader semantic repair area, but it does not itself distinguish an accepted pane payload from an unconfirmed retry.
    The prior diagnosis at data/fm-stale-supervision-notice-diagnosis/report.md:38-60 identifies the away buffer as a separate delayed prose boundary, and lines 96-104 distinguish it from current recovery projection retries.
  • PR #84: Herdr atomic admission.
    It defers when confirmed-empty admission and submit cannot be established and intentionally preserves the escalation buffer on an indeterminate result.
    That improves admission safety but leaves the accepted/unknown replay boundary open.
  • Issue #82: reentrant valid-row fallback delivery.
    It is a separate control-plane coalescing/forward-progress defect, not this retained asynchronous-buffer replay.
  • PR81: current-state/completeness reconciliation, not an event-history or delivery ledger.
    Its status log cannot distinguish one accepted payload from repeated submissions of the same retained digest.

Ordinary supervision scope

Ordinary supervision does not use the away daemon's .subsuper-escalations buffer, so it does not share this exact replay mechanism.
It does share the broader stale-advisory risk tracked by issue #74: the attended OMP branch can queue opaque precomputed prose without a final delivery-time freshness check or accepted/failed/indeterminate identity.
The smallest repair should therefore be implemented first at the away asynchronous delivery boundary and then applied to the attended OMP branch through a shared typed advisory contract, without changing watcher pause cadence or current-state semantics.

Ranked conclusion

  1. Confirmed: retained-buffer replay after accepted-but-unconfirmed submit.
  2. Not supported: fresh pause rescan; cadence would create a changed/new row, not byte-identical replays.
  3. Disconfirmed for unchanged content: catch-all status re-emission, because seen markers suppress it.
  4. Separate mechanism: durable queue replay is event-history based and does not explain the repeated accepted pane payload from the unchanged escalation buffer.

Smallest safe repair boundary

Do not repair this incident by shortening the pause cadence, changing status-tail classification, weakening the unknown safety rule, or making queue sequence/generation serve as advisory freshness.

The smallest safe repair boundary is the away asynchronous delivery contract.
It must record or reconcile an accepted payload identity, distinguish accepted/failed/indeterminate outcomes, and retire or suppress an indeterminate retry when current durable state proves the advisory obsolete.
The typed identity should include task, advisory kind, optional decision key, source status/wake revision, and source freshness generation.
The existing positive composer safety checks and generation-bound current-recovery acknowledgement must remain intact.

No tracked files or live state were modified.
No live Herdr lifecycle, formatter, linter, or project-wide test suite was used.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions