From 0038dc0fe9745f617116dbc42e8711ea67a36e74 Mon Sep 17 00:00:00 2001 From: dnth Date: Sun, 6 Sep 2026 18:42:50 +0800 Subject: [PATCH 1/7] fix(omp): deliver watcher wake batches as durable next-turn notifications The OMP primary delivered its watcher wake as a custom steer. A steer that lands while the current turn is unwinding depends on OMP's queued-message auto-continue gate, which can decline, and the session then settles idle holding a wake it never handled. Deliver the wake as a hidden nextTurn message with triggerTurn instead: OMP schedules a continuation bound to that prompt generation and consumes every wake queued during the turn in it, while the message still stays out of the editable pending-message UI. Delivery is not durable on its own, so the adapter now also keeps a claim naming the exact notification body and the durable wake sequence it covers. A replacement session or a replacement OMP process re-presents that batch exactly once; a same-session extension reload re-presents nothing. Retirement is bound to acknowledgement, not delivery: only bin/fm-wake-drain.sh retires a claim, and only once no durable row at or below its cutoff is left queued, so an interruption before acknowledgement leaves the rows and the claim durable for idempotent re-handling. Because retirement reads the queue rather than an actor, a mixed queue keeps the claim until whichever acknowledgement clears its last covered row. The shared core exposes hasPendingActionableHandoff so the adapter takes the claim over silently while the core still owes an undelivered close, and exactly one of the two mechanisms speaks after a replacement. bin/fm-omp-wake-claim-lib.sh owns the claim format and every mutation; bin/fm-omp-wake-claim.sh holds the durable wake-queue lock around one of them so the adapter drives the claim without linking that lock into the OMP process. Also restores the OMP primary suite's own shell mode after the whitespace- identity case, which had left errexit on and turned every later failure into a silent exit 1 with no "not ok" line. Claude-Session: https://claude.ai/code/session_01HfeoA87NpVLfXdajotU1EB --- .agents/skills/harness-adapters/SKILL.md | 3 +- .omp/extensions/fm-primary-omp.ts | 129 ++++++++- bin/fm-omp-wake-claim-lib.sh | 177 ++++++++++++ bin/fm-omp-wake-claim.sh | 127 +++++++++ bin/fm-primary-watch-core.ts | 14 + bin/fm-wake-drain.sh | 21 ++ docs/scripts.md | 2 + docs/verification/runtime-backends.md | 4 +- docs/verification/supervision.md | 18 ++ docs/watcher-continuity.md | 13 +- tests/fm-omp-branch-live-e2e.test.sh | 14 +- tests/fm-omp-primary-live-e2e.test.sh | 57 +++- tests/fm-omp-primary.test.sh | 338 ++++++++++++++++++++++- tests/fm-wake-queue.test.sh | 158 +++++++++++ 14 files changed, 1040 insertions(+), 35 deletions(-) create mode 100644 bin/fm-omp-wake-claim-lib.sh create mode 100755 bin/fm-omp-wake-claim.sh diff --git a/.agents/skills/harness-adapters/SKILL.md b/.agents/skills/harness-adapters/SKILL.md index 01f5f841a60..37ad69e60f1 100644 --- a/.agents/skills/harness-adapters/SKILL.md +++ b/.agents/skills/harness-adapters/SKILL.md @@ -337,7 +337,8 @@ Before killing and respawning a live OMP agent for a wedged PROVIDER STREAM, fol **Primary-session integration fact (verified 2026-07-31, OMP 17.1.8).** Plain OMP started from the Firstmate root discovers `.omp/extensions/fm-primary-omp.ts` natively, including in a fresh checkout before canonical `state/` exists; `omp -e .omp/extensions/fm-primary-omp.ts` remains the explicit recovery fallback. The adapter publishes the OMP marker shape owned by [configuration](../../../docs/configuration.md#harness-support), delivers the session-start instruction on native `session_start` and `session_switch` events, and owns watcher generations through `fm_watch_arm_omp` plus `/new` and `/resume` continuity. -It routes watcher follow-ups through OMP's `sendUserMessage(content)` with no explicit `deliverAs`, so OMP starts a turn when idle and steers while streaming instead of queueing the notification; it runs the shared turn-end predicate through native `session_stop`, and applies the shared watcher-arm, persistent-directory, and delegation-shaped tool safety checks before tool execution. +It routes watcher wakes through OMP's `sendMessage` as a hidden custom next-turn message with `triggerTurn`, which starts a handling turn from idle and, when the current turn is still unwinding, schedules the continuation that consumes every wake queued during it; [watcher continuity](../../../docs/watcher-continuity.md#actionable-wake-ordering) owns the durable claim that carries an unacknowledged batch across a replacement session or process. +It runs the shared turn-end predicate through native `session_stop`, and applies the shared watcher-arm, persistent-directory, and delegation-shaped tool safety checks before tool execution. `bin/fm-session-start.sh` rejects a missing, stale, foreign-PID, or version-mismatched loaded marker and prints both native-discovery and explicit `-e` recovery commands. The authoritative operating procedure is `docs/supervision-protocols/omp.md`. diff --git a/.omp/extensions/fm-primary-omp.ts b/.omp/extensions/fm-primary-omp.ts index 1533d70cc42..007a9a61ce4 100644 --- a/.omp/extensions/fm-primary-omp.ts +++ b/.omp/extensions/fm-primary-omp.ts @@ -1,7 +1,7 @@ // Firstmate primary integration for OMP. // OMP-native session, stop, tool-call, and shutdown events stay in this adapter. import { spawn, spawnSync } from "node:child_process"; -import { randomUUID } from "node:crypto"; +import { createHash, randomUUID } from "node:crypto"; import { mkdirSync, renameSync, unlinkSync, writeFileSync } from "node:fs"; import { dirname, isAbsolute, resolve } from "node:path"; import { fileURLToPath } from "node:url"; @@ -28,6 +28,15 @@ const state = process.env.FM_STATE_OVERRIDE || `${fmHome}/state`; const config = process.env.FM_CONFIG_OVERRIDE || `${fmHome}/config`; const marker = `${state}/.omp-primary-extension-loaded`; const operationalInputScript = `${fmRoot}/bin/fm-operational-input.sh`; +const wakeClaimScript = `${fmRoot}/bin/fm-omp-wake-claim.sh`; + +// One identity per OMP process. A same-process extension reload re-evaluates +// this module and must keep it, so it lives on the realm rather than in module +// scope; a replacement process never inherits it, which is what separates a +// reload from a restart even when the operating system reuses the former PID. +type WakeClaimGlobal = typeof globalThis & { __firstmateOmpWakeClaimInstance?: string }; +const wakeClaimGlobal = globalThis as WakeClaimGlobal; +const wakeClaimInstance = wakeClaimGlobal.__firstmateOmpWakeClaimInstance ??= randomUUID(); type ProcessResult = { code: number; @@ -135,6 +144,55 @@ function runSessionstartNudge(forceForNativeSwitch = false): string { return result.stdout.trim(); } +function claimEnv(): NodeJS.ProcessEnv { + return { + ...process.env, + FM_HOME: fmHome, + FM_ROOT_OVERRIDE: fmRoot, + FM_STATE_OVERRIDE: state, + FM_CONFIG_OVERRIDE: config, + }; +} + +// Bind the durable claim for one outstanding wake notification. Never rejects: +// the durable wake queue stays the authority for the batch itself, so a claim +// that cannot be published costs a re-presentation after a replacement, while +// failing here would either cancel the wake or make the core redeliver it. +function publishWakeClaim(session: string, content: string): Promise { + return new Promise((resolveClaim) => { + const child = spawn( + wakeClaimScript, + ["publish", "--instance", wakeClaimInstance, "--session", session], + { env: claimEnv(), stdio: ["pipe", "ignore", "ignore"] }, + ); + child.on("error", () => resolveClaim()); + child.on("close", () => resolveClaim()); + child.stdin.on("error", () => { + // The claim script may have exited before the body was written. + }); + child.stdin.end(content); + }); +} + +// Take over an outstanding claim left by a replacement session or a replacement +// process and return the exact body to re-present. Empty when nothing is +// outstanding, when this process and session already own the claim (a +// same-session extension reload), or when the handover could not complete - the +// claim then keeps its previous owner and the next replay retries it. +function takeOverWakeClaim(session: string): string { + const result = spawnSync( + wakeClaimScript, + ["replay", "--instance", wakeClaimInstance, "--session", session], + // Bounded: this runs inside a session event, so a wedged durable queue lock + // must cost one skipped re-presentation, never a hung OMP session. A + // nonzero or timed-out handover leaves the claim with its previous owner + // for the next session event to retry. + { encoding: "utf8", env: claimEnv(), maxBuffer: 4 * 1024 * 1024, timeout: 15000 }, + ); + if (result.status !== 0) return ""; + return result.stdout || ""; +} + function runChecker(script: string, flag: "--command" | "--tool", value: string): Promise { return new Promise((resolveResult) => { const child = spawn(`${fmRoot}/bin/${script}`, [flag, value], { @@ -202,6 +260,50 @@ export default function (omp: ExtensionAPI) { publishNativeProcessIdentity(); const taskInboxDoorbell = installTaskInboxDoorbell(omp); let pendingStartupNudge = ""; + // Per-session half of the claim owner identity. "unbound" until a session + // event supplies one, so a claim written before any session can still be + // matched deterministically rather than looking like a foreign owner. + let wakeClaimSession = "unbound"; + + const bindWakeClaimSession = (ctx: ExtensionContext): void => { + let sessionId = ""; + try { + sessionId = ctx.sessionManager?.getSessionId?.() ?? ""; + } catch { + // A session without a readable identity keeps the unbound placeholder. + } + wakeClaimSession = sessionId ? createHash("sha256").update(sessionId).digest("hex") : "unbound"; + }; + + // Hidden next-turn delivery with triggerTurn. OMP schedules an internal + // continuation bound to the current prompt generation, so a wake that lands + // while a turn is still unwinding still starts its handling turn instead of + // stranding an idle session, and every notification queued during that turn + // is consumed by the one continuation. The message never enters the editable + // pending-message UI, so the captain's draft is untouched. + const sendWakeNotification = (content: string): void => { + omp.sendMessage( + { + customType: "firstmate-watcher-wake", + content, + display: false, + attribution: "agent", + details: { kind: "watcher", runtime: "omp" }, + }, + { deliverAs: "nextTurn", triggerTurn: true }, + ); + }; + + // Re-present the durable batch a previous session or process notified but + // never got acknowledged. The core keeps its own handoff for a close it has + // not delivered yet and replays that itself, so this takes the claim over + // silently in that case: the handover still happens exactly once, and only + // one of the two mechanisms speaks. + const replayWakeClaim = (coreOwnsDelivery: boolean): void => { + const content = takeOverWakeClaim(wakeClaimSession); + if (!content || coreOwnsDelivery) return; + sendWakeNotification(content); + }; // Supervision-branch dispatch handshake (docs/omp-supervision-branch.md). // Build one offer per ordinary actionable wake and emit it on the shared @@ -235,17 +337,12 @@ export default function (omp: ExtensionAPI) { repairToolName: "fm_watch_arm_omp", encodeOperationalInput, sendFollowUp: async (content) => { - // Deliver a custom steer so OMP wakes idle sessions without touching the editable draft. - omp.sendMessage( - { - customType: "firstmate-watcher-wake", - content, - display: false, - attribution: "agent", - details: { kind: "watcher", runtime: "omp" }, - }, - { deliverAs: "steer", triggerTurn: true }, - ); + // Claim first, notify second: an interruption between the two leaves a + // replayable claim rather than a notification no successor can + // re-present. The claim retires only when bin/fm-wake-drain.sh + // acknowledges the durable rows it covers. + await publishWakeClaim(wakeClaimSession, content); + sendWakeNotification(content); }, offerWakeToBranch, }); @@ -280,10 +377,15 @@ export default function (omp: ExtensionAPI) { }; omp.on("session_start", (_event, ctx) => { + bindWakeClaimSession(ctx); taskInboxDoorbell.activate(); + // Read before sessionStart: activating the watch consumes the core's own + // handoff, so asking afterwards could not tell who owns the redelivery. + const coreOwnsDelivery = watch.hasPendingActionableHandoff(); watch.sessionStart(); publishSecondmateSession(ctx); deliverSessionstartNudge(); + replayWakeClaim(coreOwnsDelivery); }); omp.on("turn_start", () => { @@ -292,9 +394,12 @@ export default function (omp: ExtensionAPI) { omp.on("session_switch", async (event, ctx) => { await watch.sessionShutdown(true); + bindWakeClaimSession(ctx); publishSecondmateSession(ctx); deliverSessionstartNudge(event.reason === "new" || event.reason === "resume"); + const coreOwnsDelivery = watch.hasPendingActionableHandoff(); watch.sessionStart(); + replayWakeClaim(coreOwnsDelivery); }); omp.on("before_agent_start", (event): BeforeAgentStartEventResult | undefined => { diff --git a/bin/fm-omp-wake-claim-lib.sh b/bin/fm-omp-wake-claim-lib.sh new file mode 100644 index 00000000000..8096caef5db --- /dev/null +++ b/bin/fm-omp-wake-claim-lib.sh @@ -0,0 +1,177 @@ +#!/usr/bin/env bash +# Durable OMP primary wake-notification claim. +# +# ONE owner of the claim's file format and of every read, publish, replay +# rebind, and retirement performed on it. The claim records that a watcher wake +# notification is outstanding for the durable wake-queue rows at or below a +# sequence cutoff, so a replacement OMP session or a replacement OMP process +# re-presents that exact batch instead of losing it. The durable queue rows stay +# authoritative; the claim only carries the re-notification. +# +# Format (regular file, mode 0600, never a symlink), one field per line: +# 1 fm-omp-wake-claim-v1 +# 2 claim id minted on every write +# 3 owner instance per-OMP-process identity: a same-process extension +# reload keeps it, a replacement process never does +# 4 owner session per-OMP-session identity +# 5 cutoff highest durable wake sequence the claim covers +# 6 content base64 of the exact notification body to re-present +# +# Every _locked function requires the caller to already hold +# FM_WAKE_QUEUE_LOCK. The claim is published against the queue's own sequence +# counter and retired against the queue's own rows, so publication and +# retirement serialize on the single lock the queue already has instead of +# racing across a second one. +# +# Retirement is bound to acknowledgement, never to delivery: a claim is removed +# only once no durable row at or below its cutoff remains queued, and only +# bin/fm-wake-drain.sh's acknowledgement removes those rows. An interruption +# before that acknowledgement therefore leaves the rows and the claim durable +# for idempotent re-handling. + +FM_OMP_WAKE_CLAIM_DIR="${FM_OMP_WAKE_CLAIM_DIR:-$STATE/extensions/omp-primary-watch}" +FM_OMP_WAKE_CLAIM_FILE="${FM_OMP_WAKE_CLAIM_FILE:-$FM_OMP_WAKE_CLAIM_DIR/wake-notification}" + +# Populated by fm_omp_wake_claim_read_locked; empty whenever it returns nonzero. +FM_OMP_WAKE_CLAIM_ID= +FM_OMP_WAKE_CLAIM_INSTANCE= +FM_OMP_WAKE_CLAIM_SESSION= +FM_OMP_WAKE_CLAIM_CUTOFF= +FM_OMP_WAKE_CLAIM_CONTENT_B64= + +# Owner identities are opaque to this library: it only proves they are single +# safe tokens so a malformed claim can never be mistaken for a bound one. +fm_omp_wake_claim_token_ok() { # + case "$1" in + ''|*[!A-Za-z0-9._-]*) return 1 ;; + esac + [ "${#1}" -le 128 ] +} + +fm_omp_wake_claim_new_id() { + printf '%s.%s.%s\n' "$(fm_current_pid)" "$(date +%s)" "${RANDOM}${RANDOM}" +} + +# The highest sequence the durable queue has issued. Read under the queue lock +# so every row already queued is at or below it and every later append is above. +fm_omp_wake_claim_queue_seq_locked() { + local seq + seq=$(cat "$STATE/.wake-queue.seq" 2>/dev/null || printf 0) + case "$seq" in + ''|*[!0-9]*) seq=0 ;; + esac + printf '%s\n' "$seq" +} + +fm_omp_wake_claim_read_locked() { + local version id instance session cutoff content _extra + FM_OMP_WAKE_CLAIM_ID= + FM_OMP_WAKE_CLAIM_INSTANCE= + FM_OMP_WAKE_CLAIM_SESSION= + FM_OMP_WAKE_CLAIM_CUTOFF= + FM_OMP_WAKE_CLAIM_CONTENT_B64= + [ -f "$FM_OMP_WAKE_CLAIM_FILE" ] && [ ! -L "$FM_OMP_WAKE_CLAIM_FILE" ] || return 1 + exec 9< "$FM_OMP_WAKE_CLAIM_FILE" || return 1 + IFS= read -r version <&9 || { exec 9<&-; return 1; } + IFS= read -r id <&9 || { exec 9<&-; return 1; } + IFS= read -r instance <&9 || { exec 9<&-; return 1; } + IFS= read -r session <&9 || { exec 9<&-; return 1; } + IFS= read -r cutoff <&9 || { exec 9<&-; return 1; } + IFS= read -r content <&9 || { exec 9<&-; return 1; } + if IFS= read -r _extra <&9; then + exec 9<&- + return 1 + fi + exec 9<&- + [ "$version" = fm-omp-wake-claim-v1 ] || return 1 + fm_omp_wake_claim_token_ok "$id" || return 1 + fm_omp_wake_claim_token_ok "$instance" || return 1 + fm_omp_wake_claim_token_ok "$session" || return 1 + case "$cutoff" in + ''|*[!0-9]*) return 1 ;; + esac + case "$content" in + ''|*[!A-Za-z0-9+/=]*) return 1 ;; + esac + # shellcheck disable=SC2034 # Read by sourcing callers after a successful read. + FM_OMP_WAKE_CLAIM_ID=$id + FM_OMP_WAKE_CLAIM_INSTANCE=$instance + FM_OMP_WAKE_CLAIM_SESSION=$session + FM_OMP_WAKE_CLAIM_CUTOFF=$cutoff + FM_OMP_WAKE_CLAIM_CONTENT_B64=$content +} + +fm_omp_wake_claim_write_locked() { # + local id=$1 instance=$2 session=$3 cutoff=$4 content=$5 tmp + mkdir -p "$FM_OMP_WAKE_CLAIM_DIR" || return 1 + tmp=$(mktemp "$FM_OMP_WAKE_CLAIM_FILE.tmp.XXXXXX") || return 1 + if ! printf 'fm-omp-wake-claim-v1\n%s\n%s\n%s\n%s\n%s\n' \ + "$id" "$instance" "$session" "$cutoff" "$content" > "$tmp" \ + || ! chmod 0600 "$tmp" \ + || ! _fm_atomic_replace "$tmp" "$FM_OMP_WAKE_CLAIM_FILE"; then + rm -f -- "$tmp" + return 1 + fi +} + +# Bind one outstanding notification to the caller's process and session. A new +# claim replaces an outstanding one rather than accumulating, because OMP's +# hidden next-turn transport already coalesces every queued notification into +# one continuation turn. The cutoff only ever moves forward, so replacing a +# claim can never shorten the row span its retirement waits for. +fm_omp_wake_claim_publish_locked() { # + local instance=$1 session=$2 content=$3 cutoff + fm_omp_wake_claim_token_ok "$instance" || return 1 + fm_omp_wake_claim_token_ok "$session" || return 1 + case "$content" in + ''|*[!A-Za-z0-9+/=]*) return 1 ;; + esac + cutoff=$(fm_omp_wake_claim_queue_seq_locked) || return 1 + if fm_omp_wake_claim_read_locked && [ "$FM_OMP_WAKE_CLAIM_CUTOFF" -gt "$cutoff" ]; then + cutoff=$FM_OMP_WAKE_CLAIM_CUTOFF + fi + fm_omp_wake_claim_write_locked "$(fm_omp_wake_claim_new_id)" "$instance" "$session" "$cutoff" "$content" +} + +# Decide whether the given owner owes a re-presentation, leaving the claim in +# the read globals when it does. A claim already bound to this process and this +# session is a same-session extension reload and must not be re-presented; any +# other binding is a replacement session or a replacement process. +# 0 a replay is due, 1 invalid owner, 3 nothing to replay. +fm_omp_wake_claim_replay_pending_locked() { # + local instance=$1 session=$2 + fm_omp_wake_claim_token_ok "$instance" || return 1 + fm_omp_wake_claim_token_ok "$session" || return 1 + fm_omp_wake_claim_read_locked || return 3 + [ "$FM_OMP_WAKE_CLAIM_INSTANCE" = "$instance" ] && [ "$FM_OMP_WAKE_CLAIM_SESSION" = "$session" ] && return 3 + return 0 +} + +# Move the outstanding claim to a new owner, keeping its cutoff and body. This +# is what makes a re-presentation exactly-once per owner: a second replay under +# the same binding finds nothing to hand over. Callers rebind only after the +# body is safely handed to the new owner, so a failure anywhere earlier leaves +# the claim with its previous owner and the replay simply retries. +fm_omp_wake_claim_rebind_locked() { # + local instance=$1 session=$2 + fm_omp_wake_claim_token_ok "$instance" || return 1 + fm_omp_wake_claim_token_ok "$session" || return 1 + fm_omp_wake_claim_read_locked || return 1 + fm_omp_wake_claim_write_locked "$(fm_omp_wake_claim_new_id)" "$instance" "$session" \ + "$FM_OMP_WAKE_CLAIM_CUTOFF" "$FM_OMP_WAKE_CLAIM_CONTENT_B64" +} + +# Retire the claim exactly when the durable rows it covers are gone. Actor +# agnostic on purpose: a mixed queue can have main acknowledge some covered +# rows and the supervision branch acknowledge the rest, and the claim must +# survive until whichever acknowledgement clears the last one. +fm_omp_wake_claim_reconcile_locked() { + fm_omp_wake_claim_read_locked || return 0 + if [ -s "$FM_WAKE_QUEUE" ] && awk -F '\t' -v cutoff="$FM_OMP_WAKE_CLAIM_CUTOFF" ' + NF >= 5 && $2 ~ /^[0-9]+$/ && $2 + 0 <= cutoff + 0 { covered = 1; exit } + END { exit covered ? 0 : 1 } + ' "$FM_WAKE_QUEUE" 2>/dev/null; then + return 0 + fi + rm -f -- "$FM_OMP_WAKE_CLAIM_FILE" || return 1 +} diff --git a/bin/fm-omp-wake-claim.sh b/bin/fm-omp-wake-claim.sh new file mode 100755 index 00000000000..be92f107177 --- /dev/null +++ b/bin/fm-omp-wake-claim.sh @@ -0,0 +1,127 @@ +#!/usr/bin/env bash +# Command-line face of the durable OMP primary wake-notification claim. +# bin/fm-omp-wake-claim-lib.sh owns the format, the invariants, and every +# mutation; this script only holds the durable wake-queue lock around one of +# them so the OMP adapter can drive the claim without linking that lock into +# its own process. +# +# Usage: +# fm-omp-wake-claim.sh publish --instance --session # body on stdin +# fm-omp-wake-claim.sh replay --instance --session # body on stdout +# fm-omp-wake-claim.sh show +# +# publish binds one outstanding notification to this OMP process and session. +# replay hands an outstanding claim to a new owner exactly once and prints the +# exact body to re-present; a claim already bound to the given owner is a +# same-session extension reload and prints nothing. +# show prints "\t\t\t\t". +# +# Exit codes: 0 done, 1 failed, 2 usage, 3 nothing to replay or show, +# 4 the durable wake-queue lock stayed busy for the whole bounded wait +# (FM_OMP_WAKE_CLAIM_LOCK_ATTEMPTS attempts, 0.05s apart, default 200). +set -u + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=bin/fm-wake-lib.sh +. "$SCRIPT_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-omp-wake-claim-lib.sh +. "$SCRIPT_DIR/fm-omp-wake-claim-lib.sh" + +LOCK_HELD=false +CLAIM_TMP= + +usage() { + echo "usage: fm-omp-wake-claim.sh (publish|replay) --instance --session | show" >&2 + exit 2 +} + +# shellcheck disable=SC2317,SC2329 # Invoked by the trap handlers below. +cleanup() { + local status=$? + [ -z "$CLAIM_TMP" ] || rm -f -- "$CLAIM_TMP" 2>/dev/null || true + if [ "$LOCK_HELD" = true ]; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fi + exit "$status" +} + +# Portable decode: GNU coreutils spells it --decode, BSD base64 spells it -D. +# Decode into a file rather than straight to stdout so a rejected first attempt +# can never leave a partial body behind for the second one to append to. +decode_base64_to() { # + if printf '%s' "$1" | base64 --decode > "$2" 2>/dev/null; then return 0; fi + printf '%s' "$1" | base64 -D > "$2" 2>/dev/null +} + +acquire_queue_lock() { + local attempts=${FM_OMP_WAKE_CLAIM_LOCK_ATTEMPTS:-200} attempt=0 + case "$attempts" in + ''|*[!0-9]*|0) attempts=200 ;; + esac + while ! fm_lock_try_acquire "$FM_WAKE_QUEUE_LOCK"; do + attempt=$((attempt + 1)) + if [ "$attempt" -ge "$attempts" ]; then + echo "fm-omp-wake-claim: durable wake queue lock stayed busy" >&2 + exit 4 + fi + sleep 0.05 + done + LOCK_HELD=true +} + +COMMAND=${1:-} +[ -n "$COMMAND" ] || usage +shift || true + +INSTANCE= +SESSION= +case "$COMMAND" in + publish|replay) + while [ "$#" -gt 0 ]; do + case "$1" in + --instance) INSTANCE=${2:-}; shift 2 || usage ;; + --session) SESSION=${2:-}; shift 2 || usage ;; + *) usage ;; + esac + done + fm_omp_wake_claim_token_ok "$INSTANCE" || usage + fm_omp_wake_claim_token_ok "$SESSION" || usage + ;; + show) + [ "$#" -eq 0 ] || usage + ;; + *) usage ;; +esac + +trap cleanup EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + +case "$COMMAND" in + publish) + CLAIM_CONTENT=$(base64 | tr -d '\n') || exit 1 + [ -n "$CLAIM_CONTENT" ] || { echo "fm-omp-wake-claim: refusing to claim an empty notification" >&2; exit 1; } + acquire_queue_lock + fm_omp_wake_claim_publish_locked "$INSTANCE" "$SESSION" "$CLAIM_CONTENT" || exit 1 + ;; + replay) + acquire_queue_lock + fm_omp_wake_claim_replay_pending_locked "$INSTANCE" "$SESSION" || exit "$?" + CLAIM_TMP=$(mktemp "$STATE/.omp-wake-claim.replay.XXXXXX") || exit 1 + decode_base64_to "$FM_OMP_WAKE_CLAIM_CONTENT_B64" "$CLAIM_TMP" || exit 1 + # Hand the body over first and rebind last: an interruption before the + # rebind leaves the claim with its previous owner, so the next replay + # attempt re-presents the same batch instead of losing it. + command cat "$CLAIM_TMP" || exit 1 + fm_omp_wake_claim_rebind_locked "$INSTANCE" "$SESSION" || exit 1 + ;; + show) + acquire_queue_lock + fm_omp_wake_claim_read_locked || exit 3 + printf '%s\t%s\t%s\t%s\t%s\n' \ + "$FM_OMP_WAKE_CLAIM_ID" "$FM_OMP_WAKE_CLAIM_INSTANCE" "$FM_OMP_WAKE_CLAIM_SESSION" \ + "$FM_OMP_WAKE_CLAIM_CUTOFF" "$FM_OMP_WAKE_CLAIM_CONTENT_B64" || exit 1 + ;; +esac + +exit 0 diff --git a/bin/fm-primary-watch-core.ts b/bin/fm-primary-watch-core.ts index 052f14e7c1f..8f162825482 100644 --- a/bin/fm-primary-watch-core.ts +++ b/bin/fm-primary-watch-core.ts @@ -17,6 +17,7 @@ import { spawn, spawnSync, type ChildProcess } from "node:child_process"; import { createHash, randomUUID } from "node:crypto"; import { closeSync, + existsSync, mkdirSync, openSync, readFileSync, @@ -110,6 +111,7 @@ export type PrimaryWatchCore = { arm: () => ArmResult; armAndWait: () => Promise; acknowledgeWake: (content: string) => void; + hasPendingActionableHandoff: () => boolean; markLoaded: () => void; sessionShutdown: (replacement?: boolean) => Promise; sessionStart: () => void; @@ -1130,6 +1132,17 @@ export function createPrimaryWatchCore(options: PrimaryWatchCoreOptions): Primar return result; } + // True while the core still owns an actionable wake it has not delivered: an + // undelivered close on this generation, one handed over in process, or the + // durable replacement handoff on disk. A runtime adapter that keeps its own + // durable re-notification claim reads this before replaying that claim, so + // one wake is never delivered twice across a session replacement. + function hasPendingActionableHandoff(): boolean { + if (replacementCoordinator.pending.length > 0) return true; + if (generation.pendingActionables.some((pending) => !pending.delivered)) return true; + return existsSync(actionableHandoff); + } + function acknowledgeWake(content: string): void { for (const [token, acknowledgement] of generation.wakeAcknowledgements) { if (acknowledgement.content !== content) continue; @@ -1164,6 +1177,7 @@ export function createPrimaryWatchCore(options: PrimaryWatchCoreOptions): Primar arm: () => activateOwnedWatch(generation), armAndWait, acknowledgeWake, + hasPendingActionableHandoff, markLoaded, sessionShutdown, sessionStart, diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index eed0ef2693b..81c9f8c6162 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -21,6 +21,8 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" . "$SCRIPT_DIR/fm-line-cap-lib.sh" # shellcheck source=bin/fm-lease-lib.sh . "$SCRIPT_DIR/fm-lease-lib.sh" +# shellcheck source=bin/fm-omp-wake-claim-lib.sh +. "$SCRIPT_DIR/fm-omp-wake-claim-lib.sh" DRAIN_TMP= DRAIN_VIEW_TMP= @@ -377,6 +379,17 @@ print_status_presentation() { # [] return "$rc" } +# The OMP primary keeps a durable claim naming the wake batch it notified, so a +# replacement session or process can re-present that exact batch. Retirement is +# bound to acknowledgement rather than delivery, which makes this drain its only +# owner (bin/fm-omp-wake-claim-lib.sh). A claim that cannot be retired is a +# stale re-presentation at worst, never a lost wake, so say so and let the drain +# finish rather than failing the whole presentation. +retire_settled_omp_wake_claim() { + fm_omp_wake_claim_reconcile_locked \ + || echo "wake drain: an acknowledged OMP wake notification claim could not be retired at $FM_OMP_WAKE_CLAIM_FILE" >&2 +} + # shellcheck disable=SC2317,SC2329 # Invoked by trap handlers below. cleanup() { local status=$? @@ -394,6 +407,11 @@ trap 'exit 143' TERM fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" DRAIN_LOCK_HELD=true +# Retire an OMP primary wake claim whose durable rows are already gone. A claim +# covering rows still queued survives here untouched, so this only clears one +# whose batch a previous acknowledgement already consumed - or one that never +# covered a durable row at all, such as an extension-internal failure wake. +retire_settled_omp_wake_claim decide_scoped_locked if [ "$SCOPED" = true ]; then reclaim_stale_branch_grant_locked || exit 1 @@ -475,6 +493,9 @@ if [ -n "$ACK_THROUGH" ]; then consume_actor_rows_locked "$MAIN_ROWS_FILE" "$ACK_THROUGH" || exit 1 fi fi + # Only this acknowledgement removes durable rows, so this is the one place a + # still-covered OMP primary wake claim can become settled. + retire_settled_omp_wake_claim fm_lock_release "$FM_WAKE_QUEUE_LOCK" DRAIN_LOCK_HELD=false if [ "$ACK_REMOVED" -eq 0 ] && [ "$PRESENTED_MAX" -gt "$ACK_THROUGH" ]; then diff --git a/docs/scripts.md b/docs/scripts.md index 3625ed5d4e8..5c7ed60ed7e 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -85,6 +85,8 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-timeout-lib.sh` | Shared bounded command runner that terminates the entire process group on timeout | | `fm-gate-refuse-lib.sh` | Shared no-mistakes gate-context refusal for fleet lifecycle entrypoints | | `fm-primary-watch-core.ts` | Harness-neutral watcher lifecycle core bound by the Pi and OMP primary extensions (docs/watcher-continuity.md) | +| `fm-omp-wake-claim.sh` | Publish, hand over, or show the OMP primary's durable wake-notification claim under the durable wake-queue lock | +| `fm-omp-wake-claim-lib.sh` | Shared format, publication, replay-handover, and acknowledgement-bound retirement of that claim | | `fm-primary-watch-version-lib.sh` | The one definition of a primary watcher marker version, hashing that adapter plus the shared core | | `fm-watch-arm.sh` | Verified home-scoped watcher arm wrapper with loud cycle endings and bounded lifecycle ledger | | `fm-watch-checkpoint.sh` | Run one bounded foreground watcher checkpoint for Codex-style supervision | diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index 346add656da..89b41d83ead 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -527,7 +527,7 @@ FM_TEST_SUMMARY_FAMILY family=watcher-wake-lock count=2 duration_ms=97033 failed Every listed script ran at that head with no gate skip. The isolated Herdr role matrix emitted no queued-wake warning. The final run retained the fresh-beacon, pending-notification, queue-drain, and bounded-delivery assertions in the Herdr fixture. -The focused OMP adapter contract now delivers watcher wakes as a custom steer with `triggerTurn`, preserving the editable draft while retaining idle wake and streaming delivery. +The focused OMP adapter contract now delivers watcher wakes as a hidden custom next-turn message with `triggerTurn`, preserving the editable draft while retaining idle wake and unwinding-turn continuation delivery. The tmux role fixtures emitted their expected task-copy worktree and missing-fixture-watcher notices. The Herdr exit fixture refused an unlocked presentation close after proving normal process exit, then completed its named guarded teardown. @@ -618,7 +618,7 @@ ok - a captain-worthy wake opens exactly one follow-up turn on MAIN (real SDK) ok - OMP supervision branch live guard passed against @oh-my-pi/pi-coding-agent 17.3.4 ``` -The guard proves a broken branch (an unresolvable model pin) falls the wake back to main through the primary adapter's watcher-wake steer with triggerTurn, not sendUserMessage, leaving the wake queue durable. +The guard proves a broken branch (an unresolvable model pin) falls the wake back to main through the primary adapter's hidden watcher-wake next-turn message with triggerTurn, not sendUserMessage, leaving the wake queue durable. It proves a resident second AgentSession is created and remains re-promptable on a later wake without its turn output reaching main or replacing main's terminal resume breadcrumb, and that a routine verdict opens no new main turn while a captain verdict opens exactly one follow-up turn. The captain sub-check is skipped, not passed, on a run where the model judges the captain-worthy fixture routine. diff --git a/docs/verification/supervision.md b/docs/verification/supervision.md index 2909a50ced0..d656901486c 100644 --- a/docs/verification/supervision.md +++ b/docs/verification/supervision.md @@ -127,6 +127,24 @@ ok - OMP omp/18.1.5 primary E2E proved watcher delivery with an intact editable Before that change the same guard failed on OMP 18.1.5 at the first `/new`: the restored watcher's first wake started an agent-initiated turn, and OMP emits `before_agent_start` neither for that turn nor for a captain prompt queued into it, so an instruction staged for `before_agent_start` never reached the replacement session. +The OMP 18.1.5 idle-wake guard ran on 2026-09-06 after the adapter moved its watcher wake from a custom steer to a hidden `nextTurn` message with `triggerTurn`. +Delivery alone was already covered; this run adds the guarantee that the idle session runs the handling turn for that wake itself. + +```sh +omp --version +FM_OMP_PRIMARY_LIVE_E2E=1 tests/fm-omp-primary-live-e2e.test.sh +``` + +```text +omp/18.1.5 +ok - OMP omp/18.1.5 primary E2E proved fresh no-state and ordinary native discovery, exact ownership, once-only startup, guarded watcher startup, repeated /new continuity, shutdown, resume, and away-mode delivery +ok - OMP omp/18.1.5 primary E2E proved an idle session runs the watcher wake turn itself with an intact editable draft +``` + +The session was idle with an unsent draft when the watcher fired and nobody typed anything: it started the handling turn on its own and reached a terminal assistant record with the exact draft unchanged. +The same command refreshes this result. +A live leg that fires the wake mid-turn was attempted and left out: driving a reliably long real turn from the composer was not dependable enough for a guard, so the mid-turn continuation rests on OMP's own `sendCustomMessage` contract - `nextTurn` with `triggerTurn` schedules a post-prompt continuation bound to that prompt generation - plus the deterministic delivery-mode assertions in `tests/fm-omp-primary.test.sh`. + Standalone OMP executable compatibility is recorded in [runtime backend verification](runtime-backends.md#omp-lifecycle). Current deterministic and live entry points: diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index 55a9bdfae81..0e6516464cd 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -26,7 +26,12 @@ While supervision is still needed and away mode remains inactive, an actionable ## Actionable wake ordering After an actionable Pi, OMP, or OpenCode child close, the adapter starts and verifies one singleton successor before it delivers the original wake. -OMP delivers that follow-up through its host API as a hidden custom `steer` with `triggerTurn`, which starts an idle handling turn without touching an editable TUI draft. +OMP delivers that follow-up through its host API as a hidden custom `nextTurn` message with `triggerTurn`, which starts an idle handling turn without touching an editable TUI draft. +That mode is what makes an idle session reachable from a turn boundary: a wake that lands while the current turn is still unwinding schedules a continuation bound to that prompt generation instead of relying on a steer the finishing turn may never poll, and every wake queued during that turn is consumed by the one continuation. +OMP additionally keeps a durable claim naming the exact notification body and the durable wake sequence it covers, so a replacement session or a replacement OMP process re-presents that batch once instead of losing it; `bin/fm-omp-wake-claim-lib.sh` owns that claim's format, its per-process and per-session owner identity, and its invariants. +The claim is bound to acknowledgement, not to delivery: only `bin/fm-wake-drain.sh` retires it, and only once no durable row at or below its cutoff is left queued, so an interruption before that acknowledgement leaves the rows and the claim durable for idempotent re-handling. +Because retirement reads the queue rather than an actor, a mixed queue that main and the supervision branch acknowledge separately keeps the claim until whichever acknowledgement clears the last covered row. +The core keeps its own handoff for an actionable close it has not delivered yet, so the adapter takes the claim over silently while that handoff is outstanding and exactly one of the two mechanisms speaks. A delivered follow-up is acknowledged when the runtime starts a turn whose prompt is that wake; a wake the runtime queues into an already-running turn never starts one, so the core waits at most `FM_WATCH_WAKE_CONSUME_TIMEOUT_MS` for that acknowledgement, then treats the wake as consumed and continues the successor chain instead of parking every later actionable close behind it. It confirms the handling handoff against that successor before scheduling the follow-up, retries once against the current generation and successor, and treats a failed confirmation as a restoration failure: it classifies the error, retires a successor that is no longer alive, and surfaces exactly one typed message. A failed confirmation is never swallowed. @@ -102,8 +107,10 @@ The same suite covers Pi same-process replacement for `/new`, `/resume`, `/fork` `tests/fm-omp-primary.test.sh` drives OMP `/new`, `/resume`, `/fork`, and reload through real watcher child processes and the adapter's public `session_switch` event, proving automatic re-arm plus the same in-flight actionable handoff without using `session_shutdown` as a replacement signal. Both suites also prove the bounded acknowledgement wait: two consecutive actionable closes whose deliveries never receive `before_agent_start` still start a third arm and deliver exactly one wake per close. It also covers a fresh factory bind without a prior shutdown: the superseded binding retires its arm child, cannot arm or reclaim ownership through retained session callbacks, the live binding keeps exactly one arm child, and repeated binds never accumulate more than one process-exit fallback. -`tests/fm-omp-primary.test.sh` covers OMP's binding of the same core to its native session, watcher, and shutdown surfaces, pins the input-preserving custom-steer delivery contract, pins the recovery handling handshake OMP performs before delivering that steer, and pins the single typed wake a refused handshake produces. -The opt-in `tests/fm-omp-primary-live-e2e.test.sh` proves a real OMP watcher wake reaches the session while its exact pending draft remains intact. +`tests/fm-omp-primary.test.sh` covers OMP's binding of the same core to its native session, watcher, and shutdown surfaces, pins the input-preserving hidden next-turn delivery contract, pins the recovery handling handshake OMP performs before delivering that notification, and pins the single typed wake a refused handshake produces. +It also covers the durable wake claim end to end: publication before notification, a same-session extension reload that re-presents nothing, a replacement session and a replacement process that each re-present the exact batch once, the silent handover while the core still owns an undelivered close, and retirement only after the drain acknowledges the covered rows. +The opt-in `tests/fm-omp-primary-live-e2e.test.sh` proves a real OMP watcher wake reaches the session while its exact pending draft remains intact, and that an idle OMP primary runs a handling turn for that wake with no manual intervention. +`tests/fm-wake-queue.test.sh` covers the acknowledgement-bound retirement itself, including a partial acknowledgement that must keep the claim and a concurrent append that must not retire it. `tests/fm-watch-arm.test.sh` covers durable queue replay, real remote parent-replies ingestion into the authoritative status log, decision-only OPEN DECISIONS recovery, interrupted handling replay, generation-bound acknowledgement, a persistent live successor after recovery, a watcher close inside the handling window that must leave the printed acknowledgement valid, and the self-healing moved-generation acknowledgement that consumes its handled rows and names its remedy. `tests/fm-watch-recovery-loop.test.sh` covers the once-per-generation announcement bound with the real Pi extension against a refused handling handshake, and a handling successor that must surface a real crew event instead of going blind. Pi and OMP share `bin/fm-primary-watch-core.ts` here, so that Pi-driven coverage exercises the same confirmation, retry, and typed-failure code OMP runs. diff --git a/tests/fm-omp-branch-live-e2e.test.sh b/tests/fm-omp-branch-live-e2e.test.sh index 99d28e0d83f..cf24a37763b 100755 --- a/tests/fm-omp-branch-live-e2e.test.sh +++ b/tests/fm-omp-branch-live-e2e.test.sh @@ -101,17 +101,17 @@ const sessionFile = () => { try { return readFileSync(`${HOME}/state/.branch-ses if (MODE === "degrade") { // The broken branch falls the wake back to main through the primary adapter's - // exact main-wake mechanism: a firstmate-watcher-wake steer with triggerTurn, - // captured in the sendMessage stream (not sendUserMessage). + // exact main-wake mechanism: a hidden firstmate-watcher-wake nextTurn message + // with triggerTurn, captured in the sendMessage stream (not sendUserMessage). const fallback = sent.filter((s) => s.m.customType === "firstmate-watcher-wake"); - if (fallback.length < 1) fail("a broken branch did not fall the wake back to main via a watcher-wake steer"); - if (fallback.some((s) => s.o.triggerTurn !== true || s.o.deliverAs !== "steer")) { - fail("fallback did not use the steer+triggerTurn main-wake mechanism"); + if (fallback.length < 1) fail("a broken branch did not fall the wake back to main via a watcher-wake notification"); + if (fallback.some((s) => s.o.triggerTurn !== true || s.o.deliverAs !== "nextTurn")) { + fail("fallback did not use the hidden nextTurn+triggerTurn main-wake mechanism"); } if (sent.some((s) => s.m.customType === "fm-branch-merge")) fail("a broken branch merged into main instead of falling back"); - if (userMsgs.length !== 0) fail("fallback used sendUserMessage instead of the watcher-wake steer"); + if (userMsgs.length !== 0) fail("fallback used sendUserMessage instead of the watcher-wake notification"); if (!(readFileSync(`${HOME}/state/.wake-queue`, "utf8").trim().length > 0)) fail("the wake queue was lost on degrade"); - console.log("DRIVER_OK degrade: broken branch fell back to main via a watcher-wake steer with the wake queue intact"); + console.log("DRIVER_OK degrade: broken branch fell back to main via a hidden watcher-wake nextTurn with the wake queue intact"); process.exit(0); } diff --git a/tests/fm-omp-primary-live-e2e.test.sh b/tests/fm-omp-primary-live-e2e.test.sh index 817813a2289..b1ca2733841 100755 --- a/tests/fm-omp-primary-live-e2e.test.sh +++ b/tests/fm-omp-primary-live-e2e.test.sh @@ -106,6 +106,34 @@ wait_idle() { return 1 } +# True once the session actually ran a turn FOR the watcher wake: the hidden +# watcher-wake entry carrying is followed by a terminal assistant +# record. Delivery alone is not the guarantee - an idle session that accepts the +# notification and never turns is exactly the failure this guard exists to +# catch, and a persistent second mate is this same adapter running as the +# primary of its own home. +session_handled_watcher_wake_after() { # + local file=$1 offset=$2 marker=$3 + tail -c "+$((offset + 1))" "$file" | node -e ' + const marker = process.argv[1]; + let input = ""; + process.stdin.setEncoding("utf8"); + process.stdin.on("data", chunk => { input += chunk; }); + process.stdin.on("end", () => { + let seen = false; + for (const line of input.trimEnd().split("\n")) { + if (!line) continue; + if (line.includes("\"firstmate-watcher-wake\"") && line.includes(marker)) seen = true; + if (!seen) continue; + let entry; + try { entry = JSON.parse(line); } catch { continue; } + if (entry.message?.role === "assistant" && entry.message.stopReason === "stop") process.exit(0); + } + process.exit(1); + }); + ' "$marker" +} + session_has_terminal_assistant_after() { local file=$1 offset=$2 marker=$3 tail -c "+$((offset + 1))" "$file" | node -e ' @@ -374,7 +402,18 @@ printf 'ok - OMP %s primary E2E proved fresh no-state and ordinary native discov "$OMP_VERSION" draft="human-draft-survives-omp-watcher-wake" PATH="$WRAPPER_BIN:$PATH" tmux send-keys -t "$TARGET" -l "$draft" -[ "$(composer_state)" = pending ] && [ "$(composer_text)" = "$draft" ] \ +# The TUI renders typed text asynchronously, so wait for the exact draft rather +# than sampling once: the assertion is that this exact draft is what the wake +# must leave alone, not how fast the terminal repaints it. +draft_rendered=0 +for _ in $(seq 1 120); do + if [ "$(composer_state)" = pending ] && [ "$(composer_text)" = "$draft" ]; then + draft_rendered=1 + break + fi + sleep 0.25 +done +[ "$draft_rendered" -eq 1 ] \ || { capture >&2; fail "OMP $OMP_VERSION did not render the exact editable draft before the watcher wake"; } wake_status="$HOME_DIR/state/omp-wake-preserve-$$.status" wake_offset=$(wc -c < "$session_file" | tr -d '[:space:]') @@ -394,6 +433,20 @@ done [ "$(composer_state)" = pending ] && [ "$(composer_text)" = "$draft" ] \ || { capture >&2; fail "OMP $OMP_VERSION watcher wake changed the exact editable draft"; } -printf 'ok - OMP %s primary E2E proved watcher delivery with an intact editable draft\n' "$OMP_VERSION" +# The session was idle with a pending draft when the watcher fired, and nobody +# typed anything: hidden next-turn delivery with triggerTurn has to start the +# handling turn by itself and carry it to a terminal assistant record. +wake_handled=0 +for _ in $(seq 1 480); do + if session_handled_watcher_wake_after "$session_file" "$wake_offset" "$wake_status"; then + wake_handled=1 + break + fi + sleep 0.25 +done +[ "$wake_handled" -eq 1 ] \ + || { capture >&2; fail "OMP $OMP_VERSION left an idle session holding the watcher wake without running a handling turn"; } + +printf 'ok - OMP %s primary E2E proved an idle session runs the watcher wake turn itself with an intact editable draft\n' "$OMP_VERSION" PATH="$WRAPPER_BIN:$PATH" tmux send-keys -t "$TARGET" Escape submit_omp /exit || fail "OMP $OMP_VERSION did not accept cleanup after draft preservation" diff --git a/tests/fm-omp-primary.test.sh b/tests/fm-omp-primary.test.sh index 12765900e52..807e9de453b 100755 --- a/tests/fm-omp-primary.test.sh +++ b/tests/fm-omp-primary.test.sh @@ -377,7 +377,10 @@ extension.default(api); JS ) rc=$? - set -e + # Restore the suite's own mode. Leaving errexit on here would make every later + # test's command substitution abort the whole script silently instead of + # reporting its own "not ok" line. + set +e [ "$rc" -ne 0 ] || fail "OMP primary marker accepted a whitespace-bearing entrypoint" assert_contains "$out" 'OMP primary identity paths containing whitespace are unsupported' \ "OMP primary whitespace refusal was not actionable" @@ -640,10 +643,10 @@ if (watcherMessages.length !== 1 || !watcherMessages[0].message.content.includes } if ( watcherMessages[0].message.customType !== "firstmate-watcher-wake" || - watcherMessages[0].options?.deliverAs !== "steer" || + watcherMessages[0].options?.deliverAs !== "nextTurn" || watcherMessages[0].options?.triggerTurn !== true ) { - throw new Error(`OMP watcher notification did not preserve the editable draft delivery mode: ${JSON.stringify(watcherMessages[0])}`); + throw new Error(`OMP watcher notification did not use hidden next-turn delivery: ${JSON.stringify(watcherMessages[0])}`); } if (!existsSync(`${process.env.FM_STATE_OVERRIDE}/watch-successor-ready`)) { throw new Error("OMP actionable notification arrived before successor readiness"); @@ -782,7 +785,7 @@ JS # The shared core delivers the recovery handshake for every runtime bound to it, # so OMP must confirm a handling delivery exactly like Pi and OpenCode do: start # and verify the successor, run fm-watch-arm.sh --handling-delivered for the -# generation the successor reported, and only then deliver the wake steer. +# generation the successor reported, and only then deliver the wake notification. # Upstream covers Pi and OpenCode; this pins the fork's OMP binding of the same # contract so a future adapter change cannot silently drop it. test_native_omp_confirms_recovery_handling_delivery() { @@ -863,8 +866,8 @@ const rows = armRows(); const arms = rows.filter((row) => row.startsWith("arm=")); if (arms.length !== 2) throw new Error(`expected one successor arm, got ${arms.length}: ${rows.join(" | ")}`); if (steers !== 1) throw new Error(`expected exactly one wake steer, got ${steers}`); -if (deliveryOptions?.deliverAs !== "steer" || deliveryOptions?.triggerTurn !== true) { - throw new Error(`wake was not delivered as a turn-triggering steer: ${JSON.stringify(deliveryOptions)}`); +if (deliveryOptions?.deliverAs !== "nextTurn" || deliveryOptions?.triggerTurn !== true) { + throw new Error(`wake was not delivered as a turn-triggering hidden next-turn message: ${JSON.stringify(deliveryOptions)}`); } if (rowsAtDelivery !== 2) throw new Error(`wake delivery began before successor establishment (${rowsAtDelivery} arm rows)`); const confirmations = rows.filter((row) => row.startsWith("confirmed ")); @@ -883,8 +886,8 @@ JS ) || status=$? printf 'stop\n' > "$TMP_ROOT/native-handling-delivery.stop" 2>/dev/null || true expect_code 0 "$status" "OMP recovery handling delivery" - assert_contains "$out" omp-handling-delivery-ok "OMP did not confirm its recovery handling delivery after the wake steer" - pass "OMP confirms the recovery handling handshake after delivering its wake steer" + assert_contains "$out" omp-handling-delivery-ok "OMP did not confirm its recovery handling delivery after the wake notification" + pass "OMP confirms the recovery handling handshake after delivering its hidden next-turn wake" } # A refused handling handshake must be classified and surfaced exactly once @@ -1234,6 +1237,323 @@ JS pass "OMP unacknowledged wake delivery keeps the successor chain and delivers once per close" } +# Build an OMP primary fixture whose adapter can drive the real durable wake +# claim: the production claim scripts and wake library are copied in, so the +# adapter exercises the same publication, handover, and format the drain reads. +make_omp_claim_fixture() { # + local fixture=$TMP_ROOT/$1 + mkdir -p "$fixture/.omp/extensions/lib" "$fixture/bin" "$fixture/state" "$fixture/config" + : > "$fixture/AGENTS.md" + git init -q -b main "$fixture" + cp "$ROOT/.omp/extensions/fm-primary-omp.ts" "$fixture/.omp/extensions/fm-primary-omp.ts" + cp "$ROOT/.omp/extensions/lib/fm-branch-dispatch.ts" "$fixture/.omp/extensions/lib/fm-branch-dispatch.ts" + cp "$ROOT/.omp/extensions/lib/fm-async-exec.ts" "$fixture/.omp/extensions/lib/fm-async-exec.ts" + cp "$ROOT/.omp/extensions/lib/fm-task-inbox-doorbell.ts" "$fixture/.omp/extensions/lib/fm-task-inbox-doorbell.ts" + cp "$ROOT/bin/fm-primary-watch-core.ts" "$fixture/bin/fm-primary-watch-core.ts" + cp "$ROOT/bin/fm-pi-compatible-runtimes" "$fixture/bin/fm-pi-compatible-runtimes" + cp "$ROOT/bin/fm-wake-lib.sh" "$fixture/bin/fm-wake-lib.sh" + cp "$ROOT/bin/fm-omp-wake-claim-lib.sh" "$fixture/bin/fm-omp-wake-claim-lib.sh" + cp "$ROOT/bin/fm-omp-wake-claim.sh" "$fixture/bin/fm-omp-wake-claim.sh" + cat > "$fixture/bin/fm-gate-refuse-lib.sh" <<'SH' +fm_is_gate_agent() { return 1; } +SH + cat > "$fixture/bin/fm-primary-scope-lib.sh" <<'SH' +fm_primary_scope_matches() { return 0; } +SH + cat > "$fixture/bin/fm-operational-input.sh" <<'SH' +#!/usr/bin/env bash +printf 'encoded:%s:%s' "$2" "$(cat)" +SH + cat > "$fixture/bin/fm-sessionstart-nudge.sh" <<'SH' +#!/usr/bin/env bash +exit 0 +SH + cat > "$fixture/bin/fm-turnend-guard.sh" <<'SH' +#!/usr/bin/env bash +exit 0 +SH + for script in fm-subagent-pretool-check.sh fm-cd-pretool-check.sh fm-arm-pretool-check.sh; do + cat > "$fixture/bin/$script" <<'SH' +#!/usr/bin/env bash +exit 0 +SH + done + chmod +x "$fixture/bin/"*.sh + printf '%s\n' "$fixture" +} + +# The durable claim is the only thing that can re-present a wake batch whose +# notification a session or process never handled. Its handover therefore has to +# be exactly-once per owner: a same-process extension reload must not repeat the +# batch, while a replacement session and a replacement process each must. +test_native_omp_wake_claim_replay_is_exactly_once() { + local fixture first second status=0 + fixture=$(make_omp_claim_fixture native-wake-claim) + cat > "$fixture/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +state=${FM_STATE_OVERRIDE:?} +count=$(cat "$state/watch-count" 2>/dev/null || printf 0) +count=$((count + 1)) +printf '%s\n' "$count" > "$state/watch-count" +printf 'watcher: started pid=%s (beacon fresh)\n' "$$" +trap 'exit 0' TERM INT +if [ "$count" -eq 1 ]; then + while [ ! -e "$state/watch-trigger" ]; do sleep 0.02; done + printf 'signal: omp durable wake batch\n' + exit 0 +fi +while [ ! -e "$state/watch-stop" ]; do sleep 0.02; done +SH + chmod +x "$fixture/bin/fm-watch-arm.sh" + FM_STATE_OVERRIDE="$fixture/state" bash -c \ + '. "$1/bin/fm-wake-lib.sh"; fm_wake_append signal task-a.status "signal: task-a"' _ "$fixture" \ + || fail "the OMP claim fixture could not seed a durable wake row" + + first=$(EXTENSION="$fixture/.omp/extensions/fm-primary-omp.ts" FM_HOME="$fixture" \ + FM_ROOT_OVERRIDE="$fixture" FM_STATE_OVERRIDE="$fixture/state" FM_CONFIG_OVERRIDE="$fixture/config" \ + node --input-type=module 2>&1 <<'JS' +import { spawnSync } from "node:child_process"; +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +const state = process.env.FM_STATE_OVERRIDE; +const claimScript = `${process.env.FM_ROOT_OVERRIDE}/bin/fm-omp-wake-claim.sh`; +const showClaim = () => { + const result = spawnSync(claimScript, ["show"], { encoding: "utf8" }); + return result.status === 0 ? result.stdout.trim() : ""; +}; +const wakes = []; +const makeApi = () => ({ + zod: { object: () => ({}) }, + on(name, handler) { this.handlers.set(name, handler); }, + handlers: new Map(), + registerCommand() {}, + registerTool() {}, + sendMessage(message, options) { + if (message?.customType !== "firstmate-watcher-wake") return; + wakes.push({ content: String(message.content ?? ""), options, claimAtSend: showClaim() }); + }, +}); +const count = () => existsSync(`${state}/watch-count`) + ? Number(readFileSync(`${state}/watch-count`, "utf8").trim()) + : 0; +async function waitFor(pred, label) { + for (let i = 0; i < 500; i += 1) { + if (pred()) return; + await new Promise((resolve) => setTimeout(resolve, 10)); + } + throw new Error(`timeout waiting for ${label}`); +} +const context = (id) => ({ sessionManager: { getSessionFile: () => undefined, getSessionId: () => id } }); + +writeFileSync(`${state}/.lock`, `${process.pid}\n`); +process.argv[1] = process.env.EXTENSION; +const load = async (tag) => { + const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?${tag}=${Date.now()}`); + const api = makeApi(); + module.default(api); + return api; +}; + +const first = await load("claim"); +await first.handlers.get("session_start")({ type: "session_start" }, context("sess-one")); +await waitFor(() => count() === 1, "initial automatic OMP arm"); +writeFileSync(`${state}/watch-trigger`, "trigger\n"); +await waitFor(() => wakes.length === 1 && count() >= 2, "the durable wake batch notification"); + +const delivered = wakes[0]; +if (delivered.options?.deliverAs !== "nextTurn" || delivered.options?.triggerTurn !== true) { + throw new Error(`the wake batch was not delivered as a hidden next-turn message: ${JSON.stringify(delivered.options)}`); +} +if (!delivered.content.includes("signal: omp durable wake batch")) { + throw new Error(`the wake batch lost its reason line: ${delivered.content}`); +} +if (!delivered.claimAtSend) { + throw new Error("the durable claim was not published before the notification was delivered"); +} +const [, instanceOne, sessionOne, cutoff] = delivered.claimAtSend.split("\t"); +if (!/^[0-9]+$/.test(cutoff) || Number(cutoff) < 1) { + throw new Error(`the claim did not cover the queued durable row: ${delivered.claimAtSend}`); +} +// Acknowledging consumption clears the core's own undelivered-close handoff, so +// from here the claim is the only thing that can re-present this batch. +first.handlers.get("before_agent_start")({ type: "before_agent_start", prompt: delivered.content }, {}); + +// A same-session extension reload re-enters this process with the same session. +const reloaded = await load("reload"); +await reloaded.handlers.get("session_start")({ type: "session_start" }, context("sess-one")); +await new Promise((resolve) => setTimeout(resolve, 200)); +if (wakes.length !== 1) { + throw new Error(`a same-session extension reload repeated the batch: ${wakes.length} notifications`); +} + +// A replacement session must re-present the exact batch once, and only once. +await reloaded.handlers.get("session_switch")({ type: "session_switch", reason: "new" }, context("sess-two")); +await waitFor(() => wakes.length === 2, "the replacement-session re-presentation"); +if (wakes[1].content !== delivered.content) { + throw new Error(`the replacement session re-presented a different batch: ${wakes[1].content}`); +} +if (wakes[1].options?.deliverAs !== "nextTurn" || wakes[1].options?.triggerTurn !== true) { + throw new Error(`the re-presentation changed delivery mode: ${JSON.stringify(wakes[1].options)}`); +} +await reloaded.handlers.get("session_switch")({ type: "session_switch", reason: "resume" }, context("sess-two")); +await new Promise((resolve) => setTimeout(resolve, 200)); +if (wakes.length !== 2) { + throw new Error(`the same replacement session re-presented the batch twice: ${wakes.length} notifications`); +} +const rebound = showClaim().split("\t"); +if (rebound[1] !== instanceOne) throw new Error("a same-process replay changed the process identity"); +if (rebound[2] === sessionOne) throw new Error("the claim was not rebound to the replacement session"); + +writeFileSync(`${state}/watch-stop`, "stop\n"); +await reloaded.handlers.get("session_shutdown")({ type: "session_shutdown" }, {}); +console.log(JSON.stringify({ ok: "omp-wake-claim-replay-ok", body: delivered.content })); +JS + ) || status=$? + printf 'stop\n' > "$fixture/state/watch-stop" 2>/dev/null || true + expect_code 0 "$status" "OMP durable wake claim replay" + assert_contains "$first" omp-wake-claim-replay-ok "OMP wake claim replay did not complete: $first" + + # A replacement PROCESS re-presents the same outstanding batch exactly once, + # even though it inherits the same session and may reuse the former PID. + rm -f "$fixture/state/watch-count" "$fixture/state/watch-trigger" "$fixture/state/watch-stop" + second=$(EXTENSION="$fixture/.omp/extensions/fm-primary-omp.ts" FM_HOME="$fixture" \ + FM_ROOT_OVERRIDE="$fixture" FM_STATE_OVERRIDE="$fixture/state" FM_CONFIG_OVERRIDE="$fixture/config" \ + node --input-type=module 2>&1 <<'JS' +import { spawnSync } from "node:child_process"; +import { writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +const state = process.env.FM_STATE_OVERRIDE; +const claimScript = `${process.env.FM_ROOT_OVERRIDE}/bin/fm-omp-wake-claim.sh`; +const wakes = []; +const handlers = new Map(); +const api = { + zod: { object: () => ({}) }, + on(name, handler) { handlers.set(name, handler); }, + registerCommand() {}, + registerTool() {}, + sendMessage(message) { + if (message?.customType === "firstmate-watcher-wake") wakes.push(String(message.content ?? "")); + }, +}; +const before = spawnSync(claimScript, ["show"], { encoding: "utf8" }); +if (before.status !== 0) throw new Error("the outstanding claim did not survive the previous process"); +const previousInstance = before.stdout.trim().split("\t")[1]; + +writeFileSync(`${state}/.lock`, `${process.pid}\n`); +process.argv[1] = process.env.EXTENSION; +const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?replacementprocess=${Date.now()}`); +module.default(api); +const context = { sessionManager: { getSessionFile: () => undefined, getSessionId: () => "sess-two" } }; +await handlers.get("session_start")({ type: "session_start" }, context); +for (let i = 0; i < 200 && wakes.length === 0; i += 1) { + await new Promise((resolve) => setTimeout(resolve, 10)); +} +if (wakes.length !== 1) { + throw new Error(`a replacement process re-presented the batch ${wakes.length} times`); +} +await new Promise((resolve) => setTimeout(resolve, 200)); +if (wakes.length !== 1) throw new Error("a replacement process kept re-presenting the batch"); +const after = spawnSync(claimScript, ["show"], { encoding: "utf8" }); +if (after.status !== 0) throw new Error("the replacement process retired the claim before acknowledgement"); +if (after.stdout.trim().split("\t")[1] === previousInstance) { + throw new Error("the claim was not rebound to the replacement process"); +} +writeFileSync(`${state}/watch-stop`, "stop\n"); +await handlers.get("session_shutdown")({ type: "session_shutdown" }, {}); +console.log(JSON.stringify({ ok: "omp-wake-claim-process-replay-ok", body: wakes[0] })); +JS + ) || status=$? + printf 'stop\n' > "$fixture/state/watch-stop" 2>/dev/null || true + expect_code 0 "$status" "OMP durable wake claim process replay" + assert_contains "$second" omp-wake-claim-process-replay-ok \ + "a replacement OMP process did not re-present the outstanding batch exactly once: $second" + assert_contains "$second" "signal: omp durable wake batch" \ + "the replacement process re-presented a different batch: $second" + pass "OMP re-presents an unacknowledged wake batch once per replacement session and process, never on a reload" +} + +# Two mechanisms could re-present the same wake after a replacement: the shared +# core's own handoff for a close it never delivered, and this adapter's durable +# claim. Exactly one of them may speak, or the replacement receives the wake +# twice. +test_native_omp_wake_claim_defers_to_a_core_owned_close() { + local fixture out status=0 + fixture=$(make_omp_claim_fixture native-wake-claim-interlock) + cat > "$fixture/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +state=${FM_STATE_OVERRIDE:?} +printf 'watcher: started pid=%s (beacon fresh)\n' "$$" +trap 'exit 0' TERM INT +while [ ! -e "$state/watch-stop" ]; do sleep 0.02; done +SH + chmod +x "$fixture/bin/fm-watch-arm.sh" + mkdir -p "$fixture/state/extensions/omp-primary-watch" + printf '{"version":2,"pending":[{"version":1,"token":"1-2-3","message":"signal: core owned undelivered close","predecessorArmPid":""}]}\n' \ + > "$fixture/state/extensions/omp-primary-watch/session-replacement-actionable.json" + printf '%s' 'encoded:watcher:FIRSTMATE WATCHER WAKE: signal: claimed batch' \ + | FM_STATE_OVERRIDE="$fixture/state" "$fixture/bin/fm-omp-wake-claim.sh" \ + publish --instance inst-previous --session sess-previous \ + || fail "the interlock fixture could not publish an outstanding claim" + + out=$(EXTENSION="$fixture/.omp/extensions/fm-primary-omp.ts" FM_HOME="$fixture" \ + FM_ROOT_OVERRIDE="$fixture" FM_STATE_OVERRIDE="$fixture/state" FM_CONFIG_OVERRIDE="$fixture/config" \ + node --input-type=module 2>&1 <<'JS' +import { spawnSync } from "node:child_process"; +import { writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +const state = process.env.FM_STATE_OVERRIDE; +const claimScript = `${process.env.FM_ROOT_OVERRIDE}/bin/fm-omp-wake-claim.sh`; +const wakes = []; +const handlers = new Map(); +const api = { + zod: { object: () => ({}) }, + on(name, handler) { handlers.set(name, handler); }, + registerCommand() {}, + registerTool() {}, + sendMessage(message) { + if (message?.customType === "firstmate-watcher-wake") wakes.push(String(message.content ?? "")); + }, +}; +writeFileSync(`${state}/.lock`, `${process.pid}\n`); +process.argv[1] = process.env.EXTENSION; +const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?interlock=${Date.now()}`); +module.default(api); +await handlers.get("session_start")({ type: "session_start" }, { + sessionManager: { getSessionFile: () => undefined, getSessionId: () => "sess-new" }, +}); +for (let i = 0; i < 300 && wakes.length === 0; i += 1) { + await new Promise((resolve) => setTimeout(resolve, 10)); +} +await new Promise((resolve) => setTimeout(resolve, 300)); +if (wakes.length !== 1) { + throw new Error(`the replacement received ${wakes.length} wakes for one outstanding close: ${wakes.join(" | ")}`); +} +if (!wakes[0].includes("signal: core owned undelivered close")) { + throw new Error(`the core's undelivered close was not the wake that was delivered: ${wakes[0]}`); +} +if (wakes[0].includes("claimed batch")) { + throw new Error("the adapter re-presented its claim alongside the core's own redelivery"); +} +const claim = spawnSync(claimScript, ["show"], { encoding: "utf8" }); +if (claim.status !== 0) throw new Error("the silent handover retired the claim before acknowledgement"); +if (claim.stdout.trim().split("\t")[1] === "inst-previous") { + throw new Error("the silent handover left the claim bound to the replaced process"); +} +writeFileSync(`${state}/watch-stop`, "stop\n"); +await handlers.get("session_shutdown")({ type: "session_shutdown" }, {}); +console.log("omp-wake-claim-interlock-ok"); +JS + ) || status=$? + printf 'stop\n' > "$fixture/state/watch-stop" 2>/dev/null || true + expect_code 0 "$status" "OMP wake claim interlock" + assert_contains "$out" omp-wake-claim-interlock-ok \ + "the OMP wake claim and the core's own handoff both re-presented one close: $out" + pass "OMP hands its wake claim over silently while the shared core still owes an undelivered close" +} + test_resolve_path_uses_node_when_readlink_f_is_unavailable test_exact_bun_omp_primary_identity test_standalone_omp_primary_identity @@ -1247,3 +1567,5 @@ test_native_omp_confirms_recovery_handling_delivery test_native_omp_refused_handling_delivery_is_typed_once test_native_omp_session_switch_carries_inflight_actionable_close test_native_omp_unacknowledged_wake_keeps_successor_chain +test_native_omp_wake_claim_replay_is_exactly_once +test_native_omp_wake_claim_defers_to_a_core_owned_close diff --git a/tests/fm-wake-queue.test.sh b/tests/fm-wake-queue.test.sh index 2fc58219df8..7e71565e572 100755 --- a/tests/fm-wake-queue.test.sh +++ b/tests/fm-wake-queue.test.sh @@ -13,6 +13,21 @@ set -u WATCH="$ROOT/bin/fm-watch.sh" DRAIN="$ROOT/bin/fm-wake-drain.sh" +OMP_WAKE_CLAIM="$ROOT/bin/fm-omp-wake-claim.sh" + +# Bind an OMP primary wake-notification claim for , exactly as the OMP +# adapter does before it notifies. +publish_omp_wake_claim() { # + printf '%s' "$4" | FM_STATE_OVERRIDE="$1" "$OMP_WAKE_CLAIM" publish --instance "$2" --session "$3" +} + +omp_wake_claim_cutoff() { # + FM_STATE_OVERRIDE="$1" "$OMP_WAKE_CLAIM" show | awk -F '\t' '{ print $4 }' +} + +omp_wake_claim_outstanding() { # + FM_STATE_OVERRIDE="$1" "$OMP_WAKE_CLAIM" show > /dev/null 2>&1 +} TMP_ROOT=$(fm_test_tmproot fm-wake-tests) @@ -990,6 +1005,145 @@ test_turnend_marker_consumer_incarnation_gate() { pass "consumer fires only the live gen marker and ignores stale gens, so a delayed old gen never overwrites or drops a live completion" } +# The OMP primary's durable wake claim exists so a replacement session or +# process can re-present an unacknowledged batch. Retirement must therefore be +# bound to acknowledgement of the durable rows, never to their presentation: +# only the acknowledgement that removes the last covered row may clear it, and a +# wake appended after the claim is above its cutoff, so it neither holds the +# claim open nor is swallowed by it. +test_omp_wake_claim_retires_only_after_acknowledgement() { + local dir state sequence generation + dir=$(make_case omp-claim-ack) + state="$dir/state" + append_wake "$state" signal "task-a.status" "signal: task-a" || fail "first append failed" + append_wake "$state" heartbeat fleet "heartbeat" || fail "second append failed" + publish_omp_wake_claim "$state" inst-one sess-one 'FIRSTMATE WATCHER WAKE: signal: task-a' \ + || fail "the OMP wake claim could not be published" + [ "$(omp_wake_claim_cutoff "$state")" = 2 ] \ + || fail "the claim did not cover both queued rows: $(omp_wake_claim_cutoff "$state")" + + FM_STATE_OVERRIDE="$state" "$DRAIN" > "$dir/present.out" 2> "$dir/present.err" \ + || fail "presentation drain failed: $(cat "$dir/present.err")" + omp_wake_claim_outstanding "$state" \ + || fail "presenting the batch retired the claim before any acknowledgement" + + generation=$(recovery_marker_generation "$state/.watcher-down") + [ -n "$generation" ] || fail "presentation left no recovery generation" + + # A partial acknowledgement leaves a covered row queued, so the claim stays. + FM_STATE_OVERRIDE="$state" "$DRAIN" --ack-through 1 --recovery-generation "$generation" \ + > /dev/null 2>&1 || fail "partial acknowledgement failed" + grep -Fq "$(printf '\theartbeat\tfleet\t')" "$state/.wake-queue" \ + || fail "the partial acknowledgement consumed a row above its cutoff" + omp_wake_claim_outstanding "$state" \ + || fail "the claim was retired while a covered row was still queued" + + # A wake appended after publication is above the cutoff: it must not hold the + # claim open once every covered row is acknowledged. + append_wake "$state" signal "task-b.status" "signal: task-b" || fail "late append failed" + FM_STATE_OVERRIDE="$state" "$DRAIN" > /dev/null 2> "$dir/second.err" || fail "second presentation failed" + generation=$(recovery_marker_generation "$state/.watcher-down") + FM_STATE_OVERRIDE="$state" "$DRAIN" --ack-through 2 --recovery-generation "$generation" \ + > /dev/null 2>&1 || fail "covered acknowledgement failed" + grep -Fq "$(printf '\tsignal\ttask-b.status\t')" "$state/.wake-queue" \ + || fail "the acknowledgement consumed the later wake its cutoff never covered" + omp_wake_claim_outstanding "$state" \ + && fail "the claim survived the acknowledgement of every row it covered" + pass "an OMP wake claim is retired only by the acknowledgement that consumes its last covered row" +} + +# An interrupted handling turn must leave both halves durable: the queue rows +# for idempotent re-presentation, and the claim so a replacement re-notifies +# them. A second drain has to show exactly the same rows. +test_omp_wake_claim_survives_interrupted_handling() { + local dir state before after + dir=$(make_case omp-claim-interrupted) + state="$dir/state" + append_wake "$state" signal "task-a.status" "signal: task-a" || fail "append failed" + publish_omp_wake_claim "$state" inst-one sess-one 'FIRSTMATE WATCHER WAKE: signal: task-a' \ + || fail "the OMP wake claim could not be published" + FM_STATE_OVERRIDE="$state" "$DRAIN" > "$dir/first.out" 2>/dev/null || fail "first drain failed" + before=$(awk -F '\t' 'NF == 5 { print $2 "|" $3 "|" $4 }' "$dir/first.out") + # No acknowledgement: this is the interrupted turn. + FM_STATE_OVERRIDE="$state" "$DRAIN" > "$dir/second.out" 2>/dev/null || fail "second drain failed" + after=$(awk -F '\t' 'NF == 5 { print $2 "|" $3 "|" $4 }' "$dir/second.out") + [ -n "$before" ] && [ "$before" = "$after" ] \ + || fail "an interrupted handling turn changed the durable rows: [$before] vs [$after]" + omp_wake_claim_outstanding "$state" \ + || fail "an interrupted handling turn retired the claim for rows still queued" + pass "an interruption before acknowledgement leaves the OMP wake claim and its durable rows intact" +} + +# A mixed queue is acknowledged by two different actors. Retirement reads the +# queue rather than an actor, so the claim has to outlive whichever +# acknowledgement lands first: main's ack cannot retire a claim the supervision +# branch still owes a covered row for. +test_omp_wake_claim_waits_for_every_actor() { + local dir state grant sequence generation + grant="$ROOT/bin/fm-wake-grant.sh" + dir=$(make_case omp-claim-actors) + state="$dir/state" + append_wake "$state" check "some-poll.check.sh" "check: some-poll" || fail "check append failed" + append_wake "$state" signal "task-a.status" "signal: task-a" || fail "signal append failed" + publish_omp_wake_claim "$state" inst-one sess-one 'FIRSTMATE WATCHER WAKE: check: some-poll' \ + || fail "the OMP wake claim could not be published" + FM_STATE_OVERRIDE="$state" "$grant" activate "$$" omp-claim-actors || fail "branch owner activation failed" + FM_STATE_OVERRIDE="$state" "$grant" publish omp-claim-actors 2 || fail "branch grant publication failed" + + FM_STATE_OVERRIDE="$state" "$DRAIN" > /dev/null 2> "$dir/main.err" || fail "main drain failed" + sequence=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through \([0-9][0-9]*\) --recovery-generation [A-Za-z0-9._-][A-Za-z0-9._-]*$/\1/p' "$dir/main.err") + generation=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through [0-9][0-9]* --recovery-generation \([A-Za-z0-9._-][A-Za-z0-9._-]*\)$/\1/p' "$dir/main.err") + [ -n "$sequence" ] && [ -n "$generation" ] || fail "main drain omitted its acknowledgement boundary" + FM_STATE_OVERRIDE="$state" "$DRAIN" --ack-through "$sequence" --recovery-generation "$generation" \ + > /dev/null 2>&1 || fail "main acknowledgement failed" + omp_wake_claim_outstanding "$state" \ + || fail "main's acknowledgement retired a claim whose branch-owned row was still queued" + + FM_STATE_OVERRIDE="$state" FM_SUPERVISION_ACTOR=branch "$DRAIN" > /dev/null 2> "$dir/branch.err" \ + || fail "branch drain failed" + sequence=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through \([0-9][0-9]*\) --recovery-generation [A-Za-z0-9._-][A-Za-z0-9._-]*$/\1/p' "$dir/branch.err") + generation=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through [0-9][0-9]* --recovery-generation \([A-Za-z0-9._-][A-Za-z0-9._-]*\)$/\1/p' "$dir/branch.err") + [ -n "$sequence" ] && [ -n "$generation" ] || fail "branch drain omitted its acknowledgement boundary" + FM_STATE_OVERRIDE="$state" FM_SUPERVISION_ACTOR=branch "$DRAIN" \ + --ack-through "$sequence" --recovery-generation "$generation" > /dev/null 2>&1 \ + || fail "branch acknowledgement failed" + omp_wake_claim_outstanding "$state" \ + && fail "the claim survived after both actors acknowledged every covered row" + pass "an OMP wake claim outlives a partial actor acknowledgement and retires on the one clearing its last row" +} + +# Watcher appends and handling turns run concurrently. Publishing under the +# queue lock must keep the cutoff monotonic against those appends, so no durable +# row is lost, no claim is retired while a covered row is queued, and the full +# acknowledgement still clears it. +test_omp_wake_claim_holds_under_concurrent_appends() { + local dir state pids pid i cutoff queued + dir=$(make_case omp-claim-concurrent) + state="$dir/state" + pids= + i=1 + while [ "$i" -le 12 ]; do + append_wake "$state" signal "task-$i.status" "signal: task-$i" & + pids="$pids $!" + publish_omp_wake_claim "$state" "inst-$i" sess-one "FIRSTMATE WATCHER WAKE: signal: task-$i" & + pids="$pids $!" + i=$((i + 1)) + done + for pid in $pids; do + wait "$pid" || fail "a concurrent append or claim publication failed" + done + cutoff=$(omp_wake_claim_cutoff "$state") + case "$cutoff" in ''|*[!0-9]*) fail "concurrent publication left no readable claim cutoff" ;; esac + queued=$(awk -F '\t' 'NF == 5 && $2 ~ /^[0-9]+$/ && $2 > max { max = $2 } END { print max + 0 }' "$state/.wake-queue") + [ "$queued" -eq 12 ] || fail "concurrent appends lost a durable row: highest sequence $queued" + FM_STATE_OVERRIDE="$state" "$DRAIN" > /dev/null 2> "$dir/present.err" || fail "presentation drain failed" + omp_wake_claim_outstanding "$state" || fail "a claim was retired while its covered rows were queued" + ack_drain_err "$state" "$dir/present.err" > /dev/null 2>&1 || fail "acknowledgement failed" + [ ! -s "$state/.wake-queue" ] || fail "the acknowledgement left durable rows queued" + omp_wake_claim_outstanding "$state" && fail "the claim survived a full acknowledgement" + pass "concurrent watcher appends and claim publications lose no wake and keep retirement acknowledgement-bound" +} + test_turnend_marker_consumer_incarnation_gate test_stale_acknowledgement_names_current_presented_wake test_concurrent_append_and_drain @@ -1013,3 +1167,7 @@ test_handling_confirmation_is_bounded_by_foreign_marker_lock test_branch_actor_scoped_ack_never_swallows_a_main_owned_row test_main_drain_excludes_rows_already_granted_to_branch test_branch_owner_activation_rollback_stops_after_publication +test_omp_wake_claim_retires_only_after_acknowledgement +test_omp_wake_claim_survives_interrupted_handling +test_omp_wake_claim_waits_for_every_actor +test_omp_wake_claim_holds_under_concurrent_appends From ad6fc403c83d4912da649c06760b138e29423a93 Mon Sep 17 00:00:00 2001 From: dnth Date: Sun, 6 Sep 2026 18:57:38 +0800 Subject: [PATCH 2/7] no-mistakes(review): Documented durable fallback and added publication-failure regression test --- .omp/extensions/fm-primary-omp.ts | 8 ++- tests/fm-omp-primary.test.sh | 94 +++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+), 3 deletions(-) diff --git a/.omp/extensions/fm-primary-omp.ts b/.omp/extensions/fm-primary-omp.ts index 007a9a61ce4..48217d658d0 100644 --- a/.omp/extensions/fm-primary-omp.ts +++ b/.omp/extensions/fm-primary-omp.ts @@ -155,9 +155,11 @@ function claimEnv(): NodeJS.ProcessEnv { } // Bind the durable claim for one outstanding wake notification. Never rejects: -// the durable wake queue stays the authority for the batch itself, so a claim -// that cannot be published costs a re-presentation after a replacement, while -// failing here would either cancel the wake or make the core redeliver it. +// publication failure still delivers the wake and leaves the durable wake-queue +// row queued as the authority, so session-start drain re-presents it after a +// replacement; claim replay is best-effort re-notification layered over that +// durable-row backstop, not the loss-prevention mechanism. Failing here would +// either cancel the wake or make the core redeliver it, which is a duplicate. function publishWakeClaim(session: string, content: string): Promise { return new Promise((resolveClaim) => { const child = spawn( diff --git a/tests/fm-omp-primary.test.sh b/tests/fm-omp-primary.test.sh index 807e9de453b..ecc7ecc74ac 100755 --- a/tests/fm-omp-primary.test.sh +++ b/tests/fm-omp-primary.test.sh @@ -1474,6 +1474,99 @@ JS pass "OMP re-presents an unacknowledged wake batch once per replacement session and process, never on a reload" } +# Claim publication is best-effort: when its executable is unavailable, the +# live session still receives the wake and the durable row remains for drain. +test_native_omp_wake_claim_publication_failure_keeps_queue_authoritative() { + local fixture out status=0 + fixture=$(make_omp_claim_fixture native-wake-claim-publication-failure) + cat > "$fixture/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +state=${FM_STATE_OVERRIDE:?} +count=$(cat "$state/watch-count" 2>/dev/null || printf 0) +count=$((count + 1)) +printf '%s\n' "$count" > "$state/watch-count" +printf 'watcher: started pid=%s (beacon fresh)\n' "$$" +trap 'exit 0' TERM INT +if [ "$count" -eq 1 ]; then + while [ ! -e "$state/watch-trigger" ]; do sleep 0.02; done + printf 'signal: omp claim publication failure\n' + exit 0 +fi +while [ ! -e "$state/watch-stop" ]; do sleep 0.02; done +SH + chmod +x "$fixture/bin/fm-watch-arm.sh" + FM_STATE_OVERRIDE="$fixture/state" bash -c \ + '. "$1/bin/fm-wake-lib.sh"; fm_wake_append signal task-a.status "signal: task-a"' _ "$fixture" \ + || fail "the OMP claim publication-failure fixture could not seed a durable wake row" + chmod a-x "$fixture/bin/fm-omp-wake-claim.sh" + + out=$(EXTENSION="$fixture/.omp/extensions/fm-primary-omp.ts" FM_HOME="$fixture" \ + FM_ROOT_OVERRIDE="$fixture" FM_STATE_OVERRIDE="$fixture/state" FM_CONFIG_OVERRIDE="$fixture/config" \ + node --input-type=module 2>&1 <<'JS' +import { spawnSync } from "node:child_process"; +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +const root = process.env.FM_ROOT_OVERRIDE; +const state = process.env.FM_STATE_OVERRIDE; +const claimScript = `${root}/bin/fm-omp-wake-claim.sh`; +const wakes = []; +const handlers = new Map(); +const api = { + zod: { object: () => ({}) }, + on(name, handler) { handlers.set(name, handler); }, + registerCommand() {}, + registerTool() {}, + sendMessage(message, options) { + if (message?.customType === "firstmate-watcher-wake") wakes.push({ message, options }); + }, +}; +const count = () => existsSync(`${state}/watch-count`) + ? Number(readFileSync(`${state}/watch-count`, "utf8").trim()) + : 0; +async function waitFor(pred, label) { + for (let i = 0; i < 500; i += 1) { + if (pred()) return; + await new Promise((resolve) => setTimeout(resolve, 10)); + } + throw new Error(`timeout waiting for ${label}`); +} +writeFileSync(`${state}/.lock`, `${process.pid}\n`); +process.argv[1] = process.env.EXTENSION; +const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?publication-failure=${Date.now()}`); +module.default(api); +const context = { sessionManager: { getSessionFile: () => undefined, getSessionId: () => "sess-one" } }; +await handlers.get("session_start")({ type: "session_start" }, context); +await waitFor(() => count() === 1, "initial automatic OMP arm"); +writeFileSync(`${state}/watch-trigger`, "trigger\n"); +await waitFor(() => wakes.length === 1, "watcher wake after claim publication failure"); +await new Promise((resolve) => setTimeout(resolve, 100)); +if (wakes.length !== 1) throw new Error(`the watcher wake was delivered ${wakes.length} times`); +if (wakes[0].options?.deliverAs !== "nextTurn" || wakes[0].options?.triggerTurn !== true) { + throw new Error(`the watcher wake used the wrong delivery mode: ${JSON.stringify(wakes[0].options)}`); +} +if (!wakes[0].message.content.includes("signal: omp claim publication failure")) { + throw new Error(`the watcher wake lost its reason line: ${wakes[0].message.content}`); +} +writeFileSync(`${state}/watch-stop`, "stop\n"); +await handlers.get("session_shutdown")({ type: "session_shutdown" }, {}); +console.log("omp-wake-claim-publication-failure-delivered-once"); +JS + ) || status=$? + printf 'stop\n' > "$fixture/state/watch-stop" 2>/dev/null || true + chmod +x "$fixture/bin/fm-omp-wake-claim.sh" + expect_code 0 "$status" "OMP wake claim publication failure delivery" + assert_contains "$out" omp-wake-claim-publication-failure-delivered-once \ + "claim publication failure did not preserve one watcher wake: $out" + if "$fixture/bin/fm-omp-wake-claim.sh" show >/dev/null 2>&1; then + fail "a claim remained outstanding after publication failure" + fi + queued=$(FM_STATE_OVERRIDE="$fixture/state" bash -c \ + '. "$1/bin/fm-wake-lib.sh"; fm_wake_queued_keys signal' _ "$fixture") + [ "$queued" = task-a.status ] || fail "the durable wake row was not left queued: $queued" + pass "OMP claim publication failure falls back to the durable wake row" +} + # Two mechanisms could re-present the same wake after a replacement: the shared # core's own handoff for a close it never delivered, and this adapter's durable # claim. Exactly one of them may speak, or the replacement receives the wake @@ -1568,4 +1661,5 @@ test_native_omp_refused_handling_delivery_is_typed_once test_native_omp_session_switch_carries_inflight_actionable_close test_native_omp_unacknowledged_wake_keeps_successor_chain test_native_omp_wake_claim_replay_is_exactly_once +test_native_omp_wake_claim_publication_failure_keeps_queue_authoritative test_native_omp_wake_claim_defers_to_a_core_owned_close From 60e295a4f0913404fd922d22e71a3b444111379e Mon Sep 17 00:00:00 2001 From: dnth Date: Sun, 6 Sep 2026 19:01:37 +0800 Subject: [PATCH 3/7] no-mistakes(review): Scoped claim assertion to fixture state --- tests/fm-omp-primary.test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/fm-omp-primary.test.sh b/tests/fm-omp-primary.test.sh index ecc7ecc74ac..98a56fbf3ec 100755 --- a/tests/fm-omp-primary.test.sh +++ b/tests/fm-omp-primary.test.sh @@ -1558,7 +1558,7 @@ JS expect_code 0 "$status" "OMP wake claim publication failure delivery" assert_contains "$out" omp-wake-claim-publication-failure-delivered-once \ "claim publication failure did not preserve one watcher wake: $out" - if "$fixture/bin/fm-omp-wake-claim.sh" show >/dev/null 2>&1; then + if FM_STATE_OVERRIDE="$fixture/state" "$fixture/bin/fm-omp-wake-claim.sh" show >/dev/null 2>&1; then fail "a claim remained outstanding after publication failure" fi queued=$(FM_STATE_OVERRIDE="$fixture/state" bash -c \ From e12be6318aaa3630cc1026e562164362d92d3b9d Mon Sep 17 00:00:00 2001 From: dnth Date: Sun, 6 Sep 2026 19:14:07 +0800 Subject: [PATCH 4/7] no-mistakes(review): Synchronized wake claims and covered session-switch ownership --- .omp/extensions/fm-primary-omp.ts | 27 ++++---- tests/fm-omp-primary.test.sh | 111 ++++++++++++++++++++++++++++++ 2 files changed, 123 insertions(+), 15 deletions(-) diff --git a/.omp/extensions/fm-primary-omp.ts b/.omp/extensions/fm-primary-omp.ts index 48217d658d0..78a7856412e 100644 --- a/.omp/extensions/fm-primary-omp.ts +++ b/.omp/extensions/fm-primary-omp.ts @@ -160,20 +160,17 @@ function claimEnv(): NodeJS.ProcessEnv { // replacement; claim replay is best-effort re-notification layered over that // durable-row backstop, not the loss-prevention mechanism. Failing here would // either cancel the wake or make the core redeliver it, which is a duplicate. -function publishWakeClaim(session: string, content: string): Promise { - return new Promise((resolveClaim) => { - const child = spawn( - wakeClaimScript, - ["publish", "--instance", wakeClaimInstance, "--session", session], - { env: claimEnv(), stdio: ["pipe", "ignore", "ignore"] }, - ); - child.on("error", () => resolveClaim()); - child.on("close", () => resolveClaim()); - child.stdin.on("error", () => { - // The claim script may have exited before the body was written. - }); - child.stdin.end(content); - }); +function publishWakeClaim(session: string, content: string): void { + spawnSync( + wakeClaimScript, + ["publish", "--instance", wakeClaimInstance, "--session", session], + { + env: claimEnv(), + input: content, + stdio: ["pipe", "ignore", "ignore"], + timeout: 15000, + }, + ); } // Take over an outstanding claim left by a replacement session or a replacement @@ -343,7 +340,7 @@ export default function (omp: ExtensionAPI) { // replayable claim rather than a notification no successor can // re-present. The claim retires only when bin/fm-wake-drain.sh // acknowledges the durable rows it covers. - await publishWakeClaim(wakeClaimSession, content); + publishWakeClaim(wakeClaimSession, content); sendWakeNotification(content); }, offerWakeToBranch, diff --git a/tests/fm-omp-primary.test.sh b/tests/fm-omp-primary.test.sh index 98a56fbf3ec..6da858f123d 100755 --- a/tests/fm-omp-primary.test.sh +++ b/tests/fm-omp-primary.test.sh @@ -1567,6 +1567,116 @@ JS pass "OMP claim publication failure falls back to the durable wake row" } +test_native_omp_wake_claim_session_switch_cannot_split_owner() { + local fixture out status=0 + fixture=$(make_omp_claim_fixture native-wake-claim-session-switch) + mv "$fixture/bin/fm-omp-wake-claim.sh" "$fixture/bin/fm-omp-wake-claim.real.sh" + cat > "$fixture/bin/fm-omp-wake-claim.sh" <<'SH' +#!/usr/bin/env bash +sleep 0.4 +exec "$(dirname "$0")/fm-omp-wake-claim.real.sh" "$@" +SH + chmod +x "$fixture/bin/fm-omp-wake-claim.sh" + cat > "$fixture/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +state=${FM_STATE_OVERRIDE:?} +printf '1\n' > "$state/watch-count" +printf 'watcher: started pid=%s (beacon fresh)\n' "$$" +trap 'exit 0' TERM INT +while [ ! -e "$state/watch-trigger" ]; do sleep 0.02; done +printf 'signal: omp session-switch race\n' +exit 0 +SH + chmod +x "$fixture/bin/fm-watch-arm.sh" + FM_STATE_OVERRIDE="$fixture/state" bash -c \ + '. "$1/bin/fm-wake-lib.sh"; fm_wake_append signal task-a.status "signal: task-a"' _ "$fixture" \ + || fail "the OMP session-switch fixture could not seed a durable wake row" + + out=$(EXTENSION="$fixture/.omp/extensions/fm-primary-omp.ts" FM_HOME="$fixture" \ + FM_ROOT_OVERRIDE="$fixture" FM_STATE_OVERRIDE="$fixture/state" FM_CONFIG_OVERRIDE="$fixture/config" \ + node --input-type=module 2>&1 <<'JS' +import { createHash } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +const root = process.env.FM_ROOT_OVERRIDE; +const state = process.env.FM_STATE_OVERRIDE; +const claimScript = `${root}/bin/fm-omp-wake-claim.sh`; +const wakes = []; +const handlers = new Map(); +let currentSession = "sess-one"; +let interleaveAttempted = false; +let timer; +const api = { + zod: { object: () => ({}) }, + on(name, handler) { handlers.set(name, handler); }, + registerCommand() {}, + registerTool() {}, + sendMessage(message, options) { + if (message?.customType !== "firstmate-watcher-wake") return; + wakes.push({ message, options }); + clearTimeout(timer); + const claim = spawnSync(claimScript, ["show"], { + encoding: "utf8", + env: { ...process.env, FM_STATE_OVERRIDE: state }, + }); + if (claim.status !== 0) throw new Error("the wake was delivered without a durable claim"); + const owner = claim.stdout.trim().split("\t")[2]; + if (!owner || owner !== createHash("sha256").update(currentSession).digest("hex")) { + throw new Error(`claim owner split from live session: ${owner} vs ${currentSession}`); + } + }, +}; +const count = () => existsSync(`${state}/watch-count`) + ? Number(readFileSync(`${state}/watch-count`, "utf8").trim()) + : 0; +async function waitFor(pred, label) { + for (let i = 0; i < 500; i += 1) { + if (pred()) return; + await new Promise((resolve) => setTimeout(resolve, 10)); + } + throw new Error(`timeout waiting for ${label}`); +} +writeFileSync(`${state}/.lock`, `${process.pid}\n`); +process.argv[1] = process.env.EXTENSION; +const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?session-switch=${Date.now()}`); +module.default(api); +const context = (id) => ({ sessionManager: { getSessionFile: () => undefined, getSessionId: () => id } }); +await handlers.get("session_start")({ type: "session_start" }, context(currentSession)); +await waitFor(() => count() === 1, "initial automatic OMP arm"); +writeFileSync(`${state}/watch-trigger`, "trigger\n"); +timer = setTimeout(() => { + interleaveAttempted = true; + currentSession = "sess-two"; + handlers.get("session_switch")({ type: "session_switch", reason: "new" }, context(currentSession)); +}, 200); +await waitFor(() => wakes.length === 1, "one watcher wake"); +await new Promise((resolve) => setTimeout(resolve, 250)); +if (wakes.length !== 1) throw new Error(`the watcher wake was delivered ${wakes.length} times`); +if (interleaveAttempted) throw new Error("session_switch interleaved despite synchronous publication"); +if (wakes[0].options?.deliverAs !== "nextTurn" || wakes[0].options?.triggerTurn !== true) { + throw new Error(`the watcher wake used the wrong delivery mode: ${JSON.stringify(wakes[0].options)}`); +} +const claim = spawnSync(claimScript, ["show"], { + encoding: "utf8", + env: { ...process.env, FM_STATE_OVERRIDE: state }, +}); +if (claim.status !== 0 || claim.stdout.trim().split("\t").length !== 5) { + throw new Error("the claim did not remain bound to exactly one owner"); +} +writeFileSync(`${state}/watch-stop`, "stop\n"); +await handlers.get("session_shutdown")({ type: "session_shutdown" }, {}); +console.log("omp-wake-claim-session-switch-owner-ok"); +JS + ) || status=$? + printf 'stop\n' > "$fixture/state/watch-stop" 2>/dev/null || true + expect_code 0 "$status" "OMP wake claim session-switch ownership" + assert_contains "$out" omp-wake-claim-session-switch-owner-ok \ + "session-switch interleaving split the wake claim owner: $out" + pass "OMP wake claim publication and notification keep one session owner" +} + # Two mechanisms could re-present the same wake after a replacement: the shared # core's own handoff for a close it never delivered, and this adapter's durable # claim. Exactly one of them may speak, or the replacement receives the wake @@ -1662,4 +1772,5 @@ test_native_omp_session_switch_carries_inflight_actionable_close test_native_omp_unacknowledged_wake_keeps_successor_chain test_native_omp_wake_claim_replay_is_exactly_once test_native_omp_wake_claim_publication_failure_keeps_queue_authoritative +test_native_omp_wake_claim_session_switch_cannot_split_owner test_native_omp_wake_claim_defers_to_a_core_owned_close From c7c836b72dea871f94965d5046c429418f7d208c Mon Sep 17 00:00:00 2001 From: dnth Date: Sun, 6 Sep 2026 20:26:46 +0800 Subject: [PATCH 5/7] no-mistakes(review): Durable queue now owns OMP wake replay --- .agents/skills/harness-adapters/SKILL.md | 2 +- .omp/extensions/fm-primary-omp.ts | 115 ++---- bin/fm-omp-wake-claim-lib.sh | 177 --------- bin/fm-omp-wake-claim.sh | 127 ------- bin/fm-wake-drain.sh | 21 -- docs/scripts.md | 2 - docs/watcher-continuity.md | 11 +- tests/fm-omp-primary.test.sh | 458 +++-------------------- tests/fm-wake-queue.test.sh | 159 -------- 9 files changed, 87 insertions(+), 985 deletions(-) delete mode 100644 bin/fm-omp-wake-claim-lib.sh delete mode 100755 bin/fm-omp-wake-claim.sh diff --git a/.agents/skills/harness-adapters/SKILL.md b/.agents/skills/harness-adapters/SKILL.md index 37ad69e60f1..de751fdbabc 100644 --- a/.agents/skills/harness-adapters/SKILL.md +++ b/.agents/skills/harness-adapters/SKILL.md @@ -337,7 +337,7 @@ Before killing and respawning a live OMP agent for a wedged PROVIDER STREAM, fol **Primary-session integration fact (verified 2026-07-31, OMP 17.1.8).** Plain OMP started from the Firstmate root discovers `.omp/extensions/fm-primary-omp.ts` natively, including in a fresh checkout before canonical `state/` exists; `omp -e .omp/extensions/fm-primary-omp.ts` remains the explicit recovery fallback. The adapter publishes the OMP marker shape owned by [configuration](../../../docs/configuration.md#harness-support), delivers the session-start instruction on native `session_start` and `session_switch` events, and owns watcher generations through `fm_watch_arm_omp` plus `/new` and `/resume` continuity. -It routes watcher wakes through OMP's `sendMessage` as a hidden custom next-turn message with `triggerTurn`, which starts a handling turn from idle and, when the current turn is still unwinding, schedules the continuation that consumes every wake queued during it; [watcher continuity](../../../docs/watcher-continuity.md#actionable-wake-ordering) owns the durable claim that carries an unacknowledged batch across a replacement session or process. +It routes watcher wakes through OMP's `sendMessage` as a hidden custom next-turn message with `triggerTurn`, which starts a handling turn from idle and, when the current turn is still unwinding, schedules the continuation that consumes every wake queued during it; [watcher continuity](../../../docs/watcher-continuity.md#actionable-wake-ordering) owns the durable-row acknowledgement and session-event re-notification contract. It runs the shared turn-end predicate through native `session_stop`, and applies the shared watcher-arm, persistent-directory, and delegation-shaped tool safety checks before tool execution. `bin/fm-session-start.sh` rejects a missing, stale, foreign-PID, or version-mismatched loaded marker and prints both native-discovery and explicit `-e` recovery commands. The authoritative operating procedure is `docs/supervision-protocols/omp.md`. diff --git a/.omp/extensions/fm-primary-omp.ts b/.omp/extensions/fm-primary-omp.ts index 78a7856412e..6c6cdf90286 100644 --- a/.omp/extensions/fm-primary-omp.ts +++ b/.omp/extensions/fm-primary-omp.ts @@ -1,8 +1,8 @@ // Firstmate primary integration for OMP. // OMP-native session, stop, tool-call, and shutdown events stay in this adapter. import { spawn, spawnSync } from "node:child_process"; -import { createHash, randomUUID } from "node:crypto"; -import { mkdirSync, renameSync, unlinkSync, writeFileSync } from "node:fs"; +import { randomUUID } from "node:crypto"; +import { mkdirSync, readFileSync, renameSync, unlinkSync, writeFileSync } from "node:fs"; import { dirname, isAbsolute, resolve } from "node:path"; import { fileURLToPath } from "node:url"; import type { @@ -28,15 +28,6 @@ const state = process.env.FM_STATE_OVERRIDE || `${fmHome}/state`; const config = process.env.FM_CONFIG_OVERRIDE || `${fmHome}/config`; const marker = `${state}/.omp-primary-extension-loaded`; const operationalInputScript = `${fmRoot}/bin/fm-operational-input.sh`; -const wakeClaimScript = `${fmRoot}/bin/fm-omp-wake-claim.sh`; - -// One identity per OMP process. A same-process extension reload re-evaluates -// this module and must keep it, so it lives on the realm rather than in module -// scope; a replacement process never inherits it, which is what separates a -// reload from a restart even when the operating system reuses the former PID. -type WakeClaimGlobal = typeof globalThis & { __firstmateOmpWakeClaimInstance?: string }; -const wakeClaimGlobal = globalThis as WakeClaimGlobal; -const wakeClaimInstance = wakeClaimGlobal.__firstmateOmpWakeClaimInstance ??= randomUUID(); type ProcessResult = { code: number; @@ -144,52 +135,16 @@ function runSessionstartNudge(forceForNativeSwitch = false): string { return result.stdout.trim(); } -function claimEnv(): NodeJS.ProcessEnv { - return { - ...process.env, - FM_HOME: fmHome, - FM_ROOT_OVERRIDE: fmRoot, - FM_STATE_OVERRIDE: state, - FM_CONFIG_OVERRIDE: config, - }; -} - -// Bind the durable claim for one outstanding wake notification. Never rejects: -// publication failure still delivers the wake and leaves the durable wake-queue -// row queued as the authority, so session-start drain re-presents it after a -// replacement; claim replay is best-effort re-notification layered over that -// durable-row backstop, not the loss-prevention mechanism. Failing here would -// either cancel the wake or make the core redeliver it, which is a duplicate. -function publishWakeClaim(session: string, content: string): void { - spawnSync( - wakeClaimScript, - ["publish", "--instance", wakeClaimInstance, "--session", session], - { - env: claimEnv(), - input: content, - stdio: ["pipe", "ignore", "ignore"], - timeout: 15000, - }, - ); -} - -// Take over an outstanding claim left by a replacement session or a replacement -// process and return the exact body to re-present. Empty when nothing is -// outstanding, when this process and session already own the claim (a -// same-session extension reload), or when the handover could not complete - the -// claim then keeps its previous owner and the next replay retries it. -function takeOverWakeClaim(session: string): string { - const result = spawnSync( - wakeClaimScript, - ["replay", "--instance", wakeClaimInstance, "--session", session], - // Bounded: this runs inside a session event, so a wedged durable queue lock - // must cost one skipped re-presentation, never a hung OMP session. A - // nonzero or timed-out handover leaves the claim with its previous owner - // for the next session event to retry. - { encoding: "utf8", env: claimEnv(), maxBuffer: 4 * 1024 * 1024, timeout: 15000 }, +function durableWakeQueueHasRows(): boolean { + let queue = ""; + try { + queue = readFileSync(`${state}/.wake-queue`, "utf8"); + } catch { + return false; + } + return queue.split("\n").some((line) => + /^(?:[^\t]*)\t[0-9]+\t(?:signal|stale|check|heartbeat)\t[^\t]*\t[^\t]*$/.test(line), ); - if (result.status !== 0) return ""; - return result.stdout || ""; } function runChecker(script: string, flag: "--command" | "--tool", value: string): Promise { @@ -259,20 +214,6 @@ export default function (omp: ExtensionAPI) { publishNativeProcessIdentity(); const taskInboxDoorbell = installTaskInboxDoorbell(omp); let pendingStartupNudge = ""; - // Per-session half of the claim owner identity. "unbound" until a session - // event supplies one, so a claim written before any session can still be - // matched deterministically rather than looking like a foreign owner. - let wakeClaimSession = "unbound"; - - const bindWakeClaimSession = (ctx: ExtensionContext): void => { - let sessionId = ""; - try { - sessionId = ctx.sessionManager?.getSessionId?.() ?? ""; - } catch { - // A session without a readable identity keeps the unbound placeholder. - } - wakeClaimSession = sessionId ? createHash("sha256").update(sessionId).digest("hex") : "unbound"; - }; // Hidden next-turn delivery with triggerTurn. OMP schedules an internal // continuation bound to the current prompt generation, so a wake that lands @@ -293,15 +234,16 @@ export default function (omp: ExtensionAPI) { ); }; - // Re-present the durable batch a previous session or process notified but - // never got acknowledged. The core keeps its own handoff for a close it has - // not delivered yet and replays that itself, so this takes the claim over - // silently in that case: the handover still happens exactly once, and only - // one of the two mechanisms speaks. - const replayWakeClaim = (coreOwnsDelivery: boolean): void => { - const content = takeOverWakeClaim(wakeClaimSession); - if (!content || coreOwnsDelivery) return; - sendWakeNotification(content); + // Durable rows are the whole persistence: only fm-wake-drain acknowledgement + // removes them, so an interruption leaves the next session event able to + // re-notify. At most one notification is sent per session event, and the + // core remains the sole speaker while it owns an undelivered close. + const notifyQueuedWake = (coreOwnsDelivery: boolean): void => { + if (coreOwnsDelivery || !durableWakeQueueHasRows()) return; + sendWakeNotification(encodeOperationalInput( + "watcher", + "Durable watcher wakes are queued. Run `bin/fm-wake-drain.sh` first to present and acknowledge them.", + )); }; // Supervision-branch dispatch handshake (docs/omp-supervision-branch.md). @@ -335,14 +277,7 @@ export default function (omp: ExtensionAPI) { armReadyTimeoutEnv: "FM_OMP_ARM_READY_TIMEOUT_MS", repairToolName: "fm_watch_arm_omp", encodeOperationalInput, - sendFollowUp: async (content) => { - // Claim first, notify second: an interruption between the two leaves a - // replayable claim rather than a notification no successor can - // re-present. The claim retires only when bin/fm-wake-drain.sh - // acknowledges the durable rows it covers. - publishWakeClaim(wakeClaimSession, content); - sendWakeNotification(content); - }, + sendFollowUp: async (content) => sendWakeNotification(content), offerWakeToBranch, }); @@ -376,7 +311,6 @@ export default function (omp: ExtensionAPI) { }; omp.on("session_start", (_event, ctx) => { - bindWakeClaimSession(ctx); taskInboxDoorbell.activate(); // Read before sessionStart: activating the watch consumes the core's own // handoff, so asking afterwards could not tell who owns the redelivery. @@ -384,7 +318,7 @@ export default function (omp: ExtensionAPI) { watch.sessionStart(); publishSecondmateSession(ctx); deliverSessionstartNudge(); - replayWakeClaim(coreOwnsDelivery); + notifyQueuedWake(coreOwnsDelivery); }); omp.on("turn_start", () => { @@ -393,12 +327,11 @@ export default function (omp: ExtensionAPI) { omp.on("session_switch", async (event, ctx) => { await watch.sessionShutdown(true); - bindWakeClaimSession(ctx); publishSecondmateSession(ctx); deliverSessionstartNudge(event.reason === "new" || event.reason === "resume"); const coreOwnsDelivery = watch.hasPendingActionableHandoff(); watch.sessionStart(); - replayWakeClaim(coreOwnsDelivery); + notifyQueuedWake(coreOwnsDelivery); }); omp.on("before_agent_start", (event): BeforeAgentStartEventResult | undefined => { diff --git a/bin/fm-omp-wake-claim-lib.sh b/bin/fm-omp-wake-claim-lib.sh deleted file mode 100644 index 8096caef5db..00000000000 --- a/bin/fm-omp-wake-claim-lib.sh +++ /dev/null @@ -1,177 +0,0 @@ -#!/usr/bin/env bash -# Durable OMP primary wake-notification claim. -# -# ONE owner of the claim's file format and of every read, publish, replay -# rebind, and retirement performed on it. The claim records that a watcher wake -# notification is outstanding for the durable wake-queue rows at or below a -# sequence cutoff, so a replacement OMP session or a replacement OMP process -# re-presents that exact batch instead of losing it. The durable queue rows stay -# authoritative; the claim only carries the re-notification. -# -# Format (regular file, mode 0600, never a symlink), one field per line: -# 1 fm-omp-wake-claim-v1 -# 2 claim id minted on every write -# 3 owner instance per-OMP-process identity: a same-process extension -# reload keeps it, a replacement process never does -# 4 owner session per-OMP-session identity -# 5 cutoff highest durable wake sequence the claim covers -# 6 content base64 of the exact notification body to re-present -# -# Every _locked function requires the caller to already hold -# FM_WAKE_QUEUE_LOCK. The claim is published against the queue's own sequence -# counter and retired against the queue's own rows, so publication and -# retirement serialize on the single lock the queue already has instead of -# racing across a second one. -# -# Retirement is bound to acknowledgement, never to delivery: a claim is removed -# only once no durable row at or below its cutoff remains queued, and only -# bin/fm-wake-drain.sh's acknowledgement removes those rows. An interruption -# before that acknowledgement therefore leaves the rows and the claim durable -# for idempotent re-handling. - -FM_OMP_WAKE_CLAIM_DIR="${FM_OMP_WAKE_CLAIM_DIR:-$STATE/extensions/omp-primary-watch}" -FM_OMP_WAKE_CLAIM_FILE="${FM_OMP_WAKE_CLAIM_FILE:-$FM_OMP_WAKE_CLAIM_DIR/wake-notification}" - -# Populated by fm_omp_wake_claim_read_locked; empty whenever it returns nonzero. -FM_OMP_WAKE_CLAIM_ID= -FM_OMP_WAKE_CLAIM_INSTANCE= -FM_OMP_WAKE_CLAIM_SESSION= -FM_OMP_WAKE_CLAIM_CUTOFF= -FM_OMP_WAKE_CLAIM_CONTENT_B64= - -# Owner identities are opaque to this library: it only proves they are single -# safe tokens so a malformed claim can never be mistaken for a bound one. -fm_omp_wake_claim_token_ok() { # - case "$1" in - ''|*[!A-Za-z0-9._-]*) return 1 ;; - esac - [ "${#1}" -le 128 ] -} - -fm_omp_wake_claim_new_id() { - printf '%s.%s.%s\n' "$(fm_current_pid)" "$(date +%s)" "${RANDOM}${RANDOM}" -} - -# The highest sequence the durable queue has issued. Read under the queue lock -# so every row already queued is at or below it and every later append is above. -fm_omp_wake_claim_queue_seq_locked() { - local seq - seq=$(cat "$STATE/.wake-queue.seq" 2>/dev/null || printf 0) - case "$seq" in - ''|*[!0-9]*) seq=0 ;; - esac - printf '%s\n' "$seq" -} - -fm_omp_wake_claim_read_locked() { - local version id instance session cutoff content _extra - FM_OMP_WAKE_CLAIM_ID= - FM_OMP_WAKE_CLAIM_INSTANCE= - FM_OMP_WAKE_CLAIM_SESSION= - FM_OMP_WAKE_CLAIM_CUTOFF= - FM_OMP_WAKE_CLAIM_CONTENT_B64= - [ -f "$FM_OMP_WAKE_CLAIM_FILE" ] && [ ! -L "$FM_OMP_WAKE_CLAIM_FILE" ] || return 1 - exec 9< "$FM_OMP_WAKE_CLAIM_FILE" || return 1 - IFS= read -r version <&9 || { exec 9<&-; return 1; } - IFS= read -r id <&9 || { exec 9<&-; return 1; } - IFS= read -r instance <&9 || { exec 9<&-; return 1; } - IFS= read -r session <&9 || { exec 9<&-; return 1; } - IFS= read -r cutoff <&9 || { exec 9<&-; return 1; } - IFS= read -r content <&9 || { exec 9<&-; return 1; } - if IFS= read -r _extra <&9; then - exec 9<&- - return 1 - fi - exec 9<&- - [ "$version" = fm-omp-wake-claim-v1 ] || return 1 - fm_omp_wake_claim_token_ok "$id" || return 1 - fm_omp_wake_claim_token_ok "$instance" || return 1 - fm_omp_wake_claim_token_ok "$session" || return 1 - case "$cutoff" in - ''|*[!0-9]*) return 1 ;; - esac - case "$content" in - ''|*[!A-Za-z0-9+/=]*) return 1 ;; - esac - # shellcheck disable=SC2034 # Read by sourcing callers after a successful read. - FM_OMP_WAKE_CLAIM_ID=$id - FM_OMP_WAKE_CLAIM_INSTANCE=$instance - FM_OMP_WAKE_CLAIM_SESSION=$session - FM_OMP_WAKE_CLAIM_CUTOFF=$cutoff - FM_OMP_WAKE_CLAIM_CONTENT_B64=$content -} - -fm_omp_wake_claim_write_locked() { # - local id=$1 instance=$2 session=$3 cutoff=$4 content=$5 tmp - mkdir -p "$FM_OMP_WAKE_CLAIM_DIR" || return 1 - tmp=$(mktemp "$FM_OMP_WAKE_CLAIM_FILE.tmp.XXXXXX") || return 1 - if ! printf 'fm-omp-wake-claim-v1\n%s\n%s\n%s\n%s\n%s\n' \ - "$id" "$instance" "$session" "$cutoff" "$content" > "$tmp" \ - || ! chmod 0600 "$tmp" \ - || ! _fm_atomic_replace "$tmp" "$FM_OMP_WAKE_CLAIM_FILE"; then - rm -f -- "$tmp" - return 1 - fi -} - -# Bind one outstanding notification to the caller's process and session. A new -# claim replaces an outstanding one rather than accumulating, because OMP's -# hidden next-turn transport already coalesces every queued notification into -# one continuation turn. The cutoff only ever moves forward, so replacing a -# claim can never shorten the row span its retirement waits for. -fm_omp_wake_claim_publish_locked() { # - local instance=$1 session=$2 content=$3 cutoff - fm_omp_wake_claim_token_ok "$instance" || return 1 - fm_omp_wake_claim_token_ok "$session" || return 1 - case "$content" in - ''|*[!A-Za-z0-9+/=]*) return 1 ;; - esac - cutoff=$(fm_omp_wake_claim_queue_seq_locked) || return 1 - if fm_omp_wake_claim_read_locked && [ "$FM_OMP_WAKE_CLAIM_CUTOFF" -gt "$cutoff" ]; then - cutoff=$FM_OMP_WAKE_CLAIM_CUTOFF - fi - fm_omp_wake_claim_write_locked "$(fm_omp_wake_claim_new_id)" "$instance" "$session" "$cutoff" "$content" -} - -# Decide whether the given owner owes a re-presentation, leaving the claim in -# the read globals when it does. A claim already bound to this process and this -# session is a same-session extension reload and must not be re-presented; any -# other binding is a replacement session or a replacement process. -# 0 a replay is due, 1 invalid owner, 3 nothing to replay. -fm_omp_wake_claim_replay_pending_locked() { # - local instance=$1 session=$2 - fm_omp_wake_claim_token_ok "$instance" || return 1 - fm_omp_wake_claim_token_ok "$session" || return 1 - fm_omp_wake_claim_read_locked || return 3 - [ "$FM_OMP_WAKE_CLAIM_INSTANCE" = "$instance" ] && [ "$FM_OMP_WAKE_CLAIM_SESSION" = "$session" ] && return 3 - return 0 -} - -# Move the outstanding claim to a new owner, keeping its cutoff and body. This -# is what makes a re-presentation exactly-once per owner: a second replay under -# the same binding finds nothing to hand over. Callers rebind only after the -# body is safely handed to the new owner, so a failure anywhere earlier leaves -# the claim with its previous owner and the replay simply retries. -fm_omp_wake_claim_rebind_locked() { # - local instance=$1 session=$2 - fm_omp_wake_claim_token_ok "$instance" || return 1 - fm_omp_wake_claim_token_ok "$session" || return 1 - fm_omp_wake_claim_read_locked || return 1 - fm_omp_wake_claim_write_locked "$(fm_omp_wake_claim_new_id)" "$instance" "$session" \ - "$FM_OMP_WAKE_CLAIM_CUTOFF" "$FM_OMP_WAKE_CLAIM_CONTENT_B64" -} - -# Retire the claim exactly when the durable rows it covers are gone. Actor -# agnostic on purpose: a mixed queue can have main acknowledge some covered -# rows and the supervision branch acknowledge the rest, and the claim must -# survive until whichever acknowledgement clears the last one. -fm_omp_wake_claim_reconcile_locked() { - fm_omp_wake_claim_read_locked || return 0 - if [ -s "$FM_WAKE_QUEUE" ] && awk -F '\t' -v cutoff="$FM_OMP_WAKE_CLAIM_CUTOFF" ' - NF >= 5 && $2 ~ /^[0-9]+$/ && $2 + 0 <= cutoff + 0 { covered = 1; exit } - END { exit covered ? 0 : 1 } - ' "$FM_WAKE_QUEUE" 2>/dev/null; then - return 0 - fi - rm -f -- "$FM_OMP_WAKE_CLAIM_FILE" || return 1 -} diff --git a/bin/fm-omp-wake-claim.sh b/bin/fm-omp-wake-claim.sh deleted file mode 100755 index be92f107177..00000000000 --- a/bin/fm-omp-wake-claim.sh +++ /dev/null @@ -1,127 +0,0 @@ -#!/usr/bin/env bash -# Command-line face of the durable OMP primary wake-notification claim. -# bin/fm-omp-wake-claim-lib.sh owns the format, the invariants, and every -# mutation; this script only holds the durable wake-queue lock around one of -# them so the OMP adapter can drive the claim without linking that lock into -# its own process. -# -# Usage: -# fm-omp-wake-claim.sh publish --instance --session # body on stdin -# fm-omp-wake-claim.sh replay --instance --session # body on stdout -# fm-omp-wake-claim.sh show -# -# publish binds one outstanding notification to this OMP process and session. -# replay hands an outstanding claim to a new owner exactly once and prints the -# exact body to re-present; a claim already bound to the given owner is a -# same-session extension reload and prints nothing. -# show prints "\t\t\t\t". -# -# Exit codes: 0 done, 1 failed, 2 usage, 3 nothing to replay or show, -# 4 the durable wake-queue lock stayed busy for the whole bounded wait -# (FM_OMP_WAKE_CLAIM_LOCK_ATTEMPTS attempts, 0.05s apart, default 200). -set -u - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -# shellcheck source=bin/fm-wake-lib.sh -. "$SCRIPT_DIR/fm-wake-lib.sh" -# shellcheck source=bin/fm-omp-wake-claim-lib.sh -. "$SCRIPT_DIR/fm-omp-wake-claim-lib.sh" - -LOCK_HELD=false -CLAIM_TMP= - -usage() { - echo "usage: fm-omp-wake-claim.sh (publish|replay) --instance --session | show" >&2 - exit 2 -} - -# shellcheck disable=SC2317,SC2329 # Invoked by the trap handlers below. -cleanup() { - local status=$? - [ -z "$CLAIM_TMP" ] || rm -f -- "$CLAIM_TMP" 2>/dev/null || true - if [ "$LOCK_HELD" = true ]; then - fm_lock_release "$FM_WAKE_QUEUE_LOCK" - fi - exit "$status" -} - -# Portable decode: GNU coreutils spells it --decode, BSD base64 spells it -D. -# Decode into a file rather than straight to stdout so a rejected first attempt -# can never leave a partial body behind for the second one to append to. -decode_base64_to() { # - if printf '%s' "$1" | base64 --decode > "$2" 2>/dev/null; then return 0; fi - printf '%s' "$1" | base64 -D > "$2" 2>/dev/null -} - -acquire_queue_lock() { - local attempts=${FM_OMP_WAKE_CLAIM_LOCK_ATTEMPTS:-200} attempt=0 - case "$attempts" in - ''|*[!0-9]*|0) attempts=200 ;; - esac - while ! fm_lock_try_acquire "$FM_WAKE_QUEUE_LOCK"; do - attempt=$((attempt + 1)) - if [ "$attempt" -ge "$attempts" ]; then - echo "fm-omp-wake-claim: durable wake queue lock stayed busy" >&2 - exit 4 - fi - sleep 0.05 - done - LOCK_HELD=true -} - -COMMAND=${1:-} -[ -n "$COMMAND" ] || usage -shift || true - -INSTANCE= -SESSION= -case "$COMMAND" in - publish|replay) - while [ "$#" -gt 0 ]; do - case "$1" in - --instance) INSTANCE=${2:-}; shift 2 || usage ;; - --session) SESSION=${2:-}; shift 2 || usage ;; - *) usage ;; - esac - done - fm_omp_wake_claim_token_ok "$INSTANCE" || usage - fm_omp_wake_claim_token_ok "$SESSION" || usage - ;; - show) - [ "$#" -eq 0 ] || usage - ;; - *) usage ;; -esac - -trap cleanup EXIT -trap 'exit 130' INT -trap 'exit 143' TERM - -case "$COMMAND" in - publish) - CLAIM_CONTENT=$(base64 | tr -d '\n') || exit 1 - [ -n "$CLAIM_CONTENT" ] || { echo "fm-omp-wake-claim: refusing to claim an empty notification" >&2; exit 1; } - acquire_queue_lock - fm_omp_wake_claim_publish_locked "$INSTANCE" "$SESSION" "$CLAIM_CONTENT" || exit 1 - ;; - replay) - acquire_queue_lock - fm_omp_wake_claim_replay_pending_locked "$INSTANCE" "$SESSION" || exit "$?" - CLAIM_TMP=$(mktemp "$STATE/.omp-wake-claim.replay.XXXXXX") || exit 1 - decode_base64_to "$FM_OMP_WAKE_CLAIM_CONTENT_B64" "$CLAIM_TMP" || exit 1 - # Hand the body over first and rebind last: an interruption before the - # rebind leaves the claim with its previous owner, so the next replay - # attempt re-presents the same batch instead of losing it. - command cat "$CLAIM_TMP" || exit 1 - fm_omp_wake_claim_rebind_locked "$INSTANCE" "$SESSION" || exit 1 - ;; - show) - acquire_queue_lock - fm_omp_wake_claim_read_locked || exit 3 - printf '%s\t%s\t%s\t%s\t%s\n' \ - "$FM_OMP_WAKE_CLAIM_ID" "$FM_OMP_WAKE_CLAIM_INSTANCE" "$FM_OMP_WAKE_CLAIM_SESSION" \ - "$FM_OMP_WAKE_CLAIM_CUTOFF" "$FM_OMP_WAKE_CLAIM_CONTENT_B64" || exit 1 - ;; -esac - -exit 0 diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index 81c9f8c6162..eed0ef2693b 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -21,8 +21,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" . "$SCRIPT_DIR/fm-line-cap-lib.sh" # shellcheck source=bin/fm-lease-lib.sh . "$SCRIPT_DIR/fm-lease-lib.sh" -# shellcheck source=bin/fm-omp-wake-claim-lib.sh -. "$SCRIPT_DIR/fm-omp-wake-claim-lib.sh" DRAIN_TMP= DRAIN_VIEW_TMP= @@ -379,17 +377,6 @@ print_status_presentation() { # [] return "$rc" } -# The OMP primary keeps a durable claim naming the wake batch it notified, so a -# replacement session or process can re-present that exact batch. Retirement is -# bound to acknowledgement rather than delivery, which makes this drain its only -# owner (bin/fm-omp-wake-claim-lib.sh). A claim that cannot be retired is a -# stale re-presentation at worst, never a lost wake, so say so and let the drain -# finish rather than failing the whole presentation. -retire_settled_omp_wake_claim() { - fm_omp_wake_claim_reconcile_locked \ - || echo "wake drain: an acknowledged OMP wake notification claim could not be retired at $FM_OMP_WAKE_CLAIM_FILE" >&2 -} - # shellcheck disable=SC2317,SC2329 # Invoked by trap handlers below. cleanup() { local status=$? @@ -407,11 +394,6 @@ trap 'exit 143' TERM fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" DRAIN_LOCK_HELD=true -# Retire an OMP primary wake claim whose durable rows are already gone. A claim -# covering rows still queued survives here untouched, so this only clears one -# whose batch a previous acknowledgement already consumed - or one that never -# covered a durable row at all, such as an extension-internal failure wake. -retire_settled_omp_wake_claim decide_scoped_locked if [ "$SCOPED" = true ]; then reclaim_stale_branch_grant_locked || exit 1 @@ -493,9 +475,6 @@ if [ -n "$ACK_THROUGH" ]; then consume_actor_rows_locked "$MAIN_ROWS_FILE" "$ACK_THROUGH" || exit 1 fi fi - # Only this acknowledgement removes durable rows, so this is the one place a - # still-covered OMP primary wake claim can become settled. - retire_settled_omp_wake_claim fm_lock_release "$FM_WAKE_QUEUE_LOCK" DRAIN_LOCK_HELD=false if [ "$ACK_REMOVED" -eq 0 ] && [ "$PRESENTED_MAX" -gt "$ACK_THROUGH" ]; then diff --git a/docs/scripts.md b/docs/scripts.md index 5c7ed60ed7e..3625ed5d4e8 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -85,8 +85,6 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-timeout-lib.sh` | Shared bounded command runner that terminates the entire process group on timeout | | `fm-gate-refuse-lib.sh` | Shared no-mistakes gate-context refusal for fleet lifecycle entrypoints | | `fm-primary-watch-core.ts` | Harness-neutral watcher lifecycle core bound by the Pi and OMP primary extensions (docs/watcher-continuity.md) | -| `fm-omp-wake-claim.sh` | Publish, hand over, or show the OMP primary's durable wake-notification claim under the durable wake-queue lock | -| `fm-omp-wake-claim-lib.sh` | Shared format, publication, replay-handover, and acknowledgement-bound retirement of that claim | | `fm-primary-watch-version-lib.sh` | The one definition of a primary watcher marker version, hashing that adapter plus the shared core | | `fm-watch-arm.sh` | Verified home-scoped watcher arm wrapper with loud cycle endings and bounded lifecycle ledger | | `fm-watch-checkpoint.sh` | Run one bounded foreground watcher checkpoint for Codex-style supervision | diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index 0e6516464cd..092710c0d89 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -28,10 +28,9 @@ While supervision is still needed and away mode remains inactive, an actionable After an actionable Pi, OMP, or OpenCode child close, the adapter starts and verifies one singleton successor before it delivers the original wake. OMP delivers that follow-up through its host API as a hidden custom `nextTurn` message with `triggerTurn`, which starts an idle handling turn without touching an editable TUI draft. That mode is what makes an idle session reachable from a turn boundary: a wake that lands while the current turn is still unwinding schedules a continuation bound to that prompt generation instead of relying on a steer the finishing turn may never poll, and every wake queued during that turn is consumed by the one continuation. -OMP additionally keeps a durable claim naming the exact notification body and the durable wake sequence it covers, so a replacement session or a replacement OMP process re-presents that batch once instead of losing it; `bin/fm-omp-wake-claim-lib.sh` owns that claim's format, its per-process and per-session owner identity, and its invariants. -The claim is bound to acknowledgement, not to delivery: only `bin/fm-wake-drain.sh` retires it, and only once no durable row at or below its cutoff is left queued, so an interruption before that acknowledgement leaves the rows and the claim durable for idempotent re-handling. -Because retirement reads the queue rather than an actor, a mixed queue that main and the supervision branch acknowledge separately keeps the claim until whichever acknowledgement clears the last covered row. -The core keeps its own handoff for an actionable close it has not delivered yet, so the adapter takes the claim over silently while that handoff is outstanding and exactly one of the two mechanisms speaks. +The durable wake rows are the whole persistence: only `bin/fm-wake-drain.sh` acknowledgement removes them, so an interruption before acknowledgement leaves the rows queued for the next session event to re-notify. +On `session_start` and `session_switch`, OMP sends at most one hidden next-turn notification when structurally valid durable rows remain and the core has no undelivered actionable handoff. +The core remains the sole speaker while it owns an undelivered close, and OMP's hidden next-turn transport coalesces notifications queued during one turn into a single continuation. A delivered follow-up is acknowledged when the runtime starts a turn whose prompt is that wake; a wake the runtime queues into an already-running turn never starts one, so the core waits at most `FM_WATCH_WAKE_CONSUME_TIMEOUT_MS` for that acknowledgement, then treats the wake as consumed and continues the successor chain instead of parking every later actionable close behind it. It confirms the handling handoff against that successor before scheduling the follow-up, retries once against the current generation and successor, and treats a failed confirmation as a restoration failure: it classifies the error, retires a successor that is no longer alive, and surfaces exactly one typed message. A failed confirmation is never swallowed. @@ -108,9 +107,9 @@ The same suite covers Pi same-process replacement for `/new`, `/resume`, `/fork` Both suites also prove the bounded acknowledgement wait: two consecutive actionable closes whose deliveries never receive `before_agent_start` still start a third arm and deliver exactly one wake per close. It also covers a fresh factory bind without a prior shutdown: the superseded binding retires its arm child, cannot arm or reclaim ownership through retained session callbacks, the live binding keeps exactly one arm child, and repeated binds never accumulate more than one process-exit fallback. `tests/fm-omp-primary.test.sh` covers OMP's binding of the same core to its native session, watcher, and shutdown surfaces, pins the input-preserving hidden next-turn delivery contract, pins the recovery handling handshake OMP performs before delivering that notification, and pins the single typed wake a refused handshake produces. -It also covers the durable wake claim end to end: publication before notification, a same-session extension reload that re-presents nothing, a replacement session and a replacement process that each re-present the exact batch once, the silent handover while the core still owns an undelivered close, and retirement only after the drain acknowledges the covered rows. +It also covers durable-row re-notification on session events, acknowledgement gating, queue-empty suppression, and the silent handover while the core still owns an undelivered close. The opt-in `tests/fm-omp-primary-live-e2e.test.sh` proves a real OMP watcher wake reaches the session while its exact pending draft remains intact, and that an idle OMP primary runs a handling turn for that wake with no manual intervention. -`tests/fm-wake-queue.test.sh` covers the acknowledgement-bound retirement itself, including a partial acknowledgement that must keep the claim and a concurrent append that must not retire it. +`tests/fm-wake-queue.test.sh` covers durable-row acknowledgement and interrupted handling replay. `tests/fm-watch-arm.test.sh` covers durable queue replay, real remote parent-replies ingestion into the authoritative status log, decision-only OPEN DECISIONS recovery, interrupted handling replay, generation-bound acknowledgement, a persistent live successor after recovery, a watcher close inside the handling window that must leave the printed acknowledgement valid, and the self-healing moved-generation acknowledgement that consumes its handled rows and names its remedy. `tests/fm-watch-recovery-loop.test.sh` covers the once-per-generation announcement bound with the real Pi extension against a refused handling handshake, and a handling successor that must surface a real crew event instead of going blind. Pi and OMP share `bin/fm-primary-watch-core.ts` here, so that Pi-driven coverage exercises the same confirmation, retry, and typed-failure code OMP runs. diff --git a/tests/fm-omp-primary.test.sh b/tests/fm-omp-primary.test.sh index 6da858f123d..671f4519113 100755 --- a/tests/fm-omp-primary.test.sh +++ b/tests/fm-omp-primary.test.sh @@ -1190,7 +1190,10 @@ const api = { registerTool() {}, // The runtime queues every wake as a steer into a running turn: delivery // resolves, no turn starts, and before_agent_start is never invoked for it. - sendMessage(message) { steers.push(String(message?.content ?? "")); }, + sendMessage(message) { + const content = String(message?.content ?? ""); + if (content.includes("omp unacknowledged wake")) steers.push(content); + }, }; const state = process.env.FM_STATE_OVERRIDE; const bound = Number(process.env.FM_WATCH_WAKE_CONSUME_TIMEOUT_MS); @@ -1237,10 +1240,7 @@ JS pass "OMP unacknowledged wake delivery keeps the successor chain and delivers once per close" } -# Build an OMP primary fixture whose adapter can drive the real durable wake -# claim: the production claim scripts and wake library are copied in, so the -# adapter exercises the same publication, handover, and format the drain reads. -make_omp_claim_fixture() { # +make_omp_queue_fixture() { # local fixture=$TMP_ROOT/$1 mkdir -p "$fixture/.omp/extensions/lib" "$fixture/bin" "$fixture/state" "$fixture/config" : > "$fixture/AGENTS.md" @@ -1252,8 +1252,6 @@ make_omp_claim_fixture() { # cp "$ROOT/bin/fm-primary-watch-core.ts" "$fixture/bin/fm-primary-watch-core.ts" cp "$ROOT/bin/fm-pi-compatible-runtimes" "$fixture/bin/fm-pi-compatible-runtimes" cp "$ROOT/bin/fm-wake-lib.sh" "$fixture/bin/fm-wake-lib.sh" - cp "$ROOT/bin/fm-omp-wake-claim-lib.sh" "$fixture/bin/fm-omp-wake-claim-lib.sh" - cp "$ROOT/bin/fm-omp-wake-claim.sh" "$fixture/bin/fm-omp-wake-claim.sh" cat > "$fixture/bin/fm-gate-refuse-lib.sh" <<'SH' fm_is_gate_agent() { return 1; } SH @@ -1282,14 +1280,8 @@ SH printf '%s\n' "$fixture" } -# The durable claim is the only thing that can re-present a wake batch whose -# notification a session or process never handled. Its handover therefore has to -# be exactly-once per owner: a same-process extension reload must not repeat the -# batch, while a replacement session and a replacement process each must. -test_native_omp_wake_claim_replay_is_exactly_once() { - local fixture first second status=0 - fixture=$(make_omp_claim_fixture native-wake-claim) - cat > "$fixture/bin/fm-watch-arm.sh" <<'SH' +write_queue_watcher() { # + cat > "$1/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash state=${FM_STATE_OVERRIDE:?} count=$(cat "$state/watch-count" 2>/dev/null || printf 0) @@ -1297,219 +1289,24 @@ count=$((count + 1)) printf '%s\n' "$count" > "$state/watch-count" printf 'watcher: started pid=%s (beacon fresh)\n' "$$" trap 'exit 0' TERM INT -if [ "$count" -eq 1 ]; then - while [ ! -e "$state/watch-trigger" ]; do sleep 0.02; done - printf 'signal: omp durable wake batch\n' - exit 0 -fi while [ ! -e "$state/watch-stop" ]; do sleep 0.02; done SH - chmod +x "$fixture/bin/fm-watch-arm.sh" - FM_STATE_OVERRIDE="$fixture/state" bash -c \ - '. "$1/bin/fm-wake-lib.sh"; fm_wake_append signal task-a.status "signal: task-a"' _ "$fixture" \ - || fail "the OMP claim fixture could not seed a durable wake row" - - first=$(EXTENSION="$fixture/.omp/extensions/fm-primary-omp.ts" FM_HOME="$fixture" \ - FM_ROOT_OVERRIDE="$fixture" FM_STATE_OVERRIDE="$fixture/state" FM_CONFIG_OVERRIDE="$fixture/config" \ - node --input-type=module 2>&1 <<'JS' -import { spawnSync } from "node:child_process"; -import { existsSync, readFileSync, writeFileSync } from "node:fs"; -import { pathToFileURL } from "node:url"; - -const state = process.env.FM_STATE_OVERRIDE; -const claimScript = `${process.env.FM_ROOT_OVERRIDE}/bin/fm-omp-wake-claim.sh`; -const showClaim = () => { - const result = spawnSync(claimScript, ["show"], { encoding: "utf8" }); - return result.status === 0 ? result.stdout.trim() : ""; -}; -const wakes = []; -const makeApi = () => ({ - zod: { object: () => ({}) }, - on(name, handler) { this.handlers.set(name, handler); }, - handlers: new Map(), - registerCommand() {}, - registerTool() {}, - sendMessage(message, options) { - if (message?.customType !== "firstmate-watcher-wake") return; - wakes.push({ content: String(message.content ?? ""), options, claimAtSend: showClaim() }); - }, -}); -const count = () => existsSync(`${state}/watch-count`) - ? Number(readFileSync(`${state}/watch-count`, "utf8").trim()) - : 0; -async function waitFor(pred, label) { - for (let i = 0; i < 500; i += 1) { - if (pred()) return; - await new Promise((resolve) => setTimeout(resolve, 10)); - } - throw new Error(`timeout waiting for ${label}`); -} -const context = (id) => ({ sessionManager: { getSessionFile: () => undefined, getSessionId: () => id } }); - -writeFileSync(`${state}/.lock`, `${process.pid}\n`); -process.argv[1] = process.env.EXTENSION; -const load = async (tag) => { - const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?${tag}=${Date.now()}`); - const api = makeApi(); - module.default(api); - return api; -}; - -const first = await load("claim"); -await first.handlers.get("session_start")({ type: "session_start" }, context("sess-one")); -await waitFor(() => count() === 1, "initial automatic OMP arm"); -writeFileSync(`${state}/watch-trigger`, "trigger\n"); -await waitFor(() => wakes.length === 1 && count() >= 2, "the durable wake batch notification"); - -const delivered = wakes[0]; -if (delivered.options?.deliverAs !== "nextTurn" || delivered.options?.triggerTurn !== true) { - throw new Error(`the wake batch was not delivered as a hidden next-turn message: ${JSON.stringify(delivered.options)}`); -} -if (!delivered.content.includes("signal: omp durable wake batch")) { - throw new Error(`the wake batch lost its reason line: ${delivered.content}`); -} -if (!delivered.claimAtSend) { - throw new Error("the durable claim was not published before the notification was delivered"); -} -const [, instanceOne, sessionOne, cutoff] = delivered.claimAtSend.split("\t"); -if (!/^[0-9]+$/.test(cutoff) || Number(cutoff) < 1) { - throw new Error(`the claim did not cover the queued durable row: ${delivered.claimAtSend}`); -} -// Acknowledging consumption clears the core's own undelivered-close handoff, so -// from here the claim is the only thing that can re-present this batch. -first.handlers.get("before_agent_start")({ type: "before_agent_start", prompt: delivered.content }, {}); - -// A same-session extension reload re-enters this process with the same session. -const reloaded = await load("reload"); -await reloaded.handlers.get("session_start")({ type: "session_start" }, context("sess-one")); -await new Promise((resolve) => setTimeout(resolve, 200)); -if (wakes.length !== 1) { - throw new Error(`a same-session extension reload repeated the batch: ${wakes.length} notifications`); + chmod +x "$1/bin/fm-watch-arm.sh" } -// A replacement session must re-present the exact batch once, and only once. -await reloaded.handlers.get("session_switch")({ type: "session_switch", reason: "new" }, context("sess-two")); -await waitFor(() => wakes.length === 2, "the replacement-session re-presentation"); -if (wakes[1].content !== delivered.content) { - throw new Error(`the replacement session re-presented a different batch: ${wakes[1].content}`); -} -if (wakes[1].options?.deliverAs !== "nextTurn" || wakes[1].options?.triggerTurn !== true) { - throw new Error(`the re-presentation changed delivery mode: ${JSON.stringify(wakes[1].options)}`); -} -await reloaded.handlers.get("session_switch")({ type: "session_switch", reason: "resume" }, context("sess-two")); -await new Promise((resolve) => setTimeout(resolve, 200)); -if (wakes.length !== 2) { - throw new Error(`the same replacement session re-presented the batch twice: ${wakes.length} notifications`); -} -const rebound = showClaim().split("\t"); -if (rebound[1] !== instanceOne) throw new Error("a same-process replay changed the process identity"); -if (rebound[2] === sessionOne) throw new Error("the claim was not rebound to the replacement session"); - -writeFileSync(`${state}/watch-stop`, "stop\n"); -await reloaded.handlers.get("session_shutdown")({ type: "session_shutdown" }, {}); -console.log(JSON.stringify({ ok: "omp-wake-claim-replay-ok", body: delivered.content })); -JS - ) || status=$? - printf 'stop\n' > "$fixture/state/watch-stop" 2>/dev/null || true - expect_code 0 "$status" "OMP durable wake claim replay" - assert_contains "$first" omp-wake-claim-replay-ok "OMP wake claim replay did not complete: $first" - - # A replacement PROCESS re-presents the same outstanding batch exactly once, - # even though it inherits the same session and may reuse the former PID. - rm -f "$fixture/state/watch-count" "$fixture/state/watch-trigger" "$fixture/state/watch-stop" - second=$(EXTENSION="$fixture/.omp/extensions/fm-primary-omp.ts" FM_HOME="$fixture" \ - FM_ROOT_OVERRIDE="$fixture" FM_STATE_OVERRIDE="$fixture/state" FM_CONFIG_OVERRIDE="$fixture/config" \ - node --input-type=module 2>&1 <<'JS' -import { spawnSync } from "node:child_process"; -import { writeFileSync } from "node:fs"; -import { pathToFileURL } from "node:url"; - -const state = process.env.FM_STATE_OVERRIDE; -const claimScript = `${process.env.FM_ROOT_OVERRIDE}/bin/fm-omp-wake-claim.sh`; -const wakes = []; -const handlers = new Map(); -const api = { - zod: { object: () => ({}) }, - on(name, handler) { handlers.set(name, handler); }, - registerCommand() {}, - registerTool() {}, - sendMessage(message) { - if (message?.customType === "firstmate-watcher-wake") wakes.push(String(message.content ?? "")); - }, -}; -const before = spawnSync(claimScript, ["show"], { encoding: "utf8" }); -if (before.status !== 0) throw new Error("the outstanding claim did not survive the previous process"); -const previousInstance = before.stdout.trim().split("\t")[1]; - -writeFileSync(`${state}/.lock`, `${process.pid}\n`); -process.argv[1] = process.env.EXTENSION; -const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?replacementprocess=${Date.now()}`); -module.default(api); -const context = { sessionManager: { getSessionFile: () => undefined, getSessionId: () => "sess-two" } }; -await handlers.get("session_start")({ type: "session_start" }, context); -for (let i = 0; i < 200 && wakes.length === 0; i += 1) { - await new Promise((resolve) => setTimeout(resolve, 10)); -} -if (wakes.length !== 1) { - throw new Error(`a replacement process re-presented the batch ${wakes.length} times`); -} -await new Promise((resolve) => setTimeout(resolve, 200)); -if (wakes.length !== 1) throw new Error("a replacement process kept re-presenting the batch"); -const after = spawnSync(claimScript, ["show"], { encoding: "utf8" }); -if (after.status !== 0) throw new Error("the replacement process retired the claim before acknowledgement"); -if (after.stdout.trim().split("\t")[1] === previousInstance) { - throw new Error("the claim was not rebound to the replacement process"); -} -writeFileSync(`${state}/watch-stop`, "stop\n"); -await handlers.get("session_shutdown")({ type: "session_shutdown" }, {}); -console.log(JSON.stringify({ ok: "omp-wake-claim-process-replay-ok", body: wakes[0] })); -JS - ) || status=$? - printf 'stop\n' > "$fixture/state/watch-stop" 2>/dev/null || true - expect_code 0 "$status" "OMP durable wake claim process replay" - assert_contains "$second" omp-wake-claim-process-replay-ok \ - "a replacement OMP process did not re-present the outstanding batch exactly once: $second" - assert_contains "$second" "signal: omp durable wake batch" \ - "the replacement process re-presented a different batch: $second" - pass "OMP re-presents an unacknowledged wake batch once per replacement session and process, never on a reload" -} - -# Claim publication is best-effort: when its executable is unavailable, the -# live session still receives the wake and the durable row remains for drain. -test_native_omp_wake_claim_publication_failure_keeps_queue_authoritative() { +test_native_omp_durable_queue_session_notifications() { local fixture out status=0 - fixture=$(make_omp_claim_fixture native-wake-claim-publication-failure) - cat > "$fixture/bin/fm-watch-arm.sh" <<'SH' -#!/usr/bin/env bash -state=${FM_STATE_OVERRIDE:?} -count=$(cat "$state/watch-count" 2>/dev/null || printf 0) -count=$((count + 1)) -printf '%s\n' "$count" > "$state/watch-count" -printf 'watcher: started pid=%s (beacon fresh)\n' "$$" -trap 'exit 0' TERM INT -if [ "$count" -eq 1 ]; then - while [ ! -e "$state/watch-trigger" ]; do sleep 0.02; done - printf 'signal: omp claim publication failure\n' - exit 0 -fi -while [ ! -e "$state/watch-stop" ]; do sleep 0.02; done -SH - chmod +x "$fixture/bin/fm-watch-arm.sh" + fixture=$(make_omp_queue_fixture native-queue-session) + write_queue_watcher "$fixture" FM_STATE_OVERRIDE="$fixture/state" bash -c \ '. "$1/bin/fm-wake-lib.sh"; fm_wake_append signal task-a.status "signal: task-a"' _ "$fixture" \ - || fail "the OMP claim publication-failure fixture could not seed a durable wake row" - chmod a-x "$fixture/bin/fm-omp-wake-claim.sh" - + || fail "the OMP queue fixture could not seed a durable wake row" out=$(EXTENSION="$fixture/.omp/extensions/fm-primary-omp.ts" FM_HOME="$fixture" \ FM_ROOT_OVERRIDE="$fixture" FM_STATE_OVERRIDE="$fixture/state" FM_CONFIG_OVERRIDE="$fixture/config" \ node --input-type=module 2>&1 <<'JS' -import { spawnSync } from "node:child_process"; import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; - -const root = process.env.FM_ROOT_OVERRIDE; const state = process.env.FM_STATE_OVERRIDE; -const claimScript = `${root}/bin/fm-omp-wake-claim.sh`; const wakes = []; const handlers = new Map(); const api = { @@ -1521,240 +1318,100 @@ const api = { if (message?.customType === "firstmate-watcher-wake") wakes.push({ message, options }); }, }; -const count = () => existsSync(`${state}/watch-count`) - ? Number(readFileSync(`${state}/watch-count`, "utf8").trim()) - : 0; +const count = () => existsSync(`${state}/watch-count`) ? Number(readFileSync(`${state}/watch-count`, "utf8")) : 0; async function waitFor(pred, label) { - for (let i = 0; i < 500; i += 1) { - if (pred()) return; - await new Promise((resolve) => setTimeout(resolve, 10)); - } + for (let i = 0; i < 500; i += 1) { if (pred()) return; await new Promise((r) => setTimeout(r, 10)); } throw new Error(`timeout waiting for ${label}`); } writeFileSync(`${state}/.lock`, `${process.pid}\n`); process.argv[1] = process.env.EXTENSION; -const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?publication-failure=${Date.now()}`); +const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?queue-session=${Date.now()}`); module.default(api); const context = { sessionManager: { getSessionFile: () => undefined, getSessionId: () => "sess-one" } }; await handlers.get("session_start")({ type: "session_start" }, context); -await waitFor(() => count() === 1, "initial automatic OMP arm"); -writeFileSync(`${state}/watch-trigger`, "trigger\n"); -await waitFor(() => wakes.length === 1, "watcher wake after claim publication failure"); -await new Promise((resolve) => setTimeout(resolve, 100)); -if (wakes.length !== 1) throw new Error(`the watcher wake was delivered ${wakes.length} times`); -if (wakes[0].options?.deliverAs !== "nextTurn" || wakes[0].options?.triggerTurn !== true) { - throw new Error(`the watcher wake used the wrong delivery mode: ${JSON.stringify(wakes[0].options)}`); -} -if (!wakes[0].message.content.includes("signal: omp claim publication failure")) { - throw new Error(`the watcher wake lost its reason line: ${wakes[0].message.content}`); -} +await waitFor(() => count() === 1, "initial arm"); +await waitFor(() => wakes.length === 1, "session-start durable wake"); +if (wakes[0].options?.deliverAs !== "nextTurn" || wakes[0].options?.triggerTurn !== true) throw new Error("session-start wake used the wrong delivery mode"); +await handlers.get("session_switch")({ type: "session_switch", reason: "new" }, context); +await waitFor(() => wakes.length === 2, "session-switch durable wake"); +if (wakes.length !== 2) throw new Error(`expected two session-event notifications, got ${wakes.length}`); writeFileSync(`${state}/watch-stop`, "stop\n"); await handlers.get("session_shutdown")({ type: "session_shutdown" }, {}); -console.log("omp-wake-claim-publication-failure-delivered-once"); +console.log("omp-durable-queue-session-notifications-ok"); JS ) || status=$? printf 'stop\n' > "$fixture/state/watch-stop" 2>/dev/null || true - chmod +x "$fixture/bin/fm-omp-wake-claim.sh" - expect_code 0 "$status" "OMP wake claim publication failure delivery" - assert_contains "$out" omp-wake-claim-publication-failure-delivered-once \ - "claim publication failure did not preserve one watcher wake: $out" - if FM_STATE_OVERRIDE="$fixture/state" "$fixture/bin/fm-omp-wake-claim.sh" show >/dev/null 2>&1; then - fail "a claim remained outstanding after publication failure" - fi - queued=$(FM_STATE_OVERRIDE="$fixture/state" bash -c \ - '. "$1/bin/fm-wake-lib.sh"; fm_wake_queued_keys signal' _ "$fixture") - [ "$queued" = task-a.status ] || fail "the durable wake row was not left queued: $queued" - pass "OMP claim publication failure falls back to the durable wake row" + expect_code 0 "$status" "OMP durable queue session notifications" + assert_contains "$out" omp-durable-queue-session-notifications-ok "session events did not re-notify queued durable wakes: $out" + pass "OMP re-notifies durable wakes once on session start and switch" } -test_native_omp_wake_claim_session_switch_cannot_split_owner() { +test_native_omp_empty_queue_suppresses_session_notifications() { local fixture out status=0 - fixture=$(make_omp_claim_fixture native-wake-claim-session-switch) - mv "$fixture/bin/fm-omp-wake-claim.sh" "$fixture/bin/fm-omp-wake-claim.real.sh" - cat > "$fixture/bin/fm-omp-wake-claim.sh" <<'SH' -#!/usr/bin/env bash -sleep 0.4 -exec "$(dirname "$0")/fm-omp-wake-claim.real.sh" "$@" -SH - chmod +x "$fixture/bin/fm-omp-wake-claim.sh" - cat > "$fixture/bin/fm-watch-arm.sh" <<'SH' -#!/usr/bin/env bash -state=${FM_STATE_OVERRIDE:?} -printf '1\n' > "$state/watch-count" -printf 'watcher: started pid=%s (beacon fresh)\n' "$$" -trap 'exit 0' TERM INT -while [ ! -e "$state/watch-trigger" ]; do sleep 0.02; done -printf 'signal: omp session-switch race\n' -exit 0 -SH - chmod +x "$fixture/bin/fm-watch-arm.sh" - FM_STATE_OVERRIDE="$fixture/state" bash -c \ - '. "$1/bin/fm-wake-lib.sh"; fm_wake_append signal task-a.status "signal: task-a"' _ "$fixture" \ - || fail "the OMP session-switch fixture could not seed a durable wake row" - + fixture=$(make_omp_queue_fixture native-queue-empty) + write_queue_watcher "$fixture" out=$(EXTENSION="$fixture/.omp/extensions/fm-primary-omp.ts" FM_HOME="$fixture" \ FM_ROOT_OVERRIDE="$fixture" FM_STATE_OVERRIDE="$fixture/state" FM_CONFIG_OVERRIDE="$fixture/config" \ node --input-type=module 2>&1 <<'JS' -import { createHash } from "node:crypto"; -import { spawnSync } from "node:child_process"; -import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; - -const root = process.env.FM_ROOT_OVERRIDE; const state = process.env.FM_STATE_OVERRIDE; -const claimScript = `${root}/bin/fm-omp-wake-claim.sh`; -const wakes = []; +let wakes = 0; const handlers = new Map(); -let currentSession = "sess-one"; -let interleaveAttempted = false; -let timer; -const api = { - zod: { object: () => ({}) }, - on(name, handler) { handlers.set(name, handler); }, - registerCommand() {}, - registerTool() {}, - sendMessage(message, options) { - if (message?.customType !== "firstmate-watcher-wake") return; - wakes.push({ message, options }); - clearTimeout(timer); - const claim = spawnSync(claimScript, ["show"], { - encoding: "utf8", - env: { ...process.env, FM_STATE_OVERRIDE: state }, - }); - if (claim.status !== 0) throw new Error("the wake was delivered without a durable claim"); - const owner = claim.stdout.trim().split("\t")[2]; - if (!owner || owner !== createHash("sha256").update(currentSession).digest("hex")) { - throw new Error(`claim owner split from live session: ${owner} vs ${currentSession}`); - } - }, -}; -const count = () => existsSync(`${state}/watch-count`) - ? Number(readFileSync(`${state}/watch-count`, "utf8").trim()) - : 0; -async function waitFor(pred, label) { - for (let i = 0; i < 500; i += 1) { - if (pred()) return; - await new Promise((resolve) => setTimeout(resolve, 10)); - } - throw new Error(`timeout waiting for ${label}`); -} +const api = { zod: { object: () => ({}) }, on(name, handler) { handlers.set(name, handler); }, registerCommand() {}, registerTool() {}, sendMessage(message) { if (message?.customType === "firstmate-watcher-wake") wakes += 1; } }; writeFileSync(`${state}/.lock`, `${process.pid}\n`); process.argv[1] = process.env.EXTENSION; -const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?session-switch=${Date.now()}`); +const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?queue-empty=${Date.now()}`); module.default(api); -const context = (id) => ({ sessionManager: { getSessionFile: () => undefined, getSessionId: () => id } }); -await handlers.get("session_start")({ type: "session_start" }, context(currentSession)); -await waitFor(() => count() === 1, "initial automatic OMP arm"); -writeFileSync(`${state}/watch-trigger`, "trigger\n"); -timer = setTimeout(() => { - interleaveAttempted = true; - currentSession = "sess-two"; - handlers.get("session_switch")({ type: "session_switch", reason: "new" }, context(currentSession)); -}, 200); -await waitFor(() => wakes.length === 1, "one watcher wake"); -await new Promise((resolve) => setTimeout(resolve, 250)); -if (wakes.length !== 1) throw new Error(`the watcher wake was delivered ${wakes.length} times`); -if (interleaveAttempted) throw new Error("session_switch interleaved despite synchronous publication"); -if (wakes[0].options?.deliverAs !== "nextTurn" || wakes[0].options?.triggerTurn !== true) { - throw new Error(`the watcher wake used the wrong delivery mode: ${JSON.stringify(wakes[0].options)}`); -} -const claim = spawnSync(claimScript, ["show"], { - encoding: "utf8", - env: { ...process.env, FM_STATE_OVERRIDE: state }, -}); -if (claim.status !== 0 || claim.stdout.trim().split("\t").length !== 5) { - throw new Error("the claim did not remain bound to exactly one owner"); -} +const context = { sessionManager: { getSessionFile: () => undefined, getSessionId: () => "sess-one" } }; +await handlers.get("session_start")({ type: "session_start" }, context); +await handlers.get("session_switch")({ type: "session_switch", reason: "new" }, context); +await new Promise((r) => setTimeout(r, 100)); +if (wakes !== 0) throw new Error(`empty queue produced ${wakes} notifications`); writeFileSync(`${state}/watch-stop`, "stop\n"); await handlers.get("session_shutdown")({ type: "session_shutdown" }, {}); -console.log("omp-wake-claim-session-switch-owner-ok"); +console.log("omp-empty-queue-session-notifications-ok"); JS ) || status=$? printf 'stop\n' > "$fixture/state/watch-stop" 2>/dev/null || true - expect_code 0 "$status" "OMP wake claim session-switch ownership" - assert_contains "$out" omp-wake-claim-session-switch-owner-ok \ - "session-switch interleaving split the wake claim owner: $out" - pass "OMP wake claim publication and notification keep one session owner" + expect_code 0 "$status" "OMP empty queue session notifications" + assert_contains "$out" omp-empty-queue-session-notifications-ok "empty durable queue produced a session notification: $out" + pass "OMP suppresses session notifications when the durable queue is empty" } -# Two mechanisms could re-present the same wake after a replacement: the shared -# core's own handoff for a close it never delivered, and this adapter's durable -# claim. Exactly one of them may speak, or the replacement receives the wake -# twice. -test_native_omp_wake_claim_defers_to_a_core_owned_close() { +test_native_omp_core_handoff_suppresses_queue_notification() { local fixture out status=0 - fixture=$(make_omp_claim_fixture native-wake-claim-interlock) - cat > "$fixture/bin/fm-watch-arm.sh" <<'SH' -#!/usr/bin/env bash -state=${FM_STATE_OVERRIDE:?} -printf 'watcher: started pid=%s (beacon fresh)\n' "$$" -trap 'exit 0' TERM INT -while [ ! -e "$state/watch-stop" ]; do sleep 0.02; done -SH - chmod +x "$fixture/bin/fm-watch-arm.sh" + fixture=$(make_omp_queue_fixture native-queue-core-handoff) + write_queue_watcher "$fixture" mkdir -p "$fixture/state/extensions/omp-primary-watch" printf '{"version":2,"pending":[{"version":1,"token":"1-2-3","message":"signal: core owned undelivered close","predecessorArmPid":""}]}\n' \ > "$fixture/state/extensions/omp-primary-watch/session-replacement-actionable.json" - printf '%s' 'encoded:watcher:FIRSTMATE WATCHER WAKE: signal: claimed batch' \ - | FM_STATE_OVERRIDE="$fixture/state" "$fixture/bin/fm-omp-wake-claim.sh" \ - publish --instance inst-previous --session sess-previous \ - || fail "the interlock fixture could not publish an outstanding claim" - out=$(EXTENSION="$fixture/.omp/extensions/fm-primary-omp.ts" FM_HOME="$fixture" \ FM_ROOT_OVERRIDE="$fixture" FM_STATE_OVERRIDE="$fixture/state" FM_CONFIG_OVERRIDE="$fixture/config" \ node --input-type=module 2>&1 <<'JS' -import { spawnSync } from "node:child_process"; import { writeFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; - const state = process.env.FM_STATE_OVERRIDE; -const claimScript = `${process.env.FM_ROOT_OVERRIDE}/bin/fm-omp-wake-claim.sh`; const wakes = []; const handlers = new Map(); -const api = { - zod: { object: () => ({}) }, - on(name, handler) { handlers.set(name, handler); }, - registerCommand() {}, - registerTool() {}, - sendMessage(message) { - if (message?.customType === "firstmate-watcher-wake") wakes.push(String(message.content ?? "")); - }, -}; +const api = { zod: { object: () => ({}) }, on(name, handler) { handlers.set(name, handler); }, registerCommand() {}, registerTool() {}, sendMessage(message, options) { if (message?.customType === "firstmate-watcher-wake") wakes.push({ message, options }); } }; writeFileSync(`${state}/.lock`, `${process.pid}\n`); process.argv[1] = process.env.EXTENSION; -const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?interlock=${Date.now()}`); +const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?queue-core=${Date.now()}`); module.default(api); -await handlers.get("session_start")({ type: "session_start" }, { - sessionManager: { getSessionFile: () => undefined, getSessionId: () => "sess-new" }, -}); -for (let i = 0; i < 300 && wakes.length === 0; i += 1) { - await new Promise((resolve) => setTimeout(resolve, 10)); -} -await new Promise((resolve) => setTimeout(resolve, 300)); -if (wakes.length !== 1) { - throw new Error(`the replacement received ${wakes.length} wakes for one outstanding close: ${wakes.join(" | ")}`); -} -if (!wakes[0].includes("signal: core owned undelivered close")) { - throw new Error(`the core's undelivered close was not the wake that was delivered: ${wakes[0]}`); -} -if (wakes[0].includes("claimed batch")) { - throw new Error("the adapter re-presented its claim alongside the core's own redelivery"); -} -const claim = spawnSync(claimScript, ["show"], { encoding: "utf8" }); -if (claim.status !== 0) throw new Error("the silent handover retired the claim before acknowledgement"); -if (claim.stdout.trim().split("\t")[1] === "inst-previous") { - throw new Error("the silent handover left the claim bound to the replaced process"); -} +await handlers.get("session_start")({ type: "session_start" }, { sessionManager: { getSessionFile: () => undefined, getSessionId: () => "sess-one" } }); +for (let i = 0; i < 300 && wakes.length === 0; i += 1) await new Promise((r) => setTimeout(r, 10)); +if (wakes.length !== 1 || !wakes[0].message.content.includes("core owned undelivered close")) throw new Error(`core handoff delivery was not exclusive: ${JSON.stringify(wakes)}`); +if (wakes[0].options?.deliverAs !== "nextTurn" || wakes[0].options?.triggerTurn !== true) throw new Error("core handoff used the wrong delivery mode"); writeFileSync(`${state}/watch-stop`, "stop\n"); await handlers.get("session_shutdown")({ type: "session_shutdown" }, {}); -console.log("omp-wake-claim-interlock-ok"); +console.log("omp-core-handoff-queue-notification-ok"); JS ) || status=$? printf 'stop\n' > "$fixture/state/watch-stop" 2>/dev/null || true - expect_code 0 "$status" "OMP wake claim interlock" - assert_contains "$out" omp-wake-claim-interlock-ok \ - "the OMP wake claim and the core's own handoff both re-presented one close: $out" - pass "OMP hands its wake claim over silently while the shared core still owes an undelivered close" + expect_code 0 "$status" "OMP core handoff queue notification" + assert_contains "$out" omp-core-handoff-queue-notification-ok "core handoff was duplicated by queue notification: $out" + pass "OMP core handoff suppresses duplicate durable queue notification" } test_resolve_path_uses_node_when_readlink_f_is_unavailable @@ -1770,7 +1427,6 @@ test_native_omp_confirms_recovery_handling_delivery test_native_omp_refused_handling_delivery_is_typed_once test_native_omp_session_switch_carries_inflight_actionable_close test_native_omp_unacknowledged_wake_keeps_successor_chain -test_native_omp_wake_claim_replay_is_exactly_once -test_native_omp_wake_claim_publication_failure_keeps_queue_authoritative -test_native_omp_wake_claim_session_switch_cannot_split_owner -test_native_omp_wake_claim_defers_to_a_core_owned_close +test_native_omp_durable_queue_session_notifications +test_native_omp_empty_queue_suppresses_session_notifications +test_native_omp_core_handoff_suppresses_queue_notification diff --git a/tests/fm-wake-queue.test.sh b/tests/fm-wake-queue.test.sh index 7e71565e572..f2716c5000f 100755 --- a/tests/fm-wake-queue.test.sh +++ b/tests/fm-wake-queue.test.sh @@ -13,22 +13,6 @@ set -u WATCH="$ROOT/bin/fm-watch.sh" DRAIN="$ROOT/bin/fm-wake-drain.sh" -OMP_WAKE_CLAIM="$ROOT/bin/fm-omp-wake-claim.sh" - -# Bind an OMP primary wake-notification claim for , exactly as the OMP -# adapter does before it notifies. -publish_omp_wake_claim() { # - printf '%s' "$4" | FM_STATE_OVERRIDE="$1" "$OMP_WAKE_CLAIM" publish --instance "$2" --session "$3" -} - -omp_wake_claim_cutoff() { # - FM_STATE_OVERRIDE="$1" "$OMP_WAKE_CLAIM" show | awk -F '\t' '{ print $4 }' -} - -omp_wake_claim_outstanding() { # - FM_STATE_OVERRIDE="$1" "$OMP_WAKE_CLAIM" show > /dev/null 2>&1 -} - TMP_ROOT=$(fm_test_tmproot fm-wake-tests) # Wait briefly for to become non-empty. @@ -1005,145 +989,6 @@ test_turnend_marker_consumer_incarnation_gate() { pass "consumer fires only the live gen marker and ignores stale gens, so a delayed old gen never overwrites or drops a live completion" } -# The OMP primary's durable wake claim exists so a replacement session or -# process can re-present an unacknowledged batch. Retirement must therefore be -# bound to acknowledgement of the durable rows, never to their presentation: -# only the acknowledgement that removes the last covered row may clear it, and a -# wake appended after the claim is above its cutoff, so it neither holds the -# claim open nor is swallowed by it. -test_omp_wake_claim_retires_only_after_acknowledgement() { - local dir state sequence generation - dir=$(make_case omp-claim-ack) - state="$dir/state" - append_wake "$state" signal "task-a.status" "signal: task-a" || fail "first append failed" - append_wake "$state" heartbeat fleet "heartbeat" || fail "second append failed" - publish_omp_wake_claim "$state" inst-one sess-one 'FIRSTMATE WATCHER WAKE: signal: task-a' \ - || fail "the OMP wake claim could not be published" - [ "$(omp_wake_claim_cutoff "$state")" = 2 ] \ - || fail "the claim did not cover both queued rows: $(omp_wake_claim_cutoff "$state")" - - FM_STATE_OVERRIDE="$state" "$DRAIN" > "$dir/present.out" 2> "$dir/present.err" \ - || fail "presentation drain failed: $(cat "$dir/present.err")" - omp_wake_claim_outstanding "$state" \ - || fail "presenting the batch retired the claim before any acknowledgement" - - generation=$(recovery_marker_generation "$state/.watcher-down") - [ -n "$generation" ] || fail "presentation left no recovery generation" - - # A partial acknowledgement leaves a covered row queued, so the claim stays. - FM_STATE_OVERRIDE="$state" "$DRAIN" --ack-through 1 --recovery-generation "$generation" \ - > /dev/null 2>&1 || fail "partial acknowledgement failed" - grep -Fq "$(printf '\theartbeat\tfleet\t')" "$state/.wake-queue" \ - || fail "the partial acknowledgement consumed a row above its cutoff" - omp_wake_claim_outstanding "$state" \ - || fail "the claim was retired while a covered row was still queued" - - # A wake appended after publication is above the cutoff: it must not hold the - # claim open once every covered row is acknowledged. - append_wake "$state" signal "task-b.status" "signal: task-b" || fail "late append failed" - FM_STATE_OVERRIDE="$state" "$DRAIN" > /dev/null 2> "$dir/second.err" || fail "second presentation failed" - generation=$(recovery_marker_generation "$state/.watcher-down") - FM_STATE_OVERRIDE="$state" "$DRAIN" --ack-through 2 --recovery-generation "$generation" \ - > /dev/null 2>&1 || fail "covered acknowledgement failed" - grep -Fq "$(printf '\tsignal\ttask-b.status\t')" "$state/.wake-queue" \ - || fail "the acknowledgement consumed the later wake its cutoff never covered" - omp_wake_claim_outstanding "$state" \ - && fail "the claim survived the acknowledgement of every row it covered" - pass "an OMP wake claim is retired only by the acknowledgement that consumes its last covered row" -} - -# An interrupted handling turn must leave both halves durable: the queue rows -# for idempotent re-presentation, and the claim so a replacement re-notifies -# them. A second drain has to show exactly the same rows. -test_omp_wake_claim_survives_interrupted_handling() { - local dir state before after - dir=$(make_case omp-claim-interrupted) - state="$dir/state" - append_wake "$state" signal "task-a.status" "signal: task-a" || fail "append failed" - publish_omp_wake_claim "$state" inst-one sess-one 'FIRSTMATE WATCHER WAKE: signal: task-a' \ - || fail "the OMP wake claim could not be published" - FM_STATE_OVERRIDE="$state" "$DRAIN" > "$dir/first.out" 2>/dev/null || fail "first drain failed" - before=$(awk -F '\t' 'NF == 5 { print $2 "|" $3 "|" $4 }' "$dir/first.out") - # No acknowledgement: this is the interrupted turn. - FM_STATE_OVERRIDE="$state" "$DRAIN" > "$dir/second.out" 2>/dev/null || fail "second drain failed" - after=$(awk -F '\t' 'NF == 5 { print $2 "|" $3 "|" $4 }' "$dir/second.out") - [ -n "$before" ] && [ "$before" = "$after" ] \ - || fail "an interrupted handling turn changed the durable rows: [$before] vs [$after]" - omp_wake_claim_outstanding "$state" \ - || fail "an interrupted handling turn retired the claim for rows still queued" - pass "an interruption before acknowledgement leaves the OMP wake claim and its durable rows intact" -} - -# A mixed queue is acknowledged by two different actors. Retirement reads the -# queue rather than an actor, so the claim has to outlive whichever -# acknowledgement lands first: main's ack cannot retire a claim the supervision -# branch still owes a covered row for. -test_omp_wake_claim_waits_for_every_actor() { - local dir state grant sequence generation - grant="$ROOT/bin/fm-wake-grant.sh" - dir=$(make_case omp-claim-actors) - state="$dir/state" - append_wake "$state" check "some-poll.check.sh" "check: some-poll" || fail "check append failed" - append_wake "$state" signal "task-a.status" "signal: task-a" || fail "signal append failed" - publish_omp_wake_claim "$state" inst-one sess-one 'FIRSTMATE WATCHER WAKE: check: some-poll' \ - || fail "the OMP wake claim could not be published" - FM_STATE_OVERRIDE="$state" "$grant" activate "$$" omp-claim-actors || fail "branch owner activation failed" - FM_STATE_OVERRIDE="$state" "$grant" publish omp-claim-actors 2 || fail "branch grant publication failed" - - FM_STATE_OVERRIDE="$state" "$DRAIN" > /dev/null 2> "$dir/main.err" || fail "main drain failed" - sequence=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through \([0-9][0-9]*\) --recovery-generation [A-Za-z0-9._-][A-Za-z0-9._-]*$/\1/p' "$dir/main.err") - generation=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through [0-9][0-9]* --recovery-generation \([A-Za-z0-9._-][A-Za-z0-9._-]*\)$/\1/p' "$dir/main.err") - [ -n "$sequence" ] && [ -n "$generation" ] || fail "main drain omitted its acknowledgement boundary" - FM_STATE_OVERRIDE="$state" "$DRAIN" --ack-through "$sequence" --recovery-generation "$generation" \ - > /dev/null 2>&1 || fail "main acknowledgement failed" - omp_wake_claim_outstanding "$state" \ - || fail "main's acknowledgement retired a claim whose branch-owned row was still queued" - - FM_STATE_OVERRIDE="$state" FM_SUPERVISION_ACTOR=branch "$DRAIN" > /dev/null 2> "$dir/branch.err" \ - || fail "branch drain failed" - sequence=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through \([0-9][0-9]*\) --recovery-generation [A-Za-z0-9._-][A-Za-z0-9._-]*$/\1/p' "$dir/branch.err") - generation=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through [0-9][0-9]* --recovery-generation \([A-Za-z0-9._-][A-Za-z0-9._-]*\)$/\1/p' "$dir/branch.err") - [ -n "$sequence" ] && [ -n "$generation" ] || fail "branch drain omitted its acknowledgement boundary" - FM_STATE_OVERRIDE="$state" FM_SUPERVISION_ACTOR=branch "$DRAIN" \ - --ack-through "$sequence" --recovery-generation "$generation" > /dev/null 2>&1 \ - || fail "branch acknowledgement failed" - omp_wake_claim_outstanding "$state" \ - && fail "the claim survived after both actors acknowledged every covered row" - pass "an OMP wake claim outlives a partial actor acknowledgement and retires on the one clearing its last row" -} - -# Watcher appends and handling turns run concurrently. Publishing under the -# queue lock must keep the cutoff monotonic against those appends, so no durable -# row is lost, no claim is retired while a covered row is queued, and the full -# acknowledgement still clears it. -test_omp_wake_claim_holds_under_concurrent_appends() { - local dir state pids pid i cutoff queued - dir=$(make_case omp-claim-concurrent) - state="$dir/state" - pids= - i=1 - while [ "$i" -le 12 ]; do - append_wake "$state" signal "task-$i.status" "signal: task-$i" & - pids="$pids $!" - publish_omp_wake_claim "$state" "inst-$i" sess-one "FIRSTMATE WATCHER WAKE: signal: task-$i" & - pids="$pids $!" - i=$((i + 1)) - done - for pid in $pids; do - wait "$pid" || fail "a concurrent append or claim publication failed" - done - cutoff=$(omp_wake_claim_cutoff "$state") - case "$cutoff" in ''|*[!0-9]*) fail "concurrent publication left no readable claim cutoff" ;; esac - queued=$(awk -F '\t' 'NF == 5 && $2 ~ /^[0-9]+$/ && $2 > max { max = $2 } END { print max + 0 }' "$state/.wake-queue") - [ "$queued" -eq 12 ] || fail "concurrent appends lost a durable row: highest sequence $queued" - FM_STATE_OVERRIDE="$state" "$DRAIN" > /dev/null 2> "$dir/present.err" || fail "presentation drain failed" - omp_wake_claim_outstanding "$state" || fail "a claim was retired while its covered rows were queued" - ack_drain_err "$state" "$dir/present.err" > /dev/null 2>&1 || fail "acknowledgement failed" - [ ! -s "$state/.wake-queue" ] || fail "the acknowledgement left durable rows queued" - omp_wake_claim_outstanding "$state" && fail "the claim survived a full acknowledgement" - pass "concurrent watcher appends and claim publications lose no wake and keep retirement acknowledgement-bound" -} - test_turnend_marker_consumer_incarnation_gate test_stale_acknowledgement_names_current_presented_wake test_concurrent_append_and_drain @@ -1167,7 +1012,3 @@ test_handling_confirmation_is_bounded_by_foreign_marker_lock test_branch_actor_scoped_ack_never_swallows_a_main_owned_row test_main_drain_excludes_rows_already_granted_to_branch test_branch_owner_activation_rollback_stops_after_publication -test_omp_wake_claim_retires_only_after_acknowledgement -test_omp_wake_claim_survives_interrupted_handling -test_omp_wake_claim_waits_for_every_actor -test_omp_wake_claim_holds_under_concurrent_appends From 73ddaca0d63fa71f504e45e4f5348bb556b981dd Mon Sep 17 00:00:00 2001 From: dnth Date: Sun, 6 Sep 2026 20:31:38 +0800 Subject: [PATCH 6/7] no-mistakes(review): Validated actionable handoffs before suppressing durable wake notifications --- bin/fm-primary-watch-core.ts | 13 ++++++------ tests/fm-omp-primary.test.sh | 39 ++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 6 deletions(-) diff --git a/bin/fm-primary-watch-core.ts b/bin/fm-primary-watch-core.ts index 8f162825482..e5f7f8ad069 100644 --- a/bin/fm-primary-watch-core.ts +++ b/bin/fm-primary-watch-core.ts @@ -1132,15 +1132,16 @@ export function createPrimaryWatchCore(options: PrimaryWatchCoreOptions): Primar return result; } - // True while the core still owns an actionable wake it has not delivered: an - // undelivered close on this generation, one handed over in process, or the - // durable replacement handoff on disk. A runtime adapter that keeps its own - // durable re-notification claim reads this before replaying that claim, so - // one wake is never delivered twice across a session replacement. + // True while the core still owns an actionable wake it has not delivered. function hasPendingActionableHandoff(): boolean { if (replacementCoordinator.pending.length > 0) return true; if (generation.pendingActionables.some((pending) => !pending.delivered)) return true; - return existsSync(actionableHandoff); + try { + return validateReplacementHandoff(JSON.parse(readFileSync(actionableHandoff, "utf8"))) + .some((pending) => !pending.delivered); + } catch { + return false; + } } function acknowledgeWake(content: string): void { diff --git a/tests/fm-omp-primary.test.sh b/tests/fm-omp-primary.test.sh index 671f4519113..27ea133abd0 100755 --- a/tests/fm-omp-primary.test.sh +++ b/tests/fm-omp-primary.test.sh @@ -1414,6 +1414,44 @@ JS pass "OMP core handoff suppresses duplicate durable queue notification" } +test_native_omp_delivered_handoff_does_not_suppress_queue_notification() { + local fixture out status=0 + fixture=$(make_omp_queue_fixture native-queue-delivered-handoff) + write_queue_watcher "$fixture" + mkdir -p "$fixture/state/extensions/omp-primary-watch" + printf '{"version":2,"pending":[{"version":1,"token":"1-2-3","message":"signal: already delivered","predecessorArmPid":"","delivered":true}]}\n' \ + > "$fixture/state/extensions/omp-primary-watch/session-replacement-actionable.json" + FM_STATE_OVERRIDE="$fixture/state" bash -c \ + '. "$1/bin/fm-wake-lib.sh"; fm_wake_append signal task-a.status "signal: task-a"' _ "$fixture" \ + || fail "the delivered-handoff fixture could not seed a durable wake row" + out=$(EXTENSION="$fixture/.omp/extensions/fm-primary-omp.ts" FM_HOME="$fixture" \ + FM_ROOT_OVERRIDE="$fixture" FM_STATE_OVERRIDE="$fixture/state" FM_CONFIG_OVERRIDE="$fixture/config" \ + node --input-type=module 2>&1 <<'JS' +import { writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; +const state = process.env.FM_STATE_OVERRIDE; +const wakes = []; +const handlers = new Map(); +const api = { zod: { object: () => ({}) }, on(name, handler) { handlers.set(name, handler); }, registerCommand() {}, registerTool() {}, sendMessage(message, options) { if (message?.customType === "firstmate-watcher-wake") wakes.push({ message, options }); } }; +writeFileSync(`${state}/.lock`, `${process.pid}\n`); +process.argv[1] = process.env.EXTENSION; +const module = await import(`${pathToFileURL(process.env.EXTENSION).href}?queue-delivered-handoff=${Date.now()}`); +module.default(api); +await handlers.get("session_start")({ type: "session_start" }, { sessionManager: { getSessionFile: () => undefined, getSessionId: () => "sess-one" } }); +for (let i = 0; i < 300 && wakes.length === 0; i += 1) await new Promise((r) => setTimeout(r, 10)); +if (wakes.length !== 1) throw new Error(`delivered handoff suppressed or duplicated the queue wake: ${wakes.length}`); +if (wakes[0].options?.deliverAs !== "nextTurn" || wakes[0].options?.triggerTurn !== true) throw new Error("queue wake used the wrong delivery mode"); +writeFileSync(`${state}/watch-stop`, "stop\n"); +await handlers.get("session_shutdown")({ type: "session_shutdown" }, {}); +console.log("omp-delivered-handoff-queue-notification-ok"); +JS + ) || status=$? + printf 'stop\n' > "$fixture/state/watch-stop" 2>/dev/null || true + expect_code 0 "$status" "OMP delivered handoff queue notification" + assert_contains "$out" omp-delivered-handoff-queue-notification-ok "delivered handoff suppressed durable queue notification: $out" + pass "OMP ignores already-delivered handoffs when notifying queued wakes" +} + test_resolve_path_uses_node_when_readlink_f_is_unavailable test_exact_bun_omp_primary_identity test_standalone_omp_primary_identity @@ -1430,3 +1468,4 @@ test_native_omp_unacknowledged_wake_keeps_successor_chain test_native_omp_durable_queue_session_notifications test_native_omp_empty_queue_suppresses_session_notifications test_native_omp_core_handoff_suppresses_queue_notification +test_native_omp_delivered_handoff_does_not_suppress_queue_notification From 1f6262011b425fbed9388a2460919cf358dcd75c Mon Sep 17 00:00:00 2001 From: dnth Date: Sun, 6 Sep 2026 20:41:03 +0800 Subject: [PATCH 7/7] no-mistakes(document): Correct historical OMP transport verification and claim terminology --- bin/fm-wake-drain.sh | 4 ++-- docs/verification/runtime-backends.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index eed0ef2693b..8933b8a9d8d 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -158,8 +158,8 @@ decide_scoped_locked() { } # The highest sequence this actor has already been presented: the branch's -# grant is exactly its current prompt's rows, and main's claim file is what its -# last drain printed. Read BEFORE an ack re-claims, so a row that arrived since +# grant is exactly its current prompt's rows, and main's eligible-row snapshot +# is what its last drain printed. Read BEFORE an ack re-claims, so a row that arrived since # presentation is never named as "the current wake" the caller may acknowledge # unseen. 0 when nothing is on record. presented_max_row() { # diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index 89b41d83ead..d3cc338231a 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -527,7 +527,7 @@ FM_TEST_SUMMARY_FAMILY family=watcher-wake-lock count=2 duration_ms=97033 failed Every listed script ran at that head with no gate skip. The isolated Herdr role matrix emitted no queued-wake warning. The final run retained the fresh-beacon, pending-notification, queue-drain, and bounded-delivery assertions in the Herdr fixture. -The focused OMP adapter contract now delivers watcher wakes as a hidden custom next-turn message with `triggerTurn`, preserving the editable draft while retaining idle wake and unwinding-turn continuation delivery. +At this 2026-08-01 head, the focused OMP adapter contract delivered watcher wakes as a custom steer with `triggerTurn`, preserving the editable draft while retaining idle wake and streaming delivery. The tmux role fixtures emitted their expected task-copy worktree and missing-fixture-watcher notices. The Herdr exit fixture refused an unlocked presentation close after proving normal process exit, then completed its named guarded teardown. @@ -618,7 +618,7 @@ ok - a captain-worthy wake opens exactly one follow-up turn on MAIN (real SDK) ok - OMP supervision branch live guard passed against @oh-my-pi/pi-coding-agent 17.3.4 ``` -The guard proves a broken branch (an unresolvable model pin) falls the wake back to main through the primary adapter's hidden watcher-wake next-turn message with triggerTurn, not sendUserMessage, leaving the wake queue durable. +The guard proves a broken branch (an unresolvable model pin) falls the wake back to main through the primary adapter's custom steer with triggerTurn, not sendUserMessage, leaving the wake queue durable. It proves a resident second AgentSession is created and remains re-promptable on a later wake without its turn output reaching main or replacing main's terminal resume breadcrumb, and that a routine verdict opens no new main turn while a captain verdict opens exactly one follow-up turn. The captain sub-check is skipped, not passed, on a run where the model judges the captain-worthy fixture routine.