Skip to content

fix(watchdog): detect a remount that keeps its device number - #230

Merged
amirkhakshour merged 1 commit into
mainfrom
fix/cn-watchdog-revoked-fd
Oct 18, 2026
Merged

amirkhakshour merged 1 commit into
mainfrom
fix/cn-watchdog-revoked-fd

Conversation

@amirkhakshour

Copy link
Copy Markdown
Contributor

Problem

The data-volume watchdog fired correctly on 2026-10-14, when the drive came back
on a new BSD node (disk8s1 → disk7s1) and st_dev changed. It did not
fire on 2026-10-18, when the same drive detached and returned on the same
node: st_dev was 16777247 before and after, st_ino matched, so
classify() reported Intact forever while 11 held descriptors were revoked.
The server kept LISTENing, accepted connections, answered none, and left no
breadcrumb — the original outage with a detector that could not see it.

path identity:  stat(data_dir) == boot identity   →  Intact
reality:        every open fd on that volume      →  EBADF, unrecoverable

Fix

Arming now opens a one-byte sentinel, .volume-watchdog.sentinel, inside the
data directory and keeps the File for the life of the process. Every 5 s
sample writes through it:

  • path identity is checked first (cheap stat; Missing / Replaced return
    before any probe);
  • if the path still looks right, the held descriptor is exercised — a revoked
    vnode rejects lseek/write with EBADF immediately, which is the new
    Health::Stale verdict and counts as a miss like the other two;
  • three consecutive misses (~15 s) still trigger the existing breadcrumb +
    _exit(75) path.

The write is one byte at offset 0 and never truncates, so the sentinel stays a
single byte no matter how long the process runs; no allocation, no fsync. If the
sentinel cannot be opened at arm time, the watchdog still arms on path identity
and logs that the same-device remount is now undetected.

Verification

  • cargo +1.98.1 test --lib → 740 passed, 0 failed (16 in volume_watchdog,
    4 new)
  • cargo +1.98.1 clippy --lib --no-deps → no new findings in the changed file
  • cargo +1.98.1 fmt -- --check → clean
  • New tests: revoked descriptor reads as Stale while the path check says
    Intact; healthy sentinel keeps Intact; a missing path is not masked by a
    healthy sentinel; sentinel stays one byte over 100 samples; arming creates it
    on disk.

Not in this PR

The production process was started by hand from start-contextnest.sh, so even a
correct exit 75 has no supervisor to restart it. The launchd template already
handles this (KeepAlive SuccessfulExit=false, 15 s throttle) but is not
installed — that is an operator step (make cn-agent-install), not a code change.

🤖 Generated with Claude Code

The volume watchdog compared st_dev and st_ino at the data directory, so a
USB detach that re-enumerates back onto the same BSD node reads as Intact
forever while every file descriptor the process holds is revoked. That is
exactly what happened on 2026-10-18: same dev (16777247), same inode, 11
revoked fds, a listener that accepted and never answered, no breadcrumb.

Path identity answers "is there a directory here that looks right?", not
"can this process still use it?". Arming now also opens a one-byte sentinel
file inside the data directory and keeps the descriptor for the life of the
process; every 5 s sample seeks to 0 and writes through it. A revoked vnode
rejects that with EBADF immediately, which classifies as Health::Stale and
counts as a miss alongside the existing Missing/Replaced verdicts.

The probe is one byte at offset 0, never truncated, so the file never grows
and the tick costs no allocation or fsync. If the sentinel cannot be opened
at arm time the watchdog still arms on path identity alone and logs that the
same-device case is now undetected.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@amirkhakshour
amirkhakshour merged commit b30b32b into main Oct 18, 2026
7 checks passed
@amirkhakshour
amirkhakshour deleted the fix/cn-watchdog-revoked-fd branch October 18, 2026 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant