Repository navigation
fix(watchdog): detect a remount that keeps its device number - #230
Merged
Merged
Conversation
The volume watchdog compared st_dev and st_ino at the data directory, so a USB detach that re-enumerates back onto the same BSD node reads as Intact forever while every file descriptor the process holds is revoked. That is exactly what happened on 2026-10-18: same dev (16777247), same inode, 11 revoked fds, a listener that accepted and never answered, no breadcrumb. Path identity answers "is there a directory here that looks right?", not "can this process still use it?". Arming now also opens a one-byte sentinel file inside the data directory and keeps the descriptor for the life of the process; every 5 s sample seeks to 0 and writes through it. A revoked vnode rejects that with EBADF immediately, which classifies as Health::Stale and counts as a miss alongside the existing Missing/Replaced verdicts. The probe is one byte at offset 0, never truncated, so the file never grows and the tick costs no allocation or fsync. If the sentinel cannot be opened at arm time the watchdog still arms on path identity alone and logs that the same-device case is now undetected. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The data-volume watchdog fired correctly on 2026-10-14, when the drive came back
on a new BSD node (
disk8s1→disk7s1) andst_devchanged. It did notfire on 2026-10-18, when the same drive detached and returned on the same
node:
st_devwas16777247before and after,st_inomatched, soclassify()reportedIntactforever while 11 held descriptors were revoked.The server kept LISTENing, accepted connections, answered none, and left no
breadcrumb — the original outage with a detector that could not see it.
Fix
Arming now opens a one-byte sentinel,
.volume-watchdog.sentinel, inside thedata directory and keeps the
Filefor the life of the process. Every 5 ssample writes through it:
stat;Missing/Replacedreturnbefore any probe);
vnode rejects
lseek/writewithEBADFimmediately, which is the newHealth::Staleverdict and counts as a miss like the other two;_exit(75)path.The write is one byte at offset 0 and never truncates, so the sentinel stays a
single byte no matter how long the process runs; no allocation, no fsync. If the
sentinel cannot be opened at arm time, the watchdog still arms on path identity
and logs that the same-device remount is now undetected.
Verification
cargo +1.98.1 test --lib→ 740 passed, 0 failed (16 involume_watchdog,4 new)
cargo +1.98.1 clippy --lib --no-deps→ no new findings in the changed filecargo +1.98.1 fmt -- --check→ cleanStalewhile the path check saysIntact; healthy sentinel keepsIntact; a missing path is not masked by ahealthy sentinel; sentinel stays one byte over 100 samples; arming creates it
on disk.
Not in this PR
The production process was started by hand from
start-contextnest.sh, so even acorrect exit 75 has no supervisor to restart it. The launchd template already
handles this (
KeepAlive SuccessfulExit=false, 15 s throttle) but is notinstalled — that is an operator step (
make cn-agent-install), not a code change.🤖 Generated with Claude Code