Skip to content

engine: volume mount races devtmpfs — sysfs serial appears before the /dev node #78

Closed

Description

@CMGS

Surfaced by the PR #77 hardware round: 5/141 four-volume claims (3.5%) failed
with mount: /volumes/vol-X: /dev/vdY is not a valid block device → HTTP 500.
Zero failures in 112 single-volume claims. Reproduces on main (ce55029), so it
predates #77 — but concurrent bring-up makes multi-volume claims worth using,
so the race will be hit more.

Cause

engine/volume.go waitForVolumeDevice polls /sys/block/<dev>/serial and
returns /dev/<dev> as soon as the serial matches. The kernel publishes the
sysfs entry before devtmpfs creates the device node, so under load the guest
mount can run before /dev/<dev> exists (util-linux reports "not a valid
block device", exit 32).

Fix direction

Extend the wait predicate: after the serial matches, also confirm /dev/<dev>
exists (a silkd stat — the fs.stat verb already exists) before returning, under
the same poll interval and probe budget. Deterministic, no retry-semantics
change on the mount itself. Failed claims already clean up correctly (verified
on hardware: every failure settled to claimed==0 with warm back at target), so
the fix is purely about not failing spuriously.

Acceptance

  • A multi-volume soak (≥100 four-volume claims on a multi-vCPU tier) shows zero
    "not a valid block device" failures; before the fix the same soak reproduces
    the ~3-4% rate.
  • Single-volume latency unchanged (the extra stat rides the existing poll loop
    only until the node appears).

Activity

  1. added
    bugSomething isn't working
    on Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions