feat(racer): serve ublk block devices from dataplane workers by jveski · Pull Request #1011 · Azure/unbounded

Jordan Olshevski (jveski) · 2026-10-28T00:01:51Z

This moves the read-only ublk block device from #985 into racer, so block reads are served by racer's worker threads with no extra hop through the client socket. Each device maps to a single cache key pinned to one ETag and is configured with RACER_UBLK_DEVICES. Every I/O worker serves one ublk queue of each device using a new uring_cmd reactor operation, and writes cached pages straight into the kernel request with UBLK_F_USER_COPY. A dedicated control thread adds, starts, stops, and deletes devices, including the pinned DEL_DEV fallback needed on Linux 6.8. The design is in designs/racer-ublk.md, and hack/scripts/racer-ublk-vm-test.sh runs the root-only kernel tests in a QEMU VM on Linux 6.8 and 7.0.

Copilot

🟡 Changes recommended

Critical findings identify uninitialized padding reads and unenforced memory lifetimes for kernel access, while shutdown defects can hang the process or fail accepted reads.

5 open issues

What changed in this PR

Integrates read-only ublk devices into racer-dataplane so workers serve pinned cached objects directly, avoiding the separate daemon's client-socket hop and duplicate cache.

Changes:

Adds device configuration, queue serving, version-pinned reads, and control operations with a Linux 6.8 deletion fallback.
Extends the reactor with passthrough commands and dedicated submission capacity, and integrates ublk into worker shutdown.
Adds unit, simulation, and root-only kernel tests, a QEMU test runner, and design documentation.

File	Description
`hack/scripts/racer-dataplane-ublk-vm-test.sh`	Runs kernel tests in disposable QEMU guests.
`designs/racer-dataplane-ublk.md`	Documents configuration, architecture, lifecycle, and testing.
`cmd/racer-dataplane/src/ublk/worker.rs`	Attaches and polls each worker's device queues.
`cmd/racer-dataplane/src/ublk/uapi.rs`	Defines kernel interface layouts and command encodings.
`cmd/racer-dataplane/src/ublk/tests.rs`	Adds root-only device and lifecycle integration tests.
`cmd/racer-dataplane/src/ublk/service.rs`	Implements request serving, retries, and readahead.
`cmd/racer-dataplane/src/ublk/queue.rs`	Maps descriptors and decodes requests into page spans.
`cmd/racer-dataplane/src/ublk/mod.rs`	Organizes the new ublk module.
`cmd/racer-dataplane/src/ublk/directory.rs`	Connects pinned devices to the existing read path.
`cmd/racer-dataplane/src/ublk/control.rs`	Implements control commands and deletion fallback.
`cmd/racer-dataplane/src/test_support.rs`	Exposes read-path fixtures for ublk tests.
`cmd/racer-dataplane/src/runtime.rs`	Adds quota-backed command reservations.
`cmd/racer-dataplane/src/read/dispatch.rs`	Exposes mailbox admission state.
`cmd/racer-dataplane/src/lib.rs`	Registers the ublk module.
`cmd/racer-dataplane/src/config.rs`	Parses and validates ublk environment settings.
`cmd/racer-dataplane/src/app.rs`	Integrates startup, capacity reservation, polling, and draining.
`cmd/racer-dataplane/runtime/src/reactor/tests.rs`	Tests command results, ownership retention, and reservations.
`cmd/racer-dataplane/runtime/src/reactor/simulation.rs`	Adds scripted passthrough-command completions.
`cmd/racer-dataplane/runtime/src/reactor.rs`	Adds passthrough operations and a command-capacity partition.
`cmd/racer-dataplane/runtime/src/lib.rs`	Exposes whether a mailbox admits new commands.

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Unresolved critical issues include invalidated buffer pointers, commits before outstanding writes finish, deletion of another process’s reused device ID, and an unbounded START_DEV wait that can block the control thread.

8 open issues

2 resolved since last review

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Retained pages can exhaust the plaintext quota and cause persistent EIO on cache misses, while captured test output prevents the VM script from detecting skipped kernel checks.

6 open issues

4 resolved since last review

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Human assessment is needed because cross-worker page retention and overload-retry behavior can cause read failures or leave a device disabled until restart.

4 open issues

2 resolved since last review

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Human assessment is needed because reusing WORK_DIR can silently test a cached kernel instead of the requested GUEST_KERNEL while still passing verification.

4 open issues

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Human assessment is needed because reactor-slot exhaustion and ring-memory quota interactions can cause permanent device failure or prevent worker startup.

4 open issues

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Human assessment is needed because a shutdown race can leave requests unanswered and force process exit, while transient retry-timer overload can permanently disable a configured device.

4 open issues

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

The shared STALL flag can leave another kernel test’s read permanently pending during parallel execution, causing a hang unless stall state is isolated to the shutdown test’s backend.

5 open issues

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Human assessment is needed because a critical shutdown-ordering race can leave requests unanswered and terminate the dataplane, while the directory-backed test can pass with operations still in flight.

4 open issues

1 resolved since last review

Scope STALL state to the shutdown test backend

_{Review effort: Balanced}

Params was 108 bytes of fields padded to 112, so to_bytes read uninitialized padding. Add an explicit zeroed pad field and compile-time asserts that every UAPI struct viewed as bytes has no implicit padding.

The 16-byte URING_CMD payload is passed to the driver uninterpreted and may encode user addresses, so a safe caller could make the kernel access arbitrary memory. Mark uring_cmd and uring_cmd_reserved unsafe with a Safety contract, and justify the ublk queue call site (USER_COPY, addr 0, debug-asserted) and the test helpers.

Automatic devices wait until every fixed-id device has started or failed, so the kernel cannot hand one of our automatic devices a configured id that the stale-device path would then stop. As defense in depth, a fixed id held by a device this manager serves is never replaced.

A control loop that panicked, timed out, or lost its channel could leave queues with uncancelable FETCH commands, so the worker runtime fence waited forever. The control thread now catches panics, marks itself done, and sends STOP_DEV over a fresh /dev/ublk-control to every device that still has an attached queue. If that fails, it exits the process so the kernel aborts the fetches.

If /dev/ublkcN cannot be opened, fail that device and leave it for manual cleanup instead of issuing an unpinned DEL_DEV that could delete another process's device that reused the id.

Abandonment was raced against the whole read, so it could drop a pending USER_COPY write and send COMMIT_AND_FETCH before the copy finished. Race it only against page acquisition, await started writes, and start no new write once abandoned.

START_DEV now has an absolute deadline (control timeout, at least 5 min). On expiry the channel is poisoned and TimedOut is returned. The manager abandons the device's queues so a stuck START_DEV can finish, then tears the device down over a fresh control channel; the poisoned one is leaked.

The page cache keeps only a weak plaintext handle, so once a ublk request dropped its page, the next small read of the same page decrypted all 16 MiB again. Keep the last pages each queue used (2 per queue, 4 per worker across devices, LRU), released on abandon, finish, drop, and once the control thread is done so they never hold up shutdown or the fence.

Pages kept in the ublk page window are quota-charged plaintext owners. At the minimum valid per-worker quota ((range_window_pages + 1) pages, checked in partition_limits_with_cause) two kept 16 MiB pages could fill the quota. Windowed::page fetched before insert evicted anything, so every miss failed with EIO, and other reads sharing the quota failed too. - Size the window from the per-worker plaintext quota: a worker keeps at most what remains after range_window_pages + 1 pages in flight (zero at the minimum quota), capped at the old bounds. - On a miss, drop the page keeping the new one would evict before fetching. - If the fetch is still refused as Overloaded (for example kept pages of other queues or workers charged to the same owner), drop every kept page of the worker and retry once. Tests cover a miss with the quota exactly filled by kept pages, pre-fetch eviction, the quota-sized window at the minimum quota, refusal pass-through when nothing is kept, and window sizing for small, large, and degenerate quotas.

Libtest captures output of passing tests, so a kernel test that skipped and returned ok never printed its skip message and the harness could not reject the run. Run the guest test binary with --nocapture.

…_KERNEL

… queues

Launch the real racer-dataplane binary with RACER_UBLK_DEVICES and two I/O workers, read the whole device from CPUs on different hardware queues, check writes fail, and check SIGTERM during reads exits cleanly and removes the device it created.

Accept several test binaries and an optional DATAPLANE_BIN, bring up loopback, give scratch an O_DIRECT-capable loop-mounted ext2, and default to four CPUs so the dataplane plans two I/O workers.

…efore deleting

Jordan Olshevski (jveski) requested a balanced review from Copilot October 28, 2026 00:01

Copilot started reviewing on behalf of Jordan Olshevski (jveski) October 28, 2026 00:02

View session