You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(racer): serve Racer objects as read-only ublk devices - #985
Add racer-block, a standalone Go daemon that serves one pinned Racer object as a read-only Linux ublk device through Racer's client Unix socket. It uses an in-house io_uring binding, a chunk cache with shared fetches and readahead, and bounded retries. The command includes device lifecycle controls, build targets, and usage documentation, with no new Go dependencies or Kubernetes integration. Device and startup-failure tests run in disposable Linux 6.8 and 7.0 VMs in CI. The accompanying fixes cover Gantry e2e peer reuse and cluster isolation, and socket-close races in Racer dataplane descriptor tests.
[P2] Compare chunk indexes instead of multiplying in the loop condition. For an accepted object size such as math.MaxInt64-511, incrementing past the final partial chunk makes idx*b.chunkSize wrap negative. If buffer space remains, the loop enters newChunkLocked with an out-of-range index and panics on a negative buffer length (backend.go:380-392). This happens while b.mu is held, so the ublk handler's panic recovery still leaves later reads and shutdown blocked. Bound the loop by the final chunk index and add a test that exercises prefetch at the end of a near-MaxInt64 object; the existing overflow test only checks advance.
Validate configuration before deleting the existing device
internal/racerblock/racerblock.go:91
[P2] Validate the device configuration before deleting the existing device. For example, --id 7 --max-io-bytes 1 passes CLI validation (cmd/racer-block/main.go:237-239). This code then stops and deletes device 7 before ublk.Start rejects the unaligned I/O size (ublk/api.go:105-107). An invalid replacement invocation therefore removes the existing device without creating a replacement. Expose and reuse the complete ublk configuration validation before this destructive step, and test that invalid options leave the existing device intact.
Read beyond the header limit to detect oversized input
cmd/racer-block/main.go:293
[P2] Read one byte beyond the largest valid header file. An 8192-byte value followed by CRLF fills the current read limit, so any additional content is silently discarded. Trimming then leaves an accepted 8192-byte value rather than reporting oversized input. Increasing the limit to maxHeaderFileBytes+3 preserves valid CRLF-terminated values while detecting extra content. Add a case with an 8192-byte value followed by CRLF and a nonempty suffix.
Record queue failures synchronously before asynchronous teardown
internal/racerblock/ublk/device_linux.go:316
[P2] Record queue failures synchronously before scheduling teardown. After startup, fail only launches a teardown goroutine through stopAsync (device_linux.go:124). If cancellation lets Close or the context watcher win once.Do before that goroutine runs, teardown records nil or context.Canceled and discards the already-reported queue failure. waitAndClose then returns success (internal/racerblock/racerblock.go:158-175). Preserve the first failure under synchronization and retain it when another caller starts cleanup. Add a regression that delays asynchronous teardown until after cancellation initiates cleanup.
Allow startup reads before enabling the blocking handler
[P2] Allow startup reads before arming the blocking handler. On the supported Linux 6.8 path, START_DEV performs partition-scan reads before returning, but this handler blocks every read until cancellation. startTestDevice waits for START_DEV (device_linux.go:249-253), so the test cannot reach cancel() and hangs until the test-process timeout. Serve startup reads normally, then enable blocking after Start returns and before issuing the explicit test read.
Read beyond maximum header size to detect trailing content
cmd/racer-block/main.go:293
Read one byte beyond the largest accepted header file. A file containing 8192 value bytes followed by CRLF and additional content is currently truncated just after CRLF. Trimming then makes the length check pass, silently sending only the prefix as Authorization or Racer-Metadata. Raising the read limit to maxHeaderFileBytes+3 preserves maximum-length CRLF-terminated values while detecting oversized input. Add a regression case with content after that CRLF.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Kernel-dependent resource lifetimes and recovery require human validation with privileged VM results, alongside resolution of the shutdown error-recording race.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Preserve late serve errors during shutdown
internal/racerblock/ublk/queue_linux.go:238
If Close or cancellation sets d.closing after serve returns a genuine error but before this check, localStop() becomes true and the error never reaches d.fail. Cancellation can then be reported as a clean shutdown by racerblock.go:164-165. Preserve nonnil errors returned by serve even when teardown has started, while keeping the decision to enter queue recovery separate from failure recording. Add a regression that exercises this condition through run; the late-failure tests in stop_cause_linux_test.go call d.fail directly and bypass it.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Kernel-dependent deletion and recovery require final human review, and the retention-renewal correction remains unresolved.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Renew warm retention before the granted deadline
internal/racerblock/backend/retry.go:95
Schedule renewal from the granted retention deadline, not the requested TTL. The dataplane clamps TTL to its configured maximum (cmd/racer-dataplane/src/read/warm.rs:896-898), which defaults to one hour (cmd/racer-dataplane/src/config.rs:192). With --warm-ttl=4h and no longer existing hold, retention expires after one hour, but this loop waits two hours to refresh it. The pinned version is then eligible for eviction between renewals. Preserve the initial WarmResult.RetainUntil instead of discarding it in backend.go:158, and schedule each renewal before that deadline, updating the timer from subsequent successful Warm results. Add a regression test where the granted retention is shorter than the requested TTL.
Fail a queue right after the kernel handed it a read, and fail the EIO
loop while STOP_DEV is seen waiting in del_gendisk for a held request.
BrokenRingAtStartDev now expects the rescue path; the publication probe
the fallback relies on gets its own VM test.
A queue loop error seen after teardown marked the device closing used to
close the ring at once. STOP_DEV may still be waiting for a request that
queue owns, which Linux 7.0 then strands, so answer I/O through a rescue
ring until teardown stops the queue.
drain took every finished result off the pending list and returned at
the first prep failure, so later results were lost and their tags stayed
busy forever. A rescue sweep skips busy tags, so STOP_DEV could wait on
them. Put unprepared results back and clear busy only once a commit is
prepared.
A rescue may follow resource exhaustion, so it must not need a new
io_uring. Each queue now sets up a spare ring with its main ring, Start
fails before START_DEV if it cannot, and a rescue takes the spare. An
unused spare is closed by release.
harvestOld never enters the replaced ring, so it depends on that ring's
completion queue not overflowing. Spell out the real bound, one command
per tag plus the eventfd poll, name the ring size after it, and test it
for every supported depth. Reword the rescue comment.
Closing alone no longer ends a failed queue: run now rescues it until
the queue is stopped. Have the fixtures stop the queue once the rescue
serves, as teardown's STOP_DEV would, and assert the rescue ran.
GANTRY_E2E_CLUSTER selects the kind cluster so the suite can run beside
other clusters on a shared host. Metric waits now dump pod state, Gantry
logs, and each Gantry pod's /metrics into e2e/.artifacts before failing,
so CI uploads evidence instead of an empty directory. Add a helper that
waits for a pod's in-flight pulls to drain.
The e2e ConfigMap makes every kind node a chair with eight seeds per
digest, so the first cold pull also seeds the second worker. Its pull is
then served from its own containerd store and never reaches Gantry, and
the cluster-wide peer-hit assertion passed only when the first worker
happened to lose a race against its own seed pull.
Wait for the second worker's seed pulls to drain, evict its copy, and
assert a peer hit on that worker's Gantry pod.
GANTRY_E2E_CLUSTER only redirected kind; kubectl calls, including
port-forward and apply, still used the ambient KUBECONFIG and its current
context. Route every harness subprocess through one command helper that
sets KUBECONFIG to tmp/gantry-e2e/<cluster>/kubeconfig and pins kubectl to
kind-<cluster>. bootCluster creates into and exports to that file, also on
reuse.
Also clarify the smoke comment: origin fallback is still possible after
eviction, and the requester's peer-hit assertion is what proves reuse.
The cluster name becomes a path component of the private kubeconfig, and
kind only validates it after the harness has created that directory.
Reject empty, dot, dot-dot, absolute, and separator-containing names
(forward slash or backslash) in harnessKubeconfigPath, and resolve the
name at the start of newHarness before any filesystem write.
When startup added a device but never opened /dev/ublkc<ID>, teardown
sent STOP_DEV and DEL_DEV by numeric id with nothing pinning it, so they
could hit another device that reused the id. Send nothing by id in that
case and report the device id for manual cleanup, joined to the original
Start error.
Startup sent SET_PARAMS before opening /dev/ublkc<ID>, and a successful
open could pin a device another process added after deleting ours. Tag
ADD_DEV with a random nonzero cookie in ublksrv_flags, which the driver
stores and returns from GET_DEV_INFO without interpreting. Open the char
device first, then require a matching cookie before setting charFd; on a
mismatch or query error close the fd so teardown sends nothing by id.
Between ADD_DEV and the char device open, delete the device Start added
and add a replacement under the same id with cookie 0. Start must reject
it and leave the replacement unstarted and undeleted. Require the test in
the VM harness and clarify that only mutating commands wait for the
verified pin.
Concurrent tests spawn processes. A child keeps copies of all descriptors
until exec, so dropping a test peer socket may not close it at once.
Wait a bounded time for full disconnect and prove that disconnect waits
for the last peer descriptor copy.
net.Dialer timeouts match context.DeadlineExceeded, so listing it as
permanent failed block reads on a transient Racer dial timeout without
using the configured retries. fetch already stops once the backend
context ends.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add racer-block, a standalone Go daemon that serves one pinned Racer object as a read-only Linux ublk device through Racer's client Unix socket. It uses an in-house io_uring binding, a chunk cache with shared fetches and readahead, and bounded retries. The command includes device lifecycle controls, build targets, and usage documentation, with no new Go dependencies or Kubernetes integration. Device and startup-failure tests run in disposable Linux 6.8 and 7.0 VMs in CI. The accompanying fixes cover Gantry e2e peer reuse and cluster isolation, and socket-close races in Racer dataplane descriptor tests.