Add a usbdevfs fd with the synchronous ioctls - #366
Open
jotpalch wants to merge 1 commit into
Open
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
New src/syscall/usbdev.c/.h: typed FD_USBDEV constructor for
/dev/bus/usb/BBB/DDD (any access mode except O_PATH; the host fd is a
readiness pipe for the async stage's poll), read/pread/readv/lseek
serving the usbfs descriptors blob at a per-open position, fstat as char
189:minor, and the synchronous ioctl set against IOUSBDeviceInterface650
/ IOUSBInterfaceInterface800: CLAIM/RELEASEINTERFACE, SETINTERFACE
(implicit claim + SetAlternateInterface + pipe-map rebuild),
SETCONFIGURATION (-EBUSY while anything on the device is claimed,
device-wide like usb_interface_claimed: this fd's claims, another usbfs
fd's claims via a lock-free per-slot mirror, and bound host drivers via
one registry iterator pass), CLEAR_HALT/RESETEP
(ClearPipeStallBothEnds), GETDRIVER, GET_SPEED (the enum as the return
value), CONNECTINFO, DISCONNECT_CLAIM, USBDEVFS_IOCTL
DISCONNECT/CONNECT, sync CONTROL (DeviceRequestTO) and sync BULK
(Read/WritePipeTO). kIOReturn maps to errno per devio.c: a stall is
-EPIPE, a timeout -ETIMEDOUT, a vanished device -ENODEV, and
kIOReturnAborted is -EINTR with syscall_restart_forbid(), because the
transfer was already on the wire and a dispatcher restart would send it
twice; the eintr contract records ioret_neg_errno as 'forbids'.
The open consults the model, not the hardware. stat, access and an
O_PATH open of the same node are all answered from the model, so
resolving the IOKit service at open time made a plain O_RDONLY the one
entry point that reported ENODEV for a node the other three describe,
and it put the ELFUSE_USB_FIXTURE model, whose devices have no IOKit
service at all, out of reach of every assertion about this fd. The
service is resolved on first use instead, and each operation that needs
the wire reports -ENODEV without one. usb-sysfs models the node 0666 for
the same reason: the owner it reports is the host user, the guest's uid
need not equal it, and 0664 left access(W_OK) refusing what open(O_RDWR)
then served.
Interface arbitration is attempted, not predicted. macOS arbitrates per
IOKit object and dynamically: USBInterfaceOpen answers
kIOReturnExclusiveAccess exactly while a driver holds that interface
open and succeeds while it is idle. So the claim attempts the open and
maps what IOKit answers, rather than refusing on the presence of a
driver child -- which refused work macOS grants. Measured on the
attached ESP32-S3, with the same native IOKit probe run at the same
moment: with nothing holding /dev/cu.usbmodem1101, USBInterfaceOpen on
interface 1 (CDC data, bound to AppleUSBACMData) returns 0x00000000 and
CLAIMINTERFACE 1 now returns 0 where it returned -EBUSY; with the tty
held, both say ExclusiveAccess and -EBUSY. Interface 0 (CDC control) is
ExclusiveAccess and -EBUSY either way. Relatedly, a user client is not a
driver: IOKit publishes an AppleUSBHostInterfaceUserClient child for
every USBInterfaceOpen, so taking the first child of any class reported
a peer usbfs consumer as a bound host driver. GETDRIVER of an interface
another usbfs fd holds now reports usbfs (measured:
driver=AppleUSBHostInterfaceUserClient before, usbfs after).
Kernel-fidelity gates and check order, all of it observable:
- Every ioctl on an O_RDONLY fd is -EPERM (FMODE_WRITE, devio.c:2605),
ahead of everything else.
- The whole write family is -EBADF on an O_RDONLY fd and -EINVAL
otherwise, the order vfs_write checks FMODE_WRITE then
FMODE_CAN_WRITE. write() had this; writev, pwrite and pwritev fell
through to the readiness pipe behind the fd and answered its -EBADF
for a writable fd.
- check_ctrlrecip runs before the wLength cap (devio.c:1176-1181) and
findintfep before the transfer-length checks (devio.c:1289-1298), so
a request naming an endpoint or interface the device does not have
is -ENOENT however long it is.
- The reserved-bit test on an endpoint address (devio.c:860) lives in
the one lookup BULK, CLEAR_HALT, RESETEP and the control endpoint
recipient all reach, so the four cannot disagree about it, and it
tests the caller's whole 32-bit word, as findintfep does. Narrowing
to a byte first made ep 0x183 and 0x01000083 both name endpoint
0x83: measured on the board, all three cleared the stall on 0x83,
where Linux answers -EINVAL for the outer two.
- The interface-number bound is 64: claimintf refuses ifnum >= 8 *
sizeof(ps->ifclaimed) and ifclaimed is an unsigned long (devio.c:75,
:785). Between 64 and 32 an interface is merely absent, a question
about the device, so claimed_mask is a uint64_t. The bound also
applies to the number as it leaves the descriptor: bInterfaceNumber
is a device-supplied byte with the whole 0..255 range behind it, and
the endpoint-owner lookup reports it back as checkintf's -EINVAL
(devio.c:842) rather than handing ifaces[64] an index of 200.
- An altsetting above 255 is -EINVAL, after the implicit claim, the
order proc_setintf decides them in (devio.c:1533-1539):
bAlternateSetting is a byte, so usb_altnum_to_altsetting matches
nothing (usb.c:391) and usb_set_interface refuses (message.c:1548).
Narrowed into SetAlternateInterface's UInt8 instead, altsetting 256
selected setting 0 and took effect -- measured on the board.
- The positional and vector write paths answer in Linux's order too:
ksys_pwrite64 and do_pwritev test pos < 0 before the fd lookup, so a
negative offset is -EINVAL rather than the -EBADF this fd owes for
having no write op; vfs_readv and vfs_writev test the file's mode
before an empty vector returns 0, so readv, preadv and pwritev2 of
nothing still answer by direction; and pwritev2(RWF_APPEND) had no
dispatch of its own and reached the readiness pipe, answering
-ESPIPE.
- Every failure the constructor reports carries the errno of what
failed. ENODEV from the model -- nothing answers to this address --
is the one that becomes the -ENOENT open(2) owes for a name with
nothing behind it; an allocation, a scratch-tree mkdir, or a pipe
the host would not give reports itself, where all three used to be
flattened into -ENOENT or -EMFILE. usbdev_build_pipe_map carries the
IOReturn the same way instead of folding it into -EIO.
- proc_disconnect_claim answers -EINVAL for an interface the device
does not have (devio.c:2467-2469), the opposite of claimintf's
-ENOENT.
- USBDEVFS_IOCTL CONNECT on an interface that already has a driver is
-EBUSY (devio.c:2357-2361); only a free one reaches device_attach.
- SETINTERFACE rewrites -ENOENT and -EPROTO to -EINVAL and passes
every other answer through: the IOReturn SetAlternateInterface
produces for an absent altsetting is not in the errno table and
reached the map's default -EPROTO.
- SEEK_END is -EINVAL, and the new position is range-checked with
__builtin_add_overflow, which lseek(fd, INT64_MAX, SEEK_SET)
followed by lseek(fd, 1, SEEK_CUR) reaches.
- GET_CAPABILITIES reports 0. Every bit in that word names part of the
SUBMITURB/REAPURB machinery, which is -ENOTTY here; the async stage
raises the word as it lands each one.
- A short bulk OUT is -EIO: WritePipeTO reports no length, and
reporting the requested count would spell a partial write as a
complete one.
Locking and lifetime. A lookup pins its entry under usbdev_table_lock
and takes the per-entry lock with the table lock released. Nesting the
two meant a thread blocked on one fd's transfer held the table lock on
every other thread's behalf: measured, a GET_SPEED on an unrelated fd
waited 5517 ms behind a 6 s BULK, and it now returns in 0 ms, as do an
open and a close of the same node. Two ioctls on ONE fd still serialize;
Linux drops the device lock around the URB wait (devio.c:1219/1245,
:1337/1357) and this does not, which is recorded below rather than
claimed as kernel behavior. The teardown hook is handed a bare fd number
after the fd-table slot is already free, so a sibling thread's open can
already hold that number and two entries answer to it; matching on the
number alone tore down whichever sat lower in the table, about half the
time the live one. fd_alloc stamps a globally monotonic generation, so
the closing entry is the one with the smaller generation. Measured with
the mutation restored: 405 of 8000 freshly opened fds lost, 0 with it.
fd_alloc also publishes the fd number before the side table can bind it,
and the same hook matches on that number, so a close landing in that
window found nothing to tear down and left the entry, its blob and its
pipe held for the life of the process. The open rereads the fd's
generation once its entry is findable and retires the entry itself if
the close has been and gone. Driven by ELFUSE_USBDEV_PUBLISH_DELAY_US,
which widens the window: 32 simultaneous opens before the race and 31
after it without this, 32 with it.
Plumbing: FD_USBDEV in linux-wire.h (+LINUX_EPIPE/LINUX_ETIME), dispatch
hooks in io.c (ioctl/read/write/writev/pwrite/pwritev/pwritev2 and the
empty-vector arm), fs.c (openat constructor, lseek arm, dup refusal),
fs-stat.c (fstat, ahead of the /dev/bus path stamp, which otherwise
claimed every one of these fds and left the typed answer unreachable),
usbdev_init in syscall_init, usb-sysfs exports
usb_sysfs_device_info/node_stat.
usbdev_table_lock is registered in internal.h's lock order, whose entry
now describes what the code does -- table lock dropped before the entry
lock, refs and dead pinning the slot across the gap -- rather than a
nesting no path takes. The constructor snapshots the fd generation
before taking the table lock so fd_lock never nests under it. The
completion-pipe fds are initialized to -1 so the merged error arm cannot
close two indeterminate descriptors when pipe() itself fails.
Tests. tests/test-usbdev-ioctl.c is a new hardware-free lane: the
fixture's devices are modeled with no IOKit service behind them, which
is exactly the shape of a device the machine cannot reach, so the file
contract, the ioctl gates and the whole argument-validation surface get
69 assertions that answer the same on any host. It carries the four-
thread open/read/close churn that the teardown identity needs, and
prints each known gap as an XFAIL with both values. The
ELFUSE_USB_FIXTURE model grew the endpoint descriptor its interface
descriptor already claimed, so the endpoint-addressed paths have
something to resolve against; tests.mk keeps the fixture on the test-
usb-sysfs lane, which the earlier draft had removed after the
constructor stopped serving modeled devices.
test-usbdev-faults runs the same binary five more times, forcing one
condition each, because none of the five can be provoked from a guest on
a healthy host. ELFUSE_USB_FIXTURE=badifnum stands up a device whose one
interface declares bInterfaceNumber 200 and carries endpoint 0x81: every
byte is well formed and the range is the field's own, so nothing anyone
can plug in emits it. The answer is -EINVAL on both sides of the bound,
which is what checkintf answers, and what changes is that nothing reads
out of bounds first -- with the bound removed and the fixture in place,
UBSAN reports "index 200 out of bounds for type usbdev_iface_t[64]" at
the array read and aborts the run. ELFUSE_USBDEV_OPEN_FAULT fails the
model lookup and the descriptor copy with ENOMEM and the readiness pipe
with ENFILE, which the guest saw as ENOENT, ENOENT and EMFILE before.
ELFUSE_USBDEV_PUBLISH_DELAY_US widens the publish window for the close
race above. Both hooks are read once and do nothing when unset, the same
shape as the directory and identity lanes' hooks, and both are recorded
in docs/testing.md.
Known gaps, each printed by that lane as an XFAIL and listed in
internals.md: no disconnect gate (Linux answers -ENODEV for every ioctl
once the device is gone; the answers served from the open-time model
still report it, and noticing the disconnect needs an IOKit termination
notification on a run loop, which is the async stage's machinery); two
ioctls on one fd serialize; GETDRIVER reports the IOKit class name where
Linux reports the driver name, and DISCONNECT_CLAIM's name filters
compare against it; RESET clears the claimed pipes' stalls instead of
re-enumerating (ReEnumerate would destroy the handles); a short bulk OUT
is -EIO; the side table holds 32 open fds per process and reports
-ENOMEM past that, where Linux allocates a usb_dev_state per open;
USBDEVFS_IOCTL DISCONNECT of an interface another usbfs fd holds is
-EBUSY where Linux releases that claim; dup() is -EBADF and fork drops
the fd (fd_type_is_synthetic).
Document the layer as internals.md USB Device Passthrough.
Verified on the attached ESP32-S3 (303a:1001, modeled as
/dev/bus/usb/002/001), every line re-run against this build: CLAIM
iface2 = 0, CLAIM iface1 = 0 with the tty free and -EBUSY with it held,
CLAIM iface0 = -EBUSY; GETDRIVER(0) = AppleUSBACMControl, GETDRIVER(5) =
-ENODATA, GETDRIVER(2) from a second fd = usbfs; SETINTERFACE 2/9 =
-EINVAL, 2/0 = 0; DISCONNECT_CLAIM iface5 = -EINVAL; USBDEVFS_IOCTL
CONNECT iface1 = -EBUSY, iface5 = -EINVAL; CONTROL GET_DESCRIPTOR = 18
bytes matching the first 18 of the 116-byte read(); CLEAR_HALT 0x83 = 0,
CLEAR_HALT 0x30 = -EINVAL; BULK OUT 1 byte = 1, BULK on absent ep 0x09
with len 32 MB = -ENOENT, BULK IN on the idle ep 0x83 with a 5000 ms
timeout = -ETIMEDOUT after 5038 ms; GET_CAPABILITIES = 0, GET_SPEED = 2
(FULL), CONNECTINFO devnum 1. The arguments that used to be narrowed,
same board, iface2 claimed: CLEAR_HALT 0x183 = -EINVAL and 0x01000083 =
-EINVAL where all three used to clear the stall on 0x83 and return 0,
RESETEP 0x183 = -EINVAL, CLEAR_HALT 0x83 = 0 still, BULK ep 0x102 =
-EINVAL, SETINTERFACE 2 alt 256 = -EINVAL where it used to return 0
having selected setting 0, alt 7 = -EINVAL, alt 0 = 0, and CONTROL
GET_STATUS with wIndex 0x0183 = 2 bytes, since check_ctrlrecip masks
wIndex to its low byte before the lookup (devio.c:904) and that path
must not gain the wider test.
jotpalch
force-pushed
the
pr-c-usbdev-sync
branch
from
September 4, 2026 19:06
79ff474 to
c22f0d9
Compare
jserv
reviewed
Sep 4, 2026
| static void usbdev_retire_unpublished(usbdev_t *u) | ||
| { | ||
| pthread_mutex_lock(&usbdev_table_lock); | ||
| bool mine = !u->dead; |
Contributor
There was a problem hiding this comment.
mine = !u->dead is the only claim test, so this can retire an entry that is no longer the one this open created. If the guest closes guest_fd right after the publish at 1318, usbdev_fd_cleanup tears this entry down and usbdev_unref frees the slot (used = false); a sibling usbdev_open_path can then take that slot and bind its own live fd before 1334 runs. This call marks that entry dead, frees its blob and closes its pipe_wr, and the new fd answers EBADF on every read and ioctl. Same failure the generation tiebreak in usbdev_fd_cleanup exists to avoid.
Pass guest_fd and gen in, and claim only when u->used && !u->dead && u->guest_fd == guest_fd && u->generation == gen.
| * a guest racing close() on a guessed fd number lands in a window where | ||
| * usbdev_fd_cleanup cannot find this entry. The publish below closes it. | ||
| */ | ||
| int guest_fd = fd_alloc(FD_USBDEV, pipefd[0], usbdev_fd_cleanup); |
Contributor
There was a problem hiding this comment.
fd_alloc_from() takes an out_gen captured inside the allocating fd_lock section, which is the proof this path re-derives by hand at 1315. Re-reading after the slot is publishable means that if the guest closes guest_fd and any thread reopens the same number as another FD_USBDEV fd in the window, gen is the other allocation's stamp: the guard at 1334 compares equal, no retire happens, and both entries then match guest_fd == fd && generation == snap.generation in usbdev_acquire, which returns whichever sits lower. The orphan keeps its slot, blob and pipe write end for the life of the process, which is the leak this comment says was closed.
fd_alloc_from(0, FD_USBDEV, pipefd[0], usbdev_fd_cleanup, &gen) removes the window and one fd_lock trip.
| * readiness pipe, so nothing below applies. usbdev_ioctl re-snapshots the | ||
| * fd and pins the side-table entry by generation itself. | ||
| */ | ||
| if (fd_get_type(fd) == FD_USBDEV) |
Contributor
There was a problem hiding this comment.
do_vfs_ioctl handles FIONBIO and FIOASYNC for every file before f_op->unlocked_ioctl runs, so Linux answers them on a usbdevfs fd. This early return sends them into usbdev_ioctl, whose default arm is ENOTTY and whose FMODE_WRITE gate fires first on an O_RDONLY fd: ioctl(fd, FIONBIO, &one) returns 0 and sets O_NONBLOCK on Linux, ENOTTY here (EPERM read-only), while fcntl(fd, F_SETFL, O_NONBLOCK) on the same fd still succeeds. Two entry points onto one flag disagree.
FIOCLEX and FIONCLEX are already tested above this branch; these two belong there.
| */ | ||
| if (bt.len >= (uint32_t) INT32_MAX) | ||
| return -LINUX_EINVAL; | ||
| if (bt.len > USBDEV_BULK_MAX) |
Contributor
There was a problem hiding this comment.
usbfs_memory_mb is a global in-flight allowance charged by usbfs_increase_memory_usage and refunded on completion, not a per-call size cap. Two consequences: bt.len == USBDEV_BULK_MAX passes here, where Linux charges len + sizeof(struct urb) against the same 16 MB and returns ENOMEM; and nothing accumulates across fds, so a guest holding all 32 side-table slots can keep 32 concurrent 16 MB buffers live in the host.
An atomic byte counter charged here and refunded after the transfer gets both.
| if (!fd_snapshot(fd, &snap) || snap.type != FD_USBDEV) | ||
| return -LINUX_EBADF; | ||
| /* Every usbdev ioctl needs FMODE_WRITE (devio.c:2607-2608). */ | ||
| if ((snap.linux_flags & LINUX_O_ACCMODE) == LINUX_O_RDONLY) |
Contributor
There was a problem hiding this comment.
This tests the literal value O_RDONLY, but Linux derives the gate from OPEN_FMODE(flags) = (flags + 1) & O_ACCMODE, which gives access mode 3 neither FMODE_READ nor FMODE_WRITE. open("/dev/bus/usb/001/001", 3) succeeds (0666 node, ACC_MODE(3) is read plus write) and the fd passes this gate, so every ioctl is accepted where Linux answers EPERM. Same shape for the read gates at 1347 and 1385.
Deriving the two capability bits once from (accmode + 1) & 3 covers all four.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NOTE: piece C of the USB layer described in #319, on top of #334 and #351. It adds the usbdevfs descriptor and the synchronous half of its ioctl surface. Transfers that need the asynchronous engine (SUBMITURB, REAPURB, DISCARDURB) are the next piece and are not here; this one answers ENOTTY for them and reports no URB capability bits, which is what makes that honest.
Summary
A guest that opens
/dev/bus/usb/BBB/DDDgets a descriptor whose file contract and ioctl surface followdrivers/usb/core/devio.c: the permission answers, the write family, the file position, the argument validation the kernel does before it touches the wire, and the synchronous operations mapped ontoIOUSBDeviceInterface650andIOUSBInterfaceInterface800.CLAIMINTERFACE,RELEASEINTERFACE,SETINTERFACE,SETCONFIGURATION,CONTROL,BULK,CLEAR_HALT,RESETEP,RESET,GETDRIVER,DISCONNECT_CLAIM,IOCTL,CONNECTINFO,GET_SPEEDandGET_CAPABILITIESare implemented.Driver arbitration is per object and dynamic, not per device
The one design point worth stating up front, because it changed after measurement. macOS arbitration is not a property of a device class that can be decided in advance. A device-level open succeeds whenever the bound driver holds no exclusive device open, and an interface-level open succeeds unless that specific driver currently holds that specific interface. So the layer attempts the open and maps the IOKit result rather than refusing by class.
Measured on an attached ESP32-S3 (
303a:1001), with a native IOKit probe run at the same moment as the guest call:An earlier revision refused interface 1 outright because a driver was bound to it, which refused work macOS allows. A tool that talks to a board over the vendor interface while the CDC tty stays available to the rest of the system is the case this serves.
What the argument validation follows
Each of these is a devio.c line rather than a guess, and each has an assertion in the new lane.
O_RDONLYdescriptor isEPERM(devio.c:2605), which isFMODE_WRITEand not a permission check on the node.check_ctrlrecipruns before thewLengthcap (devio.c:1176-1181), andfindintfepbefore the transfer-length checks (devio.c:1289-1298), so an absent endpoint outranks a malformed length rather than the other way round.devio.c:860) lives in the one lookup all four endpoint ioctls share.claimintfrefusesifnum >= 8 * sizeof(ps->ifclaimed)andifclaimedis anunsigned long(devio.c:75, :785). Between 32 and 64 an interface is merely absent, which is a question about the device, so the claimed mask is auint64_t.DISCONNECT_CLAIManswersEINVALfor an interface the device does not have (devio.c:2467-2469), the opposite ofclaimintf'sENOENT.USBDEVFS_IOCTLwithCONNECTon an interface that already has a driver isEBUSY(devio.c:2357-2361); only a free one reachesdevice_attach.Deviations, recorded rather than implied
These are printed by the lane as XFAIL rows carrying both values, so they stay visible in its own output rather than living only in a commit message.
RESET: Linux re-enumerates the port. IOKit's equivalent isUSBDeviceReEnumerate, which is an unplug that tears down the user client, so this clears the claimed pipes' stalls and returns 0 instead.ENODEVfor every ioctl. This still servesGET_SPEED,CONNECTINFO,GET_CAPABILITIESandread()from the model captured at open, because noticing the departure needs an IOKit termination notification on a run loop, which is the next piece's machinery.GETDRIVERreports the IOKit class (AppleUSBACMControl) where Linux reports the driver name (cdc_acm), andDISCONNECT_CLAIM's name filters compare against it. A guest asking to detachcdc_acmby name will not match.EIO:WritePipeTOreturns no length, so the byte count Linux would report is not available.ENOMEMpast that, where Linux has no such limit.Tests
tests/test-usbdev-ioctl.cis a new lane, 54 assertions, that needs no hardware: it runs againstELFUSE_USB_FIXTURE, whose devices are modeled but have no IOKit service behind them, which is exactly the shape of a device the model knows and the machine cannot reach. It covers the half of usbdevfs decided before the wire. Run without the fixture it prints its unmet precondition and exits rather than reporting assertions that failed for the wrong reason.The half that needs a device, a real claim and a real transfer and arbitration against a bound host driver, is verified on the attached board and is not pretended to be covered by the lane.
Evidence
Summary by cubic
Implements the synchronous usbdevfs portion described in #319. Writable opens of
/dev/bus/usb/BBB/DDDpreviously returnedEACCES; they now create a typed fd that serves the descriptor blob and maps synchronous ioctls to IOKit, while async URBs remainENOTTYwith no capability bits. Wire access resolves lazily, so modeled devices can open without a matching host service and reportENODEVonly when an operation needs hardware.Behavior
EBUSYonly while the interface is held.read,pread,readv, andlseekfrom a per-open descriptor position and reportsfstatas USB char device189:minor.dup()and fork inheritance;O_PATHcontinues using the existing path-only fd.Tests and known gaps
GETDRIVER, non-reenumeratingRESET, short bulk OUT asEIO, the 32-fd side-table limit, and cross-fd disconnect behavior.Written for commit c22f0d9. Summary will update on new commits.