Take turns with host maintenance through a per-Job lock

[epompeii]

What

Make the runner and its host's maintenance take turns through a lock and markers in /run/bencher, so maintenance never runs during a Job, and the runner takes no Job while maintenance waits or runs.

• The job lock is /run/bencher/job.lock. Before each ready, the probe takes it with a nonblocking flock on a close-on-exec descriptor, and the runner holds it through the poll and any Job the poll brings. It is released on no_job, update, a receive timeout, a lost connection, a running that fails to send, and when the Job ends, before its result goes out.
• A marker is a file in /run/bencher named maintenance or maintenance.<name>. Maintenance creates its own before it waits for the lock and removes it when done.
• The probe pauses for maintenance:
  • while a marker exists, the runner sends paused with a maintenance reason and marker true, and leaves the lock free;
  • while another process holds the lock, it sends a maintenance reason with lock true;
  • only a probe that finds no reason to pause keeps the lock, so a runner paused for a RAID sync leaves it free too.
• The plain maintenance marker drains a runner. It finishes its current poll and any Job that poll brings, then takes no Job until the marker is gone. A unit uses a marker of its own, so it never removes the drain.
• runner run waits for the job lock, trying it every 250 ms and logging Waiting for host maintenance to finish once, and a stop ends the wait. It warns but runs through a RAID sync (Running during a RAID sync) or a marker (Running although host maintenance is due).
• /run/bencher is root's alone: the runner now creates it 0700 rather than 0755, for the runner lock and the job lock alike, and each lock file 0600. A directory an older runner created keeps its mode until a reboot or a drop-in's install -d -m 0700.
• A runner that is not root can open neither the lock nor the 0700 directory, so it takes Jobs without them, and runner up warns once at startup that maintenance may run during a Job. A root runner that cannot open or take the lock pauses with lock true rather than run unguarded.
• bencher_runner::maintenance, public and outside the plus feature, holds the directory, lock, and marker names and marker_present, which the probe and runner run both use, so host tooling can share them.
• Docs: a "Host Maintenance" section on the self-hosted runners page in all 9 languages: the lock, the markers, an example fstrim drop-in, and how to drain a Runner. The changelog gains a line.

Across versions:

• Like the RAID pause in the layer below, this needs a server with Let a runner pause on its channel without taking a job #1122. A server without it closes the channel on paused, and the runner reconnects no faster than once per poll.
• A runner without this change never takes the lock and ignores the markers, so a unit that waits on the lock runs at once, as it does today.

Why

Package upgrades, fstrim, and the monthly RAID check run on timers that know nothing about Jobs, so any of them can land during a Job and move its measurement.

The runner takes the lock before ready rather than when a Job arrives. Taken when a Job arrives, the lock could already be maintenance's, which would hold a Job the server has handed out while that Job's clock ran. Taken before ready, maintenance waits at most one poll plus one Job.

A waiting flock holds nothing the runner can see, and on each release the runner's own nonblocking try can win the lock back. The marker tells the runner that maintenance waits, so it steps aside and the waiter gets the lock at the next release. Markers of their own let two units overlap, such as a scrub and the morning upgrade: with one shared marker, the first to finish would remove it while the second still waits, and any unit would remove an operator's drain.

flock(2) locks whatever mode a file was opened in, so any user who can open the lock file can hold it and keep the runner paused. A root-only directory and 0600 files keep that to root. The descriptor is close-on-exec, so the jailer, the VMM, and every other child never keep the lock past the Job.

runner run is started by hand at a moment someone chose, so it warns about a sync or a marker and runs, but it never runs beside maintenance that holds the lock. Trying the lock in a loop, rather than blocking in flock(2), lets a stop end the wait at once.

Verification

• Lock tests use two independent opens of a file in a temporary directory, which flock treats as two holders even in one process. Each new test fails under a mutation of the behavior it guards:
  • a_free_lock_is_the_runners_turn and a_lock_held_elsewhere_pauses_for_maintenance: a turn that never takes the lock, and a held lock taken as free.
  • a_marker_pauses_without_taking_the_lock, with maintenance and maintenance.fstrim.service: a marker ignored, the lock kept past a marker (which starves waiting maintenance), and only the plain marker counted.
  • only_the_maintenance_names_are_markers: any file, or any name starting maintenance, read as a marker.
  • dropping_the_turn_releases_the_lock and a_child_never_holds_the_lock: a leaked descriptor, and one that is not close-on-exec.
  • the_lock_and_its_directory_are_root_only: job.lock at 0644 or 0640, and /run/bencher at 0755 or 0750.
  • a_runner_that_is_not_root_takes_its_turn_without_the_lock: a runner that is not root failing closed, and a root runner running on without the lock.
  • runner_run_waits_for_maintenance_then_takes_the_lock and a_stop_ends_the_wait_for_maintenance: a runner run that does not wait, gives up once maintenance holds the lock, or keeps waiting past a stop.
  • the_job_lock_is_held_from_ready_until_the_job_finishes and the_job_lock_is_released_whenever_no_job_follows: a release when the Job arrives, none when it ends, and none on no_job, update, a receive timeout, a lost connection, or a failed running send.
  • the_job_lock_is_kept_only_by_a_clear_probe: the lock kept through a pause, or dropped by a clear probe.
  • the_driver_releases_the_job_lock_when_told: a lock the probe took must be free to a second open after the driver runs the release. Kills a driver that ignores it, which would hold maintenance off for a whole server outage, since no probe runs until a connect succeeds.
• None of the new tests compiles on the parent, which has no job lock. The 14 tests pass 20 of 20 as an ordinary user and as root.
• In a disposable VM, a root runner up at a 5 s poll ran against a local API, with /run/bencher absent as after a boot:
  • The runner created /run/bencher 0700 and job.lock 0600.
  • A unit shaped like the next layer's drop-in, started 1 s into a 20 s Job, created its marker at once and waited. Its command started 1 ms after the runner released the lock at the Job's end, while the runner paused with marker and lock true. A second Job, submitted during the first, was claimed after the maintenance.
  • systemctl kill -s KILL of such a unit, while it waited and while it ran, left the unit failed and its marker gone, and the runner's next probe ended its pause.
  • touch /run/bencher/maintenance paused the runner once its current poll ended, with marker true and lock false. A Job submitted then read pending for 30 s and was claimed within 2 s of the rm.
• runner run behind a held lock, in the same VM, sandboxed with Firecracker, while another process held job.lock for 20 s:
  • It logged Waiting for host maintenance to finish once, started no VM until the release, then ran its benchmark.
  • While Firecracker ran, job.lock was open only in the runner, and /proc/locks showed the runner's pid holding it.
  • SIGTERM during the wait ended the run in 0.24 s, with no VM started and the holder's lock untouched.
• The scenario runner_run_takes_its_turn_at_the_job_lock puts both halves of runner run's turn in CI, in about 7 s:
  • The harness holds job.lock and starts a sandboxed runner run. The scenario requires one Waiting for host maintenance to finish, then no jail for the 2 s until the release, then the Job. Once the VM is up, neither Firecracker nor the jailer it came from may hold the lock.
  • The harness then creates /run/bencher/maintenance and runs the image on the host. The scenario requires one Running although host maintenance is due, no wait, and a successful Job whose benchmark, the runner's own child, lists no descriptor of job.lock. The host run carries the close-on-exec check, since the jailer closes every descriptor it inherits.
  • The marker and the harness's lock are gone on every path.
  • It fails when runner run never waits, when its descriptor is not close-on-exec, and when a marker makes it refuse, wait, or run without the warning. It fails on the parent, which has no job lock, and passes 20 of 20 as root.
• The runner unit tests pass as an ordinary user and as root, 441 of 441, the CLI tests pass, 14 of 14, the runner_ops tests pass, 86 of 86, cargo test-runner scenarios --build-only passes, Cargo.lock is unchanged, and clippy passes with -D warnings for Linux x86_64, Linux aarch64, and macOS on the runner, its CLI, test_runner, and runner_ops. The scenario suite passes as root, 100 of 100, and clippy passes for the whole workspace as CI runs it, on the top of this stack.

[github-actions]

Bencher Report

Project	Bencher
Branch	u/ep/runner-protocol/job-lock
Testbed	intel-v1

Click to view all benchmark results

Benchmark	Latency	Benchmark Result microseconds (µs) (Result Δ%)	Upper Boundary microseconds (µs) (Limit %)
Adapter::Json	📈 view plot 🚷 view threshold	5.21 µs (+0.49%) Baseline: 5.18 µs	5.93 µs (87.78%)
Adapter::Magic (JSON)	📈 view plot 🚷 view threshold	5.02 µs (+0.53%) Baseline: 4.99 µs	5.62 µs (89.39%)
Adapter::Magic (Rust)	📈 view plot 🚷 view threshold	24.19 µs (-9.80%) Baseline: 26.82 µs	31.37 µs (77.12%)
Adapter::Rust	📈 view plot 🚷 view threshold	4.48 µs (-1.95%) Baseline: 4.57 µs	6.05 µs (73.95%)
Adapter::RustBench	📈 view plot 🚷 view threshold	4.46 µs (-2.20%) Baseline: 4.56 µs	6.05 µs (73.70%)

🐰 View full continuous benchmarking report in Bencher