Pause the runner between Jobs while an md array syncs

[epompeii]

What

Make runner up probe its host's md arrays before each ready, and send paused instead while any array syncs, so no Job starts on disks that a resync, recovery, or scrub keeps busy. This is the runner half of paused (#1122).

• The probe runs only at the idle decision point: after any pending result is resent, before each ready, never during a Job. It reads sync_action, sync_completed, and sync_speed for each /sys/block/md* entry with an md/ directory.
  • An array is busy while sync_action is resync, recover, check, repair, or reshape and sync_completed is not none, which covers a running sync (N / M) and one queued behind another array (delayed).
  • idle, frozen, an action this runner does not know, an array with no sync thread such as raid0, and a host with no arrays never pause.
• paused carries one raid reason per busy array, with its action, progress, and speed, plus since and every ready field. since is when the pause began by the runner's clock, and it holds for the whole pause, whatever its reasons become.
• After paused the runner waits for no_job or update exactly as after ready, then probes again. The server holds the poll, so a paused runner stays connected, still updates itself, and takes Jobs again within one poll of the sync ending.
• A Job sent to a paused runner is failed unrun, with The runner was paused for a RAID sync on md0, so it ran no Job, through the usual terminal message, ACK, and retry, and no running.
• Reconnects while paused first hold off one poll timeout, ahead of the usual 5 to 10 s delay, on every path that reconnects. The hold sleeps in slices of at most 1 s, so a stop ends it within a second.
• Logs: one Paused for a RAID sync record per busy array when a pause begins (array, action, percent, speed_kib), Pause ended when it ends, and Holding off the reconnect while paused with hold_secs. A pause whose reasons change mid-pause is not logged again.
• --no-raid-pause on runner up keeps taking Jobs through a sync. The pause is on by default, and raid_pause is in the Runner starting record.
• The runner up reference documents the flag in all 9 languages, and the changelog gains a line.

With #1125 and #1130 on the server, the pause shows on the admin runner endpoints, and a pause past 12 hours mails the server admins.

Across versions:

• A server with Let a runner pause on its channel without taking a job #1122 holds the paused poll and answers no_job on the same channel.
• A server without it cannot parse paused, so it closes the channel. The runner then reconnects no faster than once per poll, so such a server sees no more connects than polls, and no self-update reaches the runner until its pause ends. The first ready after the pause takes Jobs as before.

Why

A sync reads or writes the whole array, and a Job measured during one measures the disks' contention as much as the benchmark. Probing only between Jobs leaves a running Job alone: a sync that starts mid-Job does not stop it, and the next Job waits.

sync_action alone is not enough. It reads recover while a recovery is needed but cannot run, such as on a degraded array with no spare, which would pause the runner until someone adds a disk. A running or queued sync thread is what keeps the disks busy, so sync_completed has to say one exists.

Holding the poll on the server, rather than sleeping on the runner, reuses the timers ready already has, so the runner needs no sleep of its own for the pause.

The reconnect hold is for a server without paused. Without it, a runner whose channel closes on every paused reconnects every 5 to 10 s, about 480 times an hour, and trips a runner connect limit of 256 an hour after about half an hour; connects then fail even after the pause ends, until the hour drains. With the hold, a runner reconnects at most about 3600 / (poll + 7.5) times an hour: about 58 at the default 55 s poll. A poll under about 7 s would still pass 256 an hour against such a server.

The server never claims a Job for a paused runner, so a Job that arrives anyway is failed rather than measured on busy disks.

Verification

• Each new test fails under a mutation of the behavior it guards:
  • a_running_sync_of_each_kind_pauses: a busy set that misses an action, or counts frozen, idle, or an unknown action as busy.
  • a_sync_action_with_no_sync_thread_does_not_pause: busy on sync_action alone (recover with sync_completed none).
  • a_queued_sync_pauses_with_no_progress: delayed not busy.
  • a_pause_reason_carries_the_progress_and_speed: swapped or dropped progress and speed, and an array named after anything but its block device.
  • a_host_with_no_arrays_never_pauses and an_array_with_no_sync_thread_is_listed_but_never_busy: a disk with no md/ read as an array, a host with no block directory failing, and raid0 dropped or busy.
  • a_busy_probe_pauses_and_a_clear_one_sends_ready: the probe's reasons ignored, a paused without ready's fields (the server's update check needs the runner metadata), and a no_job that does not probe again.
  • a_pause_keeps_its_start_and_is_logged_once_each_way: a since that restarts each probe or outlives the pause, and a start or end logged every poll.
  • a_paused_runner_still_updates and a_job_after_a_pause_fails_without_running: a pause that ignores update, and a paused runner that runs a Job.
  • reconnects_hold_off_a_poll_only_while_paused, at a 55 s poll, drives every path that can reconnect while paused: the channel closed, a refused connect, a receive timeout, a failed self-update, a Job failed unrun whose result is lost with the connection, and each of the 3 ways its resend can fail at the next connect. Removing the hold at any one of those 8 sites fails it, and so do a fixed 30 s hold and a hold that outlives the pause. The ninth reconnect site, a connection lost during a Job, cannot be reached while paused.
  • a_paused_reconnect_holds_off_for_the_whole_hold and a_stop_ends_a_hold_within_one_slice run the hold through the driver's effect: a hold that does not sleep, and one a stop cannot end.
  • no_raid_pause_takes_jobs_through_a_sync and the_raid_pause_is_on_unless_turned_off: the flag ignored by the probe, inverted, or off by default.
• None of the new tests compiles on the parent, which has no probe. The 14 runner tests and the CLI test pass 20 of 20 as an ordinary user and as root.
• In a disposable VM, a root runner up at a 5 s poll ran against a local API, with a RAID1 over two 2 GiB loop devices:
  • echo check > sync_action paused the runner within 4 s, with Paused for a RAID sync (md0, check), and the server stored paused with the raid reason, its progress, and its speed.
  • A Job submitted then read pending at each of 12 checks over 60 s.
  • echo idle ended the pause at the next probe, 1 s later, and the Job was claimed that second and processed 2 s after.
  • The whole run used one channel. A second run at a 7 s poll behaved the same.
• Against a server built from the commit before Let a runner pause on its channel without taking a job #1122, with its runner connect limiter on at 16 a minute and 256 an hour, a root runner up at a 20 s poll was held paused by a maintenance marker from the next layer in this stack; the hold does not depend on the reason.
  • Each paused closed the channel, and the runner logged Holding off the reconnect while paused with hold_secs 20.
  • Over 627 s it made 24 connects, 25.0 to 30.0 s apart: at most 3 in any minute, and 132 an hour. No connect drew a 429, while a burst of 20 connects with the same key at the end drew 5.
  • Once the marker was gone, the next connect probed clear and a waiting Job was claimed 9 s after the marker came off.
  • A hold that does not sleep reconnected every 5.0 to 10.0 s against the same server, 483 an hour.
  • SIGTERM 2.5 s into a 20 s hold ended runner up in 0.50 s.
• The runner unit tests pass as an ordinary user and as root, 427 of 427, the CLI tests pass, 14 of 14, the runner_ops tests pass, 86 of 86, cargo test-runner scenarios --build-only passes, Cargo.lock is unchanged, and clippy passes with -D warnings for Linux x86_64, Linux aarch64, and macOS on the runner, its CLI, test_runner, and runner_ops. The scenario suite passes as root, 99 of 99, and clippy passes for the whole workspace as CI runs it, on the top of this stack.

[github-actions]

Bencher Report

Project	Bencher
Branch	u/ep/runner-protocol/raid-pause
Testbed	intel-v1

Click to view all benchmark results

Benchmark	Latency	Benchmark Result microseconds (µs) (Result Δ%)	Upper Boundary microseconds (µs) (Limit %)
Adapter::Json	📈 view plot 🚷 view threshold	5.18 µs (+0.22%) Baseline: 5.17 µs	5.97 µs (86.79%)
Adapter::Magic (JSON)	📈 view plot 🚷 view threshold	5.02 µs (+0.74%) Baseline: 4.99 µs	5.65 µs (88.91%)
Adapter::Magic (Rust)	📈 view plot 🚷 view threshold	24.11 µs (-10.21%) Baseline: 26.85 µs	31.22 µs (77.23%)
Adapter::Rust	📈 view plot 🚷 view threshold	4.48 µs (-1.55%) Baseline: 4.55 µs	6.14 µs (72.98%)
Adapter::RustBench	📈 view plot 🚷 view threshold	4.50 µs (-0.97%) Baseline: 4.55 µs	6.14 µs (73.36%)

🐰 View full continuous benchmarking report in Bencher