Report the host's disk health with Ready and Paused

[epompeii]

What

Make the runner send its host's disk health with every ready and paused, in the health field that #1125 stores, so admins see it on the runner endpoints and #1130 mails them when it is news.

• md, at every probe: each array's array_state, degraded, and sync_action, read with the RAID pause's own read. An array with no sync thread, such as raid0, reads idle. A degraded array is a degraded finding that is failing.
• NVMe, at most hourly: each controller under /sys/class/nvme, with its model, its serial, and the SMART / health log fields that set a finding: critical warning, temperature, spare and its threshold, percentage used, and media errors. The page is read with Get Log Page (log 0x02, all namespaces, 512 bytes) through the kernel's admin passthrough, NVME_IOCTL_ADMIN_CMD, on a read-only open of /dev/<controller>. The read is cached for an hour, and only the probe reads it, so never during a Job.
• States:
  • warning: spare at or below twice its threshold, percentage used over 80, the temperature critical warning bit, or any media and data integrity error;
  • failing: any other critical warning bit, spare below its threshold, or a degraded array;
  • the host's state is its worst finding, and ok with none.
• A runner that is not root cannot use the passthrough, which needs CAP_SYS_ADMIN. It lists each controller with no log, warns once with NVMe health not read, so it is reported unreadable, and still reports md.
• Health is always sent: a host with no arrays and no controllers reports ok with empty lists.
• Nothing pauses on health. A degraded array keeps taking Jobs; only a sync pauses, as before.
• AdminCommand is the kernel's struct nvme_admin_cmd, pinned at 72 bytes by a const assertion. nix gains its ioctl feature, which adds no package to Cargo.lock.
• The changelog gains a line.

Across versions:

• A server without Keep each runner's latest status and show it to admins #1125 ignores health, since the runner messages accept unknown fields.
• paused needs a server with Let a runner pause on its channel without taking a job #1122, as in the layers below.

Why

A degraded array or a failing NVMe drive on a runner host is silent until the next disk fails too. The server half stores health and mails admins on news; this is what sends it.

md costs a few sysfs reads, so it is read at every probe. Each NVMe read is an admin command to the drive, so it runs at most hourly, and only between Jobs.

The temperature bit is a warning, not failing, because it clears by itself. Media errors warn while their count is nonzero, rather than when it rises, which would read ok again at the next hourly read.

Health is reported, never a pause. An idle degraded array barely moves a benchmark, and the mail reaches whoever can replace the disk. Health still rides on paused, because the rebuild after a disk is replaced is a pause, and that is when health matters most.

Verification

• Each new test fails under a mutation of the rule it guards:
  • The NVMe parser runs on fixture pages. a_page_parses_little_endian_at_its_offsets and a_media_error_count_past_u64_saturates: swapped temperature bytes, media errors read big-endian or from the wrong counter, spare read from the threshold byte, and a 128-bit count that wraps.
  • a_healthy_page_has_no_findings and each_critical_warning_bit_but_temperature_is_failing, over bits 0 and 2 to 7: a rule that fires on a healthy page, any one critical bit missed, and the temperature bit failing or ignored.
  • spare_at_twice_its_threshold_warns_and_below_it_fails, percentage_used_over_80_warns, and any_media_error_warns: either spare boundary or the endurance boundary moved by one, and media errors ignored or failing.
  • an_unreadable_log_still_lists_the_controller: an unreadable controller dropped, and a log read from another device.
  • a_degraded_array_is_failing and the_worst_finding_sets_the_state: a degraded array that reads healthy, every array failing, and the state taken from the first finding rather than the worst.
  • nvme_is_read_at_most_hourly, on an injected clock and reader: read once at the start, served from the cache 1 s before the hour, and read again at the hour. Kills a read every probe, and one never repeated.
  • an_unreadable_nvme_still_reports_md and every_probe_reports_md_health: a runner that is not root reporting nothing, and a probe that sends no health, or none with the RAID pause off.
  • the_probes_health_rides_on_ready_and_paused: health dropped from either message.
• None of the new tests compiles on the parent, which reads no health. The 14 tests pass 20 of 20 as an ordinary user and as root.
• A C probe against Ubuntu 24.04's linux/nvme_ioctl.h gives sizeof(struct nvme_admin_cmd) 72, NVME_IOCTL_ADMIN_CMD 0xc0484e41, and addr, data_len, and cdw10 at offsets 24, 36, and 40, which the Rust struct and ioctl_readwrite!(b'N', 0x41) match.
• In a disposable VM, an NVMe controller on the kernel's nvme-loop target, backed by a file, stood in for a drive:
  • As root, the runner's own read returned the page with status 0, and the page's data units read and host read commands equal nvme smart-log's values for that controller.
  • As an ordinary user, the read fails with EACCES, the controller is listed with no log, and md is still reported.
  • A dword count off by one fails there with status 0x600f, which the fixture pages cannot show.
• In the same kind of VM, a root runner up ran against a local API, with a RAID1 over two loop devices:
  • Healthy, the server stored health ok with md0 clean, 0 degraded, idle.
  • mdadm --fail on one member: the next report read failing, with md0 degraded failing, and the runner still ready. A Job submitted then ran.
  • mdadm --remove and --add: the runner paused for the recover sync with health still failing. Once the recovery finished, the pause ended, and the next report read ready and ok.
• The runner unit tests pass as an ordinary user and as root, 455 of 455, the CLI tests pass, 14 of 14, the runner_ops tests pass, 86 of 86, cargo test-runner scenarios --build-only passes, Cargo.lock is unchanged, and clippy passes with -D warnings for Linux x86_64, Linux aarch64, and macOS on the runner, its CLI, test_runner, and runner_ops. The scenario suite passes as root, 100 of 100, and clippy passes for the whole workspace as CI runs it, on the top of this stack.

[github-actions]

Bencher Report

Project	Bencher
Branch	u/ep/runner-protocol/health-report
Testbed	intel-v1

Click to view all benchmark results

Benchmark	Latency	Benchmark Result microseconds (µs) (Result Δ%)	Upper Boundary microseconds (µs) (Limit %)
Adapter::Json	📈 view plot 🚷 view threshold	5.18 µs (-0.10%) Baseline: 5.19 µs	5.88 µs (88.13%)
Adapter::Magic (JSON)	📈 view plot 🚷 view threshold	5.02 µs (+0.42%) Baseline: 5.00 µs	5.58 µs (90.03%)
Adapter::Magic (Rust)	📈 view plot 🚷 view threshold	24.54 µs (-8.41%) Baseline: 26.80 µs	31.51 µs (77.89%)
Adapter::Rust	📈 view plot 🚷 view threshold	4.49 µs (-2.09%) Baseline: 4.58 µs	5.96 µs (75.26%)
Adapter::RustBench	📈 view plot 🚷 view threshold	4.46 µs (-2.66%) Baseline: 4.58 µs	5.96 µs (74.76%)

🐰 View full continuous benchmarking report in Bencher