diff --git a/docs/diagrams/native-bridge-startup.mmd b/docs/diagrams/native-bridge-startup.mmd new file mode 100644 index 000000000..7cde21361 --- /dev/null +++ b/docs/diagrams/native-bridge-startup.mmd @@ -0,0 +1,23 @@ +--- +title: Native bridge startup +--- +flowchart TD + ready[Guest ready] --> check[Verify provider] + check --> inspect[Inspect all sockets] + check -->|identity changed| fail[Refuse startup] + inspect -->|unsafe identity| fail + inspect -->|ready or pending| recheck[Recheck provider] + recheck -->|identity changed| fail + recheck --> deadline{Before 10 seconds?} + deadline -->|no| fail + deadline -->|yes| complete{All sockets private?} + complete -->|no| wait[Wait up to 50ms] + wait --> check + complete -->|yes| audit[Full audit then adoption] + fail --> stop[Verified provider stop] + stop --> retained[Retain failure evidence] + class ready,check,inspect,recheck live + class wait wait + class audit ok + class fail fail + class retained idle diff --git a/docs/diagrams/native-bridge-startup.svg b/docs/diagrams/native-bridge-startup.svg new file mode 100644 index 000000000..9977810fd --- /dev/null +++ b/docs/diagrams/native-bridge-startup.svg @@ -0,0 +1,66 @@ +Native bridge startupidentity changedunsafe identityready or pendingidentity changednoyesnoyesGuest readyVerify providerInspect all socketsRefuse startupRecheck providerBefore 10 seconds?All sockets private?Wait up to 50msFull audit then adoptionVerified provider stopRetain failure evidence \ No newline at end of file diff --git a/docs/guides/native-candidate.md b/docs/guides/native-candidate.md index 8e1659535..028d26d10 100644 --- a/docs/guides/native-candidate.md +++ b/docs/guides/native-candidate.md @@ -699,6 +699,31 @@ within the macOS length limit. A clean `runtime down` keeps this alias, so no ot local user can take its name while the pool is stopped. It removes only the provider socket files the stop proved closed, through the real data directory. +On a fresh pool boot or a boot after stopping the pool, native waits up to 10 seconds +for every declared bridge socket to appear with the required private ownership and mode. +The provider's guest-ready signal can arrive before its bridge threads bind. Native +rechecks the same provider identity around each observation; it never accepts an unsafe +socket, changes its permissions or removes it to make startup pass. Existing running-pool +audits do not wait. A socket timeout or identity refusal takes the verified failed-boot +stop path and names the recovery command. Use the same selected executor and identical +`--candidate-root` argument as the failed start, followed by `runtime recover --json`. +If you explicitly selected an installed executor with `HACK_NATIVE_BINARY` and +`HACK_NATIVE_HOME`, use those same non-empty values: + +```sh +"$HACK_NATIVE_BINARY" --candidate-root "$HACK_NATIVE_HOME" runtime recover --json +``` + +Otherwise use the bundled executor path and the selected home from the failed +invocation; do not run this variable-based example with either variable unset. + +For a checkout-local executor, `--candidate-root` is the source checkout, not the native +home or `.hack-local` directory; from that checkout run `./hack-local runtime recover --json`. +Provider-verification failures during the wait carry this guidance too; recovery still +independently proves ownership before stopping anything. Then retry. This preserves +retained disks and records an unclean stop; it does not claim the guest shut down cleanly. +`hack down` may say the project was not started because no project mapping was published. + Host cleanup or a reboot can still remove the alias. For a clean stop, observation commands such as `runtime status` accept the absent alias and create nothing, and the next `runtime up`, `runtime recover` or approved-host update restores the exact alias diff --git a/docs/reference/v5-model-coverage.md b/docs/reference/v5-model-coverage.md index f4c4492ce..aeaa28ce1 100644 --- a/docs/reference/v5-model-coverage.md +++ b/docs/reference/v5-model-coverage.md @@ -36,6 +36,7 @@ must be attached to the eventual exact integration revision. | Property and critical states | Model coverage | Implementation/regression boundary | Remaining qualification | | --- | --- | --- | --- | +| Guest control ready before exposed bridge sockets bind; identity changes or the startup deadline expires | No new TLA+ model; deterministic clock schedules both readiness orders against real Unix socket nodes | `provider/bridge.rs` waits only in fresh `finish_boot`, before adoption, and keeps running audits immediate. `bridge_startup_tests.rs` covers two slots, late creation and mode tightening, permanent absence/unsafe modes, type/UID/link refusals, provider loss before/after observation and late success rejection. No wait mutates sockets or grants cleanup authority; the existing verified-provider stop handles failure. `lifecycle/boot_sequence_tests.rs` drives the real waiter through the production fresh-boot sequence and rejects audit/adoption on timeout, unsafe sockets or provider loss. `native-runtime-client.test.ts` checks the fixed runtime recover command and identical-root hint reach the frontend | Two sockets, finite scheduled events, injected provider verification and monotonic clock. Kernel process identity, provider scheduling, disk durability and successful recovery remain host boundaries. Require ten consecutive real-VM Mac Ctrl-C scenarios with pool stop/start; portable tests are not that evidence. | | CI cancellation must not lose merged-change coverage; only exact-head full evidence may qualify a prerelease | No maintained GitHub scheduler model. Finite fixture interleavings cover predecessor success/failure/cancellation, two competing full/scoped runs and two rerun attempts | `ci.yml` serializes next pushes in one non-cancelling group (GitHub cancels only the replaced pending run), cancels stale PR runs, and requires a successful before run for selection. `selectPrereleaseCiRun` reads current attempts; `requireBranchChecks` binds suite and job check IDs. `ci-trusted-policy`, `prerelease-plan` and `release-prepare-gate` regressions reject cancelled predecessors, green subsets, stale checks, skipped hosts and changed attempts | API snapshots and GitHub group/event semantics are assumptions; races after the last read remain possible. Publication rechecks the frozen protected head and full evidence but does not transactionally lock GitHub. Hosted dispatch/cancellation checks qualify wiring; no scheduler liveness or formal refinement claim. | | Native log follower versus lifecycle mutation, frontend loss, stopped container and transport loss | No new state model: the follower observes without a mutation lease and grants no cleanup authority | `provider/graph/log_follow.rs` and its native counterpart pin Ready admission, container and start generation; `provider/engine/log_follow.rs` bounds framing and distinguishes real EOF from stopped-service idle cutoff. `lifecycle_log_follow_tests.rs` keeps the production transport on a fake Engine socket active while acquiring startup and operation leases, with a same-lock contention control and output while each lease is held. `native-log-follow-source.test.ts` guards the observer-only route; process tests cover cancellation, frontend loss, real output-pipe closure while idle, replacement-container restart (including retained adopted run/owner IDs) with one marker, NDJSON admission failures, truncated/partial output and slow idle-end delivery. `native-log-restart.test.ts` covers bounded re-admission, unchanged-container refusal and prompt down; `native-log-selection.test.ts` covers strict finite admission responses and owned-child cancellation/timeouts; advisory markers grant no authority | 4 KiB payload envelopes, 8 KiB wire lines, 64 Ki-character retained payload prefixes excluding Engine timestamps, 30-second restart wait, 1000-line new-container backlog. The portable lease test fakes identity/state checks; it does not exercise host process/disk auditing, graph admission or the VM relay. Real relay EOF on stop/down/restart, VM loss and concurrent lifecycle latency require Apple Silicon acceptance. No performance or whole-product parity claim. | | Foreground group interrupt; publisher exit before authenticated stop; dead owner's pending cleanup enrollment and bridge release | native-publisher-stop | [Process adapter](../../src/backends/native-project-process.ts) isolates Compose owners regardless of hooks; Rust `process::publisher_command` isolates publishers; `legacy_cleanup_enrollment` binds the old coordinator and inventories before a distinct recovery intent. [Group regression](../../tests/native-foreground-group-interrupt.test.ts) signals the whole group for fresh/restore owners; Rust real-publisher process, frontend-loss, exact child-reaping and enrollment tests check grouping, lifetime wakeups and stale selectors. | One graph, one publisher, one crash, one identity replacement and one audited successor boot; 168 states. Publisher exit observation is separate from lease-bound reaping/proof commitment; `NoUncommittedExit` rejects owner cleanup while proof is pending. A busy lease stutters without delaying stop control. SIGHUP requests retaining cleanup; a reaped publisher enters an observable publisher-down state until explicit stop. Native process tests and frontend framing/ps tests cover signal delivery and notices. Dependency release is a separate post-completion step; `NoStrandedReservation` rejects success with a held reservation. Real reservation-file regressions check retry, capacity reuse and refusal fences. Separate stopped-receipt and intent-completion commits are collapsed; TypeScript dispatch and Rust completion-digest regressions cover that interruption window. Publication-registry sync/rename is abstracted and covered by portable commit-interruption tests. Kernel identity, socket/inode proofs, guest observations, durable storage and existing next.13 home recovery remain implementation/Apple Silicon VM gates. No eventual recovery is asserted. | diff --git a/packages/runtime-core/README.md b/packages/runtime-core/README.md index 69385d9e9..0365d473c 100644 --- a/packages/runtime-core/README.md +++ b/packages/runtime-core/README.md @@ -581,6 +581,34 @@ digest is. A mismatch refuses with `disk_template_untrusted` and changes no file removing the refused file lets SmolVM re-expand it from the verified compressed template. After a start, the boot is stopped as a failed boot without adopting its disks and needs manual recovery. +Guest control readiness does not mean exposed bridge sockets have bound. The pinned +[SmolVM 1.34.1 guest](https://github.com/smol-machines/smolvm/blob/v1.34.1/crates/smolvm-agent/src/main.rs#L642) +starts [independent expose threads](https://github.com/smol-machines/smolvm/blob/v1.34.1/crates/smolvm-agent/src/publish_socket.rs#L57) +before signalling ready, without joining them. The pinned +[launcher removes old expose paths first](https://github.com/smol-machines/smolvm/blob/v1.34.1/src/agent/launcher.rs#L1955); +freshness depends on that ordering, because private socket metadata alone does not identify +its binder. A provider-pin upgrade must re-check that contract. Fresh boot waits up to 10 seconds +for all declared bridge sockets, before disk adoption and the full boot audit. Each poll +checks the retained provider identity before and after the socket observation. Missing +nodes and same-user, single-link sockets awaiting private mode can converge; a different +type, user, link count or unreadable identity refuses immediately. Only a complete private +socket set observed before the deadline succeeds. The wait never chmods, removes or connects +to sockets. Running-pool audits remain immediate. The existing failed-boot path stops only +the verified provider, records its absence and retains `failed-boot-stopped`, without +claiming a clean guest shutdown. Socket failures and provider-verification failures during +the wait name `runtime recover --json`, using the same executor and identical +`--candidate-root` argument as the failed start. For an installed executor this argument is +the private `HACK_NATIVE_HOME`; for the checkout-local executor it is the source checkout +(the `hack-local` launcher supplies it). Do not substitute the `.hack-local` state directory. +A project may not yet +have a mapping for ordinary `hack down`. The clock-driven real-socket regression reproduces +the ready-before-bind ordering without a VM. A fresh-boot sequence regression runs the +actual waiter before audit/adoption and refuses timeout, unsafe sockets and provider loss +without adopting disks; host effects are injected and actual provider scheduling is not qualified. +HACK-2175's live gate is ten consecutive Mac Ctrl-C scenarios including pool stop/start. +The wait adds no signal handler: interruption retains the existing `booting` receipt +and stop/recovery rules, so that live gate also qualifies the extended startup window. + A start interrupted before the pool records its provider process (for example, `up` is killed) leaves the pool `creating` or `booting`, and `up` refuses. Each start clears the previous provider's already-dead identity before `booting`, so this covers every start, not only the diff --git a/packages/runtime-core/src/project/route_probe_clock_tests.rs b/packages/runtime-core/src/project/route_probe_clock_tests.rs index a6d1a5c59..6e8c698ee 100644 --- a/packages/runtime-core/src/project/route_probe_clock_tests.rs +++ b/packages/runtime-core/src/project/route_probe_clock_tests.rs @@ -56,59 +56,51 @@ fn a_route_first_answering_at_120_seconds_starts_under_the_graph_budget() { }, )])) .unwrap(); - // The old grace is a negative control: the same delayed success must fail - // early with it. The reviewed default must reach graph readiness at 120 s. - for grace in [60_000, HttpProbe::inferred_route(3000).start_period_ms] { - let mut config = HttpProbe::inferred_route(3000); - config.start_period_ms = grace; - let output = Command::new(concat!(env!("OUT_DIR"), "/http-probe-clock-host")) - .args([ - grace.to_string(), - config.interval_ms.to_string(), - config.timeout_ms.to_string(), - config.retries.to_string(), - "120000".into(), - ]) - .output() - .unwrap(); - assert!(output.status.success()); - assert!(output.stderr.is_empty()); - let samples = String::from_utf8(output.stdout) - .unwrap() - .lines() - .map(|line| { - let fields: Vec = line - .split(' ') - .map(|field| field.parse().unwrap()) - .collect(); - (fields[0], fields[1], fields[2]) - }) - .collect(); - let clock = Cell::new(Duration::ZERO); - let base = Instant::now(); - let mut driver = Probed { - clock: &clock, - config, - samples, - started: false, - ready: false, - }; - let result = run_with_clock( - &graph, - &mut driver, - Duration::from_secs(300), - |delay| clock.set(clock.get() + delay), - || base + clock.get(), - ); - assert!(driver.started); - if grace == 60_000 { - assert_eq!(result.unwrap_err().code, "graph_service_failed"); - assert!(!driver.ready); - assert!(clock.get() < Duration::from_secs(120)); - } else { - result.unwrap(); - assert!(driver.ready); - assert_eq!(clock.get(), Duration::from_secs(120)); - } - } + // Use the production default unchanged, and always require readiness. + // Restoring its grace to 60 s must fail this test, not select a passing + // branch that expects the service to become unhealthy. + let config = HttpProbe::inferred_route(3000); + let output = Command::new(concat!(env!("OUT_DIR"), "/http-probe-clock-host")) + .args([ + config.start_period_ms.to_string(), + config.interval_ms.to_string(), + config.timeout_ms.to_string(), + config.retries.to_string(), + "120000".into(), + ]) + .output() + .unwrap(); + assert!(output.status.success()); + assert!(output.stderr.is_empty()); + let samples = String::from_utf8(output.stdout) + .unwrap() + .lines() + .map(|line| { + let fields: Vec = line + .split(' ') + .map(|field| field.parse().unwrap()) + .collect(); + (fields[0], fields[1], fields[2]) + }) + .collect(); + let clock = Cell::new(Duration::ZERO); + let base = Instant::now(); + let mut driver = Probed { + clock: &clock, + config, + samples, + started: false, + ready: false, + }; + let result = run_with_clock( + &graph, + &mut driver, + Duration::from_secs(300), + |delay| clock.set(clock.get() + delay), + || base + clock.get(), + ); + assert!(driver.started); + result.expect("the production inferred probe must allow readiness at 120 s"); + assert!(driver.ready); + assert_eq!(clock.get(), Duration::from_secs(120)); } diff --git a/packages/runtime-core/src/provider/artifact.rs b/packages/runtime-core/src/provider/artifact.rs index 440c30874..df516cf83 100644 --- a/packages/runtime-core/src/provider/artifact.rs +++ b/packages/runtime-core/src/provider/artifact.rs @@ -13,6 +13,9 @@ use std::path::{Path, PathBuf}; use std::time::Duration; pub const ARCHIVE_SHA256: &str = "8ebda502e332c739fdfc13b059be982cca66da9253c5229b4586af02c92781d9"; +// Bridge freshness relies on the pinned launcher's unlink-before-guest-ready +// ordering for expose paths (src/agent/launcher.rs:1955-1972 at v1.34.1). +// A pin upgrade must re-check that contract as well as the archive digest. pub const VERSION: &str = "1.34.1"; const DIRECTORY: &str = "smolvm-1.34.1-darwin-arm64"; diff --git a/packages/runtime-core/src/provider/bridge.rs b/packages/runtime-core/src/provider/bridge.rs index 4eb07b0dc..11a083ca1 100644 --- a/packages/runtime-core/src/provider/bridge.rs +++ b/packages/runtime-core/src/provider/bridge.rs @@ -91,33 +91,123 @@ pub(super) fn verify_sockets( directory: &std::path::Path, intent: BridgeIntent, ) -> Result<(), CandidateError> { + if sockets_ready(directory, intent)? { + Ok(()) + } else { + Err(socket_identity()) + } +} + +fn socket_identity() -> CandidateError { + CandidateError::new( + "bridge_socket_identity", + "Expected a private owned bridge socket.", + ) +} + +fn private_socket(metadata: &std::fs::Metadata, uid: u32) -> Result { use std::os::unix::fs::{FileTypeExt, MetadataExt}; + if !metadata.file_type().is_socket() || metadata.uid() != uid || metadata.nlink() != 1 { + return Err(socket_identity()); + } + Ok(metadata.mode() & 0o077 == 0) +} + +/// Missing nodes and not-yet-private owned sockets may converge during boot. +/// A different type, owner, link count or unreadable identity is never pending. +fn sockets_ready( + directory: &std::path::Path, + intent: BridgeIntent, +) -> Result { + let mut ready = true; for path in intent.guest_paths() { let path = directory.join( std::path::Path::new(&path) .file_name() .expect("fixed socket name"), ); - let metadata = std::fs::symlink_metadata(path).map_err(|_| { + let metadata = match std::fs::symlink_metadata(path) { + Ok(metadata) => metadata, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + ready = false; + continue; + } + Err(_) => return Err(socket_identity()), + }; + // SAFETY: geteuid has no preconditions and cannot fail. + ready &= private_socket(&metadata, unsafe { libc::geteuid() })?; + } + Ok(ready) +} + +const BOOT_SOCKET_WAIT: std::time::Duration = std::time::Duration::from_secs(10); +const BOOT_SOCKET_POLL: std::time::Duration = std::time::Duration::from_millis(50); + +fn startup_socket_timeout() -> CandidateError { + CandidateError::new( + "bridge_socket_timeout", + "Native bridge sockets did not become private and owned before the startup deadline. Run the same selected executor with the identical --candidate-root argument used for the failed start, followed by runtime recover --json before retrying.", + ) +} + +/// Only fresh boot waits: SmolVM's guest-ready signal does not join its expose +/// threads. Caller holds the pool lease and rechecks its retained provider on +/// both sides of each complete socket observation. No socket is changed here. +/// Socket freshness also relies on the pinned launcher's removal of old expose +/// paths before guest-ready; see the upgrade contract at `artifact::VERSION`. +pub(super) fn wait_for_sockets( + directory: &std::path::Path, + intent: BridgeIntent, + verify_provider: impl FnMut() -> Result<(), CandidateError>, + clock: &impl super::process::Clock, +) -> Result<(), CandidateError> { + wait_for_sockets_with(directory, intent, verify_provider, clock, BOOT_SOCKET_WAIT) +} + +fn wait_for_sockets_with( + directory: &std::path::Path, + intent: BridgeIntent, + mut verify_provider: impl FnMut() -> Result<(), CandidateError>, + clock: &impl super::process::Clock, + budget: std::time::Duration, +) -> Result<(), CandidateError> { + let deadline = clock.now().saturating_add(budget); + loop { + if clock.now() >= deadline { + return Err(startup_socket_timeout()); + } + verify_provider().map_err(startup_provider_failure)?; + let ready = sockets_ready(directory, intent).map_err(|_| { CandidateError::new( "bridge_socket_identity", - "Expected a private owned bridge socket.", + "Bridge socket identity is unsafe; startup was refused. Run the same selected executor with the identical --candidate-root argument used for the failed start, followed by runtime recover --json before retrying.", ) })?; - if !metadata.file_type().is_socket() - || metadata.uid() != unsafe { libc::geteuid() } - || metadata.mode() & 0o077 != 0 - || metadata.nlink() != 1 - { - return Err(CandidateError::new( - "bridge_socket_identity", - "Expected a private owned bridge socket.", - )); + verify_provider().map_err(startup_provider_failure)?; + let remaining = deadline.saturating_sub(clock.now()); + if remaining.is_zero() { + return Err(startup_socket_timeout()); } + if ready { + return Ok(()); + } + clock.sleep(remaining.min(BOOT_SOCKET_POLL)); } - Ok(()) } +/// Keep the failed identity boundary's code, but never expose its dynamic text. +/// Recovery still independently proves the recorded provider before any stop. +fn startup_provider_failure(failure: CandidateError) -> CandidateError { + CandidateError::new( + failure.code, + "The retained provider could not be verified while waiting for bridge sockets; startup was refused. Run the same selected executor with the identical --candidate-root argument used for the failed start, followed by runtime recover --json before retrying.", + ) +} + +#[cfg(test)] +#[path = "bridge_startup_tests.rs"] +mod startup_tests; + #[cfg(test)] mod tests { use super::*; diff --git a/packages/runtime-core/src/provider/bridge_startup_tests.rs b/packages/runtime-core/src/provider/bridge_startup_tests.rs new file mode 100644 index 000000000..3ef108abb --- /dev/null +++ b/packages/runtime-core/src/provider/bridge_startup_tests.rs @@ -0,0 +1,257 @@ +//! Guest/control readiness and exposed-socket binding are independent events. +//! A fake monotonic clock schedules real socket nodes in either order, without a VM. +use super::*; +use crate::provider::process::Clock; +use std::{ + cell::{Cell, RefCell}, + fs, + os::unix::{ + fs::{MetadataExt, PermissionsExt, symlink}, + net::UnixListener, + }, + path::PathBuf, + time::Duration, +}; + +struct Fixture { + root: PathBuf, + listeners: RefCell>, +} +impl Fixture { + fn new() -> Self { + Self { + root: crate::unique_test_root(&std::env::temp_dir(), "bridge-wait"), + listeners: RefCell::new(Vec::new()), + } + } + fn path(&self, slot: u8) -> PathBuf { + self.root.join(format!("bridge-{slot:02}.sock")) + } + fn bind(&self, slot: u8, mode: u32) { + self.listeners + .borrow_mut() + .push(UnixListener::bind(self.path(slot)).unwrap()); + self.mode(slot, mode); + } + fn mode(&self, slot: u8, mode: u32) { + fs::set_permissions(self.path(slot), fs::Permissions::from_mode(mode)).unwrap(); + } +} +impl Drop for Fixture { + fn drop(&mut self) { + self.listeners.get_mut().clear(); + fs::remove_dir_all(&self.root).unwrap(); + } +} + +struct Scheduled { + elapsed: Cell, + step: RefCell, +} +impl Scheduled { + fn new(step: F) -> Self { + Self { + elapsed: Cell::new(Duration::ZERO), + step: RefCell::new(step), + } + } +} +impl Clock for Scheduled { + fn now(&self) -> Duration { + self.elapsed.get() + } + fn sleep(&self, duration: Duration) { + self.elapsed.set(self.elapsed.get() + duration); + (self.step.borrow_mut())(self.elapsed.get()); + } +} +const BUDGET: Duration = Duration::from_millis(200); + +#[test] +fn guest_ready_before_all_bridge_binds_waits_for_complete_private_identity() { + let fixture = Fixture::new(); + let intent = BridgeIntent::new(2).unwrap(); + // The one-shot audit reproduces the old ordering failure: the provider + // control channel is ready while its expose threads have not yet run. + assert_eq!( + verify_sockets(&fixture.root, intent).unwrap_err().code, + "bridge_socket_identity" + ); + let clock = Scheduled::new(|elapsed: Duration| match elapsed.as_millis() { + 50 => fixture.bind(0, 0o700), + 100 => fixture.bind(1, 0o777), + 150 => fixture.mode(1, 0o700), + _ => panic!("unexpected wait"), + }); + wait_for_sockets_with(&fixture.root, intent, || Ok(()), &clock, BUDGET).unwrap(); + assert_eq!(clock.now(), Duration::from_millis(150)); + verify_sockets(&fixture.root, intent).unwrap(); +} + +#[test] +fn private_sockets_ready_first_need_no_wait_but_still_verify_the_provider_twice() { + let fixture = Fixture::new(); + fixture.bind(0, 0o600); + let clock = Scheduled::new(|_: Duration| panic!("already ready")); + let checks = Cell::new(0); + wait_for_sockets_with( + &fixture.root, + BridgeIntent::new(1).unwrap(), + || { + checks.set(checks.get() + 1); + Ok(()) + }, + &clock, + BUDGET, + ) + .unwrap(); + assert_eq!(checks.get(), 2); + assert_eq!(clock.now(), Duration::ZERO); +} + +#[test] +fn missing_or_never_private_sockets_expire_without_changing_them_and_name_recovery() { + for present in [false, true] { + let fixture = Fixture::new(); + if present { + fixture.bind(0, 0o777); + } + let clock = Scheduled::new(|_: Duration| {}); + let failure = wait_for_sockets_with( + &fixture.root, + BridgeIntent::new(1).unwrap(), + || Ok(()), + &clock, + BUDGET, + ) + .unwrap_err(); + assert_eq!(failure.code, "bridge_socket_timeout"); + assert!(failure.message_fixed); + assert!(failure.message.contains("runtime recover --json")); + assert!( + failure + .message + .contains("identical --candidate-root argument") + ); + assert_eq!(clock.now(), BUDGET); + if present { + assert_eq!( + fs::symlink_metadata(fixture.path(0)).unwrap().mode() & 0o777, + 0o777 + ); + } else { + assert!(!fixture.path(0).exists()); + } + } +} + +#[test] +fn unsafe_type_alias_and_link_count_refuse_even_when_an_earlier_slot_is_missing() { + for kind in ["file", "symlink", "hardlink"] { + let fixture = Fixture::new(); + match kind { + "file" => fs::write(fixture.path(1), b"foreign").unwrap(), + "symlink" => symlink(fixture.root.join("outside"), fixture.path(1)).unwrap(), + "hardlink" => { + fixture.bind(1, 0o700); + fs::hard_link(fixture.path(1), fixture.root.join("another-link")).unwrap(); + } + _ => unreachable!(), + } + let before = fs::symlink_metadata(fixture.path(1)).unwrap(); + let clock = Scheduled::new(|_: Duration| panic!("unsafe identity must not be retried")); + let failure = wait_for_sockets_with( + &fixture.root, + BridgeIntent::new(2).unwrap(), + || Ok(()), + &clock, + BUDGET, + ) + .unwrap_err(); + assert_eq!(failure.code, "bridge_socket_identity"); + assert!(failure.message_fixed); + assert!(failure.message.contains("runtime recover --json")); + let after = fs::symlink_metadata(fixture.path(1)).unwrap(); + assert_eq!( + (before.dev(), before.ino(), before.mode(), before.nlink()), + (after.dev(), after.ino(), after.mode(), after.nlink()) + ); + assert_eq!(clock.now(), Duration::ZERO); + } +} + +#[test] +fn another_owner_is_not_pending_or_private() { + let fixture = Fixture::new(); + fixture.bind(0, 0o700); + let metadata = fs::symlink_metadata(fixture.path(0)).unwrap(); + assert!(private_socket(&metadata, metadata.uid()).unwrap()); + assert_eq!( + private_socket(&metadata, metadata.uid().wrapping_add(1)) + .unwrap_err() + .code, + "bridge_socket_identity" + ); +} + +#[test] +fn provider_loss_before_or_after_observation_is_not_readiness_or_retry_authority() { + for fail_on in [1, 2] { + let fixture = Fixture::new(); + fixture.bind(0, 0o700); + let checks = Cell::new(0); + let clock = Scheduled::new(|_: Duration| panic!("provider loss must not be retried")); + let failure = wait_for_sockets_with( + &fixture.root, + BridgeIntent::new(1).unwrap(), + || { + checks.set(checks.get() + 1); + if checks.get() == fail_on { + Err(CandidateError::new( + "process_identity_mismatch", + "Provider changed.", + )) + } else { + Ok(()) + } + }, + &clock, + BUDGET, + ) + .unwrap_err(); + assert_eq!(failure.code, "process_identity_mismatch"); + assert!(failure.message_fixed); + assert!(failure.message.contains("runtime recover --json")); + assert!( + failure + .message + .contains("identical --candidate-root argument") + ); + assert!(!failure.message.contains("Provider changed.")); + assert_eq!(checks.get(), fail_on); + assert_eq!(clock.now(), Duration::ZERO); + } +} + +#[test] +fn readiness_observed_after_the_deadline_never_counts_as_success() { + let fixture = Fixture::new(); + fixture.bind(0, 0o700); + let clock = Scheduled::new(|_: Duration| panic!("no sleep needed")); + let checks = Cell::new(0); + let failure = wait_for_sockets_with( + &fixture.root, + BridgeIntent::new(1).unwrap(), + || { + checks.set(checks.get() + 1); + if checks.get() == 2 { + clock.elapsed.set(BUDGET); + } + Ok(()) + }, + &clock, + BUDGET, + ) + .unwrap_err(); + assert_eq!(failure.code, "bridge_socket_timeout"); +} diff --git a/packages/runtime-core/src/provider/lifecycle.rs b/packages/runtime-core/src/provider/lifecycle.rs index 073953d10..88bfdf906 100644 --- a/packages/runtime-core/src/provider/lifecycle.rs +++ b/packages/runtime-core/src/provider/lifecycle.rs @@ -1604,7 +1604,7 @@ fn start_pool( if cleanup.is_ok() { phase(candidate, &mut owner, "failed-boot-stopped")?; } - cleanup?; + cleanup.map_err(failed_boot_stop_error)?; } result } @@ -1694,20 +1694,81 @@ fn verify_disk_allocation( Ok(()) } +#[cfg(test)] +#[path = "lifecycle/boot_sequence_tests.rs"] +mod boot_sequence_tests; + +fn failed_boot_stop_error(failure: CandidateError) -> CandidateError { + CandidateError::new( + failure.code, + "Failed-boot provider cleanup could not be confirmed. Run the same selected executor with the identical --candidate-root argument used for the failed start, followed by runtime recover --json before retrying.", + ) +} + fn finish_boot( candidate: &Candidate, owner: &mut Owner, previous_boot: Option<&str>, lock: &state::Lock, ) -> Result { - // Before any disk is adopted: a refusal here takes the caller's failed-boot path. - verify_templates( - candidate, + finish_boot_with( owner, - lock, - super::disk_template::Stage::AfterFirstStart, - )?; - audit_boot(candidate, owner)?; + |owner| { + verify_templates( + candidate, + owner, + lock, + super::disk_template::Stage::AfterFirstStart, + )?; + owner + .application_bridge + .map(|bridge| { + owner + .real_data_dir(candidate) + .map(|directory| (directory, bridge)) + }) + .transpose() + }, + |owner| verify_live(candidate, owner), + |owner| audit_boot(candidate, owner), + |owner| adopt_boot(candidate, owner, previous_boot, lock), + &process::MonotonicClock::new(), + ) +} + +/// Fresh boot's ordering boundary. Only effectful host/guest operations are +/// injected: the real complete-set socket wait always precedes audit/adoption. +fn finish_boot_with( + owner: &mut S, + prepare: impl FnOnce( + &S, + ) -> Result< + Option<(std::path::PathBuf, super::bridge::BridgeIntent)>, + CandidateError, + >, + mut verify_provider: impl FnMut(&S) -> Result<(), CandidateError>, + audit: impl FnOnce(&S) -> Result<(), CandidateError>, + adopt: impl FnOnce(&mut S) -> Result, + clock: &impl process::Clock, +) -> Result { + // Before any disk is adopted: a refusal here takes the caller's failed-boot path. + let bridge = prepare(owner)?; + // The provider's agent-ready signal can precede expose-thread socket binds. + // Wait before adoption, without weakening the immediate audit used by + // running-pool commands. Failure keeps the existing verified-stop path. + if let Some((directory, bridge)) = bridge { + super::bridge::wait_for_sockets(&directory, bridge, || verify_provider(owner), clock)?; + } + audit(owner)?; + adopt(owner) +} + +fn adopt_boot( + candidate: &Candidate, + owner: &mut Owner, + previous_boot: Option<&str>, + lock: &state::Lock, +) -> Result { let dir = owner.real_data_dir(candidate)?; let storage = identity::disk(&dir.join("storage.raw"))?; let overlay = identity::disk(&dir.join("overlay.raw"))?; diff --git a/packages/runtime-core/src/provider/lifecycle/boot_sequence_tests.rs b/packages/runtime-core/src/provider/lifecycle/boot_sequence_tests.rs new file mode 100644 index 000000000..fb5ab1c79 --- /dev/null +++ b/packages/runtime-core/src/provider/lifecycle/boot_sequence_tests.rs @@ -0,0 +1,146 @@ +//! The production fresh-boot sequence drives the real socket waiter before +//! audit/adoption. Host template/process/disk effects are injected, not qualified. +use super::*; +use std::{ + cell::{Cell, RefCell}, + os::unix::{fs::PermissionsExt, net::UnixListener}, +}; + +struct Clock<'a> { + elapsed: Cell, + on_sleep: RefCell>, +} +impl process::Clock for Clock<'_> { + fn now(&self) -> Duration { + self.elapsed.get() + } + fn sleep(&self, duration: Duration) { + self.elapsed.set(self.elapsed.get() + duration); + (self.on_sleep.borrow_mut())(); + } +} + +#[test] +fn fresh_boot_waits_before_audit_and_never_adopts_failed_socket_readiness() { + for mode in [ + "delayed", + "missing", + "unsafe", + "provider-before", + "provider-after", + ] { + let directory = crate::unique_test_root(&std::env::temp_dir(), "boot-bridge"); + let path = directory.join("bridge-00.sock"); + if mode == "unsafe" { + fs::write(&path, b"foreign").unwrap(); + } + let listener = RefCell::new(None); + let clock = Clock { + elapsed: Cell::new(Duration::ZERO), + on_sleep: RefCell::new(Box::new(|| { + if mode == "delayed" && listener.borrow().is_none() { + *listener.borrow_mut() = Some(UnixListener::bind(&path).unwrap()); + fs::set_permissions(&path, fs::Permissions::from_mode(0o600)).unwrap(); + } + })), + }; + let events = RefCell::new(Vec::new()); + let checks = Cell::new(0); + let mut adopted = false; + let result = finish_boot_with( + &mut adopted, + |_| { + events.borrow_mut().push("templates"); + Ok(Some(( + directory.clone(), + super::super::bridge::BridgeIntent::new(1)?, + ))) + }, + |_| { + checks.set(checks.get() + 1); + let lose = (mode == "provider-before" && checks.get() == 1) + || (mode == "provider-after" && checks.get() == 2); + if lose { + Err(CandidateError::new( + "process_identity_mismatch", + "private detail", + )) + } else { + Ok(()) + } + }, + |_| { + // Deleting the production wait now fails this assertion: the + // scheduled bind has not occurred when audit runs immediately. + super::super::bridge::verify_sockets( + &directory, + super::super::bridge::BridgeIntent::new(1)?, + )?; + events.borrow_mut().push("audit"); + Ok(()) + }, + |adopted| { + *adopted = true; + events.borrow_mut().push("adopt"); + Ok(()) + }, + &clock, + ); + if mode == "delayed" { + result.unwrap(); + assert!(adopted); + assert_eq!(clock.elapsed.get(), Duration::from_millis(50)); + assert_eq!(*events.borrow(), ["templates", "audit", "adopt"]); + } else { + let failure = result.unwrap_err(); + assert!(!adopted); + assert_eq!(*events.borrow(), ["templates"]); + let code = match mode { + "missing" => "bridge_socket_timeout", + "unsafe" => "bridge_socket_identity", + _ => "process_identity_mismatch", + }; + assert_eq!(failure.code, code); + assert!(failure.message_fixed); + assert!(failure.message.contains("runtime recover --json")); + } + drop(clock); + drop(listener.into_inner()); + fs::remove_dir_all(directory).unwrap(); + } +} + +#[test] +fn template_refusal_cannot_reach_socket_observation_audit_or_adoption() { + let clock = Clock { + elapsed: Cell::new(Duration::ZERO), + on_sleep: RefCell::new(Box::new(|| panic!("no wait"))), + }; + let failure = finish_boot_with( + &mut (), + |_| Err(CandidateError::new("disk_template_untrusted", "fixture")), + |_| panic!("no socket observation"), + |_| panic!("no audit"), + |_| -> Result<(), CandidateError> { panic!("no adoption") }, + &clock, + ) + .unwrap_err(); + assert_eq!(failure.code, "disk_template_untrusted"); +} + +#[test] +fn failed_boot_stop_preserves_its_code_and_the_recovery_command() { + let failure = failed_boot_stop_error(CandidateError::new( + "process_identity_unavailable", + String::from("private transport detail"), + )); + assert_eq!(failure.code, "process_identity_unavailable"); + assert!(failure.message_fixed); + assert!(failure.message.contains("runtime recover --json")); + assert!( + failure + .message + .contains("identical --candidate-root argument") + ); + assert!(!failure.message.contains("private transport detail")); +} diff --git a/tests/native-project-start.test.ts b/tests/native-project-start.test.ts index 18a080ac1..d9a853cdb 100644 --- a/tests/native-project-start.test.ts +++ b/tests/native-project-start.test.ts @@ -1,5 +1,6 @@ import { afterEach, expect, test } from "bun:test"; import { + chmod, mkdir, mkdtemp, readdir, @@ -59,6 +60,7 @@ import { verifyHttpsRoutes, } from "../src/backends/native-project-start.ts"; import { + invokeNativeRuntime, NativeRuntimeRequestError, nativeRuntimeFailure, } from "../src/backends/native-runtime-client.ts"; @@ -420,6 +422,38 @@ test("foreground saves after readiness and retains mapping after confirmed stop" "cleanup", ]); }); + +test("a pool bridge boot failure reaches foreground up with recovery guidance before any project mapping", async () => { + const { opts, events } = await fixture(false); + const message = + "Native bridge sockets did not become private and owned before the startup deadline. Run the same selected executor with the identical --candidate-root argument used for the failed start, followed by runtime recover --json before retrying."; + await writeFile( + opts.runtime.binary, + `#!/usr/bin/env bun +console.error(JSON.stringify({ code: "bridge_socket_timeout", message: ${JSON.stringify(message)}, message_fixed: true, private_detail: "synthetic-secret" })); +process.exit(2); +` + ); + await chmod(opts.runtime.binary, 0o755); + const invoke = opts.dependencies.invoke; + opts.dependencies.invoke = async (request) => { + if (request.args[0] === "runtime" && request.args[1] === "up") { + return await invokeNativeRuntime(request); + } + return await invoke?.(request); + }; + const failure: unknown = await startNativeProject(opts).catch( + (error: unknown) => error + ); + expect(failure).toBeInstanceOf(ClosedCliError); + expect(String(failure)).toContain("runtime recover --json"); + expect(String(failure)).toContain("--candidate-root"); + expect(String(failure)).not.toContain("synthetic-secret"); + expect(nativeRuntimeFailure(failure)?.code).toBe("bridge_socket_timeout"); + expect(events).not.toContain("save"); + expect(events).not.toContain("ready"); + expect(events).toContain("cleanup"); +}); test("foreground reports value-free digests of the selected env after publishing the mapping", async () => { const { opts, events } = await fixture(); const seen: { run: string; fingerprint: ComposeEnvFingerprintRecord }[] = []; diff --git a/tests/native-runtime-client.test.ts b/tests/native-runtime-client.test.ts index fddb8653b..61ed069ce 100644 --- a/tests/native-runtime-client.test.ts +++ b/tests/native-runtime-client.test.ts @@ -150,6 +150,7 @@ async function fixture() { if (process.argv.includes("hang")) await Bun.sleep(60_000); if (process.argv.includes("old-provider")) { console.error(JSON.stringify({code:"provider_pin_changed",message:"synthetic-secret-diagnostic"})); process.exit(2); } if (process.argv.includes("fixed-message")) { console.error(JSON.stringify({code:"graph_image",message:"Pinned image architecture or implicit volumes are incompatible.",message_fixed:true,service:"web"})); process.exit(2); } +if (process.argv.includes("bridge-socket-failure")) { console.error(JSON.stringify({code:"bridge_socket_timeout",message:"Native bridge sockets did not become private and owned before the startup deadline. Run the same selected executor with the identical --candidate-root argument used for the failed start, followed by runtime recover --json before retrying.",message_fixed:true})); process.exit(2); } if (process.argv.includes("route-probe-failure")) { console.error(JSON.stringify({code:"graph_service_failed",message:"synthetic-private-response",service:"web",route_probe:{version:1,port:3000}})); process.exit(2); } if (process.argv.includes("unsafe-fixed")) { console.error(JSON.stringify({code:"graph_image",message:"synthetic-secret-diagnostic\\nsecond line",message_fixed:true,service:"../synthetic-secret-diagnostic"})); process.exit(2); } if (process.argv.includes("plan-change")) { console.error(JSON.stringify({code:"graph_plan_reconcile",message:"synthetic-secret-diagnostic",plan_change:{version:1,service:"deps",field:"dependency_cache_volume"}})); process.exit(23); } @@ -1002,6 +1003,19 @@ test("a runtime message marked fixed is shown with its validated service", async expect(String(unsafe)).not.toContain("synthetic-secret-diagnostic"); }); +test("a failed bridge boot names its one recovery command through the runtime client", async () => { + const runtime = await fixture(); + const error: unknown = await invokeNativeRuntime({ + runtime, + cwd: runtime.home, + args: ["bridge-socket-failure"], + }).catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(NativeRuntimeRequestError); + expect(error).toHaveProperty("nativeCode", "bridge_socket_timeout"); + expect(String(error)).toContain("runtime recover --json"); + expect(String(error)).not.toContain("nothing was started"); +}); + test("structured failures expose only a bounded code, never their message", async () => { const runtime = await fixture(); await expect(