diff --git a/docs/guides/native-candidate.md b/docs/guides/native-candidate.md index 7a31e89b..010e75ff 100644 --- a/docs/guides/native-candidate.md +++ b/docs/guides/native-candidate.md @@ -118,6 +118,88 @@ complete bundle to change the candidate. binary refusing newer state is an unsupported downgrade, not a successful rollback. Preserve that state; do not rewrite receipts or adopt another home's data. +Host cleanup or a reboot can remove the temporary `/private/tmp/hkl-` HOME +alias while the private candidate home and disks remain intact. Normal commands +report `provider_home_missing`; they do not initialize another pool or recreate the +alias during observation. Explicit `runtime recover --json` can restore only that +absent exact alias, after verifying the receipt-bound dead provider, both recorded +disks, free VM lock, closed disk handles, and no active provider command. Existing +files, directories, foreign links, a live or reused PID, and uncertain ownership +refuse without replacement. Recovery rechecks the receipt before exclusive creation, +then validates socket absence and flushes the disks before recording +`recovered-unclean`. If those final checks fail, it reports +`provider_home_restored_recovery_incomplete`, keeps the exact owned alias and retained +data, and requires inspection before retry. No guest is started by alias recovery. +Interrupted starts without a recorded process or both identified disks remain +outside this repair path. + +A physical macOS reboot may also renumber the mounted filesystem device. Strict +disk and source checks still refuse a changed device number; missing-HOME recovery +does not waive them. For an offline **stock pool**, inspect the separate migration: + +```sh +./hack-native --candidate-root /absolute/private/candidate-home runtime host-filesystem-recovery --json +./hack-native --candidate-root /absolute/private/candidate-home runtime recover-host-filesystem --expect-sha256 --accept-legacy-device-rebind --json +./hack-native --candidate-root /absolute/private/candidate-home runtime recover --json +``` + +Review the inspection before selecting its hash. This explicit legacy migration +requires an absent recorded provider whose start predates the current host boot, +no active provider commands, exclusive existing operation/VM locks and closed disk +handles. Both disk inodes, sizes and ext4 UUIDs, the exact source path/inode and its +ownership must be unchanged; only one common old-to-new device-number change is +allowed. The inspection is read-only. Publication atomically changes only the +owner's disk and source device numbers, retaining its phase and process record. +Normal commands keep their strict identity checks. + +Legacy receipts have no original host boot UUID or filesystem volume UUID. Calendar +timestamps corroborate a reboot, and matching retained file identities constrain +the migration, but neither proves original volume continuity. The opt-in explicitly +accepts that limitation; copied or relocated pools are outside this procedure. +Prepared-base pools and pending owner/network/activation updates require separate +recovery and are refused. A torn owner publication preserves `owner.pending` and +blocks another migration; do not delete or adopt that file manually. + +This does not start the VM, restore HOME, retire stale sockets or rewrite historical +graph receipts. Use ordinary recovery afterward. Historical shared-source graphs +retain the prior identity and may require verified cleanup plus a new generation; +successful metadata migration alone does not establish application recovery. + +After that explicit provider recovery and one audited VM boot, a retained graph +from the immediately preceding guest boot can still carry old host device numbers +in its publisher, relay-control, and dependency-socket receipts. Inspect and select +one run's host-pin recovery separately: + +```sh +./hack-native --candidate-root /absolute/private/candidate-home graph inspect-host-pin-recovery --run-id --json +./hack-native --candidate-root /absolute/private/candidate-home graph recover-host-pins --run-id --expect-selection --accept-legacy-device-rebind --json +./hack-native --candidate-root /absolute/private/candidate-home graph recover-cleanup --run-id --expect-receipt --json +./hack-native --candidate-root /absolute/private/candidate-home graph retire-recovered-publisher --run-id --expect-owner --json +``` + +The first command only inspects. The second publishes a private, exact-run +witness before cleanup; it changes no graph, publisher, control, or dependency +receipt. It requires the provider's repaired current disks/source, unchanged +recorded inodes and raw receipts, dead owners whose recorded starts precede the +current physical host boot, refused socket listeners, and the immediate guest +boot transition. It also binds retained volume names and actual labels. Cleanup +and retirement can use the witness only for those selected old pins; ordinary +reads and later publisher generations remain strict. Missing or foreign pins, +active listeners, changed resources, pending journals, or further guest boots +refuse without adopting another run. A completed older cleanup journal is +retained as history and does not itself block a later selected generation. + +Legacy receipts do not identify the original APFS volume. This explicit +device-number rebind cannot prove pre-reboot volume continuity. Completing these +commands retains the old run's data and proves cleanup of its dead generation; +it does not migrate `graph.source.shared`, restore the application, establish +route readiness, or claim overall v5 acceptance. Same-run source continuity +requires a separate explicit witness-bound transition after cleanup and +publisher retirement. If macOS removed the foreground or relay-control +directory itself during reboot, this command refuses: the old pin receipts no +longer exist, and absent pathnames cannot stand in for their recorded owner +identities. That case requires a separate selected absence-recovery procedure. + Compatibility is qualified for specific bundle hashes and state formats. The displayed version alone does not establish frontend/executor or downgrade compatibility. V4 and candidate homes remain separate; this procedure does not diff --git a/packages/runtime-core/src/graph_cli.rs b/packages/runtime-core/src/graph_cli.rs index 9b9e3b52..b93a6872 100644 --- a/packages/runtime-core/src/graph_cli.rs +++ b/packages/runtime-core/src/graph_cli.rs @@ -25,6 +25,50 @@ pub fn command(candidate: &Candidate, args: &[&str]) -> Result run, + _ => return Err(invalid()), + }; + #[cfg(target_os = "macos")] + { + return graph::inspect_host_pin_recovery(candidate, run); + } + #[cfg(not(target_os = "macos"))] + { + let _ = run; + return Err(invalid()); + } + } + if *action == "recover-host-pins" { + let (run, expected) = match *args { + [ + "--run-id", + run, + "--expect-selection", + expected, + "--accept-legacy-device-rebind", + ] + | [ + "--run-id", + run, + "--expect-selection", + expected, + "--accept-legacy-device-rebind", + "--json", + ] => (run, expected), + _ => return Err(invalid()), + }; + #[cfg(target_os = "macos")] + { + return graph::recover_host_pins(candidate, run, expected); + } + #[cfg(not(target_os = "macos"))] + { + let _ = (run, expected); + return Err(invalid()); + } + } if ["run-selection", "run-service"].contains(action) { return one_off::command(candidate, action, args); } diff --git a/packages/runtime-core/src/main.rs b/packages/runtime-core/src/main.rs index 68bdc05e..5878fae0 100644 --- a/packages/runtime-core/src/main.rs +++ b/packages/runtime-core/src/main.rs @@ -27,6 +27,8 @@ Usage: hack-local graph recover-cleanup --run-id <32-hex> --expect-receipt [--json] hack-local graph recover-live-owner --run-id <32-hex> --expect-receipt [--json] hack-local graph retire-recovered-publisher --run-id <32-hex> --expect-owner <32-hex> [--json] + hack-local graph inspect-host-pin-recovery --run-id <32-hex> [--json] + hack-local graph recover-host-pins --run-id <32-hex> --expect-selection <64-hex> --accept-legacy-device-rebind [--json] hack-local graph logs --run-id <32-hex> --service [--tail <1..1000>] [--json] hack-local graph exec --run-id <32-hex> --service [--workdir /path] [--timeout-seconds <1..120>] [--json] -- [args...] hack-local graph dependency-plan --dependencies [--json] @@ -86,6 +88,8 @@ Usage: hack-local runtime up --profile development --internet --json hack-local runtime network extend --allow-host [--allow-host ] --json hack-local runtime up|status|down|recover [--json] + hack-local runtime host-filesystem-recovery [--json] + hack-local runtime recover-host-filesystem --expect-sha256 --accept-legacy-device-rebind [--json] hack-local node serve|status|inspect hack-local node request hack-local --version @@ -857,6 +861,32 @@ fn run() -> Result<(), CandidateError> { image, )?)?; } + ["runtime", "host-filesystem-recovery"] + | ["runtime", "host-filesystem-recovery", "--json"] => { + print_json(&hack_runtime_core::provider::host_filesystem::inspect( + &discover_candidate(&requested)?, + )?)?; + } + [ + "runtime", + "recover-host-filesystem", + "--expect-sha256", + hash, + "--accept-legacy-device-rebind", + ] + | [ + "runtime", + "recover-host-filesystem", + "--expect-sha256", + hash, + "--accept-legacy-device-rebind", + "--json", + ] => { + print_json(&hack_runtime_core::provider::host_filesystem::recover( + &discover_candidate(&requested)?, + hash, + )?)?; + } ["runtime", action] | ["runtime", action, "--json"] => { let candidate = discover_candidate(&requested)?; let result = match *action { diff --git a/packages/runtime-core/src/provider/graph/bridges/cleanup.rs b/packages/runtime-core/src/provider/graph/bridges/cleanup.rs index 8d918fe5..c76005ce 100644 --- a/packages/runtime-core/src/provider/graph/bridges/cleanup.rs +++ b/packages/runtime-core/src/provider/graph/bridges/cleanup.rs @@ -46,6 +46,14 @@ fn validate_selection( engine: &Engine<'_>, receipt: &Receipt, selection: &Selection, +) -> Result<(), CandidateError> { + validate_selection_recovery(engine, receipt, selection, None) +} +fn validate_selection_recovery( + engine: &Engine<'_>, + receipt: &Receipt, + selection: &Selection, + host_pin: Option<&super::super::host_pin_recovery::Witness>, ) -> Result<(), CandidateError> { let capacity = engine.guest().bridge_intent().map_or(0, |v| v.slots); if selection.version != 1 @@ -67,6 +75,7 @@ fn validate_selection( if predecessor_owner( receipt, selection.previous_boot.as_deref().ok_or_else(invalid)?, + host_pin, )? != *expected { return Err(invalid()); @@ -183,17 +192,39 @@ fn pinned_predecessor_eligible(receipt: &Receipt) -> bool { }) } -fn predecessor_owner(receipt: &Receipt, previous: &str) -> Result { +fn predecessor_owner( + receipt: &Receipt, + previous: &str, + host_pin: Option<&super::super::host_pin_recovery::Witness>, +) -> Result { let startup = receipt .relay_startup .as_ref() .filter(|_| pinned_predecessor_eligible(receipt)) .ok_or_else(invalid)?; - let pin = crate::provider::relay_owner::publication::PinnedEndpoint::load( - &startup.control_root, - super::super::host_relay::context(&receipt.owner, previous)?, - )?; - pin.verify_dead()?; + let context = super::super::host_relay::context(&receipt.owner, previous)?; + let pin = if let Some(selected) = host_pin { + if !selected.matches_graph(receipt) || selected.control_root() != startup.control_root { + return Err(invalid()); + } + let pin = crate::provider::relay_owner::publication::PinnedEndpoint::load_legacy_recovery( + &startup.control_root, + context, + selected.rebind(), + selected.host_boot_micros(), + )?; + if &pin.legacy_summary() != selected.control() { + return Err(invalid()); + } + pin + } else { + let pin = crate::provider::relay_owner::publication::PinnedEndpoint::load( + &startup.control_root, + context, + )?; + pin.verify_dead()?; + pin + }; Ok(pin .fingerprint() .iter() @@ -208,6 +239,7 @@ pub(crate) fn capture_previous_boot( engine: &Engine<'_>, receipt: &Receipt, previous: &str, + host_pin: Option<&super::super::host_pin_recovery::Witness>, ) -> Result { let store = strict_store(candidate, engine)?; let mut selection = Selection { @@ -271,22 +303,23 @@ pub(crate) fn capture_previous_boot( }) .collect(); } else { - selection.predecessor_owner = Some(predecessor_owner(receipt, previous)?); + selection.predecessor_owner = Some(predecessor_owner(receipt, previous, host_pin)?); } } - validate_selection(engine, receipt, &selection)?; + validate_selection_recovery(engine, receipt, &selection, host_pin)?; Ok(selection) } /// Pending recovery may resume only the exact remaining reservations. Released /// slots may be absent; a replacement or new reservation is never adopted. -pub(crate) fn verify_remaining( +pub(crate) fn verify_remaining_recovery( candidate: &Candidate, engine: &Engine<'_>, receipt: &Receipt, selection: &Selection, + host_pin: Option<&super::super::host_pin_recovery::Witness>, ) -> Result<(), CandidateError> { - validate_selection(engine, receipt, selection)?; + validate_selection_recovery(engine, receipt, selection, host_pin)?; if selection.previous_boot.is_none() { return Ok(()); } @@ -529,19 +562,28 @@ pub(crate) fn read( engine: &Engine<'_>, receipt: &Receipt, root: &std::path::Path, +) -> Result { + read_recovery(engine, receipt, root, None) +} +pub(crate) fn read_recovery( + engine: &Engine<'_>, + receipt: &Receipt, + root: &std::path::Path, + host_pin: Option<&super::super::host_pin_recovery::Witness>, ) -> Result { let selection = state::read_bounded(&selection_path(root)?, 65536)?; - validate_selection(engine, receipt, &selection)?; + validate_selection_recovery(engine, receipt, &selection, host_pin)?; Ok(selection) } -pub(crate) fn verify( +pub(crate) fn verify_recovery( candidate: &Candidate, engine: &Engine<'_>, receipt: &Receipt, selection: &Selection, + host_pin: Option<&super::super::host_pin_recovery::Witness>, ) -> Result<(), CandidateError> { - validate_selection(engine, receipt, selection)?; + validate_selection_recovery(engine, receipt, selection, host_pin)?; let store = strict_store(candidate, engine)?; if store.next_launch_serial < selection.serial || store.slots.values().any(|a| a.run == receipt.run) @@ -557,6 +599,16 @@ pub(crate) fn verify( Ok(()) } +#[cfg(test)] +pub(crate) fn verify( + candidate: &Candidate, + engine: &Engine<'_>, + receipt: &Receipt, + selection: &Selection, +) -> Result<(), CandidateError> { + verify_recovery(candidate, engine, receipt, selection, None) +} + #[cfg(test)] mod tests { use super::*; diff --git a/packages/runtime-core/src/provider/graph/dead_owner_cleanup.rs b/packages/runtime-core/src/provider/graph/dead_owner_cleanup.rs index f2d1a813..9e32a865 100644 --- a/packages/runtime-core/src/provider/graph/dead_owner_cleanup.rs +++ b/packages/runtime-core/src/provider/graph/dead_owner_cleanup.rs @@ -4,6 +4,7 @@ use super::*; use crate::provider::{identity, lifecycle, state::Owner}; use sha2::{Digest, Sha256}; +use std::os::unix::fs::MetadataExt; const FILE: &str = "dead-owner-cleanup.json"; #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] @@ -322,18 +323,98 @@ pub fn retire_recovered_publisher( } let bridges = intent.bridges.as_ref().ok_or_else(refused)?; bridges::cleanup::verify_recovery_file(&root, bridges, intent.prior_bridges.as_ref())?; + let legacy = super::host_pin_recovery::load_witness(candidate, run)?.filter(|witness| { + witness.graph_sha256() == intent.original_sha256 + && witness.publisher_sha256() == intent.owner_sha256 + && witness.matches_graph(&intent.original) + }); if intent.new_boot.as_deref() == Some(engine.guest().boot_id()) { - host_relay::inspect_cleanup(candidate, &engine, &receipt, false, &environment, bridges)?; - foreground::retire_publisher_path(candidate, run, &intent.owner_sha256, &complete)?; + if let Some(witness) = legacy { + if let Some(retired) = foreground::transport::Retired::acquire(candidate, run)? { + let control_root = witness.control_root().join("relay-control"); + let control_lock = state::Lock::acquire_existing(&control_root)?; + host_relay::inspect_cleanup_recovery( + candidate, + &engine, + &receipt, + false, + &environment, + bridges, + Some(&witness), + )?; + let lock_path = fs::symlink_metadata(control_root.join("operation.lock")) + .map_err(|_| refused())?; + if (lock_path.dev(), lock_path.ino()) != control_lock.identity()? { + return Err(refused()); + } + retired.verify_recovery_with_rebind( + candidate, + run, + &intent.owner_sha256, + &complete, + Some(witness.rebind()), + )?; + } else { + let foreground_root = foreground::transport::root(candidate, run)?; + let foreground_lock = state::Lock::acquire_existing(&foreground_root)?; + let guard = super::host_pin_recovery::acquire_for_cleanup( + candidate, + run, + &engine, + super::host_pin_recovery::CleanupProof { + original: &intent.original, + sha256: &intent.original_sha256, + current_is_original: false, + allow_absent_reservation: true, + publisher_may_be_partial: true, + }, + witness, + )?; + host_relay::inspect_cleanup_recovery( + candidate, + &engine, + &receipt, + false, + &environment, + bridges, + Some(guard.witness()), + )?; + guard.verify_lock()?; + let lock_path = fs::symlink_metadata(foreground_root.join("operation.lock")) + .map_err(|_| refused())?; + if (lock_path.dev(), lock_path.ino()) != foreground_lock.identity()? { + return Err(refused()); + } + foreground::transport::retire_recovered_publisher_locked( + candidate, + run, + &intent.owner_sha256, + &complete, + Some(guard.witness().rebind()), + &foreground_lock, + )?; + } + } else { + host_relay::inspect_cleanup( + candidate, + &engine, + &receipt, + false, + &environment, + bridges, + )?; + foreground::retire_publisher_path(candidate, run, &intent.owner_sha256, &complete)?; + } } else { // A later VM boot has a new bridge registry generation. Recheck the // retained graph and immutable prior proof, then require the publisher // to have been fully retired under the original recovery boot. - foreground::verify_recovered_publisher_retired( + foreground::transport::verify_recovered_publisher_retired_recovery( candidate, run, &intent.owner_sha256, &complete, + legacy.as_ref().map(|witness| witness.rebind()), )?; } Ok(json!({"run":run,"publisher_retired":true,"data_retained":true})) @@ -427,7 +508,13 @@ fn fresh_boot( } fn execute(candidate: &Candidate, run: &str, expected: &str) -> Result { let engine = Engine::connect_cleanup_wait(candidate)?; - let dead = foreground::DeadOwner::acquire(candidate, run)?; + let selected_pin = super::host_pin_recovery::selected_for_old_publisher(candidate, run)?; + let dead = foreground::DeadOwner::acquire_recovery( + candidate, + run, + selected_pin.as_ref().map(|pin| pin.rebind()), + selected_pin.as_ref().map(|pin| pin.host_boot_micros()), + )?; let (receipt, root) = load(candidate, &engine, run)?; if exists(&root.join("one-off-normalization.json"))? { let intent: Intent = state::read(&root.join(FILE))?; @@ -448,6 +535,36 @@ fn execute(candidate: &Candidate, run: &str, expected: &str) -> Result = if exists(&root.join(FILE))? { + Some(state::read(&root.join(FILE))?) + } else { + None + }; + let pin_guard = if let Some(witness) = selected_pin { + let original = existing_intent + .as_ref() + .map_or(&receipt, |intent| &intent.original); + let original_sha = existing_intent + .as_ref() + .map_or(expected, |intent| intent.original_sha256.as_str()); + Some(super::host_pin_recovery::acquire_for_cleanup( + candidate, + run, + &engine, + super::host_pin_recovery::CleanupProof { + original, + sha256: original_sha, + current_is_original: existing_intent.is_none(), + allow_absent_reservation: existing_intent + .as_ref() + .is_some_and(|intent| intent.complete_sha256.is_some()), + publisher_may_be_partial: false, + }, + witness, + )?) + } else { + None + }; let mut intent: Intent = if exists(&root.join(FILE))? { state::read(&root.join(FILE))? } else { @@ -479,8 +596,13 @@ fn execute(candidate: &Candidate, run: &str, expected: &str) -> Result Result Result Result Result Result Result Result Result { } } fn read(path: &Path) -> Result { + read_with_bytes(path).map(|(record, _)| record) +} +fn read_with_bytes(path: &Path) -> Result<(Record, Vec), CandidateError> { let mut file = OpenOptions::new() .read(true) .custom_flags(libc::O_NOFOLLOW | libc::O_NONBLOCK) @@ -70,7 +74,121 @@ fn read(path: &Path) -> Result { if bytes.len() as u64 != m.len() { return Err(refused()); } - serde_json::from_slice(&bytes).map_err(|_| refused()) + let record = serde_json::from_slice(&bytes).map_err(|_| refused())?; + Ok((record, bytes)) +} +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct LegacyReservation { + pub record_sha256: String, + pub sockets: BTreeMap, + pub process: ProcessIdentity, +} +pub(super) fn inspect_legacy( + candidate: &Candidate, + run: &str, + rebind: DeviceRebind, + host_boot_micros: u64, +) -> Result, CandidateError> { + let owner = state::Owner::load(candidate)?; + let directory = root(candidate); + let values = records( + &directory, + &owner.token, + owner.dependency_sockets.map_or(0, |v| v.slots), + )?; + let Some(record) = values.iter().find(|r| r.run == run) else { + return Ok(None); + }; + let (again, bytes) = read_with_bytes(&directory.join(format!("{run}.json")))?; + if &again != record || record.sockets.len() != record.slots.len() { + return Err(refused()); + } + // SAFETY: geteuid has no arguments or side effects. + identity::verify( + &record.process, + &record.process, + &record.process.executable, + unsafe { libc::geteuid() }, + )?; + rebind.definitely_dead_before_boot(&record.process, host_boot_micros)?; + for slot in record.slots.values() { + let expected = record.sockets.get(slot).ok_or_else(refused)?; + let path = socket(&owner.short_home, *slot); + let metadata = fs::symlink_metadata(&path).map_err(|_| refused())?; + if !metadata.file_type().is_socket() + || metadata.uid() != record.process.uid + || metadata.mode() & 0o7777 != 0o600 + || metadata.nlink() != 1 + || !rebind.matches(*expected, (metadata.dev(), metadata.ino())) + { + return Err(refused()); + } + crate::provider::relay_owner::publication::dead::no_listener(&path) + .map_err(|_| refused())?; + } + Ok(Some(LegacyReservation { + record_sha256: format!("{:x}", Sha256::digest(bytes)), + sockets: record.sockets.clone(), + process: record.process.clone(), + })) +} +pub(super) fn verify_legacy_remaining( + candidate: &Candidate, + run: &str, + rebind: DeviceRebind, + selected: &LegacyReservation, + allow_absent_record: bool, +) -> Result<(), CandidateError> { + let owner = state::Owner::load(candidate)?; + let path = root(candidate).join(format!("{run}.json")); + if absent(&path)? { + if !allow_absent_record { + return Err(refused()); + } + for slot in selected.sockets.keys() { + if !absent(&socket(&owner.short_home, *slot))? { + return Err(refused()); + } + } + return Ok(()); + } + let values = records( + &root(candidate), + &owner.token, + owner.dependency_sockets.map_or(0, |v| v.slots), + )?; + let record = values + .iter() + .find(|record| record.run == run) + .ok_or_else(refused)?; + let (same, bytes) = read_with_bytes(&path)?; + if &same != record + || selected.record_sha256 != format!("{:x}", Sha256::digest(bytes)) + || selected.sockets != record.sockets + || selected.process != record.process + { + return Err(refused()); + } + for slot in record.slots.values() { + let path = socket(&owner.short_home, *slot); + if absent(&path)? { + continue; + } + let expected = record.sockets.get(slot).ok_or_else(refused)?; + let observed = fs::symlink_metadata(&path).map_err(|_| refused())?; + if !observed.file_type().is_socket() + || observed.uid() != record.process.uid + || observed.mode() & 0o7777 != 0o600 + || observed.nlink() != 1 + || !rebind.matches(*expected, (observed.dev(), observed.ino())) + { + return Err(refused()); + } + crate::provider::relay_owner::publication::dead::no_listener(&path) + .map_err(|_| refused())?; + } + Ok(()) } fn valid(record: &Record, owner: &str, capacity: u8) -> bool { record.version == 1 @@ -339,7 +457,11 @@ pub fn inspect(candidate: &Candidate) -> Result Result<(), CandidateError> { +fn recover_record( + candidate: &Candidate, + record: &Record, + rebind: Option, +) -> Result<(), CandidateError> { if identity::alive(record.process.pid)? { return Err(refused()); } @@ -347,10 +469,14 @@ fn recover_record(candidate: &Candidate, record: &Record) -> Result<(), Candidat if record.owner != owner.token { return Err(refused()); } - recover_paths(&owner.short_home, record)?; + recover_paths(&owner.short_home, record, rebind)?; remove_record(&root(candidate), record) } -fn recover_paths(home: &Path, record: &Record) -> Result<(), CandidateError> { +fn recover_paths( + home: &Path, + record: &Record, + rebind: Option, +) -> Result<(), CandidateError> { // Validate every selected path before removing any. Unexpected or unrecorded // paths are never adopted, even when nobody currently listens on them. let mut selected = Vec::new(); @@ -364,7 +490,11 @@ fn recover_paths(home: &Path, record: &Record) -> Result<(), CandidateError> { || m.mode() & 0o7777 != 0o600 || m.uid() != record.process.uid || m.nlink() != 1 - || record.sockets.get(slot) != Some(&(m.dev(), m.ino())) + || !record.sockets.get(slot).is_some_and(|expected| { + rebind.map_or(*expected == (m.dev(), m.ino()), |v| { + v.matches(*expected, (m.dev(), m.ino())) + }) + }) { return Err(refused()); } @@ -402,13 +532,14 @@ pub fn recover_orphan( if fingerprint(record)? != expected || !absent(&super::directory(candidate, run)?)? { return Err(refused()); } - recover_record(candidate, record)?; + recover_record(candidate, record, None)?; Ok(serde_json::json!({"run":run,"released":true})) } /// Caller holds Engine lease and has completed dead-owner cleanup/verification. pub(super) fn recover_cleaned( candidate: &Candidate, receipt: &Receipt, + rebind: Option<(DeviceRebind, &LegacyReservation)>, ) -> Result<(), CandidateError> { let directory = root(candidate); if absent(&directory.join(format!("{}.json", receipt.run)))? { @@ -438,7 +569,17 @@ pub(super) fn recover_cleaned( { return Err(refused()); } - recover_record(candidate, record) + if let Some((_, selected)) = rebind { + let (again, bytes) = read_with_bytes(&directory.join(format!("{}.json", receipt.run)))?; + if &again != record + || selected.record_sha256 != format!("{:x}", Sha256::digest(bytes)) + || selected.sockets != record.sockets + || selected.process != record.process + { + return Err(refused()); + } + } + recover_record(candidate, record, rebind.map(|(value, _)| value)) } #[cfg(test)] @@ -555,14 +696,14 @@ mod tests { let mut b = fixture_record('2', 1); let a_listener = listener(&fixture.0, &mut a, 0); let b_listener = listener(&fixture.0, &mut b, 1); - assert!(recover_paths(&fixture.0, &a).is_err()); + assert!(recover_paths(&fixture.0, &a, None).is_err()); drop(a_listener); let old = a.sockets[&0]; a.sockets.insert(0, (old.0, old.1 + 1)); - assert!(recover_paths(&fixture.0, &a).is_err()); + assert!(recover_paths(&fixture.0, &a, None).is_err()); assert!(socket(&fixture.0, 0).exists()); a.sockets.insert(0, old); - recover_paths(&fixture.0, &a).unwrap(); + recover_paths(&fixture.0, &a, None).unwrap(); assert!(!socket(&fixture.0, 0).exists()); assert!(UnixStream::connect(socket(&fixture.0, 1)).is_ok()); drop(b_listener); @@ -575,10 +716,10 @@ mod tests { let first = listener(&fixture.0, &mut a, 0); let second = listener(&fixture.0, &mut a, 1); drop(first); - assert!(recover_paths(&fixture.0, &a).is_err()); + assert!(recover_paths(&fixture.0, &a, None).is_err()); assert!(socket(&fixture.0, 0).exists()); drop(second); - recover_paths(&fixture.0, &a).unwrap(); + recover_paths(&fixture.0, &a, None).unwrap(); assert!(!socket(&fixture.0, 0).exists()); assert!(!socket(&fixture.0, 1).exists()); } diff --git a/packages/runtime-core/src/provider/graph/foreground.rs b/packages/runtime-core/src/provider/graph/foreground.rs index 7f46e84b..366769dc 100644 --- a/packages/runtime-core/src/provider/graph/foreground.rs +++ b/packages/runtime-core/src/provider/graph/foreground.rs @@ -20,7 +20,6 @@ mod tests; pub(in crate::provider::graph) mod transport; pub(in crate::provider::graph) use transport::DeadOwner; pub(in crate::provider::graph) use transport::retire_recovered_publisher as retire_publisher_path; -pub(in crate::provider::graph) use transport::verify_recovered_publisher_retired; use transport::{Publication, WireRequest}; fn refused() -> CandidateError { CandidateError::new( diff --git a/packages/runtime-core/src/provider/graph/foreground/native_test.rs b/packages/runtime-core/src/provider/graph/foreground/native_test.rs index 016db1b0..aba7ac31 100644 --- a/packages/runtime-core/src/provider/graph/foreground/native_test.rs +++ b/packages/runtime-core/src/provider/graph/foreground/native_test.rs @@ -542,5 +542,6 @@ mod dependency_rebind; mod dependency_slots; mod startup_cancellation; +mod host_pin_recovery; mod retired_recovery_cleanup; mod same_boot_recovery; diff --git a/packages/runtime-core/src/provider/graph/foreground/native_test/host_pin_recovery.rs b/packages/runtime-core/src/provider/graph/foreground/native_test/host_pin_recovery.rs new file mode 100644 index 00000000..1ce87e51 --- /dev/null +++ b/packages/runtime-core/src/provider/graph/foreground/native_test/host_pin_recovery.rs @@ -0,0 +1,537 @@ +//! Real CLI path for explicitly witnessed legacy host-pin cleanup. An isolated +//! test VM supplies the guest; only fixture receipt device numbers are changed. +use super::dependency_rebind::{checked_cli, exec, snapshot}; +use super::*; +use std::{ + io::Write, + os::unix::fs::{MetadataExt, OpenOptionsExt}, +}; + +fn ready(owner: &mut Process, run: &str, deadline: Instant) { + loop { + assert!( + owner.poll().is_none(), + "foreground owner exited before ready" + ); + if let Some(end) = owner.out.iter().position(|byte| *byte == b'\n') { + let value: Value = serde_json::from_slice(&owner.out[..end]).unwrap(); + assert_eq!(value["kind"], "graph_foreground_ready"); + assert_eq!(value["run"], run); + return; + } + assert!(Instant::now() < deadline, "foreground readiness deadline"); + std::thread::sleep(Duration::from_millis(10)); + } +} + +fn legacy_pin(path: &Path, old: u64, before_boot: u64, control: bool) -> Vec { + let mut value: Value = state::read(path).unwrap(); + for key in if control { + ["parent", "endpoint"] + } else { + ["parent", "socket"] + } { + let pair = value[key].as_array_mut().unwrap(); + assert_ne!(pair[0].as_u64().unwrap(), old); + pair[0] = json!(old); + } + // This is a synthetic previous-physical-boot receipt. The actual owner + // exited before the test changes its private fixture bytes. + value["process"]["pid"] = json!(i32::MAX); + value["process"]["start_micros"] = json!(before_boot - 1); + state::write(path, &value).unwrap(); + fs::read(path).unwrap() +} + +fn refused_cli( + binary: &Path, + candidate: &Candidate, + args: &[&str], + expected_code: &str, + deadline: Instant, +) { + let mut process = Process::start(binary, candidate, args, None); + let status = process.wait(deadline); + assert_eq!( + status.code(), + Some(2), + "expected a CLI refusal, not a crash" + ); + assert!(process.out.is_empty(), "refused action emitted a result"); + let error: Value = serde_json::from_slice(&process.err).expect("structured CLI error"); + assert_eq!(error["code"], expected_code); + assert!(error["message"].as_str().is_some_and(|v| !v.is_empty())); +} + +fn publish_args<'a>(run: &'a str, expected: &'a str) -> [&'a str; 8] { + [ + "graph", + "recover-host-pins", + "--run-id", + run, + "--expect-selection", + expected, + "--accept-legacy-device-rebind", + "--json", + ] +} + +struct TemporarilyMoved { + original: PathBuf, + moved: PathBuf, +} +impl TemporarilyMoved { + fn new(original: PathBuf) -> Self { + let moved = original.with_extension("host-pin-test-held"); + assert!(!moved.exists()); + fs::rename(&original, &moved).unwrap(); + Self { original, moved } + } +} +impl Drop for TemporarilyMoved { + fn drop(&mut self) { + if self.moved.exists() { + assert!(!self.original.exists(), "foreign replacement at held path"); + fs::rename(&self.moved, &self.original).unwrap(); + } + } +} + +/// Preserve exact pre-mutation bytes outside managed state. A killed test can +/// be recovered from this private fixture artifact; normal unwinding restores +/// only the bytes this test itself wrote, never a foreign replacement. +struct ProviderReceiptRestore { + path: PathBuf, + original: Vec, + current: Vec, +} +impl ProviderReceiptRestore { + fn new(candidate: &Candidate, path: PathBuf) -> Self { + let original = fs::read(&path).unwrap(); + let backup = candidate + .checkout + .join("host-pin-test-provider-owner-original.json"); + let mut artifact = fs::OpenOptions::new() + .write(true) + .create_new(true) + .mode(0o600) + .open(&backup) + .unwrap(); + artifact.write_all(&original).unwrap(); + artifact.sync_all().unwrap(); + fs::File::open(&candidate.checkout) + .unwrap() + .sync_all() + .unwrap(); + Self { + path, + current: original.clone(), + original, + } + } + fn replace(&mut self, bytes: Vec) { + assert_eq!(fs::read(&self.path).unwrap(), self.current); + fs::write(&self.path, &bytes).unwrap(); + self.current = bytes; + } + fn restore(&mut self) { + assert_eq!(fs::read(&self.path).unwrap(), self.current); + fs::write(&self.path, &self.original).unwrap(); + assert_eq!(fs::read(&self.path).unwrap(), self.original); + self.current = self.original.clone(); + } +} +impl Drop for ProviderReceiptRestore { + fn drop(&mut self) { + if self.current != self.original + && fs::read(&self.path).ok().as_deref() == Some(self.current.as_slice()) + { + let _ = fs::write(&self.path, &self.original); + } + } +} + +#[test] +#[ignore = "Owned capacity-two native VM, pinned static image and external 300s watchdog required"] +fn explicit_host_pin_witness_recovers_only_selected_previous_boot_run() { + let deadline = Instant::now() + Duration::from_secs(270); + let candidate = + Candidate::discover(Path::new(&std::env::var("HACK_LOCAL_TEST_ROOT").unwrap())).unwrap(); + let binary = PathBuf::from(std::env::var("HACK_LOCAL_TEST_BINARY").unwrap()); + let image = std::env::var("HACK_LOCAL_TEST_IMAGE").unwrap(); + let fixtures = [graph::tests::Fixture::new(), graph::tests::Fixture::new()]; + let runs = [ + graph::probes::token().unwrap(), + graph::probes::token().unwrap(), + ]; + let selection_root = graph::tests::Fixture::new(); + let mut plans = Vec::new(); + let mut selections = Vec::new(); + let mut dependencies = Vec::new(); + for (index, fixture) in fixtures.iter().enumerate() { + state::write(&fixture.0.join("compose.yaml"), &json!({ + "services":{"web":{"image":image,"read_only":true,"network_mode":"none", + "init":true,"user":"0:0","entrypoint":["/bin/sleep","300"],"command":[], + "volumes":["data:/data"],"healthcheck":{"test":["CMD","/bin/hack-graph-startup-app","complete"], + "interval":"200ms","timeout":"2s","retries":10,"start_period":"500ms"}}}, + "volumes":{"data":{}} + })).unwrap(); + let review = project::plan( + &candidate, + project::PlanOptions { + branch: None, + project: &fixture.0, + compose_file: Path::new("compose.yaml"), + profiles: &[], + }, + ) + .unwrap(); + let selection = selection_root.0.join(format!("dependencies-{index}.json")); + state::write(&selection, &json!({"version":1,"plan":review.plan_id, + "artifact":"/tmp/unused-control-only-artifact","artifact_sha256":"a".repeat(64),"dependencies":[]})).unwrap(); + let dependency = checked_cli( + "dependency-plan", + &binary, + &candidate, + &[ + "graph", + "dependency-plan", + "--dependencies", + selection.to_str().unwrap(), + "--json", + ], + deadline, + ); + plans.push(review); + selections.push(selection); + dependencies.push(dependency); + } + let start = |index: usize, generation: Option<&str>| { + let normalized = fixtures[index].0.join("compose.yaml"); + let mut args = vec![ + "graph", + if generation.is_some() { + "serve-restore" + } else { + "serve" + }, + "--project", + fixtures[index].0.to_str().unwrap(), + "--file", + "compose.yaml", + "--expect-plan", + &plans[index].plan_id, + "--run-id", + &runs[index], + "--ready", + "web=healthy", + "--timeout-seconds", + "90", + "--dependencies", + selections[index].to_str().unwrap(), + "--expect-dependencies", + dependencies[index]["dependency_plan_id"].as_str().unwrap(), + "--normalized-file", + normalized.to_str().unwrap(), + "--expect-original", + &plans[index].plan.compose_sha256, + "--expect-namespace", + &plans[index].plan.namespace, + "--json", + ]; + if let Some(generation) = generation { + args.extend(["--expect-generation", generation]); + } + Process::start(&binary, &candidate, &args, None) + }; + let mut selected_owner = start(0, None); + let mut selected_cleanup = Cleanup { + binary: &binary, + candidate: &candidate, + run: &runs[0], + done: false, + }; + ready(&mut selected_owner, &runs[0], deadline); + assert_eq!( + exec(&binary, &candidate, &runs[0], "web", "write-data", deadline)["exit_code"], + 0 + ); + let original = snapshot(&candidate, &runs[0], deadline).receipt; + assert_eq!(original.phase, "ready-observed"); + let expected_receipt = format!( + "{:x}", + Sha256::digest(serde_json::to_vec_pretty(&original).unwrap()) + ); + let old_boot = crate::provider::lifecycle::status(&candidate) + .unwrap() + .guest_boot_id + .unwrap(); + selected_owner.child.kill().unwrap(); + selected_owner.wait(deadline); + assert_eq!( + crate::provider::lifecycle::down(&candidate).unwrap().phase, + "stopped" + ); + let new_boot = crate::provider::lifecycle::up(&candidate) + .unwrap() + .guest_boot_id + .unwrap(); + assert_ne!(old_boot, new_boot); + + let mut sibling_owner = start(1, None); + let mut sibling_cleanup = Cleanup { + binary: &binary, + candidate: &candidate, + run: &runs[1], + done: false, + }; + ready(&mut sibling_owner, &runs[1], deadline); + assert_eq!( + exec(&binary, &candidate, &runs[1], "web", "write-data", deadline)["exit_code"], + 0 + ); + let sibling = serde_json::to_vec(&snapshot(&candidate, &runs[1], deadline).receipt).unwrap(); + + let root = graph::directory(&candidate, &runs[0]).unwrap(); + let graph_bytes = fs::read(root.join("state.json")).unwrap(); + let startup = original.relay_startup.as_ref().unwrap(); + let publisher_root = transport::root(&candidate, &runs[0]).unwrap(); + let publisher = publisher_root.join("owner.json"); + let control = startup.control_root.join("relay-control/owner.json"); + let current = fs::symlink_metadata(&publisher_root).unwrap().dev(); + let old = current.checked_add(1).unwrap(); + let before_boot = crate::provider::lifecycle::host_filesystem::host_boot_micros().unwrap(); + let publisher_bytes = legacy_pin(&publisher, old, before_boot, false); + let control_bytes = legacy_pin(&control, old, before_boot, true); + + // Ordinary recovery remains strict; the new witness is required to use + // the legacy device-only pins and is scoped to the selected run. + assert!(graph::foreground::transport::Pin::load(&candidate, &runs[0]).is_err()); + let inspected = checked_cli( + "inspect-host-pins", + &binary, + &candidate, + &[ + "graph", + "inspect-host-pin-recovery", + "--run-id", + &runs[0], + "--json", + ], + deadline, + ); + let selection = inspected["selection_sha256"].as_str().unwrap(); + assert_eq!(selection.len(), 64); + assert_eq!(inspected["old_device"], old); + assert_eq!(inspected["new_device"], current); + assert_eq!(fs::read(root.join("state.json")).unwrap(), graph_bytes); + assert_eq!(fs::read(&publisher).unwrap(), publisher_bytes); + assert_eq!(fs::read(&control).unwrap(), control_bytes); + + let witness_path = root.join("host-pin-recovery.json"); + let inspect_args = [ + "graph", + "inspect-host-pin-recovery", + "--run-id", + &runs[0], + "--json", + ]; + // A stale selection must not publish a witness or rewrite any source + // receipt. The three temporary path removals also preserve exact bytes. + refused_cli( + &binary, + &candidate, + &publish_args(&runs[0], &"0".repeat(64)), + "graph_host_pin_recovery", + deadline, + ); + assert!(!witness_path.exists()); + assert_eq!(fs::read(root.join("state.json")).unwrap(), graph_bytes); + assert_eq!(fs::read(&publisher).unwrap(), publisher_bytes); + assert_eq!(fs::read(&control).unwrap(), control_bytes); + { + let _missing = TemporarilyMoved::new(publisher_root.clone()); + refused_cli( + &binary, + &candidate, + &inspect_args, + "provider_state", + deadline, + ); + assert!(!witness_path.exists()); + } + assert_eq!(fs::read(&publisher).unwrap(), publisher_bytes); + { + let _missing = TemporarilyMoved::new(startup.control_root.join("relay-control")); + refused_cli( + &binary, + &candidate, + &inspect_args, + "provider_state", + deadline, + ); + assert!(!witness_path.exists()); + } + assert_eq!(fs::read(&control).unwrap(), control_bytes); + + // The provider receipt is valid JSON throughout. A changed raw byte + // selection and a changed guest boot must both refuse without writes. + let provider_path = candidate.state_root.join("run/smolvm/owner.json"); + let mut owner_restore = ProviderReceiptRestore::new(&candidate, provider_path.clone()); + let provider_bytes = owner_restore.original.clone(); + let mut whitespace = provider_bytes.clone(); + whitespace.push(b' '); + owner_restore.replace(whitespace.clone()); + refused_cli( + &binary, + &candidate, + &publish_args(&runs[0], selection), + "graph_host_pin_recovery", + deadline, + ); + assert_eq!(fs::read(&provider_path).unwrap(), whitespace); + assert!(!witness_path.exists()); + owner_restore.restore(); + let mut changed_boot: Value = serde_json::from_slice(&provider_bytes).unwrap(); + changed_boot["guest_boot_id"] = json!("f".repeat(32)); + assert_ne!(changed_boot["guest_boot_id"], new_boot); + owner_restore.replace(serde_json::to_vec_pretty(&changed_boot).unwrap()); + let boot_bytes = fs::read(&provider_path).unwrap(); + refused_cli( + &binary, + &candidate, + &inspect_args, + "graph_host_pin_recovery", + deadline, + ); + assert_eq!(fs::read(&provider_path).unwrap(), boot_bytes); + assert!(!witness_path.exists()); + owner_restore.restore(); + assert_eq!(fs::read(root.join("state.json")).unwrap(), graph_bytes); + assert_eq!(fs::read(&publisher).unwrap(), publisher_bytes); + assert_eq!(fs::read(&control).unwrap(), control_bytes); + + let refreshed = checked_cli( + "inspect-after-negative-controls", + &binary, + &candidate, + &inspect_args, + deadline, + ); + let selection = refreshed["selection_sha256"].as_str().unwrap(); + let recovered = checked_cli( + "publish-host-pin-witness", + &binary, + &candidate, + &publish_args(&runs[0], selection), + deadline, + ); + assert_eq!(recovered["witness_published"], true); + assert_eq!(fs::read(root.join("state.json")).unwrap(), graph_bytes); + assert_eq!(fs::read(&publisher).unwrap(), publisher_bytes); + assert_eq!(fs::read(&control).unwrap(), control_bytes); + assert_eq!( + serde_json::to_vec(&snapshot(&candidate, &runs[1], deadline).receipt).unwrap(), + sibling + ); + + let cleanup = checked_cli( + "legacy-dead-owner-cleanup", + &binary, + &candidate, + &[ + "graph", + "recover-cleanup", + "--run-id", + &runs[0], + "--expect-receipt", + &expected_receipt, + "--json", + ], + deadline, + ); + assert_eq!(cleanup["phase"], "stopped-data-retained"); + let retained = snapshot(&candidate, &runs[0], deadline).receipt; + assert_eq!( + retained.resources["volume:data"].name, + original.resources["volume:data"].name + ); + for _ in 0..2 { + let retired = checked_cli( + "legacy-publisher-retirement", + &binary, + &candidate, + &[ + "graph", + "retire-recovered-publisher", + "--run-id", + &runs[0], + "--expect-owner", + &retained.owner, + "--json", + ], + deadline, + ); + assert_eq!(retired["publisher_retired"], true); + } + let restoration = graph::foreground::restore_selection(&candidate, &runs[0]).unwrap(); + selected_owner = start(0, Some(restoration["generation"].as_str().unwrap())); + ready(&mut selected_owner, &runs[0], deadline); + assert_eq!( + exec(&binary, &candidate, &runs[0], "web", "read-data", deadline)["exit_code"], + 0, + "selected marker must survive cleanup, retirement and same-run restore" + ); + let restored = snapshot(&candidate, &runs[0], deadline).receipt; + assert_eq!( + restored.resources["volume:data"].name, + original.resources["volume:data"].name + ); + assert_ne!( + restored.resources["container:web"].id, + original.resources["container:web"].id + ); + assert!(sibling_owner.poll().is_none()); + assert_eq!( + serde_json::to_vec(&snapshot(&candidate, &runs[1], deadline).receipt).unwrap(), + sibling + ); + assert_eq!( + exec(&binary, &candidate, &runs[1], "web", "read-data", deadline)["exit_code"], + 0 + ); + let selected_removed = checked_cli( + "remove-selected-data", + &binary, + &candidate, + &[ + "graph", + "cleanup", + "--run-id", + &runs[0], + "--remove-data", + "--json", + ], + deadline, + ); + assert_eq!(selected_removed["phase"], "removed"); + assert!(selected_owner.wait(deadline).success()); + selected_cleanup.done = true; + let removed = checked_cli( + "remove-sibling-data", + &binary, + &candidate, + &[ + "graph", + "cleanup", + "--run-id", + &runs[1], + "--remove-data", + "--json", + ], + deadline, + ); + assert_eq!(removed["phase"], "removed"); + assert!(sibling_owner.wait(deadline).success()); + sibling_cleanup.done = true; +} diff --git a/packages/runtime-core/src/provider/graph/foreground/transport.rs b/packages/runtime-core/src/provider/graph/foreground/transport.rs index 8ab4a43f..2064bfbc 100644 --- a/packages/runtime-core/src/provider/graph/foreground/transport.rs +++ b/packages/runtime-core/src/provider/graph/foreground/transport.rs @@ -1,5 +1,6 @@ use super::{Candidate, CandidateError, refused}; use crate::provider::{ + host_pin::DeviceRebind, identity::{self, ProcessIdentity}, state, }; @@ -239,6 +240,7 @@ fn selected_retirement_path( owner: &str, socket: bool, expected: (u64, u64), + device_rebind: Option, ) -> Result<(PathBuf, bool), CandidateError> { let original = root.join(if socket { "control.sock" } else { "owner.json" }); let retired = retired_path(root, owner, socket); @@ -247,8 +249,9 @@ fn selected_retirement_path( (None, Some(found)) => (retired, found, false), _ => return Err(retirement_refused()), }; - if id(¤t) != expected - || !private(¤t) + if !device_rebind.map_or(id(¤t) == expected, |v| { + v.matches(expected, id(¤t)) + }) || !private(¤t) || (socket && !current.file_type().is_socket()) || (!socket && (!current.is_file() || current.nlink() != 1 || current.len() > 8192)) { @@ -260,19 +263,26 @@ fn selected_retirement_path( fn verify_retirement( candidate: &Candidate, run: &str, - expected_owner: &str, - expected_receipt: &str, + expected: (&str, &str), root: &Path, lock: &state::Lock, intent: &Retirement, + device_rebind: Option, ) -> Result<(bool, bool), CandidateError> { + let root_id = id(&fs::symlink_metadata(root).map_err(|_| retirement_refused())?); + let lock_path_id = + id(&fs::symlink_metadata(root.join("operation.lock")).map_err(|_| retirement_refused())?); + let parent_matches = device_rebind.map_or(intent.parent == root_id, |v| { + v.matches(intent.parent, root_id) + }); if intent.version != 1 || intent.candidate != candidate.checkout || intent.run != run - || intent.owner_sha256 != expected_owner - || intent.receipt_sha256 != expected_receipt - || intent.parent != id(&fs::symlink_metadata(root).map_err(|_| retirement_refused())?) + || intent.owner_sha256 != expected.0 + || intent.receipt_sha256 != expected.1 + || !parent_matches || intent.lock != lock.identity().map_err(|_| retirement_refused())? + || lock_path_id != intent.lock || intent.owner.parent != intent.parent || intent.owner.socket != intent.socket || intent.owner.candidate != candidate.checkout @@ -282,9 +292,9 @@ fn verify_retirement( return Err(retirement_refused()); } let (socket_path, socket_original) = - selected_retirement_path(root, expected_owner, true, intent.socket)?; + selected_retirement_path(root, expected.0, true, intent.socket, device_rebind)?; let (record_path, record_original) = - selected_retirement_path(root, expected_owner, false, intent.record)?; + selected_retirement_path(root, expected.0, false, intent.record, None)?; // Socket retirement precedes record retirement. The inverse is foreign. if socket_original && !record_original { return Err(retirement_refused()); @@ -304,7 +314,7 @@ fn verify_retirement( .read_to_end(&mut bytes) .map_err(|_| retirement_refused())?; let record: Record = serde_json::from_slice(&bytes).map_err(|_| retirement_refused())?; - if record != intent.owner || format!("{:x}", Sha256::digest(&bytes)) != expected_owner { + if record != intent.owner || format!("{:x}", Sha256::digest(&bytes)) != expected.0 { return Err(retirement_refused()); } Ok((socket_original, record_original)) @@ -318,6 +328,15 @@ pub(in crate::provider::graph) fn retire_recovered_publisher( run: &str, expected_owner: &str, expected_receipt: &str, +) -> Result<(), CandidateError> { + retire_recovered_publisher_recovery(candidate, run, expected_owner, expected_receipt, None) +} +pub(in crate::provider::graph) fn retire_recovered_publisher_recovery( + candidate: &Candidate, + run: &str, + expected_owner: &str, + expected_receipt: &str, + device_rebind: Option, ) -> Result<(), CandidateError> { if !super::super::hex(expected_owner, 64) || !super::super::hex(expected_receipt, 64) { return Err(retirement_refused()); @@ -325,6 +344,24 @@ pub(in crate::provider::graph) fn retire_recovered_publisher( let root = root(candidate, run)?; state::check_private_directory(&root).map_err(|_| retirement_refused())?; let lock = state::Lock::acquire_existing(&root).map_err(|_| retirement_refused())?; + retire_recovered_publisher_locked( + candidate, + run, + expected_owner, + expected_receipt, + device_rebind, + &lock, + ) +} +pub(in crate::provider::graph) fn retire_recovered_publisher_locked( + candidate: &Candidate, + run: &str, + expected_owner: &str, + expected_receipt: &str, + device_rebind: Option, + lock: &state::Lock, +) -> Result<(), CandidateError> { + let root = root(candidate, run)?; let path = retirement_path(&root, expected_owner); let pending = path.with_extension("pending"); let name = pending @@ -335,7 +372,7 @@ pub(in crate::provider::graph) fn retire_recovered_publisher( let intent: Retirement = if metadata(&path)?.is_some() { state::read(&path).map_err(|_| retirement_refused())? } else { - let pin = Pin::read(candidate, run)?; + let pin = Pin::read_with_rebind(candidate, run, device_rebind)?; if format!("{:x}", Sha256::digest(&pin.bytes)) != expected_owner || identity::alive(pin.record.process.pid).unwrap_or(true) { @@ -360,11 +397,11 @@ pub(in crate::provider::graph) fn retire_recovered_publisher( let (socket_original, record_original) = verify_retirement( candidate, run, - expected_owner, - expected_receipt, + (expected_owner, expected_receipt), &root, - &lock, + lock, &intent, + device_rebind, )?; if socket_original { fs::rename( @@ -380,11 +417,11 @@ pub(in crate::provider::graph) fn retire_recovered_publisher( let _ = verify_retirement( candidate, run, - expected_owner, - expected_receipt, + (expected_owner, expected_receipt), &root, - &lock, + lock, &intent, + device_rebind, )?; fs::rename( root.join("owner.json"), @@ -398,11 +435,11 @@ pub(in crate::provider::graph) fn retire_recovered_publisher( let (socket_original, record_original) = verify_retirement( candidate, run, - expected_owner, - expected_receipt, + (expected_owner, expected_receipt), &root, - &lock, + lock, &intent, + device_rebind, )?; if socket_original || record_original { return Err(retirement_refused()); @@ -413,11 +450,27 @@ pub(in crate::provider::graph) fn retire_recovered_publisher( /** A later boot may resume frontend recovery only after retirement was fully * recorded. This read-only proof never begins or completes a partial move. */ +#[cfg(test)] pub(in crate::provider::graph) fn verify_recovered_publisher_retired( candidate: &Candidate, run: &str, expected_owner: &str, expected_receipt: &str, +) -> Result<(), CandidateError> { + verify_recovered_publisher_retired_recovery( + candidate, + run, + expected_owner, + expected_receipt, + None, + ) +} +pub(in crate::provider::graph) fn verify_recovered_publisher_retired_recovery( + candidate: &Candidate, + run: &str, + expected_owner: &str, + expected_receipt: &str, + device_rebind: Option, ) -> Result<(), CandidateError> { if !super::super::hex(expected_owner, 64) || !super::super::hex(expected_receipt, 64) { return Err(retirement_refused()); @@ -430,11 +483,11 @@ pub(in crate::provider::graph) fn verify_recovered_publisher_retired( let (socket_original, record_original) = verify_retirement( candidate, run, - expected_owner, - expected_receipt, + (expected_owner, expected_receipt), &root, &lock, &intent, + device_rebind, )?; if socket_original || record_original { return Err(retirement_refused()); @@ -485,11 +538,12 @@ fn peer(stream: &UnixStream) -> Result { identity::verify(&observed, &observed, &observed.executable, uid).map_err(|_| refused())?; Ok(observed) } -pub(super) struct Pin { +pub(in crate::provider::graph) struct Pin { root: PathBuf, record: Record, bytes: Vec, record_id: (u64, u64), + device_rebind: Option, } impl Pin { pub fn load(candidate: &Candidate, run: &str) -> Result { @@ -498,6 +552,13 @@ impl Pin { Ok(pin) } fn read(candidate: &Candidate, run: &str) -> Result { + Self::read_with_rebind(candidate, run, None) + } + fn read_with_rebind( + candidate: &Candidate, + run: &str, + device_rebind: Option, + ) -> Result { let root = root(candidate, run)?; state::check_private_directory(&root).map_err(|_| refused())?; let mut file = OpenOptions::new() @@ -530,6 +591,7 @@ impl Pin { record, bytes, record_id: id(&metadata), + device_rebind, }; pin.verify_files()?; Ok(pin) @@ -540,10 +602,10 @@ impl Pin { let socket = fs::symlink_metadata(self.root.join("control.sock")).map_err(|_| refused())?; let file = fs::symlink_metadata(self.root.join("owner.json")).map_err(|_| refused())?; if !parent.is_dir() - || id(&parent) != self.record.parent + || !self.matches_pin(self.record.parent, id(&parent)) || !socket.file_type().is_socket() || !private(&socket) - || id(&socket) != self.record.socket + || !self.matches_pin(self.record.socket, id(&socket)) || !file.is_file() || !private(&file) || file.nlink() != 1 @@ -570,6 +632,46 @@ impl Pin { } Ok(()) } + fn matches_pin(&self, recorded: (u64, u64), observed: (u64, u64)) -> bool { + self.device_rebind + .map_or(recorded == observed, |v| v.matches(recorded, observed)) + } + pub(in crate::provider::graph) fn legacy_summary( + candidate: &Candidate, + run: &str, + device_rebind: DeviceRebind, + host_boot_micros: u64, + ) -> Result { + let pin = Self::read_with_rebind(candidate, run, Some(device_rebind))?; + // SAFETY: geteuid has no arguments or side effects. + identity::verify( + &pin.record.process, + &pin.record.process, + &pin.record.process.executable, + unsafe { libc::geteuid() }, + )?; + device_rebind.definitely_dead_before_boot(&pin.record.process, host_boot_micros)?; + no_listener(&pin.root.join("control.sock"))?; + Ok(LegacyPublisher { + owner_sha256: format!("{:x}", Sha256::digest(&pin.bytes)), + parent: pin.record.parent, + socket: pin.record.socket, + record: pin.record_id, + process: pin.record.process, + }) + } + pub(in crate::provider::graph) fn legacy_recorded_device( + candidate: &Candidate, + run: &str, + ) -> Result { + let root = root(candidate, run)?; + state::check_private_directory(&root)?; + let record: Record = state::read_bounded(&root.join("owner.json"), 8192)?; + if record.version != 1 || record.candidate != candidate.checkout || record.run != run { + return Err(refused()); + } + Ok(record.parent.0) + } pub fn verify(&self) -> Result<(), CandidateError> { self.verify_files()?; let expected = &self.record.process; @@ -647,6 +749,16 @@ impl Pin { Ok(stream) } } + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(in crate::provider::graph) struct LegacyPublisher { + pub owner_sha256: String, + pub parent: (u64, u64), + pub socket: (u64, u64), + pub record: (u64, u64), + pub process: ProcessIdentity, +} pub(super) struct Publication { listener: UnixListener, pin: Pin, @@ -706,6 +818,7 @@ impl Publication { record, bytes, record_id, + device_rebind: None, }; pin.verify()?; Ok(Self { @@ -980,6 +1093,16 @@ impl Retired { run: &str, owner_sha256: &str, complete_sha256: &str, + ) -> Result<(), CandidateError> { + self.verify_recovery_with_rebind(candidate, run, owner_sha256, complete_sha256, None) + } + pub fn verify_recovery_with_rebind( + &self, + candidate: &Candidate, + run: &str, + owner_sha256: &str, + complete_sha256: &str, + device_rebind: Option, ) -> Result<(), CandidateError> { self.verify()?; if !super::super::hex(owner_sha256, 64) @@ -996,11 +1119,11 @@ impl Retired { let (socket_original, record_original) = verify_retirement( candidate, run, - owner_sha256, - complete_sha256, + (owner_sha256, complete_sha256), &self.root, &self._lock, &intent, + device_rebind, )?; if socket_original || record_original { return Err(retirement_refused()); @@ -1018,16 +1141,39 @@ impl DeadOwner { pub(in crate::provider::graph) fn acquire( candidate: &Candidate, run: &str, + ) -> Result { + Self::acquire_recovery(candidate, run, None, None) + } + pub(in crate::provider::graph) fn acquire_recovery( + candidate: &Candidate, + run: &str, + device_rebind: Option, + host_boot_micros: Option, ) -> Result { let lock = state::Lock::acquire_existing(&root(candidate, run)?)?; let value = Self { - pin: Pin::read(candidate, run)?, + pin: Pin::read_with_rebind(candidate, run, device_rebind)?, _lock: lock, }; + if let Some(rebind) = device_rebind { + rebind.definitely_dead_before_boot( + &value.pin.record.process, + host_boot_micros.ok_or_else(refused)?, + )?; + } value.verify()?; Ok(value) } pub(in crate::provider::graph) fn verify(&self) -> Result<(), CandidateError> { + let pathname = + fs::symlink_metadata(self.pin.root.join("operation.lock")).map_err(|_| refused())?; + if !pathname.is_file() + || pathname.nlink() != 1 + || !private(&pathname) + || id(&pathname) != self._lock.identity()? + { + return Err(refused()); + } self.pin.verify_files()?; if identity::alive(self.pin.record.process.pid)? { return Err(refused()); diff --git a/packages/runtime-core/src/provider/graph/foreground/transport_tests.rs b/packages/runtime-core/src/provider/graph/foreground/transport_tests.rs index 5b21b6a1..b81f11ee 100644 --- a/packages/runtime-core/src/provider/graph/foreground/transport_tests.rs +++ b/packages/runtime-core/src/provider/graph/foreground/transport_tests.rs @@ -201,6 +201,78 @@ fn interrupted_socket_move_resumes_but_replaced_owner_refuses() { fs::remove_dir_all(root).unwrap(); } +#[cfg(target_os = "macos")] +#[test] +fn selected_legacy_device_socket_move_resumes_only_exact_retirement() { + let (_fixture, candidate, run, _original_owner, root) = abandoned_publisher(); + let receipt = "f".repeat(64); + let current = id(&fs::symlink_metadata(&root).unwrap()).0; + let rebind = DeviceRebind { + old: current.checked_add(1).unwrap(), + current, + }; + let mut record: Record = state::read(&root.join("owner.json")).unwrap(); + assert_eq!(record.parent.0, current); + assert_eq!(record.socket.0, current); + record.parent.0 = rebind.old; + record.socket.0 = rebind.old; + let bytes = serde_json::to_vec(&record).unwrap(); + fs::write(root.join("owner.json"), &bytes).unwrap(); + let owner = format!("{:x}", Sha256::digest(&bytes)); + assert!(Pin::read(&candidate, &run).is_err()); + let lock = state::Lock::acquire_existing(&root).unwrap(); + let pin = Pin::read_with_rebind(&candidate, &run, Some(rebind)).unwrap(); + let intent = Retirement { + version: 1, + candidate: candidate.checkout.clone(), + run: run.clone(), + receipt_sha256: receipt.clone(), + owner_sha256: owner.clone(), + parent: pin.record.parent, + lock: lock.identity().unwrap(), + socket: pin.record.socket, + record: pin.record_id, + owner: pin.record, + }; + state::write(&retirement_path(&root, &owner), &intent).unwrap(); + fs::rename(root.join("control.sock"), retired_path(&root, &owner, true)).unwrap(); + drop(lock); + assert!(retire_recovered_publisher(&candidate, &run, &owner, &receipt).is_err()); + retire_recovered_publisher_recovery(&candidate, &run, &owner, &receipt, Some(rebind)).unwrap(); + let retired = Retired::acquire(&candidate, &run).unwrap().unwrap(); + retired + .verify_recovery_with_rebind(&candidate, &run, &owner, &receipt, Some(rebind)) + .unwrap(); + assert!( + retired + .verify_recovery(&candidate, &run, &owner, &receipt) + .is_err() + ); + drop(retired); + fs::remove_dir_all(root).unwrap(); +} + +#[cfg(target_os = "macos")] +#[test] +fn dead_owner_guard_refuses_replaced_foreground_lock_path() { + let (_fixture, candidate, run, _owner, root) = abandoned_publisher(); + let dead = DeadOwner::acquire(&candidate, &run).unwrap(); + dead.verify().unwrap(); + let pathname = root.join("operation.lock"); + let saved = root.join("held-operation.lock"); + fs::rename(&pathname, &saved).unwrap(); + fs::write(&pathname, b"replacement").unwrap(); + fs::set_permissions(&pathname, fs::Permissions::from_mode(0o600)).unwrap(); + let replacement = id(&fs::symlink_metadata(&pathname).unwrap()); + assert!(dead.verify().is_err()); + assert_eq!(id(&fs::symlink_metadata(&pathname).unwrap()), replacement); + assert_eq!(fs::read(&pathname).unwrap(), b"replacement"); + drop(dead); + fs::remove_file(&pathname).unwrap(); + fs::rename(&saved, &pathname).unwrap(); + fs::remove_dir_all(root).unwrap(); +} + #[cfg(target_os = "macos")] #[test] fn replaced_socket_or_inherited_listener_refuses_retirement() { diff --git a/packages/runtime-core/src/provider/graph/host_pin_recovery.rs b/packages/runtime-core/src/provider/graph/host_pin_recovery.rs new file mode 100644 index 00000000..a41c94b3 --- /dev/null +++ b/packages/runtime-core/src/provider/graph/host_pin_recovery.rs @@ -0,0 +1,704 @@ +//! Explicit, selected legacy host-pin migration for one retained graph run. +//! +//! The private witness precedes dead-owner cleanup. It never edits historical graph, +//! publisher, control, or dependency receipts and grants no ordinary replay authority. +use super::{Candidate, CandidateError, Engine, Kind, Receipt, directory, foreground, load, state}; +use crate::provider::{ + ProjectShareIntent, + host_pin::DeviceRebind, + lifecycle, + relay_owner::publication::{LegacyControl, PinnedEndpoint}, +}; +use serde::{Deserialize, Serialize}; +use serde_json::{Value, json}; +use sha2::{Digest, Sha256}; +use std::{ + collections::BTreeMap, + fs::{self, OpenOptions}, + io::{Read, Seek, SeekFrom}, + os::unix::fs::{MetadataExt, OpenOptionsExt}, + path::{Path, PathBuf}, +}; + +use super::{dependency_slots::LegacyReservation, foreground::transport::LegacyPublisher}; + +const FILE: &str = "host-pin-recovery.json"; +const LIMIT: u64 = 64 * 1024; +const GUEST_IDENTITY_ACK: &str = "host-pin-guest-identity-v1\n"; + +fn refused() -> CandidateError { + CandidateError::new( + "graph_host_pin_recovery", + "Legacy host pin selection is incomplete or changed; receipts and data were preserved.", + ) +} +fn digest(bytes: &[u8]) -> String { + format!("{:x}", Sha256::digest(bytes)) +} + +/// Guest::boot_id is sourced from the host Owner receipt. Execute a read-only +/// guest check so a changed receipt cannot impersonate the running kernel boot. +/// execute_cleanup itself compares /proc boot ID and /storage owner under the +/// held Engine/Guest lease before this fixed acknowledgement is emitted. +fn verify_guest_identity(engine: &Engine<'_>) -> Result<(), CandidateError> { + let acknowledged = engine + .guest() + .execute_cleanup("printf 'host-pin-guest-identity-v1\\n'", &[]) + .map_err(|_| refused())?; + if acknowledged != GUEST_IDENTITY_ACK { + return Err(refused()); + } + Ok(()) +} +fn read_raw(path: &Path, limit: u64) -> Result, CandidateError> { + read_raw_with(path, limit, || {}) +} +fn read_raw_with( + path: &Path, + limit: u64, + after_read: impl FnOnce(), +) -> Result, CandidateError> { + let parent = path.parent().ok_or_else(refused)?; + state::check_private_directory(parent)?; + let mut file = OpenOptions::new() + .read(true) + .custom_flags(libc::O_NOFOLLOW | libc::O_NONBLOCK) + .open(path) + .map_err(|_| refused())?; + let metadata = file.metadata().map_err(|_| refused())?; + // SAFETY: geteuid has no arguments or side effects. + if !metadata.is_file() + || metadata.nlink() != 1 + || metadata.mode() & 0o7777 != 0o600 + || metadata.uid() != unsafe { libc::geteuid() } + || metadata.len() == 0 + || metadata.len() > limit + { + return Err(refused()); + } + let mut bytes = Vec::new(); + file.by_ref() + .take(limit + 1) + .read_to_end(&mut bytes) + .map_err(|_| refused())?; + after_read(); + file.seek(SeekFrom::Start(0)).map_err(|_| refused())?; + let mut confirmation = Vec::new(); + file.by_ref() + .take(limit + 1) + .read_to_end(&mut confirmation) + .map_err(|_| refused())?; + let retained = file.metadata().map_err(|_| refused())?; + let pathname = fs::symlink_metadata(path).map_err(|_| refused())?; + if bytes != confirmation + || bytes.len() as u64 != metadata.len() + || [retained.dev(), pathname.dev()] != [metadata.dev(); 2] + || [retained.ino(), pathname.ino()] != [metadata.ino(); 2] + || [retained.len(), pathname.len()] != [metadata.len(); 2] + || [retained.uid(), pathname.uid()] != [metadata.uid(); 2] + || [retained.mode(), pathname.mode()] != [metadata.mode(); 2] + || [retained.nlink(), pathname.nlink()] != [metadata.nlink(); 2] + || [retained.mtime(), pathname.mtime()] != [metadata.mtime(); 2] + || [retained.mtime_nsec(), pathname.mtime_nsec()] != [metadata.mtime_nsec(); 2] + || [retained.ctime(), pathname.ctime()] != [metadata.ctime(); 2] + || [retained.ctime_nsec(), pathname.ctime_nsec()] != [metadata.ctime_nsec(); 2] + { + return Err(refused()); + } + Ok(bytes) +} +fn absent(path: &Path) -> Result { + match fs::symlink_metadata(path) { + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(true), + _ => Err(refused()), + } +} + +fn exact_lock_path(root: &Path, held: &state::Lock) -> Result<(), CandidateError> { + let pathname = fs::symlink_metadata(root.join("operation.lock")).map_err(|_| refused())?; + if !pathname.is_file() + || pathname.nlink() != 1 + || (pathname.dev(), pathname.ino()) != held.identity()? + { + return Err(refused()); + } + Ok(()) +} + +#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Witness { + version: u8, + candidate: PathBuf, + run: String, + owner: String, + namespace: String, + plan: String, + graph_sha256: String, + provider_sha256: String, + host_boot_micros: u64, + previous_guest_boot: String, + current_guest_boot: String, + rebind: DeviceRebind, + publisher: LegacyPublisher, + control_root: PathBuf, + control: LegacyControl, + reservation: Option, + source_shared: Option, + retained_volumes: BTreeMap, + qualification: String, +} + +impl Witness { + pub(super) fn rebind(&self) -> DeviceRebind { + self.rebind + } + pub(super) fn host_boot_micros(&self) -> u64 { + self.host_boot_micros + } + pub(super) fn publisher_sha256(&self) -> &str { + &self.publisher.owner_sha256 + } + pub(super) fn reservation(&self) -> Option<&LegacyReservation> { + self.reservation.as_ref() + } + pub(super) fn control(&self) -> &LegacyControl { + &self.control + } + pub(super) fn control_root(&self) -> &Path { + &self.control_root + } + pub(super) fn graph_sha256(&self) -> &str { + &self.graph_sha256 + } + pub(super) fn matches_graph(&self, receipt: &Receipt) -> bool { + self.run == receipt.run + && self.owner == receipt.owner + && self.namespace == receipt.namespace + && self.plan == receipt.plan_id + && self.source_shared == receipt.source.as_ref().and_then(|s| s.shared.clone()) + } +} + +pub(super) struct Guard { + witness: Witness, + witness_path: PathBuf, + control_root: PathBuf, + control_lock: state::Lock, +} + +pub(super) struct CleanupProof<'a> { + pub original: &'a Receipt, + pub sha256: &'a str, + pub current_is_original: bool, + pub allow_absent_reservation: bool, + pub publisher_may_be_partial: bool, +} +impl Guard { + pub(super) fn witness(&self) -> &Witness { + &self.witness + } + pub(super) fn verify_lock(&self) -> Result<(), CandidateError> { + let retained: Witness = + serde_json::from_slice(&read_raw(&self.witness_path, LIMIT)?).map_err(|_| refused())?; + if retained != self.witness { + return Err(refused()); + } + let context = + super::host_relay::context(&self.witness.owner, &self.witness.previous_guest_boot)?; + let control = PinnedEndpoint::load_legacy_recovery( + &self.witness.control_root, + context, + self.witness.rebind, + self.witness.host_boot_micros, + )?; + if control.legacy_summary() != self.witness.control { + return Err(refused()); + } + exact_lock_path(&self.control_root, &self.control_lock) + } +} + +fn path(candidate: &Candidate, run: &str) -> Result { + Ok(directory(candidate, run)?.join(FILE)) +} +pub(super) fn load_witness( + candidate: &Candidate, + run: &str, +) -> Result, CandidateError> { + let file = path(candidate, run)?; + absent(&file.with_extension("pending"))?; + if absent(&file).is_ok() { + return Ok(None); + } + let bytes = read_raw(&file, LIMIT)?; + let witness: Witness = serde_json::from_slice(&bytes).map_err(|_| refused())?; + if witness.version != 1 || witness.candidate != candidate.checkout || witness.run != run { + return Err(refused()); + } + Ok(Some(witness)) +} + +/// Only the exact old publisher may select the historical overlay. A successor +/// with current-device pins uses ordinary strict recovery despite retained history. +pub(super) fn selected_for_old_publisher( + candidate: &Candidate, + run: &str, +) -> Result, CandidateError> { + let Some(witness) = load_witness(candidate, run)? else { + return Ok(None); + }; + let recorded = foreground::transport::Pin::legacy_recorded_device(candidate, run)?; + if recorded == witness.rebind.current { + return Ok(None); + } + if recorded != witness.rebind.old { + return Err(refused()); + } + Ok(Some(witness)) +} + +fn verify_volume_projections( + engine: &Engine<'_>, + receipt: &Receipt, + selected: &BTreeMap, +) -> Result<(), CandidateError> { + let keys = receipt + .resources + .iter() + .filter(|(_, value)| value.kind == Kind::Volume) + .map(|(key, _)| key.clone()) + .collect::>(); + if keys.len() != selected.len() || keys.iter().any(|key| !selected.contains_key(key)) { + return Err(refused()); + } + for key in keys { + let resource = &receipt.resources[&key]; + let actual = super::inspect_resource(engine, receipt, resource)?.ok_or_else(refused)?; + let expected = selected.get(&key).ok_or_else(refused)?; + if expected["expected_name"] != resource.name + || expected["observed_name"] != actual["Name"] + || expected["observed_created_at"] != actual["CreatedAt"] + || expected["observed_mountpoint"] != actual["Mountpoint"] + || expected["observed_driver"] != actual["Driver"] + || expected["observed_labels_sha256"] + != digest(&serde_json::to_vec(&actual["Labels"]).map_err(|_| refused())?) + { + return Err(refused()); + } + let labels = expected["labels"].as_object().ok_or_else(refused)?; + if labels + .iter() + .any(|(name, value)| actual["Labels"].get(name) != Some(value)) + { + return Err(refused()); + } + } + Ok(()) +} + +/// Caller already holds the Engine lease and exact foreground owner lock. +/// The returned control lock remains held across graph cleanup effects. +pub(super) fn acquire_for_cleanup( + candidate: &Candidate, + run: &str, + engine: &Engine<'_>, + proof: CleanupProof<'_>, + witness: Witness, +) -> Result { + lifecycle::host_filesystem::no_auxiliary_update(candidate)?; + if witness.graph_sha256 != proof.sha256 + || !witness.matches_graph(proof.original) + || witness.candidate != candidate.checkout + || witness.run != run + || witness.qualification + != "explicit-legacy-device-rebind-original-volume-continuity-unproven" + || lifecycle::host_filesystem::host_boot_micros()? != witness.host_boot_micros + { + return Err(refused()); + } + witness + .rebind + .definitely_dead_before_boot(&witness.publisher.process, witness.host_boot_micros)?; + if proof.current_is_original + && digest(&read_raw( + &directory(candidate, run)?.join("state.json"), + 2 * 1024 * 1024, + )?) != witness.graph_sha256 + { + return Err(refused()); + } + let owner = state::Owner::load(candidate)?; + lifecycle::verify_disks(candidate, &owner)?; + verify_guest_identity(engine)?; + let bytes = read_raw( + &candidate.state_root.join("run/smolvm/owner.json"), + 1024 * 1024, + )?; + if digest(&bytes) != witness.provider_sha256 + || owner.token != witness.owner + || owner.guest_boot_id.as_deref() != Some(&witness.current_guest_boot) + || owner.previous_guest_boot_id.as_deref() != Some(&witness.previous_guest_boot) + || engine.guest().incarnation() != witness.owner + || engine.guest().boot_id() != witness.current_guest_boot + || owner + .storage + .as_ref() + .is_none_or(|disk| disk.device != witness.rebind.current) + || owner + .overlay + .as_ref() + .is_none_or(|disk| disk.device != witness.rebind.current) + { + return Err(refused()); + } + if !proof.publisher_may_be_partial { + let publisher = foreground::transport::Pin::legacy_summary( + candidate, + run, + witness.rebind, + witness.host_boot_micros, + )?; + if publisher != witness.publisher { + return Err(refused()); + } + } + let control_root = witness.control_root.join("relay-control"); + let control_lock = state::Lock::acquire_existing(&control_root)?; + absent(&foreground::transport::root(candidate, run)?.join("owner.pending"))?; + absent(&control_root.join("owner.pending"))?; + let context = super::host_relay::context(&witness.owner, &witness.previous_guest_boot)?; + let control = PinnedEndpoint::load_legacy_recovery( + &witness.control_root, + context, + witness.rebind, + witness.host_boot_micros, + )? + .legacy_summary(); + if control != witness.control { + return Err(refused()); + } + if let Some(selected) = &witness.reservation { + super::dependency_slots::verify_legacy_remaining( + candidate, + run, + witness.rebind, + selected, + proof.allow_absent_reservation, + )?; + } else if super::dependency_slots::inspect_legacy( + candidate, + run, + witness.rebind, + witness.host_boot_micros, + )? + .is_some() + { + return Err(refused()); + } + verify_volume_projections(engine, proof.original, &witness.retained_volumes)?; + let guard = Guard { + witness_path: path(candidate, run)?, + witness, + control_root, + control_lock, + }; + guard.verify_lock()?; + Ok(guard) +} + +struct Selected { + witness: Witness, + foreground_root: PathBuf, + foreground_lock: state::Lock, + control_lock_root: PathBuf, + control_lock: state::Lock, +} + +impl Selected { + fn verify( + &self, + candidate: &Candidate, + run: &str, + engine: &Engine<'_>, + ) -> Result<(), CandidateError> { + for (root, lock) in [ + (&self.foreground_root, &self.foreground_lock), + (&self.control_lock_root, &self.control_lock), + ] { + exact_lock_path(root, lock)?; + } + if self.foreground_root != foreground::transport::root(candidate, run)? + || self.control_lock_root != self.witness.control_root.join("relay-control") + || self.witness != select_content(candidate, run, engine)? + { + return Err(refused()); + } + Ok(()) + } +} + +fn select( + candidate: &Candidate, + run: &str, + engine: &Engine<'_>, +) -> Result { + let (receipt, _) = load(candidate, engine, run)?; + let startup = receipt.relay_startup.as_ref().ok_or_else(refused)?; + let foreground_root = foreground::transport::root(candidate, run)?; + let foreground_lock = state::Lock::acquire_existing(&foreground_root)?; + let control_lock_root = startup.control_root.join("relay-control"); + let control_lock = state::Lock::acquire_existing(&control_lock_root)?; + let selected = Selected { + witness: select_content(candidate, run, engine)?, + foreground_root, + foreground_lock, + control_lock_root, + control_lock, + }; + selected.verify(candidate, run, engine)?; + Ok(selected) +} + +fn select_content( + candidate: &Candidate, + run: &str, + engine: &Engine<'_>, +) -> Result { + lifecycle::host_filesystem::no_auxiliary_update(candidate)?; + let owner = state::Owner::load(candidate)?; + lifecycle::verify_disks(candidate, &owner)?; + verify_guest_identity(engine)?; + let host_boot_micros = lifecycle::host_filesystem::host_boot_micros()?; + let current_guest_boot = owner.guest_boot_id.clone().ok_or_else(refused)?; + let previous_guest_boot = owner.previous_guest_boot_id.clone().ok_or_else(refused)?; + if current_guest_boot == previous_guest_boot + || current_guest_boot != engine.guest().boot_id() + || owner.token != engine.guest().incarnation() + { + return Err(refused()); + } + let (receipt, root) = load(candidate, engine, run)?; + for name in [ + "state.pending", + "dead-owner-cleanup.pending", + "relay-cleanup-bridges.pending", + ] { + absent(&root.join(name))?; + } + if receipt.phase != "ready-observed" || receipt.relay_cleanup.is_some() { + return Err(refused()); + } + let startup = receipt.relay_startup.as_ref().ok_or_else(refused)?; + let graph_sha256 = digest(&read_raw(&root.join("state.json"), 2 * 1024 * 1024)?); + let provider_sha256 = digest(&read_raw( + &candidate.state_root.join("run/smolvm/owner.json"), + 1024 * 1024, + )?); + let publisher_root = foreground::transport::root(candidate, run)?; + absent(&publisher_root.join("owner.pending"))?; + absent(&startup.control_root.join("relay-control/owner.pending"))?; + let current_device = fs::symlink_metadata(&publisher_root) + .map_err(|_| refused())? + .dev(); + let rebind = DeviceRebind { + old: foreground::transport::Pin::legacy_recorded_device(candidate, run)?, + current: current_device, + }; + if rebind.old == rebind.current { + return Err(refused()); + } + if owner + .storage + .as_ref() + .is_none_or(|disk| disk.device != rebind.current) + || owner + .overlay + .as_ref() + .is_none_or(|disk| disk.device != rebind.current) + { + return Err(refused()); + } + let publisher = + foreground::transport::Pin::legacy_summary(candidate, run, rebind, host_boot_micros)?; + let context = super::host_relay::context(&receipt.owner, &previous_guest_boot)?; + let control = PinnedEndpoint::load_legacy_recovery( + &startup.control_root, + context, + rebind, + host_boot_micros, + )? + .legacy_summary(); + let reservation = + super::dependency_slots::inspect_legacy(candidate, run, rebind, host_boot_micros)?; + if receipt + .source + .as_ref() + .and_then(|source| source.shared.as_ref()) + .is_some_and(|shared| shared.device != rebind.old) + || owner.project_share.as_ref() + != receipt + .source + .as_ref() + .and_then(|s| s.shared.as_ref()) + .map(|shared| { + // Historical shared source is pinned to the old host device. Only the + // selected device may differ from the currently approved project share. + let mut expected = shared.clone(); + expected.device = rebind.current; + expected + }) + .as_ref() + { + return Err(refused()); + } + if let Some(share) = &owner.project_share { + share.validate()?; + } + let mut retained_volumes = BTreeMap::new(); + for (key, resource) in receipt + .resources + .iter() + .filter(|(_, value)| value.kind == Kind::Volume) + { + let actual = super::inspect_resource(engine, &receipt, resource)?.ok_or_else(refused)?; + let labels = super::expected_labels(&receipt, resource) + .as_object() + .ok_or_else(refused)? + .keys() + .map(|label| (label.clone(), actual["Labels"][label].clone())) + .collect::>(); + retained_volumes.insert( + key.clone(), + json!({"expected_name":resource.name,"observed_name":actual["Name"], + "observed_created_at":actual["CreatedAt"], + "observed_mountpoint":actual["Mountpoint"], + "observed_driver":actual["Driver"], + "observed_labels_sha256":digest(&serde_json::to_vec(&actual["Labels"]).map_err(|_| refused())?), + "labels":labels}), + ); + } + Ok(Witness { + version: 1, + candidate: candidate.checkout.clone(), + run: run.into(), + owner: receipt.owner, + namespace: receipt.namespace, + plan: receipt.plan_id, + graph_sha256, + provider_sha256, + host_boot_micros, + previous_guest_boot, + current_guest_boot, + rebind, + publisher, + control_root: startup.control_root.clone(), + control, + reservation, + source_shared: receipt.source.and_then(|s| s.shared), + retained_volumes, + qualification: "explicit-legacy-device-rebind-original-volume-continuity-unproven".into(), + }) +} + +fn selected_sha256(witness: &Witness) -> Result { + Ok(digest(&serde_json::to_vec(witness).map_err(|_| refused())?)) +} + +/// Read-only exact selection. Existing completed recovery history is retained. +pub fn inspect(candidate: &Candidate, run: &str) -> Result { + let engine = Engine::connect_cleanup_wait(candidate)?; + let selected = select(candidate, run, &engine)?; + Ok( + json!({"run":run,"selection_sha256":selected_sha256(&selected.witness)?,"old_device":selected.witness.rebind.old,"new_device":selected.witness.rebind.current,"qualification":selected.witness.qualification}), + ) +} + +/// Explicitly publish one private immutable witness before cleanup can consume an overlay. +pub fn recover(candidate: &Candidate, run: &str, expected: &str) -> Result { + if expected.len() != 64 || !expected.bytes().all(|b| b.is_ascii_hexdigit()) { + return Err(refused()); + } + let engine = Engine::connect_cleanup_wait(candidate)?; + let selected = select(candidate, run, &engine)?; + if selected_sha256(&selected.witness)? != expected { + return Err(refused()); + } + if let Some(existing) = load_witness(candidate, run)? { + if existing == selected.witness { + return Ok( + json!({"run":run,"selection_sha256":expected,"witness_published":true,"already_published":true,"qualification":selected.witness.qualification}), + ); + } + return Err(refused()); + } + selected.verify(candidate, run, &engine)?; + state::write(&path(candidate, run)?, &selected.witness)?; + Ok( + json!({"run":run,"selection_sha256":expected,"witness_published":true,"qualification":selected.witness.qualification}), + ) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::{io::Write, os::unix::fs::PermissionsExt}; + + #[test] + fn raw_selection_refuses_same_inode_in_place_substitution() { + let fixture = super::super::tests::Fixture::new(); + let path = fixture.0.join("selected.json"); + fs::write(&path, b"first").unwrap(); + fs::set_permissions(&path, fs::Permissions::from_mode(0o600)).unwrap(); + let before = fs::symlink_metadata(&path).unwrap(); + let result = read_raw_with(&path, 64, || { + let mut file = OpenOptions::new().write(true).open(&path).unwrap(); + file.write_all(b"other").unwrap(); + file.sync_all().unwrap(); + }); + assert!(result.is_err()); + assert_eq!(fs::symlink_metadata(&path).unwrap().ino(), before.ino()); + } + + #[test] + fn device_overlay_requires_only_one_number_change_and_exact_inode() { + let translated = DeviceRebind { + old: 10, + current: 20, + }; + assert!(translated.matches((10, 77), (20, 77))); + for (old, current) in [ + ((20, 77), (20, 77)), + ((10, 77), (10, 77)), + ((10, 77), (20, 78)), + ((10, 0), (20, 0)), + ] { + assert!(!translated.matches(old, current)); + } + } + + #[test] + fn held_lock_path_replacement_refuses_and_preserves_foreign_replacement() { + let fixture = super::super::tests::Fixture::new(); + let lock = state::Lock::acquire(&fixture.0).unwrap(); + exact_lock_path(&fixture.0, &lock).unwrap(); + let pathname = fixture.0.join("operation.lock"); + let saved = fixture.0.join("held-operation.lock"); + fs::rename(&pathname, &saved).unwrap(); + fs::write(&pathname, b"foreign replacement").unwrap(); + fs::set_permissions(&pathname, fs::Permissions::from_mode(0o600)).unwrap(); + let replacement = fs::symlink_metadata(&pathname).unwrap(); + assert!(exact_lock_path(&fixture.0, &lock).is_err()); + let after = fs::symlink_metadata(&pathname).unwrap(); + assert_eq!( + (after.dev(), after.ino()), + (replacement.dev(), replacement.ino()) + ); + assert_eq!(fs::read(&pathname).unwrap(), b"foreign replacement"); + drop(lock); + // The fixture is private and isolated; restore the original pathname + // before its directory is torn down. + fs::remove_file(&pathname).unwrap(); + fs::rename(&saved, &pathname).unwrap(); + } +} diff --git a/packages/runtime-core/src/provider/graph/host_relay.rs b/packages/runtime-core/src/provider/graph/host_relay.rs index 94a5145d..49946e7f 100644 --- a/packages/runtime-core/src/provider/graph/host_relay.rs +++ b/packages/runtime-core/src/provider/graph/host_relay.rs @@ -489,6 +489,25 @@ pub(super) fn inspect_cleanup( remove_data: bool, environment: &super::super::environment_recovery::GraphInventory, bridge_selection: &bridges::cleanup::Selection, +) -> Result<[u8; 32], CandidateError> { + inspect_cleanup_recovery( + candidate, + engine, + expected, + remove_data, + environment, + bridge_selection, + None, + ) +} +pub(super) fn inspect_cleanup_recovery( + candidate: &Candidate, + engine: &Engine<'_>, + expected: &Receipt, + remove_data: bool, + environment: &super::super::environment_recovery::GraphInventory, + bridge_selection: &bridges::cleanup::Selection, + host_pin: Option<&super::host_pin_recovery::Witness>, ) -> Result<[u8; 32], CandidateError> { let (receipt, root) = archive::load_confirmation(candidate, engine, &expected.run)?; match fs::symlink_metadata(root.join("state.pending")) { @@ -533,10 +552,10 @@ pub(super) fn inspect_cleanup( require_retained_volume(resource, value.is_some(), remove_data)?; observations.insert(key, value); } - if &bridges::cleanup::read(engine, &receipt, &root)? != bridge_selection { + if &bridges::cleanup::read_recovery(engine, &receipt, &root, host_pin)? != bridge_selection { return Err(refused()); } - bridges::cleanup::verify(candidate, engine, &receipt, bridge_selection)?; + bridges::cleanup::verify_recovery(candidate, engine, &receipt, bridge_selection, host_pin)?; environment::verify_cleanup(candidate, engine, &receipt, &root, environment)?; probes::verify_cleanup(engine, &receipt)?; startup::verify_cleanup(engine, &receipt)?; diff --git a/packages/runtime-core/src/provider/graph/live_owner_cleanup.rs b/packages/runtime-core/src/provider/graph/live_owner_cleanup.rs index 33f1aa01..d8b89dc6 100644 --- a/packages/runtime-core/src/provider/graph/live_owner_cleanup.rs +++ b/packages/runtime-core/src/provider/graph/live_owner_cleanup.rs @@ -380,7 +380,7 @@ fn finish_retirement( &intent.foreground_sha256, &digest(receipt)?, )?; - dependency_slots::recover_cleaned(candidate, receipt)?; + dependency_slots::recover_cleaned(candidate, receipt, None)?; engine.guest().verify() } diff --git a/packages/runtime-core/src/provider/graph/mod.rs b/packages/runtime-core/src/provider/graph/mod.rs index 0236bce3..d43122f1 100644 --- a/packages/runtime-core/src/provider/graph/mod.rs +++ b/packages/runtime-core/src/provider/graph/mod.rs @@ -18,9 +18,13 @@ mod dependency_hosts; #[cfg(target_os = "macos")] mod dependency_slots; #[cfg(target_os = "macos")] +mod host_pin_recovery; +#[cfg(target_os = "macos")] pub use dependency_slots::{ inspect as dependency_reservations, recover_orphan as recover_dependency_reservation, }; +#[cfg(target_os = "macos")] +pub use host_pin_recovery::{inspect as inspect_host_pin_recovery, recover as recover_host_pins}; mod initializer_cache; mod volume_subpaths; pub use dependency_hosts::dependency_address; diff --git a/packages/runtime-core/src/provider/host_pin.rs b/packages/runtime-core/src/provider/host_pin.rs new file mode 100644 index 00000000..f583198d --- /dev/null +++ b/packages/runtime-core/src/provider/host_pin.rs @@ -0,0 +1,40 @@ +//! Exact device-number translation for explicitly selected pre-reboot host pins. +//! This is never used by ordinary ownership, source replay, or guest observations. +use super::identity::{self, ProcessIdentity}; +use crate::CandidateError; +use serde::{Deserialize, Serialize}; + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct DeviceRebind { + pub old: u64, + pub current: u64, +} + +impl DeviceRebind { + pub fn matches(self, recorded: (u64, u64), observed: (u64, u64)) -> bool { + self.old != self.current + && recorded.0 == self.old + && observed.0 == self.current + && recorded.1 != 0 + && recorded.1 == observed.1 + } + + pub fn definitely_dead_before_boot( + self, + process: &ProcessIdentity, + host_boot_micros: u64, + ) -> Result<(), CandidateError> { + if host_boot_micros == 0 + || process.start_micros == 0 + || process.start_micros >= host_boot_micros + || identity::alive(process.pid)? + { + return Err(CandidateError::new( + "host_pin_recovery", + "Legacy host pin owner is not proved dead before this host boot.", + )); + } + Ok(()) + } +} diff --git a/packages/runtime-core/src/provider/lifecycle.rs b/packages/runtime-core/src/provider/lifecycle.rs index 60e7109d..36decce9 100644 --- a/packages/runtime-core/src/provider/lifecycle.rs +++ b/packages/runtime-core/src/provider/lifecycle.rs @@ -1,8 +1,10 @@ +pub mod host_filesystem; mod interrupted; mod prepared_boot; #[cfg(any(target_os = "macos", test))] mod private_child; mod relay_process; +mod short_home; use super::{ admission, agent, artifact, identity, process, state::{self, Owner, io}, @@ -1812,6 +1814,12 @@ fn finish_absent( value: &str, record_disks: bool, ) -> Result<(), CandidateError> { + let vm_lock = lock_absent_disks(candidate, owner)?; + finish_absent_locked(candidate, owner, value, record_disks, &vm_lock) +} + +/// Keep this descriptor through alias restoration and the stopped receipt. +fn lock_absent_disks(candidate: &Candidate, owner: &Owner) -> Result { let directory = owner.real_data_dir(candidate)?; let vm_lock = OpenOptions::new() .read(true) @@ -1847,6 +1855,16 @@ fn finish_absent( )); } verify_disks(candidate, owner)?; + Ok(vm_lock) +} + +fn finish_absent_locked( + candidate: &Candidate, + owner: &mut Owner, + value: &str, + record_disks: bool, + _vm_lock: &File, +) -> Result<(), CandidateError> { if record_disks && owner.storage.is_none() { owner.storage = Some(identity::disk( &owner.real_data_dir(candidate)?.join("storage.raw"), @@ -1896,7 +1914,13 @@ fn finish_absent( } pub fn recover(candidate: &Candidate) -> Result { - let initial = status(candidate)?; + let initial = match status(candidate) { + Ok(initial) => initial, + Err(error) if error.code == "provider_home_missing" => { + return short_home::recover(candidate); + } + Err(error) => return Err(error), + }; if initial.phase == "uninitialized" { return Ok(initial); } diff --git a/packages/runtime-core/src/provider/lifecycle/host_filesystem.rs b/packages/runtime-core/src/provider/lifecycle/host_filesystem.rs new file mode 100644 index 00000000..2b7ad0dd --- /dev/null +++ b/packages/runtime-core/src/provider/lifecycle/host_filesystem.rs @@ -0,0 +1,297 @@ +//! Explicit legacy device-number migration for an offline stock pool. +//! +//! Old receipts have neither a host boot UUID nor a filesystem volume UUID. Calendar start +//! time, unchanged inode/size/ext4 UUID and exact paths constrain this migration, but cannot +//! establish original volume continuity. Only an explicitly accepted, hash-selected inspection +//! may update the owner. Normal runtime operations retain their strict identity comparisons. +use super::{Owner, binary, identity, lock_absent_disks, root, state}; +use crate::{Candidate, CandidateError}; +use serde::Serialize; +use sha2::{Digest, Sha256}; +use std::{fs, os::unix::fs::MetadataExt, path::Path}; + +#[derive(Debug, Serialize, PartialEq, Eq)] +pub struct Inspection { + pub schema: &'static str, + pub selection_sha256: String, + pub owner_sha256: String, + pub machine: String, + pub host_boot_micros: u64, + pub provider_start_micros: u64, + pub old_device: u64, + pub new_device: u64, + pub pool_inode: u64, + pub storage: identity::DiskIdentity, + pub overlay: identity::DiskIdentity, + pub project_share: Option, + pub qualification: &'static str, +} + +fn refused(detail: &str) -> CandidateError { + CandidateError::new( + "host_filesystem_recovery", + format!("{detail}; no identity was changed."), + ) +} + +#[cfg(target_os = "macos")] +pub(crate) fn host_boot_micros() -> Result { + let mut time = std::mem::MaybeUninit::::zeroed(); + let mut length = std::mem::size_of::(); + // SAFETY: read-only sysctl writes an exactly sized timeval; no input or retained pointers. + if unsafe { + libc::sysctlbyname( + c"kern.boottime".as_ptr(), + time.as_mut_ptr().cast(), + &mut length, + std::ptr::null_mut(), + 0, + ) + } != 0 + || length != std::mem::size_of::() + { + return Err(refused("Native host boot time is unavailable")); + } + // SAFETY: a complete timeval was returned above. + let time = unsafe { time.assume_init() }; + let seconds = u64::try_from(time.tv_sec).map_err(|_| refused("Invalid host boot time"))?; + let micros = u64::try_from(time.tv_usec).map_err(|_| refused("Invalid host boot time"))?; + if micros >= 1_000_000 { + return Err(refused("Invalid host boot time")); + } + seconds + .checked_mul(1_000_000) + .and_then(|v| v.checked_add(micros)) + .filter(|v| *v > 0) + .ok_or_else(|| refused("Invalid host boot time")) +} + +#[cfg(not(target_os = "macos"))] +fn host_boot_micros() -> Result { + Err(CandidateError::new( + "unsupported_host", + "Legacy filesystem recovery requires macOS.", + )) +} + +pub(crate) fn no_auxiliary_update(candidate: &Candidate) -> Result<(), CandidateError> { + crate::provider::network_update::require_complete(candidate)?; + for name in [ + "owner.pending", + "prepared-base.json", + "prepared-base.json.pending", + ] { + match fs::symlink_metadata(root(candidate).join(name)) { + Err(e) if e.kind() == std::io::ErrorKind::NotFound => {} + _ => { + return Err(refused( + "Pending owner or prepared-base state requires separate recovery", + )); + } + } + } + Ok(()) +} + +fn dead_provider(candidate: &Candidate, owner: &Owner, boot: u64) -> Result<(), CandidateError> { + let process = owner + .process + .as_ref() + .ok_or_else(|| refused("No recorded provider identity"))?; + // SAFETY: geteuid has no preconditions. + identity::verify(process, process, &binary(candidate), unsafe { + libc::geteuid() + })?; + if !owner.created + || process.start_micros >= boot + || identity::alive(process.pid)? + || identity::executable_running(&binary(candidate))? + { + return Err(refused( + "Provider must be absent and its recorded start must predate this host boot", + )); + } + Ok(()) +} + +fn same_disk(before: &identity::DiskIdentity, after: &identity::DiskIdentity) -> bool { + before.device != after.device + && before.inode == after.inode + && before.bytes == after.bytes + && before.uuid == after.uuid +} + +/// A retained flock cannot fence a writer that opens a substituted lock pathname. +fn bound_locks( + candidate: &Candidate, + owner: &Owner, + operation: &state::Lock, + vm: &fs::File, +) -> Result<(), CandidateError> { + let check = |path: &Path, expected: (u64, u64)| -> Result<(), CandidateError> { + let metadata = fs::symlink_metadata(path).map_err(state::io)?; + if !metadata.is_file() + || metadata.nlink() != 1 + || (metadata.dev(), metadata.ino()) != expected + { + return Err(refused("Held lock pathname was replaced")); + } + Ok(()) + }; + check( + &root(candidate).join("operation.lock"), + operation.identity()?, + )?; + let metadata = vm.metadata().map_err(state::io)?; + check( + &owner.real_data_dir(candidate)?.join("vm.lock"), + (metadata.dev(), metadata.ino()), + ) +} + +/// Build a candidate owner in memory. This function neither adopts a new disk nor writes state. +fn selection( + candidate: &Candidate, + owner: &Owner, + boot: u64, +) -> Result<(Inspection, Owner), CandidateError> { + no_auxiliary_update(candidate)?; + dead_provider(candidate, owner, boot)?; + let data = owner.real_data_dir(candidate)?; + let storage = identity::disk(&data.join("storage.raw"))?; + let overlay = identity::disk(&data.join("overlay.raw"))?; + let before = owner + .storage + .as_ref() + .ok_or_else(|| refused("Storage identity is missing"))?; + let prior_overlay = owner + .overlay + .as_ref() + .ok_or_else(|| refused("Overlay identity is missing"))?; + let metadata = fs::symlink_metadata(root(candidate)).map_err(state::io)?; + if !same_disk(before, &storage) + || !same_disk(prior_overlay, &overlay) + || before.device != prior_overlay.device + || storage.device != overlay.device + || storage.device != metadata.dev() + { + return Err(refused( + "Only a common device-number change with unchanged disk inode, size and UUID is accepted", + )); + } + let mut next = owner.clone(); + if let Some(share) = &owner.project_share { + let observed = + crate::provider::ProjectShareIntent::approve(&share.project, share.unfiltered_source)?; + let mut expected = share.clone(); + expected.device = storage.device; + if share.device != before.device || observed != expected { + return Err(refused( + "Project share changed beyond the same filesystem device number", + )); + } + next.project_share = Some(observed); + } + next.storage = Some(storage.clone()); + next.overlay = Some(overlay.clone()); + // The canonical read is bounded and validated independently of raw-byte hashing. + let bytes = crate::provider::prepared_base::read_private( + &root(candidate).join("owner.json"), + 1024 * 1024, + )?; + if Owner::load_for_short_home_recovery(candidate)? != *owner { + return Err(refused("Owner selection changed")); + } + let mut inspection = Inspection { + schema: "hack.host-filesystem-recovery/v1", + selection_sha256: String::new(), + owner_sha256: format!("{:x}", Sha256::digest(bytes)), + machine: owner.machine.clone(), + host_boot_micros: boot, + provider_start_micros: owner + .process + .as_ref() + .ok_or_else(|| refused("No process"))? + .start_micros, + old_device: before.device, + new_device: storage.device, + pool_inode: metadata.ino(), + storage, + overlay, + project_share: next.project_share.clone(), + qualification: "explicit-legacy-migration-original-volume-continuity-unproven", + }; + inspection.selection_sha256 = format!( + "{:x}", + Sha256::digest( + serde_json::to_vec(&inspection).map_err(|_| refused("Cannot encode selection"))? + ) + ); + Ok((inspection, next)) +} + +/// Read-only inspection holds both existing locks and proves disks have no open handles. +pub fn inspect(candidate: &Candidate) -> Result { + let operation = state::Lock::acquire_existing(&root(candidate))?; + let owner = Owner::load_for_short_home_recovery(candidate)?; + let (inspection, next) = selection(candidate, &owner, host_boot_micros()?)?; + let vm = lock_absent_disks(candidate, &next)?; + if selection(candidate, &owner, host_boot_micros()?)?.0 != inspection { + return Err(refused("Inspection changed during absence verification")); + } + bound_locks(candidate, &owner, &operation, &vm)?; + Ok(inspection) +} + +/// Publish disk and source device changes in ONE atomic owner replacement. A crash leaves +/// old or new committed metadata; any unfinished owner.pending is preserved and blocks retry. +/// This does not launch, restore the alias, retire historical sockets, or rewrite graph receipts. +pub fn recover(candidate: &Candidate, expected: &str) -> Result { + recover_with_boot(candidate, expected, host_boot_micros) +} + +fn recover_with_boot( + candidate: &Candidate, + expected: &str, + boot: impl Fn() -> Result, +) -> Result { + if expected.len() != 64 || !expected.bytes().all(|b| b.is_ascii_hexdigit()) { + return Err(refused("An exact inspection SHA-256 is required")); + } + let operation = state::Lock::acquire_existing(&root(candidate))?; + let owner = Owner::load_for_short_home_recovery(candidate)?; + let (inspection, next) = selection(candidate, &owner, boot()?)?; + if inspection.selection_sha256 != expected { + return Err(refused("Inspection selection is stale")); + } + let vm = lock_absent_disks(candidate, &next)?; + if selection(candidate, &owner, boot()?)?.0 != inspection { + return Err(refused("Selected identities changed before publication")); + } + bound_locks(candidate, &owner, &operation, &vm)?; + next.save(candidate)?; + if Owner::load_for_short_home_recovery(candidate)? != next { + return Err(CandidateError::new( + "host_filesystem_recovery_incomplete", + "Owner was published but changed during confirmation; retained state requires inspection.", + )); + } + super::verify_disks(candidate, &next).map_err(|e| { + CandidateError::new( + "host_filesystem_recovery_incomplete", + format!( + "Owner was published but disk confirmation failed ({}); inspect retained state.", + e.code + ), + ) + })?; + if let Some(share) = &next.project_share { + share.validate().map_err(|e| CandidateError::new( + "host_filesystem_recovery_incomplete", format!("Owner was published but source confirmation failed ({}); inspect retained state.", e.code) + ))?; + } + Ok(inspection) +} + +#[cfg(all(test, target_os = "macos"))] +mod tests; diff --git a/packages/runtime-core/src/provider/lifecycle/host_filesystem/tests.rs b/packages/runtime-core/src/provider/lifecycle/host_filesystem/tests.rs new file mode 100644 index 00000000..4c6cc502 --- /dev/null +++ b/packages/runtime-core/src/provider/lifecycle/host_filesystem/tests.rs @@ -0,0 +1,319 @@ +use super::*; +use crate::provider::{NetworkIntent, Profile, ProjectShareIntent}; +use std::{fs::File, os::fd::AsRawFd, path::PathBuf, process::Command}; + +struct Pool { + candidate: Candidate, + owner: Owner, + directory: PathBuf, + boot: u64, +} +impl Pool { + fn new() -> Self { + let mut child = Command::new("/bin/sleep").arg("30").spawn().unwrap(); + let mut process = identity::observe(child.id() as i32).unwrap(); + child.kill().unwrap(); + child.wait().unwrap(); + let boot = process.start_micros + 1; + let directory = fs::canonicalize(std::env::temp_dir()) + .unwrap() + .join(format!( + "hack-device-recovery-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + state::private_directory(&directory).unwrap(); + let candidate = Candidate::discover(&directory).unwrap(); + let operation = state::Lock::acquire(&root(&candidate)).unwrap(); + let project = directory.join("app"); + state::private_directory(&project).unwrap(); + fs::write(project.join("package.json"), b"{}").unwrap(); + let share = ProjectShareIntent::approve(&project, true).unwrap(); + let mut owner = Owner::create_with_project_share( + &candidate, + Profile::Development, + None, + NetworkIntent::Isolated, + None, + Some(share), + ) + .unwrap(); + let data = owner.real_data_dir(&candidate).unwrap(); + state::private_directory(&data).unwrap(); + for (name, tag) in [("storage.raw", 1u8), ("overlay.raw", 2)] { + let mut bytes = vec![0; 4096]; + bytes[1080..1082].copy_from_slice(&[0x53, 0xef]); + bytes[1128..1144].fill(tag); + bytes[2048..2059].copy_from_slice(b"data-marker"); + fs::write(data.join(name), bytes).unwrap(); + } + fs::write(data.join("name"), &owner.machine).unwrap(); + fs::write(data.join("vm.lock"), b"").unwrap(); + process.executable = binary(&candidate); + owner.process = Some(process); + owner.created = true; + owner.phase = "running".into(); + let mut storage = identity::disk(&data.join("storage.raw")).unwrap(); + let mut overlay = identity::disk(&data.join("overlay.raw")).unwrap(); + let old = storage.device + 1; + storage.device = old; + overlay.device = old; + owner.storage = Some(storage); + owner.overlay = Some(overlay); + owner.project_share.as_mut().unwrap().device = old; + owner.save(&candidate).unwrap(); + fs::remove_file(&owner.short_home).unwrap(); + drop(operation); + Self { + candidate, + owner, + directory, + boot, + } + } + fn data(&self) -> PathBuf { + self.owner.real_data_dir(&self.candidate).unwrap() + } + fn selected(&self) -> Inspection { + selection(&self.candidate, &self.owner, self.boot) + .unwrap() + .0 + } + fn receipt(&self) -> Vec { + fs::read(root(&self.candidate).join("owner.json")).unwrap() + } + fn apply(&self, hash: &str) -> Result { + recover_with_boot(&self.candidate, hash, || Ok(self.boot)) + } + fn unchanged(&self, before: &[u8]) { + assert_eq!(self.receipt(), before); + assert!(self.owner.short_home.symlink_metadata().is_err()); + } +} +impl Drop for Pool { + fn drop(&mut self) { + let _ = fs::remove_file(&self.owner.short_home); + fs::remove_dir_all(&self.directory).unwrap(); + } +} + +#[test] +fn migration_changes_only_devices_and_keeps_alias_graph_history_and_data() { + let pool = Pool::new(); + let old = pool.receipt(); + let storage = fs::read(pool.data().join("storage.raw")).unwrap(); + let overlay = fs::read(pool.data().join("overlay.raw")).unwrap(); + let history = pool.directory.join("historical-graph.json"); + fs::write(&history, b"historical-source-binding").unwrap(); + let inspected = pool.selected(); + pool.unchanged(&old); + let result = pool.apply(&inspected.selection_sha256).unwrap(); + assert_eq!(result, inspected); + let mut expected = pool.owner.clone(); + expected.storage.as_mut().unwrap().device = inspected.new_device; + expected.overlay.as_mut().unwrap().device = inspected.new_device; + expected.project_share.as_mut().unwrap().device = inspected.new_device; + assert_eq!( + Owner::load_for_short_home_recovery(&pool.candidate).unwrap(), + expected + ); + assert!(pool.owner.short_home.symlink_metadata().is_err()); + assert_eq!(fs::read(history).unwrap(), b"historical-source-binding"); + assert_eq!(fs::read(pool.data().join("storage.raw")).unwrap(), storage); + assert_eq!(fs::read(pool.data().join("overlay.raw")).unwrap(), overlay); + assert!(pool.apply(&inspected.selection_sha256).is_err()); + // Ordinary recovery still owns HOME restoration and the recovered phase. + let recovered = crate::provider::recover(&pool.candidate).unwrap(); + assert_eq!(recovered.phase, "recovered-unclean"); + assert_eq!(recovered.process_alive, Some(false)); + assert_eq!(fs::read(pool.data().join("storage.raw")).unwrap(), storage); +} + +#[test] +fn disk_and_share_changes_beyond_common_device_refuse_without_writes() { + for case in 0..7 { + let mut pool = Pool::new(); + let observed_device = identity::disk(&pool.data().join("storage.raw")) + .unwrap() + .device; + let disk = pool.owner.storage.as_mut().unwrap(); + match case { + 0 => disk.inode += 1, + 1 => disk.bytes += 1, + 2 => disk.uuid.push('0'), + 3 => disk.device = observed_device, + 4 => pool.owner.overlay.as_mut().unwrap().device += 1, + 5 => pool.owner.project_share.as_mut().unwrap().inode += 1, + _ => pool.owner.project_share.as_mut().unwrap().device += 1, + } + pool.owner.save(&pool.candidate).unwrap(); + let before = pool.receipt(); + assert!(pool.apply(&"a".repeat(64)).is_err()); + pool.unchanged(&before); + } +} + +#[test] +fn stale_owner_or_host_boot_selection_refuses() { + let mut pool = Pool::new(); + let selection = pool.selected(); + assert!(pool.apply(&"a".repeat(64)).is_err()); + pool.owner.phase = "stopped".into(); + pool.owner.save(&pool.candidate).unwrap(); + let before = pool.receipt(); + assert!(pool.apply(&selection.selection_sha256).is_err()); + pool.unchanged(&before); + let selected = pool.selected(); + assert!( + recover_with_boot(&pool.candidate, &selected.selection_sha256, || Ok(pool + .boot + + 1)) + .is_err() + ); + pool.unchanged(&before); +} + +#[test] +fn same_boot_live_reused_pid_and_unknown_identity_refuse() { + for case in 0..4 { + let mut pool = Pool::new(); + if case == 0 { + pool.boot = pool.owner.process.as_ref().unwrap().start_micros; + } + if case == 1 || case == 2 { + let p = pool.owner.process.as_mut().unwrap(); + p.pid = std::process::id() as i32; + if case == 1 { + p.start_micros = identity::observe(p.pid).unwrap().start_micros; + } + } + if case == 3 { + pool.owner.process = None; + } + pool.owner.save(&pool.candidate).unwrap(); + let before = pool.receipt(); + assert!(pool.apply(&"a".repeat(64)).is_err()); + pool.unchanged(&before); + } +} + +#[test] +fn held_operation_vm_lock_and_disk_handles_refuse() { + for case in 0..3 { + let pool = Pool::new(); + let selected = pool.selected(); + let before = pool.receipt(); + let _operation = + (case == 0).then(|| state::Lock::acquire_existing(&root(&pool.candidate)).unwrap()); + let file = if case == 1 { + Some(File::open(pool.data().join("vm.lock")).unwrap()) + } else if case == 2 { + Some(File::open(pool.data().join("storage.raw")).unwrap()) + } else { + None + }; + if case == 1 { + assert_eq!( + unsafe { + libc::flock( + file.as_ref().unwrap().as_raw_fd(), + libc::LOCK_EX | libc::LOCK_NB, + ) + }, + 0 + ); + } + assert!(pool.apply(&selected.selection_sha256).is_err()); + pool.unchanged(&before); + } +} + +#[test] +fn pending_foreign_updates_and_prepared_pools_are_preserved() { + for name in [ + "owner.pending", + "network-update.json", + "network-update.pending", + "prepared-base.json", + "prepared-base.json.pending", + ] { + let pool = Pool::new(); + let selected = pool.selected(); + let before = pool.receipt(); + let path = root(&pool.candidate).join(name); + fs::write(&path, b"foreign-or-interrupted").unwrap(); + assert!(pool.apply(&selected.selection_sha256).is_err()); + pool.unchanged(&before); + assert_eq!(fs::read(path).unwrap(), b"foreign-or-interrupted"); + } +} + +#[test] +fn foreign_alias_and_source_substitution_refuse() { + for case in 0..2 { + let pool = Pool::new(); + let selected = pool.selected(); + let before = pool.receipt(); + if case == 0 { + fs::write(&pool.owner.short_home, b"foreign").unwrap(); + } else { + let project = &pool.owner.project_share.as_ref().unwrap().project; + fs::rename(project, pool.directory.join("preserved-app")).unwrap(); + state::private_directory(project).unwrap(); + fs::write(project.join("package.json"), b"{}").unwrap(); + } + assert!(pool.apply(&selected.selection_sha256).is_err()); + assert_eq!(pool.receipt(), before); + if case == 0 { + assert_eq!(fs::read(&pool.owner.short_home).unwrap(), b"foreign"); + } + } +} + +#[test] +fn final_recheck_rejects_identity_change_before_publication() { + let pool = Pool::new(); + let selected = pool.selected(); + let before = pool.receipt(); + let called = std::cell::Cell::new(false); + let result = recover_with_boot(&pool.candidate, &selected.selection_sha256, || { + if called.replace(true) { + fs::write(pool.data().join("storage.raw"), b"changed").unwrap(); + } + Ok(pool.boot) + }); + assert!(result.is_err()); + pool.unchanged(&before); + assert_eq!( + fs::read(pool.data().join("storage.raw")).unwrap(), + b"changed" + ); +} + +#[test] +fn substituted_lock_paths_do_not_authorize_publication_on_an_old_descriptor() { + for name in ["operation.lock", "vm.lock"] { + let pool = Pool::new(); + let selected = pool.selected(); + let before = pool.receipt(); + let called = std::cell::Cell::new(false); + let path = if name == "operation.lock" { + root(&pool.candidate).join(name) + } else { + pool.data().join(name) + }; + let result = recover_with_boot(&pool.candidate, &selected.selection_sha256, || { + if called.replace(true) { + fs::rename(&path, path.with_extension("preserved")).unwrap(); + fs::write(&path, b"replacement lock").unwrap(); + } + Ok(pool.boot) + }); + assert!(matches!(result, Err(e) if e.code == "host_filesystem_recovery")); + pool.unchanged(&before); + assert_eq!(fs::read(path).unwrap(), b"replacement lock"); + } +} diff --git a/packages/runtime-core/src/provider/lifecycle/short_home.rs b/packages/runtime-core/src/provider/lifecycle/short_home.rs new file mode 100644 index 00000000..6678ab60 --- /dev/null +++ b/packages/runtime-core/src/provider/lifecycle/short_home.rs @@ -0,0 +1,70 @@ +//! Explicit two-phase recovery of the receipt-bound temporary HOME after host cleanup. +//! +//! No provider is launched and no existing alias is replaced. Durable disks and VM absence +//! are audited before restoring the short socket path; socket absence is then checked before +//! committing the recovered phase. A later refusal retains the exact alias and unchanged data. +use super::{ + Owner, RuntimeStatus, binary, finish_absent_locked, identity, lock_absent_disks, root, state, + status, verify_disks, +}; +use crate::{Candidate, CandidateError}; + +fn dead_provider(candidate: &Candidate, owner: &Owner) -> Result<(), CandidateError> { + let process = owner.process.as_ref().ok_or_else(|| { + CandidateError::new( + "recovery_required", + "Missing HOME recovery requires a recorded provider identity; nothing was changed.", + ) + })?; + // SAFETY: geteuid has no preconditions. + identity::verify(process, process, &binary(candidate), unsafe { + libc::geteuid() + })?; + if identity::alive(process.pid)? || identity::executable_running(&binary(candidate))? { + return Err(CandidateError::new( + "recovery_required", + "Missing HOME recovery requires a confirmed dead provider and no active provider command; nothing was changed.", + )); + } + if !owner.created || owner.storage.is_none() || owner.overlay.is_none() { + return Err(CandidateError::new( + "recovery_required", + "Missing HOME recovery requires both previously identified disks; nothing was adopted.", + )); + } + Ok(()) +} + +pub(super) fn recover(candidate: &Candidate) -> Result { + let _operation = state::Lock::acquire_existing(&root(candidate))?; + let mut owner = Owner::load_for_short_home_recovery(candidate)?; + dead_provider(candidate, &owner)?; + let vm_lock = lock_absent_disks(candidate, &owner)?; + // Recheck immediately before the only new external effect. The locks fence cooperative + // writers; no signal, disk adoption, overwrite or receipt update happens in this phase. + dead_provider(candidate, &owner)?; + verify_disks(candidate, &owner)?; + owner.restore_missing_short_home(candidate)?; + let mut finish = || -> Result<(), CandidateError> { + let observed = Owner::load(candidate)?; + if observed != owner { + return Err(CandidateError::new( + "foreign_state", + "Provider receipt changed after HOME restoration.", + )); + } + dead_provider(candidate, &owner)?; + verify_disks(candidate, &owner)?; + finish_absent_locked(candidate, &mut owner, "recovered-unclean", false, &vm_lock) + }; + finish().map_err(|error| { + CandidateError::new( + "provider_home_restored_recovery_incomplete", + format!("Owned HOME alias restored; runtime recovery is incomplete ({}). Data retained; inspect and retry runtime recover.", error.code), + ) + })?; + status(candidate) +} + +#[cfg(all(test, target_os = "macos"))] +mod tests; diff --git a/packages/runtime-core/src/provider/lifecycle/short_home/tests.rs b/packages/runtime-core/src/provider/lifecycle/short_home/tests.rs new file mode 100644 index 00000000..f22b6219 --- /dev/null +++ b/packages/runtime-core/src/provider/lifecycle/short_home/tests.rs @@ -0,0 +1,265 @@ +use super::*; +use crate::provider::{NetworkIntent, Profile}; +use std::{ + fs, + os::{ + fd::AsRawFd, + unix::{fs::MetadataExt, net::UnixListener}, + }, + path::PathBuf, + process::Command, +}; + +struct Pool { + candidate: Candidate, + owner: Owner, + directory: PathBuf, +} +impl Pool { + fn new() -> Self { + let mut child = Command::new("/bin/sleep").arg("30").spawn().unwrap(); + let mut process = identity::observe(child.id() as i32).unwrap(); + child.kill().unwrap(); + child.wait().unwrap(); + let directory = fs::canonicalize(std::env::temp_dir()) + .unwrap() + .join(format!( + "hack-home-recovery-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + state::private_directory(&directory).unwrap(); + let candidate = Candidate::discover(&directory).unwrap(); + let operation = state::Lock::acquire(&root(&candidate)).unwrap(); + let mut owner = Owner::create( + &candidate, + Profile::Development, + None, + NetworkIntent::Isolated, + ) + .unwrap(); + let data = owner.real_data_dir(&candidate).unwrap(); + state::private_directory(&data).unwrap(); + for (name, tag) in [("storage.raw", 1u8), ("overlay.raw", 2)] { + let mut bytes = vec![0u8; 4096]; + bytes[1080..1082].copy_from_slice(&[0x53, 0xef]); + bytes[1128..1144].fill(tag); + bytes[2048..2059].copy_from_slice(b"data-marker"); + fs::write(data.join(name), bytes).unwrap(); + } + fs::write(data.join("vm.lock"), b"").unwrap(); + fs::write(data.join("name"), &owner.machine).unwrap(); + process.executable = binary(&candidate); + owner.process = Some(process); + owner.created = true; + owner.phase = "running".into(); + owner.storage = Some(identity::disk(&data.join("storage.raw")).unwrap()); + owner.overlay = Some(identity::disk(&data.join("overlay.raw")).unwrap()); + owner.save(&candidate).unwrap(); + drop(operation); + Self { + candidate, + owner, + directory, + } + } + fn remove_alias(&self) { + fs::remove_file(&self.owner.short_home).unwrap(); + } + fn receipt(&self) -> Vec { + fs::read(root(&self.candidate).join("owner.json")).unwrap() + } + fn data(&self) -> PathBuf { + self.owner.real_data_dir(&self.candidate).unwrap() + } + fn unchanged(&self, before: &[u8]) { + assert_eq!(self.receipt(), before); + assert!(self.owner.short_home.symlink_metadata().is_err()); + } +} +impl Drop for Pool { + fn drop(&mut self) { + let _ = fs::remove_file(&self.owner.short_home); + fs::remove_dir_all(&self.directory).unwrap(); + } +} + +#[test] +fn explicit_recovery_restores_only_missing_alias_and_preserves_disk_bytes() { + let pool = Pool::new(); + let storage = fs::read(pool.data().join("storage.raw")).unwrap(); + let overlay = fs::read(pool.data().join("overlay.raw")).unwrap(); + pool.remove_alias(); + assert_eq!( + status(&pool.candidate).unwrap_err().code, + "provider_home_missing" + ); + let recovered = crate::provider::recover(&pool.candidate).unwrap(); + assert_eq!(recovered.phase, "recovered-unclean"); + assert_eq!(recovered.process_alive, Some(false)); + assert_eq!( + fs::read_link(&pool.owner.short_home).unwrap(), + root(&pool.candidate).join("home") + ); + assert_eq!(fs::read(pool.data().join("storage.raw")).unwrap(), storage); + assert_eq!(fs::read(pool.data().join("overlay.raw")).unwrap(), overlay); + assert_eq!( + Owner::load(&pool.candidate).unwrap().storage, + pool.owner.storage + ); +} + +#[test] +fn live_or_reused_pid_and_missing_disk_identity_refuse_without_alias_effect() { + for case in 0..3 { + let mut pool = Pool::new(); + if case < 2 { + let p = pool.owner.process.as_mut().unwrap(); + p.pid = std::process::id() as i32; + p.start_micros = if case == 0 { + identity::observe(p.pid).unwrap().start_micros + } else { + 1 + }; + } else { + pool.owner.overlay = None; + } + pool.owner.save(&pool.candidate).unwrap(); + pool.remove_alias(); + let before = pool.receipt(); + assert_eq!( + crate::provider::recover(&pool.candidate).unwrap_err().code, + "recovery_required" + ); + pool.unchanged(&before); + } +} + +#[test] +fn changed_disk_held_vm_lock_and_open_disk_refuse_before_alias_creation() { + for case in 0..3 { + let pool = Pool::new(); + pool.remove_alias(); + let before = pool.receipt(); + let mut held = None; + if case == 0 { + fs::write(pool.data().join("overlay.raw"), b"changed disk").unwrap(); + } + if case == 1 { + let f = fs::File::open(pool.data().join("vm.lock")).unwrap(); + assert_eq!( + unsafe { libc::flock(f.as_raw_fd(), libc::LOCK_EX | libc::LOCK_NB) }, + 0 + ); + held = Some(f); + } + if case == 2 { + held = Some(fs::File::open(pool.data().join("storage.raw")).unwrap()); + } + assert!(crate::provider::recover(&pool.candidate).is_err()); + pool.unchanged(&before); + drop(held); + } +} + +#[test] +fn existing_file_directory_and_foreign_symlink_are_never_replaced() { + for case in 0..3 { + let pool = Pool::new(); + pool.remove_alias(); + let before = pool.receipt(); + if case == 0 { + fs::write(&pool.owner.short_home, b"foreign marker").unwrap(); + } + if case == 1 { + fs::create_dir(&pool.owner.short_home).unwrap(); + } + if case == 2 { + std::os::unix::fs::symlink(&pool.directory, &pool.owner.short_home).unwrap(); + } + let inode = fs::symlink_metadata(&pool.owner.short_home).unwrap().ino(); + assert_eq!( + crate::provider::recover(&pool.candidate).unwrap_err().code, + "foreign_state" + ); + assert_eq!(pool.receipt(), before); + assert_eq!( + fs::symlink_metadata(&pool.owner.short_home).unwrap().ino(), + inode + ); + if case == 1 { + fs::remove_dir(&pool.owner.short_home).unwrap(); + } + } +} + +#[test] +fn active_socket_leaves_explicit_incomplete_recovery_then_retry_finishes() { + let pool = Pool::new(); + let listener = UnixListener::bind(pool.owner.data_dir().join("agent.sock")).unwrap(); + pool.remove_alias(); + let before = pool.receipt(); + let error = crate::provider::recover(&pool.candidate).unwrap_err(); + assert_eq!(error.code, "provider_home_restored_recovery_incomplete"); + assert!(error.message.contains("stop_uncertain")); + assert_eq!(pool.receipt(), before); + assert_eq!( + fs::read_link(&pool.owner.short_home).unwrap(), + root(&pool.candidate).join("home") + ); + drop(listener); + assert_eq!( + crate::provider::recover(&pool.candidate).unwrap().phase, + "recovered-unclean" + ); +} + +#[test] +fn receipt_change_and_concurrent_alias_creation_refuse_exclusive_repair() { + let mut pool = Pool::new(); + pool.remove_alias(); + let observed = Owner::load_for_short_home_recovery(&pool.candidate).unwrap(); + pool.owner.phase = "stopped".into(); + pool.owner.save(&pool.candidate).unwrap(); + let before = pool.receipt(); + assert_eq!( + observed + .restore_missing_short_home(&pool.candidate) + .unwrap_err() + .code, + "foreign_state" + ); + pool.unchanged(&before); + std::os::unix::fs::symlink(root(&pool.candidate).join("home"), &pool.owner.short_home).unwrap(); + let inode = fs::symlink_metadata(&pool.owner.short_home).unwrap().ino(); + assert_eq!( + pool.owner + .restore_missing_short_home(&pool.candidate) + .unwrap_err() + .code, + "socket_alias_collision" + ); + assert_eq!( + fs::symlink_metadata(&pool.owner.short_home).unwrap().ino(), + inode + ); + assert_eq!(pool.receipt(), before); +} + +#[test] +fn pending_owner_update_is_preserved_and_refuses_alias_repair() { + let pool = Pool::new(); + pool.remove_alias(); + let before = pool.receipt(); + let pending = root(&pool.candidate).join("owner.pending"); + fs::write(&pending, b"interrupted update").unwrap(); + assert_eq!( + crate::provider::recover(&pool.candidate).unwrap_err().code, + "recovery_required" + ); + pool.unchanged(&before); + assert_eq!(fs::read(&pending).unwrap(), b"interrupted update"); +} diff --git a/packages/runtime-core/src/provider/mod.rs b/packages/runtime-core/src/provider/mod.rs index 30f7082b..d1203b75 100644 --- a/packages/runtime-core/src/provider/mod.rs +++ b/packages/runtime-core/src/provider/mod.rs @@ -19,6 +19,8 @@ pub mod environment; mod environment_probe_test; pub mod environment_recovery; pub mod graph; +#[cfg(target_os = "macos")] +mod host_pin; pub use engine::{EngineInfo, info as engine_info}; #[cfg(all(test, target_os = "macos", target_arch = "aarch64"))] mod gateway_probe_test; @@ -45,6 +47,7 @@ pub mod resources; pub mod storage_usage; pub use image_load::load as load_image; mod lifecycle; +pub use lifecycle::host_filesystem; mod network_intent; mod network_update; pub use network_update::{enable_internet, extend_network}; diff --git a/packages/runtime-core/src/provider/relay_owner/publication.rs b/packages/runtime-core/src/provider/relay_owner/publication.rs index 2210b12e..178e0a97 100644 --- a/packages/runtime-core/src/provider/relay_owner/publication.rs +++ b/packages/runtime-core/src/provider/relay_owner/publication.rs @@ -7,6 +7,7 @@ use super::{ use crate::{ CandidateError, provider::{ + host_pin::DeviceRebind, identity::{self, ProcessIdentity}, state, }, @@ -102,6 +103,16 @@ pub struct PinnedEndpoint { receipt: Receipt, receipt_id: FileId, bytes: Vec, + device_rebind: Option, +} +#[derive(Clone, Debug, Eq, PartialEq, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct LegacyControl { + pub receipt_sha256: String, + pub receipt_id: FileId, + pub parent: FileId, + pub endpoint: FileId, + pub process: ProcessIdentity, } impl PinnedEndpoint { pub(crate) fn runtime_root(&self) -> &Path { @@ -134,6 +145,13 @@ impl PinnedEndpoint { Self::read(Paths::new(root)?, context) } fn read(paths: Paths, context: Context) -> Result { + Self::read_with_rebind(paths, context, None) + } + fn read_with_rebind( + paths: Paths, + context: Context, + device_rebind: Option, + ) -> Result { let parent = paths.parent()?; let mut file = OpenOptions::new() .read(true) @@ -160,7 +178,9 @@ impl PinnedEndpoint { || context.boot == [0; 16] || receipt.owner == [0; 16] || receipt.socket != paths.socket - || receipt.parent != parent + || !device_rebind.map_or(receipt.parent == parent, |v| { + v.matches(receipt.parent, parent) + }) || receipt.endpoint.1 == 0 || !receipt.process.executable.is_absolute() { @@ -179,8 +199,30 @@ impl PinnedEndpoint { receipt, receipt_id: id(&m), bytes, + device_rebind, }) } + pub(crate) fn load_legacy_recovery( + root: &Path, + context: Context, + device_rebind: DeviceRebind, + host_boot_micros: u64, + ) -> Result { + let pin = Self::read_with_rebind(Paths::new(root)?, context, Some(device_rebind))?; + pin.verify_dead()?; + device_rebind.definitely_dead_before_boot(&pin.receipt.process, host_boot_micros)?; + dead::no_listener(&pin.paths.socket)?; + Ok(pin) + } + pub(crate) fn legacy_summary(&self) -> LegacyControl { + LegacyControl { + receipt_sha256: format!("{:x}", Sha256::digest(&self.bytes)), + receipt_id: self.receipt_id, + parent: self.receipt.parent, + endpoint: self.receipt.endpoint, + process: self.receipt.process.clone(), + } + } /// Read-only predecessor proof: exact private receipt/socket, absent owner. #[cfg(target_os = "macos")] pub(crate) fn verify_dead(&self) -> Result<(), CandidateError> { @@ -198,12 +240,13 @@ impl PinnedEndpoint { Sha256::digest(&self.bytes).into() } fn verify_receipt(&self) -> Result<(), CandidateError> { - let current = Self::read( + let current = Self::read_with_rebind( self.paths.clone(), Context { runtime: self.receipt.runtime, boot: self.receipt.boot, }, + self.device_rebind, )?; if current.receipt_id != self.receipt_id || current.bytes != self.bytes { return Err(refused()); @@ -211,14 +254,24 @@ impl PinnedEndpoint { Ok(()) } fn verify_socket(&self) -> Result<(), CandidateError> { - if self.paths.parent()? != self.receipt.parent { + let parent = self.paths.parent()?; + if !self + .device_rebind + .map_or(parent == self.receipt.parent, |v| { + v.matches(self.receipt.parent, parent) + }) + { return Err(refused()); } let m = fs::symlink_metadata(&self.paths.socket).map_err(|_| refused())?; if !m.file_type().is_socket() || !private(&m) || m.nlink() != 1 - || id(&m) != self.receipt.endpoint + || !self + .device_rebind + .map_or(id(&m) == self.receipt.endpoint, |v| { + v.matches(self.receipt.endpoint, id(&m)) + }) { return Err(refused()); } @@ -353,6 +406,7 @@ impl ControlListener { receipt, receipt_id, bytes, + device_rebind: None, }; endpoint.verify_receipt()?; endpoint.verify_socket()?; diff --git a/packages/runtime-core/src/provider/relay_owner/publication/dead.rs b/packages/runtime-core/src/provider/relay_owner/publication/dead.rs index edec289f..4c5390fe 100644 --- a/packages/runtime-core/src/provider/relay_owner/publication/dead.rs +++ b/packages/runtime-core/src/provider/relay_owner/publication/dead.rs @@ -41,6 +41,7 @@ impl Selection { receipt, receipt_id: self.record_id, bytes: self.bytes.clone(), + device_rebind: None, }) } } diff --git a/packages/runtime-core/src/provider/state.rs b/packages/runtime-core/src/provider/state.rs index 7f3b1a81..bc4c3bf8 100644 --- a/packages/runtime-core/src/provider/state.rs +++ b/packages/runtime-core/src/provider/state.rs @@ -57,7 +57,6 @@ impl Lock { } /// Observe existing state without initializing a directory or lock file. - #[cfg(target_os = "macos")] pub fn acquire_existing(root: &Path) -> Result { check_private_directory(root)?; let file = OpenOptions::new() @@ -177,7 +176,7 @@ impl Default for ReclamationPolicy { } } -#[derive(Debug, Serialize, Deserialize)] +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub struct Owner { #[serde(default, skip_serializing_if = "Option::is_none")] @@ -211,8 +210,34 @@ pub struct Owner { } impl Owner { pub fn load(candidate: &Candidate) -> Result { + Self::load_with_short_home(candidate, false) + } + + /// Explicit recovery may inspect a missing temporary alias, never a replaced one. + /// Loading here is read-only; the lifecycle boundary still proves provider absence. + pub(super) fn load_for_short_home_recovery( + candidate: &Candidate, + ) -> Result { + Self::load_with_short_home(candidate, true) + } + + fn load_with_short_home( + candidate: &Candidate, + allow_missing: bool, + ) -> Result { let root = candidate.state_root.join("run/smolvm"); reject_aliased_state(&root)?; + if allow_missing { + match fs::symlink_metadata(root.join("owner.pending")) { + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + _ => { + return Err(CandidateError::new( + "recovery_required", + "Provider owner update is pending or unobservable; HOME alias was not restored.", + )); + } + } + } let owner: Self = read(&root.join("owner.json"))?; owner.network.validate()?; if let Some(share) = &owner.project_share { @@ -250,20 +275,69 @@ impl Owner { "Provider owner identity does not match this checkout.", )); } - // The only permitted alias is an exact, receipt-bound short HOME for Unix sockets. - let alias = fs::symlink_metadata(&owner.short_home).map_err(io)?; + owner.check_short_home(candidate, allow_missing)?; + check_private_directory(&root.join("home"))?; + Ok(owner) + } + /// The only permitted alias is the exact receipt-bound HOME for Unix sockets. + fn check_short_home( + &self, + candidate: &Candidate, + allow_missing: bool, + ) -> Result<(), CandidateError> { + let alias = match fs::symlink_metadata(&self.short_home) { + Ok(alias) => alias, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + return if allow_missing { + Ok(()) + } else { + Err(CandidateError::new( + "provider_home_missing", + "Temporary provider HOME alias is absent. Use runtime recover to verify the stopped pool before restoring it.", + )) + }; + } + Err(error) => return Err(io(error)), + }; if !alias.file_type().is_symlink() || alias.uid() != unsafe { libc::geteuid() } - || fs::read_link(&owner.short_home).map_err(io)? != root.join("home") + || fs::read_link(&self.short_home).map_err(io)? + != candidate.state_root.join("run/smolvm/home") { return Err(CandidateError::new( "foreign_state", "Short provider HOME alias was replaced.", )); } - check_private_directory(&root.join("home"))?; - Ok(owner) + Ok(()) } + + /// Called only while the lifecycle owns the pool and VM locks and has audited absence. + /// Recheck the durable selection, then create exclusively; a concurrent alias is retained. + pub(super) fn restore_missing_short_home( + &self, + candidate: &Candidate, + ) -> Result<(), CandidateError> { + let observed = Self::load_for_short_home_recovery(candidate)?; + if observed != *self { + return Err(CandidateError::new( + "foreign_state", + "Provider owner changed during HOME recovery; no alias was restored.", + )); + } + std::os::unix::fs::symlink( + candidate.state_root.join("run/smolvm/home"), + &self.short_home, + ) + .map_err(|_| { + CandidateError::new( + "socket_alias_collision", + "Cannot exclusively restore provider HOME alias; no existing path was replaced.", + ) + })?; + self.check_short_home(candidate, false) + } + #[cfg(all(test, target_os = "macos"))] pub fn create( candidate: &Candidate,