diff --git a/docs/guides/native-candidate.md b/docs/guides/native-candidate.md index 67d63f56b..8e1659535 100644 --- a/docs/guides/native-candidate.md +++ b/docs/guides/native-candidate.md @@ -2337,10 +2337,11 @@ Host dependencies use the equivalent `--minimum-dependency-sockets N` selection; an exact capacity or a minimum. The combined physical transport limit still applies to the pool's actual capacities, including any spare slots. A minimum does not resize an existing pool or guarantee that enough slots are currently free. -Each routed service -must already declare a matching `healthcheck.x-hack-http` (see -[Native HTTP readiness probe](#native-http-readiness-probe)); command healthchecks -and missing probes are not replaced or inferred. Native review still validates +Each routed service needs a matching native route probe (see +[Native HTTP readiness probe](#native-http-readiness-probe)). If no `healthcheck` +key was authored, native infers one on the reviewed Caddy route port without +changing the Compose file. Explicit healthchecks are never replaced; command-only +or disabled checks on routed services still refuse in this increment. Native review still validates hostnames, ports, networks and labels. Foreground graph ownership supervises and retires Unix route publishers. Without explicit HTTPS selection, this enrollment does not start a hostname authority or Caddy HTTPS server. It never installs trust, configures DNS, or widens an existing pool. @@ -2859,7 +2860,59 @@ shared-source mode; active project bind mounts still use the explicit project sh #### Native HTTP readiness probe -`healthcheck.x-hack-http` declares the readiness probe native startup uses for a +For a complete Caddy route with no authored `healthcheck`, native reviews a default +probe: `GET /` on the route port, a 1000 ms timeout, three consecutive +failures, and a 240000 ms initial startup grace. It checks every second until its +first success, then every 30 seconds for the rest of that probe generation. +This allows slow first-page compilation, including a server first answering after +120 seconds. With a full timeout on each request, the grace plus three failed +startup samples takes at most about 246 seconds of probe time, leaving about +54 seconds of the usual 300-second graph budget for earlier startup work. +An inferred probe observed unhealthy names its service and route port. For a +missing listener or HTTP 5xx during boot, check `hack logs ` in the same +project/branch, fix the route +port, or declare `healthcheck.x-hack-http` with a longer `start_period_ms` and a +readiness path that answers 2xx. + +The 300-second deadline belongs to the whole graph; it is not renewed when a +service starts. Slow earlier services or `service_healthy` dependencies consume +it too. A routed service started after a dependency taking 240 seconds can reach +the graph deadline before its own probe finishes. That path still returns the +generic `graph_timeout`, without naming the waiting services. Use `hack logs` +and inspect the dependencies as well as the route. An explicit probe with a longer +grace does not extend the whole-graph deadline. +A successful TCP connection and complete HTTP/1.0 or HTTP/1.1 response headers are +required. HTTP 200–499, including `/` returning 404, is ready; 5xx, EOF, non-HTTP +bytes, a timeout or partial headers are not. Redirects are not followed. At most +four interim 1xx responses may precede the final response, within the same 8 KiB +header and timeout budgets. The request uses `Host: 127.0.0.1:`. + +This proves endpoint readiness, not application correctness. After the first +success, three consecutive failures at the 30-second interval make the service +unhealthy. Status no longer exposes a ready route endpoint, and a new publication +or rebind cannot proceed until healthy evidence returns. This does not stop or restart the +container, withdraw an existing publication, or close its existing relay: route +traffic continues to the application and may receive its error responses. A +later successful check restores health; it does not resume one-second probing. +An inferred probe does not satisfy `depends_on: condition: service_healthy`: +that condition still requires an explicit enabled healthcheck, as on Compose. +Final HTTPS publication verification accepts 2xx–4xx only with a fresh challenge added +by the upstream proxy response handler. Authority denials and routing errors do +not qualify as the application's 404. No readiness decision follows an external +redirect. Explicit probes retain the stricter policy below. The generated +`policy` field is internal and refuses in authored `x-hack-http` declarations. +The native bundle embeds the matching guest probe; its content hash binds the +published helper. An older helper refuses the new policy argument. These inferred +defaults are retained-run compatibility values and cannot change without migration. + +Native reserves `X-Hack-Probe-Request` and `X-Hack-Upstream-Proof` for the HTTPS +verification exchange. The challenge is stripped before either the authority or +application sees it. Normal traffic has no challenge, so no non-empty proof +header is emitted. The shared HTTPS reader permits up to four interim 1xx +responses before the final status for both inferred and explicit routes; explicit +final-status and reviewed-redirect rules remain unchanged. + +`healthcheck.x-hack-http` declares an explicit readiness probe native startup uses for a service. It must be the only key in that service's `healthcheck`: it cannot be combined with `test`, `disable`, `interval`, `timeout`, `start_period`, `start_interval` or `retries`. Docker Compose ignores the extension field. All six diff --git a/docs/reference/v5-model-coverage.md b/docs/reference/v5-model-coverage.md index 1dd7bf9d5..54a1671d5 100644 --- a/docs/reference/v5-model-coverage.md +++ b/docs/reference/v5-model-coverage.md @@ -49,6 +49,7 @@ must be attached to the eventual exact integration revision. | Settled retained Compose route-attempt retirement; pinned references; interrupted deletion | **Coverage gap:** filesystem journal compaction is not modeled; native-route-stop covers a different stop protocol | Instance mutation lock and exact surviving claim witnesses in [journal compaction](../../src/lib/native-compose-route-claims.ts); [claim regressions](../../tests/native-compose-route-claims.test.ts) and [retained lifecycle integration](../../tests/native-compose-adoption-routing-generation.test.ts) | 24 successful operations keep two attempt directories and constant Linux descriptors; four integrated lifecycle operations check directory counts. Injected unlink failures cover same-store and fresh-store cleanup retries. Assumes cooperative same-user writers, the caller's mutation lock and working filesystem synchronization. No process-kill/power-loss durability proof or bound on unresolved/over-budget histories. | | Graph cleanup container stop batch: bounded in-flight stops, shared worker queue, connect-stage retry, batch-wide unreachable state, deadline expiry, worker panic | **Coverage gap:** no maintained model covers scheduling inside the stop batch; native-route-stop covers a different stop protocol | `dispatch` and `attempt` in [the stop batch](../../packages/runtime-core/src/provider/engine/stop.rs): at most 4 stops in flight, every worker joined; only `connect`/`connect_timeout` failures retry (25 ms doubling to 500 ms, within the batch deadline and 30 s of a stop's first connect failure); once one stop exhausts its window, later connect failures return at once; a deadline reached during backoff keeps the connect stage; `response`, `timeout` and `transport` never retry. Regressions in the same file's `provider::engine::stop` tests (bounded in-flight socket server, late socket, missing socket, 14-stop retry counts, persistent connect failures, oversleeping backoff, non-retried stages, worker panic) and `fourteen_service_connect_failures_keep_resumable_progress_and_retry_only_unstopped` | Tests cover exact attempt counts and the in-flight bound for fixed interleavings; no model explores all worker schedules or proves at most one delivered request per stop. Assumes a failed connect sent no request bytes and that the stop budget covers fully serialized grace periods. Graph-level resume after a partial batch is covered by tests only. Host qualification of the descriptor-pressure root cause (HACK-1302) remains open. | | Concurrent stop/publication; route claim and relay lifetime | native-route-stop, shared-https-lifetime, previous-boot-shared-https, relay-lifecycle-barrier, relay-authorization, relay-staging, relay-fence | Fenced route removal, relay owner, ready/removed protocol, authenticated observations | Native DNS/TLS/browser behavior; old-boot uncertainty and external routing. | +| Inferred Caddy route readiness without authored probes | Existing probe generation, exec identity and freshness checks are reused; no new lifecycle transition | `http_probe.rs` and compiled `guest/http-probe.c` tests cover TCP refusal, complete HTTP 2xx–4xx, 5xx, EOF, timeout, partial/oversized headers and bounded interim responses. Compiled-helper startup regressions cover no listener and boot-time 5xx reaching a named service failure; route diagnostics name the port and remedies. A compiled production-transition fake-clock test drives graph readiness at 120 s, with the old 60 s grace as a negative control; it injects request outcomes and does not test real elapsed scheduling. A cadence regression covers the switch after first success. Bridge reservation regressions preserve running relay assignments when ready endpoints disappear. Project contracts preserve explicit checks and refuse inferred service_healthy dependencies; TypeScript route/preflight tests require the exact reviewed inferred policy. HTTPS tests require a fresh upstream response challenge for inferred routes | 32 route slots, 1 s startup/30 s steady interval, 1 s deadline, 8 KiB headers, four interim responses, three failed samples and 240 s initial grace. The separate 300 s graph budget includes earlier services and can still return unnamed graph_timeout. Bridge registry tests do not exercise continued route traffic; that remains a real-Caddy/VM check. No model explores HTTP parsing or Caddy handler behavior. TLS fixtures simulate the proxy marker; an actual Caddy/authority/VM run must qualify application 404 versus authority denial, SNI mismatch and missing upstream. | | Shared host helper instance: overlapping ups, session adoption, stale create decisions, replacement before finalize, epoch-record failure, interrupted cleanup, concurrent down, displaced launches, failed down session stop | shared-helper-instance | `src/lib/project-shared-helpers.ts` (shutdown generation, `assertCurrent`, `SharedHelperSessionOwnership`); `src/lib/project-lifecycle-controller.ts` session decision, the locked creation transaction (revalidate, publish, create, environment, record, stop on failure, all under one lock), launches fenced by the lock and a token check, abort and signal cleanup through the same epoch decision (`runIfOwned`), an unknown exact-presence answer failing the inspection, and the locked conditional finalize; `stopSessionBeforeReleasingHelpers` (down, restart, native retirement keep holders when the session stop fails); `runMuxCommand` bounds every lifecycle and backend mux call so it settles before the lock is released, and reports a timeout as unknown (never absence); `tests/mux-command-timeout.test.ts`; `tests/project-lifecycle-controller-ownership.test.ts`, `tests/project-shared-helpers.test.ts`, `tests/native-project-down.test.ts`, e2e `shared-host-helpers` | Helper holder/rollback and claim protocol are not modeled (claim fixtures cover them); timeout settlement is an assumption the code does not guarantee (a killed tmux client's queued command may still apply, so a late kill or window command can reach a newer session; consequences and the session-ID fence, not in this slice, are in the README); zellij is refused for shared-helper lifecycle sessions (token written after creation); the gap between a down's session stop and its release (an up that captured after the stopping mark can still succeed there) is a documented first-slice limitation; real mux/process/lock-file behavior. | | Engine restart policies: stop versus pending restart, guest engine restart, stale incarnations, terminal proof | service-restart-policy | Receipt-admitted policy, disarm-before-stop and `restarting` stop preflight in `shutdown.rs`; admission refusals for incarnation-bound delivery classes in `restart_policy.rs`; owner-driven `Rebind` after full retirement, fresh ownership/attachment checks and a new reservation in `bridges/restart_rebind.rs`; route probe sample-generation rearm; Rust tests for admission, endpoint identity, port-busy refusal, disarm and stop preflight; portable selected-child readiness guard plus macOS two-real-child polling regression prevent historical sibling exits from rejecting a replacement or misclassifying its own listener | Real guest engine behavior (crash restart, stop during backoff, disarm before deletion) needs native host evidence; several services, selected-child readiness during sibling failure, backoff timing and retry counting are not modeled. Same-port/route reachability after a policy restart, route probe rearm and per-slot child retirement still require real-VM host evidence. Rebind abstracts retirement/reservation/publication; relay lifecycle, fence and staging models cover their separate failure boundaries. | | Foreground Ctrl-C down hooks: concurrent `down`, second interrupt, crash between or during hooks, stop during `up.after`, next generation | native-foreground-down-hooks | Value-free compare-and-swap journal in [down hooks](../../src/backends/native-project-down-hooks.ts), foreground orchestration in [foreground stop](../../src/backends/native-project-foreground-stop.ts) and `hack down` in [native down](../../src/backends/native-project-down.ts); [journal](../../tests/native-project-down-hooks.test.ts), [foreground](../../tests/native-project-start-stop-hooks.test.ts) and [down](../../tests/native-project-down-journal.test.ts) regressions | Durable writes are atomic in the model; real fsync/power loss, a stale journal lock after a crash inside the lock, and the live process-group behavior of the graph owner on a host are not covered. One generation and at most two `down` invocations are explored. | diff --git a/packages/runtime-core/build.rs b/packages/runtime-core/build.rs index d97bfe0a9..1a1d15429 100644 --- a/packages/runtime-core/build.rs +++ b/packages/runtime-core/build.rs @@ -193,6 +193,21 @@ fn build_http_probe() { .status() .expect("build host HTTP probe test driver"); assert!(result.success(), "host HTTP probe build failed"); + println!("cargo:rerun-if-changed=tests/fixtures/http-probe-clock.c"); + let result = Command::new("cc") + .args([ + "-std=c11", + "-O2", + "-Wall", + "-Wextra", + "-Werror", + "tests/fixtures/http-probe-clock.c", + "-o", + ]) + .arg(out.join("http-probe-clock-host")) + .status() + .expect("build fake-clock probe test driver"); + assert!(result.success(), "fake-clock HTTP probe build failed"); } fn build_loopback_identity() { diff --git a/packages/runtime-core/guest/http-probe.c b/packages/runtime-core/guest/http-probe.c index 41a994abf..468850432 100644 --- a/packages/runtime-core/guest/http-probe.c +++ b/packages/runtime-core/guest/http-probe.c @@ -45,7 +45,7 @@ static bool wait_socket(int fd, short events, uint64_t deadline) { } return false; } -static bool request(int port, const char *path, uint64_t timeout) { +static bool request(int port, const char *path, uint64_t timeout, bool route) { uint64_t deadline = millis(CLOCK_MONOTONIC) + timeout; int fd = socket(AF_INET, SOCK_STREAM, 0); if (fd < 0) return false; @@ -71,22 +71,35 @@ static bool request(int port, const char *path, uint64_t timeout) { sent += (size_t)n; } char headers[8193]; - size_t used = 0; - while (used < sizeof(headers) - 1) { + size_t used = 0, total = 0; + unsigned interim = 0; + while (used < sizeof(headers) - 1 && total < sizeof(headers) - 1) { if (!wait_socket(fd, POLLIN, deadline)) goto done; - ssize_t n = recv(fd, headers + used, sizeof(headers) - 1 - used, 0); + ssize_t n = recv(fd, headers + used, sizeof(headers) - 1 - total, 0); if (n < 0 && (errno == EAGAIN || errno == EINTR)) continue; if (n <= 0) goto done; - size_t end = used + (size_t)n; - bool complete = false; - while (used < end) { - if (!headers[used++]) goto done; - if (used >= 4 && !memcmp(headers + used - 4, "\r\n\r\n", 4)) { complete = true; break; } - } - if (complete) { - ok = used >= 16 && (!memcmp(headers, "HTTP/1.1 ", 9) || !memcmp(headers, "HTTP/1.0 ", 9)) && - headers[9] == '2' && headers[10] >= '0' && headers[10] <= '9' && - headers[11] >= '0' && headers[11] <= '9' && headers[12] == ' '; + used += (size_t)n; + total += (size_t)n; + for (;;) { + size_t end = 0; + for (size_t i = 4; i <= used; i++) { + if (!memcmp(headers + i - 4, "\r\n\r\n", 4)) { end = i; break; } + } + if (!end) break; + if (end < 16 || memchr(headers, 0, end) || + (memcmp(headers, "HTTP/1.1 ", 9) && memcmp(headers, "HTTP/1.0 ", 9)) || + headers[9] < '1' || headers[9] > '5' || headers[10] < '0' || headers[10] > '9' || + headers[11] < '0' || headers[11] > '9' || headers[12] != ' ') goto done; + int status = (headers[9] - '0') * 100 + (headers[10] - '0') * 10 + headers[11] - '0'; + if (route && status >= 100 && status < 200 && status != 101) { + // Early hints and continue responses consume the same byte/time + // budget; they never satisfy readiness without a final status. + if (++interim > 4) goto done; + used -= end; + memmove(headers, headers + end, used); + continue; + } + ok = status >= 200 && status < (route ? 500 : 300); goto done; } } @@ -111,8 +124,23 @@ static bool publish(int directory, const char *name, const char *generation, uin // Ephemeral status is atomically replaced, never a durable readiness receipt. return ok && !renameat(directory, "status.pending", directory, name); } +// Shared with the fake-clock harness: only success ends grace early. The clock +// and request outcome come from the resident loop, not from retained samples. +static void sample_health(bool success, uint64_t elapsed, uint64_t grace, uint64_t retries, + uint64_t *failures, int *health, bool *ever_healthy) { + if (success) { *failures = 0; *health = 1; *ever_healthy = true; } + else if (*ever_healthy || elapsed >= grace) { + if (*failures < retries) ++*failures; + if (*failures >= retries) *health = 0; + } +} +static uint64_t sample_period(bool route, bool ever_healthy, uint64_t interval) { + return route && !ever_healthy && interval > 1000 ? 1000 : interval; +} int main(int argc, char **argv) { - if (argc != 9) return 125; + if (argc != 9 && argc != 10) return 125; + bool route = argc == 10; + if (route && strcmp(argv[9], "route")) return 125; int port = (int)number(argv[1], 1, 65535); size_t path_length = strlen(argv[2]); if (!path_length || path_length > 512 || argv[2][0] != '/') return 125; @@ -146,23 +174,23 @@ int main(int argc, char **argv) { if (!publish(directory, "status", argv[8], sequence, health, failures, 0)) return 125; while (!stopping) { uint64_t begun = millis(CLOCK_MONOTONIC); - bool success = request(port, argv[2], timeout); + bool success = request(port, argv[2], timeout, route); if (stopping) break; uint64_t finished = millis(CLOCK_MONOTONIC); - if (success) { failures = 0; health = 1; ever_healthy = true; } - else if (ever_healthy || finished - start >= grace) { - if (failures < retries) failures++; - if (failures >= retries) health = 0; - } + sample_health(success, finished - start, grace, retries, &failures, &health, &ever_healthy); ++sequence; // One bounded diagnostic survives later success; it never supplies current readiness. // On the first request only, health == 1 means success even during startup grace. if (sequence == 1 && !publish(directory, "first-sample", argv[8], sequence, health, failures, finished - begun)) return 125; if (!publish(directory, "status", argv[8], sequence, health, failures, finished - begun)) return 125; - uint64_t next = begun + interval; + // Generated route probes check quickly during initial compilation, then + // use the reviewed steady interval for the rest of this exec generation. + // An unhealthy sample never re-enables the fast startup cadence. + uint64_t period = sample_period(route, ever_healthy, interval); + uint64_t next = begun + period; while (!stopping && millis(CLOCK_MONOTONIC) < next) { uint64_t delay = next - millis(CLOCK_MONOTONIC); - if (delay > interval) break; + if (delay > period) break; struct timespec pause = {.tv_sec = (time_t)(delay / 1000), .tv_nsec = (long)(delay % 1000) * 1000000}; if (nanosleep(&pause, NULL) && errno != EINTR) return 125; } diff --git a/packages/runtime-core/src/error.rs b/packages/runtime-core/src/error.rs index 7ddf9b888..1272750b5 100644 --- a/packages/runtime-core/src/error.rs +++ b/packages/runtime-core/src/error.rs @@ -130,9 +130,18 @@ pub struct PlanChangeDiagnostic { pub field: PlanChangeField, } +/// Only the public reviewed port of an unhealthy inferred probe. No path, +/// response, other configuration or application output crosses this boundary. +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct RouteProbeDiagnostic { + pub version: u8, + pub port: u16, +} + /// Value-free refusal locations, boxed together so every /// `Result<_, CandidateError>` stays small; serialized flat as -/// `process_refusals` and `plan_change`. +/// `process_refusals`, `plan_change` and `route_probe`. #[derive(Debug, Default, Serialize)] pub struct RefusalDiagnostics { /// Every service whose selected process a private wrapper refused, found in @@ -142,6 +151,8 @@ pub struct RefusalDiagnostics { /// Value-free location of a refused retained-run edit; see [`PlanChangeDiagnostic`]. #[serde(skip_serializing_if = "Option::is_none")] pub plan_change: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub route_probe: Option, } /// Closed diagnostic vocabulary for the read-only log follower. No stream bytes. @@ -232,6 +243,12 @@ impl CandidateError { self } + pub fn with_route_probe(mut self, port: u16) -> Self { + self.refusals.get_or_insert_default().route_probe = + Some(RouteProbeDiagnostic { version: 1, port }); + self + } + /// Name every service and field a private process wrapper refused. pub fn with_process_refusals(mut self, refusals: Vec) -> Self { self.refusals.get_or_insert_default().process_refusals = Some(ProcessRefusalsDiagnostic { diff --git a/packages/runtime-core/src/project/compose/graph.rs b/packages/runtime-core/src/project/compose/graph.rs index 250ae11ab..759aeb3c9 100644 --- a/packages/runtime-core/src/project/compose/graph.rs +++ b/packages/runtime-core/src/project/compose/graph.rs @@ -69,10 +69,11 @@ pub(super) fn validate( )); } if condition.condition == "service_healthy" - && !target - .healthcheck - .as_ref() - .is_some_and(|h| !h.disabled && (h.test.is_some() || h.native_http.is_some())) + && !target.healthcheck.as_ref().is_some_and(|h| { + !h.disabled + && h.native_http.as_ref().is_none_or(|p| p.policy.is_none()) + && (h.test.is_some() || h.native_http.is_some()) + }) { diagnostics.push(Diagnostic::error("missing_healthcheck",&field,"A healthy dependency needs an explicit enabled healthcheck; image health defaults are not assumed.")); } diff --git a/packages/runtime-core/src/project/compose/service.rs b/packages/runtime-core/src/project/compose/service.rs index 778c4342e..4aee19da9 100644 --- a/packages/runtime-core/src/project/compose/service.rs +++ b/packages/runtime-core/src/project/compose/service.rs @@ -390,6 +390,12 @@ fn health( problem("invalid_healthcheck", "Invalid native HTTP declaration.") })?; probe.validate()?; + if probe.policy.is_some() { + return Err(problem( + "invalid_healthcheck", + "Native HTTP policy is generated from routing, not authored in a healthcheck.", + )); + } if m.keys().any(|key| key != "x-hack-http") { return Err(problem( "invalid_healthcheck", @@ -1101,6 +1107,31 @@ pub(super) fn compile( "Dependency cache volume must be mounted by its declaring service; values omitted.", )); } + let mut healthcheck = health( + m.get("healthcheck"), + &format!("{field}.healthcheck"), + diagnostics, + )?; + if !m.contains_key("healthcheck") { + if let Some(route) = &routing { + healthcheck = Some(HealthPlan { + native_http: Some(crate::provider::http_probe::HttpProbe::inferred_route( + route.port, + )), + disabled: false, + test: None, + interval_nanos: None, + timeout_nanos: None, + start_period_nanos: None, + start_interval_nanos: None, + retries: None, + }); + diagnostics.push(Diagnostic::warning( + "inferred_route_probe", &format!("{field}.healthcheck"), + "Native infers a route-port probe requiring TCP and complete HTTP 200-499 response headers. HTTP 5xx, EOF, timeouts and non-HTTP responses are not ready. No explicit healthcheck was changed.", + )); + } + } result.insert(name.clone(),ServicePlan { image,build, command:command(m.get("command"),&format!("{field}.command"))?, @@ -1111,7 +1142,7 @@ pub(super) fn compile( environment:redacted_map(m.get("environment"),true)?,extra_hosts, environment_files:environment_files(project,base,m.get("env_file"),&format!("{field}.env_file"),diagnostics)?, environment_precedence:"Compose environment keys override explicitly listed env_file inputs; bare keys and references remain unresolved; no ambient/.env values are loaded".into(), - healthcheck:health(m.get("healthcheck"),&format!("{field}.healthcheck"),diagnostics)?, + healthcheck, ports:ports(m.get("ports"),&format!("{field}.ports"),diagnostics)?,exposed_ports,networks,network_aliases,network_mode:if ["project-networks", "none"].contains(&network_mode) {network_mode.into()} else {"[unsupported]".into()}, limits:limits(m,&field,diagnostics)?,read_only:boolean(m.get("read_only"),false)?,init:boolean(m.get("init"),false)?,restart, working_dir:m.get("working_dir").map(|v| safe_absolute(string(v)?)).transpose()?, diff --git a/packages/runtime-core/src/project/execution.rs b/packages/runtime-core/src/project/execution.rs index a361ce31b..b6d57daf0 100644 --- a/packages/runtime-core/src/project/execution.rs +++ b/packages/runtime-core/src/project/execution.rs @@ -101,6 +101,12 @@ pub trait Driver { fn record(&mut self, event: Event<'_>) -> Result<(), CandidateError>; fn start(&mut self, service: &str) -> Result<(), CandidateError>; fn observe(&mut self, service: &str) -> Result; + + /// Add reviewed readiness context to an observed failure, without copying + /// application output or private configuration into the error. + fn service_failure(&self, service: &str, state: Observation) -> CandidateError { + failed_service_error("graph_service_failed", service, state) + } } fn error(code: &'static str, message: impl Into) -> CandidateError { CandidateError::new(code, message) @@ -266,7 +272,7 @@ fn inspect( observations.insert(name.into(), state); } if state.failed() { - return Err(failed_service_error("graph_service_failed", name, state)); + return Err(driver.service_failure(name, state)); } Ok(state) } @@ -293,8 +299,8 @@ pub(crate) fn failed_service_error( ) .with_service(name) } -fn before_deadline(deadline: Instant) -> Result<(), CandidateError> { - if Instant::now() >= deadline { +fn before_deadline(deadline: Instant, now: Instant) -> Result<(), CandidateError> { + if now >= deadline { return Err(error( "graph_timeout", "Graph readiness deadline exceeded; owned state is retained.", @@ -313,10 +319,20 @@ pub fn run( } fn run_with_wait( + graph: &Graph, + driver: &mut impl Driver, + timeout: Duration, + wait: impl FnMut(Duration), +) -> Result<(), CandidateError> { + run_with_clock(graph, driver, timeout, wait, Instant::now) +} + +fn run_with_clock( graph: &Graph, driver: &mut impl Driver, timeout: Duration, mut wait: impl FnMut(Duration), + mut now: impl FnMut() -> Instant, ) -> Result<(), CandidateError> { let order = graph.execution_order()?; if timeout.is_zero() || timeout > Duration::from_secs(600) { @@ -325,15 +341,15 @@ fn run_with_wait( "Graph timeout must be greater than zero and at most 600 seconds.", )); } - let deadline = Instant::now() + timeout; + let deadline = now() + timeout; let mut started = BTreeSet::::new(); let mut observations = BTreeMap::new(); loop { - before_deadline(deadline)?; + before_deadline(deadline, now())?; driver.check_cancelled()?; for name in &started { inspect(driver, name, &mut observations)?; - before_deadline(deadline)?; + before_deadline(deadline, now())?; driver.check_cancelled()?; } for name in &order { @@ -349,7 +365,7 @@ fn run_with_wait( break; } let state = inspect(driver, dependency, &mut observations)?; - before_deadline(deadline)?; + before_deadline(deadline, now())?; driver.check_cancelled()?; if !state.satisfies(*condition) { ready = false; @@ -359,10 +375,10 @@ fn run_with_wait( if !ready { continue; } - before_deadline(deadline)?; + before_deadline(deadline, now())?; driver.check_cancelled()?; driver.record(Event::StartIntent { service: name })?; - before_deadline(deadline)?; + before_deadline(deadline, now())?; driver.check_cancelled()?; if let Err(failure) = driver.start(name) { driver.record(Event::StartUncertain { service: name })?; @@ -371,14 +387,14 @@ fn run_with_wait( driver.record(Event::Started { service: name })?; started.insert(name.clone()); inspect(driver, name, &mut observations)?; - before_deadline(deadline)?; + before_deadline(deadline, now())?; driver.check_cancelled()?; } if started.len() == graph.services.len() { let mut ready = true; for (name, service) in &graph.services { let state = inspect(driver, name, &mut observations)?; - before_deadline(deadline)?; + before_deadline(deadline, now())?; driver.check_cancelled()?; ready &= state.satisfies(service.ready); } @@ -387,10 +403,14 @@ fn run_with_wait( return Ok(()); } } - wait(Duration::from_millis(100).min(deadline.saturating_duration_since(Instant::now()))); + wait(Duration::from_millis(100).min(deadline.saturating_duration_since(now()))); } } +#[cfg(all(test, feature = "native-http-probe"))] +#[path = "route_probe_clock_tests.rs"] +mod route_probe_clock_tests; + #[cfg(test)] mod tests { use super::*; diff --git a/packages/runtime-core/src/project/route_probe_clock_tests.rs b/packages/runtime-core/src/project/route_probe_clock_tests.rs new file mode 100644 index 000000000..a6d1a5c59 --- /dev/null +++ b/packages/runtime-core/src/project/route_probe_clock_tests.rs @@ -0,0 +1,114 @@ +//! Fake time drives both the production probe transition and graph deadline. +//! Socket/HTTP parsing, live exec identity and publication are tested separately. +use super::*; +use crate::provider::http_probe::HttpProbe; +use std::{cell::Cell, process::Command}; + +struct Probed<'a> { + clock: &'a Cell, + config: HttpProbe, + samples: Vec<(u64, u64, u64)>, + started: bool, + ready: bool, +} +impl Driver for Probed<'_> { + fn record(&mut self, event: Event<'_>) -> Result<(), CandidateError> { + if matches!(event, Event::Ready) { + self.ready = true; + } + Ok(()) + } + fn start(&mut self, service: &str) -> Result<(), CandidateError> { + assert_eq!(service, "web"); + assert!(!self.started, "a service must never be replayed"); + self.started = true; + Ok(()) + } + fn observe(&mut self, _: &str) -> Result { + let elapsed = self.clock.get().as_millis() as u64; + let (sequence, &(observed, health, failures)) = self + .samples + .iter() + .enumerate() + .rev() + .find(|(_, (time, _, _))| *time <= elapsed) + .unwrap(); + let generation = "0".repeat(32); + let raw = format!( + "v1 {generation} {sequence} {health} {failures} {} 0\n", + observed + 100_000 + ); + Ok(Observation::Running { + health: self + .config + .health(&raw, &generation, true, elapsed + 100_000)?, + }) + } +} + +#[test] +fn a_route_first_answering_at_120_seconds_starts_under_the_graph_budget() { + let graph = Graph::from_services(BTreeMap::from([( + "web".into(), + Service { + dependencies: BTreeMap::new(), + ready: Condition::Healthy, + }, + )])) + .unwrap(); + // The old grace is a negative control: the same delayed success must fail + // early with it. The reviewed default must reach graph readiness at 120 s. + for grace in [60_000, HttpProbe::inferred_route(3000).start_period_ms] { + let mut config = HttpProbe::inferred_route(3000); + config.start_period_ms = grace; + let output = Command::new(concat!(env!("OUT_DIR"), "/http-probe-clock-host")) + .args([ + grace.to_string(), + config.interval_ms.to_string(), + config.timeout_ms.to_string(), + config.retries.to_string(), + "120000".into(), + ]) + .output() + .unwrap(); + assert!(output.status.success()); + assert!(output.stderr.is_empty()); + let samples = String::from_utf8(output.stdout) + .unwrap() + .lines() + .map(|line| { + let fields: Vec = line + .split(' ') + .map(|field| field.parse().unwrap()) + .collect(); + (fields[0], fields[1], fields[2]) + }) + .collect(); + let clock = Cell::new(Duration::ZERO); + let base = Instant::now(); + let mut driver = Probed { + clock: &clock, + config, + samples, + started: false, + ready: false, + }; + let result = run_with_clock( + &graph, + &mut driver, + Duration::from_secs(300), + |delay| clock.set(clock.get() + delay), + || base + clock.get(), + ); + assert!(driver.started); + if grace == 60_000 { + assert_eq!(result.unwrap_err().code, "graph_service_failed"); + assert!(!driver.ready); + assert!(clock.get() < Duration::from_secs(120)); + } else { + result.unwrap(); + assert!(driver.ready); + assert_eq!(clock.get(), Duration::from_secs(120)); + } + } +} diff --git a/packages/runtime-core/src/provider/graph/bridges.rs b/packages/runtime-core/src/provider/graph/bridges.rs index d8ce59ad4..d203c4352 100644 --- a/packages/runtime-core/src/provider/graph/bridges.rs +++ b/packages/runtime-core/src/provider/graph/bridges.rs @@ -1224,7 +1224,7 @@ mod tests { assert_eq!(persisted.slots[&0].run, "b".repeat(32)); } #[test] - fn batch_commit_is_atomic_across_late_endpoint_failure_and_success() { + fn missing_ready_endpoints_refuse_new_routes_without_retiring_running_relays() { let fixture_root = super::super::tests::Fixture::new(); let candidate = Candidate::discover(&fixture_root.0).unwrap(); let owner = "a".repeat(32); @@ -1233,6 +1233,17 @@ mod tests { initialize_registry(&candidate, &owner, 3).unwrap(); let mut store = fixture(); store.owner = owner.clone(); + let existing = store.slots.get_mut(&0).unwrap(); + existing.phase = "running".into(); + existing.relay = Some(relay::Relay { + transport: relay::Transport::ReservationV1, + launch_serial: 1, + binary_sha256: "f".repeat(64), + target_pid: 42, + target_start: 123, + port: 3000, + }); + store.next_launch_serial = 1; save(&candidate, &store).unwrap(); let before = fs::read(root(&candidate).join("state.json")).unwrap(); let requested = BTreeMap::from([ @@ -1303,6 +1314,8 @@ mod tests { assert_eq!(reserved["web"].slot, 2); let persisted = read_owner_registry(&candidate, &owner, 3, false).unwrap(); assert_eq!(persisted.slots.len(), 3); + assert_eq!(persisted.slots[&0].phase, "running"); + assert_eq!(persisted.slots[&0].relay.as_ref().unwrap().launch_serial, 1); for (service, route) in reserved { assert_eq!(persisted.slots[&route.slot].run, run); assert_eq!(persisted.slots[&route.slot].service, service); diff --git a/packages/runtime-core/src/provider/graph/host_relay.rs b/packages/runtime-core/src/provider/graph/host_relay.rs index 0786c8464..590d8353e 100644 --- a/packages/runtime-core/src/provider/graph/host_relay.rs +++ b/packages/runtime-core/src/provider/graph/host_relay.rs @@ -1151,6 +1151,7 @@ mod tests { timeout_ms: 100, retries: 1, start_period_ms: 0, + policy: None, }, allocation: "1".repeat(32), generation: "2".repeat(32), diff --git a/packages/runtime-core/src/provider/graph/mod.rs b/packages/runtime-core/src/provider/graph/mod.rs index 180f4f777..b5106fbb5 100644 --- a/packages/runtime-core/src/provider/graph/mod.rs +++ b/packages/runtime-core/src/provider/graph/mod.rs @@ -992,6 +992,15 @@ impl Session<'_, '_> { } } impl Driver for Session<'_, '_> { + fn service_failure(&self, name: &str, observation: Observation) -> CandidateError { + probes::service_failure( + "graph_service_failed", + name, + observation, + self.receipt.probes.get(name).map(|probe| &probe.config), + ) + } + fn check_cancelled(&self) -> Result<(), CandidateError> { if let Some(startup) = &self.startup { startup.check_cancelled()?; diff --git a/packages/runtime-core/src/provider/graph/probes.rs b/packages/runtime-core/src/provider/graph/probes.rs index 65eb0da7d..a596f4a5b 100644 --- a/packages/runtime-core/src/provider/graph/probes.rs +++ b/packages/runtime-core/src/provider/graph/probes.rs @@ -1,4 +1,4 @@ -use super::super::http_probe::HttpProbe; +use super::super::http_probe::{HttpProbe, HttpProbePolicy}; use super::*; #[cfg(feature = "native-http-probe")] use { @@ -18,6 +18,82 @@ pub struct Probe { pub started_ms: u64, pub phase: String, } + +/// Only an inferred route's unhealthy observation gets route guidance. Explicit +/// healthchecks, exits and deaths keep their existing diagnostic semantics. +pub(super) fn service_failure( + code: &'static str, + name: &str, + observation: Observation, + probe: Option<&HttpProbe>, +) -> CandidateError { + if let Some(probe) = probe.filter(|probe| { + probe.policy == Some(HttpProbePolicy::Route) + && matches!( + observation, + Observation::Running { + health: execution::Health::Unhealthy + } + ) + }) { + return error( + code, + format!( + "Owned graph service {name} is unhealthy on inferred route port {}; no further services will start. Check hack logs {name} in the same project/branch, fix the route port, or declare healthcheck.x-hack-http with a longer start_period_ms and a readiness path, within the 300-second graph budget.", + probe.port + ), + ) + .with_service(name) + .with_route_probe(probe.port); + } + execution::failed_service_error(code, name, observation) +} + +#[cfg(test)] +mod route_failure_tests { + use super::*; + + #[test] + fn only_an_unhealthy_inferred_probe_names_its_route_and_remedies() { + let inferred = HttpProbe::inferred_route(3000); + let unhealthy = Observation::Running { + health: execution::Health::Unhealthy, + }; + for code in ["graph_service_failed", "graph_startup_application_failed"] { + let failure = service_failure(code, "web", unhealthy, Some(&inferred)); + assert_eq!(failure.code, code); + assert_eq!(failure.service.as_deref(), Some("web")); + assert!( + failure + .message + .contains("web is unhealthy on inferred route port 3000") + ); + assert!(failure.message.contains("hack logs web")); + assert!(failure.message.contains("healthcheck.x-hack-http")); + assert!(failure.message.contains("longer start_period_ms")); + let value = serde_json::to_value(failure).unwrap(); + assert_eq!(value["route_probe"], json!({"version":1,"port":3000})); + assert!(value.get("message_fixed").is_none()); + } + let mut explicit = inferred.clone(); + explicit.policy = None; + for (observation, probe) in [ + (unhealthy, None), + (unhealthy, Some(&explicit)), + (Observation::Dead, Some(&inferred)), + (Observation::Exited { code: 1 }, Some(&inferred)), + ] { + let failure = service_failure("graph_service_failed", "web", observation, probe); + assert!(!failure.message.contains("inferred route")); + assert!( + serde_json::to_value(failure) + .unwrap() + .get("route_probe") + .is_none() + ); + } + } +} pub(super) fn token() -> Result { use std::io::Read; let mut bytes = [0u8; 16]; @@ -280,6 +356,7 @@ mod observation_clock_tests { timeout_ms: 40, retries: 3, start_period_ms: 0, + policy: None, }, allocation: "a".repeat(32), generation: "b".repeat(32), diff --git a/packages/runtime-core/src/provider/graph/routes.rs b/packages/runtime-core/src/provider/graph/routes.rs index e73a651de..503339d7c 100644 --- a/packages/runtime-core/src/provider/graph/routes.rs +++ b/packages/runtime-core/src/provider/graph/routes.rs @@ -56,6 +56,10 @@ fn validate_services( || service.network_mode != "project-networks" || service.networks.is_empty() || !probe.is_some_and(|probe| probe.port == route.port && probe.validate().is_ok()) + || probe.is_some_and(|probe| { + probe.policy.is_some() + && *probe != super::super::http_probe::HttpProbe::inferred_route(route.port) + }) || !valid_intent(route) || !["caddy", "caddy.reverse_proxy", "caddy.tls"] .iter() diff --git a/packages/runtime-core/src/provider/graph/startup/runtime.rs b/packages/runtime-core/src/provider/graph/startup/runtime.rs index c6bc69c09..92f93650d 100644 --- a/packages/runtime-core/src/provider/graph/startup/runtime.rs +++ b/packages/runtime-core/src/provider/graph/startup/runtime.rs @@ -121,12 +121,12 @@ fn stage_refused(code: &'static str) -> CandidateError { /// An owned service failed while dependency relays were attached. The stage code /// stays `graph_startup_application_failed`; the message and service name the /// failed service (exit code, death or health) instead of a validation failure. -fn application_failed(name: &str, observation: Observation) -> CandidateError { - crate::project::execution::failed_service_error( - "graph_startup_application_failed", - name, - observation, - ) +fn application_failed( + name: &str, + observation: Observation, + probe: Option<&crate::provider::http_probe::HttpProbe>, +) -> CandidateError { + probes::service_failure("graph_startup_application_failed", name, observation, probe) } // Fresh and restored graphs both provision a new container generation. Retained // stopped/ready receipts cannot authorize listener attachment. @@ -864,7 +864,11 @@ impl Driver for HostRelayRuntime { state::write(&root.join("state.json"), receipt)?; // Name the failed service, as a graph without dependency // bindings does, rather than a dependency validation failure. - return Err(application_failed(name, observation)); + return Err(application_failed( + name, + observation, + receipt.probes.get(name).map(|probe| &probe.config), + )); } if let Err(error) = verify_exited_listener(receipt.readiness.get(name), &observed) { if error.code == "graph_startup_listener_unexpected_exit" @@ -1651,17 +1655,17 @@ mod application_failure_tests { use super::*; #[test] fn an_application_failure_names_the_service_not_dependency_validation() { - let error = application_failed("deps", Observation::Exited { code: 3 }); + let error = application_failed("deps", Observation::Exited { code: 3 }, None); assert_eq!(error.code, "graph_startup_application_failed"); assert_eq!(error.service.as_deref(), Some("deps")); let message = error.message.to_string(); assert!(message.contains("deps exited with code 3"), "{message}"); assert!(!message.contains("validation"), "{message}"); - let dead = application_failed("deps", Observation::Dead); + let dead = application_failed("deps", Observation::Dead, None); assert!(dead.message.to_string().contains("dead or OOM-killed")); // An invalid service identifier is never echoed as the service. assert_eq!( - application_failed("bad name", Observation::Dead).service, + application_failed("bad name", Observation::Dead, None).service, None ); } diff --git a/packages/runtime-core/src/provider/http_probe.rs b/packages/runtime-core/src/provider/http_probe.rs index 75d422c0d..375c75531 100644 --- a/packages/runtime-core/src/provider/http_probe.rs +++ b/packages/runtime-core/src/provider/http_probe.rs @@ -12,6 +12,14 @@ pub struct HttpProbe { pub timeout_ms: u64, pub retries: u32, pub start_period_ms: u64, + /// Generated route readiness only; absent for authored HTTP healthchecks. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub policy: Option, +} +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum HttpProbePolicy { + Route, } fn invalid() -> CandidateError { CandidateError::new( @@ -26,6 +34,21 @@ fn generation_valid(value: &str) -> bool { .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) } impl HttpProbe { + /// A reviewed route without an authored healthcheck gets a bounded localhost + /// TCP connection followed by a complete HTTP response from that endpoint. + pub fn inferred_route(port: u16) -> Self { + // These defaults are part of retained route admission in Rust and + // TypeScript. Changing them requires a compatibility migration. + Self { + port, + path: "/".into(), + interval_ms: 30_000, + timeout_ms: 1_000, + retries: 3, + start_period_ms: 240_000, + policy: Some(HttpProbePolicy::Route), + } + } pub fn validate(&self) -> Result<(), CandidateError> { if self.port == 0 || !self.path.starts_with('/') @@ -56,7 +79,7 @@ impl HttpProbe { { return Err(invalid()); } - Ok(vec![ + let mut arguments = vec![ self.port.to_string(), self.path.clone(), self.interval_ms.to_string(), @@ -65,7 +88,11 @@ impl HttpProbe { self.start_period_ms.to_string(), directory.into(), generation.into(), - ]) + ]; + if self.policy == Some(HttpProbePolicy::Route) { + arguments.push("route".into()); + } + Ok(arguments) } /// A historical healthy sample cannot outlive its exec, generation, or freshness window. pub fn health( @@ -125,6 +152,8 @@ impl HttpProbe { } #[cfg(feature = "native-http-probe")] pub fn guest_binary() -> &'static [u8] { + // Built from this crate's guest source by build.rs and embedded in the same + // runtime as the planner; publication addresses it by its content hash. include_bytes!(concat!(env!("OUT_DIR"), "/http-probe")) } @@ -140,6 +169,7 @@ mod tests { timeout_ms: 40, retries: 3, start_period_ms: 0, + policy: None, } } #[test] @@ -202,6 +232,46 @@ mod tests { assert!(p.validate().is_err()); } + #[test] + fn inferred_policy_is_reviewed_while_explicit_serialization_and_arguments_stay_unchanged() { + let explicit = config(); + assert!( + serde_json::to_value(&explicit) + .unwrap() + .get("policy") + .is_none() + ); + assert_eq!( + explicit + .arguments("/private/probe", GENERATION) + .unwrap() + .len(), + 8 + ); + let inferred = HttpProbe::inferred_route(3000); + inferred.validate().unwrap(); + // Even a timeout on every startup sample exhausts grace and retries + // before the frontend's 300-second graph budget. Startup cadence is at + // most one second; the steady interval is thirty seconds. + assert_eq!(inferred.interval_ms, 30_000); + assert!( + inferred.start_period_ms + u64::from(inferred.retries) * (1_000 + inferred.timeout_ms) + < 300_000 + ); + assert_eq!(serde_json::to_value(&inferred).unwrap()["policy"], "route"); + assert_eq!( + inferred + .arguments("/private/probe", GENERATION) + .unwrap() + .last() + .unwrap(), + "route" + ); + let mut invalid = serde_json::to_value(&inferred).unwrap(); + invalid["policy"] = serde_json::json!("anything"); + assert!(serde_json::from_value::(invalid).is_err()); + } + #[cfg(feature = "native-http-probe")] mod native { use super::*; @@ -235,6 +305,23 @@ mod tests { Self::with_timeout(mode, grace, 40) } fn with_timeout(mode: usize, grace: u64, timeout: u64) -> Self { + Self::with_policy(mode, grace, timeout, None) + } + fn with_policy( + mode: usize, + grace: u64, + timeout: u64, + policy: Option, + ) -> Self { + Self::with_cadence(mode, grace, timeout, policy, 100) + } + fn with_cadence( + mode: usize, + grace: u64, + timeout: u64, + policy: Option, + interval: u64, + ) -> Self { let listener = TcpListener::bind("127.0.0.1:0").unwrap(); let port = listener.local_addr().unwrap().port(); listener.set_nonblocking(true).unwrap(); @@ -274,6 +361,19 @@ mod tests { } continue; } + 7 => b"HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n", + 8 => b"HTTP/1.1 499 Client Closed\r\n\r\n", + 9 => b"HTTP/1.1 500 Failed\r\n\r\n", + 10 => b"HTTP/", + 11 => b"", + 12 => b"HTTP/1.1 103 Early Hints\r\n\r\nHTTP/1.1 200 OK\r\n\r\n", + 13 => b"HTTP/1.1 103 Early Hints\r\n\r\nHTTP/1.1 503 Failed\r\n\r\n", + 14 => b"HTTP/1.1 103 Early Hints\r\n\r\nHTTP/1.1 103 Early Hints\r\n\r\nHTTP/1.1 103 Early Hints\r\n\r\nHTTP/1.1 103 Early Hints\r\n\r\nHTTP/1.1 103 Early Hints\r\n\r\nHTTP/1.1 200 OK\r\n\r\n", + 15 => { + let response = format!("HTTP/1.1 200 OK\r\nX-Large: {}\r\n\r\n", "x".repeat(8192)); + let _ = stream.write_all(response.as_bytes()); + continue; + } _ => b"NOT-HTTP 200 OK\r\n\r\n", }; let _ = stream.write_all(response); @@ -294,6 +394,11 @@ mod tests { config.port = port; config.start_period_ms = grace; config.timeout_ms = timeout; + config.interval_ms = interval; + config.policy = policy; + if policy == Some(HttpProbePolicy::Route) { + config.path = "/".into(); + } let mut fixture = Self { root, process: None, @@ -332,7 +437,10 @@ mod tests { Some((health, raw.split(' ').nth(2).unwrap().parse().unwrap())) } fn wait(&mut self, health: Health, after: u64) -> u64 { - let deadline = Instant::now() + Duration::from_secs(4); + self.wait_for(health, after, Duration::from_secs(4)) + } + fn wait_for(&mut self, health: Health, after: u64, budget: Duration) -> u64 { + let deadline = Instant::now() + budget; loop { assert!( self.process.as_mut().unwrap().try_wait().unwrap().is_none(), @@ -358,7 +466,9 @@ mod tests { let _ = p.wait(); } self.stop.store(1, Ordering::SeqCst); - self.server.take().unwrap().join().unwrap(); + if let Some(server) = self.server.take() { + server.join().unwrap(); + } fs::remove_dir_all(&self.root).unwrap(); } } @@ -474,6 +584,118 @@ mod tests { ); } #[test] + fn inferred_boot_failures_name_the_service_before_the_graph_deadline() { + use crate::project::execution::{ + self, Condition, Driver, Event, Graph, Observation, Service, + }; + use std::collections::BTreeMap; + struct Probed(Fixture); + impl Driver for Probed { + fn record(&mut self, _: Event<'_>) -> Result<(), CandidateError> { + Ok(()) + } + fn start(&mut self, _: &str) -> Result<(), CandidateError> { + Ok(()) + } + fn observe(&mut self, _: &str) -> Result { + Ok(Observation::Running { + health: self.0.sample().map_or(Health::Starting, |s| s.0), + }) + } + } + for listening in [false, true] { + // Exercise the compiled production helper, with the same grace + // and retry rules scaled down; one case has no TCP listener, + // the other keeps answering HTTP 500 through startup. + let mut fixture = Fixture::with_policy(9, 200, 40, Some(HttpProbePolicy::Route)); + if !listening { + fixture.stop.store(1, Ordering::SeqCst); + fixture.server.take().unwrap().join().unwrap(); + } + let mut driver = Probed(fixture); + let graph = Graph::from_services(BTreeMap::from([( + "web".into(), + Service { + dependencies: BTreeMap::new(), + ready: Condition::Healthy, + }, + )])) + .unwrap(); + let failure = + execution::run(&graph, &mut driver, Duration::from_secs(3)).unwrap_err(); + assert_eq!(failure.code, "graph_service_failed"); + assert_eq!(failure.service.as_deref(), Some("web")); + assert!(failure.message.contains("web") && failure.message.contains("unhealthy")); + if listening { + assert!(driver.0.accepted.load(Ordering::SeqCst) >= 3); + } + } + } + + #[test] + fn inferred_probe_slows_after_first_success_and_does_not_resume_fast_polling() { + // Shorten only the steady interval to keep this real-helper test + // bounded. Production admission pins it to 30 seconds. + let mut f = Fixture::with_cadence(9, 20_000, 40, Some(HttpProbePolicy::Route), 6_000); + let starting = f.wait(Health::Starting, 1); + f.mode.store(7, Ordering::SeqCst); + let ready = f.wait(Health::Healthy, starting); + f.mode.store(9, Ordering::SeqCst); + thread::sleep(Duration::from_millis(1_250)); + assert_eq!(f.sample().unwrap(), (Health::Healthy, ready)); + let down = f.wait_for(Health::Unhealthy, ready, Duration::from_secs(22)); + assert!(down >= ready + 3); + // Unhealthy status does not return to the one-second startup loop. + thread::sleep(Duration::from_millis(1_250)); + assert_eq!(f.sample().unwrap(), (Health::Unhealthy, down)); + f.mode.store(7, Ordering::SeqCst); + f.wait_for(Health::Healthy, down, Duration::from_secs(8)); + } + + #[test] + fn inferred_route_accepts_http_2xx_through_4xx_but_never_tcp_only_or_http_5xx() { + for (mode, expected) in [ + (1, Health::Healthy), + (2, Health::Unhealthy), // TCP without HTTP is not a ready route. + (3, Health::Healthy), // Redirect is observed, never followed. + (5, Health::Unhealthy), + (7, Health::Healthy), + (8, Health::Healthy), + (11, Health::Unhealthy), + (0, Health::Unhealthy), + (4, Health::Unhealthy), + (6, Health::Unhealthy), // Slowly delivered partial HTTP cannot fall back to TCP. + (9, Health::Unhealthy), + (10, Health::Unhealthy), + (12, Health::Healthy), + (13, Health::Unhealthy), + (14, Health::Unhealthy), + (15, Health::Unhealthy), + ] { + let mut f = Fixture::with_policy(mode, 0, 40, Some(HttpProbePolicy::Route)); + f.wait(expected, 0); + assert!(f.accepted.load(Ordering::SeqCst) > 0); + } + for mode in [2, 3, 5, 7, 8, 11] { + let mut f = Fixture::new(mode, 0); + f.wait(Health::Unhealthy, 0); + } + } + + #[test] + fn inferred_route_requires_tcp_and_later_5xx_revokes_readiness() { + let mut f = Fixture::with_policy(7, 0, 40, Some(HttpProbePolicy::Route)); + let ready = f.wait(Health::Healthy, 0); + f.mode.store(9, Ordering::SeqCst); + let down = f.wait(Health::Unhealthy, ready); + assert!(down >= ready + 3); + f.mode.store(7, Ordering::SeqCst); + let ready = f.wait(Health::Healthy, down); + f.stop.store(1, Ordering::SeqCst); + f.server.take().unwrap().join().unwrap(); + f.wait(Health::Unhealthy, ready); + } + #[test] fn first_sample_survives_recovery_without_overriding_live_health() { for mode in [0, 1] { let mut f = Fixture::new(mode, 3000); diff --git a/packages/runtime-core/tests/fixtures/http-probe-clock.c b/packages/runtime-core/tests/fixtures/http-probe-clock.c new file mode 100644 index 000000000..44db26018 --- /dev/null +++ b/packages/runtime-core/tests/fixtures/http-probe-clock.c @@ -0,0 +1,30 @@ +// Exercise the production sample transition and cadence with a monotonic fake +// clock. Networking and publication are covered by http-probe-host regressions. +#define main resident_probe_main +#include "../../guest/http-probe.c" +#undef main + +int main(int argc, char **argv) { + if (argc != 6) return 125; + uint64_t grace = number(argv[1], 0, 3600000); + uint64_t interval = number(argv[2], 10, 3600000); + uint64_t timeout = number(argv[3], 1, 60000); + uint64_t retries = number(argv[4], 1, 100); + uint64_t ready_after = number(argv[5], 0, 3600000); + uint64_t now = 0, failures = 0; + int health = 2; + bool ever_healthy = false; + puts("0 2 0"); + while (now <= 300000) { + uint64_t begun = now; + bool success = begun >= ready_after; + // A failed sample consumes the full reviewed request timeout. + now += success ? 0 : timeout; + sample_health(success, now, grace, retries, &failures, &health, &ever_healthy); + printf("%llu %d %llu\n", (unsigned long long)now, health, + (unsigned long long)failures); + uint64_t next = begun + sample_period(true, ever_healthy, interval); + if (next > now) now = next; + } + return 0; +} diff --git a/packages/runtime-core/tests/project_contract.rs b/packages/runtime-core/tests/project_contract.rs index 88b11938e..63ebfbd98 100644 --- a/packages/runtime-core/tests/project_contract.rs +++ b/packages/runtime-core/tests/project_contract.rs @@ -66,6 +66,127 @@ impl Drop for Fixture { } const BASIC: &str = "services:\n web:\n image: alpine:3.21\n command: [echo, hello]\n"; +#[test] +fn standard_route_without_healthcheck_infers_a_reviewed_probe_without_editing_source() { + use hack_runtime_core::provider::http_probe::HttpProbe; + let document = format!( + "{BASIC} labels:\n caddy: web.example.hack\n caddy.reverse_proxy: '{{{{upstreams 3000}}}}'\n caddy.tls: internal\n" + ); + let fixture = Fixture::new(&document); + let report = fixture.plan(); + assert!(report.plan.enrollment_compatible); + let health = report.plan.services["web"].healthcheck.as_ref().unwrap(); + assert!(!health.disabled); + assert!(health.test.is_none()); + assert_eq!(health.native_http, Some(HttpProbe::inferred_route(3000))); + assert!( + report + .plan + .diagnostics + .iter() + .any(|d| d.code == "inferred_route_probe" && d.severity == "warning") + ); + assert_eq!( + fs::read_to_string(fixture.project.join("compose.yaml")).unwrap(), + document + ); + assert!(!fixture.candidate.state_root.exists()); +} + +#[test] +fn inferred_route_probe_does_not_satisfy_compose_healthy_dependencies() { + let route = format!( + "{BASIC} labels:\n caddy: web.example.hack\n caddy.reverse_proxy: '{{{{upstreams 3000}}}}'\n caddy.tls: internal\n" + ); + for condition in ["service_healthy", "service_started"] { + let dependency = format!( + " app:\n image: alpine:3.21\n depends_on:\n web: {{condition: {condition}}}\n" + ); + let fixture = Fixture::new(&format!("{route}{dependency}")); + let report = fixture.plan(); + let missing = report + .plan + .diagnostics + .iter() + .any(|d| d.code == "missing_healthcheck"); + assert_eq!(missing, condition == "service_healthy"); + assert_eq!( + report.plan.enrollment_compatible, + condition != "service_healthy" + ); + if condition == "service_healthy" { + let explicit = " healthcheck:\n x-hack-http: {port: 3000, path: /health, interval_ms: 30000, timeout_ms: 1000, retries: 3, start_period_ms: 0}\n"; + let fixture = Fixture::new(&format!("{route}{explicit}{dependency}")); + assert!(fixture.plan().plan.enrollment_compatible); + assert!(!fixture.codes().contains(&"missing_healthcheck".into())); + } + } +} + +#[test] +fn route_inference_never_replaces_authored_healthchecks_or_accepts_an_authored_policy() { + let route = format!( + "{BASIC} labels:\n caddy: web.example.hack\n caddy.reverse_proxy: '{{{{upstreams 3000}}}}'\n caddy.tls: internal\n" + ); + for authored in [ + "{}", + "{disable: true}", + "{test: [CMD, 'true']}", + "{test: [NONE]}", + ] { + let fixture = Fixture::new(&format!("{route} healthcheck: {authored}\n")); + let report = fixture.plan(); + assert!( + report.plan.services["web"] + .healthcheck + .as_ref() + .unwrap() + .native_http + .is_none(), + "{authored}" + ); + assert!( + !report + .plan + .diagnostics + .iter() + .any(|d| d.code == "inferred_route_probe") + ); + } + let probe = "{port: 3000, path: /health, interval_ms: 100, timeout_ms: 500, retries: 3, start_period_ms: 0}"; + let fixture = Fixture::new(&format!( + "{route} healthcheck: {{x-hack-http: {probe}}}\n" + )); + let report = fixture.plan(); + let health = report.plan.services["web"] + .healthcheck + .as_ref() + .unwrap() + .native_http + .as_ref() + .unwrap(); + assert_eq!(health.path, "/health"); + assert!(health.policy.is_none()); + assert!( + serde_json::to_value(health) + .unwrap() + .get("policy") + .is_none() + ); + let forged = probe.replace("path: /health", "path: /, policy: route"); + fs::write( + fixture.project.join("compose.yaml"), + format!("{route} healthcheck: {{x-hack-http: {forged}}}\n"), + ) + .unwrap(); + assert_eq!( + project::plan(&fixture.candidate, fixture.options()) + .unwrap_err() + .code, + "invalid_healthcheck" + ); +} + #[test] fn local_https_labels_have_typed_review_intent_without_publication() { let labels = " labels:\n caddy: 'search.example-app.hack, search.example-app.hack.gy'\n caddy.reverse_proxy: '{{upstreams 6980}}'\n caddy.tls: internal\n"; diff --git a/src/backends/native-http-probe-admission.ts b/src/backends/native-http-probe-admission.ts index 10ff6cb07..86957b1ba 100644 --- a/src/backends/native-http-probe-admission.ts +++ b/src/backends/native-http-probe-admission.ts @@ -11,6 +11,28 @@ const FIELDS = new Set([ "start_period_ms", ]); const PATH = /^\/[\x21-\x22\x24-\x7e]*$/; + +/** + * Exact compiler-generated defaults; this policy is never an authored opt-in. + * These values bind retained route admission; changing them requires migration. + */ +export function isInferredNativeRouteProbe( + probe: unknown, + port: unknown +): boolean { + return ( + isRecord(probe) && + bounded(port, 1, 65_535) && + Object.keys(probe).length === 7 && + probe.policy === "route" && + probe.port === port && + probe.path === "/" && + probe.interval_ms === 30_000 && + probe.timeout_ms === 1000 && + probe.retries === 3 && + probe.start_period_ms === 240_000 + ); +} function bounded(value: unknown, min: number, max: number): boolean { return ( typeof value === "number" && diff --git a/src/backends/native-preflight.ts b/src/backends/native-preflight.ts index 62a813041..04c7b5ed2 100644 --- a/src/backends/native-preflight.ts +++ b/src/backends/native-preflight.ts @@ -30,6 +30,7 @@ import { } from "./native-project-ports.ts"; import { nativeBridgeCapacity, + nativeRouteProbeRefusalDetails, reviewedNativeRoutes, } from "./native-project-routing.ts"; import { @@ -432,7 +433,12 @@ const RUNTIME_FINDINGS: Readonly> = { route_binding_required: { message: "The service declares a route, which needs native HTTPS to be set up.", - fix: "Set up native HTTPS (see the guide) and give the service a healthcheck with x-hack-http.", + fix: "Set up native HTTPS (see the guide). A route without an authored healthcheck gets a default route-port probe.", + }, + inferred_route_probe: { + message: + "Native checks TCP and HTTP on the route port because no healthcheck was authored. HTTP 2xx-4xx counts as ready; 5xx does not.", + fix: "Nothing to do. Declare an explicit x-hack-http healthcheck if this route needs a different path or timing.", }, inert_route_declaration: { message: @@ -1398,7 +1404,7 @@ function routeFindings(opts: { specs, capacity: 1, }); - } catch { + } catch (error) { found.push( finding( { @@ -1408,11 +1414,7 @@ function routeFindings(opts: { service: name, field: "healthcheck.x-hack-http", }, - { - message: - "The service has caddy routing labels but no enabled native HTTP probe on its route port, or it also declares a dependency cache.", - fix: "Give the service a healthcheck with x-hack-http on the port its route proxies to, or add one under httpProbes in the native adaptation.", - } + nativeRouteProbeRefusalDetails(error) ) ); } diff --git a/src/backends/native-project-https.ts b/src/backends/native-project-https.ts index 5acb0c1fc..62cad6722 100644 --- a/src/backends/native-project-https.ts +++ b/src/backends/native-project-https.ts @@ -39,7 +39,7 @@ import { } from "./native-runtime-client.ts"; const PROBE_PATH_BYTES = /^[\x21-\x7e]+$/; -const HTTP_STATUS = /^HTTP\/1\.[01] ([2-5][0-9]{2})(?: [^\r\n]*)?$/; +const HTTP_STATUS = /^HTTP\/1\.[01] ([1-5][0-9]{2})(?: [^\r\n]*)?$/; const SHA = /^[a-f0-9]{64}$/; const SINGLE_CERTIFICATE = /^-----BEGIN CERTIFICATE-----\r?\n[A-Za-z0-9+/=\r\n]+-----END CERTIFICATE-----$/; @@ -350,6 +350,8 @@ https://:${opts.httpsPort} { on_demand } route { + vars native_probe {http.request.header.X-Hack-Probe-Request} + request_header -X-Hack-Probe-Request request_header -X-Hack-Endpoint forward_auth ${quoted(`unix/${opts.authority}`)} { uri /route? @@ -358,6 +360,7 @@ https://:${opts.httpsPort} { reverse_proxy unix/{http.request.header.X-Hack-Endpoint} { header_up Host {hostport} header_up -X-Hack-Endpoint + header_down X-Hack-Upstream-Proof {http.vars.native_probe} } } } @@ -1117,6 +1120,8 @@ export function isNativeHttpsProbePath(path: unknown): path is string { export interface NativeHttpsResponse { readonly statusCode: number; readonly location?: string; + /** A fresh request challenge was added by the upstream proxy response handler. */ + readonly upstreamVerified?: true; } async function confirmTlsPeer(opts: { readonly verifyPeer?: (peerPort: number) => Promise; @@ -1132,7 +1137,27 @@ async function confirmTlsPeer(opts: { } const HEADER_NAME = /^[!#$%&'*+.^_`|~0-9A-Za-z-]+$/; const HEADER_VALUE = /^[\x20-\x7e\t]*$/; -export function parseNativeHttpsHeaders(header: string): NativeHttpsResponse { +function parseHttpsHeaderField(line: string): { name: string; value: string } { + const colon = line.indexOf(":"); + const name = line.slice(0, colon).toLowerCase(); + const raw = line.slice(colon + 1); + if (colon < 1 || !HEADER_NAME.test(name) || !HEADER_VALUE.test(raw)) { + throw refused(); + } + return { name, value: raw.trim() }; +} +function confirmedUpstream( + expectedProof: string | undefined, + observedProof: string | undefined +): Pick { + return expectedProof !== undefined && observedProof === expectedProof + ? { upstreamVerified: true } + : {}; +} +export function parseNativeHttpsHeaders( + header: string, + expectedProof?: string +): NativeHttpsResponse { if (header.length > 8192 || !header.endsWith("\r\n\r\n")) { throw refused(); } @@ -1143,27 +1168,60 @@ export function parseNativeHttpsHeaders(header: string): NativeHttpsResponse { throw refused(); } let location: string | undefined; + let upstreamProof: string | undefined; for (const line of lines) { - const colon = line.indexOf(":"); - const name = line.slice(0, colon).toLowerCase(); - const value = line.slice(colon + 1).trim(); - if ( - colon < 1 || - !HEADER_NAME.test(name) || - !HEADER_VALUE.test(line.slice(colon + 1)) - ) { - throw refused(); - } + const { name, value } = parseHttpsHeaderField(line); if (name === "location") { if (location !== undefined || !value || value.length > 2048) { throw refused(); } location = value; } + if (name === "x-hack-upstream-proof") { + if (upstreamProof !== undefined) { + throw refused(); + } + upstreamProof = value; + } } return { statusCode: Number(match[1]), ...(location === undefined ? {} : { location }), + ...confirmedUpstream(expectedProof, upstreamProof), + }; +} +/** One byte budget spans interim and final headers; bodies are never retained. */ +function httpsResponseReader( + upstreamProof: string +): (data: Buffer) => NativeHttpsResponse | undefined { + let prefix = ""; + let headerBytes = 0; + let interim = 0; + return (data) => { + const part = data.subarray(0, 8192 - headerBytes); + headerBytes += part.length; + prefix += part.toString("latin1"); + while (true) { + const end = prefix.indexOf("\r\n\r\n"); + if (end < 0) { + if (headerBytes >= 8192) { + throw refused(); + } + return undefined; + } + const response = parseNativeHttpsHeaders( + prefix.slice(0, end + 4), + upstreamProof + ); + if (response.statusCode >= 200) { + return response; + } + interim++; + if (response.statusCode === 101 || interim > 4) { + throw refused(); + } + prefix = prefix.slice(end + 4); + } }; } export async function verifyNativeHttpsHostname( @@ -1184,9 +1242,10 @@ export async function verifyNativeHttpsHostname( throw refused(); } const ca = await validCa(caPath); + const upstreamProof = randomBytes(16).toString("hex"); return await new Promise((resolve, reject) => { let settled = false; - let prefix = ""; + const readResponse = httpsResponseReader(upstreamProof); let phase: "handshake" | "response" = "handshake"; let peerVerified = verifyPeer === undefined; const finish = (error?: Error, response?: NativeHttpsResponse) => { @@ -1225,7 +1284,7 @@ export async function verifyNativeHttpsHostname( phase = "response"; const authority = port === 443 ? hostname : `${hostname}:${port}`; socket.write( - `GET ${path} HTTP/1.1\r\nHost: ${authority}\r\nConnection: close\r\n\r\n` + `GET ${path} HTTP/1.1\r\nHost: ${authority}\r\nX-Hack-Probe-Request: ${upstreamProof}\r\nConnection: close\r\n\r\n` ); } catch { finish(nativeHttpsVerificationError(undefined)); @@ -1238,13 +1297,11 @@ export async function verifyNativeHttpsHostname( return; } // Parse only bounded headers; discard any body bytes delivered in the same chunk. - prefix += data.subarray(0, 8192 - prefix.length).toString("latin1"); - const end = prefix.indexOf("\r\n\r\n"); - if (end < 0 && prefix.length < 8192) { - return; - } try { - finish(undefined, parseNativeHttpsHeaders(prefix.slice(0, end + 4))); + const response = readResponse(data); + if (response !== undefined) { + finish(undefined, response); + } } catch { finish(nativeHttpsVerificationError(undefined)); } diff --git a/src/backends/native-project-routing.ts b/src/backends/native-project-routing.ts index f31e38022..8076c594f 100644 --- a/src/backends/native-project-routing.ts +++ b/src/backends/native-project-routing.ts @@ -2,6 +2,7 @@ import { HackCliError } from "../lib/cli-result.ts"; import { isRecord } from "../lib/guards.ts"; import { HACK_OWNERSHIP_LABEL_NAMESPACES } from "../lib/hack-ownership-labels.ts"; import { isIgnoredHackComposeLabel } from "../lib/native-hack-compose-conventions.ts"; +import { isInferredNativeRouteProbe } from "./native-http-probe-admission.ts"; import { hasOnlyNativeCacheLabels } from "./native-project-cache.ts"; const ROUTE_LABELS = new Set(["caddy", "caddy.reverse_proxy", "caddy.tls"]); @@ -10,9 +11,53 @@ function refused(): Error { return new HackCliError({ code: "E_NATIVE_PROJECT_UNSUPPORTED", message: - "Native routes require reviewed Caddy labels and existing matching native HTTP health probes; no probe or route was invented.", + "Native routes require reviewed Caddy labels and matching native route readiness; explicit healthchecks are never replaced.", }); } + +const ROUTE_PROBE_REFUSALS = { + disabled: { + message: "The routed service explicitly disables its healthcheck.", + fix: "Replace the disabled healthcheck with an x-hack-http-only healthcheck on the route port, or remove the healthcheck declaration to use the inferred probe.", + }, + command: { + message: + "The routed service has a command-only healthcheck; native route readiness requires an HTTP probe.", + fix: "Replace the command healthcheck with an x-hack-http-only healthcheck on the route port, or remove the healthcheck declaration to use the inferred probe.", + }, + cache: { + message: + "The routed service also declares a dependency cache, which native routing does not support.", + fix: "Move the dependency cache to a separate non-routed service, or remove the dependency cache declaration.", + }, + mismatch: { + message: + "The route has no valid reviewed HTTP probe matching its route port and authored healthcheck.", + fix: "Check the route port and healthcheck.x-hack-http; remove the authored healthcheck to infer readiness, or configure a valid explicit HTTP probe on that port. If the review still differs, update Hack and rerun native preflight.", + }, +} as const; + +/** Closed reasons shared by startup admission and preflight; no authored values. */ +class RouteProbeRefusal extends HackCliError { + readonly reason: keyof typeof ROUTE_PROBE_REFUSALS; + constructor(reason: keyof typeof ROUTE_PROBE_REFUSALS) { + const detail = ROUTE_PROBE_REFUSALS[reason]; + super({ + code: "E_NATIVE_PROJECT_UNSUPPORTED", + message: `${detail.message} ${detail.fix}`, + }); + this.reason = reason; + } +} + +export function nativeRouteProbeRefusalDetails(error: unknown): { + readonly message: string; + readonly fix: string; +} { + return ROUTE_PROBE_REFUSALS[ + error instanceof RouteProbeRefusal ? error.reason : "mismatch" + ]; +} /** * Hack's authored one-shot marker. Only this exact key with the literal value `true` * (map form) or the exact entry `hack.service.one-shot=true` (list form) is admitted, @@ -141,6 +186,28 @@ function requireRouteProbe( service: Readonly>, spec: Readonly> | undefined ): void { + if ( + service.dependency_cache !== undefined && + service.dependency_cache !== null + ) { + throw new RouteProbeRefusal("cache"); + } + const authored = spec?.healthcheck; + if ( + (isRecord(service.healthcheck) && service.healthcheck.disabled === true) || + (isRecord(authored) && + (authored.disable === true || + (Array.isArray(authored.test) && authored.test[0] === "NONE"))) + ) { + throw new RouteProbeRefusal("disabled"); + } + if ( + isRecord(authored) && + !Object.hasOwn(authored, "x-hack-http") && + Object.hasOwn(authored, "test") + ) { + throw new RouteProbeRefusal("command"); + } const probe = spec && isRecord(spec.healthcheck) ? spec.healthcheck["x-hack-http"] @@ -148,22 +215,28 @@ function requireRouteProbe( const reviewedProbe = isRecord(service.healthcheck) ? service.healthcheck.native_http : undefined; + const inferred = + spec !== undefined && + !Object.hasOwn(spec, "healthcheck") && + isRecord(service.routing) && + isInferredNativeRouteProbe(reviewedProbe, service.routing.port); if ( !( isRecord(service.routing) && - isRecord(probe) && isRecord(reviewedProbe) && Number.isInteger(service.routing.port) ) || Number(service.routing.port) < 1 || Number(service.routing.port) > 65_535 || - probe.port !== service.routing.port || - reviewedProbe.port !== service.routing.port || - (isRecord(service.healthcheck) && service.healthcheck.disabled === true) || - (service.dependency_cache !== undefined && - service.dependency_cache !== null) + !( + inferred || + (isRecord(probe) && + probe.port === service.routing.port && + reviewedProbe.policy === undefined) + ) || + reviewedProbe.port !== service.routing.port ) { - throw refused(); + throw new RouteProbeRefusal("mismatch"); } } export function reviewedNativeRoutes(opts: { diff --git a/src/backends/native-project-start.ts b/src/backends/native-project-start.ts index fb8f6998d..81cc7a3fe 100644 --- a/src/backends/native-project-start.ts +++ b/src/backends/native-project-start.ts @@ -24,7 +24,10 @@ import { adaptNativeGcloudEnvironment, type NativeGcloudSelection, } from "./native-gcloud-environment.ts"; -import { admitNativeHttpProbe } from "./native-http-probe-admission.ts"; +import { + admitNativeHttpProbe, + isInferredNativeRouteProbe, +} from "./native-http-probe-admission.ts"; import { acquireNativeHttpsLease, NativeHttpsLeaseAcquisitionError, @@ -288,14 +291,44 @@ async function acquireReadyHttps(opts: { * routed service without an enabled reviewed HTTP probe, or one hostname with * conflicting probes, is the configuration refusal. */ +function reviewedHttpsProbe( + healthcheck: unknown, + port: unknown +): { path: string; inferred?: true } { + if ( + !( + isRecord(healthcheck) && + healthcheck.disabled !== true && + isRecord(healthcheck.native_http) && + isNativeHttpsProbePath(healthcheck.native_http.path) + ) + ) { + throw refused(); + } + const probe = healthcheck.native_http; + const path = healthcheck.native_http.path; + const inferred = isInferredNativeRouteProbe(probe, port); + if (probe.policy !== undefined && !inferred) { + throw refused(); + } + return { path, ...(inferred ? { inferred: true as const } : {}) }; +} export function reviewedNativeHttpsRoutes( plan: unknown, services: ReadonlySet -): readonly { hostname: string; path: string; service: string }[] { +): readonly { + hostname: string; + path: string; + service: string; + inferred?: true; +}[] { if (!(isRecord(plan) && isRecord(plan.services))) { throw unconfirmed("the reviewed runtime plan"); } - const names = new Map(); + const names = new Map< + string, + { path: string; service: string; inferred?: true } + >(); for (const name of services) { const service = plan.services[name]; if ( @@ -307,21 +340,12 @@ export function reviewedNativeHttpsRoutes( ) { throw unconfirmed("the reviewed runtime plan"); } - if ( - !( - isRecord(service.healthcheck) && - service.healthcheck.disabled !== true && - isRecord(service.healthcheck.native_http) && - isNativeHttpsProbePath(service.healthcheck.native_http.path) - ) - ) { - throw refused(); - } + const probe = reviewedHttpsProbe(service.healthcheck, service.routing.port); for (const hostname of service.routing.hostnames) { if (typeof hostname !== "string") { throw unconfirmed("the reviewed runtime plan"); } - const path = service.healthcheck.native_http.path; + const path = probe.path; if ( names.has(hostname) && (names.get(hostname)?.path !== path || @@ -329,7 +353,11 @@ export function reviewedNativeHttpsRoutes( ) { throw refused(); } - names.set(hostname, { path, service: name }); + names.set(hostname, { + path, + service: name, + ...(probe.inferred ? { inferred: true as const } : {}), + }); } } return [...names] @@ -843,6 +871,18 @@ function requireEnrollmentCompatible(plan: unknown): void { } const HTTPS_REDIRECTS = new Set([301, 302, 303, 307, 308]); type HttpsRoute = ReturnType[number]; +function httpsResponseReady( + route: HttpsRoute, + response: { statusCode: number; upstreamVerified?: true } +): boolean { + if (route.inferred && response.upstreamVerified !== true) { + throw httpsVerificationFailed("UPSTREAM_RESPONSE_UNCONFIRMED"); + } + return ( + response.statusCode >= 200 && + response.statusCode < (route.inferred ? 500 : 300) + ); +} function redirectHostname( location: string, hostname: string, @@ -909,7 +949,7 @@ export async function verifyHttpsRoutes( if (!(response && Number.isInteger(response.statusCode))) { throw httpsVerificationFailed("NO_RESPONSE"); } - if (response.statusCode >= 200 && response.statusCode < 300) { + if (httpsResponseReady(route, response)) { break; } if (!(HTTPS_REDIRECTS.has(response.statusCode) && response.location)) { diff --git a/src/backends/native-runtime-client.ts b/src/backends/native-runtime-client.ts index 959b0ab2e..fd46de9aa 100644 --- a/src/backends/native-runtime-client.ts +++ b/src/backends/native-runtime-client.ts @@ -33,7 +33,7 @@ interface NativeFailure { readonly code: string; readonly logPhase?: NativeLogFailurePhase; readonly causeCode?: string; - /** Runtime message the runtime marked as a compile-time literal (value-free). */ + /** Fixed runtime text, or fixed guidance rendered from a validated diagnostic. */ readonly message?: string; /** Admitted service the runtime named, as a plain service identifier. */ readonly service?: string; @@ -76,16 +76,43 @@ function fixedNativeMessage( : undefined; } -/** The fixed message and validated service a runtime failure carries, if any. */ +/** Render only the closed inferred-route diagnostic; never runtime error text. */ +function routeProbeFailureMessage( + value: Record, + service: string | undefined +): string | undefined { + const probe = value.route_probe; + if ( + !( + service && + ["graph_service_failed", "graph_startup_application_failed"].includes( + typeof value.code === "string" ? value.code : "" + ) && + isRecord(probe) + ) || + Object.keys(probe).length !== 2 || + probe.version !== 1 || + typeof probe.port !== "number" || + !Number.isSafeInteger(probe.port) || + probe.port < 1 || + probe.port > 65_535 + ) { + return; + } + return `Inferred route probe on port ${probe.port} is unhealthy. Check hack logs ${service} in the same project/branch, fix the route port, or declare healthcheck.x-hack-http with a longer start_period_ms and a readiness path, within the 300-second graph budget.`; +} + +/** Fixed/closed diagnostic text and the validated service a failure carries. */ export function nativeFailureDetail(value: Record): { message?: string; service?: string; } { - const message = fixedNativeMessage(value); const service = typeof value.service === "string" && NATIVE_SERVICE.test(value.service) ? value.service : undefined; + const message = + routeProbeFailureMessage(value, service) ?? fixedNativeMessage(value); return { ...(message ? { message } : {}), ...(service ? { service } : {}), @@ -124,7 +151,7 @@ export function retainNativeRuntimeFailure(opts: { /** * Value-free runtime detail appended after a closed native failure sentence: - * the fixed runtime message and the service it concerns, when known. + * fixed runtime text or closed probe guidance, and its validated service. */ export function nativeFailureNote(detail: { readonly message?: string; @@ -148,7 +175,7 @@ export class NativeRuntimeRequestError extends ClosedCliError { readonly stopSignal: "SIGINT" | "SIGTERM" | undefined; readonly nativeCode: string | undefined; readonly nativeCauseCode: string | undefined; - /** Fixed (value-free) runtime message; see {@link nativeFailureNote}. */ + /** Fixed/closed diagnostic text; see {@link nativeFailureNote}. */ readonly nativeMessage: string | undefined; readonly nativeService: string | undefined; readonly nativeStopFailures: readonly NativeStopFailure[] | undefined; @@ -158,7 +185,7 @@ export class NativeRuntimeRequestError extends ClosedCliError { readonly message: string; readonly nativeCode?: string; readonly nativeCauseCode?: string; - /** Only fixed text admitted by nativeFailureDetail; this can cross IPC. */ + /** Only safe text admitted by nativeFailureDetail; this can cross IPC. */ readonly nativeMessage?: string; readonly nativeService?: string; readonly nativeStopFailures?: readonly NativeStopFailure[]; diff --git a/tests/native-http-probe-admission.test.ts b/tests/native-http-probe-admission.test.ts index 388c1c6ce..8a811c3b4 100644 --- a/tests/native-http-probe-admission.test.ts +++ b/tests/native-http-probe-admission.test.ts @@ -55,3 +55,11 @@ test("HTTP preflight accepts runtime boundary values and command probes", () => admitNativeHttpProbe("web", { healthcheck: { test: ["CMD", "true"] } }) ).not.toThrow(); }); + +test("an authored probe cannot opt into the generated route policy", () => { + expect(() => + admitNativeHttpProbe("web", { + healthcheck: { "x-hack-http": { ...valid, policy: "route" } }, + }) + ).toThrow("healthcheck.x-hack-http"); +}); diff --git a/tests/native-preflight-cli.test.ts b/tests/native-preflight-cli.test.ts index 2b971917e..44dd53fff 100644 --- a/tests/native-preflight-cli.test.ts +++ b/tests/native-preflight-cli.test.ts @@ -713,10 +713,78 @@ test( SPAWN_TIMEOUT_MS * 2 ); +test( + "route preflight gives a separate cause and remedy for each refused probe", + async () => { + const probe = { + port: 3000, + path: "/health", + interval_ms: 1000, + timeout_ms: 1000, + retries: 3, + start_period_ms: 0, + }; + for (const entry of [ + { + authored: { disable: true }, + reviewed: { healthcheck: { disabled: true } }, + message: "explicitly disables", + fix: "Replace the disabled healthcheck", + }, + { + authored: { test: ["CMD", "true"] }, + reviewed: { healthcheck: { test: ["CMD", "true"] } }, + message: "command-only healthcheck", + fix: "Replace the command healthcheck", + }, + { + authored: { "x-hack-http": probe }, + reviewed: { healthcheck: { native_http: probe }, dependency_cache: {} }, + message: "also declares a dependency cache", + fix: "Move the dependency cache", + }, + { + authored: { "x-hack-http": probe }, + reviewed: { healthcheck: { native_http: { ...probe, port: 3001 } } }, + message: "no valid reviewed HTTP probe matching", + fix: "Check the route port", + }, + ]) { + const project = await fixture({ + services: { + web: { + image: IMAGE, + labels: { + caddy: "web.example.test", + "caddy.reverse_proxy": "{{upstreams 3000}}", + "caddy.tls": "internal", + }, + healthcheck: entry.authored, + }, + }, + }); + await project.review([], { + services: { + web: { active: true, routing: { port: 3000 }, ...entry.reviewed }, + }, + }); + const result = await preflight(project, { ...project.native, ...GRANT }); + expect(result.code).toBe(1); + const found = result.report.blockers.find( + (f) => f.code === "route_probe_required" + ); + expect(found?.service).toBe("web"); + expect(found?.message).toContain(entry.message); + expect(found?.fix).toContain(entry.fix); + } + }, + SPAWN_TIMEOUT_MS * 2 +); + test( "route admission, project-wide hosts and per-environment dependency files are checked as native up checks them", async () => { - // A routed service without a native HTTP probe: the review only warns, native up refuses. + // A route missing its reviewed probe still refuses; authored labels alone are insufficient. const routed = await fixture({ services: { web: { @@ -743,6 +811,30 @@ test( expect(route.code).toBe(1); expect(route.report.ready).toBe(false); expect(codes(route.report, "cli")).toEqual(["route_probe_required:web"]); + await routed.review([], { + services: { + web: { + active: true, + routing: { port: 3000 }, + healthcheck: { + disabled: false, + native_http: { + policy: "route", + port: 3000, + path: "/", + interval_ms: 30_000, + timeout_ms: 1000, + retries: 3, + start_period_ms: 240_000, + }, + }, + }, + }, + }); + const inferred = await preflight(routed, { ...routed.native, ...GRANT }); + expect(codes(inferred.report, "cli")).toEqual([]); + expect(inferred.code).toBe(0); + expect(inferred.report.ready).toBe(true); // Routes plus published ports over the pool's bridge limit: native up // refuses before any runtime work, so no review is needed to see it. diff --git a/tests/native-project-https.test.ts b/tests/native-project-https.test.ts index 58d05b5f1..46fdf0db5 100644 --- a/tests/native-project-https.test.ts +++ b/tests/native-project-https.test.ts @@ -245,6 +245,13 @@ test("owns authority and Caddy lifetimes, writes strict routing config and prese expect(f.config()).toContain("uri /route?"); expect(f.config()).toContain("request_header -X-Hack-Endpoint"); expect(f.config()).toContain("header_up -X-Hack-Endpoint"); + expect(f.config()).toContain( + "vars native_probe {http.request.header.X-Hack-Probe-Request}" + ); + expect(f.config()).toContain("request_header -X-Hack-Probe-Request"); + expect(f.config()).toContain( + "header_down X-Hack-Upstream-Proof {http.vars.native_probe}" + ); expect(f.config()).toContain("https://:443"); expect(f.config()).toContain("bind 127.0.0.1"); expect(f.config()).toContain("permission http http://127.0.0.1:18443/ask"); @@ -719,7 +726,11 @@ while True: if data: seen.update(request=data.decode(),remote=remote[0],remote_port=remote[1]) (root/'seen').write_text(json.dumps(seen)) - conn.sendall((root/'response').read_bytes()) + response=(root/'response').read_bytes() + if (root/'echo-proof').exists(): + proof=next((line.split(b': ',1)[1] for line in data.split(b'\r\n') if line.startswith(b'X-Hack-Probe-Request: ')),b'') + response=response.replace(b'\r\n\r\n',b'\r\nX-Hack-Upstream-Proof: '+proof+b'\r\n\r\n') + conn.sendall(response) except (OSError,ssl.SSLError): client.close() ` ); @@ -747,7 +758,11 @@ while True: ).toEqual({ statusCode: 204 }); expect(JSON.parse(await readFile(seenPath, "utf8"))).toMatchObject({ sni: "fixture.invalid", - request: `GET /health HTTP/1.1\r\nHost: fixture.invalid:${address.port}\r\nConnection: close\r\n\r\n`, + request: expect.stringMatching( + new RegExp( + `^GET /health HTTP/1\\.1\\r\\nHost: fixture\\.invalid:${address.port}\\r\\nX-Hack-Probe-Request: [a-f0-9]{32}\\r\\nConnection: close\\r\\n\\r\\n$` + ) + ), remote: "127.0.0.1", }); await rm(seenPath); @@ -809,6 +824,19 @@ while True: "/health" ) ).toEqual({ statusCode: 307, location: "https://canonical.invalid/health" }); + await writeFile(join(root, "echo-proof"), ""); + await writeFile(responsePath, "HTTP/1.1 404 Not Found\r\n\r\n"); + expect( + await verifyNativeHttpsHostname("fixture.invalid", address.port, cert, "/") + ).toEqual({ statusCode: 404, upstreamVerified: true }); + await writeFile( + responsePath, + "HTTP/1.1 103 Early Hints\r\n\r\nHTTP/1.1 404 Not Found\r\n\r\n" + ); + expect( + await verifyNativeHttpsHostname("fixture.invalid", address.port, cert, "/") + ).toEqual({ statusCode: 404, upstreamVerified: true }); + await rm(join(root, "echo-proof")); for (const invalid of [ "HTTP/1.1 100 Continue\r\n\r\n", "malformed\r\n", @@ -861,6 +889,32 @@ test("HTTPS header parsing refuses duplicate, folded and oversized fields", () = } }); +test("upstream proof requires this request's exact challenge, never merely a 4xx status", () => { + const proof = "a".repeat(32); + for (const response of [ + "HTTP/1.1 403 Denied\r\n\r\n", + "HTTP/1.1 404 Missing\r\nX-Hack-Upstream-Proof: stale\r\n\r\n", + "HTTP/1.1 421 Misdirected\r\n\r\n", + "HTTP/1.1 502 Missing Socket\r\n\r\n", + ]) { + expect( + parseNativeHttpsHeaders(response, proof).upstreamVerified + ).toBeUndefined(); + } + expect( + parseNativeHttpsHeaders( + `HTTP/1.1 404 Not Found\r\nX-Hack-Upstream-Proof: ${proof}\r\n\r\n`, + proof + ) + ).toEqual({ statusCode: 404, upstreamVerified: true }); + expect(() => + parseNativeHttpsHeaders( + `HTTP/1.1 404 Not Found\r\nX-Hack-Upstream-Proof: ${proof}\r\nx-hack-upstream-proof: ${proof}\r\n\r\n`, + proof + ) + ).toThrow(); +}); + test("HTTPS headers permit unrelated duplicates but keep Location unambiguous", () => { expect( parseNativeHttpsHeaders( diff --git a/tests/native-project-routing.test.ts b/tests/native-project-routing.test.ts index 6e26452e9..d973f3654 100644 --- a/tests/native-project-routing.test.ts +++ b/tests/native-project-routing.test.ts @@ -142,7 +142,7 @@ test("active reviewed routes request pool allocation with matching existing prob } expect(refusal).toMatchObject({ code: "E_NATIVE_PROJECT_UNSUPPORTED", - message: expect.stringContaining("reviewed Caddy labels"), + message: expect.stringContaining("no valid reviewed HTTP probe matching"), }); expect(() => reviewedNativeRoutes({ @@ -162,3 +162,65 @@ test("active reviewed routes request pool allocation with matching existing prob }).flags ).toEqual([]); }); + +test("a route without an authored healthcheck accepts only its exact reviewed inferred probe", () => { + const inferred = { + policy: "route", + port: 3000, + path: "/", + interval_ms: 30_000, + timeout_ms: 1000, + retries: 3, + start_period_ms: 240_000, + }; + const opts = { + plan: { + services: { + web: { + ...service, + healthcheck: { disabled: false, native_http: inferred }, + }, + }, + }, + specs: { web: { labels } }, + capacity: 1, + }; + expect(reviewedNativeRoutes(opts).flags).toEqual([ + "--route-slot", + "web=auto", + ]); + for (const authoredHealthcheck of [ + null, + {}, + { disable: true }, + { test: ["CMD", "true"] }, + healthcheck, + ]) { + expect(() => + reviewedNativeRoutes({ + ...opts, + specs: { web: { labels, healthcheck: authoredHealthcheck } }, + }) + ).toThrow(); + } + for (const changed of [ + { ...inferred, port: 3001 }, + { ...inferred, path: "/health" }, + { ...inferred, policy: "anything" }, + { ...inferred, timeout_ms: 60_000 }, + { ...inferred, start_period_ms: 60_000 }, + { ...inferred, extra: true }, + { port: 3000 }, + ]) { + expect(() => + reviewedNativeRoutes({ + ...opts, + plan: { + services: { + web: { ...service, healthcheck: { native_http: changed } }, + }, + }, + }) + ).toThrow(); + } +}); diff --git a/tests/native-project-start.test.ts b/tests/native-project-start.test.ts index c2e0d539e..18a080ac1 100644 --- a/tests/native-project-start.test.ts +++ b/tests/native-project-start.test.ts @@ -3157,6 +3157,82 @@ test("HTTPS startup refuses a non-2xx reviewed health response", async () => { expect(events).toContain("https-close"); }); +test("inferred HTTPS readiness accepts 2xx-4xx without following redirects and rejects 5xx", async () => { + const probe = { + policy: "route", + port: 3000, + path: "/", + interval_ms: 30_000, + timeout_ms: 1000, + retries: 3, + start_period_ms: 240_000, + }; + const plan = { + services: { + web: { + routing: { port: 3000, hostnames: ["web.example.hack"] }, + healthcheck: { disabled: false, native_http: probe }, + }, + }, + }; + for (const statusCode of [200, 204, 299, 301, 399, 400, 404, 499]) { + const calls: string[] = []; + await verifyHttpsRoutes( + { + httpsPort: 18_443, + verifyHostname: async (hostname, path) => { + calls.push(hostname); + expect(path).toBe("/"); + return { + statusCode, + location: "https://foreign.invalid/", + upstreamVerified: true, + }; + }, + }, + plan, + new Set(["web"]) + ); + expect(calls).toEqual(["web.example.hack"]); + } + for (const statusCode of [199, 500, 503, 599, 600]) { + await expect( + verifyHttpsRoutes( + { + httpsPort: 18_443, + verifyHostname: async () => ({ statusCode, upstreamVerified: true }), + }, + plan, + new Set(["web"]) + ) + ).rejects.toThrow(); + } + for (const statusCode of [200, 403, 404, 421, 502]) { + await expect( + verifyHttpsRoutes( + { httpsPort: 18_443, verifyHostname: async () => ({ statusCode }) }, + plan, + new Set(["web"]) + ) + ).rejects.toThrow("UPSTREAM_RESPONSE_UNCONFIRMED"); + } + const explicit = { + services: { + web: { + ...plan.services.web, + healthcheck: { native_http: { ...probe, policy: undefined } }, + }, + }, + }; + await expect( + verifyHttpsRoutes( + { httpsPort: 18_443, verifyHostname: async () => ({ statusCode: 404 }) }, + explicit, + new Set(["web"]) + ) + ).rejects.toThrow(); +}); + test("HTTPS redirects terminate only at verified same-service reviewed aliases", async () => { const plan = { services: { diff --git a/tests/native-runtime-client.test.ts b/tests/native-runtime-client.test.ts index c68a2e21b..fddb8653b 100644 --- a/tests/native-runtime-client.test.ts +++ b/tests/native-runtime-client.test.ts @@ -8,6 +8,7 @@ import { NATIVE_EXIT_DRAIN_BOUND_MS, NativeRuntimeRequestError, nativeCodesReportRuntimeNotRunning, + nativeFailureDetail, resolveNativeRuntimeSelection, } from "../src/backends/native-runtime-client.ts"; import { ClosedCliError } from "../src/lib/cli-result.ts"; @@ -149,6 +150,7 @@ async function fixture() { if (process.argv.includes("hang")) await Bun.sleep(60_000); if (process.argv.includes("old-provider")) { console.error(JSON.stringify({code:"provider_pin_changed",message:"synthetic-secret-diagnostic"})); process.exit(2); } if (process.argv.includes("fixed-message")) { console.error(JSON.stringify({code:"graph_image",message:"Pinned image architecture or implicit volumes are incompatible.",message_fixed:true,service:"web"})); process.exit(2); } +if (process.argv.includes("route-probe-failure")) { console.error(JSON.stringify({code:"graph_service_failed",message:"synthetic-private-response",service:"web",route_probe:{version:1,port:3000}})); process.exit(2); } if (process.argv.includes("unsafe-fixed")) { console.error(JSON.stringify({code:"graph_image",message:"synthetic-secret-diagnostic\\nsecond line",message_fixed:true,service:"../synthetic-secret-diagnostic"})); process.exit(2); } if (process.argv.includes("plan-change")) { console.error(JSON.stringify({code:"graph_plan_reconcile",message:"synthetic-secret-diagnostic",plan_change:{version:1,service:"deps",field:"dependency_cache_volume"}})); process.exit(23); } if (process.argv.includes("plan-change-unsafe")) { console.error(JSON.stringify({code:"graph_hostname_change",message:"synthetic-secret-diagnostic",plan_change:{version:1,service:"synthetic secret diagnostic",field:"hostnames"}})); process.exit(23); } @@ -1009,6 +1011,53 @@ test("structured failures expose only a bounded code, never their message", asyn ); }); +test("inferred route failures render only validated service and port guidance", async () => { + const reply = { + code: "graph_service_failed", + service: "web", + route_probe: { version: 1, port: 3000 }, + message: "synthetic-private-response", + }; + const expected = { + service: "web", + message: + "Inferred route probe on port 3000 is unhealthy. Check hack logs web in the same project/branch, fix the route port, or declare healthcheck.x-hack-http with a longer start_period_ms and a readiness path, within the 300-second graph budget.", + }; + expect(nativeFailureDetail(reply)).toEqual(expected); + const longest = nativeFailureDetail({ ...reply, service: "w".repeat(128) }); + expect(longest.message?.length).toBeLessThanOrEqual(400); + const runtime = await fixture(); + const failure: unknown = await invokeNativeRuntime({ + runtime, + cwd: runtime.home, + args: ["route-probe-failure"], + }).catch((caught: unknown) => caught); + expect(failure).toBeInstanceOf(NativeRuntimeRequestError); + expect((failure as Error).message).toContain( + `Service web: ${expected.message}` + ); + expect((failure as Error).message).not.toContain( + "synthetic-private-response" + ); + expect( + nativeFailureDetail({ ...reply, code: "graph_startup_application_failed" }) + ).toEqual(expected); + for (const patch of [ + { code: "graph_timeout" }, + { route_probe: { version: 2, port: 3000 } }, + { route_probe: { version: 1, port: 0 } }, + { route_probe: { version: 1, port: 65_536 } }, + { route_probe: { version: 1, port: 1.5 } }, + { route_probe: { version: 1, port: "synthetic-private-value" } }, + { route_probe: { version: 1, port: 3000, extra: "private" } }, + { service: "web\nprivate" }, + ]) { + const detail = nativeFailureDetail({ ...reply, ...patch }); + expect(detail.message).toBeUndefined(); + expect(JSON.stringify(detail)).not.toContain("private"); + } +}); + bunTest.each([ { argument: "foreground-stopped",