diff --git a/docs/cli.md b/docs/cli.md index 149ff4c25..60667e2f7 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -22,6 +22,7 @@ terminal). - `hack exec -- ` — command in an already-running service container - `hack session` — persistent project workspaces (tmux-first) - `hack doctor` / `hack doctor --fix` — validate and repair local setup +- `hack upgrade check` — read-only upgrade check of this machine and every registered project, ending in an ordered plan and the next command; `hack upgrade snapshot` and `hack upgrade rollback --rollback-manifest ` keep and restore the files the upgrade changes - `hack daemon` — optional local daemon for faster JSON project status; `hack daemon install` starts it at login (launchd on macOS, a systemd user unit on Linux). On Linux without linger it still stops at your last logout and does not start at boot; `loginctl enable-linger` changes that Candidate `hack-next doctor` and `hack-v5 doctor` report a missing hackd as optional diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 46880255c..c29686d99 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1845,6 +1845,108 @@ Notes: | `--version, -v` | Show version | +## `hack upgrade` + +Guided upgrade: check, snapshot and roll back + +### Usage + +```bash +hack upgrade [options] +``` + +### Subcommands + +| Command | Summary | +| --- | --- | +| `hack upgrade check` | Check this machine and every registered project for the upgrade (read-only) | +| `hack upgrade snapshot` | Keep a private copy of the files the upgrade changes | +| `hack upgrade rollback` | Put back the files kept by an upgrade snapshot | + +### Options + +| Option | Description | +| --- | --- | +| `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | +| `--help, -h` | Show help | +| `--version, -v` | Show version | + +## `hack upgrade check` + +Check this machine and every registered project for the upgrade (read-only) + +### Usage + +```bash +hack upgrade check [options] +``` + +Reads the installed versions, the daemon's service, the projects registry (dead entries apart +from unreadable ones), Hack's global services, proxy ports and DNS, the global secret store, and +for each live project its env format and key, tracked files, ignore rules, agent integrations +and (Apple Silicon) native preflight. Ends with an ordered plan and the next command. Changes +nothing and prints no value. Exits 1 when something blocks the upgrade, naming it. + +### Options + +| Option | Description | +| --- | --- | +| `--json` | Output JSON (machine-readable) | +| `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | +| `--help, -h` | Show help | +| `--version, -v` | Show version | + +## `hack upgrade snapshot` + +Keep a private copy of the files the upgrade changes + +### Usage + +```bash +hack upgrade snapshot [options] +``` + +Copies the global Hack config, the projects registry, the encrypted secret store and its key +(when used), and each live project's .hack config, env files, env layers, .hack/.gitignore and +env key files into a private directory under ~/.hack/upgrade/snapshots (0700, files 0600). +Prints the rollback command. The manifest records paths, sizes, modes and digests, never values. + +### Options + +| Option | Description | +| --- | --- | +| `--json` | Output JSON (machine-readable) | +| `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | +| `--help, -h` | Show help | +| `--version, -v` | Show version | + +## `hack upgrade rollback` + +Put back the files kept by an upgrade snapshot + +### Usage + +```bash +hack upgrade rollback [options] +``` + +Restores each file in the manifest with its exact bytes and mode. A file that changed since the +snapshot (or since the step that recorded it) is left as it is unless --overwrite is passed; +whatever a restore replaces or removes is kept in the snapshot directory, never deleted. +Without --rollback-manifest, lists the snapshots and changes nothing. + +### Options + +| Option | Description | +| --- | --- | +| `--rollback-manifest ` | The snapshot manifest to restore (from hack upgrade snapshot) | +| `--overwrite` | Also restore files that changed since the snapshot (the current copies are kept in the snapshot) | +| `--json` | Output JSON (machine-readable) | +| `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | +| `--help, -h` | Show help | +| `--version, -v` | Show version | + + ## `hack version` Print version diff --git a/docs/versions/v5/guides/upgrade-from-v4.md b/docs/versions/v5/guides/upgrade-from-v4.md index 26697a633..837e8981f 100644 --- a/docs/versions/v5/guides/upgrade-from-v4.md +++ b/docs/versions/v5/guides/upgrade-from-v4.md @@ -64,6 +64,78 @@ Sources: [candidate channel contract](https://github.com/hack-dance/hack/blob/26 ## 2. Before you start +### Check first + +v5 builds that include the upgrade assistant can check the machine before you +change anything. Run the v5 CLI (on macOS, `"$HOME/.hack-next/bin/hack-next"` +with `HACK_RUNTIME_BACKEND=compose`): + +```sh +hack upgrade check # add --json for a machine-readable report +``` + +It reads the installed versions, the daemon's login service, the projects +registry, Hack's global services, proxy ports and DNS, and the global secret +store. For each live project it reads the env format and key, tracked +machine-local files, ignore rules, agent integration files and, on Apple +Silicon, the native preflight. It changes nothing and prints no secret value. + +- **Dead registry entries** (the checkout or its config is gone) are counted + apart from live projects that cannot be read. Dead entries only need + `hack projects prune`; unreadable ones need repair. +- It ends with an ordered plan and the next command. A step either has a + command or is marked manual, with the action to take; manual text is never + printed as a command. Commands run the CLI you ran the check with (the + `hack-next` launcher during the prerelease). +- It exits 1 when something blocks the upgrade, naming the blockers. Examples + are a global secret store whose key is gone or refused while a project + reads it, a tracked secret key, an env layer conflict, or a live project + whose config cannot be read. It also exits 1 when a check the upgrade + depends on could not run, listed under "Not fully checked". Examples are a + check that timed out, or a key file that is missing while the OS keychain + may hold the key; the check never asks the keychain, because that can + prompt. + +Before you follow the plan, keep a private copy of the files it changes: + +```sh +hack upgrade snapshot +``` + +This copies the global Hack config, the projects registry, the encrypted +secret store and its key (when used), and each live project's `.hack/` +config, env files, env layers, `.hack/.gitignore` and env key files into +`~/.hack/upgrade/snapshots//` (directories 0700, files 0600). The +manifest records paths, sizes, modes and SHA-256 digests, never values. It +prints the command that puts the files back: + +```sh +hack upgrade rollback --rollback-manifest +``` + +Rollback restores each file's exact bytes and mode. A file that changed since +the snapshot is left as it is unless you add `--overwrite`. Whatever rollback +replaces or removes is kept as itself (not as a copy) in the snapshot directory +and named in the output; nothing is deleted. A file that appears at a path while +rollback runs is left alone and reported as a conflict. Rollback keeps every +temporary and replaced file inside the snapshot directory, never beside the +file in your checkout, so Git cannot pick up their contents. It therefore +needs the file and the snapshot on the same filesystem; a project on another +disk than `~/.hack` is reported for you to restore by hand. +`hack upgrade rollback` without a manifest lists the snapshots and changes +nothing. + +The snapshot also records, as absent, the env layers the migration would +create, so a rollback moves them aside. This includes layers for overlays held +only in the secret store. When the snapshot cannot cover everything, it says +so and lists what it misses (`complete: false` in `--json`). Examples are a +projects registry it could not read, a live project it could not read, a file +it skipped because it is a link, and overlays it could not list without asking +the OS keychain. A rollback cannot put back what a snapshot does not hold. The snapshot does not cover the Git index, the daemon's login +service or Docker state; the steps below still apply. + +### Record the current state + Keep the records and backups private: registry and doctor output can identify local projects, and ignored env files can contain plaintext secrets. Do not paste them into an issue. Use a new backup directory and record its location yourself: diff --git a/src/cli/spec.ts b/src/cli/spec.ts index 063875158..33f99a4a3 100644 --- a/src/cli/spec.ts +++ b/src/cli/spec.ts @@ -37,6 +37,7 @@ import { sshCommand } from "../commands/ssh.ts"; import { theCommand } from "../commands/the.ts"; import { tuiCommand } from "../commands/tui.ts"; import { updateCommand } from "../commands/update.ts"; +import { upgradeCommand } from "../commands/upgrade.ts"; import { usageCommand } from "../commands/usage.ts"; import { versionCommand } from "../commands/version.ts"; import { xCommand } from "../commands/x.ts"; @@ -105,6 +106,7 @@ export const CLI_SPEC = defineCli({ internalCommand, xCommand, updateCommand, + upgradeCommand, versionCommand, helpCommand, ], diff --git a/src/commands/doctor-native-preflight.ts b/src/commands/doctor-native-preflight.ts index 81859d344..42a0813cb 100644 --- a/src/commands/doctor-native-preflight.ts +++ b/src/commands/doctor-native-preflight.ts @@ -66,6 +66,27 @@ export async function runNativePreflightCommand(opts: { readonly profiles: readonly string[]; readonly env?: Readonly>; }): Promise { + const report = await collectProjectNativePreflight(opts); + if (opts.json) { + emitCliResult({ result: okResult({ data: report }) }); + } else { + process.stdout.write(renderNativePreflightReport(report)); + } + return report.blockers.length === 0 ? 0 : 1; +} + +/** + * The native preflight report for the project at `startDir`, without + * printing it. `runtimeReview: false` skips the native runtime's own review + * (it runs the native binary and writes a temporary directory), leaving + * only the in-process checks (HACK-2189 upgrade check). + */ +export async function collectProjectNativePreflight(opts: { + readonly startDir: string; + readonly profiles: readonly string[]; + readonly env?: Readonly>; + readonly runtimeReview?: boolean; +}): Promise>> { const env = opts.env ?? process.env; const project = await findProjectContext(opts.startDir).catch( (error: unknown) => { @@ -205,12 +226,7 @@ export async function runNativePreflightCommand(opts: { hasDefaultDependencyFile, effectiveEnvName: effectiveEnv, profiles: opts.profiles, - runtime: selectedRuntime(env), + runtime: opts.runtimeReview === false ? null : selectedRuntime(env), }); - if (opts.json) { - emitCliResult({ result: okResult({ data: report }) }); - } else { - process.stdout.write(renderNativePreflightReport(report)); - } - return report.blockers.length === 0 ? 0 : 1; + return report; } diff --git a/src/commands/upgrade.ts b/src/commands/upgrade.ts new file mode 100644 index 000000000..9e4007469 --- /dev/null +++ b/src/commands/upgrade.ts @@ -0,0 +1,405 @@ +import { resolve } from "node:path"; + +import type { CliContext, CommandArgs } from "../cli/command.ts"; +import { + CliUsageError, + defineCommand, + defineOption, + withHandler, +} from "../cli/command.ts"; +import { optJson } from "../cli/options.ts"; +import { emitCliResult, okResult } from "../lib/cli-result.ts"; +import { resolveUserHomeDir } from "../lib/home-dir.ts"; +import { escapeTerminalControls } from "../lib/terminal-text.ts"; +import { + renderUpgradeCommand, + runUpgradeCheck, + type UpgradeCheckReport, + type UpgradeFinding, + upgradeCommandPrefix, +} from "../lib/upgrade-check.ts"; +import { defaultUpgradeCheckProbes } from "../lib/upgrade-check-probes.ts"; +import { + createUpgradeSnapshot, + listUpgradeSnapshots, + type RestoreUpgradeSnapshotResult, + restoreUpgradeSnapshot, +} from "../lib/upgrade-snapshot.ts"; +import { collectUpgradeSnapshotTargets } from "../lib/upgrade-snapshot-targets.ts"; + +/** + * `hack upgrade` (HACK-2188): the guided v4 → v5 upgrade. This unit + * (HACK-2189) adds the read-only check and the snapshot and rollback that + * later apply steps build on. + */ + +const checkOptions = [optJson] as const; + +const checkSpec = defineCommand({ + name: "check", + summary: + "Check this machine and every registered project for the upgrade (read-only)", + group: "Diagnostics", + description: [ + "Reads the installed versions, the daemon's service, the projects registry (dead entries apart", + "from unreadable ones), Hack's global services, proxy ports and DNS, the global secret store, and", + "for each live project its env format and key, tracked files, ignore rules, agent integrations", + "and (Apple Silicon) native preflight. Ends with an ordered plan and the next command. Changes", + "nothing and prints no value. Exits 1 when something blocks the upgrade, naming it.", + ].join("\n"), + options: checkOptions, + positionals: [], + subcommands: [], +} as const); + +const snapshotOptions = [optJson] as const; + +const snapshotSpec = defineCommand({ + name: "snapshot", + summary: "Keep a private copy of the files the upgrade changes", + group: "Diagnostics", + description: [ + "Copies the global Hack config, the projects registry, the encrypted secret store and its key", + "(when used), and each live project's .hack config, env files, env layers, .hack/.gitignore and", + "env key files into a private directory under ~/.hack/upgrade/snapshots (0700, files 0600).", + "Prints the rollback command. The manifest records paths, sizes, modes and digests, never values.", + ].join("\n"), + options: snapshotOptions, + positionals: [], + subcommands: [], +} as const); + +const optRollbackManifest = defineOption({ + name: "rollbackManifest", + type: "string", + long: "--rollback-manifest", + valueHint: "", + description: "The snapshot manifest to restore (from hack upgrade snapshot)", +} as const); + +const optOverwrite = defineOption({ + name: "overwrite", + type: "boolean", + long: "--overwrite", + description: + "Also restore files that changed since the snapshot (the current copies are kept in the snapshot)", +} as const); + +const rollbackOptions = [optRollbackManifest, optOverwrite, optJson] as const; + +const rollbackSpec = defineCommand({ + name: "rollback", + summary: "Put back the files kept by an upgrade snapshot", + group: "Diagnostics", + description: [ + "Restores each file in the manifest with its exact bytes and mode. A file that changed since the", + "snapshot (or since the step that recorded it) is left as it is unless --overwrite is passed;", + "whatever a restore replaces or removes is kept in the snapshot directory, never deleted.", + "Without --rollback-manifest, lists the snapshots and changes nothing.", + ].join("\n"), + options: rollbackOptions, + positionals: [], + subcommands: [], +} as const); + +export const upgradeCommand = defineCommand({ + name: "upgrade", + summary: "Guided upgrade: check, snapshot and roll back", + group: "Diagnostics", + options: [], + positionals: [], + subcommands: [ + withHandler(checkSpec, handleUpgradeCheck), + withHandler(snapshotSpec, handleUpgradeSnapshot), + withHandler(rollbackSpec, handleUpgradeRollback), + ], +} as const); + +/** A command line that runs this CLI (the candidate launcher during the prerelease). */ +function command(args: readonly string[]): string { + return renderUpgradeCommand( + upgradeCommandPrefix({ env: process.env, homeDir: resolveUserHomeDir() }), + args + ); +} + +// --------------------------------------------------------------------------- +// check +// --------------------------------------------------------------------------- + +async function handleUpgradeCheck({ + ctx, + args, +}: { + readonly ctx: CliContext; + readonly args: CommandArgs; +}): Promise { + const report = await runUpgradeCheck({ + cliVersion: ctx.cli.version, + homeDir: resolveUserHomeDir(), + probes: defaultUpgradeCheckProbes(), + }); + if (args.options.json === true) { + emitCliResult({ result: okResult({ data: report }) }); + } else { + process.stdout.write(renderUpgradeCheck(report)); + } + return report.ready ? 0 : 1; +} + +const MARK: Readonly> = { + ok: "ok ", + info: "info ", + warn: "warn ", + incomplete: "UNCHK", + blocker: "BLOCK", +}; + +/** Plain-text report: every line escaped, no values. */ +export function renderUpgradeCheck(report: UpgradeCheckReport): string { + const lines: string[] = []; + const line = (text: string) => lines.push(escapeTerminalControls(text)); + line( + `hack ${report.cli.version} (${report.cli.channel}${report.cli.candidate_run ? ", candidate launcher" : ""})` + ); + line(""); + line("Machine"); + for (const finding of report.machine) { + line(` ${MARK[finding.status]} ${finding.summary}`); + } + if (report.registry.missing.length > 0) { + line( + ` Dead entries: ${report.registry.missing.map((entry) => entry.name).join(", ")}` + ); + } + for (const project of report.projects) { + line(""); + line(`Project ${project.name} (${project.root})`); + for (const finding of project.findings) { + line(` ${MARK[finding.status]} ${finding.summary}`); + } + } + line(""); + for (const text of renderPlan(report)) { + line(text); + } + return `${lines.join("\n")}\n`; +} + +function renderPlan(report: UpgradeCheckReport): string[] { + if (report.plan.length === 0) { + return report.ready + ? ["Nothing to do: this machine and its projects are ready."] + : ["No step to suggest, but the check is not complete; see above."]; + } + const lines = ["Plan", ...report.plan.flatMap(renderStep), ""]; + if (report.next_command !== null) { + lines.push(`Next: ${report.next_command}`); + } else if (report.next_action !== null) { + lines.push(`Next (manual): ${report.next_action}`); + } + if (report.blockers.length > 0) { + lines.push( + "", + `Blocked by: ${report.blockers.map((blocker) => `${blocker.scope} ${blocker.id}`).join(", ")}` + ); + } + if (report.incomplete.length > 0) { + lines.push( + "", + `Not fully checked: ${report.incomplete.map((item) => `${item.scope} ${item.id}`).join(", ")}` + ); + } + return lines; +} + +function renderStep(step: UpgradeCheckReport["plan"][number]): string[] { + const tag = + { + blocker: "[blocker] ", + incomplete: "[not checked] ", + }[step.status as string] ?? ""; + const scope = step.scope === "machine" ? "" : ` (${step.scope})`; + const head = + step.command === null + ? ` ${step.order}. ${tag}manual: ${step.action ?? step.why}${scope}` + : ` ${step.order}. ${tag}${step.command}${scope}`; + return step.command !== null && step.action !== null + ? [head, ` ${step.action}`] + : [head]; +} + +// --------------------------------------------------------------------------- +// snapshot +// --------------------------------------------------------------------------- + +async function handleUpgradeSnapshot({ + args, +}: { + readonly ctx: CliContext; + readonly args: CommandArgs; +}): Promise { + const collected = await collectUpgradeSnapshotTargets(); + const gapLines = + collected.gaps.length === 0 + ? [] + : [ + "This snapshot is not a complete recovery point; it does not cover:", + ...collected.gaps.map((gap) => ` ${gap.name}: ${gap.reason}`), + ]; + const created = await createUpgradeSnapshot({ + purpose: "hack upgrade snapshot", + targets: collected.targets, + }); + const json = args.options.json === true; + if (created.status !== "created") { + const message = + created.reason === "root" + ? `The snapshot directory ${created.root} cannot be used (${created.rootReason}); nothing was written.` + : `These files cannot be snapshotted, so nothing was written: ${created.targets.map((target) => `${target.label} (${target.reason})`).join(", ")}.`; + if (json) { + emitCliResult({ + result: okResult({ + data: { status: "refused", message, detail: created }, + }), + }); + } else { + process.stderr.write(`${escapeTerminalControls(message)}\n`); + } + return 1; + } + const { snapshot } = created; + const rollback = command([ + "upgrade", + "rollback", + "--rollback-manifest", + snapshot.manifestPath, + ]); + if (json) { + emitCliResult({ + result: okResult({ + data: { + status: "created", + id: snapshot.id, + manifest_path: snapshot.manifestPath, + files: snapshot.manifest.entries.map((entry) => ({ + path: entry.path, + label: entry.label, + })), + skipped: collected.skipped, + complete: collected.gaps.length === 0, + gaps: collected.gaps, + rollback_command: rollback, + }, + }), + }); + return 0; + } + const lines = [ + `Kept ${snapshot.manifest.entries.length} file(s) in ${snapshot.dir}.`, + ...gapLines, + "To put them back:", + ` ${rollback}`, + ]; + process.stdout.write( + `${lines.map((text) => escapeTerminalControls(text)).join("\n")}\n` + ); + return 0; +} + +// --------------------------------------------------------------------------- +// rollback +// --------------------------------------------------------------------------- + +async function handleUpgradeRollback({ + ctx, + args, +}: { + readonly ctx: CliContext; + readonly args: CommandArgs; +}): Promise { + const json = args.options.json === true; + const manifest = args.options.rollbackManifest?.trim(); + if (args.options.rollbackManifest !== undefined && !manifest) { + throw new CliUsageError("--rollback-manifest needs a manifest path."); + } + if (!manifest) { + // Never guess which snapshot to restore: list them and stop. + const snapshots = await listUpgradeSnapshots(); + if (json) { + emitCliResult({ + result: okResult({ + data: { + status: "choose", + snapshots: snapshots.map((snapshot) => ({ + id: snapshot.id, + created_at: snapshot.manifest.createdAt, + purpose: snapshot.manifest.purpose, + files: snapshot.manifest.entries.length, + manifest_path: snapshot.manifestPath, + })), + }, + }), + }); + } else { + const lines = + snapshots.length === 0 + ? [ + `No upgrade snapshots. Make one with: ${command(["upgrade", "snapshot"])}`, + ] + : [ + "Choose a snapshot to restore (nothing was changed):", + ...snapshots.map( + (snapshot) => + ` ${snapshot.manifest.createdAt} ${snapshot.manifest.purpose} (${snapshot.manifest.entries.length} file(s))\n ${command(["upgrade", "rollback", "--rollback-manifest", snapshot.manifestPath])}` + ), + ]; + process.stdout.write( + `${lines.map((text) => escapeTerminalControls(text)).join("\n")}\n` + ); + } + return 1; + } + const result = await restoreUpgradeSnapshot({ + manifestPath: resolve(ctx.cwd, manifest), + overwrite: args.options.overwrite === true, + }); + if (json) { + emitCliResult({ result: okResult({ data: rollbackJson(result) }) }); + } else { + process.stdout.write(renderRollback(result)); + } + return result.status === "restored" ? 0 : 1; +} + +function rollbackJson(result: RestoreUpgradeSnapshotResult) { + if (result.status === "invalid-manifest") { + return { status: result.status, message: result.message }; + } + return { + status: result.status, + results: result.results.map((entry) => ({ + path: entry.path, + label: entry.label, + status: entry.status, + kept_as: entry.keptAs ?? null, + message: entry.message ?? null, + })), + }; +} + +function renderRollback(result: RestoreUpgradeSnapshotResult): string { + if (result.status === "invalid-manifest") { + return `${escapeTerminalControls(result.message)}\n`; + } + const lines = result.results.map( + (entry) => + `${entry.status.padEnd(16)} ${entry.label}${entry.keptAs ? ` (previous copy kept at ${entry.keptAs})` : ""}${entry.message ? `\n ${entry.message}` : ""}` + ); + lines.push( + result.status === "restored" + ? "Every file is back as it was in the snapshot." + : "Some files were not restored; see above. Nothing was deleted." + ); + return `${lines.map((text) => escapeTerminalControls(text)).join("\n")}\n`; +} diff --git a/src/daemon/launchd.ts b/src/daemon/launchd.ts index d088278b3..336e8114b 100644 --- a/src/daemon/launchd.ts +++ b/src/daemon/launchd.ts @@ -12,6 +12,7 @@ import { } from "../lib/fs.ts"; import { resolveHackInvocation } from "../lib/hack-cli.ts"; import { normalizeHomeDir, resolveUserHomeDir } from "../lib/home-dir.ts"; +import { boundedTimeoutMs } from "../lib/process-deadline.ts"; import { findExecutableInPath } from "../lib/shell.ts"; import type { DaemonPaths } from "./paths.ts"; @@ -113,18 +114,36 @@ async function runLaunchctl(args: readonly string[]): Promise { if (!access.allowed) { return { exitCode: 1, stdout: "", stderr: access.message }; } + // Within a process deadline (withProcessDeadline), never outlive it; + // outside one, unchanged. + const limitMs = boundedTimeoutMs(); + if (limitMs !== undefined && limitMs <= 0) { + return { + exitCode: 124, + stdout: "", + stderr: "launchctl not run: deadline passed", + }; + } const proc = Bun.spawn(["launchctl", ...args], { env: childEnvironment(), stdin: "ignore", stdout: "pipe", stderr: "pipe", }); - const [exitCode, stdout, stderr] = await Promise.all([ - proc.exited, - new Response(proc.stdout).text(), - new Response(proc.stderr).text(), - ]); - return { exitCode, stdout, stderr }; + const timer = + limitMs === undefined + ? undefined + : setTimeout(() => proc.kill("SIGKILL"), limitMs); + try { + const [exitCode, stdout, stderr] = await Promise.all([ + proc.exited, + new Response(proc.stdout).text(), + new Response(proc.stderr).text(), + ]); + return { exitCode, stdout, stderr }; + } finally { + clearTimeout(timer); + } } export interface LaunchdPlistOptions { diff --git a/src/daemon/systemd.ts b/src/daemon/systemd.ts index 20d60b92e..33da37e10 100644 --- a/src/daemon/systemd.ts +++ b/src/daemon/systemd.ts @@ -29,6 +29,7 @@ import { lettingOthersCreate, walkReplaceableChain, } from "../lib/path-trust.ts"; +import { boundedTimeoutMs } from "../lib/process-deadline.ts"; import { isVirtualExecutablePath, resolveLaunchdHackBinPath, @@ -75,6 +76,15 @@ export const SYSTEMD_DISABLE_ENV = "HACK_SYSTEMD_DISABLE"; export const SYSTEMD_ALLOW_ISOLATED_ENV = "HACK_SYSTEMD_ALLOW_ISOLATED_HOME"; const COMMAND_TIMEOUT_MS = 10_000; + +/** + * A command's time limit: 10 s, capped by any process deadline + * (withProcessDeadline) so no child outlives it; null once it has passed. + */ +function commandLimitMs(): number | null { + const limit = boundedTimeoutMs(COMMAND_TIMEOUT_MS) ?? COMMAND_TIMEOUT_MS; + return limit > 0 ? limit : null; +} const MAX_OUTPUT_CHARS = 64 * 1024; type Env = Readonly>; @@ -187,6 +197,14 @@ function createDefaultRunner(opts: { stderr: `refusing to run ${bin}: ${untrusted}`, }; } + const limitMs = commandLimitMs(); + if (limitMs === null) { + return { + exitCode: 124, + stdout: "", + stderr: `${command} not run: deadline passed`, + }; + } let proc: ReturnType; try { proc = Bun.spawn([bin, ...args], { @@ -207,7 +225,7 @@ function createDefaultRunner(opts: { const timer = setTimeout(() => { timedOut = true; proc.kill("SIGKILL"); - }, COMMAND_TIMEOUT_MS); + }, limitMs); try { const [exitCode, stdout, stderr] = await Promise.all([ proc.exited, @@ -218,7 +236,7 @@ function createDefaultRunner(opts: { exitCode: timedOut ? 124 : exitCode, stdout: stdout.slice(0, MAX_OUTPUT_CHARS), stderr: timedOut - ? `${command} timed out after ${COMMAND_TIMEOUT_MS}ms` + ? `${command} timed out after ${limitMs}ms` : stderr.slice(0, MAX_OUTPUT_CHARS), }; } finally { diff --git a/src/lib/doctor-generated-files.ts b/src/lib/doctor-generated-files.ts index 9b56887a7..91d70bbff 100644 --- a/src/lib/doctor-generated-files.ts +++ b/src/lib/doctor-generated-files.ts @@ -54,7 +54,10 @@ export function buildGeneratedFilePathspecs(opts: { export async function inspectTrackedGeneratedFiles(opts: { readonly projectRoot: string; readonly projectDirName: string; + /** Bound on each git call (default 10 s); a slower git reads as null. */ + readonly timeoutMs?: number; }): Promise { + const timeoutMs = opts.timeoutMs ?? 10_000; const pathspecs = buildGeneratedFilePathspecs({ projectDirName: opts.projectDirName, }); @@ -62,7 +65,7 @@ export async function inspectTrackedGeneratedFiles(opts: { try { result = await exec( ["git", "-C", opts.projectRoot, "ls-files", "-z", "--", ...pathspecs], - { stdin: "ignore" } + { stdin: "ignore", timeoutMs } ); } catch { return null; @@ -95,7 +98,7 @@ export async function inspectTrackedGeneratedFiles(opts: { "--", `${opts.projectDirName}/.env.*`, ], - { stdin: "ignore" } + { stdin: "ignore", timeoutMs } ); } catch { return null; diff --git a/src/lib/git-worktree.ts b/src/lib/git-worktree.ts index 6a270e83e..2d7a1e25c 100644 --- a/src/lib/git-worktree.ts +++ b/src/lib/git-worktree.ts @@ -19,8 +19,10 @@ async function readGitPath(opts: { }): Promise { let result: Awaited>; try { + // A hung git reads as unknown (null), never a wait without end. result = await exec(["git", "-C", opts.repoRoot, ...opts.args], { stdin: "ignore", + timeoutMs: 10_000, }); } catch { return null; diff --git a/src/lib/process-deadline.ts b/src/lib/process-deadline.ts new file mode 100644 index 000000000..68ecaad8d --- /dev/null +++ b/src/lib/process-deadline.ts @@ -0,0 +1,36 @@ +import { AsyncLocalStorage } from "node:async_hooks"; + +/** + * A deadline for the child processes started within an async scope + * (HACK-2189). Shared helpers (`exec` and the git probes) bound their own + * timeout by it, and start nothing once it has passed, so a caller that + * gives up on a check at its deadline leaves no child running after it. + * Outside such a scope nothing changes. + */ +const scope = new AsyncLocalStorage<{ readonly deadline: number }>(); + +/** Runs `run` with every child process it starts ended by `deadline` (epoch ms). */ +export async function withProcessDeadline( + deadline: number, + run: () => Promise +): Promise { + const outer = scope.getStore()?.deadline; + return await scope.run( + { deadline: outer === undefined ? deadline : Math.min(outer, deadline) }, + run + ); +} + +/** + * The timeout a child process may have: `requested`, capped by the + * scope's remaining time. Undefined (no limit) only outside a scope with no + * request; zero or less means the scope's deadline has passed. + */ +export function boundedTimeoutMs(requested?: number): number | undefined { + const deadline = scope.getStore()?.deadline; + if (deadline === undefined) { + return requested; + } + const remaining = deadline - Date.now(); + return requested === undefined ? remaining : Math.min(requested, remaining); +} diff --git a/src/lib/project-env-config.ts b/src/lib/project-env-config.ts index 25aa07ac7..5b8a92279 100644 --- a/src/lib/project-env-config.ts +++ b/src/lib/project-env-config.ts @@ -111,6 +111,7 @@ import { readReplaceablePrivateFile, replacePrivateFileAtomic, } from "./private-file.ts"; +import { boundedTimeoutMs } from "./process-deadline.ts"; import { parseEnvConfigSelection, readProjectDefaultEnvConfig, @@ -775,6 +776,17 @@ function toGitProbePath(opts: { return relativePath.startsWith("../") ? relativePath : `./${relativePath}`; } +/** Bound on a git ignore/tracking probe; a slower git reads as unknown. */ +const GIT_PROBE_TIMEOUT_MS = 10_000; + +/** The probe's timeout within any process deadline; at least 1 ms (Bun's 0 means none). */ +function gitProbeTimeoutMs(): number { + return Math.max( + 1, + boundedTimeoutMs(GIT_PROBE_TIMEOUT_MS) ?? GIT_PROBE_TIMEOUT_MS + ); +} + async function isGitTrackedProjectEnvPath(opts: { readonly projectRoot: string; readonly path: string; @@ -797,6 +809,9 @@ async function isGitTrackedProjectEnvPath(opts: { stderr: "ignore", stdin: "ignore", stdout: "ignore", + // A hung git is unknown (null), never a wait without end. + timeout: gitProbeTimeoutMs(), + killSignal: "SIGKILL", }); } catch { return null; @@ -839,6 +854,9 @@ export async function isGitIgnoredProjectEnvPath(opts: { stderr: "ignore", stdin: "ignore", stdout: "ignore", + // A hung git is unknown (null), never a wait without end. + timeout: gitProbeTimeoutMs(), + killSignal: "SIGKILL", }); } catch { return null; @@ -4603,6 +4621,8 @@ async function isGitTrackedProjectEnvPathAnyCase(opts: { stderr: "ignore", stdin: "ignore", stdout: "pipe", + timeout: gitProbeTimeoutMs(), + killSignal: "SIGKILL", }); const [listed, exitCode] = await Promise.all([ new Response(proc.stdout).text(), @@ -10410,6 +10430,55 @@ function collectLegacyEncryptedBackendCleanupCandidates(opts: { return [...candidates].sort((left, right) => left.localeCompare(right)); } +/** + * Read-only preview for `hack upgrade snapshot` (HACK-2189): the legacy + * overlays a migration of this project would convert into env layers, from + * the migration's own selection (files plus overlays only the secret store + * holds). Writes and provisions nothing, and returns names only. When the + * contract is absent or the migration would refuse, the file-discovered + * overlays (what an undeclared-dotenv migration converts). + */ +/** + * Read-only (HACK-2189): the overlays a migration of this project finds + * without its secret store: `.env.` files and the configured default + * overlay, exactly as the migration discovers them. Empty when the + * directory cannot be read. + */ +export async function discoverLegacyMigrationOverlays(opts: { + readonly projectDir: string; +}): Promise { + return await discoverLegacyOverlayNames({ + projectDir: opts.projectDir, + includeSymlinks: false, + }).catch(() => [] as string[]); +} + +export async function previewLegacyMigrationOverlayNames(opts: { + readonly projectDir: string; + readonly projectName: string; +}): Promise { + const discovered = await discoverLegacyOverlayNames({ + projectDir: opts.projectDir, + includeSymlinks: false, + }).catch(() => [] as string[]); + const contract = await readHackEnvContract({ projectDir: opts.projectDir }); + if (!contract.exists || contract.parseError !== undefined) { + return discovered; + } + try { + return ( + await selectContractMigrationOverlays({ + projectDir: opts.projectDir, + projectName: opts.projectName, + contract: contract.contract, + composeRepairFollows: true, + }) + ).overlayNames; + } catch { + return discovered; + } +} + /** * The legacy overlays a contract migration converts, checked before anything is * written or any key material can be provisioned: diff --git a/src/lib/projects-registry.ts b/src/lib/projects-registry.ts index 90e416007..96120d690 100644 --- a/src/lib/projects-registry.ts +++ b/src/lib/projects-registry.ts @@ -750,6 +750,11 @@ function resolveGlobalRegistryRoot(): string { return resolveGlobalHackDir(); } +/** The projects registry file this process reads (`projects.json`). */ +export function resolveProjectsRegistryPath(): string { + return getRegistryPath(); +} + function getRegistryPath(): string { return resolve( resolveGlobalRegistryRoot(), diff --git a/src/lib/secret-store-retire.ts b/src/lib/secret-store-retire.ts index aa1b8202a..359baac93 100644 --- a/src/lib/secret-store-retire.ts +++ b/src/lib/secret-store-retire.ts @@ -125,6 +125,11 @@ type StoreContexts = { })[]; readonly registryRead: boolean; readonly unchecked: number; + /** + * Of `unchecked`: checkouts whose directory does not exist (a deleted + * checkout, or an unmounted volume). Retiring counts them as unchecked; + * the upgrade check counts them as dead registry entries. + */ readonly missing: number; }; @@ -288,6 +293,40 @@ export async function listProjectsUsingSecretStore(opts: { }; } +/** + * The registered checkouts that read the store at `storePath`, each with the + * key file its own configuration selects (HACK-2189): a project may share a + * store but name its own key. From configuration only; no store is opened. + */ +export async function listSecretStoreReaders(opts: { + readonly storePath: string; +}): Promise<{ + readonly readers: readonly { + readonly name: string; + readonly projectDir: string; + readonly keyPath: string; + }[]; + readonly registryRead: boolean; + /** Checkouts that could not be checked, not counting missing ones. */ + readonly unchecked: number; + /** Registered checkouts whose directory does not exist (dead entries). */ + readonly missing: number; +}> { + const found = await storeContexts(opts); + const readers: { name: string; projectDir: string; keyPath: string }[] = []; + for (const context of found.contexts) { + if (await projectUsesSecretStore({ projectDir: context.projectDir })) { + readers.push(context); + } + } + return { + readers, + registryRead: found.registryRead, + unchecked: found.unchecked - found.missing, + missing: found.missing, + }; +} + /** * Why retiring this store would take it from another registered checkout * that can still open it, or null when none can (HACK-2177). A checkout diff --git a/src/lib/secret-store.ts b/src/lib/secret-store.ts index 90b105da0..2ae5269f6 100644 --- a/src/lib/secret-store.ts +++ b/src/lib/secret-store.ts @@ -573,6 +573,75 @@ export async function keyFileOpensStore(input: { }); } +/** + * The state of an encrypted store and the key the runtime would use for it, + * for `hack upgrade check` (HACK-2189). Read-only and value-free: nothing is + * provisioned, no key or value is returned, and the OS keychain is never + * asked (asking can prompt), so a missing key file with the keychain + * fallback on is `key-uninspected`, never "the key is gone". + * + * Follows the runtime's order: `HACK_SECRETS_FILE_KEY` (from `env`) wins; + * otherwise the key file, read through the runtime's own reader, so its + * refusals (a link, group/other access, tracked by Git) are reported as + * `key-refused` even when the bytes would decrypt the store. + */ +export async function inspectStoreKeyState(input: { + readonly storePath: string; + readonly keyPath: string; + readonly env: Readonly>; +}): Promise< + | { readonly kind: "no-store" } + | { readonly kind: "store-unreadable" } + | { readonly kind: "opens"; readonly source: "env" | "file" } + | { readonly kind: "wrong-key"; readonly source: "env" | "file" } + | { readonly kind: "key-refused"; readonly message: string } + | { readonly kind: "key-missing" } + | { readonly kind: "key-uninspected" } +> { + const store = await readEncryptedStoreText(input.storePath); + if ( + store.kind === "absent" || + (store.kind === "text" && store.text.length === 0) + ) { + return { kind: "no-store" }; + } + if (store.kind !== "text") { + return { kind: "store-unreadable" }; + } + const envKey = (input.env[ENCRYPTED_FILE_KEY_ENV] ?? "").trim(); + if (envKey.length > 0) { + const opened = storeTextOpensUnder({ + storePath: input.storePath, + text: store.text, + key: envKey, + }); + return opened === "opens" + ? { kind: "opens", source: "env" } + : { kind: "wrong-key", source: "env" }; + } + let fileKey: string | null; + try { + fileKey = await readEncryptedFileKey({ keyPath: input.keyPath }); + } catch (error: unknown) { + if (error instanceof SecretStoreKeyRefusedError) { + return { kind: "key-refused", message: error.message }; + } + return { kind: "store-unreadable" }; + } + if (fileKey === null) { + return isKeychainDisabled() + ? { kind: "key-missing" } + : { kind: "key-uninspected" }; + } + return storeTextOpensUnder({ + storePath: input.storePath, + text: store.text, + key: fileKey, + }) === "opens" + ? { kind: "opens", source: "file" } + : { kind: "wrong-key", source: "file" }; +} + /** * Whether the encrypted_file key in the OS keychain (the runtime's fallback * when a project has no key file) opens the store (HACK-2177). Asked @@ -1726,7 +1795,7 @@ async function readEncryptedFileKey(input: { } if ((read.mode & 0o077) !== 0) { throw new SecretStoreKeyRefusedError( - `Refused the encrypted secret key at ${input.keyPath}: other users can read or write it (mode ${(read.mode & 0o777).toString(8)}; values omitted). Run: chmod 600 ${input.keyPath}` + `Refused the encrypted secret key at ${input.keyPath}: other users can read or write it (mode ${(read.mode & 0o777).toString(8)}; values omitted). Run: chmod 600 ${quoteShellWord(input.keyPath)}` ); } const tracked = await isKeyFileTrackedByGit(input.keyPath); diff --git a/src/lib/shell.ts b/src/lib/shell.ts index f8bcbc361..28fc4186e 100644 --- a/src/lib/shell.ts +++ b/src/lib/shell.ts @@ -6,6 +6,7 @@ import { beginNativeCpuChild, nativeCpuCommandCategory, } from "./native-cpu-diagnostics.ts"; +import { boundedTimeoutMs } from "./process-deadline.ts"; import { readSubprocessResourceUsage } from "./process-resource-usage.ts"; import { hasControllingTerminal } from "./tty-process-group.ts"; @@ -65,18 +66,24 @@ export async function exec( cmd: readonly string[], opts: ExecOptions = {} ): Promise { + // Within a process deadline (withProcessDeadline), never outlive it, and + // start nothing once it has passed. + const timeoutMs = boundedTimeoutMs(opts.timeoutMs); + if (timeoutMs !== undefined && timeoutMs <= 0) { + return { exitCode: 124, stdout: "", stderr: "" }; + } const proc = Bun.spawn([...cmd], { cwd: opts.cwd, env: buildSpawnEnv(opts.env, opts.unsetEnvKeys), stdin: opts.stdin ?? "inherit", stdout: "pipe", stderr: "pipe", - detached: opts.timeoutMs !== undefined, + detached: timeoutMs !== undefined, }); const timeout = installSubprocessTimeout({ pid: proc.pid, - timeoutMs: opts.timeoutMs, + timeoutMs, }); const observeCpu = beginNativeCpuChild(proc, nativeCpuCommandCategory(cmd)); diff --git a/src/lib/upgrade-check-probes.ts b/src/lib/upgrade-check-probes.ts new file mode 100644 index 000000000..8a7b382d2 --- /dev/null +++ b/src/lib/upgrade-check-probes.ts @@ -0,0 +1,265 @@ +import { resolve } from "node:path"; +import { resolveSystemdSummary } from "../commands/daemon-systemd.ts"; +import { collectProjectNativePreflight } from "../commands/doctor-native-preflight.ts"; +import { inspectLinuxGlobalDns } from "../commands/global-linux-dns.ts"; +import { + GLOBAL_CADDY_COMPOSE_FILENAME, + GLOBAL_CADDY_DIR_NAME, + GLOBAL_CADDY_PROJECT_NAME, +} from "../constants.ts"; +import { getLaunchdServiceStatus } from "../daemon/launchd.ts"; +import { resolveDaemonPaths } from "../daemon/paths.ts"; +import { summarizeLaunchdService } from "../daemon/service-summary.ts"; +import { readDaemonStatus } from "../daemon/status.ts"; +import { resolveGlobalHackDir } from "./config-paths.ts"; +import { pathExists } from "./fs.ts"; +import type { LinuxDnsInspection } from "./linux-dns-setup.ts"; +import { + inspectProxyLoopback, + type ProxyLoopbackReport, +} from "./proxy-loopback.ts"; +import { detectHackInstall } from "./self-update.ts"; +import { exec } from "./shell.ts"; + +/** + * The host probes `hack upgrade check` uses (HACK-2189). Every probe is + * read-only: it reads files or runs a read-only command with a time limit, + * and never starts Docker, the daemon or a service, prompts, or writes. + * Tests replace them; the project checks run against real files. + */ +export type UpgradeCheckProbes = { + /** True for the refusing stand-ins; the report is then incomplete. */ + readonly disabled?: boolean; + /** The first `hack` on PATH: where it is, how it was installed, its version. */ + readonly pathHack: () => Promise; + /** Who runs hackd and whether a login service is registered. */ + readonly daemon: () => Promise; + /** Whether the Docker engine answers. */ + readonly docker: () => Promise<"reachable" | "unreachable" | "missing">; + /** Hack's global proxy files and containers. */ + readonly globalInfra: (opts: { + readonly docker: "reachable" | "unreachable" | "missing"; + }) => Promise; + /** Linux only: the DNS route on Hack's bridge; null elsewhere. */ + readonly linuxDns: () => Promise; + /** The loopback proxy ports check (doctor's `proxy ports`). */ + readonly proxyPorts: () => Promise; + /** Apple Silicon only: the in-process native preflight for one project. */ + readonly nativePreflight: (opts: { + readonly projectRoot: string; + }) => Promise; +}; + +export type PathHackProbe = { + readonly path: string | null; + readonly install: string; + readonly version: string | null; +}; + +export type DaemonProbe = { + /** `systemd`, `launchd`, or null on a host with neither. */ + readonly manager: "systemd" | "launchd" | null; + /** A login service is registered (unit or plist installed). */ + readonly registered: boolean; + /** The service is enabled to start at login; null when unknown. */ + readonly enabled: boolean | null; + /** hackd is running (its pid file names a live process). */ + readonly running: boolean; + /** The service manager runs that hackd. */ + readonly managed: boolean; + readonly summary: string; + readonly severity: "ok" | "warn"; + readonly nextStep: string | null; +}; + +export type GlobalInfraProbe = { + /** `hack global install` has written the proxy files. */ + readonly installed: boolean; + /** The proxy containers run; null when it was not established. */ + readonly running: boolean | null; + /** True when Docker answered but listing the containers failed or timed out. */ + readonly listFailed?: boolean; +}; + +export type NativePreflightProbe = { + readonly ready: boolean; + readonly blockers: number; + readonly warnings: number; +}; + +const COMMAND_TIMEOUT_MS = 5000; +const WHITESPACE = /\s+/u; + +/** + * Hard deny: when set to `1`, no host probe runs (Docker, the Linux + * resolver, service managers, the `hack` on PATH, native preflight); each + * of their checks reports "not checked", never a pass. The test preload sets + * it, so tests never read the host. + */ +export const HOST_PROBES_DISABLE_ENV = "HACK_HOST_PROBES_DISABLE"; + +const disabled = (): Promise => + Promise.reject(new Error("Host probes are disabled.")); + +/** The real probes, or refusing stand-ins under {@link HOST_PROBES_DISABLE_ENV}. */ +export function defaultUpgradeCheckProbes( + opts: { readonly env?: Readonly> } = {} +): UpgradeCheckProbes { + const env = opts.env ?? process.env; + if ((env[HOST_PROBES_DISABLE_ENV] ?? "").trim() === "1") { + return { + disabled: true, + pathHack: disabled, + daemon: disabled, + docker: () => Promise.resolve("unreachable" as const), + globalInfra: disabled, + linuxDns: disabled, + proxyPorts: disabled, + // Reported once, as the report-level host-probes finding. + nativePreflight: () => Promise.resolve(null), + }; + } + return { + pathHack: probePathHack, + daemon: probeDaemon, + docker: probeDocker, + globalInfra: probeGlobalInfra, + linuxDns: async () => + process.platform === "linux" ? await inspectLinuxGlobalDns() : null, + proxyPorts: async () => await inspectProxyLoopback(), + nativePreflight: probeNativePreflight, + }; +} + +async function probePathHack(): Promise { + const path = Bun.which("hack"); + if (path === null) { + return { path: null, install: "missing", version: null }; + } + const install = await detectHackInstall({ path }); + const kind = install.status === "present" ? install.kind : install.status; + const result = await exec([path, "--version"], { + stdin: "ignore", + timeoutMs: COMMAND_TIMEOUT_MS, + }).catch(() => null); + const version = + result !== null && result.exitCode === 0 + ? (result.stdout.trim().split(WHITESPACE).at(-1) ?? null) + : null; + return { path, install: kind, version }; +} + +async function probeDaemon(): Promise { + const paths = resolveDaemonPaths({}); + const daemon = await readDaemonStatus({ paths }); + if (process.platform === "linux") { + const summary = await resolveSystemdSummary({ paths }); + if (summary !== null) { + return { + manager: "systemd", + registered: summary.installed, + enabled: summary.enabled, + running: daemon.running, + managed: summary.ownsDaemon, + summary: summary.summary, + severity: summary.severity === "warn" ? "warn" : "ok", + nextStep: summary.nextStep, + }; + } + } + if (process.platform === "darwin") { + const status = await getLaunchdServiceStatus({ paths }); + const summary = summarizeLaunchdService({ + status, + daemonPid: daemon.pid, + daemonRunning: daemon.running, + }); + return { + manager: "launchd", + registered: status.installed, + enabled: status.installed ? status.loaded : null, + running: daemon.running, + managed: summary.ownsDaemon, + summary: summary.summary, + severity: summary.severity === "warn" ? "warn" : "ok", + nextStep: summary.nextStep, + }; + } + return { + manager: null, + registered: false, + enabled: null, + running: daemon.running, + managed: false, + summary: "no login service manager on this host", + severity: "ok", + nextStep: null, + }; +} + +async function probeDocker(): Promise<"reachable" | "unreachable" | "missing"> { + if (Bun.which("docker") === null) { + return "missing"; + } + const result = await exec( + ["docker", "info", "--format", "{{.ServerVersion}}"], + { + stdin: "ignore", + timeoutMs: COMMAND_TIMEOUT_MS, + } + ).catch(() => null); + return result !== null && result.exitCode === 0 ? "reachable" : "unreachable"; +} + +async function probeGlobalInfra(opts: { + readonly docker: "reachable" | "unreachable" | "missing"; +}): Promise { + const installed = await pathExists( + resolve( + resolveGlobalHackDir(), + GLOBAL_CADDY_DIR_NAME, + GLOBAL_CADDY_COMPOSE_FILENAME + ) + ); + if (opts.docker !== "reachable") { + return { installed, running: null }; + } + const result = await exec( + [ + "docker", + "ps", + "--filter", + `label=com.docker.compose.project=${GLOBAL_CADDY_PROJECT_NAME}`, + "--format", + "{{.State}}", + ], + { stdin: "ignore", timeoutMs: COMMAND_TIMEOUT_MS } + ).catch(() => null); + if (result === null || result.exitCode !== 0) { + return { installed, running: null, listFailed: true }; + } + const states = result.stdout.split("\n").filter((line) => line.trim() !== ""); + return { + installed, + running: + states.length > 0 && states.every((state) => state.trim() === "running"), + }; +} + +async function probeNativePreflight(opts: { + readonly projectRoot: string; +}): Promise { + if (process.platform !== "darwin" || process.arch !== "arm64") { + return null; + } + const report = await collectProjectNativePreflight({ + startDir: opts.projectRoot, + profiles: [], + runtimeReview: false, + }); + return { + ready: report.blockers.length === 0, + blockers: report.blockers.length, + warnings: report.warnings.length, + }; +} diff --git a/src/lib/upgrade-check-project.ts b/src/lib/upgrade-check-project.ts new file mode 100644 index 000000000..e22c9ca7a --- /dev/null +++ b/src/lib/upgrade-check-project.ts @@ -0,0 +1,786 @@ +import { constants } from "node:fs"; +import { type FileHandle, open, readdir, stat } from "node:fs/promises"; +import { basename, resolve } from "node:path"; + +import { inspectDoctorAgentIntegrations } from "../commands/doctor.ts"; +import { + PROJECT_COMPOSE_FILENAME, + PROJECT_CONFIG_FILENAME, + PROJECT_CONFIG_LEGACY_FILENAME, + PROJECT_ENV_CONTRACT_FILENAME, + PROJECT_ENV_FILENAME, + PROJECT_ENV_STATE_FILENAME, +} from "../constants.ts"; +import { + gitIsLocalKeyIgnored, + gitListTrackedKeyPaths, + inspectProjectEnvKey, + resolveEnvKeyCandidates, +} from "./doctor-env-key.ts"; +import { + inspectHackDirGitignoreForDoctor, + inspectTrackedGeneratedFiles, +} from "./doctor-generated-files.ts"; +import { withProcessDeadline } from "./process-deadline.ts"; +import { + inspectProjectEnvLayerKeys, + ProjectEnvLayerPathConflictError, + projectEnvConfigExistsForMutation, +} from "./project-env-config.ts"; +import type { RegisteredProject } from "./projects-registry.ts"; +import { + inspectStoreKeyState, + resolveFileSecretStorePaths, +} from "./secret-store.ts"; +import { projectUsesSecretStore } from "./secret-store-retire.ts"; +import type { NativePreflightProbe } from "./upgrade-check-probes.ts"; + +/** + * Per-checkout checks for `hack upgrade check` (HACK-2189): the checkout's + * state in the registry, then, for a live checkout, its env format, env + * key, tracked machine-local files, ignore rules, agent integrations and + * (Apple Silicon) native preflight. Read-only; reports names, paths, counts + * and codes, never a value. + */ + +/** + * - `blocker`: continuing would fail or lose data. + * - `incomplete`: a check the upgrade's safety depends on could not be + * performed (it failed or ran out of time); readiness is not established. + * - `warn`: a step to take; `info` and `ok` need nothing. + */ +export type UpgradeFindingStatus = + | "ok" + | "info" + | "warn" + | "incomplete" + | "blocker"; + +export type UpgradeFinding = { + /** Stable id, for example `env-format` or `secret-store`. */ + readonly id: string; + readonly status: UpgradeFindingStatus; + readonly summary: string; + /** + * The Hack CLI arguments of the command that fixes it (or diagnoses it), + * run through the CLI that produced the report. Null when only a manual + * action applies. + */ + readonly args: readonly string[] | null; + /** A manual action, when no command fixes it or as its alternative. Never run. */ + readonly action: string | null; +}; + +/** Checks the upgrade's safety depends on; a failure there is `incomplete`. */ +const REQUIRED_CHECKS = new Set([ + "project-store", + "env-secrets", + "env-format", + "env-key", + "tracked-files", + "ignore-rules", +]); + +export type RegistryEntryState = + | { readonly state: "live" } + /** The checkout or its Hack config is gone: a dead entry to prune. */ + | { readonly state: "missing"; readonly reason: string } + /** It exists but cannot be read: a live project to repair, not prune. */ + | { readonly state: "unreadable"; readonly reason: string }; + +/** + * One registered checkout: a project's primary checkout or one of its + * registered linked worktrees, each checked on its own. + */ +export type RegisteredCheckout = { + /** The project name, with ` (worktree )` for a linked worktree. */ + readonly name: string; + readonly repoRoot: string; + readonly projectDir: string; + readonly projectDirName: string; +}; + +/** A registered project's primary checkout and its registered worktrees. */ +export function registeredCheckouts( + project: RegisteredProject +): RegisteredCheckout[] { + return [ + { + name: project.name, + repoRoot: project.repoRoot, + projectDir: project.projectDir, + projectDirName: project.projectDirName, + }, + ...(project.worktrees ?? []).map((worktree) => ({ + name: `${project.name} (worktree ${basename(worktree.path)})`, + repoRoot: worktree.path, + projectDir: resolve(worktree.path, project.projectDirName), + projectDirName: project.projectDirName, + })), + ]; +} + +/** Largest config file the classifier reads. */ +const MAX_CONFIG_BYTES = 1024 * 1024; + +/** + * Classifies a registered checkout from its files only. Config files are + * read only when they are regular files, through a non-blocking no-follow + * open with a size bound, so a FIFO or link at a config path cannot stall + * or redirect the check. + */ +export async function classifyRegistryEntry( + project: RegisteredCheckout +): Promise { + const root = await statState(project.repoRoot); + if (root === "absent") { + return { state: "missing", reason: "repo root is gone" }; + } + if (root !== "directory") { + return { state: "unreadable", reason: "repo root cannot be read" }; + } + const dir = await statState(project.projectDir); + if (dir === "absent") { + return { state: "missing", reason: `${project.projectDirName}/ is gone` }; + } + if (dir !== "directory") { + return { + state: "unreadable", + reason: `${project.projectDirName}/ cannot be read`, + }; + } + const config = await readConfigText( + resolve(project.projectDir, PROJECT_CONFIG_FILENAME) + ); + if (config.kind === "unreadable") { + return { + state: "unreadable", + reason: `${PROJECT_CONFIG_FILENAME} cannot be read (or is not a regular file)`, + }; + } + if (config.kind === "text") { + try { + JSON.parse(config.text); + } catch { + return { + state: "unreadable", + reason: `${PROJECT_CONFIG_FILENAME} is not valid JSON`, + }; + } + return { state: "live" }; + } + // No JSON config: the legacy TOML config or a Compose file still makes it + // a project (the same inputs project discovery and prune accept). + for (const name of [ + PROJECT_CONFIG_LEGACY_FILENAME, + PROJECT_COMPOSE_FILENAME, + ]) { + const other = await readConfigText(resolve(project.projectDir, name)); + if (other.kind === "unreadable") { + return { + state: "unreadable", + reason: `${name} cannot be read (or is not a regular file)`, + }; + } + if (other.kind === "text") { + return { state: "live" }; + } + } + return { + state: "missing", + reason: `its config (${PROJECT_CONFIG_FILENAME}, ${PROJECT_CONFIG_LEGACY_FILENAME} or ${PROJECT_COMPOSE_FILENAME}) is gone`, + }; +} + +async function readConfigText( + path: string +): Promise< + | { readonly kind: "text"; readonly text: string } + | { readonly kind: "absent" | "unreadable" } +> { + let handle: FileHandle | null = null; + try { + handle = await open( + path, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ); + const info = await handle.stat(); + if (!info.isFile() || info.size > MAX_CONFIG_BYTES) { + return { kind: "unreadable" }; + } + // Bounded even if the file grows after the stat. + const buffer = Buffer.alloc(MAX_CONFIG_BYTES + 1); + const { bytesRead } = await handle.read(buffer, 0, buffer.length, 0); + if (bytesRead > MAX_CONFIG_BYTES) { + return { kind: "unreadable" }; + } + return { + kind: "text", + text: buffer.subarray(0, bytesRead).toString("utf8"), + }; + } catch (error: unknown) { + const code = errorCode(error); + return code === "ENOENT" || code === "ENOTDIR" + ? { kind: "absent" } + : { kind: "unreadable" }; + } finally { + await handle?.close(); + } +} + +/** Env formats a project's `.hack/` holds. */ +export type EnvFormat = "layers" | "legacy-json" | "dotenv"; + +/** + * Which env formats the project dir holds: YAML layers + * (`hack.env.*.yaml`), the legacy JSON contract (`hack.env.json`) and + * legacy dotenv files (`.env`, `.env.`). Conflict when a layer + * path is a link or not a file (the same rule `hack env set` uses). + */ +export async function detectEnvFormats(opts: { + readonly projectDir: string; +}): Promise< + | { readonly kind: "formats"; readonly formats: readonly EnvFormat[] } + | { readonly kind: "conflict" } +> { + const formats: EnvFormat[] = []; + try { + if (await projectEnvConfigExistsForMutation(opts)) { + formats.push("layers"); + } + } catch (error: unknown) { + if (error instanceof ProjectEnvLayerPathConflictError) { + return { kind: "conflict" }; + } + throw error; + } + const names = await readdir(opts.projectDir); + if (names.includes(PROJECT_ENV_CONTRACT_FILENAME)) { + formats.push("legacy-json"); + } + if ( + names.some( + (name) => + name !== PROJECT_ENV_STATE_FILENAME && + (name === PROJECT_ENV_FILENAME || + (name.startsWith(`${PROJECT_ENV_FILENAME}.`) && + !name.endsWith(".json") && + !name.endsWith(".tmp"))) + ) + ) { + formats.push("dotenv"); + } + return { kind: "formats", formats }; +} + +function envFormatFinding(opts: { + readonly detected: Awaited>; + readonly root: string; +}): UpgradeFinding { + if (opts.detected.kind === "conflict") { + return { + id: "env-format", + status: "blocker", + summary: + "An env layer path (.hack/hack.env.*.yaml) is a link or not a regular file, so the env format cannot be decided.", + args: ["doctor", "--path", opts.root], + action: null, + }; + } + const { formats } = opts.detected; + const legacy = formats.filter((format) => format !== "layers"); + if (legacy.length === 0) { + return { + id: "env-format", + status: "ok", + summary: + formats.length === 0 + ? "No env configuration." + : "Env uses YAML layers (current format).", + args: null, + action: null, + }; + } + return { + id: "env-format", + status: "warn", + summary: `Env uses the legacy format (${legacy.join(", ")}${formats.includes("layers") ? ", beside YAML layers" : ""}); migrate it to YAML layers.`, + args: ["doctor", "--migrate-env-config", "--path", opts.root], + action: null, + }; +} + +async function envKeyFinding(opts: { + readonly root: string; +}): Promise { + const inspection = await inspectProjectEnvKey({ + projectRoot: opts.root, + candidates: await resolveEnvKeyCandidates({ projectRoot: opts.root }), + env: process.env, + uid: typeof process.getuid === "function" ? process.getuid() : null, + listTrackedKeyPaths: gitListTrackedKeyPaths, + isLocalKeyIgnored: gitIsLocalKeyIgnored, + }); + const worst = inspection.findings[0]; + const codes = inspection.findings + .filter((finding) => finding.status !== "ok") + .map((finding) => finding.code); + if (inspection.status === "ok" || worst === undefined) { + const present = !inspection.findings.some( + (finding) => finding.code === "env-key-absent" + ); + return { + id: "env-key", + status: "ok", + summary: present + ? "Env key present, private and untracked." + : "No env key file (whether one is needed is the env-secrets check).", + args: null, + action: null, + }; + } + return { + id: "env-key", + status: inspection.status === "error" ? "blocker" : "warn", + summary: `Env key: ${codes.join(", ")}.`, + args: ["doctor", "--path", opts.root], + action: worst.remedy ? `In the project root: ${worst.remedy}` : null, + }; +} + +async function trackedFilesFinding(opts: { + readonly root: string; + readonly projectDirName: string; + readonly timeoutMs: number; +}): Promise { + const tracked = await inspectTrackedGeneratedFiles({ + projectRoot: opts.root, + projectDirName: opts.projectDirName, + timeoutMs: opts.timeoutMs, + }); + const args = ["doctor", "--fix", "--path", opts.root]; + if (tracked === null) { + // A checkout Git failed on is not a directory without Git: whether + // sensitive files are tracked there is unknown. + return (await insideGitWorkTree(opts.root)) + ? { + id: "tracked-files", + status: "incomplete", + summary: + "Git could not list this checkout's tracked files (an unreadable index, or a repository Git refuses), so whether sensitive files are tracked is unknown.", + args: ["doctor", "--path", opts.root], + action: null, + } + : { + id: "tracked-files", + status: "info", + summary: "Not a Git checkout; no tracked files to check.", + args: null, + action: null, + }; + } + if (tracked.secretKeyTracked) { + return { + id: "tracked-files", + status: "blocker", + summary: + "The project's env secret key is tracked in Git (a committed secret).", + args, + action: null, + }; + } + const count = + tracked.trackedPaths.length + + tracked.trackedOverlayPaths.length + + tracked.unignoredOverlayPaths.length; + if (count === 0) { + return { + id: "tracked-files", + status: "ok", + summary: "No machine-local or sensitive files are tracked.", + args: null, + action: null, + }; + } + return { + id: "tracked-files", + status: "warn", + summary: `${count} machine-local or sensitive file(s) are tracked or not ignored.`, + args, + action: null, + }; +} + +async function ignoreRulesFinding(opts: { + readonly root: string; + readonly projectDir: string; +}): Promise { + const state = await inspectHackDirGitignoreForDoctor({ + projectDir: opts.projectDir, + }); + if (state === null) { + // Git failing on a checkout is not a directory without Git. + return (await insideGitWorkTree(opts.root)) + ? { + id: "ignore-rules", + status: "incomplete", + summary: + "Git could not inspect this checkout, so whether its .hack/.gitignore keeps machine-local files out is unknown.", + args: ["doctor", "--path", opts.root], + action: null, + } + : { + id: "ignore-rules", + status: "ok", + summary: "Not a Git checkout; no ignore rules needed.", + args: null, + action: null, + }; + } + if (state === "current") { + return { + id: "ignore-rules", + status: "ok", + summary: "The committed .hack/.gitignore is current.", + args: null, + action: null, + }; + } + return { + id: "ignore-rules", + status: "warn", + summary: `The committed .hack/.gitignore is ${state}.`, + args: ["doctor", "--fix", "--path", opts.root], + action: null, + }; +} + +async function agentFinding(opts: { + readonly root: string; + readonly homeDir: string; +}): Promise { + const state = await inspectDoctorAgentIntegrations({ + projectRoot: opts.root, + homeDir: opts.homeDir, + }); + return state.status === "current" + ? { + id: "agent-integrations", + status: "ok", + summary: "Agent integrations are current (or not installed).", + args: null, + action: null, + } + : { + id: "agent-integrations", + status: "warn", + summary: + "Standalone Hack agent files are stale; the hack plugin replaces them.", + args: ["setup", "migrate", "--path", opts.root], + action: null, + }; +} + +function nativeFinding( + probe: NativePreflightProbe | null +): UpgradeFinding | null { + if (probe === null) { + return null; + } + return probe.ready + ? { + id: "native-preflight", + status: "ok", + summary: `Ready for the native backend (${probe.warnings} warning(s)).`, + args: null, + action: null, + } + : { + // Native is optional: a Compose project upgrades without it. + id: "native-preflight", + status: "info", + summary: `Not ready for the native backend (${probe.blockers} blocker(s)); Compose is unaffected.`, + args: null, + action: null, + }; +} + +/** + * Whether the secrets already in this checkout's env layers open under the + * key readers would use (read-only {@link inspectProjectEnvLayerKeys}: + * counts only, never a value). Secrets with no key available, or that do + * not all open, block; a layer that cannot be inspected is incomplete. + */ +async function envSecretsFinding(opts: { + readonly root: string; + readonly projectDir: string; +}): Promise { + const inspected = await inspectProjectEnvLayerKeys({ + projectRoot: opts.root, + projectDir: opts.projectDir, + }); + const args = ["doctor", "--path", opts.root]; + const missing = inspected.reports.filter((r) => r.openCount === null); + const mismatched = inspected.reports.filter( + (r) => r.openCount !== null && r.openCount < r.secretCount + ); + const count = (reports: typeof inspected.reports) => + reports.reduce((sum, r) => sum + r.secretCount, 0); + if (missing.length > 0) { + return { + id: "env-secrets", + status: "blocker", + summary: `${count(missing)} env layer secret(s) are stored, but no project env key is available here, so they cannot be read or migrated.`, + args, + action: + "Copy .hack.secret.key (mode 600, untracked) from a checkout that has it, or set HACK_ENV_SECRET_KEY to the same key. A new key cannot open these values.", + }; + } + if (mismatched.length > 0) { + return { + id: "env-secrets", + status: "blocker", + summary: `Some env layer secrets do not open under this checkout's project env key (${mismatched.length} layer(s)).`, + args, + action: null, + }; + } + if (inspected.failures.length > 0) { + return { + id: "env-secrets", + status: "incomplete", + summary: `${inspected.failures.length} env layer(s) could not be inspected, so whether their secrets open is unknown.`, + args, + action: null, + }; + } + const total = count(inspected.reports); + return { + id: "env-secrets", + status: "ok", + summary: + total === 0 + ? "No secrets in env layers." + : `All ${total} env layer secret(s) open under the project env key.`, + args: null, + action: null, + }; +} + +/** + * The encrypted store this checkout selects for itself, when it is not the + * global one (the machine check covers that) and the checkout reads it: + * the key its configuration selects must open it. Read-only and + * value-free ({@link inspectStoreKeyState}); the keychain is never asked. + */ +async function projectStoreFinding(opts: { + readonly root: string; + readonly projectDir: string; +}): Promise { + const [own, global] = await Promise.all([ + resolveFileSecretStorePaths({ projectDir: opts.projectDir }), + resolveFileSecretStorePaths({}), + ]); + if ( + !own.active || + (global.active && (await sameFile(own.storePath, global.storePath))) || + !(await projectUsesSecretStore({ projectDir: opts.projectDir })) + ) { + return null; + } + const state = await inspectStoreKeyState({ + storePath: own.storePath, + keyPath: own.keyPath, + env: process.env, + }); + const base = { id: "project-store" } as const; + if (state.kind === "no-store" || state.kind === "opens") { + return { + ...base, + status: "ok", + summary: + state.kind === "no-store" + ? "Its own secret store file does not exist (nothing stored)." + : "Its own secret store opens with the key it selects.", + args: null, + action: null, + }; + } + if (state.kind === "store-unreadable" || state.kind === "key-uninspected") { + return { + ...base, + status: "incomplete", + summary: + state.kind === "store-unreadable" + ? "Its own secret store file cannot be read, so whether its secrets can be read is unknown." + : "Its own secret store's key file is missing; the OS keychain may hold the key, but it was not asked (asking can prompt).", + args: ["doctor", "--path", opts.root], + action: null, + }; + } + return { + ...base, + status: "blocker", + summary: + state.kind === "key-refused" + ? "Its own secret store's key is refused at runtime (a link, readable by others, or tracked by Git), so its secrets cannot be read or migrated." + : "Its own secret store does not open with the key it selects, so its secrets cannot be read or migrated.", + args: ["doctor", "--path", opts.root], + action: + state.kind === "key-refused" + ? state.message + : "Restore the store's key file (or set HACK_SECRETS_FILE_KEY), or retire the store with hack secrets retire-store --path --allow-unchecked.", + }; +} + +/** Whether two paths name the same file (resolved; same path if neither exists). */ +async function sameFile(a: string, b: string): Promise { + const [x, y] = await Promise.all([ + stat(a).catch(() => null), + stat(b).catch(() => null), + ]); + if (x !== null && y !== null) { + return x.dev === y.dev && x.ino === y.ino; + } + return resolve(a) === resolve(b); +} + +/** + * Runs one check. A check that threw or ran out of time is reported, never + * a pass: `incomplete` for a required check (readiness is then not + * established), `warn` for an optional one. It gets at most `timeoutMs`, + * and never runs past `deadline` (epoch ms). + */ +async function guarded( + id: string, + limits: { + readonly timeoutMs: number; + readonly deadline: number; + readonly root: string; + }, + run: (timeoutMs: number) => Promise +): Promise { + const notChecked = (summary: string): UpgradeFinding => ({ + id, + status: REQUIRED_CHECKS.has(id) ? "incomplete" : "warn", + summary, + args: ["doctor", "--path", limits.root], + action: null, + }); + const timeoutMs = Math.min(limits.timeoutMs, limits.deadline - Date.now()); + if (timeoutMs <= 0) { + return notChecked( + "Not checked: the upgrade check ran out of time before this check." + ); + } + let timer: ReturnType | undefined; + const timeout = new Promise<"timeout">((done) => { + timer = setTimeout(() => done("timeout"), timeoutMs); + }); + try { + // Every child process the check starts ends by the check's own limit. + const result = await Promise.race([ + withProcessDeadline(Date.now() + timeoutMs, () => run(timeoutMs)), + timeout, + ]); + return result === "timeout" + ? notChecked( + `Not checked: it took longer than ${Math.max(1, Math.round(timeoutMs / 1000))} s.` + ) + : result; + } catch { + return notChecked("Not checked: the check failed."); + } finally { + clearTimeout(timer); + } +} + +const PROJECT_CHECK_TIMEOUT_MS = 8000; + +/** The findings for one live checkout, in a fixed order. */ +export async function inspectUpgradeProject(opts: { + readonly project: RegisteredCheckout; + readonly homeDir: string; + readonly nativePreflight: (opts: { + readonly projectRoot: string; + }) => Promise; + /** Epoch ms after which no further check starts. */ + readonly deadline: number; + /** Per-check limit; defaults to 8 s. */ + readonly checkTimeoutMs?: number; +}): Promise { + const root = opts.project.repoRoot; + const projectDir = opts.project.projectDir; + const limits = { + timeoutMs: opts.checkTimeoutMs ?? PROJECT_CHECK_TIMEOUT_MS, + deadline: opts.deadline, + root, + }; + const results = await Promise.all([ + guarded("env-format", limits, async () => + envFormatFinding({ + detected: await detectEnvFormats({ projectDir }), + root, + }) + ), + guarded("env-key", limits, () => envKeyFinding({ root })), + guarded("tracked-files", limits, (timeoutMs) => + trackedFilesFinding({ + root, + projectDirName: basename(projectDir), + timeoutMs, + }) + ), + guarded("ignore-rules", limits, () => + ignoreRulesFinding({ root, projectDir }) + ), + guarded("agent-integrations", limits, () => + agentFinding({ root, homeDir: opts.homeDir }) + ), + guarded("env-secrets", limits, () => + envSecretsFinding({ root, projectDir }) + ), + guarded("project-store", limits, () => + projectStoreFinding({ root, projectDir }) + ), + guarded("native-preflight", limits, async () => + nativeFinding(await opts.nativePreflight({ projectRoot: root })) + ), + ]); + return results.filter((finding) => finding !== null); +} + +/** Whether `path` or a directory above it holds a `.git` entry. */ +async function insideGitWorkTree(path: string): Promise { + let current = resolve(path); + for (;;) { + if ((await statState(resolve(current, ".git"))) !== "absent") { + return true; + } + const parent = resolve(current, ".."); + if (parent === current) { + return false; + } + current = parent; + } +} + +async function statState( + path: string +): Promise<"directory" | "other" | "absent" | "error"> { + try { + return (await stat(path)).isDirectory() ? "directory" : "other"; + } catch (error: unknown) { + return errorCode(error) === "ENOENT" || errorCode(error) === "ENOTDIR" + ? "absent" + : "error"; + } +} + +function errorCode(error: unknown): string | undefined { + if (typeof error !== "object" || error === null || !("code" in error)) { + return; + } + const { code } = error; + return typeof code === "string" ? code : undefined; +} diff --git a/src/lib/upgrade-check.ts b/src/lib/upgrade-check.ts new file mode 100644 index 000000000..8dbc23cf5 --- /dev/null +++ b/src/lib/upgrade-check.ts @@ -0,0 +1,1243 @@ +import { join, resolve } from "node:path"; + +import { isCandidateRun } from "../daemon/client.ts"; +import { quoteShellWord } from "./logs.ts"; +import { withProcessDeadline } from "./process-deadline.ts"; +import { + readProjectsRegistryStrict, + resolveProjectsRegistryPath, +} from "./projects-registry.ts"; +import { + inspectStoreKeyState, + listRetiredEncryptedStores, + resolveFileSecretStorePaths, +} from "./secret-store.ts"; +import { + listSecretStoreReaders, + secretStoreSelectionProblem, +} from "./secret-store-retire.ts"; +import type { UpgradeCheckProbes } from "./upgrade-check-probes.ts"; +import { + classifyRegistryEntry, + inspectUpgradeProject, + type RegisteredCheckout, + type RegistryEntryState, + registeredCheckouts, + type UpgradeFinding, + type UpgradeFindingStatus, +} from "./upgrade-check-project.ts"; + +export type { + UpgradeFinding, + UpgradeFindingStatus, +} from "./upgrade-check-project.ts"; + +/** + * `hack upgrade check` (HACK-2189, the upgrade assistant HACK-2188): a + * read-only report on the machine and every registered checkout, ending in + * an ordered plan and the next command. + * + * Contract: + * - Changes nothing: no file, registry, service, container or key is + * written, started or provisioned, and nothing prompts (the OS keychain is + * never asked). Host probes are injected ({@link UpgradeCheckProbes}). + * - One deadline covers the whole check (registry discovery, classification, + * probes and checkout checks); what it cuts short is `incomplete`. + * - No value in the report: names, paths, counts, states and codes only. + * - Dead registry entries (the checkout or its config is gone) are counted + * apart from live entries that cannot be read. + * - `ready` only when nothing blocks and every required check ran: a + * `blocker` (continuing would fail or lose data) or an `incomplete` check + * (safety not established) makes it false, and the command exit non-zero. + * - A plan step carries either an executable command (built from the + * running CLI and quoted arguments, never rewritten text) or a manual + * action, never prose labelled as a command. + */ + +export const UPGRADE_CHECK_REPORT_VERSION = 1; + +/** How printed commands run the CLI that produced the report. */ +export type UpgradeCommandPrefix = { + /** Environment assignments placed before the command. */ + readonly env: Readonly>; + /** The executable (and any fixed leading arguments). */ + readonly argv: readonly string[]; +}; + +/** + * During the prerelease, v5 runs through a candidate launcher while stable + * v4 stays `hack` on PATH, so a bare `hack` would run the old CLI. The + * `hack-next` launcher lives at `~/.hack-next/bin/hack-next` (not on PATH); + * a Compose candidate run also needs `HACK_RUNTIME_BACKEND=compose`. + */ +export function upgradeCommandPrefix(opts: { + readonly env: Readonly>; + readonly homeDir: string; +}): UpgradeCommandPrefix { + const entry = (opts.env.HACK_CANDIDATE_ENTRYPOINT ?? "").trim(); + if (entry === "") { + return { env: {}, argv: ["hack"] }; + } + const launcher = + entry === "hack-next" + ? join(opts.homeDir, ".hack-next", "bin", "hack-next") + : entry; + return { + env: + (opts.env.HACK_RUNTIME_BACKEND ?? "").trim() === "compose" + ? { HACK_RUNTIME_BACKEND: "compose" } + : {}, + argv: [launcher], + }; +} + +/** A shell command line: every word quoted on its own. */ +export function renderUpgradeCommand( + prefix: UpgradeCommandPrefix, + args: readonly string[] +): string { + return [ + ...Object.entries(prefix.env).map( + ([key, value]) => `${key}=${quoteShellWord(value)}` + ), + ...[...prefix.argv, ...args].map((word) => quoteShellWord(word)), + ].join(" "); +} + +/** A finding as reported: its command rendered for the running CLI. */ +export type UpgradeReportFinding = UpgradeFinding & { + /** The command line for `args`; null when only `action` applies. */ + readonly command: string | null; +}; + +export type UpgradePlanStep = { + readonly order: number; + /** The finding this step resolves, or `snapshot`. */ + readonly id: string; + /** `machine`, or the checkout's registered name. */ + readonly scope: string; + readonly status: UpgradeFindingStatus | "step"; + /** An executable command line; null for a manual step. */ + readonly command: string | null; + /** Its Hack CLI arguments (after `cli.command`); null for a manual step. */ + readonly args: readonly string[] | null; + /** A manual action: what to do when there is no command, or instead of it. */ + readonly action: string | null; + readonly why: string; + readonly blocker: boolean; +}; + +export type UpgradeProjectReport = { + readonly name: string; + readonly root: string; + readonly state: "live" | "unreadable"; + readonly reason: string | null; + readonly findings: readonly UpgradeReportFinding[]; +}; + +type ScopedFinding = { + readonly scope: string; + readonly id: string; + readonly summary: string; +}; + +export type UpgradeCheckReport = { + readonly report_version: typeof UPGRADE_CHECK_REPORT_VERSION; + readonly generated_at: string; + readonly duration_ms: number; + readonly cli: { + readonly version: string; + readonly channel: "stable" | "next"; + readonly candidate_run: boolean; + /** How printed commands run this CLI, as a command-line prefix. */ + readonly command: string; + }; + /** `disabled` when HACK_HOST_PROBES_DISABLE=1 refused every host probe. */ + readonly host_probes: "enabled" | "disabled"; + readonly machine: readonly UpgradeReportFinding[]; + readonly registry: { + readonly state: "valid" | "absent" | "invalid" | "unreadable"; + /** Registered projects. */ + readonly projects: number; + /** Checkouts: each project's primary checkout and linked worktrees. */ + readonly total: number; + readonly live: number; + /** Dead entries: the checkout or its config is gone. */ + readonly missing: readonly RegistryEntrySummary[]; + /** Entries that exist but cannot be read. */ + readonly unreadable: readonly RegistryEntrySummary[]; + /** Entries the deadline cut short. */ + readonly unchecked: readonly RegistryEntrySummary[]; + /** Registry records that are not valid project entries. */ + readonly rejected: number; + }; + readonly projects: readonly UpgradeProjectReport[]; + readonly blockers: readonly ScopedFinding[]; + /** Required checks that could not be performed. */ + readonly incomplete: readonly ScopedFinding[]; + readonly plan: readonly UpgradePlanStep[]; + /** The first plan step's command; null when it is manual or there is nothing to do. */ + readonly next_command: string | null; + /** The first plan step's manual action when it has no command. */ + readonly next_action: string | null; + readonly ready: boolean; +}; + +type RegistryEntrySummary = { + readonly name: string; + readonly root: string; + readonly reason: string; +}; + +const PROJECT_CONCURRENCY = 8; +const CLASSIFY_CONCURRENCY = 16; +const PROBE_TIMEOUT_MS = 12_000; +/** The whole check's budget: nothing starts after it, so it ends under 30 s. */ +const CHECK_BUDGET_MS = 25_000; + +/** Runs every check and builds the plan. */ +export async function runUpgradeCheck(opts: { + readonly cliVersion: string; + readonly homeDir: string; + readonly probes: UpgradeCheckProbes; + readonly env?: Readonly>; + readonly now?: () => number; + /** Test seams: the whole budget and the per-check limit. */ + readonly budgetMs?: number; + readonly checkTimeoutMs?: number; +}): Promise { + const now = opts.now ?? Date.now; + const started = now(); + const deadline = Date.now() + (opts.budgetMs ?? CHECK_BUDGET_MS); + const env = opts.env ?? process.env; + const prefix = upgradeCommandPrefix({ env, homeDir: opts.homeDir }); + const render = (finding: UpgradeFinding): UpgradeReportFinding => ({ + ...finding, + command: + finding.args === null ? null : renderUpgradeCommand(prefix, finding.args), + }); + const [registry, machine] = await Promise.all([ + inspectRegistry({ deadline }), + inspectMachine({ + probes: opts.probes, + cliVersion: opts.cliVersion, + env, + deadline, + }), + ]); + const projects = await mapLimit( + registry.live, + PROJECT_CONCURRENCY, + async (project): Promise => ({ + name: project.name, + root: project.repoRoot, + state: "live", + reason: null, + findings: ( + await inspectUpgradeProject({ + project, + homeDir: opts.homeDir, + nativePreflight: opts.probes.nativePreflight, + deadline, + checkTimeoutMs: opts.checkTimeoutMs, + }) + ).map(render), + }) + ); + const unreadableProjects: UpgradeProjectReport[] = + registry.report.unreadable.map((entry) => ({ + name: entry.name, + root: entry.root, + state: "unreadable", + reason: entry.reason, + findings: [ + render({ + id: "project-config", + status: "blocker", + summary: `This live checkout cannot be read (${entry.reason}), so it cannot be upgraded safely.`, + args: ["doctor", "--path", entry.root], + action: + "Fix its permissions or its config file, then rerun the upgrade check.", + }), + ], + })); + const allProjects = [...projects, ...unreadableProjects].sort((a, b) => + a.name.localeCompare(b.name) + ); + const machineFindings = [...machine, ...registry.findings].map(render); + const scoped = (status: UpgradeFindingStatus): ScopedFinding[] => [ + ...machineFindings + .filter((finding) => finding.status === status) + .map((finding) => ({ + scope: "machine", + id: finding.id, + summary: finding.summary, + })), + ...allProjects.flatMap((project) => + project.findings + .filter((finding) => finding.status === status) + .map((finding) => ({ + scope: project.name, + id: finding.id, + summary: finding.summary, + })) + ), + ]; + const blockers = scoped("blocker"); + const incomplete = scoped("incomplete"); + const plan = buildPlan({ + machine: machineFindings, + projects: allProjects, + prefix, + }); + const first = plan[0]; + const version = opts.cliVersion; + return { + report_version: UPGRADE_CHECK_REPORT_VERSION, + generated_at: new Date(started).toISOString(), + duration_ms: Math.max(0, now() - started), + cli: { + version, + channel: version.includes("-next.") ? "next" : "stable", + candidate_run: isCandidateRun(env), + command: renderUpgradeCommand(prefix, []), + }, + host_probes: opts.probes.disabled === true ? "disabled" : "enabled", + machine: machineFindings, + registry: registry.report, + projects: allProjects, + blockers, + incomplete, + plan, + next_command: first?.command ?? null, + next_action: + first !== undefined && first.command === null ? first.action : null, + ready: blockers.length === 0 && incomplete.length === 0, + }; +} + +// --------------------------------------------------------------------------- +// Registry +// --------------------------------------------------------------------------- + +async function inspectRegistry(opts: { readonly deadline: number }): Promise<{ + readonly report: UpgradeCheckReport["registry"]; + readonly live: readonly RegisteredCheckout[]; + readonly findings: readonly UpgradeFinding[]; +}> { + const registryPath = resolveProjectsRegistryPath(); + const empty = { + projects: 0, + total: 0, + live: 0, + missing: [], + unreadable: [], + unchecked: [], + rejected: 0, + } as const; + const read = await withDeadline( + () => readProjectsRegistryStrict().catch(() => "error" as const), + opts.deadline + ); + if (read === "timeout" || read === "error" || read.kind !== "valid") { + return { + report: { state: registryFailureState(read), ...empty }, + live: [], + findings: [registryReadFinding({ read, registryPath })], + }; + } + const entries = read.registry.projects; + // Every registered checkout: each project's primary and linked worktrees. + const checkouts = entries.flatMap(registeredCheckouts); + const states = await mapLimit( + checkouts, + CLASSIFY_CONCURRENCY, + async (checkout) => { + const state = await withDeadline( + () => + classifyRegistryEntry(checkout).catch( + (): RegistryEntryState => ({ + state: "unreadable", + reason: "it could not be read", + }) + ), + opts.deadline + ); + return { checkout, state }; + } + ); + const summary = (checkout: RegisteredCheckout, reason: string) => ({ + name: checkout.name, + root: checkout.repoRoot, + reason, + }); + const live: RegisteredCheckout[] = []; + const missing: RegistryEntrySummary[] = []; + const unreadable: RegistryEntrySummary[] = []; + const unchecked: RegistryEntrySummary[] = []; + for (const { checkout, state } of states) { + if (state === "timeout") { + unchecked.push(summary(checkout, "not read before the check's deadline")); + } else if (state.state === "live") { + live.push(checkout); + } else if (state.state === "missing") { + missing.push(summary(checkout, state.reason)); + } else { + unreadable.push(summary(checkout, state.reason)); + } + } + const findings = registryFindings({ + projects: entries.length, + checkouts: checkouts.length, + live: live.length, + missing: missing.length, + unchecked: unchecked.length, + unreadable: unreadable.length, + rejected: read.rejected, + }); + return { + report: { + state: "valid", + projects: entries.length, + total: checkouts.length, + live: live.length, + missing, + unreadable, + unchecked, + rejected: read.rejected, + }, + live, + findings, + }; +} + +/** The registry's own findings: its counts, dead entries, cut-short and rejected records. */ +function registryFindings(counts: { + readonly projects: number; + readonly checkouts: number; + readonly live: number; + readonly missing: number; + readonly unchecked: number; + readonly unreadable: number; + readonly rejected: number; +}): UpgradeFinding[] { + const findings: UpgradeFinding[] = [ + { + id: "registry", + status: "ok", + summary: `${counts.projects} registered project(s), ${counts.checkouts} checkout(s) with worktrees: ${counts.live} live, ${counts.missing} dead, ${counts.unreadable} unreadable.`, + args: null, + action: null, + }, + ]; + if (counts.missing > 0) { + findings.push({ + id: "registry-dead", + status: "warn", + summary: `${counts.missing} registry entr${counts.missing === 1 ? "y points" : "ies point"} at a checkout or config that is gone. They are not live projects that failed to read; pruning them only edits the registry.`, + args: ["projects", "prune", "--dry-run"], + action: null, + }); + } + if (counts.unchecked > 0) { + findings.push({ + id: "registry-unchecked", + status: "incomplete", + summary: `${counts.unchecked} registered checkout(s) could not be read before the check's deadline (a slow or unavailable filesystem?); they were not checked.`, + args: ["upgrade", "check"], + action: null, + }); + } + if (counts.rejected > 0) { + findings.push({ + id: "registry-rejected", + status: "warn", + summary: `${counts.rejected} registry record(s) are not valid project entries and were skipped.`, + args: ["projects", "prune", "--dry-run"], + action: null, + }); + } + return findings; +} + +type RegistryRead = + | "timeout" + | "error" + | { readonly kind: "absent" | "invalid" }; + +function registryFailureState( + read: RegistryRead +): "unreadable" | "invalid" | "absent" { + if (read === "timeout" || read === "error") { + return "unreadable"; + } + return read.kind; +} + +/** The one finding for a registry that is absent, invalid, unreadable or slow. */ +function registryReadFinding(opts: { + readonly read: RegistryRead; + readonly registryPath: string; +}): UpgradeFinding { + const { read, registryPath } = opts; + if (read === "timeout" || read === "error") { + return { + id: "registry", + status: read === "timeout" ? "incomplete" : "blocker", + summary: + read === "timeout" + ? "The projects registry could not be read in time, so the projects to upgrade are unknown." + : "The projects registry exists but cannot be read, so the projects to upgrade are unknown.", + args: null, + action: `Check that ${registryPath} is readable, then rerun the upgrade check.`, + }; + } + if (read.kind === "invalid") { + return { + id: "registry", + status: "blocker", + summary: + "The projects registry is not valid JSON or not a registry, so the projects to upgrade are unknown.", + args: null, + action: `Repair ${registryPath}, or move it aside and run hack once in each project to register it again; then rerun the upgrade check.`, + }; + } + return { + id: "registry", + status: "ok", + summary: "No projects are registered.", + args: null, + action: null, + }; +} + +// --------------------------------------------------------------------------- +// Machine +// --------------------------------------------------------------------------- + +async function inspectMachine(opts: { + readonly probes: UpgradeCheckProbes; + readonly cliVersion: string; + readonly env: Readonly>; + readonly deadline: number; +}): Promise { + const { probes, deadline } = opts; + const docker = await probe(probes.docker, "unreachable" as const, deadline); + const [pathHack, daemon, infra, linuxDns, store] = await Promise.all([ + probe(probes.pathHack, null, deadline), + probe(probes.daemon, null, deadline), + probe(() => probes.globalInfra({ docker }), null, deadline), + probe(probes.linuxDns, undefined, deadline), + probe(() => inspectGlobalSecretStore({ env: opts.env }), null, deadline), + ]); + // Skipped while global services do not run; an attempted probe that + // fails or runs out of time is reported, never omitted. + const proxy = + infra?.running === true + ? await probe(probes.proxyPorts, "failed" as const, deadline) + : ("skipped" as const); + const findings: UpgradeFinding[] = []; + if (probes.disabled === true) { + findings.push({ + id: "host-probes", + status: "incomplete", + summary: + "Host probes are disabled (HACK_HOST_PROBES_DISABLE=1): Docker, the daemon service, global services, proxy ports and DNS were not checked.", + args: null, + action: "Unset HACK_HOST_PROBES_DISABLE, then rerun the upgrade check.", + }); + } + findings.push( + pathHackFinding({ probe: pathHack, cliVersion: opts.cliVersion }) + ); + // A probe that failed (not one disabled on purpose, which is reported + // once above) leaves its check unperformed: incomplete. + const unperformed: UpgradeFindingStatus = + probes.disabled === true ? "warn" : "incomplete"; + findings.push(daemonFinding(daemon, unperformed)); + findings.push(...infraFindings({ docker, infra, proxy, unperformed })); + if (linuxDns !== null) { + // Before `hack global install` runs, it sets the route up too. + findings.push( + infra !== null && !infra.installed + ? { + id: "linux-dns", + status: "info", + summary: + "The Linux DNS route is set up by hack global install (listed above).", + args: null, + action: null, + } + : linuxDnsFinding(linuxDns, unperformed) + ); + } + findings.push( + store ?? { + id: "secret-store", + status: "incomplete", + summary: + "Not checked: the global secret store check failed or ran out of time.", + args: ["doctor"], + action: null, + } + ); + return findings; +} + +function pathHackFinding(opts: { + readonly probe: Awaited> | null; + readonly cliVersion: string; +}): UpgradeFinding { + const found = opts.probe; + let summary: string; + if (found === null) { + summary = `Running ${opts.cliVersion}; the hack on PATH could not be checked.`; + } else if (found.path === null) { + summary = `Running ${opts.cliVersion}; no hack is on PATH.`; + } else { + summary = `Running ${opts.cliVersion}; the hack on PATH is ${found.version ?? "an unknown version"} (${found.install}) at ${found.path}.`; + } + return { id: "installed", status: "info", summary, args: null, action: null }; +} + +function daemonFinding( + daemon: Awaited> | null, + unperformed: UpgradeFindingStatus +): UpgradeFinding { + if (daemon === null) { + return { + id: "daemon", + status: unperformed, + summary: "Not checked: the daemon service could not be read.", + args: ["daemon", "status"], + action: null, + }; + } + const step = daemon.nextStep; + const hackStep = step?.startsWith("hack ") === true; + const unmanaged = daemon.running + ? "running, started directly" + : "not running"; + const owner = + daemon.manager === null + ? unmanaged + : `${daemon.manager}: ${daemon.summary}`; + return { + id: "daemon", + status: daemon.severity === "warn" ? "warn" : "info", + summary: `hackd ${daemon.running ? "is running" : "is not running"}${daemon.managed ? `, run by ${daemon.manager}` : ""}; login service ${daemon.registered ? "registered" : "not registered"} (${owner}).`, + args: hackStep ? (step?.slice("hack ".length).split(" ") ?? null) : null, + action: hackStep ? null : step, + }; +} + +function infraFindings(opts: { + readonly unperformed: UpgradeFindingStatus; + readonly docker: "reachable" | "unreachable" | "missing"; + readonly infra: Awaited> | null; + readonly proxy: + | Awaited> + | "failed" + | "skipped"; +}): UpgradeFinding[] { + const findings: UpgradeFinding[] = []; + const docker = dockerFinding(opts.docker); + if (docker !== null) { + findings.push(docker); + } + findings.push(globalInfraFinding(opts.infra, opts.unperformed)); + if (opts.proxy === "failed") { + findings.push({ + id: "proxy-ports", + status: opts.unperformed, + summary: "Not checked: the proxy ports probe failed or ran out of time.", + args: ["doctor"], + action: null, + }); + } else if (opts.proxy !== "skipped") { + findings.push({ + id: "proxy-ports", + status: opts.proxy.status === "ok" ? "ok" : "warn", + summary: opts.proxy.message, + args: opts.proxy.status === "ok" ? null : ["doctor"], + action: null, + }); + } + return findings; +} + +function dockerFinding( + docker: "reachable" | "unreachable" | "missing" +): UpgradeFinding | null { + if (docker === "reachable") { + return null; + } + return docker === "missing" + ? { + id: "docker", + status: "warn", + summary: "Docker is not installed; the Compose backend needs it.", + args: null, + action: "Install Docker Engine (or Docker Desktop on macOS).", + } + : { + id: "docker", + status: "warn", + summary: + "Docker does not answer; global services and proxy ports were not checked.", + args: null, + action: "Start Docker, then rerun the upgrade check.", + }; +} + +function globalInfraFinding( + infra: Awaited> | null, + unperformed: UpgradeFindingStatus +): UpgradeFinding { + const base = { id: "global-infra" } as const; + if (infra === null) { + return { + ...base, + status: unperformed, + summary: "Not checked: Hack's global services could not be read.", + args: ["global", "install"], + action: null, + }; + } + if (!infra.installed) { + return { + ...base, + status: "warn", + summary: "Hack's global proxy and DNS are not installed on this machine.", + args: ["global", "install"], + action: null, + }; + } + if (infra.running === false) { + return { + ...base, + status: "warn", + summary: "Hack's global proxy and DNS are installed but not running.", + args: ["global", "up"], + action: null, + }; + } + if (infra.running === null && infra.listFailed === true) { + // Docker answered but the containers could not be listed: neither they + // nor the proxy ports were checked. + return { + ...base, + status: unperformed, + summary: + "Not checked: listing Hack's global containers failed or ran out of time, so neither they nor the proxy ports were checked.", + args: ["global", "status"], + action: null, + }; + } + return { + ...base, + status: infra.running === null ? "info" : "ok", + summary: + infra.running === null + ? "Hack's global proxy is installed; whether it runs was not checked." + : "Hack's global proxy and DNS are installed and running.", + args: null, + action: null, + }; +} + +function linuxDnsFinding( + inspection: Awaited> | undefined, + unperformed: UpgradeFindingStatus +): UpgradeFinding { + if (inspection === undefined || inspection === null) { + return { + id: "linux-dns", + status: unperformed, + summary: "Not checked: the Linux DNS route could not be read.", + args: ["doctor"], + action: null, + }; + } + return inspection.state === "configured" + ? { + id: "linux-dns", + status: "ok", + summary: "The Linux DNS route for Hack's names is configured.", + args: null, + action: null, + } + : { + id: "linux-dns", + status: "warn", + summary: `Linux DNS route: ${inspection.state}${inspection.code ? ` (${inspection.code})` : ""}.`, + args: ["global", "install"], + action: inspection.fix[0] ?? null, + }; +} + +/** + * The global encrypted secret store and the key the runtime would use, read + * only ({@link inspectStoreKeyState}): never the keychain, never a + * provisioned key. A store whose key is known to be gone or wrong blocks + * only when a registered project reads it (the #779 rule). + */ +async function inspectGlobalSecretStore(opts: { + readonly env: Readonly>; +}): Promise { + const base = { id: "secret-store" } as const; + if ((await secretStoreSelectionProblem({})) !== null) { + return { + ...base, + status: "incomplete", + summary: + "The global config that selects the secret store cannot be read; the store was not checked.", + args: ["doctor"], + action: null, + }; + } + const location = await resolveFileSecretStorePaths({}); + if (!location.active) { + return { + ...base, + status: "ok", + summary: "Secrets use the OS keychain; no file store to check.", + args: null, + action: null, + }; + } + const retired = (await listRetiredEncryptedStores(location.storePath)).length; + const retiredNote = + retired > 0 + ? ` ${retired} retired store archive(s) are kept beside it.` + : ""; + const state = await inspectStoreKeyState({ + storePath: location.storePath, + keyPath: location.keyPath, + env: opts.env, + }); + if (state.kind === "no-store") { + return { + ...base, + status: "ok", + summary: `No global secret store file (nothing is stored in it).${retiredNote}`, + args: null, + action: null, + }; + } + if (state.kind === "opens") { + // A reader may name its own key file: it must open the store too. + return storeOpensFinding({ + source: state.source, + readers: await readerKeyStates({ + storePath: location.storePath, + globalKeyPath: location.keyPath, + globalState: state, + env: opts.env, + }), + retiredNote, + }); + } + if (state.kind === "store-unreadable") { + return { + ...base, + status: "incomplete", + summary: + "The global secret store file cannot be read, so its state is unknown.", + args: ["doctor"], + action: null, + }; + } + if (state.kind === "key-uninspected") { + return { + ...base, + status: "incomplete", + summary: + "The global secret store's key file is missing; the OS keychain may hold its key, but it was not asked (asking can prompt).", + args: ["doctor"], + action: + "hack doctor asks the OS keychain and reports whether the key is there.", + }; + } + // Each reader's own key decides whether it is stranded: a project may + // share this store but name a key file of its own that works. + const readers = await readerKeyStates({ + storePath: location.storePath, + globalKeyPath: location.keyPath, + globalState: state, + env: opts.env, + }); + return storeFailureFinding({ state, readers, retiredNote }); +} + +/** The finding for a global store that opens with its own key, given each reader's key. */ +function storeOpensFinding(opts: { + readonly source: "env" | "file"; + readonly readers: ReaderKeyStates; + readonly retiredNote: string; +}): UpgradeFinding { + const { readers } = opts; + if (readers.failing.length > 0) { + return { + id: "secret-store", + status: "blocker", + summary: `The global secret store opens with ${opts.source === "env" ? "HACK_SECRETS_FILE_KEY" : "its key file"}, but not with the key ${readers.failing.join(", ")} ${readers.failing.length === 1 ? "selects" : "select"} (their own encryptedFile.keyPath).`, + args: ["doctor"], + action: `Give ${readers.failing.join(", ")} the key that opens the store, or remove their own keyPath to use the global key.`, + }; + } + if (readers.uncertain.length > 0 || !readers.scanComplete) { + return uncertainReadersFinding(readers); + } + const every = + readers.opening.length > 0 + ? `, and with the key of every project that reads it (${readers.opening.join(", ")})` + : ""; + return { + id: "secret-store", + status: "ok", + summary: `The global secret store opens with ${opts.source === "env" ? "HACK_SECRETS_FILE_KEY from this shell" : "its key file"}${every}.${opts.retiredNote}`, + args: null, + action: null, + }; +} + +/** + * Some readers' keys fail while another configured key opens the store, or + * other readers' keys are unknown: fix their keys; retiring is refused (or + * not justified) then. + */ +function misconfiguredReadersFinding(readers: ReaderKeyStates): UpgradeFinding { + const others = + readers.opening.length > 0 + ? `, though ${readers.opening.join(", ")} open${readers.opening.length === 1 ? "s" : ""} it with another key` + : ", and other readers' keys could not be established"; + return { + id: "secret-store", + status: "blocker", + summary: `The global secret store does not open with the key${readers.failing.length === 1 ? "" : "s"} of ${readers.failing.join(", ")}${others}.`, + args: ["doctor"], + action: + readers.opening.length > 0 + ? `Give ${readers.failing.join(", ")} the key that opens the store (the keyPath ${readers.opening.join(", ")} use${readers.opening.length === 1 ? "s" : ""}), or HACK_SECRETS_FILE_KEY.` + : "Run hack doctor to establish every reader's key before retiring or replacing the store.", + }; +} + +/** The finding for a global store whose own key does not open it, given each reader's key. */ +function storeFailureFinding(opts: { + readonly state: Exclude< + Awaited>, + { kind: "no-store" | "opens" | "store-unreadable" | "key-uninspected" } + >; + readonly readers: ReaderKeyStates; + readonly retiredNote: string; +}): UpgradeFinding { + const { state, readers, retiredNote } = opts; + const base = { id: "secret-store" } as const; + const uncertain = readers.uncertain.length > 0 || !readers.scanComplete; + if (readers.failing.length === 0 && uncertain) { + // Not proven lost for anyone: never suggest retiring on uncertainty. + return uncertainReadersFinding(readers); + } + if (readers.failing.length === 0 && readers.opening.length > 0) { + return { + ...base, + status: "warn", + summary: `The global secret store's own key does not open it, but every registered project that reads it (${readers.opening.join(", ")}) opens it with its own key.${retiredNote}`, + args: ["doctor"], + action: null, + }; + } + const usedBy = + readers.failing.length > 0 + ? ` Read by: ${readers.failing.join(", ")}.` + : " No registered project reads it."; + const blocking: UpgradeFindingStatus = + readers.failing.length > 0 ? "blocker" : "warn"; + if (readers.opening.length > 0 || uncertain) { + // Retiring is refused while another configured key opens the store, + // and not suggested while any reader's key is unknown. Here at least + // one reader's key is known to fail. + return misconfiguredReadersFinding(readers); + } + if (state.kind === "key-refused") { + return { + ...base, + status: blocking, + summary: `The global secret store's key is refused at runtime (a link, readable by others, or tracked by Git).${usedBy}`, + args: ["doctor"], + action: state.message, + }; + } + if (state.kind === "wrong-key" && state.source === "env") { + return { + ...base, + status: blocking, + summary: `HACK_SECRETS_FILE_KEY in this shell does not open the global secret store.${usedBy}`, + args: null, + action: + "Unset HACK_SECRETS_FILE_KEY or set it to the store's key, then rerun the upgrade check.", + }; + } + return { + ...base, + status: blocking, + summary: `The global secret store ${state.kind === "key-missing" ? "has no key file (and the keychain fallback is off)" : "does not open with its key file"}.${usedBy}${retiredNote}`, + args: ["secrets", "retire-store", "--allow-unchecked"], + action: + "Or restore the store's key file (or set HACK_SECRETS_FILE_KEY) instead of retiring it.", + }; +} + +/** + * Splits the store's readers by whether the key their own configuration + * selects (with this shell's HACK_SECRETS_FILE_KEY first, as at runtime) + * opens it. Read-only; never the keychain. + */ +async function readerKeyStates(opts: { + readonly storePath: string; + readonly globalKeyPath: string; + readonly globalState: Awaited>; + readonly env: Readonly>; +}): Promise { + const found = await listSecretStoreReaders({ storePath: opts.storePath }); + const states = new Map< + string, + Promise>> + >([[resolve(opts.globalKeyPath), Promise.resolve(opts.globalState)]]); + const failing = new Set(); + const opening = new Set(); + const uncertain = new Set(); + for (const reader of found.readers) { + const keyPath = resolve(reader.keyPath); + let pending = states.get(keyPath); + if (pending === undefined) { + // One inspection per distinct key file. + pending = inspectStoreKeyState({ + storePath: opts.storePath, + keyPath, + env: opts.env, + }); + states.set(keyPath, pending); + } + const state = await pending; + if (state.kind === "opens") { + opening.add(reader.name); + } else if ( + state.kind === "key-uninspected" || + state.kind === "store-unreadable" + ) { + uncertain.add(reader.name); + } else { + failing.add(reader.name); + } + } + return { + failing: [...failing].sort(), + opening: [...opening].sort(), + uncertain: [...uncertain].sort(), + scanComplete: found.registryRead && found.unchecked === 0, + }; +} + +/** Readers of a store by whether the key each one selects opens it. */ +type ReaderKeyStates = { + readonly failing: string[]; + readonly opening: string[]; + /** Readers whose key could not be established (keychain not asked, unreadable). */ + readonly uncertain: string[]; + /** False when some registered checkouts could not be checked at all. */ + readonly scanComplete: boolean; +}; + +/** The incomplete finding for readers whose key could not be established. */ +function uncertainReadersFinding(readers: ReaderKeyStates): UpgradeFinding { + const who = + readers.uncertain.length > 0 + ? `the key of ${readers.uncertain.join(", ")} could not be established (the OS keychain was not asked, or the key file cannot be read)` + : "some registered checkouts could not be checked"; + return { + id: "secret-store", + status: "incomplete", + summary: `Whether every project that reads the global secret store can open it is unknown: ${who}.`, + args: ["doctor"], + action: null, + }; +} + +// --------------------------------------------------------------------------- +// Plan +// --------------------------------------------------------------------------- + +/** Machine fixes in the order they should run. */ +const MACHINE_ORDER = [ + "host-probes", + "registry", + "registry-unchecked", + "secret-store", + "registry-dead", + "registry-rejected", + "docker", + "global-infra", + "linux-dns", + "proxy-ports", + "daemon", +] as const; + +/** Checkout fixes in the order they should run. */ +const PROJECT_ORDER = [ + "project-config", + "project-store", + "env-secrets", + "env-format", + "env-key", + "tracked-files", + "ignore-rules", + "agent-integrations", +] as const; + +const ACTIONABLE: ReadonlySet = new Set([ + "blocker", + "incomplete", + "warn", +]); + +function buildPlan(opts: { + readonly machine: readonly UpgradeReportFinding[]; + readonly projects: readonly UpgradeProjectReport[]; + readonly prefix: UpgradeCommandPrefix; +}): UpgradePlanStep[] { + type Draft = Omit; + const actionable = (finding: UpgradeReportFinding) => + ACTIONABLE.has(finding.status) && + (finding.args !== null || finding.action !== null); + const rank = + (order: readonly string[]) => (finding: UpgradeReportFinding) => { + const index = order.indexOf(finding.id); + return index === -1 ? order.length : index; + }; + const toStep = + (scope: string) => + (finding: UpgradeReportFinding): Draft => ({ + id: finding.id, + scope, + status: finding.status, + command: finding.command, + args: finding.args, + action: finding.action, + why: finding.summary, + blocker: finding.status === "blocker", + }); + const machineSteps = [...opts.machine] + .filter(actionable) + .sort((a, b) => rank(MACHINE_ORDER)(a) - rank(MACHINE_ORDER)(b)) + .map(toStep("machine")); + const projectSteps = opts.projects.flatMap((project) => + [...project.findings] + .filter(actionable) + .sort((a, b) => rank(PROJECT_ORDER)(a) - rank(PROJECT_ORDER)(b)) + .map(toStep(project.name)) + ); + const steps = [...machineSteps, ...projectSteps]; + const severity = (step: Draft) => { + if (step.status === "blocker") { + return 0; + } + return step.status === "incomplete" ? 1 : 2; + }; + const ordered = steps + .map((step, index) => ({ step, index })) + .sort((a, b) => severity(a.step) - severity(b.step) || a.index - b.index) + .map(({ step }) => step); + // A snapshot first, before any step changes a file, so `hack upgrade + // rollback` can put back what the following steps change. It covers + // files only: not the Git index, services or containers. + const snapshot: Draft[] = + ordered.length > 0 + ? [ + { + id: "snapshot", + scope: "machine", + status: "step", + command: renderUpgradeCommand(opts.prefix, ["upgrade", "snapshot"]), + args: ["upgrade", "snapshot"], + action: null, + why: "Keep a private copy of the files the following steps change, so the upgrade rollback command can put them back.", + blocker: false, + }, + ] + : []; + return [...snapshot, ...ordered].map((step, index) => ({ + order: index + 1, + ...step, + })); +} + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +/** + * `start()`'s result, or `timeout` once `deadline` (epoch ms) passes. Work + * is not started at all when the deadline has already passed. + */ +async function withDeadline( + start: () => Promise, + deadline: number +): Promise { + const remaining = deadline - Date.now(); + if (remaining <= 0) { + return "timeout"; + } + const work = start(); + let timer: ReturnType | undefined; + const timeout = new Promise<"timeout">((done) => { + timer = setTimeout(() => done("timeout"), remaining); + }); + try { + return await Promise.race([work, timeout]); + } finally { + clearTimeout(timer); + } +} + +/** A probe's result, or `fallback` when it fails or outlives its limit. */ +async function probe( + run: () => Promise, + fallback: F, + deadline: number +): Promise { + const limit = Math.min(deadline, Date.now() + PROBE_TIMEOUT_MS); + try { + // Every child process the probe starts ends by the probe's own limit. + const result = await withDeadline( + () => withProcessDeadline(limit, run), + limit + ); + return result === "timeout" ? fallback : result; + } catch { + return fallback; + } +} + +async function mapLimit( + items: readonly T[], + limit: number, + run: (item: T) => Promise +): Promise { + const results: R[] = new Array(items.length); + let next = 0; + const workers = Array.from( + { length: Math.min(limit, items.length) }, + async () => { + while (next < items.length) { + const index = next; + next += 1; + results[index] = await run(items[index] as T); + } + } + ); + await Promise.all(workers); + return results; +} diff --git a/src/lib/upgrade-snapshot-targets.ts b/src/lib/upgrade-snapshot-targets.ts new file mode 100644 index 000000000..48d41f771 --- /dev/null +++ b/src/lib/upgrade-snapshot-targets.ts @@ -0,0 +1,363 @@ +import { lstat, readdir } from "node:fs/promises"; +import { basename, dirname, relative, resolve, sep } from "node:path"; + +import { + PROJECT_COMPOSE_FILENAME, + PROJECT_CONFIG_FILENAME, + PROJECT_ENV_CONFIG_FILENAME_PREFIX, + PROJECT_ENV_CONFIG_FILENAME_SUFFIX, + PROJECT_ENV_CONTRACT_FILENAME, + PROJECT_ENV_FILENAME, +} from "../constants.ts"; +import { resolveGlobalConfigPath } from "./config-paths.ts"; +import { resolveEnvKeyCandidates } from "./doctor-env-key.ts"; +import { findProjectContext, readProjectConfig } from "./project.ts"; +import { + discoverLegacyMigrationOverlays, + previewLegacyMigrationOverlayNames, +} from "./project-env-config.ts"; +import { + readProjectsRegistryStrict, + resolveProjectsRegistryPath, +} from "./projects-registry.ts"; +import { + inspectStoreKeyState, + resolveFileSecretStorePaths, +} from "./secret-store.ts"; +import { + classifyRegistryEntry, + registeredCheckouts, +} from "./upgrade-check-project.ts"; +import type { UpgradeSnapshotTarget } from "./upgrade-snapshot.ts"; + +/** + * The files `hack upgrade snapshot` keeps (HACK-2189): what the upgrade + * steps change. Global: the Hack config, the projects registry, and the + * encrypted secret store and its key when a file store is in use. Per live + * registered project: its `.hack/` config, env contract, dotenv files, YAML + * env layers, `.hack/.gitignore`, and env key files inside the checkout. + * + * Existing regular files are kept; a path that does not exist yet (an env + * layer the migration creates) is recorded as absent, so a rollback moves + * aside what a step put there. A link or other entry at one of these paths + * is reported in `skipped` and never followed. A step that writes other + * files snapshots them itself, through the library. + */ +export async function collectUpgradeSnapshotTargets(): Promise<{ + readonly targets: readonly UpgradeSnapshotTarget[]; + readonly skipped: readonly { + readonly path: string; + readonly label: string; + readonly reason: "symlink" | "not-a-file" | "unreadable"; + }[]; + /** + * What the snapshot does not cover, so it is not a complete recovery + * point: a registry that could not be read (no project is covered), + * registry records that are not valid entries, live checkouts that + * cannot be read, files that were skipped (a link or other entry), and + * checkouts whose migration write set could not be fully known (overlays + * only the secret store holds, when listing them would ask the keychain). + * Dead registry entries are left out on purpose and are not gaps. + */ + readonly gaps: readonly { readonly name: string; readonly reason: string }[]; +}> { + const gaps: { name: string; reason: string }[] = []; + const candidates: UpgradeSnapshotTarget[] = [ + { path: resolveGlobalConfigPath(), label: "global: hack.config.json" }, + { path: resolveProjectsRegistryPath(), label: "global: projects.json" }, + ]; + const store = await resolveFileSecretStorePaths({}).catch(() => null); + if (store?.active) { + candidates.push( + { path: store.storePath, label: "global: secret store" }, + { path: store.keyPath, label: "global: secret store key" } + ); + } + const registry = await readProjectsRegistryStrict().catch(() => null); + if (registry === null || registry.kind === "invalid") { + gaps.push({ + name: "projects registry", + reason: + "it could not be read or is not valid, so no project's files are in this snapshot", + }); + } + if (registry?.kind === "valid") { + if (registry.rejected > 0) { + gaps.push({ + name: "projects registry", + reason: `${registry.rejected} record(s) are not valid project entries, so those projects are not in this snapshot`, + }); + } + for (const project of registry.registry.projects.flatMap( + registeredCheckouts + )) { + const state = await classifyRegistryEntry(project); + if (state.state === "unreadable") { + gaps.push({ + name: project.name, + reason: `the checkout cannot be read (${state.reason}), so its files are not in this snapshot`, + }); + } + if (state.state !== "live") { + continue; + } + const found = await projectCandidates({ + name: project.name, + root: project.repoRoot, + projectDir: project.projectDir, + }); + candidates.push(...found.targets); + if (found.gap !== null) { + gaps.push({ name: project.name, reason: found.gap }); + } + } + } + const { targets, skipped } = await classifyCandidates(candidates); + for (const entry of skipped) { + gaps.push({ + name: entry.label, + reason: `${entry.path} was not kept (${entry.reason})`, + }); + } + return { targets, skipped, gaps }; +} + +type Skipped = { + readonly path: string; + readonly label: string; + readonly reason: "symlink" | "not-a-file" | "unreadable"; +}; + +/** + * Regular files and absent paths become targets (absent ones recorded as + * absent, so a rollback moves aside what a step creates there); links and + * other entries are skipped, never followed. + */ +async function classifyCandidates( + candidates: readonly UpgradeSnapshotTarget[] +): Promise<{ + readonly targets: UpgradeSnapshotTarget[]; + readonly skipped: Skipped[]; +}> { + const seen = new Set(); + const targets: UpgradeSnapshotTarget[] = []; + const skipped: Skipped[] = []; + for (const candidate of candidates) { + const path = resolve(candidate.path); + if (seen.has(path)) { + continue; + } + seen.add(path); + const entry = await lstat(path).catch((error: unknown) => + errorCode(error) === "ENOENT" || errorCode(error) === "ENOTDIR" + ? null + : ("error" as const) + ); + if (entry === null) { + // Absent now: recorded as absent, so a rollback moves aside what a + // step creates there. + targets.push({ path, label: candidate.label }); + continue; + } + if (entry === "error") { + skipped.push({ path, label: candidate.label, reason: "unreadable" }); + } else if (entry.isSymbolicLink()) { + skipped.push({ path, label: candidate.label, reason: "symlink" }); + } else if (entry.isFile()) { + targets.push({ path, label: candidate.label }); + } else { + skipped.push({ path, label: candidate.label, reason: "not-a-file" }); + } + } + return { targets, skipped }; +} + +const LEGACY_OVERLAY = /^\.env\.([A-Za-z0-9_-]+)$/u; + +async function projectCandidates(opts: { + readonly name: string; + readonly root: string; + readonly projectDir: string; +}): Promise<{ + readonly targets: UpgradeSnapshotTarget[]; + readonly gap: string | null; +}> { + const dirName = basename(opts.projectDir); + const label = (file: string) => `project ${opts.name}: ${dirName}/${file}`; + // An inventory that cannot be read is unknown, not empty: a gap. + const listed = await readdir(opts.projectDir).then( + (entries) => ({ names: entries, gap: null as string | null }), + () => ({ + names: [] as string[], + gap: `its ${basename(opts.projectDir)}/ directory could not be listed, so its env files and layers may be missing from this snapshot`, + }) + ); + const names = listed.names; + const envFiles = names.filter( + (name) => + name === PROJECT_ENV_FILENAME || + name.startsWith(`${PROJECT_ENV_FILENAME}.`) || + (name.startsWith(PROJECT_ENV_CONFIG_FILENAME_PREFIX) && + name.endsWith(PROJECT_ENV_CONFIG_FILENAME_SUFFIX)) + ); + // What the env migration writes or creates: the YAML layers (base, + // machine-local, and one pair per legacy overlay), the config, the + // Compose file it repairs, and .hack/.gitignore; absent ones are recorded + // as absent. + const migration = await migrationOverlays({ + root: opts.root, + projectDir: opts.projectDir, + names, + }); + const overlays = migration.overlays; + const layers = [ + "hack.env.default.yaml", + "hack.env.local.yaml", + "hack.env.default.local.yaml", + ...overlays.flatMap((overlay) => [ + `hack.env.${overlay}.yaml`, + `hack.env.${overlay}.local.yaml`, + ]), + ]; + const files = [ + PROJECT_CONFIG_FILENAME, + PROJECT_ENV_CONTRACT_FILENAME, + PROJECT_COMPOSE_FILENAME, + ".gitignore", + ...envFiles, + ...layers, + ]; + const keys = await resolveEnvKeyCandidates({ projectRoot: opts.root }).catch( + () => [] + ); + // The encrypted store and key this project's config selects (it may name + // its own): migration cleanup can remove them once their values are in + // env layers, so a rollback needs them back. + const backend = await resolveFileSecretStorePaths({ + projectDir: opts.projectDir, + }).catch(() => null); + const backendTargets = backend?.active + ? [ + { + path: backend.storePath, + label: `project ${opts.name}: secret store`, + }, + { + path: backend.keyPath, + label: `project ${opts.name}: secret store key`, + }, + ] + : []; + return { + targets: [ + ...backendTargets, + ...files.map((file) => ({ + path: resolve(opts.projectDir, file), + label: label(file), + })), + ...keys + .filter((key) => isInside({ path: key.path, dir: opts.root })) + .map((key) => ({ + path: key.path, + label: `project ${opts.name}: ${relative(opts.root, key.path)} (env key)`, + })), + ], + gap: + [listed.gap, migration.gap].filter((gap) => gap !== null).join("; ") || + null, + }; +} + +/** + * The overlays the env migration would turn into layers: from the files, + * plus, when the project's secret store opens with this shell's key or its + * key file (never the keychain), the overlays only the store holds, from + * the migration's own selection. When the store can only be opened through + * the keychain, those cannot be listed without a prompt: a gap. + */ +async function migrationOverlays(opts: { + readonly root: string; + readonly projectDir: string; + readonly names: readonly string[]; +}): Promise<{ + readonly overlays: readonly string[]; + readonly gap: string | null; +}> { + // The migration's own discovery: `.env.` files and the configured + // default overlay, whatever the secret backend. + const fromFiles = [ + ...new Set([ + ...opts.names.flatMap((name) => { + const match = LEGACY_OVERLAY.exec(name); + return match?.[1] && match[1] !== "state" ? [match[1]] : []; + }), + ...(await discoverLegacyMigrationOverlays({ + projectDir: opts.projectDir, + })), + ]), + ]; + const location = await resolveFileSecretStorePaths({ + projectDir: opts.projectDir, + }).catch(() => null); + if (!location?.active) { + return { overlays: fromFiles, gap: null }; + } + const state = await inspectStoreKeyState({ + storePath: location.storePath, + keyPath: location.keyPath, + env: process.env, + }); + if (state.kind === "key-uninspected") { + return { + overlays: fromFiles, + gap: "its secret store's key is not in a key file or HACK_SECRETS_FILE_KEY, so overlays held only in the store could not be listed without asking the OS keychain", + }; + } + if (state.kind !== "opens") { + // The migration cannot list the store either, so it adds no overlay + // from it. + return { overlays: fromFiles, gap: null }; + } + const preview = await previewLegacyMigrationOverlayNames({ + projectDir: opts.projectDir, + projectName: await projectName(opts), + }); + return { overlays: [...new Set([...fromFiles, ...preview])], gap: null }; +} + +/** + * The name the migration scopes store entries by, resolved as doctor does: + * the project config's name (`hack.config.json` or the legacy `hack.toml`), + * else the checkout directory's name. + */ +async function projectName(opts: { + readonly root: string; + readonly projectDir: string; +}): Promise { + try { + const project = await findProjectContext(opts.root); + if (project !== null) { + const name = (await readProjectConfig(project)).name?.trim(); + if (name) { + return name; + } + } + } catch { + // Fall back to the directory name, as the migration does. + } + return basename(opts.root); +} + +function isInside(opts: { readonly path: string; readonly dir: string }) { + const path = resolve(opts.path); + const dir = resolve(opts.dir); + return path.startsWith(`${dir}${sep}`) && dirname(path) !== ""; +} + +function errorCode(error: unknown): string | undefined { + if (typeof error !== "object" || error === null || !("code" in error)) { + return; + } + const { code } = error; + return typeof code === "string" ? code : undefined; +} diff --git a/src/lib/upgrade-snapshot.ts b/src/lib/upgrade-snapshot.ts new file mode 100644 index 000000000..ce2eed179 --- /dev/null +++ b/src/lib/upgrade-snapshot.ts @@ -0,0 +1,1422 @@ +import { createHash, randomBytes } from "node:crypto"; +import { constants, type Stats } from "node:fs"; +import { + type FileHandle, + link, + lstat, + mkdir, + open, + readdir, + realpath, + rename, + stat, + unlink, +} from "node:fs/promises"; +import { + basename, + dirname, + isAbsolute, + join, + normalize, + resolve, +} from "node:path"; + +import { resolveGlobalHackDir } from "./config-paths.ts"; +import { isRecord } from "./guards.ts"; + +/** + * Snapshots and rollback for upgrade steps (HACK-2189, the upgrade assistant + * HACK-2188). A step that changes files (the env migration, a later + * `hack upgrade apply`) snapshots them first, records what it wrote, and can + * put every file back. + * + * Contract: + * - {@link createUpgradeSnapshot} reads every target first and writes + * nothing unless all of them can be snapshotted (all or nothing). A target + * is a regular file (its exact bytes and mode are kept) or absent (the + * snapshot records that it did not exist). A symlink, directory or other + * entry at a target, a target larger than {@link MAX_SNAPSHOT_FILE_BYTES}, + * or one that cannot be read refuses the whole snapshot. + * - Storage is a private directory outside every Git working tree + * (default `~/.hack/upgrade/snapshots//`): directories 0700, files + * 0600, created with no-follow and exclusive opens, owner-checked on every + * read. File bytes live in `blobs/`. The manifest holds only + * paths, caller labels, kinds, sizes, modes and SHA-256 digests: never + * file content. Manifest writes are atomic (temporary file, fsync, + * rename). + * - {@link recordUpgradeSnapshotAfter} records each target's state after the + * step wrote it. That is the reference for "changed since": restore puts a + * target back only when it still holds what the step wrote (or already + * holds the snapshot), and refuses with `conflict` otherwise, unless + * `overwrite` is set. A target with no recorded after state (an + * interrupted step) also needs `overwrite`. + * - {@link restoreUpgradeSnapshot} never deletes data. What it replaces or + * removes (the step's output, or with `overwrite` a later edit) is kept + * as itself (the same file, not a copy) and named in the result + * (`keptAs`). A restored file gets the snapshot's exact bytes and mode, + * staged as a no-follow temporary file and published with a hard link + * that refuses an occupied name. Restore is + * idempotent: a target that already matches the snapshot is + * `already-restored`. Every target it could not restore is reported with + * a closed reason. + * - Results and errors carry paths, labels, statuses and closed messages, + * never file content. Labels are caller input and must not hold values. + * + * Restore protocol, per target, with every temporary and displaced file in + * the snapshot's private directory and none beside the target (a checkout's + * Git must never see secret bytes): stage the snapshot's copy in + * `staging/` and check that hard links work (otherwise restore nothing); + * move whatever is at the target into `displaced/` with one atomic rename + * (the same file, so a writer that still holds it open keeps writing into + * a named file); publish the staged copy with a hard link, which fails if + * anything appeared at the target meanwhile. A late edit found in the + * displaced file is linked back unless `overwrite` is set. A displaced + * non-file is never renamed back (no rename refuses an occupied name); it + * stays in `displaced/` and is reported. Because a rename and a hard link + * cross no filesystem, a target on another filesystem than the snapshot is + * refused untouched. Not atomic across targets. + * + * One writer per snapshot: `recordUpgradeSnapshotAfter` rewrites the + * manifest (read, modify, atomic rename, best-effort directory fsync), so two + * concurrent recorders on the same snapshot can lose each other's after + * states. Callers (one upgrade step at a time) own the snapshot while they + * run. + */ + +export const UPGRADE_SNAPSHOT_VERSION = 1; +export const UPGRADE_SNAPSHOT_KIND = "hack-upgrade-snapshot"; +/** Largest file a snapshot keeps; larger targets refuse the snapshot. */ +export const MAX_SNAPSHOT_FILE_BYTES = 64 * 1024 * 1024; +const MAX_MANIFEST_BYTES = 16 * 1024 * 1024; +const MANIFEST_NAME = "manifest.json"; +const BLOBS_DIR = "blobs"; +/** Displaced files, moved in as themselves (same inode), not copies. */ +const DISPLACED_DIR = "displaced"; +/** The snapshot's copies staged for publication. */ +const STAGING_DIR = "staging"; +const SHA256_HEX = /^[0-9a-f]{64}$/; +const SNAPSHOT_ID = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/; + +/** A file's state: its digest, size and permission bits, or absence. */ +export type UpgradeSnapshotState = + | { + readonly kind: "file"; + readonly size: number; + /** Permission bits (`mode & 0o7777`). */ + readonly mode: number; + readonly sha256: string; + } + | { readonly kind: "absent" }; + +export type UpgradeSnapshotEntry = { + /** Absolute, normalized target path. */ + readonly path: string; + /** Caller's description, for example `project app-a: .env`. No values. */ + readonly label: string; + readonly before: UpgradeSnapshotState; + /** Recorded by {@link recordUpgradeSnapshotAfter}; null until then. */ + readonly after: UpgradeSnapshotState | null; +}; + +export type UpgradeSnapshotManifest = { + readonly version: typeof UPGRADE_SNAPSHOT_VERSION; + readonly kind: typeof UPGRADE_SNAPSHOT_KIND; + readonly id: string; + /** What the snapshot is for, for example `env-migration app-a`. */ + readonly purpose: string; + readonly createdAt: string; + readonly updatedAt: string; + readonly entries: readonly UpgradeSnapshotEntry[]; +}; + +export type UpgradeSnapshot = { + readonly id: string; + readonly dir: string; + readonly manifestPath: string; + readonly manifest: UpgradeSnapshotManifest; +}; + +export type UpgradeSnapshotTarget = { + readonly path: string; + readonly label: string; +}; + +/** Why one target could not be snapshotted. */ +export type UpgradeSnapshotTargetRefusal = + | "not-absolute" + | "duplicate" + | "symlink" + | "not-a-file" + | "too-large" + | "unreadable" + | "inside-snapshot-root"; + +/** Why a snapshot directory could not be used. */ +export type UpgradeSnapshotRootRefusal = + | "inside-working-tree" + | "not-private" + | "not-owned" + | "symlink" + | "not-a-directory" + | "unwritable"; + +export type CreateUpgradeSnapshotResult = + | { readonly status: "created"; readonly snapshot: UpgradeSnapshot } + | { + readonly status: "refused"; + readonly reason: "targets"; + readonly targets: readonly { + readonly path: string; + readonly label: string; + readonly reason: UpgradeSnapshotTargetRefusal; + }[]; + } + | { + readonly status: "refused"; + readonly reason: "root"; + readonly root: string; + readonly rootReason: UpgradeSnapshotRootRefusal; + }; + +/** The default snapshot directory: `/upgrade/snapshots`. */ +export function defaultUpgradeSnapshotRoot(): string { + return join(resolveGlobalHackDir(), "upgrade", "snapshots"); +} + +/** + * Snapshots `targets` into a new private directory under `root`. Reads + * every target before writing anything; any refused target refuses the + * whole snapshot and nothing is written. + */ +export async function createUpgradeSnapshot(input: { + readonly purpose: string; + readonly targets: readonly UpgradeSnapshotTarget[]; + readonly root?: string; + readonly now?: Date; + /** Test seam; production uses a timestamp and random suffix. */ + readonly id?: string; +}): Promise { + const root = resolve(input.root ?? defaultUpgradeSnapshotRoot()); + const read = await readTargets({ targets: input.targets, root }); + if (read.refused.length > 0) { + return { status: "refused", reason: "targets", targets: read.refused }; + } + const rootProblem = await ensurePrivateRoot(root); + if (rootProblem !== null) { + return { status: "refused", reason: "root", root, rootReason: rootProblem }; + } + const now = input.now ?? new Date(); + const id = input.id ?? snapshotId(now); + if (!SNAPSHOT_ID.test(id)) { + throw new Error("Invalid upgrade snapshot id."); + } + const dir = join(root, id); + // Exclusive: a second snapshot with the same id never reuses a directory. + await mkdir(dir, { mode: 0o700 }); + await mkdir(join(dir, BLOBS_DIR), { mode: 0o700 }); + for (const target of read.targets) { + if (target.bytes !== null && target.state.kind === "file") { + await writeBlob({ + dir, + sha256: target.state.sha256, + bytes: target.bytes, + }); + } + } + const stamp = now.toISOString(); + const manifest: UpgradeSnapshotManifest = { + version: UPGRADE_SNAPSHOT_VERSION, + kind: UPGRADE_SNAPSHOT_KIND, + id, + purpose: input.purpose, + createdAt: stamp, + updatedAt: stamp, + entries: read.targets.map((target) => ({ + path: target.path, + label: target.label, + before: target.state, + after: null, + })), + }; + const manifestPath = join(dir, MANIFEST_NAME); + await writeManifest({ path: manifestPath, manifest }); + return { status: "created", snapshot: { id, dir, manifestPath, manifest } }; +} + +export type RecordUpgradeSnapshotAfterResult = + | { readonly status: "recorded"; readonly snapshot: UpgradeSnapshot } + | { readonly status: "invalid-manifest"; readonly message: string } + | { + readonly status: "refused"; + readonly paths: readonly { + readonly path: string; + readonly reason: UpgradeSnapshotTargetRefusal; + }[]; + }; + +/** + * Records the current state of the snapshot's targets (all of them, or + * `paths`) as what the step wrote. Call it after each write, or once after + * the step. Only digests are recorded; nothing is copied. + */ +export async function recordUpgradeSnapshotAfter(input: { + readonly manifestPath: string; + readonly paths?: readonly string[]; + readonly now?: Date; +}): Promise { + const loaded = await readUpgradeSnapshot({ + manifestPath: input.manifestPath, + }); + if (loaded.status !== "ok") { + return { status: "invalid-manifest", message: loaded.message }; + } + const { snapshot } = loaded; + const wanted = + input.paths === undefined + ? null + : new Set(input.paths.map((path) => normalize(resolve(path)))); + const refused: { path: string; reason: UpgradeSnapshotTargetRefusal }[] = []; + const entries: UpgradeSnapshotEntry[] = []; + for (const entry of snapshot.manifest.entries) { + if (wanted !== null && !wanted.has(entry.path)) { + entries.push(entry); + continue; + } + const current = await inspectTarget(entry.path); + if (current.kind === "refused") { + refused.push({ path: entry.path, reason: current.reason }); + entries.push(entry); + continue; + } + entries.push({ ...entry, after: current.state }); + } + if (wanted !== null) { + const known = new Set(snapshot.manifest.entries.map((entry) => entry.path)); + for (const path of wanted) { + if (!known.has(path)) { + throw new Error("The path is not a target of this upgrade snapshot."); + } + } + } + if (refused.length > 0) { + return { status: "refused", paths: refused }; + } + const manifest: UpgradeSnapshotManifest = { + ...snapshot.manifest, + updatedAt: (input.now ?? new Date()).toISOString(), + entries, + }; + await writeManifest({ path: snapshot.manifestPath, manifest }); + return { status: "recorded", snapshot: { ...snapshot, manifest } }; +} + +/** Per-target outcome of a restore. */ +export type UpgradeRestoreStatus = + /** The snapshot's bytes and mode were put back. */ + | "restored" + /** The target was absent in the snapshot; what the step created was moved into the snapshot directory. */ + | "moved-aside" + /** The target already matches the snapshot; nothing changed. */ + | "already-restored" + /** The target changed since the step recorded it (or no after state was recorded); left unchanged. */ + | "conflict" + /** A symlink, directory or other entry is at the target; left unchanged, even with overwrite. */ + | "not-a-file" + /** The snapshot's copy is missing or does not match its digest. */ + | "backup-damaged" + | "error"; + +export type UpgradeRestoreEntry = { + readonly path: string; + readonly label: string; + readonly status: UpgradeRestoreStatus; + /** Where the replaced or removed content was kept, when any was. */ + readonly keptAs?: string; + readonly message?: string; +}; + +export type RestoreUpgradeSnapshotResult = + | { readonly status: "invalid-manifest"; readonly message: string } + | { + /** Every target is back as it was in the snapshot. */ + readonly status: "restored" | "incomplete"; + readonly results: readonly UpgradeRestoreEntry[]; + }; + +const MESSAGES = { + conflict: + "The file changed since the upgrade step wrote it; left unchanged. Check it, then rerun with overwrite to keep the current file in the snapshot and restore the snapshot's copy.", + noAfter: + "The upgrade step did not record what it wrote here, so a change cannot be told apart from the step's own write; left unchanged. Rerun with overwrite to restore the snapshot's copy (the current file is kept in the snapshot).", + notAFile: + "A link, directory or other entry is at this path; left unchanged. Move it aside, then rerun the restore.", + damaged: + "The snapshot's copy of this file is missing or does not match its recorded digest; nothing was restored here.", + error: "The file could not be restored at this path; left as it was.", + gitEnvironment: + "GIT_DIR or GIT_WORK_TREE is set, so the snapshot directory may be part of a Git working tree; nothing was changed here. Unset them and rerun the restore.", + noWorkDir: + "The snapshot's private working directories could not be prepared; nothing was changed here.", + crossFilesystem: + "This file is on another filesystem than the snapshot, so it cannot be restored without leaving temporary copies of its contents beside it (where Git could pick them up); nothing was changed here. Put the file back by hand, or rerun with a snapshot on the same filesystem.", + noHardLinks: + "This directory's filesystem does not support hard links, so the snapshot's copy cannot be put back without risking a file that appears meanwhile; nothing was changed here.", + putBackUnrestored: + "The snapshot's copy could not be put back at this path, so the file that was there was put back; nothing was restored here.", + notPublished: + "The snapshot's copy could not be put back at this path; the file that was there is kept at keptAs.", + reoccupied: + "Another file appeared at this path during the restore; it was left as it is, so the path does not hold the snapshot's state (absent).", + nonFileAside: + "A link, directory or other entry was found at this path during the restore and is kept at keptAs; nothing was restored here. Move it back by hand if it belongs there.", + strandedAt: + "The current file could not be kept in the snapshot, and another file appeared at this path meanwhile; the current file is at keptAs. Nothing was deleted.", +} as const; + +/** + * Puts the snapshot's targets (all, or `paths`) back. See the module + * contract: never deletes, refuses changed targets unless `overwrite`, + * reports every target it could not restore. + */ +export async function restoreUpgradeSnapshot(input: { + readonly manifestPath: string; + readonly overwrite?: boolean; + readonly paths?: readonly string[]; + /** Test seam: runs after a target's checks, before it is displaced. */ + readonly beforeReplace?: (path: string) => Promise; + /** Test seam: runs after the target is displaced, before publication. */ + readonly afterDisplace?: (path: string) => Promise; +}): Promise { + const loaded = await readUpgradeSnapshot({ + manifestPath: input.manifestPath, + }); + if (loaded.status !== "ok") { + return { status: "invalid-manifest", message: loaded.message }; + } + const { snapshot } = loaded; + const wanted = + input.paths === undefined + ? null + : new Set(input.paths.map((path) => normalize(resolve(path)))); + if (wanted !== null) { + const known = new Set(snapshot.manifest.entries.map((entry) => entry.path)); + for (const path of wanted) { + if (!known.has(path)) { + throw new Error("The path is not a target of this upgrade snapshot."); + } + } + } + const results: UpgradeRestoreEntry[] = []; + for (const entry of snapshot.manifest.entries) { + if (wanted !== null && !wanted.has(entry.path)) { + continue; + } + let outcome: Omit; + try { + outcome = await restoreEntry({ + snapshot, + entry, + overwrite: input.overwrite === true, + beforeReplace: input.beforeReplace, + afterDisplace: input.afterDisplace, + }); + } catch { + outcome = { status: "error", message: MESSAGES.error }; + } + results.push({ path: entry.path, label: entry.label, ...outcome }); + } + const done = results.every( + (result) => + result.status === "restored" || + result.status === "moved-aside" || + result.status === "already-restored" + ); + return { status: done ? "restored" : "incomplete", results }; +} + +export type ReadUpgradeSnapshotResult = + | { readonly status: "ok"; readonly snapshot: UpgradeSnapshot } + | { readonly status: "invalid"; readonly message: string }; + +const INVALID_MANIFEST = + "The upgrade snapshot manifest is missing, unreadable, not private to this user, or not a valid Hack upgrade snapshot."; + +/** + * Reads and validates a manifest: owned by this user, not a link, private + * (no group or other access), and of the expected shape. + */ +export async function readUpgradeSnapshot(input: { + readonly manifestPath: string; +}): Promise { + const invalid = { status: "invalid", message: INVALID_MANIFEST } as const; + const manifestPath = resolve(input.manifestPath); + const dir = dirname(manifestPath); + if (basename(manifestPath) !== MANIFEST_NAME) { + return invalid; + } + if ( + (await privateDirectoryProblem(dir)) !== null || + (await privateDirectoryProblem(join(dir, BLOBS_DIR))) !== null + ) { + return invalid; + } + const text = await readPrivateFile({ + path: manifestPath, + maxBytes: MAX_MANIFEST_BYTES, + }); + if (text === null) { + return invalid; + } + let parsed: unknown; + try { + parsed = JSON.parse(text.toString("utf8")); + } catch { + return invalid; + } + const manifest = parseManifest(parsed); + if (manifest === null || manifest.id !== basename(dir)) { + return invalid; + } + return { + status: "ok", + snapshot: { id: manifest.id, dir, manifestPath, manifest }, + }; +} + +/** Snapshots under `root`, newest first; unreadable ones are skipped. */ +export async function listUpgradeSnapshots( + input: { readonly root?: string } = {} +): Promise { + const root = resolve(input.root ?? defaultUpgradeSnapshotRoot()); + let names: string[]; + try { + names = await readdir(root); + } catch { + return []; + } + const found: UpgradeSnapshot[] = []; + for (const name of names) { + if (!SNAPSHOT_ID.test(name)) { + continue; + } + const read = await readUpgradeSnapshot({ + manifestPath: join(root, name, MANIFEST_NAME), + }); + if (read.status === "ok") { + found.push(read.snapshot); + } + } + return found.sort((a, b) => + b.manifest.createdAt.localeCompare(a.manifest.createdAt) + ); +} + +// --------------------------------------------------------------------------- +// Reading targets +// --------------------------------------------------------------------------- + +type ReadTarget = { + readonly path: string; + readonly label: string; + readonly state: UpgradeSnapshotState; + readonly bytes: Buffer | null; +}; + +async function readTargets(input: { + readonly targets: readonly UpgradeSnapshotTarget[]; + readonly root: string; +}): Promise<{ + readonly targets: readonly ReadTarget[]; + readonly refused: readonly { + readonly path: string; + readonly label: string; + readonly reason: UpgradeSnapshotTargetRefusal; + }[]; +}> { + const targets: ReadTarget[] = []; + const refused: { + path: string; + label: string; + reason: UpgradeSnapshotTargetRefusal; + }[] = []; + const seen = new Set(); + for (const target of input.targets) { + if (!isAbsolute(target.path)) { + refused.push({ ...target, reason: "not-absolute" }); + continue; + } + const path = normalize(target.path); + if (seen.has(path)) { + refused.push({ path, label: target.label, reason: "duplicate" }); + continue; + } + seen.add(path); + if (isInside({ path, dir: input.root })) { + refused.push({ + path, + label: target.label, + reason: "inside-snapshot-root", + }); + continue; + } + const read = await readTargetBytes(path); + if (read.kind === "refused") { + refused.push({ path, label: target.label, reason: read.reason }); + continue; + } + targets.push({ + path, + label: target.label, + state: read.state, + bytes: read.bytes, + }); + } + return { targets, refused }; +} + +type TargetRead = + | { + readonly kind: "ok"; + readonly state: UpgradeSnapshotState; + readonly bytes: Buffer | null; + } + | { readonly kind: "refused"; readonly reason: UpgradeSnapshotTargetRefusal }; + +/** A target's state and bytes, read through a no-follow open. */ +async function readTargetBytes(path: string): Promise { + const entry = await lstatOrNull(path); + if (entry === "error") { + return { kind: "refused", reason: "unreadable" }; + } + if (entry === null) { + return { kind: "ok", state: { kind: "absent" }, bytes: null }; + } + if (entry.isSymbolicLink()) { + return { kind: "refused", reason: "symlink" }; + } + if (!entry.isFile()) { + return { kind: "refused", reason: "not-a-file" }; + } + if (entry.size > MAX_SNAPSHOT_FILE_BYTES) { + return { kind: "refused", reason: "too-large" }; + } + let handle: FileHandle | null = null; + try { + handle = await open( + path, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ); + const info = await handle.stat(); + if (!info.isFile() || info.ino !== entry.ino || info.dev !== entry.dev) { + return { kind: "refused", reason: "not-a-file" }; + } + if (info.size > MAX_SNAPSHOT_FILE_BYTES) { + return { kind: "refused", reason: "too-large" }; + } + const bytes = await handle.readFile(); + if (bytes.byteLength > MAX_SNAPSHOT_FILE_BYTES) { + return { kind: "refused", reason: "too-large" }; + } + return { + kind: "ok", + state: { + kind: "file", + size: bytes.byteLength, + mode: info.mode & 0o7777, + sha256: sha256(bytes), + }, + bytes, + }; + } catch (error: unknown) { + return { + kind: "refused", + reason: errorCode(error) === "ELOOP" ? "symlink" : "unreadable", + }; + } finally { + await handle?.close(); + } +} + +/** A target's current state, without keeping its bytes. */ +async function inspectTarget( + path: string +): Promise< + | { readonly kind: "ok"; readonly state: UpgradeSnapshotState } + | { readonly kind: "refused"; readonly reason: UpgradeSnapshotTargetRefusal } +> { + const read = await readTargetBytes(path); + return read.kind === "ok" ? { kind: "ok", state: read.state } : read; +} + +// --------------------------------------------------------------------------- +// Restoring +// --------------------------------------------------------------------------- + +function sameState(a: UpgradeSnapshotState, b: UpgradeSnapshotState): boolean { + if (a.kind === "absent" || b.kind === "absent") { + return a.kind === b.kind; + } + return a.sha256 === b.sha256 && a.size === b.size && a.mode === b.mode; +} + +type RestoreOutcome = Omit; + +/** + * What restoring `entry` would start from: the target's current bytes, or + * the outcome when there is nothing to do or it must not be touched. + */ +async function restoreStart(input: { + readonly entry: UpgradeSnapshotEntry; + readonly overwrite: boolean; +}): Promise< + | { readonly kind: "done"; readonly outcome: RestoreOutcome } + | { readonly kind: "go"; readonly current: TargetRead & { kind: "ok" } } +> { + const { entry } = input; + const current = await readTargetBytes(entry.path); + if (current.kind === "refused") { + const outcome: RestoreOutcome = + current.reason === "unreadable" || current.reason === "too-large" + ? { status: "error", message: MESSAGES.error } + : { status: "not-a-file", message: MESSAGES.notAFile }; + return { kind: "done", outcome }; + } + if (sameState(current.state, entry.before)) { + return { kind: "done", outcome: { status: "already-restored" } }; + } + const stepOutput = + entry.after !== null && sameState(current.state, entry.after); + if (!(stepOutput || input.overwrite)) { + return { + kind: "done", + outcome: { + status: "conflict", + message: entry.after === null ? MESSAGES.noAfter : MESSAGES.conflict, + }, + }; + } + return { kind: "go", current }; +} + +async function restoreEntry(input: { + readonly snapshot: UpgradeSnapshot; + readonly entry: UpgradeSnapshotEntry; + readonly overwrite: boolean; + readonly beforeReplace?: (path: string) => Promise; + readonly afterDisplace?: (path: string) => Promise; +}): Promise { + const { snapshot, entry } = input; + const start = await restoreStart(input); + if (start.kind === "done") { + return start.outcome; + } + const { current } = start; + const work = await restoreWorkDirs({ snapshot, path: entry.path }); + if (work.kind === "refused") { + return work.outcome; + } + let staged: string | null = null; + if (entry.before.kind === "file") { + const bytes = await readBlob({ + dir: snapshot.dir, + sha256: entry.before.sha256, + }); + if (bytes === null) { + return { status: "backup-damaged", message: MESSAGES.damaged }; + } + staged = await stageFile({ + dir: work.staging, + bytes, + mode: entry.before.mode, + }); + // Publication must refuse an occupied name, which only a hard link + // does: without hard links, restore nothing rather than risk it. + if (!(await hardLinksWork(staged))) { + await unlink(staged).catch(() => undefined); + return { status: "error", message: MESSAGES.noHardLinks }; + } + } + try { + await input.beforeReplace?.(entry.path); + const displaced = await displaceTarget({ + displacedDir: work.displaced, + path: entry.path, + expected: current.state, + overwrite: input.overwrite, + }); + if (displaced.kind === "stopped") { + return displaced.outcome; + } + await input.afterDisplace?.(entry.path); + if (staged === null) { + return await absentRestored({ + path: entry.path, + keptAs: displaced.keptAs, + }); + } + const toPublish = staged; + staged = null; + return await publishOrPutBack({ + staged: toPublish, + path: entry.path, + keptAs: displaced.keptAs, + }); + } finally { + if (staged !== null) { + await unlink(staged).catch(() => undefined); + } + } +} + +/** + * The snapshot's private `staging/` and `displaced/` directories, where + * every temporary and displaced file lives: never beside the target, where + * a checkout's Git could pick up secret bytes. They must be on the target's + * filesystem (an atomic rename and a hard link cross no filesystem); when + * they are not, nothing is touched and the target is refused. + */ +async function restoreWorkDirs(input: { + readonly snapshot: UpgradeSnapshot; + readonly path: string; +}): Promise< + | { + readonly kind: "ready"; + readonly staging: string; + readonly displaced: string; + } + | { readonly kind: "refused"; readonly outcome: RestoreOutcome } +> { + if (gitSelectedByEnvironment()) { + // Staged copies would be written where Git may track them. + return { + kind: "refused", + outcome: { status: "error", message: MESSAGES.gitEnvironment }, + }; + } + const staging = join(input.snapshot.dir, STAGING_DIR); + const displaced = join(input.snapshot.dir, DISPLACED_DIR); + for (const dir of [staging, displaced]) { + try { + await mkdir(dir, { mode: 0o700 }); + } catch (error: unknown) { + if (errorCode(error) !== "EEXIST") { + return { + kind: "refused", + outcome: { status: "error", message: MESSAGES.noWorkDir }, + }; + } + } + if ((await privateDirectoryProblem(dir)) !== null) { + return { + kind: "refused", + outcome: { status: "error", message: MESSAGES.noWorkDir }, + }; + } + } + const parent = dirname(input.path); + await mkdir(parent, { recursive: true, mode: 0o700 }); + // The directory the restore traverses: a symlinked parent counts by + // where it leads (the target's leaf stays no-follow). + const [work, target] = await Promise.all([lstat(staging), stat(parent)]); + if (work.dev !== target.dev) { + return { + kind: "refused", + outcome: { status: "error", message: MESSAGES.crossFilesystem }, + }; + } + return { kind: "ready", staging, displaced }; +} + +/** Whether hard links can be made beside `staged` (probed, then removed). */ +async function hardLinksWork(staged: string): Promise { + const probe = `${staged}.link-probe`; + try { + await link(staged, probe); + } catch { + return false; + } + await unlink(probe).catch(() => undefined); + return true; +} + +/** + * Moves whatever is at `path` right now into the snapshot's private + * `displaced/` directory with one atomic rename: the same file (inode), so + * a writer that still holds it open keeps writing into a named file, and + * nothing is left beside the target. A late edit (not what was inspected), + * without `overwrite`, is linked back into place. A displaced link, + * directory or other entry stays in `displaced/` and is reported: it is + * never renamed back, since no rename refuses an occupied name. + */ +async function displaceTarget(input: { + readonly displacedDir: string; + readonly path: string; + readonly expected: UpgradeSnapshotState; + readonly overwrite: boolean; +}): Promise< + | { readonly kind: "displaced"; readonly keptAs?: string } + | { readonly kind: "stopped"; readonly outcome: RestoreOutcome } +> { + const kept = join( + input.displacedDir, + `${new Date().toISOString().replace(ISO_SEPARATORS, "")}-${randomBytes(4).toString("hex")}-${basename(input.path)}` + ); + try { + await rename(input.path, kept); + } catch (error: unknown) { + if (errorCode(error) !== "ENOENT") { + throw error; + } + // Nothing is there now. + return input.expected.kind === "absent" || input.overwrite + ? { kind: "displaced" } + : { + kind: "stopped", + outcome: { status: "conflict", message: MESSAGES.conflict }, + }; + } + const moved = await readTargetBytes(kept); + if (moved.kind === "refused") { + const entry = await lstatOrNull(kept); + if (entry !== null && entry !== "error" && entry.isFile()) { + // A regular file that changed after the checks (now unreadable, or + // too large): a late edit, so it goes back in place. + return await lateEditBack({ kept, path: input.path }); + } + } + if (moved.kind === "refused" || moved.state.kind !== "file") { + return { + kind: "stopped", + outcome: { + status: "not-a-file", + message: MESSAGES.nonFileAside, + keptAs: kept, + }, + }; + } + if (!(sameState(moved.state, input.expected) || input.overwrite)) { + // A late edit: link that same file back into place. + return await lateEditBack({ kept, path: input.path }); + } + return { kind: "displaced", keptAs: kept }; +} + +/** + * A late edit found in the displaced file: link that same file back into + * place (refused if the name was taken meanwhile, then reported where it + * is kept). Either way the target is a conflict, never restored over it. + */ +async function lateEditBack(input: { + readonly kept: string; + readonly path: string; +}): Promise<{ readonly kind: "stopped"; readonly outcome: RestoreOutcome }> { + if (await linkBack({ from: input.kept, path: input.path })) { + return { + kind: "stopped", + outcome: { status: "conflict", message: MESSAGES.conflict }, + }; + } + return { + kind: "stopped", + outcome: { + status: "conflict", + message: MESSAGES.strandedAt, + keptAs: input.kept, + }, + }; +} + +/** + * Links a kept file back to `path`, refused when anything took the name + * meanwhile, then drops the kept name (the file is back in place). False + * when the name was taken or the link failed; the kept name stays. + */ +async function linkBack(input: { + readonly from: string; + readonly path: string; +}): Promise { + try { + await link(input.from, input.path); + } catch { + return false; + } + await unlink(input.from).catch(() => undefined); + return true; +} + +/** + * The snapshot holds absence: a file that appeared meanwhile is someone + * else's, left as it is, and the target is not restored. + */ +async function absentRestored(input: { + readonly path: string; + readonly keptAs: string | undefined; +}): Promise { + const kept = input.keptAs === undefined ? {} : { keptAs: input.keptAs }; + if ((await lstatOrNull(input.path)) !== null) { + return { status: "conflict", message: MESSAGES.reoccupied, ...kept }; + } + return input.keptAs === undefined + ? { status: "already-restored" } + : { status: "moved-aside", ...kept }; +} + +/** + * Publishes the staged copy; when that fails, puts the displaced file back + * (refused if the name was taken) or, failing that, names where it is + * kept. + */ +async function publishOrPutBack(input: { + readonly staged: string; + readonly path: string; + readonly keptAs: string | undefined; +}): Promise { + const kept = input.keptAs === undefined ? {} : { keptAs: input.keptAs }; + let published: boolean; + try { + published = await publishStaged({ + staged: input.staged, + path: input.path, + }); + } catch { + if ( + input.keptAs !== undefined && + (await linkBack({ from: input.keptAs, path: input.path })) + ) { + return { status: "error", message: MESSAGES.putBackUnrestored }; + } + return { status: "error", message: MESSAGES.notPublished, ...kept }; + } + return published + ? { status: "restored", ...kept } + : { status: "conflict", message: MESSAGES.conflict, ...kept }; +} + +/** + * Writes `bytes` with exactly `mode` to a new no-follow file in the + * snapshot's private staging directory and syncs it. + */ +async function stageFile(input: { + readonly dir: string; + readonly bytes: Buffer; + readonly mode: number; +}): Promise { + const temporary = join( + input.dir, + `restore-${randomBytes(8).toString("hex")}` + ); + const handle = await open( + temporary, + constants.O_CREAT | + constants.O_EXCL | + constants.O_NOFOLLOW | + constants.O_WRONLY, + 0o600 + ); + try { + await handle.writeFile(input.bytes); + await handle.chmod(input.mode); + await handle.sync(); + } catch (error: unknown) { + await handle.close(); + await unlink(temporary).catch(() => undefined); + throw error; + } + await handle.close(); + return temporary; +} + +/** + * Publishes a staged file at `path` only while the name is free: a hard + * link, which fails when anything appeared there since the displacement. + * False (and the staged file removed) when the name is taken; any other + * failure is thrown (the caller reports where the displaced file is kept). + */ +async function publishStaged(input: { + readonly staged: string; + readonly path: string; +}): Promise { + try { + await link(input.staged, input.path); + } catch (error: unknown) { + await unlink(input.staged).catch(() => undefined); + if (errorCode(error) === "EEXIST") { + return false; + } + throw error; + } + await unlink(input.staged).catch(() => undefined); + return true; +} + +// --------------------------------------------------------------------------- +// Private storage +// --------------------------------------------------------------------------- + +function blobPath(input: { readonly dir: string; readonly sha256: string }) { + return join(input.dir, BLOBS_DIR, input.sha256); +} + +/** + * Stores `bytes` as `blobs/` (0600, exclusive, no-follow). A blob + * that already exists with the same digest is reused. Returns its path. + */ +async function writeBlob(input: { + readonly dir: string; + readonly sha256: string; + readonly bytes: Buffer; +}): Promise { + const path = blobPath(input); + const existing = await readPrivateFile({ + path, + maxBytes: MAX_SNAPSHOT_FILE_BYTES, + }); + if (existing !== null && sha256(existing) === input.sha256) { + return path; + } + const temporary = `${path}.${randomBytes(6).toString("hex")}.tmp`; + const handle = await open( + temporary, + constants.O_CREAT | + constants.O_EXCL | + constants.O_NOFOLLOW | + constants.O_WRONLY, + 0o600 + ); + try { + await handle.writeFile(input.bytes); + await handle.sync(); + } finally { + await handle.close(); + } + await rename(temporary, path); + return path; +} + +async function readBlob(input: { + readonly dir: string; + readonly sha256: string; +}): Promise { + const bytes = await readPrivateFile({ + path: blobPath(input), + maxBytes: MAX_SNAPSHOT_FILE_BYTES, + }); + return bytes !== null && sha256(bytes) === input.sha256 ? bytes : null; +} + +async function writeManifest(input: { + readonly path: string; + readonly manifest: UpgradeSnapshotManifest; +}): Promise { + const temporary = `${input.path}.${randomBytes(6).toString("hex")}.tmp`; + const handle = await open( + temporary, + constants.O_CREAT | + constants.O_EXCL | + constants.O_NOFOLLOW | + constants.O_WRONLY, + 0o600 + ); + try { + await handle.writeFile(`${JSON.stringify(input.manifest, null, 2)}\n`); + await handle.sync(); + } finally { + await handle.close(); + } + await rename(temporary, input.path); + await syncDirectory(dirname(input.path)); +} + +/** Makes a rename in `dir` durable; best effort where directories cannot be synced. */ +async function syncDirectory(dir: string): Promise { + let handle: FileHandle | null = null; + try { + handle = await open(dir, constants.O_RDONLY); + await handle.sync(); + } catch { + // Some platforms and filesystems do not sync directories. + } finally { + await handle?.close(); + } +} + +/** + * Reads a file this user owns, that is not a link and that grants no group + * or other access, up to `maxBytes`. Null for anything else. + */ +async function readPrivateFile(input: { + readonly path: string; + readonly maxBytes: number; +}): Promise { + let handle: FileHandle | null = null; + try { + handle = await open( + input.path, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ); + const info = await handle.stat(); + if ( + !info.isFile() || + info.size > input.maxBytes || + (info.mode & 0o077) !== 0 || + !ownedByThisUser(info.uid) + ) { + return null; + } + return await handle.readFile(); + } catch { + return null; + } finally { + await handle?.close(); + } +} + +function ownedByThisUser(uid: number): boolean { + const own = process.getuid?.(); + return own === undefined || own === uid; +} + +/** Why `dir` is not a private directory of this user, or null. */ +async function privateDirectoryProblem( + dir: string +): Promise { + const entry = await lstatOrNull(dir); + if (entry === null || entry === "error") { + return "not-a-directory"; + } + if (entry.isSymbolicLink()) { + return "symlink"; + } + if (!entry.isDirectory()) { + return "not-a-directory"; + } + if (!ownedByThisUser(entry.uid)) { + return "not-owned"; + } + if ((entry.mode & 0o077) !== 0) { + return "not-private"; + } + return null; +} + +/** + * Creates the snapshot root (0700, each missing component) and checks it: + * a private directory of this user, not a link, and not inside a Git + * working tree. + */ +async function ensurePrivateRoot( + root: string +): Promise { + // Check the path as written and as resolved: a symlinked ancestor (a + // `~/.hack` that points into a dotfiles checkout) must not carry the + // snapshot into a working tree. + const canonical = await canonicalPath(root); + if (canonical === null) { + return "unwritable"; + } + if ((await insideWorkingTree(root)) || (await insideWorkingTree(canonical))) { + return "inside-working-tree"; + } + try { + await mkdir(root, { recursive: true, mode: 0o700 }); + } catch (error: unknown) { + return errorCode(error) === "EEXIST" || errorCode(error) === "ENOTDIR" + ? "not-a-directory" + : "unwritable"; + } + // Again on the created directory, before anything is written into it. + const created = await realpath(root).catch(() => null); + if (created === null) { + return "unwritable"; + } + if (await insideWorkingTree(created)) { + return "inside-working-tree"; + } + return await privateDirectoryProblem(root); +} + +/** + * The resolved form of `path`: the real path of its deepest existing + * ancestor, with the components that do not exist yet appended. Null when + * an ancestor cannot be resolved for another reason. + */ +async function canonicalPath(path: string): Promise { + const missing: string[] = []; + let current = resolve(path); + for (;;) { + try { + return join(await realpath(current), ...missing); + } catch (error: unknown) { + if (errorCode(error) !== "ENOENT") { + return null; + } + const parent = dirname(current); + if (parent === current) { + return null; + } + missing.unshift(basename(current)); + current = parent; + } + } +} + +/** + * Git selected through the environment (`GIT_DIR` or `GIT_WORK_TREE`) can + * make any directory part of a working tree without a `.git` marker above + * it, so its presence counts as inside one: fail closed. + */ +function gitSelectedByEnvironment(): boolean { + // Defined at all counts: Git takes even " " as a literal path, and an + // empty value is refused too (fail closed). + return ["GIT_DIR", "GIT_WORK_TREE"].some( + (key) => process.env[key] !== undefined + ); +} + +/** + * Whether `path` may be inside a Git working tree: a `.git` entry at or + * above it, or Git selected through the environment. + */ +async function insideWorkingTree(path: string): Promise { + if (gitSelectedByEnvironment()) { + return true; + } + let current = resolve(path); + for (;;) { + const marker = await lstatOrNull(join(current, ".git")); + if (marker !== null && marker !== "error") { + return true; + } + const parent = dirname(current); + if (parent === current) { + return false; + } + current = parent; + } +} + +// --------------------------------------------------------------------------- +// Manifest shape +// --------------------------------------------------------------------------- + +function parseState(raw: unknown): UpgradeSnapshotState | null { + if (!isRecord(raw)) { + return null; + } + if (raw.kind === "absent") { + return { kind: "absent" }; + } + const { kind, size, mode, sha256: digest } = raw; + if ( + kind !== "file" || + typeof size !== "number" || + !Number.isInteger(size) || + size < 0 || + size > MAX_SNAPSHOT_FILE_BYTES || + typeof mode !== "number" || + !Number.isInteger(mode) || + mode < 0 || + mode > 0o7777 || + typeof digest !== "string" || + !SHA256_HEX.test(digest) + ) { + return null; + } + return { kind: "file", size, mode, sha256: digest }; +} + +function parseEntry(raw: unknown): UpgradeSnapshotEntry | null { + if (!isRecord(raw)) { + return null; + } + const { path, label } = raw; + if ( + typeof path !== "string" || + !isAbsolute(path) || + normalize(path) !== path || + typeof label !== "string" + ) { + return null; + } + const before = parseState(raw.before); + const after = raw.after === null ? null : parseState(raw.after); + if (before === null || (raw.after !== null && after === null)) { + return null; + } + return { path, label, before, after }; +} + +function parseManifest(raw: unknown): UpgradeSnapshotManifest | null { + if ( + !isRecord(raw) || + raw.version !== UPGRADE_SNAPSHOT_VERSION || + raw.kind !== UPGRADE_SNAPSHOT_KIND || + typeof raw.id !== "string" || + !SNAPSHOT_ID.test(raw.id) || + typeof raw.purpose !== "string" || + typeof raw.createdAt !== "string" || + typeof raw.updatedAt !== "string" || + !Array.isArray(raw.entries) + ) { + return null; + } + const entries: UpgradeSnapshotEntry[] = []; + const seen = new Set(); + for (const value of raw.entries) { + const entry = parseEntry(value); + if (entry === null || seen.has(entry.path)) { + return null; + } + seen.add(entry.path); + entries.push(entry); + } + return { + version: UPGRADE_SNAPSHOT_VERSION, + kind: UPGRADE_SNAPSHOT_KIND, + id: raw.id, + purpose: raw.purpose, + createdAt: raw.createdAt, + updatedAt: raw.updatedAt, + entries, + }; +} + +// --------------------------------------------------------------------------- +// Small helpers +// --------------------------------------------------------------------------- + +const ISO_SEPARATORS = /[-:]/g; +const ISO_FRACTION = /\.\d+Z$/; + +function snapshotId(now: Date): string { + const stamp = now + .toISOString() + .replace(ISO_SEPARATORS, "") + .replace(ISO_FRACTION, "Z"); + return `${stamp}-${randomBytes(4).toString("hex")}`; +} + +function sha256(bytes: Uint8Array): string { + return createHash("sha256").update(bytes).digest("hex"); +} + +function isInside(input: { readonly path: string; readonly dir: string }) { + const dir = input.dir.endsWith("/") ? input.dir : `${input.dir}/`; + return input.path === input.dir || input.path.startsWith(dir); +} + +async function lstatOrNull(path: string): Promise { + try { + return await lstat(path); + } catch (error: unknown) { + return errorCode(error) === "ENOENT" ? null : "error"; + } +} + +function errorCode(error: unknown): string | undefined { + if (typeof error !== "object" || error === null || !("code" in error)) { + return; + } + const { code } = error; + return typeof code === "string" ? code : undefined; +} diff --git a/tests/preload.ts b/tests/preload.ts index e91d4abf2..c54cb7824 100644 --- a/tests/preload.ts +++ b/tests/preload.ts @@ -7,6 +7,8 @@ process.env.HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK = "1"; // fake runner, or pass an env without this key to a CLI with a fake // `systemctl` first on PATH (HACK-2052). process.env.HACK_SYSTEMD_DISABLE = "1"; +// No upgrade-check host probe (Docker, resolver, service managers, PATH hack) runs in tests. +process.env.HACK_HOST_PROBES_DISABLE = "1"; const values = new Map(); function entryKey(input: { service: string; name: string }): string { diff --git a/tests/upgrade-check.test.ts b/tests/upgrade-check.test.ts new file mode 100644 index 000000000..42f319a52 --- /dev/null +++ b/tests/upgrade-check.test.ts @@ -0,0 +1,1476 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { createHash } from "node:crypto"; +import { + chmod, + lstat, + mkdir, + mkdtemp, + readdir, + readFile, + realpath, + rm, + symlink, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { setProjectEnvValue } from "../src/lib/project-env-config.ts"; +import { resolveSecretStore } from "../src/lib/secret-store.ts"; +import { + runUpgradeCheck, + type UpgradeCheckReport, +} from "../src/lib/upgrade-check.ts"; +import { + defaultUpgradeCheckProbes, + type UpgradeCheckProbes, +} from "../src/lib/upgrade-check-probes.ts"; +import { inspectUpgradeProject } from "../src/lib/upgrade-check-project.ts"; +import { collectUpgradeSnapshotTargets } from "../src/lib/upgrade-snapshot-targets.ts"; +import { restoreEnv } from "./helpers/env.ts"; +import { fixtureGitEnv } from "./helpers/fixture-git.ts"; +import { privateCliEnv } from "./helpers/private-cli-env.ts"; + +// Values that must never appear in any report, plan or output. +const SENTINEL = ["sentinel", "upgrade", "check", "c41d07"].join("-"); +const ENTRYPOINT = resolve(import.meta.dir, "..", "index.ts"); + +const STORE_KEY = ["synthetic", "upgrade", "store", "key", "51e0"].join("-"); +const ISOLATED_ENV = [ + "HOME", + "PATH", + "HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK", + "HACK_HOME", + "HACK_GLOBAL_CONFIG_PATH", + "HACK_SECRETS_FILE_KEY", + "HACK_ENV_SECRET_KEY", +] as const; +const saved = new Map(); +let root = ""; +let home = ""; +let hackHome = ""; + +beforeEach(async () => { + for (const key of ISOLATED_ENV) { + saved.set(key, process.env[key]); + } + root = await realpath(await mkdtemp(join(tmpdir(), "hack-upgrade-check-"))); + home = join(root, "user-home"); + hackHome = join(home, ".hack"); + await mkdir(hackHome, { recursive: true }); + process.env.HOME = home; + process.env.HACK_HOME = hackHome; + process.env.HACK_GLOBAL_CONFIG_PATH = join(hackHome, "hack.config.json"); + Reflect.deleteProperty(process.env, "HACK_SECRETS_FILE_KEY"); + Reflect.deleteProperty(process.env, "HACK_ENV_SECRET_KEY"); +}); + +afterEach(async () => { + for (const key of ISOLATED_ENV) { + restoreEnv(key, saved.get(key)); + } + await rm(root, { recursive: true, force: true }); +}); + +type Entry = { + id: string; + name: string; + repoRoot: string; + projectDirName: ".hack"; + projectDir: string; + createdAt: string; + worktrees?: { path: string; branch: string; lastSeenAt: string }[]; +}; + +function entry(name: string): Entry { + const repoRoot = join(root, "repos", name); + return { + id: `id-${name}`, + name, + repoRoot, + projectDirName: ".hack", + projectDir: join(repoRoot, ".hack"), + createdAt: "2026-01-01T00:00:00.000Z", + }; +} + +async function liveProject(name: string): Promise { + const project = entry(name); + await mkdir(project.projectDir, { recursive: true }); + await writeFile( + join(project.projectDir, "hack.config.json"), + `${JSON.stringify({ name, dev_host: `${name}.hack` })}\n` + ); + return project; +} + +async function writeRegistry(projects: readonly Entry[]): Promise { + await writeFile( + join(hackHome, "projects.json"), + `${JSON.stringify({ version: 1, projects })}\n` + ); +} + +/** Probes that report a healthy, fully installed host. */ +function healthyProbes( + overrides: Partial = {} +): UpgradeCheckProbes { + return { + pathHack: async () => ({ + path: "/usr/local/bin/hack", + install: "standalone", + version: "4.3.0", + }), + daemon: async () => ({ + manager: "systemd", + registered: true, + enabled: true, + running: true, + managed: true, + summary: "installed, enabled, active (pid 42)", + severity: "ok", + nextStep: null, + }), + docker: async () => "reachable", + globalInfra: async () => ({ installed: true, running: true }), + linuxDns: async () => null, + proxyPorts: async () => ({ + name: "proxy ports", + status: "ok", + message: "Ports 80 and 443 reach Hack's proxy.", + verified: true, + reachable: true, + }), + nativePreflight: async () => null, + ...overrides, + }; +} + +async function check(probes: UpgradeCheckProbes): Promise { + return await runUpgradeCheck({ + cliVersion: "5.0.0-next.17", + homeDir: home, + probes, + }); +} + +/** Every file under `dir`: path, mode and digest, for a no-change proof. */ +async function tree(dir: string): Promise { + const out: string[] = []; + const walk = async (current: string) => { + for (const name of (await readdir(current)).sort()) { + const path = join(current, name); + const info = await lstat(path); + if (info.isDirectory()) { + out.push(`${path}/ ${info.mode & 0o7777}`); + await walk(path); + } else if (info.isFile()) { + const digest = createHash("sha256") + .update(await readFile(path)) + .digest("hex"); + out.push(`${path} ${info.mode & 0o7777} ${digest}`); + } else { + out.push(`${path} link`); + } + } + }; + await walk(dir); + return out; +} + +test("dead registry entries are counted apart from unreadable ones, and only live projects are checked", async () => { + const live = await liveProject("app-live"); + const configGone = entry("app-config-gone"); + await mkdir(configGone.projectDir, { recursive: true }); + const invalid = await liveProject("app-invalid"); + await writeFile(join(invalid.projectDir, "hack.config.json"), "{ not json"); + const dead = Array.from({ length: 76 }, (_, i) => + entry(`app-dead-${String(i).padStart(2, "0")}`) + ); + await writeRegistry([live, configGone, invalid, ...dead]); + + const report = await check(healthyProbes()); + expect(report.registry.state).toBe("valid"); + expect(report.registry.total).toBe(79); + expect(report.registry.live).toBe(1); + expect(report.registry.missing).toHaveLength(77); + expect( + report.registry.missing.find((e) => e.name === "app-config-gone")?.reason + ).toContain("is gone"); + expect(report.registry.unreadable.map((e) => e.name)).toEqual([ + "app-invalid", + ]); + expect(report.projects.map((p) => [p.name, p.state])).toEqual([ + ["app-invalid", "unreadable"], + ["app-live", "live"], + ]); + const dead77 = report.machine.find((f) => f.id === "registry-dead"); + expect(dead77?.status).toBe("warn"); + expect(dead77?.command).toBe("hack projects prune --dry-run"); + expect(dead77?.summary).toContain("not live projects that failed to read"); + // A live checkout that cannot be read blocks: its safety is unknown. + expect(report.ready).toBe(false); + expect(report.blockers).toContainEqual({ + scope: "app-invalid", + id: "project-config", + summary: expect.stringContaining("not valid JSON"), + }); + // A registry of mostly dead entries stays fast. + expect(report.duration_ms).toBeLessThan(30_000); +}); + +test("a project with only the legacy TOML config or only a Compose file is live, not dead", async () => { + const toml = entry("app-toml"); + await mkdir(toml.projectDir, { recursive: true }); + await writeFile(join(toml.projectDir, "hack.toml"), 'name = "app-toml"\n'); + const compose = entry("app-compose"); + await mkdir(compose.projectDir, { recursive: true }); + await writeFile( + join(compose.projectDir, "docker-compose.yml"), + "services: {}\n" + ); + await writeRegistry([toml, compose]); + const report = await check(healthyProbes()); + expect(report.registry.live).toBe(2); + expect(report.registry.missing).toEqual([]); +}); + +test("once the time budget is spent, project checks report not checked without running", async () => { + const project = await liveProject("app-late"); + let ran = false; + const findings = await inspectUpgradeProject({ + project, + homeDir: home, + deadline: Date.now() - 1, + nativePreflight: async () => { + ran = true; + return null; + }, + }); + expect(ran).toBe(false); + const byId = new Map(findings.map((f) => [f.id, f])); + // Required checks are incomplete (readiness not established); optional + // ones are warnings. + for (const id of ["env-format", "env-key", "tracked-files", "ignore-rules"]) { + expect(byId.get(id)?.status).toBe("incomplete"); + expect(byId.get(id)?.summary).toContain("ran out of time"); + } + expect(byId.get("agent-integrations")?.status).toBe("warn"); +}); + +test("the plan starts with the snapshot, then blockers, then machine and project steps", async () => { + // A conflicting env layer (a dangling link) blocks; legacy env is a step. + const blocked = await liveProject("app-blocked"); + await symlink( + join(root, "nowhere.yaml"), + join(blocked.projectDir, "hack.env.default.yaml") + ); + const legacy = await liveProject("app-legacy"); + await writeFile(join(legacy.projectDir, "hack.env.json"), "{}\n"); + await writeFile(join(legacy.projectDir, ".env"), `TOKEN=${SENTINEL}\n`); + await writeRegistry([blocked, legacy, entry("app-dead")]); + + const report = await check( + healthyProbes({ + globalInfra: async () => ({ installed: true, running: false }), + }) + ); + expect(report.ready).toBe(false); + expect(report.blockers).toEqual([ + { + scope: "app-blocked", + id: "env-format", + summary: expect.stringContaining("link or not a regular file"), + }, + ]); + expect( + report.plan.map((step) => [step.id, step.scope, step.blocker]) + ).toEqual([ + ["snapshot", "machine", false], + ["env-format", "app-blocked", true], + // Its dangling layer link cannot be inspected for secrets either. + ["env-secrets", "app-blocked", false], + ["registry-dead", "machine", false], + ["global-infra", "machine", false], + ["env-format", "app-legacy", false], + ]); + expect(report.next_command).toBe("hack upgrade snapshot"); + const legacyStep = report.plan.find((step) => step.scope === "app-legacy"); + expect(legacyStep?.command).toBe( + `hack doctor --migrate-env-config --path ${legacy.repoRoot}` + ); + expect(legacyStep?.why).toContain("legacy-json, dotenv"); +}); + +test("a host with nothing to do is ready, with no plan and no next command", async () => { + const current = await liveProject("app-current"); + // A YAML env layer written the way Hack writes it (no legacy env files). + await setProjectEnvValue({ + projectRoot: current.repoRoot, + projectDir: current.projectDir, + envName: null, + scope: "global", + key: "PORT", + value: "3000", + secret: false, + }); + await writeRegistry([current]); + const report = await check(healthyProbes()); + expect(report.blockers).toEqual([]); + expect(report.incomplete).toEqual([]); + expect(report.ready).toBe(true); + expect(report.plan).toEqual([]); + expect(report.next_command).toBeNull(); + expect(report.cli).toEqual({ + version: "5.0.0-next.17", + channel: "next", + candidate_run: false, + command: "hack", + }); +}); + +test("a global store whose key is gone blocks only when a registered project reads it", async () => { + const storePath = join(hackHome, "secrets.enc.json"); + await writeFile( + process.env.HACK_GLOBAL_CONFIG_PATH ?? "", + `${JSON.stringify({ + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { + path: storePath, + keyPath: join(hackHome, "secrets-file.key"), + }, + }, + }, + })}\n` + ); + // A store envelope whose key file is missing; its content is never read. + await writeFile( + storePath, + `${JSON.stringify({ version: 1, alg: "aes-256-gcm", iv: "AAAA", tag: "AAAA", data: SENTINEL })}\n`, + { mode: 0o600 } + ); + const bystander = await liveProject("app-bystander"); + await writeRegistry([bystander]); + const warned = await check(healthyProbes()); + const store = warned.machine.find((f) => f.id === "secret-store"); + expect(store?.status).toBe("warn"); + expect(warned.ready).toBe(true); + + const reader = await liveProject("app-reader"); + await writeFile( + join(reader.projectDir, "hack.env.json"), + `${JSON.stringify({ version: 1, vars: [{ key: "TOKEN", source: "keychain" }] })}\n` + ); + await writeRegistry([bystander, reader]); + const blocked = await check(healthyProbes()); + const blocking = blocked.machine.find((f) => f.id === "secret-store"); + expect(blocking?.status).toBe("blocker"); + expect(blocking?.summary).toContain("app-reader"); + expect(blocked.ready).toBe(false); + expect(blocked.plan.map((step) => step.id).slice(0, 2)).toEqual([ + "snapshot", + "secret-store", + ]); + const step = blocked.plan[1]; + expect(step?.command).toBe("hack secrets retire-store --allow-unchecked"); + expect(step?.action).toContain("restore the store's key file"); + expect(JSON.stringify(blocked)).not.toContain(SENTINEL); +}); + +test("the check changes nothing and reports no value", async () => { + const legacy = await liveProject("app-legacy"); + await writeFile(join(legacy.projectDir, "hack.env.json"), "{}\n"); + await writeFile(join(legacy.projectDir, ".env"), `TOKEN=${SENTINEL}\n`, { + mode: 0o600, + }); + await writeFile( + join(legacy.projectDir, ".env.production"), + `A=${SENTINEL}\n` + ); + await writeRegistry([legacy, entry("app-dead")]); + const before = await tree(root); + const report = await check(healthyProbes()); + expect(await tree(root)).toEqual(before); + expect(JSON.stringify(report)).not.toContain(SENTINEL); +}); + +test("probes that are disabled or fail report not checked, never a pass", async () => { + await writeRegistry([await liveProject("app-a")]); + const report = await check(defaultUpgradeCheckProbes()); + const byId = new Map(report.machine.map((f) => [f.id, f])); + expect(byId.get("daemon")?.summary).toContain("Not checked"); + expect(byId.get("global-infra")?.summary).toContain("Not checked"); + expect(byId.get("docker")?.status).toBe("warn"); + // Disabled probes leave the check incomplete, never ready. + expect(report.host_probes).toBe("disabled"); + expect(byId.get("host-probes")?.status).toBe("incomplete"); + expect(report.ready).toBe(false); + // A probe that throws or hangs does not stop the check. + const failing = await check( + healthyProbes({ + daemon: async () => { + throw new Error("boom"); + }, + }) + ); + // A proxy ports probe that fails while global services run is reported. + const proxyFailed = await check( + healthyProbes({ + proxyPorts: async () => { + throw new Error("boom"); + }, + }) + ); + expect(proxyFailed.machine.find((f) => f.id === "proxy-ports")?.status).toBe( + "incomplete" + ); + expect(proxyFailed.ready).toBe(false); + const daemon = failing.machine.find((f) => f.id === "daemon"); + expect(daemon?.summary).toContain("Not checked"); + // A probe that failed (not one disabled on purpose) leaves the check + // unperformed: not ready. + expect(daemon?.status).toBe("incomplete"); + expect(failing.ready).toBe(false); +}); + +test("tracked machine-local files and a missing .hack/.gitignore are steps, a tracked secret key blocks", async () => { + const project = await liveProject("app-git"); + const git = (args: string[]) => + Bun.spawnSync(["git", "-C", project.repoRoot, ...args], { + env: fixtureGitEnv({ env: { HOME: home } }), + }); + git(["init", "-q"]); + await writeFile(join(project.projectDir, ".env"), `TOKEN=${SENTINEL}\n`); + git(["add", "-f", ".hack/.env", ".hack/hack.config.json"]); + git(["commit", "-q", "-m", "fixture"]); + await writeRegistry([project]); + const report = await check(healthyProbes()); + const findings = new Map( + report.projects[0]?.findings.map((f) => [f.id, f]) ?? [] + ); + expect(findings.get("tracked-files")?.status).toBe("warn"); + expect(findings.get("tracked-files")?.command).toBe( + `hack doctor --fix --path ${project.repoRoot}` + ); + expect(findings.get("ignore-rules")?.status).toBe("warn"); + + await writeFile(join(project.repoRoot, ".hack.secret.key"), "x".repeat(43)); + git(["add", "-f", ".hack.secret.key"]); + git(["commit", "-q", "-m", "key"]); + const blocked = await check(healthyProbes()); + expect(blocked.ready).toBe(false); + expect( + blocked.blockers.some( + (b) => b.scope === "app-git" && b.id === "tracked-files" + ) + ).toBe(true); + expect(JSON.stringify(blocked)).not.toContain(SENTINEL); +}); + +test("a checkout Git fails on is incomplete, not a directory without Git", async () => { + if (process.getuid?.() === 0) { + return; // root reads an unreadable index anyway + } + const project = await liveProject("app-broken-git"); + const git = (args: string[]) => + Bun.spawnSync(["git", "-C", project.repoRoot, ...args], { + env: fixtureGitEnv({ env: { HOME: home } }), + }); + git(["init", "-q"]); + git(["add", "-f", ".hack/hack.config.json"]); + git(["commit", "-q", "-m", "fixture"]); + const index = join(project.repoRoot, ".git", "index"); + await chmod(index, 0o000); + await writeRegistry([project]); + try { + const report = await check(healthyProbes()); + const tracked = report.projects[0]?.findings.find( + (f) => f.id === "tracked-files" + ); + expect(tracked?.status).toBe("incomplete"); + expect(report.ready).toBe(false); + } finally { + await chmod(index, 0o644); + } +}); + +test("a checkout whose Git is broken leaves tracked files and ignore rules incomplete", async () => { + const project = await liveProject("app-broken-dotgit"); + // A .git that git cannot use: inspections fail fast (exit 128). + await mkdir(join(project.repoRoot, ".git")); + await writeRegistry([project]); + const report = await check(healthyProbes()); + const byId = new Map( + report.projects[0]?.findings.map((f) => [f.id, f.status]) ?? [] + ); + expect(byId.get("tracked-files")).toBe("incomplete"); + expect(byId.get("ignore-rules")).toBe("incomplete"); + expect(report.ready).toBe(false); +}); + +test("an expired budget starts no probe and no registry read", async () => { + await writeRegistry([await liveProject("app-a")]); + let calls = 0; + const healthy = healthyProbes(); + const counted = + (run: (...args: A) => Promise) => + (...args: A): Promise => { + calls += 1; + return run(...args); + }; + const report = await runUpgradeCheck({ + cliVersion: "5.0.0-next.17", + homeDir: home, + budgetMs: 0, + probes: { + ...healthy, + pathHack: counted(healthy.pathHack), + daemon: counted(healthy.daemon), + docker: counted(healthy.docker), + globalInfra: counted(healthy.globalInfra), + linuxDns: counted(healthy.linuxDns), + }, + }); + expect(calls).toBe(0); + expect(report.registry.state).toBe("unreadable"); + expect(report.ready).toBe(false); +}); + +test("a probe that would spawn after the deadline starts no child", async () => { + const pids = join(root, "late-pids"); + const bin = join(root, "late-bin"); + await mkdir(bin, { recursive: true }); + await writeFile( + join(bin, "late-probe"), + `#!/bin/sh\necho "$$" >> '${pids}'\nexec sleep 60\n`, + { mode: 0o755 } + ); + await chmod(join(bin, "late-probe"), 0o755); + const { exec } = await import("../src/lib/shell.ts"); + await runUpgradeCheck({ + cliVersion: "5.0.0-next.17", + homeDir: home, + budgetMs: 500, + probes: healthyProbes({ + daemon: async () => { + await Bun.sleep(1000); + await exec([join(bin, "late-probe")], { stdin: "ignore" }); + throw new Error("the late probe has no result"); + }, + }), + }); + // Give the late probe time to try. + await Bun.sleep(1200); + expect(await readFile(pids, "utf8").catch(() => "")).toBe(""); +}, 20_000); + +test("a reader key that cannot be established is incomplete, never retire advice", async () => { + const reader = await liveProject("app-unknown-key"); + await writeRegistry([reader]); + const { storePath, keyPath } = await globalStore({ reader }); + // The global key file is wrong; the reader names its own key file. + await writeFile(keyPath, "synthetic-wrong-global-key\n", { mode: 0o600 }); + const ownKey = join(root, "unknown-own.key"); + await writeFile( + join(reader.projectDir, "hack.config.json"), + `${JSON.stringify({ + name: "app-unknown-key", + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { path: storePath, keyPath: ownKey }, + }, + }, + })}\n` + ); + // Its key file is missing and the keychain fallback is on: not asked. + Reflect.deleteProperty(process.env, "HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK"); + const uninspected = await check(healthyProbes()); + process.env.HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK = "1"; + const first = uninspected.machine.find((f) => f.id === "secret-store"); + expect(first?.status).toBe("incomplete"); + expect(first?.args).toEqual(["doctor"]); + expect(uninspected.ready).toBe(false); + + // Its key file exists but cannot be read. + if (process.getuid?.() !== 0) { + await writeFile(ownKey, `${STORE_KEY}\n`, { mode: 0o600 }); + await chmod(ownKey, 0o000); + try { + const unreadable = await check(healthyProbes()); + const second = unreadable.machine.find((f) => f.id === "secret-store"); + expect(second?.status).toBe("incomplete"); + expect(second?.args).toEqual(["doctor"]); + } finally { + await chmod(ownKey, 0o600); + } + } +}); + +test("a snapshot that cannot cover everything says so", async () => { + // A malformed registry: no project is covered. + await writeFile(join(hackHome, "projects.json"), "{ not json"); + const malformed = await collectUpgradeSnapshotTargets(); + expect(malformed.gaps.map((g) => g.name)).toContain("projects registry"); + + // An unreadable live checkout, and a skipped (linked) file. + const live = await liveProject("app-live"); + await symlink( + join(root, "elsewhere.env"), + join(live.projectDir, "hack.env.json") + ); + const broken = await liveProject("app-broken"); + await writeFile(join(broken.projectDir, "hack.config.json"), "{ not json"); + await writeRegistry([live, broken]); + const partial = await collectUpgradeSnapshotTargets(); + const names = partial.gaps.map((g) => g.name); + expect(names).toContain("app-broken"); + expect(names).toContain("project app-live: .hack/hack.env.json"); +}); + +test("a registered linked worktree is checked on its own and can block", async () => { + const project = await liveProject("app-wt"); + const worktreeRoot = join(root, "worktrees", "feature"); + const worktreeDir = join(worktreeRoot, ".hack"); + await mkdir(worktreeDir, { recursive: true }); + await writeFile( + join(worktreeDir, "hack.config.json"), + `${JSON.stringify({ name: "app-wt" })}\n` + ); + await symlink( + join(root, "nowhere.yaml"), + join(worktreeDir, "hack.env.default.yaml") + ); + await writeRegistry([ + { + ...project, + worktrees: [ + { + path: worktreeRoot, + branch: "feature", + lastSeenAt: "2026-01-01T00:00:00.000Z", + }, + ], + }, + ]); + const report = await check(healthyProbes()); + expect(report.registry.live).toBe(2); + expect(report.ready).toBe(false); + expect(report.blockers.map((b) => b.scope)).toEqual([ + "app-wt (worktree feature)", + ]); +}); + +test("printed commands run the candidate launcher when the check ran through it", async () => { + const legacy = await liveProject("app-legacy"); + await writeFile(join(legacy.projectDir, "hack.env.json"), "{}\n"); + await writeRegistry([legacy]); + const report = await runUpgradeCheck({ + cliVersion: "5.0.0-next.17", + homeDir: home, + probes: healthyProbes(), + env: { + HACK_CANDIDATE_ENTRYPOINT: "hack-next", + HACK_RUNTIME_BACKEND: "compose", + }, + }); + const launcher = `HACK_RUNTIME_BACKEND=compose ${join(home, ".hack-next", "bin", "hack-next")}`; + expect(report.cli.command).toBe(launcher); + expect(report.next_command).toBe(`${launcher} upgrade snapshot`); + for (const step of report.plan) { + expect((step.command ?? `${launcher} `).startsWith(`${launcher} `)).toBe( + true + ); + } +}); + +/** A global encrypted_file store with one entry, and its key file. */ +async function globalStore(opts: { + readonly reader: Entry; + readonly entries?: Readonly>; +}) { + const storePath = join(hackHome, "secrets.enc.json"); + const keyPath = join(hackHome, "secrets-file.key"); + await writeFile( + process.env.HACK_GLOBAL_CONFIG_PATH ?? "", + `${JSON.stringify({ + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { path: storePath, keyPath }, + }, + }, + })}\n` + ); + await writeFile( + join(opts.reader.projectDir, "hack.env.json"), + `${JSON.stringify({ version: 1, vars: [{ key: "TOKEN", source: "keychain" }] })}\n` + ); + await writeFile(keyPath, `${STORE_KEY}\n`, { mode: 0o600 }); + await chmod(keyPath, 0o600); + const store = await resolveSecretStore({ + projectName: opts.reader.name, + projectDir: opts.reader.projectDir, + }); + for (const [key, value] of Object.entries( + opts.entries ?? { TOKEN: SENTINEL } + )) { + await store.set({ key, value }); + } + return { storePath, keyPath }; +} + +test("the global key is audited with the runtime's rules, not only by decrypting", async () => { + const reader = await liveProject("app-reader"); + await writeRegistry([reader]); + const { keyPath } = await globalStore({ reader }); + const storeFinding = async () => + (await check(healthyProbes())).machine.find((f) => f.id === "secret-store"); + + expect((await storeFinding())?.status).toBe("ok"); + + // A key others can read decrypts the store, but the runtime refuses it. + await chmod(keyPath, 0o644); + const permissive = await storeFinding(); + expect(permissive?.status).toBe("blocker"); + expect(permissive?.summary).toContain("refused at runtime"); + expect(permissive?.action).toContain("chmod 600"); + await chmod(keyPath, 0o600); + + // The same key tracked by Git is refused too. + const repo = join(root, "keys-repo"); + await mkdir(repo, { recursive: true }); + const trackedKey = join(repo, "secrets-file.key"); + await writeFile(trackedKey, `${STORE_KEY}\n`, { mode: 0o600 }); + await chmod(trackedKey, 0o600); + const git = (args: string[]) => + Bun.spawnSync(["git", "-C", repo, ...args], { + env: fixtureGitEnv({ env: { HOME: home } }), + }); + git(["init", "-q"]); + git(["add", "-f", "secrets-file.key"]); + git(["commit", "-q", "-m", "key"]); + const config = JSON.parse( + await readFile(process.env.HACK_GLOBAL_CONFIG_PATH ?? "", "utf8") + ) as { controlPlane: { secrets: { encryptedFile: { keyPath: string } } } }; + config.controlPlane.secrets.encryptedFile.keyPath = trackedKey; + await writeFile( + process.env.HACK_GLOBAL_CONFIG_PATH ?? "", + `${JSON.stringify(config)}\n` + ); + const tracked = await storeFinding(); + expect(tracked?.status).toBe("blocker"); + expect(tracked?.action).toContain("git tracks it"); +}); + +test("a dead registry entry does not leave a healthy store check incomplete", async () => { + // Codex review: a deleted checkout counted as an unchecked store reader, + // so a healthy machine could not be ready until stale entries were pruned. + const reader = await liveProject("app-reader"); + await writeRegistry([reader, entry("app-deleted")]); + await globalStore({ reader }); + const report = await check(healthyProbes()); + const store = report.machine.find((f) => f.id === "secret-store"); + expect(store?.status).toBe("ok"); + expect(report.incomplete).toEqual([]); + // The dead entry is reported as a registry step, not as store uncertainty. + expect(report.machine.find((f) => f.id === "registry-dead")?.status).toBe( + "warn" + ); +}); + +test("a project that names its own working key is not blocked by a broken global key", async () => { + // Codex review: the global key file decided for every reader, though a + // project may select the global store with its own keyPath. + const reader = await liveProject("app-own-key"); + await writeRegistry([reader]); + const { storePath, keyPath } = await globalStore({ reader }); + const ownKey = join(root, "own.key"); + await writeFile(ownKey, `${STORE_KEY}\n`, { mode: 0o600 }); + await chmod(ownKey, 0o600); + await writeFile( + join(reader.projectDir, "hack.config.json"), + `${JSON.stringify({ + name: "app-own-key", + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { path: storePath, keyPath: ownKey }, + }, + }, + })}\n` + ); + // The global key file is now wrong. + await writeFile(keyPath, "synthetic-wrong-global-key\n", { mode: 0o600 }); + const report = await check(healthyProbes()); + const store = report.machine.find((f) => f.id === "secret-store"); + expect(store?.status).toBe("warn"); + expect(store?.summary).toContain("app-own-key"); + expect(report.ready).toBe(true); + + // A second reader on the broken global key is blocked, and retiring is + // not suggested while another configured key opens the store. + const other = await liveProject("app-global-key"); + await writeFile( + join(other.projectDir, "hack.env.json"), + `${JSON.stringify({ version: 1, vars: [{ key: "TOKEN", source: "keychain" }] })}\n` + ); + await writeRegistry([reader, other]); + const blocked = await check(healthyProbes()); + const blocking = blocked.machine.find((f) => f.id === "secret-store"); + expect(blocking?.status).toBe("blocker"); + expect(blocking?.summary).toContain("app-global-key"); + expect(blocking?.args).toEqual(["doctor"]); + expect(blocking?.action).toContain("app-global-key"); + expect(JSON.stringify(blocked)).not.toContain(STORE_KEY); +}); + +test("a reader whose own key does not open the store blocks even when the global key works", async () => { + // Codex review: reader keys were checked only when the global key failed. + const reader = await liveProject("app-wrong-own-key"); + await writeRegistry([reader]); + const { storePath } = await globalStore({ reader }); + const ownKey = join(root, "wrong-own.key"); + await writeFile(ownKey, "synthetic-wrong-own-key\n", { mode: 0o600 }); + await chmod(ownKey, 0o600); + await writeFile( + join(reader.projectDir, "hack.config.json"), + `${JSON.stringify({ + name: "app-wrong-own-key", + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { path: storePath, keyPath: ownKey }, + }, + }, + })}\n` + ); + const report = await check(healthyProbes()); + const store = report.machine.find((f) => f.id === "secret-store"); + expect(store?.status).toBe("blocker"); + expect(store?.summary).toContain("app-wrong-own-key"); + expect(report.ready).toBe(false); +}); + +test("the global store's states: no store, an env key, a keychain not asked", async () => { + const reader = await liveProject("app-reader"); + await writeRegistry([reader]); + const { storePath, keyPath } = await globalStore({ reader }); + const finding = async (env?: Record) => { + const report = await runUpgradeCheck({ + cliVersion: "5.0.0-next.17", + homeDir: home, + probes: healthyProbes(), + // Only the variables the check reads; never the runner's environment. + env: { ...env }, + }); + return { + report, + store: report.machine.find((f) => f.id === "secret-store"), + }; + }; + // HACK_SECRETS_FILE_KEY from this shell opens it; the value never shows. + await rm(keyPath); + const viaEnv = await finding({ HACK_SECRETS_FILE_KEY: STORE_KEY }); + expect(viaEnv.store?.status).toBe("ok"); + expect(viaEnv.store?.summary).toContain("HACK_SECRETS_FILE_KEY"); + expect(JSON.stringify(viaEnv.report)).not.toContain(STORE_KEY); + expect(JSON.stringify(viaEnv.report)).not.toContain(SENTINEL); + + // No key file and the keychain fallback on: not asked, so not established. + Reflect.deleteProperty(process.env, "HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK"); + const uninspected = await finding(); + expect(uninspected.store?.status).toBe("incomplete"); + expect(uninspected.report.ready).toBe(false); + process.env.HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK = "1"; + + // No store file at all is not a store that lost its key. + await rm(storePath); + const none = await finding(); + expect(none.store?.status).toBe("ok"); + expect(none.store?.summary).toContain("No global secret store file"); +}); + +test("the snapshot covers overlays only the store holds, and reports when it cannot list them", async () => { + const reader = await liveProject("app-reader"); + await writeRegistry([reader]); + const { keyPath } = await globalStore({ + reader, + entries: { TOKEN: SENTINEL, "env.staging.TOKEN": SENTINEL }, + }); + const covered = await collectUpgradeSnapshotTargets(); + const labels = covered.targets.map((t) => t.label); + expect(labels).toContain("project app-reader: .hack/hack.env.staging.yaml"); + expect(labels).toContain( + "project app-reader: .hack/hack.env.staging.local.yaml" + ); + expect(covered.gaps).toEqual([]); + expect(JSON.stringify(covered)).not.toContain(SENTINEL); + + // Only the keychain could open the store: listing it would prompt. + await rm(keyPath); + Reflect.deleteProperty(process.env, "HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK"); + const gapped = await collectUpgradeSnapshotTargets(); + process.env.HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK = "1"; + expect(gapped.gaps).toEqual([ + { name: "app-reader", reason: expect.stringContaining("OS keychain") }, + ]); +}); + +test("the snapshot keeps a project's own secret store and key", async () => { + // Codex review: migration cleanup can remove a project-selected store and + // key; only the global backend was kept. + const project = await liveProject("app-own-store"); + const storePath = join(project.projectDir, "secrets.enc.json"); + const keyPath = join(project.repoRoot, ".store.key"); + await writeFile( + join(project.projectDir, "hack.config.json"), + `${JSON.stringify({ + name: "app-own-store", + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { path: storePath, keyPath }, + }, + }, + })}\n` + ); + await writeFile(storePath, `{"sentinel":"${SENTINEL}"}\n`, { mode: 0o600 }); + await writeFile(keyPath, `${STORE_KEY}\n`, { mode: 0o600 }); + await writeRegistry([project]); + const collected = await collectUpgradeSnapshotTargets(); + const byLabel = new Map(collected.targets.map((t) => [t.label, t.path])); + expect(byLabel.get("project app-own-store: secret store")).toBe(storePath); + expect(byLabel.get("project app-own-store: secret store key")).toBe(keyPath); + expect(JSON.stringify(collected)).not.toContain(SENTINEL); +}); + +test("a project's own secret store is checked with the key it selects", async () => { + // Codex review: only the global store was checked, so a project-selected + // store with a wrong key passed. + const project = await liveProject("app-own-store"); + const storePath = join(project.projectDir, "secrets.enc.json"); + const keyPath = join(project.repoRoot, ".store.key"); + await writeFile( + join(project.projectDir, "hack.config.json"), + `${JSON.stringify({ + name: "app-own-store", + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { path: storePath, keyPath }, + }, + }, + })}\n` + ); + await writeFile( + join(project.projectDir, "hack.env.json"), + `${JSON.stringify({ version: 1, vars: [{ key: "TOKEN", source: "keychain" }] })}\n` + ); + await writeFile(keyPath, `${STORE_KEY}\n`, { mode: 0o600 }); + await chmod(keyPath, 0o600); + await ( + await resolveSecretStore({ + projectName: "app-own-store", + projectDir: project.projectDir, + }) + ).set({ key: "TOKEN", value: SENTINEL }); + await writeRegistry([project]); + const good = await check(healthyProbes()); + const finding = (r: UpgradeCheckReport) => + r.projects[0]?.findings.find((f) => f.id === "project-store"); + expect(finding(good)?.status).toBe("ok"); + + await writeFile(keyPath, "synthetic-wrong-project-key\n", { mode: 0o600 }); + const bad = await check(healthyProbes()); + expect(finding(bad)?.status).toBe("blocker"); + expect(bad.ready).toBe(false); + expect(JSON.stringify(bad)).not.toContain(SENTINEL); + expect(JSON.stringify(bad)).not.toContain(STORE_KEY); +}); + +test("a refused project key's remedy quotes a hostile path", async () => { + // Second-family review: the runtime's "Run: chmod 600 " interpolated a + // repo-controlled key path into shell text. + const project = await liveProject("app-hostile-key"); + const storePath = join(project.projectDir, "secrets.enc.json"); + const keyPath = join( + project.repoRoot, + "key it's here; printf REVIEW_MARKER; #" + ); + await writeFile( + join(project.projectDir, "hack.config.json"), + `${JSON.stringify({ + name: "app-hostile-key", + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { path: storePath, keyPath }, + }, + }, + })}\n` + ); + await writeFile( + join(project.projectDir, "hack.env.json"), + `${JSON.stringify({ version: 1, vars: [{ key: "TOKEN", source: "keychain" }] })}\n` + ); + await writeFile(keyPath, `${STORE_KEY}\n`, { mode: 0o600 }); + await chmod(keyPath, 0o600); + await ( + await resolveSecretStore({ + projectName: "app-hostile-key", + projectDir: project.projectDir, + }) + ).set({ key: "TOKEN", value: SENTINEL }); + await chmod(keyPath, 0o644); + await writeRegistry([project]); + const report = await check(healthyProbes()); + const finding = report.projects[0]?.findings.find( + (f) => f.id === "project-store" + ); + expect(finding?.status).toBe("blocker"); + const command = (finding?.action ?? "").split("Run: ")[1] ?? ""; + // Split it the way a shell does: exactly chmod, 600 and that file. + const split = Bun.spawnSync([ + "sh", + "-c", + `eval "set -- $1"; printf '%s\\0' "$@"`, + "sh", + command, + ]); + const out = split.stdout.toString(); + expect(out).not.toContain("REVIEW_MARKER\0"); + expect(out.split("\0").slice(0, -1)).toEqual(["chmod", "600", keyPath]); +}); + +test("Docker answering but the container listing failing is incomplete", async () => { + await writeRegistry([await liveProject("app-a")]); + const report = await check( + healthyProbes({ + globalInfra: async () => ({ + installed: true, + running: null, + listFailed: true, + }), + }) + ); + expect(report.machine.find((f) => f.id === "global-infra")?.status).toBe( + "incomplete" + ); + expect(report.ready).toBe(false); +}); + +test("the snapshot covers the configured default overlay with a keychain backend", async () => { + // Codex review: with no file store active, only .env. files were + // considered, though the migration also converts the configured default + // overlay. + const project = await liveProject("app-keychain"); + await writeFile( + join(project.projectDir, "hack.config.json"), + `${JSON.stringify({ name: "app-keychain", env: { defaultOverlay: "staging" } })}\n` + ); + await writeFile(join(project.projectDir, "hack.env.json"), "{}\n"); + await writeRegistry([project]); + const collected = await collectUpgradeSnapshotTargets(); + const labels = collected.targets.map((t) => t.label); + expect(labels).toContain("project app-keychain: .hack/hack.env.staging.yaml"); + expect(labels).toContain( + "project app-keychain: .hack/hack.env.staging.local.yaml" + ); +}); + +test("store-only overlays are found under a name set in the legacy hack.toml", async () => { + // Codex review: the snapshot read the name from hack.config.json only. + const project = entry("checkout-dir"); + await mkdir(project.projectDir, { recursive: true }); + await writeFile( + join(project.projectDir, "hack.toml"), + 'name = "toml-named"\n' + ); + await writeFile( + join(project.projectDir, "docker-compose.yml"), + "services: {}\n" + ); + await writeRegistry([project]); + // A global store holding a staging-only value for "toml-named". + const storePath = join(hackHome, "secrets.enc.json"); + const keyPath = join(hackHome, "secrets-file.key"); + await writeFile( + process.env.HACK_GLOBAL_CONFIG_PATH ?? "", + `${JSON.stringify({ + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { path: storePath, keyPath }, + }, + }, + })}\n` + ); + await writeFile( + join(project.projectDir, "hack.env.json"), + `${JSON.stringify({ version: 1, vars: [{ key: "TOKEN", source: "keychain" }] })}\n` + ); + await writeFile(keyPath, `${STORE_KEY}\n`, { mode: 0o600 }); + await chmod(keyPath, 0o600); + await ( + await resolveSecretStore({ + projectName: "toml-named", + projectDir: project.projectDir, + }) + ).set({ key: "env.staging.TOKEN", value: SENTINEL }); + const collected = await collectUpgradeSnapshotTargets(); + expect(collected.targets.map((t) => t.label)).toContain( + "project checkout-dir: .hack/hack.env.staging.yaml" + ); +}); + +test("a project directory that cannot be listed is a snapshot gap, not an empty inventory", async () => { + if (process.getuid?.() === 0) { + return; // root lists any directory + } + const project = await liveProject("app-unlisted"); + await writeFile(join(project.projectDir, ".env"), `TOKEN=${SENTINEL}\n`); + await writeRegistry([project]); + // Searchable (its config still reads) but not listable. + await chmod(project.projectDir, 0o300); + try { + const collected = await collectUpgradeSnapshotTargets(); + expect(collected.gaps).toContainEqual({ + name: "app-unlisted", + reason: expect.stringContaining("could not be listed"), + }); + } finally { + await chmod(project.projectDir, 0o755); + } +}); + +test("env layer secrets must open under the project env key", async () => { + // Codex review: a missing or wrong env key passed as "none needed yet". + const project = await liveProject("app-env-secrets"); + await setProjectEnvValue({ + projectRoot: project.repoRoot, + projectDir: project.projectDir, + envName: null, + scope: "global", + key: "TOKEN", + value: SENTINEL, + secret: true, + }); + await writeRegistry([project]); + const finding = (r: UpgradeCheckReport) => + r.projects[0]?.findings.find((f) => f.id === "env-secrets"); + const good = await check(healthyProbes()); + expect(finding(good)?.status).toBe("ok"); + + const keyFile = join(project.repoRoot, ".hack.secret.key"); + const key = await readFile(keyFile, "utf8"); + await rm(keyFile); + const missing = await check(healthyProbes()); + expect(finding(missing)?.status).toBe("blocker"); + expect(missing.ready).toBe(false); + + await writeFile(keyFile, `${"A".repeat(43)}\n`, { mode: 0o600 }); + await chmod(keyFile, 0o600); + const wrong = await check(healthyProbes()); + expect(finding(wrong)?.status).toBe("blocker"); + for (const report of [good, missing, wrong]) { + expect(JSON.stringify(report)).not.toContain(SENTINEL); + expect(JSON.stringify(report)).not.toContain(key.trim()); + } +}); + +test("a config path that is a FIFO is refused at once, never waited on", async () => { + const project = entry("app-fifo"); + await mkdir(project.projectDir, { recursive: true }); + Bun.spawnSync(["mkfifo", join(project.projectDir, "hack.config.json")]); + await writeRegistry([project]); + const started = Date.now(); + const report = await check(healthyProbes()); + expect(Date.now() - started).toBeLessThan(10_000); + expect(report.registry.unreadable.map((e) => e.name)).toEqual(["app-fifo"]); + expect(report.ready).toBe(false); +}); + +test("a hanging git leaves the required check incomplete, and its children are ended", async () => { + const project = await liveProject("app-hang"); + const bin = join(root, "fake-bin"); + await mkdir(bin, { recursive: true }); + const pids = join(root, "git-pids"); + await writeFile( + join(bin, "git"), + `#!/bin/sh\necho "$$ $*" >> '${pids}'\nexec sleep 60\n`, + { mode: 0o755 } + ); + await chmod(join(bin, "git"), 0o755); + await writeRegistry([project]); + process.env.PATH = `${bin}:${process.env.PATH ?? ""}`; + const started = Date.now(); + const report = await runUpgradeCheck({ + cliVersion: "5.0.0-next.17", + homeDir: home, + probes: healthyProbes(), + budgetMs: 4000, + checkTimeoutMs: 1500, + }); + expect(Date.now() - started).toBeLessThan(10_000); + const findings = new Map( + report.projects[0]?.findings.map((f) => [f.id, f]) ?? [] + ); + expect(findings.get("tracked-files")?.status).toBe("incomplete"); + expect(report.ready).toBe(false); + expect(report.incomplete.some((i) => i.id === "tracked-files")).toBe(true); + // Every git the check started is ended by the check's own deadline. + const alive = async () => + (await readFile(pids, "utf8").catch(() => "")) + .split("\n") + .filter((line) => line.trim() !== "") + .filter((line) => { + try { + process.kill(Number(line.split(" ")[0]), 0); + return true; + } catch { + return false; + } + }); + // Each check's children end with the check: allow only kill latency. + const until = Date.now() + 3000; + while ((await alive()).length > 0 && Date.now() < until) { + await Bun.sleep(250); + } + expect(await alive()).toEqual([]); +}, 40_000); + +test("a host probe's children end with the probe, not after the report", async () => { + // Codex review: the probe deadline stopped waiting but left children. + const pids = join(root, "probe-pids"); + const bin = join(root, "probe-bin"); + await mkdir(bin, { recursive: true }); + await writeFile( + join(bin, "slow-probe"), + `#!/bin/sh\necho "$$" >> '${pids}'\nexec sleep 60\n`, + { mode: 0o755 } + ); + await chmod(join(bin, "slow-probe"), 0o755); + const { exec } = await import("../src/lib/shell.ts"); + const started = Date.now(); + await runUpgradeCheck({ + cliVersion: "5.0.0-next.17", + homeDir: home, + budgetMs: 3000, + probes: healthyProbes({ + // A probe that runs a hanging command with no limit of its own. + daemon: async () => { + await exec([join(bin, "slow-probe")], { stdin: "ignore" }); + throw new Error("unreachable"); + }, + }), + }); + expect(Date.now() - started).toBeLessThan(8000); + const until = Date.now() + 3000; + const alive = async () => + (await readFile(pids, "utf8").catch(() => "")) + .split("\n") + .filter((pid) => pid.trim() !== "") + .filter((pid) => { + try { + process.kill(Number(pid), 0); + return true; + } catch { + return false; + } + }); + while ((await alive()).length > 0 && Date.now() < until) { + await Bun.sleep(200); + } + expect(await alive()).toEqual([]); +}, 20_000); + +test("candidate commands keep a path with spaces, quotes and ' hack ' intact", async () => { + const name = "my hack repo's"; + const project = await liveProject(name); + await writeFile(join(project.projectDir, "hack.env.json"), "{}\n"); + await writeRegistry([project]); + const weirdHome = join(root, "home with space's"); + const report = await runUpgradeCheck({ + cliVersion: "5.0.0-next.17", + homeDir: weirdHome, + probes: healthyProbes(), + env: { + HACK_CANDIDATE_ENTRYPOINT: "hack-next", + HACK_RUNTIME_BACKEND: "compose", + }, + }); + const step = report.plan.find((s) => s.id === "env-format"); + expect(step?.args).toEqual([ + "doctor", + "--migrate-env-config", + "--path", + project.repoRoot, + ]); + // Split the command the way a shell does: every argument survives. + const split = Bun.spawnSync([ + "sh", + "-c", + `eval "set -- $1"; printf '%s\\0' "$@"`, + "sh", + step?.command ?? "", + ]); + const words = split.stdout.toString().split("\0").slice(0, -1); + expect(words).toEqual([ + "HACK_RUNTIME_BACKEND=compose", + join(weirdHome, ".hack-next", "bin", "hack-next"), + "doctor", + "--migrate-env-config", + "--path", + project.repoRoot, + ]); +}); + +test("a step with no command is manual: its action is never shown as a command", async () => { + await writeFile(join(hackHome, "projects.json"), "{ not json"); + const report = await check(healthyProbes()); + const registry = report.plan.find((step) => step.id === "registry"); + expect(registry?.command).toBeNull(); + expect(registry?.args).toBeNull(); + expect(registry?.action).toContain("Repair"); + expect(report.ready).toBe(false); + // The first step is the snapshot, a real command. + expect(report.next_command).toBe("hack upgrade snapshot"); + expect(report.next_action).toBeNull(); +}); + +// --------------------------------------------------------------------------- +// CLI +// --------------------------------------------------------------------------- + +async function cli(args: readonly string[]) { + const child = Bun.spawn([process.execPath, ENTRYPOINT, ...args], { + cwd: root, + env: privateCliEnv({ + root, + home, + env: { + HACK_HOME: hackHome, + HACK_GLOBAL_CONFIG_PATH: join(hackHome, "hack.config.json"), + HACK_SYSTEMD_DISABLE: "1", + HACK_HOST_PROBES_DISABLE: "1", + HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK: "1", + HACK_NO_INTERACTIVE: "1", + }, + }), + stdin: "ignore", + stdout: "pipe", + stderr: "pipe", + }); + const [stdout, stderr, code] = await Promise.all([ + new Response(child.stdout).text(), + new Response(child.stderr).text(), + child.exited, + ]); + return { code, stdout, stderr, output: `${stdout}${stderr}` }; +} + +test("hack upgrade check exits 1 naming the blockers, and prints the next command", async () => { + const blocked = await liveProject("app-blocked"); + await symlink( + join(root, "nowhere.yaml"), + join(blocked.projectDir, "hack.env.default.yaml") + ); + await writeRegistry([blocked]); + const json = await cli(["upgrade", "check", "--json"]); + expect(json.code).toBe(1); + const parsed = JSON.parse(json.stdout) as { + ok: boolean; + data: UpgradeCheckReport; + }; + expect(parsed.ok).toBe(true); + expect(parsed.data.ready).toBe(false); + expect(parsed.data.blockers[0]?.scope).toBe("app-blocked"); + + const text = await cli(["upgrade", "check"]); + expect(text.code).toBe(1); + expect(text.stdout).toContain("Blocked by: app-blocked env-format"); + expect(text.stdout).toContain("Next: hack upgrade snapshot"); + expect(text.stdout).toContain("[blocker] hack doctor --path"); +}, 60_000); + +test("hack upgrade snapshot, then rollback restores exact bytes; a changed file needs --overwrite", async () => { + const project = await liveProject("app-a"); + const envPath = join(project.projectDir, ".env"); + await writeFile(envPath, `TOKEN=${SENTINEL}\n`, { mode: 0o600 }); + await chmod(envPath, 0o600); + const configPath = join(project.projectDir, "hack.config.json"); + await writeRegistry([project]); + const envBefore = await readFile(envPath); + const configBefore = await readFile(configPath); + + const made = await cli(["upgrade", "snapshot", "--json"]); + expect(made.code).toBe(0); + const snapshot = ( + JSON.parse(made.stdout) as { + data: { + manifest_path: string; + rollback_command: string; + files: { label: string }[]; + }; + } + ).data; + const labels = snapshot.files.map((f) => f.label); + for (const label of [ + "global: projects.json", + "project app-a: .hack/.env", + "project app-a: .hack/hack.config.json", + // The env migration's write set, recorded even while absent. + "project app-a: .hack/docker-compose.yml", + "project app-a: .hack/hack.env.default.yaml", + "project app-a: .hack/.gitignore", + ]) { + expect(labels).toContain(label); + } + expect(made.output).not.toContain(SENTINEL); + expect(await readFile(snapshot.manifest_path, "utf8")).not.toContain( + SENTINEL + ); + + // Change one file, delete another and create a layer, as the env + // migration does. + await writeFile(configPath, '{"name":"app-a","v":5}\n'); + await rm(envPath); + const layerPath = join(project.projectDir, "hack.env.default.yaml"); + await writeFile(layerPath, "vars: {}\n"); + + // Without --overwrite, the changed config is refused; the deleted .env is + // put back only with --overwrite too (no step recorded what it wrote). + const refused = await cli([ + "upgrade", + "rollback", + "--rollback-manifest", + snapshot.manifest_path, + ]); + expect(refused.code).toBe(1); + expect(refused.stdout).toContain("conflict"); + expect(await readFile(configPath, "utf8")).toBe('{"name":"app-a","v":5}\n'); + + const restored = await cli([ + "upgrade", + "rollback", + "--rollback-manifest", + snapshot.manifest_path, + "--overwrite", + "--json", + ]); + expect(restored.code).toBe(0); + expect(await readFile(envPath)).toEqual(envBefore); + expect(await readFile(configPath)).toEqual(configBefore); + // The created layer is moved into the snapshot, not left in place. + expect(await lstat(layerPath).catch(() => null)).toBeNull(); + expect(restored.output).not.toContain(SENTINEL); + + // Without a manifest it lists snapshots and changes nothing. + const listed = await cli(["upgrade", "rollback"]); + expect(listed.code).toBe(1); + expect(listed.stdout).toContain("Choose a snapshot to restore"); + expect(listed.stdout).toContain(snapshot.manifest_path); +}, 60_000); diff --git a/tests/upgrade-snapshot.test.ts b/tests/upgrade-snapshot.test.ts new file mode 100644 index 000000000..8b647d4d1 --- /dev/null +++ b/tests/upgrade-snapshot.test.ts @@ -0,0 +1,668 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { + chmod, + lstat, + mkdir, + mkdtemp, + open, + readdir, + readFile, + realpath, + rm, + stat, + symlink, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { + createUpgradeSnapshot, + listUpgradeSnapshots, + readUpgradeSnapshot, + recordUpgradeSnapshotAfter, + restoreUpgradeSnapshot, + type UpgradeSnapshot, +} from "../src/lib/upgrade-snapshot.ts"; + +// A value that must never appear in a manifest, result or message. +const SENTINEL = ["sentinel", "upgrade", "value", "7f3a91"].join("-"); + +let root = ""; +let work = ""; +let snapshots = ""; + +beforeEach(async () => { + root = await realpath( + await mkdtemp(join(tmpdir(), "hack-upgrade-snapshot-")) + ); + work = join(root, "work"); + snapshots = join(root, "private", "snapshots"); + await mkdir(work, { recursive: true }); +}); + +afterEach(async () => { + await rm(root, { recursive: true, force: true }); +}); + +async function fixture() { + const env = join(work, "project", ".env"); + const config = join(work, "project", "hack.config.json"); + const created = join(work, "project", ".hack", "env.yaml"); + await mkdir(join(work, "project", ".hack"), { recursive: true }); + await writeFile(env, `TOKEN=${SENTINEL}\n`, { mode: 0o600 }); + await chmod(env, 0o600); + await writeFile(config, '{"name":"app-a"}\n', { mode: 0o644 }); + await chmod(config, 0o644); + return { env, config, created }; +} + +async function snapshotOf(paths: { + env: string; + config: string; + created: string; +}) { + const created = await createUpgradeSnapshot({ + purpose: "env-migration app-a", + root: snapshots, + targets: [ + { path: paths.env, label: "project app-a: .env" }, + { path: paths.config, label: "project app-a: hack.config.json" }, + { path: paths.created, label: "project app-a: .hack/env.yaml" }, + ], + now: new Date("2026-11-13T12:00:00Z"), + }); + if (created.status !== "created") { + throw new Error(`snapshot refused: ${JSON.stringify(created)}`); + } + return created.snapshot; +} + +async function mode(path: string): Promise { + return (await stat(path)).mode & 0o7777; +} + +test("snapshot, mutate, delete and create, then restore puts back exact bytes and modes", async () => { + const paths = await fixture(); + const envBefore = await readFile(paths.env); + const configBefore = await readFile(paths.config); + const snapshot = await snapshotOf(paths); + + // The step: delete .env, rewrite the config (and its mode), create a file. + await rm(paths.env); + await writeFile(paths.config, '{"name":"app-a","v":5}\n'); + await chmod(paths.config, 0o600); + await writeFile(paths.created, "env: migrated\n", { mode: 0o600 }); + const recorded = await recordUpgradeSnapshotAfter({ + manifestPath: snapshot.manifestPath, + }); + expect(recorded.status).toBe("recorded"); + + const restored = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + }); + expect(restored.status).toBe("restored"); + expect( + Object.fromEntries( + restored.status === "invalid-manifest" + ? [] + : restored.results.map((r) => [r.label, r.status]) + ) + ).toEqual({ + "project app-a: .env": "restored", + "project app-a: hack.config.json": "restored", + "project app-a: .hack/env.yaml": "moved-aside", + }); + expect(await readFile(paths.env)).toEqual(envBefore); + expect(await mode(paths.env)).toBe(0o600); + expect(await readFile(paths.config)).toEqual(configBefore); + expect(await mode(paths.config)).toBe(0o644); + // The created file is gone from its path but kept, never deleted. + expect(await lstat(paths.created).catch(() => null)).toBeNull(); + const moved = + restored.status === "invalid-manifest" + ? undefined + : restored.results.find((r) => r.status === "moved-aside"); + expect(moved?.keptAs).toBeDefined(); + expect(await readFile(moved?.keptAs ?? "", "utf8")).toBe("env: migrated\n"); + // The replaced config is kept too. + const replaced = + restored.status === "invalid-manifest" + ? undefined + : restored.results.find((r) => r.label.endsWith("hack.config.json")); + expect(await readFile(replaced?.keptAs ?? "", "utf8")).toBe( + '{"name":"app-a","v":5}\n' + ); + + // Idempotent: a second restore changes nothing. + const again = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + }); + expect(again.status).toBe("restored"); + expect( + again.status === "invalid-manifest" + ? [] + : again.results.map((r) => r.status) + ).toEqual(["already-restored", "already-restored", "already-restored"]); +}); + +test("a target changed since the step recorded it is refused, and kept when overwritten", async () => { + const paths = await fixture(); + const envBefore = await readFile(paths.env); + const snapshot = await snapshotOf(paths); + await writeFile(paths.env, "TOKEN=written-by-step\n"); + await recordUpgradeSnapshotAfter({ manifestPath: snapshot.manifestPath }); + // A later edit, after the step. + await writeFile(paths.env, "TOKEN=edited-later\n"); + + const refused = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + paths: [paths.env], + }); + expect(refused.status).toBe("incomplete"); + const entry = + refused.status === "invalid-manifest" ? undefined : refused.results[0]; + expect(entry?.status).toBe("conflict"); + expect(entry?.message).toContain("changed since the upgrade step wrote it"); + expect(await readFile(paths.env, "utf8")).toBe("TOKEN=edited-later\n"); + + const forced = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + paths: [paths.env], + overwrite: true, + }); + const done = + forced.status === "invalid-manifest" ? undefined : forced.results[0]; + expect(done?.status).toBe("restored"); + expect(await readFile(paths.env)).toEqual(envBefore); + expect(await readFile(done?.keptAs ?? "", "utf8")).toBe( + "TOKEN=edited-later\n" + ); +}); + +test("without a recorded after state, a changed target needs overwrite", async () => { + const paths = await fixture(); + const snapshot = await snapshotOf(paths); + await writeFile(paths.config, "{}\n"); + const refused = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + paths: [paths.config], + }); + const entry = + refused.status === "invalid-manifest" ? undefined : refused.results[0]; + expect(entry?.status).toBe("conflict"); + expect(entry?.message).toContain("did not record what it wrote"); + expect(await readFile(paths.config, "utf8")).toBe("{}\n"); +}); + +test("a target changed during the restore is left as it is", async () => { + const paths = await fixture(); + const snapshot = await snapshotOf(paths); + await writeFile(paths.config, '{"step":true}\n'); + await recordUpgradeSnapshotAfter({ manifestPath: snapshot.manifestPath }); + const result = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + paths: [paths.config], + beforeReplace: async (path) => { + await writeFile(path, '{"racing":true}\n'); + }, + }); + const entry = + result.status === "invalid-manifest" ? undefined : result.results[0]; + expect(entry?.status).toBe("conflict"); + expect(await readFile(paths.config, "utf8")).toBe('{"racing":true}\n'); +}); + +test("a file published at the target after it was displaced is never overwritten, and the step's output is kept", async () => { + // Second-family review: a replacement that landed between the last check + // and the final rename was overwritten, and only the earlier bytes kept. + const paths = await fixture(); + const snapshot = await snapshotOf(paths); + await writeFile(paths.config, '{"step":true}\n'); + await writeFile(paths.created, "env: migrated\n"); + await recordUpgradeSnapshotAfter({ manifestPath: snapshot.manifestPath }); + const result = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + paths: [paths.config, paths.created], + afterDisplace: async (path) => { + await writeFile(path, `late edit of ${path}\n`); + }, + }); + const byPath = new Map( + result.status === "invalid-manifest" + ? [] + : result.results.map((r) => [r.path, r]) + ); + // The late file stays; the step's output was kept before displacement. + expect(byPath.get(paths.config)?.status).toBe("conflict"); + expect(await readFile(paths.config, "utf8")).toBe( + `late edit of ${paths.config}\n` + ); + expect(await readFile(byPath.get(paths.config)?.keptAs ?? "", "utf8")).toBe( + '{"step":true}\n' + ); + // An absent-before target: the late file is not removed, and the target + // is not reported as restored (the snapshot holds absence). + expect(byPath.get(paths.created)?.status).toBe("conflict"); + expect(result.status).toBe("incomplete"); + expect(await readFile(paths.created, "utf8")).toBe( + `late edit of ${paths.created}\n` + ); + expect(await readFile(byPath.get(paths.created)?.keptAs ?? "", "utf8")).toBe( + "env: migrated\n" + ); + // No sibling staging or displaced file is left behind. + expect( + (await readdir(join(work, "project"))).filter((name) => + name.includes(".hack-") + ) + ).toEqual([]); +}); + +test("a late edit just before displacement goes back in place and is kept, for a created file too", async () => { + const paths = await fixture(); + const snapshot = await snapshotOf(paths); + await writeFile(paths.created, "env: migrated\n"); + await recordUpgradeSnapshotAfter({ manifestPath: snapshot.manifestPath }); + const result = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + paths: [paths.created], + beforeReplace: async (path) => { + await writeFile(path, "env: edited later\n"); + }, + }); + const entry = + result.status === "invalid-manifest" ? undefined : result.results[0]; + // The late edit is linked back into place: the same file, untouched. + expect(entry?.status).toBe("conflict"); + expect(await readFile(paths.created, "utf8")).toBe("env: edited later\n"); +}); + +test("when the snapshot's working directories cannot be prepared, nothing is touched", async () => { + const paths = await fixture(); + const snapshot = await snapshotOf(paths); + await writeFile(paths.config, '{"step":true}\n'); + await recordUpgradeSnapshotAfter({ manifestPath: snapshot.manifestPath }); + // A file where the snapshot's displaced/ directory would go. + await writeFile(join(snapshot.dir, "displaced"), "", { mode: 0o600 }); + const result = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + paths: [paths.config], + }); + const entry = + result.status === "invalid-manifest" ? undefined : result.results[0]; + expect(entry?.status).toBe("error"); + expect(await readFile(paths.config, "utf8")).toBe('{"step":true}\n'); +}); + +test("no temporary or displaced file ever appears in the checkout", async () => { + // Codex review: a sibling beside the target (for example + // `..env.hack-displaced-…`) matched no ignore rule, so Git could commit + // its secret bytes. + const paths = await fixture(); + const snapshot = await snapshotOf(paths); + await rm(paths.env); + await writeFile(paths.created, "env: migrated\n"); + await recordUpgradeSnapshotAfter({ manifestPath: snapshot.manifestPath }); + const dir = join(work, "project"); + const seen: string[][] = []; + const result = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + beforeReplace: async () => { + seen.push(await readdir(dir)); + }, + afterDisplace: async () => { + seen.push(await readdir(dir)); + }, + }); + expect(result.status).toBe("restored"); + for (const names of [...seen, await readdir(dir)]) { + expect( + names.filter( + (name) => ![".env", "hack.config.json", ".hack"].includes(name) + ) + ).toEqual([]); + } +}); + +test("a parent directory reached through a symlink from another filesystem counts by where it leads", async () => { + // Second-family review: lstat compared the symlink's own filesystem. + const shm = "/dev/shm"; + const shmInfo = await lstat(shm).catch(() => null); + if (shmInfo === null || shmInfo.dev === (await lstat(work)).dev) { + return; // no second filesystem to test with on this host + } + const shmRoot = await mkdtemp(join(shm, "hack-upgrade-snapshot-")); + try { + const paths = await fixture(); + const linkedDir = join(shmRoot, "linked-project"); + await symlink(join(work, "project"), linkedDir); + const viaLink = join(linkedDir, "hack.config.json"); + const created = await createUpgradeSnapshot({ + purpose: "symlinked parent", + root: snapshots, + targets: [{ path: viaLink, label: "config" }], + }); + if (created.status !== "created") { + throw new Error("snapshot refused"); + } + const before = await readFile(paths.config); + await writeFile(paths.config, '{"step":true}\n'); + await recordUpgradeSnapshotAfter({ + manifestPath: created.snapshot.manifestPath, + }); + const result = await restoreUpgradeSnapshot({ + manifestPath: created.snapshot.manifestPath, + }); + expect(result.status).toBe("restored"); + expect(await readFile(paths.config)).toEqual(before); + } finally { + await rm(shmRoot, { recursive: true, force: true }); + } +}); + +test("a target on another filesystem than the snapshot is refused untouched", async () => { + const shm = "/dev/shm"; + const shmInfo = await lstat(shm).catch(() => null); + if (shmInfo === null || shmInfo.dev === (await lstat(work)).dev) { + return; // no second filesystem to test with on this host + } + const shmRoot = await mkdtemp(join(shm, "hack-upgrade-snapshot-")); + try { + const paths = await fixture(); + const created = await createUpgradeSnapshot({ + purpose: "cross-filesystem", + root: join(shmRoot, "snapshots"), + targets: [{ path: paths.config, label: "config" }], + }); + if (created.status !== "created") { + throw new Error("snapshot refused"); + } + await writeFile(paths.config, '{"step":true}\n'); + await recordUpgradeSnapshotAfter({ + manifestPath: created.snapshot.manifestPath, + }); + const result = await restoreUpgradeSnapshot({ + manifestPath: created.snapshot.manifestPath, + }); + const entry = + result.status === "invalid-manifest" ? undefined : result.results[0]; + expect(entry?.status).toBe("error"); + expect(entry?.message).toContain("another filesystem"); + expect(await readFile(paths.config, "utf8")).toBe('{"step":true}\n'); + // Nothing was staged or displaced beside the target. + expect( + (await readdir(join(work, "project"))).filter( + (name) => name.includes(".hack-") || name.startsWith("restore-") + ) + ).toEqual([]); + } finally { + await rm(shmRoot, { recursive: true, force: true }); + } +}); + +test("a writer holding the displaced file open keeps its edit in a named file", async () => { + // Second-family review: copying the displaced bytes and unlinking the + // file lost an edit made through an already-open descriptor. + const paths = await fixture(); + const snapshot = await snapshotOf(paths); + await writeFile(paths.config, '{"step":true}\n'); + await recordUpgradeSnapshotAfter({ manifestPath: snapshot.manifestPath }); + let handle: Awaited> | null = null; + const result = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + paths: [paths.config], + beforeReplace: async (path) => { + handle = await open(path, "r+"); + }, + afterDisplace: async () => { + // The open descriptor still reaches the displaced file. + await handle?.write('{"written":"late"}\n', 0); + await handle?.truncate(19); + await handle?.close(); + }, + }); + const entry = + result.status === "invalid-manifest" ? undefined : result.results[0]; + expect(entry?.status).toBe("restored"); + expect(await readFile(paths.config, "utf8")).toBe('{"name":"app-a"}\n'); + expect(await readFile(entry?.keptAs ?? "", "utf8")).toBe( + '{"written":"late"}\n' + ); +}); + +test("when the snapshot's copy cannot be published, the displaced file's location is reported", async () => { + if (process.getuid?.() === 0) { + return; // root writes into a read-only directory anyway + } + const paths = await fixture(); + const snapshot = await snapshotOf(paths); + await writeFile(paths.config, '{"step":true}\n'); + await recordUpgradeSnapshotAfter({ manifestPath: snapshot.manifestPath }); + const dir = join(work, "project"); + let result: Awaited>; + try { + result = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + paths: [paths.config], + afterDisplace: async () => { + await chmod(dir, 0o500); + }, + }); + } finally { + await chmod(dir, 0o755); + } + const entry = + result.status === "invalid-manifest" ? undefined : result.results[0]; + expect(entry?.status).toBe("error"); + // Neither published nor put back (the directory refused both): the step's + // file is kept and named. + expect(await readFile(entry?.keptAs ?? "", "utf8")).toBe('{"step":true}\n'); +}); + +test("a regular file that becomes unreadable mid-restore goes back in place", async () => { + // Codex review: a regular file that could not be read after displacement + // was treated as a non-file and left out of place. + if (process.getuid?.() === 0) { + return; // root reads any file + } + const paths = await fixture(); + const snapshot = await snapshotOf(paths); + await writeFile(paths.config, '{"step":true}\n'); + await recordUpgradeSnapshotAfter({ manifestPath: snapshot.manifestPath }); + const result = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + paths: [paths.config], + beforeReplace: async (path) => { + await chmod(path, 0o000); + }, + }); + const entry = + result.status === "invalid-manifest" ? undefined : result.results[0]; + expect(entry?.status).toBe("conflict"); + // The file is back at its path, untouched. + await chmod(paths.config, 0o644); + expect(await readFile(paths.config, "utf8")).toBe('{"step":true}\n'); +}); + +test("a restore of a path the snapshot does not hold is refused, not reported as restored", async () => { + const paths = await fixture(); + const snapshot = await snapshotOf(paths); + await expect( + restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + paths: [join(work, "typo.env")], + }) + ).rejects.toThrow("not a target"); +}); + +test("no value reaches the manifest, the results or the messages; storage is private", async () => { + const paths = await fixture(); + const snapshot = await snapshotOf(paths); + await rm(paths.env); + await recordUpgradeSnapshotAfter({ manifestPath: snapshot.manifestPath }); + const restored = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + }); + const manifestText = await readFile(snapshot.manifestPath, "utf8"); + expect(manifestText).not.toContain(SENTINEL); + expect(JSON.stringify(snapshot)).not.toContain(SENTINEL); + expect(JSON.stringify(restored)).not.toContain(SENTINEL); + expect( + JSON.stringify(await listUpgradeSnapshots({ root: snapshots })) + ).not.toContain(SENTINEL); + // The bytes are kept only in private blobs. + expect(await mode(snapshots)).toBe(0o700); + expect(await mode(snapshot.dir)).toBe(0o700); + expect(await mode(join(snapshot.dir, "blobs"))).toBe(0o700); + expect(await mode(snapshot.manifestPath)).toBe(0o600); + for (const name of await readdir(join(snapshot.dir, "blobs"))) { + expect(await mode(join(snapshot.dir, "blobs", name))).toBe(0o600); + } +}); + +test("a target that cannot be snapshotted refuses the whole snapshot and writes nothing", async () => { + const paths = await fixture(); + const linked = join(work, "project", "linked.env"); + await symlink(paths.env, linked); + const directory = join(work, "project", ".hack"); + for (const targets of [ + [{ path: linked, label: "link" }], + [{ path: directory, label: "dir" }], + [{ path: "relative/.env", label: "relative" }], + [ + { path: paths.env, label: "a" }, + { path: paths.env, label: "b" }, + ], + [{ path: join(snapshots, "x"), label: "inside" }], + ]) { + const result = await createUpgradeSnapshot({ + purpose: "t", + root: snapshots, + targets: [{ path: paths.config, label: "ok" }, ...targets], + }); + expect(result.status).toBe("refused"); + } + expect(await lstat(snapshots).catch(() => null)).toBeNull(); +}); + +test("a snapshot root inside a Git working tree is refused", async () => { + const paths = await fixture(); + const repo = join(root, "repo"); + await mkdir(join(repo, ".git"), { recursive: true }); + const result = await createUpgradeSnapshot({ + purpose: "t", + root: join(repo, "snapshots"), + targets: [{ path: paths.env, label: "env" }], + }); + expect(result.status === "refused" && result.reason === "root").toBe(true); + expect( + result.status === "refused" && result.reason === "root" + ? result.rootReason + : null + ).toBe("inside-working-tree"); +}); + +test("a snapshot root reached through a symlink into a Git working tree is refused", async () => { + // Codex review: only the lexical ancestors were checked, so a symlinked + // hack home inside a dotfiles checkout received the snapshot. + const paths = await fixture(); + const repo = join(root, "dotfiles"); + await mkdir(join(repo, ".git"), { recursive: true }); + await mkdir(join(repo, "hack-home"), { recursive: true }); + const linkedHome = join(root, "linked-hack-home"); + await symlink(join(repo, "hack-home"), linkedHome); + const result = await createUpgradeSnapshot({ + purpose: "t", + root: join(linkedHome, "upgrade", "snapshots"), + targets: [{ path: paths.env, label: "env" }], + }); + expect( + result.status === "refused" && result.reason === "root" + ? result.rootReason + : null + ).toBe("inside-working-tree"); + // Nothing was created inside the checkout. + expect(await readdir(join(repo, "hack-home"))).toEqual([]); +}); + +test("Git selected through the environment refuses snapshot and restore", async () => { + // Codex security review: GIT_DIR/GIT_WORK_TREE can make $HOME a working + // tree with no .git marker above the snapshot root. + const paths = await fixture(); + const ready = await snapshotOf(paths); + await writeFile(paths.config, '{"step":true}\n'); + await recordUpgradeSnapshotAfter({ manifestPath: ready.manifestPath }); + const cases = ["GIT_DIR", "GIT_WORK_TREE"].flatMap((key) => + // Git takes " " as a literal path; an empty value is refused too. + [join(root, "elsewhere"), " ", ""].map((value) => ({ key, value })) + ); + for (const { key, value } of cases) { + const saved = process.env[key]; + process.env[key] = value; + try { + const created = await createUpgradeSnapshot({ + purpose: "t", + root: join(root, "other-snapshots"), + targets: [{ path: paths.env, label: "env" }], + }); + expect( + created.status === "refused" && created.reason === "root" + ? created.rootReason + : null + ).toBe("inside-working-tree"); + const restored = await restoreUpgradeSnapshot({ + manifestPath: ready.manifestPath, + }); + expect( + restored.status === "invalid-manifest" + ? null + : restored.results.find((r) => r.path === paths.config)?.status + ).toBe("error"); + expect(await readFile(paths.config, "utf8")).toBe('{"step":true}\n'); + } finally { + if (saved === undefined) { + Reflect.deleteProperty(process.env, key); + } else { + process.env[key] = saved; + } + } + } + expect( + await lstat(join(root, "other-snapshots")).catch(() => null) + ).toBeNull(); +}); + +test("a manifest that is not private, or a damaged copy, restores nothing", async () => { + const paths = await fixture(); + const envBefore = await readFile(paths.env); + const snapshot: UpgradeSnapshot = await snapshotOf(paths); + await writeFile(paths.env, "TOKEN=step\n"); + await recordUpgradeSnapshotAfter({ manifestPath: snapshot.manifestPath }); + + await chmod(snapshot.manifestPath, 0o644); + expect( + (await readUpgradeSnapshot({ manifestPath: snapshot.manifestPath })).status + ).toBe("invalid"); + expect( + (await restoreUpgradeSnapshot({ manifestPath: snapshot.manifestPath })) + .status + ).toBe("invalid-manifest"); + await chmod(snapshot.manifestPath, 0o600); + + // Damage the kept copy of .env: its digest no longer matches. + const entry = snapshot.manifest.entries.find((e) => e.path === paths.env); + const digest = entry?.before.kind === "file" ? entry.before.sha256 : ""; + await writeFile(join(snapshot.dir, "blobs", digest), "damaged\n", { + mode: 0o600, + }); + const result = await restoreUpgradeSnapshot({ + manifestPath: snapshot.manifestPath, + paths: [paths.env], + }); + expect( + result.status === "invalid-manifest" ? null : result.results[0]?.status + ).toBe("backup-damaged"); + expect(await readFile(paths.env, "utf8")).toBe("TOKEN=step\n"); + expect(envBefore.toString()).toContain(SENTINEL); +});