diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e766575..cf9be2c9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,188 @@ # Changelog +## 0.14.3 — 2026-10-14 + +**An agent no longer loses its key to the 0.13 → by-id upgrade.** An XMesh review of `lib/config.js` 0.14.2 returned +"do not ship" with 13 findings. Each was checked by running it in a sandbox state dir; 0.14.3 fixes the ones that lose +or expose a key and the small ones beside them, by reading the state of an identity only from records committed in one +atomic step (see "Identity and on-disk state"). The wire is unchanged; the public surface gains `relocation.abortExport` +and `IdentityExportPendingError`. + +### What was wrong + +- **A new identity could overwrite the old key.** Moving a 0.13 node (`nodes//`) to `nodes/by-id//` was + refused when that directory already existed, even if it was empty or held only a `lock.pid` (something else made it: + a lock, an interrupted run). The refusal left the node unindexed, and the next start by name did not stop there: + it took "refused" for "no such agent", made a NEW identity, moved the old directory onto the new node's path and + wrote the new `identity.json` over the old one. The agent came back under the same name with a different nodeId and + key, and its peers read what it signed as forged. A start pinned by `nodeId` was stuck instead, refusing to start + every time. +- **A 0.13 key file stayed world-readable.** 0.13 wrote `identity.json` and `e2e-keypair.json` without a mode (0644 + under the usual umask). The migration only renames, so they kept it, in a directory at 0755. A new identity in 0.14 is + written 0600; a migrated one never was. +- **A name whose identity file was missing made a replacement agent.** If the name index named a node and its + `identity.json` was not there (the `SYM_IDENTITY_DIR` volume not mounted at boot, the variable set by one embedder and + not another), a start by name made a new nodeId and key and re-pointed the name to it. +- **A start pinned by nodeId never migrated.** `loadIdentity` ran the migration only for a start by name alone. mesh-channel + starts every node as `{ name, nodeId, create: false }`, so a node it pinned to the nodeId in its `node.json` and whose + files were still the 0.13 `nodes//` could not start at all (`EIDENTITYABSENT`) until something started it by name + once; with `create: true` (the default) 0.14.2 made a new identity and lost the key. 0.14.3 loads the agent. +- Smaller, same family: a stray symlink at `nodes/` was reported as already indexed when nothing was indexed; + switching `SYM_IDENTITY_DIR` on for a 0.13 node whose identity was in its node directory reported "not a node" and + went on to make a new identity (the old key survived, orphaned in the new node's directory); an identity with a public key + and no private key loaded, and failed later at its first signature; a cosmetic rewrite of `identity.json` (a bare + Windows hostname) truncated the live key file in place; a start in the moment between the directory move and the + index write found no agent and made one; the live-lock fallback skipped the tombstone check; and nothing moved + across file systems (`EXDEV`), so a node whose directories sit on different volumes could not be migrated. + +### Who is exposed + +- Anyone who upgraded a node from 0.13 to 0.14.0, 0.14.1 or 0.14.2 and whose state root had a `nodes/by-id//` + directory for that node without an `identity.json` in it at the time the node was first started by name. Their key + was destroyed on that start; the symptom is a changed nodeId under the same name and peers rejecting its blocks. + 0.14.3 cannot bring a destroyed key back. If you have the old `identity.json` (a backup, a durable + `SYM_IDENTITY_DIR` volume, a copy of `~/.sym`), put it back at `nodes/by-id//identity.json` and set the + nodeId in the name's file `nodes/by-name/.json` to the old one. +- Anyone who switched `SYM_IDENTITY_DIR` on for an existing 0.13 node: the old key is probably still on disk, in + `nodes/by-id//identity.json` (the node that was made in its place). +- Every node migrated from 0.13 holds a private key readable by other local users until it is next loaded with 0.14.3. +- A node started by its pinned nodeId (mesh-channel's start, `create: false`) on the first start after upgrading from 0.13: + no key was lost, but it could not start until something started it by name. A pinned start with `create: true` could lose it + as above. + +### Identity and on-disk state + +The first fix for the findings above was itself reviewed (R1, on the candidate 632b95d) and failed the same way twice +over: it read the state of an identity from whatever was at a path (a by-id directory that was only partly there, an +index entry that could not be read, a node directory without `identity.json`, the content of a stale lock), and each +patch for one of those cases made another. R1 proved eleven by running them. 0.14.3 changes the assumption instead: +**the state of an identity is a record committed in one atomic step**, and the code that inferred it is deleted +(`moveTree`, `mergeMove`, `sameTree`, `legacyDirsRemain`, and the migration lock `nodes/.migrating`). + +- **The 0.13 → by-id migration copies; it never moves.** The 0.13 directory (`nodes//`, and + `SYM_IDENTITY_DIR//`) is copied into a staging directory beside its destination + (`nodes/by-id/..partial--`, on the destination's volume), links followed (a node directory kept + on another volume behind a link, an `identity.json` that is a relative link to a key file), `identity.json` last, every + file synced, the copy verified byte for byte against the source, a commit record (`.sym-commit`) written, and the + whole installed by ONE rename. Of several starts doing this at once, one rename wins; the others find the same + identity installed and discard their staging. Then the 0.13 directory is renamed aside to + `nodes/.legacy-0.13/` (key files narrowed to 0600) and the compat link `nodes/` takes its place. **Nothing + is ever deleted from the source.** A staging directory left by a killed process is cleared by the next attempt. + With no migration lock there is no wait: a multi-node process no longer blocks for up to 30 s behind another + process's migration. A move across volumes is no longer a special case: no rename crosses one. +- **A by-id directory without the commit record is a husk, never "the identity is in place".** An empty one, or one + holding only a stale lock (what 0.14.0 - 0.14.2 left), is removed with `rmdir` (which cannot remove a directory with + anything in it) and the commit made; one holding files (a node of an earlier release that ran there without its + identity) has them merged into the staged copy, and is itself kept aside at `nodes/by-id/..husk`. A file + that differs, or a different `identity.json`, is a second claim on the nodeId: the start halts and nothing changes. + A husk whose lock a live process holds refuses (`EIDENTITYLOCK`). +- **A name is claimed by an exclusive create of its index entry, after the identity is installed** (written and synced + under a temp name, then hard-linked into place; an O_EXCL create where hard links are not supported). Two first starts + of a new name both make an identity; exactly one claims the name, and the other discards its identity and loads the + winner. R1 measured 2 - 3 agents for one new name in 6 of 6 rounds with three starters; now one, every round. +- **An index entry reads as "absent" only when there is none (ENOENT).** An entry that is empty, not JSON, names another + name or cannot be read (EACCES, EMFILE) halts, says where the identity it named is, and says not to delete it. Index + writes are synced (file and directory). +- **A new identity is made only when nothing on this host is a trace of the name**: no index entry, no `nodes/` + (a link, or a directory with anything but a stale lock in it), no `SYM_IDENTITY_DIR/`, no 0.13 directory set + aside, no installed identity of that name that no name indexes. A trace that provably is the name's identity (a compat + link to it; the one unindexed identity of that name: a start killed between installing it and claiming the name, or a + lost entry) is adopted: the name is claimed for it. Any other trace halts and says what it is. **Behaviour change:** a + `nodes//` holding files and no identity used to become a new agent's directory; it now halts (with + `SYM_IDENTITY_DIR`'s volume not mounted that directory is the agent's store, and the new identity took it). +- **One path for pinned and unpinned starts.** `loadIdentity({ name, nodeId, create: false })` (mesh-channel) migrates + through the same staged commit and loads the pinned agent. A pin whose identity is in the 0.13 directory and could not + be migrated halts saying so (it used to say "No identity on this host"). +- **A start pinned to a nodeId never takes a name from another identity** (R4-9, from the mesh-channel review). It used + to re-point the name's index entry to the pin: a folder whose `.sym/node.json` pins X under "reviewer" took the name + from another folder's agent Y, whose next start by name then ran as X, and a cloned repository could choose which + identity on the host a session signs as. A pinned start may claim a name nobody has, and that is all; a name another + identity holds halts (`IdentityHaltError`) with the entry left as it is (with `create: false` and the pin not on this + host, `IdentityAbsentError` saying so), and a pinned start is never answered with the name's agent. **Behaviour change:** + to give a name to the pinned identity, rename the other agent first. +- **A live 0.13 process in the agent's 0.13 directory refuses the start** (`EIDENTITYLOCK`, naming its pid). 632b95d + loaded the identity in place and a 0.14 node ran beside the 0.13 one, on an empty store. The migration holds that + directory's own lock while it copies, so a 0.13 process cannot start in it meanwhile; another 0.14.3 start that finds + the lock marked as a migration's goes on beside it. +- **A stale lock is removed only if it is still the lock judged stale.** Removing one lock is a right of its own: a claim + file named after that lock's file identity and content (`lock.pid.break-`), itself a lock. Holding it, the lock + is read again and removed only if it is the same file (dev, ino) with the same content. R1 measured two or more + simultaneous holders in 4 of 8 rounds with six starters at a SIGKILLed holder's lock. A lock file is now created whole + (linked into place), so it is never seen without its owner. +- **Export: bundle durable first, then the tombstone beside the key, then the scrub** (`lib/relocation.js`). An intent + (`EXPORTING.json`) is written beside the key first; while it is there the source refuses to start the identity + (`IdentityExportPendingError`, `EIDENTITYEXPORTING`). An export interrupted before its tombstone is resumable (run it + again) or abortable (`relocation.abortExport`, which removes the bundle it wrote); one interrupted after it is finished + by running it again (the scrub). The tombstone is in the identity directory, so with `SYM_IDENTITY_DIR` a rebuilt node + directory no longer brings a second live copy back. The scrub also removes the private key from a 0.13 copy set aside. +- **Import** is installed through the same staged commit: directories 0700, key files 0600, and with `SYM_IDENTITY_DIR` + the private key only in the identity directory (it was in the node directory too). +- `validateName` refuses names Windows cannot hold (the rule mesh-channel's `nodeNameProblem` applies: `: * ? " < > |`, + CON, PRN, AUX, NUL, COM0-9, LPT0-9 with any extension, a trailing dot or space). **Behaviour change** for an existing + agent with such a name: it must be renamed (`renameIdentity` accepts the old name). A compat link is compared with + its target as the path it resolves to, so a Windows junction is no longer re-made on every check. +- Error texts: a halt says where the key is when it is on this host (for example `nodes/by-id//identity.json` after + `SYM_IDENTITY_DIR` was switched on), and tells the operator to remove the index entry only when no copy of the key is + anywhere on this host. `IdentityAbsentError` says what is on the host instead, and is never thrown when the identity is. +- Also in 0.14.3, from the first fix: key files 0600 and their directories 0700 (narrowed on load), atomic rewrites of + `identity.json`, a public key without a private key halts, `SYM_IDENTITY_DIR` switched on for a 0.13 node whose + identity is in its node directory halts and says how to copy it. + +### Known, not fixed here + +- `migrateIdentities()` still counts and discards any error from a single node; the node's own start raises them. +- `loadIdentity({ nodeId, name, create: true })` with a nodeId this host does not hold and no trace of the name still + makes a new identity under a new nodeId. mesh-channel passes `create: false`. +- `SYM_IDENTITY_DIR` switched on for an existing node does not copy its key for you (it halts and says how). +- The 0.13 directory set aside (`nodes/.legacy-0.13/`) and a husk kept aside are never removed by sym: they hold a + second copy of the store (and of the key, 0600, until an export scrubs it). Remove them once the agent runs on 0.14.3. + On APFS and reflink file systems the copy is a clone; elsewhere it takes the store's size again on disk. +- A process killed in the two renames between setting the 0.13 directory aside and putting the compat link in its + place leaves `nodes/` absent; the next 0.14.3 start puts the link back, but a 0.13 build started first in that + state would make a new identity. A crash anywhere else leaves the 0.13 directory, or the link to the installed copy, + where a 0.13 build reads it. +- A husk with files that was claimed by one start can, in a race with another start that never saw it, end up kept at + `..husk` without being merged (nothing is lost; the node runs without those files). It needs a 0.14.0 - 0.14.2 + node that ran in that directory beside a live 0.13 process, and two concurrent first starts. +- A claim file left by a process killed while removing a stale lock is itself removed the same way, up to three levels; + below that the start refuses (`EIDENTITYLOCK`) rather than guess. A 0.13 process still reclaims a stale lock the old, + non-atomic way: a 0.13 and a 0.14 process racing at one dead holder's lock are not covered. +- `relocation.abortExport` is not on the `sym` CLI yet; `sym node export` run again resumes an interrupted export. + +### Tests + +- `tests/identity-commit.test.js` (new): 33 tests, one or more per R1 finding (R1-1 to R1-11), R4-9, the error texts + and the guards of the staged commit; 32 fail on 632b95d (the one that passes there checks that a pinned start still + claims a name nobody has). R1-1 crashes the migration at every one of its 70 steps (78 over a lock-only husk), one + start per step, and checks the 0.13 directory is byte-identical, a 0.13 reader still finds the key, and the next start + loads the same agent with its whole store. The race the multi-process fuzz found (below) is a deterministic test. +- `tests/identity-migration-by-id.test.js`: 53 tests (was 51), 14 of them fail on 632b95d. Updated, each with the reason in + the test: a husk's stale lock is no longer kept; a halt writes no index back; a non-compat link at `nodes/` is + followed and copied; a node directory with files and no identity halts; the atomic rewrite is tested on an installed + identity; across volumes nothing is renamed and the source is kept aside; an interrupted migration leaves the 0.13 + directory as it was; the migration-lock tests are replaced by no-wait tests; a live 0.13 holder refuses; a pin whose + identity is in the 0.13 directory halts instead of "absent". +- Mutation check: 27 mutants of `lib/config.js` and `lib/relocation.js`, one per guard of the new design (the verify + before the commit, the exclusive claim, ENOENT-only, the mint gate, the adoption guard, the live 0.13 holder, the + claim and the compare in the lock reclaim, the export order, the tombstone place, the import key place, junction + compare, the Windows names, the set-aside, the link-before-aside, a pinned start taking a name, the husk re-look, and + others): every one is caught. +- Multi-process fuzz (real processes, every file call stalled 0 - 4 ms at random): four starters at a 0.13 agent, + plain / over a lock-only husk / pinned / pinned over a husk / with SYM_IDENTITY_DIR / with both: 40 rounds each, 0 + failed (960 starters, all loaded the original; one identity, the key installed, the whole store, the 0.13 data kept, + every round); six starters, 30 rounds each plain and over a husk, 0 failed. This fuzz found one race in the first + version of this design (a committed directory set aside as a husk, 9 of 40 rounds over a husk; same identity, nothing + lost), fixed before release. Six starters at a SIGKILLed holder's stale lock: 0 of 20 rounds with two holders, with + and without stalls (632b95d: 3 of 8). Three first starts of one new name: 0 of 10 rounds with two agents, 0 of 6 with + SymNode (632b95d: 6 of 6). Across a real second volume (an HFS+ disk image, EXDEV) with a stale migration lock: + 4 and 6 starters, 50 rounds, every starter the original. +- kill -9 crash fuzz: the first start is killed at a random moment, a second start follows. Same volume (over a husk, + plain, with SYM_IDENTITY_DIR; pinned and not): 360 rounds, 180 of them killed within 60 ms (all mid-run), every one + the original agent with its whole store and the key on disk. Across volumes (disk image): 120 rounds, 60 killed + mid-run, every one the same (632b95d on the same image: 4 of 10 rounds started with store files missing). +- Full suite (`--test-concurrency=2`, macOS): 1568 tests, 1564 pass, 0 fail, 4 skipped; `npm run gate:stock-nodes` passed; + `npm audit`: 0 vulnerabilities. CI (Linux, Node 20 and 22) green, the real second-filesystem test included. + ## 0.14.2 — 2026-10-14 **The daemon's start lock, hardened after review.** 0.14.1 replaced an unsound inode-number guard with a start lock diff --git a/lib/config.js b/lib/config.js index dad1ebf1..f65c57c2 100644 --- a/lib/config.js +++ b/lib/config.js @@ -53,16 +53,24 @@ const IDENTITY_RULING = * an operator who is told plainly can restore it. Regenerating is the unrecoverable one. */ class IdentityHaltError extends Error { - constructor(name, idPath, reason) { + /** + * @param {string} name + * @param {string} idPath - the file the halt is about + * @param {string} reason + * @param {string} [advice] - what to do, when the general advice (restore the keypair) is not it: the key + * may be on this host, and then the advice says where, never "start over" + */ + constructor(name, idPath, reason, advice) { super( `[SYM] REFUSING TO START — cannot establish the identity of agent "${name}".\n\n` + ` ${reason}\n` + ` identity: ${idPath}\n\n` + `${IDENTITY_RULING}\n\n` + - ` Restore the keypair from your durable copy and start again. If this agent's key is ` + - `genuinely gone, it cannot return as "${name}" — peers hold its public key and will read ` + - `anything a new key signs as forged. Set SYM_IDENTITY_DIR to a durable volume so a ` + - `rebuilt machine restores the key instead of replacing it.` + (advice ? ` ${advice}` : + ` Restore the keypair from your durable copy and start again. If this agent's key is ` + + `genuinely gone, it cannot return as "${name}" — peers hold its public key and will read ` + + `anything a new key signs as forged. Set SYM_IDENTITY_DIR to a durable volume so a ` + + `rebuilt machine restores the key instead of replacing it.`) ); this.name = 'IdentityHaltError'; this.agent = name; @@ -135,7 +143,7 @@ function normalizeMdnsHostname(h) { /** * Get the data directory for a named node. A name is an index (design D9): when the name index holds * it, this is the node's by-id directory (`nodes/by-id/`); otherwise the 0.13 path - * `nodes/`, which `loadIdentity` moves to the by-id layout when it first loads it. + * `nodes/`, which `loadIdentity` copies to the by-id layout when it first loads it. * @param {string} name — node name * @returns {string} path to the node's directory */ @@ -173,28 +181,46 @@ function uuidv7() { } /** - * Validate a node name per MMP Section 3.1.2. - * Must be valid UTF-8, 1-64 bytes, printable characters only. - * @param {string} name - * @throws {Error} if name is invalid + * Why `name` cannot be a node name at all (MMP §3.1.2, and one path component), or null. + * @private */ -function validateName(name) { - if (!name || typeof name !== 'string') { - throw new Error('Node name must be a non-empty string'); - } +function basicNameProblem(name) { + if (!name || typeof name !== 'string') return 'Node name must be a non-empty string'; const byteLength = Buffer.byteLength(name, 'utf8'); - if (byteLength < 1 || byteLength > 64) { - throw new Error(`Node name must be 1-64 bytes (got ${byteLength})`); - } + if (byteLength < 1 || byteLength > 64) return `Node name must be 1-64 bytes (got ${byteLength})`; // Reject control characters (U+0000-U+001F, U+007F-U+009F) - if (/[\x00-\x1f\x7f-\x9f]/.test(name)) { - throw new Error('Node name must not contain control characters'); - } + if (/[\x00-\x1f\x7f-\x9f]/.test(name)) return 'Node name must not contain control characters'; // A name is one path component (security review): it names `nodes/`, a compat link and an // index file, so a separator or a dot-name would reach outside the state root. - if (/[\/\\]/.test(name) || name === '.' || name === '..') { - throw new Error('Node name must not contain a path separator or be "." or ".."'); - } + if (/[\/\\]/.test(name) || name === '.' || name === '..') return 'Node name must not contain a path separator or be "." or ".."'; + return null; +} + +/** + * Why Windows could not hold `name` as a directory name, or null. The rule mesh-channel's + * nodeNameProblem applies: the characters Windows refuses in a file name, the device names it reserves + * whatever the extension (CON, NUL, COM1, ...), and a trailing dot or space, which Windows drops (so two + * names would share one directory). A name is `nodes/` on every platform, and a node moved to a + * Windows host must still have one. + * @private + */ +function windowsNameProblem(name) { + if (/[:*?"<>|]/.test(name)) return 'Node name must not contain any of : * ? " < > | (not valid in a file name on every platform)'; + if (/^(con|prn|aux|nul|com[0-9]|lpt[0-9])(\..*)?$/i.test(name)) return 'Node name must not be a name Windows reserves (CON, PRN, AUX, NUL, COM0-9, LPT0-9, with or without an extension)'; + if (/[. ]$/.test(name)) return 'Node name must not end with a dot or a space (Windows drops it, so two names would share one directory)'; + return null; +} + +/** + * Validate a node name per MMP Section 3.1.2. + * Must be valid UTF-8, 1-64 bytes, printable characters only, one path component, and a name every + * platform can hold as a directory name. + * @param {string} name + * @throws {Error} if name is invalid + */ +function validateName(name) { + const problem = basicNameProblem(name) || windowsNameProblem(name); + if (problem) throw new Error(problem); } /** @@ -217,11 +243,14 @@ function generateSigningKeyPair() { /** * Raised when an identity that must already exist does not (`loadIdentity({ create: false })`): a - * host restoring a known agent can never mint a replacement silently (design D9). + * host restoring a known agent can never mint a replacement silently (design D9). It says what IS on + * this host under the name when something is (a node's files without an identity, another agent's + * identity), so "no identity" is never said of a host that holds one: that case is an IdentityHaltError. */ class IdentityAbsentError extends Error { - constructor(what) { + constructor(what, seen = []) { super(`[SYM] No identity for ${what} on this host, and none was minted (create: false). ` + + (seen.length ? `What is here instead: ${seen.join('; ')}. ` : '') + `Restore it (sym node import) or start it where it lives; minting a new one would be a different agent.`); this.name = 'IdentityAbsentError'; this.code = 'EIDENTITYABSENT'; @@ -243,6 +272,23 @@ class IdentityTombstonedError extends Error { } } +/** + * Raised for an identity whose export began and did not finish (relocation.js wrote EXPORTING.json beside + * the key, and no tombstone yet): a bundle may exist, so the source does not start it until the export is + * finished (run it again) or abandoned (`abortExport`, which removes the bundle it wrote). + */ +class IdentityExportPendingError extends Error { + constructor(nodeId, intent) { + super(`[SYM] REFUSING TO START ${nodeId}: an export of this identity` + + `${intent && intent.out ? ` to ${intent.out}` : ''}${intent && intent.startedAt ? `, begun ${new Date(intent.startedAt).toISOString()},` : ''} did not finish. ` + + 'A bundle of it may exist; started here as well, there could be two live copies. Finish the export (run `sym node export` ' + + 'again) or abandon it (relocation.abortExport({ nodeId }), which removes the bundle it wrote; delete any copy made of that file).'); + this.name = 'IdentityExportPendingError'; + this.code = 'EIDENTITYEXPORTING'; + this.intent = intent || null; + } +} + /** Where a node lives, by its nodeId (design D9): /nodes/by-id/. */ function nodeDirById(nodeId) { if (!isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); @@ -252,76 +298,384 @@ function nodeDirById(nodeId) { /** Where a node's identity.json lives by nodeId: SYM_IDENTITY_DIR/by-id/ when set, else the node dir. */ function identityDirById(nodeId) { const root = process.env.SYM_IDENTITY_DIR; + if (root && !isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); return root ? path.join(root, 'by-id', nodeId) : nodeDirById(nodeId); } const isNodeIdText = (id) => typeof id === 'string' && /^[0-9A-Za-z][0-9A-Za-z_.-]{0,127}$/.test(id) && id !== '.' && id !== '..'; const TOMBSTONE = 'TOMBSTONE.json'; +/** Beside the key while an export is under way (relocation.js): the bundle may exist, the tombstone does not yet. */ +const EXPORTING = 'EXPORTING.json'; +/** + * The commit record of a directory this release installed: written into the staging directory before the + * one rename that puts it in place, so a directory that carries it is complete. A by-id directory without + * it was made by something else (a lock, an earlier release's interrupted move) and is never taken for + * "the identity is in place" by a migration. + */ +const COMMIT_MARK = '.sym-commit'; +/** Where a migrated 0.13 directory is set aside, under the root it was in. Nothing deletes it automatically. */ +const LEGACY_ASIDE = '.legacy-0.13'; const nameIndexPath = (name) => path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); +const lstatOrNull = (p) => { try { return fs.lstatSync(p); } catch { return null; } }; +const statOrNull = (p) => { try { return fs.statSync(p); } catch { return null; } }; +const randomTag = () => crypto.randomBytes(4).toString('hex'); +/** The errors a rename gives when its target is taken (by a directory that is not empty, on Windows by any). */ +const RENAME_TAKEN = new Set(['EEXIST', 'ENOTEMPTY', 'EPERM', 'EISDIR', 'ENOTDIR']); +/** A short synchronous pause, for the few places that let another start finish a step of a few system calls. */ +const pause = (ms) => Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); + +// ── Durable files ──────────────────────────────────────────────────────── + +/** fsync a directory, so a rename or create in it survives a crash. Best effort where unsupported (Windows). */ +function fsyncDir(dir) { + let fd; + try { fd = fs.openSync(dir, 'r'); fs.fsyncSync(fd); } catch { /* not supported here */ } finally { if (fd !== undefined) try { fs.closeSync(fd); } catch { /* */ } } +} + +/** fsync a file by path. Best effort: a file this process may not open for writing is still read back by the verify. */ +function fsyncFile(file) { + let fd; + try { fd = fs.openSync(file, 'r+'); fs.fsyncSync(fd); } catch { /* not ours to open for writing */ } finally { if (fd !== undefined) try { fs.closeSync(fd); } catch { /* */ } } +} + +/** Write `text` to a new file `file` (it must not exist), synced. A failure removes what it made (it made it: O_EXCL). */ +function writeNewFileSynced(file, text, mode) { + const fd = fs.openSync(file, 'wx', mode); + try { + try { fs.writeSync(fd, text); fs.fsyncSync(fd); } finally { fs.closeSync(fd); } + fs.chmodSync(file, mode); + } catch (e) { + try { fs.unlinkSync(file); } catch { /* */ } + throw e; + } +} + +/** A temp file beside `file`, written and synced: the first step of a durable replace or an exclusive create. */ +function writeTempBeside(file, text, mode) { + const tmp = path.join(path.dirname(file), `.${path.basename(file)}.${process.pid}-${randomTag()}.tmp`); + writeNewFileSynced(tmp, text, mode); + return tmp; +} + +/** Replace `file` with `text` in one step: a synced temp file renamed over it, and the directory synced. */ +function writeFileDurable(file, text, mode = 0o600) { + const tmp = writeTempBeside(file, text, mode); + try { fs.renameSync(tmp, file); } catch (e) { try { fs.unlinkSync(tmp); } catch { /* */ } throw e; } + fsyncDir(path.dirname(file)); +} + +/** + * Create `file` holding `text`, only if no file is there: true when this call made it, false when one already + * was. The content is written and synced under a temp name and then hard-linked into place, so the file is never + * seen empty or half written; `link` fails with EEXIST when the name is taken, and of several processes exactly + * one wins. Where hard links are not supported, an O_EXCL create written in place. + */ +function createFileExclusive(file, text, mode = 0o644) { + const tmp = writeTempBeside(file, text, mode); + try { + try { + fs.linkSync(tmp, file); + return true; + } catch (e) { + if (e.code === 'EEXIST') return false; + if (!['EPERM', 'ENOTSUP', 'EOPNOTSUPP', 'ENOSYS', 'EXDEV', 'EMLINK'].includes(e.code)) throw e; + } + try { writeNewFileSynced(file, text, mode); return true; } catch (e) { if (e.code === 'EEXIST') return false; throw e; } + } finally { + try { fs.unlinkSync(tmp); } catch { /* */ } + fsyncDir(path.dirname(file)); + } +} -/** The nodeId the name index holds for `name`, or null. */ +// ── The name index (the record of which node a name is) ────────────────── + +/** + * The nodeId the name index holds for `name`; null ONLY when there is no entry (ENOENT). An entry that + * exists and cannot be read, or does not hold a nodeId, is the record of an agent that cannot be read: it + * halts (IdentityHaltError), and is never taken for "no such agent" (a start that took it so made a new + * identity while the original was on disk). + * @param {string} name + * @returns {string|null} + */ function nodeIdForName(name) { + const file = nameIndexPath(name); + let raw; try { - const r = JSON.parse(fs.readFileSync(nameIndexPath(name), 'utf8')); - return r && isNodeIdText(r.nodeId) ? r.nodeId : null; - } catch { return null; } + raw = fs.readFileSync(file, 'utf8'); + } catch (e) { + if (e.code === 'ENOENT') return null; + throw unreadableIndex(name, file, `it could not be read (${e.code || e.message})`); + } + let r = null; + try { r = JSON.parse(raw); } catch { /* below */ } + if (!r || typeof r !== 'object' || !isNodeIdText(r.nodeId) || (r.name !== undefined && r.name !== name)) { + throw unreadableIndex(name, file, raw.length ? 'it does not hold {"name", "nodeId"} for this name' : 'it is empty'); + } + return r.nodeId; +} + +function unreadableIndex(name, file, why) { + const onDisk = identitiesOnDisk().filter((i) => i.name === name).map((i) => `${i.nodeId} (${i.file})`); + const link = compatLinkNodeId(path.join(NODES_DIR, name)); + return new IdentityHaltError(name, file, + `the name index entry for "${name}" exists, but ${why}. It is the record of which node this name is, so it is not read as "no such agent". ` + + (onDisk.length || link + ? `The identity it named is still on this host: ${[...onDisk, link && `nodes/${name} leads to by-id/${link}`].filter(Boolean).join('; ')}. ` + : '') + + 'Nothing was changed.', + `Restore the entry: ${file} holds {"name": ${JSON.stringify(name)}, "nodeId": ""}. Do not delete it to "start over": ` + + 'the key it leads to is this agent.'); +} + +/** Wait (briefly) for the entry another process is creating in place, where hard links are not supported. */ +function nodeIdForNameSettled(name) { + for (let i = 0; ; i++) { + try { return nodeIdForName(name); } catch (e) { + const st = statOrNull(nameIndexPath(name)); + if (i >= 40 || !st || st.size !== 0 || Date.now() - st.mtimeMs > 2000) throw e; + pause(5); + } + } } -/** Write the name index entry (atomic) and the 0.13-compatible `nodes/` symlink (one release). */ -function indexName(name, nodeId) { +/** + * Claim `name` for `nodeId` by an exclusive create of its index entry. Of several first starts of one name, + * exactly one makes the entry; the rest find it made. Returns the nodeId the entry holds afterwards: `nodeId` + * when this call claimed the name (or it already named it), the winner's otherwise. + */ +function claimName(name, nodeId) { ensureDir(path.join(NODES_DIR, 'by-name')); - const file = nameIndexPath(name); - const tmp = `${file}.${process.pid}.tmp`; - fs.writeFileSync(tmp, JSON.stringify({ name, nodeId, at: Date.now() })); - fs.renameSync(tmp, file); - linkCompat(path.join(NODES_DIR, name), path.join('by-id', nodeId)); - const idRoot = process.env.SYM_IDENTITY_DIR; - if (idRoot && fs.existsSync(path.join(idRoot, 'by-id', nodeId))) linkCompat(path.join(idRoot, name), path.join('by-id', nodeId)); + for (let i = 0; i < 5; i++) { + if (createFileExclusive(nameIndexPath(name), JSON.stringify({ name, nodeId, at: Date.now() }))) return nodeId; + const held = nodeIdForNameSettled(name); + if (held) return held; + } + throw new Error(`[SYM] the name index entry for "${name}" keeps appearing and disappearing; nothing was changed`); } -/** `link` → `target` (relative), unless a real directory is there (a legacy layout not yet moved). */ -function linkCompat(link, target) { - try { - const st = fs.lstatSync(link); - if (st.isSymbolicLink()) { - if (fs.readlinkSync(link) === target) return; - fs.unlinkSync(link); - } else { - return; // a real dir: migration moves it first +/** + * The halt for a start pinned to `pin` under a name whose index entry names another identity (`held`). The entry + * is left as it is: a name is one agent's on this host. + */ +function nameHeldByAnother(name, pin, held) { + const there = identityInPlace(held); + return new IdentityHaltError(name, nameIndexPath(name), + `this start is pinned to ${pin}, and the name "${name}" is another agent's on this host: its index entry names ${held}` + + `${there ? '' : ' (whose identity is not installed here)'}. A pinned start does not take a name from another identity: the next start of ` + + `"${name}" by name would run as ${pin}, and ${held} would be left without its name. The index entry was left as it is.`, + `Start ${pin} under its own name, or pin the start to ${held} if that is the agent meant. To give "${name}" to ${pin}, ` + + `rename the other agent first (renameIdentity).`); +} + +/** Every nodeId some index entry names, with the names. Entries that cannot be read are left out. */ +function indexEntries() { + const out = new Map(); + const dir = path.join(NODES_DIR, 'by-name'); + let files = []; + try { files = fs.readdirSync(dir); } catch { return out; } + for (const f of files) { + if (f.startsWith('.') || !f.endsWith('.json')) continue; + try { + const r = JSON.parse(fs.readFileSync(path.join(dir, f), 'utf8')); + if (r && isNodeIdText(r.nodeId)) { + if (!out.has(r.nodeId)) out.set(r.nodeId, []); + out.get(r.nodeId).push(typeof r.name === 'string' ? r.name : decodeURIComponent(f.slice(0, -5))); + } + } catch { /* unreadable: its own name halts when started */ } + } + return out; +} + +/** The by-id roots: nodes/by-id, and SYM_IDENTITY_DIR/by-id when set. */ +function byIdRoots() { + const roots = [path.join(NODES_DIR, 'by-id')]; + if (process.env.SYM_IDENTITY_DIR) roots.push(path.join(process.env.SYM_IDENTITY_DIR, 'by-id')); + return roots; +} + +/** Every identity.json filed by nodeId on this host: { nodeId, name, file, scrubbed }. Dot-entries (staging, husks) are not identities. */ +function identitiesOnDisk() { + const out = new Map(); + for (const root of byIdRoots()) { + let dirs = []; + try { dirs = fs.readdirSync(root); } catch { continue; } + for (const d of dirs) { + if (d.startsWith('.') || !isNodeIdText(d) || out.has(d)) continue; + const file = path.join(root, d, 'identity.json'); + try { + const j = JSON.parse(fs.readFileSync(file, 'utf8')); + if (j && j.nodeId === d) out.set(d, { nodeId: d, name: j.name, file, scrubbed: !j.privateKey }); + } catch { /* none, or not readable: not an identity this scan can name */ } } - } catch { /* absent */ } - try { fs.symlinkSync(target, link, process.platform === 'win32' ? 'junction' : 'dir'); } catch { /* best effort: the index is authoritative */ } + } + return [...out.values()]; +} + +/** True when the identity of `nodeId` is where a load reads it. */ +function identityInPlace(nodeId) { + return fs.existsSync(path.join(identityDirById(nodeId), 'identity.json')); +} + +// ── The 0.13-compatible links (nodes/ -> by-id/, for one release) ── + +/** + * True when the link at `link` leads to `target` (relative to the link's directory). A Windows junction + * reads back as an absolute path (sometimes with a \\?\ prefix) where a relative target was given, so the + * two are compared as the paths they resolve to, not as text: comparing text re-made the link on every check. + */ +function linkLeadsTo(link, target) { + let got; + try { got = fs.readlinkSync(link); } catch { return false; } + if (got === target) return true; + const fold = (s) => (process.platform === 'win32' ? s.toLowerCase() : s); + const norm = (p) => fold(path.resolve(path.dirname(link), String(p).replace(/^\\\\\?\\/, ''))).replace(/[\\/]+$/, ''); + return norm(got) === norm(target); +} + +/** The nodeId a compat link at `p` (a link to `by-id/` beside it) leads to, or null when `p` is not one. */ +function compatLinkNodeId(p) { + let target; + try { target = fs.readlinkSync(p); } catch { return null; } + const m = /(?:^|[\\/])by-id[\\/]([^\\/]+)[\\/]?$/.exec(target); + if (!m || !isNodeIdText(m[1])) return null; + return linkLeadsTo(p, path.join('by-id', m[1])) ? m[1] : null; +} + +/** Remove a link (a symlink, or a Windows junction, which unlink refuses). */ +function removeLink(p) { + try { fs.unlinkSync(p); } catch (e) { if (e.code === 'ENOENT') return; fs.rmdirSync(p); } } -/** The tombstone a moved identity left, or null. */ +/** + * Make `link` lead to `by-id/`. A real directory is not replaced (a 0.13 one is set aside by the + * migration first), except an empty one (or one holding only a stale lock), which is nothing. The link is + * made under a name of its own and renamed onto the path, so a stale link is replaced in one step. Best + * effort: the index is authoritative. + */ +function linkCompat(link, nodeId) { + const target = path.join('by-id', nodeId); + const st = lstatOrNull(link); + if (st && st.isSymbolicLink() && linkLeadsTo(link, target)) return true; + if (st && !st.isSymbolicLink() && !(st.isDirectory() && removeEmptyDir(link))) return false; + const tmp = path.join(path.dirname(link), `.${path.basename(link)}.link-${process.pid}-${randomTag()}`); + try { + fs.symlinkSync(target, tmp, process.platform === 'win32' ? 'junction' : 'dir'); + } catch { return false; } + try { + fs.renameSync(tmp, link); + return true; + } catch { + try { removeLink(tmp); } catch { /* */ } + try { + if (lstatOrNull(link) && lstatOrNull(link).isSymbolicLink()) removeLink(link); + fs.symlinkSync(target, link, process.platform === 'win32' ? 'junction' : 'dir'); + return true; + } catch { return false; } + } +} + +/** The compat links of `name`: nodes/, and SYM_IDENTITY_DIR/ when the identity is there. */ +function ensureCompatLinks(name, nodeId) { + linkCompat(path.join(NODES_DIR, name), nodeId); + const idRoot = process.env.SYM_IDENTITY_DIR; + if (idRoot && fs.existsSync(path.join(idRoot, 'by-id', nodeId))) linkCompat(path.join(idRoot, name), nodeId); +} + +/** + * Remove `dir` if it holds nothing but a lock nobody holds: rmdir removes only an empty directory, so this can + * never remove one with anything in it. Returns true when it is gone. + */ +function removeEmptyDir(dir) { + const lockPath = path.join(dir, 'lock.pid'); + const lock = readLockFile(lockPath); + if (lock) { + if (lock.pid === process.pid || lockIsHeldByLiveProcess(lock)) return false; + removeStaleLock(lockPath, lock); + } + try { fs.rmdirSync(dir); return true; } catch (e) { return e.code === 'ENOENT'; } +} + +/** The tombstone a moved identity left, or null: beside the key, or (written by 0.14.0 - 0.14.2) in the node dir. */ function readTombstone(nodeId) { - try { return JSON.parse(fs.readFileSync(path.join(nodeDirById(nodeId), TOMBSTONE), 'utf8')); } catch { return null; } + for (const d of new Set([identityDirById(nodeId), nodeDirById(nodeId)])) { + try { return JSON.parse(fs.readFileSync(path.join(d, TOMBSTONE), 'utf8')); } catch { /* none here */ } + } + return null; +} + +/** The record of an export that has begun and not finished, or null (beside the key; see relocation.js). */ +function readExportIntent(nodeId) { + for (const d of new Set([identityDirById(nodeId), nodeDirById(nodeId)])) { + try { return JSON.parse(fs.readFileSync(path.join(d, EXPORTING), 'utf8')); } catch { /* none here */ } + } + return null; } +/** The files in a node's directory that hold a PRIVATE KEY: its signing identity and the 0.13 E2E keypair. */ +const PRIVATE_KEY_FILES = ['identity.json', 'e2e-keypair.json']; + /** - * Read and validate an identity.json. Never regenerates a key: a file that exists and cannot be read - * halts (IdentityHaltError). A pre-v0.3.7 identity with no keypair gets its first key; a private key - * with no public key gets the public key derived from it. - * @private + * Narrow a node's private key files to their owner (0600), and with `dirs` the directories that hold + * them (0700). A key file a 0.13 node wrote at the umask default (0644 — world-readable) stays + * world-readable in the 0.13 directory the migration sets aside; this is what ends that. + * + * Best effort on purpose: a file this process does not own, or a filesystem without modes (Windows), + * must not stop a node from starting. Nothing here changes what a file holds. + * @param {Array} dirs */ -function readIdentityFile(idPath, name) { - let raw; +function securePrivateFiles(dirs, { narrowDirs = false } = {}) { + const narrow = (p, mode) => { try { if ((fs.statSync(p).mode & 0o077) !== 0) fs.chmodSync(p, mode); } catch { /* absent, or not ours */ } }; + for (const d of new Set(dirs.filter(Boolean))) { + for (const f of PRIVATE_KEY_FILES) narrow(path.join(d, f), 0o600); + if (narrowDirs) narrow(d, 0o700); + } +} + +/** Write a file the way a key file must be written: a private temp file, synced, renamed over the target. */ +function writeFileAtomic0600(file, text) { + const tmp = `${file}.${process.pid}.tmp`; + let fd; try { - raw = fs.readFileSync(idPath, 'utf8'); + fd = fs.openSync(tmp, 'w', 0o600); + fs.writeSync(fd, text); + fs.fsyncSync(fd); + fs.closeSync(fd); + fd = undefined; + fs.renameSync(tmp, file); + fsyncDir(path.dirname(file)); } catch (e) { - // The file EXISTS and we cannot read it. Never regenerate here — a transient read failure - // (permissions, a full disk, a partial write) would otherwise overwrite a live keypair. - throw new IdentityHaltError(name, idPath, `identity.json exists but could not be read: ${e.message}`); + if (fd !== undefined) try { fs.closeSync(fd); } catch { /* */ } + try { fs.unlinkSync(tmp); } catch { /* */ } + throw e; } - let identity; +} + +// ── Reading an identity ────────────────────────────────────────────────── + +/** + * Parse and validate the text of an identity.json, writing nothing. Never regenerates a key: text that is + * not an identity halts (IdentityHaltError), and so does one with a public key and no private key. A + * pre-v0.3.7 identity with no keypair at all gets its first key; a private key with no public key gets the + * public key derived from it; a bare hostname gets `.local`. `source` is what the file says, `identity` what + * it becomes, `mutated` whether they differ. + * @private + */ +function parseIdentity(raw, name, idPath) { + let source; try { - identity = JSON.parse(raw); + source = JSON.parse(raw); } catch (e) { throw new IdentityHaltError(name, idPath, `identity.json is not valid JSON: ${e.message}`); } - if (!identity || typeof identity !== 'object' || !identity.nodeId) { + if (!source || typeof source !== 'object' || !source.nodeId) { throw new IdentityHaltError(name, idPath, 'identity.json has no nodeId — the file is not an identity'); } + // A PUBLIC KEY BUT NO PRIVATE KEY is not "no keypair": the key existed and is gone (a truncated write, + // a scrubbed export). Neither mint one nor sign with `undefined`: a new key is a different agent. + if (source.publicKey && !source.privateKey) { + throw new IdentityHaltError(name, idPath, 'identity.json has a public key but no private key — the private key is gone, not missing, and a new one would be a different agent'); + } + const identity = { ...source }; let mutated = false; // NO KEYPAIR AT ALL (pre-v0.3.7 nodes): nothing to replace, so minting one gives an agent its first // key; the nodeId is preserved. A PRIVATE KEY BUT NO PUBLIC KEY: derive it, never generate one. @@ -346,74 +700,612 @@ function readIdentityFile(idPath, name) { identity.hostname = normalized; mutated = true; } - if (mutated) fs.writeFileSync(idPath, JSON.stringify(identity, null, 2)); + return { source, identity, mutated }; +} + +/** Read an identity.json that EXISTS: one that cannot be read halts, never regenerates. @private */ +function readIdentityText(idPath, name) { + try { + return fs.readFileSync(idPath, 'utf8'); + } catch (e) { + // The file EXISTS and we cannot read it. Never regenerate here — a transient read failure + // (permissions, a full disk, a partial write) would otherwise overwrite a live keypair. + throw new IdentityHaltError(name, idPath, `identity.json exists but could not be read: ${e.message}`); + } +} + +/** + * Read and validate an installed identity.json (see parseIdentity). A fix it needs (a first key, a derived + * public key, `.local`) is written back atomically and privately; a key file left readable by others is + * narrowed to its owner. + * @private + */ +function readIdentityFile(idPath, name) { + const { identity, mutated } = parseIdentity(readIdentityText(idPath, name), name, idPath); + // Atomic and private: a truncate-in-place that is interrupted leaves a key file with no key in it, and a + // mode passed to writeFileSync is ignored for a file that already exists. + if (mutated) writeFileAtomic0600(idPath, JSON.stringify(identity, null, 2)); + securePrivateFiles([path.dirname(idPath)]); return identity; } +/** True when `other` is the identity `source` says: the same nodeId, and every key `source` has, the same. */ +function sameIdentity(source, other) { + return Boolean(source && other) && source.nodeId === other.nodeId && + (!source.privateKey || source.privateKey === other.privateKey) && + (!source.publicKey || source.publicKey === other.publicKey); +} + +// ── Staged copies, committed by one rename ─────────────────────────────── + +/** Top-level entries a copy of a node directory leaves out: runtime state, and what the copy writes itself. */ +const COPY_SKIP_TOP = new Set(['lock.pid', 'interior.sock', TOMBSTONE, EXPORTING, COMMIT_MARK]); +const skipTop = (extra = []) => { + const names = new Set([...COPY_SKIP_TOP, ...extra]); + return (rel) => !rel.includes('/') && (names.has(rel) || rel.endsWith('.tmp') || rel.startsWith('lock.pid.') || /^\.[^/]*\.link-\d+-[0-9a-f]+$/.test(rel)); +}; + +/** Two regular files hold the same bytes (read in pieces: a store file can be large). */ +function sameBytes(a, b) { + const sa = statOrNull(a); const sb = statOrNull(b); + if (!sa || !sb || !sa.isFile() || !sb.isFile() || sa.size !== sb.size) return false; + let fa; let fb; + try { + fa = fs.openSync(a, 'r'); fb = fs.openSync(b, 'r'); + const ba = Buffer.alloc(1 << 20); const bb = Buffer.alloc(1 << 20); + for (let pos = 0; pos < sa.size;) { + const na = fs.readSync(fa, ba, 0, ba.length, pos); const nb = fs.readSync(fb, bb, 0, bb.length, pos); + if (na !== nb || na === 0 || ba.compare(bb, 0, nb, 0, na) !== 0) return false; + pos += na; + } + return true; + } catch { return false; } finally { + if (fa !== undefined) fs.closeSync(fa); + if (fb !== undefined) fs.closeSync(fb); + } +} + /** - * Move one 0.13 node (`nodes//`, and its identity at `SYM_IDENTITY_DIR//` when that is - * set) to `nodes/by-id//`, and leave `nodes/` as a symlink to it for one release (a 0.13 - * rollback reads the old path). Idempotent: a name that is already a symlink, or indexed, is left as - * it is. A directory whose lock a live process holds is skipped: that node moves itself when it - * restarts. - * - * A 0.13 identity file is read with the same reader a load uses, so one that exists and cannot be - * established (unreadable, not JSON, no nodeId) HALTS (IdentityHaltError) instead of reading as - * "no node here" — which would let the caller mint a replacement under the agent's name. - * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'} + * Copy the children of directory `src` into directory `dst` (which exists): directories are made 0700, + * files keep their bytes and mode (a key file 0600) and are synced. Links are FOLLOWED, so what a 0.13 node + * kept behind a link (its directory on another volume, an identity.json linked to a key file) is copied, not + * a link that would lead somewhere else from the new place; a link that leads nowhere is copied as the link + * it is. What is neither a file nor a directory (a socket) is not data and is left out, and so is what + * `skip(rel)` names. A directory reached twice through links is copied as a link, not followed forever. + */ +function copyChildren(src, dst, skip, rel = '', within = new Set()) { + within.add(fs.realpathSync(src)); + for (const e of fs.readdirSync(src).sort()) { + const r = rel ? `${rel}/${e}` : e; + if (skip(r)) continue; + const from = path.join(src, e); + const to = path.join(dst, e); + const st = statOrNull(from); + if (!st) { // a link that leads nowhere + const l = lstatOrNull(from); + if (l && l.isSymbolicLink()) fs.symlinkSync(fs.readlinkSync(from), to); + continue; + } + if (st.isDirectory()) { + if (within.has(fs.realpathSync(from))) { fs.symlinkSync(fs.readlinkSync(from), to); continue; } + fs.mkdirSync(to, { mode: 0o700 }); + fs.chmodSync(to, 0o700); + copyChildren(from, to, skip, r, new Set(within)); + fsyncDir(to); + } else if (st.isFile()) { + // From the file a link leads to: a clone (macOS) of a link would copy the link. + fs.copyFileSync(fs.realpathSync(from), to, fs.constants.COPYFILE_EXCL | fs.constants.COPYFILE_FICLONE); + fsyncFile(to); + fs.chmodSync(to, PRIVATE_KEY_FILES.includes(e) ? 0o600 : st.mode & 0o777); + } + } +} + +/** The first path (relative) under directory `src` whose copy under `dst` is missing or differs, or null. */ +function firstDifference(src, dst, skip, rel = '', within = new Set()) { + within.add(fs.realpathSync(src)); + for (const e of fs.readdirSync(src).sort()) { + const r = rel ? `${rel}/${e}` : e; + if (skip(r)) continue; + const from = path.join(src, e); + const to = path.join(dst, e); + const st = statOrNull(from); + if (!st) { + const l = lstatOrNull(from); const d = lstatOrNull(to); + if (l && l.isSymbolicLink() && !(d && d.isSymbolicLink() && fs.readlinkSync(to) === fs.readlinkSync(from))) return r; + continue; + } + if (st.isDirectory()) { + if (within.has(fs.realpathSync(from))) continue; + const d = lstatOrNull(to); + if (!d || !d.isDirectory()) return r; + const inner = firstDifference(from, to, skip, r, new Set(within)); + if (inner) return inner; + } else if (st.isFile() && !sameBytes(from, to)) { + return r; + } + } + return null; +} + +/** Remove leftover staging directories (`..partial--`) of processes that are gone. */ +function clearPartials(parent) { + let names = []; + try { names = fs.readdirSync(parent); } catch { return; } + for (const n of names) { + const m = /^\.[^\\/]+\.(?:partial|import)-(\d+)-[0-9a-f]+$/.exec(n); + if (!m || Number(m[1]) === process.pid || pidIsAlive(Number(m[1]))) continue; + try { fs.rmSync(path.join(parent, n), { recursive: true, force: true }); } catch { /* the next attempt */ } + } +} + +/** + * A new staging directory for `nodeId` beside where it will be installed (`parent`), so the commit is a rename + * on one volume: `..--`, 0700. Leftovers of processes that are gone are cleared first. + */ +function makeStaging(parent, nodeId, kind = 'partial') { + ensureDir(parent); + clearPartials(parent); + const staging = path.join(parent, `.${nodeId}.${kind}-${process.pid}-${randomTag()}`); + fs.mkdirSync(staging, { mode: 0o700 }); + fs.chmodSync(staging, 0o700); + return staging; +} + +/** Write the commit record into a staging directory (last), and sync the directory. */ +function markStaged(staging, nodeId, from) { + writeNewFileSynced(path.join(staging, COMMIT_MARK), JSON.stringify({ nodeId, from, at: Date.now() }) + '\n', 0o600); + fsyncDir(staging); +} + +/** Why a staged commit was refused: a different identity, or files that differ, already under the nodeId. */ +class CommitConflict extends Error {} + +/** + * What is at `target`, the by-id directory a commit for `nodeId` installs: + * - 'absent'; + * - 'committed': it carries the commit record (and, when it holds the identity, `source`'s identity); + * - 'husk': a directory without the commit record that does not contradict `source` (empty, a stale lock, + * the files of an earlier release's interrupted move or of a node that ran there without its identity); + * - 'conflict': it holds a different identity, or is not a directory. + * `holdsIdentity`: the identity lives in this directory (no SYM_IDENTITY_DIR, or the identity directory). + */ +function examineTarget(target, nodeId, source, holdsIdentity) { + const st = lstatOrNull(target); + if (!st) return { state: 'absent' }; + if (st.isSymbolicLink() || !st.isDirectory()) return { state: 'conflict', reason: `${target} is not a directory` }; + const marked = fs.existsSync(path.join(target, COMMIT_MARK)); + if (holdsIdentity) { + const f = path.join(target, 'identity.json'); + if (fs.existsSync(f)) { + let other = null; + try { other = JSON.parse(fs.readFileSync(f, 'utf8')); } catch { /* below */ } + if (!other) return { state: 'conflict', reason: `${f} is already there and cannot be read` }; + if (!sameIdentity(source, other)) return { state: 'conflict', reason: `a different identity.json is already filed under ${nodeId} (${f})` }; + } else if (marked) { + return { state: 'conflict', reason: `${target} carries a commit record and no identity.json` }; + } + } + if (marked) { + let mark = null; + try { mark = JSON.parse(fs.readFileSync(path.join(target, COMMIT_MARK), 'utf8')); } catch { /* below */ } + if (!mark || mark.nodeId !== nodeId) return { state: 'conflict', reason: `${target} carries the commit record of another node` }; + return { state: 'committed' }; + } + return { state: 'husk' }; +} + +/** + * Copy what `from` holds into `staging` where it is not there yet; a file on both sides that differs is a + * conflict (CommitConflict), and nothing of `from` is changed. How a husk's files (a node that ran in the + * by-id directory without its identity) come into the staged directory before the husk is set aside. + */ +function mergeIntoStaging(from, staging, skip, rel = '') { + for (const e of fs.readdirSync(path.join(from, rel)).sort()) { + const r = rel ? `${rel}/${e}` : e; + if (skip(r)) continue; + const src = path.join(from, r); + const dst = path.join(staging, r); + const st = statOrNull(src); + if (!st) continue; + const d = lstatOrNull(dst); + if (st.isDirectory()) { + if (d && !d.isDirectory()) throw new CommitConflict(`${src} is a directory where the 0.13 copy has a file`); + if (!d) { fs.mkdirSync(dst, { mode: 0o700 }); fs.chmodSync(dst, 0o700); } + mergeIntoStaging(from, staging, skip, r); + } else if (st.isFile()) { + if (d) { if (!sameBytes(src, dst)) throw new CommitConflict(`${src} differs from the copy of the 0.13 directory`); continue; } + fs.copyFileSync(fs.realpathSync(src), dst, fs.constants.COPYFILE_EXCL | fs.constants.COPYFILE_FICLONE); + fsyncFile(dst); + fs.chmodSync(dst, PRIVATE_KEY_FILES.includes(e) ? 0o600 : st.mode & 0o777); + if (!sameBytes(src, dst)) throw new Error(`the copy of ${src} does not match it`); + } + } +} + +/** A lock, a claim to remove one, or the temp file a lock is written under: runtime state, never a node's data. */ +const isLockArtifact = (n) => n === 'lock.pid' || n.startsWith('lock.pid.') || (n.startsWith('.lock.pid') && n.endsWith('.tmp')); + +/** + * Remove, from a husk that holds nothing else, the lock artifacts of processes that are gone: a claim to remove a + * lock that is itself gone, or a temp lock file. (lock.pid itself is removed only by removeStaleLock.) + */ +function clearDeadLockArtifacts(dir) { + if (fs.existsSync(path.join(dir, 'lock.pid'))) return; + let names = []; + try { names = fs.readdirSync(dir); } catch { return; } + // This process's own are left over from an attempt that failed: lock work is synchronous, so none is under way. + const gone = (pid) => pid === process.pid || !pidIsAlive(pid); + for (const n of names) { + if (n.startsWith('lock.pid.')) { + const l = readLockFile(path.join(dir, n)); + if (l && (l.pid === process.pid || (!lockIsLive(l) && Date.now() - l.mtimeMs > LOCK_CORRUPT_GRACE_MS))) { try { fs.unlinkSync(path.join(dir, n)); } catch { /* */ } } + } else if (n.startsWith('.lock.pid') && n.endsWith('.tmp')) { + const m = /\.(\d+)-[0-9a-f]+\.tmp$/.exec(n); + if (m && gone(Number(m[1]))) { try { fs.unlinkSync(path.join(dir, n)); } catch { /* */ } } + } + } +} + +/** + * Install a complete staging directory as `target` by ONE rename: that rename is the commit. Returns + * 'committed' when this staging went in, 'exists' when `target` already holds a committed directory (another + * start committed first, or an earlier one did): the caller checks it is the same identity and discards its + * staging. Nothing at `target` is ever overwritten by it: + * - a husk (no commit record) holding nothing, or only a lock whose holder is gone, is removed with rmdir, + * which cannot remove a directory with anything in it, and the rename is tried again; + * - a husk with files: they are copied into the staging (a file that differs is a conflict), and the husk is + * renamed aside to `..husk` beside it, kept. That name is taken once and kept, so a later rename + * can never move a directory installed since; and the staging takes in the files of a husk set aside by + * another start before each attempt; + * - a husk whose lock a live process holds: EIDENTITYLOCK, nothing changed. + */ +function commitStaged(staging, target, { nodeId, examine, skip }) { + const asideHusk = path.join(path.dirname(target), `.${nodeId}.husk`); + for (let attempt = 0; attempt < 12; attempt++) { + if (fs.existsSync(asideHusk)) mergeIntoStaging(asideHusk, staging, skip); + try { + fs.renameSync(staging, target); + fsyncDir(path.dirname(target)); + return 'committed'; + } catch (e) { + if (!RENAME_TAKEN.has(e.code)) throw e; + } + const t = examine(target); + if (t.state === 'absent') continue; + if (t.state === 'committed') return 'exists'; + if (t.state === 'conflict') throw new CommitConflict(t.reason); + const lockPath = path.join(target, 'lock.pid'); + const lock = readLockFile(lockPath); + if (lock) { + if (lockHeldInProcess(target) || (lock.pid !== process.pid && lockIsHeldByLiveProcess(lock))) { + throw lockError(`[SYM] A live process (PID ${lock.pid ?? 'unknown'}) holds ${lockPath}, but ${nodeId}'s identity is not installed there: ` + + 'a node of an earlier release is running in that directory without its identity. Stop it, then start this agent again; nothing was changed.', lock); + } + removeStaleLock(lockPath, lock); + } + let rest = []; + try { rest = fs.readdirSync(target).filter((n) => !isLockArtifact(n)); } catch { continue; } + if (!rest.length) { + clearDeadLockArtifacts(target); + try { fs.rmdirSync(target); continue; } catch (e) { + if (!['ENOENT', 'ENOTEMPTY', 'EEXIST'].includes(e.code)) throw e; + if (e.code === 'ENOENT') continue; + // Another start's claim or temp lock file is in it for a moment; if one stays, the husk is set aside with it. + if (attempt < 4) { pause(2); continue; } + } + } + // Looked at again right before it is set aside: since the look above, the husk may have been emptied and + // replaced by another start's commit (found by the multi-process fuzz: a committed directory was set aside). + // A husk with files cannot be replaced by a commit (a rename onto a directory that is not empty fails), so + // from here only another claim can move it, and then the rename below fails. + const again = examine(target); + if (again.state === 'absent') continue; + if (again.state === 'committed') return 'exists'; + if (again.state === 'conflict') throw new CommitConflict(again.reason); + mergeIntoStaging(target, staging, skip); + try { fs.renameSync(target, asideHusk); } catch (e) { if (e.code !== 'ENOENT' && !RENAME_TAKEN.has(e.code)) throw e; } + } + throw new Error(`[SYM] could not install ${target}: what was there kept changing; nothing was overwritten. Start again.`); +} + +/** + * Stage a copy of the 0.13 directory `src` (null: none) for `nodeId` and commit it as `target`. `identityText`, + * when given, is written as identity.json after everything else, then the commit record; the copy is verified + * against `src` byte for byte before the commit. Returns 'committed' | 'exists'; a conflict throws CommitConflict. */ -function migrateNodeDir(name) { - const legacy = path.join(NODES_DIR, name); +function commitCopy(src, target, { nodeId, identityText, source, holdsIdentity, from }) { + const skip = skipTop(['identity.json']); + const examine = (t) => examineTarget(t, nodeId, source, holdsIdentity); + const pre = examine(target); + if (pre.state === 'committed') return 'exists'; + if (pre.state === 'conflict') throw new CommitConflict(pre.reason); + const staging = makeStaging(path.dirname(target), nodeId); + try { + if (src) copyChildren(src, staging, skip); + if (identityText) writeNewFileSynced(path.join(staging, 'identity.json'), identityText, 0o600); + markStaged(staging, nodeId, from); + if (src) { + const d = firstDifference(src, staging, skip); + if (d) throw new Error(`[SYM] the copy of ${path.join(src, d)} does not match its source (did it change while it was copied?); nothing was installed, the source is untouched. Start again.`); + } + if (identityText && !sameIdentity(source, JSON.parse(fs.readFileSync(path.join(staging, 'identity.json'), 'utf8')))) { + throw new Error('[SYM] the staged identity.json does not read back as the identity it was written from; nothing was installed'); + } + return commitStaged(staging, target, { nodeId, skip, examine }); + } finally { + fs.rmSync(staging, { recursive: true, force: true }); + } +} + +/** + * Install a new directory for `nodeId` at `target` (which must not exist), built by `build(staging)` in a staging + * directory and committed by one rename: for a new identity, and for an import (relocation.js, whose + * `beforeCommit` sets the tombstoned copy of a node coming back aside once the new one is complete). + */ +function installNewDir(target, nodeId, build, kind = 'partial', { beforeCommit } = {}) { + const staging = makeStaging(path.dirname(target), nodeId, kind); + try { + build(staging); + markStaged(staging, nodeId, kind); + if (beforeCommit) beforeCommit(); + fs.renameSync(staging, target); + fsyncDir(path.dirname(target)); + } finally { + fs.rmSync(staging, { recursive: true, force: true }); + } +} + +// ── 0.13 directories ───────────────────────────────────────────────────── + +/** + * The 0.13 directory at `p`, or null: a real directory, or a link to a directory that is NOT a compat link (a + * 0.13 node kept its directory on another volume behind one; it is followed and copied). + */ +function legacyAt(p) { + const st = lstatOrNull(p); + if (!st) return null; + if (st.isSymbolicLink()) { + if (compatLinkNodeId(p)) return null; + const t = statOrNull(p); + return t && t.isDirectory() ? { path: p, link: true } : null; + } + return st.isDirectory() ? { path: p, link: false } : null; +} + +/** The 0.13 directories of `name`: nodes/, and SYM_IDENTITY_DIR/ (or the same) for the identity. */ +function legacySources(name) { const idRoot = process.env.SYM_IDENTITY_DIR; - const legacyIdDir = idRoot ? path.join(idRoot, name) : legacy; - const lst = (p) => { try { return fs.lstatSync(p); } catch { return null; } }; - const indexFromLink = (link) => { - const m = /by-id[\\/]([^\\/]+)$/.exec(fs.readlinkSync(link)); - if (m && isNodeIdText(m[1])) { indexName(name, m[1]); return 'indexed'; } - return 'already'; + const nodes = legacyAt(path.join(NODES_DIR, name)); + const ids = idRoot ? legacyAt(path.join(idRoot, name)) : nodes; + return { nodes, ids, idRoot }; +} + +/** True while a 0.13 identity.json sits under `name` (in nodes// or SYM_IDENTITY_DIR//). */ +function legacyIdentityUnder(name) { + const { nodes, ids } = legacySources(name); + return [nodes, ids].some((l) => l && fs.existsSync(path.join(l.path, 'identity.json'))); +} + +/** Whether `dir` holds anything but a lock. */ +function hasContent(dir) { + try { return fs.readdirSync(dir).some((n) => n !== 'lock.pid'); } catch { return false; } +} + +/** + * Hold the 0.13 directory's own lock (`nodes//lock.pid`, the file a 0.13 process of the agent takes) + * while it is copied and set aside, so no 0.13 process starts in it meanwhile. The lock says it is a + * migration's: another 0.14 start that finds it goes on beside it (the staged commit makes that safe) instead + * of waiting. A live holder that is not a migration is a process running in that directory: `liveHolder`. + */ +function takeLegacyLock(dir) { + const lockPath = path.join(dir, 'lock.pid'); + const token = randomTag() + randomTag(); + let got; + try { + got = acquireLockAt(lockPath, { migrating: true, token }); + } catch { + // The directory does not take a lock (read-only, not ours): copy it all the same, unless a process holds it. + const l = readLockFile(lockPath); + got = l && lockIsLive(l) ? { ok: false, lock: l } : { ok: true, none: true }; + if (got.ok) return { release() {} }; + } + if (!got.ok) { + const l = got.lock; + if (l && (l.migrating || l.pid === process.pid)) return { release() {} }; + return { liveHolder: (l && l.pid) || 'unknown', release() {} }; + } + return { + /** Remove the lock wherever the directory went (set aside), if it is still this one. */ + release(also = []) { + for (const p of [lockPath, ...also.map((d) => path.join(d, 'lock.pid'))]) { + const l = readLockFile(p); + if (l && l.pid === process.pid && l.token === token) { try { fs.unlinkSync(p); } catch { /* */ } } + } + }, }; - const st = lst(legacy); - if (st && st.isSymbolicLink()) return nodeIdForName(name) ? 'already' : indexFromLink(legacy); - if (st && !st.isDirectory()) return 'not-a-node'; - const idSt = idRoot ? lst(legacyIdDir) : st; - if (idRoot && idSt && idSt.isSymbolicLink()) return nodeIdForName(name) ? 'already' : indexFromLink(legacyIdDir); - const idFile = path.join(legacyIdDir, 'identity.json'); - if (!fs.existsSync(idFile)) return 'not-a-node'; - const identity = readIdentityFile(idFile, name); // halts on an identity it cannot establish - const nodeId = identity.nodeId; +} + +/** + * Rename the 0.13 directory (or link) at `at` aside, to `/.legacy-0.13/`: kept, never deleted. + * Returns where it went, or null when it is no longer there (another start set it aside). + */ +function setAside(at, root, name) { + const st = lstatOrNull(at); + if (!st || compatLinkNodeId(at)) return null; + const dir = path.join(root, LEGACY_ASIDE); + if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true, mode: 0o700 }); + for (const dest of [path.join(dir, name), path.join(dir, `${name}.${Date.now()}-${randomTag()}`)]) { + try { + fs.renameSync(at, dest); + fsyncDir(dir); + fsyncDir(root); + return dest; + } catch (e) { + if (e.code === 'ENOENT') return null; + if (!RENAME_TAKEN.has(e.code)) throw e; + const again = lstatOrNull(at); + if (!again || compatLinkNodeId(at)) return null; // set aside meanwhile; what is there now is the link + } + } + return null; +} + +/** + * After the commit: the 0.13 directories of `name` go aside and compat links take their place. The link is + * made first, under a name of its own, and renamed onto the old path right after the directory left it, so a + * 0.13 rollback finds the old path empty for one rename only. + */ +function setAsideAndLink(name, nodeId, srcs) { + const plans = [[NODES_DIR, srcs.nodes]]; + if (srcs.idRoot) plans.push([srcs.idRoot, srcs.ids]); + const aside = []; + for (const [root, src] of plans) { + const at = path.join(root, name); + if (!src) { if (root === NODES_DIR || fs.existsSync(path.join(root, 'by-id', nodeId))) linkCompat(at, nodeId); continue; } + const tmp = path.join(root, `.${name}.link-${process.pid}-${randomTag()}`); + try { + fs.symlinkSync(path.join('by-id', nodeId), tmp, process.platform === 'win32' ? 'junction' : 'dir'); + } catch { + // No link can be made here: the 0.13 directory stays where it is (a 0.13 rollback still reads it), and the + // next start tries again. The identity is installed and the name claimed either way. + continue; + } + let went = null; + try { went = setAside(at, root, name); } catch { /* the identity is installed and indexed: the next start sets it aside */ } finally { + try { fs.renameSync(tmp, at); } catch { try { removeLink(tmp); } catch { /* */ } } + linkCompat(at, nodeId); + } + if (went) { + aside.push(went); + securePrivateFiles([went], { narrowDirs: !src.link }); + } + } + return aside; +} + +/** + * Copy one 0.13 node (`nodes//`, and its identity at `SYM_IDENTITY_DIR//` when that is set) to + * `nodes/by-id//` (and `SYM_IDENTITY_DIR/by-id//`), and COMMIT it: the copy is staged beside + * its destination, identity.json last, synced, verified byte for byte against the source, and installed by one + * rename. Of several starts doing this at once each stages its own copy, one rename wins, and the rest find + * the same identity installed and discard theirs: nothing is moved, so nothing can be lost between two of + * them. Then the name is claimed (an exclusive create of its index entry), the 0.13 directory is renamed aside + * to `.legacy-0.13/` (kept, never deleted), and `nodes/` becomes a link to the by-id directory for + * one release (a 0.13 rollback reads the old path). + * + * A by-id directory already there without the commit record (an empty one, a lock, an earlier release's + * interrupted move, a node that ran there without its identity) is merged into the staged copy and replaced + * through the same commit, never read as "the identity is in place". One that holds a DIFFERENT identity, or a + * file that differs, is a second claim on the nodeId: 'skipped-duplicate', and nothing is changed. A 0.13 + * process running in the directory (its lock live): 'skipped-live', nothing changed (the start refuses with + * EIDENTITYLOCK). An identity this host exported (tombstoned): 'tombstoned'. + * + * A 0.13 identity file is read with the same parser a load uses, so one that exists and cannot be established + * (unreadable, not JSON, no nodeId) HALTS (IdentityHaltError). A 0.13 identity in `nodes//` while + * SYM_IDENTITY_DIR is set and it is not in `SYM_IDENTITY_DIR//` HALTS: it is neither where the identity dir + * looks nor safe to leave for a mint. + * + * `onlyNodeId`: a start pinned to a nodeId migrates the directory under `name` only if its identity IS that + * nodeId; another agent's directory is left as it is ('skipped-duplicate'). + * @returns {{ status: 'moved'|'indexed'|'already'|'not-a-node'|'skipped-live'|'skipped-duplicate'|'tombstoned', nodeId?: string, reason?: string }} + * @private + */ +function migrateName(name, onlyNodeId) { + const srcs = legacySources(name); + if (!srcs.nodes && !srcs.ids) { + const linked = compatLinkNodeId(path.join(NODES_DIR, name)); + const held = nodeIdForName(name); + if (held) return { status: 'already', nodeId: held }; + if (linked && identityInPlace(linked) && !indexEntries().has(linked) && (!onlyNodeId || linked === onlyNodeId)) { + return claimName(name, linked) === linked ? { status: 'indexed', nodeId: linked } : { status: 'skipped-duplicate', nodeId: linked, reason: 'another start claimed the name first' }; + } + return { status: 'not-a-node' }; + } + const idFile = srcs.ids && path.join(srcs.ids.path, 'identity.json'); + const inNodeDir = srcs.nodes && path.join(srcs.nodes.path, 'identity.json'); + if (!idFile || !fs.existsSync(idFile)) { + if (srcs.idRoot && inNodeDir && fs.existsSync(inNodeDir)) { + const want = path.join(srcs.idRoot, name, 'identity.json'); + throw new IdentityHaltError(name, inNodeDir, + `SYM_IDENTITY_DIR (${srcs.idRoot}) is set, but this agent's identity is still in its node directory and not in ${path.dirname(want)}. ` + + `It was not moved, and a new one will not be made in its place. Copy it to ${want} (keep the original until the agent has started), ` + + 'or unset SYM_IDENTITY_DIR.'); + } + return { status: 'not-a-node' }; + } + const read = parseIdentity(readIdentityText(idFile, name), name, idFile); // halts on an identity it cannot establish + const nodeId = read.identity.nodeId; if (!isNodeIdText(nodeId)) throw new IdentityHaltError(name, idFile, `the nodeId ${JSON.stringify(String(nodeId).slice(0, 40))} cannot name a directory under nodes/by-id`); - if (st) { - const lock = readLockFile(path.join(legacy, 'lock.pid')); - if (lock && lock.pid !== process.pid && lockIsHeldByLiveProcess(lock)) return 'skipped-live'; + // The operator copied the identity to SYM_IDENTITY_DIR as the halt above says, and kept the original: the + // two must be one identity. The copy in the node directory is left out of the node directory's copy. + if (srcs.idRoot && inNodeDir && fs.existsSync(inNodeDir)) { + const other = parseIdentity(readIdentityText(inNodeDir, name), name, inNodeDir).source; + if (!sameIdentity(read.source, other) || !sameIdentity(other, read.source)) { + throw new IdentityHaltError(name, inNodeDir, `two different identities sit under "${name}": ${idFile} (${nodeId}) and ${inNodeDir} (${other.nodeId}). Nothing was moved; keep the one that is this agent.`); + } } - const dest = nodeDirById(nodeId); - const idDest = idRoot && idSt && idSt.isDirectory() ? path.join(idRoot, 'by-id', nodeId) : null; - // Two dirs claim one nodeId: the first one moved stays, and nothing of this one is moved. - if ((st && fs.existsSync(dest)) || (idDest && fs.existsSync(idDest))) return 'skipped-duplicate'; - ensureDir(path.dirname(dest)); - if (st) fs.renameSync(legacy, dest); - else ensureDir(dest); - if (idDest) { - ensureDir(path.dirname(idDest)); - fs.renameSync(legacyIdDir, idDest); + if (onlyNodeId && nodeId !== onlyNodeId) return { status: 'skipped-duplicate', nodeId, reason: `the 0.13 identity under "${name}" is ${nodeId}, not the pinned ${onlyNodeId}; it was left as it is` }; + const held = nodeIdForName(name); + if (held && held !== nodeId) return { status: 'skipped-duplicate', nodeId, reason: `the name "${name}" already indexes node ${held}; the 0.13 identity ${nodeId} under it was left as it is` }; + const tomb = readTombstone(nodeId); + if (tomb) return { status: 'tombstoned', nodeId, tomb }; + const lock = srcs.nodes ? takeLegacyLock(srcs.nodes.path) : { release() {} }; + if (lock.liveHolder) return { status: 'skipped-live', nodeId, holderPid: lock.liveHolder }; + let aside = []; + try { + const nodeTarget = nodeDirById(nodeId); + const idTarget = identityDirById(nodeId); + const split = idTarget !== nodeTarget; + const text = JSON.stringify(read.identity, null, 2); + try { + commitCopy(srcs.nodes && srcs.nodes.path, nodeTarget, { nodeId, identityText: split ? null : text, source: read.source, holdsIdentity: !split, from: `0.13 nodes/${name}` }); + if (split) commitCopy(srcs.ids.path, idTarget, { nodeId, identityText: text, source: read.source, holdsIdentity: true, from: `0.13 SYM_IDENTITY_DIR/${name}` }); + } catch (e) { + if (e instanceof CommitConflict) return { status: 'skipped-duplicate', nodeId, reason: `the 0.13 identity under "${name}" was not migrated: ${e.message}` }; + throw e; + } + // The record that the name is this node: claimed once the identity is installed. Of two starts, the + // second finds the entry made and loads what it names. + const winner = claimName(name, nodeId); + if (winner !== nodeId) return { status: 'skipped-duplicate', nodeId, reason: `the name "${name}" was claimed for node ${winner} meanwhile` }; + aside = setAsideAndLink(name, nodeId, srcs); + securePrivateFiles([nodeTarget, idTarget], { narrowDirs: true }); + return { status: 'moved', nodeId }; + } finally { + lock.release(aside); } - indexName(name, nodeId); - return 'moved'; } /** - * Move every 0.13 node directory under the state root to the by-id layout (design D9, §6 of the - * design: a one-time, idempotent move). Safe to run at every start and from several processes. + * Migrate the 0.13 directory of `name` to the by-id layout (see migrateName). Idempotent, safe from several + * processes at once, and it takes no lock but the 0.13 directory's own. + * @param {string} name + * @param {string} [onlyNodeId] + * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'|'tombstoned'} + */ +function migrateNodeDir(name, onlyNodeId) { + const problem = basicNameProblem(name); + if (problem) throw new Error(problem); + return migrateName(name, onlyNodeId).status; +} + +/** + * Migrate every 0.13 node directory under the state root to the by-id layout (design D9, §6 of the + * design: a one-time, idempotent copy). Safe to run at every start and from several processes. * @returns {{ moved: number, already: number, skippedLive: number, duplicates: number }} */ function migrateIdentities() { const out = { moved: 0, already: 0, skippedLive: 0, duplicates: 0 }; - let entries; - try { entries = fs.readdirSync(NODES_DIR, { withFileTypes: true }); } catch { return out; } - for (const d of entries) { - if (d.name === 'by-id' || d.name === 'by-name' || d.name.startsWith('.')) continue; + const names = new Set(); + for (const root of [NODES_DIR, process.env.SYM_IDENTITY_DIR].filter(Boolean)) { + let entries = []; + try { entries = fs.readdirSync(root); } catch { continue; } + for (const n of entries) if (n !== 'by-id' && n !== 'by-name' && !n.startsWith('.') && !basicNameProblem(n)) names.add(n); + } + for (const name of names) { let r; - try { r = migrateNodeDir(d.name); } catch { continue; } + try { r = migrateNodeDir(name); } catch { continue; } if (r === 'moved') out.moved++; else if (r === 'already' || r === 'indexed') out.already++; else if (r === 'skipped-live') out.skippedLive++; @@ -422,52 +1314,136 @@ function migrateIdentities() { return out; } +// ── Loading ────────────────────────────────────────────────────────────── + +/** Load the installed identity of `nodeId`: tombstone and export checked, file validated (halts, never mints). */ +function loadInstalled(nodeId, label, { allowPendingExport = false } = {}) { + const tomb = readTombstone(nodeId); + if (tomb) throw new IdentityTombstonedError(nodeId, tomb); + if (!allowPendingExport) { + const intent = readExportIntent(nodeId); + if (intent) throw new IdentityExportPendingError(nodeId, intent); + } + const idPath = path.join(identityDirById(nodeId), 'identity.json'); + const identity = readIdentityFile(idPath, label); + if (identity.nodeId !== nodeId) throw new IdentityHaltError(label, idPath, `identity.json names ${identity.nodeId}, not ${nodeId}`); + securePrivateFiles([nodeDirById(nodeId)]); // the 0.13 E2E keypair beside it, when the identity lives elsewhere + return identity; +} + +/** Where on this host an identity.json for `nodeId` is, other than where a load reads it. */ +function strayCopiesOf(nodeId) { + const out = []; + const where = new Set([path.join(NODES_DIR, 'by-id', nodeId)]); + if (process.env.SYM_IDENTITY_DIR) where.add(path.join(process.env.SYM_IDENTITY_DIR, 'by-id', nodeId)); + where.delete(identityDirById(nodeId)); + for (const root of [NODES_DIR, process.env.SYM_IDENTITY_DIR].filter(Boolean)) { + let names = []; + try { names = fs.readdirSync(path.join(root, LEGACY_ASIDE)); } catch { /* none */ } + for (const n of names) where.add(path.join(root, LEGACY_ASIDE, n)); + } + for (const d of where) { + try { if (JSON.parse(fs.readFileSync(path.join(d, 'identity.json'), 'utf8')).nodeId === nodeId) out.push(path.join(d, 'identity.json')); } catch { /* none */ } + } + return out; +} + /** - * Load a node's identity by nodeId or by name (design D9). - * - * - `{ nodeId }` loads `nodes/by-id//identity.json`; - * - `{ name }` resolves the name index to a nodeId (moving a 0.13 `nodes//` dir first); - * - `create: false` throws IdentityAbsentError when the identity is absent: a host restoring a - * known agent can never mint a replacement silently; - * - `create: true` (the default, for a brand-new agent) is the ONLY path that mints, and it mints - * only when the name is unknown (a name is an index; renaming changes the index, never the - * identity); - * - a tombstoned identity (exported to another host) refuses to load: IdentityTombstonedError. - * - * @param {{ nodeId?: string, name?: string, create?: boolean }} o - * @returns {{ nodeId: string, name: string, hostname: string, createdAt: number, publicKey: string, privateKey: string }} + * The halt for an identity that is not installed where it is read: the name's index entry names it + * (`indexed`), or a start asks for it by nodeId. When a copy of it is elsewhere on this host the halt says + * where, and never tells the operator to remove the index entry. */ -function loadIdentity({ nodeId, name, create = true } = {}) { - if (name !== undefined) validateName(name); - if (!nodeId && !name) throw new Error('loadIdentity needs a nodeId or a name'); - if (nodeId && !isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); - if (!nodeId && name) { - nodeId = nodeIdForName(name); - if (!nodeId) { - const r = migrateNodeDir(name); - if (r === 'moved' || r === 'already' || r === 'indexed') nodeId = nodeIdForName(name); - if (r === 'skipped-live') { - // A 0.13 process of this name holds its old directory: load it in place, unmoved. - const idRoot = process.env.SYM_IDENTITY_DIR; - const idPath = idRoot ? path.join(idRoot, name, 'identity.json') : path.join(NODES_DIR, name, 'identity.json'); - return readIdentityFile(idPath, name); +function indexedButMissing(name, nodeId, { indexed = true } = {}) { + const idPath = path.join(identityDirById(nodeId), 'identity.json'); + const copies = strayCopiesOf(nodeId); + const link = path.join(NODES_DIR, name); + const what = indexed ? `the name "${name}" indexes node ${nodeId}, and its identity.json is not where it is read` : `the identity ${nodeId} is not installed where it is read`; + if (copies.length) { + return new IdentityHaltError(name, idPath, + `${what}; it IS on this host, at ${copies.join(' and ')}.`, + `Put a copy of it at ${idPath} (keep the original until the agent has started)` + + `${process.env.SYM_IDENTITY_DIR ? ', or unset SYM_IDENTITY_DIR if this agent\'s key was never meant to live there' : ''}. Do not remove the name's index entry: the key is this agent.`); + } + return new IdentityHaltError(name, idPath, + `${what}, and no copy of it is on this host. ` + + 'If SYM_IDENTITY_DIR changed or its volume is not mounted, restore that.', + `Restore the keypair from your durable copy to ${idPath} and start again. If this agent's key is gone for good, it cannot return as "${name}"; ` + + `to let the name start again as a DIFFERENT agent, remove ${nameIndexPath(name)} and the link ${link}.`); +} + +/** + * What on this host belongs to `name` without being an installed, indexed identity: each is a reason not to + * mint. `{ identity: bool, text }`: `identity` when it holds an identity (a halt says so; never "absent"). + */ +function nameTraces(name, migration, scan = { indexed: indexEntries(), installed: identitiesOnDisk() }) { + const out = []; + const indexed = scan.indexed; + const idRoot = process.env.SYM_IDENTITY_DIR; + for (const [root, label] of [[NODES_DIR, 'nodes'], [idRoot, 'SYM_IDENTITY_DIR']]) { + if (!root) continue; + const at = path.join(root, name); + const st = lstatOrNull(at); + if (st) { + const linked = compatLinkNodeId(at); + if (linked) { + if (!identityInPlace(linked)) out.push({ identity: false, text: `${at} leads to by-id/${linked}, whose identity is not installed` }); + else out.push({ identity: true, nodeId: indexed.has(linked) ? null : linked, text: `${at} leads to ${linked}${indexed.has(linked) ? `, which the index gives the name(s) ${indexed.get(linked).join(', ')}` : ''}` }); + } else if (st.isSymbolicLink() || st.isDirectory()) { + const l = legacyAt(at); + if (l && fs.existsSync(path.join(l.path, 'identity.json'))) { + out.push({ identity: true, legacy: true, text: `a 0.13 identity sits in ${at}${migration && migration.reason ? ` and was not migrated (${migration.reason})` : ''}` }); + } else if (st.isSymbolicLink()) { + out.push({ identity: false, text: `${at} is a link to ${fs.readlinkSync(at)} with no identity in it` }); + } else if (hasContent(at)) { + out.push({ identity: false, text: `${at} holds a node's files (${fs.readdirSync(at).filter((n) => n !== 'lock.pid').slice(0, 4).join(', ')}) and no identity` + + `${label === 'nodes' && idRoot ? `, and ${path.join(idRoot, name, 'identity.json')} is not there (is the identity volume mounted?)` : ''}` }); + } + } else { + out.push({ identity: false, text: `${at} is a file, not a node directory` }); } } - } - if (nodeId) { - const tomb = readTombstone(nodeId); - if (tomb) throw new IdentityTombstonedError(nodeId, tomb); - const idPath = path.join(identityDirById(nodeId), 'identity.json'); - if (fs.existsSync(idPath)) { - const identity = readIdentityFile(idPath, name || nodeId); - if (identity.nodeId !== nodeId) throw new IdentityHaltError(name || nodeId, idPath, `identity.json names ${identity.nodeId}, not ${nodeId}`); - if (name && nodeIdForName(name) !== nodeId) indexName(name, nodeId); - return identity; + let aside = []; + try { aside = fs.readdirSync(path.join(root, LEGACY_ASIDE)).filter((n) => n === name || n.startsWith(`${name}.`)); } catch { /* none */ } + for (const n of aside) { + const d = path.join(root, LEGACY_ASIDE, n); + let id = null; + try { id = JSON.parse(fs.readFileSync(path.join(d, 'identity.json'), 'utf8')).nodeId; } catch { /* */ } + out.push({ identity: Boolean(id), nodeId: id, text: `the 0.13 directory of "${name}" was set aside at ${d}${id ? ` (identity ${id})` : ''}` }); } - if (!create || !name) throw new IdentityAbsentError(name ? `"${name}" (${nodeId})` : nodeId); } - if (!create) throw new IdentityAbsentError(`"${name}"`); - // No identity under this name: a genuinely new agent. This is the ONLY path that mints one. + for (const i of scan.installed) { + if (i.name !== name || indexed.has(i.nodeId) || i.scrubbed) continue; + out.push({ identity: true, nodeId: i.nodeId, text: `the identity ${i.nodeId} named "${name}" is installed at ${i.file} and no name indexes it` }); + } + return out; +} + +/** + * An installed identity the name provably belongs to though its index entry is missing: a compat link + * `nodes/` to it, or the one installed identity whose `name` is this name and that no name indexes + * (a start that crashed between installing it and claiming the name; an entry lost). `{ many }` when more + * than one fits. Nothing is adoptable while any trace of the name holds another identity, or a 0.13 identity + * that was not migrated (`traces`, from nameTraces): that is two claims on the name, and a halt. + */ +function adoptable(name, traces, scan = { indexed: indexEntries(), installed: identitiesOnDisk() }) { + const indexed = scan.indexed; + const consistent = (id) => traces.every((t) => !t.identity || (!t.legacy && t.nodeId === id)); + const ok = (id) => id && identityInPlace(id) && !indexed.has(id) && !readTombstone(id) && consistent(id); + for (const p of [path.join(NODES_DIR, name), process.env.SYM_IDENTITY_DIR && path.join(process.env.SYM_IDENTITY_DIR, name)]) { + const id = p && compatLinkNodeId(p); + if (ok(id)) return { nodeId: id }; + } + const named = scan.installed.filter((i) => i.name === name && !i.scrubbed && ok(i.nodeId)); + if (named.length === 1) return { nodeId: named[0].nodeId }; + return named.length ? { many: named } : null; +} + +/** + * Make a new identity for `name`: installed by one rename (the node dir, and the identity dir when + * SYM_IDENTITY_DIR is set), THEN the name claimed by an exclusive create of its index entry. Returns null when + * another first start of the name claimed it first: this identity was never anyone's, and it is removed. + */ +function mintIdentity(name) { const kp = generateSigningKeyPair(); const identity = { nodeId: uuidv7(), @@ -477,22 +1453,137 @@ function loadIdentity({ nodeId, name, create = true } = {}) { publicKey: kp.publicKey.toString('base64url'), privateKey: kp.privateKey.toString('base64url'), }; - // A 0.13-style `nodes//` dir with no identity (made by hand, or by a test) is the new node's - // dir: it moves with the identity rather than being left behind. - const legacy = path.join(NODES_DIR, name); - let legacyDir = false; - try { legacyDir = fs.lstatSync(legacy).isDirectory(); } catch { /* absent */ } - const dir = nodeDirById(identity.nodeId); - ensureDir(path.dirname(dir)); - if (legacyDir) fs.renameSync(legacy, dir); - ensureDir(dir); - const idDir = identityDirById(identity.nodeId); - ensureDir(idDir); - fs.writeFileSync(path.join(idDir, 'identity.json'), JSON.stringify(identity, null, 2), { mode: 0o600 }); - indexName(name, identity.nodeId); + const nodeId = identity.nodeId; + const text = JSON.stringify(identity, null, 2); + const nodeTarget = nodeDirById(nodeId); + const idTarget = identityDirById(nodeId); + const split = idTarget !== nodeTarget; + const withIdentity = (staging) => writeNewFileSynced(path.join(staging, 'identity.json'), text, 0o600); + installNewDir(nodeTarget, nodeId, split ? () => {} : withIdentity); + if (split) installNewDir(idTarget, nodeId, withIdentity); + if (claimName(name, nodeId) !== nodeId) { + fs.rmSync(nodeTarget, { recursive: true, force: true }); + if (split) fs.rmSync(idTarget, { recursive: true, force: true }); + return null; + } + ensureCompatLinks(name, nodeId); return identity; } +/** + * Load a node's identity by nodeId or by name (design D9). The state of an identity is read from records + * committed in one step: an installed identity (a directory installed by one rename) and the name's index + * entry (created exclusively, after the identity was installed). Nothing is inferred from a path that merely + * exists. + * + * - `{ nodeId }` loads that identity; + * - `{ name }` (pinned or not) first migrates a 0.13 identity still under the name (one path, see + * migrateName), then resolves the name by its index entry; an entry that cannot be read halts; + * - a 0.13 directory a live process holds refuses (EIDENTITYLOCK): the agent is running there; + * - `create: false` throws IdentityAbsentError when the identity is not on this host; + * - `create: true` (the default) is the ONLY path that mints, and it mints only when there is no trace + * of the name on this host: no index entry, no `nodes/` (a link, or a directory with anything in it), + * no `SYM_IDENTITY_DIR/`, no 0.13 directory set aside, no installed identity of that name that no name + * indexes. A trace that provably is this name's identity (a compat link to it, the one unindexed identity of + * that name) is adopted: the name is claimed for it. Any other trace HALTS (IdentityHaltError), saying what + * it is. Two first starts of a new name both mint; one claims the name, the other discards its identity + * and loads the winner; + * - a tombstoned identity (exported to another host) refuses: IdentityTombstonedError; one whose export + * did not finish refuses: IdentityExportPendingError. + * + * @param {{ nodeId?: string, name?: string, create?: boolean, allowPendingExport?: boolean }} o + * @returns {{ nodeId: string, name: string, hostname: string, createdAt: number, publicKey: string, privateKey: string }} + */ +function loadIdentity({ nodeId: pin, name, create = true, allowPendingExport = false } = {}) { + if (name !== undefined) validateName(name); + if (!pin && !name) throw new Error('loadIdentity needs a nodeId or a name'); + if (pin && !isNodeIdText(pin)) throw new Error(`not a nodeId: ${String(pin).slice(0, 40)}`); + const how = { allowPendingExport }; + if (!name) { + if (identityInPlace(pin)) return loadInstalled(pin, pin, how); + if (strayCopiesOf(pin).length) throw indexedButMissing(pin, pin, { indexed: false }); + throw new IdentityAbsentError(pin); + } + // 1. A 0.13 identity under the name is migrated first: the same path for a pinned start and an unpinned one. + let migration = null; + if (legacyIdentityUnder(name)) { + try { + migration = migrateName(name, pin); + } catch (e) { + // Another start finished this migration while this one was copying (it set the 0.13 directory aside under + // this copy): what it installed is read below, by the name it claimed. Anything else is this start's to report. + if (e instanceof IdentityHaltError || e.code === 'EIDENTITYLOCK' || legacyIdentityUnder(name)) throw e; + migration = { status: 'already' }; + } + if (migration.status === 'skipped-live') { + throw lockError( + `[SYM] Agent '${name}' is running in its 0.13 directory: a live process (PID ${migration.holderPid}) holds ${path.join(NODES_DIR, name, 'lock.pid')}.\n\n` + + ' One agent runs in one process. This start is refused; nothing was moved or changed. Stop that process (it is an older copy of ' + + 'this same agent), and start again: the agent is migrated then, with its store.', { pid: migration.holderPid }); + } + if (migration.status === 'tombstoned') throw new IdentityTombstonedError(migration.nodeId, migration.tomb); + } + if (pin) { + // A pinned start never takes a name from another agent (R4-9): it may claim a name nobody has, and that is all. + // Re-pointing the entry made the other agent's next start by name run as the pinned identity and orphaned the + // other one, and let a pin in a cloned repository's node.json pick which identity on the host a session signs as. + const held = nodeIdForName(name); + if (identityInPlace(pin)) { + if (held && held !== pin) throw nameHeldByAnother(name, pin, held); + const identity = loadInstalled(pin, name, how); + const winner = held || claimName(name, pin); + if (winner !== pin) throw nameHeldByAnother(name, pin, winner); // another start claimed it this moment + ensureCompatLinks(name, pin); + return identity; + } + if (migration && migration.nodeId === pin) { + throw new IdentityHaltError(name, path.join(NODES_DIR, name, 'identity.json'), + `the identity ${pin} this start is pinned to is on this host, in the 0.13 directory of "${name}", and could not be migrated: ${migration.reason || migration.status}.`, + 'Resolve that, then start again. Nothing was changed.'); + } + if (!create) { + if (strayCopiesOf(pin).length) throw indexedButMissing(name, pin, { indexed: false }); + const elsewhere = held && held !== pin ? [`the name "${name}" is another agent's here (${held}), and was left as it is`] : []; + throw new IdentityAbsentError(`"${name}" (${pin})`, [...elsewhere, ...nameTraces(name, migration).map((t) => t.text)]); + } + // Not on this host, and the name is another identity's: the start is never answered with that one. + if (held && held !== pin) throw nameHeldByAnother(name, pin, held); + } + for (let round = 0; ; round++) { + const indexed = nodeIdForName(name); + if (indexed) { + if (!identityInPlace(indexed)) { + if (!create && !strayCopiesOf(indexed).length) throw new IdentityAbsentError(`"${name}" (${indexed})`); + throw indexedButMissing(name, indexed); + } + const identity = loadInstalled(indexed, name, how); + ensureCompatLinks(name, indexed); + return identity; + } + if (round > 40) throw new Error(`[SYM] the name "${name}" did not settle (its index entry kept being claimed and lost); nothing was minted`); + // One look at what is installed and indexed decides both what is a trace and what can be adopted; an identity + // installed after it is another first start's, and the claim of the name settles which one the name is. + const scan = { indexed: indexEntries(), installed: identitiesOnDisk() }; + const traces = nameTraces(name, migration, scan); + const adopt = adoptable(name, traces, scan); + if (nodeIdForName(name)) continue; // another start claimed the name meanwhile: load what it names + if (adopt && adopt.nodeId) { claimName(name, adopt.nodeId); continue; } // the index decides; read it again + if (adopt && adopt.many && round < 20) { pause(10); continue; } // first starts claiming it: the entry appears within ms + if (traces.length) { + if (!create && !traces.some((t) => t.identity)) throw new IdentityAbsentError(`"${name}"`, traces.map((t) => t.text)); + throw new IdentityHaltError(name, nameIndexPath(name), + `no index entry names "${name}", and this host holds a trace of it: ${traces.map((t) => t.text).join('; ')}` + + `${pin ? `; this start is pinned to ${pin}, which is not on this host` : ''}. ` + + 'Making a new identity here would orphan or overwrite its key, so none was made.', + 'If the trace is this agent, restore its index entry or its identity; if it is not, move it out of the way. Then start again.'); + } + if (!create) throw new IdentityAbsentError(pin ? `"${name}" (${pin})` : `"${name}"`); + const minted = mintIdentity(name); + if (minted) return minted; + // Another first start of this name claimed it first: the index says who it is. + } +} + /** * Load or create a persistent identity for a node (0.13 API, kept): `loadIdentity({ name, create: true })`. * See MMP §3 (Identity). @@ -503,17 +1594,25 @@ function loadOrCreateIdentity(name) { } /** - * Rename a node: the name index moves; the identity (nodeId, key) does not (design D9). + * Rename a node: the name index moves; the identity (nodeId, key) does not (design D9). The new name is + * claimed exclusively. The old name only needs to be a name at all, so an agent whose name a later rule + * refuses (one Windows cannot hold) can be renamed to one it accepts. * @returns {string} the nodeId */ function renameIdentity(oldName, newName) { - validateName(oldName); validateName(newName); + const problem = basicNameProblem(oldName); + if (problem) throw new Error(problem); + validateName(newName); const nodeId = nodeIdForName(oldName); if (!nodeId) throw new IdentityAbsentError(`"${oldName}"`); - if (nodeIdForName(newName)) throw new Error(`the name "${newName}" already indexes a node`); - indexName(newName, nodeId); + if (nodeIdForName(newName) || claimName(newName, nodeId) !== nodeId) throw new Error(`the name "${newName}" already indexes a node`); + ensureCompatLinks(newName, nodeId); try { fs.unlinkSync(nameIndexPath(oldName)); } catch { /* */ } - try { if (fs.lstatSync(path.join(NODES_DIR, oldName)).isSymbolicLink()) fs.unlinkSync(path.join(NODES_DIR, oldName)); } catch { /* */ } + fsyncDir(path.dirname(nameIndexPath(oldName))); + for (const root of [NODES_DIR, process.env.SYM_IDENTITY_DIR].filter(Boolean)) { + const link = path.join(root, oldName); + if (compatLinkNodeId(link) === nodeId) { try { removeLink(link); } catch { /* */ } } + } return nodeId; } @@ -735,34 +1834,42 @@ function selfStartTime() { * readers that `parseInt()` the whole file still get the PID from the * numeric prefix, so both directions interoperate. * - legacy: the bare PID and nothing else. + * The file is read and stat'ed through ONE open, so `ino`/`dev` and `raw` describe the same file: that is + * what lets a stale lock be removed only if it is still the lock that was judged stale (removeStaleLock). * @param {string} lockPath — absolute path to the lock.pid file - * @returns {{pid: number|null, start: string|null, mtimeMs: number}|null} + * @returns {{pid: number|null, start: string|null, mtimeMs: number, ino: string, dev: string, raw: string, migrating: boolean, token: string|null}|null} * null when the file is absent/unreadable; pid null when content is * unparseable (corrupt or mid-write). */ function readLockFile(lockPath) { - let raw, mtimeMs; + let fd; + try { fd = fs.openSync(lockPath, 'r'); } catch { return null; } + let raw; let st; try { - raw = fs.readFileSync(lockPath, 'utf8'); - mtimeMs = fs.statSync(lockPath).mtimeMs; + st = fs.fstatSync(fd, { bigint: true }); + const buf = Buffer.alloc(Math.min(Number(st.size), 4096)); + const n = buf.length ? fs.readSync(fd, buf, 0, buf.length, 0) : 0; + raw = buf.subarray(0, n).toString('utf8'); } catch { return null; + } finally { + try { fs.closeSync(fd); } catch { /* */ } } const lines = raw.split('\n'); const pid = parseInt(lines[0].trim(), 10); - let start = null; + let meta = null; if (lines.length > 1 && lines[1].trim()) { - try { - const meta = JSON.parse(lines[1]); - if (meta && typeof meta.start === 'string') start = meta.start; - } catch { - // metadata line corrupt — fall back to PID-only semantics - } + try { meta = JSON.parse(lines[1]); } catch { /* metadata line corrupt — fall back to PID-only semantics */ } } return { pid: Number.isFinite(pid) && pid > 0 ? pid : null, - start, - mtimeMs, + start: meta && typeof meta.start === 'string' ? meta.start : null, + mtimeMs: Number(st.mtimeMs), + ino: String(st.ino), + dev: String(st.dev), + raw, + migrating: Boolean(meta && meta.migrating === true), + token: meta && typeof meta.token === 'string' ? meta.token : null, }; } @@ -886,6 +1993,76 @@ function _installExitHooks() { } } +/** An EIDENTITYLOCK error: `lock` is the holder's parsed lock (its pid becomes `holderPid`). */ +function lockError(message, lock) { + const err = new Error(message); + err.code = 'EIDENTITYLOCK'; + err.holderPid = lock && lock.pid !== undefined ? lock.pid : null; + return err; +} + +/** + * Whether `lock` is held: by a live process (lockIsHeldByLiveProcess), or by this one. A lock with this + * process's pid and another start time was another process's (the pid was reused): stale. + */ +function lockIsLive(lock) { + if (lock.pid === process.pid) return !(lock.start && selfStartTime() && lock.start !== selfStartTime()); + return lockIsHeldByLiveProcess(lock); +} + +/** + * Remove the stale lock `judged` (as readLockFile read it) from `lockPath` ONLY IF it is still that lock, and + * report whether it is gone. POSIX has no compare-and-delete, and the plain sequence (read, judge stale, + * unlink, create) let two starters at a dead holder's lock both hold it: the slower one's unlink removed the + * lock the faster one had just made (4 of 8 rounds with six starters). So removing one particular lock is a + * right of its own, taken exclusively: a claim file named after that lock (`.break-`), which is itself a lock of this kind. Holding it, the lock is read again and + * removed only if it is the same file (dev, ino), with the same content, still stale. Nobody else removes a + * lock without the claim for it, and a lock that is replaced has another claim name, so between the read and + * the unlink the file cannot change. A claim whose holder died (killed in those few system calls) is stale and + * removed the same way, one level up (bounded). False when another start holds the claim (it is removing it). + */ +function removeStaleLock(lockPath, judged, depth = 0) { + if (depth > 3) return false; + const tag = crypto.createHash('sha256').update(`${judged.dev}:${judged.ino}:${judged.raw}`).digest('hex').slice(0, 16); + const claim = `${lockPath}.break-${tag}`; + let got; + try { got = acquireLockAt(claim, {}, depth + 1); } catch { return false; } + if (!got.ok) return false; + try { + const now = readLockFile(lockPath); + if (!now) return true; + // Not the lock judged stale: leave it. The same file with the same content is the same lock, and its holder, + // dead when it was judged, is dead still (a reused pid has another start time). + if (now.dev !== judged.dev || now.ino !== judged.ino || now.raw !== judged.raw) return false; + try { fs.unlinkSync(lockPath); } catch (e) { if (e.code !== 'ENOENT') throw e; } + return true; + } finally { + _releaseLockIfOurs(claim); + } +} + +/** + * Take the lock file `lockPath` (O_EXCL create; content: pid, then {start, createdAt, ...meta}). A stale lock + * (see lockIsHeldByLiveProcess) is removed with removeStaleLock and the create tried again. Returns + * `{ ok: true }` when this call made the lock, `{ ok: false, lock }` when a live process holds it (`lock`: + * the holder's, possibly this process), or the lock is being taken by another start at this moment. + * @private + */ +function acquireLockAt(lockPath, meta = {}, depth = 0) { + let last = null; + for (let attempt = 0; attempt < 8; attempt++) { + // Made whole under a temp name and linked into place (createFileExclusive): a lock is never seen without its owner. + if (createFileExclusive(lockPath, `${process.pid}\n${JSON.stringify({ start: selfStartTime(), createdAt: Date.now(), ...meta })}\n`, 0o644)) return { ok: true }; + const lock = readLockFile(lockPath); + if (!lock) continue; // released between the create and the read: take it now + last = lock; + if (lockIsLive(lock)) return { ok: false, lock }; + if (!removeStaleLock(lockPath, lock, depth)) pause(2); // another start is removing it: look again in a moment + } + return { ok: false, lock: last, contended: true }; +} + /** * NAME-SUFFIXING IS DELETED (founder ruling, agent id = node id). * @@ -909,7 +2086,7 @@ function _installExitHooks() { * duplicate-identity races on the relay (close code 4004 / 4006 loops) * and ambiguous CMB delivery. * - * Lockfile lives at ~/.sym/nodes//lock.pid: line 1 is the holder's + * Lockfile lives at ~/.sym/nodes/by-id//lock.pid: line 1 is the holder's * PID (legacy parsers read exactly this), line 2 is JSON metadata with the * holder's process start time — the disambiguator that makes staleness * detection immune to PID reuse. On acquire: @@ -917,7 +2094,8 @@ function _installExitHooks() { * 2. If the lockfile's holder is LIVE (see lockIsHeldByLiveProcess: * alive PID *and* matching start time), throw EIDENTITYLOCK. * 3. Otherwise the lock is stale (dead PID, recycled PID, pre-boot - * legacy lock, aged-out corrupt content) — reclaim it. + * legacy lock, aged-out corrupt content) — it is removed if it is still + * that same lock (removeStaleLock: one remover per lock), and step 1 runs again. * * The release function deletes the lockfile. SymNode calls it on stop(); * a process-level 'exit' hook (plus default-disposition SIGHUP/SIGINT/ @@ -926,11 +2104,10 @@ function _installExitHooks() { * the next acquire by step 3. * * @param {string} name — node name - * @param {number} [_attempt] — internal EEXIST-race retry counter * @returns {() => void} release function — call to delete the lockfile * @throws {Error} if another process already holds the lock */ -function acquireIdentityLock(name, opts = {}, _attempt = 0) { +function acquireIdentityLock(name, opts = {}) { validateName(name); // The lock lives in the node's own directory: by nodeId since 0.14 (opts.dir); `nodes/` is a // link to it for one release, so a 0.13 process of the same agent contends for the same file. @@ -947,61 +2124,34 @@ function acquireIdentityLock(name, opts = {}, _attempt = 0) { }; }; - const lock = readLockFile(lockPath); - if (lock) { - // Same-PID re-acquisition is allowed: this happens in tests that - // create multiple SymNodes with the same name in sequence, in - // hot-reload scenarios, and in recovery flows where a single - // process re-initializes after a soft failure. The lock is meant - // to catch CROSS-PROCESS duplicates (the actual bug), not - // in-process re-init. While another holder in this process still - // holds it, the re-acquirer gets a release that does nothing: its - // release must not take the lock from under the live holder (final - // re-review, Finding 7a: an in-process export freed a running node's lock). - if (lock.pid === process.pid) { - if (_heldLockPaths.has(lockPath)) return function release() { /* the first holder releases it */ }; - return makeRelease(); - } - if (lockIsHeldByLiveProcess(lock)) { - const err = new Error( - `[SYM] Agent '${name}' is already live in PID ${lock.pid ?? 'unknown'} on this host.\n\n` + + const got = acquireLockAt(lockPath); + if (got.ok) return makeRelease(); + const lock = got.lock; + // Same-PID re-acquisition is allowed: this happens in tests that + // create multiple SymNodes with the same name in sequence, in + // hot-reload scenarios, and in recovery flows where a single + // process re-initializes after a soft failure. The lock is meant + // to catch CROSS-PROCESS duplicates (the actual bug), not + // in-process re-init. While another holder in this process still + // holds it, the re-acquirer gets a release that does nothing: its + // release must not take the lock from under the live holder (final + // re-review, Finding 7a: an in-process export freed a running node's lock). + if (lock && lock.pid === process.pid && !got.contended) { + if (_heldLockPaths.has(lockPath)) return function release() { /* the first holder releases it */ }; + return makeRelease(); + } + throw lockError( + got.contended + ? `[SYM] Agent '${name}' is being started by another process at this moment (its lock, ${lockPath}, is being taken over from a holder that is gone). ` + + 'This process is refused; start again if that other start fails.' + : `[SYM] Agent '${name}' is already live in PID ${lock && lock.pid !== null ? lock.pid : 'unknown'} on this host.\n\n` + ` An agent id names an AGENT, not a process. Two processes of the same agent ARE ` + `the same agent, so they share one identity and one store, and only one may hold ` + `the write lease at a time. This process is refused; the running one keeps it.\n\n` + ` If that other process is orphaned, stop it and start again. Do NOT work around ` + `this by choosing a different name: a different agent id is a DIFFERENT AGENT, with ` + - `its own keypair and its own history, and peers will treat it as a stranger.` - ); - err.code = 'EIDENTITYLOCK'; - err.holderPid = lock.pid; - throw err; - } - // Stale lock — delete the file so the openSync('wx') below succeeds. - // Without this unlink, the EEXIST retry below could never make progress. - try { fs.unlinkSync(lockPath); } catch {} - } - - // Atomic create: O_EXCL fails if another process creates the file - // between our read above and our write. The fallback re-reads and - // re-checks the holder, bounded so pathological contention can't - // recurse forever. - let fd; - try { - fd = fs.openSync(lockPath, 'wx'); - } catch (e) { - if (e.code === 'EEXIST' && _attempt < 5) { - // Lost the race — re-acquire (retry handles the new state) - return acquireIdentityLock(name, opts, _attempt + 1); - } - throw e; - } - fs.writeSync( - fd, - `${process.pid}\n${JSON.stringify({ start: selfStartTime(), createdAt: Date.now() })}\n`, - ); - fs.closeSync(fd); - - return makeRelease(); + `its own keypair and its own history, and peers will treat it as a stranger.`, + lock); } /** @@ -1028,10 +2178,21 @@ module.exports = { migrateNodeDir, renameIdentity, readTombstone, + readExportIntent, TOMBSTONE, + EXPORTING, + COMMIT_MARK, + LEGACY_ASIDE, IdentityHaltError, IdentityAbsentError, IdentityTombstonedError, + IdentityExportPendingError, + /** For relocation.js: durable writes, and installing a staged directory by one rename. */ + writeFileDurable, + writeNewFileSynced, + fsyncDir, + installNewDir, + PRIVATE_KEY_FILES, uuidv7, validateName, generateSigningKeyPair, diff --git a/lib/relocation.js b/lib/relocation.js index 12bda04a..25701ed0 100644 --- a/lib/relocation.js +++ b/lib/relocation.js @@ -5,14 +5,21 @@ * @description Moving a node to another host (sym 0.14, design D9.2). §3.1.3: "The private key MUST * NOT leave the node" — so a node is MOVED, never copied: * - * - `exportNode` holds the identity's lock throughout (no process can start it meanwhile), writes a - * TOMBSTONE at the source FIRST, then the bundle (fsynced, renamed into place, its directory - * fsynced). The source refuses to start that identity from then on (config.js - * IdentityTombstonedError), so no copy remains live: the relay's 4004 replacement cannot produce - * two live copies, because the source is tombstoned before the bundle exists. Once the bundle - * is durable the source LOSES THE KEY (security review): identity.json is rewritten without its - * private key, and the 0.13-compatible `nodes/` link goes, so a 0.13 build on the source - * host (a rollback) cannot load the moved identity either. + * - `exportNode` holds the identity's lock throughout (no process can start it meanwhile) and goes in + * this order, each step durable before the next (0.14.3; the tombstone used to come first, so a crash + * before the bundle existed left a node refused as "exported" with no bundle anywhere, and refused a + * second export): + * 1. an export INTENT (EXPORTING.json) beside the key: from here the source refuses to start the + * identity (IdentityExportPendingError) until the export is finished or abandoned; + * 2. the bundle (fsynced, renamed into place, its directory fsynced); + * 3. the TOMBSTONE beside the key, in the identity directory (SYM_IDENTITY_DIR when set: a rebuilt + * node directory used to take the tombstone with it, and the source started a second live copy); + * 4. the source LOSES THE KEY (security review): identity.json is rewritten without its private key + * (and so is any 0.13 copy the migration set aside), and the 0.13-compatible `nodes/` link goes, + * so a 0.13 build on the source host (a rollback) cannot load the moved identity either. + * An export interrupted at any step is resumable: run it again (before the tombstone it writes the + * bundle again; after it, it finishes the scrub). Before the tombstone it is abortable: `abortExport` + * removes the bundle it wrote and the intent, and the node starts here again. * - The bundle holds the node's directory — its identity, store, key registry, grants and learned * state — encrypted under an operator passphrase (scrypt → AES-256-GCM) or to a target host's * X25519 key (ephemeral X25519 + HKDF-SHA256 → AES-256-GCM). @@ -49,7 +56,7 @@ const config = require('./config'); const { keyFingerprint, isIdentityKey } = require('./roster-keys'); const FORMAT = 'sym-node-bundle-v1'; -const SKIP = new Set(['lock.pid', config.TOMBSTONE, 'interior.sock']); +const SKIP = new Set(['lock.pid', config.TOMBSTONE, config.EXPORTING, config.COMMIT_MARK, 'interior.sock']); const X25519_SPKI_PREFIX = Buffer.from('302a300506032b656e032100', 'hex'); const X25519_PKCS8_PREFIX = Buffer.from('302e020100300506032b656e04220420', 'hex'); const ED25519_PKCS8_PREFIX = Buffer.from('302e020100300506032b657004220420', 'hex'); @@ -82,7 +89,7 @@ function listFiles(root, rel = '') { const out = []; for (const d of fs.readdirSync(path.join(root, rel), { withFileTypes: true })) { const p = path.join(rel, d.name); - if (!rel && SKIP.has(d.name)) continue; + if (!rel && (SKIP.has(d.name) || d.name.startsWith('lock.pid.'))) continue; if (d.name.endsWith('.tmp')) continue; if (d.isDirectory()) out.push(...listFiles(root, p)); else if (d.isFile()) out.push(p); @@ -162,7 +169,8 @@ function publicOf(privateKeyB64url) { } /** - * Move a node off this host: tombstone it, then write its encrypted bundle. + * Move a node off this host: an intent beside the key, its encrypted bundle, then the tombstone, then the + * source's key scrubbed. Run again after an interruption, it finishes what was begun. * @param {object} o * @param {string} [o.name] * @param {string} [o.nodeId] @@ -176,8 +184,22 @@ function exportNode({ name, nodeId, out, passphrase, toHostKey, copyable = false const id = nodeId || (name && config.nodeIdForName(name)); if (!id) throw new config.IdentityAbsentError(name ? `"${name}"` : 'the node'); const dir = config.nodeDirById(id); - if (config.readTombstone(id)) throw new Error(`relocation: ${id} is already tombstoned here (it was exported)`); - const identity = config.loadIdentity({ nodeId: id, name, create: false }); + const tomb = config.readTombstone(id); + if (tomb) { + // Tombstoned: the bundle was durable. An export interrupted before the source lost its key is finished here. + const left = readIdentityJson(config.identityDirById(id)); + if (left && left.privateKey) { + scrubSource(id, left); + return { nodeId: id, name: left.name, files: 0, out: tomb.bundle || null, fingerprint: tomb.fingerprint || keyFingerprint(left.publicKey), resumed: 'scrub' }; + } + throw new Error(`relocation: ${id} is already tombstoned here (it was exported${tomb.bundle ? ` to ${tomb.bundle}` : ''})`); + } + const identity = config.loadIdentity({ nodeId: id, name, create: false, allowPendingExport: true }); + // An export that was interrupted before its tombstone is begun again from the start; the half-written bundle it + // may have left (`..tmp`) goes. A complete bundle it wrote is left where it is: the new one supersedes it, + // and a host imports only one of them. + const pending = config.readExportIntent(id); + if (pending && typeof pending.out === 'string') removeTempBundles(pending.out); // A node running in THIS process holds the lock too (final re-review, Finding 7a). if (config.lockHeldInProcess(dir)) throw new Error(`relocation: ${identity.name} is running in this process; stop it before moving it`); const lock = config.readLockFile(path.join(dir, 'lock.pid')); @@ -200,19 +222,26 @@ function exportNode({ name, nodeId, out, passphrase, toHostKey, copyable = false } } +/** @private The identity.json in `dir`, as JSON, or null. */ +function readIdentityJson(dir) { + try { return JSON.parse(fs.readFileSync(path.join(dir, 'identity.json'), 'utf8')); } catch { return null; } +} + /** @private The export, under the identity's lock. */ function exportLocked({ id, dir, identity, out, passphrase, toHostKey }) { - // THE TOMBSTONE FIRST. From here this host refuses to start the identity, before any bundle exists. - const tomb = { nodeId: id, name: identity.name, movedAt: Date.now(), bundle: path.basename(out), fingerprint: keyFingerprint(identity.publicKey) }; - const tombPath = path.join(dir, config.TOMBSTONE); - const fd = fs.openSync(tombPath, 'wx'); - try { fs.writeSync(fd, JSON.stringify(tomb, null, 2)); fs.fsyncSync(fd); } finally { fs.closeSync(fd); } + const idDir = config.identityDirById(id); + const outAbs = path.resolve(out); + const exportedAt = Date.now(); + const fingerprint = keyFingerprint(identity.publicKey); + // 1. THE INTENT, beside the key. From here this host refuses to start the identity until this export is + // finished or abandoned: a bundle may exist from the next step on. + config.writeFileDurable(path.join(idDir, config.EXPORTING), JSON.stringify({ nodeId: id, name: identity.name, out: outAbs, startedAt: exportedAt }, null, 2), 0o600); + const tomb = { nodeId: id, name: identity.name, movedAt: exportedAt, bundle: outAbs, fingerprint }; // Each file with its mode (final re-review, Finding 7b), so a 0600 file stays 0600 on the target. - const files = listFiles(dir).map((rel) => ({ path: rel.split(path.sep).join('/'), mode: fs.statSync(path.join(dir, rel)).mode & 0o777, data: fs.readFileSync(path.join(dir, rel)).toString('base64') })); - // identity.json lives elsewhere when SYM_IDENTITY_DIR is set: it travels in the bundle all the same. - if (!files.some((f) => f.path === 'identity.json')) { - files.push({ path: 'identity.json', mode: 0o600, data: Buffer.from(JSON.stringify(identity, null, 2)).toString('base64') }); - } + const files = listFiles(dir).filter((rel) => rel !== 'identity.json') + .map((rel) => ({ path: rel.split(path.sep).join('/'), mode: fs.statSync(path.join(dir, rel)).mode & 0o777, data: fs.readFileSync(path.join(dir, rel)).toString('base64') })); + // The identity travels as it was loaded (from SYM_IDENTITY_DIR when set), never a stray copy in the node directory. + files.push({ path: 'identity.json', mode: 0o600, data: Buffer.from(JSON.stringify(identity, null, 2)).toString('base64') }); const contents = { format: FORMAT, nodeId: id, name: identity.name, publicKey: identity.publicKey, exportedAt: tomb.movedAt, files }; // The sealing header is chosen first, so the node's signature covers it (the target host key and // the ephemeral key included): the bundle is the node's own statement, for that host only. @@ -225,14 +254,60 @@ function exportLocked({ id, dir, identity, out, passphrase, toHostKey }) { const headerSig = signHeader(header, identity.privateKey); const plain = Buffer.from(JSON.stringify({ ...contents, signature }), 'utf8'); const bundle = { ...header, headerSig, ...sealWith(plain, sealing) }; + // 2. THE BUNDLE, durable. const tmp = `${out}.${process.pid}.tmp`; const bfd = fs.openSync(tmp, 'w', 0o600); try { fs.writeSync(bfd, JSON.stringify(bundle)); fs.fsyncSync(bfd); } finally { fs.closeSync(bfd); } fs.renameSync(tmp, out); - fsyncDir(path.dirname(path.resolve(out))); - // The bundle is durable: the source loses the key, and the 0.13 path to it. + fsyncDir(path.dirname(outAbs)); + // 3. THE TOMBSTONE, beside the key (it outlives a rebuilt node directory), then the intent goes. + config.writeFileDurable(path.join(idDir, config.TOMBSTONE), JSON.stringify(tomb, null, 2), 0o600); + try { fs.unlinkSync(path.join(idDir, config.EXPORTING)); } catch { /* */ } + fsyncDir(idDir); + // 4. The source loses the key, and the 0.13 path to it. scrubSource(id, identity); - return { nodeId: id, name: identity.name, files: files.length, out, fingerprint: tomb.fingerprint }; + return { nodeId: id, name: identity.name, files: files.length, out, fingerprint }; +} + +/** @private Remove the bundles an export to `out` left half written (`..tmp`). */ +function removeTempBundles(out) { + const base = `${path.basename(out)}.`; + let names = []; + try { names = fs.readdirSync(path.dirname(out)); } catch { return; } + for (const n of names) if (n.startsWith(base) && /^\d+\.tmp$/.test(n.slice(base.length))) { try { fs.unlinkSync(path.join(path.dirname(out), n)); } catch { /* */ } } + fsyncDir(path.dirname(out)); +} + +/** + * Abandon an export that began and did not finish (an intent, no tombstone): the bundle it wrote is removed, + * and the intent, so the node starts on this host again. A bundle copied elsewhere meanwhile is not reachable + * from here: delete that copy too. After the tombstone it is too late (the bundle is the node now): refused. + * @param {{ name?: string, nodeId?: string }} o + * @returns {{ nodeId: string, aborted: boolean, bundle: string|null, bundleRemoved: boolean }} + */ +function abortExport({ name, nodeId } = {}) { + const id = nodeId || (name && config.nodeIdForName(name)); + if (!id) throw new config.IdentityAbsentError(name ? `"${name}"` : 'the node'); + const tomb = config.readTombstone(id); + if (tomb) throw new Error(`relocation: the export of ${id} finished${tomb.bundle ? ` (${tomb.bundle})` : ''} and it is tombstoned here: the bundle is the node now; import it where it goes`); + const intent = config.readExportIntent(id); + if (!intent) return { nodeId: id, aborted: false, bundle: null, bundleRemoved: false }; + const dir = config.nodeDirById(id); + if (config.lockHeldInProcess(dir)) throw new Error(`relocation: ${id} is held by a node in this process`); + const release = config.acquireIdentityLock(intent.name || (name || 'node'), { dir }); + try { + let bundleRemoved = false; + if (typeof intent.out === 'string' && intent.out) { + try { fs.unlinkSync(intent.out); bundleRemoved = true; } catch (e) { if (e.code !== 'ENOENT') throw e; } + removeTempBundles(intent.out); + } + for (const d of new Set([config.identityDirById(id), dir])) { + try { fs.unlinkSync(path.join(d, config.EXPORTING)); fsyncDir(d); } catch { /* */ } + } + return { nodeId: id, aborted: true, bundle: intent.out || null, bundleRemoved }; + } finally { + try { release(); } catch { /* */ } + } } /** @private fsync a directory (a rename in it is durable after this); best effort where unsupported. */ @@ -242,14 +317,25 @@ function fsyncDir(dir) { } /** - * @private After a durable export: identity.json (in the node dir and SYM_IDENTITY_DIR) without its - * private key, written atomically, and the 0.13-compatible `nodes/` links removed. + * @private After a durable export: identity.json (in the node dir and SYM_IDENTITY_DIR, and in any 0.13 + * directory the migration set aside) without its private key, written atomically, and the 0.13-compatible + * `nodes/` links removed. */ function scrubSource(id, identity) { const { privateKey, ...rest } = identity; void privateKey; const scrubbed = JSON.stringify({ ...rest, privateKey: null, movedAt: Date.now() }, null, 2); - for (const d of new Set([config.nodeDirById(id), config.identityDirById(id)])) { + const dirs = new Set([config.nodeDirById(id), config.identityDirById(id)]); + for (const root of [config.NODES_DIR, process.env.SYM_IDENTITY_DIR].filter(Boolean)) { + let names = []; + try { names = fs.readdirSync(path.join(root, config.LEGACY_ASIDE)); } catch { /* none */ } + for (const n of names) { + const d = path.join(root, config.LEGACY_ASIDE, n); + const j = readIdentityJson(d); + if (j && j.nodeId === id && j.privateKey) dirs.add(d); + } + } + for (const d of dirs) { const f = path.join(d, 'identity.json'); if (!fs.existsSync(f)) continue; const tmp = `${f}.${process.pid}.tmp`; @@ -360,36 +446,43 @@ function importNode({ from, passphrase, hostPrivateKey, expect, name } = {}) { throw new Error(`relocation: refusing ${identity.nodeId}: the bundle was exported at or before this host moved the node away (a replay of an older move)`); } const dest = config.nodeDirById(identity.nodeId); - if (fs.existsSync(dest) && !tomb) throw new Error(`relocation: ${identity.nodeId} already lives on this host`); + const idDest = config.identityDirById(identity.nodeId); + const split = idDest !== dest; + const here = fs.existsSync(dest) || (split && fs.existsSync(path.join(idDest, 'identity.json'))); + if (here && !tomb) throw new Error(`relocation: ${identity.nodeId} already lives on this host`); const label = name || identity.name; config.validateName(label); const existing = config.nodeIdForName(label); if (existing && existing !== identity.nodeId) throw new Error(`relocation: the name "${label}" already indexes another node here; import with another name`); - // STAGED: everything is written to a directory of its own, and renamed into place last, so an - // import refused or failing on the way installs nothing (final re-review, Finding 7b). - const staged = `${dest}.import-${process.pid}-${crypto.randomBytes(4).toString('hex')}`; - try { - for (const f of plain.files) { - const rel = String(f.path); - if (rel.includes('..') || path.isAbsolute(rel) || rel.includes('\\')) throw new Error(`relocation: refusing a bundle path outside the node: ${rel}`); - const p = path.join(staged, ...rel.split('/')); - config.ensureDir(path.dirname(p)); - const mode = rel === 'identity.json' ? 0o600 : (Number.isSafeInteger(f.mode) ? (f.mode & 0o777) : 0o644); - fs.writeFileSync(p, Buffer.from(f.data, 'base64'), { mode }); - fs.chmodSync(p, mode); - } - } catch (err) { - fs.rmSync(staged, { recursive: true, force: true }); - throw err; + for (const f of plain.files) { + const rel = String(f.path); + if (rel.includes('..') || path.isAbsolute(rel) || rel.includes('\\') || !rel) throw new Error(`relocation: refusing a bundle path outside the node: ${rel}`); } - if (fs.existsSync(dest)) { - // It was moved away from here before: the tombstoned copy is archived, and the node comes back. - fs.renameSync(dest, `${dest}.moved-${Date.now()}`); + // STAGED: everything is written to a directory of its own and installed by one rename (config.installNewDir), + // so an import refused or failing on the way installs nothing (final re-review, Finding 7b). Directories are + // 0700, key files 0600, and with SYM_IDENTITY_DIR the private key goes ONLY into the identity directory. + const keyFiles = new Set(config.PRIVATE_KEY_FILES); + const write = (root, f) => { + const rel = String(f.path); + const parts = rel.split('/'); + let d = root; + for (const part of parts.slice(0, -1)) { + d = path.join(d, part); + if (!fs.existsSync(d)) { fs.mkdirSync(d, { mode: 0o700 }); fs.chmodSync(d, 0o700); } + } + const mode = keyFiles.has(parts[parts.length - 1]) ? 0o600 : (Number.isSafeInteger(f.mode) ? (f.mode & 0o777) : 0o644); + config.writeNewFileSynced(path.join(root, ...parts), Buffer.from(f.data, 'base64'), mode); + }; + const archive = (d) => { if (fs.existsSync(d)) { fs.renameSync(d, path.join(path.dirname(d), `.${identity.nodeId}.moved-${Date.now()}`)); fsyncDir(path.dirname(d)); } }; + // What a node directory holds that is this host's own state (a lock, a tombstone, an export intent, the commit + // record) is never taken from a bundle. + const hostState = (rel) => SKIP.has(rel) || rel.startsWith('lock.pid.'); + config.installNewDir(dest, identity.nodeId, (staging) => { + for (const f of plain.files) if (!(split && String(f.path) === 'identity.json') && !hostState(String(f.path))) write(staging, f); + }, 'import', { beforeCommit: () => archive(dest) }); // the tombstoned copy of a node coming back is kept aside + if (split) { + config.installNewDir(idDest, identity.nodeId, (staging) => write(staging, idFile), 'import', { beforeCommit: () => archive(idDest) }); } - fs.renameSync(staged, dest); - fsyncDir(path.dirname(dest)); - const idDirExt = config.identityDirById(identity.nodeId); - if (idDirExt !== dest) { config.ensureDir(idDirExt); fs.writeFileSync(path.join(idDirExt, 'identity.json'), Buffer.from(idFile.data, 'base64'), { mode: 0o600 }); } try { fs.appendFileSync(path.join(config.SYM_DIR, CONSUMED_FILE), JSON.stringify({ nodeId: identity.nodeId, signature: plain.signature, exportedAt: plain.exportedAt, importedAt: Date.now() }) + '\n', { mode: 0o600 }); } catch { /* best effort: the tombstone check still refuses an older bundle */ } // indexName is internal to config; loadIdentity by nodeId + name writes the index. @@ -397,4 +490,4 @@ function importNode({ from, passphrase, hostPrivateKey, expect, name } = {}) { return { nodeId: identity.nodeId, name: label, files: plain.files.length, pinnedBy }; } -module.exports = { exportNode, importNode, hostKey, FORMAT }; +module.exports = { exportNode, abortExport, importNode, hostKey, FORMAT }; diff --git a/package-lock.json b/package-lock.json index 6f21af2d..7039b1f8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "license": "Apache-2.0", "dependencies": { "bonjour-service": "^1.3.0", diff --git a/package.json b/package.json index cc55b397..01d22355 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "description": "Infrastructure and protocol for multi-agent collective intelligence", "main": "lib/node.js", "exports": { diff --git a/tests/identity-commit.test.js b/tests/identity-commit.test.js new file mode 100644 index 00000000..90f17816 --- /dev/null +++ b/tests/identity-commit.test.js @@ -0,0 +1,677 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME to a temp sandbox before lib/config loads + +/** + * 0.14.3: the state of an identity is read from records committed in one atomic step, never inferred from + * whatever is at a path. + * + * An independent review (R1) of the 0.14.3 candidate (632b95d) proved, by running them, that a partial + * destination, an unreadable index, a node directory without identity.json and a stale lock's content were + * each read as a state of the identity, and that each patch for one of them had made another. The design that + * replaces them (lib/config.js): + * - a 0.13 directory is COPIED to a staging directory beside its by-id destination, identity.json last, + * synced, verified, and installed by ONE rename; the 0.13 directory is then set aside, never deleted; + * - a name is claimed by an exclusive create of its index entry, after the identity is installed; + * - an index entry is "absent" only on ENOENT; a mint happens only when nothing on the host is a trace of + * the name; a stale lock is removed only by the one process holding the right to remove that lock. + * Every test here fails on 632b95d and passes on this branch; each names the R1 finding it holds. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const crypto = require('crypto'); +const { spawn } = require('child_process'); +const cfg = require('../lib/config'); +const relocation = require('../lib/relocation'); + +const { NODES_DIR, IdentityHaltError } = cfg; +const posix = process.platform !== 'win32'; +const modeTest = posix ? it : it.skip; +const LIB = path.resolve(__dirname, '../lib'); + +let seq = 0; +const uniq = (b) => `${b}-${process.pid}-${++seq}-${Math.random().toString(36).slice(2, 6)}`; +const read = (p) => fs.readFileSync(p, 'utf8'); +const exists = (p) => fs.existsSync(p); +const modeOf = (p) => fs.statSync(p).mode & 0o777; +const isLink = (p) => { try { return fs.lstatSync(p).isSymbolicLink(); } catch { return false; } }; +const indexFile = (name) => path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); +const ASIDE = cfg.LEGACY_ASIDE || '.legacy-0.13'; + +function makeIdentity(name, extra = {}) { + const kp = crypto.generateKeyPairSync('ed25519'); + return { + nodeId: cfg.uuidv7(), name, hostname: 'host.local', createdAt: 1, + publicKey: kp.publicKey.export({ format: 'der', type: 'spki' }).subarray(12).toString('base64url'), + privateKey: kp.privateKey.export({ format: 'der', type: 'pkcs8' }).subarray(16).toString('base64url'), + ...extra, + }; +} + +function put(file, text, mode = 0o644) { + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, text); + fs.chmodSync(file, mode); +} + +/** A 0.13 node: nodes// with identity.json and a small store. */ +function legacyNode(name, { where = path.join(NODES_DIR, name), identity = makeIdentity(name), files = 6 } = {}) { + put(path.join(where, 'identity.json'), JSON.stringify(identity, null, 2)); + for (let i = 0; i < files; i++) put(path.join(where, 'cmbs', `c${i}.json`), JSON.stringify({ i, pad: 'x'.repeat(300) })); + put(path.join(where, 'inbox.json'), '{"seq":7}'); + return identity; +} + +/** + * Every path under `dir` with its kind and bytes (links followed), sorted. The runtime files a start leaves in a + * node directory (its lock, a claim to break one, the temp file a lock is written under before it is linked into + * place) are not the node's data and are left out: a process killed at the wrong moment leaves one behind. + */ +function treeOf(dir) { + const out = []; + const walk = (d, rel) => { + for (const e of fs.readdirSync(d).sort()) { + if (!rel && (e === 'lock.pid' || e.startsWith('lock.pid.') || (e.startsWith('.lock.pid') && e.endsWith('.tmp')))) continue; + const p = path.join(d, e); const r = rel ? `${rel}/${e}` : e; + const st = fs.statSync(p); + if (st.isDirectory()) { out.push(`D ${r}`); walk(p, r); } else out.push(`F ${r} ${crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex')}`); + } + }; + walk(dir, ''); + return out; +} + +/** Installed identities (non-dot by-id entries) whose name is `name`. */ +function identitiesNamed(name) { + const found = []; + const roots = [path.join(NODES_DIR, 'by-id'), process.env.SYM_IDENTITY_DIR && path.join(process.env.SYM_IDENTITY_DIR, 'by-id')].filter(Boolean); + for (const root of roots) { + let ds = []; try { ds = fs.readdirSync(root); } catch { continue; } + for (const d of ds) { + if (d.startsWith('.')) continue; + try { if (JSON.parse(read(path.join(root, d, 'identity.json'))).name === name) found.push(d); } catch { /* */ } + } + } + return [...new Set(found)]; +} + +function withIdentityDir(fn) { + const prev = process.env.SYM_IDENTITY_DIR; + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-idroot-')); + process.env.SYM_IDENTITY_DIR = root; + const done = () => { + if (prev === undefined) delete process.env.SYM_IDENTITY_DIR; else process.env.SYM_IDENTITY_DIR = prev; + fs.rmSync(root, { recursive: true, force: true }); + }; + let r; + try { r = fn(root); } catch (e) { done(); throw e; } + if (r && typeof r.then === 'function') return r.finally(done); + done(); + return r; +} + +/** Replace fs. for the duration of `body`. */ +function patchFs(fn, wrap, body) { + const original = fs[fn]; + fs[fn] = wrap(original); + try { return body(); } finally { fs[fn] = original; } +} + +/** Replace several fs functions at once: `wraps` maps a name to (original) => replacement. */ +function patchMany(wraps, body) { + const originals = {}; + for (const [k, w] of Object.entries(wraps)) { originals[k] = fs[k]; fs[k] = w(originals[k]); } + try { return body(); } finally { for (const k of Object.keys(originals)) fs[k] = originals[k]; } +} + +const MUTATING = ['renameSync', 'copyFileSync', 'mkdirSync', 'writeFileSync', 'openSync', 'unlinkSync', 'rmdirSync', 'rmSync', 'symlinkSync', 'linkSync', 'chmodSync', 'writeSync', 'fsyncSync']; +class Injected extends Error {} + +/** Run `body`, throwing an Injected error at the k-th mutating fs call (1-based); returns the number of calls made. */ +function failAt(k, body) { + let n = 0; + const wraps = {}; + for (const fn of MUTATING) { + wraps[fn] = (orig) => function (...a) { + if (fn === 'openSync' && !/[wax+]/.test(String(a[1] || 'r'))) return orig.apply(fs, a); // a read is not a step + if (++n === k) throw new Injected(`injected at step ${k} (${fn})`); + return orig.apply(fs, a); + }; + } + // Whatever the body throws once a step failed (the failure, or what the code made of it) is the crash; the + // state it leaves is what the caller checks. + try { patchMany(wraps, body); } catch { /* the crash */ } + return n; +} + +describe('R1-1: a migration interrupted at any step changes nothing in the 0.13 directory, and the next start loads the whole store', () => { + for (const withHusk of [false, true]) { + it(`every step of the migration${withHusk ? ', over a by-id directory holding a stale lock (the 0.14.0-0.14.2 husk)' : ''}`, () => { + // 632b95d moved the store entry by entry: an interruption left part of it in by-id and the rest in nodes/, + // and the copy across volumes deleted the source as it went (R1 measured files missing after kill -9). + const probe = uniq('r11-probe'); + legacyNode(probe); + const steps = failAt(Infinity, () => cfg.migrateNodeDir(probe)); + assert.ok(steps > 10, `the migration took ${steps} steps: the test measures something`); + let checked = 0; + for (let k = 1; k <= steps; k++) { + const name = uniq('r11'); + const id = legacyNode(name); + if (withHusk) put(path.join(cfg.nodeDirById(id.nodeId), 'lock.pid'), '2147483646\n'); + const before = treeOf(path.join(NODES_DIR, name)); + failAt(k, () => cfg.migrateNodeDir(name)); + // The 0.13 directory: in place and byte-identical, or set aside byte-identical with the link in its place. + const legacy = path.join(NODES_DIR, name); + const where = isLink(legacy) ? path.join(NODES_DIR, ASIDE, name) : legacy; + if (!exists(where)) assert.fail(`step ${k}: the 0.13 directory is no longer at nodes/${name} and was not set aside (moved or deleted, not kept)`); + assert.deepStrictEqual(treeOf(where), before, `step ${k}: the 0.13 directory (${path.relative(NODES_DIR, where)}) is byte-identical`); + // What a 0.13 rollback reads: nodes//identity.json is the original key. + assert.strictEqual(JSON.parse(read(path.join(legacy, 'identity.json'))).privateKey, id.privateKey, `step ${k}: a 0.13 reader still finds the key at nodes/`); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId, `step ${k}: the same agent`); + assert.strictEqual(loaded.privateKey, id.privateKey, `step ${k}: the same key`); + const dest = cfg.nodeDirById(id.nodeId); + assert.deepStrictEqual(treeOf(dest).filter((l) => !l.includes(' identity.json') && !l.includes(` ${cfg.COMMIT_MARK || '.sym-commit'}`)), + before.filter((l) => !l.includes(' identity.json')), `step ${k}: the whole store is in by-id`); + assert.deepStrictEqual(identitiesNamed(name), [id.nodeId], `step ${k}: exactly one identity`); + checked++; + } + assert.strictEqual(checked, steps); + }); + } +}); + +describe('R1-2: a node directory without its identity is never handed to a new one', () => { + for (const pinned of [false, true]) { + it(`SYM_IDENTITY_DIR's volume not mounted at boot (${pinned ? 'pinned' : 'by name'}): the start halts, mints nothing, touches nothing; mounted, the agent loads with its store`, () => { + withIdentityDir((mnt) => { + const name = uniq('r12'); + const vol = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-keyvol-')); + const id = makeIdentity(name); + put(path.join(vol, name, 'identity.json'), JSON.stringify(id, null, 2)); // the key lives on the volume + for (let i = 0; i < 3; i++) put(path.join(NODES_DIR, name, 'cmbs', `c${i}.json`), `{"i":${i}}`); // the store + const store = treeOf(path.join(NODES_DIR, name)); + const opts = pinned ? { name, nodeId: id.nodeId, create: false } : { name }; + assert.throws(() => cfg.loadIdentity(opts), (e) => (pinned ? e.code === 'EIDENTITYABSENT' : e instanceof IdentityHaltError) && /holds a node's files/.test(e.message) && /mounted/.test(e.message)); + assert.deepStrictEqual(identitiesNamed(name), [], 'nothing was minted'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the name was not claimed'); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, name)), store, 'the store is where it was'); + // mount it + fs.rmSync(mnt, { recursive: true, force: true }); fs.renameSync(vol, mnt); + const loaded = cfg.loadIdentity(opts); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'c2.json')), 'with its store'); + }); + }); + } +}); + +describe('R1-3: what a 0.13 node kept behind a link is followed and copied', () => { + it('nodes/ a link to the node\'s directory on another volume: the original loads, the volume is untouched', () => { + const name = uniq('r13-dir'); + const durable = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-durable-')), name); + const id = legacyNode(name, { where: durable }); + const before = treeOf(durable); + fs.mkdirSync(NODES_DIR, { recursive: true }); + fs.symlinkSync(durable, path.join(NODES_DIR, name), 'dir'); + for (const opts of [{ name }, { name, nodeId: id.nodeId, create: false }]) { + const loaded = cfg.loadIdentity(opts); + assert.strictEqual(loaded.privateKey, id.privateKey, JSON.stringify(Object.keys(opts))); + } + assert.deepStrictEqual(treeOf(durable), before, 'the durable directory is byte-unchanged'); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'c5.json')), 'the store was copied'); + assert.deepStrictEqual(identitiesNamed(name), [id.nodeId]); + }); + + it('identity.json a RELATIVE link to a key file elsewhere: the original loads, and the installed identity is a file, not a link that now leads nowhere', () => { + const name = uniq('r13-rel'); + const id = makeIdentity(name); + put(path.join(cfg.SYM_DIR, 'keys', `${name}.json`), JSON.stringify(id), 0o600); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + fs.symlinkSync(`../../keys/${name}.json`, path.join(NODES_DIR, name, 'identity.json')); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.privateKey, id.privateKey); + const installed = path.join(cfg.nodeDirById(id.nodeId), 'identity.json'); + assert.ok(!isLink(installed), 'a file'); + assert.strictEqual(JSON.parse(read(installed)).privateKey, id.privateKey); + assert.strictEqual(JSON.parse(read(path.join(cfg.SYM_DIR, 'keys', `${name}.json`))).privateKey, id.privateKey, 'the key file it led to is untouched'); + }); +}); + +describe('the staged commit: what it checks before the one rename', () => { + it('a copy that does not read back as its source is not installed, and the source is untouched', () => { + const name = uniq('verify'); + const id = legacyNode(name); + const before = treeOf(path.join(NODES_DIR, name)); + assert.throws(() => patchFs('copyFileSync', (o) => function (a, b, ...r) { + const out = o.call(fs, a, b, ...r); + if (String(b).endsWith('c3.json')) fs.appendFileSync(b, ' '); // a copy that came out different + return out; + }, () => cfg.migrateNodeDir(name)), /does not match/); + assert.ok(!exists(cfg.nodeDirById(id.nodeId)), 'nothing installed'); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, name)), before, 'the source is untouched'); + assert.strictEqual(cfg.nodeIdForName(name), null); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'the next start copies it right'); + }); + + it('a directory inside the 0.13 node that is a relative link is copied as the directory it leads to', () => { + const name = uniq('sublink'); + const id = makeIdentity(name); + put(path.join(NODES_DIR, name, 'identity.json'), JSON.stringify(id)); + put(path.join(NODES_DIR, `${name}-store`, 'x.json'), '{"x":1}'); + fs.symlinkSync(`../${name}-store`, path.join(NODES_DIR, name, 'cmbs'), 'dir'); + cfg.loadIdentity({ name }); + const cmbs = path.join(cfg.nodeDirById(id.nodeId), 'cmbs'); + assert.ok(!isLink(cmbs) && read(path.join(cmbs, 'x.json')) === '{"x":1}', 'a real directory holding the files'); + }); + + it('a by-id husk whose lock a live process holds is not replaced: EIDENTITYLOCK, nothing changed', async () => { + const name = uniq('husk-live'); + const id = legacyNode(name); + const husk = cfg.nodeDirById(id.nodeId); + fs.mkdirSync(husk, { recursive: true }); + put(path.join(husk, 'inbox.json'), '{"seq":99}'); + const child = spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.join(LIB, 'config.js'))}); + c.acquireIdentityLock('h', { dir: ${JSON.stringify(husk)} }); console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + try { + await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('holder exited'))); }); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e.code === 'EIDENTITYLOCK'); + assert.strictEqual(read(path.join(husk, 'inbox.json')), '{"seq":99}', 'the husk is where it was'); + assert.ok(!exists(path.join(husk, 'identity.json'))); + assert.ok(!isLink(path.join(NODES_DIR, name)), 'the 0.13 directory is where it was'); + } finally { child.kill('SIGKILL'); await new Promise((r) => child.once('exit', r)); } + }); + + it('a husk replaced by another start\'s commit while this start looks at it is never set aside (found by the multi-process fuzz)', () => { + const name = uniq('husk-race'); + const id = legacyNode(name); + const target = cfg.nodeDirById(id.nodeId); + put(path.join(target, 'lock.pid'), '2147483646\n'); // the 0.14.0-0.14.2 husk: a stale lock + let other = false; + patchFs('readdirSync', (orig) => function (p, ...rest) { + // After this start removed the stale lock and before it reads what is left, another start commits its copy. + if (!other && String(p) === target && !exists(path.join(target, 'lock.pid')) && !exists(path.join(target, cfg.COMMIT_MARK))) { + other = true; + cfg.migrateNodeDir(name); + } + return orig.call(fs, p, ...rest); + }, () => { cfg.migrateNodeDir(name); }); + assert.ok(other, 'the other start ran in between'); + assert.ok(!exists(path.join(path.dirname(target), `.${id.nodeId}.husk`)), 'no installed directory was set aside'); + assert.strictEqual(JSON.parse(read(path.join(target, 'identity.json'))).privateKey, id.privateKey); + assert.deepStrictEqual(identitiesNamed(name), [id.nodeId]); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + }); + + it('a staging directory of a live process is left alone; one of a process that is gone is cleared', () => { + const name = uniq('partials'); + const id = legacyNode(name); + const live = path.join(NODES_DIR, 'by-id', `.${id.nodeId}.partial-${process.ppid}-aaaaaaaa`); // our parent is alive + const dead = path.join(NODES_DIR, 'by-id', `.${id.nodeId}.partial-2147483646-bbbbbbbb`); + put(path.join(live, 'x'), '1'); put(path.join(dead, 'x'), '1'); + cfg.loadIdentity({ name }); + assert.ok(exists(live), 'a live process\'s staging is not touched'); + assert.ok(!exists(dead), 'a dead one\'s is cleared'); + fs.rmSync(live, { recursive: true, force: true }); + }); +}); + +describe('R1-4: a stale identity lock is removed only if it is still the lock judged stale', () => { + it('a start that judged a dead holder\'s lock stale does not remove the lock another start took meanwhile', async () => { + // 632b95d read the lock, judged it stale, unlinked whatever was there, and created its own: with six starters at a + // SIGKILLed holder's lock, 4 of 8 rounds had two or more simultaneous holders. + const name = uniq('r14'); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-r14-')); + const lockPath = path.join(dir, 'lock.pid'); + put(lockPath, '2147483646\n{"start":"Thu Jan 1 00:00:00 1970","createdAt":1}\n'); // dead holder + const other = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { stdio: 'ignore' }); + try { + const otherLock = `${other.pid}\n${JSON.stringify({ start: cfg.processStartTime(other.pid), createdAt: Date.now() })}\n`; + let raced = false; + // The other start runs after this one READ the stale lock (its judgement) and before the first thing this one + // writes in the lock's directory (632b95d: its unlink of the lock; here: its claim to remove that lock): it + // removes the stale lock and takes its own. + let judged = false; + const inDir = (p) => path.dirname(String(p)) === dir; + const race = (p) => { if (judged && !raced && inDir(p)) { raced = true; fs.unlinkSync(lockPath); fs.writeFileSync(lockPath, otherLock); } }; + let err = null; + patchMany({ + readFileSync: (orig) => function (p, ...rest) { if (String(p) === lockPath) judged = true; return orig.call(fs, p, ...rest); }, + unlinkSync: (orig) => function (p, ...rest) { race(p); return orig.call(fs, p, ...rest); }, + linkSync: (orig) => function (a, b, ...rest) { race(b); return orig.call(fs, a, b, ...rest); }, + openSync: (orig) => function (p, flags, ...rest) { + if (/[wax+]/.test(String(flags || 'r'))) race(p); else if (String(p) === lockPath) judged = true; + return orig.call(fs, p, flags, ...rest); + }, + }, () => { + try { cfg.acquireIdentityLock(name, { dir }); } catch (e) { err = e; } + }); + assert.ok(raced, 'the other start ran between the judgement and the removal'); + assert.ok(err && err.code === 'EIDENTITYLOCK' && err.holderPid === other.pid, `refused, naming the live holder (got ${err && err.code})`); + assert.strictEqual(read(lockPath), otherLock, 'the other start\'s lock is intact'); + assert.deepStrictEqual(fs.readdirSync(dir).filter((n) => n !== 'lock.pid'), [], 'no claim file left behind'); + } finally { other.kill('SIGKILL'); fs.rmSync(dir, { recursive: true, force: true }); } + }); +}); + +describe('R1-6: two first starts of a new name never both win', () => { + it('a second start that runs while the first is between installing its identity and claiming the name gets the same agent', () => { + const name = uniq('r16'); + let ran = false; let checking = false; let second = null; + const named = () => { + try { + return fs.readdirSync(path.join(NODES_DIR, 'by-id')).some((d) => !d.startsWith('.') && + (() => { try { return JSON.parse(fs.readFileSync(path.join(NODES_DIR, 'by-id', d, 'identity.json'), 'utf8')).name === name; } catch { return false; } })()); + } catch { return false; } + }; + const hook = (orig) => function (...a) { + const r = orig.apply(fs, a); + if (!ran && !checking) { + checking = true; + try { if (named()) { ran = true; try { second = cfg.loadIdentity({ name }); } catch (e) { second = e; } } } finally { checking = false; } + } + return r; + }; + const first = patchMany(Object.fromEntries(MUTATING.map((k) => [k, hook])), () => cfg.loadIdentity({ name })); + assert.ok(ran, 'the second start ran inside the first one\'s mint'); + assert.ok(second && second.nodeId, `the second start loaded: ${second && second.message}`); + assert.strictEqual(second.nodeId, first.nodeId, 'one agent'); + assert.strictEqual(second.privateKey, first.privateKey); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId); + assert.deepStrictEqual(identitiesNamed(name), [first.nodeId], 'one identity on disk'); + }); + + it('three real processes starting one new name at once: one agent, every round', async () => { + let rounds = 0; + for (let round = 0; round < 4; round++) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-r16-')); + try { + const state = path.join(root, 'state'); + const code = `const c = require(${JSON.stringify(path.join(LIB, 'config.js'))}); + process.stdin.once('data', () => { try { const i = c.loadIdentity({ name: 'fresh' }); console.log(JSON.stringify({ id: i.nodeId })); } + catch (e) { console.log(JSON.stringify({ error: e.code || String(e.message).split('\\n')[0] })); } process.exit(0); }); console.log('ready');`; + const kids = Array.from({ length: 3 }, () => spawn(process.execPath, ['-e', code], { env: { ...process.env, SYM_STATE_DIR: state }, stdio: ['pipe', 'pipe', 'inherit'] })); + await Promise.all(kids.map((k) => new Promise((r) => k.stdout.once('data', r)))); + const outs = kids.map((k) => new Promise((r) => { let b = ''; k.stdout.on('data', (d) => { b += d; }); k.on('exit', () => r(JSON.parse(b.trim().split('\n').pop()))); })); + kids.forEach((k) => k.stdin.write('go\n')); + const res = await Promise.all(outs); + assert.deepStrictEqual(res.filter((o) => o.error), [], `round ${round}: every start loaded`); + assert.strictEqual(new Set(res.map((o) => o.id)).size, 1, `round ${round}: one agent, not ${new Set(res.map((o) => o.id)).size}`); + const installed = fs.readdirSync(path.join(state, 'nodes', 'by-id')).filter((d) => !d.startsWith('.')); + assert.deepStrictEqual(installed, [res[0].id], `round ${round}: one identity on disk`); + rounds++; + } finally { fs.rmSync(root, { recursive: true, force: true }); } + } + assert.strictEqual(rounds, 4); + }); +}); + +describe('R1-7: an index entry is "absent" only when there is none', () => { + function made() { + const name = uniq('r17'); + const orig = cfg.loadIdentity({ name }); + return { name, orig, idx: indexFile(name), link: path.join(NODES_DIR, name) }; + } + + it('deleted, and the compat link too: the original is found and the name claimed for it again (632b95d made a new identity)', () => { + const { name, orig, idx, link } = made(); + fs.unlinkSync(idx); fs.unlinkSync(link); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, orig.nodeId); + assert.strictEqual(cfg.nodeIdForName(name), orig.nodeId); + assert.deepStrictEqual(identitiesNamed(name), [orig.nodeId]); + }); + + for (const how of ['empty', 'not json', 'another name']) { + it(`${how}, and no compat link: the start halts, naming where the identity is; nothing is minted`, () => { + const { name, orig, idx, link } = made(); + fs.writeFileSync(idx, how === 'empty' ? '' : how === 'not json' ? '{"nodeId":' : JSON.stringify({ name: 'someone-else', nodeId: orig.nodeId })); + fs.unlinkSync(link); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && e.message.includes(orig.nodeId) && !/remove .*by-name/.test(e.message)); + assert.throws(() => cfg.loadIdentity({ name, create: false }), IdentityHaltError); + assert.deepStrictEqual(identitiesNamed(name), [orig.nodeId], 'no second identity'); + }); + } + + (posix && process.getuid && process.getuid() !== 0 ? it : it.skip)('unreadable (mode 000): halts, nothing minted', () => { + const { name, orig, idx, link } = made(); + fs.chmodSync(idx, 0); fs.unlinkSync(link); + try { + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.deepStrictEqual(identitiesNamed(name), [orig.nodeId]); + } finally { fs.chmodSync(idx, 0o644); } + }); + + it('a read error that is not ENOENT (EMFILE): halts, nothing minted', () => { + const { name, orig, idx, link } = made(); + fs.unlinkSync(link); + patchFs('readFileSync', (o) => function (p, ...a) { + if (String(p) === idx) throw Object.assign(new Error('EMFILE: too many open files'), { code: 'EMFILE' }); + return o.call(fs, p, ...a); + }, () => { assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); }); + assert.deepStrictEqual(identitiesNamed(name), [orig.nodeId]); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, orig.nodeId, 'once it reads, the same agent'); + }); + + it('index writes are synced: the entry is created exclusively and fsynced before it is relied on', () => { + const name = uniq('r17-sync'); + const synced = []; + const opened = new Map(); + patchMany({ + openSync: (o) => function (p, ...a) { const fd = o.call(fs, p, ...a); opened.set(fd, String(p)); return fd; }, + fsyncSync: (o) => function (fd) { synced.push(opened.get(fd)); return o.call(fs, fd); }, + }, () => cfg.loadIdentity({ name })); + const dir = path.dirname(indexFile(name)); + assert.ok(synced.some((p) => p && p.startsWith(dir + path.sep) && p.includes(encodeURIComponent(name))), 'the entry\'s content was fsynced'); + if (posix) assert.ok(synced.includes(dir), 'and the by-name directory'); + }); +}); + +describe('R1-8: an export interrupted before its bundle is durable is resumable or abortable, never a false "exported"', () => { + async function made(name) { + const id = cfg.loadIdentity({ name }); + put(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'm.json'), '{"memory":1}'); + return id; + } + const out = () => path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-bundle-')), 'node.bundle'); + + it('the start is refused saying the export did not finish (not "exported to another host"), and running the export again finishes it', async () => { + const name = uniq('r18'); + const id = await made(name); + const file = out(); + const toHostKey = relocation.hostKey().publicKey; + assert.throws(() => patchFs('renameSync', (o) => function (a, b) { + if (String(b) === file) throw Object.assign(new Error('EIO: crash before the bundle was durable'), { code: 'EIO' }); + return o.call(fs, a, b); + }, () => relocation.exportNode({ name, out: file, toHostKey })), /EIO/); + assert.ok(!exists(file), 'no bundle'); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e.code === 'EIDENTITYEXPORTING' && !/exported to another host/.test(e.message)); + assert.strictEqual(cfg.readTombstone(id.nodeId), null, 'not tombstoned: nothing was moved'); + const r = relocation.exportNode({ name, out: file, toHostKey }); + assert.strictEqual(r.nodeId, id.nodeId); + assert.ok(exists(file), 'the bundle'); + assert.ok(cfg.readTombstone(id.nodeId), 'tombstoned now'); + assert.strictEqual(JSON.parse(read(path.join(cfg.identityDirById(id.nodeId), 'identity.json'))).privateKey, null, 'the source lost the key'); + }); + + it('abortExport removes the bundle it wrote and the intent: the node starts here again', async () => { + const name = uniq('r18-abort'); + const id = await made(name); + const file = out(); + const toHostKey = relocation.hostKey().publicKey; + // interrupted after the bundle is durable and before the tombstone + assert.throws(() => patchFs('openSync', (o) => function (p, ...a) { + if (String(p).includes(cfg.TOMBSTONE)) throw Object.assign(new Error('EIO: crash before the tombstone'), { code: 'EIO' }); + return o.call(fs, p, ...a); + }, () => relocation.exportNode({ name, out: file, toHostKey })), /EIO/); + assert.ok(exists(file), 'the bundle is durable'); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e.code === 'EIDENTITYEXPORTING'); + const a = relocation.abortExport({ name }); + assert.strictEqual(a.aborted, true); + assert.strictEqual(a.bundleRemoved, true); + assert.ok(!exists(file), 'the bundle is gone'); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'the node starts again, with its key'); + }); +}); + +describe('R1-9: with SYM_IDENTITY_DIR the tombstone lives beside the key, and an import writes the key only there', () => { + it('an export killed after its bundle is durable: a rebuilt node directory does not bring a second live copy back; exporting again finishes the scrub', () => { + withIdentityDir(() => { + const name = uniq('r19'); + const id = cfg.loadIdentity({ name }); + const file = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-bundle-')), 'node.bundle'); + const toHostKey = relocation.hostKey().publicKey; + assert.throws(() => patchFs('openSync', (o) => function (p, ...a) { + if (/identity\.json\.\d+\.tmp$/.test(String(p))) throw Object.assign(new Error('EIO: crash before the scrub'), { code: 'EIO' }); + return o.call(fs, p, ...a); + }, () => relocation.exportNode({ name, out: file, toHostKey })), /EIO/); + assert.ok(exists(file), 'the bundle is durable'); + fs.rmSync(cfg.nodeDirById(id.nodeId), { recursive: true, force: true }); // the disposable node dir is rebuilt + assert.throws(() => cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }), (e) => e.code === 'EIDENTITYTOMBSTONED'); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e.code === 'EIDENTITYTOMBSTONED'); + const again = relocation.exportNode({ name, out: file + '2', toHostKey }); + assert.ok(again.resumed, 'the interrupted export was finished'); + assert.strictEqual(JSON.parse(read(path.join(cfg.identityDirById(id.nodeId), 'identity.json'))).privateKey, null, 'the source no longer holds the key'); + }); + }); + + modeTest('import: directories 0700, key files 0600, and the private key only in the identity directory', () => { + withIdentityDir(() => { + const name = uniq('r19-imp'); + const id = cfg.loadIdentity({ name }); + put(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'deep', 'm.json'), '{"memory":1}'); + put(path.join(cfg.nodeDirById(id.nodeId), 'e2e-keypair.json'), '{}', 0o600); + const file = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-bundle-')), 'node.bundle'); + relocation.exportNode({ name, out: file, toHostKey: relocation.hostKey().publicKey }); + // another host: this host's copies of the node removed + fs.rmSync(cfg.nodeDirById(id.nodeId), { recursive: true, force: true }); + fs.rmSync(cfg.identityDirById(id.nodeId), { recursive: true, force: true }); + relocation.importNode({ from: file, expect: { nodeId: id.nodeId, key: id.publicKey } }); + const nd = cfg.nodeDirById(id.nodeId); const idd = cfg.identityDirById(id.nodeId); + assert.strictEqual(modeOf(nd), 0o700); + assert.strictEqual(modeOf(path.join(nd, 'cmbs')), 0o700); + assert.strictEqual(modeOf(path.join(nd, 'cmbs', 'deep')), 0o700); + assert.strictEqual(modeOf(path.join(nd, 'e2e-keypair.json')), 0o600); + assert.strictEqual(modeOf(idd), 0o700); + assert.strictEqual(modeOf(path.join(idd, 'identity.json')), 0o600); + assert.ok(!exists(path.join(nd, 'identity.json')), 'no private key in the disposable node directory'); + assert.strictEqual(cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }).privateKey, id.privateKey); + }); + }); +}); + +describe('R1-10: a crash after the commit never leaves a state read as "new"', () => { + it('the compat link cannot be made: the 0.13 directory stays in place for a 0.13 rollback, and the next start finishes', () => { + // 632b95d had already moved everything; the failed link left nodes/ empty or absent, and a 0.13 build reading + // it would make a new identity. + const name = uniq('r110'); + const id = legacyNode(name); + patchFs('symlinkSync', (o) => function (target, p, ...a) { + if (String(p).startsWith(NODES_DIR + path.sep) && !String(p).includes(`${path.sep}by-id${path.sep}`)) throw Object.assign(new Error('EPERM: injected'), { code: 'EPERM' }); + return o.call(fs, target, p, ...a); + }, () => { try { cfg.migrateNodeDir(name); } catch { /* a failure is fine; what it leaves is the test */ } }); + assert.strictEqual(JSON.parse(read(path.join(NODES_DIR, name, 'identity.json'))).privateKey, id.privateKey, 'a 0.13 reader still finds the key'); + assert.ok(exists(path.join(NODES_DIR, name, 'cmbs', 'c0.json')), 'and its store'); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(isLink(path.join(NODES_DIR, name)), 'finished: the link'); + }); + + it('a new identity installed and the start killed before it claimed the name: the next start finds that identity, not a new one', () => { + const name = uniq('r110-mint'); + const byName = path.join(NODES_DIR, 'by-name'); + let thrown = false; + const stop = (o) => function (p, ...a) { + if (!thrown && String(p).startsWith(byName + path.sep) && (o !== fs.readFileSync)) { thrown = true; throw Object.assign(new Error('killed before the claim'), { code: 'EINTR' }); } + return o.call(fs, p, ...a); + }; + patchMany({ writeFileSync: stop, openSync: (o) => function (p, flags, ...a) { return /[wa]/.test(String(flags || 'r')) ? stop(o)(p, flags, ...a) : o.call(fs, p, flags, ...a); } }, + () => { try { cfg.loadIdentity({ name }); } catch { /* killed */ } }); + assert.ok(thrown, 'the start was stopped at the claim'); + const made = identitiesNamed(name); + assert.strictEqual(made.length, 1, 'its identity was installed'); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, made[0], 'the next start loads it'); + assert.deepStrictEqual(identitiesNamed(name), made, 'and makes no other'); + }); +}); + +describe('R1-11: names Windows cannot hold, and Windows junctions', () => { + it('validateName refuses the names Windows reserves or cannot hold (the rule of mesh-channel\'s nodeNameProblem)', () => { + for (const bad of ['CON', 'con', 'NUL', 'nul.txt', 'AUX', 'PRN.log', 'COM1', 'lpt9', 'a:b', 'a*b', 'a?b', 'a"b', 'ab', 'a|b', 'trailing.', 'trailing ']) { + assert.throws(() => cfg.validateName(bad), Error, bad); + } + for (const good of ['console', 'conman', 'nullable', 'com', 'com10x', 'a.b', 'agent-a@mesh', 'claude-code', '日本語']) { + assert.doesNotThrow(() => cfg.validateName(good), good); + } + assert.throws(() => cfg.loadIdentity({ name: 'NUL' })); + }); + + it('a compat link that reads back as an absolute path (a junction) is recognised, not re-made on every check', () => { + const name = uniq('r111'); + const id = cfg.loadIdentity({ name }); + const link = path.join(NODES_DIR, name); + const absolute = path.join(NODES_DIR, 'by-id', id.nodeId); + fs.unlinkSync(indexFile(name)); // the link is then what tells which node the name is + let remade = 0; + patchMany({ + readlinkSync: (o) => function (p, ...a) { return String(p) === link ? absolute : o.call(fs, p, ...a); }, + unlinkSync: (o) => function (p, ...a) { if (String(p) === link) remade++; return o.call(fs, p, ...a); }, + renameSync: (o) => function (a, b) { if (String(b) === link) remade++; return o.call(fs, a, b); }, + symlinkSync: (o) => function (t, p, ...a) { if (String(p) === link) remade++; return o.call(fs, t, p, ...a); }, + }, () => { + for (let i = 0; i < 3; i++) assert.strictEqual(cfg.loadIdentity({ name }).nodeId, id.nodeId); + }); + assert.strictEqual(remade, 0, 'the link was not re-made'); + }); +}); + +describe('R4-9: a start pinned to a nodeId never takes a name from another identity', () => { + // 632b95d re-pointed the name's index entry to the pin: a folder whose .sym/node.json pins X under "reviewer" took the + // name from another folder's agent Y; Y's next start by name then ran as X, and Y was left without its name. + it('the pinned identity is here and the name is another agent\'s: the start halts, the entry is left as it is', () => { + const name = uniq('r49'); + const y = cfg.loadIdentity({ name }); // the agent the name belongs to + const x = cfg.loadIdentity({ name: uniq('r49-x') }); // another agent on the host + for (const create of [false, true]) { + assert.throws(() => cfg.loadIdentity({ name, nodeId: x.nodeId, create }), + (e) => e instanceof IdentityHaltError && e.reason.includes(x.nodeId) && e.reason.includes(y.nodeId), `create: ${create}`); + assert.strictEqual(cfg.nodeIdForName(name), y.nodeId, 'the index still names Y'); + } + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, y.nodeId, 'the next start by name is Y'); + assert.strictEqual(fs.realpathSync(path.join(NODES_DIR, name)), fs.realpathSync(cfg.nodeDirById(y.nodeId)), 'and the compat link leads to Y'); + }); + + it('the pin is not on this host and the name is another agent\'s: never answered with that agent', () => { + const name = uniq('r49-absent'); + const y = cfg.loadIdentity({ name }); + const pin = cfg.uuidv7(); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin, create: true }), (e) => e instanceof IdentityHaltError && e.reason.includes(y.nodeId)); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin, create: false }), (e) => e.code === 'EIDENTITYABSENT' && e.message.includes(y.nodeId)); + assert.strictEqual(cfg.nodeIdForName(name), y.nodeId); + }); + + it('a name nobody has is claimed for the pinned identity', () => { + const x = cfg.loadIdentity({ name: uniq('r49-own') }); + const name = uniq('r49-free'); + assert.strictEqual(cfg.loadIdentity({ name, nodeId: x.nodeId, create: false }).nodeId, x.nodeId); + assert.strictEqual(cfg.nodeIdForName(name), x.nodeId); + }); +}); + +describe('error texts: the operator is never told to delete what leads to a key that is on disk', () => { + it('SYM_IDENTITY_DIR switched on after the agent was made: the halt says where the key is, and not to remove the index', () => { + const name = uniq('err-iddir'); + const id = cfg.loadIdentity({ name }); + withIdentityDir(() => { + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && + e.message.includes(path.join(cfg.nodeDirById(id.nodeId), 'identity.json')) && !e.message.includes(`remove ${indexFile(name)}`)); + assert.throws(() => cfg.loadIdentity({ name, create: false }), (e) => e instanceof IdentityHaltError && !/No identity/.test(e.message), 'not "no identity on this host"'); + assert.throws(() => cfg.loadIdentity({ nodeId: id.nodeId, create: false }), (e) => e instanceof IdentityHaltError); + }); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + }); +}); diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js new file mode 100644 index 00000000..72d772c3 --- /dev/null +++ b/tests/identity-migration-by-id.test.js @@ -0,0 +1,962 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME to a temp sandbox before lib/config loads + +/** + * The 0.13 -> by-id identity migration must never cost an agent its key. + * + * 0.14.0 - 0.14.2 had two ways to destroy one. (1) The duplicate guard asked "does nodes/by-id// + * EXIST?" instead of "does an IDENTITY already sit there?", so a by-id directory made by anything else (a + * lock, an interrupted move) left the 0.13 node where it was, unindexed, for good. (2) `loadIdentity` read + * every answer it did not recognise as "no such agent" and MINTED: it renamed the 0.13 directory (the old + * identity.json still in it) onto the new node's path and wrote the new identity over the old key. + * + * The tests below hold the line the founder drew ("the system must refuse to start rather than silently + * create a replacement identity"): a name that has an identity on this host never gets a new one. + * Every test uses its own agent name; the state root is shared by the file, as in config.test.js. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const crypto = require('crypto'); +const { spawn } = require('child_process'); +const cfg = require('../lib/config'); + +const { NODES_DIR, IdentityHaltError, IdentityTombstonedError } = cfg; +const posix = process.platform !== 'win32'; // modes are not a thing on Windows +const modeTest = posix ? it : it.skip; + +let seq = 0; +const uniq = (b) => `${b}-${process.pid}-${++seq}-${Math.random().toString(36).slice(2, 6)}`; +const modeOf = (p) => fs.statSync(p).mode & 0o777; +const read = (p) => fs.readFileSync(p, 'utf8'); +const exists = (p) => fs.existsSync(p); +const isLink = (p) => { try { return fs.lstatSync(p).isSymbolicLink(); } catch { return false; } }; + +/** A real Ed25519 identity, in the shape lib/config writes. */ +function makeIdentity(name, extra = {}) { + const kp = crypto.generateKeyPairSync('ed25519'); + return { + nodeId: cfg.uuidv7(), name, hostname: 'host.local', createdAt: 1, + publicKey: kp.publicKey.export({ format: 'der', type: 'spki' }).subarray(12).toString('base64url'), + privateKey: kp.privateKey.export({ format: 'der', type: 'pkcs8' }).subarray(16).toString('base64url'), + ...extra, + }; +} + +function put(file, text, mode = 0o644) { + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, text); + fs.chmodSync(file, mode); // explicit: the umask must not decide what a test means +} + +/** What 0.13 left on disk: nodes// with identity.json (and e2e-keypair.json) at the umask default. */ +function legacyNode(name, { e2e = false, identity = makeIdentity(name), where = path.join(NODES_DIR, name), withIdentity = true } = {}) { + if (withIdentity) put(path.join(where, 'identity.json'), JSON.stringify(identity, null, 2), 0o644); + if (e2e) put(path.join(where, 'e2e-keypair.json'), JSON.stringify({ publicKey: 'cHVi', privateKey: 'cHJpdg==' }, null, 2), 0o644); + put(path.join(where, 'cmbs', 'x.json'), '{"x":1}'); + return identity; +} + +/** A nodes/by-id// that something else made: a lock, an interrupted run. No identity in it. */ +function husk(nodeId, { lock = true } = {}) { + const d = cfg.nodeDirById(nodeId); + fs.mkdirSync(d, { recursive: true }); + if (lock) put(path.join(d, 'lock.pid'), '999999\n'); + return d; +} + +/** Every identity.json under nodes/by-id and SYM_IDENTITY_DIR/by-id whose `name` field is `name`. */ +function identitiesNamed(name) { + const found = []; + const roots = [path.join(NODES_DIR, 'by-id'), process.env.SYM_IDENTITY_DIR && path.join(process.env.SYM_IDENTITY_DIR, 'by-id')].filter(Boolean); + for (const root of roots) { + if (!exists(root)) continue; + for (const d of fs.readdirSync(root)) { + const f = path.join(root, d, 'identity.json'); + try { if (JSON.parse(read(f)).name === name) found.push(f); } catch { /* */ } + } + } + return found; +} + +function withIdentityDir(fn) { + const prev = process.env.SYM_IDENTITY_DIR; + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-idroot-')); + process.env.SYM_IDENTITY_DIR = root; + try { return fn(root); } finally { + if (prev === undefined) delete process.env.SYM_IDENTITY_DIR; else process.env.SYM_IDENTITY_DIR = prev; + fs.rmSync(root, { recursive: true, force: true }); + } +} + +/** Every path under `dir` with its kind and bytes (links followed), sorted: two equal trees hold the same data. */ +function treeOf(dir) { + const out = []; + const walk = (d, rel) => { + for (const e of fs.readdirSync(d).sort()) { + const p = path.join(d, e); const r = rel ? `${rel}/${e}` : e; + const st = fs.statSync(p); + if (st.isDirectory()) { out.push(`D ${r}`); walk(p, r); } else out.push(`F ${r} ${crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex')}`); + } + }; + walk(dir, ''); + return out; +} + +/** Replace fs. for the duration of `body`; `wrap(original)` returns the replacement. */ +function patchFs(fn, wrap, body) { + const original = fs[fn]; + fs[fn] = wrap(original); + try { return body(); } finally { fs[fn] = original; } +} +const exdev = (msg) => Object.assign(new Error(`EXDEV: cross-device link not permitted (${msg})`), { code: 'EXDEV' }); + +/** + * T9: every case asserts it measured something. A run over nothing returns moved: 0 and would "pass" a + * test that only checked for the absence of damage. + */ +function assertMoved(result, atLeast = 1) { + assert.ok(result && result.moved >= atLeast, `the migration moved ${result && result.moved}, expected at least ${atLeast}: the test measured nothing`); +} + +describe('T1/T2: the migration leaves key files private', () => { + modeTest('identity.json written 0644 by 0.13 ends at 0600, and its directory at 0700', () => { + const name = uniq('t1'); + const id = legacyNode(name); + assert.strictEqual(modeOf(path.join(NODES_DIR, name, 'identity.json')), 0o644, 'setup: a 0.13 key file, world-readable'); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const dir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + assert.strictEqual(JSON.parse(read(path.join(dir, 'identity.json'))).privateKey, id.privateKey, 'the key is the same key'); + }); + + modeTest('e2e-keypair.json ends at 0600 with its bytes unchanged', () => { + const name = uniq('t2'); + const id = legacyNode(name, { e2e: true }); + const before = read(path.join(NODES_DIR, name, 'e2e-keypair.json')); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const f = path.join(cfg.nodeDirById(id.nodeId), 'e2e-keypair.json'); + assert.strictEqual(modeOf(f), 0o600); + assert.strictEqual(read(f), before); + }); + + modeTest('with SYM_IDENTITY_DIR: the identity file is 0600 and both directories 0700 (T8)', () => { + withIdentityDir((idRoot) => { + const name = uniq('t8'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + put(path.join(NODES_DIR, name, 'e2e-keypair.json'), '{}', 0o644); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const idDir = path.join(idRoot, 'by-id', id.nodeId); + const nodeDir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(idDir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(idDir), 0o700); + assert.strictEqual(modeOf(nodeDir), 0o700); + assert.strictEqual(modeOf(path.join(nodeDir, 'e2e-keypair.json')), 0o600); + assert.ok(exists(path.join(nodeDir, 'cmbs', 'x.json')), 'the node dir moved with its store'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId, 'the index resolves'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + }); + + modeTest('loading narrows a key file an earlier release already migrated at 0644 (and the E2E keypair beside it)', () => { + const name = uniq('heal'); + const id = legacyNode(name, { e2e: true }); + cfg.migrateNodeDir(name); + const dir = cfg.nodeDirById(id.nodeId); + fs.chmodSync(path.join(dir, 'identity.json'), 0o644); fs.chmodSync(path.join(dir, 'e2e-keypair.json'), 0o644); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(path.join(dir, 'e2e-keypair.json')), 0o600); + }); + + modeTest('a rewrite of identity.json (bare Windows hostname) lands at 0600, not at the old file mode', () => { + const name = uniq('rewrite'); + const id = legacyNode(name, { identity: makeIdentity(name, { hostname: 'bare-host' }) }); + cfg.migrateNodeDir(name); + const f = path.join(cfg.nodeDirById(id.nodeId), 'identity.json'); + const onDisk = JSON.parse(read(f)); onDisk.hostname = 'bare-host'; + put(f, JSON.stringify(onDisk, null, 2), 0o644); + const loaded = cfg.loadIdentity({ name, create: false }); + assert.strictEqual(loaded.hostname, 'bare-host.local'); + assert.strictEqual(JSON.parse(read(f)).hostname, 'bare-host.local', 'the rewrite happened'); + assert.strictEqual(modeOf(f), 0o600); + }); + + modeTest('a newly made agent: identity 0600, its directory 0700', () => { + const name = uniq('fresh'); + const id = cfg.loadIdentity({ name }); + const dir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + }); +}); + +describe('T3-T5: a by-id directory that merely exists is not a second claim on the nodeId', () => { + it('T3: the node is merged into an existing by-id dir that holds only lock.pid', () => { + const name = uniq('t3'); + const id = legacyNode(name); + const dest = husk(id.nodeId); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey, 'the identity is under the nodeId, key unchanged'); + // 0.14.3 changed this: the husk is REPLACED through the staged commit (one rename), and its stale lock is removed + // (compare-and-delete, what the next acquire would do anyway) rather than kept; a live one would refuse (EIDENTITYLOCK). + assert.ok(!exists(path.join(dest, 'lock.pid')), 'the stale lock is not carried into the installed directory'); + assert.ok(exists(path.join(dest, cfg.COMMIT_MARK)), 'the directory carries the commit record'); + assert.ok(exists(path.join(dest, 'cmbs', 'x.json')), 'the store came with it'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + assert.ok(isLink(path.join(NODES_DIR, name)), 'nodes/ is the compat link'); + assert.strictEqual(fs.realpathSync(path.join(NODES_DIR, name)), fs.realpathSync(dest)); + }); + + it('T3: an EMPTY by-id dir is merged into too', () => { + const name = uniq('t3e'); + const id = legacyNode(name); + husk(id.nodeId, { lock: false }); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'identity.json'))); + }); + + it('T4: a pinned start recovers once the node is migrated', () => { + const name = uniq('t4'); + const id = legacyNode(name); + husk(id.nodeId); + assertMoved(cfg.migrateIdentities()); + const loaded = cfg.loadIdentity({ nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + }); + + it('T5: an unpinned start never mints over a husk: same nodeId, same key, and no second identity exists', () => { + const name = uniq('t5'); + const id = legacyNode(name); + husk(id.nodeId); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + const files = identitiesNamed(name); + assert.strictEqual(files.length, 1, `exactly one identity answers to ${name}: ${files.join(', ')}`); + assert.strictEqual(JSON.parse(read(files[0])).privateKey, id.privateKey, 'and it still holds the original private key'); + }); + + it('the reported repro: a 0.13 dir plus a pre-existing EMPTY by-id// keeps the old key (0.14.2 minted and overwrote it)', () => { + const name = uniq('repro'); + const old = legacyNode(name); + fs.mkdirSync(path.join(NODES_DIR, 'by-id', old.nodeId), { recursive: true }); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, old.nodeId, 'not a new nodeId'); + assert.strictEqual(loaded.privateKey, old.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + assert.strictEqual(JSON.parse(read(path.join(cfg.nodeDirById(old.nodeId), 'identity.json'))).privateKey, old.privateKey, 'the old private key is still on disk'); + assert.strictEqual(cfg.nodeIdForName(name), old.nodeId); + }); + + it('the same identity on both sides is an interrupted move, resumed, not a duplicate', () => { + const name = uniq('resume-same'); + const id = legacyNode(name); + const dest = cfg.nodeDirById(id.nodeId); + put(path.join(dest, 'identity.json'), read(path.join(NODES_DIR, name, 'identity.json')), 0o600); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(exists(path.join(dest, 'cmbs', 'x.json'))); + assert.ok(isLink(path.join(NODES_DIR, name))); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); +}); + +describe('T6/T7: a real duplicate is refused, and refusing is a halt, never a mint', () => { + function duplicate(name) { + const mine = legacyNode(name); + const other = makeIdentity(name, { nodeId: mine.nodeId }); // another key, filed under the same nodeId + const dest = cfg.nodeDirById(mine.nodeId); + put(path.join(dest, 'identity.json'), JSON.stringify(other, null, 2), 0o600); + return { mine, other, dest }; + } + + it('T6: a DIFFERENT identity.json already under the nodeId: skipped-duplicate, and nothing is overwritten', () => { + const name = uniq('t6'); + const { mine, dest } = duplicate(name); + const legacyBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const destBefore = read(path.join(dest, 'identity.json')); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), legacyBefore, 'the 0.13 identity is byte-unchanged'); + assert.strictEqual(read(path.join(dest, 'identity.json')), destBefore, 'the one already filed is byte-unchanged'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'nothing was indexed'); + assert.ok(!exists(path.join(dest, 'cmbs')), 'and nothing was merged in beside it'); + assert.ok(mine.privateKey); + }); + + it('T7: loading that name halts (IdentityHaltError) and does not touch the 0.13 file', () => { + const name = uniq('t7'); + const { dest } = duplicate(name); + const legacyBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const destBefore = read(path.join(dest, 'identity.json')); + const dirsBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /0\.13 identity/.test(e.reason)); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), legacyBefore); + assert.strictEqual(read(path.join(dest, 'identity.json')), destBefore); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, dirsBefore, 'no new node directory was made'); + assert.strictEqual(cfg.nodeIdForName(name), null); + }); + + it('a differing store file in an existing by-id dir is a conflict too: nothing moves', () => { + const name = uniq('conflict'); + const id = legacyNode(name); + const dest = husk(id.nodeId, { lock: false }); + put(path.join(dest, 'cmbs', 'x.json'), '{"x":2}'); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the 0.13 node is where it was'); + assert.strictEqual(read(path.join(dest, 'cmbs', 'x.json')), '{"x":2}'); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + }); + + it('a name the index already gives to ANOTHER node is not re-pointed by a stray 0.13 directory', () => { + const name = uniq('held'); + const held = cfg.loadIdentity({ name }); + const stray = makeIdentity(name); + fs.unlinkSync(path.join(NODES_DIR, name)); // a real directory where the index says there is a link + legacyNode(name, { identity: stray }); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.strictEqual(cfg.nodeIdForName(name), held.nodeId, 'the index is untouched'); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, held.nodeId, 'and the indexed agent still loads'); + assert.strictEqual(JSON.parse(read(path.join(NODES_DIR, name, 'identity.json'))).privateKey, stray.privateKey, 'the stray key is not destroyed'); + }); +}); + +describe('F4: a name that indexes a node whose identity is missing halts', () => { + it('create (the default) halts and neither mints nor re-points the index', () => { + const name = uniq('f4'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + const dirsBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && e.reason.includes(first.nodeId)); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId, 'the index still names the agent'); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, dirsBefore, 'nothing was minted'); + }); + + it('the halt names what to remove, and removing exactly that lets the name start again as a different agent', () => { + const name = uniq('f4reset'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + let halt = null; + try { cfg.loadIdentity({ name }); } catch (e) { halt = e; } + assert.ok(halt instanceof IdentityHaltError); + const index = path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); + const link = path.join(NODES_DIR, name); + // 0.14.3: what to do is the halt's advice (in the message), and it is given only when no copy of the key is on this host. + assert.ok(halt.message.includes(index) && halt.message.includes(link), 'the message names the index file and the link'); + fs.unlinkSync(index); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError, 'the index alone is not enough: the link is a trace of the name'); + // 0.14.3: a halt writes nothing (632b95d re-created the index from the link on its way to the halt). + assert.ok(!exists(index), 'the halt did not write the index back'); + fs.unlinkSync(link); + const again = cfg.loadIdentity({ name }); + assert.notStrictEqual(again.nodeId, first.nodeId, 'a different agent, as the message says'); + }); + + it('create: false is still IdentityAbsentError (EIDENTITYABSENT)', () => { + const name = uniq('f4b'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + assert.throws(() => cfg.loadIdentity({ name, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.throws(() => cfg.loadIdentity({ nodeId: first.nodeId, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + }); + + it('an identity volume that is not mounted at boot halts instead of making a replacement agent', () => { + withIdentityDir((idRoot) => { + const name = uniq('f4v'); + const first = cfg.loadIdentity({ name }); + assert.ok(exists(path.join(idRoot, 'by-id', first.nodeId, 'identity.json')), 'setup: the key is on the identity volume'); + const away = `${idRoot}.away`; + fs.renameSync(idRoot, away); + try { + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId); + } finally { fs.rmSync(idRoot, { recursive: true, force: true }); fs.renameSync(away, idRoot); } + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, first.privateKey, 'mounted again: the same agent'); + }); + }); +}); + +describe('the safety net: the mint path never buries a 0.13 identity', () => { + it('F5: SYM_IDENTITY_DIR switched on for a 0.13 node whose identity is in its node dir halts, and moves nothing', () => { + withIdentityDir(() => { + const name = uniq('f5'); + const id = legacyNode(name); + const before = read(path.join(NODES_DIR, name, 'identity.json')); + assert.throws(() => cfg.migrateNodeDir(name), (e) => e instanceof IdentityHaltError && /SYM_IDENTITY_DIR/.test(e.reason)); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), before); + assert.ok(!isLink(path.join(NODES_DIR, name)) && exists(path.join(NODES_DIR, name, 'cmbs')), 'the node dir was not moved'); + assert.strictEqual(identitiesNamed(name).length, 0, 'and no identity was made'); + assert.ok(id.privateKey); + }); + }); + + it('F7 (changed in 0.14.3, R1): a symlink at nodes/ that is not a compat link is a 0.13 directory kept elsewhere: it is followed and copied, its target untouched', () => { + // 632b95d halted here, and minted when the link was a 0.13 node's directory on a durable volume. A link that is + // not `by-id/` is the agent's own directory: copy what it leads to, set the link aside, never touch the target. + const name = uniq('f7'); + const elsewhere = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-stray-')), 'agent'); + const id = legacyNode(name, { where: elsewhere }); + const before = treeOf(elsewhere); + fs.mkdirSync(NODES_DIR, { recursive: true }); + fs.symlinkSync(elsewhere, path.join(NODES_DIR, name), 'dir'); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + const dest = cfg.nodeDirById(id.nodeId); + assert.ok(!isLink(path.join(dest, 'cmbs')) && exists(path.join(dest, 'cmbs', 'x.json')), 'the store was copied, not linked'); + assert.deepStrictEqual(treeOf(elsewhere), before, 'what the link led to is byte-unchanged'); + assert.strictEqual(fs.readlinkSync(path.join(NODES_DIR, cfg.LEGACY_ASIDE, name)), elsewhere, 'the link itself was set aside'); + assert.strictEqual(cfg.migrateNodeDir(name), 'already'); + }); + + it('a nodes// holding files and NO identity halts (changed in 0.14.3, R1-2): it is a trace of an agent, never a new one\'s', () => { + // 632b95d gave such a directory to a NEW identity. With SYM_IDENTITY_DIR's volume not mounted that is exactly an + // existing agent's store whose key is elsewhere: the new identity took its store and the original was orphaned. + const name = uniq('nodir-id'); + put(path.join(NODES_DIR, name, 'cmbs', 'kept.json'), '{"k":1}'); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /holds a node's files/.test(e.reason)); + assert.strictEqual(identitiesNamed(name).length, 0, 'nothing was minted'); + assert.throws(() => cfg.loadIdentity({ name, create: false }), (e) => e.code === 'EIDENTITYABSENT' && /holds a node's files/.test(e.message), 'create: false says what is here'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'cmbs', 'kept.json')), '{"k":1}'); + }); + + it('an EMPTY nodes// (or one with only a stale lock) is no trace: the new agent is made and the link takes its place', () => { + const name = uniq('empty-dir'); + fs.mkdirSync(path.join(NODES_DIR, name), { recursive: true }); + put(path.join(NODES_DIR, name, 'lock.pid'), '2147483646\n'); + const id = cfg.loadIdentity({ name }); + assert.ok(isLink(path.join(NODES_DIR, name))); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); + }); + + it('a brand-new name still mints exactly one identity', () => { + const name = uniq('brand-new'); + const a = cfg.loadIdentity({ name }); + const b = cfg.loadIdentity({ name }); + assert.strictEqual(a.nodeId, b.nodeId); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F10: a public key without a private key halts', () => { + it('is not loaded, not repaired, and the file is untouched', () => { + const name = uniq('f10'); + const id = makeIdentity(name); delete id.privateKey; + legacyNode(name, { identity: id }); + const f = path.join(NODES_DIR, name, 'identity.json'); + const before = read(f); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /no private key/.test(e.reason)); + assert.strictEqual(read(f), before); + }); +}); + +describe('F6: rewriting identity.json is atomic', () => { + it('a failure while writing leaves the original key file intact, and no temp file behind', () => { + // 0.14.3: the migration never rewrites a 0.13 file (it copies, normalised); the rewrite is of an installed identity. + const name = uniq('f6'); + const id = cfg.loadIdentity({ name }); + const f = path.join(cfg.nodeDirById(id.nodeId), 'identity.json'); + const onDisk = JSON.parse(read(f)); onDisk.hostname = 'bare-host'; + put(f, JSON.stringify(onDisk, null, 2), 0o600); + const before = read(f); + let injected = 0; + const tempFds = new Set(); // the fds of identity.json's temp file: the lock file's writes are left alone + patchFs('openSync', (open) => function (p, ...rest) { + const fd = open.call(fs, p, ...rest); + if (/identity\.json\.\d+\.tmp$/.test(String(p))) tempFds.add(fd); + return fd; + }, () => patchFs('writeSync', (write) => function (fd, ...rest) { + if (tempFds.has(fd)) { injected++; throw Object.assign(new Error('ENOSPC: no space left on device (injected)'), { code: 'ENOSPC' }); } + return write.call(fs, fd, ...rest); + }, () => { + assert.throws(() => cfg.loadIdentity({ name }), /ENOSPC/); + })); + assert.ok(injected >= 1, 'the failure was reached, in the write of the key file'); + assert.strictEqual(read(f), before, 'the key file is byte-identical: it was never truncated'); + assert.deepStrictEqual(fs.readdirSync(path.dirname(f)).filter((n) => n.endsWith('.tmp')), []); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'and the next start succeeds with the same key'); + }); +}); + +/** Which tree a path is in: a rename between two of them would cross a volume when each is mounted on its own. */ +function volumeOf(p, idRoot) { + const roots = [path.join(idRoot, 'by-id'), idRoot, path.join(NODES_DIR, 'by-id'), NODES_DIR]; + return roots.find((r) => String(p) === r || String(p).startsWith(r + path.sep)) || 'other'; +} + +describe('F9: a copy across filesystems: copies, verifies, commits; the source is kept', () => { + it('EXDEV between the trees: the identity arrives intact at 0600, and the source is set aside, not deleted', () => { + // 0.14.3: no rename ever crosses from a 0.13 directory into by-id (the staging is made beside its destination), + // so every rename across the trees below fails with EXDEV here, and the migration still completes. + withIdentityDir((idRoot) => { + const name = uniq('f9'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(idRoot, name, 'sub', 'a.txt'), 'a'); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + let crossed = 0; + const result = patchFs('renameSync', (orig) => function (a, b) { + if (volumeOf(a, idRoot) !== volumeOf(b, idRoot)) { crossed++; throw exdev('simulated'); } + return orig.call(fs, a, b); + }, () => cfg.migrateNodeDir(name)); + assert.strictEqual(crossed, 0, 'no rename tried to cross a volume'); + assert.strictEqual(result, 'moved'); + const dest = path.join(idRoot, 'by-id', id.nodeId); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + assert.strictEqual(read(path.join(dest, 'sub', 'a.txt')), 'a'); + if (posix) assert.strictEqual(modeOf(path.join(dest, 'identity.json')), 0o600); + assert.ok(isLink(path.join(idRoot, name)), 'SYM_IDENTITY_DIR/ is the compat link'); + assert.strictEqual(JSON.parse(read(path.join(idRoot, cfg.LEGACY_ASIDE, name, 'identity.json'))).privateKey, id.privateKey, 'the 0.13 source is kept, set aside'); + if (posix) assert.strictEqual(modeOf(path.join(idRoot, cfg.LEGACY_ASIDE, name, 'identity.json')), 0o600, 'and narrowed to its owner'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + }); + + it('a copy that fails leaves the source untouched and no half-copy at the destination', () => { + withIdentityDir((idRoot) => { + const name = uniq('f9b'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o600); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const src = read(path.join(idRoot, name, 'identity.json')); + assert.throws(() => patchFs('copyFileSync', () => function () { throw Object.assign(new Error('EIO: injected'), { code: 'EIO' }); }, + () => cfg.migrateNodeDir(name)), /EIO/); + assert.strictEqual(read(path.join(idRoot, name, 'identity.json')), src, 'the identity is where it was'); + assert.ok(!exists(path.join(idRoot, 'by-id', id.nodeId)), 'no half-copy'); + assert.ok(!exists(cfg.nodeDirById(id.nodeId)), 'and no node directory installed either'); + assert.deepStrictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).filter((n) => n.includes(id.nodeId)), [], 'no staging left behind'); + // the fault clears: the next run completes the move + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + assert.ok(exists(path.join(idRoot, 'by-id', id.nodeId, 'identity.json'))); + }); + }); +}); + +describe('F9 on a real second filesystem (Linux: nodes/by-id lives on /dev/shm, a tmpfs)', () => { + const shm = '/dev/shm'; + const second = process.platform === 'linux' && exists(shm) && fs.statSync(shm).dev !== fs.statSync(os.tmpdir()).dev; + (second ? it : it.skip)('by-id on another filesystem for real: the node arrives whole, private and unchanged, and no rename crosses the volumes', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-exdev-')); + const far = fs.mkdtempSync(path.join(shm, 'sym-exdev-')); + try { + const state = path.join(root, 'state'); + fs.mkdirSync(path.join(state, 'nodes'), { recursive: true }); + fs.symlinkSync(far, path.join(state, 'nodes', 'by-id'), 'dir'); // everything filed by nodeId lands on the other filesystem + const script = ` + const fs = require('fs'), path = require('path'); + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + const name = 'exdev-agent'; + const id = JSON.parse(process.env.LEGACY_IDENTITY); + const dir = path.join(c.NODES_DIR, name); + fs.mkdirSync(path.join(dir, 'cmbs'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'identity.json'), JSON.stringify(id, null, 2), { mode: 0o644 }); fs.chmodSync(path.join(dir, 'identity.json'), 0o644); + fs.writeFileSync(path.join(dir, 'e2e-keypair.json'), '{}'); fs.chmodSync(path.join(dir, 'e2e-keypair.json'), 0o644); + fs.writeFileSync(path.join(dir, 'cmbs', 'x.json'), '{"x":1}'); + let sawExdev = false; // 0.14.3 never renames across the volumes: every commit is a rename beside its destination + const orig = fs.renameSync; + fs.renameSync = function (a, b) { try { return orig.call(fs, a, b); } catch (e) { if (e.code === 'EXDEV') sawExdev = true; throw e; } }; + const loaded = c.loadIdentity({ name }); + const dest = c.nodeDirById(id.nodeId); + console.log(JSON.stringify({ + sawExdev, same: loaded.nodeId === id.nodeId && loaded.privateKey === id.privateKey, + sourceKept: fs.lstatSync(dir).isSymbolicLink() && fs.readFileSync(path.join(c.NODES_DIR, c.LEGACY_ASIDE, name, 'cmbs', 'x.json'), 'utf8') === '{"x":1}', + store: fs.readFileSync(path.join(dest, 'cmbs', 'x.json'), 'utf8'), + idMode: fs.statSync(path.join(dest, 'identity.json')).mode & 0o777, e2eMode: fs.statSync(path.join(dest, 'e2e-keypair.json')).mode & 0o777, + dirMode: fs.statSync(dest).mode & 0o777, onOtherFs: fs.realpathSync(dest).startsWith(fs.realpathSync(${JSON.stringify(far)})), + }));`; + const id = makeIdentity('exdev-agent'); + const out = await new Promise((resolve, reject) => { + const child = spawn(process.execPath, ['-e', script], { + env: { ...process.env, SYM_STATE_DIR: state, LEGACY_IDENTITY: JSON.stringify(id) }, stdio: ['ignore', 'pipe', 'pipe'], + }); + let so = ''; let se = ''; + child.stdout.on('data', (d) => { so += d; }); child.stderr.on('data', (d) => { se += d; }); + child.on('exit', (code) => (code === 0 ? resolve(so) : reject(new Error(`child exited ${code}: ${se}`)))); + }); + const r = JSON.parse(out); + assert.strictEqual(r.onOtherFs, true, 'setup: the node lives on the other filesystem'); + assert.strictEqual(r.sawExdev, false, 'no rename crossed the volumes: each copy is staged beside its destination'); + assert.strictEqual(r.same, true); + assert.strictEqual(r.sourceKept, true, 'the 0.13 directory is set aside, whole'); + assert.strictEqual(r.store, '{"x":1}'); + assert.strictEqual(r.idMode, 0o600); + assert.strictEqual(r.e2eMode, 0o600); + assert.strictEqual(r.dirMode, 0o700); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + fs.rmSync(far, { recursive: true, force: true }); + } + }); +}); + +describe('F11: the window between the move and the index is not a mint race', () => { + it('a second starter that arrives mid-move finds the agent instead of making a new one', () => { + const name = uniq('f11'); + const id = legacyNode(name); + let racer = null; + patchFs('renameSync', (orig) => function (a, b) { + const r = orig.call(fs, a, b); + if (racer === null && a === path.join(NODES_DIR, name)) { // the node dir has just moved; the index is not final yet + racer = 'running'; + try { racer = cfg.loadIdentity({ name }); } catch (e) { racer = e; } + } + return r; + }, () => { assert.strictEqual(cfg.loadIdentity({ name }).nodeId, id.nodeId); }); + assert.ok(racer && racer.nodeId, `the racing starter loaded an identity: ${racer && racer.message}`); + assert.strictEqual(racer.nodeId, id.nodeId, 'the same agent'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); + + it('an interrupted migration leaves the 0.13 directory as it was, and the next start completes it (changed in 0.14.3: a copy, committed by one rename)', () => { + // 632b95d merged entry by entry, so an interruption left the store split between the two places. Now nothing is + // moved: the copy is staged, and a failure before the commit installs nothing and changes nothing in the source. + const name = uniq('f11r'); + const id = legacyNode(name); + put(path.join(NODES_DIR, name, 'aaa.json'), '{}'); put(path.join(NODES_DIR, name, 'zzz.json'), '{}'); + husk(id.nodeId); + const before = treeOf(path.join(NODES_DIR, name)); + patchFs('renameSync', (orig) => function (a, b) { + if (String(b) === cfg.nodeDirById(id.nodeId)) throw Object.assign(new Error('EIO: injected at the commit'), { code: 'EIO' }); + return orig.call(fs, a, b); + }, () => { assert.throws(() => cfg.migrateNodeDir(name), /EIO/); }); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, name)), before, 'the 0.13 directory is byte-unchanged'); + assert.ok(!exists(path.join(cfg.nodeDirById(id.nodeId), 'identity.json')), 'nothing was installed'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'and the name was not claimed'); + assert.deepStrictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).filter((n) => n.startsWith(`.${id.nodeId}.partial-`)), [], 'the staging was removed'); + // a staging left by a process that was killed is cleared by the next attempt + const dead = path.join(NODES_DIR, 'by-id', `.${id.nodeId}.partial-2147483646-deadbeef`); + put(path.join(dead, 'identity.json'), '{"junk":1}'); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(!exists(dead), 'the dead process\'s staging is gone'); + for (const f of ['aaa.json', 'zzz.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), f)), `the whole store: ${f}`); + assert.ok(isLink(path.join(NODES_DIR, name)), 'finished: nodes/ is the compat link'); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, cfg.LEGACY_ASIDE, name)).filter((l) => !l.includes(' lock.pid')), before.filter((l) => !l.includes(' lock.pid')), 'and the 0.13 directory is kept, set aside, as it was'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F11 (0.14.3): no migration lock; concurrent starts are made safe by the staged commit', () => { + /** A process that holds `lockPath` (written as `meta` says) until killed. */ + function holder(dir, meta = {}) { + return spawn(process.execPath, ['-e', ` + const fs = require('fs'), path = require('path'); + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + fs.mkdirSync(${JSON.stringify(dir)}, { recursive: true }); + fs.writeFileSync(path.join(${JSON.stringify(dir)}, 'lock.pid'), process.pid + '\\n' + JSON.stringify({ start: c.processStartTime(process.pid), createdAt: Date.now(), ...${JSON.stringify(meta)} }) + '\\n'); + console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + } + const held = (child) => new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('the holder exited'))); }); + + it('a start beside another start\'s migration of the same agent does not wait for it: it stages its own copy, and one commit wins', async () => { + // 632b95d serialised every migration in the process tree behind nodes/.migrating, blocking the event loop of every + // node in a multi-node process for up to 30 s. The 0.13 directory's own lock, held by a 0.14.3 migration, is passed. + const name = uniq('f11-beside'); + const id = legacyNode(name); + const child = holder(path.join(NODES_DIR, name), { migrating: true, token: 'other' }); + try { + await held(child); + const t0 = Date.now(); + const loaded = cfg.loadIdentity({ name }); + assert.ok(Date.now() - t0 < 5000, `no wait (${Date.now() - t0} ms)`); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'x.json')), 'migrated, with its store'); + assert.ok(!exists(path.join(NODES_DIR, '.migrating')), 'no migration lock exists'); + } finally { child.kill(); } + }); + + it('a start is not held up by a 632b95d-era migration lock (nodes/.migrating) that another process holds', async () => { + const name = uniq('f11-oldlock'); + const id = legacyNode(name); + const child = holder(path.join(NODES_DIR, '.migrating')); + try { + await held(child); + if (cfg._setMigrationLockWait) cfg._setMigrationLockWait(3000); // (632b95d only: so its wait fails in 3 s, not 30) + const t0 = Date.now(); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + assert.ok(Date.now() - t0 < 2500, `no wait (${Date.now() - t0} ms)`); + } finally { if (cfg._setMigrationLockWait) cfg._setMigrationLockWait(30000); child.kill(); } + }); + + it('a stale 0.13 lock (its process is gone) is reclaimed, not waited for', () => { + const name = uniq('f11-stale'); + const id = legacyNode(name); + put(path.join(NODES_DIR, name, 'lock.pid'), '2147483646\n', 0o644); // a pid that is not running + const t0 = Date.now(); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(Date.now() - t0 < 2000, 'no wait'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + + it('nothing to move: no lock is taken and nothing is written', () => { + const name = uniq('f11-nolock'); + const id = cfg.loadIdentity({ name }); // a new agent: indexed, nodes/ is the link + const lockDir = path.join(NODES_DIR, '.migrating'); + fs.rmSync(lockDir, { recursive: true, force: true }); + assert.strictEqual(cfg.migrateNodeDir(name), 'already'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); + assert.ok(!exists(lockDir), 'the migration lock directory was not even made'); + }); + + it('a path that leads through a link to the destination is never "merged" onto itself (that would delete the only copy)', () => { + const name = uniq('f11-alias'); + const id = legacyNode(name); + put(path.join(NODES_DIR, name, 'aaa.json'), '{"a":1}'); + const legacy = path.join(NODES_DIR, name); + const dest = husk(id.nodeId); // forces an entry-by-entry merge + let helped = false; let listings = 0; + // Something not holding the lock finishes the move while this merge holds a list of what to move (it cannot + // happen among cooperating processes; the check is a last guard on a key file). + patchFs('readdirSync', (orig) => function (p, ...rest) { + const names = orig.call(fs, p, ...rest); + if (!helped && p === legacy && ++listings === 2) { // the 2nd listing is the merge's own, after the conflict check's + helped = true; + for (const n of names) fs.renameSync(path.join(legacy, n), path.join(dest, n)); + fs.rmdirSync(legacy); + fs.symlinkSync(path.join('by-id', id.nodeId), legacy, 'dir'); + } + return names; + }, () => { try { cfg.migrateNodeDir(name); } catch { /* the directory became a link under it: an error is fine, a deletion is not */ } }); + assert.ok(helped, 'the other actor ran'); + for (const f of ['identity.json', 'aaa.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(dest, f)), `${f} survived`); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + }); +}); + +describe('F11: a starter that read the index too early', () => { + it('finds the agent by the index after migrating nothing, instead of making a new one', () => { + // The other process has moved everything and written the index, but not yet made the compat link: there is no + // 0.13 directory, so there is nothing to migrate, and this starter read the index (as missing) just before. + const name = uniq('f11-early'); + const id = legacyNode(name); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + fs.unlinkSync(path.join(NODES_DIR, name)); // the link not yet made + const index = path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); + let fakedMissing = false; + const loaded = patchFs('readFileSync', (orig) => function (p, ...rest) { + if (!fakedMissing && String(p) === index) { fakedMissing = true; throw Object.assign(new Error('ENOENT (before the other process wrote it)'), { code: 'ENOENT' }); } + return orig.call(fs, p, ...rest); + }, () => cfg.loadIdentity({ name })); + assert.ok(fakedMissing, 'the early read happened'); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F11 with real processes: several starters at once never make a second identity', () => { + /** Four processes call loadIdentity({ name }) at the same moment on a 0.13 node, in a state root of their own. */ + async function race({ withHusk, pinned = false, stall = false }) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-race-')); + try { + const state = path.join(root, 'state'); + const nodes = path.join(state, 'nodes'); + const name = 'racing-agent'; + const id = makeIdentity(name); + fs.mkdirSync(path.join(nodes, name, 'cmbs'), { recursive: true }); + for (let i = 0; i < 30; i++) fs.writeFileSync(path.join(nodes, name, 'cmbs', `c${i}.json`), '{}'); + fs.writeFileSync(path.join(nodes, name, 'identity.json'), JSON.stringify(id, null, 2)); + if (withHusk) { fs.mkdirSync(path.join(nodes, 'by-id', id.nodeId), { recursive: true }); fs.writeFileSync(path.join(nodes, 'by-id', id.nodeId, 'lock.pid'), '999999\n'); } + // `stall`: every file call may pause a few ms first, which spreads the four processes' steps through each other. + const stalls = stall ? ` + const sab = new Int32Array(new SharedArrayBuffer(4)); + for (const k of ['renameSync', 'lstatSync', 'statSync', 'readdirSync', 'mkdirSync', 'existsSync', 'readFileSync', 'writeFileSync', 'rmdirSync', 'rmSync', + 'symlinkSync', 'unlinkSync', 'realpathSync', 'readlinkSync', 'chmodSync', 'openSync', 'copyFileSync']) { + const orig = fs[k]; fs[k] = function (...a) { if (Math.random() < 0.35) Atomics.wait(sab, 0, 0, Math.random() * 4); return orig.apply(fs, a); }; + }` : ''; + // `pinned`: the start mesh-channel makes, { name, nodeId, create: false }. + const options = pinned ? `{ name: ${JSON.stringify(name)}, nodeId: ${JSON.stringify(id.nodeId)}, create: false }` : `{ name: ${JSON.stringify(name)} }`; + const code = ` + const fs = require('fs');${stalls} + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + process.stdin.once('data', () => { + try { const i = c.loadIdentity(${options}); console.log(JSON.stringify({ id: i.nodeId, key: i.privateKey })); } + catch (e) { console.log(JSON.stringify({ error: e.name + ': ' + String(e.message).split('\\n')[0].slice(0, 200) })); } + process.exit(0); + }); + console.log('ready');`; + const kids = Array.from({ length: 4 }, () => spawn(process.execPath, ['-e', code], { env: { ...process.env, SYM_STATE_DIR: state }, stdio: ['pipe', 'pipe', 'inherit'] })); + await Promise.all(kids.map((k) => new Promise((resolve) => k.stdout.once('data', resolve)))); + const done = kids.map((k) => new Promise((resolve) => { let b = ''; k.stdout.on('data', (d) => { b += d; }); k.on('exit', () => resolve(JSON.parse(b.trim().split('\n').pop()))); })); + kids.forEach((k) => k.stdin.write('go\n')); + const outcomes = await Promise.all(done); + const kept = JSON.parse(fs.readFileSync(path.join(nodes, 'by-id', id.nodeId, 'identity.json'), 'utf8')); + return { id, outcomes, kept }; + } finally { fs.rmSync(root, { recursive: true, force: true }); } + } + + for (const [withHusk, pinned, stall] of [[false, false, false], [true, false, false], [false, true, true], [true, true, true]]) { + it(`${pinned ? 'pinned starts (nodeId + name, create: false), stalled' : 'starts by name'}, ${withHusk ? 'with a by-id dir that already exists' : 'plain'}: every starter that loads gets the original agent, and the key survives`, async () => { + let loaded = 0; + for (let round = 0; round < 5; round++) { + const { id, outcomes, kept } = await race({ withHusk, pinned, stall }); + assert.strictEqual(kept.privateKey, id.privateKey, 'the original private key is still the one on disk'); + for (const o of outcomes) { + if (o.error) continue; // a starter may be told to start again; it must not become somebody else + loaded++; + assert.strictEqual(o.id, id.nodeId, 'the same nodeId'); + assert.strictEqual(o.key, id.privateKey, 'the same key'); + } + } + assert.ok(loaded >= 5, `starters loaded the agent (${loaded}): the test measured something`); + }); + } +}); + +describe('R1-5: a live 0.13 process in the agent\'s 0.13 directory refuses the start (EIDENTITYLOCK)', () => { + // 632b95d loaded the identity "in place" and the 0.14 node ran beside the 0.13 one: the same identity twice, the + // 0.14 one on an empty by-id store. One agent runs in one process. + async function liveLegacy(name) { + const dir = path.join(NODES_DIR, name); + const child = spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + c.acquireIdentityLock(${JSON.stringify(name)}, { dir: ${JSON.stringify(dir)} }); + console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('lock holder exited'))); }); + return child; + } + + for (const pinned of [false, true]) { + it(`${pinned ? 'pinned' : 'by name'}: refused with EIDENTITYLOCK naming the holder, nothing moved or made; after it stops, the agent migrates`, async () => { + const name = uniq(pinned ? 'r15-pin' : 'r15'); + const id = legacyNode(name); + const before = treeOf(path.join(NODES_DIR, name)); + const child = await liveLegacy(name); + const opts = pinned ? { name, nodeId: id.nodeId, create: false } : { name }; + try { + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-live'); + assert.throws(() => cfg.loadIdentity(opts), (e) => e.code === 'EIDENTITYLOCK' && e.holderPid === child.pid); + assert.ok(!exists(cfg.nodeDirById(id.nodeId)), 'no by-id directory was made for a second copy to run in'); + assert.strictEqual(cfg.nodeIdForName(name), null); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, name)).filter((l) => !l.includes(' lock.pid')), before); + if (pinned) assert.throws(() => cfg.loadIdentity({ name, nodeId: cfg.uuidv7(), create: false }), (e) => e.code === 'EIDENTITYABSENT', 'a different pin does not get it'); + } finally { child.kill('SIGKILL'); await new Promise((r) => child.once('exit', r)); } + const loaded = cfg.loadIdentity(opts); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'x.json')), 'migrated with its store once the 0.13 process is gone'); + }); + } + + it('an identity this host exported does not load from an unmigrated 0.13 directory (tombstone checked)', () => { + const name = uniq('f13'); + const id = legacyNode(name); + put(path.join(cfg.nodeDirById(id.nodeId), cfg.TOMBSTONE), JSON.stringify({ movedAt: Date.now() })); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityTombstonedError); + assert.throws(() => cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }), (e) => e instanceof IdentityTombstonedError); + assert.ok(!isLink(path.join(NODES_DIR, name)), 'and it was not migrated'); + }); +}); + +describe('a start pinned to a nodeId migrates too (mesh-channel starts every node as { name, nodeId, create: false })', () => { + it('create: false, a plain 0.13 dir: the pinned agent loads, moved, with its key files private', () => { + const name = uniq('pin-plain'); + const id = legacyNode(name, { e2e: true }); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + const dir = cfg.nodeDirById(id.nodeId); + assert.ok(exists(path.join(dir, 'cmbs', 'x.json')), 'the store moved with it'); + assert.ok(isLink(path.join(NODES_DIR, name)), 'nodes/ is the compat link'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + if (posix) { + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(path.join(dir, 'e2e-keypair.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + } + assert.strictEqual(cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }).privateKey, id.privateKey, 'and the second start is the same'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); + + it('create: false, a by-id/ dir that already exists (empty, or holding only a lock): the pinned agent loads', () => { + for (const lock of [false, true]) { + const name = uniq(`pin-husk${lock ? '-lock' : ''}`); + const id = legacyNode(name); + husk(id.nodeId, { lock }); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + } + }); + + it('the default create: true loads the original and does not mint (0.14.2 made a new nodeId and lost the key)', () => { + for (const withHusk of [false, true]) { + const name = uniq(`pin-create${withHusk ? '-husk' : ''}`); + const id = legacyNode(name); + if (withHusk) husk(id.nodeId); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1, 'no second identity answers to the name'); + } + }); + + it('with SYM_IDENTITY_DIR: the identity under SYM_IDENTITY_DIR/ moves and loads for the pin', () => { + withIdentityDir((idRoot) => { + const name = uniq('pin-idroot'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(idRoot, 'by-id', id.nodeId, 'identity.json'))); + if (posix) assert.strictEqual(modeOf(path.join(idRoot, 'by-id', id.nodeId, 'identity.json')), 0o600); + }); + }); + + it('a pin that is a DIFFERENT nodeId than the 0.13 directory\'s: the pin stays absent, the directory is left exactly as it is, nothing is made', () => { + const name = uniq('pin-other'); + const other = legacyNode(name); // the agent whose directory this is + const pin = cfg.uuidv7(); // the start asks for another + const identityBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const byIdBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin }), (e) => e instanceof IdentityHaltError && /pinned to/.test(e.reason), 'create: true halts; it does not mint under the name'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), identityBefore, 'the other agent\'s key is byte-unchanged'); + assert.ok(!isLink(path.join(NODES_DIR, name)) && exists(path.join(NODES_DIR, name, 'cmbs', 'x.json')), 'and its directory is where it was, unmoved'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the name was not indexed to either'); + assert.ok(!exists(cfg.nodeDirById(other.nodeId)) && !exists(cfg.nodeDirById(pin)), 'no directory was made for either'); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, byIdBefore); + assert.strictEqual(identitiesNamed(name).length, 0); + // the unpinned start still moves it, as before + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, other.nodeId); + }); + + it('nothing 0.13 to move (only a link to another agent): a pinned start does not index the name from it', () => { + const name = uniq('pin-link'); + const owner = cfg.loadIdentity({ name: uniq('pin-link-owner') }); // some other agent, filed by id + fs.symlinkSync(path.join('by-id', owner.nodeId), path.join(NODES_DIR, name), 'dir'); // a link under this name, name unindexed + assert.throws(() => cfg.loadIdentity({ name, nodeId: cfg.uuidv7(), create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the pinned start wrote no index entry for the name'); + }); + + it('the name is indexed to another agent and the pin\'s 0.13 directory sits under it: the start halts saying the pin IS here (changed in 0.14.3), and does not answer with the indexed agent', () => { + // 632b95d threw IdentityAbsentError ("No identity ... on this host") while the pinned identity sat in nodes/. + const name = uniq('pin-held'); + const indexed = cfg.loadIdentity({ name }); // N -> indexed + fs.unlinkSync(path.join(NODES_DIR, name)); // the link out of the way: a real directory for the pin + const pin = legacyNode(name); + const before = read(path.join(NODES_DIR, name, 'identity.json')); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin.nodeId, create: false }), + (e) => e instanceof IdentityHaltError && e.reason.includes(pin.nodeId) && /is on this host/.test(e.reason) && e.reason.includes(indexed.nodeId)); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), before, 'the pin\'s key is where it was'); + assert.strictEqual(cfg.nodeIdForName(name), indexed.nodeId, 'and the index still says what it said'); + }); +}); + +describe('T9: the checks measure something', () => { + it('assertMoved fails a run that moved nothing, and a populated run passes it', () => { + const name = uniq('t9'); + legacyNode(name); + assertMoved(cfg.migrateIdentities()); + assert.throws(() => assertMoved(cfg.migrateIdentities()), /measured nothing/, 'a second pass moves nothing and must not satisfy the guard'); + }); +});