From d7fa80a8053980bb6f92a07b11b56efb6cd020d9 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 09:54:33 +0100 Subject: [PATCH 01/15] Identity migration: an agent no longer loses its key to the 0.13 -> by-id upgrade An XMesh review of lib/config.js 0.14.2 returned DO-NOT-SHIP with 13 findings. Each was reproduced in a sandbox state dir before anything was changed; the data-loss one was also reported from a developer's machine. What lost the key: migrateNodeDir refused to move a 0.13 node when nodes/by-id// existed, even empty or holding only lock.pid, and left the name unindexed. loadIdentity then read "refused" as "no such agent", made a new identity, renamed the 0.13 directory onto the new node's path and wrote the new identity.json over the old one. A pinned start was bricked instead. The guard asked whether a directory existed; it must ask whether an identity does. Root causes, changed rather than patched: - A second claim on a nodeId is a different identity (or content that cannot merge) already filed under it. An empty directory or a stale lock claims nothing: the node is merged into it. The same identity on both sides is an interrupted move and is finished. A real duplicate is still refused and nothing is overwritten. - loadIdentity treated every answer it did not recognise as "mint". It now mints only when nothing on the host is known under the name: a 0.13 identity still under the name, or a name indexing a node whose identity is missing, halts (IdentityHaltError). - The name is indexed before anything moves and the identity moves last, so a second starter finds the agent and an interrupted move is re-entered by the identity's absence. A step that fails because another process just did it is re-run; a path that became an alias of its destination is never merged onto itself. - Key files: rename keeps the mode 0.13 wrote (0644). identity.json and e2e-keypair.json now end at 0600 and their directories at 0700 after a migration; loading narrows an existing key file, which heals nodes already migrated by 0.14.0-0.14.2. A rewrite of identity.json is atomic. - A move across file systems copies, verifies byte for byte, then removes the source. - Smaller: an identity with a public key and no private key halts; the live-lock fallback checks the tombstone; a symlink that is not a by-id link no longer reads as already indexed; SYM_IDENTITY_DIR switched on for a node whose identity is in its node dir halts with instructions instead of making a new identity. Not fixed, listed in the CHANGELOG: migrateIdentities still discards per-node errors; loadIdentity({nodeId, name}) re-points a name that indexes another node; SYM_IDENTITY_DIR first-time does not move the key for the operator; an explicit nodeId unknown to the index is ignored when a new identity is made. tests/identity-migration-by-id.test.js: 36 tests, 30 of which fail on 0.14.2. Four real processes starting one 0.13 agent at once, 320 starters: every one loaded the original agent. Seventeen mutants of lib/config.js, each caught. --- CHANGELOG.md | 88 ++++ lib/config.js | 285 +++++++++- tests/identity-migration-by-id.test.js | 685 +++++++++++++++++++++++++ 3 files changed, 1031 insertions(+), 27 deletions(-) create mode 100644 tests/identity-migration-by-id.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e766575..8876107d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,93 @@ # Changelog +## 0.14.3 — 2026-10-14 + +**An agent no longer loses its key to the 0.13 → by-id upgrade.** An XMesh review of `lib/config.js` 0.14.2 returned +"do not ship" with 13 findings. Each was checked by running it in a sandbox state dir; 0.14.3 fixes the ones that lose +or expose a key and the small ones beside them. The wire and the public surface are unchanged. + +### What was wrong + +- **A new identity could overwrite the old key.** Moving a 0.13 node (`nodes//`) to `nodes/by-id//` was + refused when that directory already existed, even if it was empty or held only a `lock.pid` (something else made it: + a lock, an interrupted run). The refusal left the node unindexed, and the next start by name did not stop there: + it took "refused" for "no such agent", made a NEW identity, moved the old directory onto the new node's path and + wrote the new `identity.json` over the old one. The agent came back under the same name with a different nodeId and + key, and its peers read what it signed as forged. A start pinned by `nodeId` was stuck instead, refusing to start + every time. +- **A 0.13 key file stayed world-readable.** 0.13 wrote `identity.json` and `e2e-keypair.json` without a mode (0644 + under the usual umask). The migration only renames, so they kept it, in a directory at 0755. A new identity in 0.14 is + written 0600; a migrated one never was. +- **A name whose identity file was missing made a replacement agent.** If the name index named a node and its + `identity.json` was not there (the `SYM_IDENTITY_DIR` volume not mounted at boot, the variable set by one embedder and + not another), a start by name made a new nodeId and key and re-pointed the name to it. +- Smaller, same family: a stray symlink at `nodes/` was reported as already indexed when nothing was indexed; + switching `SYM_IDENTITY_DIR` on for a 0.13 node whose identity was in its node directory reported "not a node" and + went on to make a new identity (the old key survived, orphaned in the new node's directory); an identity with a public key + and no private key loaded, and failed later at its first signature; a cosmetic rewrite of `identity.json` (a bare + Windows hostname) truncated the live key file in place; a start in the moment between the directory move and the + index write found no agent and made one; the live-lock fallback skipped the tombstone check; and nothing moved + across file systems (`EXDEV`), so a node whose directories sit on different volumes could not be migrated. + +### Who is exposed + +- Anyone who upgraded a node from 0.13 to 0.14.0, 0.14.1 or 0.14.2 and whose state root had a `nodes/by-id//` + directory for that node without an `identity.json` in it at the time the node was first started by name. Their key + was destroyed on that start; the symptom is a changed nodeId under the same name and peers rejecting its blocks. + 0.14.3 cannot bring a destroyed key back. If you have the old `identity.json` (a backup, a durable + `SYM_IDENTITY_DIR` volume, a copy of `~/.sym`), put it back at `nodes/by-id//identity.json` and set the + nodeId in the name's file `nodes/by-name/.json` to the old one. +- Anyone who switched `SYM_IDENTITY_DIR` on for an existing 0.13 node: the old key is probably still on disk, in + `nodes/by-id//identity.json` (the node that was made in its place). +- Every node migrated from 0.13 holds a private key readable by other local users until it is next loaded with 0.14.3. + +### Changed + +- **The duplicate check looks at the identity, not the directory.** A second claim on a nodeId is a DIFFERENT + `identity.json` (or other content that cannot be merged) already filed under it. A directory that is empty, or holds a + stale lock, claims nothing: the node is MERGED into it. The same identity on both sides is an interrupted move, and + the next start finishes it. A real duplicate is still refused: nothing is moved and nothing is overwritten. +- **The name is indexed before anything moves, and the identity moves last.** A second starter that arrives in the + middle finds the agent; an interrupted move is re-entered on the next start by the identity's absence. A step that + fails because another process just did it is re-run (up to three times) against what that process left, and a path + that has become an alias of its destination through the other process's compat link is never "merged" onto itself. + Measured with real processes: four starters at once, 320 starters in all (with and without a by-id directory already + in place), every one loaded the original agent and the original key was intact every time. The same run on 0.14.2 + made a new identity in 14 of 100 starters, and destroyed the key in every round that had a by-id directory. +- **A start never makes an identity where one exists.** Before it makes one it checks that no 0.13 `identity.json` + sits under the name (in `nodes//` or `SYM_IDENTITY_DIR//`); if one does and could not be migrated it + halts (`IdentityHaltError`) and says why. A name that indexes a node whose identity is missing halts too. + `create: false` still throws `IdentityAbsentError`. Behaviour change: a node whose `identity.json` was deleted by hand + to "start over" now halts instead of silently becoming a new agent; to start the name again as a different agent, + remove the file the message names (`nodes/by-name/.json`). +- **Key files are private.** After a migration `identity.json` and `e2e-keypair.json` are 0600 and the directories that + hold them 0700; loading an identity narrows an existing key file (and the E2E keypair beside it), so nodes already + migrated by 0.14.0 to 0.14.2 are healed on their next start. A new identity's directory is 0700. Best effort on a + file this process does not own, and a no-op on Windows. +- **A rewrite of `identity.json` is atomic** (a private temp file, synced, renamed over it). +- **A move across file systems copies, verifies byte for byte, then removes the source.** A failed copy is removed and + the source left as it was. +- An identity with a public key and no private key halts. A live 0.13 directory's identity is checked against the + tombstone. A symlink at `nodes/` that is not a by-id link is reported `not-a-node`. +- `SYM_IDENTITY_DIR` switched on for a 0.13 node whose identity is still in `nodes//` halts and tells the + operator to copy it to `SYM_IDENTITY_DIR//identity.json`. It does not move it for them (see below). + +### Known, not fixed here + +- `migrateIdentities()` still counts and discards any error from a single node (including the halts above); only the + counters it returns. The node's own start (`loadIdentity`) raises them, so a start is loud. Whether the bulk call + should report them is open. +- `loadIdentity({ nodeId, name })` still re-points a name that already indexes a different node. +- `SYM_IDENTITY_DIR` switched on for an existing node does not move its key for you (it halts and says how). +- `loadIdentity({ nodeId, name, create: true })` with a `nodeId` the name index does not hold and no identity file still + makes a new identity under a new nodeId (it does not use the one given). + +### Tests + +- `tests/identity-migration-by-id.test.js`: 36 tests (35 run on macOS; one real cross-filesystem test runs on Linux). + 30 of them fail on 0.14.2; five pin behaviour that did not change, and one runs only on Linux. The code is mutation-tested: seventeen + mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. + ## 0.14.2 — 2026-10-14 **The daemon's start lock, hardened after review.** 0.14.1 replaced an unsound inode-number guard with a start lock diff --git a/lib/config.js b/lib/config.js index dad1ebf1..f7a224b6 100644 --- a/lib/config.js +++ b/lib/config.js @@ -267,13 +267,18 @@ function nodeIdForName(name) { } catch { return null; } } -/** Write the name index entry (atomic) and the 0.13-compatible `nodes/` symlink (one release). */ -function indexName(name, nodeId) { +/** Write the name index entry, atomically. The index is authoritative for what a name means. */ +function writeNameIndex(name, nodeId) { ensureDir(path.join(NODES_DIR, 'by-name')); const file = nameIndexPath(name); const tmp = `${file}.${process.pid}.tmp`; fs.writeFileSync(tmp, JSON.stringify({ name, nodeId, at: Date.now() })); fs.renameSync(tmp, file); +} + +/** Write the name index entry (atomic) and the 0.13-compatible `nodes/` symlink (one release). */ +function indexName(name, nodeId) { + writeNameIndex(name, nodeId); linkCompat(path.join(NODES_DIR, name), path.join('by-id', nodeId)); const idRoot = process.env.SYM_IDENTITY_DIR; if (idRoot && fs.existsSync(path.join(idRoot, 'by-id', nodeId))) linkCompat(path.join(idRoot, name), path.join('by-id', nodeId)); @@ -298,10 +303,165 @@ function readTombstone(nodeId) { try { return JSON.parse(fs.readFileSync(path.join(nodeDirById(nodeId), TOMBSTONE), 'utf8')); } catch { return null; } } +const lstatOrNull = (p) => { try { return fs.lstatSync(p); } catch { return null; } }; + +/** The files in a node's directory that hold a PRIVATE KEY: its signing identity and the 0.13 E2E keypair. */ +const PRIVATE_KEY_FILES = ['identity.json', 'e2e-keypair.json']; + +/** + * Narrow a node's private key files to their owner (0600), and with `dirs` the directories that hold + * them (0700). `rename` keeps a file's mode, so a key file a 0.13 node wrote at the umask default + * (0644 — world-readable) stayed world-readable through every move; this is what ends that. + * + * Best effort on purpose: a file this process does not own, or a filesystem without modes (Windows), + * must not stop a node from starting. Nothing here changes what a file holds. + * @param {Array} dirs + */ +function securePrivateFiles(dirs, { narrowDirs = false } = {}) { + const narrow = (p, mode) => { try { if ((fs.statSync(p).mode & 0o077) !== 0) fs.chmodSync(p, mode); } catch { /* absent, or not ours */ } }; + for (const d of new Set(dirs.filter(Boolean))) { + for (const f of PRIVATE_KEY_FILES) narrow(path.join(d, f), 0o600); + if (narrowDirs) narrow(d, 0o700); + } +} + +/** Write a file the way a key file must be written: a private temp file, synced, renamed over the target. */ +function writeFileAtomic0600(file, text) { + const tmp = `${file}.${process.pid}.tmp`; + let fd; + try { + fd = fs.openSync(tmp, 'w', 0o600); + fs.writeSync(fd, text); + fs.fsyncSync(fd); + fs.closeSync(fd); + fd = undefined; + fs.renameSync(tmp, file); + } catch (e) { + if (fd !== undefined) try { fs.closeSync(fd); } catch { /* */ } + try { fs.unlinkSync(tmp); } catch { /* */ } + throw e; + } +} + +/** True when two trees are the same: same names, same bytes, same link targets. */ +function sameTree(a, b) { + const sa = lstatOrNull(a); const sb = lstatOrNull(b); + if (!sa || !sb) return false; + if (sa.isSymbolicLink() || sb.isSymbolicLink()) return sa.isSymbolicLink() && sb.isSymbolicLink() && fs.readlinkSync(a) === fs.readlinkSync(b); + if (sa.isDirectory() || sb.isDirectory()) { + if (!(sa.isDirectory() && sb.isDirectory())) return false; + const ea = fs.readdirSync(a).sort(); const eb = fs.readdirSync(b).sort(); + return ea.length === eb.length && ea.every((n, i) => n === eb[i] && sameTree(path.join(a, n), path.join(b, n))); + } + if (sa.size !== sb.size) return false; + try { return Buffer.compare(fs.readFileSync(a), fs.readFileSync(b)) === 0; } catch { return false; } // unreadable (a socket): not provably the same +} + +/** Copy a tree to a path that does not exist yet (fails with EEXIST, creating nothing, if it does). */ +function copyTree(src, dst) { + const st = fs.lstatSync(src); + if (st.isSymbolicLink()) { fs.symlinkSync(fs.readlinkSync(src), dst); return; } + if (st.isDirectory()) { + fs.mkdirSync(dst, { mode: 0o700 }); + for (const e of fs.readdirSync(src)) copyTree(path.join(src, e), path.join(dst, e)); + fs.chmodSync(dst, st.mode & 0o777); + return; + } + fs.copyFileSync(src, dst, fs.constants.COPYFILE_EXCL); + fs.chmodSync(dst, st.mode & 0o777); +} + +/** + * Move a file or directory to a path that does not exist. `rename` cannot cross a filesystem + * (EXDEV) — and SYM_IDENTITY_DIR exists to put the identity on ANOTHER volume — so then: copy, check + * the copy is the same as the source, and only then remove the source. A failed copy is removed and + * the source is left untouched; nothing is ever deleted that has not been verified elsewhere. + */ +function moveTree(src, dst) { + try { fs.renameSync(src, dst); return; } catch (e) { if (e.code !== 'EXDEV') throw e; } + try { + copyTree(src, dst); + if (!sameTree(src, dst)) throw new Error(`the copy of ${src} at ${dst} does not match its source; the source is untouched`); + } catch (e) { + if (e.code !== 'EEXIST') fs.rmSync(dst, { recursive: true, force: true }); + throw e; + } + fs.rmSync(src, { recursive: true, force: true }); +} + +/** + * The first path under `dst` that `src` could not be merged onto without replacing something that is + * not the same, or null. A stale `lock.pid` never conflicts (its holder was checked; the one already in + * place stands). Two copies of the same file do not conflict; two DIFFERENT files do — and an + * `identity.json` that differs is exactly a second claim on a nodeId. + */ +function mergeConflict(src, dst) { + const s = lstatOrNull(src); const d = lstatOrNull(dst); + if (!s || !d) return null; + if (s.isDirectory() && d.isDirectory()) { + for (const e of fs.readdirSync(src)) { + const c = mergeConflict(path.join(src, e), path.join(dst, e)); + if (c) return c; + } + return null; + } + if (path.basename(src) === 'lock.pid') return null; + return sameTree(src, dst) ? null : dst; +} + +/** True when both paths lead to one file or directory (through a symlink, say). */ +function sameFile(a, b) { + try { return fs.realpathSync(a) === fs.realpathSync(b); } catch { return false; } +} + +/** + * Move `src` onto `dst`, MERGING when `dst` already exists (check `mergeConflict` first): what is not + * in `dst` moves in, what is already there stays. A directory that merely exists is not a reason to + * leave a node where it was. `identity.json` moves LAST, so "the identity is in place" means "everything + * else is too" — an interrupted move is re-entered by the identity's absence. An entry already moved by + * another process (ENOENT) is skipped: the move is safe to run from several processes. + */ +function mergeMove(src, dst) { + const s = lstatOrNull(src); + if (!s) return; + const d = lstatOrNull(dst); + // Already the same file: the other process finished and replaced the old directory with the compat link, so + // this path now leads to `dst`. Moving it onto itself is a no-op at best; deleting it would delete `dst`. + if (d && sameFile(src, dst)) return; + if (!d) { + try { moveTree(src, dst); } catch (e) { if (e.code === 'ENOENT' && !lstatOrNull(src)) return; throw e; } + return; + } + if (s.isDirectory() && d.isDirectory()) { + const names = fs.readdirSync(src).sort((a, b) => (a === 'identity.json') - (b === 'identity.json')); + for (const n of names) mergeMove(path.join(src, n), path.join(dst, n)); + try { fs.rmdirSync(src); } catch (e) { if (e.code !== 'ENOENT') throw e; } + return; + } + // The same file on both sides (or a stale lock): the one in place stands. Checked again here, because + // this deletes: another process may have changed either side since mergeConflict ran. + if (path.basename(src) !== 'lock.pid' && !sameTree(src, dst)) throw new Error(`cannot merge ${src} onto ${dst}: they differ`); + fs.rmSync(src, { recursive: true, force: true }); +} + +/** + * Where a 0.13 identity for `name` still sits (SYM_IDENTITY_DIR/, else nodes/), or null. + * `existsSync` follows a symlink, so a link that leads to an identity counts. + */ +function legacyIdentityPath(name) { + const idRoot = process.env.SYM_IDENTITY_DIR; + for (const p of [idRoot && path.join(idRoot, name, 'identity.json'), path.join(NODES_DIR, name, 'identity.json')]) { + if (p && fs.existsSync(p)) return p; + } + return null; +} + /** * Read and validate an identity.json. Never regenerates a key: a file that exists and cannot be read - * halts (IdentityHaltError). A pre-v0.3.7 identity with no keypair gets its first key; a private key - * with no public key gets the public key derived from it. + * halts (IdentityHaltError), and so does one with a public key and no private key. A pre-v0.3.7 + * identity with no keypair at all gets its first key; a private key with no public key gets the public + * key derived from it. A key file left readable by others (0.13 wrote it at the umask default) is + * narrowed to its owner. * @private */ function readIdentityFile(idPath, name) { @@ -322,6 +482,11 @@ function readIdentityFile(idPath, name) { if (!identity || typeof identity !== 'object' || !identity.nodeId) { throw new IdentityHaltError(name, idPath, 'identity.json has no nodeId — the file is not an identity'); } + // A PUBLIC KEY BUT NO PRIVATE KEY is not "no keypair": the key existed and is gone (a truncated write, + // a scrubbed export). Neither mint one nor sign with `undefined`: a new key is a different agent. + if (identity.publicKey && !identity.privateKey) { + throw new IdentityHaltError(name, idPath, 'identity.json has a public key but no private key — the private key is gone, not missing, and a new one would be a different agent'); + } let mutated = false; // NO KEYPAIR AT ALL (pre-v0.3.7 nodes): nothing to replace, so minting one gives an agent its first // key; the nodeId is preserved. A PRIVATE KEY BUT NO PUBLIC KEY: derive it, never generate one. @@ -346,7 +511,10 @@ function readIdentityFile(idPath, name) { identity.hostname = normalized; mutated = true; } - if (mutated) fs.writeFileSync(idPath, JSON.stringify(identity, null, 2)); + // Atomic and private: a truncate-in-place that is interrupted leaves a key file with no key in it, and a + // mode passed to writeFileSync is ignored for a file that already exists. + if (mutated) writeFileAtomic0600(idPath, JSON.stringify(identity, null, 2)); + securePrivateFiles([path.dirname(idPath)]); return identity; } @@ -360,25 +528,56 @@ function readIdentityFile(idPath, name) { * A 0.13 identity file is read with the same reader a load uses, so one that exists and cannot be * established (unreadable, not JSON, no nodeId) HALTS (IdentityHaltError) instead of reading as * "no node here" — which would let the caller mint a replacement under the agent's name. + * + * What makes a second claim on a nodeId is a DIFFERENT identity (or content that cannot merge) already + * filed under it — not a directory that merely exists. `nodes/by-id//` is also made by a lock, or + * by an earlier interrupted move, and an empty one used to leave the node where it was, unindexed, for + * good. Now the node is MERGED into it. The same identity on both sides is an interrupted move, resumed. + * The name is indexed BEFORE anything moves, so a second process starting meanwhile finds the agent + * rather than concluding there is none; the identity moves last, so an interrupted move is re-entered + * by the identity's absence (`loadIdentity` does it). + * + * Key files end at 0600 (their directory at 0700): `rename` keeps the mode 0.13 wrote them with. + * A move across filesystems (EXDEV) is copied, verified and only then removed. + * + * A 0.13 identity in `nodes//` while SYM_IDENTITY_DIR is set (the volume switched on after the + * agent was made) HALTS: it is neither where the identity dir looks nor safe to leave for a mint. * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'} */ function migrateNodeDir(name) { + // Safe from several processes: a step that fails because another process just did it (an entry gone, a + // directory already replaced by the compat link) is re-run against the state that process left, which + // reads as moved. What persists is thrown, the last one seen. + for (let attempt = 1; ; attempt++) { + try { return migrateNodeDirOnce(name); } catch (e) { if (attempt >= 3) throw e; } + } +} + +function migrateNodeDirOnce(name) { const legacy = path.join(NODES_DIR, name); const idRoot = process.env.SYM_IDENTITY_DIR; const legacyIdDir = idRoot ? path.join(idRoot, name) : legacy; - const lst = (p) => { try { return fs.lstatSync(p); } catch { return null; } }; const indexFromLink = (link) => { const m = /by-id[\\/]([^\\/]+)$/.exec(fs.readlinkSync(link)); if (m && isNodeIdText(m[1])) { indexName(name, m[1]); return 'indexed'; } - return 'already'; + return 'not-a-node'; // a link to somewhere else: nothing was indexed, so it must not read as 'already' }; - const st = lst(legacy); + const st = lstatOrNull(legacy); if (st && st.isSymbolicLink()) return nodeIdForName(name) ? 'already' : indexFromLink(legacy); if (st && !st.isDirectory()) return 'not-a-node'; - const idSt = idRoot ? lst(legacyIdDir) : st; + const idSt = idRoot ? lstatOrNull(legacyIdDir) : st; if (idRoot && idSt && idSt.isSymbolicLink()) return nodeIdForName(name) ? 'already' : indexFromLink(legacyIdDir); const idFile = path.join(legacyIdDir, 'identity.json'); - if (!fs.existsSync(idFile)) return 'not-a-node'; + if (!fs.existsSync(idFile)) { + const inNodeDir = path.join(legacy, 'identity.json'); + if (idRoot && st && fs.existsSync(inNodeDir)) { + throw new IdentityHaltError(name, inNodeDir, + `SYM_IDENTITY_DIR (${idRoot}) is set, but this agent's identity is still in its node directory and not in ${legacyIdDir}. ` + + `It was not moved, and a new one will not be made in its place. Copy it to ${idFile} (keep the original until the agent has started), ` + + 'or unset SYM_IDENTITY_DIR.'); + } + return 'not-a-node'; + } const identity = readIdentityFile(idFile, name); // halts on an identity it cannot establish const nodeId = identity.nodeId; if (!isNodeIdText(nodeId)) throw new IdentityHaltError(name, idFile, `the nodeId ${JSON.stringify(String(nodeId).slice(0, 40))} cannot name a directory under nodes/by-id`); @@ -386,17 +585,24 @@ function migrateNodeDir(name) { const lock = readLockFile(path.join(legacy, 'lock.pid')); if (lock && lock.pid !== process.pid && lockIsHeldByLiveProcess(lock)) return 'skipped-live'; } + // The index gives this name to another node: two identities claim one name, and neither is moved over the other. + const held = nodeIdForName(name); + if (held && held !== nodeId) return 'skipped-duplicate'; const dest = nodeDirById(nodeId); const idDest = idRoot && idSt && idSt.isDirectory() ? path.join(idRoot, 'by-id', nodeId) : null; - // Two dirs claim one nodeId: the first one moved stays, and nothing of this one is moved. - if ((st && fs.existsSync(dest)) || (idDest && fs.existsSync(idDest))) return 'skipped-duplicate'; + // Two dirs claim one nodeId only when what is filed under it differs: the first one moved stays, and + // nothing of this one is moved. An empty directory, or one holding a stale lock, claims nothing. + if ((st && mergeConflict(legacy, dest)) || (idDest && mergeConflict(legacyIdDir, idDest))) return 'skipped-duplicate'; + securePrivateFiles([legacy, legacyIdDir]); // before the move: a copy across volumes must not start from a world-readable file + writeNameIndex(name, nodeId); ensureDir(path.dirname(dest)); - if (st) fs.renameSync(legacy, dest); + if (st) mergeMove(legacy, dest); else ensureDir(dest); if (idDest) { ensureDir(path.dirname(idDest)); - fs.renameSync(legacyIdDir, idDest); + mergeMove(legacyIdDir, idDest); } + securePrivateFiles([dest, idDest], { narrowDirs: true }); indexName(name, nodeId); return 'moved'; } @@ -430,8 +636,10 @@ function migrateIdentities() { * - `create: false` throws IdentityAbsentError when the identity is absent: a host restoring a * known agent can never mint a replacement silently; * - `create: true` (the default, for a brand-new agent) is the ONLY path that mints, and it mints - * only when the name is unknown (a name is an index; renaming changes the index, never the - * identity); + * only when nothing on this host is known under the name: a name that indexes a node whose + * identity is missing HALTS (IdentityHaltError), and so does a 0.13 identity still sitting under + * the name that could not be moved. Minting there would replace a key (a name is an index; + * renaming changes the index, never the identity); * - a tombstoned identity (exported to another host) refuses to load: IdentityTombstonedError. * * @param {{ nodeId?: string, name?: string, create?: boolean }} o @@ -441,17 +649,21 @@ function loadIdentity({ nodeId, name, create = true } = {}) { if (name !== undefined) validateName(name); if (!nodeId && !name) throw new Error('loadIdentity needs a nodeId or a name'); if (nodeId && !isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); + let migrated = null; if (!nodeId && name) { nodeId = nodeIdForName(name); - if (!nodeId) { - const r = migrateNodeDir(name); - if (r === 'moved' || r === 'already' || r === 'indexed') nodeId = nodeIdForName(name); - if (r === 'skipped-live') { + // Not indexed, or indexed but the identity is not in place: the 0.13 layout may still hold it (a move + // another process is making, or one that was interrupted). Migrating is idempotent. + if (!nodeId || !fs.existsSync(path.join(identityDirById(nodeId), 'identity.json'))) { + migrated = migrateNodeDir(name); + if (migrated === 'skipped-live') { // A 0.13 process of this name holds its old directory: load it in place, unmoved. - const idRoot = process.env.SYM_IDENTITY_DIR; - const idPath = idRoot ? path.join(idRoot, name, 'identity.json') : path.join(NODES_DIR, name, 'identity.json'); - return readIdentityFile(idPath, name); + const live = readIdentityFile(legacyIdentityPath(name), name); + const liveTomb = readTombstone(live.nodeId); + if (liveTomb) throw new IdentityTombstonedError(live.nodeId, liveTomb); + return live; } + if (!nodeId) nodeId = nodeIdForName(name); // the index, whichever process wrote it, says who this name is } } if (nodeId) { @@ -461,13 +673,31 @@ function loadIdentity({ nodeId, name, create = true } = {}) { if (fs.existsSync(idPath)) { const identity = readIdentityFile(idPath, name || nodeId); if (identity.nodeId !== nodeId) throw new IdentityHaltError(name || nodeId, idPath, `identity.json names ${identity.nodeId}, not ${nodeId}`); + securePrivateFiles([nodeDirById(nodeId)]); // the 0.13 E2E keypair beside it, when the identity lives elsewhere if (name && nodeIdForName(name) !== nodeId) indexName(name, nodeId); return identity; } if (!create || !name) throw new IdentityAbsentError(name ? `"${name}" (${nodeId})` : nodeId); + // The name indexes a node that exists, and its identity file is not where it should be (a volume not + // mounted, SYM_IDENTITY_DIR set by one embedder and not another, a file removed). That is a known agent + // that cannot be loaded, never a new one. + if (nodeIdForName(name) === nodeId) { + throw new IdentityHaltError(name, idPath, + `the name "${name}" indexes node ${nodeId}, but its identity.json is not there. If SYM_IDENTITY_DIR changed or its volume is not mounted, ` + + `restore that. If this agent is gone for good, remove ${nameIndexPath(name)} to let the name start again as a different agent.`); + } } if (!create) throw new IdentityAbsentError(`"${name}"`); - // No identity under this name: a genuinely new agent. This is the ONLY path that mints one. + // No identity under this name: a genuinely new agent. This is the ONLY path that mints one — and only + // once it is established that no 0.13 identity sits under the name waiting to be moved. A directory with + // an identity.json in it is not "a directory with no identity"; renaming it onto the new node's path would + // bury a live key (and write the new identity over it). + const stranded = legacyIdentityPath(name); + if (stranded) { + throw new IdentityHaltError(name, stranded, + `a 0.13 identity exists under this name and was not migrated (migration said: ${migrated || 'nothing'}). ` + + 'Making a new identity here would orphan or overwrite its key.'); + } const kp = generateSigningKeyPair(); const identity = { nodeId: uuidv7(), @@ -480,15 +710,16 @@ function loadIdentity({ nodeId, name, create = true } = {}) { // A 0.13-style `nodes//` dir with no identity (made by hand, or by a test) is the new node's // dir: it moves with the identity rather than being left behind. const legacy = path.join(NODES_DIR, name); - let legacyDir = false; - try { legacyDir = fs.lstatSync(legacy).isDirectory(); } catch { /* absent */ } + const legacySt = lstatOrNull(legacy); + const legacyDir = Boolean(legacySt && legacySt.isDirectory()); const dir = nodeDirById(identity.nodeId); ensureDir(path.dirname(dir)); - if (legacyDir) fs.renameSync(legacy, dir); + if (legacyDir) moveTree(legacy, dir); ensureDir(dir); const idDir = identityDirById(identity.nodeId); ensureDir(idDir); fs.writeFileSync(path.join(idDir, 'identity.json'), JSON.stringify(identity, null, 2), { mode: 0o600 }); + securePrivateFiles([dir, idDir], { narrowDirs: true }); indexName(name, identity.nodeId); return identity; } diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js new file mode 100644 index 00000000..7438d0cc --- /dev/null +++ b/tests/identity-migration-by-id.test.js @@ -0,0 +1,685 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME to a temp sandbox before lib/config loads + +/** + * The 0.13 -> by-id identity migration must never cost an agent its key. + * + * 0.14.0 - 0.14.2 had two ways to destroy one. (1) The duplicate guard asked "does nodes/by-id// + * EXIST?" instead of "does an IDENTITY already sit there?", so a by-id directory made by anything else (a + * lock, an interrupted move) left the 0.13 node where it was, unindexed, for good. (2) `loadIdentity` read + * every answer it did not recognise as "no such agent" and MINTED: it renamed the 0.13 directory (the old + * identity.json still in it) onto the new node's path and wrote the new identity over the old key. + * + * The tests below hold the line the founder drew ("the system must refuse to start rather than silently + * create a replacement identity"): a name that has an identity on this host never gets a new one. + * Every test uses its own agent name; the state root is shared by the file, as in config.test.js. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const crypto = require('crypto'); +const { spawn } = require('child_process'); +const cfg = require('../lib/config'); + +const { NODES_DIR, IdentityHaltError, IdentityTombstonedError } = cfg; +const posix = process.platform !== 'win32'; // modes are not a thing on Windows +const modeTest = posix ? it : it.skip; + +let seq = 0; +const uniq = (b) => `${b}-${process.pid}-${++seq}-${Math.random().toString(36).slice(2, 6)}`; +const modeOf = (p) => fs.statSync(p).mode & 0o777; +const read = (p) => fs.readFileSync(p, 'utf8'); +const exists = (p) => fs.existsSync(p); +const isLink = (p) => { try { return fs.lstatSync(p).isSymbolicLink(); } catch { return false; } }; + +/** A real Ed25519 identity, in the shape lib/config writes. */ +function makeIdentity(name, extra = {}) { + const kp = crypto.generateKeyPairSync('ed25519'); + return { + nodeId: cfg.uuidv7(), name, hostname: 'host.local', createdAt: 1, + publicKey: kp.publicKey.export({ format: 'der', type: 'spki' }).subarray(12).toString('base64url'), + privateKey: kp.privateKey.export({ format: 'der', type: 'pkcs8' }).subarray(16).toString('base64url'), + ...extra, + }; +} + +function put(file, text, mode = 0o644) { + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, text); + fs.chmodSync(file, mode); // explicit: the umask must not decide what a test means +} + +/** What 0.13 left on disk: nodes// with identity.json (and e2e-keypair.json) at the umask default. */ +function legacyNode(name, { e2e = false, identity = makeIdentity(name), where = path.join(NODES_DIR, name), withIdentity = true } = {}) { + if (withIdentity) put(path.join(where, 'identity.json'), JSON.stringify(identity, null, 2), 0o644); + if (e2e) put(path.join(where, 'e2e-keypair.json'), JSON.stringify({ publicKey: 'cHVi', privateKey: 'cHJpdg==' }, null, 2), 0o644); + put(path.join(where, 'cmbs', 'x.json'), '{"x":1}'); + return identity; +} + +/** A nodes/by-id// that something else made: a lock, an interrupted run. No identity in it. */ +function husk(nodeId, { lock = true } = {}) { + const d = cfg.nodeDirById(nodeId); + fs.mkdirSync(d, { recursive: true }); + if (lock) put(path.join(d, 'lock.pid'), '999999\n'); + return d; +} + +/** Every identity.json under nodes/by-id and SYM_IDENTITY_DIR/by-id whose `name` field is `name`. */ +function identitiesNamed(name) { + const found = []; + const roots = [path.join(NODES_DIR, 'by-id'), process.env.SYM_IDENTITY_DIR && path.join(process.env.SYM_IDENTITY_DIR, 'by-id')].filter(Boolean); + for (const root of roots) { + if (!exists(root)) continue; + for (const d of fs.readdirSync(root)) { + const f = path.join(root, d, 'identity.json'); + try { if (JSON.parse(read(f)).name === name) found.push(f); } catch { /* */ } + } + } + return found; +} + +function withIdentityDir(fn) { + const prev = process.env.SYM_IDENTITY_DIR; + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-idroot-')); + process.env.SYM_IDENTITY_DIR = root; + try { return fn(root); } finally { + if (prev === undefined) delete process.env.SYM_IDENTITY_DIR; else process.env.SYM_IDENTITY_DIR = prev; + fs.rmSync(root, { recursive: true, force: true }); + } +} + +/** Replace fs. for the duration of `body`; `wrap(original)` returns the replacement. */ +function patchFs(fn, wrap, body) { + const original = fs[fn]; + fs[fn] = wrap(original); + try { return body(); } finally { fs[fn] = original; } +} +const exdev = (msg) => Object.assign(new Error(`EXDEV: cross-device link not permitted (${msg})`), { code: 'EXDEV' }); + +/** + * T9: every case asserts it measured something. A run over nothing returns moved: 0 and would "pass" a + * test that only checked for the absence of damage. + */ +function assertMoved(result, atLeast = 1) { + assert.ok(result && result.moved >= atLeast, `the migration moved ${result && result.moved}, expected at least ${atLeast}: the test measured nothing`); +} + +describe('T1/T2: the migration leaves key files private', () => { + modeTest('identity.json written 0644 by 0.13 ends at 0600, and its directory at 0700', () => { + const name = uniq('t1'); + const id = legacyNode(name); + assert.strictEqual(modeOf(path.join(NODES_DIR, name, 'identity.json')), 0o644, 'setup: a 0.13 key file, world-readable'); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const dir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + assert.strictEqual(JSON.parse(read(path.join(dir, 'identity.json'))).privateKey, id.privateKey, 'the key is the same key'); + }); + + modeTest('e2e-keypair.json ends at 0600 with its bytes unchanged', () => { + const name = uniq('t2'); + const id = legacyNode(name, { e2e: true }); + const before = read(path.join(NODES_DIR, name, 'e2e-keypair.json')); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const f = path.join(cfg.nodeDirById(id.nodeId), 'e2e-keypair.json'); + assert.strictEqual(modeOf(f), 0o600); + assert.strictEqual(read(f), before); + }); + + modeTest('with SYM_IDENTITY_DIR: the identity file is 0600 and both directories 0700 (T8)', () => { + withIdentityDir((idRoot) => { + const name = uniq('t8'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + put(path.join(NODES_DIR, name, 'e2e-keypair.json'), '{}', 0o644); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + const idDir = path.join(idRoot, 'by-id', id.nodeId); + const nodeDir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(idDir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(idDir), 0o700); + assert.strictEqual(modeOf(nodeDir), 0o700); + assert.strictEqual(modeOf(path.join(nodeDir, 'e2e-keypair.json')), 0o600); + assert.ok(exists(path.join(nodeDir, 'cmbs', 'x.json')), 'the node dir moved with its store'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId, 'the index resolves'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + }); + + modeTest('loading narrows a key file an earlier release already migrated at 0644 (and the E2E keypair beside it)', () => { + const name = uniq('heal'); + const id = legacyNode(name, { e2e: true }); + cfg.migrateNodeDir(name); + const dir = cfg.nodeDirById(id.nodeId); + fs.chmodSync(path.join(dir, 'identity.json'), 0o644); fs.chmodSync(path.join(dir, 'e2e-keypair.json'), 0o644); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(path.join(dir, 'e2e-keypair.json')), 0o600); + }); + + modeTest('a rewrite of identity.json (bare Windows hostname) lands at 0600, not at the old file mode', () => { + const name = uniq('rewrite'); + const id = legacyNode(name, { identity: makeIdentity(name, { hostname: 'bare-host' }) }); + cfg.migrateNodeDir(name); + const f = path.join(cfg.nodeDirById(id.nodeId), 'identity.json'); + const onDisk = JSON.parse(read(f)); onDisk.hostname = 'bare-host'; + put(f, JSON.stringify(onDisk, null, 2), 0o644); + const loaded = cfg.loadIdentity({ name, create: false }); + assert.strictEqual(loaded.hostname, 'bare-host.local'); + assert.strictEqual(JSON.parse(read(f)).hostname, 'bare-host.local', 'the rewrite happened'); + assert.strictEqual(modeOf(f), 0o600); + }); + + modeTest('a newly made agent: identity 0600, its directory 0700', () => { + const name = uniq('fresh'); + const id = cfg.loadIdentity({ name }); + const dir = cfg.nodeDirById(id.nodeId); + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + }); +}); + +describe('T3-T5: a by-id directory that merely exists is not a second claim on the nodeId', () => { + it('T3: the node is merged into an existing by-id dir that holds only lock.pid', () => { + const name = uniq('t3'); + const id = legacyNode(name); + const dest = husk(id.nodeId); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey, 'the identity is under the nodeId, key unchanged'); + assert.ok(exists(path.join(dest, 'lock.pid')), 'what was already there stays'); + assert.ok(exists(path.join(dest, 'cmbs', 'x.json')), 'the store came with it'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + assert.ok(isLink(path.join(NODES_DIR, name)), 'nodes/ is the compat link'); + assert.strictEqual(fs.realpathSync(path.join(NODES_DIR, name)), fs.realpathSync(dest)); + }); + + it('T3: an EMPTY by-id dir is merged into too', () => { + const name = uniq('t3e'); + const id = legacyNode(name); + husk(id.nodeId, { lock: false }); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'identity.json'))); + }); + + it('T4: a pinned start recovers once the node is migrated', () => { + const name = uniq('t4'); + const id = legacyNode(name); + husk(id.nodeId); + assertMoved(cfg.migrateIdentities()); + const loaded = cfg.loadIdentity({ nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + }); + + it('T5: an unpinned start never mints over a husk: same nodeId, same key, and no second identity exists', () => { + const name = uniq('t5'); + const id = legacyNode(name); + husk(id.nodeId); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + const files = identitiesNamed(name); + assert.strictEqual(files.length, 1, `exactly one identity answers to ${name}: ${files.join(', ')}`); + assert.strictEqual(JSON.parse(read(files[0])).privateKey, id.privateKey, 'and it still holds the original private key'); + }); + + it('the reported repro: a 0.13 dir plus a pre-existing EMPTY by-id// keeps the old key (0.14.2 minted and overwrote it)', () => { + const name = uniq('repro'); + const old = legacyNode(name); + fs.mkdirSync(path.join(NODES_DIR, 'by-id', old.nodeId), { recursive: true }); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, old.nodeId, 'not a new nodeId'); + assert.strictEqual(loaded.privateKey, old.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + assert.strictEqual(JSON.parse(read(path.join(cfg.nodeDirById(old.nodeId), 'identity.json'))).privateKey, old.privateKey, 'the old private key is still on disk'); + assert.strictEqual(cfg.nodeIdForName(name), old.nodeId); + }); + + it('the same identity on both sides is an interrupted move, resumed, not a duplicate', () => { + const name = uniq('resume-same'); + const id = legacyNode(name); + const dest = cfg.nodeDirById(id.nodeId); + put(path.join(dest, 'identity.json'), read(path.join(NODES_DIR, name, 'identity.json')), 0o600); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(exists(path.join(dest, 'cmbs', 'x.json'))); + assert.ok(isLink(path.join(NODES_DIR, name))); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); +}); + +describe('T6/T7: a real duplicate is refused, and refusing is a halt, never a mint', () => { + function duplicate(name) { + const mine = legacyNode(name); + const other = makeIdentity(name, { nodeId: mine.nodeId }); // another key, filed under the same nodeId + const dest = cfg.nodeDirById(mine.nodeId); + put(path.join(dest, 'identity.json'), JSON.stringify(other, null, 2), 0o600); + return { mine, other, dest }; + } + + it('T6: a DIFFERENT identity.json already under the nodeId: skipped-duplicate, and nothing is overwritten', () => { + const name = uniq('t6'); + const { mine, dest } = duplicate(name); + const legacyBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const destBefore = read(path.join(dest, 'identity.json')); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), legacyBefore, 'the 0.13 identity is byte-unchanged'); + assert.strictEqual(read(path.join(dest, 'identity.json')), destBefore, 'the one already filed is byte-unchanged'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'nothing was indexed'); + assert.ok(!exists(path.join(dest, 'cmbs')), 'and nothing was merged in beside it'); + assert.ok(mine.privateKey); + }); + + it('T7: loading that name halts (IdentityHaltError) and does not touch the 0.13 file', () => { + const name = uniq('t7'); + const { dest } = duplicate(name); + const legacyBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const destBefore = read(path.join(dest, 'identity.json')); + const dirsBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /0\.13 identity/.test(e.reason)); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), legacyBefore); + assert.strictEqual(read(path.join(dest, 'identity.json')), destBefore); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, dirsBefore, 'no new node directory was made'); + assert.strictEqual(cfg.nodeIdForName(name), null); + }); + + it('a differing store file in an existing by-id dir is a conflict too: nothing moves', () => { + const name = uniq('conflict'); + const id = legacyNode(name); + const dest = husk(id.nodeId, { lock: false }); + put(path.join(dest, 'cmbs', 'x.json'), '{"x":2}'); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the 0.13 node is where it was'); + assert.strictEqual(read(path.join(dest, 'cmbs', 'x.json')), '{"x":2}'); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + }); + + it('a name the index already gives to ANOTHER node is not re-pointed by a stray 0.13 directory', () => { + const name = uniq('held'); + const held = cfg.loadIdentity({ name }); + const stray = makeIdentity(name); + fs.unlinkSync(path.join(NODES_DIR, name)); // a real directory where the index says there is a link + legacyNode(name, { identity: stray }); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-duplicate'); + assert.strictEqual(cfg.nodeIdForName(name), held.nodeId, 'the index is untouched'); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, held.nodeId, 'and the indexed agent still loads'); + assert.strictEqual(JSON.parse(read(path.join(NODES_DIR, name, 'identity.json'))).privateKey, stray.privateKey, 'the stray key is not destroyed'); + }); +}); + +describe('F4: a name that indexes a node whose identity is missing halts', () => { + it('create (the default) halts and neither mints nor re-points the index', () => { + const name = uniq('f4'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + const dirsBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && e.reason.includes(first.nodeId)); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId, 'the index still names the agent'); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, dirsBefore, 'nothing was minted'); + }); + + it('create: false is still IdentityAbsentError (EIDENTITYABSENT)', () => { + const name = uniq('f4b'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + assert.throws(() => cfg.loadIdentity({ name, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.throws(() => cfg.loadIdentity({ nodeId: first.nodeId, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + }); + + it('an identity volume that is not mounted at boot halts instead of making a replacement agent', () => { + withIdentityDir((idRoot) => { + const name = uniq('f4v'); + const first = cfg.loadIdentity({ name }); + assert.ok(exists(path.join(idRoot, 'by-id', first.nodeId, 'identity.json')), 'setup: the key is on the identity volume'); + const away = `${idRoot}.away`; + fs.renameSync(idRoot, away); + try { + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId); + } finally { fs.rmSync(idRoot, { recursive: true, force: true }); fs.renameSync(away, idRoot); } + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, first.privateKey, 'mounted again: the same agent'); + }); + }); +}); + +describe('the safety net: the mint path never buries a 0.13 identity', () => { + it('F5: SYM_IDENTITY_DIR switched on for a 0.13 node whose identity is in its node dir halts, and moves nothing', () => { + withIdentityDir(() => { + const name = uniq('f5'); + const id = legacyNode(name); + const before = read(path.join(NODES_DIR, name, 'identity.json')); + assert.throws(() => cfg.migrateNodeDir(name), (e) => e instanceof IdentityHaltError && /SYM_IDENTITY_DIR/.test(e.reason)); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), before); + assert.ok(!isLink(path.join(NODES_DIR, name)) && exists(path.join(NODES_DIR, name, 'cmbs')), 'the node dir was not moved'); + assert.strictEqual(identitiesNamed(name).length, 0, 'and no identity was made'); + assert.ok(id.privateKey); + }); + }); + + it('F7: a stray symlink at nodes/ that leads to an identity but is not a by-id link does not read as indexed, and halts', () => { + const name = uniq('f7'); + const elsewhere = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-stray-')), 'agent'); + const id = legacyNode(name, { where: elsewhere }); + fs.mkdirSync(NODES_DIR, { recursive: true }); + fs.symlinkSync(elsewhere, path.join(NODES_DIR, name), 'dir'); + assert.strictEqual(cfg.migrateNodeDir(name), 'not-a-node', 'nothing was indexed, so not "already"'); + assert.strictEqual(cfg.nodeIdForName(name), null); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.strictEqual(fs.readlinkSync(path.join(NODES_DIR, name)), elsewhere, 'the link is left alone'); + assert.strictEqual(JSON.parse(read(path.join(elsewhere, 'identity.json'))).privateKey, id.privateKey); + }); + + it('a hand-made nodes// with NO identity still becomes the new agent\'s directory (documented, unchanged)', () => { + const name = uniq('nodir-id'); + put(path.join(NODES_DIR, name, 'cmbs', 'kept.json'), '{"k":1}'); + const id = cfg.loadIdentity({ name }); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'kept.json')), 'the store moved with the new identity'); + assert.ok(isLink(path.join(NODES_DIR, name))); + }); + + it('a brand-new name still mints exactly one identity', () => { + const name = uniq('brand-new'); + const a = cfg.loadIdentity({ name }); + const b = cfg.loadIdentity({ name }); + assert.strictEqual(a.nodeId, b.nodeId); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F10: a public key without a private key halts', () => { + it('is not loaded, not repaired, and the file is untouched', () => { + const name = uniq('f10'); + const id = makeIdentity(name); delete id.privateKey; + legacyNode(name, { identity: id }); + const f = path.join(NODES_DIR, name, 'identity.json'); + const before = read(f); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /no private key/.test(e.reason)); + assert.strictEqual(read(f), before); + }); +}); + +describe('F6: rewriting identity.json is atomic', () => { + it('a failure while writing leaves the original key file intact, and no temp file behind', () => { + const name = uniq('f6'); + const id = legacyNode(name, { identity: makeIdentity(name, { hostname: 'bare-host' }) }); + const f = path.join(NODES_DIR, name, 'identity.json'); + const before = read(f); + let injected = 0; + patchFs('writeSync', (orig) => function (fd, ...rest) { + injected++; + throw Object.assign(new Error('ENOSPC: no space left on device (injected)'), { code: 'ENOSPC' }); + }, () => { + assert.throws(() => cfg.loadIdentity({ name }), /ENOSPC/); + }); + assert.ok(injected >= 1, 'the failure was reached'); + assert.strictEqual(read(f), before, 'the key file is byte-identical: it was never truncated'); + assert.deepStrictEqual(fs.readdirSync(path.join(NODES_DIR, name)).filter((n) => n.endsWith('.tmp')), []); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'and the next start succeeds with the same key'); + }); +}); + +describe('F9: a move across filesystems copies, verifies, then removes', () => { + it('EXDEV on the identity dir: the identity arrives intact at 0600 and the source is gone', () => { + withIdentityDir((idRoot) => { + const name = uniq('f9'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(idRoot, name, 'sub', 'a.txt'), 'a'); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const result = patchFs('renameSync', (orig) => function (a, b) { + if (String(a).startsWith(idRoot) && String(b).startsWith(idRoot)) throw exdev('simulated'); + return orig.call(fs, a, b); + }, () => cfg.migrateNodeDir(name)); + assert.strictEqual(result, 'moved'); + const dest = path.join(idRoot, 'by-id', id.nodeId); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + assert.strictEqual(read(path.join(dest, 'sub', 'a.txt')), 'a'); + if (posix) assert.strictEqual(modeOf(path.join(dest, 'identity.json')), 0o600); + assert.ok(!exists(path.join(idRoot, name, 'identity.json')) || isLink(path.join(idRoot, name)), 'the source is gone (only the compat link is left)'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + }); + + it('a copy that fails leaves the source untouched and no half-copy at the destination', () => { + withIdentityDir((idRoot) => { + const name = uniq('f9b'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o600); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const src = read(path.join(idRoot, name, 'identity.json')); + assert.throws(() => patchFs('renameSync', (orig) => function (a, b) { + if (String(a).startsWith(idRoot) && String(b).startsWith(idRoot)) throw exdev('simulated'); + return orig.call(fs, a, b); + }, () => patchFs('copyFileSync', () => function () { throw Object.assign(new Error('EIO: injected'), { code: 'EIO' }); }, + () => cfg.migrateNodeDir(name))), /EIO/); + assert.strictEqual(read(path.join(idRoot, name, 'identity.json')), src, 'the identity is where it was'); + assert.ok(!exists(path.join(idRoot, 'by-id', id.nodeId)), 'no half-copy'); + // the fault clears: the next run completes the move + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + assert.ok(exists(path.join(idRoot, 'by-id', id.nodeId, 'identity.json'))); + }); + }); +}); + +describe('F9 on a real second filesystem (Linux: nodes/by-id lives on /dev/shm, a tmpfs)', () => { + const shm = '/dev/shm'; + const second = process.platform === 'linux' && exists(shm) && fs.statSync(shm).dev !== fs.statSync(os.tmpdir()).dev; + (second ? it : it.skip)('rename fails with EXDEV for real, and the node still arrives whole, private, and unchanged', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-exdev-')); + const far = fs.mkdtempSync(path.join(shm, 'sym-exdev-')); + try { + const state = path.join(root, 'state'); + fs.mkdirSync(path.join(state, 'nodes'), { recursive: true }); + fs.symlinkSync(far, path.join(state, 'nodes', 'by-id'), 'dir'); // everything filed by nodeId lands on the other filesystem + const script = ` + const fs = require('fs'), path = require('path'); + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + const name = 'exdev-agent'; + const id = JSON.parse(process.env.LEGACY_IDENTITY); + const dir = path.join(c.NODES_DIR, name); + fs.mkdirSync(path.join(dir, 'cmbs'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'identity.json'), JSON.stringify(id, null, 2), { mode: 0o644 }); fs.chmodSync(path.join(dir, 'identity.json'), 0o644); + fs.writeFileSync(path.join(dir, 'e2e-keypair.json'), '{}'); fs.chmodSync(path.join(dir, 'e2e-keypair.json'), 0o644); + fs.writeFileSync(path.join(dir, 'cmbs', 'x.json'), '{"x":1}'); + let sawExdev = false; + const orig = fs.renameSync; + fs.renameSync = function (a, b) { try { return orig.call(fs, a, b); } catch (e) { if (e.code === 'EXDEV') sawExdev = true; throw e; } }; + const loaded = c.loadIdentity({ name }); + const dest = c.nodeDirById(id.nodeId); + console.log(JSON.stringify({ + sawExdev, same: loaded.nodeId === id.nodeId && loaded.privateKey === id.privateKey, + sourceGone: !fs.existsSync(path.join(dir, 'cmbs')) || fs.lstatSync(dir).isSymbolicLink(), + store: fs.readFileSync(path.join(dest, 'cmbs', 'x.json'), 'utf8'), + idMode: fs.statSync(path.join(dest, 'identity.json')).mode & 0o777, e2eMode: fs.statSync(path.join(dest, 'e2e-keypair.json')).mode & 0o777, + dirMode: fs.statSync(dest).mode & 0o777, onOtherFs: fs.realpathSync(dest).startsWith(fs.realpathSync(${JSON.stringify(far)})), + }));`; + const id = makeIdentity('exdev-agent'); + const out = await new Promise((resolve, reject) => { + const child = spawn(process.execPath, ['-e', script], { + env: { ...process.env, SYM_STATE_DIR: state, LEGACY_IDENTITY: JSON.stringify(id) }, stdio: ['ignore', 'pipe', 'pipe'], + }); + let so = ''; let se = ''; + child.stdout.on('data', (d) => { so += d; }); child.stderr.on('data', (d) => { se += d; }); + child.on('exit', (code) => (code === 0 ? resolve(so) : reject(new Error(`child exited ${code}: ${se}`)))); + }); + const r = JSON.parse(out); + assert.strictEqual(r.onOtherFs, true, 'setup: the node lives on the other filesystem'); + assert.strictEqual(r.sawExdev, true, 'the kernel really refused the rename'); + assert.strictEqual(r.same, true); + assert.strictEqual(r.sourceGone, true); + assert.strictEqual(r.store, '{"x":1}'); + assert.strictEqual(r.idMode, 0o600); + assert.strictEqual(r.e2eMode, 0o600); + assert.strictEqual(r.dirMode, 0o700); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + fs.rmSync(far, { recursive: true, force: true }); + } + }); +}); + +describe('F11: the window between the move and the index is not a mint race', () => { + it('a second starter that arrives mid-move finds the agent instead of making a new one', () => { + const name = uniq('f11'); + const id = legacyNode(name); + let racer = null; + patchFs('renameSync', (orig) => function (a, b) { + const r = orig.call(fs, a, b); + if (racer === null && a === path.join(NODES_DIR, name)) { // the node dir has just moved; the index is not final yet + racer = 'running'; + try { racer = cfg.loadIdentity({ name }); } catch (e) { racer = e; } + } + return r; + }, () => { assert.strictEqual(cfg.loadIdentity({ name }).nodeId, id.nodeId); }); + assert.ok(racer && racer.nodeId, `the racing starter loaded an identity: ${racer && racer.message}`); + assert.strictEqual(racer.nodeId, id.nodeId, 'the same agent'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); + + it('an interrupted move is resumed by the next start: the identity moves last, and its absence re-enters the move', () => { + const name = uniq('f11r'); + const id = legacyNode(name); + put(path.join(NODES_DIR, name, 'aaa.json'), '{}'); put(path.join(NODES_DIR, name, 'zzz.json'), '{}'); // names on both sides of identity.json + husk(id.nodeId); // forces a merge, entry by entry + patchFs('renameSync', (orig) => function (a, b) { + if (path.basename(String(a)) === 'identity.json' && String(a).startsWith(NODES_DIR)) throw Object.assign(new Error('EIO: injected'), { code: 'EIO' }); + return orig.call(fs, a, b); + }, () => { assert.throws(() => cfg.migrateNodeDir(name), /EIO/); }); + assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the identity had not moved yet'); + for (const f of ['aaa.json', 'zzz.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), f)), `but the rest had: ${f}`); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId, 'the name was indexed first'); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(isLink(path.join(NODES_DIR, name)), 'finished: nodes/ is the compat link'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + +describe('F11: another process finishing the move underneath this one', () => { + it('a step that fails because the other process just did it is re-run against what it left, and reads as moved', () => { + const name = uniq('f11-other'); + const id = legacyNode(name); + const legacy = path.join(NODES_DIR, name); + const dest = cfg.nodeDirById(id.nodeId); + let helped = false; + const result = patchFs('renameSync', (orig) => function (a, b) { + if (!helped && a === legacy) { // the other process gets there first: it moves the directory and leaves the link + helped = true; + orig.call(fs, a, b); + fs.symlinkSync(path.join('by-id', id.nodeId), legacy, 'dir'); + throw Object.assign(new Error(`ENOENT: no such file or directory, rename '${a}' -> '${b}'`), { code: 'ENOENT' }); + } + return orig.call(fs, a, b); + }, () => cfg.migrateNodeDir(name)); + assert.ok(helped, 'the other process ran'); + assert.strictEqual(result, 'already'); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); + }); + + it('a path that now leads through the other process\'s link to the destination is never "merged" onto itself (that would delete it)', () => { + const name = uniq('f11-alias'); + const id = legacyNode(name); + put(path.join(NODES_DIR, name, 'aaa.json'), '{"a":1}'); + const legacy = path.join(NODES_DIR, name); + const dest = husk(id.nodeId); // forces an entry-by-entry merge + let helped = false; let listings = 0; + const result = patchFs('readdirSync', (orig) => function (p, ...rest) { + const names = orig.call(fs, p, ...rest); + if (!helped && p === legacy && ++listings === 2) { // the 2nd listing is the merge's own, after the conflict check's + helped = true; // the other process finishes the whole move while this one holds the list + for (const n of names) fs.renameSync(path.join(legacy, n), path.join(dest, n)); + fs.rmdirSync(legacy); + fs.symlinkSync(path.join('by-id', id.nodeId), legacy, 'dir'); + } + return names; + }, () => cfg.migrateNodeDir(name)); + assert.ok(helped, 'the other process ran'); + assert.strictEqual(result, 'already'); + for (const f of ['identity.json', 'aaa.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(dest, f)), `${f} survived`); + assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + }); +}); + +describe('F11 with real processes: several starters at once never make a second identity', () => { + /** Four processes call loadIdentity({ name }) at the same moment on a 0.13 node, in a state root of their own. */ + async function race({ withHusk }) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-race-')); + try { + const state = path.join(root, 'state'); + const nodes = path.join(state, 'nodes'); + const name = 'racing-agent'; + const id = makeIdentity(name); + fs.mkdirSync(path.join(nodes, name, 'cmbs'), { recursive: true }); + for (let i = 0; i < 30; i++) fs.writeFileSync(path.join(nodes, name, 'cmbs', `c${i}.json`), '{}'); + fs.writeFileSync(path.join(nodes, name, 'identity.json'), JSON.stringify(id, null, 2)); + if (withHusk) { fs.mkdirSync(path.join(nodes, 'by-id', id.nodeId), { recursive: true }); fs.writeFileSync(path.join(nodes, 'by-id', id.nodeId, 'lock.pid'), '999999\n'); } + const code = ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + process.stdin.once('data', () => { + try { const i = c.loadIdentity({ name: ${JSON.stringify(name)} }); console.log(JSON.stringify({ id: i.nodeId, key: i.privateKey })); } + catch (e) { console.log(JSON.stringify({ error: e.name + ': ' + String(e.message).split('\\n')[0].slice(0, 200) })); } + process.exit(0); + }); + console.log('ready');`; + const kids = Array.from({ length: 4 }, () => spawn(process.execPath, ['-e', code], { env: { ...process.env, SYM_STATE_DIR: state }, stdio: ['pipe', 'pipe', 'inherit'] })); + await Promise.all(kids.map((k) => new Promise((resolve) => k.stdout.once('data', resolve)))); + const done = kids.map((k) => new Promise((resolve) => { let b = ''; k.stdout.on('data', (d) => { b += d; }); k.on('exit', () => resolve(JSON.parse(b.trim().split('\n').pop()))); })); + kids.forEach((k) => k.stdin.write('go\n')); + const outcomes = await Promise.all(done); + const kept = JSON.parse(fs.readFileSync(path.join(nodes, 'by-id', id.nodeId, 'identity.json'), 'utf8')); + return { id, outcomes, kept }; + } finally { fs.rmSync(root, { recursive: true, force: true }); } + } + + for (const withHusk of [false, true]) { + it(`${withHusk ? 'with a by-id dir that already exists' : 'plain'}: every starter that loads gets the original agent, and the key survives`, async () => { + let loaded = 0; + for (let round = 0; round < 5; round++) { + const { id, outcomes, kept } = await race({ withHusk }); + assert.strictEqual(kept.privateKey, id.privateKey, 'the original private key is still the one on disk'); + for (const o of outcomes) { + if (o.error) continue; // a starter may be told to start again; it must not become somebody else + loaded++; + assert.strictEqual(o.id, id.nodeId, 'the same nodeId'); + assert.strictEqual(o.key, id.privateKey, 'the same key'); + } + } + assert.ok(loaded >= 5, `starters loaded the agent (${loaded}): the test measured something`); + }); + } +}); + +describe('F13: the live-lock fallback checks the tombstone', () => { + it('an identity exported away does not load from an unmoved 0.13 directory a live process holds', async () => { + const name = uniq('f13'); + const id = legacyNode(name); + const dir = path.join(NODES_DIR, name); + const child = spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + c.acquireIdentityLock(${JSON.stringify(name)}, { dir: ${JSON.stringify(dir)} }); + console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + try { + await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('lock holder exited'))); }); + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-live', 'setup: a live process holds the old directory'); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, id.nodeId, 'not tombstoned: it loads in place'); + put(path.join(cfg.nodeDirById(id.nodeId), cfg.TOMBSTONE), JSON.stringify({ movedAt: Date.now() })); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityTombstonedError); + } finally { child.kill(); } + }); +}); + +describe('T9: the checks measure something', () => { + it('assertMoved fails a run that moved nothing, and a populated run passes it', () => { + const name = uniq('t9'); + legacyNode(name); + assertMoved(cfg.migrateIdentities()); + assert.throws(() => assertMoved(cfg.migrateIdentities()), /measured nothing/, 'a second pass moves nothing and must not satisfy the guard'); + }); +}); From 0e31e380d9d7ec7068fc3de24357658a304ae8b1 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 09:54:49 +0100 Subject: [PATCH 02/15] 0.14.3 --- package-lock.json | 4 ++-- package.json | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/package-lock.json b/package-lock.json index 6f21af2d..7039b1f8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "license": "Apache-2.0", "dependencies": { "bonjour-service": "^1.3.0", diff --git a/package.json b/package.json index cc55b397..01d22355 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@sym-bot/sym", - "version": "0.14.2", + "version": "0.14.3", "description": "Infrastructure and protocol for multi-agent collective intelligence", "main": "lib/node.js", "exports": { From f0e6905d9e6d172918a391c186b681811b1f88fc Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 09:57:10 +0100 Subject: [PATCH 03/15] Identity halt: say that the link has to go too A name whose identity file is missing halts and names what to remove to start the name again as a different agent. It named only nodes/by-name/.json, but nodes/ is a link to the old by-id directory and re-indexes the name from it on the next start, so removing the index alone halts again. The message now names both, and a test follows the message to the end (the first removal is shown not to be enough). The test count in the CHANGELOG is now 37 (31 fail on 0.14.2). --- CHANGELOG.md | 6 +++--- lib/config.js | 2 +- tests/identity-migration-by-id.test.js | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8876107d..16ba428b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -59,7 +59,7 @@ or expose a key and the small ones beside them. The wire and the public surface halts (`IdentityHaltError`) and says why. A name that indexes a node whose identity is missing halts too. `create: false` still throws `IdentityAbsentError`. Behaviour change: a node whose `identity.json` was deleted by hand to "start over" now halts instead of silently becoming a new agent; to start the name again as a different agent, - remove the file the message names (`nodes/by-name/.json`). + remove the two paths the message names (`nodes/by-name/.json` and the link `nodes/`). - **Key files are private.** After a migration `identity.json` and `e2e-keypair.json` are 0600 and the directories that hold them 0700; loading an identity narrows an existing key file (and the E2E keypair beside it), so nodes already migrated by 0.14.0 to 0.14.2 are healed on their next start. A new identity's directory is 0700. Best effort on a @@ -84,8 +84,8 @@ or expose a key and the small ones beside them. The wire and the public surface ### Tests -- `tests/identity-migration-by-id.test.js`: 36 tests (35 run on macOS; one real cross-filesystem test runs on Linux). - 30 of them fail on 0.14.2; five pin behaviour that did not change, and one runs only on Linux. The code is mutation-tested: seventeen +- `tests/identity-migration-by-id.test.js`: 37 tests (36 run on macOS; one real cross-filesystem test runs on Linux). + 31 of them fail on 0.14.2; five pin behaviour that did not change, and one runs only on Linux. The code is mutation-tested: seventeen mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. ## 0.14.2 — 2026-10-14 diff --git a/lib/config.js b/lib/config.js index f7a224b6..d3d1beba 100644 --- a/lib/config.js +++ b/lib/config.js @@ -684,7 +684,7 @@ function loadIdentity({ nodeId, name, create = true } = {}) { if (nodeIdForName(name) === nodeId) { throw new IdentityHaltError(name, idPath, `the name "${name}" indexes node ${nodeId}, but its identity.json is not there. If SYM_IDENTITY_DIR changed or its volume is not mounted, ` + - `restore that. If this agent is gone for good, remove ${nameIndexPath(name)} to let the name start again as a different agent.`); + `restore that. If this agent is gone for good, remove ${nameIndexPath(name)} and the link ${path.join(NODES_DIR, name)} to let the name start again as a different agent.`); } } if (!create) throw new IdentityAbsentError(`"${name}"`); diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js index 7438d0cc..d7bcf364 100644 --- a/tests/identity-migration-by-id.test.js +++ b/tests/identity-migration-by-id.test.js @@ -322,6 +322,24 @@ describe('F4: a name that indexes a node whose identity is missing halts', () => assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, dirsBefore, 'nothing was minted'); }); + it('the halt names what to remove, and removing exactly that lets the name start again as a different agent', () => { + const name = uniq('f4reset'); + const first = cfg.loadIdentity({ name }); + fs.rmSync(path.join(cfg.nodeDirById(first.nodeId), 'identity.json')); + let halt = null; + try { cfg.loadIdentity({ name }); } catch (e) { halt = e; } + assert.ok(halt instanceof IdentityHaltError); + const index = path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); + const link = path.join(NODES_DIR, name); + assert.ok(halt.reason.includes(index) && halt.reason.includes(link), 'the message names the index file and the link'); + fs.unlinkSync(index); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError, 'the index alone is not enough: the link re-indexes the name'); + assert.ok(exists(index), '(and it has: the index is back)'); + fs.unlinkSync(index); fs.unlinkSync(link); + const again = cfg.loadIdentity({ name }); + assert.notStrictEqual(again.nodeId, first.nodeId, 'a different agent, as the message says'); + }); + it('create: false is still IdentityAbsentError (EIDENTITYABSENT)', () => { const name = uniq('f4b'); const first = cfg.loadIdentity({ name }); From 6bd36ec426bd435a104a589626a6abc5f5950cd8 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 10:45:43 +0100 Subject: [PATCH 04/15] Identity migration: one migrator at a time (a stalled four-process run lost the key) The merge into an existing by-id directory was safe only if no other process was doing the same. It assumed each step was idempotent, so several processes could interleave them. They cannot: when one finishes it replaces nodes/ with the compat link, and another, still holding the old path, then sees that path lead to the destination, finds the "duplicate" file identical to itself and deletes it - the only copy of identity.json. Plain concurrent runs passed 320 starters. A fuzzer that stalls every file call for a few ms in four real processes lost the identity file in 2 of 300 rounds (everything else filed, identity.json gone). A Linux run under a throttled CPU (node 22, 0.3 cpu) failed the new race test twice in 40 with the same symptom: identity.json missing from by-id after every starter had finished. The design assumption was "a retry makes concurrent migration safe". It does not, and a guard on the alias only narrows the window. The migrator now holds a lock (nodes/.migrating/lock.pid, the identity lock) while it moves something; others wait, read the state again on their turn, and find it done. It is taken only when a 0.13 directory is there to move. The retry loop and its ENOENT tolerance are gone; the alias check stays as a last guard on a key file. With the lock the fuzzer ran 1,200 starters (300 rounds, husk) and 1,600 (400 rounds, none) with every starter loading the original agent and no errors. Also fixed in this commit: the F6 test passed for the wrong reason once the lock existed (its injected write failure hit the lock file's write); it now fails only the identity temp file. A starter that read the index just before another process wrote it, and then migrates nothing, re-reads the index instead of making a new identity; that line had no test and now has one. Tests: 41 (33 fail on 0.14.2); 17 mutants, each caught. --- CHANGELOG.md | 21 ++-- lib/config.js | 75 ++++++++++---- tests/identity-migration-by-id.test.js | 129 +++++++++++++++++++------ 3 files changed, 170 insertions(+), 55 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 16ba428b..653d430d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,12 +47,17 @@ or expose a key and the small ones beside them. The wire and the public surface `identity.json` (or other content that cannot be merged) already filed under it. A directory that is empty, or holds a stale lock, claims nothing: the node is MERGED into it. The same identity on both sides is an interrupted move, and the next start finishes it. A real duplicate is still refused: nothing is moved and nothing is overwritten. -- **The name is indexed before anything moves, and the identity moves last.** A second starter that arrives in the - middle finds the agent; an interrupted move is re-entered on the next start by the identity's absence. A step that - fails because another process just did it is re-run (up to three times) against what that process left, and a path - that has become an alias of its destination through the other process's compat link is never "merged" onto itself. - Measured with real processes: four starters at once, 320 starters in all (with and without a by-id directory already - in place), every one loaded the original agent and the original key was intact every time. The same run on 0.14.2 +- **The name is indexed before anything moves, and the identity moves last.** A crash in the middle leaves the name + pointing at the agent and the identity still where the next start will find it, so an interrupted move is re-entered + by the identity's absence instead of being mistaken for "no such agent". +- **One process migrates at a time.** Moving a directory is many steps, and two processes interleaving them could each + delete what the other had just filed. Found by running four real processes at it with a stall injected into every + file call (about one round in a hundred lost the identity file), after plain concurrent runs had passed. The migrator + now holds a lock (`nodes/.migrating/lock.pid`, the identity lock: reclaimed when its holder is gone) while it moves + something; the others wait (up to 30 s, then fail loudly), read the state again and find it done. It is taken only + when a 0.13 directory is there to move. Under that lock: 1,200 starters in 300 rounds, then 1,600 in 400 rounds + without a husk, every one loaded the original agent, no errors, the key in place every round. A path that has + become an alias of its destination is never deleted as a "duplicate" of it. The same four-process run on 0.14.2 made a new identity in 14 of 100 starters, and destroyed the key in every round that had a by-id directory. - **A start never makes an identity where one exists.** Before it makes one it checks that no 0.13 `identity.json` sits under the name (in `nodes//` or `SYM_IDENTITY_DIR//`); if one does and could not be migrated it @@ -84,8 +89,8 @@ or expose a key and the small ones beside them. The wire and the public surface ### Tests -- `tests/identity-migration-by-id.test.js`: 37 tests (36 run on macOS; one real cross-filesystem test runs on Linux). - 31 of them fail on 0.14.2; five pin behaviour that did not change, and one runs only on Linux. The code is mutation-tested: seventeen +- `tests/identity-migration-by-id.test.js`: 41 tests (40 run on macOS; one real cross-filesystem test runs on Linux). + 33 of them fail on 0.14.2 (34 on Linux); seven pin behaviour that did not change. The code is mutation-tested: seventeen mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. ## 0.14.2 — 2026-10-14 diff --git a/lib/config.js b/lib/config.js index d3d1beba..e311eaed 100644 --- a/lib/config.js +++ b/lib/config.js @@ -418,28 +418,27 @@ function sameFile(a, b) { * Move `src` onto `dst`, MERGING when `dst` already exists (check `mergeConflict` first): what is not * in `dst` moves in, what is already there stays. A directory that merely exists is not a reason to * leave a node where it was. `identity.json` moves LAST, so "the identity is in place" means "everything - * else is too" — an interrupted move is re-entered by the identity's absence. An entry already moved by - * another process (ENOENT) is skipped: the move is safe to run from several processes. + * else is too" — an interrupted move is re-entered by the identity's absence. + * + * One process at a time: a merge is many steps, and two processes interleaving them can each delete what the + * other has just filed (see `acquireMigrationLock`). Callers hold that lock. */ function mergeMove(src, dst) { const s = lstatOrNull(src); if (!s) return; const d = lstatOrNull(dst); - // Already the same file: the other process finished and replaced the old directory with the compat link, so - // this path now leads to `dst`. Moving it onto itself is a no-op at best; deleting it would delete `dst`. + // Never delete a path that leads to the copy it was verified against (through a symlink, say): that would + // delete the only copy. if (d && sameFile(src, dst)) return; - if (!d) { - try { moveTree(src, dst); } catch (e) { if (e.code === 'ENOENT' && !lstatOrNull(src)) return; throw e; } - return; - } + if (!d) { moveTree(src, dst); return; } if (s.isDirectory() && d.isDirectory()) { const names = fs.readdirSync(src).sort((a, b) => (a === 'identity.json') - (b === 'identity.json')); for (const n of names) mergeMove(path.join(src, n), path.join(dst, n)); - try { fs.rmdirSync(src); } catch (e) { if (e.code !== 'ENOENT') throw e; } + fs.rmdirSync(src); return; } // The same file on both sides (or a stale lock): the one in place stands. Checked again here, because - // this deletes: another process may have changed either side since mergeConflict ran. + // this deletes. if (path.basename(src) !== 'lock.pid' && !sameTree(src, dst)) throw new Error(`cannot merge ${src} onto ${dst}: they differ`); fs.rmSync(src, { recursive: true, force: true }); } @@ -533,9 +532,10 @@ function readIdentityFile(idPath, name) { * filed under it — not a directory that merely exists. `nodes/by-id//` is also made by a lock, or * by an earlier interrupted move, and an empty one used to leave the node where it was, unindexed, for * good. Now the node is MERGED into it. The same identity on both sides is an interrupted move, resumed. - * The name is indexed BEFORE anything moves, so a second process starting meanwhile finds the agent - * rather than concluding there is none; the identity moves last, so an interrupted move is re-entered - * by the identity's absence (`loadIdentity` does it). + * The name is indexed BEFORE anything moves, so a start that has no lock to wait on (a crash in the middle, + * a process that read the index too early) finds the agent rather than concluding there is none; the identity + * moves last, so an interrupted move is re-entered by the identity's absence (`loadIdentity` does it). Only one + * process migrates at a time (`acquireMigrationLock`): the others wait, then find it done. * * Key files end at 0600 (their directory at 0700): `rename` keeps the mode 0.13 wrote them with. * A move across filesystems (EXDEV) is copied, verified and only then removed. @@ -545,11 +545,48 @@ function readIdentityFile(idPath, name) { * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'} */ function migrateNodeDir(name) { - // Safe from several processes: a step that fails because another process just did it (an entry gone, a - // directory already replaced by the compat link) is re-run against the state that process left, which - // reads as moved. What persists is thrown, the last one seen. - for (let attempt = 1; ; attempt++) { - try { return migrateNodeDirOnce(name); } catch (e) { if (attempt >= 3) throw e; } + // One migrator per name. Moving a directory is many steps; two processes doing it at once (a 0.14 start + // beside a CLI call, say) interleave them, and each can delete what the other just filed: found by running + // four real processes against it with a stall injected into every file call, which lost the key in about one + // round in a hundred. The lock is taken only when there is a 0.13 directory to move, and the state is read + // again under it: a process that waited finds the move done. + if (!legacyDirsRemain(name)) return migrateNodeDirOnce(name); + const release = acquireMigrationLock(name); + try { return migrateNodeDirOnce(name); } finally { release(); } +} + +/** True while a 0.13 directory for `name` is still in its old place (a symlink there is the move, done). */ +function legacyDirsRemain(name) { + const realDir = (p) => { const st = lstatOrNull(p); return Boolean(st && st.isDirectory()); }; + const idRoot = process.env.SYM_IDENTITY_DIR; + return realDir(path.join(NODES_DIR, name)) || Boolean(idRoot && realDir(path.join(idRoot, name))); +} + +/** How long a start waits for another process's migration of the same name before it gives up, loudly. */ +let migrationLockWaitMs = 30000; + +/** + * Take the exclusive right to migrate 0.13 directories, waiting for another process that holds it. It is the + * identity lock (`acquireIdentityLock`: a lock.pid carrying the holder's start time, reclaimed when its holder is + * gone) on a directory of its own beside the nodes, `nodes/.migrating/`. One lock for every name: a move is quick, + * it happens once per node, and a lock per name would need a directory made and removed around every move. + * @returns {() => void} release + */ +function acquireMigrationLock(name) { + const dir = path.join(NODES_DIR, '.migrating'); + const sleeper = new Int32Array(new SharedArrayBuffer(4)); + const deadline = Date.now() + migrationLockWaitMs; + for (;;) { + try { + return acquireIdentityLock(name, { dir }); + } catch (e) { + if (e.code !== 'EIDENTITYLOCK') throw e; + if (Date.now() >= deadline) { + throw new Error(`[SYM] another process (pid ${e.holderPid ?? 'unknown'}) has been moving 0.13 nodes for ${migrationLockWaitMs / 1000}s and has not finished ` + + `(it holds ${path.join(dir, 'lock.pid')}). Wait for it, or stop it and start again.`); + } + } + Atomics.wait(sleeper, 0, 0, 20); } } @@ -1284,6 +1321,8 @@ module.exports = { _startTimeWarned = false; }, _createStartTimeCache: createStartTimeCache, + /** Test hook: how long (ms) a start waits for another process's migration of the same name. */ + _setMigrationLockWait: (ms) => { migrationLockWaitMs = ms; }, /** Test hook (win32 only): the raw batched PowerShell lookup. */ _windowsStartTimes: windowsStartTimes, readLockFile, diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js index d7bcf364..1c4be069 100644 --- a/tests/identity-migration-by-id.test.js +++ b/tests/identity-migration-by-id.test.js @@ -428,13 +428,18 @@ describe('F6: rewriting identity.json is atomic', () => { const f = path.join(NODES_DIR, name, 'identity.json'); const before = read(f); let injected = 0; - patchFs('writeSync', (orig) => function (fd, ...rest) { - injected++; - throw Object.assign(new Error('ENOSPC: no space left on device (injected)'), { code: 'ENOSPC' }); + const tempFds = new Set(); // the fds of identity.json's temp file: the lock file's writes are left alone + patchFs('openSync', (open) => function (p, ...rest) { + const fd = open.call(fs, p, ...rest); + if (/identity\.json\.\d+\.tmp$/.test(String(p))) tempFds.add(fd); + return fd; + }, () => patchFs('writeSync', (write) => function (fd, ...rest) { + if (tempFds.has(fd)) { injected++; throw Object.assign(new Error('ENOSPC: no space left on device (injected)'), { code: 'ENOSPC' }); } + return write.call(fs, fd, ...rest); }, () => { assert.throws(() => cfg.loadIdentity({ name }), /ENOSPC/); - }); - assert.ok(injected >= 1, 'the failure was reached'); + })); + assert.ok(injected >= 1, 'the failure was reached, in the write of the key file'); assert.strictEqual(read(f), before, 'the key file is byte-identical: it was never truncated'); assert.deepStrictEqual(fs.readdirSync(path.join(NODES_DIR, name)).filter((n) => n.endsWith('.tmp')), []); assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'and the next start succeeds with the same key'); @@ -579,52 +584,118 @@ describe('F11: the window between the move and the index is not a mint race', () }); }); -describe('F11: another process finishing the move underneath this one', () => { - it('a step that fails because the other process just did it is re-run against what it left, and reads as moved', () => { - const name = uniq('f11-other'); +describe('F11: one migrator at a time', () => { + /** A second process that holds the migration lock for `ms`, then moves the node itself and lets go. */ + function holder(name, ms) { + return spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + const release = c.acquireIdentityLock('migration', { dir: ${JSON.stringify(path.join(NODES_DIR, '.migrating'))} }); + console.log('held'); + setTimeout(() => { c.migrateNodeDir(${JSON.stringify(name)}); release(); console.log('released'); process.exit(0); }, ${ms});`], + { stdio: ['ignore', 'pipe', 'inherit'] }); + } + const held = (child) => new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('the holder exited'))); }); + + it('a start waits for the process that is moving the node, then finds it moved', async () => { + const name = uniq('f11-wait'); const id = legacyNode(name); - const legacy = path.join(NODES_DIR, name); - const dest = cfg.nodeDirById(id.nodeId); - let helped = false; - const result = patchFs('renameSync', (orig) => function (a, b) { - if (!helped && a === legacy) { // the other process gets there first: it moves the directory and leaves the link - helped = true; - orig.call(fs, a, b); - fs.symlinkSync(path.join('by-id', id.nodeId), legacy, 'dir'); - throw Object.assign(new Error(`ENOENT: no such file or directory, rename '${a}' -> '${b}'`), { code: 'ENOENT' }); - } - return orig.call(fs, a, b); - }, () => cfg.migrateNodeDir(name)); - assert.ok(helped, 'the other process ran'); - assert.strictEqual(result, 'already'); - assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); + husk(id.nodeId); // a merge, not a single rename + const child = holder(name, 400); + try { + await held(child); + const t0 = Date.now(); + const loaded = cfg.loadIdentity({ name }); // blocks until the holder lets go + const waited = Date.now() - t0; + assert.ok(waited >= 250, `it waited for the holder (${waited} ms)`); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + assert.ok(isLink(path.join(NODES_DIR, name)), 'the holder finished the move'); + assert.ok(!exists(path.join(NODES_DIR, '.migrating', 'lock.pid')), 'and the lock was let go'); + } finally { child.kill(); } + }); + + it('the lock of a process that is gone is reclaimed, not waited for', () => { + const name = uniq('f11-stale'); + const id = legacyNode(name); + put(path.join(NODES_DIR, '.migrating', 'lock.pid'), '2147483646\n', 0o644); // a pid that is not running + const t0 = Date.now(); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.ok(Date.now() - t0 < 2000, 'no wait'); assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); }); - it('a path that now leads through the other process\'s link to the destination is never "merged" onto itself (that would delete it)', () => { + it('a holder that never lets go makes the start fail loudly after the wait, and moves nothing', async () => { + const name = uniq('f11-timeout'); + const id = legacyNode(name); + const child = holder(name, 60000); + try { + await held(child); + cfg._setMigrationLockWait(300); + assert.throws(() => cfg.migrateNodeDir(name), /has been moving 0\.13 nodes/); + assert.throws(() => cfg.loadIdentity({ name }), /has been moving 0\.13 nodes/, 'and a start does not make a new identity meanwhile'); + assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the node is where it was'); + assert.strictEqual(identitiesNamed(name).length, 0); + assert.ok(id.privateKey); + } finally { cfg._setMigrationLockWait(30000); child.kill(); } + }); + + it('nothing to move: no lock is taken and nothing is written', () => { + const name = uniq('f11-nolock'); + const id = cfg.loadIdentity({ name }); // a new agent: indexed, nodes/ is the link + const lockDir = path.join(NODES_DIR, '.migrating'); + fs.rmSync(lockDir, { recursive: true, force: true }); + assert.strictEqual(cfg.migrateNodeDir(name), 'already'); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); + assert.ok(!exists(lockDir), 'the migration lock directory was not even made'); + }); + + it('a path that leads through a link to the destination is never "merged" onto itself (that would delete the only copy)', () => { const name = uniq('f11-alias'); const id = legacyNode(name); put(path.join(NODES_DIR, name, 'aaa.json'), '{"a":1}'); const legacy = path.join(NODES_DIR, name); const dest = husk(id.nodeId); // forces an entry-by-entry merge let helped = false; let listings = 0; - const result = patchFs('readdirSync', (orig) => function (p, ...rest) { + // Something not holding the lock finishes the move while this merge holds a list of what to move (it cannot + // happen among cooperating processes; the check is a last guard on a key file). + patchFs('readdirSync', (orig) => function (p, ...rest) { const names = orig.call(fs, p, ...rest); if (!helped && p === legacy && ++listings === 2) { // the 2nd listing is the merge's own, after the conflict check's - helped = true; // the other process finishes the whole move while this one holds the list + helped = true; for (const n of names) fs.renameSync(path.join(legacy, n), path.join(dest, n)); fs.rmdirSync(legacy); fs.symlinkSync(path.join('by-id', id.nodeId), legacy, 'dir'); } return names; - }, () => cfg.migrateNodeDir(name)); - assert.ok(helped, 'the other process ran'); - assert.strictEqual(result, 'already'); + }, () => { try { cfg.migrateNodeDir(name); } catch { /* the directory became a link under it: an error is fine, a deletion is not */ } }); + assert.ok(helped, 'the other actor ran'); for (const f of ['identity.json', 'aaa.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(dest, f)), `${f} survived`); assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); }); }); +describe('F11: a starter that read the index too early', () => { + it('finds the agent by the index after migrating nothing, instead of making a new one', () => { + // The other process has moved everything and written the index, but not yet made the compat link: there is no + // 0.13 directory, so there is nothing to migrate, and this starter read the index (as missing) just before. + const name = uniq('f11-early'); + const id = legacyNode(name); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + fs.unlinkSync(path.join(NODES_DIR, name)); // the link not yet made + const index = path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); + let fakedMissing = false; + const loaded = patchFs('readFileSync', (orig) => function (p, ...rest) { + if (!fakedMissing && String(p) === index) { fakedMissing = true; throw Object.assign(new Error('ENOENT (before the other process wrote it)'), { code: 'ENOENT' }); } + return orig.call(fs, p, ...rest); + }, () => cfg.loadIdentity({ name })); + assert.ok(fakedMissing, 'the early read happened'); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + }); +}); + describe('F11 with real processes: several starters at once never make a second identity', () => { /** Four processes call loadIdentity({ name }) at the same moment on a 0.13 node, in a state root of their own. */ async function race({ withHusk }) { From 4f1240786a2fedaa138d73325532b74f01e56a07 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 12:00:18 +0100 Subject: [PATCH 05/15] Test: the waiting start is judged by when it returned, not by how long it waited The test asserted the second start waited at least 250 ms of a 400 ms hold. Under a throttled cpu (node 20 in a container at 0.3 cpu) the holder's timer had run for 150-170 ms before the test process was told it held the lock, so 4 of 30 runs measured 225-243 ms and failed although the start had waited. It now has the holder report the time it let go, and asserts the start returned at or after that time (a start that moved the node alongside the holder returns well before it). Mutant "no migration lock" still fails it. --- tests/identity-migration-by-id.test.js | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js index 1c4be069..ba577f51 100644 --- a/tests/identity-migration-by-id.test.js +++ b/tests/identity-migration-by-id.test.js @@ -591,22 +591,28 @@ describe('F11: one migrator at a time', () => { const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); const release = c.acquireIdentityLock('migration', { dir: ${JSON.stringify(path.join(NODES_DIR, '.migrating'))} }); console.log('held'); - setTimeout(() => { c.migrateNodeDir(${JSON.stringify(name)}); release(); console.log('released'); process.exit(0); }, ${ms});`], + setTimeout(() => { c.migrateNodeDir(${JSON.stringify(name)}); release(); console.log('released:' + Date.now()); process.exit(0); }, ${ms});`], { stdio: ['ignore', 'pipe', 'inherit'] }); } const held = (child) => new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('the holder exited'))); }); + /** When the holder let go (its clock, ms), read after the fact. */ + const releasedAt = (child) => new Promise((resolve, reject) => { + let buf = ''; + child.stdout.on('data', (d) => { buf += d; const m = /released:(\d+)/.exec(buf); if (m) resolve(Number(m[1])); }); + child.once('exit', () => { const m = /released:(\d+)/.exec(buf); if (m) resolve(Number(m[1])); else reject(new Error('the holder never released')); }); + }); it('a start waits for the process that is moving the node, then finds it moved', async () => { const name = uniq('f11-wait'); const id = legacyNode(name); husk(id.nodeId); // a merge, not a single rename - const child = holder(name, 400); + const child = holder(name, 1000); try { await held(child); - const t0 = Date.now(); const loaded = cfg.loadIdentity({ name }); // blocks until the holder lets go - const waited = Date.now() - t0; - assert.ok(waited >= 250, `it waited for the holder (${waited} ms)`); + const returnedAt = Date.now(); + const released = await releasedAt(child); + assert.ok(returnedAt >= released, `it returned (${returnedAt}) only after the holder let go (${released}): it waited, it did not move the node alongside it`); assert.strictEqual(loaded.nodeId, id.nodeId); assert.strictEqual(loaded.privateKey, id.privateKey); assert.strictEqual(identitiesNamed(name).length, 1); From 051f44bccb4c1b14a3dbe83d3a2fad566e7f5c03 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 12:15:30 +0100 Subject: [PATCH 06/15] CHANGELOG 0.14.3: the fuzzer's numbers as measured --- CHANGELOG.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 653d430d..390f25c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,11 +52,12 @@ or expose a key and the small ones beside them. The wire and the public surface by the identity's absence instead of being mistaken for "no such agent". - **One process migrates at a time.** Moving a directory is many steps, and two processes interleaving them could each delete what the other had just filed. Found by running four real processes at it with a stall injected into every - file call (about one round in a hundred lost the identity file), after plain concurrent runs had passed. The migrator + file call (about one round in 150 lost the identity file), after plain concurrent runs had passed. The migrator now holds a lock (`nodes/.migrating/lock.pid`, the identity lock: reclaimed when its holder is gone) while it moves something; the others wait (up to 30 s, then fail loudly), read the state again and find it done. It is taken only - when a 0.13 directory is there to move. Under that lock: 1,200 starters in 300 rounds, then 1,600 in 400 rounds - without a husk, every one loaded the original agent, no errors, the key in place every round. A path that has + when a 0.13 directory is there to move. Under that lock, in the same stall fuzzer: 1,200 starters in 300 rounds, 1,600 + in 400 rounds without a by-id directory in place, and 3,600 in 600 rounds with six starters at once: every one loaded + the original agent, no errors, the key in place every round. A path that has become an alias of its destination is never deleted as a "duplicate" of it. The same four-process run on 0.14.2 made a new identity in 14 of 100 starters, and destroyed the key in every round that had a by-id directory. - **A start never makes an identity where one exists.** Before it makes one it checks that no 0.13 `identity.json` From 632b95d19e8019b2f0374c0e9105580f2adf6a29 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 12:28:56 +0100 Subject: [PATCH 07/15] A start pinned to a nodeId migrates the 0.13 directory too loadIdentity ran the migration only for a start by name alone (`!nodeId && name`). mesh-channel starts every node as { name, nodeId: , create: false }, so a pinned node whose files were still the 0.13 nodes// could not start (EIDENTITYABSENT) until something started it by name; with the default create: true, 0.14.2 made a new identity and lost the key, and the first 0.14.3 draft halted. Reproduced in a sandbox state dir on fe285a3 and on the branch before this change (plain, empty by-id/, create true/false). When the pin's identity is not in place and a 0.13 directory under the name is still there, loadIdentity now runs the same locked, idempotent migrateNodeDir(name) and goes on to load the pinned agent. It is one path: the unpinned and pinned starts share the block; migrateNodeDir takes the pin as an optional second argument. It is not run when there is no 0.13 directory to move, so a pinned start writes nothing it did not before. Decision, a pin that is a DIFFERENT nodeId than the directory's: that directory is another agent's, so it is left where it is, unindexed (migrateNodeDir answers 'skipped-duplicate' before it moves or indexes anything). The pinned identity stays absent and the start fails as before: IdentityAbsentError with create: false, and with create: true the halt that names the stranded 0.13 identity and the pin. Nothing is lost and nothing is made. That agent moves when it is started by name or by its own pin. Tests (tests/identity-migration-by-id.test.js, 51 now): plain and empty/locked by-id/ with create false; create true loads the original and does not mint; SYM_IDENTITY_DIR; a pin that is another agent's; a link to another agent does not get indexed by a pinned start; the name indexed to another agent; a live 0.13 holder and the tombstone; four real processes with injected stalls, with and without a by-id dir already in place. Eight of them fail on 0.14.3 as pushed (051f44b) and on fe285a3; two guard against over-migrating. Four new mutants (no pinned migrate, no pin guard, no legacy-dir gate, adopting the index) are each caught. --- CHANGELOG.md | 30 +++++- lib/config.js | 40 +++++--- tests/identity-migration-by-id.test.js | 132 ++++++++++++++++++++++++- 3 files changed, 180 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 390f25c3..1f725f6a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,10 @@ or expose a key and the small ones beside them. The wire and the public surface - **A name whose identity file was missing made a replacement agent.** If the name index named a node and its `identity.json` was not there (the `SYM_IDENTITY_DIR` volume not mounted at boot, the variable set by one embedder and not another), a start by name made a new nodeId and key and re-pointed the name to it. +- **A start pinned by nodeId never migrated.** `loadIdentity` ran the migration only for a start by name alone. mesh-channel + starts every node as `{ name, nodeId, create: false }`, so a node it pinned to the nodeId in its `node.json` and whose + files were still the 0.13 `nodes//` could not start at all (`EIDENTITYABSENT`) until something started it by name + once; with `create: true` (the default) 0.14.2 made a new identity and lost the key. 0.14.3 loads the agent. - Smaller, same family: a stray symlink at `nodes/` was reported as already indexed when nothing was indexed; switching `SYM_IDENTITY_DIR` on for a 0.13 node whose identity was in its node directory reported "not a node" and went on to make a new identity (the old key survived, orphaned in the new node's directory); an identity with a public key @@ -40,6 +44,9 @@ or expose a key and the small ones beside them. The wire and the public surface - Anyone who switched `SYM_IDENTITY_DIR` on for an existing 0.13 node: the old key is probably still on disk, in `nodes/by-id//identity.json` (the node that was made in its place). - Every node migrated from 0.13 holds a private key readable by other local users until it is next loaded with 0.14.3. +- A node started by its pinned nodeId (mesh-channel's start, `create: false`) on the first start after upgrading from 0.13: + no key was lost, but it could not start until something started it by name. A pinned start with `create: true` could lose it + as above. ### Changed @@ -47,6 +54,14 @@ or expose a key and the small ones beside them. The wire and the public surface `identity.json` (or other content that cannot be merged) already filed under it. A directory that is empty, or holds a stale lock, claims nothing: the node is MERGED into it. The same identity on both sides is an interrupted move, and the next start finishes it. A real duplicate is still refused: nothing is moved and nothing is overwritten. +- **A start pinned to a nodeId migrates too.** `loadIdentity({ nodeId, name })` whose identity is not in place and for + which a 0.13 directory under `name` is still there runs the same locked, idempotent migration as a start by name, then + loads the pinned agent. It is one code path, not a copy, and it is not run when there is no 0.13 directory to move. + **A pin that is a different nodeId than the directory's does not migrate it:** the directory belongs to another agent, + so it is left exactly where it is, unindexed (nothing moved, nothing lost), the pinned identity stays absent, and the + start fails as it would have: `IdentityAbsentError` with `create: false`, and with `create: true` the halt that says a + 0.13 identity sits under the name (naming the pin). That agent moves when it is started by name or by its own pin. A + 0.13 process holding the pinned agent's directory still lets it load in place (tombstone checked), as for a start by name. - **The name is indexed before anything moves, and the identity moves last.** A crash in the middle leaves the name pointing at the agent and the identity still where the next start will find it, so an interrupted move is re-entered by the identity's absence instead of being mistaken for "no such agent". @@ -85,14 +100,19 @@ or expose a key and the small ones beside them. The wire and the public surface should report them is open. - `loadIdentity({ nodeId, name })` still re-points a name that already indexes a different node. - `SYM_IDENTITY_DIR` switched on for an existing node does not move its key for you (it halts and says how). -- `loadIdentity({ nodeId, name, create: true })` with a `nodeId` the name index does not hold and no identity file still - makes a new identity under a new nodeId (it does not use the one given). +- `loadIdentity({ nodeId, name, create: true })` with a `nodeId` the name index does not hold, no identity file, and no 0.13 + directory under the name still makes a new identity under a new nodeId (it does not use the one given). mesh-channel + passes `create: false`, so a pin is never ignored there. +- A pinned start leaves another agent's 0.13 directory (same name, different nodeId) unmigrated, as above; with + `SYM_IDENTITY_DIR` set and that identity in the node directory it halts with the instructions given for that case. ### Tests -- `tests/identity-migration-by-id.test.js`: 41 tests (40 run on macOS; one real cross-filesystem test runs on Linux). - 33 of them fail on 0.14.2 (34 on Linux); seven pin behaviour that did not change. The code is mutation-tested: seventeen - mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. +- `tests/identity-migration-by-id.test.js`: 51 tests (50 run on macOS; one real cross-filesystem test runs on Linux). + 41 of them fail on 0.14.2 (42 on Linux); nine pin behaviour that did not change (two of them keep a pinned start from + migrating what is not its own). Of the 51, ten are for pinned starts, among them four real processes starting one 0.13 + agent at once under injected stalls, with and without a by-id directory already in place. The code is mutation-tested: + twenty-one mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. ## 0.14.2 — 2026-10-14 diff --git a/lib/config.js b/lib/config.js index e311eaed..161cd102 100644 --- a/lib/config.js +++ b/lib/config.js @@ -542,17 +542,24 @@ function readIdentityFile(idPath, name) { * * A 0.13 identity in `nodes//` while SYM_IDENTITY_DIR is set (the volume switched on after the * agent was made) HALTS: it is neither where the identity dir looks nor safe to leave for a mint. + * + * `onlyNodeId`: a start pinned to a nodeId (`loadIdentity({ nodeId, name })`) migrates the directory under + * `name` only if its identity IS that nodeId. Another agent's directory is left unmoved and unindexed + * ('skipped-duplicate'): the pin asked for one agent, and moving or indexing a different one is not its + * business. + * @param {string} name + * @param {string} [onlyNodeId] * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'} */ -function migrateNodeDir(name) { +function migrateNodeDir(name, onlyNodeId) { // One migrator per name. Moving a directory is many steps; two processes doing it at once (a 0.14 start // beside a CLI call, say) interleave them, and each can delete what the other just filed: found by running // four real processes against it with a stall injected into every file call, which lost the key in about one // round in a hundred. The lock is taken only when there is a 0.13 directory to move, and the state is read // again under it: a process that waited finds the move done. - if (!legacyDirsRemain(name)) return migrateNodeDirOnce(name); + if (!legacyDirsRemain(name)) return migrateNodeDirOnce(name, onlyNodeId); const release = acquireMigrationLock(name); - try { return migrateNodeDirOnce(name); } finally { release(); } + try { return migrateNodeDirOnce(name, onlyNodeId); } finally { release(); } } /** True while a 0.13 directory for `name` is still in its old place (a symlink there is the move, done). */ @@ -590,7 +597,7 @@ function acquireMigrationLock(name) { } } -function migrateNodeDirOnce(name) { +function migrateNodeDirOnce(name, onlyNodeId) { const legacy = path.join(NODES_DIR, name); const idRoot = process.env.SYM_IDENTITY_DIR; const legacyIdDir = idRoot ? path.join(idRoot, name) : legacy; @@ -618,6 +625,9 @@ function migrateNodeDirOnce(name) { const identity = readIdentityFile(idFile, name); // halts on an identity it cannot establish const nodeId = identity.nodeId; if (!isNodeIdText(nodeId)) throw new IdentityHaltError(name, idFile, `the nodeId ${JSON.stringify(String(nodeId).slice(0, 40))} cannot name a directory under nodes/by-id`); + // A start pinned to one nodeId moves that agent and no other: the identity under this name is another + // agent's, and stays exactly where it is (nothing is indexed or moved, nothing lost). + if (onlyNodeId && nodeId !== onlyNodeId) return 'skipped-duplicate'; if (st) { const lock = readLockFile(path.join(legacy, 'lock.pid')); if (lock && lock.pid !== process.pid && lockIsHeldByLiveProcess(lock)) return 'skipped-live'; @@ -687,14 +697,19 @@ function loadIdentity({ nodeId, name, create = true } = {}) { if (!nodeId && !name) throw new Error('loadIdentity needs a nodeId or a name'); if (nodeId && !isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); let migrated = null; - if (!nodeId && name) { - nodeId = nodeIdForName(name); - // Not indexed, or indexed but the identity is not in place: the 0.13 layout may still hold it (a move - // another process is making, or one that was interrupted). Migrating is idempotent. - if (!nodeId || !fs.existsSync(path.join(identityDirById(nodeId), 'identity.json'))) { - migrated = migrateNodeDir(name); + if (name) { + const pinned = nodeId; + if (!pinned) nodeId = nodeIdForName(name); + // The identity is not in place: the 0.13 layout may still hold it. Unpinned, that is when the name is not + // indexed, or is but the identity is missing (a move another process is making, or one that was + // interrupted). Pinned (mesh-channel starts every node as `{ name, nodeId, create: false }`), when a 0.13 + // directory for the name is still there. Migrating is idempotent, and one path serves both. + const inPlace = Boolean(nodeId) && fs.existsSync(path.join(identityDirById(nodeId), 'identity.json')); + if (!inPlace && (!pinned || legacyDirsRemain(name))) { + migrated = migrateNodeDir(name, pinned); if (migrated === 'skipped-live') { - // A 0.13 process of this name holds its old directory: load it in place, unmoved. + // A 0.13 process of this name holds its old directory: load it in place, unmoved. (Pinned: it is the + // pinned agent's, or the migration would have said 'skipped-duplicate'.) const live = readIdentityFile(legacyIdentityPath(name), name); const liveTomb = readTombstone(live.nodeId); if (liveTomb) throw new IdentityTombstonedError(live.nodeId, liveTomb); @@ -732,7 +747,8 @@ function loadIdentity({ nodeId, name, create = true } = {}) { const stranded = legacyIdentityPath(name); if (stranded) { throw new IdentityHaltError(name, stranded, - `a 0.13 identity exists under this name and was not migrated (migration said: ${migrated || 'nothing'}). ` + + `a 0.13 identity exists under this name and was not migrated (migration said: ${migrated || 'nothing'}` + + `${nodeId && migrated === 'skipped-duplicate' ? `; this start is pinned to ${nodeId}, and that identity is another agent's` : ''}). ` + 'Making a new identity here would orphan or overwrite its key.'); } const kp = generateSigningKeyPair(); diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js index ba577f51..16212c6c 100644 --- a/tests/identity-migration-by-id.test.js +++ b/tests/identity-migration-by-id.test.js @@ -704,7 +704,7 @@ describe('F11: a starter that read the index too early', () => { describe('F11 with real processes: several starters at once never make a second identity', () => { /** Four processes call loadIdentity({ name }) at the same moment on a 0.13 node, in a state root of their own. */ - async function race({ withHusk }) { + async function race({ withHusk, pinned = false, stall = false }) { const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-race-')); try { const state = path.join(root, 'state'); @@ -715,10 +715,20 @@ describe('F11 with real processes: several starters at once never make a second for (let i = 0; i < 30; i++) fs.writeFileSync(path.join(nodes, name, 'cmbs', `c${i}.json`), '{}'); fs.writeFileSync(path.join(nodes, name, 'identity.json'), JSON.stringify(id, null, 2)); if (withHusk) { fs.mkdirSync(path.join(nodes, 'by-id', id.nodeId), { recursive: true }); fs.writeFileSync(path.join(nodes, 'by-id', id.nodeId, 'lock.pid'), '999999\n'); } + // `stall`: every file call may pause a few ms first, which spreads the four processes' steps through each other. + const stalls = stall ? ` + const sab = new Int32Array(new SharedArrayBuffer(4)); + for (const k of ['renameSync', 'lstatSync', 'statSync', 'readdirSync', 'mkdirSync', 'existsSync', 'readFileSync', 'writeFileSync', 'rmdirSync', 'rmSync', + 'symlinkSync', 'unlinkSync', 'realpathSync', 'readlinkSync', 'chmodSync', 'openSync', 'copyFileSync']) { + const orig = fs[k]; fs[k] = function (...a) { if (Math.random() < 0.35) Atomics.wait(sab, 0, 0, Math.random() * 4); return orig.apply(fs, a); }; + }` : ''; + // `pinned`: the start mesh-channel makes, { name, nodeId, create: false }. + const options = pinned ? `{ name: ${JSON.stringify(name)}, nodeId: ${JSON.stringify(id.nodeId)}, create: false }` : `{ name: ${JSON.stringify(name)} }`; const code = ` + const fs = require('fs');${stalls} const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); process.stdin.once('data', () => { - try { const i = c.loadIdentity({ name: ${JSON.stringify(name)} }); console.log(JSON.stringify({ id: i.nodeId, key: i.privateKey })); } + try { const i = c.loadIdentity(${options}); console.log(JSON.stringify({ id: i.nodeId, key: i.privateKey })); } catch (e) { console.log(JSON.stringify({ error: e.name + ': ' + String(e.message).split('\\n')[0].slice(0, 200) })); } process.exit(0); }); @@ -733,11 +743,11 @@ describe('F11 with real processes: several starters at once never make a second } finally { fs.rmSync(root, { recursive: true, force: true }); } } - for (const withHusk of [false, true]) { - it(`${withHusk ? 'with a by-id dir that already exists' : 'plain'}: every starter that loads gets the original agent, and the key survives`, async () => { + for (const [withHusk, pinned, stall] of [[false, false, false], [true, false, false], [false, true, true], [true, true, true]]) { + it(`${pinned ? 'pinned starts (nodeId + name, create: false), stalled' : 'starts by name'}, ${withHusk ? 'with a by-id dir that already exists' : 'plain'}: every starter that loads gets the original agent, and the key survives`, async () => { let loaded = 0; for (let round = 0; round < 5; round++) { - const { id, outcomes, kept } = await race({ withHusk }); + const { id, outcomes, kept } = await race({ withHusk, pinned, stall }); assert.strictEqual(kept.privateKey, id.privateKey, 'the original private key is still the one on disk'); for (const o of outcomes) { if (o.error) continue; // a starter may be told to start again; it must not become somebody else @@ -770,6 +780,118 @@ describe('F13: the live-lock fallback checks the tombstone', () => { }); }); +describe('a start pinned to a nodeId migrates too (mesh-channel starts every node as { name, nodeId, create: false })', () => { + it('create: false, a plain 0.13 dir: the pinned agent loads, moved, with its key files private', () => { + const name = uniq('pin-plain'); + const id = legacyNode(name, { e2e: true }); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + const dir = cfg.nodeDirById(id.nodeId); + assert.ok(exists(path.join(dir, 'cmbs', 'x.json')), 'the store moved with it'); + assert.ok(isLink(path.join(NODES_DIR, name)), 'nodes/ is the compat link'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + if (posix) { + assert.strictEqual(modeOf(path.join(dir, 'identity.json')), 0o600); + assert.strictEqual(modeOf(path.join(dir, 'e2e-keypair.json')), 0o600); + assert.strictEqual(modeOf(dir), 0o700); + } + assert.strictEqual(cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }).privateKey, id.privateKey, 'and the second start is the same'); + assert.strictEqual(identitiesNamed(name).length, 1); + }); + + it('create: false, a by-id/ dir that already exists (empty, or holding only a lock): the pinned agent loads', () => { + for (const lock of [false, true]) { + const name = uniq(`pin-husk${lock ? '-lock' : ''}`); + const id = legacyNode(name); + husk(id.nodeId, { lock }); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1); + } + }); + + it('the default create: true loads the original and does not mint (0.14.2 made a new nodeId and lost the key)', () => { + for (const withHusk of [false, true]) { + const name = uniq(`pin-create${withHusk ? '-husk' : ''}`); + const id = legacyNode(name); + if (withHusk) husk(id.nodeId); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId }); + assert.strictEqual(loaded.nodeId, id.nodeId); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.strictEqual(identitiesNamed(name).length, 1, 'no second identity answers to the name'); + } + }); + + it('with SYM_IDENTITY_DIR: the identity under SYM_IDENTITY_DIR/ moves and loads for the pin', () => { + withIdentityDir((idRoot) => { + const name = uniq('pin-idroot'); + const id = makeIdentity(name); + put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + const loaded = cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(idRoot, 'by-id', id.nodeId, 'identity.json'))); + if (posix) assert.strictEqual(modeOf(path.join(idRoot, 'by-id', id.nodeId, 'identity.json')), 0o600); + }); + }); + + it('a pin that is a DIFFERENT nodeId than the 0.13 directory\'s: the pin stays absent, the directory is left exactly as it is, nothing is made', () => { + const name = uniq('pin-other'); + const other = legacyNode(name); // the agent whose directory this is + const pin = cfg.uuidv7(); // the start asks for another + const identityBefore = read(path.join(NODES_DIR, name, 'identity.json')); + const byIdBefore = fs.readdirSync(path.join(NODES_DIR, 'by-id')).length; + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin }), (e) => e instanceof IdentityHaltError && /pinned to/.test(e.reason), 'create: true halts; it does not mint under the name'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), identityBefore, 'the other agent\'s key is byte-unchanged'); + assert.ok(!isLink(path.join(NODES_DIR, name)) && exists(path.join(NODES_DIR, name, 'cmbs', 'x.json')), 'and its directory is where it was, unmoved'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the name was not indexed to either'); + assert.ok(!exists(cfg.nodeDirById(other.nodeId)) && !exists(cfg.nodeDirById(pin)), 'no directory was made for either'); + assert.strictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).length, byIdBefore); + assert.strictEqual(identitiesNamed(name).length, 0); + // the unpinned start still moves it, as before + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, other.nodeId); + }); + + it('nothing 0.13 to move (only a link to another agent): a pinned start does not index the name from it', () => { + const name = uniq('pin-link'); + const owner = cfg.loadIdentity({ name: uniq('pin-link-owner') }); // some other agent, filed by id + fs.symlinkSync(path.join('by-id', owner.nodeId), path.join(NODES_DIR, name), 'dir'); // a link under this name, name unindexed + assert.throws(() => cfg.loadIdentity({ name, nodeId: cfg.uuidv7(), create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the pinned start wrote no index entry for the name'); + }); + + it('the name is indexed to another agent and the pin\'s 0.13 directory sits under it: the pin stays absent, and the start does not answer with the indexed agent', () => { + const name = uniq('pin-held'); + const indexed = cfg.loadIdentity({ name }); // N -> indexed + fs.unlinkSync(path.join(NODES_DIR, name)); // the link out of the way: a real directory for the pin + const pin = legacyNode(name); + const before = read(path.join(NODES_DIR, name, 'identity.json')); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin.nodeId, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), before, 'the pin\'s key is where it was'); + assert.strictEqual(cfg.nodeIdForName(name), indexed.nodeId, 'and the index still says what it said'); + }); + + it('a 0.13 process holds the pinned agent\'s directory: it loads in place, and a tombstone still refuses it', async () => { + const name = uniq('pin-live'); + const id = legacyNode(name); + const dir = path.join(NODES_DIR, name); + const child = spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); + c.acquireIdentityLock(${JSON.stringify(name)}, { dir: ${JSON.stringify(dir)} }); + console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + try { + await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('lock holder exited'))); }); + assert.strictEqual(cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }).privateKey, id.privateKey); + assert.ok(exists(path.join(dir, 'cmbs', 'x.json')) && !isLink(dir), 'unmoved: its process is running there'); + assert.throws(() => cfg.loadIdentity({ name, nodeId: cfg.uuidv7(), create: false }), (e) => e.code === 'EIDENTITYABSENT', 'a different pin does not get it'); + put(path.join(cfg.nodeDirById(id.nodeId), cfg.TOMBSTONE), JSON.stringify({ movedAt: Date.now() })); + assert.throws(() => cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }), (e) => e instanceof IdentityTombstonedError); + } finally { child.kill(); } + }); +}); + describe('T9: the checks measure something', () => { it('assertMoved fails a run that moved nothing, and a populated run passes it', () => { const name = uniq('t9'); From eb76263f1560f9f2873e62cb89fe42631a2a6e02 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 16:23:41 +0100 Subject: [PATCH 08/15] Identity state from committed records: staged copy + one-rename commit, exclusive name claim The design assumption that failed twice: the state of an identity was inferred from whatever was at a path (a partial destination, an unreadable index, a node dir without identity.json, a stale lock's content). Replaced: - 0.13 -> by-id migration COPIES: staged beside its destination (nodes/by-id/..partial-*), identity.json last, synced, verified byte for byte, installed by ONE rename. The loser of two commits finds the same identity and discards its staging. The 0.13 directory is then renamed aside to .legacy-0.13/ (never deleted) and the compat link takes its place. A by-id dir without the commit record (.sym-commit) is a husk: emptied with rmdir, or its files merged into the staging and the husk kept at ..husk. mergeMove, moveTree and the nodes/.migrating lock (Atomics.wait up to 30 s) are gone. - A name is claimed by an exclusive create (link) of its index entry after the commit; a losing first start discards its minted identity and loads the winner. The index reads as absent only on ENOENT; anything else halts. Index writes are fsynced. - A mint happens only when there is no trace of the name; a trace that provably is the name's identity is adopted, any other halts with what it is. - A live 0.13 process in the 0.13 directory refuses the start (EIDENTITYLOCK). - A stale lock is removed only by the holder of a claim named after that lock's file identity and content (compare-and-delete); locks are created whole via link. - relocation: intent, bundle, tombstone beside the key, scrub; resumable and abortable (abortExport); imports staged with 0700 dirs, 0600 keys, the key only in the identity dir. - validateName refuses names Windows cannot hold; compat links compared as resolved paths. tests/identity-commit.test.js: 25 tests, all failing on 632b95d. --- lib/config.js | 1748 +++++++++++++++++------- lib/relocation.js | 198 ++- tests/identity-commit.test.js | 565 ++++++++ tests/identity-migration-by-id.test.js | 278 ++-- 4 files changed, 2160 insertions(+), 629 deletions(-) create mode 100644 tests/identity-commit.test.js diff --git a/lib/config.js b/lib/config.js index 161cd102..b338764d 100644 --- a/lib/config.js +++ b/lib/config.js @@ -53,16 +53,24 @@ const IDENTITY_RULING = * an operator who is told plainly can restore it. Regenerating is the unrecoverable one. */ class IdentityHaltError extends Error { - constructor(name, idPath, reason) { + /** + * @param {string} name + * @param {string} idPath - the file the halt is about + * @param {string} reason + * @param {string} [advice] - what to do, when the general advice (restore the keypair) is not it: the key + * may be on this host, and then the advice says where, never "start over" + */ + constructor(name, idPath, reason, advice) { super( `[SYM] REFUSING TO START — cannot establish the identity of agent "${name}".\n\n` + ` ${reason}\n` + ` identity: ${idPath}\n\n` + `${IDENTITY_RULING}\n\n` + - ` Restore the keypair from your durable copy and start again. If this agent's key is ` + - `genuinely gone, it cannot return as "${name}" — peers hold its public key and will read ` + - `anything a new key signs as forged. Set SYM_IDENTITY_DIR to a durable volume so a ` + - `rebuilt machine restores the key instead of replacing it.` + (advice ? ` ${advice}` : + ` Restore the keypair from your durable copy and start again. If this agent's key is ` + + `genuinely gone, it cannot return as "${name}" — peers hold its public key and will read ` + + `anything a new key signs as forged. Set SYM_IDENTITY_DIR to a durable volume so a ` + + `rebuilt machine restores the key instead of replacing it.`) ); this.name = 'IdentityHaltError'; this.agent = name; @@ -135,7 +143,7 @@ function normalizeMdnsHostname(h) { /** * Get the data directory for a named node. A name is an index (design D9): when the name index holds * it, this is the node's by-id directory (`nodes/by-id/`); otherwise the 0.13 path - * `nodes/`, which `loadIdentity` moves to the by-id layout when it first loads it. + * `nodes/`, which `loadIdentity` copies to the by-id layout when it first loads it. * @param {string} name — node name * @returns {string} path to the node's directory */ @@ -173,28 +181,46 @@ function uuidv7() { } /** - * Validate a node name per MMP Section 3.1.2. - * Must be valid UTF-8, 1-64 bytes, printable characters only. - * @param {string} name - * @throws {Error} if name is invalid + * Why `name` cannot be a node name at all (MMP §3.1.2, and one path component), or null. + * @private */ -function validateName(name) { - if (!name || typeof name !== 'string') { - throw new Error('Node name must be a non-empty string'); - } +function basicNameProblem(name) { + if (!name || typeof name !== 'string') return 'Node name must be a non-empty string'; const byteLength = Buffer.byteLength(name, 'utf8'); - if (byteLength < 1 || byteLength > 64) { - throw new Error(`Node name must be 1-64 bytes (got ${byteLength})`); - } + if (byteLength < 1 || byteLength > 64) return `Node name must be 1-64 bytes (got ${byteLength})`; // Reject control characters (U+0000-U+001F, U+007F-U+009F) - if (/[\x00-\x1f\x7f-\x9f]/.test(name)) { - throw new Error('Node name must not contain control characters'); - } + if (/[\x00-\x1f\x7f-\x9f]/.test(name)) return 'Node name must not contain control characters'; // A name is one path component (security review): it names `nodes/`, a compat link and an // index file, so a separator or a dot-name would reach outside the state root. - if (/[\/\\]/.test(name) || name === '.' || name === '..') { - throw new Error('Node name must not contain a path separator or be "." or ".."'); - } + if (/[\/\\]/.test(name) || name === '.' || name === '..') return 'Node name must not contain a path separator or be "." or ".."'; + return null; +} + +/** + * Why Windows could not hold `name` as a directory name, or null. The rule mesh-channel's + * nodeNameProblem applies: the characters Windows refuses in a file name, the device names it reserves + * whatever the extension (CON, NUL, COM1, ...), and a trailing dot or space, which Windows drops (so two + * names would share one directory). A name is `nodes/` on every platform, and a node moved to a + * Windows host must still have one. + * @private + */ +function windowsNameProblem(name) { + if (/[:*?"<>|]/.test(name)) return 'Node name must not contain any of : * ? " < > | (not valid in a file name on every platform)'; + if (/^(con|prn|aux|nul|com[0-9]|lpt[0-9])(\..*)?$/i.test(name)) return 'Node name must not be a name Windows reserves (CON, PRN, AUX, NUL, COM0-9, LPT0-9, with or without an extension)'; + if (/[. ]$/.test(name)) return 'Node name must not end with a dot or a space (Windows drops it, so two names would share one directory)'; + return null; +} + +/** + * Validate a node name per MMP Section 3.1.2. + * Must be valid UTF-8, 1-64 bytes, printable characters only, one path component, and a name every + * platform can hold as a directory name. + * @param {string} name + * @throws {Error} if name is invalid + */ +function validateName(name) { + const problem = basicNameProblem(name) || windowsNameProblem(name); + if (problem) throw new Error(problem); } /** @@ -217,11 +243,14 @@ function generateSigningKeyPair() { /** * Raised when an identity that must already exist does not (`loadIdentity({ create: false })`): a - * host restoring a known agent can never mint a replacement silently (design D9). + * host restoring a known agent can never mint a replacement silently (design D9). It says what IS on + * this host under the name when something is (a node's files without an identity, another agent's + * identity), so "no identity" is never said of a host that holds one: that case is an IdentityHaltError. */ class IdentityAbsentError extends Error { - constructor(what) { + constructor(what, seen = []) { super(`[SYM] No identity for ${what} on this host, and none was minted (create: false). ` + + (seen.length ? `What is here instead: ${seen.join('; ')}. ` : '') + `Restore it (sym node import) or start it where it lives; minting a new one would be a different agent.`); this.name = 'IdentityAbsentError'; this.code = 'EIDENTITYABSENT'; @@ -243,6 +272,23 @@ class IdentityTombstonedError extends Error { } } +/** + * Raised for an identity whose export began and did not finish (relocation.js wrote EXPORTING.json beside + * the key, and no tombstone yet): a bundle may exist, so the source does not start it until the export is + * finished (run it again) or abandoned (`abortExport`, which removes the bundle it wrote). + */ +class IdentityExportPendingError extends Error { + constructor(nodeId, intent) { + super(`[SYM] REFUSING TO START ${nodeId}: an export of this identity` + + `${intent && intent.out ? ` to ${intent.out}` : ''}${intent && intent.startedAt ? `, begun ${new Date(intent.startedAt).toISOString()},` : ''} did not finish. ` + + 'A bundle of it may exist; started here as well, there could be two live copies. Finish the export (run `sym node export` ' + + 'again) or abandon it (relocation.abortExport({ nodeId }), which removes the bundle it wrote; delete any copy made of that file).'); + this.name = 'IdentityExportPendingError'; + this.code = 'EIDENTITYEXPORTING'; + this.intent = intent || null; + } +} + /** Where a node lives, by its nodeId (design D9): /nodes/by-id/. */ function nodeDirById(nodeId) { if (!isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); @@ -252,66 +298,318 @@ function nodeDirById(nodeId) { /** Where a node's identity.json lives by nodeId: SYM_IDENTITY_DIR/by-id/ when set, else the node dir. */ function identityDirById(nodeId) { const root = process.env.SYM_IDENTITY_DIR; + if (root && !isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); return root ? path.join(root, 'by-id', nodeId) : nodeDirById(nodeId); } const isNodeIdText = (id) => typeof id === 'string' && /^[0-9A-Za-z][0-9A-Za-z_.-]{0,127}$/.test(id) && id !== '.' && id !== '..'; const TOMBSTONE = 'TOMBSTONE.json'; +/** Beside the key while an export is under way (relocation.js): the bundle may exist, the tombstone does not yet. */ +const EXPORTING = 'EXPORTING.json'; +/** + * The commit record of a directory this release installed: written into the staging directory before the + * one rename that puts it in place, so a directory that carries it is complete. A by-id directory without + * it was made by something else (a lock, an earlier release's interrupted move) and is never taken for + * "the identity is in place" by a migration. + */ +const COMMIT_MARK = '.sym-commit'; +/** Where a migrated 0.13 directory is set aside, under the root it was in. Nothing deletes it automatically. */ +const LEGACY_ASIDE = '.legacy-0.13'; const nameIndexPath = (name) => path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); +const lstatOrNull = (p) => { try { return fs.lstatSync(p); } catch { return null; } }; +const statOrNull = (p) => { try { return fs.statSync(p); } catch { return null; } }; +const randomTag = () => crypto.randomBytes(4).toString('hex'); +/** The errors a rename gives when its target is taken (by a directory that is not empty, on Windows by any). */ +const RENAME_TAKEN = new Set(['EEXIST', 'ENOTEMPTY', 'EPERM', 'EISDIR', 'ENOTDIR']); +/** A short synchronous pause, for the few places that let another start finish a step of a few system calls. */ +const pause = (ms) => Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); + +// ── Durable files ──────────────────────────────────────────────────────── + +/** fsync a directory, so a rename or create in it survives a crash. Best effort where unsupported (Windows). */ +function fsyncDir(dir) { + let fd; + try { fd = fs.openSync(dir, 'r'); fs.fsyncSync(fd); } catch { /* not supported here */ } finally { if (fd !== undefined) try { fs.closeSync(fd); } catch { /* */ } } +} + +/** fsync a file by path. Best effort: a file this process may not open for writing is still read back by the verify. */ +function fsyncFile(file) { + let fd; + try { fd = fs.openSync(file, 'r+'); fs.fsyncSync(fd); } catch { /* not ours to open for writing */ } finally { if (fd !== undefined) try { fs.closeSync(fd); } catch { /* */ } } +} + +/** Write `text` to a new file `file` (it must not exist), synced. A failure removes what it made (it made it: O_EXCL). */ +function writeNewFileSynced(file, text, mode) { + const fd = fs.openSync(file, 'wx', mode); + try { + try { fs.writeSync(fd, text); fs.fsyncSync(fd); } finally { fs.closeSync(fd); } + fs.chmodSync(file, mode); + } catch (e) { + try { fs.unlinkSync(file); } catch { /* */ } + throw e; + } +} -/** The nodeId the name index holds for `name`, or null. */ +/** A temp file beside `file`, written and synced: the first step of a durable replace or an exclusive create. */ +function writeTempBeside(file, text, mode) { + const tmp = path.join(path.dirname(file), `.${path.basename(file)}.${process.pid}-${randomTag()}.tmp`); + writeNewFileSynced(tmp, text, mode); + return tmp; +} + +/** Replace `file` with `text` in one step: a synced temp file renamed over it, and the directory synced. */ +function writeFileDurable(file, text, mode = 0o600) { + const tmp = writeTempBeside(file, text, mode); + try { fs.renameSync(tmp, file); } catch (e) { try { fs.unlinkSync(tmp); } catch { /* */ } throw e; } + fsyncDir(path.dirname(file)); +} + +/** + * Create `file` holding `text`, only if no file is there: true when this call made it, false when one already + * was. The content is written and synced under a temp name and then hard-linked into place, so the file is never + * seen empty or half written; `link` fails with EEXIST when the name is taken, and of several processes exactly + * one wins. Where hard links are not supported, an O_EXCL create written in place. + */ +function createFileExclusive(file, text, mode = 0o644) { + const tmp = writeTempBeside(file, text, mode); + try { + try { + fs.linkSync(tmp, file); + return true; + } catch (e) { + if (e.code === 'EEXIST') return false; + if (!['EPERM', 'ENOTSUP', 'EOPNOTSUPP', 'ENOSYS', 'EXDEV', 'EMLINK'].includes(e.code)) throw e; + } + try { writeNewFileSynced(file, text, mode); return true; } catch (e) { if (e.code === 'EEXIST') return false; throw e; } + } finally { + try { fs.unlinkSync(tmp); } catch { /* */ } + fsyncDir(path.dirname(file)); + } +} + +// ── The name index (the record of which node a name is) ────────────────── + +/** + * The nodeId the name index holds for `name`; null ONLY when there is no entry (ENOENT). An entry that + * exists and cannot be read, or does not hold a nodeId, is the record of an agent that cannot be read: it + * halts (IdentityHaltError), and is never taken for "no such agent" (a start that took it so made a new + * identity while the original was on disk). + * @param {string} name + * @returns {string|null} + */ function nodeIdForName(name) { + const file = nameIndexPath(name); + let raw; try { - const r = JSON.parse(fs.readFileSync(nameIndexPath(name), 'utf8')); - return r && isNodeIdText(r.nodeId) ? r.nodeId : null; - } catch { return null; } + raw = fs.readFileSync(file, 'utf8'); + } catch (e) { + if (e.code === 'ENOENT') return null; + throw unreadableIndex(name, file, `it could not be read (${e.code || e.message})`); + } + let r = null; + try { r = JSON.parse(raw); } catch { /* below */ } + if (!r || typeof r !== 'object' || !isNodeIdText(r.nodeId) || (r.name !== undefined && r.name !== name)) { + throw unreadableIndex(name, file, raw.length ? 'it does not hold {"name", "nodeId"} for this name' : 'it is empty'); + } + return r.nodeId; +} + +function unreadableIndex(name, file, why) { + const onDisk = identitiesOnDisk().filter((i) => i.name === name).map((i) => `${i.nodeId} (${i.file})`); + const link = compatLinkNodeId(path.join(NODES_DIR, name)); + return new IdentityHaltError(name, file, + `the name index entry for "${name}" exists, but ${why}. It is the record of which node this name is, so it is not read as "no such agent". ` + + (onDisk.length || link + ? `The identity it named is still on this host: ${[...onDisk, link && `nodes/${name} leads to by-id/${link}`].filter(Boolean).join('; ')}. ` + : '') + + 'Nothing was changed.', + `Restore the entry: ${file} holds {"name": ${JSON.stringify(name)}, "nodeId": ""}. Do not delete it to "start over": ` + + 'the key it leads to is this agent.'); +} + +/** Wait (briefly) for the entry another process is creating in place, where hard links are not supported. */ +function nodeIdForNameSettled(name) { + for (let i = 0; ; i++) { + try { return nodeIdForName(name); } catch (e) { + const st = statOrNull(nameIndexPath(name)); + if (i >= 40 || !st || st.size !== 0 || Date.now() - st.mtimeMs > 2000) throw e; + pause(5); + } + } +} + +/** + * Claim `name` for `nodeId` by an exclusive create of its index entry. Of several first starts of one name, + * exactly one makes the entry; the rest find it made. Returns the nodeId the entry holds afterwards: `nodeId` + * when this call claimed the name (or it already named it), the winner's otherwise. + */ +function claimName(name, nodeId) { + ensureDir(path.join(NODES_DIR, 'by-name')); + for (let i = 0; i < 5; i++) { + if (createFileExclusive(nameIndexPath(name), JSON.stringify({ name, nodeId, at: Date.now() }))) return nodeId; + const held = nodeIdForNameSettled(name); + if (held) return held; + } + throw new Error(`[SYM] the name index entry for "${name}" keeps appearing and disappearing; nothing was changed`); } -/** Write the name index entry, atomically. The index is authoritative for what a name means. */ -function writeNameIndex(name, nodeId) { +/** Point `name` at `nodeId` whatever it named (a start pinned to a nodeId: known, kept), durably. */ +function repointName(name, nodeId) { ensureDir(path.join(NODES_DIR, 'by-name')); - const file = nameIndexPath(name); - const tmp = `${file}.${process.pid}.tmp`; - fs.writeFileSync(tmp, JSON.stringify({ name, nodeId, at: Date.now() })); - fs.renameSync(tmp, file); + writeFileDurable(nameIndexPath(name), JSON.stringify({ name, nodeId, at: Date.now() }), 0o644); } -/** Write the name index entry (atomic) and the 0.13-compatible `nodes/` symlink (one release). */ -function indexName(name, nodeId) { - writeNameIndex(name, nodeId); - linkCompat(path.join(NODES_DIR, name), path.join('by-id', nodeId)); - const idRoot = process.env.SYM_IDENTITY_DIR; - if (idRoot && fs.existsSync(path.join(idRoot, 'by-id', nodeId))) linkCompat(path.join(idRoot, name), path.join('by-id', nodeId)); +/** Every nodeId some index entry names, with the names. Entries that cannot be read are left out. */ +function indexEntries() { + const out = new Map(); + const dir = path.join(NODES_DIR, 'by-name'); + let files = []; + try { files = fs.readdirSync(dir); } catch { return out; } + for (const f of files) { + if (f.startsWith('.') || !f.endsWith('.json')) continue; + try { + const r = JSON.parse(fs.readFileSync(path.join(dir, f), 'utf8')); + if (r && isNodeIdText(r.nodeId)) { + if (!out.has(r.nodeId)) out.set(r.nodeId, []); + out.get(r.nodeId).push(typeof r.name === 'string' ? r.name : decodeURIComponent(f.slice(0, -5))); + } + } catch { /* unreadable: its own name halts when started */ } + } + return out; } -/** `link` → `target` (relative), unless a real directory is there (a legacy layout not yet moved). */ -function linkCompat(link, target) { - try { - const st = fs.lstatSync(link); - if (st.isSymbolicLink()) { - if (fs.readlinkSync(link) === target) return; - fs.unlinkSync(link); - } else { - return; // a real dir: migration moves it first +/** The by-id roots: nodes/by-id, and SYM_IDENTITY_DIR/by-id when set. */ +function byIdRoots() { + const roots = [path.join(NODES_DIR, 'by-id')]; + if (process.env.SYM_IDENTITY_DIR) roots.push(path.join(process.env.SYM_IDENTITY_DIR, 'by-id')); + return roots; +} + +/** Every identity.json filed by nodeId on this host: { nodeId, name, file, scrubbed }. Dot-entries (staging, husks) are not identities. */ +function identitiesOnDisk() { + const out = new Map(); + for (const root of byIdRoots()) { + let dirs = []; + try { dirs = fs.readdirSync(root); } catch { continue; } + for (const d of dirs) { + if (d.startsWith('.') || !isNodeIdText(d) || out.has(d)) continue; + const file = path.join(root, d, 'identity.json'); + try { + const j = JSON.parse(fs.readFileSync(file, 'utf8')); + if (j && j.nodeId === d) out.set(d, { nodeId: d, name: j.name, file, scrubbed: !j.privateKey }); + } catch { /* none, or not readable: not an identity this scan can name */ } } - } catch { /* absent */ } - try { fs.symlinkSync(target, link, process.platform === 'win32' ? 'junction' : 'dir'); } catch { /* best effort: the index is authoritative */ } + } + return [...out.values()]; +} + +/** True when the identity of `nodeId` is where a load reads it. */ +function identityInPlace(nodeId) { + return fs.existsSync(path.join(identityDirById(nodeId), 'identity.json')); +} + +// ── The 0.13-compatible links (nodes/ -> by-id/, for one release) ── + +/** + * True when the link at `link` leads to `target` (relative to the link's directory). A Windows junction + * reads back as an absolute path (sometimes with a \\?\ prefix) where a relative target was given, so the + * two are compared as the paths they resolve to, not as text: comparing text re-made the link on every check. + */ +function linkLeadsTo(link, target) { + let got; + try { got = fs.readlinkSync(link); } catch { return false; } + if (got === target) return true; + const fold = (s) => (process.platform === 'win32' ? s.toLowerCase() : s); + const norm = (p) => fold(path.resolve(path.dirname(link), String(p).replace(/^\\\\\?\\/, ''))).replace(/[\\/]+$/, ''); + return norm(got) === norm(target); +} + +/** The nodeId a compat link at `p` (a link to `by-id/` beside it) leads to, or null when `p` is not one. */ +function compatLinkNodeId(p) { + let target; + try { target = fs.readlinkSync(p); } catch { return null; } + const m = /(?:^|[\\/])by-id[\\/]([^\\/]+)[\\/]?$/.exec(target); + if (!m || !isNodeIdText(m[1])) return null; + return linkLeadsTo(p, path.join('by-id', m[1])) ? m[1] : null; +} + +/** Remove a link (a symlink, or a Windows junction, which unlink refuses). */ +function removeLink(p) { + try { fs.unlinkSync(p); } catch (e) { if (e.code === 'ENOENT') return; fs.rmdirSync(p); } } -/** The tombstone a moved identity left, or null. */ +/** + * Make `link` lead to `by-id/`. A real directory is not replaced (a 0.13 one is set aside by the + * migration first), except an empty one (or one holding only a stale lock), which is nothing. The link is + * made under a name of its own and renamed onto the path, so a stale link is replaced in one step. Best + * effort: the index is authoritative. + */ +function linkCompat(link, nodeId) { + const target = path.join('by-id', nodeId); + const st = lstatOrNull(link); + if (st && st.isSymbolicLink() && linkLeadsTo(link, target)) return true; + if (st && !st.isSymbolicLink() && !(st.isDirectory() && removeEmptyDir(link))) return false; + const tmp = path.join(path.dirname(link), `.${path.basename(link)}.link-${process.pid}-${randomTag()}`); + try { + fs.symlinkSync(target, tmp, process.platform === 'win32' ? 'junction' : 'dir'); + } catch { return false; } + try { + fs.renameSync(tmp, link); + return true; + } catch { + try { removeLink(tmp); } catch { /* */ } + try { + if (lstatOrNull(link) && lstatOrNull(link).isSymbolicLink()) removeLink(link); + fs.symlinkSync(target, link, process.platform === 'win32' ? 'junction' : 'dir'); + return true; + } catch { return false; } + } +} + +/** The compat links of `name`: nodes/, and SYM_IDENTITY_DIR/ when the identity is there. */ +function ensureCompatLinks(name, nodeId) { + linkCompat(path.join(NODES_DIR, name), nodeId); + const idRoot = process.env.SYM_IDENTITY_DIR; + if (idRoot && fs.existsSync(path.join(idRoot, 'by-id', nodeId))) linkCompat(path.join(idRoot, name), nodeId); +} + +/** + * Remove `dir` if it holds nothing but a lock nobody holds: rmdir removes only an empty directory, so this can + * never remove one with anything in it. Returns true when it is gone. + */ +function removeEmptyDir(dir) { + const lockPath = path.join(dir, 'lock.pid'); + const lock = readLockFile(lockPath); + if (lock) { + if (lock.pid === process.pid || lockIsHeldByLiveProcess(lock)) return false; + removeStaleLock(lockPath, lock); + } + try { fs.rmdirSync(dir); return true; } catch (e) { return e.code === 'ENOENT'; } +} + +/** The tombstone a moved identity left, or null: beside the key, or (written by 0.14.0 - 0.14.2) in the node dir. */ function readTombstone(nodeId) { - try { return JSON.parse(fs.readFileSync(path.join(nodeDirById(nodeId), TOMBSTONE), 'utf8')); } catch { return null; } + for (const d of new Set([identityDirById(nodeId), nodeDirById(nodeId)])) { + try { return JSON.parse(fs.readFileSync(path.join(d, TOMBSTONE), 'utf8')); } catch { /* none here */ } + } + return null; } -const lstatOrNull = (p) => { try { return fs.lstatSync(p); } catch { return null; } }; +/** The record of an export that has begun and not finished, or null (beside the key; see relocation.js). */ +function readExportIntent(nodeId) { + for (const d of new Set([identityDirById(nodeId), nodeDirById(nodeId)])) { + try { return JSON.parse(fs.readFileSync(path.join(d, EXPORTING), 'utf8')); } catch { /* none here */ } + } + return null; +} /** The files in a node's directory that hold a PRIVATE KEY: its signing identity and the 0.13 E2E keypair. */ const PRIVATE_KEY_FILES = ['identity.json', 'e2e-keypair.json']; /** * Narrow a node's private key files to their owner (0600), and with `dirs` the directories that hold - * them (0700). `rename` keeps a file's mode, so a key file a 0.13 node wrote at the umask default - * (0644 — world-readable) stayed world-readable through every move; this is what ends that. + * them (0700). A key file a 0.13 node wrote at the umask default (0644 — world-readable) stays + * world-readable in the 0.13 directory the migration sets aside; this is what ends that. * * Best effort on purpose: a file this process does not own, or a filesystem without modes (Windows), * must not stop a node from starting. Nothing here changes what a file holds. @@ -336,6 +634,7 @@ function writeFileAtomic0600(file, text) { fs.closeSync(fd); fd = undefined; fs.renameSync(tmp, file); + fsyncDir(path.dirname(file)); } catch (e) { if (fd !== undefined) try { fs.closeSync(fd); } catch { /* */ } try { fs.unlinkSync(tmp); } catch { /* */ } @@ -343,149 +642,32 @@ function writeFileAtomic0600(file, text) { } } -/** True when two trees are the same: same names, same bytes, same link targets. */ -function sameTree(a, b) { - const sa = lstatOrNull(a); const sb = lstatOrNull(b); - if (!sa || !sb) return false; - if (sa.isSymbolicLink() || sb.isSymbolicLink()) return sa.isSymbolicLink() && sb.isSymbolicLink() && fs.readlinkSync(a) === fs.readlinkSync(b); - if (sa.isDirectory() || sb.isDirectory()) { - if (!(sa.isDirectory() && sb.isDirectory())) return false; - const ea = fs.readdirSync(a).sort(); const eb = fs.readdirSync(b).sort(); - return ea.length === eb.length && ea.every((n, i) => n === eb[i] && sameTree(path.join(a, n), path.join(b, n))); - } - if (sa.size !== sb.size) return false; - try { return Buffer.compare(fs.readFileSync(a), fs.readFileSync(b)) === 0; } catch { return false; } // unreadable (a socket): not provably the same -} - -/** Copy a tree to a path that does not exist yet (fails with EEXIST, creating nothing, if it does). */ -function copyTree(src, dst) { - const st = fs.lstatSync(src); - if (st.isSymbolicLink()) { fs.symlinkSync(fs.readlinkSync(src), dst); return; } - if (st.isDirectory()) { - fs.mkdirSync(dst, { mode: 0o700 }); - for (const e of fs.readdirSync(src)) copyTree(path.join(src, e), path.join(dst, e)); - fs.chmodSync(dst, st.mode & 0o777); - return; - } - fs.copyFileSync(src, dst, fs.constants.COPYFILE_EXCL); - fs.chmodSync(dst, st.mode & 0o777); -} - -/** - * Move a file or directory to a path that does not exist. `rename` cannot cross a filesystem - * (EXDEV) — and SYM_IDENTITY_DIR exists to put the identity on ANOTHER volume — so then: copy, check - * the copy is the same as the source, and only then remove the source. A failed copy is removed and - * the source is left untouched; nothing is ever deleted that has not been verified elsewhere. - */ -function moveTree(src, dst) { - try { fs.renameSync(src, dst); return; } catch (e) { if (e.code !== 'EXDEV') throw e; } - try { - copyTree(src, dst); - if (!sameTree(src, dst)) throw new Error(`the copy of ${src} at ${dst} does not match its source; the source is untouched`); - } catch (e) { - if (e.code !== 'EEXIST') fs.rmSync(dst, { recursive: true, force: true }); - throw e; - } - fs.rmSync(src, { recursive: true, force: true }); -} +// ── Reading an identity ────────────────────────────────────────────────── /** - * The first path under `dst` that `src` could not be merged onto without replacing something that is - * not the same, or null. A stale `lock.pid` never conflicts (its holder was checked; the one already in - * place stands). Two copies of the same file do not conflict; two DIFFERENT files do — and an - * `identity.json` that differs is exactly a second claim on a nodeId. - */ -function mergeConflict(src, dst) { - const s = lstatOrNull(src); const d = lstatOrNull(dst); - if (!s || !d) return null; - if (s.isDirectory() && d.isDirectory()) { - for (const e of fs.readdirSync(src)) { - const c = mergeConflict(path.join(src, e), path.join(dst, e)); - if (c) return c; - } - return null; - } - if (path.basename(src) === 'lock.pid') return null; - return sameTree(src, dst) ? null : dst; -} - -/** True when both paths lead to one file or directory (through a symlink, say). */ -function sameFile(a, b) { - try { return fs.realpathSync(a) === fs.realpathSync(b); } catch { return false; } -} - -/** - * Move `src` onto `dst`, MERGING when `dst` already exists (check `mergeConflict` first): what is not - * in `dst` moves in, what is already there stays. A directory that merely exists is not a reason to - * leave a node where it was. `identity.json` moves LAST, so "the identity is in place" means "everything - * else is too" — an interrupted move is re-entered by the identity's absence. - * - * One process at a time: a merge is many steps, and two processes interleaving them can each delete what the - * other has just filed (see `acquireMigrationLock`). Callers hold that lock. - */ -function mergeMove(src, dst) { - const s = lstatOrNull(src); - if (!s) return; - const d = lstatOrNull(dst); - // Never delete a path that leads to the copy it was verified against (through a symlink, say): that would - // delete the only copy. - if (d && sameFile(src, dst)) return; - if (!d) { moveTree(src, dst); return; } - if (s.isDirectory() && d.isDirectory()) { - const names = fs.readdirSync(src).sort((a, b) => (a === 'identity.json') - (b === 'identity.json')); - for (const n of names) mergeMove(path.join(src, n), path.join(dst, n)); - fs.rmdirSync(src); - return; - } - // The same file on both sides (or a stale lock): the one in place stands. Checked again here, because - // this deletes. - if (path.basename(src) !== 'lock.pid' && !sameTree(src, dst)) throw new Error(`cannot merge ${src} onto ${dst}: they differ`); - fs.rmSync(src, { recursive: true, force: true }); -} - -/** - * Where a 0.13 identity for `name` still sits (SYM_IDENTITY_DIR/, else nodes/), or null. - * `existsSync` follows a symlink, so a link that leads to an identity counts. - */ -function legacyIdentityPath(name) { - const idRoot = process.env.SYM_IDENTITY_DIR; - for (const p of [idRoot && path.join(idRoot, name, 'identity.json'), path.join(NODES_DIR, name, 'identity.json')]) { - if (p && fs.existsSync(p)) return p; - } - return null; -} - -/** - * Read and validate an identity.json. Never regenerates a key: a file that exists and cannot be read - * halts (IdentityHaltError), and so does one with a public key and no private key. A pre-v0.3.7 - * identity with no keypair at all gets its first key; a private key with no public key gets the public - * key derived from it. A key file left readable by others (0.13 wrote it at the umask default) is - * narrowed to its owner. + * Parse and validate the text of an identity.json, writing nothing. Never regenerates a key: text that is + * not an identity halts (IdentityHaltError), and so does one with a public key and no private key. A + * pre-v0.3.7 identity with no keypair at all gets its first key; a private key with no public key gets the + * public key derived from it; a bare hostname gets `.local`. `source` is what the file says, `identity` what + * it becomes, `mutated` whether they differ. * @private */ -function readIdentityFile(idPath, name) { - let raw; +function parseIdentity(raw, name, idPath) { + let source; try { - raw = fs.readFileSync(idPath, 'utf8'); - } catch (e) { - // The file EXISTS and we cannot read it. Never regenerate here — a transient read failure - // (permissions, a full disk, a partial write) would otherwise overwrite a live keypair. - throw new IdentityHaltError(name, idPath, `identity.json exists but could not be read: ${e.message}`); - } - let identity; - try { - identity = JSON.parse(raw); + source = JSON.parse(raw); } catch (e) { throw new IdentityHaltError(name, idPath, `identity.json is not valid JSON: ${e.message}`); } - if (!identity || typeof identity !== 'object' || !identity.nodeId) { + if (!source || typeof source !== 'object' || !source.nodeId) { throw new IdentityHaltError(name, idPath, 'identity.json has no nodeId — the file is not an identity'); } // A PUBLIC KEY BUT NO PRIVATE KEY is not "no keypair": the key existed and is gone (a truncated write, // a scrubbed export). Neither mint one nor sign with `undefined`: a new key is a different agent. - if (identity.publicKey && !identity.privateKey) { + if (source.publicKey && !source.privateKey) { throw new IdentityHaltError(name, idPath, 'identity.json has a public key but no private key — the private key is gone, not missing, and a new one would be a different agent'); } + const identity = { ...source }; let mutated = false; // NO KEYPAIR AT ALL (pre-v0.3.7 nodes): nothing to replace, so minting one gives an agent its first // key; the nodeId is preserved. A PRIVATE KEY BUT NO PUBLIC KEY: derive it, never generate one. @@ -510,6 +692,28 @@ function readIdentityFile(idPath, name) { identity.hostname = normalized; mutated = true; } + return { source, identity, mutated }; +} + +/** Read an identity.json that EXISTS: one that cannot be read halts, never regenerates. @private */ +function readIdentityText(idPath, name) { + try { + return fs.readFileSync(idPath, 'utf8'); + } catch (e) { + // The file EXISTS and we cannot read it. Never regenerate here — a transient read failure + // (permissions, a full disk, a partial write) would otherwise overwrite a live keypair. + throw new IdentityHaltError(name, idPath, `identity.json exists but could not be read: ${e.message}`); + } +} + +/** + * Read and validate an installed identity.json (see parseIdentity). A fix it needs (a first key, a derived + * public key, `.local`) is written back atomically and privately; a key file left readable by others is + * narrowed to its owner. + * @private + */ +function readIdentityFile(idPath, name) { + const { identity, mutated } = parseIdentity(readIdentityText(idPath, name), name, idPath); // Atomic and private: a truncate-in-place that is interrupted leaves a key file with no key in it, and a // mode passed to writeFileSync is ignored for a file that already exists. if (mutated) writeFileAtomic0600(idPath, JSON.stringify(identity, null, 2)); @@ -517,156 +721,546 @@ function readIdentityFile(idPath, name) { return identity; } +/** True when `other` is the identity `source` says: the same nodeId, and every key `source` has, the same. */ +function sameIdentity(source, other) { + return Boolean(source && other) && source.nodeId === other.nodeId && + (!source.privateKey || source.privateKey === other.privateKey) && + (!source.publicKey || source.publicKey === other.publicKey); +} + +// ── Staged copies, committed by one rename ─────────────────────────────── + +/** Top-level entries a copy of a node directory leaves out: runtime state, and what the copy writes itself. */ +const COPY_SKIP_TOP = new Set(['lock.pid', 'interior.sock', TOMBSTONE, EXPORTING, COMMIT_MARK]); +const skipTop = (extra = []) => { + const names = new Set([...COPY_SKIP_TOP, ...extra]); + return (rel) => !rel.includes('/') && (names.has(rel) || rel.endsWith('.tmp') || rel.startsWith('lock.pid.') || /^\.[^/]*\.link-\d+-[0-9a-f]+$/.test(rel)); +}; + +/** Two regular files hold the same bytes (read in pieces: a store file can be large). */ +function sameBytes(a, b) { + const sa = statOrNull(a); const sb = statOrNull(b); + if (!sa || !sb || !sa.isFile() || !sb.isFile() || sa.size !== sb.size) return false; + let fa; let fb; + try { + fa = fs.openSync(a, 'r'); fb = fs.openSync(b, 'r'); + const ba = Buffer.alloc(1 << 20); const bb = Buffer.alloc(1 << 20); + for (let pos = 0; pos < sa.size;) { + const na = fs.readSync(fa, ba, 0, ba.length, pos); const nb = fs.readSync(fb, bb, 0, bb.length, pos); + if (na !== nb || na === 0 || ba.compare(bb, 0, nb, 0, na) !== 0) return false; + pos += na; + } + return true; + } catch { return false; } finally { + if (fa !== undefined) fs.closeSync(fa); + if (fb !== undefined) fs.closeSync(fb); + } +} + /** - * Move one 0.13 node (`nodes//`, and its identity at `SYM_IDENTITY_DIR//` when that is - * set) to `nodes/by-id//`, and leave `nodes/` as a symlink to it for one release (a 0.13 - * rollback reads the old path). Idempotent: a name that is already a symlink, or indexed, is left as - * it is. A directory whose lock a live process holds is skipped: that node moves itself when it - * restarts. - * - * A 0.13 identity file is read with the same reader a load uses, so one that exists and cannot be - * established (unreadable, not JSON, no nodeId) HALTS (IdentityHaltError) instead of reading as - * "no node here" — which would let the caller mint a replacement under the agent's name. - * - * What makes a second claim on a nodeId is a DIFFERENT identity (or content that cannot merge) already - * filed under it — not a directory that merely exists. `nodes/by-id//` is also made by a lock, or - * by an earlier interrupted move, and an empty one used to leave the node where it was, unindexed, for - * good. Now the node is MERGED into it. The same identity on both sides is an interrupted move, resumed. - * The name is indexed BEFORE anything moves, so a start that has no lock to wait on (a crash in the middle, - * a process that read the index too early) finds the agent rather than concluding there is none; the identity - * moves last, so an interrupted move is re-entered by the identity's absence (`loadIdentity` does it). Only one - * process migrates at a time (`acquireMigrationLock`): the others wait, then find it done. - * - * Key files end at 0600 (their directory at 0700): `rename` keeps the mode 0.13 wrote them with. - * A move across filesystems (EXDEV) is copied, verified and only then removed. - * - * A 0.13 identity in `nodes//` while SYM_IDENTITY_DIR is set (the volume switched on after the - * agent was made) HALTS: it is neither where the identity dir looks nor safe to leave for a mint. - * - * `onlyNodeId`: a start pinned to a nodeId (`loadIdentity({ nodeId, name })`) migrates the directory under - * `name` only if its identity IS that nodeId. Another agent's directory is left unmoved and unindexed - * ('skipped-duplicate'): the pin asked for one agent, and moving or indexing a different one is not its - * business. - * @param {string} name - * @param {string} [onlyNodeId] - * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'} + * Copy the children of directory `src` into directory `dst` (which exists): directories are made 0700, + * files keep their bytes and mode (a key file 0600) and are synced. Links are FOLLOWED, so what a 0.13 node + * kept behind a link (its directory on another volume, an identity.json linked to a key file) is copied, not + * a link that would lead somewhere else from the new place; a link that leads nowhere is copied as the link + * it is. What is neither a file nor a directory (a socket) is not data and is left out, and so is what + * `skip(rel)` names. A directory reached twice through links is copied as a link, not followed forever. */ -function migrateNodeDir(name, onlyNodeId) { - // One migrator per name. Moving a directory is many steps; two processes doing it at once (a 0.14 start - // beside a CLI call, say) interleave them, and each can delete what the other just filed: found by running - // four real processes against it with a stall injected into every file call, which lost the key in about one - // round in a hundred. The lock is taken only when there is a 0.13 directory to move, and the state is read - // again under it: a process that waited finds the move done. - if (!legacyDirsRemain(name)) return migrateNodeDirOnce(name, onlyNodeId); - const release = acquireMigrationLock(name); - try { return migrateNodeDirOnce(name, onlyNodeId); } finally { release(); } -} - -/** True while a 0.13 directory for `name` is still in its old place (a symlink there is the move, done). */ -function legacyDirsRemain(name) { - const realDir = (p) => { const st = lstatOrNull(p); return Boolean(st && st.isDirectory()); }; - const idRoot = process.env.SYM_IDENTITY_DIR; - return realDir(path.join(NODES_DIR, name)) || Boolean(idRoot && realDir(path.join(idRoot, name))); +function copyChildren(src, dst, skip, rel = '', within = new Set()) { + within.add(fs.realpathSync(src)); + for (const e of fs.readdirSync(src).sort()) { + const r = rel ? `${rel}/${e}` : e; + if (skip(r)) continue; + const from = path.join(src, e); + const to = path.join(dst, e); + const st = statOrNull(from); + if (!st) { // a link that leads nowhere + const l = lstatOrNull(from); + if (l && l.isSymbolicLink()) fs.symlinkSync(fs.readlinkSync(from), to); + continue; + } + if (st.isDirectory()) { + if (within.has(fs.realpathSync(from))) { fs.symlinkSync(fs.readlinkSync(from), to); continue; } + fs.mkdirSync(to, { mode: 0o700 }); + fs.chmodSync(to, 0o700); + copyChildren(from, to, skip, r, new Set(within)); + fsyncDir(to); + } else if (st.isFile()) { + // From the file a link leads to: a clone (macOS) of a link would copy the link. + fs.copyFileSync(fs.realpathSync(from), to, fs.constants.COPYFILE_EXCL | fs.constants.COPYFILE_FICLONE); + fsyncFile(to); + fs.chmodSync(to, PRIVATE_KEY_FILES.includes(e) ? 0o600 : st.mode & 0o777); + } + } +} + +/** The first path (relative) under directory `src` whose copy under `dst` is missing or differs, or null. */ +function firstDifference(src, dst, skip, rel = '', within = new Set()) { + within.add(fs.realpathSync(src)); + for (const e of fs.readdirSync(src).sort()) { + const r = rel ? `${rel}/${e}` : e; + if (skip(r)) continue; + const from = path.join(src, e); + const to = path.join(dst, e); + const st = statOrNull(from); + if (!st) { + const l = lstatOrNull(from); const d = lstatOrNull(to); + if (l && l.isSymbolicLink() && !(d && d.isSymbolicLink() && fs.readlinkSync(to) === fs.readlinkSync(from))) return r; + continue; + } + if (st.isDirectory()) { + if (within.has(fs.realpathSync(from))) continue; + const d = lstatOrNull(to); + if (!d || !d.isDirectory()) return r; + const inner = firstDifference(from, to, skip, r, new Set(within)); + if (inner) return inner; + } else if (st.isFile() && !sameBytes(from, to)) { + return r; + } + } + return null; +} + +/** Remove leftover staging directories (`..partial--`) of processes that are gone. */ +function clearPartials(parent) { + let names = []; + try { names = fs.readdirSync(parent); } catch { return; } + for (const n of names) { + const m = /^\.[^\\/]+\.(?:partial|import)-(\d+)-[0-9a-f]+$/.exec(n); + if (!m || Number(m[1]) === process.pid || pidIsAlive(Number(m[1]))) continue; + try { fs.rmSync(path.join(parent, n), { recursive: true, force: true }); } catch { /* the next attempt */ } + } } -/** How long a start waits for another process's migration of the same name before it gives up, loudly. */ -let migrationLockWaitMs = 30000; +/** + * A new staging directory for `nodeId` beside where it will be installed (`parent`), so the commit is a rename + * on one volume: `..--`, 0700. Leftovers of processes that are gone are cleared first. + */ +function makeStaging(parent, nodeId, kind = 'partial') { + ensureDir(parent); + clearPartials(parent); + const staging = path.join(parent, `.${nodeId}.${kind}-${process.pid}-${randomTag()}`); + fs.mkdirSync(staging, { mode: 0o700 }); + fs.chmodSync(staging, 0o700); + return staging; +} + +/** Write the commit record into a staging directory (last), and sync the directory. */ +function markStaged(staging, nodeId, from) { + writeNewFileSynced(path.join(staging, COMMIT_MARK), JSON.stringify({ nodeId, from, at: Date.now() }) + '\n', 0o600); + fsyncDir(staging); +} + +/** Why a staged commit was refused: a different identity, or files that differ, already under the nodeId. */ +class CommitConflict extends Error {} /** - * Take the exclusive right to migrate 0.13 directories, waiting for another process that holds it. It is the - * identity lock (`acquireIdentityLock`: a lock.pid carrying the holder's start time, reclaimed when its holder is - * gone) on a directory of its own beside the nodes, `nodes/.migrating/`. One lock for every name: a move is quick, - * it happens once per node, and a lock per name would need a directory made and removed around every move. - * @returns {() => void} release + * What is at `target`, the by-id directory a commit for `nodeId` installs: + * - 'absent'; + * - 'committed': it carries the commit record (and, when it holds the identity, `source`'s identity); + * - 'husk': a directory without the commit record that does not contradict `source` (empty, a stale lock, + * the files of an earlier release's interrupted move or of a node that ran there without its identity); + * - 'conflict': it holds a different identity, or is not a directory. + * `holdsIdentity`: the identity lives in this directory (no SYM_IDENTITY_DIR, or the identity directory). */ -function acquireMigrationLock(name) { - const dir = path.join(NODES_DIR, '.migrating'); - const sleeper = new Int32Array(new SharedArrayBuffer(4)); - const deadline = Date.now() + migrationLockWaitMs; - for (;;) { +function examineTarget(target, nodeId, source, holdsIdentity) { + const st = lstatOrNull(target); + if (!st) return { state: 'absent' }; + if (st.isSymbolicLink() || !st.isDirectory()) return { state: 'conflict', reason: `${target} is not a directory` }; + const marked = fs.existsSync(path.join(target, COMMIT_MARK)); + if (holdsIdentity) { + const f = path.join(target, 'identity.json'); + if (fs.existsSync(f)) { + let other = null; + try { other = JSON.parse(fs.readFileSync(f, 'utf8')); } catch { /* below */ } + if (!other) return { state: 'conflict', reason: `${f} is already there and cannot be read` }; + if (!sameIdentity(source, other)) return { state: 'conflict', reason: `a different identity.json is already filed under ${nodeId} (${f})` }; + } else if (marked) { + return { state: 'conflict', reason: `${target} carries a commit record and no identity.json` }; + } + } + if (marked) { + let mark = null; + try { mark = JSON.parse(fs.readFileSync(path.join(target, COMMIT_MARK), 'utf8')); } catch { /* below */ } + if (!mark || mark.nodeId !== nodeId) return { state: 'conflict', reason: `${target} carries the commit record of another node` }; + return { state: 'committed' }; + } + return { state: 'husk' }; +} + +/** + * Copy what `from` holds into `staging` where it is not there yet; a file on both sides that differs is a + * conflict (CommitConflict), and nothing of `from` is changed. How a husk's files (a node that ran in the + * by-id directory without its identity) come into the staged directory before the husk is set aside. + */ +function mergeIntoStaging(from, staging, skip, rel = '') { + for (const e of fs.readdirSync(path.join(from, rel)).sort()) { + const r = rel ? `${rel}/${e}` : e; + if (skip(r)) continue; + const src = path.join(from, r); + const dst = path.join(staging, r); + const st = statOrNull(src); + if (!st) continue; + const d = lstatOrNull(dst); + if (st.isDirectory()) { + if (d && !d.isDirectory()) throw new CommitConflict(`${src} is a directory where the 0.13 copy has a file`); + if (!d) { fs.mkdirSync(dst, { mode: 0o700 }); fs.chmodSync(dst, 0o700); } + mergeIntoStaging(from, staging, skip, r); + } else if (st.isFile()) { + if (d) { if (!sameBytes(src, dst)) throw new CommitConflict(`${src} differs from the copy of the 0.13 directory`); continue; } + fs.copyFileSync(fs.realpathSync(src), dst, fs.constants.COPYFILE_EXCL | fs.constants.COPYFILE_FICLONE); + fsyncFile(dst); + fs.chmodSync(dst, PRIVATE_KEY_FILES.includes(e) ? 0o600 : st.mode & 0o777); + if (!sameBytes(src, dst)) throw new Error(`the copy of ${src} does not match it`); + } + } +} + +/** + * Install a complete staging directory as `target` by ONE rename: that rename is the commit. Returns + * 'committed' when this staging went in, 'exists' when `target` already holds a committed directory (another + * start committed first, or an earlier one did): the caller checks it is the same identity and discards its + * staging. Nothing at `target` is ever overwritten by it: + * - a husk (no commit record) holding nothing, or only a lock whose holder is gone, is removed with rmdir, + * which cannot remove a directory with anything in it, and the rename is tried again; + * - a husk with files: they are copied into the staging (a file that differs is a conflict), and the husk is + * renamed aside to `..husk` beside it, kept. That name is taken once and kept, so a later rename + * can never move a directory installed since; and the staging takes in the files of a husk set aside by + * another start before each attempt; + * - a husk whose lock a live process holds: EIDENTITYLOCK, nothing changed. + */ +function commitStaged(staging, target, { nodeId, examine, skip }) { + const asideHusk = path.join(path.dirname(target), `.${nodeId}.husk`); + for (let attempt = 0; attempt < 12; attempt++) { + if (fs.existsSync(asideHusk)) mergeIntoStaging(asideHusk, staging, skip); try { - return acquireIdentityLock(name, { dir }); + fs.renameSync(staging, target); + fsyncDir(path.dirname(target)); + return 'committed'; } catch (e) { - if (e.code !== 'EIDENTITYLOCK') throw e; - if (Date.now() >= deadline) { - throw new Error(`[SYM] another process (pid ${e.holderPid ?? 'unknown'}) has been moving 0.13 nodes for ${migrationLockWaitMs / 1000}s and has not finished ` + - `(it holds ${path.join(dir, 'lock.pid')}). Wait for it, or stop it and start again.`); + if (!RENAME_TAKEN.has(e.code)) throw e; + } + const t = examine(target); + if (t.state === 'absent') continue; + if (t.state === 'committed') return 'exists'; + if (t.state === 'conflict') throw new CommitConflict(t.reason); + const lockPath = path.join(target, 'lock.pid'); + const lock = readLockFile(lockPath); + if (lock) { + if (lockHeldInProcess(target) || (lock.pid !== process.pid && lockIsHeldByLiveProcess(lock))) { + throw lockError(`[SYM] A live process (PID ${lock.pid ?? 'unknown'}) holds ${lockPath}, but ${nodeId}'s identity is not installed there: ` + + 'a node of an earlier release is running in that directory without its identity. Stop it, then start this agent again; nothing was changed.', lock); } + removeStaleLock(lockPath, lock); + } + let rest = []; + try { rest = fs.readdirSync(target).filter((n) => n !== 'lock.pid'); } catch { continue; } + if (!rest.length) { + try { fs.rmdirSync(target); } catch (e) { if (!['ENOENT', 'ENOTEMPTY', 'EEXIST'].includes(e.code)) throw e; } + continue; } - Atomics.wait(sleeper, 0, 0, 20); + mergeIntoStaging(target, staging, skip); + try { fs.renameSync(target, asideHusk); } catch (e) { if (e.code !== 'ENOENT' && !RENAME_TAKEN.has(e.code)) throw e; } + } + throw new Error(`[SYM] could not install ${target}: what was there kept changing; nothing was overwritten. Start again.`); +} + +/** + * Stage a copy of the 0.13 directory `src` (null: none) for `nodeId` and commit it as `target`. `identityText`, + * when given, is written as identity.json after everything else, then the commit record; the copy is verified + * against `src` byte for byte before the commit. Returns 'committed' | 'exists'; a conflict throws CommitConflict. + */ +function commitCopy(src, target, { nodeId, identityText, source, holdsIdentity, from }) { + const skip = skipTop(['identity.json']); + const examine = (t) => examineTarget(t, nodeId, source, holdsIdentity); + const pre = examine(target); + if (pre.state === 'committed') return 'exists'; + if (pre.state === 'conflict') throw new CommitConflict(pre.reason); + const staging = makeStaging(path.dirname(target), nodeId); + try { + if (src) copyChildren(src, staging, skip); + if (identityText) writeNewFileSynced(path.join(staging, 'identity.json'), identityText, 0o600); + markStaged(staging, nodeId, from); + if (src) { + const d = firstDifference(src, staging, skip); + if (d) throw new Error(`[SYM] the copy of ${path.join(src, d)} does not match its source (did it change while it was copied?); nothing was installed, the source is untouched. Start again.`); + } + if (identityText && !sameIdentity(source, JSON.parse(fs.readFileSync(path.join(staging, 'identity.json'), 'utf8')))) { + throw new Error('[SYM] the staged identity.json does not read back as the identity it was written from; nothing was installed'); + } + return commitStaged(staging, target, { nodeId, skip, examine }); + } finally { + fs.rmSync(staging, { recursive: true, force: true }); + } +} + +/** + * Install a new directory for `nodeId` at `target` (which must not exist), built by `build(staging)` in a staging + * directory and committed by one rename: for a new identity, and for an import (relocation.js, whose + * `beforeCommit` sets the tombstoned copy of a node coming back aside once the new one is complete). + */ +function installNewDir(target, nodeId, build, kind = 'partial', { beforeCommit } = {}) { + const staging = makeStaging(path.dirname(target), nodeId, kind); + try { + build(staging); + markStaged(staging, nodeId, kind); + if (beforeCommit) beforeCommit(); + fs.renameSync(staging, target); + fsyncDir(path.dirname(target)); + } finally { + fs.rmSync(staging, { recursive: true, force: true }); } } -function migrateNodeDirOnce(name, onlyNodeId) { - const legacy = path.join(NODES_DIR, name); +// ── 0.13 directories ───────────────────────────────────────────────────── + +/** + * The 0.13 directory at `p`, or null: a real directory, or a link to a directory that is NOT a compat link (a + * 0.13 node kept its directory on another volume behind one; it is followed and copied). + */ +function legacyAt(p) { + const st = lstatOrNull(p); + if (!st) return null; + if (st.isSymbolicLink()) { + if (compatLinkNodeId(p)) return null; + const t = statOrNull(p); + return t && t.isDirectory() ? { path: p, link: true } : null; + } + return st.isDirectory() ? { path: p, link: false } : null; +} + +/** The 0.13 directories of `name`: nodes/, and SYM_IDENTITY_DIR/ (or the same) for the identity. */ +function legacySources(name) { const idRoot = process.env.SYM_IDENTITY_DIR; - const legacyIdDir = idRoot ? path.join(idRoot, name) : legacy; - const indexFromLink = (link) => { - const m = /by-id[\\/]([^\\/]+)$/.exec(fs.readlinkSync(link)); - if (m && isNodeIdText(m[1])) { indexName(name, m[1]); return 'indexed'; } - return 'not-a-node'; // a link to somewhere else: nothing was indexed, so it must not read as 'already' + const nodes = legacyAt(path.join(NODES_DIR, name)); + const ids = idRoot ? legacyAt(path.join(idRoot, name)) : nodes; + return { nodes, ids, idRoot }; +} + +/** True while a 0.13 identity.json sits under `name` (in nodes// or SYM_IDENTITY_DIR//). */ +function legacyIdentityUnder(name) { + const { nodes, ids } = legacySources(name); + return [nodes, ids].some((l) => l && fs.existsSync(path.join(l.path, 'identity.json'))); +} + +/** Whether `dir` holds anything but a lock. */ +function hasContent(dir) { + try { return fs.readdirSync(dir).some((n) => n !== 'lock.pid'); } catch { return false; } +} + +/** + * Hold the 0.13 directory's own lock (`nodes//lock.pid`, the file a 0.13 process of the agent takes) + * while it is copied and set aside, so no 0.13 process starts in it meanwhile. The lock says it is a + * migration's: another 0.14 start that finds it goes on beside it (the staged commit makes that safe) instead + * of waiting. A live holder that is not a migration is a process running in that directory: `liveHolder`. + */ +function takeLegacyLock(dir) { + const lockPath = path.join(dir, 'lock.pid'); + const token = randomTag() + randomTag(); + let got; + try { + got = acquireLockAt(lockPath, { migrating: true, token }); + } catch { + // The directory does not take a lock (read-only, not ours): copy it all the same, unless a process holds it. + const l = readLockFile(lockPath); + got = l && lockIsLive(l) ? { ok: false, lock: l } : { ok: true, none: true }; + if (got.ok) return { release() {} }; + } + if (!got.ok) { + const l = got.lock; + if (l && (l.migrating || l.pid === process.pid)) return { release() {} }; + return { liveHolder: (l && l.pid) || 'unknown', release() {} }; + } + return { + /** Remove the lock wherever the directory went (set aside), if it is still this one. */ + release(also = []) { + for (const p of [lockPath, ...also.map((d) => path.join(d, 'lock.pid'))]) { + const l = readLockFile(p); + if (l && l.pid === process.pid && l.token === token) { try { fs.unlinkSync(p); } catch { /* */ } } + } + }, }; - const st = lstatOrNull(legacy); - if (st && st.isSymbolicLink()) return nodeIdForName(name) ? 'already' : indexFromLink(legacy); - if (st && !st.isDirectory()) return 'not-a-node'; - const idSt = idRoot ? lstatOrNull(legacyIdDir) : st; - if (idRoot && idSt && idSt.isSymbolicLink()) return nodeIdForName(name) ? 'already' : indexFromLink(legacyIdDir); - const idFile = path.join(legacyIdDir, 'identity.json'); - if (!fs.existsSync(idFile)) { - const inNodeDir = path.join(legacy, 'identity.json'); - if (idRoot && st && fs.existsSync(inNodeDir)) { +} + +/** + * Rename the 0.13 directory (or link) at `at` aside, to `/.legacy-0.13/`: kept, never deleted. + * Returns where it went, or null when it is no longer there (another start set it aside). + */ +function setAside(at, root, name) { + const st = lstatOrNull(at); + if (!st || compatLinkNodeId(at)) return null; + const dir = path.join(root, LEGACY_ASIDE); + if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true, mode: 0o700 }); + for (const dest of [path.join(dir, name), path.join(dir, `${name}.${Date.now()}-${randomTag()}`)]) { + try { + fs.renameSync(at, dest); + fsyncDir(dir); + fsyncDir(root); + return dest; + } catch (e) { + if (e.code === 'ENOENT') return null; + if (!RENAME_TAKEN.has(e.code)) throw e; + const again = lstatOrNull(at); + if (!again || compatLinkNodeId(at)) return null; // set aside meanwhile; what is there now is the link + } + } + return null; +} + +/** + * After the commit: the 0.13 directories of `name` go aside and compat links take their place. The link is + * made first, under a name of its own, and renamed onto the old path right after the directory left it, so a + * 0.13 rollback finds the old path empty for one rename only. + */ +function setAsideAndLink(name, nodeId, srcs) { + const plans = [[NODES_DIR, srcs.nodes]]; + if (srcs.idRoot) plans.push([srcs.idRoot, srcs.ids]); + const aside = []; + for (const [root, src] of plans) { + const at = path.join(root, name); + if (!src) { if (root === NODES_DIR || fs.existsSync(path.join(root, 'by-id', nodeId))) linkCompat(at, nodeId); continue; } + const tmp = path.join(root, `.${name}.link-${process.pid}-${randomTag()}`); + try { + fs.symlinkSync(path.join('by-id', nodeId), tmp, process.platform === 'win32' ? 'junction' : 'dir'); + } catch { + // No link can be made here: the 0.13 directory stays where it is (a 0.13 rollback still reads it), and the + // next start tries again. The identity is installed and the name claimed either way. + continue; + } + let went = null; + try { went = setAside(at, root, name); } catch { /* the identity is installed and indexed: the next start sets it aside */ } finally { + try { fs.renameSync(tmp, at); } catch { try { removeLink(tmp); } catch { /* */ } } + linkCompat(at, nodeId); + } + if (went) { + aside.push(went); + securePrivateFiles([went], { narrowDirs: !src.link }); + } + } + return aside; +} + +/** + * Copy one 0.13 node (`nodes//`, and its identity at `SYM_IDENTITY_DIR//` when that is set) to + * `nodes/by-id//` (and `SYM_IDENTITY_DIR/by-id//`), and COMMIT it: the copy is staged beside + * its destination, identity.json last, synced, verified byte for byte against the source, and installed by one + * rename. Of several starts doing this at once each stages its own copy, one rename wins, and the rest find + * the same identity installed and discard theirs: nothing is moved, so nothing can be lost between two of + * them. Then the name is claimed (an exclusive create of its index entry), the 0.13 directory is renamed aside + * to `.legacy-0.13/` (kept, never deleted), and `nodes/` becomes a link to the by-id directory for + * one release (a 0.13 rollback reads the old path). + * + * A by-id directory already there without the commit record (an empty one, a lock, an earlier release's + * interrupted move, a node that ran there without its identity) is merged into the staged copy and replaced + * through the same commit, never read as "the identity is in place". One that holds a DIFFERENT identity, or a + * file that differs, is a second claim on the nodeId: 'skipped-duplicate', and nothing is changed. A 0.13 + * process running in the directory (its lock live): 'skipped-live', nothing changed (the start refuses with + * EIDENTITYLOCK). An identity this host exported (tombstoned): 'tombstoned'. + * + * A 0.13 identity file is read with the same parser a load uses, so one that exists and cannot be established + * (unreadable, not JSON, no nodeId) HALTS (IdentityHaltError). A 0.13 identity in `nodes//` while + * SYM_IDENTITY_DIR is set and it is not in `SYM_IDENTITY_DIR//` HALTS: it is neither where the identity dir + * looks nor safe to leave for a mint. + * + * `onlyNodeId`: a start pinned to a nodeId migrates the directory under `name` only if its identity IS that + * nodeId; another agent's directory is left as it is ('skipped-duplicate'). + * @returns {{ status: 'moved'|'indexed'|'already'|'not-a-node'|'skipped-live'|'skipped-duplicate'|'tombstoned', nodeId?: string, reason?: string }} + * @private + */ +function migrateName(name, onlyNodeId) { + const srcs = legacySources(name); + if (!srcs.nodes && !srcs.ids) { + const linked = compatLinkNodeId(path.join(NODES_DIR, name)); + const held = nodeIdForName(name); + if (held) return { status: 'already', nodeId: held }; + if (linked && identityInPlace(linked) && !indexEntries().has(linked) && (!onlyNodeId || linked === onlyNodeId)) { + return claimName(name, linked) === linked ? { status: 'indexed', nodeId: linked } : { status: 'skipped-duplicate', nodeId: linked, reason: 'another start claimed the name first' }; + } + return { status: 'not-a-node' }; + } + const idFile = srcs.ids && path.join(srcs.ids.path, 'identity.json'); + const inNodeDir = srcs.nodes && path.join(srcs.nodes.path, 'identity.json'); + if (!idFile || !fs.existsSync(idFile)) { + if (srcs.idRoot && inNodeDir && fs.existsSync(inNodeDir)) { + const want = path.join(srcs.idRoot, name, 'identity.json'); throw new IdentityHaltError(name, inNodeDir, - `SYM_IDENTITY_DIR (${idRoot}) is set, but this agent's identity is still in its node directory and not in ${legacyIdDir}. ` + - `It was not moved, and a new one will not be made in its place. Copy it to ${idFile} (keep the original until the agent has started), ` + + `SYM_IDENTITY_DIR (${srcs.idRoot}) is set, but this agent's identity is still in its node directory and not in ${path.dirname(want)}. ` + + `It was not moved, and a new one will not be made in its place. Copy it to ${want} (keep the original until the agent has started), ` + 'or unset SYM_IDENTITY_DIR.'); } - return 'not-a-node'; + return { status: 'not-a-node' }; } - const identity = readIdentityFile(idFile, name); // halts on an identity it cannot establish - const nodeId = identity.nodeId; + const read = parseIdentity(readIdentityText(idFile, name), name, idFile); // halts on an identity it cannot establish + const nodeId = read.identity.nodeId; if (!isNodeIdText(nodeId)) throw new IdentityHaltError(name, idFile, `the nodeId ${JSON.stringify(String(nodeId).slice(0, 40))} cannot name a directory under nodes/by-id`); - // A start pinned to one nodeId moves that agent and no other: the identity under this name is another - // agent's, and stays exactly where it is (nothing is indexed or moved, nothing lost). - if (onlyNodeId && nodeId !== onlyNodeId) return 'skipped-duplicate'; - if (st) { - const lock = readLockFile(path.join(legacy, 'lock.pid')); - if (lock && lock.pid !== process.pid && lockIsHeldByLiveProcess(lock)) return 'skipped-live'; - } - // The index gives this name to another node: two identities claim one name, and neither is moved over the other. + // The operator copied the identity to SYM_IDENTITY_DIR as the halt above says, and kept the original: the + // two must be one identity. The copy in the node directory is left out of the node directory's copy. + if (srcs.idRoot && inNodeDir && fs.existsSync(inNodeDir)) { + const other = parseIdentity(readIdentityText(inNodeDir, name), name, inNodeDir).source; + if (!sameIdentity(read.source, other) || !sameIdentity(other, read.source)) { + throw new IdentityHaltError(name, inNodeDir, `two different identities sit under "${name}": ${idFile} (${nodeId}) and ${inNodeDir} (${other.nodeId}). Nothing was moved; keep the one that is this agent.`); + } + } + if (onlyNodeId && nodeId !== onlyNodeId) return { status: 'skipped-duplicate', nodeId, reason: `the 0.13 identity under "${name}" is ${nodeId}, not the pinned ${onlyNodeId}; it was left as it is` }; const held = nodeIdForName(name); - if (held && held !== nodeId) return 'skipped-duplicate'; - const dest = nodeDirById(nodeId); - const idDest = idRoot && idSt && idSt.isDirectory() ? path.join(idRoot, 'by-id', nodeId) : null; - // Two dirs claim one nodeId only when what is filed under it differs: the first one moved stays, and - // nothing of this one is moved. An empty directory, or one holding a stale lock, claims nothing. - if ((st && mergeConflict(legacy, dest)) || (idDest && mergeConflict(legacyIdDir, idDest))) return 'skipped-duplicate'; - securePrivateFiles([legacy, legacyIdDir]); // before the move: a copy across volumes must not start from a world-readable file - writeNameIndex(name, nodeId); - ensureDir(path.dirname(dest)); - if (st) mergeMove(legacy, dest); - else ensureDir(dest); - if (idDest) { - ensureDir(path.dirname(idDest)); - mergeMove(legacyIdDir, idDest); - } - securePrivateFiles([dest, idDest], { narrowDirs: true }); - indexName(name, nodeId); - return 'moved'; -} - -/** - * Move every 0.13 node directory under the state root to the by-id layout (design D9, §6 of the - * design: a one-time, idempotent move). Safe to run at every start and from several processes. + if (held && held !== nodeId) return { status: 'skipped-duplicate', nodeId, reason: `the name "${name}" already indexes node ${held}; the 0.13 identity ${nodeId} under it was left as it is` }; + const tomb = readTombstone(nodeId); + if (tomb) return { status: 'tombstoned', nodeId, tomb }; + const lock = srcs.nodes ? takeLegacyLock(srcs.nodes.path) : { release() {} }; + if (lock.liveHolder) return { status: 'skipped-live', nodeId, holderPid: lock.liveHolder }; + let aside = []; + try { + const nodeTarget = nodeDirById(nodeId); + const idTarget = identityDirById(nodeId); + const split = idTarget !== nodeTarget; + const text = JSON.stringify(read.identity, null, 2); + try { + commitCopy(srcs.nodes && srcs.nodes.path, nodeTarget, { nodeId, identityText: split ? null : text, source: read.source, holdsIdentity: !split, from: `0.13 nodes/${name}` }); + if (split) commitCopy(srcs.ids.path, idTarget, { nodeId, identityText: text, source: read.source, holdsIdentity: true, from: `0.13 SYM_IDENTITY_DIR/${name}` }); + } catch (e) { + if (e instanceof CommitConflict) return { status: 'skipped-duplicate', nodeId, reason: `the 0.13 identity under "${name}" was not migrated: ${e.message}` }; + throw e; + } + // The record that the name is this node: claimed once the identity is installed. Of two starts, the + // second finds the entry made and loads what it names. + const winner = claimName(name, nodeId); + if (winner !== nodeId) return { status: 'skipped-duplicate', nodeId, reason: `the name "${name}" was claimed for node ${winner} meanwhile` }; + aside = setAsideAndLink(name, nodeId, srcs); + securePrivateFiles([nodeTarget, idTarget], { narrowDirs: true }); + return { status: 'moved', nodeId }; + } finally { + lock.release(aside); + } +} + +/** + * Migrate the 0.13 directory of `name` to the by-id layout (see migrateName). Idempotent, safe from several + * processes at once, and it takes no lock but the 0.13 directory's own. + * @param {string} name + * @param {string} [onlyNodeId] + * @returns {'moved'|'indexed'|'skipped-live'|'skipped-duplicate'|'not-a-node'|'already'|'tombstoned'} + */ +function migrateNodeDir(name, onlyNodeId) { + const problem = basicNameProblem(name); + if (problem) throw new Error(problem); + return migrateName(name, onlyNodeId).status; +} + +/** + * Migrate every 0.13 node directory under the state root to the by-id layout (design D9, §6 of the + * design: a one-time, idempotent copy). Safe to run at every start and from several processes. * @returns {{ moved: number, already: number, skippedLive: number, duplicates: number }} */ function migrateIdentities() { const out = { moved: 0, already: 0, skippedLive: 0, duplicates: 0 }; - let entries; - try { entries = fs.readdirSync(NODES_DIR, { withFileTypes: true }); } catch { return out; } - for (const d of entries) { - if (d.name === 'by-id' || d.name === 'by-name' || d.name.startsWith('.')) continue; + const names = new Set(); + for (const root of [NODES_DIR, process.env.SYM_IDENTITY_DIR].filter(Boolean)) { + let entries = []; + try { entries = fs.readdirSync(root); } catch { continue; } + for (const n of entries) if (n !== 'by-id' && n !== 'by-name' && !n.startsWith('.') && !basicNameProblem(n)) names.add(n); + } + for (const name of names) { let r; - try { r = migrateNodeDir(d.name); } catch { continue; } + try { r = migrateNodeDir(name); } catch { continue; } if (r === 'moved') out.moved++; else if (r === 'already' || r === 'indexed') out.already++; else if (r === 'skipped-live') out.skippedLive++; @@ -675,82 +1269,136 @@ function migrateIdentities() { return out; } +// ── Loading ────────────────────────────────────────────────────────────── + +/** Load the installed identity of `nodeId`: tombstone and export checked, file validated (halts, never mints). */ +function loadInstalled(nodeId, label, { allowPendingExport = false } = {}) { + const tomb = readTombstone(nodeId); + if (tomb) throw new IdentityTombstonedError(nodeId, tomb); + if (!allowPendingExport) { + const intent = readExportIntent(nodeId); + if (intent) throw new IdentityExportPendingError(nodeId, intent); + } + const idPath = path.join(identityDirById(nodeId), 'identity.json'); + const identity = readIdentityFile(idPath, label); + if (identity.nodeId !== nodeId) throw new IdentityHaltError(label, idPath, `identity.json names ${identity.nodeId}, not ${nodeId}`); + securePrivateFiles([nodeDirById(nodeId)]); // the 0.13 E2E keypair beside it, when the identity lives elsewhere + return identity; +} + +/** Where on this host an identity.json for `nodeId` is, other than where a load reads it. */ +function strayCopiesOf(nodeId) { + const out = []; + const where = new Set([path.join(NODES_DIR, 'by-id', nodeId)]); + if (process.env.SYM_IDENTITY_DIR) where.add(path.join(process.env.SYM_IDENTITY_DIR, 'by-id', nodeId)); + where.delete(identityDirById(nodeId)); + for (const root of [NODES_DIR, process.env.SYM_IDENTITY_DIR].filter(Boolean)) { + let names = []; + try { names = fs.readdirSync(path.join(root, LEGACY_ASIDE)); } catch { /* none */ } + for (const n of names) where.add(path.join(root, LEGACY_ASIDE, n)); + } + for (const d of where) { + try { if (JSON.parse(fs.readFileSync(path.join(d, 'identity.json'), 'utf8')).nodeId === nodeId) out.push(path.join(d, 'identity.json')); } catch { /* none */ } + } + return out; +} + /** - * Load a node's identity by nodeId or by name (design D9). - * - * - `{ nodeId }` loads `nodes/by-id//identity.json`; - * - `{ name }` resolves the name index to a nodeId (moving a 0.13 `nodes//` dir first); - * - `create: false` throws IdentityAbsentError when the identity is absent: a host restoring a - * known agent can never mint a replacement silently; - * - `create: true` (the default, for a brand-new agent) is the ONLY path that mints, and it mints - * only when nothing on this host is known under the name: a name that indexes a node whose - * identity is missing HALTS (IdentityHaltError), and so does a 0.13 identity still sitting under - * the name that could not be moved. Minting there would replace a key (a name is an index; - * renaming changes the index, never the identity); - * - a tombstoned identity (exported to another host) refuses to load: IdentityTombstonedError. - * - * @param {{ nodeId?: string, name?: string, create?: boolean }} o - * @returns {{ nodeId: string, name: string, hostname: string, createdAt: number, publicKey: string, privateKey: string }} + * The halt for an identity that is not installed where it is read: the name's index entry names it + * (`indexed`), or a start asks for it by nodeId. When a copy of it is elsewhere on this host the halt says + * where, and never tells the operator to remove the index entry. */ -function loadIdentity({ nodeId, name, create = true } = {}) { - if (name !== undefined) validateName(name); - if (!nodeId && !name) throw new Error('loadIdentity needs a nodeId or a name'); - if (nodeId && !isNodeIdText(nodeId)) throw new Error(`not a nodeId: ${String(nodeId).slice(0, 40)}`); - let migrated = null; - if (name) { - const pinned = nodeId; - if (!pinned) nodeId = nodeIdForName(name); - // The identity is not in place: the 0.13 layout may still hold it. Unpinned, that is when the name is not - // indexed, or is but the identity is missing (a move another process is making, or one that was - // interrupted). Pinned (mesh-channel starts every node as `{ name, nodeId, create: false }`), when a 0.13 - // directory for the name is still there. Migrating is idempotent, and one path serves both. - const inPlace = Boolean(nodeId) && fs.existsSync(path.join(identityDirById(nodeId), 'identity.json')); - if (!inPlace && (!pinned || legacyDirsRemain(name))) { - migrated = migrateNodeDir(name, pinned); - if (migrated === 'skipped-live') { - // A 0.13 process of this name holds its old directory: load it in place, unmoved. (Pinned: it is the - // pinned agent's, or the migration would have said 'skipped-duplicate'.) - const live = readIdentityFile(legacyIdentityPath(name), name); - const liveTomb = readTombstone(live.nodeId); - if (liveTomb) throw new IdentityTombstonedError(live.nodeId, liveTomb); - return live; - } - if (!nodeId) nodeId = nodeIdForName(name); // the index, whichever process wrote it, says who this name is - } +function indexedButMissing(name, nodeId, { indexed = true } = {}) { + const idPath = path.join(identityDirById(nodeId), 'identity.json'); + const copies = strayCopiesOf(nodeId); + const link = path.join(NODES_DIR, name); + const what = indexed ? `the name "${name}" indexes node ${nodeId}, and its identity.json is not where it is read` : `the identity ${nodeId} is not installed where it is read`; + if (copies.length) { + return new IdentityHaltError(name, idPath, + `${what}; it IS on this host, at ${copies.join(' and ')}.`, + `Put a copy of it at ${idPath} (keep the original until the agent has started)` + + `${process.env.SYM_IDENTITY_DIR ? ', or unset SYM_IDENTITY_DIR if this agent\'s key was never meant to live there' : ''}. Do not remove the name's index entry: the key is this agent.`); } - if (nodeId) { - const tomb = readTombstone(nodeId); - if (tomb) throw new IdentityTombstonedError(nodeId, tomb); - const idPath = path.join(identityDirById(nodeId), 'identity.json'); - if (fs.existsSync(idPath)) { - const identity = readIdentityFile(idPath, name || nodeId); - if (identity.nodeId !== nodeId) throw new IdentityHaltError(name || nodeId, idPath, `identity.json names ${identity.nodeId}, not ${nodeId}`); - securePrivateFiles([nodeDirById(nodeId)]); // the 0.13 E2E keypair beside it, when the identity lives elsewhere - if (name && nodeIdForName(name) !== nodeId) indexName(name, nodeId); - return identity; + return new IdentityHaltError(name, idPath, + `${what}, and no copy of it is on this host. ` + + 'If SYM_IDENTITY_DIR changed or its volume is not mounted, restore that.', + `Restore the keypair from your durable copy to ${idPath} and start again. If this agent's key is gone for good, it cannot return as "${name}"; ` + + `to let the name start again as a DIFFERENT agent, remove ${nameIndexPath(name)} and the link ${link}.`); +} + +/** + * What on this host belongs to `name` without being an installed, indexed identity: each is a reason not to + * mint. `{ identity: bool, text }`: `identity` when it holds an identity (a halt says so; never "absent"). + */ +function nameTraces(name, migration, scan = { indexed: indexEntries(), installed: identitiesOnDisk() }) { + const out = []; + const indexed = scan.indexed; + const idRoot = process.env.SYM_IDENTITY_DIR; + for (const [root, label] of [[NODES_DIR, 'nodes'], [idRoot, 'SYM_IDENTITY_DIR']]) { + if (!root) continue; + const at = path.join(root, name); + const st = lstatOrNull(at); + if (st) { + const linked = compatLinkNodeId(at); + if (linked) { + if (!identityInPlace(linked)) out.push({ identity: false, text: `${at} leads to by-id/${linked}, whose identity is not installed` }); + else out.push({ identity: true, nodeId: indexed.has(linked) ? null : linked, text: `${at} leads to ${linked}${indexed.has(linked) ? `, which the index gives the name(s) ${indexed.get(linked).join(', ')}` : ''}` }); + } else if (st.isSymbolicLink() || st.isDirectory()) { + const l = legacyAt(at); + if (l && fs.existsSync(path.join(l.path, 'identity.json'))) { + out.push({ identity: true, legacy: true, text: `a 0.13 identity sits in ${at}${migration && migration.reason ? ` and was not migrated (${migration.reason})` : ''}` }); + } else if (st.isSymbolicLink()) { + out.push({ identity: false, text: `${at} is a link to ${fs.readlinkSync(at)} with no identity in it` }); + } else if (hasContent(at)) { + out.push({ identity: false, text: `${at} holds a node's files (${fs.readdirSync(at).filter((n) => n !== 'lock.pid').slice(0, 4).join(', ')}) and no identity` + + `${label === 'nodes' && idRoot ? `, and ${path.join(idRoot, name, 'identity.json')} is not there (is the identity volume mounted?)` : ''}` }); + } + } else { + out.push({ identity: false, text: `${at} is a file, not a node directory` }); + } } - if (!create || !name) throw new IdentityAbsentError(name ? `"${name}" (${nodeId})` : nodeId); - // The name indexes a node that exists, and its identity file is not where it should be (a volume not - // mounted, SYM_IDENTITY_DIR set by one embedder and not another, a file removed). That is a known agent - // that cannot be loaded, never a new one. - if (nodeIdForName(name) === nodeId) { - throw new IdentityHaltError(name, idPath, - `the name "${name}" indexes node ${nodeId}, but its identity.json is not there. If SYM_IDENTITY_DIR changed or its volume is not mounted, ` + - `restore that. If this agent is gone for good, remove ${nameIndexPath(name)} and the link ${path.join(NODES_DIR, name)} to let the name start again as a different agent.`); + let aside = []; + try { aside = fs.readdirSync(path.join(root, LEGACY_ASIDE)).filter((n) => n === name || n.startsWith(`${name}.`)); } catch { /* none */ } + for (const n of aside) { + const d = path.join(root, LEGACY_ASIDE, n); + let id = null; + try { id = JSON.parse(fs.readFileSync(path.join(d, 'identity.json'), 'utf8')).nodeId; } catch { /* */ } + out.push({ identity: Boolean(id), nodeId: id, text: `the 0.13 directory of "${name}" was set aside at ${d}${id ? ` (identity ${id})` : ''}` }); } } - if (!create) throw new IdentityAbsentError(`"${name}"`); - // No identity under this name: a genuinely new agent. This is the ONLY path that mints one — and only - // once it is established that no 0.13 identity sits under the name waiting to be moved. A directory with - // an identity.json in it is not "a directory with no identity"; renaming it onto the new node's path would - // bury a live key (and write the new identity over it). - const stranded = legacyIdentityPath(name); - if (stranded) { - throw new IdentityHaltError(name, stranded, - `a 0.13 identity exists under this name and was not migrated (migration said: ${migrated || 'nothing'}` + - `${nodeId && migrated === 'skipped-duplicate' ? `; this start is pinned to ${nodeId}, and that identity is another agent's` : ''}). ` + - 'Making a new identity here would orphan or overwrite its key.'); + for (const i of scan.installed) { + if (i.name !== name || indexed.has(i.nodeId) || i.scrubbed) continue; + out.push({ identity: true, nodeId: i.nodeId, text: `the identity ${i.nodeId} named "${name}" is installed at ${i.file} and no name indexes it` }); } + return out; +} + +/** + * An installed identity the name provably belongs to though its index entry is missing: a compat link + * `nodes/` to it, or the one installed identity whose `name` is this name and that no name indexes + * (a start that crashed between installing it and claiming the name; an entry lost). `{ many }` when more + * than one fits. Nothing is adoptable while any trace of the name holds another identity, or a 0.13 identity + * that was not migrated (`traces`, from nameTraces): that is two claims on the name, and a halt. + */ +function adoptable(name, traces, scan = { indexed: indexEntries(), installed: identitiesOnDisk() }) { + const indexed = scan.indexed; + const consistent = (id) => traces.every((t) => !t.identity || (!t.legacy && t.nodeId === id)); + const ok = (id) => id && identityInPlace(id) && !indexed.has(id) && !readTombstone(id) && consistent(id); + for (const p of [path.join(NODES_DIR, name), process.env.SYM_IDENTITY_DIR && path.join(process.env.SYM_IDENTITY_DIR, name)]) { + const id = p && compatLinkNodeId(p); + if (ok(id)) return { nodeId: id }; + } + const named = scan.installed.filter((i) => i.name === name && !i.scrubbed && ok(i.nodeId)); + if (named.length === 1) return { nodeId: named[0].nodeId }; + return named.length ? { many: named } : null; +} + +/** + * Make a new identity for `name`: installed by one rename (the node dir, and the identity dir when + * SYM_IDENTITY_DIR is set), THEN the name claimed by an exclusive create of its index entry. Returns null when + * another first start of the name claimed it first: this identity was never anyone's, and it is removed. + */ +function mintIdentity(name) { const kp = generateSigningKeyPair(); const identity = { nodeId: uuidv7(), @@ -760,23 +1408,123 @@ function loadIdentity({ nodeId, name, create = true } = {}) { publicKey: kp.publicKey.toString('base64url'), privateKey: kp.privateKey.toString('base64url'), }; - // A 0.13-style `nodes//` dir with no identity (made by hand, or by a test) is the new node's - // dir: it moves with the identity rather than being left behind. - const legacy = path.join(NODES_DIR, name); - const legacySt = lstatOrNull(legacy); - const legacyDir = Boolean(legacySt && legacySt.isDirectory()); - const dir = nodeDirById(identity.nodeId); - ensureDir(path.dirname(dir)); - if (legacyDir) moveTree(legacy, dir); - ensureDir(dir); - const idDir = identityDirById(identity.nodeId); - ensureDir(idDir); - fs.writeFileSync(path.join(idDir, 'identity.json'), JSON.stringify(identity, null, 2), { mode: 0o600 }); - securePrivateFiles([dir, idDir], { narrowDirs: true }); - indexName(name, identity.nodeId); + const nodeId = identity.nodeId; + const text = JSON.stringify(identity, null, 2); + const nodeTarget = nodeDirById(nodeId); + const idTarget = identityDirById(nodeId); + const split = idTarget !== nodeTarget; + const withIdentity = (staging) => writeNewFileSynced(path.join(staging, 'identity.json'), text, 0o600); + installNewDir(nodeTarget, nodeId, split ? () => {} : withIdentity); + if (split) installNewDir(idTarget, nodeId, withIdentity); + if (claimName(name, nodeId) !== nodeId) { + fs.rmSync(nodeTarget, { recursive: true, force: true }); + if (split) fs.rmSync(idTarget, { recursive: true, force: true }); + return null; + } + ensureCompatLinks(name, nodeId); return identity; } +/** + * Load a node's identity by nodeId or by name (design D9). The state of an identity is read from records + * committed in one step: an installed identity (a directory installed by one rename) and the name's index + * entry (created exclusively, after the identity was installed). Nothing is inferred from a path that merely + * exists. + * + * - `{ nodeId }` loads that identity; + * - `{ name }` (pinned or not) first migrates a 0.13 identity still under the name (one path, see + * migrateName), then resolves the name by its index entry; an entry that cannot be read halts; + * - a 0.13 directory a live process holds refuses (EIDENTITYLOCK): the agent is running there; + * - `create: false` throws IdentityAbsentError when the identity is not on this host; + * - `create: true` (the default) is the ONLY path that mints, and it mints only when there is no trace + * of the name on this host: no index entry, no `nodes/` (a link, or a directory with anything in it), + * no `SYM_IDENTITY_DIR/`, no 0.13 directory set aside, no installed identity of that name that no name + * indexes. A trace that provably is this name's identity (a compat link to it, the one unindexed identity of + * that name) is adopted: the name is claimed for it. Any other trace HALTS (IdentityHaltError), saying what + * it is. Two first starts of a new name both mint; one claims the name, the other discards its identity + * and loads the winner; + * - a tombstoned identity (exported to another host) refuses: IdentityTombstonedError; one whose export + * did not finish refuses: IdentityExportPendingError. + * + * @param {{ nodeId?: string, name?: string, create?: boolean, allowPendingExport?: boolean }} o + * @returns {{ nodeId: string, name: string, hostname: string, createdAt: number, publicKey: string, privateKey: string }} + */ +function loadIdentity({ nodeId: pin, name, create = true, allowPendingExport = false } = {}) { + if (name !== undefined) validateName(name); + if (!pin && !name) throw new Error('loadIdentity needs a nodeId or a name'); + if (pin && !isNodeIdText(pin)) throw new Error(`not a nodeId: ${String(pin).slice(0, 40)}`); + const how = { allowPendingExport }; + if (!name) { + if (identityInPlace(pin)) return loadInstalled(pin, pin, how); + if (strayCopiesOf(pin).length) throw indexedButMissing(pin, pin, { indexed: false }); + throw new IdentityAbsentError(pin); + } + // 1. A 0.13 identity under the name is migrated first: the same path for a pinned start and an unpinned one. + let migration = null; + if (legacyIdentityUnder(name)) { + migration = migrateName(name, pin); + if (migration.status === 'skipped-live') { + throw lockError( + `[SYM] Agent '${name}' is running in its 0.13 directory: a live process (PID ${migration.holderPid}) holds ${path.join(NODES_DIR, name, 'lock.pid')}.\n\n` + + ' One agent runs in one process. This start is refused; nothing was moved or changed. Stop that process (it is an older copy of ' + + 'this same agent), and start again: the agent is migrated then, with its store.', { pid: migration.holderPid }); + } + if (migration.status === 'tombstoned') throw new IdentityTombstonedError(migration.nodeId, migration.tomb); + } + if (pin) { + if (identityInPlace(pin)) { + const identity = loadInstalled(pin, name, how); + const held = nodeIdForName(name); + if (!held) claimName(name, pin); + else if (held !== pin) repointName(name, pin); // known: a pinned start re-points the name + ensureCompatLinks(name, pin); + return identity; + } + if (migration && migration.nodeId === pin) { + throw new IdentityHaltError(name, path.join(NODES_DIR, name, 'identity.json'), + `the identity ${pin} this start is pinned to is on this host, in the 0.13 directory of "${name}", and could not be migrated: ${migration.reason || migration.status}.`, + 'Resolve that, then start again. Nothing was changed.'); + } + if (!create) { + if (strayCopiesOf(pin).length) throw indexedButMissing(name, pin, { indexed: false }); + throw new IdentityAbsentError(`"${name}" (${pin})`, nameTraces(name, migration).map((t) => t.text)); + } + } + for (let round = 0; ; round++) { + const indexed = nodeIdForName(name); + if (indexed) { + if (!identityInPlace(indexed)) { + if (!create && !strayCopiesOf(indexed).length) throw new IdentityAbsentError(`"${name}" (${indexed})`); + throw indexedButMissing(name, indexed); + } + const identity = loadInstalled(indexed, name, how); + ensureCompatLinks(name, indexed); + return identity; + } + if (round > 40) throw new Error(`[SYM] the name "${name}" did not settle (its index entry kept being claimed and lost); nothing was minted`); + // One look at what is installed and indexed decides both what is a trace and what can be adopted; an identity + // installed after it is another first start's, and the claim of the name settles which one the name is. + const scan = { indexed: indexEntries(), installed: identitiesOnDisk() }; + const traces = nameTraces(name, migration, scan); + const adopt = adoptable(name, traces, scan); + if (nodeIdForName(name)) continue; // another start claimed the name meanwhile: load what it names + if (adopt && adopt.nodeId) { claimName(name, adopt.nodeId); continue; } // the index decides; read it again + if (adopt && adopt.many && round < 20) { pause(10); continue; } // first starts claiming it: the entry appears within ms + if (traces.length) { + if (!create && !traces.some((t) => t.identity)) throw new IdentityAbsentError(`"${name}"`, traces.map((t) => t.text)); + throw new IdentityHaltError(name, nameIndexPath(name), + `no index entry names "${name}", and this host holds a trace of it: ${traces.map((t) => t.text).join('; ')}` + + `${pin ? `; this start is pinned to ${pin}, which is not on this host` : ''}. ` + + 'Making a new identity here would orphan or overwrite its key, so none was made.', + 'If the trace is this agent, restore its index entry or its identity; if it is not, move it out of the way. Then start again.'); + } + if (!create) throw new IdentityAbsentError(pin ? `"${name}" (${pin})` : `"${name}"`); + const minted = mintIdentity(name); + if (minted) return minted; + // Another first start of this name claimed it first: the index says who it is. + } +} + /** * Load or create a persistent identity for a node (0.13 API, kept): `loadIdentity({ name, create: true })`. * See MMP §3 (Identity). @@ -787,17 +1535,25 @@ function loadOrCreateIdentity(name) { } /** - * Rename a node: the name index moves; the identity (nodeId, key) does not (design D9). + * Rename a node: the name index moves; the identity (nodeId, key) does not (design D9). The new name is + * claimed exclusively. The old name only needs to be a name at all, so an agent whose name a later rule + * refuses (one Windows cannot hold) can be renamed to one it accepts. * @returns {string} the nodeId */ function renameIdentity(oldName, newName) { - validateName(oldName); validateName(newName); + const problem = basicNameProblem(oldName); + if (problem) throw new Error(problem); + validateName(newName); const nodeId = nodeIdForName(oldName); if (!nodeId) throw new IdentityAbsentError(`"${oldName}"`); - if (nodeIdForName(newName)) throw new Error(`the name "${newName}" already indexes a node`); - indexName(newName, nodeId); + if (nodeIdForName(newName) || claimName(newName, nodeId) !== nodeId) throw new Error(`the name "${newName}" already indexes a node`); + ensureCompatLinks(newName, nodeId); try { fs.unlinkSync(nameIndexPath(oldName)); } catch { /* */ } - try { if (fs.lstatSync(path.join(NODES_DIR, oldName)).isSymbolicLink()) fs.unlinkSync(path.join(NODES_DIR, oldName)); } catch { /* */ } + fsyncDir(path.dirname(nameIndexPath(oldName))); + for (const root of [NODES_DIR, process.env.SYM_IDENTITY_DIR].filter(Boolean)) { + const link = path.join(root, oldName); + if (compatLinkNodeId(link) === nodeId) { try { removeLink(link); } catch { /* */ } } + } return nodeId; } @@ -1019,34 +1775,42 @@ function selfStartTime() { * readers that `parseInt()` the whole file still get the PID from the * numeric prefix, so both directions interoperate. * - legacy: the bare PID and nothing else. + * The file is read and stat'ed through ONE open, so `ino`/`dev` and `raw` describe the same file: that is + * what lets a stale lock be removed only if it is still the lock that was judged stale (removeStaleLock). * @param {string} lockPath — absolute path to the lock.pid file - * @returns {{pid: number|null, start: string|null, mtimeMs: number}|null} + * @returns {{pid: number|null, start: string|null, mtimeMs: number, ino: string, dev: string, raw: string, migrating: boolean, token: string|null}|null} * null when the file is absent/unreadable; pid null when content is * unparseable (corrupt or mid-write). */ function readLockFile(lockPath) { - let raw, mtimeMs; + let fd; + try { fd = fs.openSync(lockPath, 'r'); } catch { return null; } + let raw; let st; try { - raw = fs.readFileSync(lockPath, 'utf8'); - mtimeMs = fs.statSync(lockPath).mtimeMs; + st = fs.fstatSync(fd, { bigint: true }); + const buf = Buffer.alloc(Math.min(Number(st.size), 4096)); + const n = buf.length ? fs.readSync(fd, buf, 0, buf.length, 0) : 0; + raw = buf.subarray(0, n).toString('utf8'); } catch { return null; + } finally { + try { fs.closeSync(fd); } catch { /* */ } } const lines = raw.split('\n'); const pid = parseInt(lines[0].trim(), 10); - let start = null; + let meta = null; if (lines.length > 1 && lines[1].trim()) { - try { - const meta = JSON.parse(lines[1]); - if (meta && typeof meta.start === 'string') start = meta.start; - } catch { - // metadata line corrupt — fall back to PID-only semantics - } + try { meta = JSON.parse(lines[1]); } catch { /* metadata line corrupt — fall back to PID-only semantics */ } } return { pid: Number.isFinite(pid) && pid > 0 ? pid : null, - start, - mtimeMs, + start: meta && typeof meta.start === 'string' ? meta.start : null, + mtimeMs: Number(st.mtimeMs), + ino: String(st.ino), + dev: String(st.dev), + raw, + migrating: Boolean(meta && meta.migrating === true), + token: meta && typeof meta.token === 'string' ? meta.token : null, }; } @@ -1170,6 +1934,76 @@ function _installExitHooks() { } } +/** An EIDENTITYLOCK error: `lock` is the holder's parsed lock (its pid becomes `holderPid`). */ +function lockError(message, lock) { + const err = new Error(message); + err.code = 'EIDENTITYLOCK'; + err.holderPid = lock && lock.pid !== undefined ? lock.pid : null; + return err; +} + +/** + * Whether `lock` is held: by a live process (lockIsHeldByLiveProcess), or by this one. A lock with this + * process's pid and another start time was another process's (the pid was reused): stale. + */ +function lockIsLive(lock) { + if (lock.pid === process.pid) return !(lock.start && selfStartTime() && lock.start !== selfStartTime()); + return lockIsHeldByLiveProcess(lock); +} + +/** + * Remove the stale lock `judged` (as readLockFile read it) from `lockPath` ONLY IF it is still that lock, and + * report whether it is gone. POSIX has no compare-and-delete, and the plain sequence (read, judge stale, + * unlink, create) let two starters at a dead holder's lock both hold it: the slower one's unlink removed the + * lock the faster one had just made (4 of 8 rounds with six starters). So removing one particular lock is a + * right of its own, taken exclusively: a claim file named after that lock (`.break-`), which is itself a lock of this kind. Holding it, the lock is read again and + * removed only if it is the same file (dev, ino), with the same content, still stale. Nobody else removes a + * lock without the claim for it, and a lock that is replaced has another claim name, so between the read and + * the unlink the file cannot change. A claim whose holder died (killed in those few system calls) is stale and + * removed the same way, one level up (bounded). False when another start holds the claim (it is removing it). + */ +function removeStaleLock(lockPath, judged, depth = 0) { + if (depth > 3) return false; + const tag = crypto.createHash('sha256').update(`${judged.dev}:${judged.ino}:${judged.raw}`).digest('hex').slice(0, 16); + const claim = `${lockPath}.break-${tag}`; + let got; + try { got = acquireLockAt(claim, {}, depth + 1); } catch { return false; } + if (!got.ok) return false; + try { + const now = readLockFile(lockPath); + if (!now) return true; + // Not the lock judged stale: leave it. The same file with the same content is the same lock, and its holder, + // dead when it was judged, is dead still (a reused pid has another start time). + if (now.dev !== judged.dev || now.ino !== judged.ino || now.raw !== judged.raw) return false; + try { fs.unlinkSync(lockPath); } catch (e) { if (e.code !== 'ENOENT') throw e; } + return true; + } finally { + _releaseLockIfOurs(claim); + } +} + +/** + * Take the lock file `lockPath` (O_EXCL create; content: pid, then {start, createdAt, ...meta}). A stale lock + * (see lockIsHeldByLiveProcess) is removed with removeStaleLock and the create tried again. Returns + * `{ ok: true }` when this call made the lock, `{ ok: false, lock }` when a live process holds it (`lock`: + * the holder's, possibly this process), or the lock is being taken by another start at this moment. + * @private + */ +function acquireLockAt(lockPath, meta = {}, depth = 0) { + let last = null; + for (let attempt = 0; attempt < 8; attempt++) { + // Made whole under a temp name and linked into place (createFileExclusive): a lock is never seen without its owner. + if (createFileExclusive(lockPath, `${process.pid}\n${JSON.stringify({ start: selfStartTime(), createdAt: Date.now(), ...meta })}\n`, 0o644)) return { ok: true }; + const lock = readLockFile(lockPath); + if (!lock) continue; // released between the create and the read: take it now + last = lock; + if (lockIsLive(lock)) return { ok: false, lock }; + if (!removeStaleLock(lockPath, lock, depth)) pause(2); // another start is removing it: look again in a moment + } + return { ok: false, lock: last, contended: true }; +} + /** * NAME-SUFFIXING IS DELETED (founder ruling, agent id = node id). * @@ -1193,7 +2027,7 @@ function _installExitHooks() { * duplicate-identity races on the relay (close code 4004 / 4006 loops) * and ambiguous CMB delivery. * - * Lockfile lives at ~/.sym/nodes//lock.pid: line 1 is the holder's + * Lockfile lives at ~/.sym/nodes/by-id//lock.pid: line 1 is the holder's * PID (legacy parsers read exactly this), line 2 is JSON metadata with the * holder's process start time — the disambiguator that makes staleness * detection immune to PID reuse. On acquire: @@ -1201,7 +2035,8 @@ function _installExitHooks() { * 2. If the lockfile's holder is LIVE (see lockIsHeldByLiveProcess: * alive PID *and* matching start time), throw EIDENTITYLOCK. * 3. Otherwise the lock is stale (dead PID, recycled PID, pre-boot - * legacy lock, aged-out corrupt content) — reclaim it. + * legacy lock, aged-out corrupt content) — it is removed if it is still + * that same lock (removeStaleLock: one remover per lock), and step 1 runs again. * * The release function deletes the lockfile. SymNode calls it on stop(); * a process-level 'exit' hook (plus default-disposition SIGHUP/SIGINT/ @@ -1210,11 +2045,10 @@ function _installExitHooks() { * the next acquire by step 3. * * @param {string} name — node name - * @param {number} [_attempt] — internal EEXIST-race retry counter * @returns {() => void} release function — call to delete the lockfile * @throws {Error} if another process already holds the lock */ -function acquireIdentityLock(name, opts = {}, _attempt = 0) { +function acquireIdentityLock(name, opts = {}) { validateName(name); // The lock lives in the node's own directory: by nodeId since 0.14 (opts.dir); `nodes/` is a // link to it for one release, so a 0.13 process of the same agent contends for the same file. @@ -1231,61 +2065,34 @@ function acquireIdentityLock(name, opts = {}, _attempt = 0) { }; }; - const lock = readLockFile(lockPath); - if (lock) { - // Same-PID re-acquisition is allowed: this happens in tests that - // create multiple SymNodes with the same name in sequence, in - // hot-reload scenarios, and in recovery flows where a single - // process re-initializes after a soft failure. The lock is meant - // to catch CROSS-PROCESS duplicates (the actual bug), not - // in-process re-init. While another holder in this process still - // holds it, the re-acquirer gets a release that does nothing: its - // release must not take the lock from under the live holder (final - // re-review, Finding 7a: an in-process export freed a running node's lock). - if (lock.pid === process.pid) { - if (_heldLockPaths.has(lockPath)) return function release() { /* the first holder releases it */ }; - return makeRelease(); - } - if (lockIsHeldByLiveProcess(lock)) { - const err = new Error( - `[SYM] Agent '${name}' is already live in PID ${lock.pid ?? 'unknown'} on this host.\n\n` + + const got = acquireLockAt(lockPath); + if (got.ok) return makeRelease(); + const lock = got.lock; + // Same-PID re-acquisition is allowed: this happens in tests that + // create multiple SymNodes with the same name in sequence, in + // hot-reload scenarios, and in recovery flows where a single + // process re-initializes after a soft failure. The lock is meant + // to catch CROSS-PROCESS duplicates (the actual bug), not + // in-process re-init. While another holder in this process still + // holds it, the re-acquirer gets a release that does nothing: its + // release must not take the lock from under the live holder (final + // re-review, Finding 7a: an in-process export freed a running node's lock). + if (lock && lock.pid === process.pid && !got.contended) { + if (_heldLockPaths.has(lockPath)) return function release() { /* the first holder releases it */ }; + return makeRelease(); + } + throw lockError( + got.contended + ? `[SYM] Agent '${name}' is being started by another process at this moment (its lock, ${lockPath}, is being taken over from a holder that is gone). ` + + 'This process is refused; start again if that other start fails.' + : `[SYM] Agent '${name}' is already live in PID ${lock && lock.pid !== null ? lock.pid : 'unknown'} on this host.\n\n` + ` An agent id names an AGENT, not a process. Two processes of the same agent ARE ` + `the same agent, so they share one identity and one store, and only one may hold ` + `the write lease at a time. This process is refused; the running one keeps it.\n\n` + ` If that other process is orphaned, stop it and start again. Do NOT work around ` + `this by choosing a different name: a different agent id is a DIFFERENT AGENT, with ` + - `its own keypair and its own history, and peers will treat it as a stranger.` - ); - err.code = 'EIDENTITYLOCK'; - err.holderPid = lock.pid; - throw err; - } - // Stale lock — delete the file so the openSync('wx') below succeeds. - // Without this unlink, the EEXIST retry below could never make progress. - try { fs.unlinkSync(lockPath); } catch {} - } - - // Atomic create: O_EXCL fails if another process creates the file - // between our read above and our write. The fallback re-reads and - // re-checks the holder, bounded so pathological contention can't - // recurse forever. - let fd; - try { - fd = fs.openSync(lockPath, 'wx'); - } catch (e) { - if (e.code === 'EEXIST' && _attempt < 5) { - // Lost the race — re-acquire (retry handles the new state) - return acquireIdentityLock(name, opts, _attempt + 1); - } - throw e; - } - fs.writeSync( - fd, - `${process.pid}\n${JSON.stringify({ start: selfStartTime(), createdAt: Date.now() })}\n`, - ); - fs.closeSync(fd); - - return makeRelease(); + `its own keypair and its own history, and peers will treat it as a stranger.`, + lock); } /** @@ -1312,10 +2119,21 @@ module.exports = { migrateNodeDir, renameIdentity, readTombstone, + readExportIntent, TOMBSTONE, + EXPORTING, + COMMIT_MARK, + LEGACY_ASIDE, IdentityHaltError, IdentityAbsentError, IdentityTombstonedError, + IdentityExportPendingError, + /** For relocation.js: durable writes, and installing a staged directory by one rename. */ + writeFileDurable, + writeNewFileSynced, + fsyncDir, + installNewDir, + PRIVATE_KEY_FILES, uuidv7, validateName, generateSigningKeyPair, @@ -1337,8 +2155,6 @@ module.exports = { _startTimeWarned = false; }, _createStartTimeCache: createStartTimeCache, - /** Test hook: how long (ms) a start waits for another process's migration of the same name. */ - _setMigrationLockWait: (ms) => { migrationLockWaitMs = ms; }, /** Test hook (win32 only): the raw batched PowerShell lookup. */ _windowsStartTimes: windowsStartTimes, readLockFile, diff --git a/lib/relocation.js b/lib/relocation.js index 12bda04a..cddbf3c6 100644 --- a/lib/relocation.js +++ b/lib/relocation.js @@ -5,14 +5,21 @@ * @description Moving a node to another host (sym 0.14, design D9.2). §3.1.3: "The private key MUST * NOT leave the node" — so a node is MOVED, never copied: * - * - `exportNode` holds the identity's lock throughout (no process can start it meanwhile), writes a - * TOMBSTONE at the source FIRST, then the bundle (fsynced, renamed into place, its directory - * fsynced). The source refuses to start that identity from then on (config.js - * IdentityTombstonedError), so no copy remains live: the relay's 4004 replacement cannot produce - * two live copies, because the source is tombstoned before the bundle exists. Once the bundle - * is durable the source LOSES THE KEY (security review): identity.json is rewritten without its - * private key, and the 0.13-compatible `nodes/` link goes, so a 0.13 build on the source - * host (a rollback) cannot load the moved identity either. + * - `exportNode` holds the identity's lock throughout (no process can start it meanwhile) and goes in + * this order, each step durable before the next (0.14.3; the tombstone used to come first, so a crash + * before the bundle existed left a node refused as "exported" with no bundle anywhere, and refused a + * second export): + * 1. an export INTENT (EXPORTING.json) beside the key: from here the source refuses to start the + * identity (IdentityExportPendingError) until the export is finished or abandoned; + * 2. the bundle (fsynced, renamed into place, its directory fsynced); + * 3. the TOMBSTONE beside the key, in the identity directory (SYM_IDENTITY_DIR when set: a rebuilt + * node directory used to take the tombstone with it, and the source started a second live copy); + * 4. the source LOSES THE KEY (security review): identity.json is rewritten without its private key + * (and so is any 0.13 copy the migration set aside), and the 0.13-compatible `nodes/` link goes, + * so a 0.13 build on the source host (a rollback) cannot load the moved identity either. + * An export interrupted at any step is resumable: run it again (before the tombstone it writes the + * bundle again; after it, it finishes the scrub). Before the tombstone it is abortable: `abortExport` + * removes the bundle it wrote and the intent, and the node starts here again. * - The bundle holds the node's directory — its identity, store, key registry, grants and learned * state — encrypted under an operator passphrase (scrypt → AES-256-GCM) or to a target host's * X25519 key (ephemeral X25519 + HKDF-SHA256 → AES-256-GCM). @@ -49,7 +56,7 @@ const config = require('./config'); const { keyFingerprint, isIdentityKey } = require('./roster-keys'); const FORMAT = 'sym-node-bundle-v1'; -const SKIP = new Set(['lock.pid', config.TOMBSTONE, 'interior.sock']); +const SKIP = new Set(['lock.pid', config.TOMBSTONE, config.EXPORTING, config.COMMIT_MARK, 'interior.sock']); const X25519_SPKI_PREFIX = Buffer.from('302a300506032b656e032100', 'hex'); const X25519_PKCS8_PREFIX = Buffer.from('302e020100300506032b656e04220420', 'hex'); const ED25519_PKCS8_PREFIX = Buffer.from('302e020100300506032b657004220420', 'hex'); @@ -82,7 +89,7 @@ function listFiles(root, rel = '') { const out = []; for (const d of fs.readdirSync(path.join(root, rel), { withFileTypes: true })) { const p = path.join(rel, d.name); - if (!rel && SKIP.has(d.name)) continue; + if (!rel && (SKIP.has(d.name) || d.name.startsWith('lock.pid.'))) continue; if (d.name.endsWith('.tmp')) continue; if (d.isDirectory()) out.push(...listFiles(root, p)); else if (d.isFile()) out.push(p); @@ -162,7 +169,8 @@ function publicOf(privateKeyB64url) { } /** - * Move a node off this host: tombstone it, then write its encrypted bundle. + * Move a node off this host: an intent beside the key, its encrypted bundle, then the tombstone, then the + * source's key scrubbed. Run again after an interruption, it finishes what was begun. * @param {object} o * @param {string} [o.name] * @param {string} [o.nodeId] @@ -176,8 +184,22 @@ function exportNode({ name, nodeId, out, passphrase, toHostKey, copyable = false const id = nodeId || (name && config.nodeIdForName(name)); if (!id) throw new config.IdentityAbsentError(name ? `"${name}"` : 'the node'); const dir = config.nodeDirById(id); - if (config.readTombstone(id)) throw new Error(`relocation: ${id} is already tombstoned here (it was exported)`); - const identity = config.loadIdentity({ nodeId: id, name, create: false }); + const tomb = config.readTombstone(id); + if (tomb) { + // Tombstoned: the bundle was durable. An export interrupted before the source lost its key is finished here. + const left = readIdentityJson(config.identityDirById(id)); + if (left && left.privateKey) { + scrubSource(id, left); + return { nodeId: id, name: left.name, files: 0, out: tomb.bundle || null, fingerprint: tomb.fingerprint || keyFingerprint(left.publicKey), resumed: 'scrub' }; + } + throw new Error(`relocation: ${id} is already tombstoned here (it was exported${tomb.bundle ? ` to ${tomb.bundle}` : ''})`); + } + const identity = config.loadIdentity({ nodeId: id, name, create: false, allowPendingExport: true }); + // An export that was interrupted before its tombstone is begun again from the start; the half-written bundle it + // may have left (`..tmp`) goes. A complete bundle it wrote is left where it is: the new one supersedes it, + // and a host imports only one of them. + const pending = config.readExportIntent(id); + if (pending && typeof pending.out === 'string') removeTempBundles(pending.out); // A node running in THIS process holds the lock too (final re-review, Finding 7a). if (config.lockHeldInProcess(dir)) throw new Error(`relocation: ${identity.name} is running in this process; stop it before moving it`); const lock = config.readLockFile(path.join(dir, 'lock.pid')); @@ -200,19 +222,26 @@ function exportNode({ name, nodeId, out, passphrase, toHostKey, copyable = false } } +/** @private The identity.json in `dir`, as JSON, or null. */ +function readIdentityJson(dir) { + try { return JSON.parse(fs.readFileSync(path.join(dir, 'identity.json'), 'utf8')); } catch { return null; } +} + /** @private The export, under the identity's lock. */ function exportLocked({ id, dir, identity, out, passphrase, toHostKey }) { - // THE TOMBSTONE FIRST. From here this host refuses to start the identity, before any bundle exists. - const tomb = { nodeId: id, name: identity.name, movedAt: Date.now(), bundle: path.basename(out), fingerprint: keyFingerprint(identity.publicKey) }; - const tombPath = path.join(dir, config.TOMBSTONE); - const fd = fs.openSync(tombPath, 'wx'); - try { fs.writeSync(fd, JSON.stringify(tomb, null, 2)); fs.fsyncSync(fd); } finally { fs.closeSync(fd); } + const idDir = config.identityDirById(id); + const outAbs = path.resolve(out); + const exportedAt = Date.now(); + const fingerprint = keyFingerprint(identity.publicKey); + // 1. THE INTENT, beside the key. From here this host refuses to start the identity until this export is + // finished or abandoned: a bundle may exist from the next step on. + config.writeFileDurable(path.join(idDir, config.EXPORTING), JSON.stringify({ nodeId: id, name: identity.name, out: outAbs, startedAt: exportedAt }, null, 2), 0o600); + const tomb = { nodeId: id, name: identity.name, movedAt: exportedAt, bundle: outAbs, fingerprint }; // Each file with its mode (final re-review, Finding 7b), so a 0600 file stays 0600 on the target. - const files = listFiles(dir).map((rel) => ({ path: rel.split(path.sep).join('/'), mode: fs.statSync(path.join(dir, rel)).mode & 0o777, data: fs.readFileSync(path.join(dir, rel)).toString('base64') })); - // identity.json lives elsewhere when SYM_IDENTITY_DIR is set: it travels in the bundle all the same. - if (!files.some((f) => f.path === 'identity.json')) { - files.push({ path: 'identity.json', mode: 0o600, data: Buffer.from(JSON.stringify(identity, null, 2)).toString('base64') }); - } + const files = listFiles(dir).filter((rel) => rel !== 'identity.json') + .map((rel) => ({ path: rel.split(path.sep).join('/'), mode: fs.statSync(path.join(dir, rel)).mode & 0o777, data: fs.readFileSync(path.join(dir, rel)).toString('base64') })); + // The identity travels as it was loaded (from SYM_IDENTITY_DIR when set), never a stray copy in the node directory. + files.push({ path: 'identity.json', mode: 0o600, data: Buffer.from(JSON.stringify(identity, null, 2)).toString('base64') }); const contents = { format: FORMAT, nodeId: id, name: identity.name, publicKey: identity.publicKey, exportedAt: tomb.movedAt, files }; // The sealing header is chosen first, so the node's signature covers it (the target host key and // the ephemeral key included): the bundle is the node's own statement, for that host only. @@ -225,14 +254,60 @@ function exportLocked({ id, dir, identity, out, passphrase, toHostKey }) { const headerSig = signHeader(header, identity.privateKey); const plain = Buffer.from(JSON.stringify({ ...contents, signature }), 'utf8'); const bundle = { ...header, headerSig, ...sealWith(plain, sealing) }; + // 2. THE BUNDLE, durable. const tmp = `${out}.${process.pid}.tmp`; const bfd = fs.openSync(tmp, 'w', 0o600); try { fs.writeSync(bfd, JSON.stringify(bundle)); fs.fsyncSync(bfd); } finally { fs.closeSync(bfd); } fs.renameSync(tmp, out); - fsyncDir(path.dirname(path.resolve(out))); - // The bundle is durable: the source loses the key, and the 0.13 path to it. + fsyncDir(path.dirname(outAbs)); + // 3. THE TOMBSTONE, beside the key (it outlives a rebuilt node directory), then the intent goes. + config.writeFileDurable(path.join(idDir, config.TOMBSTONE), JSON.stringify(tomb, null, 2), 0o600); + try { fs.unlinkSync(path.join(idDir, config.EXPORTING)); } catch { /* */ } + fsyncDir(idDir); + // 4. The source loses the key, and the 0.13 path to it. scrubSource(id, identity); - return { nodeId: id, name: identity.name, files: files.length, out, fingerprint: tomb.fingerprint }; + return { nodeId: id, name: identity.name, files: files.length, out, fingerprint }; +} + +/** @private Remove the bundles an export to `out` left half written (`..tmp`). */ +function removeTempBundles(out) { + const base = `${path.basename(out)}.`; + let names = []; + try { names = fs.readdirSync(path.dirname(out)); } catch { return; } + for (const n of names) if (n.startsWith(base) && /^\d+\.tmp$/.test(n.slice(base.length))) { try { fs.unlinkSync(path.join(path.dirname(out), n)); } catch { /* */ } } + fsyncDir(path.dirname(out)); +} + +/** + * Abandon an export that began and did not finish (an intent, no tombstone): the bundle it wrote is removed, + * and the intent, so the node starts on this host again. A bundle copied elsewhere meanwhile is not reachable + * from here: delete that copy too. After the tombstone it is too late (the bundle is the node now): refused. + * @param {{ name?: string, nodeId?: string }} o + * @returns {{ nodeId: string, aborted: boolean, bundle: string|null, bundleRemoved: boolean }} + */ +function abortExport({ name, nodeId } = {}) { + const id = nodeId || (name && config.nodeIdForName(name)); + if (!id) throw new config.IdentityAbsentError(name ? `"${name}"` : 'the node'); + const tomb = config.readTombstone(id); + if (tomb) throw new Error(`relocation: the export of ${id} finished${tomb.bundle ? ` (${tomb.bundle})` : ''} and it is tombstoned here: the bundle is the node now; import it where it goes`); + const intent = config.readExportIntent(id); + if (!intent) return { nodeId: id, aborted: false, bundle: null, bundleRemoved: false }; + const dir = config.nodeDirById(id); + if (config.lockHeldInProcess(dir)) throw new Error(`relocation: ${id} is held by a node in this process`); + const release = config.acquireIdentityLock(intent.name || (name || 'node'), { dir }); + try { + let bundleRemoved = false; + if (typeof intent.out === 'string' && intent.out) { + try { fs.unlinkSync(intent.out); bundleRemoved = true; } catch (e) { if (e.code !== 'ENOENT') throw e; } + removeTempBundles(intent.out); + } + for (const d of new Set([config.identityDirById(id), dir])) { + try { fs.unlinkSync(path.join(d, config.EXPORTING)); fsyncDir(d); } catch { /* */ } + } + return { nodeId: id, aborted: true, bundle: intent.out || null, bundleRemoved }; + } finally { + try { release(); } catch { /* */ } + } } /** @private fsync a directory (a rename in it is durable after this); best effort where unsupported. */ @@ -242,14 +317,25 @@ function fsyncDir(dir) { } /** - * @private After a durable export: identity.json (in the node dir and SYM_IDENTITY_DIR) without its - * private key, written atomically, and the 0.13-compatible `nodes/` links removed. + * @private After a durable export: identity.json (in the node dir and SYM_IDENTITY_DIR, and in any 0.13 + * directory the migration set aside) without its private key, written atomically, and the 0.13-compatible + * `nodes/` links removed. */ function scrubSource(id, identity) { const { privateKey, ...rest } = identity; void privateKey; const scrubbed = JSON.stringify({ ...rest, privateKey: null, movedAt: Date.now() }, null, 2); - for (const d of new Set([config.nodeDirById(id), config.identityDirById(id)])) { + const dirs = new Set([config.nodeDirById(id), config.identityDirById(id)]); + for (const root of [config.NODES_DIR, process.env.SYM_IDENTITY_DIR].filter(Boolean)) { + let names = []; + try { names = fs.readdirSync(path.join(root, config.LEGACY_ASIDE)); } catch { /* none */ } + for (const n of names) { + const d = path.join(root, config.LEGACY_ASIDE, n); + const j = readIdentityJson(d); + if (j && j.nodeId === id && j.privateKey) dirs.add(d); + } + } + for (const d of dirs) { const f = path.join(d, 'identity.json'); if (!fs.existsSync(f)) continue; const tmp = `${f}.${process.pid}.tmp`; @@ -360,36 +446,40 @@ function importNode({ from, passphrase, hostPrivateKey, expect, name } = {}) { throw new Error(`relocation: refusing ${identity.nodeId}: the bundle was exported at or before this host moved the node away (a replay of an older move)`); } const dest = config.nodeDirById(identity.nodeId); - if (fs.existsSync(dest) && !tomb) throw new Error(`relocation: ${identity.nodeId} already lives on this host`); + const idDest = config.identityDirById(identity.nodeId); + const split = idDest !== dest; + const here = fs.existsSync(dest) || (split && fs.existsSync(path.join(idDest, 'identity.json'))); + if (here && !tomb) throw new Error(`relocation: ${identity.nodeId} already lives on this host`); const label = name || identity.name; config.validateName(label); const existing = config.nodeIdForName(label); if (existing && existing !== identity.nodeId) throw new Error(`relocation: the name "${label}" already indexes another node here; import with another name`); - // STAGED: everything is written to a directory of its own, and renamed into place last, so an - // import refused or failing on the way installs nothing (final re-review, Finding 7b). - const staged = `${dest}.import-${process.pid}-${crypto.randomBytes(4).toString('hex')}`; - try { - for (const f of plain.files) { - const rel = String(f.path); - if (rel.includes('..') || path.isAbsolute(rel) || rel.includes('\\')) throw new Error(`relocation: refusing a bundle path outside the node: ${rel}`); - const p = path.join(staged, ...rel.split('/')); - config.ensureDir(path.dirname(p)); - const mode = rel === 'identity.json' ? 0o600 : (Number.isSafeInteger(f.mode) ? (f.mode & 0o777) : 0o644); - fs.writeFileSync(p, Buffer.from(f.data, 'base64'), { mode }); - fs.chmodSync(p, mode); - } - } catch (err) { - fs.rmSync(staged, { recursive: true, force: true }); - throw err; + for (const f of plain.files) { + const rel = String(f.path); + if (rel.includes('..') || path.isAbsolute(rel) || rel.includes('\\') || !rel) throw new Error(`relocation: refusing a bundle path outside the node: ${rel}`); } - if (fs.existsSync(dest)) { - // It was moved away from here before: the tombstoned copy is archived, and the node comes back. - fs.renameSync(dest, `${dest}.moved-${Date.now()}`); + // STAGED: everything is written to a directory of its own and installed by one rename (config.installNewDir), + // so an import refused or failing on the way installs nothing (final re-review, Finding 7b). Directories are + // 0700, key files 0600, and with SYM_IDENTITY_DIR the private key goes ONLY into the identity directory. + const keyFiles = new Set(config.PRIVATE_KEY_FILES); + const write = (root, f) => { + const rel = String(f.path); + const parts = rel.split('/'); + let d = root; + for (const part of parts.slice(0, -1)) { + d = path.join(d, part); + if (!fs.existsSync(d)) { fs.mkdirSync(d, { mode: 0o700 }); fs.chmodSync(d, 0o700); } + } + const mode = keyFiles.has(parts[parts.length - 1]) ? 0o600 : (Number.isSafeInteger(f.mode) ? (f.mode & 0o777) : 0o644); + config.writeNewFileSynced(path.join(root, ...parts), Buffer.from(f.data, 'base64'), mode); + }; + const archive = (d) => { if (fs.existsSync(d)) { fs.renameSync(d, path.join(path.dirname(d), `.${identity.nodeId}.moved-${Date.now()}`)); fsyncDir(path.dirname(d)); } }; + config.installNewDir(dest, identity.nodeId, (staging) => { + for (const f of plain.files) if (!(split && String(f.path) === 'identity.json')) write(staging, f); + }, 'import', { beforeCommit: () => archive(dest) }); // the tombstoned copy of a node coming back is kept aside + if (split) { + config.installNewDir(idDest, identity.nodeId, (staging) => write(staging, idFile), 'import', { beforeCommit: () => archive(idDest) }); } - fs.renameSync(staged, dest); - fsyncDir(path.dirname(dest)); - const idDirExt = config.identityDirById(identity.nodeId); - if (idDirExt !== dest) { config.ensureDir(idDirExt); fs.writeFileSync(path.join(idDirExt, 'identity.json'), Buffer.from(idFile.data, 'base64'), { mode: 0o600 }); } try { fs.appendFileSync(path.join(config.SYM_DIR, CONSUMED_FILE), JSON.stringify({ nodeId: identity.nodeId, signature: plain.signature, exportedAt: plain.exportedAt, importedAt: Date.now() }) + '\n', { mode: 0o600 }); } catch { /* best effort: the tombstone check still refuses an older bundle */ } // indexName is internal to config; loadIdentity by nodeId + name writes the index. @@ -397,4 +487,4 @@ function importNode({ from, passphrase, hostPrivateKey, expect, name } = {}) { return { nodeId: identity.nodeId, name: label, files: plain.files.length, pinnedBy }; } -module.exports = { exportNode, importNode, hostKey, FORMAT }; +module.exports = { exportNode, abortExport, importNode, hostKey, FORMAT }; diff --git a/tests/identity-commit.test.js b/tests/identity-commit.test.js new file mode 100644 index 00000000..e70eb28a --- /dev/null +++ b/tests/identity-commit.test.js @@ -0,0 +1,565 @@ +'use strict'; + +require('./_isolate-home'); // redirect $HOME to a temp sandbox before lib/config loads + +/** + * 0.14.3: the state of an identity is read from records committed in one atomic step, never inferred from + * whatever is at a path. + * + * An independent review (R1) of the 0.14.3 candidate (632b95d) proved, by running them, that a partial + * destination, an unreadable index, a node directory without identity.json and a stale lock's content were + * each read as a state of the identity, and that each patch for one of them had made another. The design that + * replaces them (lib/config.js): + * - a 0.13 directory is COPIED to a staging directory beside its by-id destination, identity.json last, + * synced, verified, and installed by ONE rename; the 0.13 directory is then set aside, never deleted; + * - a name is claimed by an exclusive create of its index entry, after the identity is installed; + * - an index entry is "absent" only on ENOENT; a mint happens only when nothing on the host is a trace of + * the name; a stale lock is removed only by the one process holding the right to remove that lock. + * Every test here fails on 632b95d and passes on this branch; each names the R1 finding it holds. + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const crypto = require('crypto'); +const { spawn } = require('child_process'); +const cfg = require('../lib/config'); +const relocation = require('../lib/relocation'); + +const { NODES_DIR, IdentityHaltError } = cfg; +const posix = process.platform !== 'win32'; +const modeTest = posix ? it : it.skip; +const LIB = path.resolve(__dirname, '../lib'); + +let seq = 0; +const uniq = (b) => `${b}-${process.pid}-${++seq}-${Math.random().toString(36).slice(2, 6)}`; +const read = (p) => fs.readFileSync(p, 'utf8'); +const exists = (p) => fs.existsSync(p); +const modeOf = (p) => fs.statSync(p).mode & 0o777; +const isLink = (p) => { try { return fs.lstatSync(p).isSymbolicLink(); } catch { return false; } }; +const indexFile = (name) => path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); +const ASIDE = cfg.LEGACY_ASIDE || '.legacy-0.13'; + +function makeIdentity(name, extra = {}) { + const kp = crypto.generateKeyPairSync('ed25519'); + return { + nodeId: cfg.uuidv7(), name, hostname: 'host.local', createdAt: 1, + publicKey: kp.publicKey.export({ format: 'der', type: 'spki' }).subarray(12).toString('base64url'), + privateKey: kp.privateKey.export({ format: 'der', type: 'pkcs8' }).subarray(16).toString('base64url'), + ...extra, + }; +} + +function put(file, text, mode = 0o644) { + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, text); + fs.chmodSync(file, mode); +} + +/** A 0.13 node: nodes// with identity.json and a small store. */ +function legacyNode(name, { where = path.join(NODES_DIR, name), identity = makeIdentity(name), files = 6 } = {}) { + put(path.join(where, 'identity.json'), JSON.stringify(identity, null, 2)); + for (let i = 0; i < files; i++) put(path.join(where, 'cmbs', `c${i}.json`), JSON.stringify({ i, pad: 'x'.repeat(300) })); + put(path.join(where, 'inbox.json'), '{"seq":7}'); + return identity; +} + +/** + * Every path under `dir` with its kind and bytes (links followed), sorted. The runtime files a start leaves in a + * node directory (its lock, a claim to break one, the temp file a lock is written under before it is linked into + * place) are not the node's data and are left out: a process killed at the wrong moment leaves one behind. + */ +function treeOf(dir) { + const out = []; + const walk = (d, rel) => { + for (const e of fs.readdirSync(d).sort()) { + if (!rel && (e === 'lock.pid' || e.startsWith('lock.pid.') || (e.startsWith('.lock.pid') && e.endsWith('.tmp')))) continue; + const p = path.join(d, e); const r = rel ? `${rel}/${e}` : e; + const st = fs.statSync(p); + if (st.isDirectory()) { out.push(`D ${r}`); walk(p, r); } else out.push(`F ${r} ${crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex')}`); + } + }; + walk(dir, ''); + return out; +} + +/** Installed identities (non-dot by-id entries) whose name is `name`. */ +function identitiesNamed(name) { + const found = []; + const roots = [path.join(NODES_DIR, 'by-id'), process.env.SYM_IDENTITY_DIR && path.join(process.env.SYM_IDENTITY_DIR, 'by-id')].filter(Boolean); + for (const root of roots) { + let ds = []; try { ds = fs.readdirSync(root); } catch { continue; } + for (const d of ds) { + if (d.startsWith('.')) continue; + try { if (JSON.parse(read(path.join(root, d, 'identity.json'))).name === name) found.push(d); } catch { /* */ } + } + } + return [...new Set(found)]; +} + +function withIdentityDir(fn) { + const prev = process.env.SYM_IDENTITY_DIR; + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-idroot-')); + process.env.SYM_IDENTITY_DIR = root; + const done = () => { + if (prev === undefined) delete process.env.SYM_IDENTITY_DIR; else process.env.SYM_IDENTITY_DIR = prev; + fs.rmSync(root, { recursive: true, force: true }); + }; + let r; + try { r = fn(root); } catch (e) { done(); throw e; } + if (r && typeof r.then === 'function') return r.finally(done); + done(); + return r; +} + +/** Replace fs. for the duration of `body`. */ +function patchFs(fn, wrap, body) { + const original = fs[fn]; + fs[fn] = wrap(original); + try { return body(); } finally { fs[fn] = original; } +} + +/** Replace several fs functions at once: `wraps` maps a name to (original) => replacement. */ +function patchMany(wraps, body) { + const originals = {}; + for (const [k, w] of Object.entries(wraps)) { originals[k] = fs[k]; fs[k] = w(originals[k]); } + try { return body(); } finally { for (const k of Object.keys(originals)) fs[k] = originals[k]; } +} + +const MUTATING = ['renameSync', 'copyFileSync', 'mkdirSync', 'writeFileSync', 'openSync', 'unlinkSync', 'rmdirSync', 'rmSync', 'symlinkSync', 'linkSync', 'chmodSync', 'writeSync', 'fsyncSync']; +class Injected extends Error {} + +/** Run `body`, throwing an Injected error at the k-th mutating fs call (1-based); returns the number of calls made. */ +function failAt(k, body) { + let n = 0; + const wraps = {}; + for (const fn of MUTATING) { + wraps[fn] = (orig) => function (...a) { + if (fn === 'openSync' && !/[wax+]/.test(String(a[1] || 'r'))) return orig.apply(fs, a); // a read is not a step + if (++n === k) throw new Injected(`injected at step ${k} (${fn})`); + return orig.apply(fs, a); + }; + } + // Whatever the body throws once a step failed (the failure, or what the code made of it) is the crash; the + // state it leaves is what the caller checks. + try { patchMany(wraps, body); } catch { /* the crash */ } + return n; +} + +describe('R1-1: a migration interrupted at any step changes nothing in the 0.13 directory, and the next start loads the whole store', () => { + for (const withHusk of [false, true]) { + it(`every step of the migration${withHusk ? ', over a by-id directory holding a stale lock (the 0.14.0-0.14.2 husk)' : ''}`, () => { + // 632b95d moved the store entry by entry: an interruption left part of it in by-id and the rest in nodes/, + // and the copy across volumes deleted the source as it went (R1 measured files missing after kill -9). + const probe = uniq('r11-probe'); + legacyNode(probe); + const steps = failAt(Infinity, () => cfg.migrateNodeDir(probe)); + assert.ok(steps > 10, `the migration took ${steps} steps: the test measures something`); + let checked = 0; + for (let k = 1; k <= steps; k++) { + const name = uniq('r11'); + const id = legacyNode(name); + if (withHusk) put(path.join(cfg.nodeDirById(id.nodeId), 'lock.pid'), '2147483646\n'); + const before = treeOf(path.join(NODES_DIR, name)); + failAt(k, () => cfg.migrateNodeDir(name)); + // The 0.13 directory: in place and byte-identical, or set aside byte-identical with the link in its place. + const legacy = path.join(NODES_DIR, name); + const where = isLink(legacy) ? path.join(NODES_DIR, ASIDE, name) : legacy; + if (!exists(where)) assert.fail(`step ${k}: the 0.13 directory is no longer at nodes/${name} and was not set aside (moved or deleted, not kept)`); + assert.deepStrictEqual(treeOf(where), before, `step ${k}: the 0.13 directory (${path.relative(NODES_DIR, where)}) is byte-identical`); + // What a 0.13 rollback reads: nodes//identity.json is the original key. + assert.strictEqual(JSON.parse(read(path.join(legacy, 'identity.json'))).privateKey, id.privateKey, `step ${k}: a 0.13 reader still finds the key at nodes/`); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.nodeId, id.nodeId, `step ${k}: the same agent`); + assert.strictEqual(loaded.privateKey, id.privateKey, `step ${k}: the same key`); + const dest = cfg.nodeDirById(id.nodeId); + assert.deepStrictEqual(treeOf(dest).filter((l) => !l.includes(' identity.json') && !l.includes(` ${cfg.COMMIT_MARK || '.sym-commit'}`)), + before.filter((l) => !l.includes(' identity.json')), `step ${k}: the whole store is in by-id`); + assert.deepStrictEqual(identitiesNamed(name), [id.nodeId], `step ${k}: exactly one identity`); + checked++; + } + assert.strictEqual(checked, steps); + }); + } +}); + +describe('R1-2: a node directory without its identity is never handed to a new one', () => { + for (const pinned of [false, true]) { + it(`SYM_IDENTITY_DIR's volume not mounted at boot (${pinned ? 'pinned' : 'by name'}): the start halts, mints nothing, touches nothing; mounted, the agent loads with its store`, () => { + withIdentityDir((mnt) => { + const name = uniq('r12'); + const vol = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-keyvol-')); + const id = makeIdentity(name); + put(path.join(vol, name, 'identity.json'), JSON.stringify(id, null, 2)); // the key lives on the volume + for (let i = 0; i < 3; i++) put(path.join(NODES_DIR, name, 'cmbs', `c${i}.json`), `{"i":${i}}`); // the store + const store = treeOf(path.join(NODES_DIR, name)); + const opts = pinned ? { name, nodeId: id.nodeId, create: false } : { name }; + assert.throws(() => cfg.loadIdentity(opts), (e) => (pinned ? e.code === 'EIDENTITYABSENT' : e instanceof IdentityHaltError) && /holds a node's files/.test(e.message) && /mounted/.test(e.message)); + assert.deepStrictEqual(identitiesNamed(name), [], 'nothing was minted'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'the name was not claimed'); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, name)), store, 'the store is where it was'); + // mount it + fs.rmSync(mnt, { recursive: true, force: true }); fs.renameSync(vol, mnt); + const loaded = cfg.loadIdentity(opts); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'c2.json')), 'with its store'); + }); + }); + } +}); + +describe('R1-3: what a 0.13 node kept behind a link is followed and copied', () => { + it('nodes/ a link to the node\'s directory on another volume: the original loads, the volume is untouched', () => { + const name = uniq('r13-dir'); + const durable = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-durable-')), name); + const id = legacyNode(name, { where: durable }); + const before = treeOf(durable); + fs.mkdirSync(NODES_DIR, { recursive: true }); + fs.symlinkSync(durable, path.join(NODES_DIR, name), 'dir'); + for (const opts of [{ name }, { name, nodeId: id.nodeId, create: false }]) { + const loaded = cfg.loadIdentity(opts); + assert.strictEqual(loaded.privateKey, id.privateKey, JSON.stringify(Object.keys(opts))); + } + assert.deepStrictEqual(treeOf(durable), before, 'the durable directory is byte-unchanged'); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'c5.json')), 'the store was copied'); + assert.deepStrictEqual(identitiesNamed(name), [id.nodeId]); + }); + + it('identity.json a RELATIVE link to a key file elsewhere: the original loads, and the installed identity is a file, not a link that now leads nowhere', () => { + const name = uniq('r13-rel'); + const id = makeIdentity(name); + put(path.join(cfg.SYM_DIR, 'keys', `${name}.json`), JSON.stringify(id), 0o600); + put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + fs.symlinkSync(`../../keys/${name}.json`, path.join(NODES_DIR, name, 'identity.json')); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.privateKey, id.privateKey); + const installed = path.join(cfg.nodeDirById(id.nodeId), 'identity.json'); + assert.ok(!isLink(installed), 'a file'); + assert.strictEqual(JSON.parse(read(installed)).privateKey, id.privateKey); + assert.strictEqual(JSON.parse(read(path.join(cfg.SYM_DIR, 'keys', `${name}.json`))).privateKey, id.privateKey, 'the key file it led to is untouched'); + }); +}); + +describe('R1-4: a stale identity lock is removed only if it is still the lock judged stale', () => { + it('a start that judged a dead holder\'s lock stale does not remove the lock another start took meanwhile', async () => { + // 632b95d read the lock, judged it stale, unlinked whatever was there, and created its own: with six starters at a + // SIGKILLed holder's lock, 4 of 8 rounds had two or more simultaneous holders. + const name = uniq('r14'); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-r14-')); + const lockPath = path.join(dir, 'lock.pid'); + put(lockPath, '2147483646\n{"start":"Thu Jan 1 00:00:00 1970","createdAt":1}\n'); // dead holder + const other = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { stdio: 'ignore' }); + try { + const otherLock = `${other.pid}\n${JSON.stringify({ start: cfg.processStartTime(other.pid), createdAt: Date.now() })}\n`; + let raced = false; + // The other start runs after this one READ the stale lock (its judgement) and before the first thing this one + // writes in the lock's directory (632b95d: its unlink of the lock; here: its claim to remove that lock): it + // removes the stale lock and takes its own. + let judged = false; + const inDir = (p) => path.dirname(String(p)) === dir; + const race = (p) => { if (judged && !raced && inDir(p)) { raced = true; fs.unlinkSync(lockPath); fs.writeFileSync(lockPath, otherLock); } }; + let err = null; + patchMany({ + readFileSync: (orig) => function (p, ...rest) { if (String(p) === lockPath) judged = true; return orig.call(fs, p, ...rest); }, + unlinkSync: (orig) => function (p, ...rest) { race(p); return orig.call(fs, p, ...rest); }, + linkSync: (orig) => function (a, b, ...rest) { race(b); return orig.call(fs, a, b, ...rest); }, + openSync: (orig) => function (p, flags, ...rest) { + if (/[wax+]/.test(String(flags || 'r'))) race(p); else if (String(p) === lockPath) judged = true; + return orig.call(fs, p, flags, ...rest); + }, + }, () => { + try { cfg.acquireIdentityLock(name, { dir }); } catch (e) { err = e; } + }); + assert.ok(raced, 'the other start ran between the judgement and the removal'); + assert.ok(err && err.code === 'EIDENTITYLOCK' && err.holderPid === other.pid, `refused, naming the live holder (got ${err && err.code})`); + assert.strictEqual(read(lockPath), otherLock, 'the other start\'s lock is intact'); + assert.deepStrictEqual(fs.readdirSync(dir).filter((n) => n !== 'lock.pid'), [], 'no claim file left behind'); + } finally { other.kill('SIGKILL'); fs.rmSync(dir, { recursive: true, force: true }); } + }); +}); + +describe('R1-6: two first starts of a new name never both win', () => { + it('a second start that runs while the first is between installing its identity and claiming the name gets the same agent', () => { + const name = uniq('r16'); + let ran = false; let checking = false; let second = null; + const named = () => { + try { + return fs.readdirSync(path.join(NODES_DIR, 'by-id')).some((d) => !d.startsWith('.') && + (() => { try { return JSON.parse(fs.readFileSync(path.join(NODES_DIR, 'by-id', d, 'identity.json'), 'utf8')).name === name; } catch { return false; } })()); + } catch { return false; } + }; + const hook = (orig) => function (...a) { + const r = orig.apply(fs, a); + if (!ran && !checking) { + checking = true; + try { if (named()) { ran = true; try { second = cfg.loadIdentity({ name }); } catch (e) { second = e; } } } finally { checking = false; } + } + return r; + }; + const first = patchMany(Object.fromEntries(MUTATING.map((k) => [k, hook])), () => cfg.loadIdentity({ name })); + assert.ok(ran, 'the second start ran inside the first one\'s mint'); + assert.ok(second && second.nodeId, `the second start loaded: ${second && second.message}`); + assert.strictEqual(second.nodeId, first.nodeId, 'one agent'); + assert.strictEqual(second.privateKey, first.privateKey); + assert.strictEqual(cfg.nodeIdForName(name), first.nodeId); + assert.deepStrictEqual(identitiesNamed(name), [first.nodeId], 'one identity on disk'); + }); + + it('three real processes starting one new name at once: one agent, every round', async () => { + let rounds = 0; + for (let round = 0; round < 4; round++) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-r16-')); + try { + const state = path.join(root, 'state'); + const code = `const c = require(${JSON.stringify(path.join(LIB, 'config.js'))}); + process.stdin.once('data', () => { try { const i = c.loadIdentity({ name: 'fresh' }); console.log(JSON.stringify({ id: i.nodeId })); } + catch (e) { console.log(JSON.stringify({ error: e.code || String(e.message).split('\\n')[0] })); } process.exit(0); }); console.log('ready');`; + const kids = Array.from({ length: 3 }, () => spawn(process.execPath, ['-e', code], { env: { ...process.env, SYM_STATE_DIR: state }, stdio: ['pipe', 'pipe', 'inherit'] })); + await Promise.all(kids.map((k) => new Promise((r) => k.stdout.once('data', r)))); + const outs = kids.map((k) => new Promise((r) => { let b = ''; k.stdout.on('data', (d) => { b += d; }); k.on('exit', () => r(JSON.parse(b.trim().split('\n').pop()))); })); + kids.forEach((k) => k.stdin.write('go\n')); + const res = await Promise.all(outs); + assert.deepStrictEqual(res.filter((o) => o.error), [], `round ${round}: every start loaded`); + assert.strictEqual(new Set(res.map((o) => o.id)).size, 1, `round ${round}: one agent, not ${new Set(res.map((o) => o.id)).size}`); + const installed = fs.readdirSync(path.join(state, 'nodes', 'by-id')).filter((d) => !d.startsWith('.')); + assert.deepStrictEqual(installed, [res[0].id], `round ${round}: one identity on disk`); + rounds++; + } finally { fs.rmSync(root, { recursive: true, force: true }); } + } + assert.strictEqual(rounds, 4); + }); +}); + +describe('R1-7: an index entry is "absent" only when there is none', () => { + function made() { + const name = uniq('r17'); + const orig = cfg.loadIdentity({ name }); + return { name, orig, idx: indexFile(name), link: path.join(NODES_DIR, name) }; + } + + it('deleted, and the compat link too: the original is found and the name claimed for it again (632b95d made a new identity)', () => { + const { name, orig, idx, link } = made(); + fs.unlinkSync(idx); fs.unlinkSync(link); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, orig.nodeId); + assert.strictEqual(cfg.nodeIdForName(name), orig.nodeId); + assert.deepStrictEqual(identitiesNamed(name), [orig.nodeId]); + }); + + for (const how of ['empty', 'not json', 'another name']) { + it(`${how}, and no compat link: the start halts, naming where the identity is; nothing is minted`, () => { + const { name, orig, idx, link } = made(); + fs.writeFileSync(idx, how === 'empty' ? '' : how === 'not json' ? '{"nodeId":' : JSON.stringify({ name: 'someone-else', nodeId: orig.nodeId })); + fs.unlinkSync(link); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && e.message.includes(orig.nodeId) && !/remove .*by-name/.test(e.message)); + assert.throws(() => cfg.loadIdentity({ name, create: false }), IdentityHaltError); + assert.deepStrictEqual(identitiesNamed(name), [orig.nodeId], 'no second identity'); + }); + } + + (posix && process.getuid && process.getuid() !== 0 ? it : it.skip)('unreadable (mode 000): halts, nothing minted', () => { + const { name, orig, idx, link } = made(); + fs.chmodSync(idx, 0); fs.unlinkSync(link); + try { + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); + assert.deepStrictEqual(identitiesNamed(name), [orig.nodeId]); + } finally { fs.chmodSync(idx, 0o644); } + }); + + it('a read error that is not ENOENT (EMFILE): halts, nothing minted', () => { + const { name, orig, idx, link } = made(); + fs.unlinkSync(link); + patchFs('readFileSync', (o) => function (p, ...a) { + if (String(p) === idx) throw Object.assign(new Error('EMFILE: too many open files'), { code: 'EMFILE' }); + return o.call(fs, p, ...a); + }, () => { assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); }); + assert.deepStrictEqual(identitiesNamed(name), [orig.nodeId]); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, orig.nodeId, 'once it reads, the same agent'); + }); + + it('index writes are synced: the entry is created exclusively and fsynced before it is relied on', () => { + const name = uniq('r17-sync'); + const synced = []; + const opened = new Map(); + patchMany({ + openSync: (o) => function (p, ...a) { const fd = o.call(fs, p, ...a); opened.set(fd, String(p)); return fd; }, + fsyncSync: (o) => function (fd) { synced.push(opened.get(fd)); return o.call(fs, fd); }, + }, () => cfg.loadIdentity({ name })); + const dir = path.dirname(indexFile(name)); + assert.ok(synced.some((p) => p && p.startsWith(dir + path.sep) && p.includes(encodeURIComponent(name))), 'the entry\'s content was fsynced'); + if (posix) assert.ok(synced.includes(dir), 'and the by-name directory'); + }); +}); + +describe('R1-8: an export interrupted before its bundle is durable is resumable or abortable, never a false "exported"', () => { + async function made(name) { + const id = cfg.loadIdentity({ name }); + put(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'm.json'), '{"memory":1}'); + return id; + } + const out = () => path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-bundle-')), 'node.bundle'); + + it('the start is refused saying the export did not finish (not "exported to another host"), and running the export again finishes it', async () => { + const name = uniq('r18'); + const id = await made(name); + const file = out(); + const toHostKey = relocation.hostKey().publicKey; + assert.throws(() => patchFs('renameSync', (o) => function (a, b) { + if (String(b) === file) throw Object.assign(new Error('EIO: crash before the bundle was durable'), { code: 'EIO' }); + return o.call(fs, a, b); + }, () => relocation.exportNode({ name, out: file, toHostKey })), /EIO/); + assert.ok(!exists(file), 'no bundle'); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e.code === 'EIDENTITYEXPORTING' && !/exported to another host/.test(e.message)); + assert.strictEqual(cfg.readTombstone(id.nodeId), null, 'not tombstoned: nothing was moved'); + const r = relocation.exportNode({ name, out: file, toHostKey }); + assert.strictEqual(r.nodeId, id.nodeId); + assert.ok(exists(file), 'the bundle'); + assert.ok(cfg.readTombstone(id.nodeId), 'tombstoned now'); + assert.strictEqual(JSON.parse(read(path.join(cfg.identityDirById(id.nodeId), 'identity.json'))).privateKey, null, 'the source lost the key'); + }); + + it('abortExport removes the bundle it wrote and the intent: the node starts here again', async () => { + const name = uniq('r18-abort'); + const id = await made(name); + const file = out(); + const toHostKey = relocation.hostKey().publicKey; + // interrupted after the bundle is durable and before the tombstone + assert.throws(() => patchFs('openSync', (o) => function (p, ...a) { + if (String(p).includes(cfg.TOMBSTONE)) throw Object.assign(new Error('EIO: crash before the tombstone'), { code: 'EIO' }); + return o.call(fs, p, ...a); + }, () => relocation.exportNode({ name, out: file, toHostKey })), /EIO/); + assert.ok(exists(file), 'the bundle is durable'); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e.code === 'EIDENTITYEXPORTING'); + const a = relocation.abortExport({ name }); + assert.strictEqual(a.aborted, true); + assert.strictEqual(a.bundleRemoved, true); + assert.ok(!exists(file), 'the bundle is gone'); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'the node starts again, with its key'); + }); +}); + +describe('R1-9: with SYM_IDENTITY_DIR the tombstone lives beside the key, and an import writes the key only there', () => { + it('an export killed after its bundle is durable: a rebuilt node directory does not bring a second live copy back; exporting again finishes the scrub', () => { + withIdentityDir(() => { + const name = uniq('r19'); + const id = cfg.loadIdentity({ name }); + const file = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-bundle-')), 'node.bundle'); + const toHostKey = relocation.hostKey().publicKey; + assert.throws(() => patchFs('openSync', (o) => function (p, ...a) { + if (/identity\.json\.\d+\.tmp$/.test(String(p))) throw Object.assign(new Error('EIO: crash before the scrub'), { code: 'EIO' }); + return o.call(fs, p, ...a); + }, () => relocation.exportNode({ name, out: file, toHostKey })), /EIO/); + assert.ok(exists(file), 'the bundle is durable'); + fs.rmSync(cfg.nodeDirById(id.nodeId), { recursive: true, force: true }); // the disposable node dir is rebuilt + assert.throws(() => cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }), (e) => e.code === 'EIDENTITYTOMBSTONED'); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e.code === 'EIDENTITYTOMBSTONED'); + const again = relocation.exportNode({ name, out: file + '2', toHostKey }); + assert.ok(again.resumed, 'the interrupted export was finished'); + assert.strictEqual(JSON.parse(read(path.join(cfg.identityDirById(id.nodeId), 'identity.json'))).privateKey, null, 'the source no longer holds the key'); + }); + }); + + modeTest('import: directories 0700, key files 0600, and the private key only in the identity directory', () => { + withIdentityDir(() => { + const name = uniq('r19-imp'); + const id = cfg.loadIdentity({ name }); + put(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'deep', 'm.json'), '{"memory":1}'); + put(path.join(cfg.nodeDirById(id.nodeId), 'e2e-keypair.json'), '{}', 0o600); + const file = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-bundle-')), 'node.bundle'); + relocation.exportNode({ name, out: file, toHostKey: relocation.hostKey().publicKey }); + // another host: this host's copies of the node removed + fs.rmSync(cfg.nodeDirById(id.nodeId), { recursive: true, force: true }); + fs.rmSync(cfg.identityDirById(id.nodeId), { recursive: true, force: true }); + relocation.importNode({ from: file, expect: { nodeId: id.nodeId, key: id.publicKey } }); + const nd = cfg.nodeDirById(id.nodeId); const idd = cfg.identityDirById(id.nodeId); + assert.strictEqual(modeOf(nd), 0o700); + assert.strictEqual(modeOf(path.join(nd, 'cmbs')), 0o700); + assert.strictEqual(modeOf(path.join(nd, 'cmbs', 'deep')), 0o700); + assert.strictEqual(modeOf(path.join(nd, 'e2e-keypair.json')), 0o600); + assert.strictEqual(modeOf(idd), 0o700); + assert.strictEqual(modeOf(path.join(idd, 'identity.json')), 0o600); + assert.ok(!exists(path.join(nd, 'identity.json')), 'no private key in the disposable node directory'); + assert.strictEqual(cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }).privateKey, id.privateKey); + }); + }); +}); + +describe('R1-10: a crash after the commit never leaves a state read as "new"', () => { + it('the compat link cannot be made: the 0.13 directory stays in place for a 0.13 rollback, and the next start finishes', () => { + // 632b95d had already moved everything; the failed link left nodes/ empty or absent, and a 0.13 build reading + // it would make a new identity. + const name = uniq('r110'); + const id = legacyNode(name); + patchFs('symlinkSync', (o) => function (target, p, ...a) { + if (String(p).startsWith(NODES_DIR + path.sep) && !String(p).includes(`${path.sep}by-id${path.sep}`)) throw Object.assign(new Error('EPERM: injected'), { code: 'EPERM' }); + return o.call(fs, target, p, ...a); + }, () => { try { cfg.migrateNodeDir(name); } catch { /* a failure is fine; what it leaves is the test */ } }); + assert.strictEqual(JSON.parse(read(path.join(NODES_DIR, name, 'identity.json'))).privateKey, id.privateKey, 'a 0.13 reader still finds the key'); + assert.ok(exists(path.join(NODES_DIR, name, 'cmbs', 'c0.json')), 'and its store'); + const loaded = cfg.loadIdentity({ name }); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(isLink(path.join(NODES_DIR, name)), 'finished: the link'); + }); + + it('a new identity installed and the start killed before it claimed the name: the next start finds that identity, not a new one', () => { + const name = uniq('r110-mint'); + const byName = path.join(NODES_DIR, 'by-name'); + let thrown = false; + const stop = (o) => function (p, ...a) { + if (!thrown && String(p).startsWith(byName + path.sep) && (o !== fs.readFileSync)) { thrown = true; throw Object.assign(new Error('killed before the claim'), { code: 'EINTR' }); } + return o.call(fs, p, ...a); + }; + patchMany({ writeFileSync: stop, openSync: (o) => function (p, flags, ...a) { return /[wa]/.test(String(flags || 'r')) ? stop(o)(p, flags, ...a) : o.call(fs, p, flags, ...a); } }, + () => { try { cfg.loadIdentity({ name }); } catch { /* killed */ } }); + assert.ok(thrown, 'the start was stopped at the claim'); + const made = identitiesNamed(name); + assert.strictEqual(made.length, 1, 'its identity was installed'); + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, made[0], 'the next start loads it'); + assert.deepStrictEqual(identitiesNamed(name), made, 'and makes no other'); + }); +}); + +describe('R1-11: names Windows cannot hold, and Windows junctions', () => { + it('validateName refuses the names Windows reserves or cannot hold (the rule of mesh-channel\'s nodeNameProblem)', () => { + for (const bad of ['CON', 'con', 'NUL', 'nul.txt', 'AUX', 'PRN.log', 'COM1', 'lpt9', 'a:b', 'a*b', 'a?b', 'a"b', 'ab', 'a|b', 'trailing.', 'trailing ']) { + assert.throws(() => cfg.validateName(bad), Error, bad); + } + for (const good of ['console', 'conman', 'nullable', 'com', 'com10x', 'a.b', 'agent-a@mesh', 'claude-code', '日本語']) { + assert.doesNotThrow(() => cfg.validateName(good), good); + } + assert.throws(() => cfg.loadIdentity({ name: 'NUL' })); + }); + + it('a compat link that reads back as an absolute path (a junction) is recognised, not re-made on every check', () => { + const name = uniq('r111'); + const id = cfg.loadIdentity({ name }); + const link = path.join(NODES_DIR, name); + const absolute = path.join(NODES_DIR, 'by-id', id.nodeId); + fs.unlinkSync(indexFile(name)); // the link is then what tells which node the name is + let remade = 0; + patchMany({ + readlinkSync: (o) => function (p, ...a) { return String(p) === link ? absolute : o.call(fs, p, ...a); }, + unlinkSync: (o) => function (p, ...a) { if (String(p) === link) remade++; return o.call(fs, p, ...a); }, + renameSync: (o) => function (a, b) { if (String(b) === link) remade++; return o.call(fs, a, b); }, + symlinkSync: (o) => function (t, p, ...a) { if (String(p) === link) remade++; return o.call(fs, t, p, ...a); }, + }, () => { + for (let i = 0; i < 3; i++) assert.strictEqual(cfg.loadIdentity({ name }).nodeId, id.nodeId); + }); + assert.strictEqual(remade, 0, 'the link was not re-made'); + }); +}); + +describe('error texts: the operator is never told to delete what leads to a key that is on disk', () => { + it('SYM_IDENTITY_DIR switched on after the agent was made: the halt says where the key is, and not to remove the index', () => { + const name = uniq('err-iddir'); + const id = cfg.loadIdentity({ name }); + withIdentityDir(() => { + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && + e.message.includes(path.join(cfg.nodeDirById(id.nodeId), 'identity.json')) && !e.message.includes(`remove ${indexFile(name)}`)); + assert.throws(() => cfg.loadIdentity({ name, create: false }), (e) => e instanceof IdentityHaltError && !/No identity/.test(e.message), 'not "no identity on this host"'); + assert.throws(() => cfg.loadIdentity({ nodeId: id.nodeId, create: false }), (e) => e instanceof IdentityHaltError); + }); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + }); +}); diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js index 16212c6c..7013e7dc 100644 --- a/tests/identity-migration-by-id.test.js +++ b/tests/identity-migration-by-id.test.js @@ -93,6 +93,20 @@ function withIdentityDir(fn) { } } +/** Every path under `dir` with its kind and bytes (links followed), sorted: two equal trees hold the same data. */ +function treeOf(dir) { + const out = []; + const walk = (d, rel) => { + for (const e of fs.readdirSync(d).sort()) { + const p = path.join(d, e); const r = rel ? `${rel}/${e}` : e; + const st = fs.statSync(p); + if (st.isDirectory()) { out.push(`D ${r}`); walk(p, r); } else out.push(`F ${r} ${crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex')}`); + } + }; + walk(dir, ''); + return out; +} + /** Replace fs. for the duration of `body`; `wrap(original)` returns the replacement. */ function patchFs(fn, wrap, body) { const original = fs[fn]; @@ -191,7 +205,10 @@ describe('T3-T5: a by-id directory that merely exists is not a second claim on t const dest = husk(id.nodeId); assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey, 'the identity is under the nodeId, key unchanged'); - assert.ok(exists(path.join(dest, 'lock.pid')), 'what was already there stays'); + // 0.14.3 changed this: the husk is REPLACED through the staged commit (one rename), and its stale lock is removed + // (compare-and-delete, what the next acquire would do anyway) rather than kept; a live one would refuse (EIDENTITYLOCK). + assert.ok(!exists(path.join(dest, 'lock.pid')), 'the stale lock is not carried into the installed directory'); + assert.ok(exists(path.join(dest, cfg.COMMIT_MARK)), 'the directory carries the commit record'); assert.ok(exists(path.join(dest, 'cmbs', 'x.json')), 'the store came with it'); assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); assert.ok(isLink(path.join(NODES_DIR, name)), 'nodes/ is the compat link'); @@ -331,11 +348,13 @@ describe('F4: a name that indexes a node whose identity is missing halts', () => assert.ok(halt instanceof IdentityHaltError); const index = path.join(NODES_DIR, 'by-name', `${encodeURIComponent(name)}.json`); const link = path.join(NODES_DIR, name); - assert.ok(halt.reason.includes(index) && halt.reason.includes(link), 'the message names the index file and the link'); + // 0.14.3: what to do is the halt's advice (in the message), and it is given only when no copy of the key is on this host. + assert.ok(halt.message.includes(index) && halt.message.includes(link), 'the message names the index file and the link'); fs.unlinkSync(index); - assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError, 'the index alone is not enough: the link re-indexes the name'); - assert.ok(exists(index), '(and it has: the index is back)'); - fs.unlinkSync(index); fs.unlinkSync(link); + assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError, 'the index alone is not enough: the link is a trace of the name'); + // 0.14.3: a halt writes nothing (632b95d re-created the index from the link on its way to the halt). + assert.ok(!exists(index), 'the halt did not write the index back'); + fs.unlinkSync(link); const again = cfg.loadIdentity({ name }); assert.notStrictEqual(again.nodeId, first.nodeId, 'a different agent, as the message says'); }); @@ -379,25 +398,43 @@ describe('the safety net: the mint path never buries a 0.13 identity', () => { }); }); - it('F7: a stray symlink at nodes/ that leads to an identity but is not a by-id link does not read as indexed, and halts', () => { + it('F7 (changed in 0.14.3, R1): a symlink at nodes/ that is not a compat link is a 0.13 directory kept elsewhere: it is followed and copied, its target untouched', () => { + // 632b95d halted here, and minted when the link was a 0.13 node's directory on a durable volume. A link that is + // not `by-id/` is the agent's own directory: copy what it leads to, set the link aside, never touch the target. const name = uniq('f7'); const elsewhere = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sym-stray-')), 'agent'); const id = legacyNode(name, { where: elsewhere }); + const before = treeOf(elsewhere); fs.mkdirSync(NODES_DIR, { recursive: true }); fs.symlinkSync(elsewhere, path.join(NODES_DIR, name), 'dir'); - assert.strictEqual(cfg.migrateNodeDir(name), 'not-a-node', 'nothing was indexed, so not "already"'); - assert.strictEqual(cfg.nodeIdForName(name), null); - assert.throws(() => cfg.loadIdentity({ name }), IdentityHaltError); - assert.strictEqual(fs.readlinkSync(path.join(NODES_DIR, name)), elsewhere, 'the link is left alone'); - assert.strictEqual(JSON.parse(read(path.join(elsewhere, 'identity.json'))).privateKey, id.privateKey); + assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); + assert.strictEqual(cfg.nodeIdForName(name), id.nodeId); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + const dest = cfg.nodeDirById(id.nodeId); + assert.ok(!isLink(path.join(dest, 'cmbs')) && exists(path.join(dest, 'cmbs', 'x.json')), 'the store was copied, not linked'); + assert.deepStrictEqual(treeOf(elsewhere), before, 'what the link led to is byte-unchanged'); + assert.strictEqual(fs.readlinkSync(path.join(NODES_DIR, cfg.LEGACY_ASIDE, name)), elsewhere, 'the link itself was set aside'); + assert.strictEqual(cfg.migrateNodeDir(name), 'already'); }); - it('a hand-made nodes// with NO identity still becomes the new agent\'s directory (documented, unchanged)', () => { + it('a nodes// holding files and NO identity halts (changed in 0.14.3, R1-2): it is a trace of an agent, never a new one\'s', () => { + // 632b95d gave such a directory to a NEW identity. With SYM_IDENTITY_DIR's volume not mounted that is exactly an + // existing agent's store whose key is elsewhere: the new identity took its store and the original was orphaned. const name = uniq('nodir-id'); put(path.join(NODES_DIR, name, 'cmbs', 'kept.json'), '{"k":1}'); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityHaltError && /holds a node's files/.test(e.reason)); + assert.strictEqual(identitiesNamed(name).length, 0, 'nothing was minted'); + assert.throws(() => cfg.loadIdentity({ name, create: false }), (e) => e.code === 'EIDENTITYABSENT' && /holds a node's files/.test(e.message), 'create: false says what is here'); + assert.strictEqual(read(path.join(NODES_DIR, name, 'cmbs', 'kept.json')), '{"k":1}'); + }); + + it('an EMPTY nodes// (or one with only a stale lock) is no trace: the new agent is made and the link takes its place', () => { + const name = uniq('empty-dir'); + fs.mkdirSync(path.join(NODES_DIR, name), { recursive: true }); + put(path.join(NODES_DIR, name, 'lock.pid'), '2147483646\n'); const id = cfg.loadIdentity({ name }); - assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'kept.json')), 'the store moved with the new identity'); assert.ok(isLink(path.join(NODES_DIR, name))); + assert.strictEqual(cfg.loadIdentity({ name, create: false }).nodeId, id.nodeId); }); it('a brand-new name still mints exactly one identity', () => { @@ -423,9 +460,12 @@ describe('F10: a public key without a private key halts', () => { describe('F6: rewriting identity.json is atomic', () => { it('a failure while writing leaves the original key file intact, and no temp file behind', () => { + // 0.14.3: the migration never rewrites a 0.13 file (it copies, normalised); the rewrite is of an installed identity. const name = uniq('f6'); - const id = legacyNode(name, { identity: makeIdentity(name, { hostname: 'bare-host' }) }); - const f = path.join(NODES_DIR, name, 'identity.json'); + const id = cfg.loadIdentity({ name }); + const f = path.join(cfg.nodeDirById(id.nodeId), 'identity.json'); + const onDisk = JSON.parse(read(f)); onDisk.hostname = 'bare-host'; + put(f, JSON.stringify(onDisk, null, 2), 0o600); const before = read(f); let injected = 0; const tempFds = new Set(); // the fds of identity.json's temp file: the lock file's writes are left alone @@ -441,29 +481,41 @@ describe('F6: rewriting identity.json is atomic', () => { })); assert.ok(injected >= 1, 'the failure was reached, in the write of the key file'); assert.strictEqual(read(f), before, 'the key file is byte-identical: it was never truncated'); - assert.deepStrictEqual(fs.readdirSync(path.join(NODES_DIR, name)).filter((n) => n.endsWith('.tmp')), []); + assert.deepStrictEqual(fs.readdirSync(path.dirname(f)).filter((n) => n.endsWith('.tmp')), []); assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'and the next start succeeds with the same key'); }); }); -describe('F9: a move across filesystems copies, verifies, then removes', () => { - it('EXDEV on the identity dir: the identity arrives intact at 0600 and the source is gone', () => { +/** Which tree a path is in: a rename between two of them would cross a volume when each is mounted on its own. */ +function volumeOf(p, idRoot) { + const roots = [path.join(idRoot, 'by-id'), idRoot, path.join(NODES_DIR, 'by-id'), NODES_DIR]; + return roots.find((r) => String(p) === r || String(p).startsWith(r + path.sep)) || 'other'; +} + +describe('F9: a copy across filesystems: copies, verifies, commits; the source is kept', () => { + it('EXDEV between the trees: the identity arrives intact at 0600, and the source is set aside, not deleted', () => { + // 0.14.3: no rename ever crosses from a 0.13 directory into by-id (the staging is made beside its destination), + // so every rename across the trees below fails with EXDEV here, and the migration still completes. withIdentityDir((idRoot) => { const name = uniq('f9'); const id = makeIdentity(name); put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o644); put(path.join(idRoot, name, 'sub', 'a.txt'), 'a'); put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); + let crossed = 0; const result = patchFs('renameSync', (orig) => function (a, b) { - if (String(a).startsWith(idRoot) && String(b).startsWith(idRoot)) throw exdev('simulated'); + if (volumeOf(a, idRoot) !== volumeOf(b, idRoot)) { crossed++; throw exdev('simulated'); } return orig.call(fs, a, b); }, () => cfg.migrateNodeDir(name)); + assert.strictEqual(crossed, 0, 'no rename tried to cross a volume'); assert.strictEqual(result, 'moved'); const dest = path.join(idRoot, 'by-id', id.nodeId); assert.strictEqual(JSON.parse(read(path.join(dest, 'identity.json'))).privateKey, id.privateKey); assert.strictEqual(read(path.join(dest, 'sub', 'a.txt')), 'a'); if (posix) assert.strictEqual(modeOf(path.join(dest, 'identity.json')), 0o600); - assert.ok(!exists(path.join(idRoot, name, 'identity.json')) || isLink(path.join(idRoot, name)), 'the source is gone (only the compat link is left)'); + assert.ok(isLink(path.join(idRoot, name)), 'SYM_IDENTITY_DIR/ is the compat link'); + assert.strictEqual(JSON.parse(read(path.join(idRoot, cfg.LEGACY_ASIDE, name, 'identity.json'))).privateKey, id.privateKey, 'the 0.13 source is kept, set aside'); + if (posix) assert.strictEqual(modeOf(path.join(idRoot, cfg.LEGACY_ASIDE, name, 'identity.json')), 0o600, 'and narrowed to its owner'); assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); }); }); @@ -475,13 +527,12 @@ describe('F9: a move across filesystems copies, verifies, then removes', () => { put(path.join(idRoot, name, 'identity.json'), JSON.stringify(id, null, 2), 0o600); put(path.join(NODES_DIR, name, 'cmbs', 'x.json'), '{}'); const src = read(path.join(idRoot, name, 'identity.json')); - assert.throws(() => patchFs('renameSync', (orig) => function (a, b) { - if (String(a).startsWith(idRoot) && String(b).startsWith(idRoot)) throw exdev('simulated'); - return orig.call(fs, a, b); - }, () => patchFs('copyFileSync', () => function () { throw Object.assign(new Error('EIO: injected'), { code: 'EIO' }); }, - () => cfg.migrateNodeDir(name))), /EIO/); + assert.throws(() => patchFs('copyFileSync', () => function () { throw Object.assign(new Error('EIO: injected'), { code: 'EIO' }); }, + () => cfg.migrateNodeDir(name)), /EIO/); assert.strictEqual(read(path.join(idRoot, name, 'identity.json')), src, 'the identity is where it was'); assert.ok(!exists(path.join(idRoot, 'by-id', id.nodeId)), 'no half-copy'); + assert.ok(!exists(cfg.nodeDirById(id.nodeId)), 'and no node directory installed either'); + assert.deepStrictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).filter((n) => n.includes(id.nodeId)), [], 'no staging left behind'); // the fault clears: the next run completes the move assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); assert.ok(exists(path.join(idRoot, 'by-id', id.nodeId, 'identity.json'))); @@ -509,14 +560,14 @@ describe('F9 on a real second filesystem (Linux: nodes/by-id lives on /dev/shm, fs.writeFileSync(path.join(dir, 'identity.json'), JSON.stringify(id, null, 2), { mode: 0o644 }); fs.chmodSync(path.join(dir, 'identity.json'), 0o644); fs.writeFileSync(path.join(dir, 'e2e-keypair.json'), '{}'); fs.chmodSync(path.join(dir, 'e2e-keypair.json'), 0o644); fs.writeFileSync(path.join(dir, 'cmbs', 'x.json'), '{"x":1}'); - let sawExdev = false; + let sawExdev = false; // 0.14.3 never renames across the volumes: every commit is a rename beside its destination const orig = fs.renameSync; fs.renameSync = function (a, b) { try { return orig.call(fs, a, b); } catch (e) { if (e.code === 'EXDEV') sawExdev = true; throw e; } }; const loaded = c.loadIdentity({ name }); const dest = c.nodeDirById(id.nodeId); console.log(JSON.stringify({ sawExdev, same: loaded.nodeId === id.nodeId && loaded.privateKey === id.privateKey, - sourceGone: !fs.existsSync(path.join(dir, 'cmbs')) || fs.lstatSync(dir).isSymbolicLink(), + sourceKept: fs.lstatSync(dir).isSymbolicLink() && fs.readFileSync(path.join(c.NODES_DIR, c.LEGACY_ASIDE, name, 'cmbs', 'x.json'), 'utf8') === '{"x":1}', store: fs.readFileSync(path.join(dest, 'cmbs', 'x.json'), 'utf8'), idMode: fs.statSync(path.join(dest, 'identity.json')).mode & 0o777, e2eMode: fs.statSync(path.join(dest, 'e2e-keypair.json')).mode & 0o777, dirMode: fs.statSync(dest).mode & 0o777, onOtherFs: fs.realpathSync(dest).startsWith(fs.realpathSync(${JSON.stringify(far)})), @@ -532,9 +583,9 @@ describe('F9 on a real second filesystem (Linux: nodes/by-id lives on /dev/shm, }); const r = JSON.parse(out); assert.strictEqual(r.onOtherFs, true, 'setup: the node lives on the other filesystem'); - assert.strictEqual(r.sawExdev, true, 'the kernel really refused the rename'); + assert.strictEqual(r.sawExdev, false, 'no rename crossed the volumes: each copy is staged beside its destination'); assert.strictEqual(r.same, true); - assert.strictEqual(r.sourceGone, true); + assert.strictEqual(r.sourceKept, true, 'the 0.13 directory is set aside, whole'); assert.strictEqual(r.store, '{"x":1}'); assert.strictEqual(r.idMode, 0o600); assert.strictEqual(r.e2eMode, 0o600); @@ -564,88 +615,88 @@ describe('F11: the window between the move and the index is not a mint race', () assert.strictEqual(identitiesNamed(name).length, 1); }); - it('an interrupted move is resumed by the next start: the identity moves last, and its absence re-enters the move', () => { + it('an interrupted migration leaves the 0.13 directory as it was, and the next start completes it (changed in 0.14.3: a copy, committed by one rename)', () => { + // 632b95d merged entry by entry, so an interruption left the store split between the two places. Now nothing is + // moved: the copy is staged, and a failure before the commit installs nothing and changes nothing in the source. const name = uniq('f11r'); const id = legacyNode(name); - put(path.join(NODES_DIR, name, 'aaa.json'), '{}'); put(path.join(NODES_DIR, name, 'zzz.json'), '{}'); // names on both sides of identity.json - husk(id.nodeId); // forces a merge, entry by entry + put(path.join(NODES_DIR, name, 'aaa.json'), '{}'); put(path.join(NODES_DIR, name, 'zzz.json'), '{}'); + husk(id.nodeId); + const before = treeOf(path.join(NODES_DIR, name)); patchFs('renameSync', (orig) => function (a, b) { - if (path.basename(String(a)) === 'identity.json' && String(a).startsWith(NODES_DIR)) throw Object.assign(new Error('EIO: injected'), { code: 'EIO' }); + if (String(b) === cfg.nodeDirById(id.nodeId)) throw Object.assign(new Error('EIO: injected at the commit'), { code: 'EIO' }); return orig.call(fs, a, b); }, () => { assert.throws(() => cfg.migrateNodeDir(name), /EIO/); }); - assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the identity had not moved yet'); - for (const f of ['aaa.json', 'zzz.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), f)), `but the rest had: ${f}`); - assert.strictEqual(cfg.nodeIdForName(name), id.nodeId, 'the name was indexed first'); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, name)), before, 'the 0.13 directory is byte-unchanged'); + assert.ok(!exists(path.join(cfg.nodeDirById(id.nodeId), 'identity.json')), 'nothing was installed'); + assert.strictEqual(cfg.nodeIdForName(name), null, 'and the name was not claimed'); + assert.deepStrictEqual(fs.readdirSync(path.join(NODES_DIR, 'by-id')).filter((n) => n.startsWith(`.${id.nodeId}.partial-`)), [], 'the staging was removed'); + // a staging left by a process that was killed is cleared by the next attempt + const dead = path.join(NODES_DIR, 'by-id', `.${id.nodeId}.partial-2147483646-deadbeef`); + put(path.join(dead, 'identity.json'), '{"junk":1}'); const loaded = cfg.loadIdentity({ name }); assert.strictEqual(loaded.nodeId, id.nodeId); assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(!exists(dead), 'the dead process\'s staging is gone'); + for (const f of ['aaa.json', 'zzz.json', path.join('cmbs', 'x.json')]) assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), f)), `the whole store: ${f}`); assert.ok(isLink(path.join(NODES_DIR, name)), 'finished: nodes/ is the compat link'); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, cfg.LEGACY_ASIDE, name)).filter((l) => !l.includes(' lock.pid')), before.filter((l) => !l.includes(' lock.pid')), 'and the 0.13 directory is kept, set aside, as it was'); assert.strictEqual(identitiesNamed(name).length, 1); }); }); -describe('F11: one migrator at a time', () => { - /** A second process that holds the migration lock for `ms`, then moves the node itself and lets go. */ - function holder(name, ms) { +describe('F11 (0.14.3): no migration lock; concurrent starts are made safe by the staged commit', () => { + /** A process that holds `lockPath` (written as `meta` says) until killed. */ + function holder(dir, meta = {}) { return spawn(process.execPath, ['-e', ` + const fs = require('fs'), path = require('path'); const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); - const release = c.acquireIdentityLock('migration', { dir: ${JSON.stringify(path.join(NODES_DIR, '.migrating'))} }); - console.log('held'); - setTimeout(() => { c.migrateNodeDir(${JSON.stringify(name)}); release(); console.log('released:' + Date.now()); process.exit(0); }, ${ms});`], - { stdio: ['ignore', 'pipe', 'inherit'] }); + fs.mkdirSync(${JSON.stringify(dir)}, { recursive: true }); + fs.writeFileSync(path.join(${JSON.stringify(dir)}, 'lock.pid'), process.pid + '\\n' + JSON.stringify({ start: c.processStartTime(process.pid), createdAt: Date.now(), ...${JSON.stringify(meta)} }) + '\\n'); + console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); } const held = (child) => new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('the holder exited'))); }); - /** When the holder let go (its clock, ms), read after the fact. */ - const releasedAt = (child) => new Promise((resolve, reject) => { - let buf = ''; - child.stdout.on('data', (d) => { buf += d; const m = /released:(\d+)/.exec(buf); if (m) resolve(Number(m[1])); }); - child.once('exit', () => { const m = /released:(\d+)/.exec(buf); if (m) resolve(Number(m[1])); else reject(new Error('the holder never released')); }); - }); - it('a start waits for the process that is moving the node, then finds it moved', async () => { - const name = uniq('f11-wait'); + it('a start beside another start\'s migration of the same agent does not wait for it: it stages its own copy, and one commit wins', async () => { + // 632b95d serialised every migration in the process tree behind nodes/.migrating, blocking the event loop of every + // node in a multi-node process for up to 30 s. The 0.13 directory's own lock, held by a 0.14.3 migration, is passed. + const name = uniq('f11-beside'); const id = legacyNode(name); - husk(id.nodeId); // a merge, not a single rename - const child = holder(name, 1000); + const child = holder(path.join(NODES_DIR, name), { migrating: true, token: 'other' }); try { await held(child); - const loaded = cfg.loadIdentity({ name }); // blocks until the holder lets go - const returnedAt = Date.now(); - const released = await releasedAt(child); - assert.ok(returnedAt >= released, `it returned (${returnedAt}) only after the holder let go (${released}): it waited, it did not move the node alongside it`); - assert.strictEqual(loaded.nodeId, id.nodeId); + const t0 = Date.now(); + const loaded = cfg.loadIdentity({ name }); + assert.ok(Date.now() - t0 < 5000, `no wait (${Date.now() - t0} ms)`); assert.strictEqual(loaded.privateKey, id.privateKey); - assert.strictEqual(identitiesNamed(name).length, 1); - assert.ok(isLink(path.join(NODES_DIR, name)), 'the holder finished the move'); - assert.ok(!exists(path.join(NODES_DIR, '.migrating', 'lock.pid')), 'and the lock was let go'); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'x.json')), 'migrated, with its store'); + assert.ok(!exists(path.join(NODES_DIR, '.migrating')), 'no migration lock exists'); } finally { child.kill(); } }); - it('the lock of a process that is gone is reclaimed, not waited for', () => { + it('a start is not held up by a 632b95d-era migration lock (nodes/.migrating) that another process holds', async () => { + const name = uniq('f11-oldlock'); + const id = legacyNode(name); + const child = holder(path.join(NODES_DIR, '.migrating')); + try { + await held(child); + if (cfg._setMigrationLockWait) cfg._setMigrationLockWait(3000); // (632b95d only: so its wait fails in 3 s, not 30) + const t0 = Date.now(); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + assert.ok(Date.now() - t0 < 2500, `no wait (${Date.now() - t0} ms)`); + } finally { if (cfg._setMigrationLockWait) cfg._setMigrationLockWait(30000); child.kill(); } + }); + + it('a stale 0.13 lock (its process is gone) is reclaimed, not waited for', () => { const name = uniq('f11-stale'); const id = legacyNode(name); - put(path.join(NODES_DIR, '.migrating', 'lock.pid'), '2147483646\n', 0o644); // a pid that is not running + put(path.join(NODES_DIR, name, 'lock.pid'), '2147483646\n', 0o644); // a pid that is not running const t0 = Date.now(); assert.strictEqual(cfg.migrateNodeDir(name), 'moved'); assert.ok(Date.now() - t0 < 2000, 'no wait'); assert.strictEqual(cfg.loadIdentity({ name, create: false }).privateKey, id.privateKey); }); - it('a holder that never lets go makes the start fail loudly after the wait, and moves nothing', async () => { - const name = uniq('f11-timeout'); - const id = legacyNode(name); - const child = holder(name, 60000); - try { - await held(child); - cfg._setMigrationLockWait(300); - assert.throws(() => cfg.migrateNodeDir(name), /has been moving 0\.13 nodes/); - assert.throws(() => cfg.loadIdentity({ name }), /has been moving 0\.13 nodes/, 'and a start does not make a new identity meanwhile'); - assert.ok(exists(path.join(NODES_DIR, name, 'identity.json')), 'the node is where it was'); - assert.strictEqual(identitiesNamed(name).length, 0); - assert.ok(id.privateKey); - } finally { cfg._setMigrationLockWait(30000); child.kill(); } - }); - it('nothing to move: no lock is taken and nothing is written', () => { const name = uniq('f11-nolock'); const id = cfg.loadIdentity({ name }); // a new agent: indexed, nodes/ is the link @@ -761,22 +812,47 @@ describe('F11 with real processes: several starters at once never make a second } }); -describe('F13: the live-lock fallback checks the tombstone', () => { - it('an identity exported away does not load from an unmoved 0.13 directory a live process holds', async () => { - const name = uniq('f13'); - const id = legacyNode(name); +describe('R1-5: a live 0.13 process in the agent\'s 0.13 directory refuses the start (EIDENTITYLOCK)', () => { + // 632b95d loaded the identity "in place" and the 0.14 node ran beside the 0.13 one: the same identity twice, the + // 0.14 one on an empty by-id store. One agent runs in one process. + async function liveLegacy(name) { const dir = path.join(NODES_DIR, name); const child = spawn(process.execPath, ['-e', ` const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); c.acquireIdentityLock(${JSON.stringify(name)}, { dir: ${JSON.stringify(dir)} }); console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); - try { - await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('lock holder exited'))); }); - assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-live', 'setup: a live process holds the old directory'); - assert.strictEqual(cfg.loadIdentity({ name }).nodeId, id.nodeId, 'not tombstoned: it loads in place'); - put(path.join(cfg.nodeDirById(id.nodeId), cfg.TOMBSTONE), JSON.stringify({ movedAt: Date.now() })); - assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityTombstonedError); - } finally { child.kill(); } + await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('lock holder exited'))); }); + return child; + } + + for (const pinned of [false, true]) { + it(`${pinned ? 'pinned' : 'by name'}: refused with EIDENTITYLOCK naming the holder, nothing moved or made; after it stops, the agent migrates`, async () => { + const name = uniq(pinned ? 'r15-pin' : 'r15'); + const id = legacyNode(name); + const before = treeOf(path.join(NODES_DIR, name)); + const child = await liveLegacy(name); + const opts = pinned ? { name, nodeId: id.nodeId, create: false } : { name }; + try { + assert.strictEqual(cfg.migrateNodeDir(name), 'skipped-live'); + assert.throws(() => cfg.loadIdentity(opts), (e) => e.code === 'EIDENTITYLOCK' && e.holderPid === child.pid); + assert.ok(!exists(cfg.nodeDirById(id.nodeId)), 'no by-id directory was made for a second copy to run in'); + assert.strictEqual(cfg.nodeIdForName(name), null); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, name)).filter((l) => !l.includes(' lock.pid')), before); + if (pinned) assert.throws(() => cfg.loadIdentity({ name, nodeId: cfg.uuidv7(), create: false }), (e) => e.code === 'EIDENTITYABSENT', 'a different pin does not get it'); + } finally { child.kill('SIGKILL'); await new Promise((r) => child.once('exit', r)); } + const loaded = cfg.loadIdentity(opts); + assert.strictEqual(loaded.privateKey, id.privateKey); + assert.ok(exists(path.join(cfg.nodeDirById(id.nodeId), 'cmbs', 'x.json')), 'migrated with its store once the 0.13 process is gone'); + }); + } + + it('an identity this host exported does not load from an unmigrated 0.13 directory (tombstone checked)', () => { + const name = uniq('f13'); + const id = legacyNode(name); + put(path.join(cfg.nodeDirById(id.nodeId), cfg.TOMBSTONE), JSON.stringify({ movedAt: Date.now() })); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e instanceof IdentityTombstonedError); + assert.throws(() => cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }), (e) => e instanceof IdentityTombstonedError); + assert.ok(!isLink(path.join(NODES_DIR, name)), 'and it was not migrated'); }); }); @@ -862,34 +938,18 @@ describe('a start pinned to a nodeId migrates too (mesh-channel starts every nod assert.strictEqual(cfg.nodeIdForName(name), null, 'the pinned start wrote no index entry for the name'); }); - it('the name is indexed to another agent and the pin\'s 0.13 directory sits under it: the pin stays absent, and the start does not answer with the indexed agent', () => { + it('the name is indexed to another agent and the pin\'s 0.13 directory sits under it: the start halts saying the pin IS here (changed in 0.14.3), and does not answer with the indexed agent', () => { + // 632b95d threw IdentityAbsentError ("No identity ... on this host") while the pinned identity sat in nodes/. const name = uniq('pin-held'); const indexed = cfg.loadIdentity({ name }); // N -> indexed fs.unlinkSync(path.join(NODES_DIR, name)); // the link out of the way: a real directory for the pin const pin = legacyNode(name); const before = read(path.join(NODES_DIR, name, 'identity.json')); - assert.throws(() => cfg.loadIdentity({ name, nodeId: pin.nodeId, create: false }), (e) => e.code === 'EIDENTITYABSENT'); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin.nodeId, create: false }), + (e) => e instanceof IdentityHaltError && e.reason.includes(pin.nodeId) && /is on this host/.test(e.reason) && e.reason.includes(indexed.nodeId)); assert.strictEqual(read(path.join(NODES_DIR, name, 'identity.json')), before, 'the pin\'s key is where it was'); assert.strictEqual(cfg.nodeIdForName(name), indexed.nodeId, 'and the index still says what it said'); }); - - it('a 0.13 process holds the pinned agent\'s directory: it loads in place, and a tombstone still refuses it', async () => { - const name = uniq('pin-live'); - const id = legacyNode(name); - const dir = path.join(NODES_DIR, name); - const child = spawn(process.execPath, ['-e', ` - const c = require(${JSON.stringify(path.resolve(__dirname, '../lib/config.js'))}); - c.acquireIdentityLock(${JSON.stringify(name)}, { dir: ${JSON.stringify(dir)} }); - console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); - try { - await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('lock holder exited'))); }); - assert.strictEqual(cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }).privateKey, id.privateKey); - assert.ok(exists(path.join(dir, 'cmbs', 'x.json')) && !isLink(dir), 'unmoved: its process is running there'); - assert.throws(() => cfg.loadIdentity({ name, nodeId: cfg.uuidv7(), create: false }), (e) => e.code === 'EIDENTITYABSENT', 'a different pin does not get it'); - put(path.join(cfg.nodeDirById(id.nodeId), cfg.TOMBSTONE), JSON.stringify({ movedAt: Date.now() })); - assert.throws(() => cfg.loadIdentity({ name, nodeId: id.nodeId, create: false }), (e) => e instanceof IdentityTombstonedError); - } finally { child.kill(); } - }); }); describe('T9: the checks measure something', () => { From b5bb5e180c654c62068e81b301929de7bcbd05ca Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 16:29:20 +0100 Subject: [PATCH 09/15] Identity: a start beside another start's migration loads what that start installed A start copying the 0.13 directory while another start sets it aside can see it go mid-copy. That start's error is not the agent's state: the other start installed the identity and claimed the name, so loadIdentity reads it by the index instead of failing. Tests: four guard tests for the staged commit (a copy that does not read back is not installed; a linked subdirectory is copied as a directory; a husk with a live lock refuses; a live process's staging is left alone). --- lib/config.js | 9 +++++- tests/identity-commit.test.js | 58 +++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+), 1 deletion(-) diff --git a/lib/config.js b/lib/config.js index b338764d..957219bb 100644 --- a/lib/config.js +++ b/lib/config.js @@ -1462,7 +1462,14 @@ function loadIdentity({ nodeId: pin, name, create = true, allowPendingExport = f // 1. A 0.13 identity under the name is migrated first: the same path for a pinned start and an unpinned one. let migration = null; if (legacyIdentityUnder(name)) { - migration = migrateName(name, pin); + try { + migration = migrateName(name, pin); + } catch (e) { + // Another start finished this migration while this one was copying (it set the 0.13 directory aside under + // this copy): what it installed is read below, by the name it claimed. Anything else is this start's to report. + if (e instanceof IdentityHaltError || e.code === 'EIDENTITYLOCK' || legacyIdentityUnder(name)) throw e; + migration = { status: 'already' }; + } if (migration.status === 'skipped-live') { throw lockError( `[SYM] Agent '${name}' is running in its 0.13 directory: a live process (PID ${migration.holderPid}) holds ${path.join(NODES_DIR, name, 'lock.pid')}.\n\n` + diff --git a/tests/identity-commit.test.js b/tests/identity-commit.test.js index e70eb28a..fa4ab3f5 100644 --- a/tests/identity-commit.test.js +++ b/tests/identity-commit.test.js @@ -242,6 +242,64 @@ describe('R1-3: what a 0.13 node kept behind a link is followed and copied', () }); }); +describe('the staged commit: what it checks before the one rename', () => { + it('a copy that does not read back as its source is not installed, and the source is untouched', () => { + const name = uniq('verify'); + const id = legacyNode(name); + const before = treeOf(path.join(NODES_DIR, name)); + assert.throws(() => patchFs('copyFileSync', (o) => function (a, b, ...r) { + const out = o.call(fs, a, b, ...r); + if (String(b).endsWith('c3.json')) fs.appendFileSync(b, ' '); // a copy that came out different + return out; + }, () => cfg.migrateNodeDir(name)), /does not match/); + assert.ok(!exists(cfg.nodeDirById(id.nodeId)), 'nothing installed'); + assert.deepStrictEqual(treeOf(path.join(NODES_DIR, name)), before, 'the source is untouched'); + assert.strictEqual(cfg.nodeIdForName(name), null); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey, 'the next start copies it right'); + }); + + it('a directory inside the 0.13 node that is a relative link is copied as the directory it leads to', () => { + const name = uniq('sublink'); + const id = makeIdentity(name); + put(path.join(NODES_DIR, name, 'identity.json'), JSON.stringify(id)); + put(path.join(NODES_DIR, `${name}-store`, 'x.json'), '{"x":1}'); + fs.symlinkSync(`../${name}-store`, path.join(NODES_DIR, name, 'cmbs'), 'dir'); + cfg.loadIdentity({ name }); + const cmbs = path.join(cfg.nodeDirById(id.nodeId), 'cmbs'); + assert.ok(!isLink(cmbs) && read(path.join(cmbs, 'x.json')) === '{"x":1}', 'a real directory holding the files'); + }); + + it('a by-id husk whose lock a live process holds is not replaced: EIDENTITYLOCK, nothing changed', async () => { + const name = uniq('husk-live'); + const id = legacyNode(name); + const husk = cfg.nodeDirById(id.nodeId); + fs.mkdirSync(husk, { recursive: true }); + put(path.join(husk, 'inbox.json'), '{"seq":99}'); + const child = spawn(process.execPath, ['-e', ` + const c = require(${JSON.stringify(path.join(LIB, 'config.js'))}); + c.acquireIdentityLock('h', { dir: ${JSON.stringify(husk)} }); console.log('held'); setInterval(() => {}, 1000);`], { stdio: ['ignore', 'pipe', 'inherit'] }); + try { + await new Promise((resolve, reject) => { child.stdout.once('data', resolve); child.once('exit', () => reject(new Error('holder exited'))); }); + assert.throws(() => cfg.loadIdentity({ name }), (e) => e.code === 'EIDENTITYLOCK'); + assert.strictEqual(read(path.join(husk, 'inbox.json')), '{"seq":99}', 'the husk is where it was'); + assert.ok(!exists(path.join(husk, 'identity.json'))); + assert.ok(!isLink(path.join(NODES_DIR, name)), 'the 0.13 directory is where it was'); + } finally { child.kill('SIGKILL'); await new Promise((r) => child.once('exit', r)); } + }); + + it('a staging directory of a live process is left alone; one of a process that is gone is cleared', () => { + const name = uniq('partials'); + const id = legacyNode(name); + const live = path.join(NODES_DIR, 'by-id', `.${id.nodeId}.partial-${process.ppid}-aaaaaaaa`); // our parent is alive + const dead = path.join(NODES_DIR, 'by-id', `.${id.nodeId}.partial-2147483646-bbbbbbbb`); + put(path.join(live, 'x'), '1'); put(path.join(dead, 'x'), '1'); + cfg.loadIdentity({ name }); + assert.ok(exists(live), 'a live process\'s staging is not touched'); + assert.ok(!exists(dead), 'a dead one\'s is cleared'); + fs.rmSync(live, { recursive: true, force: true }); + }); +}); + describe('R1-4: a stale identity lock is removed only if it is still the lock judged stale', () => { it('a start that judged a dead holder\'s lock stale does not remove the lock another start took meanwhile', async () => { // 632b95d read the lock, judged it stale, unlinked whatever was there, and created its own: with six starters at a From 92cfb0fb8ea525fc1b31e45cc0856107bfafbfab Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 16:30:12 +0100 Subject: [PATCH 10/15] relocation: an import never takes this host's own state files (lock, tombstone, export intent, commit record) from a bundle --- lib/relocation.js | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/relocation.js b/lib/relocation.js index cddbf3c6..25701ed0 100644 --- a/lib/relocation.js +++ b/lib/relocation.js @@ -474,8 +474,11 @@ function importNode({ from, passphrase, hostPrivateKey, expect, name } = {}) { config.writeNewFileSynced(path.join(root, ...parts), Buffer.from(f.data, 'base64'), mode); }; const archive = (d) => { if (fs.existsSync(d)) { fs.renameSync(d, path.join(path.dirname(d), `.${identity.nodeId}.moved-${Date.now()}`)); fsyncDir(path.dirname(d)); } }; + // What a node directory holds that is this host's own state (a lock, a tombstone, an export intent, the commit + // record) is never taken from a bundle. + const hostState = (rel) => SKIP.has(rel) || rel.startsWith('lock.pid.'); config.installNewDir(dest, identity.nodeId, (staging) => { - for (const f of plain.files) if (!(split && String(f.path) === 'identity.json')) write(staging, f); + for (const f of plain.files) if (!(split && String(f.path) === 'identity.json') && !hostState(String(f.path))) write(staging, f); }, 'import', { beforeCommit: () => archive(dest) }); // the tombstoned copy of a node coming back is kept aside if (split) { config.installNewDir(idDest, identity.nodeId, (staging) => write(staging, idFile), 'import', { beforeCommit: () => archive(idDest) }); From 4a845e0fe3b2a43a0dae31ea2447ecf14699a0b7 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 16:39:45 +0100 Subject: [PATCH 11/15] Identity: a start pinned to a nodeId never takes a name from another identity (R4-9) loadIdentity({ name, nodeId: pin }) re-pointed the name's index entry to the pin whenever it named another identity. mesh-channel pins node_id from a repository's .sym/node.json, so a folder pinning X under "reviewer" silently took the name from another folder's agent Y: Y's next start by name ran as X and Y was left without its name, and a cloned repository could choose which identity on the host a session signs as. Now a pinned start may claim a name nobody has, and that is all. A name whose entry names another identity halts (IdentityHaltError, saying both and how to resolve it) and the entry is left as it is; with create: false and the pin not on this host it is IdentityAbsentError, saying the name is another agent's. A pinned start is never answered with the name's agent. --- lib/config.js | 31 +++++++++++++++++++++++-------- tests/identity-commit.test.js | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+), 8 deletions(-) diff --git a/lib/config.js b/lib/config.js index 957219bb..a6a685c9 100644 --- a/lib/config.js +++ b/lib/config.js @@ -453,10 +453,18 @@ function claimName(name, nodeId) { throw new Error(`[SYM] the name index entry for "${name}" keeps appearing and disappearing; nothing was changed`); } -/** Point `name` at `nodeId` whatever it named (a start pinned to a nodeId: known, kept), durably. */ -function repointName(name, nodeId) { - ensureDir(path.join(NODES_DIR, 'by-name')); - writeFileDurable(nameIndexPath(name), JSON.stringify({ name, nodeId, at: Date.now() }), 0o644); +/** + * The halt for a start pinned to `pin` under a name whose index entry names another identity (`held`). The entry + * is left as it is: a name is one agent's on this host. + */ +function nameHeldByAnother(name, pin, held) { + const there = identityInPlace(held); + return new IdentityHaltError(name, nameIndexPath(name), + `this start is pinned to ${pin}, and the name "${name}" is another agent's on this host: its index entry names ${held}` + + `${there ? '' : ' (whose identity is not installed here)'}. A pinned start does not take a name from another identity: the next start of ` + + `"${name}" by name would run as ${pin}, and ${held} would be left without its name. The index entry was left as it is.`, + `Start ${pin} under its own name, or pin the start to ${held} if that is the agent meant. To give "${name}" to ${pin}, ` + + `rename the other agent first (renameIdentity).`); } /** Every nodeId some index entry names, with the names. Entries that cannot be read are left out. */ @@ -1479,11 +1487,15 @@ function loadIdentity({ nodeId: pin, name, create = true, allowPendingExport = f if (migration.status === 'tombstoned') throw new IdentityTombstonedError(migration.nodeId, migration.tomb); } if (pin) { + // A pinned start never takes a name from another agent (R4-9): it may claim a name nobody has, and that is all. + // Re-pointing the entry made the other agent's next start by name run as the pinned identity and orphaned the + // other one, and let a pin in a cloned repository's node.json pick which identity on the host a session signs as. + const held = nodeIdForName(name); if (identityInPlace(pin)) { + if (held && held !== pin) throw nameHeldByAnother(name, pin, held); const identity = loadInstalled(pin, name, how); - const held = nodeIdForName(name); - if (!held) claimName(name, pin); - else if (held !== pin) repointName(name, pin); // known: a pinned start re-points the name + const winner = held || claimName(name, pin); + if (winner !== pin) throw nameHeldByAnother(name, pin, winner); // another start claimed it this moment ensureCompatLinks(name, pin); return identity; } @@ -1494,8 +1506,11 @@ function loadIdentity({ nodeId: pin, name, create = true, allowPendingExport = f } if (!create) { if (strayCopiesOf(pin).length) throw indexedButMissing(name, pin, { indexed: false }); - throw new IdentityAbsentError(`"${name}" (${pin})`, nameTraces(name, migration).map((t) => t.text)); + const elsewhere = held && held !== pin ? [`the name "${name}" is another agent's here (${held}), and was left as it is`] : []; + throw new IdentityAbsentError(`"${name}" (${pin})`, [...elsewhere, ...nameTraces(name, migration).map((t) => t.text)]); } + // Not on this host, and the name is another identity's: the start is never answered with that one. + if (held && held !== pin) throw nameHeldByAnother(name, pin, held); } for (let round = 0; ; round++) { const indexed = nodeIdForName(name); diff --git a/tests/identity-commit.test.js b/tests/identity-commit.test.js index fa4ab3f5..1dfd51c8 100644 --- a/tests/identity-commit.test.js +++ b/tests/identity-commit.test.js @@ -608,6 +608,39 @@ describe('R1-11: names Windows cannot hold, and Windows junctions', () => { }); }); +describe('R4-9: a start pinned to a nodeId never takes a name from another identity', () => { + // 632b95d re-pointed the name's index entry to the pin: a folder whose .sym/node.json pins X under "reviewer" took the + // name from another folder's agent Y; Y's next start by name then ran as X, and Y was left without its name. + it('the pinned identity is here and the name is another agent\'s: the start halts, the entry is left as it is', () => { + const name = uniq('r49'); + const y = cfg.loadIdentity({ name }); // the agent the name belongs to + const x = cfg.loadIdentity({ name: uniq('r49-x') }); // another agent on the host + for (const create of [false, true]) { + assert.throws(() => cfg.loadIdentity({ name, nodeId: x.nodeId, create }), + (e) => e instanceof IdentityHaltError && e.reason.includes(x.nodeId) && e.reason.includes(y.nodeId), `create: ${create}`); + assert.strictEqual(cfg.nodeIdForName(name), y.nodeId, 'the index still names Y'); + } + assert.strictEqual(cfg.loadIdentity({ name }).nodeId, y.nodeId, 'the next start by name is Y'); + assert.strictEqual(fs.realpathSync(path.join(NODES_DIR, name)), fs.realpathSync(cfg.nodeDirById(y.nodeId)), 'and the compat link leads to Y'); + }); + + it('the pin is not on this host and the name is another agent\'s: never answered with that agent', () => { + const name = uniq('r49-absent'); + const y = cfg.loadIdentity({ name }); + const pin = cfg.uuidv7(); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin, create: true }), (e) => e instanceof IdentityHaltError && e.reason.includes(y.nodeId)); + assert.throws(() => cfg.loadIdentity({ name, nodeId: pin, create: false }), (e) => e.code === 'EIDENTITYABSENT' && e.message.includes(y.nodeId)); + assert.strictEqual(cfg.nodeIdForName(name), y.nodeId); + }); + + it('a name nobody has is claimed for the pinned identity', () => { + const x = cfg.loadIdentity({ name: uniq('r49-own') }); + const name = uniq('r49-free'); + assert.strictEqual(cfg.loadIdentity({ name, nodeId: x.nodeId, create: false }).nodeId, x.nodeId); + assert.strictEqual(cfg.nodeIdForName(name), x.nodeId); + }); +}); + describe('error texts: the operator is never told to delete what leads to a key that is on disk', () => { it('SYM_IDENTITY_DIR switched on after the agent was made: the halt says where the key is, and not to remove the index', () => { const name = uniq('err-iddir'); From 788f75e45beb2cc00882d936f64d628691c91462 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 16:40:17 +0100 Subject: [PATCH 12/15] CHANGELOG 0.14.3: identity and on-disk state, the design as built (numbers to follow) --- CHANGELOG.md | 159 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 99 insertions(+), 60 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f725f6a..cd992e8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,9 @@ **An agent no longer loses its key to the 0.13 → by-id upgrade.** An XMesh review of `lib/config.js` 0.14.2 returned "do not ship" with 13 findings. Each was checked by running it in a sandbox state dir; 0.14.3 fixes the ones that lose -or expose a key and the small ones beside them. The wire and the public surface are unchanged. +or expose a key and the small ones beside them, by reading the state of an identity only from records committed in one +atomic step (see "Identity and on-disk state"). The wire is unchanged; the public surface gains `relocation.abortExport` +and `IdentityExportPendingError`. ### What was wrong @@ -48,71 +50,108 @@ or expose a key and the small ones beside them. The wire and the public surface no key was lost, but it could not start until something started it by name. A pinned start with `create: true` could lose it as above. -### Changed - -- **The duplicate check looks at the identity, not the directory.** A second claim on a nodeId is a DIFFERENT - `identity.json` (or other content that cannot be merged) already filed under it. A directory that is empty, or holds a - stale lock, claims nothing: the node is MERGED into it. The same identity on both sides is an interrupted move, and - the next start finishes it. A real duplicate is still refused: nothing is moved and nothing is overwritten. -- **A start pinned to a nodeId migrates too.** `loadIdentity({ nodeId, name })` whose identity is not in place and for - which a 0.13 directory under `name` is still there runs the same locked, idempotent migration as a start by name, then - loads the pinned agent. It is one code path, not a copy, and it is not run when there is no 0.13 directory to move. - **A pin that is a different nodeId than the directory's does not migrate it:** the directory belongs to another agent, - so it is left exactly where it is, unindexed (nothing moved, nothing lost), the pinned identity stays absent, and the - start fails as it would have: `IdentityAbsentError` with `create: false`, and with `create: true` the halt that says a - 0.13 identity sits under the name (naming the pin). That agent moves when it is started by name or by its own pin. A - 0.13 process holding the pinned agent's directory still lets it load in place (tombstone checked), as for a start by name. -- **The name is indexed before anything moves, and the identity moves last.** A crash in the middle leaves the name - pointing at the agent and the identity still where the next start will find it, so an interrupted move is re-entered - by the identity's absence instead of being mistaken for "no such agent". -- **One process migrates at a time.** Moving a directory is many steps, and two processes interleaving them could each - delete what the other had just filed. Found by running four real processes at it with a stall injected into every - file call (about one round in 150 lost the identity file), after plain concurrent runs had passed. The migrator - now holds a lock (`nodes/.migrating/lock.pid`, the identity lock: reclaimed when its holder is gone) while it moves - something; the others wait (up to 30 s, then fail loudly), read the state again and find it done. It is taken only - when a 0.13 directory is there to move. Under that lock, in the same stall fuzzer: 1,200 starters in 300 rounds, 1,600 - in 400 rounds without a by-id directory in place, and 3,600 in 600 rounds with six starters at once: every one loaded - the original agent, no errors, the key in place every round. A path that has - become an alias of its destination is never deleted as a "duplicate" of it. The same four-process run on 0.14.2 - made a new identity in 14 of 100 starters, and destroyed the key in every round that had a by-id directory. -- **A start never makes an identity where one exists.** Before it makes one it checks that no 0.13 `identity.json` - sits under the name (in `nodes//` or `SYM_IDENTITY_DIR//`); if one does and could not be migrated it - halts (`IdentityHaltError`) and says why. A name that indexes a node whose identity is missing halts too. - `create: false` still throws `IdentityAbsentError`. Behaviour change: a node whose `identity.json` was deleted by hand - to "start over" now halts instead of silently becoming a new agent; to start the name again as a different agent, - remove the two paths the message names (`nodes/by-name/.json` and the link `nodes/`). -- **Key files are private.** After a migration `identity.json` and `e2e-keypair.json` are 0600 and the directories that - hold them 0700; loading an identity narrows an existing key file (and the E2E keypair beside it), so nodes already - migrated by 0.14.0 to 0.14.2 are healed on their next start. A new identity's directory is 0700. Best effort on a - file this process does not own, and a no-op on Windows. -- **A rewrite of `identity.json` is atomic** (a private temp file, synced, renamed over it). -- **A move across file systems copies, verifies byte for byte, then removes the source.** A failed copy is removed and - the source left as it was. -- An identity with a public key and no private key halts. A live 0.13 directory's identity is checked against the - tombstone. A symlink at `nodes/` that is not a by-id link is reported `not-a-node`. -- `SYM_IDENTITY_DIR` switched on for a 0.13 node whose identity is still in `nodes//` halts and tells the - operator to copy it to `SYM_IDENTITY_DIR//identity.json`. It does not move it for them (see below). +### Identity and on-disk state + +The first fix for the findings above was itself reviewed (R1, on the candidate 632b95d) and failed the same way twice +over: it read the state of an identity from whatever was at a path (a by-id directory that was only partly there, an +index entry that could not be read, a node directory without `identity.json`, the content of a stale lock), and each +patch for one of those cases made another. R1 proved eleven by running them. 0.14.3 changes the assumption instead: +**the state of an identity is a record committed in one atomic step**, and the code that inferred it is deleted +(`moveTree`, `mergeMove`, `sameTree`, `legacyDirsRemain`, and the migration lock `nodes/.migrating`). + +- **The 0.13 → by-id migration copies; it never moves.** The 0.13 directory (`nodes//`, and + `SYM_IDENTITY_DIR//`) is copied into a staging directory beside its destination + (`nodes/by-id/..partial--`, on the destination's volume), links followed (a node directory kept + on another volume behind a link, an `identity.json` that is a relative link to a key file), `identity.json` last, every + file synced, the copy verified byte for byte against the source, a commit record (`.sym-commit`) written, and the + whole installed by ONE rename. Of several starts doing this at once, one rename wins; the others find the same + identity installed and discard their staging. Then the 0.13 directory is renamed aside to + `nodes/.legacy-0.13/` (key files narrowed to 0600) and the compat link `nodes/` takes its place. **Nothing + is ever deleted from the source.** A staging directory left by a killed process is cleared by the next attempt. + With no migration lock there is no wait: a multi-node process no longer blocks for up to 30 s behind another + process's migration. A move across volumes is no longer a special case: no rename crosses one. +- **A by-id directory without the commit record is a husk, never "the identity is in place".** An empty one, or one + holding only a stale lock (what 0.14.0 - 0.14.2 left), is removed with `rmdir` (which cannot remove a directory with + anything in it) and the commit made; one holding files (a node of an earlier release that ran there without its + identity) has them merged into the staged copy, and is itself kept aside at `nodes/by-id/..husk`. A file + that differs, or a different `identity.json`, is a second claim on the nodeId: the start halts and nothing changes. + A husk whose lock a live process holds refuses (`EIDENTITYLOCK`). +- **A name is claimed by an exclusive create of its index entry, after the identity is installed** (written and synced + under a temp name, then hard-linked into place; an O_EXCL create where hard links are not supported). Two first starts + of a new name both make an identity; exactly one claims the name, and the other discards its identity and loads the + winner. R1 measured 2 - 3 agents for one new name in 6 of 6 rounds with three starters; now one, every round. +- **An index entry reads as "absent" only when there is none (ENOENT).** An entry that is empty, not JSON, names another + name or cannot be read (EACCES, EMFILE) halts, says where the identity it named is, and says not to delete it. Index + writes are synced (file and directory). +- **A new identity is made only when nothing on this host is a trace of the name**: no index entry, no `nodes/` + (a link, or a directory with anything but a stale lock in it), no `SYM_IDENTITY_DIR/`, no 0.13 directory set + aside, no installed identity of that name that no name indexes. A trace that provably is the name's identity (a compat + link to it; the one unindexed identity of that name: a start killed between installing it and claiming the name, or a + lost entry) is adopted: the name is claimed for it. Any other trace halts and says what it is. **Behaviour change:** a + `nodes//` holding files and no identity used to become a new agent's directory; it now halts (with + `SYM_IDENTITY_DIR`'s volume not mounted that directory is the agent's store, and the new identity took it). +- **One path for pinned and unpinned starts.** `loadIdentity({ name, nodeId, create: false })` (mesh-channel) migrates + through the same staged commit and loads the pinned agent. A pin whose identity is in the 0.13 directory and could not + be migrated halts saying so (it used to say "No identity on this host"). +- **A start pinned to a nodeId never takes a name from another identity** (R4-9, from the mesh-channel review). It used + to re-point the name's index entry to the pin: a folder whose `.sym/node.json` pins X under "reviewer" took the name + from another folder's agent Y, whose next start by name then ran as X, and a cloned repository could choose which + identity on the host a session signs as. A pinned start may claim a name nobody has, and that is all; a name another + identity holds halts (`IdentityHaltError`) with the entry left as it is (with `create: false` and the pin not on this + host, `IdentityAbsentError` saying so), and a pinned start is never answered with the name's agent. **Behaviour change:** + to give a name to the pinned identity, rename the other agent first. +- **A live 0.13 process in the agent's 0.13 directory refuses the start** (`EIDENTITYLOCK`, naming its pid). 632b95d + loaded the identity in place and a 0.14 node ran beside the 0.13 one, on an empty store. The migration holds that + directory's own lock while it copies, so a 0.13 process cannot start in it meanwhile; another 0.14.3 start that finds + the lock marked as a migration's goes on beside it. +- **A stale lock is removed only if it is still the lock judged stale.** Removing one lock is a right of its own: a claim + file named after that lock's file identity and content (`lock.pid.break-`), itself a lock. Holding it, the lock + is read again and removed only if it is the same file (dev, ino) with the same content. R1 measured two or more + simultaneous holders in 4 of 8 rounds with six starters at a SIGKILLed holder's lock. A lock file is now created whole + (linked into place), so it is never seen without its owner. +- **Export: bundle durable first, then the tombstone beside the key, then the scrub** (`lib/relocation.js`). An intent + (`EXPORTING.json`) is written beside the key first; while it is there the source refuses to start the identity + (`IdentityExportPendingError`, `EIDENTITYEXPORTING`). An export interrupted before its tombstone is resumable (run it + again) or abortable (`relocation.abortExport`, which removes the bundle it wrote); one interrupted after it is finished + by running it again (the scrub). The tombstone is in the identity directory, so with `SYM_IDENTITY_DIR` a rebuilt node + directory no longer brings a second live copy back. The scrub also removes the private key from a 0.13 copy set aside. +- **Import** is installed through the same staged commit: directories 0700, key files 0600, and with `SYM_IDENTITY_DIR` + the private key only in the identity directory (it was in the node directory too). +- `validateName` refuses names Windows cannot hold (the rule mesh-channel's `nodeNameProblem` applies: `: * ? " < > |`, + CON, PRN, AUX, NUL, COM0-9, LPT0-9 with any extension, a trailing dot or space). **Behaviour change** for an existing + agent with such a name: it must be renamed (`renameIdentity` accepts the old name). A compat link is compared with + its target as the path it resolves to, so a Windows junction is no longer re-made on every check. +- Error texts: a halt says where the key is when it is on this host (for example `nodes/by-id//identity.json` after + `SYM_IDENTITY_DIR` was switched on), and tells the operator to remove the index entry only when no copy of the key is + anywhere on this host. `IdentityAbsentError` says what is on the host instead, and is never thrown when the identity is. +- Also in 0.14.3, from the first fix: key files 0600 and their directories 0700 (narrowed on load), atomic rewrites of + `identity.json`, a public key without a private key halts, `SYM_IDENTITY_DIR` switched on for a 0.13 node whose + identity is in its node directory halts and says how to copy it. ### Known, not fixed here -- `migrateIdentities()` still counts and discards any error from a single node (including the halts above); only the - counters it returns. The node's own start (`loadIdentity`) raises them, so a start is loud. Whether the bulk call - should report them is open. -- `loadIdentity({ nodeId, name })` still re-points a name that already indexes a different node. -- `SYM_IDENTITY_DIR` switched on for an existing node does not move its key for you (it halts and says how). -- `loadIdentity({ nodeId, name, create: true })` with a `nodeId` the name index does not hold, no identity file, and no 0.13 - directory under the name still makes a new identity under a new nodeId (it does not use the one given). mesh-channel - passes `create: false`, so a pin is never ignored there. -- A pinned start leaves another agent's 0.13 directory (same name, different nodeId) unmigrated, as above; with - `SYM_IDENTITY_DIR` set and that identity in the node directory it halts with the instructions given for that case. +- `migrateIdentities()` still counts and discards any error from a single node; the node's own start raises them. +- `loadIdentity({ nodeId, name, create: true })` with a nodeId this host does not hold and no trace of the name still + makes a new identity under a new nodeId. mesh-channel passes `create: false`. +- `SYM_IDENTITY_DIR` switched on for an existing node does not copy its key for you (it halts and says how). +- The 0.13 directory set aside (`nodes/.legacy-0.13/`) and a husk kept aside are never removed by sym: they hold a + second copy of the store (and of the key, 0600, until an export scrubs it). Remove them once the agent runs on 0.14.3. + On APFS and reflink file systems the copy is a clone; elsewhere it takes the store's size again on disk. +- A process killed in the two renames between setting the 0.13 directory aside and putting the compat link in its + place leaves `nodes/` absent; the next 0.14.3 start puts the link back, but a 0.13 build started first in that + state would make a new identity. A crash anywhere else leaves the 0.13 directory, or the link to the installed copy, + where a 0.13 build reads it. +- A husk with files that was claimed by one start can, in a race with another start that never saw it, end up kept at + `..husk` without being merged (nothing is lost; the node runs without those files). It needs a 0.14.0 - 0.14.2 + node that ran in that directory beside a live 0.13 process, and two concurrent first starts. +- A claim file left by a process killed while removing a stale lock is itself removed the same way, up to three levels; + below that the start refuses (`EIDENTITYLOCK`) rather than guess. A 0.13 process still reclaims a stale lock the old, + non-atomic way: a 0.13 and a 0.14 process racing at one dead holder's lock are not covered. +- `relocation.abortExport` is not on the `sym` CLI yet; `sym node export` run again resumes an interrupted export. ### Tests -- `tests/identity-migration-by-id.test.js`: 51 tests (50 run on macOS; one real cross-filesystem test runs on Linux). - 41 of them fail on 0.14.2 (42 on Linux); nine pin behaviour that did not change (two of them keep a pinned start from - migrating what is not its own). Of the 51, ten are for pinned starts, among them four real processes starting one 0.13 - agent at once under injected stalls, with and without a by-id directory already in place. The code is mutation-tested: - twenty-one mutants of `lib/config.js` (each guard and each ordering removed in turn), each caught by at least one test. +FUZZ_NUMBERS ## 0.14.2 — 2026-10-14 From ad03aedb78154255ae6c95cd2d76b3ff3d6782a4 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 17:09:42 +0100 Subject: [PATCH 13/15] Identity: a husk is looked at again right before it is set aside (found by the multi-process fuzz) Four starts at a 0.13 node over a lock-only by-id husk: one start judged the target a husk, another emptied it (stale lock removed, rmdir) and committed its copy there, and the first then read the directory's entries (the committed copy's), took them for a husk's files and renamed the committed directory aside to ..husk. Same identity, nothing lost, but an installed directory moved: 9 of 40 rounds (6 of 40 pinned). The husk is now examined again immediately before it is set aside; a commit found there is the other start's ('exists'). A husk with files cannot be replaced by a commit (a rename onto a non-empty directory fails), so after that look only another claim can move it, and then the rename aside fails. Lock artifacts (a claim, a temp lock file) are not a husk's content: dead ones (and this process's own, left by a failed attempt) are removed before the rmdir; one that stays has the husk set aside with it. The real cross-filesystem test is named for what it now checks: no rename crosses the volumes. --- lib/config.js | 43 ++++++++++++++++++++++++-- tests/identity-migration-by-id.test.js | 2 +- 2 files changed, 41 insertions(+), 4 deletions(-) diff --git a/lib/config.js b/lib/config.js index a6a685c9..f65c57c2 100644 --- a/lib/config.js +++ b/lib/config.js @@ -923,6 +923,30 @@ function mergeIntoStaging(from, staging, skip, rel = '') { } } +/** A lock, a claim to remove one, or the temp file a lock is written under: runtime state, never a node's data. */ +const isLockArtifact = (n) => n === 'lock.pid' || n.startsWith('lock.pid.') || (n.startsWith('.lock.pid') && n.endsWith('.tmp')); + +/** + * Remove, from a husk that holds nothing else, the lock artifacts of processes that are gone: a claim to remove a + * lock that is itself gone, or a temp lock file. (lock.pid itself is removed only by removeStaleLock.) + */ +function clearDeadLockArtifacts(dir) { + if (fs.existsSync(path.join(dir, 'lock.pid'))) return; + let names = []; + try { names = fs.readdirSync(dir); } catch { return; } + // This process's own are left over from an attempt that failed: lock work is synchronous, so none is under way. + const gone = (pid) => pid === process.pid || !pidIsAlive(pid); + for (const n of names) { + if (n.startsWith('lock.pid.')) { + const l = readLockFile(path.join(dir, n)); + if (l && (l.pid === process.pid || (!lockIsLive(l) && Date.now() - l.mtimeMs > LOCK_CORRUPT_GRACE_MS))) { try { fs.unlinkSync(path.join(dir, n)); } catch { /* */ } } + } else if (n.startsWith('.lock.pid') && n.endsWith('.tmp')) { + const m = /\.(\d+)-[0-9a-f]+\.tmp$/.exec(n); + if (m && gone(Number(m[1]))) { try { fs.unlinkSync(path.join(dir, n)); } catch { /* */ } } + } + } +} + /** * Install a complete staging directory as `target` by ONE rename: that rename is the commit. Returns * 'committed' when this staging went in, 'exists' when `target` already holds a committed directory (another @@ -961,11 +985,24 @@ function commitStaged(staging, target, { nodeId, examine, skip }) { removeStaleLock(lockPath, lock); } let rest = []; - try { rest = fs.readdirSync(target).filter((n) => n !== 'lock.pid'); } catch { continue; } + try { rest = fs.readdirSync(target).filter((n) => !isLockArtifact(n)); } catch { continue; } if (!rest.length) { - try { fs.rmdirSync(target); } catch (e) { if (!['ENOENT', 'ENOTEMPTY', 'EEXIST'].includes(e.code)) throw e; } - continue; + clearDeadLockArtifacts(target); + try { fs.rmdirSync(target); continue; } catch (e) { + if (!['ENOENT', 'ENOTEMPTY', 'EEXIST'].includes(e.code)) throw e; + if (e.code === 'ENOENT') continue; + // Another start's claim or temp lock file is in it for a moment; if one stays, the husk is set aside with it. + if (attempt < 4) { pause(2); continue; } + } } + // Looked at again right before it is set aside: since the look above, the husk may have been emptied and + // replaced by another start's commit (found by the multi-process fuzz: a committed directory was set aside). + // A husk with files cannot be replaced by a commit (a rename onto a directory that is not empty fails), so + // from here only another claim can move it, and then the rename below fails. + const again = examine(target); + if (again.state === 'absent') continue; + if (again.state === 'committed') return 'exists'; + if (again.state === 'conflict') throw new CommitConflict(again.reason); mergeIntoStaging(target, staging, skip); try { fs.renameSync(target, asideHusk); } catch (e) { if (e.code !== 'ENOENT' && !RENAME_TAKEN.has(e.code)) throw e; } } diff --git a/tests/identity-migration-by-id.test.js b/tests/identity-migration-by-id.test.js index 7013e7dc..72d772c3 100644 --- a/tests/identity-migration-by-id.test.js +++ b/tests/identity-migration-by-id.test.js @@ -543,7 +543,7 @@ describe('F9: a copy across filesystems: copies, verifies, commits; the source i describe('F9 on a real second filesystem (Linux: nodes/by-id lives on /dev/shm, a tmpfs)', () => { const shm = '/dev/shm'; const second = process.platform === 'linux' && exists(shm) && fs.statSync(shm).dev !== fs.statSync(os.tmpdir()).dev; - (second ? it : it.skip)('rename fails with EXDEV for real, and the node still arrives whole, private, and unchanged', async () => { + (second ? it : it.skip)('by-id on another filesystem for real: the node arrives whole, private and unchanged, and no rename crosses the volumes', async () => { const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sym-exdev-')); const far = fs.mkdtempSync(path.join(shm, 'sym-exdev-')); try { From a0e9663d687121d64cb90e8a99aae5f753fbf913 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 17:10:10 +0100 Subject: [PATCH 14/15] Test: a husk replaced by another start's commit mid-look is never set aside (the fuzz's race, made deterministic) --- tests/identity-commit.test.js | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/tests/identity-commit.test.js b/tests/identity-commit.test.js index 1dfd51c8..90f17816 100644 --- a/tests/identity-commit.test.js +++ b/tests/identity-commit.test.js @@ -287,6 +287,27 @@ describe('the staged commit: what it checks before the one rename', () => { } finally { child.kill('SIGKILL'); await new Promise((r) => child.once('exit', r)); } }); + it('a husk replaced by another start\'s commit while this start looks at it is never set aside (found by the multi-process fuzz)', () => { + const name = uniq('husk-race'); + const id = legacyNode(name); + const target = cfg.nodeDirById(id.nodeId); + put(path.join(target, 'lock.pid'), '2147483646\n'); // the 0.14.0-0.14.2 husk: a stale lock + let other = false; + patchFs('readdirSync', (orig) => function (p, ...rest) { + // After this start removed the stale lock and before it reads what is left, another start commits its copy. + if (!other && String(p) === target && !exists(path.join(target, 'lock.pid')) && !exists(path.join(target, cfg.COMMIT_MARK))) { + other = true; + cfg.migrateNodeDir(name); + } + return orig.call(fs, p, ...rest); + }, () => { cfg.migrateNodeDir(name); }); + assert.ok(other, 'the other start ran in between'); + assert.ok(!exists(path.join(path.dirname(target), `.${id.nodeId}.husk`)), 'no installed directory was set aside'); + assert.strictEqual(JSON.parse(read(path.join(target, 'identity.json'))).privateKey, id.privateKey); + assert.deepStrictEqual(identitiesNamed(name), [id.nodeId]); + assert.strictEqual(cfg.loadIdentity({ name }).privateKey, id.privateKey); + }); + it('a staging directory of a live process is left alone; one of a process that is gone is cleared', () => { const name = uniq('partials'); const id = legacyNode(name); From 379d75dec3bfb361af8c3bf2789af08b3c7f8075 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 17:27:05 +0100 Subject: [PATCH 15/15] CHANGELOG 0.14.3: the measured numbers (tests, mutation, multi-process and kill -9 fuzz, suite, gate, audit) --- CHANGELOG.md | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cd992e8c..cf9be2c9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -151,7 +151,37 @@ patch for one of those cases made another. R1 proved eleven by running them. 0.1 ### Tests -FUZZ_NUMBERS +- `tests/identity-commit.test.js` (new): 33 tests, one or more per R1 finding (R1-1 to R1-11), R4-9, the error texts + and the guards of the staged commit; 32 fail on 632b95d (the one that passes there checks that a pinned start still + claims a name nobody has). R1-1 crashes the migration at every one of its 70 steps (78 over a lock-only husk), one + start per step, and checks the 0.13 directory is byte-identical, a 0.13 reader still finds the key, and the next start + loads the same agent with its whole store. The race the multi-process fuzz found (below) is a deterministic test. +- `tests/identity-migration-by-id.test.js`: 53 tests (was 51), 14 of them fail on 632b95d. Updated, each with the reason in + the test: a husk's stale lock is no longer kept; a halt writes no index back; a non-compat link at `nodes/` is + followed and copied; a node directory with files and no identity halts; the atomic rewrite is tested on an installed + identity; across volumes nothing is renamed and the source is kept aside; an interrupted migration leaves the 0.13 + directory as it was; the migration-lock tests are replaced by no-wait tests; a live 0.13 holder refuses; a pin whose + identity is in the 0.13 directory halts instead of "absent". +- Mutation check: 27 mutants of `lib/config.js` and `lib/relocation.js`, one per guard of the new design (the verify + before the commit, the exclusive claim, ENOENT-only, the mint gate, the adoption guard, the live 0.13 holder, the + claim and the compare in the lock reclaim, the export order, the tombstone place, the import key place, junction + compare, the Windows names, the set-aside, the link-before-aside, a pinned start taking a name, the husk re-look, and + others): every one is caught. +- Multi-process fuzz (real processes, every file call stalled 0 - 4 ms at random): four starters at a 0.13 agent, + plain / over a lock-only husk / pinned / pinned over a husk / with SYM_IDENTITY_DIR / with both: 40 rounds each, 0 + failed (960 starters, all loaded the original; one identity, the key installed, the whole store, the 0.13 data kept, + every round); six starters, 30 rounds each plain and over a husk, 0 failed. This fuzz found one race in the first + version of this design (a committed directory set aside as a husk, 9 of 40 rounds over a husk; same identity, nothing + lost), fixed before release. Six starters at a SIGKILLed holder's stale lock: 0 of 20 rounds with two holders, with + and without stalls (632b95d: 3 of 8). Three first starts of one new name: 0 of 10 rounds with two agents, 0 of 6 with + SymNode (632b95d: 6 of 6). Across a real second volume (an HFS+ disk image, EXDEV) with a stale migration lock: + 4 and 6 starters, 50 rounds, every starter the original. +- kill -9 crash fuzz: the first start is killed at a random moment, a second start follows. Same volume (over a husk, + plain, with SYM_IDENTITY_DIR; pinned and not): 360 rounds, 180 of them killed within 60 ms (all mid-run), every one + the original agent with its whole store and the key on disk. Across volumes (disk image): 120 rounds, 60 killed + mid-run, every one the same (632b95d on the same image: 4 of 10 rounds started with store files missing). +- Full suite (`--test-concurrency=2`, macOS): 1568 tests, 1564 pass, 0 fail, 4 skipped; `npm run gate:stock-nodes` passed; + `npm audit`: 0 vulnerabilities. CI (Linux, Node 20 and 22) green, the real second-filesystem test included. ## 0.14.2 — 2026-10-14