diff --git a/CHANGELOG.md b/CHANGELOG.md index 68c5b475..536818e6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ "do not ship" with 13 findings. Each was checked by running it in a sandbox state dir; 0.14.3 fixes the ones that lose or expose a key and the small ones beside them, by reading the state of an identity only from records committed in one atomic step (see "Identity and on-disk state"). The wire is unchanged; the public surface gains `relocation.abortExport` -and `IdentityExportPendingError`. +(and `sym node abort-export`) and `IdentityExportPendingError`. ### What was wrong @@ -52,102 +52,134 @@ and `IdentityExportPendingError`. ### Identity and on-disk state -The first fix for the findings above was itself reviewed (R1, on the candidate 632b95d) and failed the same way twice -over: it read the state of an identity from whatever was at a path (a by-id directory that was only partly there, an -index entry that could not be read, a node directory without `identity.json`, the content of a stale lock), and each -patch for one of those cases made another. R1 proved eleven by running them. 0.14.3 changes the assumption instead: -**the state of an identity is a record committed in one atomic step**, and the code that inferred it is deleted -(`moveTree`, `mergeMove`, `sameTree`, `legacyDirsRemain`, and the migration lock `nodes/.migrating`). - -- **The 0.13 → by-id migration copies; it never moves.** The 0.13 directory (`nodes//`, and - `SYM_IDENTITY_DIR//`) is copied into a staging directory beside its destination - (`nodes/by-id/..partial--`, on the destination's volume), links followed (a node directory kept - on another volume behind a link, an `identity.json` that is a relative link to a key file), `identity.json` last, every - file synced, the copy verified byte for byte against the source, a commit record (`.sym-commit`) written, and the - whole installed by ONE rename. Of several starts doing this at once, one rename wins; the others find the same - identity installed and discard their staging. Then the 0.13 directory is renamed aside to - `nodes/.legacy-0.13/` (key files narrowed to 0600) and the compat link `nodes/` takes its place. **Nothing - is ever deleted from the source.** A staging directory left by a killed process is cleared by the next attempt. - With no migration lock there is no wait: a multi-node process no longer blocks for up to 30 s behind another - process's migration. A move across volumes is no longer a special case: no rename crosses one. -- **A by-id directory without the commit record is a husk, never "the identity is in place".** An empty one, or one - holding only a stale lock (what 0.14.0 - 0.14.2 left), is removed with `rmdir` (which cannot remove a directory with - anything in it) and the commit made; one holding files (a node of an earlier release that ran there without its - identity) has them merged into the staged copy, and is itself kept aside at `nodes/by-id/..husk`. A file - that differs, or a different `identity.json`, is a second claim on the nodeId: the start halts and nothing changes. - A husk whose lock a live process holds refuses (`EIDENTITYLOCK`). -- **A name is claimed by an exclusive create of its index entry, after the identity is installed** (written and synced - under a temp name, then hard-linked into place; an O_EXCL create where hard links are not supported). Two first starts - of a new name both make an identity; exactly one claims the name, and the other discards its identity and loads the - winner. R1 measured 2 - 3 agents for one new name in 6 of 6 rounds with three starters; now one, every round. -- **An index entry reads as "absent" only when there is none (ENOENT).** An entry that is empty, not JSON, names another - name or cannot be read (EACCES, EMFILE) halts, says where the identity it named is, and says not to delete it. Index - writes are synced (file and directory). -- **A new identity is made only when nothing on this host is a trace of the name**: no index entry, no `nodes/` - (a link, or a directory with anything but a stale lock in it), no `SYM_IDENTITY_DIR/`, no 0.13 directory set - aside, no installed identity of that name that no name indexes. A trace that provably is the name's identity (a compat - link to it; the one unindexed identity of that name: a start killed between installing it and claiming the name, or a - lost entry) is adopted: the name is claimed for it. Any other trace halts and says what it is. **Behaviour change:** a - `nodes//` holding files and no identity used to become a new agent's directory; it now halts (with - `SYM_IDENTITY_DIR`'s volume not mounted that directory is the agent's store, and the new identity took it). -- **One path for pinned and unpinned starts.** `loadIdentity({ name, nodeId, create: false })` (mesh-channel) migrates - through the same staged commit and loads the pinned agent. A pin whose identity is in the 0.13 directory and could not - be migrated halts saying so (it used to say "No identity on this host"). -- **A start pinned to a nodeId never takes a name from another identity** (R4-9, from the mesh-channel review). It used - to re-point the name's index entry to the pin: a folder whose `.sym/node.json` pins X under "reviewer" took the name - from another folder's agent Y, whose next start by name then ran as X, and a cloned repository could choose which - identity on the host a session signs as. A pinned start may claim a name nobody has, and that is all; a name another - identity holds halts (`IdentityHaltError`) with the entry left as it is (with `create: false` and the pin not on this - host, `IdentityAbsentError` saying so), and a pinned start is never answered with the name's agent. **Behaviour change:** - to give a name to the pinned identity, rename the other agent first. -- **A live 0.13 process in the agent's 0.13 directory refuses the start** (`EIDENTITYLOCK`, naming its pid). 632b95d - loaded the identity in place and a 0.14 node ran beside the 0.13 one, on an empty store. The migration holds that - directory's own lock while it copies, so a 0.13 process cannot start in it meanwhile; another 0.14.3 start that finds - the lock marked as a migration's goes on beside it. -- **A stale lock is removed only if it is still the lock judged stale.** Removing one lock is a right of its own: a claim - file named after that lock's file identity and content (`lock.pid.break-`), itself a lock. Holding it, the lock - is read again and removed only if it is the same file (dev, ino) with the same content. R1 measured two or more - simultaneous holders in 4 of 8 rounds with six starters at a SIGKILLed holder's lock. A lock file is now created whole - (linked into place), so it is never seen without its owner. -- **Export: bundle durable first, then the tombstone beside the key, then the scrub** (`lib/relocation.js`). An intent - (`EXPORTING.json`) is written beside the key first; while it is there the source refuses to start the identity - (`IdentityExportPendingError`, `EIDENTITYEXPORTING`). An export interrupted before its tombstone is resumable (run it - again) or abortable (`relocation.abortExport`, which removes the bundle it wrote); one interrupted after it is finished - by running it again (the scrub). The tombstone is in the identity directory, so with `SYM_IDENTITY_DIR` a rebuilt node - directory no longer brings a second live copy back. The scrub also removes the private key from a 0.13 copy set aside. -- **Import** is installed through the same staged commit: directories 0700, key files 0600, and with `SYM_IDENTITY_DIR` - the private key only in the identity directory (it was in the node directory too). -- `validateName` refuses names Windows cannot hold (the rule mesh-channel's `nodeNameProblem` applies: `: * ? " < > |`, - CON, PRN, AUX, NUL, COM0-9, LPT0-9 with any extension, a trailing dot or space). **Behaviour change** for an existing - agent with such a name: it must be renamed (`renameIdentity` accepts the old name). A compat link is compared with - its target as the path it resolves to, so a Windows junction is no longer re-made on every check. -- Error texts: a halt says where the key is when it is on this host (for example `nodes/by-id//identity.json` after - `SYM_IDENTITY_DIR` was switched on), and tells the operator to remove the index entry only when no copy of the key is - anywhere on this host. `IdentityAbsentError` says what is on the host instead, and is never thrown when the identity is. -- Also in 0.14.3, from the first fix: key files 0600 and their directories 0700 (narrowed on load), atomic rewrites of - `identity.json`, a public key without a private key halts, `SYM_IDENTITY_DIR` switched on for a 0.13 node whose - identity is in its node directory halts and says how to copy it. +Two designs for the fixes above failed review before this one. 0.14.2 inferred the state of an identity from whatever +was at a path. The first 0.14.3 candidate (9f6847c) copied a 0.13 agent into a staging directory, verified the copy and +committed it by one rename; an independent re-review reproduced what the copy cost: a store that did not fit twice on a +file system that cannot clone (ext4, HFS+) gave ENOSPC on every start; a 5,000-file first start took 21 s instead of +61 ms, and 12 agents in one XMesh process blocked its constructor for 15.5 s; private-key copies were left at 0644 in +husks, killed starters' staging and set-aside directories, outside the export's scrub; a store on a mount point was +copied onto the state volume; a relative link was set aside dangling; and the export wrote its bundle before its +tombstone, so an interrupted export retried to another destination left two importable bundles (two live copies of one +key). 0.14.3 ships the design that replaces both: + +- **One source of truth.** An agent exists exactly when its identity is committed: `nodes/by-id//identity.json` + (with `SYM_IDENTITY_DIR`, `SYM_IDENTITY_DIR/by-id//identity.json`). The name index (`nodes/by-name/`) and the + 0.13-compatible links (`nodes/`) are caches of which agent a name is, written after the commit. An entry that + is missing or cannot be read is re-derived from the committed identities (their `name`), with the compat link as a + second witness: an entry whose content is not an entry is written again; one that could not be read for an I/O error + (EACCES, EMFILE) is left as it is. Neither is ever read as "no such agent". An entry that names an identity which is + not committed here halts (a `SYM_IDENTITY_DIR` volume not mounted). Two committed identities that are one name halt, + naming both; each still loads by its nodeId. An entry 0.14.0 - 0.14.2 wrote for a rename or a pinned start (it names + an identity under another name) keeps leading where it led, and when it is that identity's only name the identity + takes it, so it can be derived from then on. `renameIdentity` writes the identity's own `name` (the commit of a + rename), after claiming the new entry. +- **Names are compared without regard to case on every platform** (Unicode NFC, lower case): the entry is + `nodes/by-name/.json` and holds the exact name. 0.14.0 - 0.14.2's entry under the exact name is read when + the folded one is missing, and written beside it for a rollback. Two 0.13 agents whose names fold together (a + case-sensitive file system let both exist): the second halts and is left as it is. +- **A new identity is minted only when nothing on the host holds the name**: no committed identity of that name (or + that its entry or link leads to), no 0.13 directory holding an identity, no identity set aside or in another start's + staging, and no other trace (a `nodes/` holding files, an entry that cannot be read, a link to an identity not + in place). A mint holds the name's short mint lock (`nodes/by-id/.mint-.lock`, with the atomic stale-lock + reclaim) from its staging to its claim: identity.json is written and synced in a staging directory, committed by one + rename, the compat link put in place, and the name claimed by an exclusive create (`link()`) of its entry. A start + killed between the commit and the claim leaves an identity the next start adopts. +- **A 0.13 agent is MOVED, never copied.** On one volume, ONE rename of the whole directory onto + `nodes/by-id/` (with `SYM_IDENTITY_DIR`, the node directory first and the identity directory last: the + identity in place is the commit). Atomic, O(1), no disk: the 5,000-file first start takes 61 ms, and 12 agents in one + process start in 0.6 s. The compat link is made before the commit and renamed onto the 0.13 path right after it, + before any sync, so a 0.14.2 or 0.13 build finds neither for one system call; no 0.13 directory is moved out from under + a path no link can then take. A 0.13 directory that is a link, a mount point, on another volume than `nodes/by-id`, + or holding a relative link that leads out of it is not moved and not copied: `by-id/` becomes an absolute + link to the directory it is, and the store stays where its operator put it (a link is replaced by the compat link; a + mount point stays the real directory). Across volumes only an identity directory's key files (`identity.json`, the + 0.13 `e2e-keypair.json`) are copied: staged beside their destination, synced, read back and compared, committed by + one rename, after a free-space check that halts with the bytes needed (the source untouched); the source is then set + aside. +- **A husk** (a `by-id/` directory an earlier release made without the identity: a lock, a node that ran there + beside a live 0.13 process) is set aside whole to `by-id/..husk-