From b7d0634799283514091e8a59cceff22d8bb88504 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 19:01:15 +0100 Subject: [PATCH 1/3] 0.14.3 records: actions by assertion, per-assertion summaries, well-formed text, in-memory recall Four defects the review of 9f6847c found, each fixed at the design assumption that made it. S3 (R2-6). An actionable record (a signed application section) is a message as well as a memory: memory keeps one record per cognition key, and the action is identified by its assertion identity wherever it is sent, de-duplicated and delivered. The frame handler's one delivery decision (_surfaceUnstored) is now taken on every branch where memory stored nothing, the admit branch included (a record SVAF admitted under a key the store held reached nobody). remember() sends an action whose cognition memory holds (HEAD's, an earlier action's, a peer's) by its own assertion (_sendAction: nothing stored again, HEAD unmoved, `cognitionHeld: true` and a `delivery` report), and an action whose store write failed; the same assertion given twice is a duplicate. lib/record.js: recordActionable, recordAssertion. B2 (R2-8). aggregateAttestations summarises one named assertion: given, held, or the sole one the attestations under the key name (a node without the record: XMesh's observers); several give no single verdict and byAssertion has each. Adds basis, assertions, byAssertion; the old fields stay. M2 (R2-13). The signed object is UTF-8 bytes, where a lone surrogate can only be U+FFFD: every string a record signs is brought to its well-formed spelling at each boundary (createCMB, a caller's record in remember(), the canonical projection, the Legacy Import route, the public verifier, the store's read) instead of refusing it, so a 0.14.2 author's text cut inside a pair verifies and is held as one string. The signing step still refuses a string no boundary normalised. sym's own cuts (message preview, wire names, SVAF text, context, error texts, decision labels) fall on code point boundaries (lib/core/text.js: cutText). M1 (R2-3). recall() matches on a bounded search index kept apart from the entry cache (each entry's lowercased text, 4,096 characters at most, 32 M in all, the oldest let go first) and reads only the entries it returns; recall(query, { limit, where }) and `total`. A scan reads through the cache, never into it; the cache starts with the newest entries; the timestamp ratchet and the lifecycle are read from the index. The primer, daemon, interior and mesh agent pass limits. Tests: tests/security-0.14.3-records.test.js (13, real nodes on the wire paths) and tests/integration/mixed-0142-records.js (4, against a real 0.14.2 tree, SYM_0142_DIR) fail on 9f6847c and pass here. R2-13's tests and remember-payload's collapse test now state the design as built. --- bin/sym-daemon.js | 6 +- bin/sym.js | 3 +- docs/reference.md | 2 +- lib/core/claude-memory-bridge.js | 5 +- lib/core/cmb-encoder.js | 97 +++- lib/core/cmb-signing.js | 22 +- lib/core/context-encoder.js | 4 +- lib/core/record-canonical.js | 22 +- lib/core/text.js | 49 +++ lib/frame-handler.js | 74 ++-- lib/interior.js | 12 +- lib/legacy-import.js | 5 +- lib/memory-store.js | 267 +++++++++-- lib/mesh-agent.js | 3 +- lib/node.js | 212 ++++++--- lib/record.js | 22 + lib/wire-identity.js | 4 +- tests/integration/e2e-payload-receive.js | 2 +- tests/integration/mixed-0142-records.js | 170 +++++++ tests/remember-payload.test.js | 21 +- tests/security-0.14.3-records.test.js | 537 +++++++++++++++++++++++ tests/security-0.14.3-verify.test.js | 49 ++- 22 files changed, 1389 insertions(+), 199 deletions(-) create mode 100644 lib/core/text.js create mode 100644 tests/integration/mixed-0142-records.js create mode 100644 tests/security-0.14.3-records.test.js diff --git a/bin/sym-daemon.js b/bin/sym-daemon.js index c804073b..ca16fef9 100755 --- a/bin/sym-daemon.js +++ b/bin/sym-daemon.js @@ -519,8 +519,10 @@ function handleIPCMessage(socketId, socket, msg) { break; case 'recall': { - let results = node.recall(msg.query || ''); - if (msg.limit && msg.limit > 0) results = results.slice(0, msg.limit); + // Only the entries returned are read (review M1). + const limit = Number.isSafeInteger(msg.limit) && msg.limit > 0 ? msg.limit : undefined; + let results = node.recall(msg.query || '', limit ? { limit } : undefined); + if (limit) results = results.slice(0, limit); sendIPC(socket, { type: 'result', action: 'recall', results }); break; } diff --git a/bin/sym.js b/bin/sym.js index 03bff582..39596323 100755 --- a/bin/sym.js +++ b/bin/sym.js @@ -3,6 +3,7 @@ const { recordCreatedBy } = require('../lib/record'); const { logSafe } = require('../lib/wire-identity'); // peer-chosen text on the terminal (R2-18) +const { cutText } = require('../lib/core/text'); /** * sym — CLI for the SYM mesh. @@ -963,7 +964,7 @@ function gatherMeshMemory(question, limit) { const pool = matched.length > 0 ? matched : all.sort((a, b) => b.timestamp - a.timestamp); return pool.slice(0, limit).map((c) => ({ ...c, - content: c.content.length > 400 ? c.content.slice(0, 400) + '…' : c.content, + content: c.content.length > 400 ? cutText(c.content, 400) + '…' : c.content, })); } diff --git a/docs/reference.md b/docs/reference.md index bf7f8182..5cece6ec 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -406,7 +406,7 @@ anything (a lifecycle change, an action): it refuses every record that is not v2 | `error` | Meaning | |---|---| | `unsigned-member` | the record carries a member its signature does not cover (`detail` names it) | -| `malformed-record` | it is not a well-formed v2.0 record: a type, a bound, NFC or well-formed Unicode (`detail`) | +| `malformed-record` | it is not a well-formed v2.0 record: a type, a bound or NFC (`detail`). A lone surrogate is not one of these: the signed bytes are UTF-8, where it can only be U+FFFD, so the record is read, and its `record` given, in that spelling (0.14.3) | | `assertion-id-mismatch` | its `assertionId` is not the hash of its signed preimage | | `bad-signature` | the signature does not verify under the key given | | `content-mismatch` | a category key or the record's address is not that of its text | diff --git a/lib/core/claude-memory-bridge.js b/lib/core/claude-memory-bridge.js index b072c55d..86869004 100644 --- a/lib/core/claude-memory-bridge.js +++ b/lib/core/claude-memory-bridge.js @@ -22,6 +22,7 @@ const fs = require('fs'); const path = require('path'); const os = require('os'); +const { cutText } = require('./text'); /** * Bridges Claude Code's project memory directory with the SYM mesh. @@ -227,7 +228,7 @@ class ClaudeMemoryBridge { const ts = Date.now(); const filename = `mesh_${safeName}_${ts}.md`; const filePath = path.join(this._memoryDir, filename); - const preview = (entry.content || '').slice(0, 80).replace(/\n/g, ' '); + const preview = cutText(entry.content || '', 80).replace(/\n/g, ' '); const md = [ '---', @@ -265,7 +266,7 @@ class ClaudeMemoryBridge { index = index.trimEnd() + '\n\n## Mesh\n'; } - const preview = (entry.content || '').slice(0, 60).replace(/\n/g, ' '); + const preview = cutText(entry.content || '', 60).replace(/\n/g, ' '); const line = `- [${filename}](${filename}) — [mesh: ${peerName}] ${preview}\n`; // Insert under ## Mesh diff --git a/lib/core/cmb-encoder.js b/lib/core/cmb-encoder.js index b881990b..0dc45e00 100644 --- a/lib/core/cmb-encoder.js +++ b/lib/core/cmb-encoder.js @@ -53,22 +53,78 @@ const { resolveCategories, unreadableReason } = require('./record-shape'); // portability hazard. Vectors are excluded (local, encoder-specific artifacts). const CMB_KEY_DOMAIN = 'mmp-cmb-v1\n'; +// Well-formed text (review M2): one definition, in a module that requires nothing (lib/core/text.js). +// A record is signed as UTF-8 bytes (§8.8.4), where a lone surrogate can only be U+FFFD, so a string +// that is not well formed is normalised wherever a record crosses a boundary (wellFormedRecord), never +// refused. +const { isWellFormedText, toWellFormedText, cutText } = require('./text'); + +/** `s` well formed when it is a string; anything else as it is (the schema checks types elsewhere). */ +const wf = (s) => (typeof s === 'string' && !isWellFormedText(s) ? toWellFormedText(s) : s); + +/** A list of references, each well formed; the same array when every one already is. */ +function wfList(list) { + if (!Array.isArray(list) || list.every((r) => typeof r !== 'string' || isWellFormedText(r))) return list; + return list.map(wf); +} + /** - * Whether `s` is well-formed Unicode: no lone surrogate (security review R2-13). UTF-8 encodes a lone - * surrogate as U+FFFD, so "\ud800" and "\ufffd" hash and sign as the same bytes: one signature would - * verify two different strings. A receiver refuses a signed string that is not well formed (record-canonical, - * the signing preimages), and an emitter mints only well-formed text (toWellFormedText). - * @param {string} s - * @returns {boolean} + * The record with every string it signs in its well-formed spelling (sym 0.14.3, review M2): each + * category's text and parents, `createdBy`, `room`, `to`, the lineage's parents and method, the + * application's schema. The signed object is UTF-8 bytes, where a lone surrogate can only be U+FFFD, + * so this changes no signed byte: a record verifies exactly as before, and what is held is the one + * string its signature covers (two different strings never share one signature). Applied at every + * boundary a record crosses: minting (createCMB, remember() for a caller's record) before it is + * signed, the canonical projection of a received or stored record, the Legacy Import route, and the + * store's read. + * + * Returns the same object when nothing needs it (the common case), otherwise a copy: the caller's + * object is never changed. + * @param {object} cmb + * @returns {object} */ -const isWellFormedText = typeof String.prototype.isWellFormed === 'function' - ? (s) => typeof s === 'string' && s.isWellFormed() - : (s) => typeof s === 'string' && !/\p{Cs}/u.test(s); - -/** `s` with each lone surrogate replaced by U+FFFD: the text UTF-8 would carry anyway, now spelled once. */ -const toWellFormedText = typeof String.prototype.toWellFormed === 'function' - ? (s) => s.toWellFormed() - : (s) => s.replace(/\p{Cs}/gu, '\ufffd'); +function wellFormedRecord(cmb) { + if (!cmb || typeof cmb !== 'object') return cmb; + let changed = false; + const cats = cmb.categories; + let categories = cats; + if (cats && typeof cats === 'object' && !Array.isArray(cats)) { + for (const [f, v] of Object.entries(cats)) { + let nv = v; + if (typeof v === 'string') nv = wf(v); + else if (v && typeof v === 'object') { + const text = wf(v.text); + const parents = v.meta && typeof v.meta === 'object' ? wfList(v.meta.parents) : undefined; + if (text !== v.text || (v.meta && parents !== v.meta.parents)) { + nv = { ...v, text, ...(v.meta && typeof v.meta === 'object' ? { meta: { ...v.meta, parents } } : {}) }; + } + } + if (nv !== v) { + if (categories === cats) categories = { ...cats }; + categories[f] = nv; + changed = true; + } + } + } + const m = cmb.metadata; + let metadata = m; + if (m && typeof m === 'object' && !Array.isArray(m)) { + const patch = {}; + for (const k of ['createdBy', 'room', 'to']) { const x = wf(m[k]); if (x !== m[k]) patch[k] = x; } + if (m.lineage && typeof m.lineage === 'object') { + const parents = wfList(m.lineage.parents); + const method = wf(m.lineage.method); + if (parents !== m.lineage.parents || method !== m.lineage.method) patch.lineage = { ...m.lineage, parents, ...(m.lineage.method !== undefined ? { method } : {}) }; + } + if (m.application && typeof m.application === 'object') { + const schema = wf(m.application.schema); + if (schema !== m.application.schema) patch.application = { ...m.application, schema }; + } + if (Object.keys(patch).length) { metadata = { ...m, ...patch }; changed = true; } + } + if (!changed) return cmb; + return { ...cmb, ...(categories !== cats ? { categories } : {}), ...(metadata !== m ? { metadata } : {}) }; +} /** Netstring length-prefix: `:`. Injection-proof * (a delimiter inside the text cannot shift a category boundary) and trivially @@ -588,8 +644,8 @@ function parentSet(list) { */ function mintLineage(lineage) { if (!lineage || typeof lineage !== 'object' || !Array.isArray(lineage.parents) || lineage.parents.length === 0) return null; - const out = { parents: parentSet(lineage.parents) }; - if (lineage.method !== undefined) out.method = lineage.method; + const out = { parents: parentSet(wfList(lineage.parents)) }; + if (lineage.method !== undefined) out.method = wf(lineage.method); return out; } @@ -600,8 +656,9 @@ function mintLineage(lineage) { */ function mintCategoryText(value) { const text = typeof value === 'string' ? value : (value && typeof value.text === 'string' ? value.text : ''); - // Well formed, then NFC (R2-13): a receiver refuses a lone surrogate, so one is minted as the U+FFFD - // its UTF-8 encoding carried anyway (a host that cut a string inside a surrogate pair keeps working). + // Well formed, then NFC (review M2): a lone surrogate is minted as the U+FFFD its UTF-8 encoding + // carries anyway, so the record holds the one string its signature covers (a host that cut a string + // inside a surrogate pair keeps working). return toWellFormedText(text || 'neutral').normalize('NFC'); } @@ -683,7 +740,7 @@ function createCMB(opts = {}) { for (const f of CAT7_CATEGORIES) { categories[f].meta = { key: categoryKeyV1(f, categories[f].text), - parents: Array.isArray(categoryParents[f]) ? parentSet(categoryParents[f]) : [], + parents: Array.isArray(categoryParents[f]) ? parentSet(wfList(categoryParents[f])) : [], }; } @@ -830,6 +887,8 @@ module.exports = { mintCategoryText, isWellFormedText, toWellFormedText, + wellFormedRecord, + cutText, recordBytes, textBytes, DEFAULT_ROOM, diff --git a/lib/core/cmb-signing.js b/lib/core/cmb-signing.js index 6b041897..79018b13 100644 --- a/lib/core/cmb-signing.js +++ b/lib/core/cmb-signing.js @@ -24,14 +24,17 @@ const { resolveCategories, unreadableReason } = require('./record-shape.js'); const crypto = require('crypto'); const { verifyStrict } = require('./ed25519'); -const { lp, keyScheme, sameKey, blockKeyV2, categoryKeyV1, isWellFormedText } = require('./cmb-encoder'); +const { lp, keyScheme, sameKey, blockKeyV2, categoryKeyV1, isWellFormedText, wellFormedRecord } = require('./cmb-encoder'); const { CAT7_CATEGORIES } = require('./cmb'); const { canonicalRecordV2_0 } = require('./record-canonical'); /** - * A string that enters a signing preimage is well-formed Unicode, or nothing is signed or verified over it - * (sym 0.14.3, R2-13): UTF-8 writes a lone surrogate as U+FFFD, so "\ud800" and "\ufffd" would sign as the - * same bytes and one signature would verify both. Returns the string. + * A string that enters a signing preimage is well-formed Unicode, or nothing is signed or verified over it. + * UTF-8 writes a lone surrogate as U+FFFD, so "\ud800" and "\ufffd" sign as the same bytes: a record is + * brought to its well-formed spelling at the boundary it crosses (wellFormedRecord: minting, the canonical + * projection, the Legacy Import route, the store's read; review M2), and one that reaches a preimage + * still holding a lone surrogate came by a path that skipped it. That is refused, so the object a + * signature is computed or checked over is always the one string it covers (R2-13). Returns the string. */ function wellFormed(v, what, suite) { const str = String(v); @@ -409,8 +412,9 @@ function verifyCMB(cmb, publicKeyB64url) { // and telling the caller which one is the whole value of the message. return { signed: true, valid: false, error: unreadableReason(cmb) }; } - // A text that is not well-formed Unicode addresses as its U+FFFD spelling (R2-13): refused, so one - // signature verifies one text. + // A text that is not well-formed Unicode addresses as its U+FFFD spelling: this step is run on a + // record already brought to that spelling (the projection, wellFormedRecord), and one that was not + // is refused, so one signature verifies one text (R2-13, review M2). for (const f of CAT7_CATEGORIES) { const c = cats[f]; const t = typeof c === 'string' ? c : (c && c.text); @@ -596,10 +600,12 @@ function verifyRecord(cmb, publicKeyB64url, opts = {}) { if (!m.sig || m.sigAlg !== 'ed25519') return { signed: false, valid: false, suite, coreSecure: false }; if (suite === MMP_SIGNATURE_SUITE) return verifyRecordV2_0(cmb, publicKeyB64url); if (requireCoreSecure) return { signed: true, valid: false, error: 'not-core-secure', suite, coreSecure: false }; - const why = legacyUnsigned(cmb); + // The 0.13 suite signs UTF-8 bytes too: its strings are read in their well-formed spelling (review M2). + const rec = wellFormedRecord(cmb); + const why = legacyUnsigned(rec); if (why === 'content-mismatch') return { signed: true, valid: false, error: 'content-mismatch', suite, coreSecure: false }; if (why) return { signed: true, valid: false, error: 'unsigned-member', detail: why, suite, coreSecure: false }; - return { ...verifyCMB(cmb, publicKeyB64url), suite, coreSecure: false }; + return { ...verifyCMB(rec, publicKeyB64url), suite, coreSecure: false }; } module.exports = { signingPayloadV2, signingPayloadV2_0, applicationCommitmentV1, assertionIdV2_0, categoryParentsCommitment, sigDigestV2, signCMB, verifyCMB, verifyRecord, checkAudience, privateKeyObject, publicKeyObject }; diff --git a/lib/core/context-encoder.js b/lib/core/context-encoder.js index b74b767d..206c4862 100644 --- a/lib/core/context-encoder.js +++ b/lib/core/context-encoder.js @@ -253,9 +253,11 @@ function svafKernel() { * CPU without bound. The semantic model reads its first few hundred tokens anyway; the n-gram kernel * now reads the first SVAF_TEXT_MAX characters. A text no longer than this encodes as before. */ +const { cutText } = require('./text'); const SVAF_TEXT_MAX = 16 * 1024; function svafText(text) { - return typeof text === 'string' && text.length > SVAF_TEXT_MAX ? text.slice(0, SVAF_TEXT_MAX) : text; + // Cut on a code point boundary (review M2): the kernel never reads half a surrogate pair. + return cutText(text, SVAF_TEXT_MAX); } /** diff --git a/lib/core/record-canonical.js b/lib/core/record-canonical.js index c20b6476..398c97ca 100644 --- a/lib/core/record-canonical.js +++ b/lib/core/record-canonical.js @@ -19,8 +19,10 @@ * - every signed string (each category's text, `createdBy`, `application.schema`) is already NFC: * a value that is not is refused, never normalised, as no type is coerced (MMP 2.0 update 1, * §8.8.5 step 1); `room` is a §5.8 room identifier, which is ASCII; - * - every string is well-formed Unicode (sym 0.14.3, R2-13): UTF-8 writes a lone surrogate as U+FFFD, - * so a record spelling one would verify under the signature of the record spelling U+FFFD; + * - every string is held in its well-formed spelling (sym 0.14.3, review M2; wellFormedRecord): the + * signed object is UTF-8 bytes, where a lone surrogate can only be U+FFFD, so a lone surrogate is + * held as U+FFFD. The record verifies as its author signed it (a 0.14.2 author's text cut inside a + * surrogate pair included), and two spellings of one signed text are held as one string (R2-13); * - `sig` is spelled canonically (R2-14): 86 base64url characters, the last one of A, Q, g, w; * - `createdBy` and each parent reference are at most 256 characters, counted as JSON Schema * counts them (code points), and a parents list holds at most 256 entries; @@ -45,7 +47,7 @@ */ const { CAT7_CATEGORIES } = require('./cmb'); -const { categoryKeyV1, textBytes, isWellFormedText, MAX_CATEGORY_BYTES, MAX_RECORD_TEXT_BYTES, MAX_RECORD_BYTES } = require('./cmb-encoder'); +const { categoryKeyV1, textBytes, wellFormedRecord, MAX_CATEGORY_BYTES, MAX_RECORD_TEXT_BYTES, MAX_RECORD_BYTES } = require('./cmb-encoder'); const { isRoomId } = require('./room-id'); const LOWER_UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; @@ -84,9 +86,6 @@ function codePoints(s) { } /** A signed string is NFC already, or the record is refused (§8.8.5 step 1): never normalised. */ const isNFC = (s) => s === s.normalize('NFC'); -/** A signed string is well-formed Unicode, or the record is refused (R2-13): a lone surrogate encodes as - * U+FFFD, so one signature would verify two strings. */ -const isWellFormed = isWellFormedText; /** Parents are signed as a set sorted by their UTF-8 bytes (§8.8.4); the projection holds that order. */ const sortedBytewise = (list) => [...list].sort((a, b) => Buffer.compare(Buffer.from(a, 'utf8'), Buffer.from(b, 'utf8'))); @@ -103,7 +102,7 @@ function refs(list, what, pattern) { if (list.length > MAX_PARENTS) throw new RecordShapeError(`${what} has more than ${MAX_PARENTS} entries`); const seen = new Set(); for (const r of list) { - if (typeof r !== 'string' || r.length === 0 || !isWellFormed(r) || codePoints(r) > MAX_REF || (pattern && !pattern.test(r))) throw new RecordShapeError(`${what} holds a value that is not a reference`); + if (typeof r !== 'string' || r.length === 0 || codePoints(r) > MAX_REF || (pattern && !pattern.test(r))) throw new RecordShapeError(`${what} holds a value that is not a reference`); if (seen.has(r)) throw new RecordShapeError(`${what} repeats an entry`); seen.add(r); } @@ -115,7 +114,6 @@ function category(name, v) { const allowed = name === 'mood' ? new Set(['text', 'meta', 'valence', 'arousal']) : new Set(['text', 'meta']); for (const k of Object.keys(v)) if (!allowed.has(k)) throw new RecordShapeError(`category ${name} carries an unknown member`); if (typeof v.text !== 'string') throw new RecordShapeError(`category ${name} has no text`); - if (!isWellFormed(v.text)) throw new RecordShapeError(`category ${name} text is not well-formed Unicode`); if (!isNFC(v.text)) throw new RecordShapeError(`category ${name} text is not NFC`); if (name === 'mood') { // Validated as the schema types them, then dropped: unsigned (§8.8.4). @@ -140,7 +138,7 @@ function application(a) { if (!isPlain(a)) throw new RecordShapeError('application is not an object'); for (const k of Object.keys(a)) if (!APPLICATION_MEMBERS.includes(k)) throw new RecordShapeError('application carries an unknown member'); if (typeof a.mediaType !== 'string' || !MEDIA.test(a.mediaType)) throw new RecordShapeError('application mediaType is malformed'); - if (typeof a.schema !== 'string' || a.schema.length > 2048 || !isWellFormed(a.schema)) throw new RecordShapeError('application schema is not text'); + if (typeof a.schema !== 'string' || a.schema.length > 2048) throw new RecordShapeError('application schema is not text'); // Signed as NFC(schema) (§8.8.3), so a free spelling would be a second form of one assertion: it is // NFC already, or refused (§8.8.5 step 1, MMP 2.0 update 1). if (!isNFC(a.schema)) throw new RecordShapeError('application schema is not NFC'); @@ -181,6 +179,9 @@ function sizeRefusal(cmb) { function canonicalRecordV2_0(cmb) { if (!isPlain(cmb)) throw new RecordShapeError('not an object'); sizeRefusal(cmb); + // Every string in its well-formed spelling before anything reads it (review M2): what the signature + // covers is its UTF-8 bytes, so this changes nothing it covers, and the projection holds one string. + cmb = wellFormedRecord(cmb); for (const k of Object.keys(cmb)) if (k !== 'categories' && k !== 'metadata') throw new RecordShapeError('carries a member outside categories and metadata'); const c = cmb.categories; if (!isPlain(c)) throw new RecordShapeError('no categories section'); @@ -199,7 +200,6 @@ function canonicalRecordV2_0(cmb) { if (m.signatureSuite !== 'mmp-sig-v2.0') throw new RecordShapeError('signatureSuite is not mmp-sig-v2.0'); if (!isLowerUuid(m.createdByNodeId)) throw new RecordShapeError('createdByNodeId is not a lowercase UUID'); if (typeof m.createdBy !== 'string' || m.createdBy.length === 0 || codePoints(m.createdBy) > 256) throw new RecordShapeError('createdBy is not a label'); - if (!isWellFormed(m.createdBy)) throw new RecordShapeError('createdBy is not well-formed Unicode'); if (!isNFC(m.createdBy)) throw new RecordShapeError('createdBy is not NFC'); if (!Number.isSafeInteger(m.createdTimestamp) || m.createdTimestamp < 0) throw new RecordShapeError('createdTimestamp is not an integer'); // A §5.8 room identifier (MMP 2.0 update 1: one room grammar everywhere). Being ASCII it is its own @@ -212,7 +212,7 @@ function canonicalRecordV2_0(cmb) { if (!isPlain(m.lineage)) throw new RecordShapeError('lineage is not an object'); for (const k of Object.keys(m.lineage)) if (k !== 'parents' && k !== 'method') throw new RecordShapeError('lineage carries an unknown member'); // Optional, unsigned and dropped (§8.8.4), but typed as the schema types it. - if (m.lineage.method !== undefined && (typeof m.lineage.method !== 'string' || m.lineage.method.length === 0 || !isWellFormed(m.lineage.method))) throw new RecordShapeError('lineage.method is not text'); + if (m.lineage.method !== undefined && (typeof m.lineage.method !== 'string' || m.lineage.method.length === 0)) throw new RecordShapeError('lineage.method is not text'); const parents = refs(m.lineage.parents, 'lineage.parents', CMB_KEY); // No parents signs as no lineage (§8.8.4), so it is held as none. lineage = parents.length > 0 ? { parents } : null; diff --git a/lib/core/text.js b/lib/core/text.js new file mode 100644 index 00000000..1faf6e69 --- /dev/null +++ b/lib/core/text.js @@ -0,0 +1,49 @@ +'use strict'; + +/** + * @module sym/core/text + * @description Well-formed text (sym 0.14.3, review M2). A record is signed as UTF-8 bytes (§8.8.4), and + * UTF-8 has no spelling for a lone surrogate: an encoder writes U+FFFD in its place. These are the one + * definition of "the string its bytes say" and of a cut that never makes a lone surrogate, used by the + * record boundary (cmb-encoder's wellFormedRecord) and by every place sym truncates a text it passes on. + * It requires nothing, so any core module can use it without a cycle. + * + * @copyright 2026 SYM.BOT Ltd. + * @license Apache-2.0 + */ + +/** + * Whether `s` is well-formed Unicode: no lone surrogate. + * @param {string} s + * @returns {boolean} + */ +const isWellFormedText = typeof String.prototype.isWellFormed === 'function' + ? (s) => typeof s === 'string' && s.isWellFormed() + : (s) => typeof s === 'string' && !/\p{Cs}/u.test(s); + +/** + * `s` with each lone surrogate replaced by U+FFFD: the string its UTF-8 bytes decode to, and so the one + * spelling of what a signature over them covers (String.prototype.toWellFormed, Node 20+). + * @param {string} s + * @returns {string} + */ +const toWellFormedText = typeof String.prototype.toWellFormed === 'function' + ? (s) => s.toWellFormed() + : (s) => s.replace(/\p{Cs}/gu, '�'); + +/** + * At most the first `max` UTF-16 units of `s`, cut on a code point boundary: a surrogate pair is never + * split, so a cut never makes a lone surrogate. One unit shorter when `max` falls inside a pair. + * @param {string} s + * @param {number} max + * @returns {string} + */ +function cutText(s, max) { + if (typeof s !== 'string' || s.length <= max) return s; + if (!(max > 0)) return ''; + const last = s.charCodeAt(max - 1); + const next = s.charCodeAt(max); + return s.slice(0, last >= 0xd800 && last <= 0xdbff && next >= 0xdc00 && next <= 0xdfff ? max - 1 : max); +} + +module.exports = { isWellFormedText, toWellFormedText, cutText }; diff --git a/lib/frame-handler.js b/lib/frame-handler.js index 3fee8140..532a4ebc 100644 --- a/lib/frame-handler.js +++ b/lib/frame-handler.js @@ -2,7 +2,7 @@ const crypto = require('crypto'); -const { recordKey, recordLineage, recordParents, recordCreatedBy, setRecordLineage } = require('./record'); +const { recordKey, recordLineage, recordParents, recordCreatedBy, setRecordLineage, recordActionable } = require('./record'); const { wireNodeId, wireName } = require('./wire-identity'); const { canonicalRecordV2_0 } = require('./core/record-canonical'); const { moodFrameReason } = require('./core/mood-frame'); @@ -28,6 +28,7 @@ const { ATTEST_FRAME, fromWireAttestation, fromWireCheckpoint, fromWireWitness, const fs = require('fs'); const path = require('path'); +const { cutText } = require('./core/text'); const { encode, cosineSimilarity, processHeuristicSVAF, buildFusedRecord, computeCategoryVerdicts, resolveTetherAnchor, tetherOfRecord, signTetherAttestation, verifyTetherAttestation, classifyAddress, categoriesAsSigned, recordAsSigned } = require('./core'); /** The schema of the signed application section that marks a directed record as a message (design D1). */ @@ -747,25 +748,36 @@ class FrameHandler { * @private */ /** - * Surface a directed (peer-bound) CMB to the application layer when SVAF - * has REJECTED it for memory. MMP §4.4.4: a CMB addressed to this node is a - * request between two agents and MUST reach the agent regardless of the SVAF - * verdict — SVAF governs memory admission only, not delivery. On the ADMIT - * path the stored remix already surfaces via the receiveFromPeer cmb-accepted - * emit, so this is invoked only from reject/redundant branches to fill the - * one gap where a directed CMB would otherwise be dropped. Returns true if it - * surfaced. The §9.3 mood channel is separate (`mood-delivered`) and is still - * delivered for a rejected directed CMB: it has no directed exemption. + * DELIVERY IS NOT MEMORY. A record memory did not store — SVAF rejected it or found it redundant, + * memory already holds its cognition key, or the store's write failed — is still delivered when its + * delivery does not depend on memory, and this is the one place that decides it: + * + * - a DIRECTED record (MMP §4.4.4) is a request between two agents and MUST reach the agent + * whatever SVAF decided: SVAF governs memory admission only, not delivery; + * - an ACTIONABLE room record (a signed application section, §8.8.2) is a message as well as a + * memory: its action is identified by its assertion identity, so memory holding its cognition + * (another author's record under the same categories, or another action of the same author) + * never suppresses it. It is delivered unless SVAF rejected it for drift (receiver-autonomous + * attention: a room record is gated) (_surfaceActionableDuplicate). + * + * Called from every branch where nothing was stored: the reject and redundant branches, and the + * admit branch when the store returned null (security review S3: the admit branch surfaced only a + * directed record, so an actionable record SVAF admitted under a cognition key memory already held + * reached nobody). A replay of the same assertion never gets here: the delivery ledger stopped it + * before SVAF (_handleMemoryShare's incoming key). Returns true if it surfaced. The §9.3 mood + * channel is separate (`mood-delivered`) and is still delivered for a rejected directed CMB: it has + * no directed exemption. * * The surfaced entry carries `remixed: false` — the receiver delivered the * CMB to the agent but did NOT ingest it into memory (no remix, no lineage). - * Consumers check this flag to distinguish a delivered-only directed CMB from + * Consumers check this flag to distinguish a delivered-only CMB from * one that was ingested (see node.js receiveFromPeer, which sets remixed:true). - * `decision` says why it was not stored: the SVAF verdict (redundant/rejected), - * or 'not-stored' when SVAF admitted it and the store's write failed. + * `decision` says why it was not stored: the SVAF verdict (redundant/rejected), 'redundant' when + * SVAF admitted it and memory already holds its cognition key, or 'not-stored' when the store's + * write failed. * @private */ - _surfaceDirectedReject(msg, peerName, peerId, now, decision) { + _surfaceUnstored(msg, peerName, peerId, now, decision) { if (!msg._directedToMe) return this._surfaceActionableDuplicate(msg, peerName, peerId, now, decision); const entry = this._markProvenance({ ...msg, @@ -794,7 +806,7 @@ class FrameHandler { */ _surfaceActionableDuplicate(msg, peerName, peerId, now, decision) { if (msg._cmbVerified !== true || msg._legacyImport) return false; - if (!(msg.cmb && msg.cmb.metadata && msg.cmb.metadata.application != null)) return false; + if (!recordActionable(msg.cmb)) return false; if (decision !== 'redundant' && decision !== 'not-stored') return false; const entry = this._markProvenance({ ...msg, @@ -960,8 +972,10 @@ class FrameHandler { * @private */ _rejectLegacyImport(peerId, peerName, msg) { - const { verifyCMB } = require('./core'); - const cmb = msg.cmb; + const { verifyCMB, wellFormedRecord } = require('./core'); + // The record in its well-formed spelling (review M2): a 0.13 suite signs UTF-8 bytes too, so this + // changes nothing its signature covers, and what is stored is the one string it covers. + const cmb = msg.cmb = wellFormedRecord(msg.cmb); const pinned = msg._legacyPinnedKey; const k = recordKey(cmb); // Only records the routed node itself authored (security review E): its pinned key signing a @@ -1611,7 +1625,7 @@ class FrameHandler { totalDrift: total_drift, categoryDrifts: category_drifts || null, gateValues: gate_values || null, - focusLabel: String((msg.cmb && msg.cmb.categories && msg.cmb.categories.focus && msg.cmb.categories.focus.text) || msg.content || '').slice(0, 120), + focusLabel: cutText(String((msg.cmb && msg.cmb.categories && msg.cmb.categories.focus && msg.cmb.categories.focus.text) || msg.content || ''), 120), }); if (decision === 'rejected') { @@ -1620,7 +1634,7 @@ class FrameHandler { // MMP §4.4.4: a directed (peer-bound) CMB surfaces even when SVAF rejects // it for memory — delivery is unconditional, memory admission is not. - this._surfaceDirectedReject(msg, peerName, peerId, now, 'rejected'); + this._surfaceUnstored(msg, peerName, peerId, now, 'rejected'); // Attest the REJECT too — a refusal is the compliance-critical gating event, // and the per-attester chain must cover every gate (omission-evidence), even @@ -1709,11 +1723,12 @@ class FrameHandler { severLineage = this._applyLineageTether(fusedEntry, tether, now); } // The admitted-path surface is the store's own cmb-accepted emit. The store returns null when - // it already holds the key ('redundant') or the write failed ('not-stored'), and a directed CMB - // then reached nobody: surface it. Everything after this runs exactly as for any admit. + // it already holds the key ('redundant') or the write failed ('not-stored'), and the record then + // reached nobody: a directed or actionable one is delivered anyway (_surfaceUnstored). Everything + // after this runs exactly as for any admit. const storedNeural = this._node._store.receiveFromPeer(peerId, fusedEntry, { creatorRole: this._getCreatorRole(peerId, msg), severLineage, authorityOver: this._authorityOver(msg) }); - if (!storedNeural && msg._directedToMe) { - this._surfaceDirectedReject(msg, peerName, peerId, now, this._node._store.get(fusedEntry.key) ? 'redundant' : 'not-stored'); + if (!storedNeural) { + this._surfaceUnstored(msg, peerName, peerId, now, this._node._store.get(fusedEntry.key) ? 'redundant' : 'not-stored'); } this._reencodeLocalStateAfterAdmit(); @@ -1795,7 +1810,7 @@ class FrameHandler { changeSignal: result.changeSignal ?? null, categoryDrifts: result.categoryDrifts || null, gateValues: result.gateValues || null, - focusLabel: String((msg.cmb && msg.cmb.categories && msg.cmb.categories.focus && msg.cmb.categories.focus.text) || msg.content || '').slice(0, 120), + focusLabel: cutText(String((msg.cmb && msg.cmb.categories && msg.cmb.categories.focus && msg.cmb.categories.focus.text) || msg.content || ''), 120), }); if (!result.accepted) { @@ -1807,7 +1822,7 @@ class FrameHandler { // MMP §4.4.4: a directed (peer-bound) CMB surfaces even when SVAF rejects // or deems it redundant for memory — delivery is unconditional. - this._surfaceDirectedReject(msg, peerName, peerId, now, result.decision); + this._surfaceUnstored(msg, peerName, peerId, now, result.decision); // Attest the reject/redundant gate too — the chain must cover every gating // event (omission-evidence). The Phase-A heuristic already produced the @@ -1840,11 +1855,12 @@ class FrameHandler { const severLineage = this._applyLineageTether(result.fusedEntry, result.tether, now); // The admitted-path surface is the store's own cmb-accepted emit. The store returns null when - // it already holds the key ('redundant') or the write failed ('not-stored'), and a directed CMB - // then reached nobody: surface it. Everything after this runs exactly as for any admit. + // it already holds the key ('redundant') or the write failed ('not-stored'), and the record then + // reached nobody: a directed or actionable one is delivered anyway (_surfaceUnstored). Everything + // after this runs exactly as for any admit. const storedHeuristic = this._node._store.receiveFromPeer(peerId, result.fusedEntry, { creatorRole: this._getCreatorRole(peerId, msg), severLineage, authorityOver: this._authorityOver(msg) }); - if (!storedHeuristic && msg._directedToMe) { - this._surfaceDirectedReject(msg, peerName, peerId, now, this._node._store.get(result.fusedEntry.key) ? 'redundant' : 'not-stored'); + if (!storedHeuristic) { + this._surfaceUnstored(msg, peerName, peerId, now, this._node._store.get(result.fusedEntry.key) ? 'redundant' : 'not-stored'); } this._reencodeLocalStateAfterAdmit(); diff --git a/lib/interior.js b/lib/interior.js index 4ce28921..6a68b2d9 100644 --- a/lib/interior.js +++ b/lib/interior.js @@ -76,7 +76,7 @@ const fs = require('fs'); const net = require('net'); const path = require('path'); const crypto = require('crypto'); -const { CAT7_CATEGORIES } = require('./core'); +const { CAT7_CATEGORIES, cutText } = require('./core'); const { signedProjection } = require('./core/record-canonical'); const recordOf = (cmb) => { try { return signedProjection(cmb); } catch { return null; } }; const { nestedTooDeep } = require('./core/json-depth'); @@ -157,7 +157,7 @@ class Interior { const kinds = Array.isArray(m.kinds) ? m.kinds.filter((k) => typeof k === 'string' && k) : []; if (kinds.length === 0) throw new Error('interior: a mission declares the kinds of submission it may make'); return { - id: typeof m.id === 'string' ? m.id.slice(0, 128) : `mission-${Date.now()}`, + id: typeof m.id === 'string' ? cutText(m.id, 128) : `mission-${Date.now()}`, kinds: new Set(kinds), allowTo: new Set(Array.isArray(m.allowTo) ? m.allowTo.filter((t) => typeof t === 'string') : []), // Records the host gives the mind to read and cite (its view holds nothing from before it started). @@ -434,11 +434,13 @@ class Interior { const own = this._node.nodeId; const scope = this._scopeKeys(m); const items = []; - for (const e of this._node.recall(req.query) || []) { + // Only what the mission may see (ruling C), and a peer's record only when it verified: decided + // newest first, so only the entries returned are read (review M1). + const seen = (e) => !!e && scope.has(e.key) && (!e.peerId || e.verified === true); + for (const e of this._node.recall(req.query, { limit, where: seen }) || []) { if (items.length >= limit) break; - if (!e || !scope.has(e.key)) continue; // only what the mission may see (ruling C) + if (!seen(e)) continue; const mine = !e.peerId; - if (!mine && e.verified !== true) continue; // a peer's record only when it verified const md = e.cmb && e.cmb.metadata; items.push({ key: e.key, record: e.cmb && md && md.signatureSuite === 'mmp-sig-v2.0' ? recordOf(e.cmb) : null, verified: mine ? true : e.verified === true, storedAt: e.storedAt ?? null, author: mine ? { name: this._node.name, nodeId: own } : (e.author || null) }); } diff --git a/lib/legacy-import.js b/lib/legacy-import.js index 52d9e7ea..c52a4aaa 100644 --- a/lib/legacy-import.js +++ b/lib/legacy-import.js @@ -59,6 +59,7 @@ const LEGACY_DEAD_MS = 45_000; const ROUTES_FILE = 'legacy-routes.json'; const LEGACY_ENCRYPTION = 'legacy (X25519 + AES-256-GCM per connection; no forward secrecy, no transcript proof)'; const { isCanonicalNodeId } = require('./core/record-canonical'); +const { toWellFormedText, cutText } = require('./core/text'); const isNodeId = (id) => isCanonicalNodeId(id); /** The X25519 SubjectPublicKeyInfo prefix (a 0.13 key travels as SPKI DER, base64). */ const X25519_SPKI = Buffer.from('302a300506032b656e032100', 'hex'); @@ -100,7 +101,7 @@ function checkRoute(r) { if (!fingerprint) throw new Error(`legacy route ${r.nodeId.slice(0, 8)}: the identity key fingerprint is mandatory`); const e2e = x25519Key(r.e2eKey); if (!e2e) throw new Error(`legacy route ${r.nodeId.slice(0, 8)}: e2eKey (the 0.13 node's persistent X25519 public key, base64) is mandatory`); - return { nodeId: r.nodeId, endpoint, key, fingerprint, e2eKey: e2e.toString('base64'), name: typeof r.name === 'string' ? r.name.slice(0, 64) : null }; + return { nodeId: r.nodeId, endpoint, key, fingerprint, e2eKey: e2e.toString('base64'), name: typeof r.name === 'string' ? cutText(toWellFormedText(r.name), 64) : null }; } /** @@ -204,7 +205,7 @@ class LegacySession extends EventEmitter { }, LEGACY_PING_MS); if (this._beat.unref) this._beat.unref(); this.nodeId = h.nodeId; - this.name = typeof h.name === 'string' && h.name ? h.name.slice(0, 256) : (this.route.name || 'legacy'); + this.name = typeof h.name === 'string' && h.name ? cutText(toWellFormedText(h.name), 256) : (this.route.name || 'legacy'); this.identityKey = h.publicKey; this.state = 'confirmed'; this.confirmedAt = Date.now(); diff --git a/lib/memory-store.js b/lib/memory-store.js index 69eabad5..2b6898f6 100644 --- a/lib/memory-store.js +++ b/lib/memory-store.js @@ -22,7 +22,7 @@ const fs = require('fs'); const path = require('path'); const crypto = require('crypto'); const { ensureDir } = require('./config'); -const { isLineageSevered } = require('./core'); +const { isLineageSevered, wellFormedRecord, isWellFormedText, toWellFormedText, cutText } = require('./core'); // ── In-Memory Index ────────────────────────────────────────── @@ -67,6 +67,12 @@ const ENTRY_ANNOTATIONS = Object.freeze(['admission', 'tether', 'provenance', 'c * metadata) is left alone: its top-level members are its own sections. */ function asStored(entry) { + // Every string the record signs, and the text rendered from it, in its well-formed spelling (review + // M2): a store written by 0.14.2 can hold a lone surrogate a peer's record carried. Nothing signed changes. + if (entry && typeof entry === 'object') { + if (entry.cmb && typeof entry.cmb === 'object') entry.cmb = wellFormedRecord(entry.cmb); + if (typeof entry.content === 'string' && !isWellFormedText(entry.content)) entry.content = toWellFormedText(entry.content); + } const cmb = entry && entry.cmb; if (!cmb || typeof cmb !== 'object' || !cmb.metadata || typeof cmb.metadata !== 'object') return entry; const own = (k) => Object.prototype.hasOwnProperty.call(cmb, k); @@ -124,6 +130,11 @@ class EntryCache { this._map.set(key, slot); return slot.entry; } + /** The entry, without making it the most recently used: a scan reads through the cache, never into it. */ + peek(key) { + const slot = this._map.get(key); + return slot ? slot.entry : undefined; + } set(key, entry) { const old = this._map.get(key); if (old) { this._weight -= old.weight; this._map.delete(key); } @@ -150,6 +161,107 @@ class EntryCache { [Symbol.iterator]() { return this.entries(); } } +/** + * THE SEARCH INDEX (review M1, 0.14.3), kept apart from the entry cache. recall() matches a query + * against each entry's text; with the cache bounded (R2-3) that read a file for every entry the cache + * did not hold, on every call, synchronously: 6,000 entries took 140 ms a call (0.14.2, everything in + * memory: 4 ms). What a match needs is far smaller than an entry: its rendered text, lowercased. This + * index holds only that, per entry at most SEARCH_ENTRY_CHARS (an entry longer than that is marked + * `partial`, and only a query its indexed text does not match reads it), and SEARCH_MAX_CHARS in all + * (past that the oldest entries' text is let go, and only those are read). Its text is the entry's + * content (which renders every category that is not 'neutral', mood with its numbers), its tags, and any + * category text the content does not contain; the key is matched as it is. A query matches within one + * of those, never across two. + */ +const SEARCH_ENTRY_CHARS = 4096; +const SEARCH_MAX_CHARS = 32 * 1024 * 1024; +/** What one indexed entry costs beyond its text (the slot and the Map's own), counted against the bound. */ +const SEARCH_SLOT_CHARS = 64; + +/** The lowercased text recall() matches an entry by, before the per-entry cap (see SearchIndex). */ +function searchText(entry) { + const content = typeof entry.content === 'string' ? entry.content : ''; + const parts = [content]; + for (const t of entry.tags || []) if (typeof t === 'string' && t) parts.push(t); + const cats = entry.cmb && entry.cmb.categories; + if (cats && typeof cats === 'object') { + const seen = new Set(); + for (const v of Object.values(cats)) { + const t = v && typeof v.text === 'string' ? v.text : null; + if (t && !seen.has(t) && !content.includes(t)) { seen.add(t); parts.push(t); } + } + } + return parts.join('\n').toLowerCase(); +} + +class SearchIndex { + constructor({ maxChars = SEARCH_MAX_CHARS, entryChars = SEARCH_ENTRY_CHARS } = {}) { + this._map = new Map(); // key -> { text, partial }, the oldest stored first + this._chars = 0; + this._maxChars = maxChars; + this._entryChars = entryChars; + this._staged = null; + } + get size() { return this._map.size; } + /** The characters held, slots included. */ + get chars() { return this._chars; } + get(key) { return this._map.get(key); } + /** @private An entry's slot: its text, at most its share, and whether that is all of it. */ + _slot(entry) { + const full = searchText(entry); + const text = cutText(full, this._entryChars); + return { text, partial: text.length < full.length }; + } + /** @private Put a slot in as the newest, then let the oldest go while over the bound. */ + _put(key, slot) { + this.delete(key); + this._map.set(key, slot); + this._chars += slot.text.length + SEARCH_SLOT_CHARS; + for (const k of this._map.keys()) { + if (this._chars <= this._maxChars) break; + this.delete(k); + } + } + /** Index an entry just stored (the newest). */ + add(key, entry) { this._put(key, this._slot(entry)); } + /** + * Index an entry read while the store loads, in any order: held aside (its slot only) until commit(), + * which puts them in the order they were stored, so the bound lets the oldest go. + */ + stage(key, entry, storedAt) { + const slot = this._slot(entry); + (this._staged ||= []).push([storedAt || 0, key, slot]); + this._stagedChars = (this._stagedChars || 0) + slot.text.length + SEARCH_SLOT_CHARS; + // Held aside within twice the bound: past it, only the newest that fit the bound are kept. + if (this._stagedChars > 2 * this._maxChars) this._keepNewestStaged(); + } + /** @private */ + _keepNewestStaged() { + this._staged.sort((a, b) => b[0] - a[0]); + let n = 0; let i = 0; + for (; i < this._staged.length; i++) { + const c = this._staged[i][2].text.length + SEARCH_SLOT_CHARS; + if (n + c > this._maxChars) break; + n += c; + } + this._staged.length = i; + this._stagedChars = n; + } + commit() { + const staged = this._staged || []; + this._staged = null; + this._stagedChars = 0; + staged.sort((a, b) => a[0] - b[0]); + for (const [, key, slot] of staged) this._put(key, slot); + } + delete(key) { + const slot = this._map.get(key); + if (!slot) return false; + this._chars -= slot.text.length + SEARCH_SLOT_CHARS; + return this._map.delete(key); + } +} + class MemoryIndex { /** Create an empty index. */ constructor() { @@ -230,6 +342,29 @@ class MemoryIndex { } } +/** + * The newest entries read while a store is loaded, at most `max` of them held at once (twice that + * between trims): what the entry cache is filled with when the load ends, so it starts with the entries + * read newest first (recall's order, the anchors', recent()'s), not with whichever files were read last. + */ +class NewestEntries { + constructor(max) { this._max = max; this._list = []; } + add(entry) { + this._list.push(entry); + if (this._list.length > 2 * this._max) this._trim(); + } + _trim() { + this._list.sort((a, b) => (b.storedAt || b.timestamp || 0) - (a.storedAt || a.timestamp || 0)); + this._list.length = Math.min(this._list.length, this._max); + } + /** Put them in `cache`, the oldest first, so the newest are the most recently used. */ + fill(cache) { + this._trim(); + for (let i = this._list.length - 1; i >= 0; i--) cache.set(this._list[i].key, this._list[i]); + this._list = []; + } +} + // ── Memory Store ───────────────────────────────────────────── class MemoryStore { @@ -245,7 +380,8 @@ class MemoryStore { this.__indexed = false; // has the index been built (by either path)? this.__loading = null; // the in-flight asynchronous load, if any this._index = new MemoryIndex(); - this._cache = new EntryCache(); // key → full entry (avoids disk reads on recall); bounded (R2-3) + this._cache = new EntryCache(); // key → full entry (avoids disk reads); bounded (R2-3) + this._search = new SearchIndex(); // key → the text recall() matches (review M1); bounded ensureDir(this._dir); @@ -273,6 +409,8 @@ class MemoryStore { set _index(v) { this.__index = v; } get _cache() { this._ensureIndex(); return this.__cache; } set _cache(v) { this.__cache = v; } + get _search() { this._ensureIndex(); return this.__search; } + set _search(v) { this.__search = v; } /** Build the index synchronously if nothing has yet — the old behaviour, now the fallback. */ _ensureIndex() { @@ -295,6 +433,8 @@ class MemoryStore { this.__loading = (async () => { const index = new MemoryIndex(); const cache = new EntryCache(); + const search = new SearchIndex(); + const newest = new NewestEntries(CACHE_MAX_ENTRIES); let files = []; try { files = (await fs.promises.readdir(this._dir)).filter(isStoreFile); } catch { files = []; } for (let i = 0; i < files.length; i += batch) { @@ -306,13 +446,18 @@ class MemoryStore { })); for (const p of parsed) { if (!p) continue; - cache.set(p.entry.key, p.entry); - index.add(this._indexEntryFor(p.entry, p.filePath)); + const ie = this._indexEntryFor(p.entry, p.filePath); + if (!index.add(ie)) continue; + search.stage(p.entry.key, p.entry, ie.storedAt); + newest.add(p.entry); } } if (this.__indexed) return; // a synchronous fallback won the race; its index is authoritative + newest.fill(cache); + search.commit(); this.__index = index; this.__cache = cache; + this.__search = search; this.__indexed = true; })(); try { await this.__loading; } finally { this.__loading = null; } @@ -325,6 +470,8 @@ class MemoryStore { return { key: entry.key, storedAt: entry.storedAt || entry.timestamp || 0, + createdAt: recordCreatedAt(entry.cmb), + lifecycle: entry.lifecycle || 'observed', createdBy: recordCreatedBy(entry.cmb) || entry.source || this._source, peerId: entry.peerId || null, parents: severed ? [] : (entry.lineage?.parents || recordParents(entry.cmb)), @@ -374,6 +521,8 @@ class MemoryStore { this._index.add({ key, storedAt: now, + createdAt: recordCreatedAt(entry.cmb), + lifecycle: 'observed', createdBy: this._source, peerId: null, parents: lineage?.parents || [], @@ -381,6 +530,7 @@ class MemoryStore { tier: 'hot', filePath: this._filePath(key), }); + this._searchAdd(key, entry); // Section 6.4: advance parent CMB lifecycle when lineage exists. // A CMB with parents = remix → parent advances to 'remixed'. @@ -438,6 +588,8 @@ class MemoryStore { this._index.add({ key, storedAt: now, + createdAt: recordCreatedAt(stored.cmb), + lifecycle: 'observed', createdBy: recordCreatedBy(cmb) || entry.source || peerId, peerId, parents: lineage?.parents || [], @@ -445,6 +597,7 @@ class MemoryStore { tier: 'hot', filePath: this._filePath(key), }); + this._searchAdd(key, stored); // Section 6.4: advance parent CMB lifecycle. this._advanceParentLifecycle(lineage, 'remixed'); @@ -538,15 +691,15 @@ class MemoryStore { /** * The newest `createdTimestamp` among this node's own stored records (those no peer delivered), over - * the whole index, not a recent window: the timestamp ratchet starts from it. O(records), once; it - * reads without filling the cache. + * the whole index, not a recent window: the timestamp ratchet starts from it. Read from the index, + * which holds each record's createdTimestamp: no file is read (review M1). * @returns {number} 0 when there is none */ newestOwnCreatedAt() { let max = 0; - for (const [key, idx] of this._index.byKey) { + for (const idx of this._index.byKey.values()) { if (idx.peerId != null) continue; - const t = recordCreatedAt((this._cache.get(key) || this._load(idx.filePath))?.cmb); + const t = idx.createdAt; if (Number.isFinite(t) && t > max) max = t; } return max; @@ -660,42 +813,66 @@ class MemoryStore { // ── Search (user-facing) ───────────────────────────────── /** - * Keyword search across CMB category texts, content, and tags. - * For CLI `sym recall "..."` — not for LLM context curation. + * Keyword search across CMB category texts, content, and tags: the entries whose text contains the + * query (case-insensitive), the newest stored first. For CLI `sym recall "..."` — not for LLM + * context curation. No query returns every entry (the agent decides relevance, not the store). + * + * MATCHED IN MEMORY, READ ONLY FOR WHAT IS RETURNED (review M1): the match runs on the search index + * (SearchIndex), so a query costs no file read for an entry it does not return; an entry is read + * (when the cache does not hold it) only to be returned, and with `limit` only that many are. A scan + * reads through the cache and never into it, so a recall does not push the working set out. + * + * @param {string} query + * @param {{ limit?: number, where?: function(object): boolean }} [opts] + * limit — at most this many entries are returned (and read); `where` — only entries it accepts + * are returned (checked newest first, so only as many are read as it takes to fill `limit`) + * @returns {object[]} the entries; the array's non-enumerable `total` is the number of entries the + * query matched, whatever the limit */ - recall(query) { - const results = []; - + recall(query, opts = {}) { + const q = typeof query === 'string' && query.trim() !== '' ? query.toLowerCase() : null; + const limit = Number.isSafeInteger(opts.limit) && opts.limit > 0 ? opts.limit : Infinity; + const where = typeof opts.where === 'function' ? opts.where : null; + const hits = []; for (const [key, idx] of this._index.byKey) { - const entry = this._cache.get(key) || this._load(idx.filePath); + if (q === null || this._matches(key, idx, q)) hits.push(idx); + } + // Newest stored first (a stable sort: entries stored in one millisecond keep the index's order). + hits.sort((a, b) => (b.storedAt || 0) - (a.storedAt || 0)); + const results = []; + for (const idx of hits) { + if (results.length >= limit) break; + const entry = this._cache.peek(idx.key) || this._load(idx.filePath); if (!entry) continue; - - // No query = return all (the agent decides relevance, not the store) - if (!query || query.trim() === '') { - results.push(entry); - continue; - } - - // Keyword filter when query provided - const q = query.toLowerCase(); - const parts = [entry.content || '', entry.key || '', ...(entry.tags || [])]; - - if (entry.cmb?.categories) { - for (const value of Object.values(entry.cmb.categories)) { - if (value?.text) parts.push(value.text); - } - } - - if (parts.join(' ').toLowerCase().includes(q)) { - results.push(entry); - } + if (where && !where(entry)) continue; + results.push(entry); } + Object.defineProperty(results, 'total', { value: hits.length, enumerable: false, configurable: true }); + return results; + } - return results.sort((a, b) => (b.storedAt || b.timestamp || 0) - (a.storedAt || a.timestamp || 0)); + /** + * Whether an entry's text contains `q` (already lowercased): its key, then its indexed text. Only an + * entry the index holds no text for (past SEARCH_MAX_CHARS), or holds a prefix of that does not + * match, is read. + * @private + */ + _matches(key, idx, q) { + if (key.toLowerCase().includes(q)) return true; + const slot = this._search.get(key); + if (slot && slot.text.includes(q)) return true; + if (slot && !slot.partial) return false; + const entry = this._cache.peek(key) || this._load(idx.filePath); + return !!entry && searchText(entry).includes(q); + } + + /** Index an entry just stored for recall(), within the search index's bound. @private */ + _searchAdd(key, entry) { + this._search.add(key, entry); } /** Backward compat alias */ - search(query) { return this.recall(query); } + search(query, opts) { return this.recall(query, opts); } /** * Check if a CMB key exists in local entries (authored by this node). @@ -837,6 +1014,7 @@ class MemoryStore { } this._index.remove(key); this._cache.delete(key); // fully evict — a purged CMB must not resurrect from cache via get() + this._search.delete(key); removed++; } @@ -988,6 +1166,9 @@ class MemoryStore { * @returns {string} lifecycle state or 'observed' */ getLifecycle(key) { + // The index holds it (kept by _persist), so compaction and purge read no file for it (review M1). + const idx = this._index.get(key); + if (idx && typeof idx.lifecycle === 'string') return idx.lifecycle; const entry = this.get(key); return entry?.lifecycle || 'observed'; } @@ -1031,6 +1212,9 @@ class MemoryStore { _persist(entry) { try { fs.writeFileSync(this._filePath(entry.key), JSON.stringify(entry, null, 2)); + // The index's copy of the lifecycle follows every write of it (validate, canonize, dismiss, remix). + const idx = this.__index && this.__index.get(entry.key); + if (idx) idx.lifecycle = entry.lifecycle || 'observed'; return true; } catch (e) { console.error(`[SYM] cmbs: write failed: ${e.message}`); @@ -1050,16 +1234,21 @@ class MemoryStore { if (!fs.existsSync(this._dir)) return; const files = fs.readdirSync(this._dir).filter(isStoreFile); + const newest = new NewestEntries(CACHE_MAX_ENTRIES); for (const file of files) { const filePath = path.join(this._dir, file); try { const entry = asStored(JSON.parse(fs.readFileSync(filePath, 'utf8'))); - this._cache.set(entry.key, entry); - this._index.add(this._indexEntryFor(entry, filePath)); + const ie = this._indexEntryFor(entry, filePath); + if (!this._index.add(ie)) continue; + this._search.stage(entry.key, entry, ie.storedAt); + newest.add(entry); } catch (e) { console.error(`[SYM] cmbs: failed to index ${file}: ${e.message}`); } } + newest.fill(this._cache); + this._search.commit(); } // ── Migration ──────────────────────────────────────────── @@ -1167,4 +1356,4 @@ class MemoryStore { } } -module.exports = { MemoryStore, EntryCache, CACHE_MAX_ENTRIES, CACHE_MAX_CHARS }; +module.exports = { MemoryStore, EntryCache, SearchIndex, CACHE_MAX_ENTRIES, CACHE_MAX_CHARS, SEARCH_ENTRY_CHARS, SEARCH_MAX_CHARS }; diff --git a/lib/mesh-agent.js b/lib/mesh-agent.js index 5cc0d49b..a283c9bc 100644 --- a/lib/mesh-agent.js +++ b/lib/mesh-agent.js @@ -425,7 +425,8 @@ class MeshAgent { // ── Helpers ──────────────────────────────────────────── _meshContext() { - return (this._node.recall('') || []) + // The five newest entries another agent wrote, and only those are read (review M1). + return (this._node.recall('', { limit: 5, where: (r) => r.source !== this._name }) || []) .filter(r => r.source !== this._name) .slice(0, 5) .map(r => ({ diff --git a/lib/node.js b/lib/node.js index 67d0e6d0..46442f7b 100644 --- a/lib/node.js +++ b/lib/node.js @@ -1,6 +1,6 @@ 'use strict'; -const { recordKey, recordCreatedBy, recordParents } = require('./record'); +const { recordKey, recordCreatedBy, recordParents, recordActionable, recordAssertion } = require('./record'); /** * SymNode — sovereign mesh node with cognitive coupling. @@ -54,7 +54,7 @@ const { WakeManager, signCMB, assertionIdV2_0, mintRemixKey, signAttestation, verifyAttestation, verifyAttestationRole, merkleRoot, signCheckpoint, verifyCheckpoint, signWitness, verifyWitness, - CAT7_CATEGORIES, assertRecordSendable, + CAT7_CATEGORIES, assertRecordSendable, wellFormedRecord, toWellFormedText, cutText, attestMerkleRoot, attestCheckpointRoot, toWireAttestation, toWireCheckpoint, toWireWitness, ATTEST_FRAME, } = require('./core'); /** What a checkpoint made since 0.14 commits (the sym-attest-v1 root); one without it is 0.13's. */ @@ -1330,12 +1330,12 @@ class SymNode extends EventEmitter { } /** - * Aggregate every attestation about a CMB into a verdict WEIGHTED BY EARNED AUTHORITY - * (MMP §6.4/§6.5). The per-attestation `verifyAttestationRole` check answers "is this - * one attester's claimed role real"; this answers "what does the roster, weighted by - * who actually holds rank, conclude about this CMB". An anchor's admit outweighs a - * participant's; a node over-claiming a role it never earned is down-weighted to its - * resolved rank and surfaced as evidence, not silently trusted. + * Aggregate the attestations about ONE ASSERTION into a verdict WEIGHTED BY EARNED AUTHORITY + * (MMP §6.4/§6.5). The per-attestation `verifyAttestationRole` check answers "is this one + * attester's claimed role real"; this answers "what does the roster, weighted by who actually + * holds rank, conclude about this assertion". An anchor's admit outweighs a participant's; a node + * over-claiming a role it never earned is down-weighted to its resolved rank and surfaced as + * evidence, not silently trusted. * * Each attestation is (1) signature-verified against the attester's key from the roster * registry — unverifiable ones are EXCLUDED, never weighted — and (2) role-resolved against @@ -1345,23 +1345,77 @@ class SymNode extends EventEmitter { * overall and per-CAT7-category verdicts are weighted tallies; `confidence` is the dominant * verdict's share of total weight. * - * ONE ASSERTION, NOT ONE COGNITION KEY (security review R2-8; sym-attest-v1 §6: a receiver that weighs - * an attestation finds the attested record by `assertionId`). Records of different authors, or of one - * author with other signed application bytes, share a cognition key; only attestations naming the - * assertion are weighed: the held record's under `cmbKey`, or `opts.assertionId`. Any other is - * excluded (`other-assertion`), and with no assertion to match, every one is (`assertion-unknown`). + * A SUMMARY IS ALWAYS ABOUT ONE NAMED ASSERTION (security review R2-8 and B2; sym-attest-v1 §6: a + * receiver that weighs an attestation finds the attested record by `assertionId`). Records of + * different authors, or of one author with other signed application bytes, share a cognition key, + * and an attestation about one is never a vote on another. Which assertion (`basis`): + * + * 'given' `opts.assertionId`; + * 'held' the record this node holds under `cmbKey`; + * 'sole' neither, and every attestation under the key names the same assertion: that one, + * returned as `assertionId` (a node without the record, an observer with no store); + * 'several' neither, and they name more than one: there is no single verdict to give, so + * `assertionId` is null, `overall.dominant` is null and `categories` is empty, while + * `total`, `weight` and `byRole` count every verified attestation under the key (they + * were made; they are not a vote on one record). `byAssertion` has the verdicts; + * 'none' no attestation under the key names an assertion. + * + * `byAssertion` always holds one summary per assertion named under the key, so a caller can choose. + * An attestation naming no assertion is excluded as `assertion-unknown`; in the 'given' and 'held' + * cases one naming another assertion is excluded as `other-assertion` (and weighed in its own + * `byAssertion` summary). * @param {string} cmbKey — the gated CMB key (`of`) - * @param {{ assertionId?: string }} [opts] — the assertion to weigh, when this node does not hold the - * record (or to weigh another assertion under the same key) - * @returns {{ of: string, assertionId: string|null, total: number, weight: number, byRole: object, - * overall: object, categories: object, mismatches: object[], excluded: object[] }} + * @param {{ assertionId?: string }} [opts] — the assertion to weigh (when this node does not hold + * the record, or to weigh another assertion under the same key) + * @returns {{ of: string, assertionId: string|null, basis: string, total: number, weight: number, + * byRole: object, overall: object, categories: object, mismatches: object[], excluded: object[], + * assertions: number, byAssertion: object }} */ aggregateAttestations(cmbKey, opts = {}) { let assertionId = typeof opts.assertionId === 'string' && opts.assertionId ? opts.assertionId : null; + let basis = assertionId ? 'given' : null; if (!assertionId) { - try { const e = this._store.get(cmbKey); const a = e && e.cmb && e.cmb.metadata && e.cmb.metadata.assertionId; if (typeof a === 'string' && a) assertionId = a; } catch { /* none held */ } + try { const e = this._store.get(cmbKey); const a = e && e.cmb && e.cmb.metadata && e.cmb.metadata.assertionId; if (typeof a === 'string' && a) { assertionId = a; basis = 'held'; } } catch { /* none held */ } + } + // Every attestation under the key, by the assertion it names; one naming none is weighed for nothing. + const groups = new Map(); + const unnamed = []; + for (const att of this._attestations.byCmb(cmbKey)) { + const a = att && typeof att.assertionId === 'string' && att.assertionId ? att.assertionId : null; + if (!a) { unnamed.push({ by: att && att.by, reason: 'assertion-unknown' }); continue; } + if (!groups.has(a)) groups.set(a, []); + groups.get(a).push(att); + } + const byAssertion = Object.create(null); + for (const [a, list] of groups) byAssertion[a] = this._weighAttestations(list); + if (!assertionId) { + if (groups.size === 1) { assertionId = groups.keys().next().value; basis = 'sole'; } + else basis = groups.size > 1 ? 'several' : 'none'; + } + const out = { of: cmbKey, assertionId, basis, assertions: groups.size, byAssertion }; + if (assertionId) { + const own = byAssertion[assertionId] || this._weighAttestations([]); + const others = []; + for (const [a, list] of groups) if (a !== assertionId) for (const att of list) others.push({ by: att.by, reason: 'other-assertion' }); + return { ...out, ...own, excluded: [...unnamed, ...others, ...own.excluded] }; } - const atts = this._attestations.byCmb(cmbKey); + // No one assertion: what was attested is counted, and no verdict is given for it. + const byRole = { participant: 0, issuer: 0, validator: 0, admin: 0, anchor: 0 }; + let total = 0; let weight = 0; + const mismatches = []; const excluded = [...unnamed]; + for (const s of Object.values(byAssertion)) { + total += s.total; weight += s.weight; + for (const [r, n] of Object.entries(s.byRole)) byRole[r] = (byRole[r] || 0) + n; + mismatches.push(...s.mismatches); excluded.push(...s.excluded); + } + return { ...out, total, weight, byRole, overall: summarizeTally({}, 0), categories: {}, mismatches, excluded }; + } + + /** + * The authority-weighted tally of attestations that all name one assertion (see aggregateAttestations). + * @private + */ + _weighAttestations(atts) { const overall = Object.create(null); // verdict -> summed weight const categories = Object.create(null); // cat7 category -> { verdict -> weight } const byRole = { participant: 0, issuer: 0, validator: 0, admin: 0, anchor: 0 }; @@ -1369,11 +1423,7 @@ class SymNode extends EventEmitter { const excluded = []; let total = 0; let weight = 0; - for (const att of atts) { - // (0) the assertion: an attestation about another record under this key is not a vote on this one. - if (!assertionId) { excluded.push({ by: att.by, reason: 'assertion-unknown' }); continue; } - if (att.assertionId !== assertionId) { excluded.push({ by: att.by, reason: 'other-assertion' }); continue; } // (1) signature gate — an attestation we cannot verify is evidence, not a vote. const key = this._identityKey(att.by); if (!key) { excluded.push({ by: att.by, reason: 'unknown-key' }); continue; } @@ -1383,7 +1433,6 @@ class SymNode extends EventEmitter { const rr = verifyAttestationRole(att, () => this._attesterRole(att)); if (!rr.matches) mismatches.push({ by: att.by, claimed: rr.claimed, resolved: rr.resolved }); const w = 2 ** authorityMixin.lifecycleRank(rr.resolved); // participant and issuer 1, validator 2, admin and anchor 4 - total += 1; weight += w; byRole[rr.resolved] = (byRole[rr.resolved] || 0) + 1; @@ -1393,10 +1442,7 @@ class SymNode extends EventEmitter { categories[f][v] = (categories[f][v] || 0) + w; } } - return { - of: cmbKey, - assertionId, total, weight, byRole, @@ -2249,7 +2295,7 @@ class SymNode extends EventEmitter { let left = CONTEXT_MAX_CHARS; const take = (text) => { if (left <= 0 || typeof text !== 'string' || !text) return; - const t = text.length > Math.min(CONTEXT_ENTRY_CHARS, left) ? text.slice(0, Math.min(CONTEXT_ENTRY_CHARS, left)) : text; + const t = cutText(text, Math.min(CONTEXT_ENTRY_CHARS, left)); parts.push(t); left -= t.length + 1; }; @@ -3089,6 +3135,9 @@ class SymNode extends EventEmitter { // broken must say so, not quietly emit records every verifying peer will refuse. this._signOrThrow(opts.cmb, 'CMB signing failed'); } + // A caller's own record is held in the well-formed spelling of what its signature covers (review + // M2): the same signed bytes, one string. createCMB mints every record it builds that way. + if (!builtHere) opts.cmb = wellFormedRecord(opts.cmb); // A record no transport can carry is refused here, before it is stored as sent (ECMBSIZE). // createCMB bounds what it builds; a payload, or a caller's own record, is bounded only here. assertRecordSendable(opts.cmb); @@ -3104,6 +3153,18 @@ class SymNode extends EventEmitter { // discarded, so the store looks right and the timeline is wrong. Nothing is minted, HEAD // does not advance, and the caller is handed the address that already says this. const mintKey = opts.cmb?.metadata?.key; + // AN ACTIONABLE RECORD IS A MESSAGE AS WELL AS A MEMORY (§8.8.2; security review S3). Its cognition + // is remembered under the cognition key, once per key; its action is sent and delivered by its + // assertion identity. So memory already holding this cognition (it is this node's HEAD, or the + // store holds the key: an earlier action under the same categories, or a peer's record) decides only + // that nothing is stored again, never that the action is not sent: it goes out (_sendAction). The + // one record not sent again is this very assertion, already stored as it is (a caller's record + // given twice): it went out when it was stored, and is a duplicate below. + if (mintKey && recordActionable(opts.cmb)) { + const held = this._store.get(mintKey); + const same = !!held && recordAssertion(held.cmb) !== null && recordAssertion(held.cmb) === recordAssertion(opts.cmb); + if (!same && (held || (this._head && mintKey === this._head))) return this._sendAction(opts, mintKey, held, builtHere); + } if (mintKey && this._head && mintKey === this._head) { // Clear the lineage before handing the record back. It was populated during construction // with [own HEAD] per Rule A — and since this address IS the HEAD, that lineage is exactly @@ -3144,13 +3205,15 @@ class SymNode extends EventEmitter { // 0.11.0), never the null a refusal or a failure returns. if (!entry) { const existingKey = opts.cmb?.metadata?.key || opts.cmb?.key || null; - if (!opts.to) { - if (existingKey && this._store.get(existingKey)) return { key: existingKey, duplicate: true }; + const stored = existingKey ? this._store.get(existingKey) : null; + // A broadcast memory is not sent again, nor an action already stored as this very assertion. An + // action whose store write failed is still sent, as a directed send is: its delivery is not memory. + if (!opts.to && (stored || !recordActionable(opts.cmb))) { + if (stored) return { key: existingKey, duplicate: true }; return null; // the store write failed: nothing was stored } // Entry-shaped, like any other return, but built from the caller's own record: the stored entry // under this key may be a peer's admission, and its provenance is not this send's (re-review F5). - const stored = existingKey ? this._store.get(existingKey) : null; // The store returns null both for a key it already holds and for a write that failed; only // the first is a duplicate (r3 F3). const existing = { @@ -3164,7 +3227,7 @@ class SymNode extends EventEmitter { persisted: !!stored, }; if (!stored) this._log(`STORE WRITE FAILED for ${String(existingKey).slice(0, 16)} — sending, but not stored locally`); - this._dispatchExisting(existing, existingKey, opts.to); + this._dispatchExisting(existing, existingKey, opts.to || null); return existing; } @@ -3306,11 +3369,15 @@ class SymNode extends EventEmitter { * See MMP v0.2.0 Section 6 (Memory). * * @param {string} query — search keyword - * @returns {Array} matching entries sorted by recency + * @param {{ limit?: number, where?: function(object): boolean }} [opts] — return at most `limit` + * entries, only those `where` accepts; matching is in memory, and only returned entries are read + * (MemoryStore.recall) + * @returns {Array} matching entries sorted by recency; the array's non-enumerable `total` + * is how many entries the query matched */ - recall(query) { + recall(query, opts) { this._metrics.recalls++; - return this._store.search(query); + return this._store.search(query, opts); } // ── Startup Primer ───────────────────────────────────────── @@ -3348,16 +3415,17 @@ class SymNode extends EventEmitter { const maxAgeMs = Number.isInteger(opts.maxAgeMs) && opts.maxAgeMs > 0 ? opts.maxAgeMs : 86_400_000; // 24h const cutoff = Date.now() - maxAgeMs; - // recall('') returns every entry sorted newest-first — the remix store - // is the agent's memory, not a curated view. No tag-filter gating. - const all = this._store.search(''); - const totalInStore = all.length; + // recall('') returns entries sorted newest-first — the remix store is the agent's memory, not a + // curated view. No tag-filter gating. Only the newest maxCount are read (review M1): the window + // is a prefix of that order, so it cuts the same entries out of them as out of the whole store. + const newest = this._store.search('', { limit: maxCount }); + const totalInStore = newest.total; if (!totalInStore) { return { text: '', count: 0, dropped: 0, totalInStore: 0 }; } // Apply recency window first, then count cap. - const withinWindow = all.filter((e) => (e.storedAt || e.timestamp || 0) >= cutoff); + const withinWindow = newest.filter((e) => (e.storedAt || e.timestamp || 0) >= cutoff); const kept = withinWindow.slice(0, maxCount); const dropped = totalInStore - kept.length; @@ -3709,6 +3777,8 @@ class SymNode extends EventEmitter { /** @private A signed v2.0 directed record carrying a message to `to`. */ _mintMessage(content, to) { assertAddressee(to); + // The text as its UTF-8 bytes say it (review M2), cut for the preview on a code point boundary. + content = toWellFormedText(String(content)); const bytes = Buffer.from(content, 'utf8'); const application = { mediaType: 'text/plain', @@ -3718,7 +3788,7 @@ class SymNode extends EventEmitter { digest: `sha256-${crypto.createHash('sha256').update(bytes).digest('hex')}`, data: bytes.toString('base64url'), }; - const preview = content.length > 4096 ? `${content.slice(0, 4096)}…` : content; + const preview = content.length > 4096 ? `${cutText(content, 4096)}…` : content; const cmb = createCMB({ categories: { focus: preview || '(empty message)', intent: 'message' }, createdBy: this.name, room: this._room, to, emitV2: true, createdByNodeId: this.nodeId, application }); const clock = cmb.metadata.createdTimestamp; const ts = Math.max(clock, this._timestampRatchetBase() + 1); @@ -3749,7 +3819,7 @@ class SymNode extends EventEmitter { } const peer = this._peers.get(peerId); if (!peer) return; - peer.transport.send({ type: 'error', code, message: String(message ?? '').slice(0, 200), ...(detail ? { detail: String(detail).slice(0, 200) } : {}) }); + peer.transport.send({ type: 'error', code, message: cutText(String(message ?? ''), 200), ...(detail ? { detail: cutText(String(detail), 200) } : {}) }); this._log(`Error sent to ${peer.name}: ${code} ${message}`); } @@ -5288,21 +5358,65 @@ class SymNode extends EventEmitter { } /** - * Deliver the record of a send that minted nothing (collapsed onto HEAD, or already stored) to - * its one addressed peer, and attach the same `delivery` result a fresh send carries, so the - * send is answerable instead of looking delivered. The record is the caller's freshly signed - * one: a new assertion of the same cognition (MMP §8.8.2), not a replay of an earlier send. + * Send an ACTIONABLE record whose cognition memory already holds (remember(); §8.8.2, security + * review S3): it is this node's HEAD, or the store holds its cognition key under another assertion. + * The action is a message and goes out — to its addressed peer, or to the room — while nothing is + * stored: memory keeps the record it holds under that key (memory de-duplication uses the cognition + * key) and HEAD does not move. The result is entry-shaped and built from THIS record (its own + * assertion, never the held one's), with `cognitionHeld: true` and the same `delivery` a fresh send + * carries. + * + * A record that names its own key as a parent (K -> K: it cites the record whose cognition it + * repeats) is sent without that one parent, as the collapse onto HEAD drops the self-edge: a record + * this node built is re-signed so; a caller's record cannot be, and is not sent (said in its + * `delivery`, and as undelivered for a directed send). + * @private + */ + _sendAction(opts, key, held, builtHere) { + const cmb = opts.cmb; + const tags = opts.tags || []; + const result = { key, content: held?.content ?? renderContent(cmb), source: this.name, cmb, storedAt: held?.storedAt ?? null, tags, persisted: !!held, cognitionHeld: true }; + if (key === this._head) { + this._metrics.collapsed = (this._metrics.collapsed ?? 0) + 1; + this.emit('metric', { type: 'cmb-collapsed', key, reason: 'identical-to-own-head' }); + } + if (recordParents(cmb).includes(key)) { + if (!builtHere) { + if (opts.to) this._markUndelivered(result, key, opts.to, 'caller-supplied record names its own key as a parent'); + else Object.defineProperty(result, 'delivery', { value: Object.freeze({ directed: false, to: null, targets: this._peers.size, dispatched: 0, undelivered: false, reason: 'cites-itself' }), enumerable: false, configurable: true }); + return result; + } + const parents = recordParents(cmb).filter((p) => p !== key); + cmb.metadata.lineage = parents.length ? { ...cmb.metadata.lineage, parents } : null; + cmb.metadata.assertionId = assertionIdV2_0(cmb); + this._signOrThrow(cmb, 'CMB re-signing without its self-parent failed'); + } + this._log(`Action ${String(recordAssertion(cmb)).slice(0, 16)}… under a cognition memory holds (${String(key).slice(0, 16)}…) — sent by its assertion, not stored again`); + this._dispatchExisting(result, key, opts.to || null); + return result; + } + + /** + * Send the record of a send that minted nothing (collapsed onto HEAD, or already stored) — to its + * one addressed peer, or (an actionable record, `to` null) to the room — and attach the same + * `delivery` result a fresh send carries, so the send is answerable instead of looking delivered. + * The record is the caller's freshly signed one: a new assertion of the same cognition (MMP + * §8.8.2), not a replay of an earlier send. * @private */ _dispatchExisting(result, key, to) { const cmb = result.cmb || null; - const { dispatched: shared, reason } = cmb ? this._dispatchCmb(cmb, Date.now(), to) : { dispatched: 0, reason: NOT_SENT.SEND_FAILED }; + const { dispatched: shared, reason } = cmb ? this._dispatchCmb(cmb, Date.now(), to || null) : { dispatched: 0, reason: NOT_SENT.SEND_FAILED }; + // A room send made no addressed promise, so it is never `undelivered` (as remember()'s own). + const delivery = to + ? { directed: true, to, targets: this._peers.has(to) ? 1 : 0, dispatched: shared, undelivered: shared === 0, ...(shared === 0 ? { reason } : {}) } + : { directed: false, to: null, targets: this._peers.size, dispatched: shared, undelivered: false }; Object.defineProperty(result, 'delivery', { - value: Object.freeze({ directed: true, to, targets: this._peers.has(to) ? 1 : 0, dispatched: shared, undelivered: shared === 0, ...(shared === 0 ? { reason } : {}) }), + value: Object.freeze(delivery), enumerable: false, configurable: true, }); - if (shared === 0) { + if (to && shared === 0) { this._log(`UNDELIVERED (directed): ${to.slice(0, 8)} ${NOT_SENT_SAID[reason] || reason} — record ${String(key).slice(0, 16)} NOT sent`); this.emit('metric', { type: 'cmb-undelivered', to, key, reason }); } diff --git a/lib/record.js b/lib/record.js index 2b51c80f..e99fa95f 100644 --- a/lib/record.js +++ b/lib/record.js @@ -70,6 +70,28 @@ function setRecordLineage(cmb, value) { return cmb; } +/** + * Whether a record is ACTIONABLE: it carries a signed application section (§8.8.3, `metadata.application` + * not null). An actionable record is a message as well as a memory (§8.8.2; review S3): its cognition is + * remembered under its cognition key, one record per key (memory de-duplication uses the cognition key), + * and its action is identified by its assertion identity wherever it is sent, de-duplicated, delivered + * and kept (actionable delivery de-duplication MUST use the assertion identity). Two of them with the + * same categories (two authors, or one author's two actions) share a cognition key and are two actions: + * whether memory holds the cognition never decides whether an action is sent or delivered. + * @param {object} cmb + * @returns {boolean} + */ +function recordActionable(cmb) { + return !!(cmb && cmb.metadata && typeof cmb.metadata === 'object' && cmb.metadata.application != null); +} + +/** A record's assertion identity (`metadata.assertionId`, §8.8.2), or null. */ +function recordAssertion(cmb) { + const a = cmb && cmb.metadata ? cmb.metadata.assertionId : null; + return typeof a === 'string' && a ? a : null; +} + module.exports = { recordKey, recordCreatedBy, recordCreatedAt, recordLineage, recordParents, setRecordLineage, + recordActionable, recordAssertion, }; diff --git a/lib/wire-identity.js b/lib/wire-identity.js index f2f456b3..041aa8aa 100644 --- a/lib/wire-identity.js +++ b/lib/wire-identity.js @@ -19,6 +19,7 @@ */ const { isCanonicalNodeId } = require('./core/record-canonical'); +const { toWellFormedText, cutText } = require('./core/text'); const MAX_NODE_ID = 256; const MAX_NAME = 256; @@ -39,7 +40,8 @@ function wireNodeId(x) { * peerName)`), so it is never kept, printed or passed on as given. */ function wireName(x, fallback = 'unknown') { - return typeof x === 'string' && x.length > 0 ? x.slice(0, MAX_NAME) : fallback; + // Well formed, and cut on a code point boundary (review M2): a label is passed on as text. + return typeof x === 'string' && x.length > 0 ? cutText(toWellFormedText(x), MAX_NAME) : fallback; } /** A public key off the wire (base64url text), or null when it is not one. */ diff --git a/tests/integration/e2e-payload-receive.js b/tests/integration/e2e-payload-receive.js index 0c51392c..a887ea71 100644 --- a/tests/integration/e2e-payload-receive.js +++ b/tests/integration/e2e-payload-receive.js @@ -16,7 +16,7 @@ require('../_isolate-home'); // sandbox HOME/USERPROFILE: node state must never * SVAF admit vs reject depends on the receiver's own memory drift, and the two * verdicts surface the CMB through different objects: * - * - directed REJECT → _surfaceDirectedReject({...msg}) — raw wire CMB + * - directed REJECT → _surfaceUnstored({...msg}) — raw wire CMB * - SVAF ADMIT → receiveFromPeer → store entry → emit — store copy * * This test wires two in-process nodes and asserts the payload survives to diff --git a/tests/integration/mixed-0142-records.js b/tests/integration/mixed-0142-records.js new file mode 100644 index 00000000..e6913558 --- /dev/null +++ b/tests/integration/mixed-0142-records.js @@ -0,0 +1,170 @@ +'use strict'; + +require('../_isolate-home'); // sandbox HOME: node state must never land in the real ~/.sym +process.env.SYM_SEMANTIC_ENCODER = process.env.SYM_SEMANTIC_ENCODER || 'off'; + +/** + * The four record defects of the 0.14.3 review (S3, B2, M2; tests/security-0.14.3-records.test.js) across + * versions: 0.14.3 nodes and REAL 0.14.2 nodes in one process, joined through the real §5.2 handshake over + * in-memory pipes. The 0.14.2 tree is SYM_0142_DIR (default: the sibling worktree …/wt/sym-0.14.2; a + * `git archive` of the 0.14.2 tag with node_modules beside it will do). Skipped when it is absent. + * + * npm run test:integration + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); +const path = require('path'); +const { SymNode } = require('../../lib/node'); +const { NullDiscovery } = require('../../lib/discovery'); +const { nodeDirById } = require('../../lib/config'); +const { until } = require('../_core-secure'); + +const DIR = process.env.SYM_0142_DIR || path.resolve(__dirname, '..', '..', '..', 'sym-0.14.2'); +let OLD = null; +try { if (JSON.parse(fs.readFileSync(path.join(DIR, 'package.json'), 'utf8')).version === '0.14.2') OLD = { SymNode: require(path.join(DIR, 'lib', 'node')).SymNode, NullDiscovery: require(path.join(DIR, 'lib', 'discovery')).NullDiscovery }; } catch { /* absent */ } +const skip = OLD ? false : `no 0.14.2 tree at ${DIR} (set SYM_0142_DIR)`; + +const ROOM = 'mixed'; +const made = []; +function mk(version, base, extra = {}) { + const Impl = version === '0.14.2' ? OLD : { SymNode, NullDiscovery }; + const n = new Impl.SymNode({ name: `${base}-${process.pid}-${Math.random().toString(36).slice(2, 7)}`, silent: true, discovery: new Impl.NullDiscovery(), room: ROOM, checkpointInterval: 0, ...extra }); + made.push(n); + return n; +} +async function stopAll() { + for (const n of made.splice(0)) { + try { await n.stop(); } catch { /* */ } + try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } + } +} +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); + +/** In-memory pipes sharing one wire: frames sent in one turn arrive in one later turn, as on a socket. */ +function sharedWire() { + const queue = []; + let scheduled = false; + const flush = () => { scheduled = false; for (const [end, frame] of queue.splice(0)) if (!end._closed) end.emit('message', frame); }; + const mkEnd = () => { + const l = {}; + return { + _closed: false, + on(ev, fn) { (l[ev] ||= []).push(fn); return this; }, + once(ev, fn) { const w = (...a) => { this.off(ev, w); fn(...a); }; return this.on(ev, w); }, + off(ev, fn) { l[ev] = (l[ev] || []).filter((f) => f !== fn); return this; }, + removeListener(ev, fn) { return this.off(ev, fn); }, + removeAllListeners(ev) { if (ev) l[ev] = []; else for (const k of Object.keys(l)) l[k] = []; return this; }, + emit(ev, ...a) { for (const f of [...(l[ev] || [])]) f(...a); }, + }; + }; + return async function connect(client, server) { + const a = mkEnd(); const b = mkEnd(); + for (const [self, other] of [[a, b], [b, a]]) { + self.trySend = (frame) => { + if (self._closed) return { ok: false, reason: 'not-connected' }; + queue.push([other, JSON.parse(JSON.stringify(frame))]); + if (!scheduled) { scheduled = true; setImmediate(flush); } + return { ok: true }; + }; + self.send = (frame) => self.trySend(frame).ok; + self.close = () => { if (self._closed) return; self._closed = true; setImmediate(() => self.emit('close')); if (!other._closed) { other._closed = true; setImmediate(() => other.emit('close')); } }; + self.destroy = self.close; + self.end = () => setImmediate(() => self.close()); + } + server.connectTransport(b, { role: 'server' }); + client.connectTransport(a, { role: 'client', expectNodeId: server.nodeId }); + const up = (n, o) => { const p = n._peers.get(o.nodeId); return !!(p && p.transport); }; + await until(() => up(client, server) && up(server, client), 8000); + if (!up(client, server) || !up(server, client)) throw new Error('the pair did not confirm'); + }; +} +const CATS = { focus: 'deploy the build', issue: 'release', intent: 'act', motivation: 'ship', commitment: 'today', perspective: 'operator', mood: { text: 'calm', valence: 0, arousal: 0 } }; +const authorOf = (e) => (e.author && e.author.name) || (e.cmb && e.cmb.metadata && e.cmb.metadata.createdBy); +function collect(node) { + const got = []; + node.on('cmb-accepted', (e) => got.push({ who: authorOf(e), payload: e.cmb && e.cmb.payload, text: e.cmb && e.cmb.categories && e.cmb.categories.focus && e.cmb.categories.focus.text, remixed: e.remixed, verified: e.verified })); + return got; +} +function refusals(node) { + const out = []; + node.on('metric', (m) => { if (/refus|reject|forg|malformed|penal|closed|altered|unattributable/.test(m.type)) out.push(m.type); }); + return out; +} + +describe('0.14.2 and 0.14.3 nodes, the four record defects (S3, B2, M2)', { skip }, () => { + it('S3: two 0.14.2 authors under the same categories, different actions: a 0.14.3 receiver delivers both', async () => { + try { + const A = mk('0.14.2', 'old-a'); const C = mk('0.14.2', 'old-c'); const B = mk('0.14.3', 'new-b'); + await Promise.all([A, B, C].map((n) => n.start())); + const connect = sharedWire(); + await connect(A, B); await connect(C, B); + const got = collect(B); + A.remember(CATS, { payload: { act: 'a' } }); + C.remember(CATS, { payload: { act: 'c' } }); + await until(() => got.length >= 2, 4000); await sleep(300); + // 9f6847c: only A's (C's was admitted under the key A's record holds, and reached nobody). + assert.deepStrictEqual(got.map((g) => `${g.who}:${JSON.stringify(g.payload)}`).sort(), [`${A.name}:{"act":"a"}`, `${C.name}:{"act":"c"}`].sort()); + } finally { await stopAll(); } + }); + + it('S3: a 0.14.3 author\'s two actions under one cognition reach a 0.14.2 receiver as signed records it verifies (it keeps the first: 0.14.2 de-duplicates by cognition key)', async () => { + try { + const X = mk('0.14.3', 'new-x'); const Y = mk('0.14.2', 'old-y'); + await Promise.all([X, Y].map((n) => n.start())); + await sharedWire()(X, Y); + const got = collect(Y); const bad = refusals(Y); + const r1 = X.remember(CATS, { payload: { n: 1 } }); + const r2 = X.remember(CATS, { payload: { n: 2 } }); + assert.ok(r2.delivery && r2.delivery.dispatched === 1, 'the second action is sent'); + await until(() => got.length >= 1, 4000); await sleep(300); + assert.deepStrictEqual(got.map((g) => JSON.stringify(g.payload)), ['{"n":1}']); + assert.deepStrictEqual(bad, [], 'nothing refused or charged at the 0.14.2 receiver'); + assert.ok(Y._peers.get(X.nodeId) && Y._peers.get(X.nodeId).transport, 'the session stays up'); + assert.notStrictEqual(r1.cmb.metadata.assertionId, r2.cmb.metadata.assertionId); + } finally { await stopAll(); } + }); + + it('B2: a 0.14.3 observer with no store weighs the attestation a 0.14.2 gater made about a 0.14.2 author\'s record', async () => { + try { + const P = mk('0.14.2', 'old-author'); const Q = mk('0.14.2', 'old-gater'); const O = mk('0.14.3', 'new-observer', { cliHostMode: true }); + await Promise.all([P, Q, O].map((n) => n.start())); + const connect = sharedWire(); + await connect(P, Q); await connect(Q, O); await connect(P, O); + const r = P.remember(CATS); + await until(() => O.attestationsFor(r.key).length > 0, 6000); await sleep(300); + const agg = O.aggregateAttestations(r.key); + // 9f6847c: total 0 (assertion-unknown). + assert.strictEqual(agg.total, 1, JSON.stringify(agg.excluded)); + assert.strictEqual(agg.basis, 'sole'); + assert.strictEqual(agg.assertionId, r.cmb.metadata.assertionId); + } finally { await stopAll(); } + }); + + it('M2: a 0.14.2 author\'s text cut inside a surrogate pair is delivered by 0.14.3 (room and directed), as U+FFFD; and 0.14.3\'s reaches 0.14.2', async () => { + try { + const S = mk('0.14.2', 'old-sender'); const R = mk('0.14.3', 'new-recv'); + await Promise.all([S, R].map((n) => n.start())); + await sharedWire()(S, R); + const got = collect(R); const bad = refusals(R); const sbad = refusals(S); + const cut = 'Release note 🚀🚀 deploy'.slice(0, 16); + assert.ok(!cut.isWellFormed(), 'the scene: a 0.14.2 host cut a pair'); + const e1 = S.remember({ ...CATS, focus: `D ${cut}` }, { to: R.nodeId }); + S.remember({ ...CATS, focus: `R ${cut}` }); + assert.ok(!e1.cmb.categories.focus.text.isWellFormed(), 'and 0.14.2 signed it so'); + await until(() => got.length >= 2, 4000); await sleep(300); + // 9f6847c: both dropped, as malformed (a lone surrogate). + assert.deepStrictEqual(got.map((g) => g.text).sort(), ['D Release note 🚀�', 'R Release note 🚀�']); + assert.ok(got.every((g) => g.verified === true)); + assert.deepStrictEqual(bad, []); + assert.ok(R._peers.get(S.nodeId) && R._peers.get(S.nodeId).transport, 'the session stays up'); + // The other way: 0.14.3 mints the cut text as U+FFFD; 0.14.2 verifies and delivers it. + const back = collect(S); + R.remember({ ...CATS, focus: `B ${cut}` }, { to: S.nodeId }); + await until(() => back.length >= 1, 4000); await sleep(200); + assert.deepStrictEqual(back.map((g) => g.text), ['B Release note 🚀�']); + assert.deepStrictEqual(sbad, []); + } finally { await stopAll(); } + }); +}); diff --git a/tests/remember-payload.test.js b/tests/remember-payload.test.js index a693bb19..6857896d 100644 --- a/tests/remember-payload.test.js +++ b/tests/remember-payload.test.js @@ -111,7 +111,7 @@ describe('remember({payload}) — opaque payload riding CMBs alongside CAT7', () }); }); - it('payload does not affect cmbKey — same CAT7 + different payloads dedupe', async () => { + it('payload does not affect cmbKey — same CAT7 + different payloads share one memory; each is its own action', async () => { await withNode('payload-not-in-key', async (node) => { const entry1 = node.remember({ focus: 'identical-cat7', @@ -135,13 +135,20 @@ describe('remember({payload}) — opaque payload riding CMBs alongside CAT7', () assert.ok(entry1, 'first emit should land'); - // The payload is NOT addressed: identical CAT7 with a different payload is the same - // content and therefore the same address. This used to assert `null` — "nothing came - // back" — which the collapse contract replaced with something stronger: the second call - // returns a CITATION of the address that already says it, so a caller can tell - // "already held" apart from "refused", which `null` conflated. - assert.ok(entry2?.collapsed, 'the second emit collapses rather than minting a sibling'); + // The payload is NOT addressed: identical CAT7 with a different payload is the same cognition and + // therefore the same address, and memory holds it once: nothing is minted, HEAD does not move. But + // the payload is a signed action (§8.8.2, security review S3): the second record is another + // assertion, and it is SENT, not deduplicated away as the collapse used to (the collapse result + // `collapsed: true` read as "already said" to mesh-channel). The result says so: the same key, + // this record's own assertion, `cognitionHeld`, and a delivery report. + assert.ok(entry2 && entry2.cognitionHeld === true, 'memory already held the cognition'); + assert.ok(!entry2.collapsed && !entry2.duplicate, 'and the action is not a duplicate: it is sent'); assert.strictEqual(entry2.key, entry1.key, 'and cites the address the payload did not change'); + assert.notStrictEqual(entry2.cmb.metadata.assertionId, entry1.cmb.metadata.assertionId, 'its own assertion'); + assert.deepStrictEqual(entry2.cmb.payload, { v: 2 }); + assert.ok(entry2.delivery && entry2.delivery.directed === false, 'with the delivery a room send carries'); + assert.strictEqual(node._store.count(), 1, 'one memory'); + assert.deepStrictEqual(node._store.get(entry1.key).cmb.payload, { v: 1 }, 'memory keeps the record it holds'); // The assertion that actually distinguishes a mint-level refusal from a store-level dedup: // a dedup writes nothing while HEAD has already moved. assert.strictEqual(node._head, entry1.key, 'HEAD MUST NOT advance — nothing new was said'); diff --git a/tests/security-0.14.3-records.test.js b/tests/security-0.14.3-records.test.js new file mode 100644 index 00000000..93bf5f2d --- /dev/null +++ b/tests/security-0.14.3-records.test.js @@ -0,0 +1,537 @@ +'use strict'; + +process.env.SYM_SEMANTIC_ENCODER = process.env.SYM_SEMANTIC_ENCODER || 'off'; +require('./_isolate-home'); // redirect $HOME before lib/config loads + +/** + * The independent review of release/0.14.3 (9f6847c): four defects in records, attestation summaries, + * text and recall, each a regression test that fails on 9f6847c for its own reason and passes here. + * The same defects against a real 0.14.2 node: tests/integration/mixed-0142-records.js. + * + * S3 an actionable record was keyed by its cognition key on two paths: a room record SVAF admitted + * under a key the store held was never delivered, and remember() never sent a second action + * under the same categories (the R2-6 fix covered only the redundant branch) + * B2 aggregateAttestations returned total 0 on a node that does not hold the record (XMesh's + * observers), instead of weighing the one assertion its attestations name + * M2 a record whose text holds a lone surrogate (a 0.14.2 author's text cut inside a pair) was + * refused: the signed bytes are UTF-8, where it can only be U+FFFD, so it is held as U+FFFD + * M1 with the store's cache bounded, recall() read a file for every entry the cache did not hold + */ + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const crypto = require('crypto'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const core = require('../lib/core'); +const pub = require('../lib/public/core'); +const { SymNode } = require('../lib/node'); +const { NullDiscovery } = require('../lib/discovery'); +const { nodeDirById } = require('../lib/config'); +const MS = require('../lib/memory-store'); +const { identity, connectNodes, admitAs, deliver, signedRecord, applicationFor, until } = require('./_core-secure'); +const { settle } = require('./_settle'); + +const ROOM = 'sec'; +const made = []; +function mk(base, extra = {}) { + const n = new SymNode({ name: `${base}-${process.pid}-${Math.random().toString(36).slice(2, 7)}`, silent: true, discovery: new NullDiscovery(), room: ROOM, checkpointInterval: 0, ...extra }); + made.push(n); + return n; +} +async function stopAll() { + for (const n of made.splice(0)) { + try { await n.stop(); } catch { /* not started */ } + try { fs.rmSync(nodeDirById(n.nodeId), { recursive: true, force: true }); } catch { /* */ } + } +} +const CATS = { focus: 'deploy the build', issue: 'release 0.14.3', intent: 'act', motivation: 'ship', commitment: 'today', perspective: 'operator', mood: { text: 'calm' } }; +const payloadOf = (e) => JSON.stringify(e.cmb && e.cmb.payload !== undefined ? e.cmb.payload : null); +const authorOf = (e) => (e.author && e.author.name) || (e.cmb && e.cmb.metadata && e.cmb.metadata.createdBy); +/** What a node surfaces to its host (cmb-accepted), as `author:payload:remixed`. */ +function deliveries(node) { + const got = []; + node.on('cmb-accepted', (e) => got.push(`${authorOf(e)}:${payloadOf(e)}:${e.remixed}`)); + return got; +} +/** + * Connect nodes through the real handshake over in-memory pipes that share ONE wire: every frame sent + * in one turn is delivered in one later turn, as a socket hands several frames to one read. Records + * sent together so reach SVAF together, before any of them is stored: the aligned path the review + * measured on the LAN (SVAF admits each, and the store then holds the key for all but the first). + */ +function sharedWire() { + const queue = []; + let scheduled = false; + const flush = () => { scheduled = false; for (const [end, frame] of queue.splice(0)) if (!end._closed) end.emit('message', frame); }; + function pipe() { + const mk = () => { + const listeners = {}; + return { + _closed: false, + on(ev, fn) { (listeners[ev] ||= []).push(fn); return this; }, + once(ev, fn) { const w = (...a) => { this.off(ev, w); fn(...a); }; return this.on(ev, w); }, + off(ev, fn) { listeners[ev] = (listeners[ev] || []).filter((f) => f !== fn); return this; }, + removeListener(ev, fn) { return this.off(ev, fn); }, + removeAllListeners(ev) { if (ev) listeners[ev] = []; else for (const k of Object.keys(listeners)) listeners[k] = []; return this; }, + emit(ev, ...a) { for (const f of [...(listeners[ev] || [])]) f(...a); }, + }; + }; + const a = mk(); const b = mk(); + for (const [self, other] of [[a, b], [b, a]]) { + self.trySend = (frame) => { + if (self._closed) return { ok: false, reason: 'not-connected' }; + queue.push([other, JSON.parse(JSON.stringify(frame))]); + if (!scheduled) { scheduled = true; setImmediate(flush); } + return { ok: true }; + }; + self.send = (frame) => self.trySend(frame).ok; + self.close = () => { + if (self._closed) return; + self._closed = true; setImmediate(() => self.emit('close')); + if (!other._closed) { other._closed = true; setImmediate(() => other.emit('close')); } + }; + self.destroy = self.close; + self.end = () => setImmediate(() => self.close()); + } + return [a, b]; + } + return async function connect(client, server) { + const [tc, ts] = pipe(); + server.connectTransport(ts, { role: 'server' }); + client.connectTransport(tc, { role: 'client', expectNodeId: server.nodeId }); + const joined = (n, o) => { const s = n._peers.get(o.nodeId)?.transports?.get('bonjour'); return !!(s && s.confirmed && !s.closed); }; + await until(() => joined(client, server) && joined(server, client), 5000); + if (!joined(client, server) || !joined(server, client)) throw new Error('the pair did not confirm'); + }; +} + +/** The SVAF decisions a node records, by the payload of the record decided. */ +function decisions(node) { + const out = []; + const rec = node._recordDecision.bind(node); + node._recordDecision = (d) => { out.push(d.decision); return rec(d); }; + return out; +} + +describe('S3: an actionable record is a message, identified by its assertion everywhere', () => { + it('S3: real nodes, the room path SVAF admits: two authors under the same categories, and one author\'s two actions, are each delivered once', async () => { + try { + const A = mk('s3-a'); const B = mk('s3-b'); const C = mk('s3-c'); + await Promise.all([A, B, C].map((n) => n.start())); + const connect = sharedWire(); + await connect(A, B); await connect(C, B); + const got = deliveries(B); + const decided = decisions(B); + const a1 = A.remember(CATS, { payload: { act: 'v1' } }); + const a2 = A.remember(CATS, { payload: { act: 'v2' } }); + const c1 = C.remember(CATS, { payload: { act: 'c1' } }); + assert.strictEqual(a1.key, a2.key, 'the scene: one cognition key'); + assert.strictEqual(a1.key, c1.key); + await until(() => got.length >= 3, 4000); await settle(200); + // 9f6847c: A's second action was never sent (collapsed onto HEAD) and C's, which SVAF admitted under a + // key B's store held, reached nobody: only A's first was delivered. + assert.ok(a2.delivery && a2.delivery.dispatched === 1, 'the second action is sent'); + assert.deepStrictEqual([...got].sort(), [`${A.name}:{"act":"v1"}:true`, `${A.name}:{"act":"v2"}:false`, `${C.name}:{"act":"c1"}:false`].sort()); + assert.ok(decided.slice(1).every((d) => d === 'aligned' || d === 'guarded'), `the aligned path (SVAF admitted each): ${decided}`); + // A true replay of each (the same signed record again) is not delivered again. + A._dispatchCmb(a1.cmb, Date.now(), null); A._dispatchCmb(a2.cmb, Date.now(), null); C._dispatchCmb(c1.cmb, Date.now(), null); + await settle(300); + assert.strictEqual(got.length, 3, 'a replay of the same assertion is delivered once'); + assert.strictEqual(B._store.count(), 1, 'memory holds the cognition once'); + } finally { await stopAll(); } + }); + + it('S3: the neural admit path: a record SVAF admits under a key memory holds is delivered by its assertion', async () => { + try { + const node = mk('s3-neural'); + await node._store.load(); + node._svafEvaluator = { evaluate: async () => ({ decision: 'aligned', total_drift: 0.05, category_drifts: {}, gate_values: {} }) }; + const decided = decisions(node); + const mallory = identity('mallory'); const alice = identity('alice'); + const sm = admitAs(node, mallory); const sa = admitAs(node, alice); + const cats = { focus: 'release 0.14.3 approved', intent: 'deploy' }; + const front = signedRecord(mallory, { categories: cats, room: ROOM, application: applicationFor(Buffer.from('{"deploy":"mallory-build"}')) }); + const real = signedRecord(alice, { categories: cats, room: ROOM, application: applicationFor(Buffer.from('{"deploy":"official-build"}')) }); + const memo = signedRecord(alice, { categories: cats, room: ROOM }); + deliver(node, sm, { type: 'cmb', cmb: front }); await settle(100); + deliver(node, sa, { type: 'cmb', cmb: real }); await settle(100); + deliver(node, sa, { type: 'cmb', cmb: memo }); await settle(100); + for (const r of [front, real, memo]) { deliver(node, sa, { type: 'cmb', cmb: JSON.parse(JSON.stringify(r)) }); await settle(30); } + assert.deepStrictEqual(decided, ['aligned', 'aligned', 'aligned'], 'the neural gate admitted each'); + const app = (m) => (m.record && m.record.metadata.application ? Buffer.from(m.record.metadata.application.data, 'base64url').toString() : 'none'); + const got = node.inbox({ peek: true }).messages.map((m) => `${m.from}:${app(m)}:${m.remixed}`); + // 9f6847c: alice's action, admitted under the key mallory's record holds, was not delivered. + assert.deepStrictEqual(got.sort(), ['alice:{"deploy":"official-build"}:false', 'mallory:{"deploy":"mallory-build"}:true'].sort(), + 'both actions, once each; a memory memory already holds (no application) is not delivered again'); + } finally { await stopAll(); } + }); + + it('S3: remember() sends every action by its own assertion, whatever memory holds; the same assertion given twice is not sent again', async () => { + try { + const X = mk('s3-x'); const Y = mk('s3-y'); + await Promise.all([X, Y].map((n) => n.start())); + await connectNodes(X, Y); + const got = deliveries(Y); + const r1 = X.remember(CATS, { payload: { n: 1 } }); + // HEAD's cognition: 9f6847c collapsed it ({ collapsed, duplicate }, not sent). + const r2 = X.remember(CATS, { payload: { n: 2 } }); + assert.ok(r2.delivery && r2.delivery.dispatched === 1 && r2.delivery.directed === false, 'sent to the room'); + assert.strictEqual(r2.cognitionHeld, true); + assert.ok(!r2.collapsed && !r2.duplicate, 'not a duplicate: mesh-channel reads it as published'); + assert.notStrictEqual(r2.cmb.metadata.assertionId, r1.cmb.metadata.assertionId); + assert.strictEqual(X._head, r1.key, 'HEAD does not move'); + // Held, not HEAD: 9f6847c returned { key, duplicate: true } and sent nothing. + const other = X.remember({ ...CATS, focus: 'another observation' }); + const r3 = X.remember(CATS, { payload: { n: 3 } }); + assert.ok(r3.delivery && r3.delivery.dispatched === 1, 'an action under a held (not HEAD) cognition is sent'); + assert.strictEqual(X._head, other.key); + // Directed. + const r4 = X.remember(CATS, { payload: { n: 4 }, to: Y.nodeId }); + assert.ok(r4.delivery && r4.delivery.directed === true && r4.delivery.dispatched === 1); + // A peer's record holds the cognition: X's own action under it still goes. + const ycats = { ...CATS, focus: 'the peer said it first' }; + Y.remember(ycats, { payload: { y: 1 } }); + await until(() => X._store.count() >= 3, 3000); await settle(100); + const r5 = X.remember(ycats, { payload: { x: 1 } }); + assert.ok(r5.delivery && r5.delivery.dispatched === 1, 'an action under a peer\'s cognition is sent'); + // The same assertion, already stored as it is: a duplicate, not sent again. + const again = X.remember(CATS, { cmb: JSON.parse(JSON.stringify(r1.cmb)) }); + assert.strictEqual(again.duplicate, true); + assert.ok(!again.delivery, 'not sent'); + await until(() => got.length >= 6, 4000); await settle(200); + const mine = got.filter((g) => g.startsWith(`${X.name}:`)).map((g) => g.split(':').slice(1).join(':')); + assert.deepStrictEqual(mine.sort(), ['{"n":1}:true', '{"n":2}:false', '{"n":3}:false', '{"n":4}:false', '{"x":1}:false', 'null:true'].sort()); + assert.strictEqual(X._store.count(), 3, 'memory: three cognitions'); + } finally { await stopAll(); } + }); + + it('S3: an action is sent when memory cannot store it, and one citing the record whose cognition it repeats is sent without that self-parent', async () => { + try { + const X = mk('s3-fail'); const Y = mk('s3-fail-y'); + await Promise.all([X, Y].map((n) => n.start())); + await connectNodes(X, Y); + const got = deliveries(Y); + const persist = X._store._persist; + X._store._persist = () => false; + const lost = X.remember({ ...CATS, focus: 'memory cannot take this' }, { payload: { go: 1 } }); + X._store._persist = persist; + assert.ok(lost && lost.delivery && lost.delivery.dispatched === 1, 'its delivery is not memory: sent'); + assert.strictEqual(lost.persisted, false); + const memo = X.remember({ ...CATS, focus: 'a memory nobody could store' }); + assert.ok(memo, 'the scene: a memory'); + X._store._persist = () => false; + assert.strictEqual(X.remember({ ...CATS, focus: 'another memory' }), null, 'a memory the store cannot write is still not sent'); + X._store._persist = persist; + const first = X.remember(CATS, { payload: { v: 1 } }); + const cites = X.remember(CATS, { payload: { v: 2 }, parents: [first] }); + assert.ok(cites.delivery && cites.delivery.dispatched === 1); + assert.strictEqual(cites.cmb.metadata.lineage, null, 'K -> K dropped, as the collapse onto HEAD drops it'); + assert.strictEqual(pub.verifyCMB(cites.cmb, X._identity.publicKey).valid, true, 're-signed'); + await until(() => got.length >= 4, 4000); await settle(200); + assert.deepStrictEqual(got.map((g) => g.split(':').slice(1).join(':')).sort(), ['{"go":1}:true', 'null:true', '{"v":1}:true', '{"v":2}:false'].sort()); + } finally { await stopAll(); } + }); +}); + +describe('B2: an attestation summary is about one named assertion', () => { + const hex = () => crypto.randomBytes(32).toString('hex'); + const ACATS = { focus: 'admit', issue: 'admit', intent: 'admit', motivation: 'admit', commitment: 'admit', perspective: 'admit', mood: 'admit' }; + function attestation(by, seq, extra = {}) { + const a = { of: `cmb-${hex()}`, assertionId: `asrt-${hex()}`, by: by.nodeId, at: 1, roster: ROOM, method: 'neural', verdict: 'aligned', categories: ACATS, role: 'participant', seq, prev: 'genesis', ...extra }; + core.signAttestation(a, by.privateKey); + return a; + } + const attest = (V, T, a) => V._receiveSessionFrame(admitAs(V, T), { type: 'sym-attest-attestation', attestation: core.toWireAttestation(a) }); + + it('B2: real nodes: an observer without the record (cliHostMode, no store) weighs the one assertion its attestations name', async () => { + try { + const P = mk('b2-author'); const Q = mk('b2-gater'); const O = mk('b2-observer', { cliHostMode: true }); + await Promise.all([P, Q, O].map((n) => n.start())); + await connectNodes(P, Q); await connectNodes(Q, O); await connectNodes(P, O); + const r = P.remember(CATS); + const key = r.key; + await until(() => O.attestationsFor(key).length > 0, 4000); await settle(200); + assert.strictEqual(O._store.get(key), null, 'the scene: the observer does not hold the record'); + const agg = O.aggregateAttestations(key); + // 9f6847c: total 0, every attestation excluded as assertion-unknown. + assert.strictEqual(agg.total, 1, JSON.stringify(agg.excluded)); + assert.strictEqual(agg.basis, 'sole'); + assert.strictEqual(agg.assertionId, r.cmb.metadata.assertionId, 'and says which assertion'); + assert.ok(agg.overall.dominant, 'a verdict'); + assert.deepStrictEqual(Object.keys(agg.byAssertion), [r.cmb.metadata.assertionId]); + const held = Q.aggregateAttestations(key); + assert.strictEqual(held.basis, 'held'); + assert.strictEqual(held.total, 1); + } finally { await stopAll(); } + }); + + it('B2: attestations naming several assertions: no single verdict, every one in byAssertion; the held or given one is still weighed alone (R2-8)', async () => { + try { + const V = mk('b2-several'); + const key = `cmb-${hex()}`; const X = `asrt-${hex()}`; const Y = `asrt-${hex()}`; + const t1 = identity('t1'); const t2 = identity('t2'); const t3 = identity('t3'); + attest(V, t1, attestation(t1, 1, { of: key, assertionId: X, verdict: 'aligned' })); + attest(V, t2, attestation(t2, 1, { of: key, assertionId: X, verdict: 'aligned' })); + attest(V, t3, attestation(t3, 1, { of: key, assertionId: Y, verdict: 'rejected' })); + assert.strictEqual(V.attestationsFor(key).length, 3); + const agg = V.aggregateAttestations(key); + // 9f6847c: total 0, no per-assertion summaries. + assert.strictEqual(agg.total, 3, 'every verified attestation is counted, never a silent 0'); + assert.strictEqual(agg.basis, 'several'); + assert.strictEqual(agg.assertionId, null); + assert.strictEqual(agg.overall.dominant, null, 'no single verdict'); + assert.deepStrictEqual(agg.categories, {}); + assert.strictEqual(agg.assertions, 2); + assert.strictEqual(agg.byAssertion[X].total, 2); + assert.strictEqual(agg.byAssertion[X].overall.dominant, 'aligned'); + assert.strictEqual(agg.byAssertion[Y].overall.dominant, 'rejected'); + const given = V.aggregateAttestations(key, { assertionId: Y }); + assert.strictEqual(given.basis, 'given'); + assert.strictEqual(given.total, 1); + assert.strictEqual(given.overall.dominant, 'rejected'); + assert.deepStrictEqual(given.excluded.map((e) => e.reason), ['other-assertion', 'other-assertion'], 'an attestation about another assertion is never a vote on this one'); + assert.strictEqual(V.aggregateAttestations(`cmb-${hex()}`).basis, 'none'); + } finally { await stopAll(); } + }); +}); + +describe('M2: well-formed text is normalised, not refused', () => { + /** A record as a 0.14.2 author signed it: its text cut inside a surrogate pair (the same signed bytes). */ + function cutRecord(author, extra = {}) { + const rec = signedRecord(author, { categories: { focus: 'Release note 🚀�', issue: 'surrogate' }, room: ROOM, ...extra }); + const wire = JSON.parse(JSON.stringify(rec).replace('🚀�', '🚀\\ud83d')); + assert.ok(!wire.categories.focus.text.isWellFormed(), 'the scene: a lone surrogate on the wire'); + return { rec, wire }; + } + + it('M2: a record holding a lone surrogate is delivered, verified, and held as its U+FFFD spelling (room and directed)', async () => { + try { + const node = mk('m2-recv'); + await node._store.load(); + const alice = identity('alice'); + const s = admitAs(node, alice); + const refused = []; node.on('metric', (m) => { if (/refus|reject|malformed/.test(m.type)) refused.push(m.type); }); + const room = cutRecord(alice); + const directed = cutRecord(alice, { to: node.nodeId, categories: { focus: 'Direct 🚀�', issue: 'surrogate' } }); + deliver(node, s, { type: 'cmb', cmb: room.wire }); await settle(100); + deliver(node, s, { type: 'cmb', cmb: directed.wire }); await settle(100); + const got = node.inbox({ peek: true }).messages; + // 9f6847c: both refused as malformed (a lone surrogate), silently. + assert.deepStrictEqual(got.map((m) => m.categories.focus.text).sort(), ['Direct 🚀�', 'Release note 🚀�'].sort()); + assert.ok(got.every((m) => m.verified === true && m.record.categories.focus.text.isWellFormed())); + assert.deepStrictEqual(refused, []); + const stored = node._store.get(room.rec.metadata.key); + assert.strictEqual(stored.cmb.categories.focus.text, 'Release note 🚀�', 'held as the one string its signature covers'); + assert.strictEqual(pub.verifyCMB(stored.cmb, alice.publicKey).valid, true); + // Two spellings of one record are one record: the U+FFFD spelling is a replay. + deliver(node, s, { type: 'cmb', cmb: room.rec }); await settle(100); + assert.strictEqual(node.inbox({ peek: true }).messages.length, 2); + } finally { await stopAll(); } + }); + + it('M2: sym\'s own cuts fall on code point boundaries (a message preview, a wire name, the SVAF text)', async () => { + try { + const X = mk('m2-msg'); + const to = identity('peer').nodeId; + const text = 'x'.repeat(4095) + '🚀' + 'tail'; + const cmb = X._mintMessage(text, to); + // 9f6847c: the preview was cut inside the pair and minted as U+FFFD. + assert.ok(!cmb.categories.focus.text.includes('�'), 'no U+FFFD where sym cut the text'); + assert.ok(cmb.categories.focus.text.endsWith('x…')); + } finally { await stopAll(); } + const enc = require('../lib/core/context-encoder'); + const svaf = 'y'.repeat(16 * 1024 - 1) + '🚀'; + const a = await enc.svafKernel().encode(svaf); + const b = await enc.svafKernel().encode('y'.repeat(16 * 1024 - 1)); + assert.deepStrictEqual(a.h1, b.h1, 'the kernel reads up to the pair, never half of it'); + const { wireName } = require('../lib/wire-identity'); + const longName = 'n'.repeat(255) + '🚀🚀'; + assert.ok(wireName(longName).isWellFormed(), 'a wire name'); + assert.ok(wireName('lone \ud83d end').isWellFormed()); + assert.strictEqual(core.cutText('ab🚀cd', 3), 'ab', 'a pair is never split'); + assert.strictEqual(core.cutText('ab🚀cd', 4), 'ab🚀'); + assert.strictEqual(core.cutText('abc', 9), 'abc'); + }); + + it('M2: a store written by 0.14.2 that holds a lone surrogate is read back well formed', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'm2-store-')); + try { + const alice = identity('alice'); + const { rec } = cutRecord(alice); + const entry = { key: rec.metadata.key, content: 'focus: Release note 🚀\ud83d', source: 'old', tags: [], storedAt: 1, timestamp: 1, peerId: alice.nodeId, tier: 'hot', lifecycle: 'observed', cmb: JSON.parse(JSON.stringify(rec).replace('🚀�', '🚀\\ud83d')), lineage: { parents: [], ancestors: [], method: null } }; + fs.writeFileSync(path.join(dir, `${entry.key}.json`), JSON.stringify(entry, null, 2)); + const store = new MS.MemoryStore(dir, 'reader'); + const got = store.get(entry.key); + // 9f6847c: the lone surrogate as stored. + assert.strictEqual(got.cmb.categories.focus.text, 'Release note 🚀�'); + assert.ok(got.content.isWellFormed()); + assert.strictEqual(pub.verifyCMB(got.cmb, alice.publicKey).valid, true); + } finally { fs.rmSync(dir, { recursive: true, force: true }); } + }); + + it('M2: a caller\'s own record and a Legacy Import record are held in their well-formed spelling; createCMB mints references so', async () => { + try { + const X = mk('m2-caller'); + const own = { nodeId: X.nodeId, name: X.name, publicKey: X._identity.publicKey, privateKey: X._identity.privateKey }; + const rec = signedRecord(own, { categories: { focus: 'caller cut 🚀\ufffd' }, room: ROOM }); + const wire = JSON.parse(JSON.stringify(rec).replace('🚀\ufffd', '🚀\\ud83d')); + const e = X.remember(null, { cmb: wire }); + assert.strictEqual(X._store.get(e.key).cmb.categories.focus.text, 'caller cut 🚀\ufffd', 'stored as the one string its signature covers'); + assert.ok(!wire.categories.focus.text.isWellFormed(), 'the caller\'s object is not changed'); + // Given again: nothing is stored (it collapses onto HEAD), and the record handed back is the canonical one. + const back = X.remember(null, { cmb: JSON.parse(JSON.stringify(wire)) }); + assert.ok(back.collapsed && back.cmb.categories.focus.text.isWellFormed(), 'what remember() hands back holds the one string'); + // A 0.13-suite record on a Legacy Import route. + const alice = identity('alice'); + const legacy = core.createCMB({ categories: { focus: 'legacy cut 🚀\ufffd' }, createdBy: 'alice', room: ROOM }); + core.signCMB(legacy, alice.privateKey); + const msg = { type: 'cmb', cmb: JSON.parse(JSON.stringify(legacy).replace('🚀\ufffd', '🚀\\ud83d')), _legacyImport: true, _legacyPinnedKey: alice.publicKey, _legacyRoutedNodeId: alice.nodeId }; + // 9f6847c: refused (malformed-string). + assert.strictEqual(X._frameHandler._rejectLegacyImport(alice.nodeId, 'alice', msg), false, 'taken, quarantined as before'); + assert.strictEqual(msg.cmb.categories.focus.text, 'legacy cut 🚀\ufffd'); + const minted = core.createCMB({ categories: { focus: 'p' }, categoryParents: { focus: ['ref-\ud800'] }, lineage: { parents: [`cmb-${'a'.repeat(64)}`], method: 'm\udc00' }, createdBy: 'x', room: ROOM }); + assert.deepStrictEqual(minted.categories.focus.meta.parents, ['ref-\ufffd']); + assert.strictEqual(minted.metadata.lineage.method, 'm\ufffd'); + } finally { await stopAll(); } + }); +}); + +describe('M1: recall is matched in memory and reads only what it returns', () => { + const N = 3000; // past the entry cache's 2,048 + function filled(dir, n = N, opts = {}) { + const store = new MS.MemoryStore(dir, 'rp'); + let t = 1_000_000; + const now = Date.now; + try { + Date.now = () => t; + for (let i = 0; i < n; i++) { + t += 1; + const cmb = core.createCMB({ categories: { focus: `entry ${i} ${opts.long && i % 500 === 0 ? 'z'.repeat(6000) + ` deepneedle${i}` : ''}`, issue: 'recall perf ' + 'x'.repeat(100), intent: 'i', motivation: 'm', commitment: 'c', perspective: 'p', mood: { text: 'calm', valence: 0, arousal: 0 } }, createdBy: 'rp', room: 'default' }); + store.write(core.renderContent(cmb), { cmb, tags: i % 7 === 0 ? ['Tagged'] : [] }); + } + } finally { Date.now = now; } + return store; + } + /** The recall of 9f6847c, over every entry read whole: the reference for what recall returns. */ + function reference(store, query) { + const out = []; + for (const idx of store._index.byKey.values()) { + const e = store._load(idx.filePath); + if (!query || !query.trim()) { out.push(e); continue; } + const parts = [e.content || '', e.key || '', ...(e.tags || [])]; + for (const v of Object.values(e.cmb.categories)) if (v && v.text) parts.push(v.text); + if (parts.join(' ').toLowerCase().includes(query.toLowerCase())) out.push(e); + } + return out.sort((a, b) => (b.storedAt || 0) - (a.storedAt || 0)).map((e) => e.key); + } + function counting(store) { + const c = { n: 0 }; + const load = store._load.bind(store); + store._load = (p) => { c.n++; return load(p); }; + return c; + } + + it('M1: a query reads no file for an entry it does not return, and with a limit only that many; what it returns is unchanged', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'm1-recall-')); + try { + filled(dir); + const store = new MS.MemoryStore(dir, 'rp'); + await store.load(); + assert.strictEqual(store.count(), N); + const reads = counting(store); + assert.deepStrictEqual(store.recall('needle-not-there'), []); + // 9f6847c: one file read for each of the 952 entries the cache did not hold. + assert.strictEqual(reads.n, 0, 'a miss reads nothing'); + const few = store.recall('entry 123 '); + assert.deepStrictEqual(few.map((e) => e.key), reference(store, 'entry 123 ')); + reads.n = 0; + const limited = store.recall('entry 1', { limit: 5 }); + assert.strictEqual(limited.length, 5); + assert.ok(reads.n <= 5, `a limit reads at most that many (${reads.n})`); + assert.strictEqual(limited.total, reference(store, 'entry 1').length, '`total` counts every match'); + assert.deepStrictEqual(limited.map((e) => e.key), reference(store, 'entry 1').slice(0, 5)); + for (const q of ['', 'entry 1', 'ENTRY 29', 'tagged', 'recall perf', 'calm (v:0', 'neutral', limited[0].key.slice(4, 20), 'focus: entry 2999']) { + assert.deepStrictEqual(store.recall(q).map((e) => e.key), reference(store, q), `the same entries for ${JSON.stringify(q)}`); + } + reads.n = 0; + const where = store.recall('', { limit: 3, where: (e) => e.tags.includes('Tagged') }); + assert.ok(where.length === 3 && where.every((e) => e.tags.includes('Tagged'))); + assert.ok(reads.n <= 21, `only as many read as it takes to fill the limit (${reads.n})`); + reads.n = 0; + store.recall('needle-not-there'); + assert.strictEqual(store.newestOwnCreatedAt() > 0, true); + const oldest = store._index.byTime[store._index.byTime.length - 1]; + assert.ok(!store.__cache.has(oldest), 'the scene: an entry the cache does not hold'); + assert.strictEqual(store.getLifecycle(oldest), 'observed'); + assert.strictEqual(reads.n, 0, 'nor do the timestamp ratchet\'s scan and the lifecycle read a file'); + const old = reference(store, 'entry 7 ')[0]; + assert.strictEqual(store.validateCMB(old, { lifecycle: 'validated' }).ok, true); + store.__cache.delete(old); + reads.n = 0; + assert.strictEqual(store.getLifecycle(old), 'validated', 'the index follows every write of the lifecycle'); + assert.strictEqual(reads.n, 0); + assert.strictEqual(new MS.MemoryStore(dir, 'rp').getLifecycle(old), 'validated', 'and is rebuilt from the files'); + } finally { fs.rmSync(dir, { recursive: true, force: true }); } + }); + + it('M1: the cache starts with the newest entries; a recall reads through it, never into it', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'm1-cache-')); + try { + filled(dir); + for (const sync of [false, true]) { + const store = new MS.MemoryStore(dir, 'rp'); + if (sync) store._ensureIndex(); else await store.load(); + const newest = store._index.recentKeys(MS.CACHE_MAX_ENTRIES); + // 9f6847c: the cache held the last 2,048 files read (the directory's order). + assert.deepStrictEqual([...store.__cache.keys()].sort(), [...newest].sort(), `the newest ${MS.CACHE_MAX_ENTRIES} (${sync ? 'sync' : 'async'} build)`); + const before = [...store.__cache.keys()]; + store.recall(''); + assert.deepStrictEqual([...store.__cache.keys()], before, 'a scan does not reorder or replace the working set'); + } + } finally { fs.rmSync(dir, { recursive: true, force: true }); } + }); + + it('M1: the search index is bounded; an entry past the bound, or longer than its share, is still found', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'm1-bound-')); + try { + filled(dir, 1200, { long: true }); + const store = new MS.MemoryStore(dir, 'rp'); + await store.load(); + const s = store.__search; + // 9f6847c: no index apart from the cache; a recall read every entry it did not hold. + assert.ok(s && typeof s.chars === 'number' && s.size === 1200, 'a search index, apart from the entry cache, holds every entry\'s text'); + assert.ok(s.chars <= MS.SEARCH_MAX_CHARS); + for (const v of s._map.values()) assert.ok(v.text.length <= MS.SEARCH_ENTRY_CHARS, 'each entry\'s share is bounded'); + assert.deepStrictEqual(store.recall('deepneedle500').map((e) => e.key), reference(store, 'deepneedle500'), 'a match past an entry\'s share is found'); + // A small bound: only the newest entries keep their text; the rest are read, and still found. + const small = new MS.SearchIndex({ maxChars: 40_000, entryChars: 256 }); + for (const [k, idx] of store._index.byKey) small.stage(k, store._load(idx.filePath), idx.storedAt); + small.commit(); + store.__search = small; + assert.ok(small.chars <= 40_000 && small.size < 1200, `bounded (${small.size} entries, ${small.chars} characters)`); + const newestKey = store._index.byTime[0]; + assert.ok(small.get(newestKey), 'the newest keep their text'); + assert.ok(!small.get(store._index.byTime[store._index.byTime.length - 1]), 'the oldest let theirs go'); + const tiny = new MS.SearchIndex({ maxChars: 2000, entryChars: 256 }); + for (const [k, idx] of store._index.byKey) tiny.stage(k, store._load(idx.filePath), idx.storedAt); + assert.ok(tiny._stagedChars <= 4000, 'held aside within twice the bound while a store loads'); + tiny.commit(); + assert.ok(tiny.chars <= 2000 && tiny.get(newestKey)); + const written = new MS.MemoryStore(dir, 'rp'); + written.__search = new MS.SearchIndex({ maxChars: 3000, entryChars: 256 }); + const fresh = core.createCMB({ categories: { focus: 'freshly written needle' }, createdBy: 'rp', room: 'default' }); + written.write(core.renderContent(fresh), { cmb: fresh }); + assert.ok(written.__search.get(fresh.metadata.key) && written.__search.chars <= 3000, 'a new entry is indexed as the newest, within the bound'); + for (const q of ['entry 3 ', 'entry 1199 ', 'deepneedle1000', 'tagged']) { + assert.deepStrictEqual(store.recall(q).map((e) => e.key), reference(store, q), `found past the bound: ${q}`); + } + const gone = store._index.byTime[0]; + store._index.get(gone).tier = 'cold'; + assert.strictEqual(store.purge(), 1); + assert.strictEqual(small.get(gone), undefined, 'a purged entry\'s text goes with it'); + assert.deepStrictEqual(store.recall('').map((e) => e.key), reference(store, '')); + } finally { fs.rmSync(dir, { recursive: true, force: true }); } + }); +}); diff --git a/tests/security-0.14.3-verify.test.js b/tests/security-0.14.3-verify.test.js index 8c1f652b..20d8698a 100644 --- a/tests/security-0.14.3-verify.test.js +++ b/tests/security-0.14.3-verify.test.js @@ -8,7 +8,8 @@ require('./_isolate-home'); // redirect $HOME before lib/config loads * * R2-12 the public verifyCMB reported a record carrying unsigned members, a wrong assertionId, or a * 0.13-suite record with an attached application, as valid:true - * R2-13 a lone surrogate encodes as U+FFFD, so one signature verified two strings + * R2-13 a lone surrogate encodes as U+FFFD, so one signature verified two strings (and, review M2, + * refusing it dropped a 0.14.2 author's record cut inside a surrogate pair: it is held as U+FFFD) * R2-14 non-canonical base64url spellings of one key or signature were accepted * R2-21 an invite parse error echoed the URL, relay token included */ @@ -135,32 +136,40 @@ describe('R2-12: the public verifyCMB checks the record\'s own bytes, and says w }); }); -describe('R2-13: one signature verifies one string', () => { - it('R2-13: a text re-spelt with a lone surrogate does not verify under the signature of its U+FFFD spelling', () => { - const rec = signedRecord(alice, { categories: { focus: 'build � ok' } }); +describe('R2-13 (review M2): one signature, one string; a lone surrogate is held as U+FFFD, never refused', () => { + // The signed object is UTF-8 bytes, where a lone surrogate can only be U+FFFD: "\ud800" and "\ufffd" are + // one signed text. 0.14.3 first refused the lone-surrogate spelling, which dropped a 0.14.2 author's + // record whose text was cut inside a surrogate pair; it now holds it as the U+FFFD spelling, so the + // record verifies as signed and two different strings are never held under one signature. + it('R2-13: a text re-spelt with a lone surrogate is the same record: it verifies, and is held as its U+FFFD spelling', () => { + const rec = signedRecord(alice, { categories: { focus: 'build \ufffd ok' } }); assert.strictEqual(pub.verifyCMB(copy(rec), alice.publicKey).valid, true); - const forged = JSON.parse(JSON.stringify(rec).replace('�', '\\ud800')); - assert.notStrictEqual(forged.categories.focus.text, rec.categories.focus.text, 'the scene: two different strings'); - assert.throws(() => canonicalRecordV2_0(forged), (e) => e.code === 'ERECORDSHAPE' && /well-formed/.test(e.reason)); - assert.strictEqual(pub.verifyCMB(forged, alice.publicKey).valid, false); - assert.strictEqual(core.verifyCMB(forged, alice.publicKey).valid, false, 'core\'s signature step refuses it too'); + const respelt = JSON.parse(JSON.stringify(rec).replace('\ufffd', '\\ud800')); + assert.notStrictEqual(respelt.categories.focus.text, rec.categories.focus.text, 'the scene: two different strings'); + assert.deepStrictEqual(canonicalRecordV2_0(respelt), canonicalRecordV2_0(copy(rec)), 'one projection: one string'); + const v = pub.verifyCMB(respelt, alice.publicKey); + assert.strictEqual(v.valid, true, JSON.stringify(v)); + assert.strictEqual(v.record.categories.focus.text, 'build \ufffd ok', 'what verifies is the U+FFFD spelling'); + assert.strictEqual(core.verifyCMB(respelt, alice.publicKey).valid, false, 'core\'s signature step refuses an object no boundary brought to its well-formed spelling (fail closed)'); + assert.strictEqual(core.verifyCMB(canonicalRecordV2_0(respelt), alice.publicKey).valid, true, 'and verifies the projection'); }); - it('R2-13: a signed label re-spelt with a lone surrogate does not verify', () => { - const rec = signedRecord({ ...alice, name: 'ali�ce' }); - const forged = JSON.parse(JSON.stringify(rec).replace('�', '\\udfff')); - assert.throws(() => canonicalRecordV2_0(forged), (e) => e.code === 'ERECORDSHAPE'); - assert.strictEqual(core.verifyCMB(forged, alice.publicKey).valid, false); - assert.strictEqual(pub.verifyCMB(forged, alice.publicKey).valid, false); - const legacy = legacyRecord({ ...alice, name: 'ali�ce' }); - const legacyForged = JSON.parse(JSON.stringify(legacy).replace('�', '\\udfff')); - assert.strictEqual(core.verifyCMB(legacyForged, alice.publicKey).valid, false, 'the 0.13 suite too'); + it('R2-13: a signed label re-spelt with a lone surrogate is the same label, in both suites', () => { + const rec = signedRecord({ ...alice, name: 'ali\ufffdce' }); + const respelt = JSON.parse(JSON.stringify(rec).replace('\ufffd', '\\udfff')); + assert.strictEqual(canonicalRecordV2_0(respelt).metadata.createdBy, 'ali\ufffdce'); + assert.strictEqual(core.verifyCMB(respelt, alice.publicKey).valid, false, 'unprojected: refused'); + assert.strictEqual(pub.verifyCMB(respelt, alice.publicKey).valid, true); + const legacy = legacyRecord({ ...alice, name: 'ali\ufffdce' }); + const legacyRespelt = JSON.parse(JSON.stringify(legacy).replace('\ufffd', '\\udfff')); + assert.strictEqual(core.verifyCMB(legacyRespelt, alice.publicKey).valid, false, 'the 0.13 suite: core\'s step refuses the unnormalised object'); + assert.strictEqual(pub.verifyCMB(legacyRespelt, alice.publicKey).valid, true, 'the public verifier reads it as its bytes say'); }); it('R2-13: a record is minted well formed: a lone surrogate a host passes is minted as U+FFFD, and verifies', () => { const r = core.createCMB({ categories: { focus: 'cut inside a pair \ud83d' }, createdBy: 'ag\udc00ent', createdByNodeId: alice.nodeId, emitV2: true, room: 'default' }); - assert.strictEqual(r.categories.focus.text, 'cut inside a pair �'); - assert.strictEqual(r.metadata.createdBy, 'ag�ent'); + assert.strictEqual(r.categories.focus.text, 'cut inside a pair \ufffd'); + assert.strictEqual(r.metadata.createdBy, 'ag\ufffdent'); r.metadata.assertionId = core.assertionIdV2_0(r); core.signCMB(r, alice.privateKey); assert.strictEqual(pub.verifyCMB(copy(r), alice.publicKey).valid, true); From 6d8ebb8dbfce2f6e7fbb833cb2e93bc660e3edc9 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 19:41:49 +0100 Subject: [PATCH 2/3] recall refuses a query that is not text, never reading it as "no query" The search-index recall read a non-string query as none and returned every entry; the store's old scan threw on it by accident (query.trim), which tests/daemon-relay-only relies on to see a throwing handler answered. A TypeError now, said on purpose; null, undefined and '' are still no query. --- lib/memory-store.js | 3 +++ tests/security-0.14.3-records.test.js | 2 ++ 2 files changed, 5 insertions(+) diff --git a/lib/memory-store.js b/lib/memory-store.js index 2b6898f6..b4463128 100644 --- a/lib/memory-store.js +++ b/lib/memory-store.js @@ -830,6 +830,9 @@ class MemoryStore { * query matched, whatever the limit */ recall(query, opts = {}) { + // A query is text, or none at all (null, undefined, ''): anything else is the caller's error, said, never + // read as "no query" (which returns every entry). + if (query != null && typeof query !== 'string') throw new TypeError(`recall: the query must be a string (got ${typeof query})`); const q = typeof query === 'string' && query.trim() !== '' ? query.toLowerCase() : null; const limit = Number.isSafeInteger(opts.limit) && opts.limit > 0 ? opts.limit : Infinity; const where = typeof opts.where === 'function' ? opts.where : null; diff --git a/tests/security-0.14.3-records.test.js b/tests/security-0.14.3-records.test.js index 93bf5f2d..68e3b49f 100644 --- a/tests/security-0.14.3-records.test.js +++ b/tests/security-0.14.3-records.test.js @@ -455,6 +455,8 @@ describe('M1: recall is matched in memory and reads only what it returns', () => for (const q of ['', 'entry 1', 'ENTRY 29', 'tagged', 'recall perf', 'calm (v:0', 'neutral', limited[0].key.slice(4, 20), 'focus: entry 2999']) { assert.deepStrictEqual(store.recall(q).map((e) => e.key), reference(store, q), `the same entries for ${JSON.stringify(q)}`); } + assert.throws(() => store.recall(123), TypeError, 'a query that is not text is refused, never read as "no query"'); + assert.strictEqual(store.recall(null).length, N); reads.n = 0; const where = store.recall('', { limit: 3, where: (e) => e.tags.includes('Tagged') }); assert.ok(where.length === 3 && where.every((e) => e.tags.includes('Tagged'))); From 3cfbaf8cdeaaff41b2dfa9cd05fc0d06b1528908 Mon Sep 17 00:00:00 2001 From: "SYM.BOT" Date: Wed, 14 Oct 2026 19:41:49 +0100 Subject: [PATCH 3/3] CHANGELOG 0.14.3: R2-3 (recall), R2-6, R2-8 and R2-13 as built after the review of 9f6847c Each text states the design as built: actionable records sent and delivered by assertion with memory keeping one record per cognition key; attestation summaries about one named assertion; well-formed text normalised at every boundary, not refused; recall matched on a bounded index apart from the entry cache, reading only what it returns, with the numbers measured at 6,000 and 50,000 entries and the memory used. --- CHANGELOG.md | 61 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 50 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 68c5b475..8ce21cb3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -153,6 +153,9 @@ patch for one of those cases made another. R1 proved eleven by running them. 0.1 An independent security review (R2) of the 0.14 core reproduced each finding below; each fix has a test in `tests/security-0.14.3*.test.js` that fails on 0.14.3's base (632b95d) and passes here. +A review of the candidate (9f6847c) found that four of those fixes did not hold as designed (R2-3's recall, R2-6, +R2-8, R2-13); they are corrected below, each with tests in `tests/security-0.14.3-records.test.js` and, against a +real 0.14.2 node, `tests/integration/mixed-0142-records.js`, that fail on 9f6847c and pass here. - **A grant binds a key only when it is mesh-wide and rooted in the anchor through admins (R2-1, high).** Any holder of a delegating grant, even one scoped to one world, could bind any unbound nodeId to its own key at every node @@ -173,23 +176,53 @@ An independent security review (R2) of the 0.14 core reproduced each finding bel verification and record budgets are also charged by size (1 MiB/s burst 4 MiB before verification, 128 KiB/s burst 1.5 MiB before SVAF); the SVAF kernel reads at most 16 KiB of a text; the local-state context reads at most 2 KiB of an entry and 16 KiB in all, and is re-encoded at most once a second; the store's entry cache is an LRU - (2,048 entries, 32 M characters); the inbox holds at most 32 M characters (read items make room, oldest first; an - unread one never goes, and a delivery that does not fit is refused and counted, `inbox-full-refused`). The review's - 24 records of 2 × 200 KB: 15 s CPU before, 0.4 s now. The store itself is still unbounded on disk by default - (retention is unlimited unless configured); the size budget bounds how fast one session can grow it. + (2,048 entries, 32 M characters), filled at load with the newest entries; the inbox holds at most 32 M characters + (read items make room, oldest first; an unread one never goes, and a delivery that does not fit is refused and + counted, `inbox-full-refused`). The review's 24 records of 2 × 200 KB: 15 s CPU before, 0.4 s now. The store + itself is still unbounded on disk by default (retention is unlimited unless configured); the size budget bounds + how fast one session can grow it. +- **Recall stays in memory with the cache bounded (R2-3, review M1).** The bounded cache first made `recall()` read a + file, synchronously, for every entry the cache did not hold, on every call. It now matches on a search index kept + apart from the cache: each entry's rendered text, lowercased, at most 4,096 characters of it (an entry longer than + that is read only when its indexed text does not match) and 32 M characters in all (past that the oldest entries' + text is let go, and only those are read). A query reads only the entries it returns: `recall(query, { limit, + where })` returns at most `limit` of them, only those `where` accepts, and the array's `total` counts every match. + A scan reads through the cache, never into it. A query matches within one field (the content, a tag, a category's + text, the key), no longer across two. The timestamp ratchet and the lifecycle (compaction, purge) are read from the + index too, never a file. The startup primer, the daemon, the interior and the mesh agent pass a limit. A query + that is not text is refused (`TypeError`), never read as "no query". Measured on macOS (one call, short entries, + 9f6847c → 0.14.3, with 0.14.2's everything-in-memory in brackets): 6,000 entries, a miss 74-103 ms → 1-2 ms + (2-4 ms), 1,111 matches 74-86 → 28-38 ms (3-5 ms), the same with `limit: 10` 2 ms; 50,000 entries, a miss + 5.0-5.7 s → 9-13 ms (20-26 ms), 11,111 matches 5.0 s → 0.3-0.7 s, with `limit: 10` 15 ms. Heap after load: + 6,000 entries 8.4 → 10.4 MB, 50,000 entries 28.9 → 50.7 MB (18 and 147 MB); the index holds 2.4 M and 20.1 M + characters. A query matching most of a large store, with no limit, still reads what it returns: `recall('')` + over 50,000 entries 5.1 s → 2.9 s (21 ms). Pass a limit. - Gossip signature checks are bounded whatever the number of sessions: every new attestation, checkpoint, witness or authority statement takes one global token before verification, through the sending peer's fair share. The ceiling after verification is gone (R2-4). - LAN client-hellos are rate-limited before any crypto: a token bucket per remote host (8/s, burst 32) and one over all hosts (64/s, burst 256). A hello past either is closed unanswered (R2-5). -- Actionable delivery is de-duplicated by assertion identity (§8.8.2, R2-6): two room records with the same - categories and different signed application sections are both delivered, and one memory de-duplicates is still - delivered (`remixed: false`); a replay of either is not. An actionable room record signed more than 30 days ago is +- An actionable record (one with a signed application section) is a message as well as a memory (§8.8.2, R2-6, + review S3): memory keeps one record per cognition key, and the action is identified by its assertion identity + wherever it is sent, de-duplicated and delivered. Whether memory already holds its categories never decides whether + it is sent or delivered. A receiver delivers an actionable room record SVAF does not reject for drift even when + memory holds its cognition key (`remixed: false`): another author's record under the same categories, or another + action of the same author. The first fix covered SVAF's redundant verdict only; a record SVAF admitted under a key + the store held reached nobody. `remember()` sends an action whose categories memory holds (its HEAD's, an earlier + action's, a peer's) to the room or its addressee, stores nothing again and does not move HEAD; it returns the record + with `cognitionHeld: true` and its `delivery` (it used to return `{ collapsed, duplicate }` and send nothing). An + action whose store write failed is still sent. A replay of the same assertion is delivered once, and the same + assertion given to `remember()` twice is a duplicate. An actionable room record signed more than 30 days ago is refused as stale (its delivery mark is kept that long). - Equivocation evidence and this node's own-witness memory are never evicted. When full, new attesters are refused or not witnessed, and that is counted (R2-7). -- `aggregateAttestations` weighs only attestations naming the held record's `assertionId`, or `opts.assertionId`. The - result carries `assertionId` (R2-8). +- `aggregateAttestations` summarises one named assertion (R2-8, review B2), never an attestation about another as a + vote on it: `opts.assertionId`, else the held record's, else, on a node that does not hold the record (XMesh's + observers, a `cliHostMode` node), the one every attestation under the key names. When they name several there is no + single verdict (`assertionId` and `overall.dominant` null, `categories` empty; `total`, `weight` and `byRole` count + every verified attestation). The result keeps its fields and adds `basis` (`given`, `held`, `sole`, `several`, + `none`), `assertions` and `byAssertion` (one summary per assertion). The first fix returned total 0 on a node + without the record. - Earned key bindings age out (R2-9): a new one (everything it verified fell within a day) after 30 days unseen, and new bindings hold at most a quarter of the registry; an established one after 180 days unseen. One directed record from a fresh identity used to earn a binding that never expired. @@ -200,8 +233,14 @@ An independent security review (R2) of the 0.14 core reproduced each finding bel says `coreSecure`, and a valid v2.0 record comes with its frozen signed projection (`record`). A 0.13-suite record whose sections carry an application, a node id or anything else 0.13 never minted is invalid. A genuine 0.13 record stays valid unless the caller passes `{ coreSecure: true }` (R2-12). See docs/reference.md. -- One signature verifies one string: a signed string with a lone surrogate is refused, and createCMB mints such text - as U+FFFD (R2-13). +- One signature, one string (R2-13, review M2). The signed object is UTF-8 bytes, where a lone surrogate can only be + U+FFFD, so every string a record signs is brought to its well-formed spelling wherever a record crosses a boundary: + minting (createCMB, a caller's own record given to `remember()`), the canonical projection of a received or stored + record, the Legacy Import route, the public verifier, the store's read. The record verifies unchanged, and two + different strings are never held under one signature. The first fix refused a lone surrogate instead, and dropped a + 0.14.2 author's record whose text was cut inside a surrogate pair. sym's own cuts (a message's preview, wire names, + the SVAF text, the local-state context, error texts, decision labels) fall on code point boundaries. The signing step + still refuses a string no boundary normalised. - Keys and signatures are accepted only in canonical unpadded base64url, one spelling each, in the strict verifier, the record projection, the key registry, room-join grants, handshake proofs and attestations (R2-14). - A checkpoint that does not link to the chain held, next to a held checkpoint, is refused as `malformed-link` and is