Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 50 additions & 11 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,9 @@ patch for one of those cases made another. R1 proved eleven by running them. 0.1

An independent security review (R2) of the 0.14 core reproduced each finding below; each fix has a test in
`tests/security-0.14.3*.test.js` that fails on 0.14.3's base (632b95d) and passes here.
A review of the candidate (9f6847c) found that four of those fixes did not hold as designed (R2-3's recall, R2-6,
R2-8, R2-13); they are corrected below, each with tests in `tests/security-0.14.3-records.test.js` and, against a
real 0.14.2 node, `tests/integration/mixed-0142-records.js`, that fail on 9f6847c and pass here.

- **A grant binds a key only when it is mesh-wide and rooted in the anchor through admins (R2-1, high).** Any holder
of a delegating grant, even one scoped to one world, could bind any unbound nodeId to its own key at every node
Expand All @@ -173,23 +176,53 @@ An independent security review (R2) of the 0.14 core reproduced each finding bel
verification and record budgets are also charged by size (1 MiB/s burst 4 MiB before verification, 128 KiB/s
burst 1.5 MiB before SVAF); the SVAF kernel reads at most 16 KiB of a text; the local-state context reads at most
2 KiB of an entry and 16 KiB in all, and is re-encoded at most once a second; the store's entry cache is an LRU
(2,048 entries, 32 M characters); the inbox holds at most 32 M characters (read items make room, oldest first; an
unread one never goes, and a delivery that does not fit is refused and counted, `inbox-full-refused`). The review's
24 records of 2 × 200 KB: 15 s CPU before, 0.4 s now. The store itself is still unbounded on disk by default
(retention is unlimited unless configured); the size budget bounds how fast one session can grow it.
(2,048 entries, 32 M characters), filled at load with the newest entries; the inbox holds at most 32 M characters
(read items make room, oldest first; an unread one never goes, and a delivery that does not fit is refused and
counted, `inbox-full-refused`). The review's 24 records of 2 × 200 KB: 15 s CPU before, 0.4 s now. The store
itself is still unbounded on disk by default (retention is unlimited unless configured); the size budget bounds
how fast one session can grow it.
- **Recall stays in memory with the cache bounded (R2-3, review M1).** The bounded cache first made `recall()` read a
file, synchronously, for every entry the cache did not hold, on every call. It now matches on a search index kept
apart from the cache: each entry's rendered text, lowercased, at most 4,096 characters of it (an entry longer than
that is read only when its indexed text does not match) and 32 M characters in all (past that the oldest entries'
text is let go, and only those are read). A query reads only the entries it returns: `recall(query, { limit,
where })` returns at most `limit` of them, only those `where` accepts, and the array's `total` counts every match.
A scan reads through the cache, never into it. A query matches within one field (the content, a tag, a category's
text, the key), no longer across two. The timestamp ratchet and the lifecycle (compaction, purge) are read from the
index too, never a file. The startup primer, the daemon, the interior and the mesh agent pass a limit. A query
that is not text is refused (`TypeError`), never read as "no query". Measured on macOS (one call, short entries,
9f6847c → 0.14.3, with 0.14.2's everything-in-memory in brackets): 6,000 entries, a miss 74-103 ms → 1-2 ms
(2-4 ms), 1,111 matches 74-86 → 28-38 ms (3-5 ms), the same with `limit: 10` 2 ms; 50,000 entries, a miss
5.0-5.7 s → 9-13 ms (20-26 ms), 11,111 matches 5.0 s → 0.3-0.7 s, with `limit: 10` 15 ms. Heap after load:
6,000 entries 8.4 → 10.4 MB, 50,000 entries 28.9 → 50.7 MB (18 and 147 MB); the index holds 2.4 M and 20.1 M
characters. A query matching most of a large store, with no limit, still reads what it returns: `recall('')`
over 50,000 entries 5.1 s → 2.9 s (21 ms). Pass a limit.
- Gossip signature checks are bounded whatever the number of sessions: every new attestation, checkpoint, witness or
authority statement takes one global token before verification, through the sending peer's fair share. The ceiling
after verification is gone (R2-4).
- LAN client-hellos are rate-limited before any crypto: a token bucket per remote host (8/s, burst 32) and one over
all hosts (64/s, burst 256). A hello past either is closed unanswered (R2-5).
- Actionable delivery is de-duplicated by assertion identity (§8.8.2, R2-6): two room records with the same
categories and different signed application sections are both delivered, and one memory de-duplicates is still
delivered (`remixed: false`); a replay of either is not. An actionable room record signed more than 30 days ago is
- An actionable record (one with a signed application section) is a message as well as a memory (§8.8.2, R2-6,
review S3): memory keeps one record per cognition key, and the action is identified by its assertion identity
wherever it is sent, de-duplicated and delivered. Whether memory already holds its categories never decides whether
it is sent or delivered. A receiver delivers an actionable room record SVAF does not reject for drift even when
memory holds its cognition key (`remixed: false`): another author's record under the same categories, or another
action of the same author. The first fix covered SVAF's redundant verdict only; a record SVAF admitted under a key
the store held reached nobody. `remember()` sends an action whose categories memory holds (its HEAD's, an earlier
action's, a peer's) to the room or its addressee, stores nothing again and does not move HEAD; it returns the record
with `cognitionHeld: true` and its `delivery` (it used to return `{ collapsed, duplicate }` and send nothing). An
action whose store write failed is still sent. A replay of the same assertion is delivered once, and the same
assertion given to `remember()` twice is a duplicate. An actionable room record signed more than 30 days ago is
refused as stale (its delivery mark is kept that long).
- Equivocation evidence and this node's own-witness memory are never evicted. When full, new attesters are refused
or not witnessed, and that is counted (R2-7).
- `aggregateAttestations` weighs only attestations naming the held record's `assertionId`, or `opts.assertionId`. The
result carries `assertionId` (R2-8).
- `aggregateAttestations` summarises one named assertion (R2-8, review B2), never an attestation about another as a
vote on it: `opts.assertionId`, else the held record's, else, on a node that does not hold the record (XMesh's
observers, a `cliHostMode` node), the one every attestation under the key names. When they name several there is no
single verdict (`assertionId` and `overall.dominant` null, `categories` empty; `total`, `weight` and `byRole` count
every verified attestation). The result keeps its fields and adds `basis` (`given`, `held`, `sole`, `several`,
`none`), `assertions` and `byAssertion` (one summary per assertion). The first fix returned total 0 on a node
without the record.
- Earned key bindings age out (R2-9): a new one (everything it verified fell within a day) after 30 days unseen, and
new bindings hold at most a quarter of the registry; an established one after 180 days unseen. One directed record
from a fresh identity used to earn a binding that never expired.
Expand All @@ -200,8 +233,14 @@ An independent security review (R2) of the 0.14 core reproduced each finding bel
says `coreSecure`, and a valid v2.0 record comes with its frozen signed projection (`record`). A 0.13-suite record
whose sections carry an application, a node id or anything else 0.13 never minted is invalid. A genuine 0.13 record
stays valid unless the caller passes `{ coreSecure: true }` (R2-12). See docs/reference.md.
- One signature verifies one string: a signed string with a lone surrogate is refused, and createCMB mints such text
as U+FFFD (R2-13).
- One signature, one string (R2-13, review M2). The signed object is UTF-8 bytes, where a lone surrogate can only be
U+FFFD, so every string a record signs is brought to its well-formed spelling wherever a record crosses a boundary:
minting (createCMB, a caller's own record given to `remember()`), the canonical projection of a received or stored
record, the Legacy Import route, the public verifier, the store's read. The record verifies unchanged, and two
different strings are never held under one signature. The first fix refused a lone surrogate instead, and dropped a
0.14.2 author's record whose text was cut inside a surrogate pair. sym's own cuts (a message's preview, wire names,
the SVAF text, the local-state context, error texts, decision labels) fall on code point boundaries. The signing step
still refuses a string no boundary normalised.
- Keys and signatures are accepted only in canonical unpadded base64url, one spelling each, in the strict verifier,
the record projection, the key registry, room-join grants, handshake proofs and attestations (R2-14).
- A checkpoint that does not link to the chain held, next to a held checkpoint, is refused as `malformed-link` and is
Expand Down
6 changes: 4 additions & 2 deletions bin/sym-daemon.js
Original file line number Diff line number Diff line change
Expand Up @@ -519,8 +519,10 @@ function handleIPCMessage(socketId, socket, msg) {
break;

case 'recall': {
let results = node.recall(msg.query || '');
if (msg.limit && msg.limit > 0) results = results.slice(0, msg.limit);
// Only the entries returned are read (review M1).
const limit = Number.isSafeInteger(msg.limit) && msg.limit > 0 ? msg.limit : undefined;
let results = node.recall(msg.query || '', limit ? { limit } : undefined);
if (limit) results = results.slice(0, limit);
sendIPC(socket, { type: 'result', action: 'recall', results });
break;
}
Expand Down
3 changes: 2 additions & 1 deletion bin/sym.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

const { recordCreatedBy } = require('../lib/record');
const { logSafe } = require('../lib/wire-identity'); // peer-chosen text on the terminal (R2-18)
const { cutText } = require('../lib/core/text');

/**
* sym — CLI for the SYM mesh.
Expand Down Expand Up @@ -963,7 +964,7 @@ function gatherMeshMemory(question, limit) {
const pool = matched.length > 0 ? matched : all.sort((a, b) => b.timestamp - a.timestamp);
return pool.slice(0, limit).map((c) => ({
...c,
content: c.content.length > 400 ? c.content.slice(0, 400) + '…' : c.content,
content: c.content.length > 400 ? cutText(c.content, 400) + '…' : c.content,
}));
}

Expand Down
2 changes: 1 addition & 1 deletion docs/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -406,7 +406,7 @@ anything (a lifecycle change, an action): it refuses every record that is not v2
| `error` | Meaning |
|---|---|
| `unsigned-member` | the record carries a member its signature does not cover (`detail` names it) |
| `malformed-record` | it is not a well-formed v2.0 record: a type, a bound, NFC or well-formed Unicode (`detail`) |
| `malformed-record` | it is not a well-formed v2.0 record: a type, a bound or NFC (`detail`). A lone surrogate is not one of these: the signed bytes are UTF-8, where it can only be U+FFFD, so the record is read, and its `record` given, in that spelling (0.14.3) |
| `assertion-id-mismatch` | its `assertionId` is not the hash of its signed preimage |
| `bad-signature` | the signature does not verify under the key given |
| `content-mismatch` | a category key or the record's address is not that of its text |
Expand Down
5 changes: 3 additions & 2 deletions lib/core/claude-memory-bridge.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
const fs = require('fs');
const path = require('path');
const os = require('os');
const { cutText } = require('./text');

/**
* Bridges Claude Code's project memory directory with the SYM mesh.
Expand Down Expand Up @@ -227,7 +228,7 @@ class ClaudeMemoryBridge {
const ts = Date.now();
const filename = `mesh_${safeName}_${ts}.md`;
const filePath = path.join(this._memoryDir, filename);
const preview = (entry.content || '').slice(0, 80).replace(/\n/g, ' ');
const preview = cutText(entry.content || '', 80).replace(/\n/g, ' ');

const md = [
'---',
Expand Down Expand Up @@ -265,7 +266,7 @@ class ClaudeMemoryBridge {
index = index.trimEnd() + '\n\n## Mesh\n';
}

const preview = (entry.content || '').slice(0, 60).replace(/\n/g, ' ');
const preview = cutText(entry.content || '', 60).replace(/\n/g, ' ');
const line = `- [${filename}](${filename}) — [mesh: ${peerName}] ${preview}\n`;

// Insert under ## Mesh
Expand Down
97 changes: 78 additions & 19 deletions lib/core/cmb-encoder.js
Original file line number Diff line number Diff line change
Expand Up @@ -53,22 +53,78 @@ const { resolveCategories, unreadableReason } = require('./record-shape');
// portability hazard. Vectors are excluded (local, encoder-specific artifacts).
const CMB_KEY_DOMAIN = 'mmp-cmb-v1\n';

// Well-formed text (review M2): one definition, in a module that requires nothing (lib/core/text.js).
// A record is signed as UTF-8 bytes (§8.8.4), where a lone surrogate can only be U+FFFD, so a string
// that is not well formed is normalised wherever a record crosses a boundary (wellFormedRecord), never
// refused.
const { isWellFormedText, toWellFormedText, cutText } = require('./text');

/** `s` well formed when it is a string; anything else as it is (the schema checks types elsewhere). */
const wf = (s) => (typeof s === 'string' && !isWellFormedText(s) ? toWellFormedText(s) : s);

/** A list of references, each well formed; the same array when every one already is. */
function wfList(list) {
if (!Array.isArray(list) || list.every((r) => typeof r !== 'string' || isWellFormedText(r))) return list;
return list.map(wf);
}

/**
* Whether `s` is well-formed Unicode: no lone surrogate (security review R2-13). UTF-8 encodes a lone
* surrogate as U+FFFD, so "\ud800" and "\ufffd" hash and sign as the same bytes: one signature would
* verify two different strings. A receiver refuses a signed string that is not well formed (record-canonical,
* the signing preimages), and an emitter mints only well-formed text (toWellFormedText).
* @param {string} s
* @returns {boolean}
* The record with every string it signs in its well-formed spelling (sym 0.14.3, review M2): each
* category's text and parents, `createdBy`, `room`, `to`, the lineage's parents and method, the
* application's schema. The signed object is UTF-8 bytes, where a lone surrogate can only be U+FFFD,
* so this changes no signed byte: a record verifies exactly as before, and what is held is the one
* string its signature covers (two different strings never share one signature). Applied at every
* boundary a record crosses: minting (createCMB, remember() for a caller's record) before it is
* signed, the canonical projection of a received or stored record, the Legacy Import route, and the
* store's read.
*
* Returns the same object when nothing needs it (the common case), otherwise a copy: the caller's
* object is never changed.
* @param {object} cmb
* @returns {object}
*/
const isWellFormedText = typeof String.prototype.isWellFormed === 'function'
? (s) => typeof s === 'string' && s.isWellFormed()
: (s) => typeof s === 'string' && !/\p{Cs}/u.test(s);

/** `s` with each lone surrogate replaced by U+FFFD: the text UTF-8 would carry anyway, now spelled once. */
const toWellFormedText = typeof String.prototype.toWellFormed === 'function'
? (s) => s.toWellFormed()
: (s) => s.replace(/\p{Cs}/gu, '\ufffd');
function wellFormedRecord(cmb) {
if (!cmb || typeof cmb !== 'object') return cmb;
let changed = false;
const cats = cmb.categories;
let categories = cats;
if (cats && typeof cats === 'object' && !Array.isArray(cats)) {
for (const [f, v] of Object.entries(cats)) {
let nv = v;
if (typeof v === 'string') nv = wf(v);
else if (v && typeof v === 'object') {
const text = wf(v.text);
const parents = v.meta && typeof v.meta === 'object' ? wfList(v.meta.parents) : undefined;
if (text !== v.text || (v.meta && parents !== v.meta.parents)) {
nv = { ...v, text, ...(v.meta && typeof v.meta === 'object' ? { meta: { ...v.meta, parents } } : {}) };
}
}
if (nv !== v) {
if (categories === cats) categories = { ...cats };
categories[f] = nv;
changed = true;
}
}
}
const m = cmb.metadata;
let metadata = m;
if (m && typeof m === 'object' && !Array.isArray(m)) {
const patch = {};
for (const k of ['createdBy', 'room', 'to']) { const x = wf(m[k]); if (x !== m[k]) patch[k] = x; }
if (m.lineage && typeof m.lineage === 'object') {
const parents = wfList(m.lineage.parents);
const method = wf(m.lineage.method);
if (parents !== m.lineage.parents || method !== m.lineage.method) patch.lineage = { ...m.lineage, parents, ...(m.lineage.method !== undefined ? { method } : {}) };
}
if (m.application && typeof m.application === 'object') {
const schema = wf(m.application.schema);
if (schema !== m.application.schema) patch.application = { ...m.application, schema };
}
if (Object.keys(patch).length) { metadata = { ...m, ...patch }; changed = true; }
}
if (!changed) return cmb;
return { ...cmb, ...(categories !== cats ? { categories } : {}), ...(metadata !== m ? { metadata } : {}) };
}

/** Netstring length-prefix: `<utf8-byte-length>:<utf8 bytes>`. Injection-proof
* (a delimiter inside the text cannot shift a category boundary) and trivially
Expand Down Expand Up @@ -588,8 +644,8 @@ function parentSet(list) {
*/
function mintLineage(lineage) {
if (!lineage || typeof lineage !== 'object' || !Array.isArray(lineage.parents) || lineage.parents.length === 0) return null;
const out = { parents: parentSet(lineage.parents) };
if (lineage.method !== undefined) out.method = lineage.method;
const out = { parents: parentSet(wfList(lineage.parents)) };
if (lineage.method !== undefined) out.method = wf(lineage.method);
return out;
}

Expand All @@ -600,8 +656,9 @@ function mintLineage(lineage) {
*/
function mintCategoryText(value) {
const text = typeof value === 'string' ? value : (value && typeof value.text === 'string' ? value.text : '');
// Well formed, then NFC (R2-13): a receiver refuses a lone surrogate, so one is minted as the U+FFFD
// its UTF-8 encoding carried anyway (a host that cut a string inside a surrogate pair keeps working).
// Well formed, then NFC (review M2): a lone surrogate is minted as the U+FFFD its UTF-8 encoding
// carries anyway, so the record holds the one string its signature covers (a host that cut a string
// inside a surrogate pair keeps working).
return toWellFormedText(text || 'neutral').normalize('NFC');
}

Expand Down Expand Up @@ -683,7 +740,7 @@ function createCMB(opts = {}) {
for (const f of CAT7_CATEGORIES) {
categories[f].meta = {
key: categoryKeyV1(f, categories[f].text),
parents: Array.isArray(categoryParents[f]) ? parentSet(categoryParents[f]) : [],
parents: Array.isArray(categoryParents[f]) ? parentSet(wfList(categoryParents[f])) : [],
};
}

Expand Down Expand Up @@ -830,6 +887,8 @@ module.exports = {
mintCategoryText,
isWellFormedText,
toWellFormedText,
wellFormedRecord,
cutText,
recordBytes,
textBytes,
DEFAULT_ROOM,
Expand Down
Loading
Loading