Scope audit: attribute every hop, verify declared regions, derive region keys (M0-M2) - #11
Merged
Conversation
…t honesty The Scope Audit reports "declared but not observed" for regions this instance holds no region key for, presenting a config gap as a finding about someone else's repeater. Verified on the live instance: packet 0a065d41d51f1f77 carries code1=9209, which is exactly the derived code for #fm-112 over that packet's own payload, yet is stored with scope_name = "" because #fm-112 is not in hashRegions. Across the network, 260 of 613 notObserved entries (42%) name a region that never appeared under any name in a 7-day window. Spec covers two independent halves: - M1, server + frontend only: count the unmatched packets that ScopeAuditForwarding currently discards and surface them as a caveat, so an unnameable region stops reading as a confirmed gap. - M2, ingestor: a two-tier regionKeySet (explicit hashRegions plus a capped tier derived from node_declared_regions), refreshed behind an atomic snapshot so the ingest hot path stays lock-free, and matchScope returning a reason instead of a bare string. Opt-in, default off. scope-repair must build the same key set or it would unname every derived row. M3 (path-evidence tie-break) is deliberately gated on M2's measured ambiguity volume rather than built on expectation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…aveat Five TDD tasks, server and frontend only: count the unmatched packets ScopeAuditForwarding discards today, expose them as observedUnmatchedPackets, render a muted caveat chip beside the grey ones, document the field, verify against the real e3d3f4d7 row. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eleven TDD tasks in cmd/ingestor: an opt-in autoRegionKeys config block (default off), a filter and deterministic ranking for declared names, a two-tier regionKeySet behind an atomic snapshot, tiered matching where an explicit hashRegions key beats a derived one, DeclaredRegionStats, startup plus ticker refresh, scope-repair pulled onto the same key set, and a key-set-size benchmark. The scope-repair task is a data-loss guard, not a nicety: repairing against the explicit tier alone would classify every derived name as 'named -> unmatched' and erase it. Tier 3 (path-evidence tie-break) is deliberately absent. The ambiguity counter added here is what measures whether it is worth building. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found on BE-HHE-LAAK-EDG-01 (cf7903ce…7e12): /api/nodes/{pk}/scopes returns
observed:[] with the whole route mix zero for 1h/24h/7d, while the same page's
node header reports transported_scopes ["#be","#be-vli","#eu","#fm-112"] and
relay_count_24h 306. Both read the same database.
Cause: scopeConformanceQuery and scopeAuditForwarderScanQuery credit a
transmission to path[last] only. On a flood route every forwarder appends its
hash to the END of the path, so path[last] means "heard directly by an uplinked
observer", not "forwarded it". Both queries already filter route_type IN (0,1),
where the DIRECT-route hazard that rule guards against cannot arise.
Measured live (24h): the rule keeps 394 of 2789 hop observations (14%, mean path
7.08); 85% of nodes seen forwarding never appear as a last hop; 133 of 205
repeaters (65%) have zero attributable evidence, of which 110 relayed traffic in
the window and 55 already carry a full-key-attributed transported_scopes.
ambiguousHops is 0 on all 205 rows. Of 642 notObserved entries, 238 (37%) name a
region this instance demonstrably can name.
The approved 42% first-cause figure was an upper bound, as recorded — but not a
lower bound, which was not visible then. Amendment adds M0 (drop the two
predicates, restate the doc comments, re-measure) ahead of M1 and M2: until it
lands, M2's effect on the audit is exactly zero for 65% of repeaters, which is
indistinguishable from M2 not working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Frontend half of M1. The server field it reads (observedUnmatchedPackets) does not exist yet — the chip renders nothing until it does, which is asserted, so this is safe to land ahead of the backend and ahead of M0. A declared region absent from this instance's hashRegions can never turn green however much traffic the repeater forwards: the ingestor stores those packets with an empty scope_name, so there is no name to match the declaration against and the row reports "not observed" — a finding about someone else's repeater that is really a gap in our own config. The chip says which of the two it is. Deliberately reuses .sa-chip-ambiguous's muted dashed treatment: both are caveats on the row's finding, not findings, and neither may compete with the red/green scope chips. Six assertions, including that a non-numeric server value cannot become an injection point — the count is escaped, unlike the raw interpolation ambiguousCaveat still does. Untouched on purpose while another session holds them dirty: cmd/server/scopes.go (M0 + the counter this chip reads) and docs/api-spec.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he last
M0 of docs/specs/2026-09-07-auto-region-keys-design.md.
scopeConformanceQuery and scopeAuditForwarderScanQuery credited a transmission
to path[last] only. On a flood route every forwarder appends its own hash to the
END of the path (internal/packetpath/route.go), so path[last] does not mean
"forwarded it" — it means "was the transmission an uplinked observer heard
directly". Every earlier hop forwarded the same packet and was discarded.
Both queries already filter route_type IN (0,1), where the DIRECT-route hazard
the last-hop rule guards against cannot arise: a DIRECT route consumes hops from
the front, so its path is the route's remaining plan rather than a record of who
transmitted. Inside these two queries the restriction only threw evidence away.
That filter is now the sole guard against misattribution, so it gets explicit
tests rather than being left implied by the route constants.
Found on BE-HHE-LAAK-EDG-01 (cf7903ce…7e12), whose Scopes card was empty — whole
route mix zero for 1h/24h/7d — while its own node header listed
transported_scopes ["#be","#be-vli","#eu","#fm-112"] from the same database,
because byPathHop indexes every hop. Replaying its real 14-day traffic (373
transmissions, live path_json/route_type/scope_name) through both queries:
old: 0 transmissions
new: 156 — #be 127, #fm-112 7, #eu 1, unmatched 1, unscoped 20,
route mix 136 transportFlood / 20 flood
Network-wide the old rule kept 394 of 2789 hop observations (14%, mean path
7.08); 85% of nodes seen forwarding never appeared as a last hop, and 133 of 205
declared repeaters (65%) had no evidence of any kind, so every region they
declare read "declared, not observed". Expect ambiguousHops to start firing —
it was 0 on all 205 rows because the machinery never had enough hops.
Widening the join makes ScopeConformance's EXISTS and ScopeAuditForwarding's
"<target>|<txID>" de-duplication load-bearing rather than belt-and-braces: one
path can now match the same target on several hops (a routing loop, or two hops
colliding on the same truncated prefix). Both are pinned by test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
M0 (spec b610d46) changes what both plans can claim, so both say so now rather than reading as if attribution were sound. M1: dependency stated up front — the unmatchedPackets counter counts unmatched rows among the rows attribution admits, so before M0 it reads zero for the same 65% of repeaters and invites the same wrong conclusion in a new field. Tasks 1, 2 and 5 are blocked; Task 3 is marked done (79f38ef), since the chip is inert until the server sends the field. Adds a verification step for a risk M0 creates: widening attribution from 222 distinct last-hops to 964 distinct prefixes will start ambiguousHops firing, and two caveat chips that are both permanently lit say less than one. M2: no file overlaps M0 or M1 — it is entirely cmd/ingestor — but its measurement is worthless before M0, since a derived key that correctly names a scope still leaves the declaring repeater in notObserved when its hops were discarded first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…iscards Task 1 of docs/plans/2026-09-07-scope-audit-unmatched-caveat.md, now that M0 (d93b446) has freed this file. The unmatched skip threw away the one fact that separates "this repeater is not forwarding that region" from "this instance holds no key for it". Both look identical in notObserved, and on live data the second explains a large share of the entries. Kept out of unscopedPackets on purpose: unscoped means the packet carried no scope at all and is what '*' governs, unmatched means it IS scoped and we cannot name it, so '*' says nothing about it. Two tests. The mid-path one is the case that carries almost all the real volume after M0 — before it, a repeater deep in a flood path contributed nothing — and it also pins that a target appearing twice in one path still counts once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 2 of docs/plans/2026-09-07-scope-audit-unmatched-caveat.md. Carries the Task 1 counter to the client. The row still lists the declared region under notObserved — an unmatched packet names no region, so it cannot satisfy a declaration, and pretending otherwise would trade one wrong answer for another — but the client can now say which of the two kinds of absence it is looking at. The test pins that it does not leak into wildcardContradiction: unmatched traffic is scoped, so it is evidence about a region key, not about '*'. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 4 of docs/plans/2026-09-07-scope-audit-unmatched-caveat.md. States the consequence a client has to design around: a repeater forwarding a region this instance cannot name is reported exactly like one forwarding nothing. Placed beside the ambiguousHops note and drawn against it, since the two look alike on the wire but have different causes and only one of them is the reader's to fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tasks 1, 2, 3 and 4 are committed (70e6bcd, ca464b5, 79f38ef, 93a0c38) and all four automated suites pass. Task 5 steps 3-5 cannot run here and are marked deferred rather than skipped: test-fixtures/e2e-fixture.db predates the feature entirely (no node_declared_regions table, no scope_name column) and the live instance still runs pre-M0 code. They run against staging or live after deploy. The plan is not complete until they have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
M1 marks a declared region this instance cannot name with a row-level
caveat but leaves its chip grey, and grey reads as "declared but not
forwarding" — a claim the data supports no better than green. There are
three states and the column has two colours.
The question is answerable from data already stored. For repeater R
declaring region X, derive SHA256("#X")[:16] and HMAC R's own unmatched
packets with it: the same computation matchingRegions performs, with
the candidate set narrowed to R's own declarations. Demonstrated by
hand on e3d3f4d7: of 36 unmatched rows, 23 derive to fm-112 and 3 to
behss, so both declared regions are provably forwarded and the page
shows both grey.
Narrow beats global for this question on three counts. It tests ~9
independently-supported hypotheses instead of ~180 unrelated ones, so
it needs none of the cap/ranking/tie-break machinery section 1 exists
to contain. It can require corroboration, which ingest-time naming
structurally cannot: a chip turns green on two or more matching
packets, making coincidence (1/65536)^2 rather than 1/65536. And it
writes nothing, so a wrong answer expires with the window instead of
sitting in scope_name until scope-repair runs.
Query shape is load-bearing: the hop scan returns 19049 rows per 2000
packets after M0, so raw_hex comes from a second narrow query over the
~400 unmatched transmissions instead of widening that scan.
M2 is re-scoped accordingly. It no longer fixes the audit; it fixes the
packets page, scope-stats, default_scope and the per-node observed
list. Real, but different value and lower urgency than this document
originally claimed, and it should be sized against what M1b leaves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight TDD tasks, cmd/server and public/ only. Nothing written, no config, no schema change. Two design points carry the plan. The corroboration threshold is 2 and the arithmetic is the argument: code1 is two bytes, so one match happens by chance with probability 1/65536, and across ~400 unmatched packets and ~124 declared names that is roughly one false positive per refresh; two on the same region for the same repeater is (1/65536)^2. And the memo is not an optimisation — the naive shape is targets x names x packets (~740k HMACs, ~0.7s), while the work only depends on (region, transmission), which is ~50k and ~50ms. It is built in from the first task and asserted by a test that watches hmacCount. Verification deliberately does not call DecodePacket: decodePayload attempts decryption and signature validation this has no use for. It walks the offsets reusing decodeHeader/isTransportRoute/decodePath, so the arithmetic is not duplicated from DecodePacket either. The unit tests are built on transmission 0a065d41d51f1f77 from the live instance rather than a synthetic fixture — a hand-built packet would only prove the parser agrees with itself, whereas this one's code1 is exactly what #fm-112 derives over its own payload, which is the bug. M1's caveat chip is narrowed rather than removed: it now reports only what verification could not explain, which is a sharper finding — traffic for a region the repeater does not declare and this instance cannot name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… payload Task 1 of docs/plans/2026-09-07-declared-region-verification.md. scopeHMACInputs walks a raw_hex to the three values a region hypothesis needs - payload type, payload bytes, code1 - reusing decodeHeader, isTransportRoute and decodePath rather than duplicating the offsets. It does not call DecodePacket: decodePayload attempts decryption and signature validation this has no use for, on every unmatched packet on every refresh. regionCode is the forward direction of matchingRegions in the ingestor and must stay in step with it, including that case is significant - the key is a hash over the raw bytes of '#name'. Tests run on transmission 0a065d41d51f1f77 from the live instance, not a synthetic fixture: its code1 is exactly what #fm-112 derives over its own payload, which is the bug this milestone exists for. The plan's reserved-hash_size case was wrong (pathByte 0x41 with 0x80 as the first hop, rather than pathByte 0xC0); the test caught it and both the test and the plan are corrected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 2 of docs/plans/2026-09-07-declared-region-verification.md. M1's counter says how many; verification needs to know which. The ids inherit the same (target, txID) de-duplication the counter has, which matters more here than there: without it one packet reaching a target by two hops would corroborate twice and clear the two-packet threshold on its own. The list is capped at scopeVerifyMaxPacketsPerTarget while the count is not - deliberately asymmetric, since the count is the honest total and the list only has to carry enough evidence to settle a threshold of 2. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 3 of docs/plans/2026-09-07-declared-region-verification.md. Selects only scope_name = '' transmissions inside the window on FLOOD-family routes - roughly 400 rows over 7 days on the reference deployment. Deliberately a second query rather than widening scopeAuditForwarderScanQuery, which returns one row per hop per flood packet (19049 on a 2000-packet sample after M0); carrying raw_hex on all of those to serve a few hundred would load the hot path for nothing. Selection only: an unparseable raw_hex is still returned and dropped by the verifier, so the row count keeps saying how many candidates the window really held. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gofmt applies the old godoc typographic substitution inside doc comments and rewrites a two-single-quote digraph into a closing curly quote. The comment explaining that this query keys on an EMPTY scope_name therefore rendered as a quotation mark - silently misstating the one value the query depends on, and reverting on every gofmt run. Written out in words instead, with a note so the next person does not put the literal back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ket threshold Task 4 of docs/plans/2026-09-07-declared-region-verification.md. scopeVerifyMinCorroboration = 2 is the load-bearing constant. code1 is two bytes, so an unrelated name matches a given packet with probability 1/65536; across ~400 unmatched packets and ~124 declared names that is roughly one false positive per refresh. Two matches on the same region for the same repeater is (1/65536)^2. Lowering it to 1 does not make the feature noisy, it makes it unsound. The memo is not an optimisation either. Naively the pass is targets x names x packets (205 x 9 x 400, ~740k HMACs, ~0.7s); the work depends only on (region, transmission), which is ~50k and ~50ms. hmacCount exists so a test can assert the memo still holds after a refactor rather than trusting the shape of the code. An unparseable raw_hex is kept with ok=false rather than dropped, so repeat lookups still short-circuit, and one malformed row in the window cannot blank verification for every repeater. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nnameable traffic Task 5 of docs/plans/2026-09-07-declared-region-verification.md. This is the milestone's point: a chip turns green when the instance can prove the repeater forwards that region, instead of staying grey with a footnote. Only regions with no NAMED evidence are tested. That is not just an optimisation - it keeps the candidate set to exactly the open questions, so the verifier's work stays proportional to the problem rather than to the fleet. notObserved remains the single source of chip colour; RegionEvidence only says HOW a region was established, and explains a region that got exactly one hit and therefore stayed grey. Two fields that can disagree about the same fact is how this column got confusing in the first place. A failed verification query degrades to no verification rather than failing the request - the audit was useful before this existed. All 18 handler tests pass, including the sort tests the plan flagged as at risk: a verified region leaving notObserved changes a row's rank. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t stays unexplained Task 6 of docs/plans/2026-09-07-declared-region-verification.md. A region established by verification renders in the same green as one named from a configured key - it is observed either way - with a dotted underline and a tooltip saying how. A third colour would compete for attention in a column that already carries two. A region with exactly one corroborating packet stays grey and its tooltip says why: one match in 65536 by chance alone, two would be evidence. Silence there would look like an arbitrary refusal. The M1 caveat is narrowed rather than removed. It now reports only observedUnmatchedPackets minus what verification explained, which is a sharper finding than the old total: this repeater forwards a region it does not declare AND that this instance cannot name. Fully explained traffic raises no caveat at all, so the chip stops re-asking a question the Scopes column has just answered. 692 frontend assertions pass, up from 686. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 7 of docs/plans/2026-09-07-declared-region-verification.md. States the threshold and why it is 2, since a client that treated one hit as evidence would draw a false green about once per refresh. Also amends the observedUnmatchedPackets note, which predates verification: part of that count is now explained, and a client showing it as a caveat should subtract regionEvidence and report only the remainder. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cket) Task 8 of docs/plans/2026-09-07-declared-region-verification.md, plus the correction the benchmark forced. The plan claimed ~50ms on the grounds that caching per (region, transmission) cuts the HMACs from ~10M to ~50k. Measured, it was 501ms - the HMACs had indeed become a rounding error, but the ITERATION was still targets x names x packets: 10.2M map lookups at ~49ns each. Caching the expensive operation is not the same as removing the expensive loop. Keyed per region instead, holding the set of transmissions that derive to it. A region is HMACed over every packet once; a target then asks one question per declared region rather than one per (region, packet), and since most declared regions match nothing the common case is a single lookup with no packet loop. before 501254220 ns/op after 35977350 ns/op (14x, AMD Ryzen 7 PRO 8845HS) That is worst case: every one of 205 targets declaring all 124 names over all 400 packets. Real rows declare ~9. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…erred All eight tasks committed and every automated suite green: cmd/server full (119.9s), 692 frontend assertions, 99 packet-filter, 18 aging, gofmt and vet clean. Task 8 Step 5 stays open rather than being ticked: confirming the e3d3f4d7 row shows fm-112 and behss green cannot run here, since the e2e fixture predates the feature entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 1 of docs/plans/2026-09-07-auto-derived-region-keys.md. Opt-in, matching the existing clientRxObservations / clientRfSamples / clientRegions flags. An absent block leaves behaviour byte-for-byte what it is today, which is asserted rather than assumed. Zero and negative overrides fall back to the defaults instead of being honoured: a zero is indistinguishable from an omitted key after json.Unmarshal, and a zero refresh interval would panic time.NewTicker. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d region names
Task 2 of docs/plans/2026-09-07-auto-derived-region-keys.md.
The filter is structural, never about a name's meaning. The declared
set contains entries that look like junk ('null', 'bierhuis', 'sol3')
and they are kept: a blocklist on string values is unmaintainable, and
one bad name costs a single slot out of maxDerived plus a 1-in-65536
collision chance. What is rejected is anything that cannot have come
off the firmware intact - a comma (the regions_csv delimiter), a '#'
(the firmware strips it), non-ASCII or whitespace (the key is a hash
over raw bytes), a NUL (untrimmed block-cipher padding).
Ranking breaks ties on name as its last step, which is what makes the
cap deterministic: without it the derived tier would churn between
refreshes on equally-ranked names and the add/drop logging would be
noise rather than signal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 3 of docs/plans/2026-09-07-auto-derived-region-keys.md. The ingest hot path reads via one atomic load; a refresh builds the replacement map off to the side and swaps the pointer, so a rebuild never blocks a packet (AGENTS.md rule 0). A snapshot already handed to a packet keeps its contents, which a test pins - an alias would let the key set change under a match in progress. refreshDerived REPLACES the derived tier rather than merging into it, so a region that stops being declared leaves the key set. Merging would make the cap meaningless within days. A declared name duplicating an explicit key is skipped rather than re-added: the explicit tier has to stay authoritative for the tie-break, and demoting a configured region because some repeater also declares it would invert the whole rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ie-break Task 4 of docs/plans/2026-09-07-auto-derived-region-keys.md, minus the matchScope deletion - that goes with the call-site update in the next commit so the tree builds at every point (AGENTS.md rule 6). The plan split them and would have left one commit broken. Three tiers. One match names it. Several with exactly one from hashRegions names that one: operator config outranks a name picked up over RF, and this absorbs the bulk of the ambiguity auto-derivation introduces. Anything else abstains, which is Kpa-clawbot#1609's rule unchanged - naming a packet wrongly is worse than leaving it unnamed. scopeMatch carries the reason and the candidate list so the outcome is auditable in logs without a schema change. The tier-3 path-evidence tie-break is deliberately absent; the ambiguous counter is what will say whether it is worth building. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tasks 5, 6 and 7 of docs/plans/2026-09-07-auto-derived-region-keys.md, in one commit because they are entangled across the same four files and splitting them would leave a commit that does not build (AGENTS.md rule 6). - *regionKeySet replaces map[string][]byte at all thirteen call sites. snapshot() is nil-safe: a nil set means 'no region keys', exactly what a nil map meant before, and panicking there would turn an absent key set into a crash on the ingest path. - matchScope is removed. Its doc comment's suggestion of a 'pre-indexed lookup table' beyond ~50 regions goes with it, and a note in its place says why the idea must not come back: code1 is an HMAC over the payload, so there is nothing payload-independent to index on. That is precisely why maxDerived exists. - DeclaredRegionStats aggregates the latest answer per target, by observed_at and never ingested_at, so a drive buffered offline cannot resurrect a region a repeater has since dropped. '*' is excluded - it is the wildcard, not a region. - refreshFromStore runs at startup and on its own ticker, not the daily retention one: declared answers arrive continuously and waiting 24h to name a new region would defeat the point. A DB error keeps the CURRENT snapshot; an empty key set would silently unname all traffic, which looks exactly like the bug this fixes. Existing tests keep their old shape via two shims rather than being rewritten alongside the change they guard: regionSetFromKeys treats every key as explicit, which is what a hashRegions map meant, so the 1609 ambiguity assertions still hold. Full ingestor suite: only the pre-existing Windows symlink-privilege failure, which is byte-identical to master. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 10 of docs/plans/2026-09-07-auto-derived-region-keys.md. Names the trap the sibling opt-in flags document: the block is top-level, and json.Unmarshal without DisallowUnknownFields means a mis-nested key is silently ignored with the feature staying off and no error. client-regions.md gains the consequence operators would otherwise meet by surprise: retention.clientRegionsDays now also bounds how long a region stays derivable, since a pruned declaration takes its key with it on the next refresh. Regions you want named permanently belong in hashRegions, which nothing prunes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 8 of docs/plans/2026-09-07-auto-derived-region-keys.md. runScopeRepair built its keys with loadRegionKeys, the explicit tier only. Left that way, a repair run would find no key for any automatically-named row, classify it as 'named -> unmatched', and write an empty scope_name over the name - a maintenance tool silently erasing exactly the data M2 exists to produce. It now builds the same two-tier set and refreshes the derived tier before scanning, and logs how many keys are in force so an operator can see at a glance whether the run was working from what they expected. The regression test builds a packet whose code1 IS #behss's derived code, so a revert fails it deterministically rather than depending on which rows happen to be in the database. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 9 of docs/plans/2026-09-07-auto-derived-region-keys.md. AGENTS.md rule 0: this triples the key set, so the claim needs proof. A sweep rather than a before/after, because the explicit tier's size is operator config and varies per deployment: keys=16 10861 ns/op keys=58 36984 ns/op keys=180 114349 ns/op keys=314 216683 ns/op (AMD Ryzen 7 PRO 8845HS) Linear at ~0.65us per key, as it must be: code1 is an HMAC over the payload, so there is nothing payload-independent to index on and the cost cannot be made sublinear. That is the whole reason maxDerived exists. Against real load it is nothing. The reference deployment sees 22126 transport-scoped packets per 7 days, 0.037/s; at the 314-key ceiling that is 0.0008% of one core. It stays under 3% even at a sustained 100 packets/s, which is three orders of magnitude above what this network does. Unlike M1b's verifier there is no loop to hoist here - one HMAC per key per packet is the irreducible shape. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All eleven tasks committed. Two deliberate deviations recorded: the matchScope removal moved into the call-site commit, and tasks 5-7 landed together, because splitting either would have left a commit that does not build. Deferred and marked open rather than ticked: go test -race cannot run on this machine (needs cgo, no gcc) and belongs in CI on Linux, and the three real-data steps need the feature switched on somewhere live. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Written for a fresh session on the build/publish machine, which has the server access this one lacks. Carries only what git cannot: why the code is shaped this way, what is deliberately unfinished, and which measurements decide what happens next. Names the four real-data measurements and marks which are decisions rather than tick-boxes - the ambiguity rate decides whether M3 gets built, and the double-caveat count decides whether M0's widened attribution has made both caveat chips meaningless. Also records the traps: gofmt rewriting a two-quote digraph in doc comments, and that caching the expensive operation is not the same as removing the expensive loop (M1b measured 501ms with the HMACs already cached away). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…er hop The forwarder scan carried scope_name and first_seen on every hop row. Both are columns of transmissions, and a 7d window on the live-shaped staging database returns 3,470,188 hop rows against 79,652 transmissions: 43 hop rows re-reading the same two values for each transmission. They now come from one separate query over the same window, keyed by transmission id, and the hop scan carries two columns instead of four. Both queries run in one read transaction, so a transmission arriving between them cannot appear in the hop scan with no metadata to attribute it by. The hop itself is lower-cased into a stack buffer rather than through strings.ToLower: 1,026,814 of the 1,284,897 hops in a 24h window are stored uppercase (packetpath.DecodePathFromRawHex writes them that way) and the great majority match no declared target, so that allocation was paid millions of times to answer "no". The (target, txID) de-duplication key is a struct for the same reason. Measured on that database at window 7d before this change: 16.7s total, of which 2.7s is SQLite. Every SQL-side reduction measured worse and was rejected: a first-4-hex prefix filter against the declared targets 20.9s, GROUP BY t.id, hop 38.0s, SELECT DISTINCT t.id, path_json 17.7s. The 3.47M rows are inherent to 1,368,761 observations carrying a path. Behaviour is unchanged. TestScopeAuditForwardingAttributesLongerHopByItsOwnLength is new and covers what the rewrite touches and the existing tests did not: they all seed 4-char hops, so nothing pinned that attribution keys on the hop's own length. Case folding was already covered, since seedTransmissionPathAt has always upper-cased its hops. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
… own TTL
handleScopeAudit computed outside the cache mutex, so every request that
arrived on a cold window ran its own full scan alongside the others. On the 7d
window that scan is 16.7s over 3.47M hop rows, which is the shape where a
thundering herd costs real work rather than a duplicated map build. The compute
moves into computeScopeAudit and the handler wraps it in a singleflight, the
same treatment /api/observers and /api/nodes/{pubkey}/reach already have.
The TTL becomes per-window: 5 minutes for 7d, unchanged at 30s for 1h and 24h.
Measured cold on the live-shaped staging database: 0.15s for 1h, 4.0s for 24h,
16.7s for 7d. At 30s a single reader with the 7d window open kept the instance
recomputing more than half the time, for an aggregate covering a week of
traffic where five minutes of staleness is not something the reader can act on
differently.
Two tests: one pins the per-window TTL so collapsing it back to a single
constant has to delete the reason, one pins that a second request inside the
TTL is served from the cache rather than recomputed. The cache path itself had
no test before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
Browser validation (AGENTS.md rule 2) is done for M0, M1 and M1b, on BE-LML-RP01 rather than on the e3d3f4d7 row both plans name: #behss and #fm-112 went into the live hashRegions on 2026-09-07 08:42, so that row is green by ordinary naming now and can no longer exercise verification either way. The double-caveat check is done and found nothing to raise: 2 of 206 rows carry ambiguousHops, 63 carry the unexplained chip, none carry both. The M3 gate stays open but its first tally reads ambiguous=0 in 1306 scoped packets, and the derived tier turned out to add exactly one key on this network because the other 123 declared names were already configured by hand. The M0 section gains the cost of reading every hop: 3,470,188 rows and 16.7s for a 7d window, the three SQL-side reductions that measured worse, the 80% uppercase hops that make a case-sensitive hop comparison silently wrong, and what 942761c and b7515ce brought it down to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four milestones fixing one user-visible defect: the Scope Audit reported regions as "declared but not observed" for repeaters that were provably forwarding them.
Draft on purpose. Browser validation (AGENTS.md rule 2), deployment, and four real-data measurements are still outstanding — see
docs/plans/for what is ticked and what is not.The defect
Repeater
e3d3f4d7…c0b1declares nine regions;behssandfm-112showed grey. Packet0a065d41d51f1f77decodes tocode1=9209, which is exactly the code#fm-112derives over that packet's own payload — the repeater was forwarding it. Two independent causes:scope_name = '', and everything comparing declared against observed reads that as absent.path[last]only. On flood routes every hop appends its own hash (internal/packetpath/route.go:20), sopath[last]means "heard directly by an uplinked observer", not "forwarded it". Measured live: 133 of 205 repeaters had zero attributable evidence.Milestones
code1is two bytes, so one match happens by chance with p=1/65536, two is (1/65536)².node_declared_regions. Opt-in, default off. Re-scoped by M1b: this no longer fixes the audit, it fixes the packets page, scope-stats anddefault_scope.Review notes
notObservedstays the single source of chip colour;regionEvidencesays only how a region was established.scope-repair.scope-repairhad to be pulled onto the same key set or it would have erased every derived name.Known gap
go test -racecovers the server only (deploy.yml:134, added for PR Kpa-clawbot#1208's atomic.Pointer migration). M2 introduces anatomic.Pointerin the ingestor, which line 143 tests without-race. Not fixed here — it is a CI change, and a separate decision.Specs and plans:
docs/specs/2026-09-07-auto-region-keys-design.md,docs/plans/2026-09-07-*.md.🤖 Generated with Claude Code