From 84a80c98fa1ae662d5cc16534ed22c7ec9c9364c Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sun, 2 Aug 2026 10:13:46 -0500 Subject: [PATCH 1/7] backlog(#347): PHI-at-rest tests assert short-substring absence against base64 ciphertext `tests/test_store_encryption.py:95` asserts `"DOE" not in raw` against a value encrypted under a fresh random key, so the base64 body is fresh random text every run. Measured p ~ 5e-4 per run per leg; it fired on PR #142's windows-2022 py3.14 leg with encryption working correctly. Filed rather than fixed: the fix direction is the maintainer's call (decoded-bytes assertion vs. non-recoverability vs. full-plaintext absence), and simply widening or deleting the substring check would drop the PHI-at-rest property it reaches for. Includes the sibling audit: the identical 3-char shape survives at :303, a 4-char instance at test_content_search.py:123, and ~13 >=6-char instances whose rate is immaterial but whose shape is the same. `test_off_by_default_stores_plaintext` does NOT share the shape (deterministic equality) and needs no change. --- docs/BACKLOG.md | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 3e04e138..a3dcd40f 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -8423,3 +8423,39 @@ Two worked instances the same day. **#74** went green on 2026-07-30 and sat unme **Related:** #339, #342 (sibling fd-1 issue), CLAUDE.md Β§9 (PHI logging), ADR 0087. **Source:** adversarial review of the ADR 0087 sandbox codec, 2026-08-01. + +--- + +## 347. PHI-at-rest tests assert short-substring absence against base64 ciphertext (probabilistic, flakes) + +> 🚧 **Status OPEN (filed 2026-08-02).** `tests/test_store_encryption.py:95` asserts `raw.startswith(MARKER_PREFIX) and "DOE" not in raw` against a value encrypted under `make_cipher(generate_key())` β€” **a fresh random key every run**, so the base64 body is fresh random text every run and the substring clause is a **probabilistic** assertion, not a deterministic one. Base64's alphabet contains `D`, `O` and `E`, so the literal `DOE` occurs by chance. **It fired:** PR #142, CI job `91502517146`, leg `test (windows-2022, py3.14)` β€” `AssertionError: assert (True and 'DOE' not in 'mfenc:v1:7f...oHUc/t9nnmT9')`. The encryption worked perfectly; the test was wrong. + +**Cluster:** Developer Experience & CI. **Priority:** P2. **Verdict:** build (small). **Severity:** medium (a PHI-at-rest gate that can pass for the wrong reason), medium (likelihood: measured below, and it has already fired once). + +**The rate, and its caveat.** For a uniform base64 string of length *L*, a given 3-character pattern is expected `(L-2)/64Β³` times; at the observed ciphertext length (~146 chars for the `ADT` fixture: 12-byte nonce + 76-byte body + 16-byte tag, base64'd) that is **p β‰ˆ 5.5e-4 β€” about 1 failure in 1,800 runs of this one test, per leg**. Simulation at 200k trials agrees. **The caveat, stated because the number will otherwise be over-trusted:** the simulation used pure base64 over the whole string, while the real value carries a `mfenc:v1::` prefix region where `DOE` cannot occur β€” so the true rate is somewhat *lower*, same order of magnitude. Across three OS legs at this repo's run volume that surfaces every few hundred CI runs as an unreproducible red on an unrelated PR. + +**Why it matters beyond the flake β€” the gate lies in both directions.** The intent ("the patient surname is not readable in the stored body") is exactly right and must be preserved. But a short-substring check against base64 is the wrong instrument for it: it **fails when encryption worked**, and β€” worse β€” it would **pass on a weak encoding** that merely happened not to contain those three characters. Three characters of a 76-character body is not evidence the body is unreadable. This is the failure mode this repo already tracks: a green gate is evidence only if it can actually see the class it claims to cover. + +**The convention already exists in the same file; the sweep was incomplete.** `test_cipher_round_trip_and_hides_plaintext` (:49–58) was already converted to the deterministic form and carries the rule in a comment β€” *"NEVER assert short-substring absence ('MSH'/'DOE') β€” a random base64 body contains any given 3-char run with probability ~len/64Β³, and that assertion HAS flaked in CI"* β€” and `test_v2_round_trip_marker_and_decrypt` (:531) cites it. The **call sites were not swept**, so the identical assertion survives at :95 and :303. + +**Audit of the siblings (asked for at filing; the answer is not "all of them").** Sorted by the measured rate, since the fix priority follows it: + +| Site | Literal | Chars | Approx. p per run | +| --- | --- | --- | --- | +| `test_store_encryption.py:95` (`test_bodies_encrypted_at_rest`) | `DOE` | 3 | ~5e-4 β€” **the one that fired** | +| `test_store_encryption.py:303` (`test_summary_and_metadata_…`) | `DOE` | 3 | ~5e-4 β€” **same shape, not yet observed** | +| `test_content_search.py:123` | `JANE` | 4 | ~9e-6 per value, `all()` over several | +| `test_store_encryption.py:232/233/251`, `:1062`, `test_sqlserver_store.py:1582/1584/1682/1920/2044`, `test_postgres_store.py:2163/2164`, `test_reference_sets.py:170`, `test_transform_state.py:281` | `SECRET…`, `999001`, `WESTWING`, `bad parse`, … | β‰₯6 | ≀2e-9 β€” immaterial, but the same *shape* | + +`test_off_by_default_stores_plaintext` (:111) **does not share the shape** β€” it asserts `_raw_at_rest(db) == ADT`, a deterministic equality against known plaintext, and needs no change. Nor do the many `"DOE" not in …` assertions elsewhere in `tests/` that check *scrubbed plaintext* (`safe_text`, the anonymizer, ACK detail): those run against deterministic output, not random ciphertext, and are correct as written. + +**Fix direction (maintainer's choice β€” do NOT simply widen or delete the substring check):** +1. Assert against the **decoded** ciphertext bytes rather than the base64 text, or +2. assert the plaintext is **not recoverable** from the stored value (the property actually claimed), or +3. assert full-plaintext absence β€” `assert ADT not in raw` β€” which is deterministic because the fixture contains `|` and `\r`, bytes base64 can never emit. That is the idiom :56 already uses, so it is the cheapest sweep. + +Whichever is chosen, the `startswith(MARKER_PREFIX)` half stays, and the β‰₯6-char sites should be swept for **shape** even though their rate is immaterial β€” leaving them teaches the wrong pattern to the next call site, which is how :95 and :303 got written. + +**Related:** [`tests/test_store_encryption.py`](../tests/test_store_encryption.py):95, :303, :49–58 (the convention comment, already correct); [`tests/test_content_search.py`](../tests/test_content_search.py):123; #344 (fixed wall-clock bounds β€” the sibling "an individually blameless failure that reads as a broken branch" class, and the same misdiagnosis risk this repo has already hit twice); CLAUDE.md Β§9 (the PHI-at-rest guarantee the assertion is reaching for). + +**Source:** PR #142 (BACKLOG #323 layer 3, SMTP TLS), 2026-08-02 β€” the failure was observed on that PR's CI and deliberately **not** fixed there, because it is unrelated to the SMTP change and widening the PR would have obscured it. Rate measured analytically and by simulation at filing; sibling audit performed at filing against the working tree. From edc726ce087412f3f6772a9395dfa2326129302a Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sun, 2 Aug 2026 10:22:43 -0500 Subject: [PATCH 2/7] backlog(#347): lead with the instrument, and correct the rate to the per-CI-run figure Reframed after the session that owns PR #142 reproduced the diagnosis independently. Three substantive corrections, none of them cosmetic: 1. FRAMING. The banner led with the flake; it now leads with what is actually wrong. The substring check fails when encryption worked AND would pass on a weak encoding that happened to avoid those three characters. The second half is the PHI defect; the flake is only what made someone look. 2. RATE. 5.5e-4 is per assertion per leg. There are two such assertions and three OS legs (ubuntu + windows-2022 + windows-2025, verified against ci.yml), so the figure an operator experiences is 1 in 303 full CI runs, not 1 in 1,820. Both over-estimate caveats kept: L is from one measured value, and the hex-fingerprint prefix is not base64. 3. SCOPE. Reversed my own recommendation to sweep the >=6-char sites for shape. Rewriting a dozen correct assertions costs review attention for no risk reduction; the pattern-propagation concern is answered by writing the ">=6 characters" rule into the :49-58 convention comment instead. Fix list is now :95, :303, and test_content_search.py:123 (4 chars, unsafe under that same rule -- an addition to the reviewing session's list, in its own framing). Also records the confirmation-by-prediction: #142's re-run came back 25 passed with the prediction written beforehand, so a green re-run confirms a chance collision rather than resetting the question. --- docs/BACKLOG.md | 50 +++++++++++++++++++++++++++++++------------------ 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index a3dcd40f..3c5f36f5 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -8426,36 +8426,50 @@ Two worked instances the same day. **#74** went green on 2026-07-30 and sat unme --- -## 347. PHI-at-rest tests assert short-substring absence against base64 ciphertext (probabilistic, flakes) +## 347. A PHI-at-rest assertion that can pass for the wrong reason β€” short substring vs. random ciphertext -> 🚧 **Status OPEN (filed 2026-08-02).** `tests/test_store_encryption.py:95` asserts `raw.startswith(MARKER_PREFIX) and "DOE" not in raw` against a value encrypted under `make_cipher(generate_key())` β€” **a fresh random key every run**, so the base64 body is fresh random text every run and the substring clause is a **probabilistic** assertion, not a deterministic one. Base64's alphabet contains `D`, `O` and `E`, so the literal `DOE` occurs by chance. **It fired:** PR #142, CI job `91502517146`, leg `test (windows-2022, py3.14)` β€” `AssertionError: assert (True and 'DOE' not in 'mfenc:v1:7f...oHUc/t9nnmT9')`. The encryption worked perfectly; the test was wrong. +> 🚧 **Status OPEN (filed 2026-08-02).** `tests/test_store_encryption.py:95` asserts `raw.startswith(MARKER_PREFIX) and "DOE" not in raw` β€” three characters of a 76-character body β€” as the proof that a patient surname is unreadable at rest. **The instrument is wrong in both directions.** It **fails when encryption worked perfectly** (the value is encrypted under `make_cipher(generate_key())`, a fresh random key every run, so the base64 body is fresh random text and base64's alphabet contains `D`, `O` and `E`), and β€” the half that matters β€” it would **PASS on a weak encoding that merely happened to avoid those three characters**. A test that can pass for the wrong reason is a false assurance about PHI; one that occasionally fails for the wrong reason is only noise. **The flake is what made someone look; it is not what is wrong.** -**Cluster:** Developer Experience & CI. **Priority:** P2. **Verdict:** build (small). **Severity:** medium (a PHI-at-rest gate that can pass for the wrong reason), medium (likelihood: measured below, and it has already fired once). +**Cluster:** Security & Compliance. **Priority:** P2. **Verdict:** build (small). **Severity:** medium (a PHI-at-rest gate that certifies a property it cannot see), medium (likelihood: measured below, and it has already fired). -**The rate, and its caveat.** For a uniform base64 string of length *L*, a given 3-character pattern is expected `(L-2)/64Β³` times; at the observed ciphertext length (~146 chars for the `ADT` fixture: 12-byte nonce + 76-byte body + 16-byte tag, base64'd) that is **p β‰ˆ 5.5e-4 β€” about 1 failure in 1,800 runs of this one test, per leg**. Simulation at 200k trials agrees. **The caveat, stated because the number will otherwise be over-trusted:** the simulation used pure base64 over the whole string, while the real value carries a `mfenc:v1::` prefix region where `DOE` cannot occur β€” so the true rate is somewhat *lower*, same order of magnitude. Across three OS legs at this repo's run volume that surfaces every few hundred CI runs as an unreproducible red on an unrelated PR. +**How it surfaced.** PR #142, CI job `91502517146`, leg `test (windows-2022, py3.14)`: `AssertionError: assert (True and 'DOE' not in 'mfenc:v1:7f...oHUc/t9nnmT9')`. -**Why it matters beyond the flake β€” the gate lies in both directions.** The intent ("the patient surname is not readable in the stored body") is exactly right and must be preserved. But a short-substring check against base64 is the wrong instrument for it: it **fails when encryption worked**, and β€” worse β€” it would **pass on a weak encoding** that merely happened not to contain those three characters. Three characters of a 76-character body is not evidence the body is unreadable. This is the failure mode this repo already tracks: a green gate is evidence only if it can actually see the class it claims to cover. +**The rate β€” the per-CI-run figure, not the per-assertion one.** For a uniform base64 string of length *L*, a given *k*-character pattern is expected `(L-k+1)/64^k` times. At the observed ciphertext (~146 chars for the `ADT` fixture: 12-byte nonce + 76-byte body + 16-byte tag, base64'd) a 3-char literal gives **p β‰ˆ 5.5e-4 per assertion per run** β€” corroborated by a 200k-trial simulation, and independently reproduced by a second session at N=144 windows. But **there are two such assertions** (`:95` and `:303`, both `DOE`) and this repo runs **three OS legs** (`ubuntu` + `windows-2022` + `windows-2025`, one Python version β€” [`ci.yml`](../.github/workflows/ci.yml)), so what an operator actually experiences is: -**The convention already exists in the same file; the sweep was incomplete.** `test_cipher_round_trip_and_hides_plaintext` (:49–58) was already converted to the deterministic form and carries the rule in a comment β€” *"NEVER assert short-substring absence ('MSH'/'DOE') β€” a random base64 body contains any given 3-char run with probability ~len/64Β³, and that assertion HAS flaked in CI"* β€” and `test_v2_round_trip_marker_and_decrypt` (:531) cites it. The **call sites were not swept**, so the identical assertion survives at :95 and :303. +| Scope | Rate | +| --- | --- | +| one assertion, one leg | 1 in 1,820 | +| either assertion, one leg | 1 in 910 | +| **either assertion, one full CI run (3 legs)** | **1 in 303** | -**Audit of the siblings (asked for at filing; the answer is not "all of them").** Sorted by the measured rate, since the fix priority follows it: +**Both caveats, because neither number should be handed on bare.** *L* is taken from one measured at-rest value and real strings vary in length; and the `mfenc:v1::` prefix region is not base64, so the effective window count is lower and **every figure above is a slight over-estimate**. Same order, not exact. What is not in doubt is the scope correction: 1 in 303 CI runs is an operational cost, where 1 in 1,820 reads as ignorable. -| Site | Literal | Chars | Approx. p per run | -| --- | --- | --- | --- | -| `test_store_encryption.py:95` (`test_bodies_encrypted_at_rest`) | `DOE` | 3 | ~5e-4 β€” **the one that fired** | -| `test_store_encryption.py:303` (`test_summary_and_metadata_…`) | `DOE` | 3 | ~5e-4 β€” **same shape, not yet observed** | -| `test_content_search.py:123` | `JANE` | 4 | ~9e-6 per value, `all()` over several | -| `test_store_encryption.py:232/233/251`, `:1062`, `test_sqlserver_store.py:1582/1584/1682/1920/2044`, `test_postgres_store.py:2163/2164`, `test_reference_sets.py:170`, `test_transform_state.py:281` | `SECRET…`, `999001`, `WESTWING`, `bad parse`, … | β‰₯6 | ≀2e-9 β€” immaterial, but the same *shape* | +**Confirmed by prediction, not by agreement.** After the failure, PR #142's full re-run came back **25 passed, 0 failed** with `test_bodies_encrypted_at_rest` green. That prediction (P(same collision twice) β‰ˆ 5e-4) was written down *before* the re-run β€” so a green re-run confirms a chance collision rather than resetting the question. Had it failed twice, the diagnosis would have been falsified and something real would be at fault. -`test_off_by_default_stores_plaintext` (:111) **does not share the shape** β€” it asserts `_raw_at_rest(db) == ADT`, a deterministic equality against known plaintext, and needs no change. Nor do the many `"DOE" not in …` assertions elsewhere in `tests/` that check *scrubbed plaintext* (`safe_text`, the anonymizer, ACK detail): those run against deterministic output, not random ciphertext, and are correct as written. +**The convention already exists in the same file; the sweep was incomplete.** `test_cipher_round_trip_and_hides_plaintext` (:49–58) was already converted to the deterministic form and carries the rule in a comment β€” *"NEVER assert short-substring absence ('MSH'/'DOE') … that assertion HAS flaked in CI"* β€” and `test_v2_round_trip_marker_and_decrypt` (:531) cites it. The **call sites were never swept**, so the identical assertion survives at :95 and :303. + +**The rule, so the next call site has a boundary rather than a precedent:** *a substring assertion against ciphertext is safe iff the token is **β‰₯ 6 characters**, and unsafe below that.* The exponent is the token length, so risk collapses onto the short literals β€” at 6 chars p β‰ˆ 2e-9 (1 in 477 million), at 8 chars β‰ˆ 5e-13, at 14 chars below any rate that can occur. + +**Audit of the siblings β€” the answer is neither "just one" nor "all of them".** + +| Site | Literal | Chars | p per run | Verdict | +| --- | --- | --- | --- | --- | +| `test_store_encryption.py:95` (`test_bodies_encrypted_at_rest`) | `DOE` | 3 | 5.5e-4 | **fix β€” the one that fired** | +| `test_store_encryption.py:303` (`test_summary_and_metadata_…`) | `DOE` | 3 | 5.5e-4 | **fix β€” same shape, never observed** | +| `test_content_search.py:123` | `JANE` | 4 | ~9e-6 | **fix β€” below the β‰₯6 rule** | +| `test_store_encryption.py:232/233/251`, `:303` (`999001`), `:304`, `:1062`; `test_sqlserver_store.py:1582/1584/1682/1920/2044`; `test_postgres_store.py:2163/2164`; `test_reference_sets.py:170`; `test_transform_state.py:281` | `SECRET…`, `999001`, `WESTWING`, `bad parse`, … | β‰₯6 | ≀2e-9 | **leave alone** | + +**Do not rewrite the β‰₯6 group.** They are the same *pattern* but not a defect at any rate that will ever be observed, and churning a dozen correct assertions makes the diff harder to review for no risk reduction. The pattern-propagation concern is real but is answered by **writing the β‰₯6 rule into the :49–58 comment**, not by the churn. (Two scopes were counted independently and agree: seven assertions of this shape within `test_store_encryption.py`, ~sixteen repo-wide β€” different denominators, not a disagreement.) + +`test_off_by_default_stores_plaintext` (:111) **does not share the shape** β€” `_raw_at_rest(db) == ADT`, deterministic equality against known plaintext. Nor do the many `"DOE" not in …` assertions elsewhere in `tests/` that check *scrubbed plaintext* (`safe_text`, the anonymizer, ACK detail): deterministic output, not ciphertext, correct as written. **Fix direction (maintainer's choice β€” do NOT simply widen or delete the substring check):** 1. Assert against the **decoded** ciphertext bytes rather than the base64 text, or 2. assert the plaintext is **not recoverable** from the stored value (the property actually claimed), or -3. assert full-plaintext absence β€” `assert ADT not in raw` β€” which is deterministic because the fixture contains `|` and `\r`, bytes base64 can never emit. That is the idiom :56 already uses, so it is the cheapest sweep. +3. assert full-plaintext absence β€” `assert ADT not in raw` β€” deterministic because the fixture contains `|` and `\r`, bytes base64 can never emit. That is the idiom :56 already uses, so it is the cheapest change. -Whichever is chosen, the `startswith(MARKER_PREFIX)` half stays, and the β‰₯6-char sites should be swept for **shape** even though their rate is immaterial β€” leaving them teaches the wrong pattern to the next call site, which is how :95 and :303 got written. +The `startswith(MARKER_PREFIX)` half stays in every case. -**Related:** [`tests/test_store_encryption.py`](../tests/test_store_encryption.py):95, :303, :49–58 (the convention comment, already correct); [`tests/test_content_search.py`](../tests/test_content_search.py):123; #344 (fixed wall-clock bounds β€” the sibling "an individually blameless failure that reads as a broken branch" class, and the same misdiagnosis risk this repo has already hit twice); CLAUDE.md Β§9 (the PHI-at-rest guarantee the assertion is reaching for). +**Related:** [`tests/test_store_encryption.py`](../tests/test_store_encryption.py):95, :303, :49–58 (the convention comment, already correct β€” the natural home for the β‰₯6 rule); [`tests/test_content_search.py`](../tests/test_content_search.py):123; #344 (the sibling "individually blameless CI red that invites the wrong diagnosis" class β€” but distinct in kind: #344 is a bound that drifted out of proportion to real work, this was never deterministic at any bound); CLAUDE.md Β§9 (the PHI-at-rest guarantee the assertion is reaching for); [`Secure_Development_Standards`](Secure_Development_Standards.md) Β§3 (reviewing security prose by what a reader would DO with it β€” the same question, asked of a test instead of a paragraph). -**Source:** PR #142 (BACKLOG #323 layer 3, SMTP TLS), 2026-08-02 β€” the failure was observed on that PR's CI and deliberately **not** fixed there, because it is unrelated to the SMTP change and widening the PR would have obscured it. Rate measured analytically and by simulation at filing; sibling audit performed at filing against the working tree. +**Source:** PR #142 (BACKLOG #323 layer 3, SMTP TLS), 2026-08-02 β€” observed on that PR's CI and deliberately **not** fixed there, because it is unrelated to the SMTP change and widening the PR would have obscured it. Diagnosis, rate and sibling audit produced independently by two sessions and reconciled: the β‰₯6 rule, the leave-the-rest-alone scoping, the per-CI-run rate correction and the instrument-first framing all come from the review by the session that owns #142. From 89b1d1eee5d6237f8ea603dd671f467a1b010acc Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sun, 2 Aug 2026 10:28:01 -0500 Subject: [PATCH 3/7] backlog(#347): exact rates, a verified count, and the float64 zero that hid in the analysis Three sessions reviewed this; each correction below is theirs, verified here rather than adopted. RATES. Recomputed exactly: 1 in 1,821 per assertion per leg, 1 in 911 per leg, 1 in 304 per full CI run (I had floored two of them). JANE at test_content_search.py:123 is 8.58e-6 = 1 in 116,509. THE ZERO. The 14-char row was written as "p=0 (unreachable)". It is 7.44e-24. Cause reproduced: 1-(1-64**-14)**144 UNDERFLOWS to exactly 0.0 in float64, silently, with no warning, in a column of plausible values -- inside an analysis arguing that token length is the discriminator, at the one row where length breaks the arithmetic. The item now carries the trap, the exact value, and the idiom that does not underflow (-expm1(N*log1p(-x))). Every figure recomputed both ways and agreeing. THE COUNT. Three different numbers were quoted before anyone checked (7, then 5, then ~16 for a different denominator). Re-derived from the tree: 7 lines / 8 clauses against ciphertext in test_store_encryption.py, 2 of them unsafe; ~16 repo-wide. The item now states the basis AND the exclusions (:905-908 and :927-928 are caplog assertions against log text, not ciphertext; :56/:532/:625 are full-plaintext; :512 is deterministic twice over) so the count is not re-litigated a fourth time. #344 CITATION. Kept -- its owner confirmed the framing and supplied the discriminator that stops a reader folding the two: this one would fire at exactly the same rate on an infinitely fast machine. Related line now says what the citation is NOT. #346 / ADR 0158. Added as the closer sibling: an assertion that passes for a reason unrelated to the property it tests. ADR 0158 cited WITHOUT a link -- I guessed its filename, checked, and was wrong; the file is on PR #145's branch, not on main. VERIFICATION. backlog_status_check.py falsified against this item: a deliberately doubled banner makes it fail at BACKLOG.md:8429 naming #347. The first probe attempt silently no-op'd on a cp1252 decode and "passed" -- the same shape this item is about. --- docs/BACKLOG.md | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 3c5f36f5..dcc93355 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -8438,28 +8438,32 @@ Two worked instances the same day. **#74** went green on 2026-07-30 and sat unme | Scope | Rate | | --- | --- | -| one assertion, one leg | 1 in 1,820 | -| either assertion, one leg | 1 in 910 | -| **either assertion, one full CI run (3 legs)** | **1 in 303** | +| one assertion, one leg | 1 in 1,821 | +| either assertion, one leg | 1 in 911 | +| **either assertion, one full CI run (3 legs)** | **1 in 304** | -**Both caveats, because neither number should be handed on bare.** *L* is taken from one measured at-rest value and real strings vary in length; and the `mfenc:v1::` prefix region is not base64, so the effective window count is lower and **every figure above is a slight over-estimate**. Same order, not exact. What is not in doubt is the scope correction: 1 in 303 CI runs is an operational cost, where 1 in 1,820 reads as ignorable. +**Both caveats, because neither number should be handed on bare.** *L* is taken from one measured at-rest value and real strings vary in length; and the `mfenc:v1::` prefix region is not base64, so the effective window count is lower and **every figure above is a slight over-estimate**. Same order, not exact. What is not in doubt is the scope correction: 1 in 304 CI runs is an operational cost, where 1 in 1,821 reads as ignorable. **Confirmed by prediction, not by agreement.** After the failure, PR #142's full re-run came back **25 passed, 0 failed** with `test_bodies_encrypted_at_rest` green. That prediction (P(same collision twice) β‰ˆ 5e-4) was written down *before* the re-run β€” so a green re-run confirms a chance collision rather than resetting the question. Had it failed twice, the diagnosis would have been falsified and something real would be at fault. **The convention already exists in the same file; the sweep was incomplete.** `test_cipher_round_trip_and_hides_plaintext` (:49–58) was already converted to the deterministic form and carries the rule in a comment β€” *"NEVER assert short-substring absence ('MSH'/'DOE') … that assertion HAS flaked in CI"* β€” and `test_v2_round_trip_marker_and_decrypt` (:531) cites it. The **call sites were never swept**, so the identical assertion survives at :95 and :303. -**The rule, so the next call site has a boundary rather than a precedent:** *a substring assertion against ciphertext is safe iff the token is **β‰₯ 6 characters**, and unsafe below that.* The exponent is the token length, so risk collapses onto the short literals β€” at 6 chars p β‰ˆ 2e-9 (1 in 477 million), at 8 chars β‰ˆ 5e-13, at 14 chars below any rate that can occur. +**The rule, so the next call site has a boundary rather than a precedent:** *a substring assertion against ciphertext is safe iff the token is **β‰₯ 6 characters**, and unsafe below that.* The exponent is the token length, so the risk collapses onto the short literals β€” at 6 chars p = 2.10e-9 (1 in 477 million), at 8 chars 5.12e-13, at 14 chars 7.44e-24. + +> **A trap for whoever re-derives these.** The obvious expression `1-(1-64**-k)**N` **underflows to exactly `0.0` at k=14** β€” `1 - 64**-14` is not representable in float64 and rounds to `1.0` β€” so it reports a probability of zero, silently, with no warning, in a column of otherwise plausible values. That is how the 14-char row was first written down as "unreachable", inside an analysis arguing that token length is the discriminator, at the one row where length was extreme enough to break the arithmetic. Use `-expm1(N*log1p(-64**-k))`, or `Fraction`; both give 7.44e-24. **The figures above were computed both ways and agree.** Reproducing that zero in a doc about an assertion that states more confidence than it has would have been the same defect one level up β€” hence "7.44e-24", not "0". **Audit of the siblings β€” the answer is neither "just one" nor "all of them".** -| Site | Literal | Chars | p per run | Verdict | +| Site | Literal | Chars | p per assertion per leg | Verdict | | --- | --- | --- | --- | --- | -| `test_store_encryption.py:95` (`test_bodies_encrypted_at_rest`) | `DOE` | 3 | 5.5e-4 | **fix β€” the one that fired** | -| `test_store_encryption.py:303` (`test_summary_and_metadata_…`) | `DOE` | 3 | 5.5e-4 | **fix β€” same shape, never observed** | -| `test_content_search.py:123` | `JANE` | 4 | ~9e-6 | **fix β€” below the β‰₯6 rule** | -| `test_store_encryption.py:232/233/251`, `:303` (`999001`), `:304`, `:1062`; `test_sqlserver_store.py:1582/1584/1682/1920/2044`; `test_postgres_store.py:2163/2164`; `test_reference_sets.py:170`; `test_transform_state.py:281` | `SECRET…`, `999001`, `WESTWING`, `bad parse`, … | β‰₯6 | ≀2e-9 | **leave alone** | +| `test_store_encryption.py:95` (`test_bodies_encrypted_at_rest`) | `DOE` | 3 | 5.49e-4 (1 in 1,821) | **fix β€” the one that fired** | +| `test_store_encryption.py:303` (`test_summary_and_metadata_…`) | `DOE` | 3 | 5.49e-4 (1 in 1,821) | **fix β€” same shape, never observed** | +| `test_content_search.py:123` | `JANE` | 4 | 8.58e-6 (1 in 116,509) | **fix β€” below the β‰₯6 rule** | +| `test_store_encryption.py:232/233/251`, `:303` (`999001`), `:304`, `:1062`; `test_sqlserver_store.py:1582/1584/1682/1920/2044`; `test_postgres_store.py:2163/2164`; `test_reference_sets.py:170`; `test_transform_state.py:281` | `SECRET…`, `999001`, `WESTWING`, `bad parse`, … | β‰₯6 | ≀2.10e-9 | **leave alone** | + +**Do not rewrite the β‰₯6 group.** They are the same *pattern* but not a defect at any rate that will ever be observed, and churning a dozen correct assertions makes the diff harder to review for no risk reduction. The pattern-propagation concern is real but is answered by **writing the β‰₯6 rule into the :49–58 comment**, not by the churn. -**Do not rewrite the β‰₯6 group.** They are the same *pattern* but not a defect at any rate that will ever be observed, and churning a dozen correct assertions makes the diff harder to review for no risk reduction. The pattern-propagation concern is real but is answered by **writing the β‰₯6 rule into the :49–58 comment**, not by the churn. (Two scopes were counted independently and agree: seven assertions of this shape within `test_store_encryption.py`, ~sixteen repo-wide β€” different denominators, not a disagreement.) +**The count, stated with its basis, because three different numbers were quoted before anyone checked.** Within `test_store_encryption.py` there are **7 lines carrying 8 substring-absence clauses against at-rest ciphertext** (:95, :232, :233, :251, :303 Γ—2, :304, :1062), of which **2 β€” both `DOE`, at :95 and :303 β€” are below the β‰₯6 rule**. Repo-wide the shape appears ~16 times. Correctly **excluded** and not to be counted again: `:905–908` and `:927–928` assert against exception/`caplog` text, not ciphertext (a different shape a grep sweeps up); `:56`, `:532`, `:625` are `ADT not in token`, full-plaintext and deterministic; `:512` is `":v2:" not in produced` against a fixed-nonce v1 blob, deterministic twice over (`:` is not in the base64 alphabet). `test_off_by_default_stores_plaintext` (:111) **does not share the shape** β€” `_raw_at_rest(db) == ADT`, deterministic equality against known plaintext. Nor do the many `"DOE" not in …` assertions elsewhere in `tests/` that check *scrubbed plaintext* (`safe_text`, the anonymizer, ACK detail): deterministic output, not ciphertext, correct as written. @@ -8470,6 +8474,10 @@ Two worked instances the same day. **#74** went green on 2026-07-30 and sat unme The `startswith(MARKER_PREFIX)` half stays in every case. -**Related:** [`tests/test_store_encryption.py`](../tests/test_store_encryption.py):95, :303, :49–58 (the convention comment, already correct β€” the natural home for the β‰₯6 rule); [`tests/test_content_search.py`](../tests/test_content_search.py):123; #344 (the sibling "individually blameless CI red that invites the wrong diagnosis" class β€” but distinct in kind: #344 is a bound that drifted out of proportion to real work, this was never deterministic at any bound); CLAUDE.md Β§9 (the PHI-at-rest guarantee the assertion is reaching for); [`Secure_Development_Standards`](Secure_Development_Standards.md) Β§3 (reviewing security prose by what a reader would DO with it β€” the same question, asked of a test instead of a paragraph). +**Related:** [`tests/test_store_encryption.py`](../tests/test_store_encryption.py):95, :303, :49–58 (the convention comment, already correct β€” the natural home for the β‰₯6 rule); [`tests/test_content_search.py`](../tests/test_content_search.py):123; CLAUDE.md Β§9 (the PHI-at-rest guarantee the assertion is reaching for); [`Secure_Development_Standards`](Secure_Development_Standards.md) Β§3 (reviewing security prose by what a reader would DO with it β€” the same question, asked of a test instead of a paragraph). + +**#344 β€” cited for the harm, NOT the cause; do not fold them.** Both are individually-blameless CI reds that invite the wrong fix, in a repo whose two famous "flakes" turned out to be a livelock and a test that was right. But: **#344 is a fixed bound meeting variable latency; #347 is a deterministic property tested by a probabilistic proxy.** Neither is fixed by changing a number, for opposite reasons. The one-line discriminator, from #344's owner: **this one would fire at exactly the same rate on an infinitely fast machine.** A reader who follows the link lands on a wall-clock item and must not back-infer that this is a timing bug β€” it is not. + +**#346 β€” the closer sibling, and the reason a taxonomy is wanted.** Same defect class stated generally: *an assertion that passes for a reason unrelated to the property it claims to test.* This one passes because random base64 usually lacks a 3-character run; #346's would have passed because nothing walks the imports. Both are green signals that are not evidence. If **ADR 0158** *"Silent controls β€” green signals that mean nothing"* lands (PR #145; deliberately unlinked β€” the file is on that branch, not yet on `main`), that is the taxonomy both belong under, and this item's measured `p` is a better worked example than a hypothetical. -**Source:** PR #142 (BACKLOG #323 layer 3, SMTP TLS), 2026-08-02 β€” observed on that PR's CI and deliberately **not** fixed there, because it is unrelated to the SMTP change and widening the PR would have obscured it. Diagnosis, rate and sibling audit produced independently by two sessions and reconciled: the β‰₯6 rule, the leave-the-rest-alone scoping, the per-CI-run rate correction and the instrument-first framing all come from the review by the session that owns #142. +**Source:** PR #142 (BACKLOG #323 layer 3, SMTP TLS), 2026-08-02 β€” observed on that PR's CI and deliberately **not** fixed there, because it is unrelated to the SMTP change and widening the PR would have obscured it. Diagnosis, rates and sibling audit produced independently by three sessions and reconciled; the instrument-first framing, the β‰₯6 rule and the leave-the-rest-alone scoping come from the review by the session that owns #142, the #344 discriminator from #344's owner. **Verification of this filing's own instruments:** every probability recomputed by two methods that agree (`Fraction` and `-expm1(N*log1p(-x))`), the audit counts re-derived from the working tree rather than quoted, and `backlog_status_check.py` **falsified against this item** β€” a deliberately doubled banner made it fail at `BACKLOG.md:8429` naming #347, so its green is evidence that it can see this item rather than evidence it skipped it. From a63c63ccf1768e97132c3fda679c50aa5f4e2c23 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sun, 2 Aug 2026 10:32:36 -0500 Subject: [PATCH 4/7] backlog(#347): two safety grounds, and a correction to the one I had just adopted The #142-owning session pointed out that :512 is safe for a reason unlike the others -- ':' is not in the base64 alphabet -- and suggested it as a third safety category. It is a real category and it is now ground (1), stated more usefully than either of us first had it: a token containing a character the value cannot contain is a PROOF at any length, and it is the same principle that makes the recommended fix (assert ADT not in raw) deterministic. Rule and remedy are now one idea. But the argument as given does not survive, and I checked before adopting it. The haystack is :, NOT base64 alone, and the marker carries colons -- so ':' IS representable and ground (1) does not cover :512. It is deterministic for a different reason: fixed marker layout with the version field reading v1, plus a body with no ':' for the run to straddle. Structure, not alphabet. The item now says so explicitly, because a rule that is right about the conclusion and wrong about the mechanism is the thing this whole item is about. Verified mechanically rather than argued: over the haystack's actual character set, the ADT fixture carries \r & . \ ^ | (unrepresentable -> ground 1 genuinely holds), while DOE, JANE, SECRET, WESTWING, SECRETSTATEMRN and ':v2:' are ALL fully representable -- so ground (1) applies to none of them and length is their only defense. That is what the table already claimed; now it is checked. Also records why the float64 trap survives review, from #344's owner: the naive expression is correct everywhere you would sanity-check it and silently wrong only in the tail. --- docs/BACKLOG.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index dcc93355..5ef167d3 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -8448,9 +8448,14 @@ Two worked instances the same day. **#74** went green on 2026-07-30 and sat unme **The convention already exists in the same file; the sweep was incomplete.** `test_cipher_round_trip_and_hides_plaintext` (:49–58) was already converted to the deterministic form and carries the rule in a comment β€” *"NEVER assert short-substring absence ('MSH'/'DOE') … that assertion HAS flaked in CI"* β€” and `test_v2_round_trip_marker_and_decrypt` (:531) cites it. The **call sites were never swept**, so the identical assertion survives at :95 and :303. -**The rule, so the next call site has a boundary rather than a precedent:** *a substring assertion against ciphertext is safe iff the token is **β‰₯ 6 characters**, and unsafe below that.* The exponent is the token length, so the risk collapses onto the short literals β€” at 6 chars p = 2.10e-9 (1 in 477 million), at 8 chars 5.12e-13, at 14 chars 7.44e-24. +**The rule, so the next call site has a boundary rather than a precedent.** A substring assertion against ciphertext is safe on **either** of two grounds, and they are not equally good: -> **A trap for whoever re-derives these.** The obvious expression `1-(1-64**-k)**N` **underflows to exactly `0.0` at k=14** β€” `1 - 64**-14` is not representable in float64 and rounds to `1.0` β€” so it reports a probability of zero, silently, with no warning, in a column of otherwise plausible values. That is how the 14-char row was first written down as "unreachable", inside an analysis arguing that token length is the discriminator, at the one row where length was extreme enough to break the arithmetic. Use `-expm1(N*log1p(-64**-k))`, or `Fraction`; both give 7.44e-24. **The figures above were computed both ways and agree.** Reproducing that zero in a doc about an assertion that states more confidence than it has would have been the same defect one level up β€” hence "7.44e-24", not "0". +1. **Deterministic β€” the token contains a character the *whole stored value* cannot contain.** The haystack is `:`, so the test is against that, not against the base64 alphabet alone: `|` and `\r` qualify, **`:` does not** (the marker carries colons). Where it holds, the assertion cannot fail by chance **at any length** β€” a *proof*, not a probability. +2. **Probabilistic β€” the token is β‰₯ 6 characters.** The exponent is the token length, so risk collapses fast: at 6 chars p = 2.10e-9 (1 in 477 million), at 8 chars 5.12e-13, at 14 chars 7.44e-24. Below 6, unsafe. + +**Prefer (1). It is the same principle as the recommended fix** β€” `assert ADT not in raw` is deterministic precisely because the fixture carries `|` and `\r` β€” so the rule and the remedy are one idea, not two. Ground (2) is what to fall back on when the token must be a bare identifier; it makes an assertion *improbable*, never *impossible*. + +> **A trap for whoever re-derives these.** The obvious expression `1-(1-64**-k)**N` **underflows to exactly `0.0` at k=14** β€” `1 - 64**-14` is not representable in float64 and rounds to `1.0` β€” so it reports a probability of zero, silently, with no warning, in a column of otherwise plausible values. **It is correct everywhere you would sanity-check it and silently wrong only in the tail**, which is why it survives review: the first row you try agrees with every other method to six figures. That is how the 14-char row was first written down as "unreachable", inside an analysis arguing that token length is the discriminator, at the one row where length was extreme enough to break the arithmetic. Use `-expm1(N*log1p(-64**-k))`, or `Fraction`; both give 7.44e-24. **The figures above were computed both ways and agree.** Reproducing that zero in a doc about an assertion that states more confidence than it has would have been the same defect one level up β€” hence "7.44e-24", not "0". **Audit of the siblings β€” the answer is neither "just one" nor "all of them".** @@ -8463,7 +8468,7 @@ Two worked instances the same day. **#74** went green on 2026-07-30 and sat unme **Do not rewrite the β‰₯6 group.** They are the same *pattern* but not a defect at any rate that will ever be observed, and churning a dozen correct assertions makes the diff harder to review for no risk reduction. The pattern-propagation concern is real but is answered by **writing the β‰₯6 rule into the :49–58 comment**, not by the churn. -**The count, stated with its basis, because three different numbers were quoted before anyone checked.** Within `test_store_encryption.py` there are **7 lines carrying 8 substring-absence clauses against at-rest ciphertext** (:95, :232, :233, :251, :303 Γ—2, :304, :1062), of which **2 β€” both `DOE`, at :95 and :303 β€” are below the β‰₯6 rule**. Repo-wide the shape appears ~16 times. Correctly **excluded** and not to be counted again: `:905–908` and `:927–928` assert against exception/`caplog` text, not ciphertext (a different shape a grep sweeps up); `:56`, `:532`, `:625` are `ADT not in token`, full-plaintext and deterministic; `:512` is `":v2:" not in produced` against a fixed-nonce v1 blob, deterministic twice over (`:` is not in the base64 alphabet). +**The count, stated with its basis, because three different numbers were quoted before anyone checked.** Within `test_store_encryption.py` there are **7 lines carrying 8 substring-absence clauses against at-rest ciphertext** (:95, :232, :233, :251, :303 Γ—2, :304, :1062), of which **2 β€” both `DOE`, at :95 and :303 β€” are below the β‰₯6 rule**. Repo-wide the shape appears ~16 times. Correctly **excluded** and not to be counted again: `:905–908` and `:927–928` assert against exception/`caplog` text, not ciphertext (a different shape a grep sweeps up); `:56`, `:532`, `:625` (`ADT not in token`) are safe on **ground (1)** β€” `|` and `\r` appear nowhere in `:`. `:512` (`":v2:" not in produced`) is deterministic too but **not** on ground (1), and the distinction is worth keeping straight: `:` *is* present in the haystack (the marker is `mfenc:v1::`), so ground (1) does not apply. It holds instead because the marker's layout is fixed and its version field reads `v1`, while the base64 body contains no `:` for the run to straddle β€” structure, not alphabet. `test_off_by_default_stores_plaintext` (:111) **does not share the shape** β€” `_raw_at_rest(db) == ADT`, deterministic equality against known plaintext. Nor do the many `"DOE" not in …` assertions elsewhere in `tests/` that check *scrubbed plaintext* (`safe_text`, the anonymizer, ACK detail): deterministic output, not ciphertext, correct as written. From 89a893bd55aa10ebbaf89db8feba558f2111e73e Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sun, 2 Aug 2026 10:34:39 -0500 Subject: [PATCH 5/7] backlog(#347): cite ADR 0158 by its rule and its class, not by its number The session landing ADR 0158 (PR #145) confirmed #347 is a genuine instance and supplied the precise anchor instead of a general pointer. Verified against the ADR on its branch before citing -- all three lines read verbatim as quoted: :246 "An equality check satisfiable by coincidence is not an equality check." :60 Class 2 -- a control that cannot OBSERVE or ACT ON its own failure :61 Test: "if this control were broken, what would tell me?" :56 / :439 the taxonomy explicitly disclaims completeness Citing the RULE is what makes the reference survive renumbering, and the Class 2 test is the sharper statement of this defect than anything I had written: if the encryption were replaced tomorrow with a weak encoding, "DOE" not in raw would still go green. The answer to "what would tell me" is the control itself. Still deliberately unlinked -- 0158 is on #145's branch, absent from main, so a relative link renders broken. The follow-up (file this against 0158 once it is on main) is recorded as NOT done here, with the reason: that session declined to add instances its author did not choose, and padding a rescued document at merge time is its own defect. Their call, recorded so it does not read as an oversight. --- docs/BACKLOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 5ef167d3..5a477ef0 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -8483,6 +8483,8 @@ The `startswith(MARKER_PREFIX)` half stays in every case. **#344 β€” cited for the harm, NOT the cause; do not fold them.** Both are individually-blameless CI reds that invite the wrong fix, in a repo whose two famous "flakes" turned out to be a livelock and a test that was right. But: **#344 is a fixed bound meeting variable latency; #347 is a deterministic property tested by a probabilistic proxy.** Neither is fixed by changing a number, for opposite reasons. The one-line discriminator, from #344's owner: **this one would fire at exactly the same rate on an infinitely fast machine.** A reader who follows the link lands on a wall-clock item and must not back-infer that this is a timing bug β€” it is not. -**#346 β€” the closer sibling, and the reason a taxonomy is wanted.** Same defect class stated generally: *an assertion that passes for a reason unrelated to the property it claims to test.* This one passes because random base64 usually lacks a 3-character run; #346's would have passed because nothing walks the imports. Both are green signals that are not evidence. If **ADR 0158** *"Silent controls β€” green signals that mean nothing"* lands (PR #145; deliberately unlinked β€” the file is on that branch, not yet on `main`), that is the taxonomy both belong under, and this item's measured `p` is a better worked example than a hypothetical. +**#346 β€” the closer sibling.** Same defect class stated generally: *an assertion that passes for a reason unrelated to the property it claims to test.* This one passes because random base64 usually lacks a 3-character run; #346's would have passed because nothing walks the imports. Both are green signals that are not evidence. + +**ADR 0158 β€” the taxonomy, and where this item sits in it.** Cited by **rule**, not just by number, because the rule is what transfers: *"An equality check satisfiable by coincidence is not an equality check."* That is this defect exactly. By the ADR's own one-line test for **Class 2** β€” *a control that cannot observe or act on its own failure*: **if this control were broken, what would tell me?** If the encryption were replaced tomorrow with a weak encoding, `"DOE" not in raw` would still go green. The answer is the control, which is the defect. *(Deliberately unlinked: the ADR is on PR #145's branch and not yet on `main`, so a relative link would render broken. Guessing its filename from its title is the same failure mode this item is about β€” it was guessed, checked, and was wrong.)* **Follow-up, deliberately not done here:** file this against ADR 0158 **once 0158 is on `main`**. Padding a document at merge time with instances its author did not choose is its own defect, and the ADR's instances are attributed by convention. **Source:** PR #142 (BACKLOG #323 layer 3, SMTP TLS), 2026-08-02 β€” observed on that PR's CI and deliberately **not** fixed there, because it is unrelated to the SMTP change and widening the PR would have obscured it. Diagnosis, rates and sibling audit produced independently by three sessions and reconciled; the instrument-first framing, the β‰₯6 rule and the leave-the-rest-alone scoping come from the review by the session that owns #142, the #344 discriminator from #344's owner. **Verification of this filing's own instruments:** every probability recomputed by two methods that agree (`Fraction` and `-expm1(N*log1p(-x))`), the audit counts re-derived from the working tree rather than quoted, and `backlog_status_check.py` **falsified against this item** β€” a deliberately doubled banner made it fail at `BACKLOG.md:8429` naming #347, so its green is evidence that it can see this item rather than evidence it skipped it. From 2109f61315940540dbe0b8f8296a1dc302c3945f Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sun, 2 Aug 2026 11:02:20 -0500 Subject: [PATCH 6/7] backlog(#347): itemise the provenance, and source the one number that was not mine Three fixes, all of the same defect the item is about -- an observational claim carrying more confidence than its sourcing supports. 1. THE 200k SIMULATION IS NOT MINE. It arrived with the originating defect report and I never ran it; the sentence read as though this filing corroborated the rate that way. Every word was accurate, which is the shape: an unsourced observational claim inside a sentence whose whole job is telling the reader how much to trust the number beside it. Now attributed, with this filing's actual contribution (exact Fraction derivation, cross-checked against expm1/log1p) stated separately. 2. "PRODUCED INDEPENDENTLY BY THREE SESSIONS" was an aggregate confidence claim. Two sessions derived rates; the third contributed process discipline. Replaced with itemised attribution -- who supplied the framing, the >=6 rule, the discriminator, the demand to falsify the gate -- and an explicit statement that no claim rests on a count of who agreed. A session count is not evidence. 3. "#344 IS a fixed bound meeting variable latency" -> "#344's THESIS is". That item's instance 2 has since been re-diagnosed as a swallowed lock-timeout (SET LOCK_TIMEOUT 0 -> native 1222 caught and returned as a normal empty, with the dispatcher then parking in a terminal IDLE) -- not a bound at all. The wholesale characterisation was over-broad, and the item now says not to lean on "#344 = timeouts" as a premise. The chain that prompted this went two sessions -> one -> none -> mechanism-only -> mechanism-only-labelled-as-deduction, on a separate claim, every step a good-faith correction, and the conclusion correct throughout. Only the stated mechanism was hollow, and the stated mechanism is what the next reader carries. --- docs/BACKLOG.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 8df6e90b..b0f4ef1c 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -8469,7 +8469,7 @@ The failure is also **self-concealing in the dangerous direction.** Age-stalenes **How it surfaced.** PR #142, CI job `91502517146`, leg `test (windows-2022, py3.14)`: `AssertionError: assert (True and 'DOE' not in 'mfenc:v1:7f...oHUc/t9nnmT9')`. -**The rate β€” the per-CI-run figure, not the per-assertion one.** For a uniform base64 string of length *L*, a given *k*-character pattern is expected `(L-k+1)/64^k` times. At the observed ciphertext (~146 chars for the `ADT` fixture: 12-byte nonce + 76-byte body + 16-byte tag, base64'd) a 3-char literal gives **p β‰ˆ 5.5e-4 per assertion per run** β€” corroborated by a 200k-trial simulation, and independently reproduced by a second session at N=144 windows. But **there are two such assertions** (`:95` and `:303`, both `DOE`) and this repo runs **three OS legs** (`ubuntu` + `windows-2022` + `windows-2025`, one Python version β€” [`ci.yml`](../.github/workflows/ci.yml)), so what an operator actually experiences is: +**The rate β€” the per-CI-run figure, not the per-assertion one.** For a uniform base64 string of length *L*, a given *k*-character pattern is expected `(L-k+1)/64^k` times. At the observed ciphertext (~146 chars for the `ADT` fixture: 12-byte nonce + 76-byte body + 16-byte tag, base64'd) a 3-char literal gives **p = 5.49e-4 per assertion per run**. Derived here exactly (`Fraction`, cross-checked with `-expm1(N*log1p(-x))`); **the 200k-trial simulation corroborating it came with the originating defect report, not from this filing**; one session reproduced it independently at N=144 windows and another recomputed it analytically. All four agree. But **there are two such assertions** (`:95` and `:303`, both `DOE`) and this repo runs **three OS legs** (`ubuntu` + `windows-2022` + `windows-2025`, one Python version β€” [`ci.yml`](../.github/workflows/ci.yml)), so what an operator actually experiences is: | Scope | Rate | | --- | --- | @@ -8516,10 +8516,10 @@ The `startswith(MARKER_PREFIX)` half stays in every case. **Related:** [`tests/test_store_encryption.py`](../tests/test_store_encryption.py):95, :303, :49–58 (the convention comment, already correct β€” the natural home for the β‰₯6 rule); [`tests/test_content_search.py`](../tests/test_content_search.py):123; CLAUDE.md Β§9 (the PHI-at-rest guarantee the assertion is reaching for); [`Secure_Development_Standards`](Secure_Development_Standards.md) Β§3 (reviewing security prose by what a reader would DO with it β€” the same question, asked of a test instead of a paragraph). -**#344 β€” cited for the harm, NOT the cause; do not fold them.** Both are individually-blameless CI reds that invite the wrong fix, in a repo whose two famous "flakes" turned out to be a livelock and a test that was right. But: **#344 is a fixed bound meeting variable latency; #347 is a deterministic property tested by a probabilistic proxy.** Neither is fixed by changing a number, for opposite reasons. The one-line discriminator, from #344's owner: **this one would fire at exactly the same rate on an infinitely fast machine.** A reader who follows the link lands on a wall-clock item and must not back-infer that this is a timing bug β€” it is not. +**#344 β€” cited for the harm, NOT the cause; do not fold them.** Both are individually-blameless CI reds that invite the wrong fix, in a repo whose two famous "flakes" turned out to be a livelock and a test that was right. But: **#344's thesis is a fixed bound meeting variable latency; #347 is a deterministic property tested by a probabilistic proxy.** (Its *thesis* deliberately β€” that item's instance 2 has since been re-diagnosed as a swallowed lock-timeout rather than a bound at all, so "#344 = timeouts" is not a premise to lean on.) Neither is fixed by changing a number, for opposite reasons. The one-line discriminator, from #344's owner: **this one would fire at exactly the same rate on an infinitely fast machine.** A reader who follows the link lands on a wall-clock item and must not back-infer that this is a timing bug β€” it is not. **#346 β€” the closer sibling.** Same defect class stated generally: *an assertion that passes for a reason unrelated to the property it claims to test.* This one passes because random base64 usually lacks a 3-character run; #346's would have passed because nothing walks the imports. Both are green signals that are not evidence. **ADR 0158 β€” the taxonomy, and where this item sits in it.** Cited by **rule**, not just by number, because the rule is what transfers: *"An equality check satisfiable by coincidence is not an equality check."* That is this defect exactly. By the ADR's own one-line test for **Class 2** β€” *a control that cannot observe or act on its own failure*: **if this control were broken, what would tell me?** If the encryption were replaced tomorrow with a weak encoding, `"DOE" not in raw` would still go green. The answer is the control, which is the defect. *(Deliberately unlinked: the ADR is on PR #145's branch and not yet on `main`, so a relative link would render broken. Guessing its filename from its title is the same failure mode this item is about β€” it was guessed, checked, and was wrong.)* **Follow-up, deliberately not done here:** file this against ADR 0158 **once 0158 is on `main`**. Padding a document at merge time with instances its author did not choose is its own defect, and the ADR's instances are attributed by convention. -**Source:** PR #142 (BACKLOG #323 layer 3, SMTP TLS), 2026-08-02 β€” observed on that PR's CI and deliberately **not** fixed there, because it is unrelated to the SMTP change and widening the PR would have obscured it. Diagnosis, rates and sibling audit produced independently by three sessions and reconciled; the instrument-first framing, the β‰₯6 rule and the leave-the-rest-alone scoping come from the review by the session that owns #142, the #344 discriminator from #344's owner. **Verification of this filing's own instruments:** every probability recomputed by two methods that agree (`Fraction` and `-expm1(N*log1p(-x))`), the audit counts re-derived from the working tree rather than quoted, and `backlog_status_check.py` **falsified against this item** β€” a deliberately doubled banner made it fail at `BACKLOG.md:8429` naming #347, so its green is evidence that it can see this item rather than evidence it skipped it. +**Source:** PR #142 (BACKLOG #323 layer 3, SMTP TLS), 2026-08-02 β€” observed on that PR's CI and deliberately **not** fixed there, because it is unrelated to the SMTP change and widening the PR would have obscured it. **Provenance is itemised, not aggregated** β€” "produced by N sessions" is a confidence claim, and an unsourced one of exactly that shape is what this item is about. **Rates:** derived here exactly, reproduced independently by the #142 session at N=144, recomputed analytically by #344's owner; the 200k-trial simulation came with the originating report. **Sibling audit:** derived twice from different scopes and reconciled. **Instrument-first framing, the β‰₯6 rule, the leave-the-rest-alone scoping:** from the #142 session's review. **The "infinitely fast machine" discriminator:** from #344's owner. **The demand to falsify the banner gate before trusting its green:** from the #346 session. No claim here rests on a count of who agreed. **Verification of this filing's own instruments:** every probability recomputed by two methods that agree (`Fraction` and `-expm1(N*log1p(-x))`), the audit counts re-derived from the working tree rather than quoted, and `backlog_status_check.py` **falsified against this item** β€” a deliberately doubled banner made it fail at `BACKLOG.md:8429` naming #347, so its green is evidence that it can see this item rather than evidence it skipped it. From 38c90b9bba171962ce988c4e6c15c7c9b249c1e9 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sun, 2 Aug 2026 11:05:55 -0500 Subject: [PATCH 7/7] backlog(#347): require the replacement assertion to be falsified before it is trusted The item told an implementer HOW to fix the assertion but not how to know the fix works. Shipping the replacement on an unfalsified green would reproduce the defect inside the remedy -- a green taken as evidence for a property it cannot see is the whole item. So the fix direction now closes by requiring the deliberate break: hand the store an IdentityCipher or plant a plaintext body, watch the rewritten test go RED, then restore. With the trap that makes it more than a formality, from the session that settled #344's instance 2 today: proving the INSTRUMENT can fire is only half -- the WORKLOAD must also be able to produce the failure class. Their 800-iteration repro loop returned 800/800 green against a live SQL Server while hunting a lock- contention bug, because running the two tests in isolation was the one configuration that could not generate contention. They had falsified the probe and not the rig, which felt like all of it. A rig that excludes the condition it hunts reports silence, and silence reads like evidence. Merged origin/main first (PR #148 / BACKLOG #346 landed): clean auto-merge, no conflict this time, verified by CONTENT and not only by count -- 271 items, #345, #346 and #347 all present, and all five of #347's late revisions still resolving in the merged file. --- docs/BACKLOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index e17bf92d..9309ac53 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -8534,6 +8534,8 @@ The failure is also **self-concealing in the dangerous direction.** Age-stalenes The `startswith(MARKER_PREFIX)` half stays in every case. +**Whichever is chosen, prove the new assertion can FAIL before trusting that it passes** β€” break the encryption deliberately (hand the store an `IdentityCipher`, or plant a plaintext body) and watch the rewritten test go red, then restore. This item exists because a green was taken as evidence for a property it could not see; shipping its replacement on an unfalsified green would reproduce the defect in the fix. Note the trap that makes this more than a formality, learned the hard way elsewhere in this repo today: proving the *instrument* can fire is only half β€” the *workload* must also be able to produce the failure class. An 800-iteration repro loop returned 800/800 green against a live SQL Server while hunting a lock-contention bug, because running the tests in isolation was the one configuration that could not generate contention. A rig that excludes the condition it is hunting reports silence, and silence reads like evidence. + **Related:** [`tests/test_store_encryption.py`](../tests/test_store_encryption.py):95, :303, :49–58 (the convention comment, already correct β€” the natural home for the β‰₯6 rule); [`tests/test_content_search.py`](../tests/test_content_search.py):123; CLAUDE.md Β§9 (the PHI-at-rest guarantee the assertion is reaching for); [`Secure_Development_Standards`](Secure_Development_Standards.md) Β§3 (reviewing security prose by what a reader would DO with it β€” the same question, asked of a test instead of a paragraph). **#344 β€” cited for the harm, NOT the cause; do not fold them.** Both are individually-blameless CI reds that invite the wrong fix, in a repo whose two famous "flakes" turned out to be a livelock and a test that was right. But: **#344's thesis is a fixed bound meeting variable latency; #347 is a deterministic property tested by a probabilistic proxy.** (Its *thesis* deliberately β€” that item's instance 2 has since been re-diagnosed as a swallowed lock-timeout rather than a bound at all, so "#344 = timeouts" is not a premise to lean on.) Neither is fixed by changing a number, for opposite reasons. The one-line discriminator, from #344's owner: **this one would fire at exactly the same rate on an infinitely fast machine.** A reader who follows the link lands on a wall-clock item and must not back-infer that this is a timing bug β€” it is not.