From 5d33cbc52a481740df5641553cb5784cca44a34c Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 15:13:54 -0600 Subject: [PATCH 01/41] rotate-key opens its record through an open_store hook; serve refuses a key change the audit chain cannot follow Owner rulings of 2026-11-04 on engine PR 2745, comment 6627309111. BACKLOG #4936 step 3 ("Build the open_store hook"): open_store takes before_first_write, run on every writable open right after the store id pin check and before the salt bind, cipher reserve, at-rest sweep and any audit row, on SQLite, PostgreSQL and SQL Server. rotate-key opens its rotation record there, on every store, so a run killed inside the open leaves the record, and the next start stamps nothing on a store with no keyed chain too. BACKLOG #4934 step 3 ("Refuse, with override"): serve refuses to start, before any audit row, when the store key changed, the keyed chain has not rolled to it, and the chain's current range key is not configured. [secret_rotation].allow_lost_audit_range_key = true is the override, named as a security loosening. Proposed PR title: rotate-key opens its record through an open_store hook, and serve refuses a key change the audit chain cannot follow (BACKLOG #4934, #4936) Proposed banners: #4934: SHIPPED -- step 3 built: serve refuses on a changed key, an unrolled keyed chain and an unconfigured current range key, with [secret_rotation].allow_lost_audit_range_key as the named-loosening override; both tested through the serve lifespan. #4936: SHIPPED -- step 3 built: the rotation record opens in an open_store before_first_write hook, before any write under the key, on every store; a kill after the hook leaves no stamp on a store with no keyed chain, tested. --- changelog.d/4934.security.md | 15 + docs/CONFIGURATION.md | 1 + docs/SECURITY-LOOSENING.md | 4 +- messagefoundry/__main__.py | 31 +- messagefoundry/api/app.py | 20 ++ messagefoundry/config/settings.py | 18 ++ messagefoundry/pipeline/secret_rotation.py | 54 ++++ messagefoundry/store/base.py | 20 +- messagefoundry/store/postgres.py | 2 + messagefoundry/store/sqlserver.py | 2 + messagefoundry/store/store.py | 45 +++ scripts/security/crypto_inventory_check.py | 4 + ...rotate_key_open_hook_and_lost_range_key.py | 292 ++++++++++++++++++ tests/test_rotate_key_rotation_record.py | 5 +- 14 files changed, 499 insertions(+), 14 deletions(-) create mode 100644 changelog.d/4934.security.md create mode 100644 tests/test_rotate_key_open_hook_and_lost_range_key.py diff --git a/changelog.d/4934.security.md b/changelog.d/4934.security.md new file mode 100644 index 0000000000..8c24a9cc4a --- /dev/null +++ b/changelog.d/4934.security.md @@ -0,0 +1,15 @@ +- **`rotate-key` opens its rotation record inside the store open, before the open's first write + under the key, on every store.** `open_store` takes a `before_first_write` hook, run on every + writable open right after the store id pin check and before the salt bind, the cipher block + reserve, the at-rest sweep and any audit row; a read-only open does not run it. A run killed + anywhere after the hook leaves the record open, so the next `serve` start stamps nothing, on a + store with no keyed audit chain too. The open's schema step runs before the hook, so a kill + there leaves no record. + **`serve` refuses to start when the store key differs from its stamp, or has none, the keyed + audit chain has not been rolled to it, and the key the chain's current range is under is not + configured.** That start's first audit row would break the chain. The refusal says to put the + prior key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. + `[secret_rotation].allow_lost_audit_range_key = true` starts anyway, for a prior key that is + truly lost; it logs the refusal as a WARNING, the chain breaks at that start's first audit row, + and it is reported as a security loosening. + (`BACKLOG #4934`, `BACKLOG #4936`) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index b6fc98cde1..35c71f5240 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -3005,6 +3005,7 @@ tracked; set `warn_days = 0` to disable the reminder. | `secret_max_age_days` | int | 365 | max age for the **non-DEK** tracked secret classes (connector/AD/SMTP/Vault/OIDC), alerted this many days after their last observed fingerprint change | | `enforce_grace_days` | int | 30 | under `[security].enforcement=enforce`, a DEK older than `store_key_max_age_days + this` escalates its rotation alert **and refuses to start** (see `enforce_store_key_expiry`) | | `enforce_store_key_expiry` | bool | `true` | under `[security].enforcement=enforce`, a store DEK past `store_key_max_age_days + enforce_grace_days` — or one whose age cannot be determined — **aborts engine start**. `false` keeps the alert, drops the refusal, and is reported as a **security loosening** | +| `allow_lost_audit_range_key` | bool | `false` | `serve` refuses to start when the store key differs from its stamp (or has none), the keyed audit chain has not been rolled to it, and the key the chain's current range is under is not configured: put the prior key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. `true` starts anyway, for a prior key that is truly lost; the chain breaks at that start's first audit row, and it is reported as a **security loosening** | | `enforce_secret_expiry_classes` | list | `[]` | non-DEK classes whose calendar expiry **aborts engine start** under `[security].enforcement=enforce`, past `secret_max_age_days + enforce_grace_days` or with no recorded age. Entries are the `MEFOR_*` class names above or `connector`; an unknown name is refused at load. Empty keeps every non-DEK class alert-only | ```toml diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index a62abe6237..5f4273f1c0 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -242,6 +242,7 @@ Earlier versions of this guide called this an open owner question. The 2026-10-2 | | `[approvals].operations` | `["connection_purge", "dead_letter_replay"]` (*conditional* — a loosening only while `[approvals].enabled` is on; a list that leaves out either default operation is named, and so is an empty list, which holds nothing; named as `approvals.operations`) | | | `[api].trusted_proxies` | `[]` (any network covering more than one address is named, once, a `/31` or `/127` included, because every host inside it may set `X-Forwarded-For`. Entries covering every address, such as `0.0.0.0/0` or `::/0`, trust it from every peer, as the refused `*` would, and get their own wording) | | | `[secret_rotation].enforce_store_key_expiry` | `true` (a calendar-overdue store DEK refuses to start) | +| | `[secret_rotation].allow_lost_audit_range_key` | `false` (`true` starts the engine on a changed store key the keyed audit chain has not rolled to while the key of the chain's current range is not configured; the chain breaks at that start's first audit row) | | | `[cert_monitor].warn_days`, `[secret_rotation].warn_days` | `30` / `14` (`0` turns that credential reminder off, and a smaller value makes it later; named as `cert_monitor.warn_days` and `secret_rotation.warn_days`) | | | `[[alerts.rules]]` | `[]` (a rule that can send a credential reminder to no transport: `mute = true`, `transports = []`, or an `escalate` tier with `transports = []`; named as `alerts.rules`) | | | `[alerts].security_signals`, its five detector counts and `security_window_seconds` | `true`, each count's default, `300` (`false` turns the security-signal layer off and is named as `alerts.security_signals`; with the layer on, a count at `0` turns that one detector off, a count above its default needs a larger burst, a shorter window lets a burst spread past it raise nothing, and each is named as `alerts.`, the window only while any count is above `0`) | @@ -276,7 +277,8 @@ keys, the three named `[auth].phi_read_rate_limit_*` keys, the four `[auth].admi `[auth].mfa_recovery_code_count`, `[auth].password_min_length`, the four other `[auth].ad_session_*` reconciler keys, `[auth].ad_reconcile_outage_passes`, the five `[auth].oidc_*` lifetime and skew keys, `[approvals].min_dwell_seconds`, `[approvals].expiry_hours`, `[approvals].operations`, -`[secret_rotation].enforce_store_key_expiry`, `[secret_rotation].warn_days`, +`[secret_rotation].enforce_store_key_expiry`, `[secret_rotation].allow_lost_audit_range_key`, +`[secret_rotation].warn_days`, `[cert_monitor].warn_days`, `[[alerts.rules]]`, `[alerts].security_signals`, its counts and `security_window_seconds`, `[api].trusted_proxies`, `[api].plaintext_upstream_hop_acknowledged`, `[api].expose_docs`, `[backup].allow_unencrypted`, diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index 9eb6e7e6e1..3f4a3a294a 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -11695,8 +11695,9 @@ def _rotate_key(args: argparse.Namespace) -> int: own fingerprint (``active_key_id``, stored scheme-tagged, BACKLOG #3421), never the sub-key (ADR 0196 AC-7). - **What the order of the steps guarantees (BACKLOG #3737).** The rotation record opens before - either data pass, the audit range rolls after both, and the stamp and the record's close come + **What the order of the steps guarantees (BACKLOG #3737).** The rotation record opens inside + the store open, before its first write under the key, on every store (BACKLOG #4936), so before + either data pass too; the audit range rolls after both, and the stamp and the record's close come last, on a run with no skip, an uploads directory both its pre-check and its uploads pass could list (BACKLOG #4940; a listable directory that is not the one holding the uploads passes both, which docs/PHI.md states) and a good roll. One it cannot list at the start refuses before @@ -11854,16 +11855,26 @@ async def run() -> tuple[int, ResealResult, tuple[bool, str], bool, ResealOutcom # A key the open cannot resolve, or a store hop the posture refuses, exits 2 as one at # resolve_active_key above does. vault_transit mode never gets here (BACKLOG #2109). _warn_on_an_unrevoked_store_hop(settings) # serve's WARNING at warn (#2532) - store = await _open_store_or_refuse(open_store(settings.store, posture=posture)) + today = datetime.datetime.now(tz=datetime.UTC).date().isoformat() + # Whether the record names this key: a run with the key already in use opens none. + record_named = False + + async def open_record(opening: object) -> None: + # BACKLOG #3737, #4936: inside the store open, before its first write under the key, + # on every store, so a kill anywhere after it, the open's own writes and either data + # pass included, leaves the record open and no serve start stamps this key. + nonlocal record_named + # Every backend's store has cipher_info(); the hook is typed for any of them. + info = getattr(opening, "cipher_info", None) + key = info().active_key_id if callable(info) else None + if isinstance(opening, SecretRotationMetaStore) and isinstance(key, str) and key: + record_named = await open_rotation_record(opening, key, today) + + store = await _open_store_or_refuse( + open_store(settings.store, posture=posture, before_first_write=open_record) + ) try: - today = datetime.datetime.now(tz=datetime.UTC).date().isoformat() key_id = store.cipher_info().active_key_id - # BACKLOG #3737: before either data pass, so a kill or an abort in one leaves it open too. - # The store open above can already have written under this key. - # Whether it names this key: a run with the key already in use opens none. - record_named = False - if isinstance(store, SecretRotationMetaStore) and key_id: - record_named = await open_rotation_record(store, key_id, today) count = await store.reencrypt_to_active() # Vault BACKLOG #5635: what the store pass left under a key it holds and could not # re-seal, for the note below. A record under a key it does not hold aborted the pass. diff --git a/messagefoundry/api/app.py b/messagefoundry/api/app.py index f7aea6292f..5e8d53b874 100644 --- a/messagefoundry/api/app.py +++ b/messagefoundry/api/app.py @@ -12652,6 +12652,26 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]: # store; left open, aiosqlite's non-daemon worker holds the process at exit (#1257). The # engine has not started anywhere in this span, so the unwind never calls engine.stop(). try: + # BACKLOG #4934 (owner ruling 2026-11-04): first, before anything below writes an audit + # row, refuse a changed store key the keyed audit chain cannot follow. The open settled + # the chain; the only I/O is the stamp read. + from messagefoundry.pipeline.secret_rotation import refuse_unrolled_key_change + from messagefoundry.store.store import ( + SecretRotationMetaStore, + audit_chain_rolled, + audit_range_key_missing, + ) + + if isinstance(store, SecretRotationMetaStore): + await refuse_unrolled_key_change( + store, + dek_key_id=store.cipher_info().active_key_id, + audit_rolled=audit_chain_rolled(store), + range_key_missing=audit_range_key_missing(store), + allow_lost=( + secret_rotation_settings or SecretRotationSettings() + ).allow_lost_audit_range_key, + ) # Offline uploaded-logs store (BACKLOG #125/#126, ADR 0134), on the LIVE store's cipher instance. # DISABLED (None) unless [store].uploads_dir is set, so no PHI-at-rest surface exists unless an # operator opts in; every uploaded-logs route 503s when None. diff --git a/messagefoundry/config/settings.py b/messagefoundry/config/settings.py index b83bc7ae7f..0e1ae7fb6f 100644 --- a/messagefoundry/config/settings.py +++ b/messagefoundry/config/settings.py @@ -6852,6 +6852,12 @@ class SecretRotationSettings(_Section): # build would buy the setting without the posture. Setting it false is a LOOSENING and # security_loosenings() names it, so the opt-out is never silent. enforce_store_key_expiry: bool = True + # BACKLOG #4934 — serve refuses to start when the store key changed, the keyed audit chain has + # not rolled to it, and the key the chain's current range is under is not configured: the first + # audit row that start wrote would break the chain, and no later rotate-key could roll it. True + # starts anyway, for a prior key that is truly lost; it is a LOOSENING, named by + # security_loosenings(), and the chain breaks at that start's first audit row. + allow_lost_audit_range_key: bool = False # ASVS 13.3.4 / BACKLOG #1932 — the calendar refusal for the NON-DEK classes, OPT-IN per class. # Each entry names one class from ENFORCEABLE_SECRET_EXPIRY_CLASSES. Under # [security].enforcement=ENFORCE, a named class the engine holds that is past @@ -10666,6 +10672,18 @@ def security_loosenings( "key's 2**32-encrypt usage ceiling still refuses unconditionally)", ) ) + # BACKLOG #4934: named whatever the store holds, as enforce_store_key_expiry is, because the + # setting says what the next start may do, not what it found. + if secret_rotation.allow_lost_audit_range_key: + out.append( + ( + "allow_lost_audit_range_key", + "the engine starts even when the store key changed, the audit chain has not been " + "rolled to it and the key its current range is under is not configured, so the " + "audit chain breaks at that start's first audit row and no rotate-key run can roll " + "it afterwards", + ) + ) # --- the credential reminders (ASVS 6.4.5; BACKLOG #2227 and #2008 step 4). Owner answer to # #2006 (a): a silent weakening keeps the cell at partial. _reminder_loosenings says what each # entry covers. diff --git a/messagefoundry/pipeline/secret_rotation.py b/messagefoundry/pipeline/secret_rotation.py index 2b85357269..ef3a97c79c 100644 --- a/messagefoundry/pipeline/secret_rotation.py +++ b/messagefoundry/pipeline/secret_rotation.py @@ -84,6 +84,7 @@ "SecretRotationRunner", "SecretStamp", "StoreKeyRotationOverdueError", + "UnrolledStoreKeyRefused", "close_rotation_record", "dek_changed", "dek_fingerprint", @@ -93,6 +94,7 @@ "fingerprints_equal", "open_rotation_record", "reconcile_rotation_meta", + "refuse_unrolled_key_change", "secrets_from_settings", "secrets_from_settings_and_stamps", ] @@ -393,6 +395,58 @@ def _rotation_hold(record: SecretRotationMetaRow | None, fingerprint: str) -> st return None +class UnrolledStoreKeyRefused(RuntimeError): + """``serve`` refused to start on a changed store key the keyed audit chain cannot follow + (BACKLOG #4934); see :func:`refuse_unrolled_key_change`.""" + + +#: The refusal :func:`refuse_unrolled_key_change` raises, and the WARNING it logs under the override. +_UNROLLED_REFUSAL = ( + "the store key in use differs from its stamp, or has none, the audit chain has not been " + "rolled to it, and the key the chain's current range is under is not configured. This start's " + "first audit row would go into that range under another key, and the chain would break there, " + "so no later `messagefoundry rotate-key` could roll it. Put the prior store key back in " + "MEFOR_STORE_ENCRYPTION_KEYS_RETIRED, keep the new key active, and run `messagefoundry " + "rotate-key` until it exits 0, then start. Only where the prior key is truly lost, set " + "[secret_rotation].allow_lost_audit_range_key = true to start anyway: that is a security " + "loosening, and the chain breaks at this start's first audit row." +) + + +async def refuse_unrolled_key_change( + meta_store: SecretRotationMetaStore, + *, + dek_key_id: str | None, + audit_rolled: bool | None, + range_key_missing: bool, + allow_lost: bool, +) -> None: + """Raise :class:`UnrolledStoreKeyRefused` when the store key changed, the keyed audit chain has + not rolled to it, and the key the chain's current range is under is not configured (BACKLOG + #4934, owner ruling 2026-11-04). ``serve`` calls it right after the store open, before it writes + any audit row. ``audit_rolled`` is :func:`~messagefoundry.store.store.audit_chain_rolled` and + ``range_key_missing`` is :func:`~messagefoundry.store.store.audit_range_key_missing`, both for + the open store. + + The key counts as changed as :func:`dek_changed` says, so a store with no stamp, or a stamp + under a scheme this release does not know, counts too. A stamp naming the key in use does not + refuse, even over such a chain. ``allow_lost``, ``[secret_rotation].allow_lost_audit_range_key``, + logs the refusal as a WARNING and returns instead of raising.""" + if not dek_key_id or audit_rolled is not False or not range_key_missing: + return + prior = (await meta_store.get_secret_rotation_meta()).get(_DEK_CLASS_ID) + if dek_changed(prior, dek_fingerprint(dek_key_id)) is False: + return + if allow_lost: + log.warning( + "secret rotation: [secret_rotation].allow_lost_audit_range_key is set, so the engine " + "starts over this refusal: %s", + _UNROLLED_REFUSAL, + ) + return + raise UnrolledStoreKeyRefused(f"refusing to start: {_UNROLLED_REFUSAL}") + + #: Bytes of the per-row random HMAC key a sealed fingerprint cell holds (BACKLOG #3472). _CELL_KEY_BYTES = 32 _LOWER_HEX = "0123456789abcdef" diff --git a/messagefoundry/store/base.py b/messagefoundry/store/base.py index 4605c5a5d4..1dab2802bd 100644 --- a/messagefoundry/store/base.py +++ b/messagefoundry/store/base.py @@ -28,7 +28,15 @@ import sys import threading import weakref -from collections.abc import AsyncIterator, Callable, Collection, Iterable, Mapping, Sequence +from collections.abc import ( + AsyncIterator, + Awaitable, + Callable, + Collection, + Iterable, + Mapping, + Sequence, +) from dataclasses import dataclass from functools import cache, partial from pathlib import Path @@ -80,6 +88,7 @@ from messagefoundry.store.row_seal import AnyRowSealer, ResealOutcome from messagefoundry.store.store import ( ANY_CLAIM_OWNER, + OPEN_STORE_HOOK, OPEN_STORE_PIN, OPEN_STORE_PIN_CHECK, UPLOAD_RESERVATION_STALE_AFTER, @@ -4766,6 +4775,7 @@ async def open_store( refusal_hook: UnmarkedRefusalHook | None = None, custody: HostBlob | None = None, recovery_wrap: bytes = b"", + before_first_write: Callable[[Any], Awaitable[None]] | None = None, ) -> Store: """Open the store for the configured backend — the single backend-selection seam. @@ -4849,6 +4859,12 @@ async def open_store( Every custody refusal is a :class:`~messagefoundry.store.keyprovider.KeyProviderError`, with a fixed text and no cause or context; :func:`_host_custody_for_open` says which errors on the custody path are replaced (BACKLOG #4882). + + ``before_first_write`` (BACKLOG #4936) is awaited with the half-opened backend store on every + writable open, right after the pin check: the schema is in place, and nothing has yet bound a + salt, reserved a cipher block, sealed a row or written an audit row. ``rotate-key`` opens its + rotation record there. The schema step before it can already have run its DDL or a migration. + A read-only open does not run it. What it raises fails the open, with the handle closed. """ if create and read_only: raise ValueError("open_store: create and read_only exclude each other") @@ -4901,6 +4917,7 @@ async def open_store( # The batch failures a backend typed itself, and the pin refusals, are re-raised as they are. check = PinCheck() if minting_open() and settings.backend is not StoreBackend.SQLITE else None checking = OPEN_STORE_PIN_CHECK.set(check) + hooked = OPEN_STORE_HOOK.set(before_first_write) try: store = await _open_backend( settings, @@ -4917,6 +4934,7 @@ async def open_store( raise raise StoreCreateBatchFailedBeforeKeyUse(exc) from exc finally: + OPEN_STORE_HOOK.reset(hooked) OPEN_STORE_PIN_CHECK.reset(checking) OPEN_STORE_PIN.reset(pinned) if keyless_chain_refusal is not None and audit_mac_key is None and audit_mac_fn is None: diff --git a/messagefoundry/store/postgres.py b/messagefoundry/store/postgres.py index accf59f283..1cf1d0b867 100644 --- a/messagefoundry/store/postgres.py +++ b/messagefoundry/store/postgres.py @@ -388,6 +388,7 @@ roles_if_held, roll_audit_key_range, rotation_reseal, + run_open_store_hook, scope_if_source, scope_withdrawal, seal_approval_async, @@ -2214,6 +2215,7 @@ async def open( await store._ensure_schema(create=create, read_only=read_only) if not read_only: await check_open_store_pin(store) # vault BACKLOG #3411: before any write + await run_open_store_hook(store) # BACKLOG #4936: after the pin, before any write # ADR 0196: bind this store's salt BEFORE the first checkpoint, so the block # reserved next and every value sealed after it land under this store's own data # sub-key. A read-only handle seals nothing; every stored value names its own salt. diff --git a/messagefoundry/store/sqlserver.py b/messagefoundry/store/sqlserver.py index 02a953801a..0e989ed95f 100644 --- a/messagefoundry/store/sqlserver.py +++ b/messagefoundry/store/sqlserver.py @@ -396,6 +396,7 @@ roles_if_held, roll_audit_key_range, rotation_reseal, + run_open_store_hook, scope_if_source, scope_withdrawal, seal_approval_async, @@ -4294,6 +4295,7 @@ async def open( await store._gate_claim_prepared() if not read_only: await check_open_store_pin(store) # vault BACKLOG #3411: before any write + await run_open_store_hook(store) # BACKLOG #4936: after the pin, before any write # ADR 0196: bind this store's salt BEFORE the first checkpoint, so the block # reserved next and every value sealed after it land under this store's own data # sub-key. A read-only handle seals nothing; every stored value names its own salt. diff --git a/messagefoundry/store/store.py b/messagefoundry/store/store.py index 7c5c3ae4ab..144d8c5812 100644 --- a/messagefoundry/store/store.py +++ b/messagefoundry/store/store.py @@ -6890,6 +6890,24 @@ def __init__(self, failure: Exception) -> None: #: construction; ``None`` for every open that is not under the ``host`` key provider. OPEN_STORE_PIN: ContextVar[StorePin | None] = ContextVar("mefor_open_store_pin", default=None) +#: The ``before_first_write`` hook of the store open in progress, set by ``store.base.open_store`` +#: around the backend open and run by :func:`run_open_store_hook` (BACKLOG #4936). A context +#: variable for :data:`OPEN_STORE_PIN`'s reason; ``None`` for an open that passes no hook. +OPEN_STORE_HOOK: ContextVar[Callable[[Any], Awaitable[None]] | None] = ContextVar( + "mefor_open_store_hook", default=None +) + + +async def run_open_store_hook(host: object) -> None: + """Run the open's ``before_first_write`` hook on the half-opened ``host`` (BACKLOG #4936). Each + backend's writable open calls this right after :func:`check_open_store_pin`, so once the schema + is in place and before it binds a salt, reserves a cipher block, seals a row or writes an audit + row. A read-only open never calls it. A no-op when no hook is set. What the hook raises fails + the open, which closes the handle.""" + hook = OPEN_STORE_HOOK.get() + if hook is not None: + await hook(host) + class PinCheck: """Whether the open in progress has passed its store id pin check (:func:`check_open_store_pin`), @@ -8107,6 +8125,32 @@ def audit_chain_rolled(store: object) -> bool | None: return bool(isinstance(keys, list) and keys and keys[-1] == active) +def audit_range_key_missing(store: object) -> bool: + """Whether the keyed audit chain's current range, as the open authenticated it, is keyed under + an audit key this store does not hold (BACKLOG #4934). The open then moves new rows to the + active key (:func:`settle_audit_ranges`), so the first row a start writes lands in a range keyed + under another key, and the chain breaks there; :func:`roll_audit_key_range` refuses the same + state. ``False`` with no keyed chain, and for ranges the open could not authenticate, which + say nothing about the current range's key. Read by attribute, as :func:`audit_chain_rolled` + reads; no I/O.""" + if not getattr(store, "_audit_chain_keyed", False): + return False + if not getattr(store, "_audit_ranges_trusted", False): + return False + keys = getattr(store, "_audit_range_keys", None) + if not (isinstance(keys, list) and keys): + return False + mac_keys = getattr(store, "_audit_mac_keys", None) + return ( + _audit_secret_for( + keys[-1], + mac_keys if isinstance(mac_keys, Mapping) else {}, + getattr(store, "_audit_mac_fn", None), + ) + is None + ) + + # #233 (ADR 0111): the event name for a Send declined because its target outbound is present-but-NOT- # deployed. Recorded once per declined destination (:func:`not_deployed_detail` for the detail). It has # TWO load-bearing roles, and both fail unless it is in the audit floor below: (1) it is the operator- @@ -11568,6 +11612,7 @@ async def open( # has not bound, a failure before the bind included (BACKLOG #5720). store._cipher_foreign = True await check_open_store_pin(store) # vault BACKLOG #3411: before any write + await run_open_store_hook(store) # BACKLOG #4936: after the pin, before any write # ADR 0196: bind this store's salt BEFORE the first checkpoint, so the block # reserved next and every value sealed after it land under this store's own data # sub-key. A read-only handle seals nothing; every stored value names its own salt. diff --git a/scripts/security/crypto_inventory_check.py b/scripts/security/crypto_inventory_check.py index ab0e727636..6be2730d53 100644 --- a/scripts/security/crypto_inventory_check.py +++ b/scripts/security/crypto_inventory_check.py @@ -1138,6 +1138,10 @@ "compare:via messagefoundry.config.fingerprint", "hash:via messagefoundry.auth.trust_anchors", "tls_context:via messagefoundry.config.tls_policy", + # BACKLOG #4934: the start refusal compares the DEK key id with its stamp + # (dek_changed), and reads the audit chain's range keys the open settled. + "compare:via messagefoundry.pipeline.secret_rotation", + "hash:via messagefoundry.store.store", } ), "messagefoundry/api/security.py": frozenset( diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py new file mode 100644 index 0000000000..bcba1c8852 --- /dev/null +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -0,0 +1,292 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 MessageFoundry Foundation, LLC and contributors +"""BACKLOG #4936 and #4934, the owner rulings of 2026-11-04. + +#4936: ``rotate-key`` opens its rotation record through ``open_store``'s ``before_first_write`` +hook, on every store, so a run killed inside the store open after the hook leaves a record, and the +next start stamps nothing, on a store with no keyed audit chain too. Before the hook the record +opened only after the open returned, and that start stamped the new key. + +#4934: ``serve`` refuses to start when the store key changed, the keyed audit chain has not rolled +to it, and the key the chain's current range is under is not configured, with +``[secret_rotation].allow_lost_audit_range_key`` as the override for a prior key that is truly +lost.""" + +from __future__ import annotations + +import asyncio +import logging +from pathlib import Path +from typing import Any + +import pytest + +from messagefoundry.__main__ import main +from messagefoundry.config.ai_policy import SecurityEnforcement +from messagefoundry.config.settings import SecretRotationSettings, StoreSettings +from messagefoundry.pipeline.secret_rotation import ( + ROTATION_RECORD_ID, + UnrolledStoreKeyRefused, + dek_fingerprint, + reconcile_rotation_meta, + refuse_unrolled_key_change, +) +from messagefoundry.store.base import open_store +from messagefoundry.store.crypto import generate_key +from messagefoundry.store.store import ( + MessageStore, + audit_chain_rolled, + audit_range_key_missing, +) +from tests.test_rotate_key_rotation_record import ( + _DEK, + _OLD_DATE, + _env, + _key_id, + _open, + _rows, + _seed, + _seed_keyless_chain, +) + + +class _Killed(Exception): + """Stands in for a kill inside the store open, after the hook ran.""" + + +def _kill_after_the_hook(monkeypatch: pytest.MonkeyPatch) -> None: + async def killed(*_args: object, **_kwargs: object) -> None: + raise _Killed("killed inside the store open") + + # The salt bind is the open's first write under the key, right after the hook. + monkeypatch.setattr("messagefoundry.store.store.bind_store_salt", killed) + + +async def _reconcile(db: Path, key: str, *retired: str) -> dict[str, Any]: + store = await _open(db, key, *retired) + try: + await reconcile_rotation_meta( + store, + SecretRotationSettings(), + dek_key_id=_key_id(key), + enforcement=SecurityEnforcement.WARN, + env_values={}, + audit_rolled=audit_chain_rolled(store), + ) + return dict(await store.get_secret_rotation_meta()) + finally: + await store.close() + + +def test_a_run_killed_inside_the_open_on_a_store_with_no_keyed_chain_leaves_no_stamp( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """#4936 step 3. On the code before the hook no record existed at the kill, and this start + stamped the new key, because a store with no keyed chain has nothing else to hold it.""" + monkeypatch.chdir(tmp_path) + db, uploads, key_a, key_b = ( + tmp_path / "k.db", + tmp_path / "uploads", + generate_key(), + generate_key(), + ) + asyncio.run(_seed_keyless_chain(db, uploads, key_a)) + _env(monkeypatch, key_b, key_a, uploads) + with monkeypatch.context() as patch: + _kill_after_the_hook(patch) + # run_guarded turns the uncaught kill into a failing exit. + assert main(["rotate-key", "--db", str(db)]) != 0 + capsys.readouterr() + rows = asyncio.run(_reconcile(db, key_b, key_a)) + assert rows[ROTATION_RECORD_ID].fingerprint == dek_fingerprint(_key_id(key_b)) + assert rows[_DEK].fingerprint == dek_fingerprint(_key_id(key_a)) + assert rows[_DEK].last_rotated == _OLD_DATE + + +def test_the_hook_runs_before_the_first_write_and_not_on_a_read_only_open(tmp_path: Path) -> None: + """The hook sees the store before the salt bind and the at-rest sweep, and a read-only open + never runs it.""" + db, key = tmp_path / "h.db", generate_key() + asyncio.run(_seed(db, tmp_path / "uploads", key)) + seen: list[str] = [] + + async def hook(store: object) -> None: + assert isinstance(store, MessageStore) + assert store._cipher_foreign # the salt is not bound yet + seen.append("hook") + + async def opened(read_only: bool) -> None: + settings = StoreSettings(path=str(db), encryption_key=key) + store = await open_store( + settings, read_only=read_only, keyless_chain_refusal=None, before_first_write=hook + ) + await store.close() + + asyncio.run(opened(False)) + assert seen == ["hook"] + asyncio.run(opened(True)) + assert seen == ["hook"] + + +async def _refusal(db: Path, key: str, *retired: str, allow_lost: bool = False) -> None: + store = await _open(db, key, *retired) + try: + await refuse_unrolled_key_change( + store, + dek_key_id=_key_id(key), + audit_rolled=audit_chain_rolled(store), + range_key_missing=audit_range_key_missing(store), + allow_lost=allow_lost, + ) + finally: + await store.close() + + +def test_a_start_on_the_new_key_alone_refuses_before_any_audit_row(tmp_path: Path) -> None: + """#4934 step 3: changed key, chain not rolled, current range key not configured.""" + db, key_a, key_b = tmp_path / "r.db", generate_key(), generate_key() + asyncio.run(_seed(db, tmp_path / "uploads", key_a)) + with pytest.raises(UnrolledStoreKeyRefused, match="allow_lost_audit_range_key"): + asyncio.run(_refusal(db, key_b)) + + +@pytest.mark.parametrize("case", ["prior_key_configured", "key_unchanged", "keyless_chain"]) +def test_the_refusal_needs_all_three_conditions(tmp_path: Path, case: str) -> None: + """Each condition alone failing lets the start through.""" + db, key_a, key_b = tmp_path / "c.db", generate_key(), generate_key() + if case == "keyless_chain": + asyncio.run(_seed_keyless_chain(db, tmp_path / "uploads", key_a)) + asyncio.run(_refusal(db, key_b)) + return + asyncio.run(_seed(db, tmp_path / "uploads", key_a)) + if case == "prior_key_configured": + asyncio.run(_refusal(db, key_b, key_a)) + else: + asyncio.run(_refusal(db, key_a)) + + +def test_the_override_starts_with_a_warning_and_is_a_named_loosening( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + db, key_a, key_b = tmp_path / "o.db", generate_key(), generate_key() + asyncio.run(_seed(db, tmp_path / "uploads", key_a)) + with caplog.at_level(logging.WARNING, logger="messagefoundry.pipeline.secret_rotation"): + asyncio.run(_refusal(db, key_b, allow_lost=True)) + assert "allow_lost_audit_range_key is set" in caplog.text + + +def _named(settings: SecretRotationSettings) -> dict[str, str]: + from messagefoundry.config.settings import ( + AiSettings, + AlertsSettings, + ApiSettings, + ApprovalsSettings, + AuthSettings, + BackupSettings, + CertMonitorSettings, + SecretsSettings, + SecuritySettings, + security_loosenings, + ) + + return dict( + security_loosenings( + SecuritySettings(), + StoreSettings(), + AuthSettings(), + AlertsSettings(), + settings, + cleartext_hops=(), + expiry_relaxed_hops=(), + hostname_unchecked_hops=(), + query_credential_hops=(), + unverified_db_hops=(), + attested_hops=(), + revocation_attested_hops=(), + path_form_fhir_hops=(), + ech_cleartext_sidecar_hops=(), + api=ApiSettings(), + approvals=ApprovalsSettings(), + cert_monitor=CertMonitorSettings(), + backup=BackupSettings(), + secrets=SecretsSettings(), + ai=AiSettings(), + store_privilege=None, + audit_chain_unkeyed=None, + remote_debug=None, + startup=None, + service_env=None, + ) + ) + + +def test_the_override_is_a_named_loosening_and_off_by_default() -> None: + assert SecretRotationSettings().allow_lost_audit_range_key is False + assert "allow_lost_audit_range_key" not in _named(SecretRotationSettings()) + risk = _named(SecretRotationSettings(allow_lost_audit_range_key=True))[ + "allow_lost_audit_range_key" + ] + assert "audit chain breaks" in risk + + +def test_the_range_key_answer_tells_the_states_apart(tmp_path: Path) -> None: + keyed, keyless = tmp_path / "keyed.db", tmp_path / "keyless.db" + key_a, key_b = generate_key(), generate_key() + asyncio.run(_seed(keyed, tmp_path / "u1", key_a)) + asyncio.run(_seed_keyless_chain(keyless, tmp_path / "u2", key_a)) + + async def answer(db: Path, key: str, *retired: str) -> bool: + store = await _open(db, key, *retired) + try: + return audit_range_key_missing(store) + finally: + await store.close() + + assert asyncio.run(answer(keyed, key_a)) is False + assert asyncio.run(answer(keyed, key_b, key_a)) is False + assert asyncio.run(answer(keyed, key_b)) is True + assert asyncio.run(answer(keyless, key_b)) is False + assert audit_range_key_missing(object()) is False + + +def test_the_rows_after_a_refused_start_are_unchanged(tmp_path: Path) -> None: + """The refusal writes nothing: the DEK stamp and the record are as the seed left them.""" + db, key_a, key_b = tmp_path / "u.db", generate_key(), generate_key() + asyncio.run(_seed(db, tmp_path / "uploads", key_a)) + with pytest.raises(UnrolledStoreKeyRefused): + asyncio.run(_refusal(db, key_b)) + rows = asyncio.run(_rows(db, key_b, key_a)) + assert rows[_DEK].fingerprint == dek_fingerprint(_key_id(key_a)) + assert ROTATION_RECORD_ID not in rows + + +def _app_on_the_new_key_alone(tmp_path: Path, allow_lost: bool) -> Any: + from messagefoundry.api import create_managed_app + from messagefoundry.config.settings import EgressSettings + + db, key_a, key_b = tmp_path / "serve.db", generate_key(), generate_key() + asyncio.run(_seed(db, tmp_path / "uploads", key_a)) + return create_managed_app( + store_settings=StoreSettings(path=str(db), encryption_key=key_b), + poll_interval=0.05, + # The seed's DEK stamp is old; the calendar refusal is not under test here. + secret_rotation_settings=SecretRotationSettings( + allow_lost_audit_range_key=allow_lost, enforce_store_key_expiry=False + ), + egress_settings=EgressSettings(deny_by_default=False), + ) + + +async def test_serve_refuses_a_start_on_the_new_key_alone(tmp_path: Path) -> None: + """#4934 step 3, the wiring: the refusal aborts the serve lifespan.""" + app = await asyncio.to_thread(_app_on_the_new_key_alone, tmp_path, False) + with pytest.raises(UnrolledStoreKeyRefused): + async with app.router.lifespan_context(app): + pass # pragma: no cover -- startup must not reach here + + +async def test_serve_starts_on_the_new_key_alone_under_the_override(tmp_path: Path) -> None: + """Positive control on the arm above: the same store starts with the override.""" + app = await asyncio.to_thread(_app_on_the_new_key_alone, tmp_path, True) + async with app.router.lifespan_context(app): + assert app.state.engine is not None diff --git a/tests/test_rotate_key_rotation_record.py b/tests/test_rotate_key_rotation_record.py index dc38074555..b26fd1d45f 100644 --- a/tests/test_rotate_key_rotation_record.py +++ b/tests/test_rotate_key_rotation_record.py @@ -345,8 +345,9 @@ def test_a_run_killed_before_it_opened_the_record_leaves_no_stamp( monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], ) -> None: - """BACKLOG #4936. The run stops after its store open and before the record opens, the window a - kill inside the open stands in. No record exists, and the next start must still not stamp.""" + """BACKLOG #4936. The record open itself fails, inside the store open's hook and before any + write under the key. No record exists, and the next start on a keyed chain must still not + stamp.""" db, uploads, key_a, key_b = rig _env(monkeypatch, key_b, key_a, uploads) From b8ab12d685ce977b376e45cabe9cabdb54f7cc1b Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 15:55:16 -0600 Subject: [PATCH 02/41] supervise takes the unrolled-key refusal before its API pair audit row; docs and tests close review round 1 Review round 1 (xhigh, round-e29864a2) on the rotate-key hook and lost-range-key refusal: - supervise opened the store and wrote the renewed API pair's audit row before any shard's lifespan could refuse, so a refused fleet had already broken the chain. The supervisor now runs refuse_unrolled_key_change first, under the same override. - docs/PHI.md still said a serve start on the new key alone keys new rows under it; it now states the refusal, the override, and that the admin commands do not refuse. - docs/SECURITY-LOOSENING.md gets the override's own section; the security_loosenings() docstring registry names it. - The refusal names the key the chain's range is under rather than "the prior key", which can be an older one; CONFIGURATION.md and the changelog fragment match. - Tests: the stamp arm alone, a refused serve writes no audit row, and supervise refuses before its audit row and records under the override. Negative controls: disabling the supervise call or the stamp arm turns those tests red. - crypto inventory: __main__.py reads the audit range keys through store.store. BACKLOG #4934, #4936 --- changelog.d/4934.security.md | 12 +-- docs/CONFIGURATION.md | 2 +- docs/PHI.md | 12 ++- docs/SECURITY-LOOSENING.md | 23 ++++++ messagefoundry/__main__.py | 16 ++++ messagefoundry/config/settings.py | 3 +- messagefoundry/pipeline/secret_rotation.py | 8 +- scripts/security/crypto_inventory_check.py | 3 + ...rotate_key_open_hook_and_lost_range_key.py | 76 +++++++++++++++++++ 9 files changed, 141 insertions(+), 14 deletions(-) diff --git a/changelog.d/4934.security.md b/changelog.d/4934.security.md index 8c24a9cc4a..e801b60c61 100644 --- a/changelog.d/4934.security.md +++ b/changelog.d/4934.security.md @@ -7,9 +7,11 @@ there leaves no record. **`serve` refuses to start when the store key differs from its stamp, or has none, the keyed audit chain has not been rolled to it, and the key the chain's current range is under is not - configured.** That start's first audit row would break the chain. The refusal says to put the - prior key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. - `[secret_rotation].allow_lost_audit_range_key = true` starts anyway, for a prior key that is - truly lost; it logs the refusal as a WARNING, the chain breaks at that start's first audit row, - and it is reported as a security loosening. + configured.** That start's first audit row would break the chain. `supervise` refuses the same + state before it records a renewed API certificate. The refusal says to put the key that range + is under back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. + `[secret_rotation].allow_lost_audit_range_key = true` starts anyway, for a key that is truly + lost; it logs the refusal as a WARNING, the chain breaks at that start's first audit row, and it + is reported as a security loosening. Other commands that open the store and write an audit row, + such as the `admin-*` commands, do not refuse. (`BACKLOG #4934`, `BACKLOG #4936`) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 35c71f5240..3e4a197372 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -3005,7 +3005,7 @@ tracked; set `warn_days = 0` to disable the reminder. | `secret_max_age_days` | int | 365 | max age for the **non-DEK** tracked secret classes (connector/AD/SMTP/Vault/OIDC), alerted this many days after their last observed fingerprint change | | `enforce_grace_days` | int | 30 | under `[security].enforcement=enforce`, a DEK older than `store_key_max_age_days + this` escalates its rotation alert **and refuses to start** (see `enforce_store_key_expiry`) | | `enforce_store_key_expiry` | bool | `true` | under `[security].enforcement=enforce`, a store DEK past `store_key_max_age_days + enforce_grace_days` — or one whose age cannot be determined — **aborts engine start**. `false` keeps the alert, drops the refusal, and is reported as a **security loosening** | -| `allow_lost_audit_range_key` | bool | `false` | `serve` refuses to start when the store key differs from its stamp (or has none), the keyed audit chain has not been rolled to it, and the key the chain's current range is under is not configured: put the prior key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. `true` starts anyway, for a prior key that is truly lost; the chain breaks at that start's first audit row, and it is reported as a **security loosening** | +| `allow_lost_audit_range_key` | bool | `false` | `serve` (and `supervise`, before it records a renewed API certificate) refuses to start when the store key differs from its stamp (or has none), the keyed audit chain has not been rolled to it, and the key the chain's current range is under is not configured: put that key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. `true` starts anyway, for a key that is truly lost; the chain breaks at that start's first audit row, and it is reported as a **security loosening** | | `enforce_secret_expiry_classes` | list | `[]` | non-DEK classes whose calendar expiry **aborts engine start** under `[security].enforcement=enforce`, past `secret_max_age_days + enforce_grace_days` or with no recorded age. Entries are the `MEFOR_*` class names above or `connector`; an unknown name is refused at load. Empty keeps every non-DEK class alert-only | ```toml diff --git a/docs/PHI.md b/docs/PHI.md index 34898c6b0b..2cc273f436 100644 --- a/docs/PHI.md +++ b/docs/PHI.md @@ -770,10 +770,14 @@ for defense-in-depth without swapping the `aiosqlite` connector. store with no keyed chain to ask, such as one whose audit rows open with no genesis row naming its key, so there a stamp proves only a changed key. On such a store, do not start `serve` between setting a new key and running `rotate-key`: that stamp stands however the run ends. On a - store with a keyed chain, do not start `serve` on the new key alone before a run's roll succeeds: - the chain's current range is then under a key it does not hold, so it keys new audit rows under - the new key, those rows never verify inside that range, and no later `rotate-key` can roll the - chain. **A roll that refuses for good holds the key for good (BACKLOG #4934, #4935).** The roll + store with a keyed chain, `serve` refuses to start on a changed key the chain has not rolled to + while the key the chain's current range is under is not configured (BACKLOG #4934): its first + audit row would go into that range under the new key, never verify there, and leave a chain no + later `rotate-key` can roll. Put that key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run + `rotate-key` until it exits 0. `[secret_rotation].allow_lost_audit_range_key = true` starts + anyway, for a key that is truly lost; it is a named security loosening, and the chain breaks at + that start's first audit row. Other commands that open the store and write an audit row, such as + the `admin-*` commands, do not refuse, so do not run them in that state either. **A roll that refuses for good holds the key for good (BACKLOG #4934, #4935).** The roll verifies the whole chain first, so a chain whose rows do not verify, or whose rows open with no genesis row, refuses under every key, a new one included, and the record stays open. A newest range under a key that is not configured is not one of these, though `audit-verify` reports it diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index 5f4273f1c0..f00f6dd626 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -834,6 +834,29 @@ This section is kept rather than deleted, because the claim it used to make is t - **Reversible:** yes, immediately — set it back to `true` (or delete the line) and restart. Nothing about the key or the store changes either way; only whether the engine agrees to start. +### `[secret_rotation].allow_lost_audit_range_key = true` — the engine starts on a key change the audit chain cannot follow +Named as `allow_lost_audit_range_key` (BACKLOG #4934), whatever the store holds, because it says what +the next start may do. +- **What you lose:** the start refusal that keeps the keyed audit chain whole across a store key + change. With it off, `serve` (and each `supervise` shard, and the supervisor before it records a + renewed API certificate) refuses to start when the store key differs from its stamp, or has none, + the chain has not been rolled to it, and the key the chain's current range is under is not + configured. With it on, that start goes ahead: its first audit row goes into that range under the + new key, never verifies there, and the chain breaks at that row. `audit-verify` reports the break, + and `rotate-key` cannot roll the chain afterwards. +- **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, + and every other start check. Rows before the break still verify. The setting changes nothing on a + start whose key is unchanged, whose chain has rolled, or whose range key is configured. +- **When acceptable:** only when the key the chain's current range is under is truly lost, so no + `rotate-key` run can roll the chain anyway. If the key exists anywhere, put it back in + `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0 instead. +- **Compensating controls:** none that restore the chain. Record when and why the key was lost, keep + the `audit-verify` output from before the start, and treat the break as a known event in any audit + review. +- **Reversible:** the setting is, the break is not. The chain stays broken at that row, and turning + the setting back off makes the next start refuse again, because the chain's range is still under + the lost key. + ### `[cert_monitor].warn_days` or `[secret_rotation].warn_days` at `0`, or below its default — a credential reminder is off or late Named as `cert_monitor.warn_days` and `secret_rotation.warn_days` (ASVS 6.4.5, BACKLOG #2227). A value below the shipped default (`30` and `14` days) is named too, as the sign-in limits are when looser diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index 3f4a3a294a..49c34ec868 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -5238,7 +5238,13 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P from messagefoundry.config.anchor import anchored_store_path, resolve_project_root from messagefoundry.config.settings import hop_posture_from_ai, keyless_opt_out_refusal from messagefoundry.last_resort import run_guarded + from messagefoundry.pipeline.secret_rotation import refuse_unrolled_key_change from messagefoundry.store import open_store + from messagefoundry.store.store import ( + SecretRotationMetaStore, + audit_chain_rolled, + audit_range_key_missing, + ) cwd = Path.cwd() root = resolve_project_root(settings.environments.base_dir or None, cwd=cwd) @@ -5267,6 +5273,16 @@ async def _audit() -> None: keyless_chain_refusal=keyless_opt_out_refusal(shard_store, settings.security), ) try: + # BACKLOG #4934: the audit row below is this store's first since the open, so it takes + # the shards' start refusal first, or a refused start would already have broken the chain. + if isinstance(store, SecretRotationMetaStore): + await refuse_unrolled_key_change( + store, + dek_key_id=store.cipher_info().active_key_id, + audit_rolled=audit_chain_rolled(store), + range_key_missing=audit_range_key_missing(store), + allow_lost=settings.secret_rotation.allow_lost_audit_range_key, + ) await record_generated_pair_replacements(store, replaced) finally: await store.close() diff --git a/messagefoundry/config/settings.py b/messagefoundry/config/settings.py index 0e1ae7fb6f..de67c05def 100644 --- a/messagefoundry/config/settings.py +++ b/messagefoundry/config/settings.py @@ -10262,7 +10262,8 @@ def security_loosenings( ``[alerts].webhook_tls_revocation_attested`` (BACKLOG #2193), ``[ai].endpoint_revocation_attested`` (BACKLOG #2193 slice (e)), ``[alerts].email_use_tls``/``email_tls_verify`` (#323 - layer 3), ``[secret_rotation].enforce_store_key_expiry`` (#1004), the two credential-reminder + layer 3), ``[secret_rotation].enforce_store_key_expiry`` (#1004), + ``[secret_rotation].allow_lost_audit_range_key`` (BACKLOG #4934), the two credential-reminder leads ``[cert_monitor].warn_days`` and ``[secret_rotation].warn_days`` and an ``[[alerts.rules]]`` entry that can silence a reminder (#2227, #2008), the per-connection deviations — ``cleartext_accepted``, ``tls_allow_expired``, ``tls_check_hostname=false`` (ASVS diff --git a/messagefoundry/pipeline/secret_rotation.py b/messagefoundry/pipeline/secret_rotation.py index ef3a97c79c..3612cc6467 100644 --- a/messagefoundry/pipeline/secret_rotation.py +++ b/messagefoundry/pipeline/secret_rotation.py @@ -405,9 +405,11 @@ class UnrolledStoreKeyRefused(RuntimeError): "the store key in use differs from its stamp, or has none, the audit chain has not been " "rolled to it, and the key the chain's current range is under is not configured. This start's " "first audit row would go into that range under another key, and the chain would break there, " - "so no later `messagefoundry rotate-key` could roll it. Put the prior store key back in " - "MEFOR_STORE_ENCRYPTION_KEYS_RETIRED, keep the new key active, and run `messagefoundry " - "rotate-key` until it exits 0, then start. Only where the prior key is truly lost, set " + "so no later `messagefoundry rotate-key` could roll it. Put the key that range is under back in " + "MEFOR_STORE_ENCRYPTION_KEYS_RETIRED (it can be older than the last key, where an earlier key " + "change was never rolled; the store open's ERROR names the range's first row), keep the new key " + "active, and run `messagefoundry rotate-key` until it exits 0, then start. Only where that key " + "is truly lost, set " "[secret_rotation].allow_lost_audit_range_key = true to start anyway: that is a security " "loosening, and the chain breaks at this start's first audit row." ) diff --git a/scripts/security/crypto_inventory_check.py b/scripts/security/crypto_inventory_check.py index 6be2730d53..a24142f1e3 100644 --- a/scripts/security/crypto_inventory_check.py +++ b/scripts/security/crypto_inventory_check.py @@ -1073,6 +1073,9 @@ "compare:via messagefoundry.api.tls", # BACKLOG #1167: evaluate_anchor's pin compare "compare:via messagefoundry.auth.totp", "compare:via messagefoundry.pipeline.secret_rotation", + # BACKLOG #4934: supervise's start refusal reads the audit chain's range keys the open + # settled, as serve's lifespan does. + "hash:via messagefoundry.store.store", # BACKLOG #3470: anchor_refusal checks the forwarder's and the SMTP relay's CA before # those hops are built, through evaluate_anchor: its SHA-256 fingerprint, and its pin # compare, which these specs never reach since they carry no pin. diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index bcba1c8852..161267242d 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -16,6 +16,8 @@ import asyncio import logging +import sqlite3 +from contextlib import closing from pathlib import Path from typing import Any @@ -165,6 +167,31 @@ def test_the_refusal_needs_all_three_conditions(tmp_path: Path, case: str) -> No asyncio.run(_refusal(db, key_a)) +def test_a_stamp_naming_the_key_in_use_does_not_refuse(tmp_path: Path) -> None: + """The stamp arm alone: chain not rolled and its range key not configured, but the stamp + already names the key in use, so the start goes through. The other cases return before the + stamp is read.""" + db, key_a, key_b = tmp_path / "s.db", generate_key(), generate_key() + asyncio.run(_seed(db, tmp_path / "uploads", key_a)) + + async def stamp_key_b() -> None: + store = await _open(db, key_b) + try: + await store.upsert_secret_rotation_meta( + _DEK, + fingerprint=dek_fingerprint(_key_id(key_b)), + tracked_since=_OLD_DATE, + last_rotated=_OLD_DATE, + ) + assert audit_chain_rolled(store) is False + assert audit_range_key_missing(store) is True + finally: + await store.close() + + asyncio.run(stamp_key_b()) + asyncio.run(_refusal(db, key_b)) + + def test_the_override_starts_with_a_warning_and_is_a_named_loosening( tmp_path: Path, caplog: pytest.LogCaptureFixture ) -> None: @@ -260,6 +287,52 @@ def test_the_rows_after_a_refused_start_are_unchanged(tmp_path: Path) -> None: assert ROTATION_RECORD_ID not in rows +@pytest.mark.parametrize("allow_lost", [False, True]) +def test_supervise_refuses_before_it_records_a_renewed_api_pair( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, allow_lost: bool +) -> None: + """The supervisor writes an audit row for a renewed API pair before any shard starts, so it + takes the shards' refusal first, or a refused fleet would already have broken the chain. Under + the override it records the renewal, as the shards then start.""" + from messagefoundry import __main__ as cli + from messagefoundry.api import tls + from messagefoundry.config.settings import ServiceSettings + + db, key_a, key_b = tmp_path / "sup.db", generate_key(), generate_key() + asyncio.run(_seed(db, tmp_path / "uploads", key_a)) + recorded: list[object] = [] + + def renewed(*_args: object, replacements: list[object], **_kwargs: object) -> None: + replacements.append(object()) + + async def record(_store: object, replaced: list[object]) -> None: + recorded.extend(replaced) + + monkeypatch.setattr(tls, "ensure_api_tls_material", renewed) + monkeypatch.setattr(tls, "record_generated_pair_replacements", record) + settings = ServiceSettings( + store=StoreSettings(path=str(db), encryption_key=key_b), + secret_rotation=SecretRotationSettings(allow_lost_audit_range_key=allow_lost), + ) + before = _audit_rows(db) + if allow_lost: + cli._renew_api_tls_before_spawning(settings, str(db)) + assert len(recorded) == 1 + return + with pytest.raises(UnrolledStoreKeyRefused): + cli._renew_api_tls_before_spawning(settings, str(db)) + assert recorded == [] + assert _audit_rows(db) == before + + +def _audit_rows(db: Path) -> int: + """The audit_log row count of a synthetic test store, read without opening it as a store.""" + with closing(sqlite3.connect(db)) as conn: + (count,) = conn.execute("SELECT COUNT(*) FROM audit_log").fetchone() + assert isinstance(count, int) + return count + + def _app_on_the_new_key_alone(tmp_path: Path, allow_lost: bool) -> Any: from messagefoundry.api import create_managed_app from messagefoundry.config.settings import EgressSettings @@ -280,9 +353,12 @@ def _app_on_the_new_key_alone(tmp_path: Path, allow_lost: bool) -> Any: async def test_serve_refuses_a_start_on_the_new_key_alone(tmp_path: Path) -> None: """#4934 step 3, the wiring: the refusal aborts the serve lifespan.""" app = await asyncio.to_thread(_app_on_the_new_key_alone, tmp_path, False) + before = _audit_rows(tmp_path / "serve.db") with pytest.raises(UnrolledStoreKeyRefused): async with app.router.lifespan_context(app): pass # pragma: no cover -- startup must not reach here + # The point of the refusal: the refused start wrote no audit row into the unrolled range. + assert _audit_rows(tmp_path / "serve.db") == before async def test_serve_starts_on_the_new_key_alone_under_the_override(tmp_path: Path) -> None: From 3cb08115758be65a4f886be3e5e64ceeae054107 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 16:14:33 -0600 Subject: [PATCH 03/41] rotate-key opens its record through an open_store hook; serve and supervise refuse a key change the audit chain cannot follow Review round 2 (xhigh, round-09b4f4af): - supervise now catches UnrolledStoreKeyRefused from the API pair renewal and prints it in full with exit 2, as its sibling start refusals do, rather than letting the CLI floor cut the message (and its remedy) at 200 characters with exit 1. - docs/SECURITY-LOOSENING.md: the override is not reversible once a start has used it; the store then needs it on for every later start. Proposed PR title: rotate-key opens its record through an open_store hook; serve and supervise refuse a key change the audit chain cannot follow (BACKLOG #4934, #4936) Proposed ledger banner: #4936 SHIPPED (open_store before_first_write hook, run in all three backend opens; rotate-key's record opens through it). #4934 SHIPPED (serve and supervise refuse an unrolled key change with the range key missing; [secret_rotation].allow_lost_audit_range_key is the named loosening override). Open review findings filed as items on the row. BACKLOG #4934, #4936 --- docs/SECURITY-LOOSENING.md | 7 ++++--- messagefoundry/__main__.py | 4 ++++ 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index f00f6dd626..b8322111f7 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -853,9 +853,10 @@ the next start may do. - **Compensating controls:** none that restore the chain. Record when and why the key was lost, keep the `audit-verify` output from before the start, and treat the break as a known event in any audit review. -- **Reversible:** the setting is, the break is not. The chain stays broken at that row, and turning - the setting back off makes the next start refuse again, because the chain's range is still under - the lost key. +- **Reversible:** no, once a start has used it. The chain stays broken at that row, and the store + needs the setting on for every later start: turned back off, the next start refuses again, + because the chain's range is still under the lost key, and no `rotate-key` run can move it. Set + only and never used, it changes nothing and can be removed. ### `[cert_monitor].warn_days` or `[secret_rotation].warn_days` at `0`, or below its default — a credential reminder is off or late Named as `cert_monitor.warn_days` and `secret_rotation.warn_days` (ASVS 6.4.5, BACKLOG #2227). A diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index 49c34ec868..3812d9aafc 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -5505,6 +5505,7 @@ def _supervise(args: argparse.Namespace) -> int: # Vault BACKLOG #2601: the two key-file checks each shard's `serve` makes, for the same reason # as the gates above: every shard would refuse, and the supervisor would only restart them. The # store key file is checked before the renewal below can open the store and read it. + from messagefoundry.pipeline.secret_rotation import UnrolledStoreKeyRefused from messagefoundry.restricted_file import RestrictedFileError from messagefoundry.store.base import KeylessAuditChainRefused @@ -5573,6 +5574,9 @@ def configure_with_forwarder(forward: SyslogForward | None) -> bool: except KeylessAuditChainRefused as exc: # #1916: a named key the provider did not resolve print(f"error: {exc}", file=sys.stderr) return 2 + except UnrolledStoreKeyRefused as exc: # BACKLOG #4934: in full, since it carries the remedy + print(f"error: {exc}", file=sys.stderr) + return 2 except RestrictedFileError as exc: # the TLS key could not be created restricted print(f"error: {exc}; refusing to start the fleet.", file=sys.stderr) return 2 From 5e27a478b89251d07724a2b935fa105c45c3ce0b Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 17:09:02 -0600 Subject: [PATCH 04/41] A refused start renews no API pair; the refusal's transit exception is stated Repairs the Lander hold on engine PR 2991 at 3cb0811575 (BACKLOG #4934, #4936). B1: supervise now opens its store on every start and takes the unrolled-key refusal before ensure_api_tls_material, so a refused fleet start renews nothing on disk and does not restart shards that would each refuse. serve mints before its lifespan opens the store, so a start that may replace the generated pair (renewal or recovery, read with the new read-only generated_pair_may_be_replaced) opens the store and takes the refusal first, exiting 2 with the remedy. A first-run mint replaces nothing and opens nothing early. Tests: real pair due for renewal stays byte-identical on a refused supervise and serve start (renewed and audited under the override), and supervise's catch exits 2 with the remedy. B2: SECURITY-LOOSENING.md states the vault_transit exception (no local store key, so the refusal never fires); CONFIGURATION.md and the changelog point to it. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/4934.security.md | 4 +- docs/CONFIGURATION.md | 2 +- docs/SECURITY-LOOSENING.md | 11 ++- messagefoundry/__main__.py | 96 ++++++++++++++----- messagefoundry/api/tls.py | 18 ++++ tests/test_key_file_access.py | 7 +- tests/test_keyless_chain_every_command.py | 3 + ...rotate_key_open_hook_and_lost_range_key.py | 92 ++++++++++++++---- tests/test_vault_posture_census.py | 2 + 9 files changed, 189 insertions(+), 46 deletions(-) diff --git a/changelog.d/4934.security.md b/changelog.d/4934.security.md index e801b60c61..ff8f9dce14 100644 --- a/changelog.d/4934.security.md +++ b/changelog.d/4934.security.md @@ -8,7 +8,9 @@ **`serve` refuses to start when the store key differs from its stamp, or has none, the keyed audit chain has not been rolled to it, and the key the chain's current range is under is not configured.** That start's first audit row would break the chain. `supervise` refuses the same - state before it records a renewed API certificate. The refusal says to put the key that range + state on every start, before it renews the API certificate, and `serve` refuses before it + writes the generated pair, so a refused start renews nothing. A `vault_transit` store is the one + exception, stated in `docs/SECURITY-LOOSENING.md`. The refusal says to put the key that range is under back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. `[secret_rotation].allow_lost_audit_range_key = true` starts anyway, for a key that is truly lost; it logs the refusal as a WARNING, the chain breaks at that start's first audit row, and it diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 3e4a197372..4f7b0b9946 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -3005,7 +3005,7 @@ tracked; set `warn_days = 0` to disable the reminder. | `secret_max_age_days` | int | 365 | max age for the **non-DEK** tracked secret classes (connector/AD/SMTP/Vault/OIDC), alerted this many days after their last observed fingerprint change | | `enforce_grace_days` | int | 30 | under `[security].enforcement=enforce`, a DEK older than `store_key_max_age_days + this` escalates its rotation alert **and refuses to start** (see `enforce_store_key_expiry`) | | `enforce_store_key_expiry` | bool | `true` | under `[security].enforcement=enforce`, a store DEK past `store_key_max_age_days + enforce_grace_days` — or one whose age cannot be determined — **aborts engine start**. `false` keeps the alert, drops the refusal, and is reported as a **security loosening** | -| `allow_lost_audit_range_key` | bool | `false` | `serve` (and `supervise`, before it records a renewed API certificate) refuses to start when the store key differs from its stamp (or has none), the keyed audit chain has not been rolled to it, and the key the chain's current range is under is not configured: put that key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. `true` starts anyway, for a key that is truly lost; the chain breaks at that start's first audit row, and it is reported as a **security loosening** | +| `allow_lost_audit_range_key` | bool | `false` | `serve` (and `supervise`, before it renews the API certificate) refuses to start when the store key differs from its stamp (or has none), the keyed audit chain has not been rolled to it, and the key the chain's current range is under is not configured, except on a `vault_transit` store, as [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) states: put that key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. `true` starts anyway, for a key that is truly lost; the chain breaks at that start's first audit row, and it is reported as a **security loosening** | | `enforce_secret_expiry_classes` | list | `[]` | non-DEK classes whose calendar expiry **aborts engine start** under `[security].enforcement=enforce`, past `secret_max_age_days + enforce_grace_days` or with no recorded age. Entries are the `MEFOR_*` class names above or `connector`; an unknown name is refused at load. Empty keeps every non-DEK class alert-only | ```toml diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index b8322111f7..64ad2e788d 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -838,10 +838,13 @@ This section is kept rather than deleted, because the claim it used to make is t Named as `allow_lost_audit_range_key` (BACKLOG #4934), whatever the store holds, because it says what the next start may do. - **What you lose:** the start refusal that keeps the keyed audit chain whole across a store key - change. With it off, `serve` (and each `supervise` shard, and the supervisor before it records a - renewed API certificate) refuses to start when the store key differs from its stamp, or has none, - the chain has not been rolled to it, and the key the chain's current range is under is not - configured. With it on, that start goes ahead: its first audit row goes into that range under the + change. With it off, `serve` (and each `supervise` shard, and the supervisor before it renews the + API certificate) refuses to start when the store key differs from its stamp, or has none, the + chain has not been rolled to it, and the key the chain's current range is under is not + configured. A refused start renews no API certificate. The one exception is a store under + `[store].cipher_provider = "vault_transit"`: Transit holds that store's key, so the engine has no + local store key to compare with the stamp, the refusal never fires, and this setting changes + nothing there. With it on, that start goes ahead: its first audit row goes into that range under the new key, never verifies there, and the chain breaks at that row. `audit-verify` reports the break, and `rotate-key` cannot roll the chain afterwards. - **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index 3812d9aafc..e4cd57d997 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -4899,6 +4899,7 @@ def registry_filter(reg: Registry) -> Registry: # noqa: F811 (local shard-bound from messagefoundry.api.tls import ( GeneratedPairReplaced, ensure_api_tls_material, + generated_pair_may_be_replaced, generated_state_dir, ) from messagefoundry.restricted_file import RestrictedFileError @@ -4907,6 +4908,18 @@ def registry_filter(reg: Registry) -> Registry: # noqa: F811 (local shard-bound # the store is open, which is after this point (ADR 0172 decision 6: never silent). _replaced: list[GeneratedPairReplaced] = [] _state_dir = generated_state_dir(settings.store.path) + # BACKLOG #4934: the lifespan's start refusal runs after this mint, so a start that may replace + # the pair takes it here first, and a refused start renews nothing on disk. A first-run mint + # replaces nothing, so it leaves no renewal without its audit row and needs no early open. + if generated_pair_may_be_replaced(settings.api, state_dir=_state_dir, renew=args.shard is None): + from messagefoundry.pipeline.secret_rotation import UnrolledStoreKeyRefused + from messagefoundry.store.base import KeylessAuditChainRefused + + try: + _refuse_unrolled_key_before_tls_write(settings) + except (UnrolledStoreKeyRefused, KeylessAuditChainRefused) as exc: + print(f"error: {exc}", file=sys.stderr) + return 2 try: _material = ensure_api_tls_material( settings.api, @@ -5225,9 +5238,10 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P supervisor forwards, or the settings file). ``db_base`` is anchored only by ``--project-root`` here, so a base_dir set in the file is applied too, or the two would name different pairs. - A re-mint is audited like serve's, in the store the shards are about to open, over the same - derived store-hop posture serve's lifespan opens it with. A failure to open propagates, as it - would in serve: the WARNING the renewal logged, with both fingerprints, is then its record. + The store the shards are about to open is opened first, on every start, over the same derived + store-hop posture serve's lifespan opens it with, and takes serve's start refusal before the + renewal runs (BACKLOG #4934), so a refused start writes nothing to the pair. A re-mint is then + audited like serve's, in that store. A failure to open propagates before any renewal. """ from messagefoundry.api.tls import ( GeneratedPairReplaced, @@ -5238,13 +5252,7 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P from messagefoundry.config.anchor import anchored_store_path, resolve_project_root from messagefoundry.config.settings import hop_posture_from_ai, keyless_opt_out_refusal from messagefoundry.last_resort import run_guarded - from messagefoundry.pipeline.secret_rotation import refuse_unrolled_key_change from messagefoundry.store import open_store - from messagefoundry.store.store import ( - SecretRotationMetaStore, - audit_chain_rolled, - audit_range_key_missing, - ) cwd = Path.cwd() root = resolve_project_root(settings.environments.base_dir or None, cwd=cwd) @@ -5252,9 +5260,6 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P state_dir = generated_state_dir(store_path) replaced: list[GeneratedPairReplaced] = [] - ensure_api_tls_material(settings.api, state_dir=state_dir, replacements=replaced) - if not replaced: - return state_dir posture = ( hop_posture_from_ai(settings.ai, enforcement=settings.security.enforcement) if settings.ai is not None @@ -5273,16 +5278,11 @@ async def _audit() -> None: keyless_chain_refusal=keyless_opt_out_refusal(shard_store, settings.security), ) try: - # BACKLOG #4934: the audit row below is this store's first since the open, so it takes - # the shards' start refusal first, or a refused start would already have broken the chain. - if isinstance(store, SecretRotationMetaStore): - await refuse_unrolled_key_change( - store, - dek_key_id=store.cipher_info().active_key_id, - audit_rolled=audit_chain_rolled(store), - range_key_missing=audit_range_key_missing(store), - allow_lost=settings.secret_rotation.allow_lost_audit_range_key, - ) + # BACKLOG #4934: the shards' start refusal, on every start and before the renewal, so a + # refused start renews nothing on disk and the supervisor does not restart shards that + # would each refuse. The audit row below is then this store's first since the open. + await _refuse_unrolled_key_on(store, settings) + ensure_api_tls_material(settings.api, state_dir=state_dir, replacements=replaced) await record_generated_pair_replacements(store, replaced) finally: await store.close() @@ -5291,6 +5291,58 @@ async def _audit() -> None: return state_dir +async def _refuse_unrolled_key_on(store: Store, settings: ServiceSettings) -> None: + """``serve``'s start refusal of a changed store key the keyed audit chain cannot follow + (BACKLOG #4934), on an open store; see + :func:`~messagefoundry.pipeline.secret_rotation.refuse_unrolled_key_change`.""" + from messagefoundry.pipeline.secret_rotation import refuse_unrolled_key_change + from messagefoundry.store.store import ( + SecretRotationMetaStore, + audit_chain_rolled, + audit_range_key_missing, + ) + + if isinstance(store, SecretRotationMetaStore): + await refuse_unrolled_key_change( + store, + dek_key_id=store.cipher_info().active_key_id, + audit_rolled=audit_chain_rolled(store), + range_key_missing=audit_range_key_missing(store), + allow_lost=settings.secret_rotation.allow_lost_audit_range_key, + ) + + +def _refuse_unrolled_key_before_tls_write(settings: ServiceSettings) -> None: + """Open ``serve``'s store and run :func:`_refuse_unrolled_key_on` before the start may replace + the generated API pair (BACKLOG #4934), so a refused start renews nothing on disk. The lifespan + refuses again after its own open; this one exists only because the mint comes first. Raises + :class:`~messagefoundry.pipeline.secret_rotation.UnrolledStoreKeyRefused`, and whatever the + open raises, as the lifespan's open would.""" + from messagefoundry.config.settings import hop_posture_from_ai, keyless_opt_out_refusal + from messagefoundry.last_resort import run_guarded + from messagefoundry.store import open_store + + posture = ( + hop_posture_from_ai(settings.ai, enforcement=settings.security.enforcement) + if settings.ai is not None + else None + ) + + async def _check() -> None: + store = await open_store( + settings.store, + create=True, + posture=posture, + keyless_chain_refusal=keyless_opt_out_refusal(settings.store, settings.security), + ) + try: + await _refuse_unrolled_key_on(store, settings) + finally: + await store.close() + + run_guarded(_check()) + + def _supervise(args: argparse.Namespace) -> int: """L3 multi-process sharding (messagefoundry/pipeline/supervisor.py): discover the shard ids in the config and run one `serve --shard ` subprocess per shard, each with its own SQLite db file and diff --git a/messagefoundry/api/tls.py b/messagefoundry/api/tls.py index 2608bbbe83..84d45c0560 100644 --- a/messagefoundry/api/tls.py +++ b/messagefoundry/api/tls.py @@ -60,6 +60,7 @@ "api_tls_source", "build_api_ssl_context", "ensure_api_tls_material", + "generated_pair_may_be_replaced", "generated_state_dir", "off_loopback_certificate_remedy", "plan_api_tls_material", @@ -628,6 +629,23 @@ def _let_local_users_read_cert(cert_path: Path) -> None: _grant_read(cert_path, _LOCAL_USERS_SID) +def generated_pair_may_be_replaced( + api: ApiSettings, *, state_dir: Path, renew: bool = True +) -> bool: + """True when :func:`ensure_api_tls_material`, given the same arguments, may replace a generated + pair already on disk: a renewal, or the recovery of an unusable or half-written pair, each of + which it reports as a :class:`GeneratedPairReplaced`. A first-run mint replaces nothing and + answers ``False``. Read-only: it writes nothing and takes no lock, so a start can run a + refusal first and leave the pair as it found it on a refused start (BACKLOG #4934). A pair that + changes on disk between this call and the mint is decided again there.""" + if plan_api_tls_material(api, state_dir=state_dir).source != "generated": + return False + cert_path, key_path = _generated_pair(state_dir) + if not (cert_path.exists() or key_path.exists()): + return False + return not _reusable_without_the_lock(cert_path, key_path, renew=renew) + + def ensure_api_tls_material( api: ApiSettings, *, diff --git a/tests/test_key_file_access.py b/tests/test_key_file_access.py index a0c02a01ee..7a668cba36 100644 --- a/tests/test_key_file_access.py +++ b/tests/test_key_file_access.py @@ -699,7 +699,12 @@ async def fake_supervise(config: str, **kwargs: Any) -> int: return 0 monkeypatch.chdir(tmp_path) - monkeypatch.setenv("MEFOR_STORE_ENCRYPTION_KEY", generate_key()) + # One key per test: supervise opens the store on every start, and a second start on another + # key over the first start's store takes the unrolled-key refusal (BACKLOG #4934). + key_file = tmp_path / "test-store-key.txt" + if not key_file.exists(): + key_file.write_text(generate_key(), encoding="utf-8") + monkeypatch.setenv("MEFOR_STORE_ENCRYPTION_KEY", key_file.read_text(encoding="utf-8")) monkeypatch.setattr("messagefoundry.pipeline.supervisor.supervise", fake_supervise) monkeypatch.setattr(cli, "configure_logging", lambda *args, **kwargs: None) setenv_declared_egress(monkeypatch) diff --git a/tests/test_keyless_chain_every_command.py b/tests/test_keyless_chain_every_command.py index 6c29e8144d..9baa696155 100644 --- a/tests/test_keyless_chain_every_command.py +++ b/tests/test_keyless_chain_every_command.py @@ -1358,6 +1358,9 @@ def test_a_service_config_that_cannot_be_read_exits_2( "_provision_admin.page_granted.reopen", # the supervisor's audit of a re-minted API pair, on the verdict serve's lifespan passes "_renew_api_tls_before_spawning._audit", + # BACKLOG #4934: serve's open for the unrolled-key refusal before it writes the API pair, on + # the verdict its lifespan passes + "_refuse_unrolled_key_before_tls_write._check", "_audit_verify.run", "_audit_anchor.run", # BACKLOG #1719: cipher-audit's read-only open of a PostgreSQL or SQL Server store diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index 161267242d..3a7c086f9c 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -287,29 +287,32 @@ def test_the_rows_after_a_refused_start_are_unchanged(tmp_path: Path) -> None: assert ROTATION_RECORD_ID not in rows +def _mint_a_pair_due_for_renewal(state_dir: Path, monkeypatch: pytest.MonkeyPatch) -> bytes: + """A real generated API pair in ``state_dir``, reported due for renewal from now on, so the + next renewing start writes a new one. Returns the certificate's bytes as minted.""" + from messagefoundry.api import tls + from messagefoundry.config.settings import ApiSettings + + tls.ensure_api_tls_material(ApiSettings(), state_dir=state_dir) + monkeypatch.setattr(tls, "_renewal_due", tls._cert_facts) + return (state_dir / "api-generated-cert.pem").read_bytes() + + @pytest.mark.parametrize("allow_lost", [False, True]) -def test_supervise_refuses_before_it_records_a_renewed_api_pair( +def test_a_refused_supervise_start_leaves_the_api_pair_unchanged( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, allow_lost: bool ) -> None: - """The supervisor writes an audit row for a renewed API pair before any shard starts, so it - takes the shards' refusal first, or a refused fleet would already have broken the chain. Under - the override it records the renewal, as the shards then start.""" + """#4934: the refusal runs before the renewal, so a refused fleet start renews nothing on disk + and leaves no renewal without its audit row. Under the override the same start renews the pair + and records it, the positive control that the pair was due.""" from messagefoundry import __main__ as cli - from messagefoundry.api import tls + from messagefoundry.api.tls import generated_state_dir from messagefoundry.config.settings import ServiceSettings db, key_a, key_b = tmp_path / "sup.db", generate_key(), generate_key() asyncio.run(_seed(db, tmp_path / "uploads", key_a)) - recorded: list[object] = [] - - def renewed(*_args: object, replacements: list[object], **_kwargs: object) -> None: - replacements.append(object()) - - async def record(_store: object, replaced: list[object]) -> None: - recorded.extend(replaced) - - monkeypatch.setattr(tls, "ensure_api_tls_material", renewed) - monkeypatch.setattr(tls, "record_generated_pair_replacements", record) + state_dir = generated_state_dir(str(db)) + minted = _mint_a_pair_due_for_renewal(state_dir, monkeypatch) settings = ServiceSettings( store=StoreSettings(path=str(db), encryption_key=key_b), secret_rotation=SecretRotationSettings(allow_lost_audit_range_key=allow_lost), @@ -317,14 +320,69 @@ async def record(_store: object, replaced: list[object]) -> None: before = _audit_rows(db) if allow_lost: cli._renew_api_tls_before_spawning(settings, str(db)) - assert len(recorded) == 1 + assert (state_dir / "api-generated-cert.pem").read_bytes() != minted + assert _audit_rows(db) == before + 1 return with pytest.raises(UnrolledStoreKeyRefused): cli._renew_api_tls_before_spawning(settings, str(db)) - assert recorded == [] + assert (state_dir / "api-generated-cert.pem").read_bytes() == minted + assert _audit_rows(db) == before + + +def test_a_refused_serve_start_leaves_the_api_pair_unchanged( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """#4934: serve mints before its lifespan opens the store, so a start that would write the + pair takes the refusal first, exits 2 with the remedy, and renews nothing on disk.""" + db, key_a, key_b = tmp_path / "messagefoundry.db", generate_key(), generate_key() + asyncio.run(_seed(db, tmp_path / "uploads", key_a)) + minted = _mint_a_pair_due_for_renewal(tmp_path, monkeypatch) + started: list[object] = [] + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("MEFOR_STORE_ENCRYPTION_KEY", key_b) + # The enforce-only start gates (an off-box log collector, ...) are not under test here. + monkeypatch.setenv("MEFOR_SECURITY_ENFORCEMENT", "warn") + monkeypatch.setattr("uvicorn.run", lambda *a, **k: started.append(k)) + (tmp_path / "messagefoundry.toml").write_text( + "security.block_unlisted_outbound = true\n" + "alerts.security_notifications_required = false\n" + "security.local_access_only = true\n", + encoding="utf-8", + ) + samples = Path(__file__).resolve().parent.parent / "samples" / "config" + before = _audit_rows(db) + assert main(["serve", "--config", str(samples), "--env", "dev"]) == 2 + err = capsys.readouterr().err + assert "MEFOR_STORE_ENCRYPTION_KEYS_RETIRED" in err, err[-2000:] + assert started == [] + assert (tmp_path / "api-generated-cert.pem").read_bytes() == minted assert _audit_rows(db) == before +def test_supervise_exits_2_with_the_remedy_on_the_refusal( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """#4934: ``supervise`` catches the refusal, prints it in full and spawns no shard.""" + from messagefoundry import __main__ as cli + from tests.test_remote_debug_start_refusal import ( + _OFF, + _stand_down_the_gates_before_logging, + _supervise, + ) + + def refused(*_args: object, **_kwargs: object) -> Path: + raise UnrolledStoreKeyRefused("refusing to start: TEST REMEDY") + + _stand_down_the_gates_before_logging(monkeypatch) + monkeypatch.setattr(cli, "_keyless_store_gate", lambda settings: None) + monkeypatch.setattr(cli, "_start_logging", lambda *a, **k: None) + monkeypatch.setattr(cli, "_renew_api_tls_before_spawning", refused) + rc, spawned = _supervise(tmp_path, monkeypatch, _OFF) + err = capsys.readouterr().err + assert rc == 2 and not spawned, err[-2000:] + assert "error: refusing to start: TEST REMEDY" in err + + def _audit_rows(db: Path) -> int: """The audit_log row count of a synthetic test store, read without opening it as a store.""" with closing(sqlite3.connect(db)) as conn: diff --git a/tests/test_vault_posture_census.py b/tests/test_vault_posture_census.py index 8bf625f486..020e8f6a30 100644 --- a/tests/test_vault_posture_census.py +++ b/tests/test_vault_posture_census.py @@ -309,6 +309,7 @@ def _found() -> tuple[Row, ...]: ("__main__.py", "_provision_admin.run", "open_store", "posture"), ("__main__.py", "_provision_admin.server_store_found", "open_store", "posture"), ("__main__.py", "_renew_api_tls_before_spawning._audit", "open_store", "posture"), + ("__main__.py", "_refuse_unrolled_key_before_tls_write._check", "open_store", "posture"), ("__main__.py", "_rotate_key", "resolve_active_key", "posture"), ("__main__.py", "_rotate_key.run", "open_store", "posture"), ("__main__.py", "_store_transit_bound.run", "open_store", "posture"), @@ -523,6 +524,7 @@ def test_every_place_a_posture_is_built_is_listed() -> None: ("__main__.py", "_offline_security_notifier", "settings.security.enforcement"), ("__main__.py", "_provision_admin", "settings.security.enforcement"), ("__main__.py", "_renew_api_tls_before_spawning", "settings.security.enforcement"), + ("__main__.py", "_refuse_unrolled_key_before_tls_write", "settings.security.enforcement"), ("__main__.py", "_start_logging", "settings.security.enforcement"), ("__main__.py", "_store_provision_schema", "settings.security.enforcement"), # At warn, the store-opening commands' PostgreSQL revocation WARNING (BACKLOG #2532). From 1b0d44a71db56fe6810c5345a5f860276c5d2d1e Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 17:31:29 -0600 Subject: [PATCH 05/41] supervise opens its store early only on a start that may replace the pair Repairs findings of review round-bb323c84 on the B1/B2 repair of engine PR 2991 (BACKLOG #4934, #4936). supervise opened its store on every start, which sent an at-open cipher refusal to the log instead of a shard lifespan's notifier and left most open refusals to the dispatch floor on ordinary starts. It now opens early only when generated_pair_may_be_replaced says the start may renew or recover the pair, as serve does; the refusal still runs before that write, which runs off the event loop. serve's early open maps the open refusals _open_store_or_refuse maps to exit 2. The docs no longer claim vault_transit is the only store the refusal skips: unauthenticated audit ranges skip it too. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/4934.security.md | 10 +++---- docs/CONFIGURATION.md | 2 +- docs/SECURITY-LOOSENING.md | 10 ++++--- messagefoundry/__main__.py | 55 +++++++++++++++++++++++++---------- tests/test_key_file_access.py | 7 +---- 5 files changed, 53 insertions(+), 31 deletions(-) diff --git a/changelog.d/4934.security.md b/changelog.d/4934.security.md index ff8f9dce14..592d7147ea 100644 --- a/changelog.d/4934.security.md +++ b/changelog.d/4934.security.md @@ -7,11 +7,11 @@ there leaves no record. **`serve` refuses to start when the store key differs from its stamp, or has none, the keyed audit chain has not been rolled to it, and the key the chain's current range is under is not - configured.** That start's first audit row would break the chain. `supervise` refuses the same - state on every start, before it renews the API certificate, and `serve` refuses before it - writes the generated pair, so a refused start renews nothing. A `vault_transit` store is the one - exception, stated in `docs/SECURITY-LOOSENING.md`. The refusal says to put the key that range - is under back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. + configured.** That start's first audit row would break the chain. `serve`, and + `supervise` before it renews the API certificate, refuse before they may replace the generated + pair, so a refused start renews nothing. `docs/SECURITY-LOOSENING.md` names the stores the + refusal does not cover, at least a `vault_transit` store. The refusal says to put the key that + range is under back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. `[secret_rotation].allow_lost_audit_range_key = true` starts anyway, for a key that is truly lost; it logs the refusal as a WARNING, the chain breaks at that start's first audit row, and it is reported as a security loosening. Other commands that open the store and write an audit row, diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 4f7b0b9946..06568767bd 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -3005,7 +3005,7 @@ tracked; set `warn_days = 0` to disable the reminder. | `secret_max_age_days` | int | 365 | max age for the **non-DEK** tracked secret classes (connector/AD/SMTP/Vault/OIDC), alerted this many days after their last observed fingerprint change | | `enforce_grace_days` | int | 30 | under `[security].enforcement=enforce`, a DEK older than `store_key_max_age_days + this` escalates its rotation alert **and refuses to start** (see `enforce_store_key_expiry`) | | `enforce_store_key_expiry` | bool | `true` | under `[security].enforcement=enforce`, a store DEK past `store_key_max_age_days + enforce_grace_days` — or one whose age cannot be determined — **aborts engine start**. `false` keeps the alert, drops the refusal, and is reported as a **security loosening** | -| `allow_lost_audit_range_key` | bool | `false` | `serve` (and `supervise`, before it renews the API certificate) refuses to start when the store key differs from its stamp (or has none), the keyed audit chain has not been rolled to it, and the key the chain's current range is under is not configured, except on a `vault_transit` store, as [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) states: put that key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. `true` starts anyway, for a key that is truly lost; the chain breaks at that start's first audit row, and it is reported as a **security loosening** | +| `allow_lost_audit_range_key` | bool | `false` | `serve` (and `supervise`, before it renews the API certificate) refuses to start when the store key differs from its stamp (or has none), the keyed audit chain has not been rolled to it, and the key the chain's current range is under is not configured, except on the stores [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) names, at least a `vault_transit` store: put that key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. `true` starts anyway, for a key that is truly lost; the chain breaks at that start's first audit row, and it is reported as a **security loosening** | | `enforce_secret_expiry_classes` | list | `[]` | non-DEK classes whose calendar expiry **aborts engine start** under `[security].enforcement=enforce`, past `secret_max_age_days + enforce_grace_days` or with no recorded age. Entries are the `MEFOR_*` class names above or `connector`; an unknown name is refused at load. Empty keeps every non-DEK class alert-only | ```toml diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index 64ad2e788d..baad7913c4 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -841,10 +841,12 @@ the next start may do. change. With it off, `serve` (and each `supervise` shard, and the supervisor before it renews the API certificate) refuses to start when the store key differs from its stamp, or has none, the chain has not been rolled to it, and the key the chain's current range is under is not - configured. A refused start renews no API certificate. The one exception is a store under - `[store].cipher_provider = "vault_transit"`: Transit holds that store's key, so the engine has no - local store key to compare with the stamp, the refusal never fires, and this setting changes - nothing there. With it on, that start goes ahead: its first audit row goes into that range under the + configured. A refused start renews no API certificate. The refusal never fires on at least two + kinds of store, so this setting changes nothing there: a store under + `[store].cipher_provider = "vault_transit"`, because Transit holds its key and the engine has no + local store key to compare with the stamp; and a store whose audit ranges the open could not + authenticate, because the open then cannot say which key the current range is under, though + that start's first audit row may still break the chain. With it on, that start goes ahead: its first audit row goes into that range under the new key, never verifies there, and the chain breaks at that row. `audit-verify` reports the break, and `rotate-key` cannot roll the chain afterwards. - **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index e4cd57d997..9f29970b30 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -4920,6 +4920,9 @@ def registry_filter(reg: Registry) -> Registry: # noqa: F811 (local shard-bound except (UnrolledStoreKeyRefused, KeylessAuditChainRefused) as exc: print(f"error: {exc}", file=sys.stderr) return 2 + except _StoreOpenRefused as exc: # its text already went through codec_safe_str + print(f"error: cannot open the store: {exc}; refusing to start.", file=sys.stderr) + return 2 try: _material = ensure_api_tls_material( settings.api, @@ -5238,14 +5241,18 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P supervisor forwards, or the settings file). ``db_base`` is anchored only by ``--project-root`` here, so a base_dir set in the file is applied too, or the two would name different pairs. - The store the shards are about to open is opened first, on every start, over the same derived - store-hop posture serve's lifespan opens it with, and takes serve's start refusal before the - renewal runs (BACKLOG #4934), so a refused start writes nothing to the pair. A re-mint is then - audited like serve's, in that store. A failure to open propagates before any renewal. + A re-mint is audited like serve's, in the store the shards are about to open, over the same + derived store-hop posture serve's lifespan opens it with. On a start that may replace the pair, + that store is opened first and takes serve's start refusal before the renewal runs (BACKLOG + #4934), so a refused start writes nothing to the pair, and a failure to open propagates before + any renewal. A start that replaced a pair it did not expect to opens it after, as before. """ + import asyncio + from messagefoundry.api.tls import ( GeneratedPairReplaced, ensure_api_tls_material, + generated_pair_may_be_replaced, generated_state_dir, record_generated_pair_replacements, ) @@ -5260,6 +5267,14 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P state_dir = generated_state_dir(store_path) replaced: list[GeneratedPairReplaced] = [] + # BACKLOG #4934: a start that may replace the pair opens the store and takes the shards' start + # refusal before it writes anything, so a refused start renews nothing on disk. Any other start + # mints or reuses as before, and opens the store only if that replaced a pair after all. + renew_after_refusal = generated_pair_may_be_replaced(settings.api, state_dir=state_dir) + if not renew_after_refusal: + ensure_api_tls_material(settings.api, state_dir=state_dir, replacements=replaced) + if not replaced: + return state_dir posture = ( hop_posture_from_ai(settings.ai, enforcement=settings.security.enforcement) if settings.ai is not None @@ -5278,11 +5293,17 @@ async def _audit() -> None: keyless_chain_refusal=keyless_opt_out_refusal(shard_store, settings.security), ) try: - # BACKLOG #4934: the shards' start refusal, on every start and before the renewal, so a - # refused start renews nothing on disk and the supervisor does not restart shards that - # would each refuse. The audit row below is then this store's first since the open. + # BACKLOG #4934: before the renewal and before the audit row below, which is this + # store's first since the open. await _refuse_unrolled_key_on(store, settings) - ensure_api_tls_material(settings.api, state_dir=state_dir, replacements=replaced) + if renew_after_refusal: + # Off the loop: key generation, file writes and the pair lock's wait block. + await asyncio.to_thread( + ensure_api_tls_material, + settings.api, + state_dir=state_dir, + replacements=replaced, + ) await record_generated_pair_replacements(store, replaced) finally: await store.close() @@ -5316,8 +5337,10 @@ def _refuse_unrolled_key_before_tls_write(settings: ServiceSettings) -> None: """Open ``serve``'s store and run :func:`_refuse_unrolled_key_on` before the start may replace the generated API pair (BACKLOG #4934), so a refused start renews nothing on disk. The lifespan refuses again after its own open; this one exists only because the mint comes first. Raises - :class:`~messagefoundry.pipeline.secret_rotation.UnrolledStoreKeyRefused`, and whatever the - open raises, as the lifespan's open would.""" + :class:`~messagefoundry.pipeline.secret_rotation.UnrolledStoreKeyRefused`, the keyless-chain + refusal, :class:`_StoreOpenRefused` for an open refusal :func:`_open_store_or_refuse` maps, and + anything else the open raises. This open passes no ``refusal_hook``, so an at-open cipher + refusal reaches the log, not the alert notifier the lifespan's open would send it to.""" from messagefoundry.config.settings import hop_posture_from_ai, keyless_opt_out_refusal from messagefoundry.last_resort import run_guarded from messagefoundry.store import open_store @@ -5329,11 +5352,13 @@ def _refuse_unrolled_key_before_tls_write(settings: ServiceSettings) -> None: ) async def _check() -> None: - store = await open_store( - settings.store, - create=True, - posture=posture, - keyless_chain_refusal=keyless_opt_out_refusal(settings.store, settings.security), + store = await _open_store_or_refuse( + open_store( + settings.store, + create=True, + posture=posture, + keyless_chain_refusal=keyless_opt_out_refusal(settings.store, settings.security), + ) ) try: await _refuse_unrolled_key_on(store, settings) diff --git a/tests/test_key_file_access.py b/tests/test_key_file_access.py index 7a668cba36..a0c02a01ee 100644 --- a/tests/test_key_file_access.py +++ b/tests/test_key_file_access.py @@ -699,12 +699,7 @@ async def fake_supervise(config: str, **kwargs: Any) -> int: return 0 monkeypatch.chdir(tmp_path) - # One key per test: supervise opens the store on every start, and a second start on another - # key over the first start's store takes the unrolled-key refusal (BACKLOG #4934). - key_file = tmp_path / "test-store-key.txt" - if not key_file.exists(): - key_file.write_text(generate_key(), encoding="utf-8") - monkeypatch.setenv("MEFOR_STORE_ENCRYPTION_KEY", key_file.read_text(encoding="utf-8")) + monkeypatch.setenv("MEFOR_STORE_ENCRYPTION_KEY", generate_key()) monkeypatch.setattr("messagefoundry.pipeline.supervisor.supervise", fake_supervise) monkeypatch.setattr(cli, "configure_logging", lambda *args, **kwargs: None) setenv_declared_egress(monkeypatch) From 47e461251209d0c975b6a81fc690a60082ed4e01 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 17:50:34 -0600 Subject: [PATCH 06/41] The rotation record docstring and PHI.md say where rotate-key opens it now Round-2 finding 6 on engine PR 2991 (BACKLOG #4934, #4936): both still said the store open can already have written under the key before the record opens. rotate-key now opens it from the open's before_first_write hook, after the schema step and the pin check, before any write under the key. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- docs/PHI.md | 5 +++-- messagefoundry/pipeline/secret_rotation.py | 6 ++++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/PHI.md b/docs/PHI.md index 2cc273f436..73c18233d8 100644 --- a/docs/PHI.md +++ b/docs/PHI.md @@ -738,8 +738,9 @@ for defense-in-depth without swapping the `aiosqlite` connector. uses, where the run prints a note naming the path it read. Before retiring the prior key, confirm the volume is mounted and the path is the one `serve` uses. **A `serve` start does not stamp a changed key while the rotation record names it or cannot be read (BACKLOG - #3737).** `rotate-key` opens a rotation record in `secret_rotation_meta` before either data pass - (the store open can already have written under the key) and closes it only when a run completes, + #3737).** `rotate-key` opens a rotation record in `secret_rotation_meta` inside the store open, + before the open's first write under the key (BACKLOG #4936; only the open's schema step, which + writes nothing under the key, can run before it), and closes it only when a run completes, so a run killed or failed after that point leaves it open. **A sealed record under a key the run was not given stops it (vault BACKLOG #5635).** When a user, role, open approval or directory-map seal names a key the run does not hold, a pre-flight read finds it before any re-seal or cell diff --git a/messagefoundry/pipeline/secret_rotation.py b/messagefoundry/pipeline/secret_rotation.py index 3612cc6467..94be98b6b4 100644 --- a/messagefoundry/pipeline/secret_rotation.py +++ b/messagefoundry/pipeline/secret_rotation.py @@ -279,8 +279,10 @@ async def open_rotation_record( meta_store: SecretRotationMetaStore, key_id: str, day: str, *, keep_another: bool = False ) -> bool: """Record that ``rotate-key`` is moving this store to the DEK ``key_id``, before either data - pass, so a crash or a failed step in one leaves the record standing (BACKLOG #3737). The store - open has run by then and can already have written under the key. A ``serve`` start calls it too, + pass, so a crash or a failed step in one leaves the record standing (BACKLOG #3737). + ``rotate-key`` calls it from the store open's ``before_first_write`` hook (BACKLOG #4936), so + before the open's first write under the key; only the open's schema step, which writes nothing + under the key, can have run before it. A ``serve`` start calls it too, for a changed key the keyed audit chain has not rolled to (BACKLOG #4942), with ``keep_another``: it then writes nothing over a record open for another key. That record holds its own key, which a run may have rolled the chain to with an upload still under the prior key, so replacing it would From 839f0c384a15b32448f0f5115e0789c3343487ab Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 18:00:11 -0600 Subject: [PATCH 07/41] What can run before rotate-key's record opens is an at-least list Review round-71d04055 on engine PR 2991 (BACKLOG #4934, #4936): "only the schema step" was false; the pin check and, on SQL Server, the database option switch run before the before_first_write hook too, none under the key. The docstring also says a serve start opens the record after its own open wrote. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- docs/PHI.md | 5 +++-- messagefoundry/pipeline/secret_rotation.py | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/PHI.md b/docs/PHI.md index 73c18233d8..269b211172 100644 --- a/docs/PHI.md +++ b/docs/PHI.md @@ -739,8 +739,9 @@ for defense-in-depth without swapping the `aiosqlite` connector. the volume is mounted and the path is the one `serve` uses. **A `serve` start does not stamp a changed key while the rotation record names it or cannot be read (BACKLOG #3737).** `rotate-key` opens a rotation record in `secret_rotation_meta` inside the store open, - before the open's first write under the key (BACKLOG #4936; only the open's schema step, which - writes nothing under the key, can run before it), and closes it only when a run completes, + before the open's first write under the key (BACKLOG #4936; steps that write nothing under the + key can run before it, at least the schema step and, on SQL Server, the database option switch), + and closes it only when a run completes, so a run killed or failed after that point leaves it open. **A sealed record under a key the run was not given stops it (vault BACKLOG #5635).** When a user, role, open approval or directory-map seal names a key the run does not hold, a pre-flight read finds it before any re-seal or cell diff --git a/messagefoundry/pipeline/secret_rotation.py b/messagefoundry/pipeline/secret_rotation.py index 94be98b6b4..685afff3ca 100644 --- a/messagefoundry/pipeline/secret_rotation.py +++ b/messagefoundry/pipeline/secret_rotation.py @@ -281,8 +281,9 @@ async def open_rotation_record( """Record that ``rotate-key`` is moving this store to the DEK ``key_id``, before either data pass, so a crash or a failed step in one leaves the record standing (BACKLOG #3737). ``rotate-key`` calls it from the store open's ``before_first_write`` hook (BACKLOG #4936), so - before the open's first write under the key; only the open's schema step, which writes nothing - under the key, can have run before it. A ``serve`` start calls it too, + before the open's first write under the key. Steps that write nothing under the key can run + before it, at least the schema step, the store id pin check and, on SQL Server, the database + option switch. A ``serve`` start calls it too, after its own open has written under the key, for a changed key the keyed audit chain has not rolled to (BACKLOG #4942), with ``keep_another``: it then writes nothing over a record open for another key. That record holds its own key, which a run may have rolled the chain to with an upload still under the prior key, so replacing it would From 19e0049a9a53ef2b0b2ab60008175b62a202d616 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 18:14:28 -0600 Subject: [PATCH 08/41] The rotation record changelog fragment says where rotate-key opens it now changelog.d/3737.security.md said "after its store open", false since this PR's BACKLOG #4936 change moved the open into the store open's before_first_write hook, before the open's first write under the key, as the BACKLOG #4934 fragment says. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/3737.security.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index 6af3c3ad0c..861c35184f 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -1,6 +1,7 @@ - **`rotate-key` keeps a rotation record, so a `serve` start no longer stamps a key once a run on it has opened that record and not finished.** The command opens the record in - `secret_rotation_meta` before either data pass, after its store open, and closes it only when a + `secret_rotation_meta` before either data pass, from its store open's `before_first_write` hook, + before that open's first write under the key (`BACKLOG #4936`), and closes it only when a run completes. When a start finds that the key in use differs from its stamp, or has none, and the record names that key or cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a WARNING. A run killed inside the store open, before the record opens, leaves no From 0d65b2cadbd7ab05bd598ae9bfa789994ec3831d Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 20:23:20 -0600 Subject: [PATCH 09/41] Rotation record and pin check docs match the open order and the serve caller Lander hold at 19e0049a9a on engine PR 2991, items 1-4 (BACKLOG #4934, #4936): the open_rotation_record docstring names the serve caller's rolled chain case; the PinCheck docstring puts the before_first_write hook between the pin check and the first write under the key; the rotation record changelog fragment drops the commas that made the whole open precede the record, and says no record opens when the stamp already names the key. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/3737.security.md | 4 ++-- messagefoundry/pipeline/secret_rotation.py | 4 +++- messagefoundry/store/store.py | 9 +++++---- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index 861c35184f..6689fb8d3c 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -4,8 +4,8 @@ before that open's first write under the key (`BACKLOG #4936`), and closes it only when a run completes. When a start finds that the key in use differs from its stamp, or has none, and the record names that key or cannot be read, `serve` writes no stamp, does not reset the key's age, - and logs a WARNING. A run killed inside the store open, before the record opens, leaves no - record. On a store with a keyed audit chain, a start that then finds the key changed opens one + and logs a WARNING. A run killed inside the store open before the record opens leaves no + record. No record opens when the key's stamp already names that key. On a store with a keyed audit chain, a start that then finds the key changed opens one itself, as does a start before the first run; the `BACKLOG #4942` entry says when. **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload pass opens only a regular file, and reads whole only one of a size an upload can be.** A FIFO no diff --git a/messagefoundry/pipeline/secret_rotation.py b/messagefoundry/pipeline/secret_rotation.py index 685afff3ca..5553a8b9f6 100644 --- a/messagefoundry/pipeline/secret_rotation.py +++ b/messagefoundry/pipeline/secret_rotation.py @@ -284,7 +284,9 @@ async def open_rotation_record( before the open's first write under the key. Steps that write nothing under the key can run before it, at least the schema step, the store id pin check and, on SQL Server, the database option switch. A ``serve`` start calls it too, after its own open has written under the key, - for a changed key the keyed audit chain has not rolled to (BACKLOG #4942), with ``keep_another``: + through :func:`reconcile_rotation_meta`, for a changed key the keyed audit chain has not rolled + to, or one it has rolled to while a record stands open for another key (BACKLOG #4942), with + ``keep_another``: it then writes nothing over a record open for another key. That record holds its own key, which a run may have rolled the chain to with an upload still under the prior key, so replacing it would let a later start on that key stamp it. ``rotate-key`` replaces it, keeping its dates. A start on diff --git a/messagefoundry/store/store.py b/messagefoundry/store/store.py index 144d8c5812..23bd7f0b17 100644 --- a/messagefoundry/store/store.py +++ b/messagefoundry/store/store.py @@ -6910,10 +6910,11 @@ async def run_open_store_hook(host: object) -> None: class PinCheck: - """Whether the open in progress has passed its store id pin check (:func:`check_open_store_pin`), - the last step before its first write under the store key. Under ``auto`` schema management the - open may already have committed its schema, and on SQL Server altered the database's options, - before it; neither is under the key. A mutable cell rather than a flag in a context variable, + """Whether the open in progress has passed its store id pin check (:func:`check_open_store_pin`). + The open's ``before_first_write`` hook (:func:`run_open_store_hook`, BACKLOG #4936) runs after + the pin check, and the open's first write under the store key after both. Under ``auto`` + schema management the open may already have committed its schema, and on SQL Server altered + the database's options, before the pin check; neither is under the key. A mutable cell rather than a flag in a context variable, so a pass recorded under a copied context still reaches the open that set it (vault BACKLOG #8119 item 2).""" From 8abc2c7b07c5b08c21abed2fe3c1076af4bd2e9c Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 20:36:37 -0600 Subject: [PATCH 10/41] The serve caller list is an at-least list; the no-record sentence moves up Review round-98cd5c1d on engine PR 2991 (BACKLOG #4934, #4936): the docstring's two cases of a serve start opening the record read as complete, and the fragment's new no-record sentence broke the "then" of the next one. The docstring now defers to reconcile_rotation_meta with an at-least list, and the sentence sits after the open and close it qualifies. Lines rewrapped. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/3737.security.md | 8 ++++---- messagefoundry/pipeline/secret_rotation.py | 6 +++--- messagefoundry/store/store.py | 6 +++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index 6689fb8d3c..2eddbece13 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -2,10 +2,10 @@ it has opened that record and not finished.** The command opens the record in `secret_rotation_meta` before either data pass, from its store open's `before_first_write` hook, before that open's first write under the key (`BACKLOG #4936`), and closes it only when a - run completes. When a start finds that the key in use differs from its stamp, or has none, and the - record names that key or cannot be read, `serve` writes no stamp, does not reset the key's age, - and logs a WARNING. A run killed inside the store open before the record opens leaves no - record. No record opens when the key's stamp already names that key. On a store with a keyed audit chain, a start that then finds the key changed opens one + run completes. No record opens when the key's stamp already names that key. When a start finds + that the key in use differs from its stamp, or has none, and the record names that key or + cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a WARNING. A run killed inside the store open before the record opens leaves no + record. On a store with a keyed audit chain, a start that then finds the key changed opens one itself, as does a start before the first run; the `BACKLOG #4942` entry says when. **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload pass opens only a regular file, and reads whole only one of a size an upload can be.** A FIFO no diff --git a/messagefoundry/pipeline/secret_rotation.py b/messagefoundry/pipeline/secret_rotation.py index 5553a8b9f6..f30707e4b3 100644 --- a/messagefoundry/pipeline/secret_rotation.py +++ b/messagefoundry/pipeline/secret_rotation.py @@ -284,9 +284,9 @@ async def open_rotation_record( before the open's first write under the key. Steps that write nothing under the key can run before it, at least the schema step, the store id pin check and, on SQL Server, the database option switch. A ``serve`` start calls it too, after its own open has written under the key, - through :func:`reconcile_rotation_meta`, for a changed key the keyed audit chain has not rolled - to, or one it has rolled to while a record stands open for another key (BACKLOG #4942), with - ``keep_another``: + through :func:`reconcile_rotation_meta`, which says when: at least for a changed key the keyed + audit chain has not rolled to, and for one it has rolled to while a record stands open for + another key (BACKLOG #4942). It passes ``keep_another``: it then writes nothing over a record open for another key. That record holds its own key, which a run may have rolled the chain to with an upload still under the prior key, so replacing it would let a later start on that key stamp it. ``rotate-key`` replaces it, keeping its dates. A start on diff --git a/messagefoundry/store/store.py b/messagefoundry/store/store.py index 23bd7f0b17..ae61c5ccd0 100644 --- a/messagefoundry/store/store.py +++ b/messagefoundry/store/store.py @@ -6914,9 +6914,9 @@ class PinCheck: The open's ``before_first_write`` hook (:func:`run_open_store_hook`, BACKLOG #4936) runs after the pin check, and the open's first write under the store key after both. Under ``auto`` schema management the open may already have committed its schema, and on SQL Server altered - the database's options, before the pin check; neither is under the key. A mutable cell rather than a flag in a context variable, - so a pass recorded under a copied context still reaches the open that set it (vault BACKLOG - #8119 item 2).""" + the database's options, before the pin check; neither is under the key. A mutable cell rather + than a flag in a context variable, so a pass recorded under a copied context still reaches the + open that set it (vault BACKLOG #8119 item 2).""" __slots__ = ("passed",) From 674323371178696d888a3c830d5dcdd4ecdba989 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 20:50:30 -0600 Subject: [PATCH 11/41] Four rotation record statements say only what the code does Review round-3caf5116 on engine PR 2991 (BACKLOG #4934, #4936): the serve caller case excludes a record that names the key or cannot be read; on a keyed chain a record for another key holds the start, rolled or not; the fragment's headline holds while the record still names the key; a run killed before the hook opens no record, which is not leaving none. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/3737.security.md | 6 +++--- messagefoundry/pipeline/secret_rotation.py | 12 +++++++----- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index 2eddbece13..04bd0fe171 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -1,10 +1,10 @@ - **`rotate-key` keeps a rotation record, so a `serve` start no longer stamps a key once a run on - it has opened that record and not finished.** The command opens the record in - `secret_rotation_meta` before either data pass, from its store open's `before_first_write` hook, + it has opened that record and not finished, while the record still names that key.** The + command opens the record in `secret_rotation_meta` before either data pass, from its store open's `before_first_write` hook, before that open's first write under the key (`BACKLOG #4936`), and closes it only when a run completes. No record opens when the key's stamp already names that key. When a start finds that the key in use differs from its stamp, or has none, and the record names that key or - cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a WARNING. A run killed inside the store open before the record opens leaves no + cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a WARNING. A run killed inside the store open before the record opens opens no record. On a store with a keyed audit chain, a start that then finds the key changed opens one itself, as does a start before the first run; the `BACKLOG #4942` entry says when. **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload diff --git a/messagefoundry/pipeline/secret_rotation.py b/messagefoundry/pipeline/secret_rotation.py index f30707e4b3..2c4e2b0dc0 100644 --- a/messagefoundry/pipeline/secret_rotation.py +++ b/messagefoundry/pipeline/secret_rotation.py @@ -285,8 +285,9 @@ async def open_rotation_record( before it, at least the schema step, the store id pin check and, on SQL Server, the database option switch. A ``serve`` start calls it too, after its own open has written under the key, through :func:`reconcile_rotation_meta`, which says when: at least for a changed key the keyed - audit chain has not rolled to, and for one it has rolled to while a record stands open for - another key (BACKLOG #4942). It passes ``keep_another``: + audit chain has not rolled to, unless the record already names that key or cannot be read, + and for one it has rolled to while a record stands open for another key (BACKLOG #4942). It + passes ``keep_another``: it then writes nothing over a record open for another key. That record holds its own key, which a run may have rolled the chain to with an upload still under the prior key, so replacing it would let a later start on that key stamp it. ``rotate-key`` replaces it, keeping its dates. A start on @@ -372,9 +373,10 @@ def _rotation_hold(record: SecretRotationMetaRow | None, fingerprint: str) -> st It holds for a record naming this key. It also holds for a record that cannot be read, because this function fails toward NOT stamping: an unstamped key over-reports its age, and a false stamp is what would lift an overdue refusal. A record naming another key does not hold here: the key - in use was changed by hand. Where the keyed audit chain has not rolled to that key, - :func:`reconcile_rotation_meta` holds it anyway and keeps the record (BACKLOG #4942); elsewhere - it stamps as any such change does.""" + in use was changed by hand. On a store with a keyed audit chain, + :func:`reconcile_rotation_meta` holds it anyway and keeps the record (BACKLOG #4942), whether or + not the chain has rolled to the key in use; on a store with none it stamps as any such change + does.""" if record is None or record.fingerprint == _ROTATION_RECORD_CLOSED: return None # A key id has one length, so one of any other length is a damaged record, not another key. From 6747e24957c58015e21748ece9a515ed937d7f97 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 21:03:11 -0600 Subject: [PATCH 12/41] The rotation record changelog fragment states when a record opens, closes and holds Lander hold at 8abc2c7b07 on engine PR 2991 (BACKLOG #4934, #4936): the fragment's record statements are rewritten against the code at the head, by store kind: the open from the before_first_write hook and when it opens none, the close only after both passes, no skip and a roll that does not refuse (same-key runs included), what a kill before or after the hook leaves, the unknown-scheme stamp, and the keyed-chain holds versus the stamp on a store with no keyed chain. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/3737.security.md | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index 04bd0fe171..702061a6b0 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -1,12 +1,19 @@ -- **`rotate-key` keeps a rotation record, so a `serve` start no longer stamps a key once a run on - it has opened that record and not finished, while the record still names that key.** The - command opens the record in `secret_rotation_meta` before either data pass, from its store open's `before_first_write` hook, - before that open's first write under the key (`BACKLOG #4936`), and closes it only when a - run completes. No record opens when the key's stamp already names that key. When a start finds - that the key in use differs from its stamp, or has none, and the record names that key or - cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a WARNING. A run killed inside the store open before the record opens opens no - record. On a store with a keyed audit chain, a start that then finds the key changed opens one - itself, as does a start before the first run; the `BACKLOG #4942` entry says when. **Once a run +- **`rotate-key` keeps a rotation record, and a `serve` start does not stamp a key while that + record names it.** The command opens the record in `secret_rotation_meta` inside its store + open, before that open's first write under the key (`BACKLOG #4936`). It opens none when the + store-key stamp already names the key the run moves to. A record already open for another key + is replaced, keeping its dates. The command closes the record only when a run gets through both + data passes with no upload skipped and its audit roll does not refuse; a run on the key already + in use closes an open record too. So a run that stops short of that leaves an open record open, + and so does one killed inside the store open after the record opened. A run killed before that point + opens no record and leaves any record an earlier run opened as it was. When a start finds that + the key in use differs from its stamp, has none, or has one under a scheme this release does not + know, and the record names that key or cannot be read, `serve` writes no stamp, does not reset + the key's age, and logs a WARNING. On a store with a keyed audit chain, a start also holds such a + key at least where the open does not find the chain rolled to it, and opens the record for it + if none is open, and where a record stands open for another key, which it keeps. On a store with + no keyed audit chain, a record naming another key holds nothing, and the start stamps the key. + The `BACKLOG #4942` entry says more. **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload pass opens only a regular file, and reads whole only one of a size an upload can be.** A FIFO no longer blocks it, a symlink is a skip and is not followed, and a file too large for an upload is a From 43d07a41f9e8f0096da4a2c1ef05aedb2bad56a2 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 21:17:28 -0600 Subject: [PATCH 13/41] The rotation record fragment conditions the replace, the close and the start hold Review round-d7ec0bc2 on engine PR 2991 (BACKLOG #4934, #4936): a run whose stamp names its key replaces no record; no run closes the record on a store whose audit rows open with no genesis row; a kill before the hook leaves any open record, a start's included; the keyed-chain start holds are two separate cases, the kept record and the start-opened one. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/3737.security.md | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index 702061a6b0..fe86e75169 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -1,19 +1,22 @@ - **`rotate-key` keeps a rotation record, and a `serve` start does not stamp a key while that record names it.** The command opens the record in `secret_rotation_meta` inside its store - open, before that open's first write under the key (`BACKLOG #4936`). It opens none when the - store-key stamp already names the key the run moves to. A record already open for another key - is replaced, keeping its dates. The command closes the record only when a run gets through both - data passes with no upload skipped and its audit roll does not refuse; a run on the key already - in use closes an open record too. So a run that stops short of that leaves an open record open, - and so does one killed inside the store open after the record opened. A run killed before that point - opens no record and leaves any record an earlier run opened as it was. When a start finds that - the key in use differs from its stamp, has none, or has one under a scheme this release does not - know, and the record names that key or cannot be read, `serve` writes no stamp, does not reset - the key's age, and logs a WARNING. On a store with a keyed audit chain, a start also holds such a - key at least where the open does not find the chain rolled to it, and opens the record for it - if none is open, and where a record stands open for another key, which it keeps. On a store with - no keyed audit chain, a record naming another key holds nothing, and the start stamps the key. - The `BACKLOG #4942` entry says more. **Once a run + open, before that open's first write under the key (`BACKLOG #4936`), unless the store-key + stamp already names the key the run moves to. When it opens one, a record already open for + another key is replaced, keeping its dates; when it does not, any open record is left in place. + The command closes the record only when a run gets through both data passes with no upload + skipped and its audit roll does not refuse, and a run on the key already in use that gets that + far closes an open record too. On a store whose audit rows open with no genesis row, the roll + refuses on every run, so no run closes the record there. A run that stops short leaves an open + record open, and so does one killed inside the store open after the record opened. A run killed + before that point opens no record and leaves any record already open, whoever opened it, as it + was. When a start finds that the key in use differs from its stamp, has none, or has one under a + scheme this release does not know, and the record names that key or cannot be read, `serve` + writes no stamp, does not reset the key's age, and logs a WARNING. On a store with a keyed audit + chain, a start holds such a key in at least two more cases. Where a record stands open for + another key, the start keeps that record and holds the key. Where none is open and the open does + not find the chain rolled to the key, the start opens the record for the key itself. On a store + with no keyed audit chain, a record naming another key holds nothing, and the start stamps the + key. The `BACKLOG #4942` entry says more. **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload pass opens only a regular file, and reads whole only one of a size an upload can be.** A FIFO no longer blocks it, a symlink is a skip and is not followed, and a file too large for an upload is a From 99fa0f70522725fbbd3a8ceb579e0d50d7790d58 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 21:33:48 -0600 Subject: [PATCH 14/41] The rotation record fragment names the refusal, the unreadable record and the no-genesis store Review round-17f71f50 on engine PR 2991 (BACKLOG #4934, #4936): a start that would open the record refuses instead when the current range key is missing; an unreadable record is replaced too; the store with no keyed chain is the one whose rows open with no genesis row, named once; a closed record proves no complete re-seal. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/3737.security.md | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index fe86e75169..c9cb23e9b5 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -2,21 +2,24 @@ record names it.** The command opens the record in `secret_rotation_meta` inside its store open, before that open's first write under the key (`BACKLOG #4936`), unless the store-key stamp already names the key the run moves to. When it opens one, a record already open for - another key is replaced, keeping its dates; when it does not, any open record is left in place. - The command closes the record only when a run gets through both data passes with no upload - skipped and its audit roll does not refuse, and a run on the key already in use that gets that - far closes an open record too. On a store whose audit rows open with no genesis row, the roll - refuses on every run, so no run closes the record there. A run that stops short leaves an open - record open, and so does one killed inside the store open after the record opened. A run killed - before that point opens no record and leaves any record already open, whoever opened it, as it - was. When a start finds that the key in use differs from its stamp, has none, or has one under a - scheme this release does not know, and the record names that key or cannot be read, `serve` - writes no stamp, does not reset the key's age, and logs a WARNING. On a store with a keyed audit - chain, a start holds such a key in at least two more cases. Where a record stands open for - another key, the start keeps that record and holds the key. Where none is open and the open does - not find the chain rolled to the key, the start opens the record for the key itself. On a store - with no keyed audit chain, a record naming another key holds nothing, and the start stamps the - key. The `BACKLOG #4942` entry says more. **Once a run + another key, or one that cannot be read, is replaced, keeping its dates; when it does not, any + open record is left in place. The command closes the record only when a run gets through both + data passes with no upload skipped and its audit roll does not refuse, and a run on the key + already in use that gets that far closes an open record too. A closed record does not prove that + nothing is left under the prior key; [PHI.md](../docs/PHI.md) names cases. A run that stops + short leaves an open record open, and so does one killed inside the store open after the record + opened. A run killed before that point opens no record and leaves any record already open, + whoever opened it, as it was. When a start finds that the key in use differs from its stamp, has + none, or has one under a scheme this release does not know, and the record names that key or + cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a WARNING. On a + store with a keyed audit chain, a start holds such a key in at least two more cases. Where a + record stands open for another key, the start keeps that record and holds the key. Where none is + open and the open does not find the chain rolled to the key, the start opens the record for the + key itself, unless it refuses to start because the key of the chain's current range is not + configured (`BACKLOG #4934`). A store with a key whose audit rows open with no genesis row has no + keyed chain: there the roll refuses on every run, so no run closes the record, and a record + naming another key holds nothing, so a start stamps the key. The `BACKLOG #4942` entry says + more. **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload pass opens only a regular file, and reads whole only one of a size an upload can be.** A FIFO no longer blocks it, a symlink is a skip and is not followed, and a file too large for an upload is a From 0f339d61cb73818ee4a662d94941220b59256a43 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 21:49:14 -0600 Subject: [PATCH 15/41] The rotation record fragment puts the unrolled-key refusal ahead of every start case Review round-adaa4d20 on engine PR 2991 (BACKLOG #4934, #4936): the start refusal for a missing range key fires whatever the record says, so it now precedes the start behaviour instead of qualifying one case; on a store whose rows open with no genesis row the record holds its key until a run on another key replaces it. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/3737.security.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index c9cb23e9b5..25cbc66a34 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -9,17 +9,18 @@ nothing is left under the prior key; [PHI.md](../docs/PHI.md) names cases. A run that stops short leaves an open record open, and so does one killed inside the store open after the record opened. A run killed before that point opens no record and leaves any record already open, - whoever opened it, as it was. When a start finds that the key in use differs from its stamp, has - none, or has one under a scheme this release does not know, and the record names that key or - cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a WARNING. On a - store with a keyed audit chain, a start holds such a key in at least two more cases. Where a + whoever opened it, as it was. A `serve` start that refuses because the chain's current range is + under a key that is not configured (`BACKLOG #4934`) refuses whatever the record says, and none of + the start behaviour below runs. Otherwise, when a start finds that the key in use differs from its + stamp, has none, or has one under a scheme this release does not know, and the record names that + key or cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a + WARNING. On a store with a keyed audit chain, a start holds such a key in at least two more cases. Where a record stands open for another key, the start keeps that record and holds the key. Where none is open and the open does not find the chain rolled to the key, the start opens the record for the - key itself, unless it refuses to start because the key of the chain's current range is not - configured (`BACKLOG #4934`). A store with a key whose audit rows open with no genesis row has no - keyed chain: there the roll refuses on every run, so no run closes the record, and a record - naming another key holds nothing, so a start stamps the key. The `BACKLOG #4942` entry says - more. **Once a run + key itself. A store with a key whose audit rows open with no genesis row has no keyed chain. + There the roll refuses on every run, so no run closes the record, and it holds its key until a + run on another key replaces it; a record naming another key holds nothing, so a start stamps the + key. The `BACKLOG #4942` entry says more. **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload pass opens only a regular file, and reads whole only one of a size an upload can be.** A FIFO no longer blocks it, a symlink is a skip and is not followed, and a file too large for an upload is a From 8949b96029a29411f33a92e9afbf98f8e9576a1c Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 22:05:11 -0600 Subject: [PATCH 16/41] The rotation record fragment names the host provider and the start's own open Review round-41c1a6ae on engine PR 2991 (BACKLOG #4934, #4936): under the host key provider the pin check refuses every open of a store whose rows open with no genesis row, so the roll statement excludes it; a refused store open opens no record either; the chain check is the start's own open. Paragraph rewrapped. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/3737.security.md | 45 ++++++++++++++++++------------------ 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index 25cbc66a34..5e32e46a8b 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -1,26 +1,27 @@ -- **`rotate-key` keeps a rotation record, and a `serve` start does not stamp a key while that - record names it.** The command opens the record in `secret_rotation_meta` inside its store - open, before that open's first write under the key (`BACKLOG #4936`), unless the store-key - stamp already names the key the run moves to. When it opens one, a record already open for - another key, or one that cannot be read, is replaced, keeping its dates; when it does not, any - open record is left in place. The command closes the record only when a run gets through both - data passes with no upload skipped and its audit roll does not refuse, and a run on the key - already in use that gets that far closes an open record too. A closed record does not prove that - nothing is left under the prior key; [PHI.md](../docs/PHI.md) names cases. A run that stops - short leaves an open record open, and so does one killed inside the store open after the record - opened. A run killed before that point opens no record and leaves any record already open, - whoever opened it, as it was. A `serve` start that refuses because the chain's current range is - under a key that is not configured (`BACKLOG #4934`) refuses whatever the record says, and none of - the start behaviour below runs. Otherwise, when a start finds that the key in use differs from its - stamp, has none, or has one under a scheme this release does not know, and the record names that - key or cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a - WARNING. On a store with a keyed audit chain, a start holds such a key in at least two more cases. Where a +- **`rotate-key` keeps a rotation record, and a `serve` start does not stamp a key while that record + names it.** The command opens the record in `secret_rotation_meta` inside its store open, before + that open's first write under the key (`BACKLOG #4936`), unless the store-key stamp already names + the key the run moves to. When it opens one, a record already open for another key, or one that + cannot be read, is replaced, keeping its dates; when it does not, any open record is left in + place. The command closes the record only when a run gets through both data passes with no upload + skipped and its audit roll does not refuse, and a run on the key already in use that gets that far + closes an open record too. A closed record does not prove that nothing is left under the prior + key; [PHI.md](../docs/PHI.md) names cases. A run that stops short leaves an open record open, and + so does one killed inside the store open after the record opened. A run killed, or whose store + open refuses, before that point opens no record and leaves any record already open, whoever opened + it, as it was. A `serve` start that refuses because the chain's current range is under a key that + is not configured (`BACKLOG #4934`) refuses whatever the record says, and none of the start + behaviour below runs. Otherwise, when a start finds that the key in use differs from its stamp, + has none, or has one under a scheme this release does not know, and the record names that key or + cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a WARNING. On a + store with a keyed audit chain, a start holds such a key in at least two more cases. Where a record stands open for another key, the start keeps that record and holds the key. Where none is - open and the open does not find the chain rolled to the key, the start opens the record for the - key itself. A store with a key whose audit rows open with no genesis row has no keyed chain. - There the roll refuses on every run, so no run closes the record, and it holds its key until a - run on another key replaces it; a record naming another key holds nothing, so a start stamps the - key. The `BACKLOG #4942` entry says more. **Once a run + open and the start's own store open does not find the chain rolled to the key, the start opens the + record for the key itself. A store with a key whose audit rows open with no genesis row has no + keyed chain. There, except under `key_provider = "host"`, whose store id pin check refuses every + open of such a store, the roll refuses on every run, so no run closes the record, and it holds its + key until a run on another key replaces it; a record naming another key holds nothing, so a start + stamps the key. The `BACKLOG #4942` entry says more. **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload pass opens only a regular file, and reads whole only one of a size an upload can be.** A FIFO no longer blocks it, a symlink is a skip and is not followed, and a file too large for an upload is a From 309d478dc066394277e1cfd4304fdc8a9d453737 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 22:20:59 -0600 Subject: [PATCH 17/41] The rotation record fragment states the start refusal's full condition Review round-1ff6c1cf on engine PR 2991 (BACKLOG #4934, #4936): the start refusal is stated with all its conditions and the override; the no-record case for the command is an at-least list. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/3737.security.md | 46 +++++++++++++++++++----------------- 1 file changed, 24 insertions(+), 22 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index 5e32e46a8b..b81552c345 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -1,27 +1,29 @@ - **`rotate-key` keeps a rotation record, and a `serve` start does not stamp a key while that record names it.** The command opens the record in `secret_rotation_meta` inside its store open, before - that open's first write under the key (`BACKLOG #4936`), unless the store-key stamp already names - the key the run moves to. When it opens one, a record already open for another key, or one that - cannot be read, is replaced, keeping its dates; when it does not, any open record is left in - place. The command closes the record only when a run gets through both data passes with no upload - skipped and its audit roll does not refuse, and a run on the key already in use that gets that far - closes an open record too. A closed record does not prove that nothing is left under the prior - key; [PHI.md](../docs/PHI.md) names cases. A run that stops short leaves an open record open, and - so does one killed inside the store open after the record opened. A run killed, or whose store - open refuses, before that point opens no record and leaves any record already open, whoever opened - it, as it was. A `serve` start that refuses because the chain's current range is under a key that - is not configured (`BACKLOG #4934`) refuses whatever the record says, and none of the start - behaviour below runs. Otherwise, when a start finds that the key in use differs from its stamp, - has none, or has one under a scheme this release does not know, and the record names that key or - cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a WARNING. On a - store with a keyed audit chain, a start holds such a key in at least two more cases. Where a - record stands open for another key, the start keeps that record and holds the key. Where none is - open and the start's own store open does not find the chain rolled to the key, the start opens the - record for the key itself. A store with a key whose audit rows open with no genesis row has no - keyed chain. There, except under `key_provider = "host"`, whose store id pin check refuses every - open of such a store, the roll refuses on every run, so no run closes the record, and it holds its - key until a run on another key replaces it; a record naming another key holds nothing, so a start - stamps the key. The `BACKLOG #4942` entry says more. **Once a run + that open's first write under the key (`BACKLOG #4936`). It opens none at least when the store-key + stamp already names the key the run moves to. When it opens one, a record already open for another + key, or one that cannot be read, is replaced, keeping its dates; when it does not, any open record + is left in place. The command closes the record only when a run gets through both data passes with + no upload skipped and its audit roll does not refuse, and a run on the key already in use that + gets that far closes an open record too. A closed record does not prove that nothing is left under + the prior key; [PHI.md](../docs/PHI.md) names cases. A run that stops short leaves an open record + open, and so does one killed inside the store open after the record opened. A run killed, or whose + store open refuses, before that point opens no record and leaves any record already open, whoever + opened it, as it was. A `serve` start that refuses under `BACKLOG #4934` (the key in use differs + from its stamp, has none, or has one under a scheme this release does not know, the keyed chain + has not rolled to it, the key of the chain's current range is not configured, and + `[secret_rotation].allow_lost_audit_range_key` is off) refuses whatever the record says, and none + of the start behaviour below runs. Otherwise, when a start finds that the key in use differs from + its stamp, has none, or has one under a scheme this release does not know, and the record names + that key or cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a + WARNING. On a store with a keyed audit chain, a start holds such a key in at least two more cases. + Where a record stands open for another key, the start keeps that record and holds the key. Where + none is open and the start's own store open does not find the chain rolled to the key, the start + opens the record for the key itself. A store with a key whose audit rows open with no genesis row + has no keyed chain. There, except under the `host` key provider, whose store id pin check refuses + every open of such a store, the roll refuses on every run, so no run closes the record, and it + holds its key until a run on another key replaces it; a record naming another key holds nothing, + so a start stamps the key. The `BACKLOG #4942` entry says more. **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload pass opens only a regular file, and reads whole only one of a size an upload can be.** A FIFO no longer blocks it, a symlink is a skip and is not followed, and a file too large for an upload is a From ce66165839b9e20dac6a566ae85ce8474b97cdf6 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 22:36:58 -0600 Subject: [PATCH 18/41] On a no-genesis store only a run on an unstamped key releases the record Review round-5ca3ef97 on engine PR 2991 (BACKLOG #4934, #4936): a run on a key the stamp already names opens no record, so it does not replace one; the fragment now says which runs release a held key there. Proposed PR title: rotate-key opens its record inside the store open, and serve refuses an unrolled store key change (BACKLOG #4934, #4936) Proposed ledger banner: SHIPPED in engine PR 2991 --- changelog.d/3737.security.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index b81552c345..da4b96fcf8 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -22,8 +22,8 @@ opens the record for the key itself. A store with a key whose audit rows open with no genesis row has no keyed chain. There, except under the `host` key provider, whose store id pin check refuses every open of such a store, the roll refuses on every run, so no run closes the record, and it - holds its key until a run on another key replaces it; a record naming another key holds nothing, - so a start stamps the key. The `BACKLOG #4942` entry says more. **Once a run + holds its key until a run on a key the stamp does not name replaces it; a record naming another + key holds nothing, so a start stamps the key. The `BACKLOG #4942` entry says more. **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload pass opens only a regular file, and reads whole only one of a size an upload can be.** A FIFO no longer blocks it, a symlink is a skip and is not followed, and a file too large for an upload is a From 7b3d835803ca05713b7c048efd973e425bb97b8f Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 23:24:45 -0600 Subject: [PATCH 19/41] The 3737 and 4934 changelog fragments state the rotation record and the start refusal in a short headline Eight review rounds each found a new false statement in the 3737 fragment, because it listed every case of when the rotation record opens, closes, holds a start or is left behind. Both fragments now say only what changed, and point to docs/PHI.md and the open_rotation_record docstring for the conditions. The 4934 fragment drops three statements that were false in some case: that the next serve start stamps nothing after a killed run, that a kill in the schema step leaves no record, and that a refused start renews nothing. Fragments only; no code changes. BACKLOG #4934, #3737 --- changelog.d/3737.security.md | 29 +++-------------------------- changelog.d/4934.security.md | 28 ++++++++++------------------ 2 files changed, 13 insertions(+), 44 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index da4b96fcf8..d6f1545799 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -1,29 +1,6 @@ -- **`rotate-key` keeps a rotation record, and a `serve` start does not stamp a key while that record - names it.** The command opens the record in `secret_rotation_meta` inside its store open, before - that open's first write under the key (`BACKLOG #4936`). It opens none at least when the store-key - stamp already names the key the run moves to. When it opens one, a record already open for another - key, or one that cannot be read, is replaced, keeping its dates; when it does not, any open record - is left in place. The command closes the record only when a run gets through both data passes with - no upload skipped and its audit roll does not refuse, and a run on the key already in use that - gets that far closes an open record too. A closed record does not prove that nothing is left under - the prior key; [PHI.md](../docs/PHI.md) names cases. A run that stops short leaves an open record - open, and so does one killed inside the store open after the record opened. A run killed, or whose - store open refuses, before that point opens no record and leaves any record already open, whoever - opened it, as it was. A `serve` start that refuses under `BACKLOG #4934` (the key in use differs - from its stamp, has none, or has one under a scheme this release does not know, the keyed chain - has not rolled to it, the key of the chain's current range is not configured, and - `[secret_rotation].allow_lost_audit_range_key` is off) refuses whatever the record says, and none - of the start behaviour below runs. Otherwise, when a start finds that the key in use differs from - its stamp, has none, or has one under a scheme this release does not know, and the record names - that key or cannot be read, `serve` writes no stamp, does not reset the key's age, and logs a - WARNING. On a store with a keyed audit chain, a start holds such a key in at least two more cases. - Where a record stands open for another key, the start keeps that record and holds the key. Where - none is open and the start's own store open does not find the chain rolled to the key, the start - opens the record for the key itself. A store with a key whose audit rows open with no genesis row - has no keyed chain. There, except under the `host` key provider, whose store id pin check refuses - every open of such a store, the roll refuses on every run, so no run closes the record, and it - holds its key until a run on a key the stamp does not name replaces it; a record naming another - key holds nothing, so a start stamps the key. The `BACKLOG #4942` entry says more. **Once a run +- **`rotate-key` keeps a rotation record in `secret_rotation_meta`, and a `serve` start does not + stamp a changed store key while that record names it or cannot be read.** [PHI.md](../docs/PHI.md) + and the `open_rotation_record` docstring give the conditions. **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload pass opens only a regular file, and reads whole only one of a size an upload can be.** A FIFO no longer blocks it, a symlink is a skip and is not followed, and a file too large for an upload is a diff --git a/changelog.d/4934.security.md b/changelog.d/4934.security.md index 592d7147ea..e22cd86b9f 100644 --- a/changelog.d/4934.security.md +++ b/changelog.d/4934.security.md @@ -1,19 +1,11 @@ -- **`rotate-key` opens its rotation record inside the store open, before the open's first write - under the key, on every store.** `open_store` takes a `before_first_write` hook, run on every - writable open right after the store id pin check and before the salt bind, the cipher block - reserve, the at-rest sweep and any audit row; a read-only open does not run it. A run killed - anywhere after the hook leaves the record open, so the next `serve` start stamps nothing, on a - store with no keyed audit chain too. The open's schema step runs before the hook, so a kill - there leaves no record. - **`serve` refuses to start when the store key differs from its stamp, or has none, the keyed - audit chain has not been rolled to it, and the key the chain's current range is under is not - configured.** That start's first audit row would break the chain. `serve`, and - `supervise` before it renews the API certificate, refuse before they may replace the generated - pair, so a refused start renews nothing. `docs/SECURITY-LOOSENING.md` names the stores the - refusal does not cover, at least a `vault_transit` store. The refusal says to put the key that - range is under back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. - `[secret_rotation].allow_lost_audit_range_key = true` starts anyway, for a key that is truly - lost; it logs the refusal as a WARNING, the chain breaks at that start's first audit row, and it - is reported as a security loosening. Other commands that open the store and write an audit row, - such as the `admin-*` commands, do not refuse. +- **`rotate-key` now opens its rotation record, when it opens one, from a new `open_store` hook, + `before_first_write`, which a writable open runs before its first write under the store key.** A + read-only open does not run it. [PHI.md](../docs/PHI.md) and the `open_rotation_record` docstring + give the conditions. + **`serve` can now refuse to start on a changed store key the keyed audit chain cannot follow.** + The refusal names its conditions and a remedy, and `docs/SECURITY-LOOSENING.md` names stores it + does not cover. `[secret_rotation].allow_lost_audit_range_key = true` starts anyway, for a key + that is truly lost; it logs the refusal as a WARNING, the chain breaks at that start's first + audit row, and it is reported as a security loosening. Other commands that open the store and + write an audit row, such as the `admin-*` commands, do not refuse. (`BACKLOG #4934`, `BACKLOG #4936`) From e528a4dd776ac6ffc2b3396b95532552af158b21 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 23:36:47 -0600 Subject: [PATCH 20/41] The 4934 fragment and the unrolled-key refusal drop two statements a review round found false The fragment no longer says every command but serve does not refuse, since supervise refuses too; it names the admin-* commands only. It no longer says the refusal names its conditions. The refusal text says the store key in use is not confirmed by its stamp, which also holds for a stamp under a scheme this release does not know, where it said the key differs from its stamp or has none. No code behaviour changes. BACKLOG #4934 --- changelog.d/4934.security.md | 9 ++++----- messagefoundry/pipeline/secret_rotation.py | 4 ++-- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/changelog.d/4934.security.md b/changelog.d/4934.security.md index e22cd86b9f..27a2500e76 100644 --- a/changelog.d/4934.security.md +++ b/changelog.d/4934.security.md @@ -3,9 +3,8 @@ read-only open does not run it. [PHI.md](../docs/PHI.md) and the `open_rotation_record` docstring give the conditions. **`serve` can now refuse to start on a changed store key the keyed audit chain cannot follow.** - The refusal names its conditions and a remedy, and `docs/SECURITY-LOOSENING.md` names stores it - does not cover. `[secret_rotation].allow_lost_audit_range_key = true` starts anyway, for a key - that is truly lost; it logs the refusal as a WARNING, the chain breaks at that start's first - audit row, and it is reported as a security loosening. Other commands that open the store and - write an audit row, such as the `admin-*` commands, do not refuse. + `docs/SECURITY-LOOSENING.md` names stores the refusal does not cover. + `[secret_rotation].allow_lost_audit_range_key = true` starts anyway, for a key that is truly + lost; it logs the refusal as a WARNING, the chain breaks at that start's first audit row, and it + is reported as a security loosening. The `admin-*` commands do not refuse. (`BACKLOG #4934`, `BACKLOG #4936`) diff --git a/messagefoundry/pipeline/secret_rotation.py b/messagefoundry/pipeline/secret_rotation.py index 2c4e2b0dc0..a277094da3 100644 --- a/messagefoundry/pipeline/secret_rotation.py +++ b/messagefoundry/pipeline/secret_rotation.py @@ -409,8 +409,8 @@ class UnrolledStoreKeyRefused(RuntimeError): #: The refusal :func:`refuse_unrolled_key_change` raises, and the WARNING it logs under the override. _UNROLLED_REFUSAL = ( - "the store key in use differs from its stamp, or has none, the audit chain has not been " - "rolled to it, and the key the chain's current range is under is not configured. This start's " + "the store key in use is not confirmed by its stamp, the audit chain has not been rolled to " + "it, and the key the chain's current range is under is not configured. This start's " "first audit row would go into that range under another key, and the chain would break there, " "so no later `messagefoundry rotate-key` could roll it. Put the key that range is under back in " "MEFOR_STORE_ENCRYPTION_KEYS_RETIRED (it can be older than the last key, where an earlier key " From 8d815d327ff242e0b09d2c0968c9298ca2f3c57c Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Mon, 16 Nov 2026 23:55:22 -0600 Subject: [PATCH 21/41] The fragments point to their sources without claiming what they hold, and the refusal names its range test and drops a remedy that can fail Both fragments now say See, rather than that PHI.md and the open_rotation_record docstring give the conditions, which a review round found the two do not each do. The unrolled-key refusal now says the chain's current range is not under the key in use, which is what audit_chain_rolled tests, where it said the chain had not been rolled to the key. It no longer tells the operator to keep the new key active and run rotate-key until it exits 0, which never happens where the active key already keyed an earlier range; it says to run rotate-key before starting. No code behaviour changes. BACKLOG #4934 --- changelog.d/3737.security.md | 4 ++-- changelog.d/4934.security.md | 3 +-- messagefoundry/pipeline/secret_rotation.py | 8 ++++---- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index d6f1545799..f939374f51 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -1,6 +1,6 @@ - **`rotate-key` keeps a rotation record in `secret_rotation_meta`, and a `serve` start does not - stamp a changed store key while that record names it or cannot be read.** [PHI.md](../docs/PHI.md) - and the `open_rotation_record` docstring give the conditions. **Once a run + stamp a changed store key while that record names it or cannot be read.** See + [PHI.md](../docs/PHI.md) and the `open_rotation_record` docstring. **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload pass opens only a regular file, and reads whole only one of a size an upload can be.** A FIFO no longer blocks it, a symlink is a skip and is not followed, and a file too large for an upload is a diff --git a/changelog.d/4934.security.md b/changelog.d/4934.security.md index 27a2500e76..d6d42cd0b2 100644 --- a/changelog.d/4934.security.md +++ b/changelog.d/4934.security.md @@ -1,7 +1,6 @@ - **`rotate-key` now opens its rotation record, when it opens one, from a new `open_store` hook, `before_first_write`, which a writable open runs before its first write under the store key.** A - read-only open does not run it. [PHI.md](../docs/PHI.md) and the `open_rotation_record` docstring - give the conditions. + read-only open does not run it. See the `open_rotation_record` docstring. **`serve` can now refuse to start on a changed store key the keyed audit chain cannot follow.** `docs/SECURITY-LOOSENING.md` names stores the refusal does not cover. `[secret_rotation].allow_lost_audit_range_key = true` starts anyway, for a key that is truly diff --git a/messagefoundry/pipeline/secret_rotation.py b/messagefoundry/pipeline/secret_rotation.py index a277094da3..34b6581b8d 100644 --- a/messagefoundry/pipeline/secret_rotation.py +++ b/messagefoundry/pipeline/secret_rotation.py @@ -409,13 +409,13 @@ class UnrolledStoreKeyRefused(RuntimeError): #: The refusal :func:`refuse_unrolled_key_change` raises, and the WARNING it logs under the override. _UNROLLED_REFUSAL = ( - "the store key in use is not confirmed by its stamp, the audit chain has not been rolled to " - "it, and the key the chain's current range is under is not configured. This start's " + "the store key in use is not confirmed by its stamp, the audit chain's current range is not " + "under it, and the key that range is under is not configured. This start's " "first audit row would go into that range under another key, and the chain would break there, " "so no later `messagefoundry rotate-key` could roll it. Put the key that range is under back in " "MEFOR_STORE_ENCRYPTION_KEYS_RETIRED (it can be older than the last key, where an earlier key " - "change was never rolled; the store open's ERROR names the range's first row), keep the new key " - "active, and run `messagefoundry rotate-key` until it exits 0, then start. Only where that key " + "change was never rolled; the store open's ERROR names the range's first row) and run " + "`messagefoundry rotate-key` before you start. Only where that key " "is truly lost, set " "[secret_rotation].allow_lost_audit_range_key = true to start anyway: that is a security " "loosening, and the chain breaks at this start's first audit row." From a46ecd402d6966c674a99fa655e0a5963fbcebf0 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 00:21:06 -0600 Subject: [PATCH 22/41] SECURITY-LOOSENING.md and CONFIGURATION.md state the unrolled-key refusal as the refusal text does Both now give the refusal's condition in the words of _UNROLLED_REFUSAL: the key in use is not confirmed by its stamp and the chain's current range is not under it, where they said the key differs from its stamp or has none and the chain has not been rolled to it. Both say run rotate-key, not run it until it exits 0, which never happens where the active key already keyed an earlier range. SECURITY-LOOSENING.md drops 'A refused start renews no API certificate', and both drop the supervisor-before-renewal clause, which the generated_pair_may_be_replaced race makes false in some case. BACKLOG #4934 --- docs/CONFIGURATION.md | 2 +- docs/SECURITY-LOOSENING.md | 9 ++++----- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 06568767bd..3f5b5f787e 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -3005,7 +3005,7 @@ tracked; set `warn_days = 0` to disable the reminder. | `secret_max_age_days` | int | 365 | max age for the **non-DEK** tracked secret classes (connector/AD/SMTP/Vault/OIDC), alerted this many days after their last observed fingerprint change | | `enforce_grace_days` | int | 30 | under `[security].enforcement=enforce`, a DEK older than `store_key_max_age_days + this` escalates its rotation alert **and refuses to start** (see `enforce_store_key_expiry`) | | `enforce_store_key_expiry` | bool | `true` | under `[security].enforcement=enforce`, a store DEK past `store_key_max_age_days + enforce_grace_days` — or one whose age cannot be determined — **aborts engine start**. `false` keeps the alert, drops the refusal, and is reported as a **security loosening** | -| `allow_lost_audit_range_key` | bool | `false` | `serve` (and `supervise`, before it renews the API certificate) refuses to start when the store key differs from its stamp (or has none), the keyed audit chain has not been rolled to it, and the key the chain's current range is under is not configured, except on the stores [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) names, at least a `vault_transit` store: put that key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0. `true` starts anyway, for a key that is truly lost; the chain breaks at that start's first audit row, and it is reported as a **security loosening** | +| `allow_lost_audit_range_key` | bool | `false` | `serve` refuses to start when the store key in use is not confirmed by its stamp, the keyed audit chain's current range is not under it, and the key that range is under is not configured, except on the stores [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) names, at least a `vault_transit` store: put that key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key`. `true` starts anyway, for a key that is truly lost; the chain breaks at that start's first audit row, and it is reported as a **security loosening** | | `enforce_secret_expiry_classes` | list | `[]` | non-DEK classes whose calendar expiry **aborts engine start** under `[security].enforcement=enforce`, past `secret_max_age_days + enforce_grace_days` or with no recorded age. Entries are the `MEFOR_*` class names above or `connector`; an unknown name is refused at load. Empty keeps every non-DEK class alert-only | ```toml diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index baad7913c4..6192cfbb4c 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -838,10 +838,9 @@ This section is kept rather than deleted, because the claim it used to make is t Named as `allow_lost_audit_range_key` (BACKLOG #4934), whatever the store holds, because it says what the next start may do. - **What you lose:** the start refusal that keeps the keyed audit chain whole across a store key - change. With it off, `serve` (and each `supervise` shard, and the supervisor before it renews the - API certificate) refuses to start when the store key differs from its stamp, or has none, the - chain has not been rolled to it, and the key the chain's current range is under is not - configured. A refused start renews no API certificate. The refusal never fires on at least two + change. With it off, `serve` (and each `supervise` shard) refuses to start when the store key in + use is not confirmed by its stamp, the chain's current range is not under it, and the key that + range is under is not configured. The refusal never fires on at least two kinds of store, so this setting changes nothing there: a store under `[store].cipher_provider = "vault_transit"`, because Transit holds its key and the engine has no local store key to compare with the stamp; and a store whose audit ranges the open could not @@ -854,7 +853,7 @@ the next start may do. start whose key is unchanged, whose chain has rolled, or whose range key is configured. - **When acceptable:** only when the key the chain's current range is under is truly lost, so no `rotate-key` run can roll the chain anyway. If the key exists anywhere, put it back in - `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` until it exits 0 instead. + `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` instead. - **Compensating controls:** none that restore the chain. Record when and why the key was lost, keep the `audit-verify` output from before the start, and treat the break as a known event in any audit review. From 6324aebc4c0323066c9ff699f68b9253678982da Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 00:45:51 -0600 Subject: [PATCH 23/41] The CONFIGURATION.md row drops a remedy that read as applying to the stores the refusal skips After the colon, the remedy followed the exception clause, so it read as covering a vault_transit store, where it cannot work. SECURITY-LOOSENING.md keeps the remedy. BACKLOG #4934 --- docs/CONFIGURATION.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 3f5b5f787e..ee212275dc 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -3005,7 +3005,7 @@ tracked; set `warn_days = 0` to disable the reminder. | `secret_max_age_days` | int | 365 | max age for the **non-DEK** tracked secret classes (connector/AD/SMTP/Vault/OIDC), alerted this many days after their last observed fingerprint change | | `enforce_grace_days` | int | 30 | under `[security].enforcement=enforce`, a DEK older than `store_key_max_age_days + this` escalates its rotation alert **and refuses to start** (see `enforce_store_key_expiry`) | | `enforce_store_key_expiry` | bool | `true` | under `[security].enforcement=enforce`, a store DEK past `store_key_max_age_days + enforce_grace_days` — or one whose age cannot be determined — **aborts engine start**. `false` keeps the alert, drops the refusal, and is reported as a **security loosening** | -| `allow_lost_audit_range_key` | bool | `false` | `serve` refuses to start when the store key in use is not confirmed by its stamp, the keyed audit chain's current range is not under it, and the key that range is under is not configured, except on the stores [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) names, at least a `vault_transit` store: put that key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key`. `true` starts anyway, for a key that is truly lost; the chain breaks at that start's first audit row, and it is reported as a **security loosening** | +| `allow_lost_audit_range_key` | bool | `false` | `serve` refuses to start when the store key in use is not confirmed by its stamp, the keyed audit chain's current range is not under it, and the key that range is under is not configured, except on the stores [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) names, at least a `vault_transit` store. `true` starts anyway, for a key that is truly lost; the chain breaks at that start's first audit row, and it is reported as a **security loosening** | | `enforce_secret_expiry_classes` | list | `[]` | non-DEK classes whose calendar expiry **aborts engine start** under `[security].enforcement=enforce`, past `secret_max_age_days + enforce_grace_days` or with no recorded age. Entries are the `MEFOR_*` class names above or `connector`; an unknown name is refused at load. Empty keeps every non-DEK class alert-only | ```toml From 84a81359d8152b8c91dfb367d3b07cedf5b2e858 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 01:56:26 -0600 Subject: [PATCH 24/41] The SECURITY-LOOSENING entry is the one statement of the unrolled-key refusal, and other copies point to it The allow_lost_audit_range_key entry now states the refusal's condition as refuse_unrolled_key_change tests it, the unknown-scheme stamp and a key in use that keyed an earlier range included, and a way out rotate-key completes. CONFIGURATION.md, PHI.md, the SECURITY-LOOSENING table row, the fragments, the settings comment and posture text, the refuse_unrolled_key_change docstring and the test module docstring point there instead of restating it. Cut as false: the comments that a refused start renews nothing, which supervise contradicts when the pair prediction is False; PHI.md's other-commands-do-not-refuse sentence; the needs-the-setting-on-for-every-later-start claim; the hook comment's no-serve-start-stamps-this-key. _OPENED_AT_START and _HELD_FOR_ANOTHER_RECORD drop until it exits 0 and say not confirmed by its stamp. No behaviour changes. BACKLOG #4934 --- changelog.d/3737.security.md | 3 +- changelog.d/4934.security.md | 9 +++-- docs/CONFIGURATION.md | 2 +- docs/PHI.md | 11 ++---- docs/SECURITY-LOOSENING.md | 36 +++++++++---------- messagefoundry/__main__.py | 15 ++++---- messagefoundry/config/settings.py | 15 ++++---- messagefoundry/pipeline/secret_rotation.py | 14 ++++---- ...rotate_key_open_hook_and_lost_range_key.py | 13 +++---- 9 files changed, 50 insertions(+), 68 deletions(-) diff --git a/changelog.d/3737.security.md b/changelog.d/3737.security.md index f939374f51..28135946c6 100644 --- a/changelog.d/3737.security.md +++ b/changelog.d/3737.security.md @@ -1,6 +1,5 @@ - **`rotate-key` keeps a rotation record in `secret_rotation_meta`, and a `serve` start does not - stamp a changed store key while that record names it or cannot be read.** See - [PHI.md](../docs/PHI.md) and the `open_rotation_record` docstring. **Once a run + stamp a changed store key while that record names it or cannot be read.** **Once a run has rolled a keyed audit chain to a new key there is no way back to the prior key**, and a run that skipped an uploaded file rolls like any other. **The upload pass opens only a regular file, and reads whole only one of a size an upload can be.** A FIFO no longer blocks it, a symlink is a skip and is not followed, and a file too large for an upload is a diff --git a/changelog.d/4934.security.md b/changelog.d/4934.security.md index d6d42cd0b2..61bf26e359 100644 --- a/changelog.d/4934.security.md +++ b/changelog.d/4934.security.md @@ -1,9 +1,8 @@ - **`rotate-key` now opens its rotation record, when it opens one, from a new `open_store` hook, `before_first_write`, which a writable open runs before its first write under the store key.** A - read-only open does not run it. See the `open_rotation_record` docstring. + read-only open does not run it. **`serve` can now refuse to start on a changed store key the keyed audit chain cannot follow.** - `docs/SECURITY-LOOSENING.md` names stores the refusal does not cover. - `[secret_rotation].allow_lost_audit_range_key = true` starts anyway, for a key that is truly - lost; it logs the refusal as a WARNING, the chain breaks at that start's first audit row, and it - is reported as a security loosening. The `admin-*` commands do not refuse. + `[secret_rotation].allow_lost_audit_range_key = true` starts anyway and is reported as a security + loosening; its entry in `docs/SECURITY-LOOSENING.md` states when the refusal fires and the way + out. The `admin-*` commands do not refuse. (`BACKLOG #4934`, `BACKLOG #4936`) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index ee212275dc..8b6284cd92 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -3005,7 +3005,7 @@ tracked; set `warn_days = 0` to disable the reminder. | `secret_max_age_days` | int | 365 | max age for the **non-DEK** tracked secret classes (connector/AD/SMTP/Vault/OIDC), alerted this many days after their last observed fingerprint change | | `enforce_grace_days` | int | 30 | under `[security].enforcement=enforce`, a DEK older than `store_key_max_age_days + this` escalates its rotation alert **and refuses to start** (see `enforce_store_key_expiry`) | | `enforce_store_key_expiry` | bool | `true` | under `[security].enforcement=enforce`, a store DEK past `store_key_max_age_days + enforce_grace_days` — or one whose age cannot be determined — **aborts engine start**. `false` keeps the alert, drops the refusal, and is reported as a **security loosening** | -| `allow_lost_audit_range_key` | bool | `false` | `serve` refuses to start when the store key in use is not confirmed by its stamp, the keyed audit chain's current range is not under it, and the key that range is under is not configured, except on the stores [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) names, at least a `vault_transit` store. `true` starts anyway, for a key that is truly lost; the chain breaks at that start's first audit row, and it is reported as a **security loosening** | +| `allow_lost_audit_range_key` | bool | `false` | `true` starts `serve` over its refusal of a store key change the keyed audit chain cannot follow, and is reported as a **security loosening**. [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) states when that refusal fires and the way out | | `enforce_secret_expiry_classes` | list | `[]` | non-DEK classes whose calendar expiry **aborts engine start** under `[security].enforcement=enforce`, past `secret_max_age_days + enforce_grace_days` or with no recorded age. Entries are the `MEFOR_*` class names above or `connector`; an unknown name is refused at load. Empty keeps every non-DEK class alert-only | ```toml diff --git a/docs/PHI.md b/docs/PHI.md index 269b211172..c96ee5e6b8 100644 --- a/docs/PHI.md +++ b/docs/PHI.md @@ -772,14 +772,9 @@ for defense-in-depth without swapping the `aiosqlite` connector. store with no keyed chain to ask, such as one whose audit rows open with no genesis row naming its key, so there a stamp proves only a changed key. On such a store, do not start `serve` between setting a new key and running `rotate-key`: that stamp stands however the run ends. On a - store with a keyed chain, `serve` refuses to start on a changed key the chain has not rolled to - while the key the chain's current range is under is not configured (BACKLOG #4934): its first - audit row would go into that range under the new key, never verify there, and leave a chain no - later `rotate-key` can roll. Put that key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run - `rotate-key` until it exits 0. `[secret_rotation].allow_lost_audit_range_key = true` starts - anyway, for a key that is truly lost; it is a named security loosening, and the chain breaks at - that start's first audit row. Other commands that open the store and write an audit row, such as - the `admin-*` commands, do not refuse, so do not run them in that state either. **A roll that refuses for good holds the key for good (BACKLOG #4934, #4935).** The roll + store with a keyed chain, `serve` can refuse to start on a changed key (BACKLOG #4934); the + `[secret_rotation].allow_lost_audit_range_key` entry in + [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) states when, and the way out. **A roll that refuses for good holds the key for good (BACKLOG #4934, #4935).** The roll verifies the whole chain first, so a chain whose rows do not verify, or whose rows open with no genesis row, refuses under every key, a new one included, and the record stays open. A newest range under a key that is not configured is not one of these, though `audit-verify` reports it diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index 6192cfbb4c..107c1834a1 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -242,7 +242,7 @@ Earlier versions of this guide called this an open owner question. The 2026-10-2 | | `[approvals].operations` | `["connection_purge", "dead_letter_replay"]` (*conditional* — a loosening only while `[approvals].enabled` is on; a list that leaves out either default operation is named, and so is an empty list, which holds nothing; named as `approvals.operations`) | | | `[api].trusted_proxies` | `[]` (any network covering more than one address is named, once, a `/31` or `/127` included, because every host inside it may set `X-Forwarded-For`. Entries covering every address, such as `0.0.0.0/0` or `::/0`, trust it from every peer, as the refused `*` would, and get their own wording) | | | `[secret_rotation].enforce_store_key_expiry` | `true` (a calendar-overdue store DEK refuses to start) | -| | `[secret_rotation].allow_lost_audit_range_key` | `false` (`true` starts the engine on a changed store key the keyed audit chain has not rolled to while the key of the chain's current range is not configured; the chain breaks at that start's first audit row) | +| | `[secret_rotation].allow_lost_audit_range_key` | `false` (`true` starts the engine over the store key refusal its entry below states) | | | `[cert_monitor].warn_days`, `[secret_rotation].warn_days` | `30` / `14` (`0` turns that credential reminder off, and a smaller value makes it later; named as `cert_monitor.warn_days` and `secret_rotation.warn_days`) | | | `[[alerts.rules]]` | `[]` (a rule that can send a credential reminder to no transport: `mute = true`, `transports = []`, or an `escalate` tier with `transports = []`; named as `alerts.rules`) | | | `[alerts].security_signals`, its five detector counts and `security_window_seconds` | `true`, each count's default, `300` (`false` turns the security-signal layer off and is named as `alerts.security_signals`; with the layer on, a count at `0` turns that one detector off, a count above its default needs a larger burst, a shorter window lets a burst spread past it raise nothing, and each is named as `alerts.`, the window only while any count is above `0`) | @@ -838,29 +838,27 @@ This section is kept rather than deleted, because the claim it used to make is t Named as `allow_lost_audit_range_key` (BACKLOG #4934), whatever the store holds, because it says what the next start may do. - **What you lose:** the start refusal that keeps the keyed audit chain whole across a store key - change. With it off, `serve` (and each `supervise` shard) refuses to start when the store key in - use is not confirmed by its stamp, the chain's current range is not under it, and the key that - range is under is not configured. The refusal never fires on at least two - kinds of store, so this setting changes nothing there: a store under - `[store].cipher_provider = "vault_transit"`, because Transit holds its key and the engine has no - local store key to compare with the stamp; and a store whose audit ranges the open could not - authenticate, because the open then cannot say which key the current range is under, though - that start's first audit row may still break the chain. With it on, that start goes ahead: its first audit row goes into that range under the - new key, never verifies there, and the chain breaks at that row. `audit-verify` reports the break, - and `rotate-key` cannot roll the chain afterwards. + change. This entry is the statement of when that refusal fires and of the way out. With it off, + `serve` (and each `supervise` shard) refuses to start when all of these hold: the engine holds a + local store key, so not under `[store].cipher_provider = "vault_transit"`; the stamp does not + name that key, which covers a different key, no stamp, and a stamp under a scheme this release + does not know; the store has a keyed audit chain whose ranges the open authenticated; and the + key the chain's newest range is under is not configured. It refuses whether or not the key in + use keyed an earlier range. With it on, that start goes ahead: its first audit row goes into + that range under the key in use, never verifies there, and the chain breaks at that row. Once it + has broken, `audit-verify` reports the break, and no `rotate-key` run can roll the chain. - **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, and every other start check. Rows before the break still verify. The setting changes nothing on a - start whose key is unchanged, whose chain has rolled, or whose range key is configured. -- **When acceptable:** only when the key the chain's current range is under is truly lost, so no - `rotate-key` run can roll the chain anyway. If the key exists anywhere, put it back in - `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` instead. + start the refusal would not stop. +- **When acceptable:** only when the key the chain's newest range is under is truly lost. If the + key exists anywhere, put it back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` + instead. Its audit chain step says whether it rolled the chain and, where it refuses, why. Where + the key in use already keyed an earlier range, that step refuses and asks for a new key. - **Compensating controls:** none that restore the chain. Record when and why the key was lost, keep the `audit-verify` output from before the start, and treat the break as a known event in any audit review. -- **Reversible:** no, once a start has used it. The chain stays broken at that row, and the store - needs the setting on for every later start: turned back off, the next start refuses again, - because the chain's range is still under the lost key, and no `rotate-key` run can move it. Set - only and never used, it changes nothing and can be removed. +- **Reversible:** no, once a start has used it: the chain stays broken at that row. Set only and + never used, it changes nothing and can be removed. ### `[cert_monitor].warn_days` or `[secret_rotation].warn_days` at `0`, or below its default — a credential reminder is off or late Named as `cert_monitor.warn_days` and `secret_rotation.warn_days` (ASVS 6.4.5, BACKLOG #2227). A diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index 9f29970b30..b98c8b7ebb 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -4909,8 +4909,7 @@ def registry_filter(reg: Registry) -> Registry: # noqa: F811 (local shard-bound _replaced: list[GeneratedPairReplaced] = [] _state_dir = generated_state_dir(settings.store.path) # BACKLOG #4934: the lifespan's start refusal runs after this mint, so a start that may replace - # the pair takes it here first, and a refused start renews nothing on disk. A first-run mint - # replaces nothing, so it leaves no renewal without its audit row and needs no early open. + # the pair takes it here first. A first-run mint replaces nothing, so it needs no early open. if generated_pair_may_be_replaced(settings.api, state_dir=_state_dir, renew=args.shard is None): from messagefoundry.pipeline.secret_rotation import UnrolledStoreKeyRefused from messagefoundry.store.base import KeylessAuditChainRefused @@ -5244,8 +5243,8 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P A re-mint is audited like serve's, in the store the shards are about to open, over the same derived store-hop posture serve's lifespan opens it with. On a start that may replace the pair, that store is opened first and takes serve's start refusal before the renewal runs (BACKLOG - #4934), so a refused start writes nothing to the pair, and a failure to open propagates before - any renewal. A start that replaced a pair it did not expect to opens it after, as before. + #4934), and a failure to open propagates before any renewal. A start that replaced a pair it + did not expect to opens it after, as before. """ import asyncio @@ -5268,8 +5267,8 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P state_dir = generated_state_dir(store_path) replaced: list[GeneratedPairReplaced] = [] # BACKLOG #4934: a start that may replace the pair opens the store and takes the shards' start - # refusal before it writes anything, so a refused start renews nothing on disk. Any other start - # mints or reuses as before, and opens the store only if that replaced a pair after all. + # refusal before it writes anything. Any other start mints or reuses as before, and opens the + # store only if that replaced a pair after all. renew_after_refusal = generated_pair_may_be_replaced(settings.api, state_dir=state_dir) if not renew_after_refusal: ensure_api_tls_material(settings.api, state_dir=state_dir, replacements=replaced) @@ -5335,7 +5334,7 @@ async def _refuse_unrolled_key_on(store: Store, settings: ServiceSettings) -> No def _refuse_unrolled_key_before_tls_write(settings: ServiceSettings) -> None: """Open ``serve``'s store and run :func:`_refuse_unrolled_key_on` before the start may replace - the generated API pair (BACKLOG #4934), so a refused start renews nothing on disk. The lifespan + the generated API pair (BACKLOG #4934). The lifespan refuses again after its own open; this one exists only because the mint comes first. Raises :class:`~messagefoundry.pipeline.secret_rotation.UnrolledStoreKeyRefused`, the keyless-chain refusal, :class:`_StoreOpenRefused` for an open refusal :func:`_open_store_or_refuse` maps, and @@ -11959,7 +11958,7 @@ async def run() -> tuple[int, ResealResult, tuple[bool, str], bool, ResealOutcom async def open_record(opening: object) -> None: # BACKLOG #3737, #4936: inside the store open, before its first write under the key, # on every store, so a kill anywhere after it, the open's own writes and either data - # pass included, leaves the record open and no serve start stamps this key. + # pass included, leaves the record open. nonlocal record_named # Every backend's store has cipher_info(); the hook is typed for any of them. info = getattr(opening, "cipher_info", None) diff --git a/messagefoundry/config/settings.py b/messagefoundry/config/settings.py index de67c05def..d0f4070270 100644 --- a/messagefoundry/config/settings.py +++ b/messagefoundry/config/settings.py @@ -6852,11 +6852,9 @@ class SecretRotationSettings(_Section): # build would buy the setting without the posture. Setting it false is a LOOSENING and # security_loosenings() names it, so the opt-out is never silent. enforce_store_key_expiry: bool = True - # BACKLOG #4934 — serve refuses to start when the store key changed, the keyed audit chain has - # not rolled to it, and the key the chain's current range is under is not configured: the first - # audit row that start wrote would break the chain, and no later rotate-key could roll it. True - # starts anyway, for a prior key that is truly lost; it is a LOOSENING, named by - # security_loosenings(), and the chain breaks at that start's first audit row. + # BACKLOG #4934 — True starts serve over its refusal of a store key change the keyed audit + # chain cannot follow; a LOOSENING, named by security_loosenings(). The refusal's condition and + # way out are stated once, in this setting's entry in docs/SECURITY-LOOSENING.md. allow_lost_audit_range_key: bool = False # ASVS 13.3.4 / BACKLOG #1932 — the calendar refusal for the NON-DEK classes, OPT-IN per class. # Each entry names one class from ENFORCEABLE_SECRET_EXPIRY_CLASSES. Under @@ -10679,10 +10677,9 @@ def security_loosenings( out.append( ( "allow_lost_audit_range_key", - "the engine starts even when the store key changed, the audit chain has not been " - "rolled to it and the key its current range is under is not configured, so the " - "audit chain breaks at that start's first audit row and no rotate-key run can roll " - "it afterwards", + "the engine starts even where its refusal of a store key change the keyed audit " + "chain cannot follow would stop it, and the audit chain breaks at that start's " + "first audit row; docs/SECURITY-LOOSENING.md states that refusal", ) ) # --- the credential reminders (ASVS 6.4.5; BACKLOG #2227 and #2008 step 4). Owner answer to diff --git a/messagefoundry/pipeline/secret_rotation.py b/messagefoundry/pipeline/secret_rotation.py index 34b6581b8d..80fb659e4c 100644 --- a/messagefoundry/pipeline/secret_rotation.py +++ b/messagefoundry/pipeline/secret_rotation.py @@ -346,23 +346,22 @@ async def close_rotation_record(meta_store: SecretRotationMetaStore, day: str) - #: The WARNING a start logs when it opened the rotation record itself (BACKLOG #4942). _OPENED_AT_START = ( - "secret rotation: the store key in use differs from its stamp, or has none, and the audit " + "secret rotation: the store key in use is not confirmed by its stamp, and the audit " "chain has not been rolled to it, so no `messagefoundry rotate-key` run has completed on it. A " "rotation record was opened for it, its rotation date was not stamped and its age was not " "reset. Stop the engine and run `messagefoundry rotate-key`, with the prior key in " - "MEFOR_STORE_ENCRYPTION_KEYS_RETIRED, until it exits 0. If it refuses at its audit chain step, " + "MEFOR_STORE_ENCRYPTION_KEYS_RETIRED. If it refuses at its audit chain step, " "its error says whether a re-run can clear that, and the way out where none can." ) #: The WARNING a start logs when it held a key and kept a record open for another key (BACKLOG #: #4942): see :func:`open_rotation_record`'s ``keep_another``. _HELD_FOR_ANOTHER_RECORD = ( - "secret rotation: the store key in use differs from its stamp, or has none, while the " + "secret rotation: the store key in use is not confirmed by its stamp, while the " "rotation record stands open for another store key that no `messagefoundry rotate-key` run has " "completed on. That record was kept, this key's " "rotation date was not stamped and its age was not reset. Stop the engine and run " - "`messagefoundry rotate-key`, with every prior key in MEFOR_STORE_ENCRYPTION_KEYS_RETIRED, " - "until it exits 0." + "`messagefoundry rotate-key`, with every prior key in MEFOR_STORE_ENCRYPTION_KEYS_RETIRED." ) @@ -430,9 +429,8 @@ async def refuse_unrolled_key_change( range_key_missing: bool, allow_lost: bool, ) -> None: - """Raise :class:`UnrolledStoreKeyRefused` when the store key changed, the keyed audit chain has - not rolled to it, and the key the chain's current range is under is not configured (BACKLOG - #4934, owner ruling 2026-11-04). ``serve`` calls it right after the store open, before it writes + """Raise :class:`UnrolledStoreKeyRefused` on the condition the ``allow_lost_audit_range_key`` + entry in docs/SECURITY-LOOSENING.md states (BACKLOG #4934, owner ruling 2026-11-04). ``serve`` calls it right after the store open, before it writes any audit row. ``audit_rolled`` is :func:`~messagefoundry.store.store.audit_chain_rolled` and ``range_key_missing`` is :func:`~messagefoundry.store.store.audit_range_key_missing`, both for the open store. diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index 3a7c086f9c..409105718a 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -3,14 +3,11 @@ """BACKLOG #4936 and #4934, the owner rulings of 2026-11-04. #4936: ``rotate-key`` opens its rotation record through ``open_store``'s ``before_first_write`` -hook, on every store, so a run killed inside the store open after the hook leaves a record, and the -next start stamps nothing, on a store with no keyed audit chain too. Before the hook the record -opened only after the open returned, and that start stamped the new key. - -#4934: ``serve`` refuses to start when the store key changed, the keyed audit chain has not rolled -to it, and the key the chain's current range is under is not configured, with -``[secret_rotation].allow_lost_audit_range_key`` as the override for a prior key that is truly -lost.""" +hook, on every store, so a run killed inside the store open after the hook leaves a record. + +#4934: ``serve``'s start refusal of a store key change the keyed audit chain cannot follow, with +``[secret_rotation].allow_lost_audit_range_key`` as its override; docs/SECURITY-LOOSENING.md +states when it fires.""" from __future__ import annotations From d5046efb5ee3d9668313e1c0d3924f3701507c4f Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 02:18:07 -0600 Subject: [PATCH 25/41] The refusal entry states the range test the code makes, and five other texts drop what a review round found false SECURITY-LOOSENING.md says the open found no range that fails to authenticate, not that it authenticated them, since a range under a key not held is never checked. _OPENED_AT_START says the open did not find the chain rolled to the key, and drops the claim that no run completed and that the roll's error names a way out; its test's assertion follows. The supervise comment says the refusal comes before the renewal, not before any write. The hook comment and the test module docstring say any record it opened. The settings comment drops 'stated once'. No behaviour changes. BACKLOG #4934 --- docs/SECURITY-LOOSENING.md | 5 +++-- messagefoundry/__main__.py | 4 ++-- messagefoundry/config/settings.py | 4 ++-- messagefoundry/pipeline/secret_rotation.py | 7 +++---- tests/test_rotate_key_open_hook_and_lost_range_key.py | 3 ++- tests/test_rotate_key_rotation_record.py | 2 +- 6 files changed, 13 insertions(+), 12 deletions(-) diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index 107c1834a1..1b6ded8a54 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -838,11 +838,12 @@ This section is kept rather than deleted, because the claim it used to make is t Named as `allow_lost_audit_range_key` (BACKLOG #4934), whatever the store holds, because it says what the next start may do. - **What you lose:** the start refusal that keeps the keyed audit chain whole across a store key - change. This entry is the statement of when that refusal fires and of the way out. With it off, + change. This entry states when that refusal fires and the way out. With it off, `serve` (and each `supervise` shard) refuses to start when all of these hold: the engine holds a local store key, so not under `[store].cipher_provider = "vault_transit"`; the stamp does not name that key, which covers a different key, no stamp, and a stamp under a scheme this release - does not know; the store has a keyed audit chain whose ranges the open authenticated; and the + does not know; the store has a keyed audit chain in which the open found no range that fails to + authenticate; and the key the chain's newest range is under is not configured. It refuses whether or not the key in use keyed an earlier range. With it on, that start goes ahead: its first audit row goes into that range under the key in use, never verifies there, and the chain breaks at that row. Once it diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index b98c8b7ebb..d6d45be6e0 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -5267,7 +5267,7 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P state_dir = generated_state_dir(store_path) replaced: list[GeneratedPairReplaced] = [] # BACKLOG #4934: a start that may replace the pair opens the store and takes the shards' start - # refusal before it writes anything. Any other start mints or reuses as before, and opens the + # refusal before it renews the pair. Any other start mints or reuses as before, and opens the # store only if that replaced a pair after all. renew_after_refusal = generated_pair_may_be_replaced(settings.api, state_dir=state_dir) if not renew_after_refusal: @@ -11958,7 +11958,7 @@ async def run() -> tuple[int, ResealResult, tuple[bool, str], bool, ResealOutcom async def open_record(opening: object) -> None: # BACKLOG #3737, #4936: inside the store open, before its first write under the key, # on every store, so a kill anywhere after it, the open's own writes and either data - # pass included, leaves the record open. + # pass included, leaves any record it opened open. nonlocal record_named # Every backend's store has cipher_info(); the hook is typed for any of them. info = getattr(opening, "cipher_info", None) diff --git a/messagefoundry/config/settings.py b/messagefoundry/config/settings.py index d0f4070270..1c1936d2a1 100644 --- a/messagefoundry/config/settings.py +++ b/messagefoundry/config/settings.py @@ -6853,8 +6853,8 @@ class SecretRotationSettings(_Section): # security_loosenings() names it, so the opt-out is never silent. enforce_store_key_expiry: bool = True # BACKLOG #4934 — True starts serve over its refusal of a store key change the keyed audit - # chain cannot follow; a LOOSENING, named by security_loosenings(). The refusal's condition and - # way out are stated once, in this setting's entry in docs/SECURITY-LOOSENING.md. + # chain cannot follow; a LOOSENING, named by security_loosenings(). This setting's entry in + # docs/SECURITY-LOOSENING.md states the refusal's condition and way out. allow_lost_audit_range_key: bool = False # ASVS 13.3.4 / BACKLOG #1932 — the calendar refusal for the NON-DEK classes, OPT-IN per class. # Each entry names one class from ENFORCEABLE_SECRET_EXPIRY_CLASSES. Under diff --git a/messagefoundry/pipeline/secret_rotation.py b/messagefoundry/pipeline/secret_rotation.py index 80fb659e4c..7cdf32ddab 100644 --- a/messagefoundry/pipeline/secret_rotation.py +++ b/messagefoundry/pipeline/secret_rotation.py @@ -346,12 +346,11 @@ async def close_rotation_record(meta_store: SecretRotationMetaStore, day: str) - #: The WARNING a start logs when it opened the rotation record itself (BACKLOG #4942). _OPENED_AT_START = ( - "secret rotation: the store key in use is not confirmed by its stamp, and the audit " - "chain has not been rolled to it, so no `messagefoundry rotate-key` run has completed on it. A " + "secret rotation: the store key in use is not confirmed by its stamp, and the open did not " + "find the audit chain rolled to it. A " "rotation record was opened for it, its rotation date was not stamped and its age was not " "reset. Stop the engine and run `messagefoundry rotate-key`, with the prior key in " - "MEFOR_STORE_ENCRYPTION_KEYS_RETIRED. If it refuses at its audit chain step, " - "its error says whether a re-run can clear that, and the way out where none can." + "MEFOR_STORE_ENCRYPTION_KEYS_RETIRED." ) #: The WARNING a start logs when it held a key and kept a record open for another key (BACKLOG diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index 409105718a..f784cb37a2 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -3,7 +3,8 @@ """BACKLOG #4936 and #4934, the owner rulings of 2026-11-04. #4936: ``rotate-key`` opens its rotation record through ``open_store``'s ``before_first_write`` -hook, on every store, so a run killed inside the store open after the hook leaves a record. +hook, on every store, so a run killed inside the store open after the hook leaves any record it +opened. #4934: ``serve``'s start refusal of a store key change the keyed audit chain cannot follow, with ``[secret_rotation].allow_lost_audit_range_key`` as its override; docs/SECURITY-LOOSENING.md diff --git a/tests/test_rotate_key_rotation_record.py b/tests/test_rotate_key_rotation_record.py index b26fd1d45f..0eae5cc024 100644 --- a/tests/test_rotate_key_rotation_record.py +++ b/tests/test_rotate_key_rotation_record.py @@ -322,7 +322,7 @@ def test_a_serve_start_before_the_first_run_opens_the_record_and_stamps_nothing( assert rows[_DEK].last_rotated == _OLD_DATE record = rows[ROTATION_RECORD_ID] assert record.fingerprint == dek_fingerprint(_key_id(key_b)) - assert "the audit chain has not been rolled to it" in caplog.text + assert "did not find the audit chain rolled to it" in caplog.text caplog.clear() with caplog.at_level(logging.WARNING, logger="messagefoundry.pipeline.secret_rotation"): From 2b83e86f75bb77ef95dd1a0025c3f166009d8db8 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 02:33:35 -0600 Subject: [PATCH 26/41] The refusal entry drops two audit-verify claims, and the hook comment bounds its kill While the newest range's key is lost, audit-verify does not check the row an overridden start writes, so the entry no longer says it reports that break or that rows before it still verify. The rotate-key hook comment says a kill after the hook leaves any record it opened open until a run closes it. No behaviour changes. BACKLOG #4934 --- docs/SECURITY-LOOSENING.md | 5 ++--- messagefoundry/__main__.py | 4 ++-- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index 1b6ded8a54..81d2aa04ae 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -846,10 +846,9 @@ the next start may do. authenticate; and the key the chain's newest range is under is not configured. It refuses whether or not the key in use keyed an earlier range. With it on, that start goes ahead: its first audit row goes into - that range under the key in use, never verifies there, and the chain breaks at that row. Once it - has broken, `audit-verify` reports the break, and no `rotate-key` run can roll the chain. + that range under the key in use, never verifies there, and the chain breaks at that row. - **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, - and every other start check. Rows before the break still verify. The setting changes nothing on a + and every other start check. The setting changes nothing on a start the refusal would not stop. - **When acceptable:** only when the key the chain's newest range is under is truly lost. If the key exists anywhere, put it back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index d6d45be6e0..f86ad2c9e0 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -11957,8 +11957,8 @@ async def run() -> tuple[int, ResealResult, tuple[bool, str], bool, ResealOutcom async def open_record(opening: object) -> None: # BACKLOG #3737, #4936: inside the store open, before its first write under the key, - # on every store, so a kill anywhere after it, the open's own writes and either data - # pass included, leaves any record it opened open. + # on every store, so a kill after it, the open's own writes and either data pass + # included, leaves any record it opened open until a run closes it. nonlocal record_named # Every backend's store has cipher_info(); the hook is typed for any of them. info = getattr(opening, "cipher_info", None) From 9d5517ccec2454a927995cf2f1c5c7d0834b04b9 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 02:58:03 -0600 Subject: [PATCH 27/41] The override entry drops two changes-nothing claims and states that a switch-back refuses again As the Lander ruled on PR 2991: SECURITY-LOOSENING.md drops 'The setting changes nothing on a start the refusal would not stop' and 'it changes nothing and' from the set-only sentence, since security_loosenings() names the switch at every start, and adds the one sentence that turned back off the next start refuses again while the lost range key is still not configured. The rotate-key hook comment and the test module docstring drop 'on every store'. Comments and docs only. BACKLOG #4934 --- docs/SECURITY-LOOSENING.md | 8 ++++---- messagefoundry/__main__.py | 2 +- tests/test_rotate_key_open_hook_and_lost_range_key.py | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index 81d2aa04ae..8283ccc7cc 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -848,8 +848,7 @@ the next start may do. use keyed an earlier range. With it on, that start goes ahead: its first audit row goes into that range under the key in use, never verifies there, and the chain breaks at that row. - **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, - and every other start check. The setting changes nothing on a - start the refusal would not stop. + and every other start check. - **When acceptable:** only when the key the chain's newest range is under is truly lost. If the key exists anywhere, put it back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` instead. Its audit chain step says whether it rolled the chain and, where it refuses, why. Where @@ -857,8 +856,9 @@ the next start may do. - **Compensating controls:** none that restore the chain. Record when and why the key was lost, keep the `audit-verify` output from before the start, and treat the break as a known event in any audit review. -- **Reversible:** no, once a start has used it: the chain stays broken at that row. Set only and - never used, it changes nothing and can be removed. +- **Reversible:** no, once a start has used it: the chain stays broken at that row. Turned back + off, the next start refuses again while the lost range key is still not configured. Set only and + never used, it can be removed. ### `[cert_monitor].warn_days` or `[secret_rotation].warn_days` at `0`, or below its default — a credential reminder is off or late Named as `cert_monitor.warn_days` and `secret_rotation.warn_days` (ASVS 6.4.5, BACKLOG #2227). A diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index f86ad2c9e0..11c3bc2c8f 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -11957,7 +11957,7 @@ async def run() -> tuple[int, ResealResult, tuple[bool, str], bool, ResealOutcom async def open_record(opening: object) -> None: # BACKLOG #3737, #4936: inside the store open, before its first write under the key, - # on every store, so a kill after it, the open's own writes and either data pass + # so a kill after it, the open's own writes and either data pass # included, leaves any record it opened open until a run closes it. nonlocal record_named # Every backend's store has cipher_info(); the hook is typed for any of them. diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index f784cb37a2..a5c576d4ba 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -3,7 +3,7 @@ """BACKLOG #4936 and #4934, the owner rulings of 2026-11-04. #4936: ``rotate-key`` opens its rotation record through ``open_store``'s ``before_first_write`` -hook, on every store, so a run killed inside the store open after the hook leaves any record it +hook, so a run killed inside the store open after the hook leaves any record it opened. #4934: ``serve``'s start refusal of a store key change the keyed audit chain cannot follow, with From 686d421faf509a19011b0726fb3f49a5903db37b Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 03:24:53 -0600 Subject: [PATCH 28/41] The override entry restores the cost and the untrusted-store caveat, and drops whole As the Lander held on PR 2991 at 9d5517cc: SECURITY-LOOSENING.md and the security_loosenings() text again say rotate-key cannot roll the chain after an overridden start, and the entry again says an untrusted store's first audit row may still break the chain, both word for word from the text before 84a81359. The admin-* warning PHI.md carried called every other command refusal-free, which supervise contradicts, so no true old text exists and 'whole' is cut instead. No behaviour changes. BACKLOG #4934 --- docs/SECURITY-LOOSENING.md | 8 +++++--- messagefoundry/config/settings.py | 3 ++- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index 8283ccc7cc..c7d3f5de37 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -837,16 +837,18 @@ This section is kept rather than deleted, because the claim it used to make is t ### `[secret_rotation].allow_lost_audit_range_key = true` — the engine starts on a key change the audit chain cannot follow Named as `allow_lost_audit_range_key` (BACKLOG #4934), whatever the store holds, because it says what the next start may do. -- **What you lose:** the start refusal that keeps the keyed audit chain whole across a store key +- **What you lose:** the start refusal that keeps the keyed audit chain across a store key change. This entry states when that refusal fires and the way out. With it off, `serve` (and each `supervise` shard) refuses to start when all of these hold: the engine holds a local store key, so not under `[store].cipher_provider = "vault_transit"`; the stamp does not name that key, which covers a different key, no stamp, and a stamp under a scheme this release does not know; the store has a keyed audit chain in which the open found no range that fails to - authenticate; and the + authenticate (where it finds one, the refusal does not fire, though that start's first audit row + may still break the chain); and the key the chain's newest range is under is not configured. It refuses whether or not the key in use keyed an earlier range. With it on, that start goes ahead: its first audit row goes into - that range under the key in use, never verifies there, and the chain breaks at that row. + that range under the key in use, never verifies there, and the chain breaks at that row, and + `rotate-key` cannot roll the chain afterwards. - **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, and every other start check. - **When acceptable:** only when the key the chain's newest range is under is truly lost. If the diff --git a/messagefoundry/config/settings.py b/messagefoundry/config/settings.py index 1c1936d2a1..2fa2c2d7c0 100644 --- a/messagefoundry/config/settings.py +++ b/messagefoundry/config/settings.py @@ -10679,7 +10679,8 @@ def security_loosenings( "allow_lost_audit_range_key", "the engine starts even where its refusal of a store key change the keyed audit " "chain cannot follow would stop it, and the audit chain breaks at that start's " - "first audit row; docs/SECURITY-LOOSENING.md states that refusal", + "first audit row and no rotate-key run can roll it afterwards; " + "docs/SECURITY-LOOSENING.md states that refusal", ) ) # --- the credential reminders (ASVS 6.4.5; BACKLOG #2227 and #2008 step 4). Owner answer to From 6059920c047d9ce0b3006f8811e42065b2f5340b Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 03:40:46 -0600 Subject: [PATCH 29/41] The override entry names the serve and supervise start refusal and warns that admin-* commands do not refuse As the Lander ruled on PR 2991 at 686d421f: the What you lose line now claims the refusal only for the serve and supervise start, which are the only callers of refuse_unrolled_key_change (the lifespan, the serve early TLS check and the supervise renewal), and the entry adds that the admin-* commands do not refuse, so do not run them in that state. Docs only. BACKLOG #4934 --- docs/SECURITY-LOOSENING.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index c7d3f5de37..77bf11c85c 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -837,7 +837,7 @@ This section is kept rather than deleted, because the claim it used to make is t ### `[secret_rotation].allow_lost_audit_range_key = true` — the engine starts on a key change the audit chain cannot follow Named as `allow_lost_audit_range_key` (BACKLOG #4934), whatever the store holds, because it says what the next start may do. -- **What you lose:** the start refusal that keeps the keyed audit chain across a store key +- **What you lose:** the `serve` and `supervise` start refusal that keeps the keyed audit chain across a store key change. This entry states when that refusal fires and the way out. With it off, `serve` (and each `supervise` shard) refuses to start when all of these hold: the engine holds a local store key, so not under `[store].cipher_provider = "vault_transit"`; the stamp does not @@ -848,7 +848,8 @@ the next start may do. key the chain's newest range is under is not configured. It refuses whether or not the key in use keyed an earlier range. With it on, that start goes ahead: its first audit row goes into that range under the key in use, never verifies there, and the chain breaks at that row, and - `rotate-key` cannot roll the chain afterwards. + `rotate-key` cannot roll the chain afterwards. The `admin-*` commands do not refuse, so do not + run them in that state. - **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, and every other start check. - **When acceptable:** only when the key the chain's newest range is under is truly lost. If the From fa48b9a5aa5d828cc4f3eae12099402c3c936329 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 04:06:31 -0600 Subject: [PATCH 30/41] The override entry names the commands that do not refuse and conditions its way out on none having run As the Lander ruled on PR 2991 at 6059920c: SECURITY-LOOSENING.md replaces the admin-* sentence with the dictated at-least list and its cost, conditions the way out and the Reversible bullet on no audit-writing command having run in that state, the 4934 fragment points to the entry for which commands do not refuse, and PHI.md drops the sentence that told the reader to put the key back and re-run. Docs only. BACKLOG #4934 --- changelog.d/4934.security.md | 2 +- docs/PHI.md | 4 +--- docs/SECURITY-LOOSENING.md | 13 ++++++++----- 3 files changed, 10 insertions(+), 9 deletions(-) diff --git a/changelog.d/4934.security.md b/changelog.d/4934.security.md index 61bf26e359..1942fa3542 100644 --- a/changelog.d/4934.security.md +++ b/changelog.d/4934.security.md @@ -4,5 +4,5 @@ **`serve` can now refuse to start on a changed store key the keyed audit chain cannot follow.** `[secret_rotation].allow_lost_audit_range_key = true` starts anyway and is reported as a security loosening; its entry in `docs/SECURITY-LOOSENING.md` states when the refusal fires and the way - out. The `admin-*` commands do not refuse. + out, and which commands do not refuse. (`BACKLOG #4934`, `BACKLOG #4936`) diff --git a/docs/PHI.md b/docs/PHI.md index c96ee5e6b8..f33e68b64e 100644 --- a/docs/PHI.md +++ b/docs/PHI.md @@ -776,9 +776,7 @@ for defense-in-depth without swapping the `aiosqlite` connector. `[secret_rotation].allow_lost_audit_range_key` entry in [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) states when, and the way out. **A roll that refuses for good holds the key for good (BACKLOG #4934, #4935).** The roll verifies the whole chain first, so a chain whose rows do not verify, or whose rows open with no - genesis row, refuses under every key, a new one included, and the record stays open. A newest - range under a key that is not configured is not one of these, though `audit-verify` reports it - too: put that key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and re-run. Where a run that + genesis row, refuses under every key, a new one included, and the record stays open. Where a run that opened the record meets a refusal for good, it names at least two ways out: `[secret_rotation].store_key_last_rotated`, set to the date of a run that re-sealed the data with no skip, and `[secret_rotation].enforce_store_key_expiry = false`, which lifts only the start diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index 77bf11c85c..4f12d3026f 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -848,18 +848,21 @@ the next start may do. key the chain's newest range is under is not configured. It refuses whether or not the key in use keyed an earlier range. With it on, that start goes ahead: its first audit row goes into that range under the key in use, never verifies there, and the chain breaks at that row, and - `rotate-key` cannot roll the chain afterwards. The `admin-*` commands do not refuse, so do not - run them in that state. + `rotate-key` cannot roll the chain afterwards. At least the `admin-*` commands, + `issue-enrolment-code`, `provision-admin` and `backup` do not refuse; one run of any of them in + that state breaks the chain, and `rotate-key` then refuses for good, even after the key is put + back. - **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, and every other start check. - **When acceptable:** only when the key the chain's newest range is under is truly lost. If the - key exists anywhere, put it back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` - instead. Its audit chain step says whether it rolled the chain and, where it refuses, why. Where + key exists anywhere and no command that writes an audit row has run in that state, put it back in + `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` instead. Its audit chain step says whether it rolled the chain and, where it refuses, why. Where the key in use already keyed an earlier range, that step refuses and asks for a new key. - **Compensating controls:** none that restore the chain. Record when and why the key was lost, keep the `audit-verify` output from before the start, and treat the break as a known event in any audit review. -- **Reversible:** no, once a start has used it: the chain stays broken at that row. Turned back +- **Reversible:** no, once a start has used it, or any command that writes an audit row has run in + that state: the chain stays broken at that row. Turned back off, the next start refuses again while the lost range key is still not configured. Set only and never used, it can be removed. From 8dd4b60fa0ec6a8da9084524319c54a2344f6335 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 04:35:45 -0600 Subject: [PATCH 31/41] Revert fa48b9a5 and state the audit-writing command cost in one sentence As the Lander ruled on PR 2991 at fa48b9a5: the text returns to 6059920c's, and the admin-* sentence in the allow_lost_audit_range_key entry becomes: A command that writes an audit row in that state breaks the chain, and rotate-key then cannot roll it, even after the key is put back. Docs only. BACKLOG #4934 --- changelog.d/4934.security.md | 2 +- docs/PHI.md | 4 +++- docs/SECURITY-LOOSENING.md | 13 +++++-------- 3 files changed, 9 insertions(+), 10 deletions(-) diff --git a/changelog.d/4934.security.md b/changelog.d/4934.security.md index 1942fa3542..61bf26e359 100644 --- a/changelog.d/4934.security.md +++ b/changelog.d/4934.security.md @@ -4,5 +4,5 @@ **`serve` can now refuse to start on a changed store key the keyed audit chain cannot follow.** `[secret_rotation].allow_lost_audit_range_key = true` starts anyway and is reported as a security loosening; its entry in `docs/SECURITY-LOOSENING.md` states when the refusal fires and the way - out, and which commands do not refuse. + out. The `admin-*` commands do not refuse. (`BACKLOG #4934`, `BACKLOG #4936`) diff --git a/docs/PHI.md b/docs/PHI.md index f33e68b64e..c96ee5e6b8 100644 --- a/docs/PHI.md +++ b/docs/PHI.md @@ -776,7 +776,9 @@ for defense-in-depth without swapping the `aiosqlite` connector. `[secret_rotation].allow_lost_audit_range_key` entry in [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) states when, and the way out. **A roll that refuses for good holds the key for good (BACKLOG #4934, #4935).** The roll verifies the whole chain first, so a chain whose rows do not verify, or whose rows open with no - genesis row, refuses under every key, a new one included, and the record stays open. Where a run that + genesis row, refuses under every key, a new one included, and the record stays open. A newest + range under a key that is not configured is not one of these, though `audit-verify` reports it + too: put that key back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and re-run. Where a run that opened the record meets a refusal for good, it names at least two ways out: `[secret_rotation].store_key_last_rotated`, set to the date of a run that re-sealed the data with no skip, and `[secret_rotation].enforce_store_key_expiry = false`, which lifts only the start diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index 4f12d3026f..bebb3db5c6 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -848,21 +848,18 @@ the next start may do. key the chain's newest range is under is not configured. It refuses whether or not the key in use keyed an earlier range. With it on, that start goes ahead: its first audit row goes into that range under the key in use, never verifies there, and the chain breaks at that row, and - `rotate-key` cannot roll the chain afterwards. At least the `admin-*` commands, - `issue-enrolment-code`, `provision-admin` and `backup` do not refuse; one run of any of them in - that state breaks the chain, and `rotate-key` then refuses for good, even after the key is put - back. + `rotate-key` cannot roll the chain afterwards. A command that writes an audit row in that state + breaks the chain, and `rotate-key` then cannot roll it, even after the key is put back. - **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, and every other start check. - **When acceptable:** only when the key the chain's newest range is under is truly lost. If the - key exists anywhere and no command that writes an audit row has run in that state, put it back in - `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` instead. Its audit chain step says whether it rolled the chain and, where it refuses, why. Where + key exists anywhere, put it back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` + instead. Its audit chain step says whether it rolled the chain and, where it refuses, why. Where the key in use already keyed an earlier range, that step refuses and asks for a new key. - **Compensating controls:** none that restore the chain. Record when and why the key was lost, keep the `audit-verify` output from before the start, and treat the break as a known event in any audit review. -- **Reversible:** no, once a start has used it, or any command that writes an audit row has run in - that state: the chain stays broken at that row. Turned back +- **Reversible:** no, once a start has used it: the chain stays broken at that row. Turned back off, the next start refuses again while the lost range key is still not configured. Set only and never used, it can be removed. From ddfee4ce2ebe6a06fdf5da0dce768b30ecd67e91 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 05:40:20 -0600 Subject: [PATCH 32/41] PHI.md restores the new-key-alone rule for every audit-writing command, and the override entry states its lasting cost As the Lander held on PR 2991 at 8dd4b60f: PHI.md again says not to start serve, or run any other command that writes an audit row, on the new key alone before a run's roll succeeds, beside the pointer to the refusal. The 4934 fragment names at least the admin-* commands, issue-enrolment-code and provision-admin. The allow_lost_audit_range_key entry says every later start on that key refuses again without the switch, and that while on it also lets through an unrelated later lost-key change. 'shard' becomes 'engine shard' in the entry and the supervise comment. Docs and a comment only. BACKLOG #4934 --- changelog.d/4934.security.md | 2 +- docs/PHI.md | 6 +++++- docs/SECURITY-LOOSENING.md | 9 +++++---- messagefoundry/__main__.py | 2 +- 4 files changed, 12 insertions(+), 7 deletions(-) diff --git a/changelog.d/4934.security.md b/changelog.d/4934.security.md index 61bf26e359..f02af63d47 100644 --- a/changelog.d/4934.security.md +++ b/changelog.d/4934.security.md @@ -4,5 +4,5 @@ **`serve` can now refuse to start on a changed store key the keyed audit chain cannot follow.** `[secret_rotation].allow_lost_audit_range_key = true` starts anyway and is reported as a security loosening; its entry in `docs/SECURITY-LOOSENING.md` states when the refusal fires and the way - out. The `admin-*` commands do not refuse. + out. At least the `admin-*` commands, `issue-enrolment-code` and `provision-admin` do not refuse. (`BACKLOG #4934`, `BACKLOG #4936`) diff --git a/docs/PHI.md b/docs/PHI.md index c96ee5e6b8..388e57b4de 100644 --- a/docs/PHI.md +++ b/docs/PHI.md @@ -772,7 +772,11 @@ for defense-in-depth without swapping the `aiosqlite` connector. store with no keyed chain to ask, such as one whose audit rows open with no genesis row naming its key, so there a stamp proves only a changed key. On such a store, do not start `serve` between setting a new key and running `rotate-key`: that stamp stands however the run ends. On a - store with a keyed chain, `serve` can refuse to start on a changed key (BACKLOG #4934); the + store with a keyed chain, do not start `serve`, or run any other command that writes an audit + row, on the new key alone before a run's roll succeeds: the chain's current range is then under + a key it does not hold, so it keys new audit rows under the new key, those rows never verify + inside that range, and no later `rotate-key` can roll the chain. `serve` can refuse to start on + a changed key (BACKLOG #4934); the `[secret_rotation].allow_lost_audit_range_key` entry in [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) states when, and the way out. **A roll that refuses for good holds the key for good (BACKLOG #4934, #4935).** The roll verifies the whole chain first, so a chain whose rows do not verify, or whose rows open with no diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index bebb3db5c6..e4eb69a19c 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -839,7 +839,7 @@ Named as `allow_lost_audit_range_key` (BACKLOG #4934), whatever the store holds, the next start may do. - **What you lose:** the `serve` and `supervise` start refusal that keeps the keyed audit chain across a store key change. This entry states when that refusal fires and the way out. With it off, - `serve` (and each `supervise` shard) refuses to start when all of these hold: the engine holds a + `serve` (and each `supervise` engine shard) refuses to start when all of these hold: the engine holds a local store key, so not under `[store].cipher_provider = "vault_transit"`; the stamp does not name that key, which covers a different key, no stamp, and a stamp under a scheme this release does not know; the store has a keyed audit chain in which the open found no range that fails to @@ -859,9 +859,10 @@ the next start may do. - **Compensating controls:** none that restore the chain. Record when and why the key was lost, keep the `audit-verify` output from before the start, and treat the break as a known event in any audit review. -- **Reversible:** no, once a start has used it: the chain stays broken at that row. Turned back - off, the next start refuses again while the lost range key is still not configured. Set only and - never used, it can be removed. +- **Reversible:** no, once a start has used it: the chain stays broken at that row. Once it has let + a start through on a key whose range key is truly lost, every later start on that key refuses + again without it. While it stays on, it also lets through a later, unrelated store key change + whose range key is lost. Set only and never used, it can be removed. ### `[cert_monitor].warn_days` or `[secret_rotation].warn_days` at `0`, or below its default — a credential reminder is off or late Named as `cert_monitor.warn_days` and `secret_rotation.warn_days` (ASVS 6.4.5, BACKLOG #2227). A diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index 11c3bc2c8f..6b7b56afb8 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -5266,7 +5266,7 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P state_dir = generated_state_dir(store_path) replaced: list[GeneratedPairReplaced] = [] - # BACKLOG #4934: a start that may replace the pair opens the store and takes the shards' start + # BACKLOG #4934: a start that may replace the pair opens the store and takes the engine shards' start # refusal before it renews the pair. Any other start mints or reuses as before, and opens the # store only if that replaced a pair after all. renew_after_refusal = generated_pair_may_be_replaced(settings.api, state_dir=state_dir) From 8835939a6b8bc6998967f66ad289223f0fc97183 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 07:16:23 -0600 Subject: [PATCH 33/41] The early store open before the API pair mint no longer swallows the store-cipher alert Lander hold 6948655708 on PR 2991, item 1 (BACKLOG #4934). serve: when the open in _refuse_unrolled_key_before_tls_write is refused on an unmarked value (UnmarkedCipherError only, not CipherError), the start no longer exits there. It goes on to the mint and the lifespan, whose open passes refusal_hook, is refused on the same value and sends the store-cipher integrity_drift alert, the order main ran. supervise: _renew_api_tls_before_spawning falls back to main's path on that refusal from the pre-renewal open: mint, return if nothing was replaced, otherwise open the store to audit the replacement. Both docstrings are rewritten true. Two tests, red on ddfee4ce, pin the alert reaching the sink for serve and for supervise. --- messagefoundry/__main__.py | 76 +++++++----- ...rotate_key_open_hook_and_lost_range_key.py | 114 ++++++++++++++++++ 2 files changed, 163 insertions(+), 27 deletions(-) diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index 6b7b56afb8..fd68134ce8 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -5243,8 +5243,13 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P A re-mint is audited like serve's, in the store the shards are about to open, over the same derived store-hop posture serve's lifespan opens it with. On a start that may replace the pair, that store is opened first and takes serve's start refusal before the renewal runs (BACKLOG - #4934), and a failure to open propagates before any renewal. A start that replaced a pair it - did not expect to opens it after, as before. + #4934), and a failure to open propagates before any renewal, except one: an open refused on an + unmarked value (``UnmarkedCipherError``). That open passes no refusal hook, so the start falls + back to the order it had before #4934: mint, return if nothing was replaced, so that each engine + shard's hooked open sends the store-cipher alert, and otherwise open the store to audit the + replacement, which raises the refusal again. Any other start mints first and opens the store + only if that replaced a pair, which then takes the start refusal before the audit row. In both + of those orders a refusal raised after the mint leaves the replaced pair unaudited. """ import asyncio @@ -5259,6 +5264,7 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P from messagefoundry.config.settings import hop_posture_from_ai, keyless_opt_out_refusal from messagefoundry.last_resort import run_guarded from messagefoundry.store import open_store + from messagefoundry.store.crypto import UnmarkedCipherError cwd = Path.cwd() root = resolve_project_root(settings.environments.base_dir or None, cwd=cwd) @@ -5266,14 +5272,6 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P state_dir = generated_state_dir(store_path) replaced: list[GeneratedPairReplaced] = [] - # BACKLOG #4934: a start that may replace the pair opens the store and takes the engine shards' start - # refusal before it renews the pair. Any other start mints or reuses as before, and opens the - # store only if that replaced a pair after all. - renew_after_refusal = generated_pair_may_be_replaced(settings.api, state_dir=state_dir) - if not renew_after_refusal: - ensure_api_tls_material(settings.api, state_dir=state_dir, replacements=replaced) - if not replaced: - return state_dir posture = ( hop_posture_from_ai(settings.ai, enforcement=settings.security.enforcement) if settings.ai is not None @@ -5284,13 +5282,19 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P # as it does there, and a store with no key and no opt-out is refused here as serve would refuse. shard_store = settings.store.model_copy(update={"path": store_path}) - async def _audit() -> None: - store = await open_store( - shard_store, - create=True, - posture=posture, - keyless_chain_refusal=keyless_opt_out_refusal(shard_store, settings.security), - ) + async def _audit(*, renew_after_refusal: bool) -> bool: + """``False`` only when the open ahead of a renewal is refused on an unmarked value.""" + try: + store = await open_store( + shard_store, + create=True, + posture=posture, + keyless_chain_refusal=keyless_opt_out_refusal(shard_store, settings.security), + ) + except UnmarkedCipherError: + if not renew_after_refusal: + raise + return False try: # BACKLOG #4934: before the renewal and before the audit row below, which is this # store's first since the open. @@ -5306,8 +5310,18 @@ async def _audit() -> None: await record_generated_pair_replacements(store, replaced) finally: await store.close() + return True - run_guarded(_audit()) + # BACKLOG #4934: a start that may replace the pair opens the store and takes the engine shards' + # start refusal before it renews the pair. Any other start, and one whose early open is refused + # on an unmarked value, mints or reuses first and opens the store only if that replaced a pair. + if generated_pair_may_be_replaced(settings.api, state_dir=state_dir) and run_guarded( + _audit(renew_after_refusal=True) + ): + return state_dir + ensure_api_tls_material(settings.api, state_dir=state_dir, replacements=replaced) + if replaced: + run_guarded(_audit(renew_after_refusal=False)) return state_dir @@ -5338,11 +5352,14 @@ def _refuse_unrolled_key_before_tls_write(settings: ServiceSettings) -> None: refuses again after its own open; this one exists only because the mint comes first. Raises :class:`~messagefoundry.pipeline.secret_rotation.UnrolledStoreKeyRefused`, the keyless-chain refusal, :class:`_StoreOpenRefused` for an open refusal :func:`_open_store_or_refuse` maps, and - anything else the open raises. This open passes no ``refusal_hook``, so an at-open cipher - refusal reaches the log, not the alert notifier the lifespan's open would send it to.""" + anything else the open raises, except one. This open passes no ``refusal_hook``, so an open + refused on an unmarked value (``UnmarkedCipherError``) returns without the check instead: the + start goes on to the mint and the lifespan, whose open passes the hook, is refused on the same + value and sends the store-cipher alert.""" from messagefoundry.config.settings import hop_posture_from_ai, keyless_opt_out_refusal from messagefoundry.last_resort import run_guarded from messagefoundry.store import open_store + from messagefoundry.store.crypto import UnmarkedCipherError posture = ( hop_posture_from_ai(settings.ai, enforcement=settings.security.enforcement) @@ -5351,14 +5368,19 @@ def _refuse_unrolled_key_before_tls_write(settings: ServiceSettings) -> None: ) async def _check() -> None: - store = await _open_store_or_refuse( - open_store( - settings.store, - create=True, - posture=posture, - keyless_chain_refusal=keyless_opt_out_refusal(settings.store, settings.security), + try: + store = await _open_store_or_refuse( + open_store( + settings.store, + create=True, + posture=posture, + keyless_chain_refusal=keyless_opt_out_refusal( + settings.store, settings.security + ), + ) ) - ) + except UnmarkedCipherError: + return # the lifespan's hooked open refuses it again and alerts, as before #4934 try: await _refuse_unrolled_key_on(store, settings) finally: diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index a5c576d4ba..31b8f4c20f 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -422,3 +422,117 @@ async def test_serve_starts_on_the_new_key_alone_under_the_override(tmp_path: Pa app = await asyncio.to_thread(_app_on_the_new_key_alone, tmp_path, True) async with app.router.lifespan_context(app): assert app.state.engine is not None + + +def _seed_a_planted_state_value(db: Path, key: str) -> None: + """A keyed synthetic store whose eagerly loaded ``state`` table holds an unmarked value beside a + sealed one, so every keyed open of it raises ``UnmarkedCipherError``.""" + from tests.test_store_strict_ciphertext import _keyed, _plant_state + from tests.test_store_strict_ciphertext import _seed as _seed_messages + + asyncio.run(_seed_messages(db, 1, _keyed(key))) + _plant_state(db, key) + + +def _sink_for_the_lifespan(monkeypatch: pytest.MonkeyPatch) -> Any: + """The notifier every lifespan built from here on uses, recording what it is sent.""" + import importlib + + from tests.test_store_strict_ciphertext import _LifecycleSink + + sink = _LifecycleSink() + app_mod = importlib.import_module("messagefoundry.api.app") + monkeypatch.setattr(app_mod, "notifier_from_settings", lambda *a, **k: sink) + return sink + + +def _assert_the_store_cipher_alert(sink: Any) -> None: + assert sink.events and {e[0] for e in sink.events} == {"store-cipher"}, sink.events + assert all("state.value" in reason for _subject, reason, _count in sink.events), sink.events + + +def test_serve_alerts_on_an_unmarked_state_value_on_a_renewal_due_start( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """#4934 repair: the early open ahead of the mint passes no refusal hook, so an unmarked value + it is refused on must not end the start there. The start goes on to the mint and the lifespan, + whose hooked open is refused again and sends the store-cipher alert, as on main.""" + pytest.importorskip("psutil") + from fastapi.testclient import TestClient + + from messagefoundry.store.crypto import CipherError + + db, key = tmp_path / "messagefoundry.db", generate_key() + _seed_a_planted_state_value(db, key) + _mint_a_pair_due_for_renewal(tmp_path, monkeypatch) + sink = _sink_for_the_lifespan(monkeypatch) + refused: list[BaseException] = [] + + def run_the_lifespan(app: Any, *_a: object, **_k: object) -> None: + try: + with TestClient(app): + pass # pragma: no cover -- the open is refused + except CipherError as exc: + refused.append(exc) + + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("MEFOR_STORE_ENCRYPTION_KEY", key) + # The enforce-only start gates (an off-box log collector, ...) are not under test here. + monkeypatch.setenv("MEFOR_SECURITY_ENFORCEMENT", "warn") + monkeypatch.setattr("uvicorn.run", run_the_lifespan) + (tmp_path / "messagefoundry.toml").write_text( + "security.block_unlisted_outbound = true\n" + "alerts.security_notifications_required = false\n" + "security.local_access_only = true\n", + encoding="utf-8", + ) + samples = Path(__file__).resolve().parent.parent / "samples" / "config" + main(["serve", "--config", str(samples), "--env", "dev"]) + assert refused, "the lifespan never ran its hooked open" + _assert_the_store_cipher_alert(sink) + + +def test_supervise_alerts_on_an_unmarked_state_value_when_the_pair_is_not_replaced( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """#4934 repair: a fleet start whose pair may be replaced opens the store before the renewal, + with no refusal hook. An unmarked value refused there falls back to main's path, so a start + that then replaces nothing returns and its engine shards start, and an engine shard's lifespan + open is hooked and sends the store-cipher alert. Before the repair the early open ended the + start, and no engine shard ran to send it. On a start that does replace the pair, main's own + open after the renewal passed no hook either, so that arm has no alert to restore.""" + pytest.importorskip("psutil") + from fastapi.testclient import TestClient + + from messagefoundry import __main__ as cli + from messagefoundry.api import create_managed_app, tls + from messagefoundry.api.tls import generated_state_dir + from messagefoundry.config.settings import ( + AlertsSettings, + ApiSettings, + EgressSettings, + ServiceSettings, + ) + from messagefoundry.store.crypto import CipherError + + db, key = tmp_path / "sup.db", generate_key() + _seed_a_planted_state_value(db, key) + state_dir = generated_state_dir(str(db)) + tls.ensure_api_tls_material(ApiSettings(), state_dir=state_dir) + minted = (state_dir / "api-generated-cert.pem").read_bytes() + # A pair the read-only check cannot clear, which the mint under the lock then reuses. + monkeypatch.setattr(tls, "generated_pair_may_be_replaced", lambda *a, **k: True) + settings = ServiceSettings(store=StoreSettings(path=str(db), encryption_key=key)) + assert cli._renew_api_tls_before_spawning(settings, str(db)) == state_dir + assert (state_dir / "api-generated-cert.pem").read_bytes() == minted + # The engine shard the fleet now starts on that store. + sink = _sink_for_the_lifespan(monkeypatch) + app = create_managed_app( + store_settings=StoreSettings(path=str(db), encryption_key=key), + alerts_settings=AlertsSettings(), + poll_interval=0.05, + egress_settings=EgressSettings(deny_by_default=False), + ) + with pytest.raises(CipherError), TestClient(app): + pass # pragma: no cover -- the open is refused + _assert_the_store_cipher_alert(sink) From 738e9e9e3c7812c5c88806db7a29e8b0c57f7143 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 07:47:45 -0600 Subject: [PATCH 34/41] The unmarked-value fallback logs a WARNING, and its prose says what it leaves unaudited Review round 1 on the repair of Lander hold 6948655708 (BACKLOG #4934). Both UnmarkedCipherError handlers log a WARNING rather than swallow the refusal silently. The serve call-site comment, the two docstrings and the test assertion are rewritten true: a pair the fallback mint replaces is left unaudited, as on main, and the supervise arm that replaced the pair sends no alert. The test's alert check now also requires the alert to name no planted content and to have been sent, not queued. --- messagefoundry/__main__.py | 34 ++++++++++++++----- ...rotate_key_open_hook_and_lost_range_key.py | 5 ++- 2 files changed, 29 insertions(+), 10 deletions(-) diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index fd68134ce8..6eba3af4ea 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -4910,6 +4910,9 @@ def registry_filter(reg: Registry) -> Registry: # noqa: F811 (local shard-bound _state_dir = generated_state_dir(settings.store.path) # BACKLOG #4934: the lifespan's start refusal runs after this mint, so a start that may replace # the pair takes it here first. A first-run mint replaces nothing, so it needs no early open. + # An early open refused on an unmarked value is the exception: that start goes on to the mint, + # which may replace the pair, and the lifespan's hooked open then refuses it and alerts before + # the replacement is audited, the order main ran. if generated_pair_may_be_replaced(settings.api, state_dir=_state_dir, renew=args.shard is None): from messagefoundry.pipeline.secret_rotation import UnrolledStoreKeyRefused from messagefoundry.store.base import KeylessAuditChainRefused @@ -5245,11 +5248,13 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P that store is opened first and takes serve's start refusal before the renewal runs (BACKLOG #4934), and a failure to open propagates before any renewal, except one: an open refused on an unmarked value (``UnmarkedCipherError``). That open passes no refusal hook, so the start falls - back to the order it had before #4934: mint, return if nothing was replaced, so that each engine - shard's hooked open sends the store-cipher alert, and otherwise open the store to audit the - replacement, which raises the refusal again. Any other start mints first and opens the store - only if that replaced a pair, which then takes the start refusal before the audit row. In both - of those orders a refusal raised after the mint leaves the replaced pair unaudited. + back to the order main ran: it logs a WARNING, mints, and returns if nothing was replaced, and + an engine shard that then starts sends the store-cipher alert from its hooked open. If the mint + replaced the pair, it opens the store again to audit the replacement, with no hook, as main + did, and that open raises the refusal again: the start is refused, the replaced pair is left + unaudited, and no alert is sent. Any other start mints first and opens the store only if that + replaced a pair, which then takes the start refusal before the audit row; a refusal there also + leaves the replaced pair unaudited. """ import asyncio @@ -5294,6 +5299,10 @@ async def _audit(*, renew_after_refusal: bool) -> bool: except UnmarkedCipherError: if not renew_after_refusal: raise + logging.getLogger(__name__).warning( + "the store open before the API pair renewal was refused on an unmarked value; the " + "start refusal was not checked, and the fleet start goes on to the mint" + ) return False try: # BACKLOG #4934: before the renewal and before the audit row below, which is this @@ -5353,9 +5362,11 @@ def _refuse_unrolled_key_before_tls_write(settings: ServiceSettings) -> None: :class:`~messagefoundry.pipeline.secret_rotation.UnrolledStoreKeyRefused`, the keyless-chain refusal, :class:`_StoreOpenRefused` for an open refusal :func:`_open_store_or_refuse` maps, and anything else the open raises, except one. This open passes no ``refusal_hook``, so an open - refused on an unmarked value (``UnmarkedCipherError``) returns without the check instead: the - start goes on to the mint and the lifespan, whose open passes the hook, is refused on the same - value and sends the store-cipher alert.""" + refused on an unmarked value (``UnmarkedCipherError``) logs a WARNING and returns without the + check instead. The start goes on to the mint, which may replace the pair, and, unless a gate + between the two refuses first, to the lifespan, whose open passes the hook, is refused on the + same value and sends the store-cipher alert. A pair the mint replaced is then left unaudited, + as on main.""" from messagefoundry.config.settings import hop_posture_from_ai, keyless_opt_out_refusal from messagefoundry.last_resort import run_guarded from messagefoundry.store import open_store @@ -5380,7 +5391,12 @@ async def _check() -> None: ) ) except UnmarkedCipherError: - return # the lifespan's hooked open refuses it again and alerts, as before #4934 + # The lifespan's hooked open refuses it again and alerts, as before #4934. + logging.getLogger(__name__).warning( + "the store open before the API pair mint was refused on an unmarked value; the " + "start refusal was not checked, and the start goes on to the mint and the store open" + ) + return try: await _refuse_unrolled_key_on(store, settings) finally: diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index 31b8f4c20f..04e6476cb8 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -447,8 +447,11 @@ def _sink_for_the_lifespan(monkeypatch: pytest.MonkeyPatch) -> Any: def _assert_the_store_cipher_alert(sink: Any) -> None: + """The alert names the table and column and no planted content, and was sent, not queued.""" assert sink.events and {e[0] for e in sink.events} == {"store-cipher"}, sink.events - assert all("state.value" in reason for _subject, reason, _count in sink.events), sink.events + for _subject, reason, _count in sink.events: + assert "state.value" in reason and "666" not in reason and "row-key-9" not in reason + assert sink.started and sink.drained def test_serve_alerts_on_an_unmarked_state_value_on_a_renewal_due_start( From 2cab09db6b2473bc29a45e205844926f478fb1b4 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 08:09:47 -0600 Subject: [PATCH 35/41] The unmarked-value fallback's prose names its exit and the sweep's unhooked report, and its tests pin the WARNINGs Review round 2 on the repair of Lander hold 6948655708 (BACKLOG #4934). The serve call-site comment points to the one docstring that states the fallback, instead of restating it. The supervise docstring names the dispatch-floor exit on the arm that replaced the pair. The serve docstring restores the true statement that a planted row the early open's sweep reports, without raising, reaches only the log. Both tests now require the fallback's WARNING. --- messagefoundry/__main__.py | 14 ++++++----- ...rotate_key_open_hook_and_lost_range_key.py | 25 +++++++++++++++++++ 2 files changed, 33 insertions(+), 6 deletions(-) diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index 6eba3af4ea..042a519a37 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -4910,9 +4910,8 @@ def registry_filter(reg: Registry) -> Registry: # noqa: F811 (local shard-bound _state_dir = generated_state_dir(settings.store.path) # BACKLOG #4934: the lifespan's start refusal runs after this mint, so a start that may replace # the pair takes it here first. A first-run mint replaces nothing, so it needs no early open. - # An early open refused on an unmarked value is the exception: that start goes on to the mint, - # which may replace the pair, and the lifespan's hooked open then refuses it and alerts before - # the replacement is audited, the order main ran. + # An early open refused on an unmarked value is the exception; _refuse_unrolled_key_before_tls_write + # says what that start does instead. if generated_pair_may_be_replaced(settings.api, state_dir=_state_dir, renew=args.shard is None): from messagefoundry.pipeline.secret_rotation import UnrolledStoreKeyRefused from messagefoundry.store.base import KeylessAuditChainRefused @@ -5251,8 +5250,9 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P back to the order main ran: it logs a WARNING, mints, and returns if nothing was replaced, and an engine shard that then starts sends the store-cipher alert from its hooked open. If the mint replaced the pair, it opens the store again to audit the replacement, with no hook, as main - did, and that open raises the refusal again: the start is refused, the replaced pair is left - unaudited, and no alert is sent. Any other start mints first and opens the store only if that + did, and that open raises ``UnmarkedCipherError`` again, which ``supervise`` does not catch: it + ends the start at the dispatch floor, the replaced pair is left unaudited, and no alert is + sent. Any other start mints first and opens the store only if that replaced a pair, which then takes the start refusal before the audit row; a refusal there also leaves the replaced pair unaudited. """ @@ -5366,7 +5366,9 @@ def _refuse_unrolled_key_before_tls_write(settings: ServiceSettings) -> None: check instead. The start goes on to the mint, which may replace the pair, and, unless a gate between the two refuses first, to the lifespan, whose open passes the hook, is refused on the same value and sends the store-cipher alert. A pair the mint replaced is then left unaudited, - as on main.""" + as on main. A planted row the open's at-open sweep finds and leaves in place, without raising, + is reported to no hook, so it reaches only the log; if the check then refuses, the start ends + here and no hooked open runs to alert on it.""" from messagefoundry.config.settings import hop_posture_from_ai, keyless_opt_out_refusal from messagefoundry.last_resort import run_guarded from messagefoundry.store import open_store diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index 04e6476cb8..af80ebfd1f 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -446,6 +446,27 @@ def _sink_for_the_lifespan(monkeypatch: pytest.MonkeyPatch) -> Any: return sink +class _Recorded(logging.Handler): + """The WARNINGs ``messagefoundry.__main__`` logs. A handler on that logger, since ``serve`` + reconfigures the root handlers that ``caplog`` relies on.""" + + def __init__(self) -> None: + super().__init__(logging.WARNING) + self.lines: list[str] = [] + + def emit(self, record: logging.LogRecord) -> None: + self.lines.append(record.getMessage()) + + +def _record_the_cli_warnings(monkeypatch: pytest.MonkeyPatch) -> _Recorded: + recorded = _Recorded() + cli_log = logging.getLogger("messagefoundry.__main__") + # monkeypatch restores the logger's own handler list and state after the test. + monkeypatch.setattr(cli_log, "handlers", [*cli_log.handlers, recorded]) + monkeypatch.setattr(cli_log, "disabled", False) + return recorded + + def _assert_the_store_cipher_alert(sink: Any) -> None: """The alert names the table and column and no planted content, and was sent, not queued.""" assert sink.events and {e[0] for e in sink.events} == {"store-cipher"}, sink.events @@ -469,6 +490,7 @@ def test_serve_alerts_on_an_unmarked_state_value_on_a_renewal_due_start( _seed_a_planted_state_value(db, key) _mint_a_pair_due_for_renewal(tmp_path, monkeypatch) sink = _sink_for_the_lifespan(monkeypatch) + recorded = _record_the_cli_warnings(monkeypatch) refused: list[BaseException] = [] def run_the_lifespan(app: Any, *_a: object, **_k: object) -> None: @@ -493,6 +515,7 @@ def run_the_lifespan(app: Any, *_a: object, **_k: object) -> None: main(["serve", "--config", str(samples), "--env", "dev"]) assert refused, "the lifespan never ran its hooked open" _assert_the_store_cipher_alert(sink) + assert any("before the API pair mint was refused" in line for line in recorded.lines) def test_supervise_alerts_on_an_unmarked_state_value_when_the_pair_is_not_replaced( @@ -526,8 +549,10 @@ def test_supervise_alerts_on_an_unmarked_state_value_when_the_pair_is_not_replac # A pair the read-only check cannot clear, which the mint under the lock then reuses. monkeypatch.setattr(tls, "generated_pair_may_be_replaced", lambda *a, **k: True) settings = ServiceSettings(store=StoreSettings(path=str(db), encryption_key=key)) + recorded = _record_the_cli_warnings(monkeypatch) assert cli._renew_api_tls_before_spawning(settings, str(db)) == state_dir assert (state_dir / "api-generated-cert.pem").read_bytes() == minted + assert any("before the API pair renewal was refused" in line for line in recorded.lines) # The engine shard the fleet now starts on that store. sink = _sink_for_the_lifespan(monkeypatch) app = create_managed_app( From 2b48c688335227b24c897c55ccecf8be51bf0ee4 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 09:47:13 -0600 Subject: [PATCH 36/41] The API pair renewal is decided once, before the start refusal, and two claims are made true Lander hold 6952403017 on PR 2991, items 2 to 4 (BACKLOG #4934). serve and supervise pass the read-only generated_pair_may_be_replaced answer to the mint as its renew, so a pair that falls due after that check is not renewed past the start refusal it skipped. Two tests, red on 2cab09db, pin it with a renewal clock that changes between the calls. docs/SECURITY-LOOSENING.md no longer says a start "goes ahead" under allow_lost_audit_range_key; it says this refusal does not fire. The crypto inventory comment credits serve's early check as well as supervise's. Item 1, a refusal hook on the early opens, is not in this commit. --- docs/SECURITY-LOOSENING.md | 8 +-- messagefoundry/__main__.py | 21 ++++-- scripts/security/crypto_inventory_check.py | 4 +- ...rotate_key_open_hook_and_lost_range_key.py | 65 +++++++++++++++++++ 4 files changed, 85 insertions(+), 13 deletions(-) diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index e4eb69a19c..d595f5c68f 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -846,10 +846,10 @@ the next start may do. authenticate (where it finds one, the refusal does not fire, though that start's first audit row may still break the chain); and the key the chain's newest range is under is not configured. It refuses whether or not the key in - use keyed an earlier range. With it on, that start goes ahead: its first audit row goes into - that range under the key in use, never verifies there, and the chain breaks at that row, and - `rotate-key` cannot roll the chain afterwards. A command that writes an audit row in that state - breaks the chain, and `rotate-key` then cannot roll it, even after the key is put back. + use keyed an earlier range. With it on, this refusal does not fire. An audit row written in that + state, by a start or any other command, goes into that range under the key in use and never + verifies there, so the chain breaks at that row, and `rotate-key` then cannot roll it, even after + the key is put back. - **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, and every other start check. - **When acceptable:** only when the key the chain's newest range is under is truly lost. If the diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index 042a519a37..3cace2b8c2 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -4911,8 +4911,12 @@ def registry_filter(reg: Registry) -> Registry: # noqa: F811 (local shard-bound # BACKLOG #4934: the lifespan's start refusal runs after this mint, so a start that may replace # the pair takes it here first. A first-run mint replaces nothing, so it needs no early open. # An early open refused on an unmarked value is the exception; _refuse_unrolled_key_before_tls_write - # says what that start does instead. - if generated_pair_may_be_replaced(settings.api, state_dir=_state_dir, renew=args.shard is None): + # says what that start does instead. The answer is also the mint's `renew`, so renewal is decided + # once: a pair that falls due after this check is not renewed past the refusal it skipped. + _may_replace = generated_pair_may_be_replaced( + settings.api, state_dir=_state_dir, renew=args.shard is None + ) + if _may_replace: from messagefoundry.pipeline.secret_rotation import UnrolledStoreKeyRefused from messagefoundry.store.base import KeylessAuditChainRefused @@ -4932,7 +4936,7 @@ def registry_filter(reg: Registry) -> Registry: # noqa: F811 (local shard-bound # An engine shard never renews: `supervise` renews before it spawns the whole fleet, so # a lone restarted shard cannot leave its siblings serving a different certificate # (#1276). - renew=args.shard is None, + renew=args.shard is None and _may_replace, ) except RestrictedFileError as exc: # The TLS key could not be created restricted, and no usable pair exists to fall back on @@ -5324,11 +5328,14 @@ async def _audit(*, renew_after_refusal: bool) -> bool: # BACKLOG #4934: a start that may replace the pair opens the store and takes the engine shards' # start refusal before it renews the pair. Any other start, and one whose early open is refused # on an unmarked value, mints or reuses first and opens the store only if that replaced a pair. - if generated_pair_may_be_replaced(settings.api, state_dir=state_dir) and run_guarded( - _audit(renew_after_refusal=True) - ): + # The check's answer is also the mint's `renew`, so renewal is decided once: a pair that falls + # due after the check is not renewed past the refusal it skipped. + may_replace = generated_pair_may_be_replaced(settings.api, state_dir=state_dir) + if may_replace and run_guarded(_audit(renew_after_refusal=True)): return state_dir - ensure_api_tls_material(settings.api, state_dir=state_dir, replacements=replaced) + ensure_api_tls_material( + settings.api, state_dir=state_dir, replacements=replaced, renew=may_replace + ) if replaced: run_guarded(_audit(renew_after_refusal=False)) return state_dir diff --git a/scripts/security/crypto_inventory_check.py b/scripts/security/crypto_inventory_check.py index a24142f1e3..6614c73982 100644 --- a/scripts/security/crypto_inventory_check.py +++ b/scripts/security/crypto_inventory_check.py @@ -1073,8 +1073,8 @@ "compare:via messagefoundry.api.tls", # BACKLOG #1167: evaluate_anchor's pin compare "compare:via messagefoundry.auth.totp", "compare:via messagefoundry.pipeline.secret_rotation", - # BACKLOG #4934: supervise's start refusal reads the audit chain's range keys the open - # settled, as serve's lifespan does. + # BACKLOG #4934: the start refusal that serve and supervise run before the API pair + # mint reads the audit chain's range keys the open settled. "hash:via messagefoundry.store.store", # BACKLOG #3470: anchor_refusal checks the forwarder's and the SMTP relay's CA before # those hops are built, through evaluate_anchor: its SHA-256 fingerprint, and its pin diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index af80ebfd1f..437a83ff67 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -296,6 +296,71 @@ def _mint_a_pair_due_for_renewal(state_dir: Path, monkeypatch: pytest.MonkeyPatc return (state_dir / "api-generated-cert.pem").read_bytes() +def _mint_a_pair_that_falls_due_after_the_first_check( + state_dir: Path, monkeypatch: pytest.MonkeyPatch +) -> bytes: + """As :func:`_mint_a_pair_due_for_renewal`, but the pair is reported fresh to the first renewal + check after this and due to every later one: a pair crossing the renewal threshold between the + start's read-only check and its mint.""" + from messagefoundry.api import tls + from messagefoundry.config.settings import ApiSettings + + tls.ensure_api_tls_material(ApiSettings(), state_dir=state_dir) + checks: list[Path] = [] + + def due_after_the_first_check(cert_path: Path) -> Any: + checks.append(cert_path) + return None if len(checks) == 1 else tls._cert_facts(cert_path) + + monkeypatch.setattr(tls, "_renewal_due", due_after_the_first_check) + return (state_dir / "api-generated-cert.pem").read_bytes() + + +def test_a_pair_falling_due_mid_start_is_not_renewed_by_a_refused_supervise_start( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """#4934: renewal is decided once, by the check the refusal runs on. A pair that falls due after + that check is not renewed by the mint, so a start its engine shards will refuse leaves it + unchanged. Before, the mint renewed it and the refusal then left the renewal unaudited.""" + from messagefoundry import __main__ as cli + from messagefoundry.api.tls import generated_state_dir + from messagefoundry.config.settings import ServiceSettings + + db, key_a, key_b = tmp_path / "sup.db", generate_key(), generate_key() + asyncio.run(_seed(db, tmp_path / "uploads", key_a)) + state_dir = generated_state_dir(str(db)) + minted = _mint_a_pair_that_falls_due_after_the_first_check(state_dir, monkeypatch) + settings = ServiceSettings(store=StoreSettings(path=str(db), encryption_key=key_b)) + before = _audit_rows(db) + # Nothing is replaced, so the store is not opened here; each engine shard's lifespan refuses. + assert cli._renew_api_tls_before_spawning(settings, str(db)) == state_dir + assert (state_dir / "api-generated-cert.pem").read_bytes() == minted + assert _audit_rows(db) == before + + +def test_a_pair_falling_due_mid_start_is_not_renewed_by_a_refused_serve_start( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """#4934: as the supervise test above, for serve, whose lifespan refuses after the mint.""" + db, key_a, key_b = tmp_path / "messagefoundry.db", generate_key(), generate_key() + asyncio.run(_seed(db, tmp_path / "uploads", key_a)) + minted = _mint_a_pair_that_falls_due_after_the_first_check(tmp_path, monkeypatch) + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("MEFOR_STORE_ENCRYPTION_KEY", key_b) + # The enforce-only start gates (an off-box log collector, ...) are not under test here. + monkeypatch.setenv("MEFOR_SECURITY_ENFORCEMENT", "warn") + monkeypatch.setattr("uvicorn.run", lambda *a, **k: None) + (tmp_path / "messagefoundry.toml").write_text( + "security.block_unlisted_outbound = true\n" + "alerts.security_notifications_required = false\n" + "security.local_access_only = true\n", + encoding="utf-8", + ) + samples = Path(__file__).resolve().parent.parent / "samples" / "config" + main(["serve", "--config", str(samples), "--env", "dev"]) + assert (tmp_path / "api-generated-cert.pem").read_bytes() == minted + + @pytest.mark.parametrize("allow_lost", [False, True]) def test_a_refused_supervise_start_leaves_the_api_pair_unchanged( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, allow_lost: bool From 3b3f8ecba260b84372b094cb6932a68f915c137b Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 10:08:19 -0600 Subject: [PATCH 37/41] The renewal-once change's contracts, test control and two prose claims say what the code does Review round 1 on the repair of Lander hold 6952403017 (BACKLOG #4934). The tls.py docstrings of generated_pair_may_be_replaced and ensure_api_tls_material now say serve and supervise pass the check's False to the mint as renew=False, and that recovery is still decided again at the mint. The shard test's mutation note names both clauses (measured: both dropped is red, either alone stays green). The new serve test asserts exit 0 and that the server was reached. The loosening bullet says other commands are not refused whatever the switch; the inventory comment no longer says the refusal runs only before the mint. --- docs/SECURITY-LOOSENING.md | 6 +++--- messagefoundry/api/tls.py | 10 +++++++--- scripts/security/crypto_inventory_check.py | 4 ++-- tests/test_api_tls.py | 3 ++- .../test_rotate_key_open_hook_and_lost_range_key.py | 13 +++++++++---- 5 files changed, 23 insertions(+), 13 deletions(-) diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index d595f5c68f..e73cc1768b 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -847,9 +847,9 @@ the next start may do. may still break the chain); and the key the chain's newest range is under is not configured. It refuses whether or not the key in use keyed an earlier range. With it on, this refusal does not fire. An audit row written in that - state, by a start or any other command, goes into that range under the key in use and never - verifies there, so the chain breaks at that row, and `rotate-key` then cannot roll it, even after - the key is put back. + state goes into that range under the key in use and never verifies there, so the chain breaks at + that row, and `rotate-key` then cannot roll it, even after the key is put back. Other commands + are not refused in that state, whatever the switch. - **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, and every other start check. - **When acceptable:** only when the key the chain's newest range is under is truly lost. If the diff --git a/messagefoundry/api/tls.py b/messagefoundry/api/tls.py index 84d45c0560..2a5fab8801 100644 --- a/messagefoundry/api/tls.py +++ b/messagefoundry/api/tls.py @@ -636,8 +636,10 @@ def generated_pair_may_be_replaced( pair already on disk: a renewal, or the recovery of an unusable or half-written pair, each of which it reports as a :class:`GeneratedPairReplaced`. A first-run mint replaces nothing and answers ``False``. Read-only: it writes nothing and takes no lock, so a start can run a - refusal first and leave the pair as it found it on a refused start (BACKLOG #4934). A pair that - changes on disk between this call and the mint is decided again there.""" + refusal first and leave the pair as it found it on a refused start (BACKLOG #4934). ``serve`` + and ``supervise`` pass a ``False`` answer to the mint as ``renew=False``, so a pair that falls + due after this call is not renewed there; whether a pair that stops loading after this call is + recovered is still decided again at the mint.""" if plan_api_tls_material(api, state_dir=state_dir).source != "generated": return False cert_path, key_path = _generated_pair(state_dir) @@ -686,7 +688,9 @@ def ensure_api_tls_material( certificate while its siblings keep the old one in memory. So ``supervise`` renews before it spawns any shard, and a shard only reuses a pair that loads, due or not. It still mints on a first run and still recovers a pair that does not load, because it has nothing else to serve. - **A file or lock failure never costs a start the old pair can still serve**: any ``OSError`` on + A plain ``serve`` and ``supervise`` also pass ``renew=False`` when + :func:`generated_pair_may_be_replaced` found nothing to replace, so they renew only a pair that + check found due (BACKLOG #4934). **A file or lock failure never costs a start the old pair can still serve**: any ``OSError`` on the locked path -- including a lock it could not take -- falls back to that pair with a WARNING, and only a pair that does not load or has expired lets the error propagate. Other errors, such as a ``[api].host`` the certificate builder refuses, still stop the start. diff --git a/scripts/security/crypto_inventory_check.py b/scripts/security/crypto_inventory_check.py index 6614c73982..8d1e96bcb1 100644 --- a/scripts/security/crypto_inventory_check.py +++ b/scripts/security/crypto_inventory_check.py @@ -1073,8 +1073,8 @@ "compare:via messagefoundry.api.tls", # BACKLOG #1167: evaluate_anchor's pin compare "compare:via messagefoundry.auth.totp", "compare:via messagefoundry.pipeline.secret_rotation", - # BACKLOG #4934: the start refusal that serve and supervise run before the API pair - # mint reads the audit chain's range keys the open settled. + # BACKLOG #4934: the start refusal serve and supervise run here reads the audit + # chain's range keys the open settled. "hash:via messagefoundry.store.store", # BACKLOG #3470: anchor_refusal checks the forwarder's and the SMTP relay's CA before # those hops are built, through evaluate_anchor: its SHA-256 fingerprint, and its pin diff --git a/tests/test_api_tls.py b/tests/test_api_tls.py index 1fa4fa7651..f7fc1042c6 100644 --- a/tests/test_api_tls.py +++ b/tests/test_api_tls.py @@ -5452,7 +5452,8 @@ def test_plain_serve_renews_and_an_engine_shard_never_does( reuses it untouched: `supervise` renews for the whole fleet before spawning, so one restarted shard can never serve a certificate its siblings do not. - Mutation: drop ``renew=args.shard is None``. Red: the shard row renews.""" + Mutation: drop ``args.shard is None`` from both the read-only check's ``renew`` and the mint's. + Red: the shard row renews. Dropping it from either one alone stays green.""" cert = tmp_path / _GENERATED_CERT_NAME _plant_generated_pair(tmp_path, lived_days=300, left_days=65) old = cert.read_bytes() diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index 437a83ff67..42bec0b901 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -338,10 +338,12 @@ def test_a_pair_falling_due_mid_start_is_not_renewed_by_a_refused_supervise_star assert _audit_rows(db) == before -def test_a_pair_falling_due_mid_start_is_not_renewed_by_a_refused_serve_start( +def test_a_pair_falling_due_mid_start_is_not_renewed_by_serve( tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: - """#4934: as the supervise test above, for serve, whose lifespan refuses after the mint.""" + """#4934: as the supervise test above, for serve. The read-only check finds the pair fresh, so + no early refusal runs, and the mint must not renew it either: this store's lifespan would + refuse before auditing a renewal. The lifespan itself is not run here.""" db, key_a, key_b = tmp_path / "messagefoundry.db", generate_key(), generate_key() asyncio.run(_seed(db, tmp_path / "uploads", key_a)) minted = _mint_a_pair_that_falls_due_after_the_first_check(tmp_path, monkeypatch) @@ -349,7 +351,8 @@ def test_a_pair_falling_due_mid_start_is_not_renewed_by_a_refused_serve_start( monkeypatch.setenv("MEFOR_STORE_ENCRYPTION_KEY", key_b) # The enforce-only start gates (an off-box log collector, ...) are not under test here. monkeypatch.setenv("MEFOR_SECURITY_ENFORCEMENT", "warn") - monkeypatch.setattr("uvicorn.run", lambda *a, **k: None) + started: list[object] = [] + monkeypatch.setattr("uvicorn.run", lambda *a, **k: started.append(a)) (tmp_path / "messagefoundry.toml").write_text( "security.block_unlisted_outbound = true\n" "alerts.security_notifications_required = false\n" @@ -357,7 +360,9 @@ def test_a_pair_falling_due_mid_start_is_not_renewed_by_a_refused_serve_start( encoding="utf-8", ) samples = Path(__file__).resolve().parent.parent / "samples" / "config" - main(["serve", "--config", str(samples), "--env", "dev"]) + # Positive control: the start ran past the mint and reached the server. + assert main(["serve", "--config", str(samples), "--env", "dev"]) == 0 + assert len(started) == 1 assert (tmp_path / "api-generated-cert.pem").read_bytes() == minted From 45c6a0cc58a2a5e621495a428fd3b976135aff82 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 10:29:24 -0600 Subject: [PATCH 38/41] The renewal contract is stated by the library, true for an engine shard, and the new tests have controls Review round 2 on the repair of Lander hold 6952403017 (BACKLOG #4934). The tls.py docstrings describe what renew=False does and what a caller may pass it for, instead of restating what serve and supervise do. The serve comment no longer says the check's answer is the mint's renew, which is false for an engine shard. The loosening bullet again names the start itself as a writer of the breaking row, and no longer claims that every other command is unrefused. Both renewal-once tests assert the renewal clock was consulted, and their names no longer claim a refusal. --- docs/SECURITY-LOOSENING.md | 6 ++--- messagefoundry/__main__.py | 4 ++-- messagefoundry/api/tls.py | 23 +++++++++---------- ...rotate_key_open_hook_and_lost_range_key.py | 14 ++++++----- 4 files changed, 24 insertions(+), 23 deletions(-) diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index e73cc1768b..87febc5417 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -847,9 +847,9 @@ the next start may do. may still break the chain); and the key the chain's newest range is under is not configured. It refuses whether or not the key in use keyed an earlier range. With it on, this refusal does not fire. An audit row written in that - state goes into that range under the key in use and never verifies there, so the chain breaks at - that row, and `rotate-key` then cannot roll it, even after the key is put back. Other commands - are not refused in that state, whatever the switch. + state, by that start or by a command this refusal does not cover, goes into that range under the + key in use and never verifies there, so the chain breaks at that row, and `rotate-key` then + cannot roll it, even after the key is put back. - **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, and every other start check. - **When acceptable:** only when the key the chain's newest range is under is truly lost. If the diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index 3cace2b8c2..41be7f0eec 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -4911,8 +4911,8 @@ def registry_filter(reg: Registry) -> Registry: # noqa: F811 (local shard-bound # BACKLOG #4934: the lifespan's start refusal runs after this mint, so a start that may replace # the pair takes it here first. A first-run mint replaces nothing, so it needs no early open. # An early open refused on an unmarked value is the exception; _refuse_unrolled_key_before_tls_write - # says what that start does instead. The answer is also the mint's `renew`, so renewal is decided - # once: a pair that falls due after this check is not renewed past the refusal it skipped. + # says what that start does instead. A False answer also makes the mint's `renew` False, so a + # pair that falls due after this check is not renewed past the refusal it skipped. _may_replace = generated_pair_may_be_replaced( settings.api, state_dir=_state_dir, renew=args.shard is None ) diff --git a/messagefoundry/api/tls.py b/messagefoundry/api/tls.py index 2a5fab8801..324c3a2315 100644 --- a/messagefoundry/api/tls.py +++ b/messagefoundry/api/tls.py @@ -636,10 +636,10 @@ def generated_pair_may_be_replaced( pair already on disk: a renewal, or the recovery of an unusable or half-written pair, each of which it reports as a :class:`GeneratedPairReplaced`. A first-run mint replaces nothing and answers ``False``. Read-only: it writes nothing and takes no lock, so a start can run a - refusal first and leave the pair as it found it on a refused start (BACKLOG #4934). ``serve`` - and ``supervise`` pass a ``False`` answer to the mint as ``renew=False``, so a pair that falls - due after this call is not renewed there; whether a pair that stops loading after this call is - recovered is still decided again at the mint.""" + refusal first and leave the pair as it found it on a refused start (BACKLOG #4934). The mint + decides again what it finds on disk; a caller that passes a ``False`` answer on as the mint's + ``renew=False`` keeps a pair that falls due after this call from being renewed there, but not a + pair that stops loading from being recovered.""" if plan_api_tls_material(api, state_dir=state_dir).source != "generated": return False cert_path, key_path = _generated_pair(state_dir) @@ -683,14 +683,13 @@ def ensure_api_tls_material( Nothing renews mid-run: ``CertExpiryRunner`` watches the served certificate, because ``serve`` hands it the path this function RETURNS, and it stays the alarm for an engine never restarted. - **``renew=False`` is for an engine shard (``serve --shard``).** Every process that serves the - pair must start together when it is renewed, or a lone restarted shard would serve a new - certificate while its siblings keep the old one in memory. So ``supervise`` renews before it - spawns any shard, and a shard only reuses a pair that loads, due or not. It still mints on a - first run and still recovers a pair that does not load, because it has nothing else to serve. - A plain ``serve`` and ``supervise`` also pass ``renew=False`` when - :func:`generated_pair_may_be_replaced` found nothing to replace, so they renew only a pair that - check found due (BACKLOG #4934). **A file or lock failure never costs a start the old pair can still serve**: any ``OSError`` on + **``renew=False`` reuses a pair that loads, due or not.** An engine shard (``serve --shard``) + always passes it: every process that serves the pair must start together when it is renewed, + or a lone restarted shard would serve a new certificate while its siblings keep the old one in + memory. So ``supervise`` renews before it spawns any shard. A caller may also pass it to keep a + ``False`` answer from :func:`generated_pair_may_be_replaced` (BACKLOG #4934). It still mints on + a first run and still recovers a pair that does not load, because it has nothing else to serve. + **A file or lock failure never costs a start the old pair can still serve**: any ``OSError`` on the locked path -- including a lock it could not take -- falls back to that pair with a WARNING, and only a pair that does not load or has expired lets the error propagate. Other errors, such as a ``[api].host`` the certificate builder refuses, still stop the start. diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index 42bec0b901..d4655bfb4e 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -298,10 +298,10 @@ def _mint_a_pair_due_for_renewal(state_dir: Path, monkeypatch: pytest.MonkeyPatc def _mint_a_pair_that_falls_due_after_the_first_check( state_dir: Path, monkeypatch: pytest.MonkeyPatch -) -> bytes: +) -> tuple[bytes, list[Path]]: """As :func:`_mint_a_pair_due_for_renewal`, but the pair is reported fresh to the first renewal check after this and due to every later one: a pair crossing the renewal threshold between the - start's read-only check and its mint.""" + start's read-only check and its mint. Also returns the list each renewal check appends to.""" from messagefoundry.api import tls from messagefoundry.config.settings import ApiSettings @@ -313,10 +313,10 @@ def due_after_the_first_check(cert_path: Path) -> Any: return None if len(checks) == 1 else tls._cert_facts(cert_path) monkeypatch.setattr(tls, "_renewal_due", due_after_the_first_check) - return (state_dir / "api-generated-cert.pem").read_bytes() + return (state_dir / "api-generated-cert.pem").read_bytes(), checks -def test_a_pair_falling_due_mid_start_is_not_renewed_by_a_refused_supervise_start( +def test_a_pair_falling_due_mid_start_is_not_renewed_by_supervise( tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: """#4934: renewal is decided once, by the check the refusal runs on. A pair that falls due after @@ -329,11 +329,12 @@ def test_a_pair_falling_due_mid_start_is_not_renewed_by_a_refused_supervise_star db, key_a, key_b = tmp_path / "sup.db", generate_key(), generate_key() asyncio.run(_seed(db, tmp_path / "uploads", key_a)) state_dir = generated_state_dir(str(db)) - minted = _mint_a_pair_that_falls_due_after_the_first_check(state_dir, monkeypatch) + minted, checks = _mint_a_pair_that_falls_due_after_the_first_check(state_dir, monkeypatch) settings = ServiceSettings(store=StoreSettings(path=str(db), encryption_key=key_b)) before = _audit_rows(db) # Nothing is replaced, so the store is not opened here; each engine shard's lifespan refuses. assert cli._renew_api_tls_before_spawning(settings, str(db)) == state_dir + assert checks, "positive control: the read-only check never consulted the renewal clock" assert (state_dir / "api-generated-cert.pem").read_bytes() == minted assert _audit_rows(db) == before @@ -346,7 +347,7 @@ def test_a_pair_falling_due_mid_start_is_not_renewed_by_serve( refuse before auditing a renewal. The lifespan itself is not run here.""" db, key_a, key_b = tmp_path / "messagefoundry.db", generate_key(), generate_key() asyncio.run(_seed(db, tmp_path / "uploads", key_a)) - minted = _mint_a_pair_that_falls_due_after_the_first_check(tmp_path, monkeypatch) + minted, checks = _mint_a_pair_that_falls_due_after_the_first_check(tmp_path, monkeypatch) monkeypatch.chdir(tmp_path) monkeypatch.setenv("MEFOR_STORE_ENCRYPTION_KEY", key_b) # The enforce-only start gates (an off-box log collector, ...) are not under test here. @@ -363,6 +364,7 @@ def test_a_pair_falling_due_mid_start_is_not_renewed_by_serve( # Positive control: the start ran past the mint and reached the server. assert main(["serve", "--config", str(samples), "--env", "dev"]) == 0 assert len(started) == 1 + assert checks, "positive control: the read-only check never consulted the renewal clock" assert (tmp_path / "api-generated-cert.pem").read_bytes() == minted From 910c9d70e8acecf872d57efe806d6ee0ea7e7b45 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 11:19:01 -0600 Subject: [PATCH 39/41] serve and supervise start as on main: the early unrolled-key refusal and its wiring come out Re-cut of PR 2991, Repair Manager decision d20261117T165237004-e56620. Removes the BACKLOG #4934 start refusal from the startup path. serve's early store open before the API pair mint, supervise's pre-renewal open in _renew_api_tls_before_spawning, the lifespan's refuse_unrolled_key_change call in api/app.py, and tls.generated_pair_may_be_replaced with its renew plumbing are reverted to the merge base. The census, keyless-chain and api_tls test entries for them, the crypto inventory entries they added, and the PR tests that drove them go too. The rotate-key before_first_write hunks in __main__.py stay. --- messagefoundry/__main__.py | 171 +-------- messagefoundry/api/app.py | 20 - messagefoundry/api/tls.py | 31 +- scripts/security/crypto_inventory_check.py | 7 - tests/test_api_tls.py | 3 +- tests/test_keyless_chain_every_command.py | 3 - ...rotate_key_open_hook_and_lost_range_key.py | 355 ------------------ tests/test_vault_posture_census.py | 2 - 8 files changed, 20 insertions(+), 572 deletions(-) diff --git a/messagefoundry/__main__.py b/messagefoundry/__main__.py index 41be7f0eec..abdf9798d0 100644 --- a/messagefoundry/__main__.py +++ b/messagefoundry/__main__.py @@ -4899,7 +4899,6 @@ def registry_filter(reg: Registry) -> Registry: # noqa: F811 (local shard-bound from messagefoundry.api.tls import ( GeneratedPairReplaced, ensure_api_tls_material, - generated_pair_may_be_replaced, generated_state_dir, ) from messagefoundry.restricted_file import RestrictedFileError @@ -4908,26 +4907,6 @@ def registry_filter(reg: Registry) -> Registry: # noqa: F811 (local shard-bound # the store is open, which is after this point (ADR 0172 decision 6: never silent). _replaced: list[GeneratedPairReplaced] = [] _state_dir = generated_state_dir(settings.store.path) - # BACKLOG #4934: the lifespan's start refusal runs after this mint, so a start that may replace - # the pair takes it here first. A first-run mint replaces nothing, so it needs no early open. - # An early open refused on an unmarked value is the exception; _refuse_unrolled_key_before_tls_write - # says what that start does instead. A False answer also makes the mint's `renew` False, so a - # pair that falls due after this check is not renewed past the refusal it skipped. - _may_replace = generated_pair_may_be_replaced( - settings.api, state_dir=_state_dir, renew=args.shard is None - ) - if _may_replace: - from messagefoundry.pipeline.secret_rotation import UnrolledStoreKeyRefused - from messagefoundry.store.base import KeylessAuditChainRefused - - try: - _refuse_unrolled_key_before_tls_write(settings) - except (UnrolledStoreKeyRefused, KeylessAuditChainRefused) as exc: - print(f"error: {exc}", file=sys.stderr) - return 2 - except _StoreOpenRefused as exc: # its text already went through codec_safe_str - print(f"error: cannot open the store: {exc}; refusing to start.", file=sys.stderr) - return 2 try: _material = ensure_api_tls_material( settings.api, @@ -4936,7 +4915,7 @@ def registry_filter(reg: Registry) -> Registry: # noqa: F811 (local shard-bound # An engine shard never renews: `supervise` renews before it spawns the whole fleet, so # a lone restarted shard cannot leave its siblings serving a different certificate # (#1276). - renew=args.shard is None and _may_replace, + renew=args.shard is None, ) except RestrictedFileError as exc: # The TLS key could not be created restricted, and no usable pair exists to fall back on @@ -5247,25 +5226,12 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P here, so a base_dir set in the file is applied too, or the two would name different pairs. A re-mint is audited like serve's, in the store the shards are about to open, over the same - derived store-hop posture serve's lifespan opens it with. On a start that may replace the pair, - that store is opened first and takes serve's start refusal before the renewal runs (BACKLOG - #4934), and a failure to open propagates before any renewal, except one: an open refused on an - unmarked value (``UnmarkedCipherError``). That open passes no refusal hook, so the start falls - back to the order main ran: it logs a WARNING, mints, and returns if nothing was replaced, and - an engine shard that then starts sends the store-cipher alert from its hooked open. If the mint - replaced the pair, it opens the store again to audit the replacement, with no hook, as main - did, and that open raises ``UnmarkedCipherError`` again, which ``supervise`` does not catch: it - ends the start at the dispatch floor, the replaced pair is left unaudited, and no alert is - sent. Any other start mints first and opens the store only if that - replaced a pair, which then takes the start refusal before the audit row; a refusal there also - leaves the replaced pair unaudited. + derived store-hop posture serve's lifespan opens it with. A failure to open propagates, as it + would in serve: the WARNING the renewal logged, with both fingerprints, is then its record. """ - import asyncio - from messagefoundry.api.tls import ( GeneratedPairReplaced, ensure_api_tls_material, - generated_pair_may_be_replaced, generated_state_dir, record_generated_pair_replacements, ) @@ -5273,7 +5239,6 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P from messagefoundry.config.settings import hop_posture_from_ai, keyless_opt_out_refusal from messagefoundry.last_resort import run_guarded from messagefoundry.store import open_store - from messagefoundry.store.crypto import UnmarkedCipherError cwd = Path.cwd() root = resolve_project_root(settings.environments.base_dir or None, cwd=cwd) @@ -5281,6 +5246,9 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P state_dir = generated_state_dir(store_path) replaced: list[GeneratedPairReplaced] = [] + ensure_api_tls_material(settings.api, state_dir=state_dir, replacements=replaced) + if not replaced: + return state_dir posture = ( hop_posture_from_ai(settings.ai, enforcement=settings.security.enforcement) if settings.ai is not None @@ -5291,129 +5259,22 @@ def _renew_api_tls_before_spawning(settings: ServiceSettings, db_base: str) -> P # as it does there, and a store with no key and no opt-out is refused here as serve would refuse. shard_store = settings.store.model_copy(update={"path": store_path}) - async def _audit(*, renew_after_refusal: bool) -> bool: - """``False`` only when the open ahead of a renewal is refused on an unmarked value.""" - try: - store = await open_store( - shard_store, - create=True, - posture=posture, - keyless_chain_refusal=keyless_opt_out_refusal(shard_store, settings.security), - ) - except UnmarkedCipherError: - if not renew_after_refusal: - raise - logging.getLogger(__name__).warning( - "the store open before the API pair renewal was refused on an unmarked value; the " - "start refusal was not checked, and the fleet start goes on to the mint" - ) - return False + async def _audit() -> None: + store = await open_store( + shard_store, + create=True, + posture=posture, + keyless_chain_refusal=keyless_opt_out_refusal(shard_store, settings.security), + ) try: - # BACKLOG #4934: before the renewal and before the audit row below, which is this - # store's first since the open. - await _refuse_unrolled_key_on(store, settings) - if renew_after_refusal: - # Off the loop: key generation, file writes and the pair lock's wait block. - await asyncio.to_thread( - ensure_api_tls_material, - settings.api, - state_dir=state_dir, - replacements=replaced, - ) await record_generated_pair_replacements(store, replaced) finally: await store.close() - return True - # BACKLOG #4934: a start that may replace the pair opens the store and takes the engine shards' - # start refusal before it renews the pair. Any other start, and one whose early open is refused - # on an unmarked value, mints or reuses first and opens the store only if that replaced a pair. - # The check's answer is also the mint's `renew`, so renewal is decided once: a pair that falls - # due after the check is not renewed past the refusal it skipped. - may_replace = generated_pair_may_be_replaced(settings.api, state_dir=state_dir) - if may_replace and run_guarded(_audit(renew_after_refusal=True)): - return state_dir - ensure_api_tls_material( - settings.api, state_dir=state_dir, replacements=replaced, renew=may_replace - ) - if replaced: - run_guarded(_audit(renew_after_refusal=False)) + run_guarded(_audit()) return state_dir -async def _refuse_unrolled_key_on(store: Store, settings: ServiceSettings) -> None: - """``serve``'s start refusal of a changed store key the keyed audit chain cannot follow - (BACKLOG #4934), on an open store; see - :func:`~messagefoundry.pipeline.secret_rotation.refuse_unrolled_key_change`.""" - from messagefoundry.pipeline.secret_rotation import refuse_unrolled_key_change - from messagefoundry.store.store import ( - SecretRotationMetaStore, - audit_chain_rolled, - audit_range_key_missing, - ) - - if isinstance(store, SecretRotationMetaStore): - await refuse_unrolled_key_change( - store, - dek_key_id=store.cipher_info().active_key_id, - audit_rolled=audit_chain_rolled(store), - range_key_missing=audit_range_key_missing(store), - allow_lost=settings.secret_rotation.allow_lost_audit_range_key, - ) - - -def _refuse_unrolled_key_before_tls_write(settings: ServiceSettings) -> None: - """Open ``serve``'s store and run :func:`_refuse_unrolled_key_on` before the start may replace - the generated API pair (BACKLOG #4934). The lifespan - refuses again after its own open; this one exists only because the mint comes first. Raises - :class:`~messagefoundry.pipeline.secret_rotation.UnrolledStoreKeyRefused`, the keyless-chain - refusal, :class:`_StoreOpenRefused` for an open refusal :func:`_open_store_or_refuse` maps, and - anything else the open raises, except one. This open passes no ``refusal_hook``, so an open - refused on an unmarked value (``UnmarkedCipherError``) logs a WARNING and returns without the - check instead. The start goes on to the mint, which may replace the pair, and, unless a gate - between the two refuses first, to the lifespan, whose open passes the hook, is refused on the - same value and sends the store-cipher alert. A pair the mint replaced is then left unaudited, - as on main. A planted row the open's at-open sweep finds and leaves in place, without raising, - is reported to no hook, so it reaches only the log; if the check then refuses, the start ends - here and no hooked open runs to alert on it.""" - from messagefoundry.config.settings import hop_posture_from_ai, keyless_opt_out_refusal - from messagefoundry.last_resort import run_guarded - from messagefoundry.store import open_store - from messagefoundry.store.crypto import UnmarkedCipherError - - posture = ( - hop_posture_from_ai(settings.ai, enforcement=settings.security.enforcement) - if settings.ai is not None - else None - ) - - async def _check() -> None: - try: - store = await _open_store_or_refuse( - open_store( - settings.store, - create=True, - posture=posture, - keyless_chain_refusal=keyless_opt_out_refusal( - settings.store, settings.security - ), - ) - ) - except UnmarkedCipherError: - # The lifespan's hooked open refuses it again and alerts, as before #4934. - logging.getLogger(__name__).warning( - "the store open before the API pair mint was refused on an unmarked value; the " - "start refusal was not checked, and the start goes on to the mint and the store open" - ) - return - try: - await _refuse_unrolled_key_on(store, settings) - finally: - await store.close() - - run_guarded(_check()) - - def _supervise(args: argparse.Namespace) -> int: """L3 multi-process sharding (messagefoundry/pipeline/supervisor.py): discover the shard ids in the config and run one `serve --shard ` subprocess per shard, each with its own SQLite db file and @@ -5628,7 +5489,6 @@ def _supervise(args: argparse.Namespace) -> int: # Vault BACKLOG #2601: the two key-file checks each shard's `serve` makes, for the same reason # as the gates above: every shard would refuse, and the supervisor would only restart them. The # store key file is checked before the renewal below can open the store and read it. - from messagefoundry.pipeline.secret_rotation import UnrolledStoreKeyRefused from messagefoundry.restricted_file import RestrictedFileError from messagefoundry.store.base import KeylessAuditChainRefused @@ -5697,9 +5557,6 @@ def configure_with_forwarder(forward: SyslogForward | None) -> bool: except KeylessAuditChainRefused as exc: # #1916: a named key the provider did not resolve print(f"error: {exc}", file=sys.stderr) return 2 - except UnrolledStoreKeyRefused as exc: # BACKLOG #4934: in full, since it carries the remedy - print(f"error: {exc}", file=sys.stderr) - return 2 except RestrictedFileError as exc: # the TLS key could not be created restricted print(f"error: {exc}; refusing to start the fleet.", file=sys.stderr) return 2 diff --git a/messagefoundry/api/app.py b/messagefoundry/api/app.py index 5e8d53b874..f7aea6292f 100644 --- a/messagefoundry/api/app.py +++ b/messagefoundry/api/app.py @@ -12652,26 +12652,6 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]: # store; left open, aiosqlite's non-daemon worker holds the process at exit (#1257). The # engine has not started anywhere in this span, so the unwind never calls engine.stop(). try: - # BACKLOG #4934 (owner ruling 2026-11-04): first, before anything below writes an audit - # row, refuse a changed store key the keyed audit chain cannot follow. The open settled - # the chain; the only I/O is the stamp read. - from messagefoundry.pipeline.secret_rotation import refuse_unrolled_key_change - from messagefoundry.store.store import ( - SecretRotationMetaStore, - audit_chain_rolled, - audit_range_key_missing, - ) - - if isinstance(store, SecretRotationMetaStore): - await refuse_unrolled_key_change( - store, - dek_key_id=store.cipher_info().active_key_id, - audit_rolled=audit_chain_rolled(store), - range_key_missing=audit_range_key_missing(store), - allow_lost=( - secret_rotation_settings or SecretRotationSettings() - ).allow_lost_audit_range_key, - ) # Offline uploaded-logs store (BACKLOG #125/#126, ADR 0134), on the LIVE store's cipher instance. # DISABLED (None) unless [store].uploads_dir is set, so no PHI-at-rest surface exists unless an # operator opts in; every uploaded-logs route 503s when None. diff --git a/messagefoundry/api/tls.py b/messagefoundry/api/tls.py index 324c3a2315..2608bbbe83 100644 --- a/messagefoundry/api/tls.py +++ b/messagefoundry/api/tls.py @@ -60,7 +60,6 @@ "api_tls_source", "build_api_ssl_context", "ensure_api_tls_material", - "generated_pair_may_be_replaced", "generated_state_dir", "off_loopback_certificate_remedy", "plan_api_tls_material", @@ -629,25 +628,6 @@ def _let_local_users_read_cert(cert_path: Path) -> None: _grant_read(cert_path, _LOCAL_USERS_SID) -def generated_pair_may_be_replaced( - api: ApiSettings, *, state_dir: Path, renew: bool = True -) -> bool: - """True when :func:`ensure_api_tls_material`, given the same arguments, may replace a generated - pair already on disk: a renewal, or the recovery of an unusable or half-written pair, each of - which it reports as a :class:`GeneratedPairReplaced`. A first-run mint replaces nothing and - answers ``False``. Read-only: it writes nothing and takes no lock, so a start can run a - refusal first and leave the pair as it found it on a refused start (BACKLOG #4934). The mint - decides again what it finds on disk; a caller that passes a ``False`` answer on as the mint's - ``renew=False`` keeps a pair that falls due after this call from being renewed there, but not a - pair that stops loading from being recovered.""" - if plan_api_tls_material(api, state_dir=state_dir).source != "generated": - return False - cert_path, key_path = _generated_pair(state_dir) - if not (cert_path.exists() or key_path.exists()): - return False - return not _reusable_without_the_lock(cert_path, key_path, renew=renew) - - def ensure_api_tls_material( api: ApiSettings, *, @@ -683,12 +663,11 @@ def ensure_api_tls_material( Nothing renews mid-run: ``CertExpiryRunner`` watches the served certificate, because ``serve`` hands it the path this function RETURNS, and it stays the alarm for an engine never restarted. - **``renew=False`` reuses a pair that loads, due or not.** An engine shard (``serve --shard``) - always passes it: every process that serves the pair must start together when it is renewed, - or a lone restarted shard would serve a new certificate while its siblings keep the old one in - memory. So ``supervise`` renews before it spawns any shard. A caller may also pass it to keep a - ``False`` answer from :func:`generated_pair_may_be_replaced` (BACKLOG #4934). It still mints on - a first run and still recovers a pair that does not load, because it has nothing else to serve. + **``renew=False`` is for an engine shard (``serve --shard``).** Every process that serves the + pair must start together when it is renewed, or a lone restarted shard would serve a new + certificate while its siblings keep the old one in memory. So ``supervise`` renews before it + spawns any shard, and a shard only reuses a pair that loads, due or not. It still mints on a + first run and still recovers a pair that does not load, because it has nothing else to serve. **A file or lock failure never costs a start the old pair can still serve**: any ``OSError`` on the locked path -- including a lock it could not take -- falls back to that pair with a WARNING, and only a pair that does not load or has expired lets the error propagate. Other errors, such diff --git a/scripts/security/crypto_inventory_check.py b/scripts/security/crypto_inventory_check.py index 8d1e96bcb1..ab0e727636 100644 --- a/scripts/security/crypto_inventory_check.py +++ b/scripts/security/crypto_inventory_check.py @@ -1073,9 +1073,6 @@ "compare:via messagefoundry.api.tls", # BACKLOG #1167: evaluate_anchor's pin compare "compare:via messagefoundry.auth.totp", "compare:via messagefoundry.pipeline.secret_rotation", - # BACKLOG #4934: the start refusal serve and supervise run here reads the audit - # chain's range keys the open settled. - "hash:via messagefoundry.store.store", # BACKLOG #3470: anchor_refusal checks the forwarder's and the SMTP relay's CA before # those hops are built, through evaluate_anchor: its SHA-256 fingerprint, and its pin # compare, which these specs never reach since they carry no pin. @@ -1141,10 +1138,6 @@ "compare:via messagefoundry.config.fingerprint", "hash:via messagefoundry.auth.trust_anchors", "tls_context:via messagefoundry.config.tls_policy", - # BACKLOG #4934: the start refusal compares the DEK key id with its stamp - # (dek_changed), and reads the audit chain's range keys the open settled. - "compare:via messagefoundry.pipeline.secret_rotation", - "hash:via messagefoundry.store.store", } ), "messagefoundry/api/security.py": frozenset( diff --git a/tests/test_api_tls.py b/tests/test_api_tls.py index f7fc1042c6..1fa4fa7651 100644 --- a/tests/test_api_tls.py +++ b/tests/test_api_tls.py @@ -5452,8 +5452,7 @@ def test_plain_serve_renews_and_an_engine_shard_never_does( reuses it untouched: `supervise` renews for the whole fleet before spawning, so one restarted shard can never serve a certificate its siblings do not. - Mutation: drop ``args.shard is None`` from both the read-only check's ``renew`` and the mint's. - Red: the shard row renews. Dropping it from either one alone stays green.""" + Mutation: drop ``renew=args.shard is None``. Red: the shard row renews.""" cert = tmp_path / _GENERATED_CERT_NAME _plant_generated_pair(tmp_path, lived_days=300, left_days=65) old = cert.read_bytes() diff --git a/tests/test_keyless_chain_every_command.py b/tests/test_keyless_chain_every_command.py index 9baa696155..6c29e8144d 100644 --- a/tests/test_keyless_chain_every_command.py +++ b/tests/test_keyless_chain_every_command.py @@ -1358,9 +1358,6 @@ def test_a_service_config_that_cannot_be_read_exits_2( "_provision_admin.page_granted.reopen", # the supervisor's audit of a re-minted API pair, on the verdict serve's lifespan passes "_renew_api_tls_before_spawning._audit", - # BACKLOG #4934: serve's open for the unrolled-key refusal before it writes the API pair, on - # the verdict its lifespan passes - "_refuse_unrolled_key_before_tls_write._check", "_audit_verify.run", "_audit_anchor.run", # BACKLOG #1719: cipher-audit's read-only open of a PostgreSQL or SQL Server store diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index d4655bfb4e..d414c490a8 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -14,8 +14,6 @@ import asyncio import logging -import sqlite3 -from contextlib import closing from pathlib import Path from typing import Any @@ -283,356 +281,3 @@ def test_the_rows_after_a_refused_start_are_unchanged(tmp_path: Path) -> None: rows = asyncio.run(_rows(db, key_b, key_a)) assert rows[_DEK].fingerprint == dek_fingerprint(_key_id(key_a)) assert ROTATION_RECORD_ID not in rows - - -def _mint_a_pair_due_for_renewal(state_dir: Path, monkeypatch: pytest.MonkeyPatch) -> bytes: - """A real generated API pair in ``state_dir``, reported due for renewal from now on, so the - next renewing start writes a new one. Returns the certificate's bytes as minted.""" - from messagefoundry.api import tls - from messagefoundry.config.settings import ApiSettings - - tls.ensure_api_tls_material(ApiSettings(), state_dir=state_dir) - monkeypatch.setattr(tls, "_renewal_due", tls._cert_facts) - return (state_dir / "api-generated-cert.pem").read_bytes() - - -def _mint_a_pair_that_falls_due_after_the_first_check( - state_dir: Path, monkeypatch: pytest.MonkeyPatch -) -> tuple[bytes, list[Path]]: - """As :func:`_mint_a_pair_due_for_renewal`, but the pair is reported fresh to the first renewal - check after this and due to every later one: a pair crossing the renewal threshold between the - start's read-only check and its mint. Also returns the list each renewal check appends to.""" - from messagefoundry.api import tls - from messagefoundry.config.settings import ApiSettings - - tls.ensure_api_tls_material(ApiSettings(), state_dir=state_dir) - checks: list[Path] = [] - - def due_after_the_first_check(cert_path: Path) -> Any: - checks.append(cert_path) - return None if len(checks) == 1 else tls._cert_facts(cert_path) - - monkeypatch.setattr(tls, "_renewal_due", due_after_the_first_check) - return (state_dir / "api-generated-cert.pem").read_bytes(), checks - - -def test_a_pair_falling_due_mid_start_is_not_renewed_by_supervise( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """#4934: renewal is decided once, by the check the refusal runs on. A pair that falls due after - that check is not renewed by the mint, so a start its engine shards will refuse leaves it - unchanged. Before, the mint renewed it and the refusal then left the renewal unaudited.""" - from messagefoundry import __main__ as cli - from messagefoundry.api.tls import generated_state_dir - from messagefoundry.config.settings import ServiceSettings - - db, key_a, key_b = tmp_path / "sup.db", generate_key(), generate_key() - asyncio.run(_seed(db, tmp_path / "uploads", key_a)) - state_dir = generated_state_dir(str(db)) - minted, checks = _mint_a_pair_that_falls_due_after_the_first_check(state_dir, monkeypatch) - settings = ServiceSettings(store=StoreSettings(path=str(db), encryption_key=key_b)) - before = _audit_rows(db) - # Nothing is replaced, so the store is not opened here; each engine shard's lifespan refuses. - assert cli._renew_api_tls_before_spawning(settings, str(db)) == state_dir - assert checks, "positive control: the read-only check never consulted the renewal clock" - assert (state_dir / "api-generated-cert.pem").read_bytes() == minted - assert _audit_rows(db) == before - - -def test_a_pair_falling_due_mid_start_is_not_renewed_by_serve( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """#4934: as the supervise test above, for serve. The read-only check finds the pair fresh, so - no early refusal runs, and the mint must not renew it either: this store's lifespan would - refuse before auditing a renewal. The lifespan itself is not run here.""" - db, key_a, key_b = tmp_path / "messagefoundry.db", generate_key(), generate_key() - asyncio.run(_seed(db, tmp_path / "uploads", key_a)) - minted, checks = _mint_a_pair_that_falls_due_after_the_first_check(tmp_path, monkeypatch) - monkeypatch.chdir(tmp_path) - monkeypatch.setenv("MEFOR_STORE_ENCRYPTION_KEY", key_b) - # The enforce-only start gates (an off-box log collector, ...) are not under test here. - monkeypatch.setenv("MEFOR_SECURITY_ENFORCEMENT", "warn") - started: list[object] = [] - monkeypatch.setattr("uvicorn.run", lambda *a, **k: started.append(a)) - (tmp_path / "messagefoundry.toml").write_text( - "security.block_unlisted_outbound = true\n" - "alerts.security_notifications_required = false\n" - "security.local_access_only = true\n", - encoding="utf-8", - ) - samples = Path(__file__).resolve().parent.parent / "samples" / "config" - # Positive control: the start ran past the mint and reached the server. - assert main(["serve", "--config", str(samples), "--env", "dev"]) == 0 - assert len(started) == 1 - assert checks, "positive control: the read-only check never consulted the renewal clock" - assert (tmp_path / "api-generated-cert.pem").read_bytes() == minted - - -@pytest.mark.parametrize("allow_lost", [False, True]) -def test_a_refused_supervise_start_leaves_the_api_pair_unchanged( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch, allow_lost: bool -) -> None: - """#4934: the refusal runs before the renewal, so a refused fleet start renews nothing on disk - and leaves no renewal without its audit row. Under the override the same start renews the pair - and records it, the positive control that the pair was due.""" - from messagefoundry import __main__ as cli - from messagefoundry.api.tls import generated_state_dir - from messagefoundry.config.settings import ServiceSettings - - db, key_a, key_b = tmp_path / "sup.db", generate_key(), generate_key() - asyncio.run(_seed(db, tmp_path / "uploads", key_a)) - state_dir = generated_state_dir(str(db)) - minted = _mint_a_pair_due_for_renewal(state_dir, monkeypatch) - settings = ServiceSettings( - store=StoreSettings(path=str(db), encryption_key=key_b), - secret_rotation=SecretRotationSettings(allow_lost_audit_range_key=allow_lost), - ) - before = _audit_rows(db) - if allow_lost: - cli._renew_api_tls_before_spawning(settings, str(db)) - assert (state_dir / "api-generated-cert.pem").read_bytes() != minted - assert _audit_rows(db) == before + 1 - return - with pytest.raises(UnrolledStoreKeyRefused): - cli._renew_api_tls_before_spawning(settings, str(db)) - assert (state_dir / "api-generated-cert.pem").read_bytes() == minted - assert _audit_rows(db) == before - - -def test_a_refused_serve_start_leaves_the_api_pair_unchanged( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] -) -> None: - """#4934: serve mints before its lifespan opens the store, so a start that would write the - pair takes the refusal first, exits 2 with the remedy, and renews nothing on disk.""" - db, key_a, key_b = tmp_path / "messagefoundry.db", generate_key(), generate_key() - asyncio.run(_seed(db, tmp_path / "uploads", key_a)) - minted = _mint_a_pair_due_for_renewal(tmp_path, monkeypatch) - started: list[object] = [] - monkeypatch.chdir(tmp_path) - monkeypatch.setenv("MEFOR_STORE_ENCRYPTION_KEY", key_b) - # The enforce-only start gates (an off-box log collector, ...) are not under test here. - monkeypatch.setenv("MEFOR_SECURITY_ENFORCEMENT", "warn") - monkeypatch.setattr("uvicorn.run", lambda *a, **k: started.append(k)) - (tmp_path / "messagefoundry.toml").write_text( - "security.block_unlisted_outbound = true\n" - "alerts.security_notifications_required = false\n" - "security.local_access_only = true\n", - encoding="utf-8", - ) - samples = Path(__file__).resolve().parent.parent / "samples" / "config" - before = _audit_rows(db) - assert main(["serve", "--config", str(samples), "--env", "dev"]) == 2 - err = capsys.readouterr().err - assert "MEFOR_STORE_ENCRYPTION_KEYS_RETIRED" in err, err[-2000:] - assert started == [] - assert (tmp_path / "api-generated-cert.pem").read_bytes() == minted - assert _audit_rows(db) == before - - -def test_supervise_exits_2_with_the_remedy_on_the_refusal( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] -) -> None: - """#4934: ``supervise`` catches the refusal, prints it in full and spawns no shard.""" - from messagefoundry import __main__ as cli - from tests.test_remote_debug_start_refusal import ( - _OFF, - _stand_down_the_gates_before_logging, - _supervise, - ) - - def refused(*_args: object, **_kwargs: object) -> Path: - raise UnrolledStoreKeyRefused("refusing to start: TEST REMEDY") - - _stand_down_the_gates_before_logging(monkeypatch) - monkeypatch.setattr(cli, "_keyless_store_gate", lambda settings: None) - monkeypatch.setattr(cli, "_start_logging", lambda *a, **k: None) - monkeypatch.setattr(cli, "_renew_api_tls_before_spawning", refused) - rc, spawned = _supervise(tmp_path, monkeypatch, _OFF) - err = capsys.readouterr().err - assert rc == 2 and not spawned, err[-2000:] - assert "error: refusing to start: TEST REMEDY" in err - - -def _audit_rows(db: Path) -> int: - """The audit_log row count of a synthetic test store, read without opening it as a store.""" - with closing(sqlite3.connect(db)) as conn: - (count,) = conn.execute("SELECT COUNT(*) FROM audit_log").fetchone() - assert isinstance(count, int) - return count - - -def _app_on_the_new_key_alone(tmp_path: Path, allow_lost: bool) -> Any: - from messagefoundry.api import create_managed_app - from messagefoundry.config.settings import EgressSettings - - db, key_a, key_b = tmp_path / "serve.db", generate_key(), generate_key() - asyncio.run(_seed(db, tmp_path / "uploads", key_a)) - return create_managed_app( - store_settings=StoreSettings(path=str(db), encryption_key=key_b), - poll_interval=0.05, - # The seed's DEK stamp is old; the calendar refusal is not under test here. - secret_rotation_settings=SecretRotationSettings( - allow_lost_audit_range_key=allow_lost, enforce_store_key_expiry=False - ), - egress_settings=EgressSettings(deny_by_default=False), - ) - - -async def test_serve_refuses_a_start_on_the_new_key_alone(tmp_path: Path) -> None: - """#4934 step 3, the wiring: the refusal aborts the serve lifespan.""" - app = await asyncio.to_thread(_app_on_the_new_key_alone, tmp_path, False) - before = _audit_rows(tmp_path / "serve.db") - with pytest.raises(UnrolledStoreKeyRefused): - async with app.router.lifespan_context(app): - pass # pragma: no cover -- startup must not reach here - # The point of the refusal: the refused start wrote no audit row into the unrolled range. - assert _audit_rows(tmp_path / "serve.db") == before - - -async def test_serve_starts_on_the_new_key_alone_under_the_override(tmp_path: Path) -> None: - """Positive control on the arm above: the same store starts with the override.""" - app = await asyncio.to_thread(_app_on_the_new_key_alone, tmp_path, True) - async with app.router.lifespan_context(app): - assert app.state.engine is not None - - -def _seed_a_planted_state_value(db: Path, key: str) -> None: - """A keyed synthetic store whose eagerly loaded ``state`` table holds an unmarked value beside a - sealed one, so every keyed open of it raises ``UnmarkedCipherError``.""" - from tests.test_store_strict_ciphertext import _keyed, _plant_state - from tests.test_store_strict_ciphertext import _seed as _seed_messages - - asyncio.run(_seed_messages(db, 1, _keyed(key))) - _plant_state(db, key) - - -def _sink_for_the_lifespan(monkeypatch: pytest.MonkeyPatch) -> Any: - """The notifier every lifespan built from here on uses, recording what it is sent.""" - import importlib - - from tests.test_store_strict_ciphertext import _LifecycleSink - - sink = _LifecycleSink() - app_mod = importlib.import_module("messagefoundry.api.app") - monkeypatch.setattr(app_mod, "notifier_from_settings", lambda *a, **k: sink) - return sink - - -class _Recorded(logging.Handler): - """The WARNINGs ``messagefoundry.__main__`` logs. A handler on that logger, since ``serve`` - reconfigures the root handlers that ``caplog`` relies on.""" - - def __init__(self) -> None: - super().__init__(logging.WARNING) - self.lines: list[str] = [] - - def emit(self, record: logging.LogRecord) -> None: - self.lines.append(record.getMessage()) - - -def _record_the_cli_warnings(monkeypatch: pytest.MonkeyPatch) -> _Recorded: - recorded = _Recorded() - cli_log = logging.getLogger("messagefoundry.__main__") - # monkeypatch restores the logger's own handler list and state after the test. - monkeypatch.setattr(cli_log, "handlers", [*cli_log.handlers, recorded]) - monkeypatch.setattr(cli_log, "disabled", False) - return recorded - - -def _assert_the_store_cipher_alert(sink: Any) -> None: - """The alert names the table and column and no planted content, and was sent, not queued.""" - assert sink.events and {e[0] for e in sink.events} == {"store-cipher"}, sink.events - for _subject, reason, _count in sink.events: - assert "state.value" in reason and "666" not in reason and "row-key-9" not in reason - assert sink.started and sink.drained - - -def test_serve_alerts_on_an_unmarked_state_value_on_a_renewal_due_start( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """#4934 repair: the early open ahead of the mint passes no refusal hook, so an unmarked value - it is refused on must not end the start there. The start goes on to the mint and the lifespan, - whose hooked open is refused again and sends the store-cipher alert, as on main.""" - pytest.importorskip("psutil") - from fastapi.testclient import TestClient - - from messagefoundry.store.crypto import CipherError - - db, key = tmp_path / "messagefoundry.db", generate_key() - _seed_a_planted_state_value(db, key) - _mint_a_pair_due_for_renewal(tmp_path, monkeypatch) - sink = _sink_for_the_lifespan(monkeypatch) - recorded = _record_the_cli_warnings(monkeypatch) - refused: list[BaseException] = [] - - def run_the_lifespan(app: Any, *_a: object, **_k: object) -> None: - try: - with TestClient(app): - pass # pragma: no cover -- the open is refused - except CipherError as exc: - refused.append(exc) - - monkeypatch.chdir(tmp_path) - monkeypatch.setenv("MEFOR_STORE_ENCRYPTION_KEY", key) - # The enforce-only start gates (an off-box log collector, ...) are not under test here. - monkeypatch.setenv("MEFOR_SECURITY_ENFORCEMENT", "warn") - monkeypatch.setattr("uvicorn.run", run_the_lifespan) - (tmp_path / "messagefoundry.toml").write_text( - "security.block_unlisted_outbound = true\n" - "alerts.security_notifications_required = false\n" - "security.local_access_only = true\n", - encoding="utf-8", - ) - samples = Path(__file__).resolve().parent.parent / "samples" / "config" - main(["serve", "--config", str(samples), "--env", "dev"]) - assert refused, "the lifespan never ran its hooked open" - _assert_the_store_cipher_alert(sink) - assert any("before the API pair mint was refused" in line for line in recorded.lines) - - -def test_supervise_alerts_on_an_unmarked_state_value_when_the_pair_is_not_replaced( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """#4934 repair: a fleet start whose pair may be replaced opens the store before the renewal, - with no refusal hook. An unmarked value refused there falls back to main's path, so a start - that then replaces nothing returns and its engine shards start, and an engine shard's lifespan - open is hooked and sends the store-cipher alert. Before the repair the early open ended the - start, and no engine shard ran to send it. On a start that does replace the pair, main's own - open after the renewal passed no hook either, so that arm has no alert to restore.""" - pytest.importorskip("psutil") - from fastapi.testclient import TestClient - - from messagefoundry import __main__ as cli - from messagefoundry.api import create_managed_app, tls - from messagefoundry.api.tls import generated_state_dir - from messagefoundry.config.settings import ( - AlertsSettings, - ApiSettings, - EgressSettings, - ServiceSettings, - ) - from messagefoundry.store.crypto import CipherError - - db, key = tmp_path / "sup.db", generate_key() - _seed_a_planted_state_value(db, key) - state_dir = generated_state_dir(str(db)) - tls.ensure_api_tls_material(ApiSettings(), state_dir=state_dir) - minted = (state_dir / "api-generated-cert.pem").read_bytes() - # A pair the read-only check cannot clear, which the mint under the lock then reuses. - monkeypatch.setattr(tls, "generated_pair_may_be_replaced", lambda *a, **k: True) - settings = ServiceSettings(store=StoreSettings(path=str(db), encryption_key=key)) - recorded = _record_the_cli_warnings(monkeypatch) - assert cli._renew_api_tls_before_spawning(settings, str(db)) == state_dir - assert (state_dir / "api-generated-cert.pem").read_bytes() == minted - assert any("before the API pair renewal was refused" in line for line in recorded.lines) - # The engine shard the fleet now starts on that store. - sink = _sink_for_the_lifespan(monkeypatch) - app = create_managed_app( - store_settings=StoreSettings(path=str(db), encryption_key=key), - alerts_settings=AlertsSettings(), - poll_interval=0.05, - egress_settings=EgressSettings(deny_by_default=False), - ) - with pytest.raises(CipherError), TestClient(app): - pass # pragma: no cover -- the open is refused - _assert_the_store_cipher_alert(sink) diff --git a/tests/test_vault_posture_census.py b/tests/test_vault_posture_census.py index 020e8f6a30..8bf625f486 100644 --- a/tests/test_vault_posture_census.py +++ b/tests/test_vault_posture_census.py @@ -309,7 +309,6 @@ def _found() -> tuple[Row, ...]: ("__main__.py", "_provision_admin.run", "open_store", "posture"), ("__main__.py", "_provision_admin.server_store_found", "open_store", "posture"), ("__main__.py", "_renew_api_tls_before_spawning._audit", "open_store", "posture"), - ("__main__.py", "_refuse_unrolled_key_before_tls_write._check", "open_store", "posture"), ("__main__.py", "_rotate_key", "resolve_active_key", "posture"), ("__main__.py", "_rotate_key.run", "open_store", "posture"), ("__main__.py", "_store_transit_bound.run", "open_store", "posture"), @@ -524,7 +523,6 @@ def test_every_place_a_posture_is_built_is_listed() -> None: ("__main__.py", "_offline_security_notifier", "settings.security.enforcement"), ("__main__.py", "_provision_admin", "settings.security.enforcement"), ("__main__.py", "_renew_api_tls_before_spawning", "settings.security.enforcement"), - ("__main__.py", "_refuse_unrolled_key_before_tls_write", "settings.security.enforcement"), ("__main__.py", "_start_logging", "settings.security.enforcement"), ("__main__.py", "_store_provision_schema", "settings.security.enforcement"), # At warn, the store-opening commands' PostgreSQL revocation WARNING (BACKLOG #2532). From 7a81779f3cee32d8371dcd76e9529080f37f5e42 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 11:32:31 -0600 Subject: [PATCH 40/41] The unrolled-key refusal's library code, its switch and their docs come out; the hook stays Re-cut of PR 2991, Repair Manager decision d20261117T165237004-e56620. With every caller gone, refuse_unrolled_key_change, UnrolledStoreKeyRefused and its refusal text (secret_rotation.py) and audit_range_key_missing (store.py) are reverted, with [secret_rotation].allow_lost_audit_range_key, its security_loosenings entry, its CONFIGURATION.md row and its SECURITY-LOOSENING.md entry and table rows. PHI.md drops its pointer to the refusal and keeps the widened rule that no audit-writing command runs on the new key alone before a roll succeeds. The 4934 changelog fragment becomes 4936's and states the hook alone. The refusal's tests go. Kept: the before_first_write hook (store base, SQLite, PostgreSQL, SQL Server), rotate-key's record opened through it, and the rotation record docstrings and WARNING texts. --- changelog.d/4934.security.md | 8 - changelog.d/4936.security.md | 3 + docs/CONFIGURATION.md | 1 - docs/PHI.md | 5 +- docs/SECURITY-LOOSENING.md | 34 +--- messagefoundry/config/settings.py | 19 +- messagefoundry/pipeline/secret_rotation.py | 55 ------ messagefoundry/store/store.py | 26 --- ...rotate_key_open_hook_and_lost_range_key.py | 173 +----------------- 9 files changed, 9 insertions(+), 315 deletions(-) delete mode 100644 changelog.d/4934.security.md create mode 100644 changelog.d/4936.security.md diff --git a/changelog.d/4934.security.md b/changelog.d/4934.security.md deleted file mode 100644 index f02af63d47..0000000000 --- a/changelog.d/4934.security.md +++ /dev/null @@ -1,8 +0,0 @@ -- **`rotate-key` now opens its rotation record, when it opens one, from a new `open_store` hook, - `before_first_write`, which a writable open runs before its first write under the store key.** A - read-only open does not run it. - **`serve` can now refuse to start on a changed store key the keyed audit chain cannot follow.** - `[secret_rotation].allow_lost_audit_range_key = true` starts anyway and is reported as a security - loosening; its entry in `docs/SECURITY-LOOSENING.md` states when the refusal fires and the way - out. At least the `admin-*` commands, `issue-enrolment-code` and `provision-admin` do not refuse. - (`BACKLOG #4934`, `BACKLOG #4936`) diff --git a/changelog.d/4936.security.md b/changelog.d/4936.security.md new file mode 100644 index 0000000000..abd275b281 --- /dev/null +++ b/changelog.d/4936.security.md @@ -0,0 +1,3 @@ +- **`rotate-key` now opens its rotation record, when it opens one, from a new `open_store` hook, + `before_first_write`, which a writable open runs before its first write under the store key.** A + read-only open does not run it. (`BACKLOG #4936`) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 8b6284cd92..b6fc98cde1 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -3005,7 +3005,6 @@ tracked; set `warn_days = 0` to disable the reminder. | `secret_max_age_days` | int | 365 | max age for the **non-DEK** tracked secret classes (connector/AD/SMTP/Vault/OIDC), alerted this many days after their last observed fingerprint change | | `enforce_grace_days` | int | 30 | under `[security].enforcement=enforce`, a DEK older than `store_key_max_age_days + this` escalates its rotation alert **and refuses to start** (see `enforce_store_key_expiry`) | | `enforce_store_key_expiry` | bool | `true` | under `[security].enforcement=enforce`, a store DEK past `store_key_max_age_days + enforce_grace_days` — or one whose age cannot be determined — **aborts engine start**. `false` keeps the alert, drops the refusal, and is reported as a **security loosening** | -| `allow_lost_audit_range_key` | bool | `false` | `true` starts `serve` over its refusal of a store key change the keyed audit chain cannot follow, and is reported as a **security loosening**. [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) states when that refusal fires and the way out | | `enforce_secret_expiry_classes` | list | `[]` | non-DEK classes whose calendar expiry **aborts engine start** under `[security].enforcement=enforce`, past `secret_max_age_days + enforce_grace_days` or with no recorded age. Entries are the `MEFOR_*` class names above or `connector`; an unknown name is refused at load. Empty keeps every non-DEK class alert-only | ```toml diff --git a/docs/PHI.md b/docs/PHI.md index 388e57b4de..bc74619a1d 100644 --- a/docs/PHI.md +++ b/docs/PHI.md @@ -775,10 +775,7 @@ for defense-in-depth without swapping the `aiosqlite` connector. store with a keyed chain, do not start `serve`, or run any other command that writes an audit row, on the new key alone before a run's roll succeeds: the chain's current range is then under a key it does not hold, so it keys new audit rows under the new key, those rows never verify - inside that range, and no later `rotate-key` can roll the chain. `serve` can refuse to start on - a changed key (BACKLOG #4934); the - `[secret_rotation].allow_lost_audit_range_key` entry in - [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) states when, and the way out. **A roll that refuses for good holds the key for good (BACKLOG #4934, #4935).** The roll + inside that range, and no later `rotate-key` can roll the chain. **A roll that refuses for good holds the key for good (BACKLOG #4934, #4935).** The roll verifies the whole chain first, so a chain whose rows do not verify, or whose rows open with no genesis row, refuses under every key, a new one included, and the record stays open. A newest range under a key that is not configured is not one of these, though `audit-verify` reports it diff --git a/docs/SECURITY-LOOSENING.md b/docs/SECURITY-LOOSENING.md index 87febc5417..a62abe6237 100644 --- a/docs/SECURITY-LOOSENING.md +++ b/docs/SECURITY-LOOSENING.md @@ -242,7 +242,6 @@ Earlier versions of this guide called this an open owner question. The 2026-10-2 | | `[approvals].operations` | `["connection_purge", "dead_letter_replay"]` (*conditional* — a loosening only while `[approvals].enabled` is on; a list that leaves out either default operation is named, and so is an empty list, which holds nothing; named as `approvals.operations`) | | | `[api].trusted_proxies` | `[]` (any network covering more than one address is named, once, a `/31` or `/127` included, because every host inside it may set `X-Forwarded-For`. Entries covering every address, such as `0.0.0.0/0` or `::/0`, trust it from every peer, as the refused `*` would, and get their own wording) | | | `[secret_rotation].enforce_store_key_expiry` | `true` (a calendar-overdue store DEK refuses to start) | -| | `[secret_rotation].allow_lost_audit_range_key` | `false` (`true` starts the engine over the store key refusal its entry below states) | | | `[cert_monitor].warn_days`, `[secret_rotation].warn_days` | `30` / `14` (`0` turns that credential reminder off, and a smaller value makes it later; named as `cert_monitor.warn_days` and `secret_rotation.warn_days`) | | | `[[alerts.rules]]` | `[]` (a rule that can send a credential reminder to no transport: `mute = true`, `transports = []`, or an `escalate` tier with `transports = []`; named as `alerts.rules`) | | | `[alerts].security_signals`, its five detector counts and `security_window_seconds` | `true`, each count's default, `300` (`false` turns the security-signal layer off and is named as `alerts.security_signals`; with the layer on, a count at `0` turns that one detector off, a count above its default needs a larger burst, a shorter window lets a burst spread past it raise nothing, and each is named as `alerts.`, the window only while any count is above `0`) | @@ -277,8 +276,7 @@ keys, the three named `[auth].phi_read_rate_limit_*` keys, the four `[auth].admi `[auth].mfa_recovery_code_count`, `[auth].password_min_length`, the four other `[auth].ad_session_*` reconciler keys, `[auth].ad_reconcile_outage_passes`, the five `[auth].oidc_*` lifetime and skew keys, `[approvals].min_dwell_seconds`, `[approvals].expiry_hours`, `[approvals].operations`, -`[secret_rotation].enforce_store_key_expiry`, `[secret_rotation].allow_lost_audit_range_key`, -`[secret_rotation].warn_days`, +`[secret_rotation].enforce_store_key_expiry`, `[secret_rotation].warn_days`, `[cert_monitor].warn_days`, `[[alerts.rules]]`, `[alerts].security_signals`, its counts and `security_window_seconds`, `[api].trusted_proxies`, `[api].plaintext_upstream_hop_acknowledged`, `[api].expose_docs`, `[backup].allow_unencrypted`, @@ -834,36 +832,6 @@ This section is kept rather than deleted, because the claim it used to make is t - **Reversible:** yes, immediately — set it back to `true` (or delete the line) and restart. Nothing about the key or the store changes either way; only whether the engine agrees to start. -### `[secret_rotation].allow_lost_audit_range_key = true` — the engine starts on a key change the audit chain cannot follow -Named as `allow_lost_audit_range_key` (BACKLOG #4934), whatever the store holds, because it says what -the next start may do. -- **What you lose:** the `serve` and `supervise` start refusal that keeps the keyed audit chain across a store key - change. This entry states when that refusal fires and the way out. With it off, - `serve` (and each `supervise` engine shard) refuses to start when all of these hold: the engine holds a - local store key, so not under `[store].cipher_provider = "vault_transit"`; the stamp does not - name that key, which covers a different key, no stamp, and a stamp under a scheme this release - does not know; the store has a keyed audit chain in which the open found no range that fails to - authenticate (where it finds one, the refusal does not fire, though that start's first audit row - may still break the chain); and the - key the chain's newest range is under is not configured. It refuses whether or not the key in - use keyed an earlier range. With it on, this refusal does not fire. An audit row written in that - state, by that start or by a command this refusal does not cover, goes into that range under the - key in use and never verifies there, so the chain breaks at that row, and `rotate-key` then - cannot roll it, even after the key is put back. -- **What you keep:** the refusal's text, logged as a WARNING on every start the switch lets through, - and every other start check. -- **When acceptable:** only when the key the chain's newest range is under is truly lost. If the - key exists anywhere, put it back in `MEFOR_STORE_ENCRYPTION_KEYS_RETIRED` and run `rotate-key` - instead. Its audit chain step says whether it rolled the chain and, where it refuses, why. Where - the key in use already keyed an earlier range, that step refuses and asks for a new key. -- **Compensating controls:** none that restore the chain. Record when and why the key was lost, keep - the `audit-verify` output from before the start, and treat the break as a known event in any audit - review. -- **Reversible:** no, once a start has used it: the chain stays broken at that row. Once it has let - a start through on a key whose range key is truly lost, every later start on that key refuses - again without it. While it stays on, it also lets through a later, unrelated store key change - whose range key is lost. Set only and never used, it can be removed. - ### `[cert_monitor].warn_days` or `[secret_rotation].warn_days` at `0`, or below its default — a credential reminder is off or late Named as `cert_monitor.warn_days` and `secret_rotation.warn_days` (ASVS 6.4.5, BACKLOG #2227). A value below the shipped default (`30` and `14` days) is named too, as the sign-in limits are when looser diff --git a/messagefoundry/config/settings.py b/messagefoundry/config/settings.py index 2fa2c2d7c0..b83bc7ae7f 100644 --- a/messagefoundry/config/settings.py +++ b/messagefoundry/config/settings.py @@ -6852,10 +6852,6 @@ class SecretRotationSettings(_Section): # build would buy the setting without the posture. Setting it false is a LOOSENING and # security_loosenings() names it, so the opt-out is never silent. enforce_store_key_expiry: bool = True - # BACKLOG #4934 — True starts serve over its refusal of a store key change the keyed audit - # chain cannot follow; a LOOSENING, named by security_loosenings(). This setting's entry in - # docs/SECURITY-LOOSENING.md states the refusal's condition and way out. - allow_lost_audit_range_key: bool = False # ASVS 13.3.4 / BACKLOG #1932 — the calendar refusal for the NON-DEK classes, OPT-IN per class. # Each entry names one class from ENFORCEABLE_SECRET_EXPIRY_CLASSES. Under # [security].enforcement=ENFORCE, a named class the engine holds that is past @@ -10260,8 +10256,7 @@ def security_loosenings( ``[alerts].webhook_tls_revocation_attested`` (BACKLOG #2193), ``[ai].endpoint_revocation_attested`` (BACKLOG #2193 slice (e)), ``[alerts].email_use_tls``/``email_tls_verify`` (#323 - layer 3), ``[secret_rotation].enforce_store_key_expiry`` (#1004), - ``[secret_rotation].allow_lost_audit_range_key`` (BACKLOG #4934), the two credential-reminder + layer 3), ``[secret_rotation].enforce_store_key_expiry`` (#1004), the two credential-reminder leads ``[cert_monitor].warn_days`` and ``[secret_rotation].warn_days`` and an ``[[alerts.rules]]`` entry that can silence a reminder (#2227, #2008), the per-connection deviations — ``cleartext_accepted``, ``tls_allow_expired``, ``tls_check_hostname=false`` (ASVS @@ -10671,18 +10666,6 @@ def security_loosenings( "key's 2**32-encrypt usage ceiling still refuses unconditionally)", ) ) - # BACKLOG #4934: named whatever the store holds, as enforce_store_key_expiry is, because the - # setting says what the next start may do, not what it found. - if secret_rotation.allow_lost_audit_range_key: - out.append( - ( - "allow_lost_audit_range_key", - "the engine starts even where its refusal of a store key change the keyed audit " - "chain cannot follow would stop it, and the audit chain breaks at that start's " - "first audit row and no rotate-key run can roll it afterwards; " - "docs/SECURITY-LOOSENING.md states that refusal", - ) - ) # --- the credential reminders (ASVS 6.4.5; BACKLOG #2227 and #2008 step 4). Owner answer to # #2006 (a): a silent weakening keeps the cell at partial. _reminder_loosenings says what each # entry covers. diff --git a/messagefoundry/pipeline/secret_rotation.py b/messagefoundry/pipeline/secret_rotation.py index 7cdf32ddab..af7fb6161b 100644 --- a/messagefoundry/pipeline/secret_rotation.py +++ b/messagefoundry/pipeline/secret_rotation.py @@ -84,7 +84,6 @@ "SecretRotationRunner", "SecretStamp", "StoreKeyRotationOverdueError", - "UnrolledStoreKeyRefused", "close_rotation_record", "dek_changed", "dek_fingerprint", @@ -94,7 +93,6 @@ "fingerprints_equal", "open_rotation_record", "reconcile_rotation_meta", - "refuse_unrolled_key_change", "secrets_from_settings", "secrets_from_settings_and_stamps", ] @@ -400,59 +398,6 @@ def _rotation_hold(record: SecretRotationMetaRow | None, fingerprint: str) -> st return None -class UnrolledStoreKeyRefused(RuntimeError): - """``serve`` refused to start on a changed store key the keyed audit chain cannot follow - (BACKLOG #4934); see :func:`refuse_unrolled_key_change`.""" - - -#: The refusal :func:`refuse_unrolled_key_change` raises, and the WARNING it logs under the override. -_UNROLLED_REFUSAL = ( - "the store key in use is not confirmed by its stamp, the audit chain's current range is not " - "under it, and the key that range is under is not configured. This start's " - "first audit row would go into that range under another key, and the chain would break there, " - "so no later `messagefoundry rotate-key` could roll it. Put the key that range is under back in " - "MEFOR_STORE_ENCRYPTION_KEYS_RETIRED (it can be older than the last key, where an earlier key " - "change was never rolled; the store open's ERROR names the range's first row) and run " - "`messagefoundry rotate-key` before you start. Only where that key " - "is truly lost, set " - "[secret_rotation].allow_lost_audit_range_key = true to start anyway: that is a security " - "loosening, and the chain breaks at this start's first audit row." -) - - -async def refuse_unrolled_key_change( - meta_store: SecretRotationMetaStore, - *, - dek_key_id: str | None, - audit_rolled: bool | None, - range_key_missing: bool, - allow_lost: bool, -) -> None: - """Raise :class:`UnrolledStoreKeyRefused` on the condition the ``allow_lost_audit_range_key`` - entry in docs/SECURITY-LOOSENING.md states (BACKLOG #4934, owner ruling 2026-11-04). ``serve`` calls it right after the store open, before it writes - any audit row. ``audit_rolled`` is :func:`~messagefoundry.store.store.audit_chain_rolled` and - ``range_key_missing`` is :func:`~messagefoundry.store.store.audit_range_key_missing`, both for - the open store. - - The key counts as changed as :func:`dek_changed` says, so a store with no stamp, or a stamp - under a scheme this release does not know, counts too. A stamp naming the key in use does not - refuse, even over such a chain. ``allow_lost``, ``[secret_rotation].allow_lost_audit_range_key``, - logs the refusal as a WARNING and returns instead of raising.""" - if not dek_key_id or audit_rolled is not False or not range_key_missing: - return - prior = (await meta_store.get_secret_rotation_meta()).get(_DEK_CLASS_ID) - if dek_changed(prior, dek_fingerprint(dek_key_id)) is False: - return - if allow_lost: - log.warning( - "secret rotation: [secret_rotation].allow_lost_audit_range_key is set, so the engine " - "starts over this refusal: %s", - _UNROLLED_REFUSAL, - ) - return - raise UnrolledStoreKeyRefused(f"refusing to start: {_UNROLLED_REFUSAL}") - - #: Bytes of the per-row random HMAC key a sealed fingerprint cell holds (BACKLOG #3472). _CELL_KEY_BYTES = 32 _LOWER_HEX = "0123456789abcdef" diff --git a/messagefoundry/store/store.py b/messagefoundry/store/store.py index ae61c5ccd0..8fdd77723a 100644 --- a/messagefoundry/store/store.py +++ b/messagefoundry/store/store.py @@ -8126,32 +8126,6 @@ def audit_chain_rolled(store: object) -> bool | None: return bool(isinstance(keys, list) and keys and keys[-1] == active) -def audit_range_key_missing(store: object) -> bool: - """Whether the keyed audit chain's current range, as the open authenticated it, is keyed under - an audit key this store does not hold (BACKLOG #4934). The open then moves new rows to the - active key (:func:`settle_audit_ranges`), so the first row a start writes lands in a range keyed - under another key, and the chain breaks there; :func:`roll_audit_key_range` refuses the same - state. ``False`` with no keyed chain, and for ranges the open could not authenticate, which - say nothing about the current range's key. Read by attribute, as :func:`audit_chain_rolled` - reads; no I/O.""" - if not getattr(store, "_audit_chain_keyed", False): - return False - if not getattr(store, "_audit_ranges_trusted", False): - return False - keys = getattr(store, "_audit_range_keys", None) - if not (isinstance(keys, list) and keys): - return False - mac_keys = getattr(store, "_audit_mac_keys", None) - return ( - _audit_secret_for( - keys[-1], - mac_keys if isinstance(mac_keys, Mapping) else {}, - getattr(store, "_audit_mac_fn", None), - ) - is None - ) - - # #233 (ADR 0111): the event name for a Send declined because its target outbound is present-but-NOT- # deployed. Recorded once per declined destination (:func:`not_deployed_detail` for the detail). It has # TWO load-bearing roles, and both fail unless it is in the audit floor below: (1) it is the operator- diff --git a/tests/test_rotate_key_open_hook_and_lost_range_key.py b/tests/test_rotate_key_open_hook_and_lost_range_key.py index d414c490a8..c29fd54e74 100644 --- a/tests/test_rotate_key_open_hook_and_lost_range_key.py +++ b/tests/test_rotate_key_open_hook_and_lost_range_key.py @@ -1,19 +1,13 @@ # SPDX-License-Identifier: AGPL-3.0-or-later # Copyright (C) 2026 MessageFoundry Foundation, LLC and contributors -"""BACKLOG #4936 and #4934, the owner rulings of 2026-11-04. +"""BACKLOG #4936, the owner ruling of 2026-11-04. -#4936: ``rotate-key`` opens its rotation record through ``open_store``'s ``before_first_write`` -hook, so a run killed inside the store open after the hook leaves any record it -opened. - -#4934: ``serve``'s start refusal of a store key change the keyed audit chain cannot follow, with -``[secret_rotation].allow_lost_audit_range_key`` as its override; docs/SECURITY-LOOSENING.md -states when it fires.""" +``rotate-key`` opens its rotation record through ``open_store``'s ``before_first_write`` hook, so a +run killed inside the store open after the hook leaves any record it opened.""" from __future__ import annotations import asyncio -import logging from pathlib import Path from typing import Any @@ -24,17 +18,14 @@ from messagefoundry.config.settings import SecretRotationSettings, StoreSettings from messagefoundry.pipeline.secret_rotation import ( ROTATION_RECORD_ID, - UnrolledStoreKeyRefused, dek_fingerprint, reconcile_rotation_meta, - refuse_unrolled_key_change, ) from messagefoundry.store.base import open_store from messagefoundry.store.crypto import generate_key from messagefoundry.store.store import ( MessageStore, audit_chain_rolled, - audit_range_key_missing, ) from tests.test_rotate_key_rotation_record import ( _DEK, @@ -42,7 +33,6 @@ _env, _key_id, _open, - _rows, _seed, _seed_keyless_chain, ) @@ -124,160 +114,3 @@ async def opened(read_only: bool) -> None: assert seen == ["hook"] asyncio.run(opened(True)) assert seen == ["hook"] - - -async def _refusal(db: Path, key: str, *retired: str, allow_lost: bool = False) -> None: - store = await _open(db, key, *retired) - try: - await refuse_unrolled_key_change( - store, - dek_key_id=_key_id(key), - audit_rolled=audit_chain_rolled(store), - range_key_missing=audit_range_key_missing(store), - allow_lost=allow_lost, - ) - finally: - await store.close() - - -def test_a_start_on_the_new_key_alone_refuses_before_any_audit_row(tmp_path: Path) -> None: - """#4934 step 3: changed key, chain not rolled, current range key not configured.""" - db, key_a, key_b = tmp_path / "r.db", generate_key(), generate_key() - asyncio.run(_seed(db, tmp_path / "uploads", key_a)) - with pytest.raises(UnrolledStoreKeyRefused, match="allow_lost_audit_range_key"): - asyncio.run(_refusal(db, key_b)) - - -@pytest.mark.parametrize("case", ["prior_key_configured", "key_unchanged", "keyless_chain"]) -def test_the_refusal_needs_all_three_conditions(tmp_path: Path, case: str) -> None: - """Each condition alone failing lets the start through.""" - db, key_a, key_b = tmp_path / "c.db", generate_key(), generate_key() - if case == "keyless_chain": - asyncio.run(_seed_keyless_chain(db, tmp_path / "uploads", key_a)) - asyncio.run(_refusal(db, key_b)) - return - asyncio.run(_seed(db, tmp_path / "uploads", key_a)) - if case == "prior_key_configured": - asyncio.run(_refusal(db, key_b, key_a)) - else: - asyncio.run(_refusal(db, key_a)) - - -def test_a_stamp_naming_the_key_in_use_does_not_refuse(tmp_path: Path) -> None: - """The stamp arm alone: chain not rolled and its range key not configured, but the stamp - already names the key in use, so the start goes through. The other cases return before the - stamp is read.""" - db, key_a, key_b = tmp_path / "s.db", generate_key(), generate_key() - asyncio.run(_seed(db, tmp_path / "uploads", key_a)) - - async def stamp_key_b() -> None: - store = await _open(db, key_b) - try: - await store.upsert_secret_rotation_meta( - _DEK, - fingerprint=dek_fingerprint(_key_id(key_b)), - tracked_since=_OLD_DATE, - last_rotated=_OLD_DATE, - ) - assert audit_chain_rolled(store) is False - assert audit_range_key_missing(store) is True - finally: - await store.close() - - asyncio.run(stamp_key_b()) - asyncio.run(_refusal(db, key_b)) - - -def test_the_override_starts_with_a_warning_and_is_a_named_loosening( - tmp_path: Path, caplog: pytest.LogCaptureFixture -) -> None: - db, key_a, key_b = tmp_path / "o.db", generate_key(), generate_key() - asyncio.run(_seed(db, tmp_path / "uploads", key_a)) - with caplog.at_level(logging.WARNING, logger="messagefoundry.pipeline.secret_rotation"): - asyncio.run(_refusal(db, key_b, allow_lost=True)) - assert "allow_lost_audit_range_key is set" in caplog.text - - -def _named(settings: SecretRotationSettings) -> dict[str, str]: - from messagefoundry.config.settings import ( - AiSettings, - AlertsSettings, - ApiSettings, - ApprovalsSettings, - AuthSettings, - BackupSettings, - CertMonitorSettings, - SecretsSettings, - SecuritySettings, - security_loosenings, - ) - - return dict( - security_loosenings( - SecuritySettings(), - StoreSettings(), - AuthSettings(), - AlertsSettings(), - settings, - cleartext_hops=(), - expiry_relaxed_hops=(), - hostname_unchecked_hops=(), - query_credential_hops=(), - unverified_db_hops=(), - attested_hops=(), - revocation_attested_hops=(), - path_form_fhir_hops=(), - ech_cleartext_sidecar_hops=(), - api=ApiSettings(), - approvals=ApprovalsSettings(), - cert_monitor=CertMonitorSettings(), - backup=BackupSettings(), - secrets=SecretsSettings(), - ai=AiSettings(), - store_privilege=None, - audit_chain_unkeyed=None, - remote_debug=None, - startup=None, - service_env=None, - ) - ) - - -def test_the_override_is_a_named_loosening_and_off_by_default() -> None: - assert SecretRotationSettings().allow_lost_audit_range_key is False - assert "allow_lost_audit_range_key" not in _named(SecretRotationSettings()) - risk = _named(SecretRotationSettings(allow_lost_audit_range_key=True))[ - "allow_lost_audit_range_key" - ] - assert "audit chain breaks" in risk - - -def test_the_range_key_answer_tells_the_states_apart(tmp_path: Path) -> None: - keyed, keyless = tmp_path / "keyed.db", tmp_path / "keyless.db" - key_a, key_b = generate_key(), generate_key() - asyncio.run(_seed(keyed, tmp_path / "u1", key_a)) - asyncio.run(_seed_keyless_chain(keyless, tmp_path / "u2", key_a)) - - async def answer(db: Path, key: str, *retired: str) -> bool: - store = await _open(db, key, *retired) - try: - return audit_range_key_missing(store) - finally: - await store.close() - - assert asyncio.run(answer(keyed, key_a)) is False - assert asyncio.run(answer(keyed, key_b, key_a)) is False - assert asyncio.run(answer(keyed, key_b)) is True - assert asyncio.run(answer(keyless, key_b)) is False - assert audit_range_key_missing(object()) is False - - -def test_the_rows_after_a_refused_start_are_unchanged(tmp_path: Path) -> None: - """The refusal writes nothing: the DEK stamp and the record are as the seed left them.""" - db, key_a, key_b = tmp_path / "u.db", generate_key(), generate_key() - asyncio.run(_seed(db, tmp_path / "uploads", key_a)) - with pytest.raises(UnrolledStoreKeyRefused): - asyncio.run(_refusal(db, key_b)) - rows = asyncio.run(_rows(db, key_b, key_a)) - assert rows[_DEK].fingerprint == dek_fingerprint(_key_id(key_a)) - assert ROTATION_RECORD_ID not in rows From e406cdfae0a53005af1b6655e56d206192bf9611 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 17 Nov 2026 11:58:49 -0600 Subject: [PATCH 41/41] The 4936 fragment states what an operator sees: a run killed inside the open leaves its record Review round 1 on the re-cut of PR 2991. --- changelog.d/4936.security.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/changelog.d/4936.security.md b/changelog.d/4936.security.md index abd275b281..39daa1cba2 100644 --- a/changelog.d/4936.security.md +++ b/changelog.d/4936.security.md @@ -1,3 +1,4 @@ -- **`rotate-key` now opens its rotation record, when it opens one, from a new `open_store` hook, - `before_first_write`, which a writable open runs before its first write under the store key.** A - read-only open does not run it. (`BACKLOG #4936`) +- **A `rotate-key` run killed inside its store open, after the open's new `before_first_write` + hook has opened the rotation record, now leaves that record open, so a later `serve` start does + not stamp the new key.** A writable open runs the hook before its first write under the store + key; a read-only open does not run it. (`BACKLOG #4936`)