From 701c45fbe773701dcc52679c3ffb67ad8f554359 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sun, 2 Aug 2026 19:11:02 -0500 Subject: [PATCH 1/2] docs(asvs): make the scope boundary explicit, and correct a published attestation that was not true Two public documents, one cause: the assessment method never wrote down where the software ends and the substrate begins, so a platform-property requirement had no principled home and churned between verdicts -- and separately, a published attestation claimed more than the record supports. ASVS-ASSESSMENT-METHOD.md * Section 2 now states the boundary the "assessed as source" phrase already implied: the subject is three SOFTWARE ARTIFACTS, not the host, hypervisor, CPU, firmware or network. Where a requirement's VERB names a property of that substrate, rule 1 applies. This is ASVS's own principle and is now quoted verbatim from 0x03 at the v5.0.0 tag rather than paraphrased -- "excludes requirements ... where configuration is outside the application's responsibility", with the DNS example. * Two guards, because a scope clause grows to swallow inconvenient cells: reporting on a platform property is not providing it AND is not disqualifying either -- ask what the verb requires, and of what. * Explicitly declines the fork route. 0x03 does say organizations are "strongly encouraged" to fork, but its omission examples are technology-not-used (GraphQL/WebSockets/SOAP) and forking changes what you claim conformance TO. Rule 1 against a declared scope is narrower and leaves the target unchanged. * New section 2.1: what an out-of-scope cell does NOT buy. 4.0's "may still claim full ASVS compliance" clause was DROPPED in 5.0, and OWASP retains normative authority over which requirements sit at which level -- so an L3 claim omitting an L3 requirement is non-conformant on OWASP's own terms. * 11.7.1 moves from the worked example of a rule-3 fail to the worked example of a rule-1 na, with the objection it has to answer stated rather than omitted. Secure_Development_Standards.md * "verified against OWASP ASVS 5.0 Level 3" -> "assessed against ASVS 5.0 using Level 3 as the target -- an assessment IN PROGRESS". The old wording was unsupportable three ways: the survey is incomplete and unread cells are explicitly not passes; open requirements exist including at L2; and an L3 requirement is scoped out, which under 5.0 does not preserve an L3 claim. * The correction is stated in the document rather than made quietly, because the claim was PUBLISHED. No figure is restated here, deliberately, so this page cannot go stale against the record. * Notes that no accredited L3 pathway exists to appeal to: OWASP certifies nobody, and the one ASVS-based scheme accredits testing FIRMS at L1-L2. --- docs/ASVS-ASSESSMENT-METHOD.md | 51 +++++++++++++++++++++++++++- docs/Secure_Development_Standards.md | 11 +++++- 2 files changed, 60 insertions(+), 2 deletions(-) diff --git a/docs/ASVS-ASSESSMENT-METHOD.md b/docs/ASVS-ASSESSMENT-METHOD.md index 13dc557a..97807652 100644 --- a/docs/ASVS-ASSESSMENT-METHOD.md +++ b/docs/ASVS-ASSESSMENT-METHOD.md @@ -62,7 +62,7 @@ These are the real disputes. They are here so the next assessor reaches the same |---|---|---| | **5.4.3** | `fail` | Rule 3. A scan *hook* exists but its only shipped implementation is `_no_scan`, and there is **no configuration key at all** — an operator must **author** the scanner. Supplying a control is not configuring one. | | **15.2.5** | `partial` | Rule 5, **not** rule 3. `[sandbox].mode` ships `off`, but `subprocess` mode is real and was verified by executing it. A working control that ships off. | -| **11.7.1** | `fail` | Rule 3. The read-out is all-`None` off Linux and the only refusing branch keys on a **TOML declaration**, never on the measured property. Zero percent of the verb is satisfied by shipped code. | +| **11.7.1** | `na` | **Rule 1 — the hardest call in this table, and it moved.** The verb is *"full memory encryption is in use"*: a property of the **CPU, firmware and hypervisor**, not of the three software artifacts in §2. Outside the declared scope, so rule 1 fires before rule 3 is ever reached. **The objection this has to answer, because it is a good one:** ADR 0152's rungs 1–2 *do* ship in-engine, so the engine is not silent on this cell. But that code **reports on and gates against** the platform property — it never provides it. Reporting is not implementing (§2's first guard). ⚠️ Previously scored `fail` under rule 3 and cited here as the worked example of one. That reading was not wrong on its own terms; it answered *"does code implement the verb"* without first asking *"is the verb's subject in scope"*, and rule 1 runs first. **This cell has moved four times in eighteen days — it is CLOSED by owner decision (2026-08-02); do not re-derive it.** ⛔ It buys **no** Level 3 claim: see §2.1. | | **3.7.3** | `fail` | Rule 3. One off-site navigation, a bare 303, no interstitial and no cancel. `oidc_enabled=False` removes the **trigger**, not a control. | | **10.5.5** | `na` | Rule 1. The requirement is conditional — *"**when using** OIDC back-channel logout"* — and the precondition is false and unreachable by configuration. **Building it would create applicability.** | | **12.2.2** | `na` | Rule 1. No external-facing services on the declared scope. *Also a scoping error worth remembering: this row spent months scoring 12.3.1's verb.* | @@ -86,6 +86,55 @@ Per ASVS's guidance, stated as what **is** included rather than what is excluded **Included:** the MessageFoundry engine, the web console, and the IDE extension, assessed as source, at a named commit, against **all 345 ASVS 5.0.0 requirements** at **Level 3**. +**The boundary that phrase implies, stated explicitly because it decides verdicts.** The subject of +this assessment is those three **software artifacts**. It is not the host, the hypervisor, the CPU, the +firmware, or the network the operator deploys onto. Where a requirement's **verb** names a property of +that substrate rather than of the software, the requirement is outside the declared scope and takes +`na` under rule 1 — with the rationale written, as always. + +**This is ASVS's own principle, not a local invention.** Verbatim from `0x03-What-is-the-ASVS.md` at the +`v5.0.0` tag: + +> "Conversely, ASVS generally excludes requirements that are not directly relevant to the application or +> **where configuration is outside the application's responsibility**. For example, DNS issues are +> typically managed by a separate team or function." + +⚠️ **Do not over-read that, and do not reach for the fork clause to do this job.** The same chapter says +organizations are *"strongly encouraged to create an organization- or domain-specific fork that adjusts +requirements"* — but its worked examples of omission are **technology-not-used** (*"omitting irrelevant +sections (e.g., GraphQL, WebSockets, SOAP, if unused)"*), which is the functionality-based shape, and +**forking changes what you are claiming conformance TO**: your tailored ASVS, not stock ASVS 5.0. This +project does **not** fork. It applies rule 1 against a positively-declared scope, which is narrower, +cheaper to defend, and leaves the conformance target unchanged. + +Two guards on that boundary, because it is exactly the kind of clause that grows to swallow +inconvenient cells: + +* **Shipping code that *reports on* a platform property is not the same as *providing* it, and neither + direction is decisive on its own.** A cell does not become in-scope merely because the engine + observes the substrate; nor does it leave scope merely because the substrate is involved. Ask what + the **verb** requires to be true, and of what. +* ⛔ **This boundary does not shrink the conformance claim's denominator.** A requirement excluded here + is still a requirement OWASP assigns to Level 3. See §2.1. + +### 2.1 What an out-of-scope cell does NOT buy + +**It does not preserve an unqualified Level 3 claim.** Two facts, both from ASVS 5.0.0 itself: + +* **4.0's clause that an organization excluding requirements "may still claim full ASVS compliance" + was DROPPED in 5.0.** The 5.0 text says only that non-applicability must be noted in the report. + There is no longer any standard text saying a documented exclusion preserves a compliance claim, and + a rationale that cites the older wording is citing a superseded standard. +* **OWASP does not certify anyone — but it does retain normative authority over the requirement SET.** + It assigns each requirement to a level. So a Level 3 claim that silently omits a requirement OWASP + places at Level 3 is non-conformant **on OWASP's own terms**, regardless of how well-argued the + exclusion is. + +**Therefore:** scoping a cell out is a statement about *what was assessed*, never a statement that the +level was achieved anyway. Any published attestation must say which requirements were excluded, or say +something weaker than "verified at Level 3". Writing `na` in the record and "Level 3 verified" in a +brochure is the failure mode this section exists to prevent. + **The configuration assessed** — one posture, not a matrix: > On-premises single hospital · private network, never internet-facing · SQL Server store · operator diff --git a/docs/Secure_Development_Standards.md b/docs/Secure_Development_Standards.md index 5658f067..9302bed6 100644 --- a/docs/Secure_Development_Standards.md +++ b/docs/Secure_Development_Standards.md @@ -364,7 +364,16 @@ The project maintains a current evidence set so any claim is backed: - **Per-project applicability profile** (Appendix A and onward). - A **claims register** recording each published claim, its wording, and the evidence behind it. -**Attestation posture.** The software is self-attested as NIST SSDF–aligned, tested per NIST SP 800-115, verified against OWASP ASVS 5.0 Level 3, and built to support HIPAA-compliant deployment (controls mapped to NIST SP 800-66 Rev. 2). Third-party validation of the SSDF attestation and the ASVS 5.0 Level 3 assessment raises the weight of these claims. **Attestations are published with releases** so adopters can rely on them; each adopter still performs its own deployment risk assessment (§7.3). None of these is a NIST certificate; displayable certificates (SOC 2, ISO 27001, HITRUST) are a separate, organization-level track. +**Attestation posture.** The software is self-attested as NIST SSDF–aligned, tested per NIST SP 800-115, **assessed against** OWASP ASVS 5.0 **using Level 3 as the target** — an assessment **in progress**, not a completed verification — and built to support HIPAA-compliant deployment (controls mapped to NIST SP 800-66 Rev. 2). Third-party validation of the SSDF attestation and of the ASVS assessment raises the weight of these claims. + +> **The ASVS wording above changed on 2026-08-02, and the previous wording was wrong.** It read *"verified against OWASP ASVS 5.0 Level 3."* Three facts make that unsupportable, and they are stated here rather than quietly corrected because the claim was **published**: +> 1. **The survey is incomplete.** A minority of the 345 requirements have been read against the ASVS text at a known commit; the remainder are recorded as *unverified*, which is explicitly **not** a pass. "Verified" asserted a completed verification that had not occurred. +> 2. **Open requirements exist**, including at Level 2 — so the claim was not rescuable by narrowing it to a lower level. +> 3. **At least one Level 3 requirement is scoped out** as a hosting-platform property outside the assessed software. Under ASVS 5.0 that does **not** preserve a Level 3 claim: 4.0's "may still claim full ASVS compliance" clause was **dropped** in 5.0, and OWASP retains normative authority over which requirements sit at which level. A Level 3 claim omitting a Level 3 requirement is non-conformant on OWASP's own terms. +> +> **No accredited Level 3 pathway exists to appeal to.** OWASP certifies no vendor or software and states that any trust mark claiming ASVS compliance is not officially endorsed by it; the one ASVS-based accreditation scheme accredits testing *firms*, and its published scope is Levels 1–2. So "self-attested" is the only honest register available here, and this section now uses it. +> +> Current status is held in the project's private assessment record, which is the count of record; **no figure is restated here**, deliberately, so this page cannot go stale against it. **Attestations are published with releases** so adopters can rely on them; each adopter still performs its own deployment risk assessment (§7.3). None of these is a NIST certificate; displayable certificates (SOC 2, ISO 27001, HITRUST) are a separate, organization-level track. --- From fe44fac026c982e6a48c7ef2ea7ab123502f3651 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sun, 2 Aug 2026 19:14:42 -0500 Subject: [PATCH 2/2] docs(asvs): a count movement is not a posture movement -- write down the four causes When 11.7.1 was scoped out under rule 1, the fail count went 3 -> 2 with ZERO lines of engine code changed. A reader comparing only the totals across those two days concludes a defect was fixed. Nothing was. The rationale sits on the cell and the boundary sits in section 2, but neither is visible to someone reading a number. New section 2.2 names the four causes a bucket total moves, of which exactly one is an improvement: a control shipped on by default (real); a cell read for the first time (the survey advanced -- unverified -> pass is a DISCOVERY, not a fix); a scope boundary stated (identical code, smaller denominator); a rule applied more carefully (more accurate, and these move DOWN too). Two obligations, both cheap: never report a total as a trend without naming which cause moved it, and when a count improves, state what would have had to happen for it to MEAN improvement and whether that happened -- the negative-control discipline applied to a metric. Written down because it cuts AGAINST us more often than for us. The survey is incomplete, so most future movement is cause 2, and the aggregate will get worse before it gets better as unverified cells resolve into real verdicts. That is the survey working, and it should be reported as such rather than defended against. Sits directly under 2.1 (an out-of-scope cell buys no Level 3 claim), because the two failure modes are the same one seen from opposite ends: 2.1 stops a scoped-out cell becoming a conformance claim, 2.2 stops it becoming a trend line. Prompted by the ASVS-11.7.1-summary session, whose objection -- that the rule-3 observations about the code stay literally true even after the verdict moves -- is what made the gap visible. --- docs/ASVS-ASSESSMENT-METHOD.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/docs/ASVS-ASSESSMENT-METHOD.md b/docs/ASVS-ASSESSMENT-METHOD.md index 97807652..7c0e2bdd 100644 --- a/docs/ASVS-ASSESSMENT-METHOD.md +++ b/docs/ASVS-ASSESSMENT-METHOD.md @@ -135,6 +135,37 @@ level was achieved anyway. Any published attestation must say which requirements something weaker than "verified at Level 3". Writing `na` in the record and "Level 3 verified" in a brochure is the failure mode this section exists to prevent. +### 2.2 A count movement is not a posture movement — read the counts accordingly + +**The single most misreadable thing this method produces is a change in the bucket totals.** Counts move +for four different reasons, and only one of them means the software got better: + +| The count moved because… | Did the posture improve? | What actually happened | +|---|---|---| +| A control was **built or turned on by default** | **Yes** | The verb is now satisfied by shipped code | +| A cell was **read for the first time** (`unverified` → anything) | **No** | The survey advanced. A cell moving `unverified` → `pass` is a *discovery*, not an improvement | +| A **scope boundary was stated** (→ `na`) | **No** | The requirement left the denominator. Identical code, smaller question | +| A **rule was applied more carefully** (re-grade in either direction) | **No** | The assessment got more accurate. Some of these move *down* | + +**The worked example, and it is recent.** On 2026-08-02 the fail count went **3 → 2** when 11.7.1 was +scoped out under rule 1. **Zero lines of engine code changed.** A reader comparing only the fail counts +across those two days would conclude a defect was fixed; nothing was. The rationale is on the cell and +the boundary is in §2, but neither is visible to someone reading a total. + +**Two obligations follow, and they are cheap:** + +1. **Never report a bucket total as a trend without naming which of the four causes moved it.** "Fails + went from 3 to 2" is not a finding. "One cell was scoped out; no code changed" is. +2. **When a count improves, state what would have had to happen for it to mean an improvement, and + whether that happened.** This is the same discipline as a negative control: a number that can only + go one way is not measuring anything. + +⚠️ **This cuts against us more often than for us, which is why it is written down.** The survey is +incomplete, so most future movement will come from cause 2 — cells being read for the first time — and +the aggregate will get *worse* before it gets better as `unverified` cells resolve into real verdicts. +**That is the survey working, not a regression**, and it should be reported as such rather than +defended against. + **The configuration assessed** — one posture, not a matrix: > On-premises single hospital · private network, never internet-facing · SQL Server store · operator