Skip to content

fix(security): reconcile inconsistent aggregate exception markers #4523

Description

@devantler

🤖 Generated by the Agentic Engineer

Problem

A named control-detail read can report status: passed with subStatus: w/exceptions even though all of its rules report an unsuppressed pass and every rule has no applied ignore rule. The current exception definitions do not match the affected record. The stored workload independently carries the required security fields.

This was verified through named reads and current declarations on 2026-10-05. The public example is intentionally synthetic; full receipts remain private:

{"status":{"status":"passed","subStatus":"w/exceptions"},"rules":[{"status":{"status":"passed","subStatus":""},"appliedIgnoreRules":null}]}

Impact

The aggregate exception indicator cannot currently establish whether the control really passed or whether a suppression remains. Presence, rule count and a bare passed status are insufficient to clear the security acceptance.

Work

Trace where the aggregate marker is computed and persisted, using a bounded synthetic reproduction. Repair the responsible write or aggregation path rather than rewriting production scan data or weakening exception scope. Distinguish this from the earlier stale-detail investigation in #3275 and the deleted-record retention work in #4505.

Acceptance criteria

  • Reproduce a stale aggregate exception marker when the final suppression is removed, or establish another verified cause.
  • The repaired record's aggregate status agrees with its actual rule-level evaluation after a new scan.
  • A genuinely excepted control retains its exception marker; an unsuppressed pass clears it.
  • Failed or incomplete observations cannot produce a clean verdict, and storage permissions remain unchanged.
  • Verify the result by named live read after delivery, without treating creation time alone as scan freshness.

Part of #2824

Remaining work: kubescape v4.0.16 (published 2026-10-14) contains kubescape/kubescape#4044; platform#4911 moves the scanner onto it. After it deploys, read a scan on the new release back live (last criterion).

Blocker: the first scheduled Kubescape posture scan after the Sigstore egress fix, due 2026-10-17 09:12 UTC | outcome | last-verified 2026-10-17: #4966 closed and its fix #4967 merged 2026-10-16 22:41 UTC; the stored posture objects still describe the last pre-upgrade scan. After the scan, confirm it completed, then re-read the named control detail for the stale aggregate exception marker

Activity

  1. devantler commented on Oct 5, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Read-only persistence inspection is now bound to the deployed storage base, kubescape/storage v0.0.297 (b35788b68337134fc2514574cde1ba7f1225fd43), rather than current upstream main.

    The workload scan endpoint uses the generic storage backend. Its creation/update strategy leaves the supplied object unchanged, and the backend's default processor has no scan-status transformation. The virtual namespace summary sums existing workload-summary severities; it does not reconstruct a workload control's exception marker from its rules. The Platform compatibility patches inspected for this path do not introduce such a recomputation either.

    This narrows the investigation: repairing the virtual namespace summary alone would not repair the contradictory marker in the named workload scan. Next proof needs the scanner's status calculation and a scan receipt that joins the resulting aggregate and rule details. A stable resourceVersion or creation time alone cannot establish that a fresh scan produced this contradiction. No stored scan data, permissions or exception definitions were changed.

    Source references: endpoint assembly, workload strategy, default processor, virtual summary.

  2. added a commit that references this issue on Oct 5, 2026
  3. devantler commented on Oct 5, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Cause established: the scanner merges the previously stored control status into every rescan

    Read from the released source of the scanner version the platform deploys (kubescape image v4.0.14, which pins opa-utils v0.0.312), not from upstream main. No scan data, permissions or exception definitions were touched.

    What happens on a rescan of a workload that already has a stored record

    1. The scanner first tries to create the workload scan. The deployed storage (v0.0.297) refuses a create for an existing key, so every rescan takes the update path (StoreWorkloadConfigurationScanResult).
    2. The update path does not write the new result. It calls mergeWorkloadConfigurationScanSpec, which, for each control present in both records:
      • computes the control's status with CompareStatusAndSubStatus(existing, new, …), and
      • replaces rules with the new scan's rules.
    3. CompareStatusAndSubStatus returns the more significant of its two inputs: for two passed inputs it returns w/exceptions when either side carries it.

    So once a control has been stored as passed / w/exceptions, a later scan in which the suppression no longer applies produces exactly the record this issue describes: the control keeps subStatus: w/exceptions (inherited from the old record) while its rules are the new ones, each an unsuppressed pass with appliedIgnoreRules: null. The aggregate marker and the rule details come from two different scans.

    The per-workload summary is merged the same way (mergeWorkloadConfigurationScanSummarySpec), so it inherits the same marker.

    Consequences that follow from the same code (source-derived; not yet observed live)

    • The merge only ever keeps the more significant status, so it cannot turn a failing or excepted control into a clean pass. The fourth acceptance criterion (no clean verdict from an incomplete observation) is not at risk from this path.
    • The same rule makes failed sticky: Compare returns failed when either side is failed. A control that failed once and was then remediated would keep reading failed in the stored record until that record is deleted and created afresh. This needs one named live read to confirm or refute, because it would mean stored failures can outlive their fix, which bears on the measured compliance floor (Ratchet the measured combined security compliance floor to 98 percent #4520) as an under-count, never an over-count.
    • A control that disappears from the new scan is dropped from the record, and a brand-new record is written exactly as scanned. Only controls present in both scans are affected.

    Not fixed upstream yet. The merge function is unchanged in the newest released scanner (v4.0.15) apart from an unrelated metadata copy, so a version bump does not clear it.

    Acceptance criteria against this finding

    • Reproduce … or establish another verified cause — cause established from the deployed version's source, as above. A synthetic reproduction is three lines against the public function: CompareStatusAndSubStatus(passed, passed, "w/exceptions", "") returns passed, w/exceptions.
    • Repaired record agrees with rule-level evaluation — open, needs the repair below.
    • Genuine exception keeps its marker, unsuppressed pass clears it — open.
    • No clean verdict from an incomplete observation; storage permissions unchanged — holds by construction (the merge is monotone towards the more significant status), and nothing here changes permissions.
    • Named live read after delivery — open.

    Repair options

    Option What it changes Cost
    A. Fix the merge upstream so a full rescan's control status replaces the stored one The scanner, for everyone Needs a third-party contribution, which an unattended run may not open; a released fix then arrives as an ordinary version bump
    B. Carry a reviewed scanner patch, the way the storage and node-agent images already are One more platform-built image to maintain until A ships Highest ongoing cost; removes the contradiction at the source without waiting
    C. Recreate a workload's stored record when its exception definitions change A platform-side reset so the next scan is written as scanned Smallest change, but it deletes stored scan data, which this issue asked to avoid, and it does not help the sticky-failed case unless it also runs after remediation

    Recommendation: A, with the sticky-failed live read done first — if that read confirms stored failures outlive their fix, the under-count is worth the interim patch in B; if it refutes it, the contradiction is cosmetic on the safe side and A alone is enough. Until one of them lands, a w/exceptions marker on a stored control means "was excepted in some earlier scan", and only the rule-level appliedIgnoreRules says whether an exception applies now.

  4. devantler commented on Oct 5, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Live read: the sticky-failed consequence is not observed; the stale exception marker is, and it is rare

    Follow-up to the cause recorded above. Read-only named reads of 45 stored workload scans (Deployments, StatefulSets, DaemonSets and CronJobs across namespaces, excluding the scanner's own), 2026-10-05 ~15:45Z. A list was not used, since it does not return control detail. Nothing was changed.

    Control status in the stored record Controls
    passed 2,097
    passed / w/exceptions 271
    skipped / configuration 164
    failed 100
    • failed with every rule passing: 0 of 100. The source allows a remediated control to stay failed, but this sample shows no instance. That is evidence against it mattering in practice here, not proof it cannot happen; a control remediated and rescanned inside the sample would have shown it.
    • w/exceptions with every rule an unsuppressed pass and no applied ignore rule: 2 of 271, one each on C-0077 and C-0211, in two different records. This is the contradiction this issue describes, reproduced by named read, and it matches the merge behaviour: the marker survives from an earlier scan while the rules are the current ones.

    An earlier pass over 120 records of other kinds (Secrets, Endpoints, volume claims) found only plain passed controls, so the effect is confined to workload records that have carried an exception.

    What this changes

    • The measured compliance figure is not being held down by stale failures in this sample, so the interim scanner patch (option B) is not justified by it.
    • The stale marker over-reports exceptions slightly (under 1% of excepted controls sampled) and never reports a clean pass that is not one.
    • Recommendation: option A only — correct the merge upstream so a full rescan's control status replaces the stored one, and take it as a normal version bump. Opening that upstream contribution needs the maintainer's go-ahead; it is not something an unattended run may do. Until then, read appliedIgnoreRules at rule level to decide whether an exception applies now, as the exception guard already documents for C-0211.

    Remaining acceptance criteria (repaired record, marker cleared on an unsuppressed pass, live read after delivery) stay open behind that fix.

  5. devantler commented on Oct 6, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Maintainer answer, given in the Slack thread under the ask on 2026-10-05 and missed until now (no run was reading thread replies; the maintainer pointed this out on 2026-10-06):

    2

    Meaning: option 2 of the ask — an upstream issue and a fix pull request on the scanner are approved. This is no longer blocked on a decision. Next: write the upstream issue from the released source with a synthetic example (nothing about this cluster), then the fix pull request, following the third-party contribution rules; link both here.

  6. devantler commented on Oct 6, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The upstream report and a tested fix are written and ready to submit

    The maintainer approved an upstream issue and a fix pull request (option 2, recorded above). Both are prepared below. Neither has been opened yet: this scheduled run's dispatch lists only devantler-tech pull requests and issues as write actions, so it stopped at preparation. Any run or session whose dispatch covers third-party contributions can submit them as they stand.

    What changed in my understanding of the cause

    Keeping the more significant status is deliberate upstream: a table test named existing failed wins over new passed pins it. So "replace instead of merge" would be rejected as written. What is missing is a distinction between results from the same report and results from a later one. The newest scanner release (v4.0.15) records each report's generation time on the stored record, which makes that distinction possible.

    The fix

    When the incoming scan comes from a newer report than the stored record, its status replaces the stored one. Results from the same report keep today's behaviour. A scan with no timestamp, or one older than the stored record, also keeps today's behaviour, so a late or unplaceable result can never clear a stored failure.

    Proof, run against the released v4.0.15 source in a disposable copy:

    Check Result
    Whole storage package with the fix (existing tests plus 8 new cases) 75 passed, go vet clean
    The 8 new cases against the unpatched source 3 fail: exception removed, control remediated, legacy record without a timestamp
    The other 5 new cases (still-excepted, regression, same report, late older report, no timestamp) pass on both, so the fix changes nothing for them

    This covers acceptance criteria 2 and 3 at source level. The fourth still holds: nothing can produce a clean verdict from a scan that cannot be placed in time.

    Two things the fix does not settle

    1. The per-workload summary has no timestamp, so it cannot make the same decision by itself. The report asks upstream which way they prefer; the patch leaves the summary alone.
    2. Production cannot use this fix by a scanner bump alone. Production runs scanner v4.0.14 with the platform-built storage based on v0.0.297; the timestamp field arrives with v4.0.15 and a newer storage API. The fix only takes effect once the platform's storage image is rebased far enough to persist that field. That rebase is separate work and should be filed when the upstream fix is accepted.

    Before submitting

    • The project's contribution rules live in its central governance repository, which I did not read. Check them, including any sign-off or AI-assistance requirement, and follow the target's disclosure policy.
    • Open the issue first and link the pull request to it; the report asks two design questions that may change the patch.
    • The pull request template asks for an overview, how to test (go test ./httphandler/storage) and the related issue.
    Upstream issue text

    Title: Rescans merge the previously stored control status, so a remediated control stays failed and a removed exception stays w/exceptions

    Description

    When the in-cluster scanner stores a WorkloadConfigurationScan for a workload that already has a stored record, it does not write the new result. It merges the stored control status into the new one and keeps the more significant of the two, while replacing the control's rules with the new scan's rules.

    The result is a stored control whose status and rules come from two different scans:

    • a control that failed once and was later fixed keeps status: failed, although every rule in the same record now passes;
    • a control that was once excepted keeps subStatus: w/exceptions after the exception is removed, although no rule in the same record has an applied ignore rule.

    The per-workload WorkloadConfigurationScanSummary is merged the same way, so its severity counts inherit the stale status.

    Where

    httphandler/storage/apiserver.go, at v4.0.15:

    • StoreWorkloadConfigurationScanResult takes the update path whenever the record already exists and calls mergeWorkloadConfigurationScanSpec.
    • mergeWorkloadConfigurationScanSpec computes the status with apis.CompareStatusAndSubStatus(existing, new, …) and sets Rules: newCtrl.Rules.
    • mergeWorkloadConfigurationScanSummarySpec does the same for the summary.

    CompareStatusAndSubStatus (opa-utils v0.0.312) returns failed when either side is failed, and for two passed inputs returns w/exceptions when either side carries it.

    Reproduction

    No cluster is needed. Against the current function:

    existing := v1beta1.WorkloadConfigurationScanSpec{Controls: map[string]v1beta1.ScannedControl{
        "C-001": {ControlID: "C-001", Status: v1beta1.ScannedControlStatus{Status: "failed"},
            Rules: []v1beta1.ScannedControlRule{{Name: "rule", Status: v1beta1.RuleStatus{Status: "failed"}}}},
    }}
    rescan := v1beta1.WorkloadConfigurationScanSpec{Controls: map[string]v1beta1.ScannedControl{
        "C-001": {ControlID: "C-001", Status: v1beta1.ScannedControlStatus{Status: "passed"},
            Rules: []v1beta1.ScannedControlRule{{Name: "rule", Status: v1beta1.RuleStatus{Status: "passed"}}}},
    }}
    got := mergeWorkloadConfigurationScanSpec(existing, rescan).Controls["C-001"]
    // got.Status.Status == "failed", got.Rules[0].Status.Status == "passed"

    The existing table test existing failed wins over new passed, rules replaced by new scan pins this behaviour.

    In a cluster I have observed the exception variant: reading stored WorkloadConfigurationScan records by name, a small share of controls marked passed / w/exceptions had only unsuppressed passing rules with appliedIgnoreRules: null. I have not observed the failed variant in a cluster; it follows from the same code path and the snippet above.

    Expected behaviour

    A stored control's status agrees with the rules stored beside it. A scan from a newer report replaces the status from an older one.

    Suggested fix

    Keeping the more significant status looks intended for results that belong to the same report. v4.0.15 records the report generation time in spec.metadata.report.createdAt, which makes that distinction available: keep the current merge when both sides come from the same report, and let the new scan's status replace the stored one when its report is newer. A new scan without a timestamp, or one older than the stored record, keeps the current behaviour, so a late or unplaceable result can never clear a stored failure.

    I have a small patch for mergeWorkloadConfigurationScanSpec along those lines, with table tests, and can open a pull request. Two things I would like your view on first:

    1. Is the merge meant only for multiple results within one report, or is carrying status across reports relied on somewhere?
    2. WorkloadConfigurationScanSummarySpec has no report timestamp, so the summary cannot make the same decision by itself. Would you prefer the summary to be rebuilt from the merged scan, or to carry the timestamp too?

    Environment

    • Kubescape v4.0.14 in-cluster (operator-triggered continuous scanning); merge code read at v4.0.14 and v4.0.15, unchanged in substance.
    Patch against v4.0.15
    --- a/httphandler/storage/apiserver.go
    +++ b/httphandler/storage/apiserver.go
    @@ -299,10 +299,26 @@
     	return nil
     }
     
    +// isNewerReport reports whether newMeta comes from a later posture report than existingMeta.
    +// A stored scan without a report timestamp predates the field, so any timestamped report is
    +// newer. A new scan without one cannot be placed in time and is treated as the same report.
    +func isNewerReport(existingMeta, newMeta *v1beta1.WorkloadConfigurationScanMeta) bool {
    +	if newMeta == nil || newMeta.Report.CreatedAt.IsZero() {
    +		return false
    +	}
    +	if existingMeta == nil || existingMeta.Report.CreatedAt.IsZero() {
    +		return true
    +	}
    +	return newMeta.Report.CreatedAt.Time.After(existingMeta.Report.CreatedAt.Time)
    +}
    +
     func mergeWorkloadConfigurationScanSpec(existingSpec v1beta1.WorkloadConfigurationScanSpec, newSpec v1beta1.WorkloadConfigurationScanSpec) v1beta1.WorkloadConfigurationScanSpec {
     	if existingSpec.Controls == nil {
     		existingSpec.Controls = make(map[string]v1beta1.ScannedControl)
     	}
    +	// A status from an earlier report describes an earlier state of the workload. Carrying it into
    +	// a newer report leaves the control's status and its rules describing two different scans.
    +	supersedes := isNewerReport(existingSpec.Metadata, newSpec.Metadata)
     	for ctrlID, newCtrl := range newSpec.Controls {
     		existingCtrl, found := existingSpec.Controls[ctrlID]
     		if !found {
    @@ -314,10 +330,13 @@
     		existingSubStatus := apis.ScanningSubStatus(existingCtrl.Status.SubStatus)
     		newStatus := apis.ScanningStatus(newCtrl.Status.Status)
     		newSubStatus := apis.ScanningSubStatus(newCtrl.Status.SubStatus)
    -		mergedStatus, mergedSubStatus := apis.CompareStatusAndSubStatus(existingStatus, newStatus, existingSubStatus, newSubStatus)
    +		mergedStatus, mergedSubStatus := newStatus, newSubStatus
    +		if !supersedes {
    +			mergedStatus, mergedSubStatus = apis.CompareStatusAndSubStatus(existingStatus, newStatus, existingSubStatus, newSubStatus)
    +		}
     
     		info := newCtrl.Status.Info
    -		if info == "" {
    +		if info == "" && !supersedes {
     			info = existingCtrl.Status.Info
     		}
     
    New test file: httphandler/storage/apiserver_merge_report_test.go
    package storage
    
    import (
    	"testing"
    	"time"
    
    	"github.com/kubescape/opa-utils/reporthandling/apis"
    	"github.com/kubescape/storage/pkg/apis/softwarecomposition/v1beta1"
    	"github.com/stretchr/testify/assert"
    	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    )
    
    func reportMeta(t time.Time) *v1beta1.WorkloadConfigurationScanMeta {
    	return &v1beta1.WorkloadConfigurationScanMeta{Report: v1beta1.ReportMeta{CreatedAt: metav1.NewTime(t)}}
    }
    
    func controlSpec(meta *v1beta1.WorkloadConfigurationScanMeta, status apis.ScanningStatus, subStatus apis.ScanningSubStatus, rule string) v1beta1.WorkloadConfigurationScanSpec {
    	return v1beta1.WorkloadConfigurationScanSpec{
    		Metadata: meta,
    		Controls: map[string]v1beta1.ScannedControl{
    			"C-001": {
    				ControlID: "C-001",
    				Name:      "Control 1",
    				Status:    v1beta1.ScannedControlStatus{Status: string(status), SubStatus: string(subStatus)},
    				Rules:     []v1beta1.ScannedControlRule{{Name: rule}},
    			},
    		},
    	}
    }
    
    func TestMergeWorkloadConfigurationScanSpec_NewerReportReplacesStatus(t *testing.T) {
    	earlier := time.Date(2026, time.September, 1, 8, 0, 0, 0, time.UTC)
    	later := earlier.Add(24 * time.Hour)
    
    	tests := []struct {
    		name              string
    		existing          v1beta1.WorkloadConfigurationScanSpec
    		new               v1beta1.WorkloadConfigurationScanSpec
    		expectedStatus    apis.ScanningStatus
    		expectedSubStatus apis.ScanningSubStatus
    	}{
    		{
    			name:              "exception removed: newer unsuppressed pass clears the exception marker",
    			existing:          controlSpec(reportMeta(earlier), apis.StatusPassed, apis.SubStatusException, "rule-old"),
    			new:               controlSpec(reportMeta(later), apis.StatusPassed, "", "rule-new"),
    			expectedStatus:    apis.StatusPassed,
    			expectedSubStatus: "",
    		},
    		{
    			name:              "remediated: newer pass replaces an earlier failure",
    			existing:          controlSpec(reportMeta(earlier), apis.StatusFailed, "", "rule-old"),
    			new:               controlSpec(reportMeta(later), apis.StatusPassed, "", "rule-new"),
    			expectedStatus:    apis.StatusPassed,
    			expectedSubStatus: "",
    		},
    		{
    			name:              "regressed: newer failure replaces an earlier pass",
    			existing:          controlSpec(reportMeta(earlier), apis.StatusPassed, "", "rule-old"),
    			new:               controlSpec(reportMeta(later), apis.StatusFailed, "", "rule-new"),
    			expectedStatus:    apis.StatusFailed,
    			expectedSubStatus: "",
    		},
    		{
    			name:              "exception still applies: newer excepted pass keeps the marker",
    			existing:          controlSpec(reportMeta(earlier), apis.StatusPassed, apis.SubStatusException, "rule-old"),
    			new:               controlSpec(reportMeta(later), apis.StatusPassed, apis.SubStatusException, "rule-new"),
    			expectedStatus:    apis.StatusPassed,
    			expectedSubStatus: apis.SubStatusException,
    		},
    		{
    			name:              "legacy stored scan without a timestamp is superseded by a timestamped report",
    			existing:          controlSpec(nil, apis.StatusFailed, "", "rule-old"),
    			new:               controlSpec(reportMeta(later), apis.StatusPassed, "", "rule-new"),
    			expectedStatus:    apis.StatusPassed,
    			expectedSubStatus: "",
    		},
    		{
    			name:              "same report: the more significant status is kept",
    			existing:          controlSpec(reportMeta(later), apis.StatusFailed, "", "rule-old"),
    			new:               controlSpec(reportMeta(later), apis.StatusPassed, "", "rule-new"),
    			expectedStatus:    apis.StatusFailed,
    			expectedSubStatus: "",
    		},
    		{
    			name:              "older report arriving late does not clear a newer failure",
    			existing:          controlSpec(reportMeta(later), apis.StatusFailed, "", "rule-old"),
    			new:               controlSpec(reportMeta(earlier), apis.StatusPassed, "", "rule-new"),
    			expectedStatus:    apis.StatusFailed,
    			expectedSubStatus: "",
    		},
    		{
    			name:              "new scan without a timestamp cannot clear a stored failure",
    			existing:          controlSpec(reportMeta(earlier), apis.StatusFailed, "", "rule-old"),
    			new:               controlSpec(nil, apis.StatusPassed, "", "rule-new"),
    			expectedStatus:    apis.StatusFailed,
    			expectedSubStatus: "",
    		},
    	}
    	for _, tt := range tests {
    		t.Run(tt.name, func(t *testing.T) {
    			got := mergeWorkloadConfigurationScanSpec(tt.existing, tt.new).Controls["C-001"]
    			assert.Equal(t, string(tt.expectedStatus), got.Status.Status)
    			assert.Equal(t, string(tt.expectedSubStatus), got.Status.SubStatus)
    			assert.Equal(t, []v1beta1.ScannedControlRule{{Name: "rule-new"}}, got.Rules)
    		})
    	}
    }
  7. devantler commented on Oct 7, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer (interactive Claude Code session)

    Maintainer confirmation, given in an interactive session on 2026-10-07: open the approved upstream contribution now, from this session. No AI attribution unless the target project requires it.

    Next (this session): re-check the target project's contribution and AI policy and that the prepared change still applies upstream, open it, and link it here.

  8. devantler commented on Oct 7, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer (interactive Claude Code session)

    The upstream contribution is not opened, because upstream fixed this defect on 2026-10-06. Checked on 2026-10-07 before submitting anything.

    What this issue waits for now: a Kubescape release that contains the fix. The newest release is v4.0.15 from 2026-09-29, which predates it. The blocker line is updated to that.

    One thing to re-check when the release arrives. The earlier note says production cannot use a fix by a scanner upgrade alone, because the platform-built storage is older than the report-time field. The merged fix also writes the report time as an annotation from the scanner side, which may make the scanner upgrade sufficient without rebasing the storage image. That is unverified: confirm it against the released code and a named live read before closing the acceptance criteria.

  9. devantler commented on Oct 15, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Pre-deployment verification baseline

    A fresh read-only sample retrieved 105 workload scan records by name (6,669 controls), not through the spec-stripping list response. It found 17 controls carrying an aggregate exception marker despite only unsuppressed passing rules, including both original affected controls. It also found 803 controls with genuinely applied exceptions. This is a wider sample than the earlier investigation, not a like-for-like regression-rate comparison.

    No sampled record carries a usable report identity. The released scanner writes kubescape.io/report-created-at as an annotation as well as the newer spec metadata, so the post-deployment check can bind a genuinely new report even when older storage drops the spec field. That storage compatibility remains a live verification requirement, not a source-derived success claim.

    The scanner was still on the pre-fix release at collection. #4911 remains queued; keep this issue open until a new-release report clears the stale markers while retaining genuine exceptions. No stored results, exception definitions or permissions were changed.

  10. devantler commented on Oct 15, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    PR #4911 is now actually merged: 7cde9dcfb748e5d867d48f56be39b13d8a17ca10 at 11:06:32 UTC. OIDC readback confirms scanner v4.0.16 is running, Deployment generation 42 is observed with 1/1 ready, and the HelmRelease reports UpgradeSucceeded. The new server started at 10:57:38 UTC during the queue deployment.

    Verification remains open: all 2,801 stored posture objects still lack kubescape.io/report-created-at; the original C-0077 object has the same resourceVersion and contradiction as the pre-upgrade sample. The daily scan completed before this deployment, so existing records are not evidence that the new scanner has rescanned them. No reports were deleted or patched and no local production scan was started. Wait for a fresh scan before assessing the 17-control baseline and preserving genuine exception results.

  11. devantler commented on Oct 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer (interactive Claude Code session)

    The fresh scan this verification waits for has not happened. The first scheduled scan on the new scanner (16 October, 09:12 UTC) aborted before scanning. The release added Sigstore verification of its rule bundle, and the namespace's egress policy blocks the trust-root host that verification needs. The scheduler Job still reported Completed. The stored posture objects therefore still describe the last pre-upgrade scan, and nothing here can be assessed yet.

    That cause is #4966, with the one-host egress fix in #4967. This issue is now natively blocked by #4966. Once #4967 deploys and a scan completes, the 17-control baseline above can be re-read.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Fields

    No fields configured for Security.

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions