Skip to content

[release-5.0] OCPBUGS-120715: Add Degraded=True exception for authentication operator during upgrade - #31608

Open
openshift-cherrypick-robot wants to merge 4 commits into
openshift:release-5.0from
openshift-cherrypick-robot:cherry-pick-31535-to-release-5.0
Open

[release-5.0] OCPBUGS-120715: Add Degraded=True exception for authentication operator during upgrade#31608
openshift-cherrypick-robot wants to merge 4 commits into
openshift:release-5.0from
openshift-cherrypick-robot:cherry-pick-31535-to-release-5.0

Conversation

@openshift-cherrypick-robot

Copy link
Copy Markdown

This is an automated cherry-pick of #31535

/assign pskrbasu

…or during upgrade

The authentication operator transiently goes Degraded for ~8.5 seconds during
upgrade rollout when oauth-apiserver and oauth-server pods are briefly
unavailable. This is the same pattern already excepted for kube-apiserver
(OCPBUGS-38661), kube-controller-manager (OCPBUGS-38662), and kube-scheduler
(OCPBUGS-38663).

Add a narrow exception scoped to UnavailablePod reasons only, so transient pod
rollout states are classified as flakes rather than hard failures. This
unblocks the OKD SCOS 5.0 promoted upgrade job which has been failing
consistently since ec.4 (OKD-424).
… upgrade

Verifies that the authentication operator's transient Degraded=True state
during upgrade is correctly excepted for UnavailablePod reasons, while
unrelated Degraded reasons remain hard failures.
- Use strings.Contains for reason matching instead of exact matches,
  preserving the original breadth and avoiding regressions on two-node
  clusters
- Return a descriptive message with bug ID instead of a bare URL
- Update tests to match the new strings.Contains behavior
…oNode guard

Address review feedback from hongkailiu and Prashanth684:
- Restore the original isTwoNode guard for DualReplica clusters
- Add separate isOKD guard for OKD SCOS clusters only, keeping OCP HA
  uncovered so regressions are detected
- Return full bug URL for easy copy-paste from job logs
- Add isOKDCluster() helper following the existing isVSphere() pattern
- Mark the OKD exception as temporary until the auth operator is fixed
- Update tests to cover DualReplica and HA non-OKD paths separately
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Sep 4, 2026
@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 5f8eed2a-0678-44c6-99a9-63a8832b6f50

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from deads2k and sjenning September 4, 2026 04:57
@openshift-ci-robot

Copy link
Copy Markdown

@openshift-cherrypick-robot: Jira Issue OCPBUGS-111997 has been cloned as Jira Issue OCPBUGS-120715. Will retitle bug to link to clone.
/retitle [release-5.0] OCPBUGS-120715: Add Degraded=True exception for authentication operator during upgrade

Details

In response to this:

This is an automated cherry-pick of #31535

/assign pskrbasu

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot changed the title [release-5.0] OCPBUGS-111997: Add Degraded=True exception for authentication operator during upgrade [release-5.0] OCPBUGS-120715: Add Degraded=True exception for authentication operator during upgrade Sep 4, 2026
@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Sep 4, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-120715, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This is an automated cherry-pick of #31535

/assign pskrbasu

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@Prashanth684 Prashanth684 added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Sep 4, 2026
@openshift-ci

openshift-ci Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: openshift-cherrypick-robot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@pskrbasu

pskrbasu commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

/verified

@openshift-ci-robot

Copy link
Copy Markdown

@pskrbasu: The /verified command must be used with one of the following actions: by, later, remove, or bypass. See https://docs.ci.openshift.org/docs/architecture/jira/#premerge-verification for more information.

Details

In response to this:

/verified

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor

/override-sticky ci/prow/e2e-vsphere-ovn-upi

Automated triage: This failure appears unrelated to the PR changes.

Job classification: Eligible long-running presubmit e2e/integration job: pull-ci-openshift-origin-release-5.0-e2e-vsphere-ovn-upi, using the openshift-e2e-vsphere-upi workflow, vsphere-elastic cluster profile, UPI vSphere provisioning, and the common OpenShift conformance suite via openshift-e2e-test.

Revision check: Incoming/run SHA f393c48ac4dbceed33e164c7d289588e3966ee03; current PR HEAD f393c48ac4dbceed33e164c7d289588e3966ee03; match. The build log independently records source resolution merging PR #31608 at this SHA. The Prow metadata endpoint was unavailable for a separate prowjob.json check.

Execution status: Tests executed. The pre phase succeeded, the e2e test ran for 2h4m33s, and the suite reported 2336 pass, 1 blocking fail, 2 informing fail, 1 flaky, 2217 skip. The blocking failure was CRDValidationRatcheting ... oldSelf = nil ... optionalOldSelf is true; its cleanup failed deleting a CRD with context deadline exceeded, with a nearby ConnectionErrorDuringSingleNodeAPIServerTargetDown event.

Completed supporting jobs: ci/prow/e2e-vsphere-ovn passed on the same SHA. Standard checks including unit, lint, verify, verify-deps, go-verify-deps, images, and okd-scos-images passed. Pending: tide. ci/prow/e2e-metal-ipi-ovn-ipv6 also failed and is not used as positive signal.

Overlap assessment: The PR changes only CVO legacy monitor logic and its unit tests: an OKD authentication-operator Degraded exception during pod rollout and tests for that behavior. The failed test exercises API-server CRD validation/ratching behavior and has no direct or indirect overlap with the changed CVO/authentication monitor code.

Missing-coverage risk: Low. This job executed the conformance suite; the only blocking failure is the isolated CRD validation test. Test history records prior failures for this exact test and no open bug, while this run shows a CRD cleanup timeout coincident with an API-server-down event.

Rationale: The failure is a known, previously failing API-server/cleanup test signal and is unrelated to the PR's CVO authentication-monitor changes. The override is therefore appropriate for this completed long-running e2e run.

If you disagree with this assessment, rerun the current job with /test e2e-vsphere-ovn-upi.


AI-generated. Review for accuracy.

@openshift-ci

openshift-ci Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: Overrode contexts on behalf of redhat-chai-bot: ci/prow/e2e-vsphere-ovn-upi

These overrides will persist across retests on the current HEAD SHA. Pushing a new commit will clear them. Use /override-cancel to remove them.

Details

In response to this:

/override-sticky ci/prow/e2e-vsphere-ovn-upi

Automated triage: This failure appears unrelated to the PR changes.

Job classification: Eligible long-running presubmit e2e/integration job: pull-ci-openshift-origin-release-5.0-e2e-vsphere-ovn-upi, using the openshift-e2e-vsphere-upi workflow, vsphere-elastic cluster profile, UPI vSphere provisioning, and the common OpenShift conformance suite via openshift-e2e-test.

Revision check: Incoming/run SHA f393c48ac4dbceed33e164c7d289588e3966ee03; current PR HEAD f393c48ac4dbceed33e164c7d289588e3966ee03; match. The build log independently records source resolution merging PR #31608 at this SHA. The Prow metadata endpoint was unavailable for a separate prowjob.json check.

Execution status: Tests executed. The pre phase succeeded, the e2e test ran for 2h4m33s, and the suite reported 2336 pass, 1 blocking fail, 2 informing fail, 1 flaky, 2217 skip. The blocking failure was CRDValidationRatcheting ... oldSelf = nil ... optionalOldSelf is true; its cleanup failed deleting a CRD with context deadline exceeded, with a nearby ConnectionErrorDuringSingleNodeAPIServerTargetDown event.

Completed supporting jobs: ci/prow/e2e-vsphere-ovn passed on the same SHA. Standard checks including unit, lint, verify, verify-deps, go-verify-deps, images, and okd-scos-images passed. Pending: tide. ci/prow/e2e-metal-ipi-ovn-ipv6 also failed and is not used as positive signal.

Overlap assessment: The PR changes only CVO legacy monitor logic and its unit tests: an OKD authentication-operator Degraded exception during pod rollout and tests for that behavior. The failed test exercises API-server CRD validation/ratching behavior and has no direct or indirect overlap with the changed CVO/authentication monitor code.

Missing-coverage risk: Low. This job executed the conformance suite; the only blocking failure is the isolated CRD validation test. Test history records prior failures for this exact test and no open bug, while this run shows a CRD cleanup timeout coincident with an API-server-down event.

Rationale: The failure is a known, previously failing API-server/cleanup test signal and is unrelated to the PR's CVO authentication-monitor changes. The override is therefore appropriate for this completed long-running e2e run.

If you disagree with this assessment, rerun the current job with /test e2e-vsphere-ovn-upi.


AI-generated. Review for accuracy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci

openshift-ci Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

@openshift-cherrypick-robot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-vsphere-ovn-upi f393c48 link true /test e2e-vsphere-ovn-upi
ci/prow/e2e-metal-ipi-ovn-ipv6 f393c48 link true /test e2e-metal-ipi-ovn-ipv6

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants