feat: airgaps - #105
Draft
butler54 wants to merge 82 commits into
Draft
Conversation
oc-mirror v2 config targeting quay.apac-tech-lab.net/mirror: - OCP 4.21.24 single release - Full baremetal operator set (OSC 1.13, Trustee 1.2, ACM, ESO, GitOps, cert-manager, Kyverno, NFD, LVM, CNV, GPU, Intel) - Application images (ubi9/httpd-24, vault) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…iguration - Add 7 VP Helm OCI charts as additionalImages (clustergroup, acm, vault, eso, trustee, sandbox, policies) - Add Kyverno chart OCI artifact (ghcr.io/kyverno/charts/kyverno:3.7.2) - Add VP utility + imperative container images - Add oc-debug support-tools image (D-05) - Add ose-cli image for wait-for-vault job - Add Intel DCAP images (tag-based + digest-pinned from chart templates) - Add ubi9 workload images - Replace commented chart list with live additionalImages entries
- values-global: SSH git URL, helmRepoUrl, tokenSecret, CatalogSource sources - values-global: clusterGroupName set to baremetal, patternsOperator + gitops sources - values-baremetal: source overrides on all 10 subscriptions (8 redhat, 2 certified) - values-baremetal: Kyverno repoURL changed to OCI quay mirror path - Implements DPAT-01,03,04,06,14
- bootstrap_secrets v2.0 format for ArgoCD repo credential - Creates private-repo Secret with SSH key (~/.ssh/id_rsa) + git URL - URL matches values-global.yaml repoURL exactly for ArgoCD prefix matching - Implements DPAT-02
The patterns-operator's getUserFromURL() only supports SCP-style URLs (user@host:path), not ssh:// protocol URLs. Also use IPv4 address since cluster DNS resolves hostname to unreachable IPv6. - values-global.yaml: chbutler@172.25.36.135:/home/chbutler/coco-pattern - values-secret.yaml.template: same URL in bootstrap_secrets Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
patterns-operator webhook rejects SSH URLs unless they start with git@, and creating a git user requires sudo (unavailable on jump host). Switch to dumb HTTP via Apache UserDir (~/public_html/git/). - values-global.yaml: http://172.25.36.135/~chbutler/git/coco-pattern.git - values-secret.yaml.template: same URL in bootstrap_secrets Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
patterns-operator is not in redhat-operator-index for OCP 4.21. It's in the community-operator-index. Add it to the mirroring config so oc-mirror handles it and the operator installs via OLM normally. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- scripts/airgap-post-install.sh: idempotent bootstrap for disconnected clusters (disable default catalogs, mirror OCI Helm charts, add CA certs to ArgoCD, create bare git repos for HTTP serving, optional routingViaHost for test-lab networks) - scripts/git-http-server.py: smart HTTP fallback server if Apache dumb HTTP doesn't work with go-git - Makefile: airgap-post-install and airgap-sync-repos targets - values-global.yaml: remove tokenSecret (HTTP repos need no auth) - values-secret.yaml.template: remove bootstrap_secrets section Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Generic target wrapping oc-mirror with configurable MIRROR_REGISTRY, AUTHFILE, and IMAGESET_CONFIG variables. Workflow: make airgap-mirror MIRROR_REGISTRY=quay.example.com:443/mirror make airgap-post-install make install Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
oc-mirror reads auth from containers/auth.json or docker config.json automatically. Remove the -a flag (not supported by oc-mirror). Users should podman login or copy pull-secret.json to the right path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
skopeo is only needed for listing pattern-install tags. Make it optional — the script warns and skips that step if skopeo is missing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
patterns-operator is in community-operator-index, not redhat-operator-index. The previous value caused the subscription to reference a non-existent CatalogSource on the disconnected cluster. Added comments clarifying these are mirrored CatalogSource names from labctl apply-mirror-resources. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major rewrite of the disconnected deployment bootstrap: Layer B (general disconnected OpenShift): - Create all 3 mirrored CatalogSources (redhat, certified, community) - Create ITMS for tag-based image pulls (ubi-minimal, VP images) - Fix manifest list failures from oc-mirror (skopeo fallback) - Auto-extract CA from cluster trust bundle for ArgoCD Layer C (validated patterns disconnected): - Deploy Pattern CR directly via oc apply (bypass pattern.sh) - Create patterns-operator-config ConfigMap (override gitops channel) - Add gitops.channel: latest to values-global.yaml - Add patternsOperator.channel: fast to values-global.yaml - Smart HTTP git server setup (go-git can't handle Apache dumb HTTP) imageset-config.yaml: - Add imperative-container:v1 (VP uses :v1, not :latest) - Add pattern-ui-catalog:stable-v1 (operator console plugin) - Add ubi-minimal:latest explicitly Makefile: - airgap-deploy-pattern: deploy Pattern CR directly - airgap-fix-manifests: fix oc-mirror manifest list failures Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add step 7b: configure_argocd_helm_auth - Extracts registry credentials from cluster pull-secret - Creates helm-registry-config secret in ArgoCD namespace - Patches ArgoCD CR to mount HELM_REGISTRY_CONFIG - Runs automatically during --deploy-pattern mode This fixes the 401 errors when ArgoCD tries to pull OCI Helm charts from private Quay repositories. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ArgoCD's OCI registry credential system is broken (bugs #25513, #26311). The ArgoCD operator reverts any custom env/volume patches. Making Quay repos public requires admin access we don't have. Solution: extract charts from OCI mirror, serve as traditional Helm HTTP repo via Apache UserDir. No auth needed, no OCI issues. - helmRepoUrl changed to http://jump-host/~user/helm-charts - Charts extracted via helm pull from OCI, index.yaml generated Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Quay Helm chart repos made public — ArgoCD can pull OCI charts without auth. Reverts the HTTP Helm repo workaround. helmRepoUrl: quay.apac-tech-lab.net:443/mirror/validatedpatterns Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Covers the full workflow: mirror → install OCP → bootstrap → deploy pattern → load secrets. Documents all known issues, workarounds, and Quay requirements. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The clustergroup chart checks global.singleArgoCD, not main.singleArgoCD. Move it to global: so the chart reads it correctly and only creates one ArgoCD instance. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The VP Helm chart uses registry.connect.redhat.com/hashicorp/vault (Red Hat certified UBI build), not docker.io/hashicorp/vault. Fix the imageset-config and add ITMS entry for registry.connect.redhat.com/hashicorp. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extracts ArgoCD admin credentials from the cluster and logs in with the argocd CLI. Falls back to printing credentials + install instructions if the CLI isn't available. make argocd-login Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extracts the argocd binary from the running ArgoCD server image on the cluster — works in disconnected environments without internet access to GitHub releases. make argocd-install # installs to ~/.local/bin/ make argocd-install ARGOCD_CLI_DIR=~/bin # custom path Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove 2>/dev/null so KUBECONFIG and API errors are visible. Add explicit error messages for missing ArgoCD instance or deployment. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OpenShift GitOps prefixes deployment names with the instance name (e.g. vp-gitops-server), so app.kubernetes.io/name doesn't match a static 'argocd-server'. Use component=server + part-of=argocd. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The reencrypt route handles TLS termination; let the CLI negotiate the connection without forcing gRPC-web transport. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OpenShift reencrypt routes don't pass gRPC traffic. Use --core mode which talks directly to the Kubernetes API instead of the ArgoCD server, avoiding all route/TLS/gRPC issues. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Kyverno chart was under mirror/kyverno/charts which has nested path issues with Quay repo visibility. Re-mirrored under mirror/validatedpatterns/kyverno (same public namespace as VP charts). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Confidential container workloads pull images inside the kata guest VM, which has no access to IDMS mirror mappings. The image references must point directly to the mirror registry. Parameterized image refs in hello-openshift, kbs-access-curl, and kbs-access-sealed charts via values.yaml. Baremetal topology overrides set them to the Quay mirror path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire the full pipeline: values-secret -> Vault -> ESO -> Ansible -> initdata. The init-data-gzipper reads a registry-ca Secret from the imperative namespace, splits multi-cert PEM chains into individual certs, and renders them as extra_root_certificates in cdh.toml. When no CA secret exists, behavior is unchanged (backward compatible with connected clusters). Also adds cache-registry-ca Makefile target and enables kbs.registryCa in the baremetal topology. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the broken yq|jq pipeline with Python3+yaml that auto-detects both old format (reference-values.json array) and new format (reference_value dict with base64-encoded RVPS entries). Fixes extraction for coco-tools 0.5.1 / Trustee 1.2. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ACM's MCH hardcodes source: redhat-operators for the MCE child subscription. In disconnected environments the default CatalogSource is disabled and replaced with cs-redhat-operator-index-v4-21. Without this override, MCH stays stuck in Installing forever because it can never find the MCE operator. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ACM creates an MCE subscription as a dependency. OLM needs the multicluster-engine package in the mirrored catalog to resolve it. Without this, MCH stays stuck in Installing with "no operators found in package multicluster-engine". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Manual-approval subscriptions (sandbox, trustee, GPU) block the parent app sync at wave 0, preventing vault init and SA creation. Moving them to wave 10 lets vault initialize at wave 5 first. Sandbox and trustee already had wave 10. GPU was missing it. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
VP operator images use hybrid OCI/Docker manifests (OCI index parent with Docker v2 children) that Quay 3.17 rejects with MANIFEST_INVALID. Workaround: extract amd64 single-arch manifest and create IDMS mapping the original parent index digest to the amd64 child digest. Upstream bug: validatedpatterns/patterns-operator#778 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
osc-pccs-rhel9:latest and osc-tdx-qgs-rhel9:latest do not exist on registry.redhat.io. Only digest-pinned refs work. Removing the tag entries eliminates 2 oc-mirror errors per run. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Quay has ANONYMOUS_ACCESS: true — Helm chart repos are public. Mounting the pull-secret as HELM_REGISTRY_CONFIG causes ArgoCD to send Basic auth that Quay rejects (expects Bearer tokens). Unauthenticated pulls work correctly. Also: mirror-registry step 4b not needed — all images go to Quay. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Intel certified operator controller images (intel-deviceplugin-operator, intel-tdx-dcap-operator) fail oc-mirror due to missing cosign signatures. Add them to the fixup list alongside VP operator images. Also consolidate the IDMS to cover all certified operator sources (VP, Intel, NVIDIA, Hashicorp) in a single idms-certified-operators resource. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NVIDIA bundle image fails oc-mirror (signature lookup). GPU operator subscription at wave 10 blocks trustee (wave 20) and workloads (wave 30). Disabled: gpu-operator subscription, nvidia-gpu app, gpu-workload app. Re-enable when NVIDIA bundle is mirrored or upstream fix lands. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
oc-mirror generates idms-oc-mirror.yaml and itms-oc-mirror.yaml with operator and release image mirror mappings, but they were never applied to the cluster. Without them, tag-based operator image pulls (NFD, etc.) can't resolve to the Quay mirror. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
oc-mirror fails to mirror 10 additional images from certified operator catalogs because cosign signature lookup fails. Add all of them to the fixup mirror list so they're pushed to Quay by airgap-post-install.sh. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Intel TDX DCAP operator creates QGS pods requiring privileged access (hostPath, runAsUser 0) but its CSV doesn't declare SCC requirements. Add ClusterRoleBinding granting privileged SCC to the default SA in openshift-operators namespace. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
QGS DaemonSet uses serviceAccount intel-tdx-dcap, not default. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Uses pcsclient.py from the Intel PCS Client Tool repo (intel/confidential-computing.tee.dcap) instead of pck-cert-tool extracted from the QGS container (which no longer contains it). Usage: make pck-register INTEL_PCS_API_KEY=<key> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All operator subscriptions now at sync-wave 10 so wave 0 only contains infrastructure (namespaces, SA, CRBs, CronJobs, AppProjects). This lets the vault hook at wave 5 run after the imperative SA is created, before any subscription health check can block the parent sync. Testing whether this eliminates the vault init deadlock. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ESO and LVM are vault infrastructure dependencies: - LVM provides storage for vault PVC - ESO delivers secrets from vault to workload namespaces Moving them to wave 10 blocked vault initialization (PVC unbound). Only non-infrastructure subscriptions should be at wave 10. Wave 0: ESO, LVM (vault dependencies) Wave 10: ACM, sandbox, trustee, cert-manager, CNV, NFD, Intel, GPU Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…lt PVC CNV (kubevirt-hyperconverged) deploys the hostpath provisioner used for vault storage on SNO. Must be at wave 0 with ESO and LVM. Wave 0: ESO, LVM, CNV (vault infrastructure) Wave 10: ACM, sandbox, trustee, cert-manager, NFD, Intel, GPU Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The vault hook must wait for the full chain: - CNV operator install (~5 min) → hostpath provisioner - StorageClass creation + PVC bind (~2 min) - Vault pod startup (~2 min) - unsealvault-cronjob init + unseal (up to 10 min) - 120s post-unseal sleep in hook script 600s (10 min) was too short. 1800s (30 min) provides adequate margin. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
values-baremetal.yaml set serviceAccountName: imperative-admin-sa, which collides with the chart's adminServiceAccountName default (also imperative-admin-sa). This caused ArgoCD RepeatedResourceWarning. Remove the override — let the chart use its defaults: - imperative-sa (serviceAccountCreate) - imperative-admin-sa (adminServiceAccountCreate) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The imperative cronjob (init-data-gzipper, unsealvault) uses imperative-sa which defaults to read-only (get/list/watch). These jobs need to create/update ConfigMaps, secrets, and vault resources. Override clusterRoleYaml to grant full admin access to imperative-sa, matching what imperative-admin-sa already has. This avoids the duplicate SA issue from overriding serviceAccountName to imperative-admin-sa. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OLM pulls patterns-operator bundle by tag from quay.io/community-operator-pipeline-prod. Without an ITMS entry, the tag-based pull goes to the upstream (blackholed) registry. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…d to Quay Step 6b now has two separate image lists: - VP hybrid manifest images → mirror-registry (--keep-manifest-list=true, preserves parent OCI index digest) - Certified operator images → Quay (single-arch, signature failures only) Also creates separate IDMS for each registry: - idms-mirror-registry: VP operator → 172.25.36.135:8443 - idms-certified-operators: Intel/NVIDIA/Hashicorp → Quay No manual steps 9-10 needed — all handled by the script. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
kata-cc emptyDir volume ownership doesn't apply fsGroup correctly, causing the init container's shell redirect to fail with permission denied when writing to /var/www/html/secret.txt. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
kata-cc with restricted-v2 SCC prevents direct writes to the emptyDir mount. Write to /tmp first then copy. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tibility kata-cc with restricted-v2 SCC doesn't apply fsGroup to emptyDir mounts, making /var/www/html unwritable. Mount at /tmp/shared instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ptyDir writes kata-cc + restricted-v2 SCC prevents writes to emptyDir mounts. Add a dedicated ServiceAccount with privileged SCC binding so the pod runs as root and can write to the shared volume. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three-phase workflow separated by connectivity requirement: - make snp-collect-vcek-urls (cluster-connected, no internet) - make snp-download-vcek (internet-connected, no cluster) - make snp-gen-overrides (local, generates chart values + secret entries) Also updates values-hw-amd-snp.yaml to enable SNP attestation and disable intel-tdx-dcap subscription. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.