Skip to content

Mount the whole regenerated /etc/pki into workshop sessions and widen prune-builds - #1233

Merged
jorgemoralespou merged 2 commits into
educates:developfrom
jorgemoralespou:cherrypick-ca-changes
Oct 14, 2026
Merged

jorgemoralespou merged 2 commits into
educates:developfrom
jorgemoralespou:cherrypick-ca-changes

Conversation

@jorgemoralespou

Copy link
Copy Markdown
Collaborator

Summary

Splits two self-contained commits by @GrahamDumpleton out of #1232 so they
can land now, while the other two commits there (the CA delivery fix for the
CLI and operator, 615bf14, and the start-script tls.crt fallback,
abbbb90) are reworked. Both are cherry-picked unchanged: same patch IDs,
original author kept.

fix(session-manager): mount the whole regenerated /etc/pki into workshop
sessions
(from 9aae3fc)

On Fedora 44, mounting only the regenerated /etc/pki/ca-trust leaves
OpenSSL's default CApath (/etc/pki/tls/certs) as the image's stale copy,
and the /etc/pki/tls/cert.pem CAfile symlink no longer exists. Tools that
read a bundle file (curl, git, Go) trust the ingress CA; anything on
OpenSSL's default verify paths, Fedora's system Python among them, does not.

 ca-trust-store-initialization (setup-certificates)
   update-ca-trust
+  ln -sf /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /etc/pki/tls/cert.pem
-  copy /etc/pki/ca-trust -> workshop-ca-trust volume
+  copy /etc/pki          -> workshop-ca-trust volume   # ca-trust, tls, java, rpm-gpg (4 MB)
 workshop container
-  workshop-ca-trust -> /etc/pki/ca-trust (readOnly)
+  workshop-ca-trust -> /etc/pki          (read-write, copied files root-owned)
 downloads init container
-  workshop-ca-trust -> /etc/pki/ca-trust (readOnly)
+  workshop-ca-trust -> /etc/pki          (readOnly)

The init container only runs when session-manager has an ingress CA Secret
(INGRESS_CA_SECRET), so this takes effect where that is already set, for
example Inline mode with caCertificateSecretRef. The Managed-mode custom
CA path only reaches it once the reworked CA delivery fix from #1232 lands.

build(make): have prune-builds remove every untracked build artifact
(from c935f87)

 prune-builds
+  rm -rf workshop-images/base-environment/client-programs
+  rm -rf {lookup-service,secrets-manager,session-manager,tunnel-manager}/venv
+  make -C installer/operator clean-local-charts   # new target, driven by LOCAL_SUBCHARTS

clean-local-charts removes only the packaged runtime subchart tarballs;
the upstream vendored tarballs recorded in SHA256SUMS and scratch/ are
untouched. The target is documented in
developer-docs/build-instructions.md.

Evidence

  • Before: in a workshop session on a kind cluster with a self-signed
    CA, Python ssl and requests against the training portal URL fail
    verification while curl in the same terminal succeeds.
    After: Python's default context verifies with the CAfile and with the
    CApath alone; requests, curl, git, helm and kubectl all verify. (Graham's
    verification in Deliver the custom ingress CA to platform components and workshop sessions #1232.)
  • For this split: both commits apply cleanly to develop (9526449) with
    patch IDs identical to the originals; workshopsession.py compiles,
    setup-certificates passes bash -n, and make -n prune-builds expands
    clean-local-charts to the five LOCAL_SUBCHARTS tarballs only.

Merge Danger

Door: two-way

Both commits revert cleanly. Nothing persistent changes: the trust store
volume is rebuilt for every session.

Blast Radius: workshop-sessions

Every workshop session with an ingress CA now gets all of /etc/pki from
the init container rather than from its own image, and the workshop
container's mount is no longer read-only. The init container runs the base
environment image, not the workshop's image, so a custom workshop image that
adds files under /etc/pki outside ca-trust (for example a repository
signing key in /etc/pki/rpm-gpg referenced by a .repo file) no longer
sees them when an ingress CA is configured. Before, only ca-trust was
replaced this way. prune-builds only affects local builds.

9aae3fc also needs a backport to support/3.8.x, which moved to Fedora 44
as well; as noted in #1232, only this commit applies there and it will need
a release-notes entry on that line. No release-notes entry here, as the
defect has not shipped in a tagged 4.0 release.

…hop sessions

The ca-trust-store-initialization init container ran update-ca-trust and
copied only /etc/pki/ca-trust into the volume the workshop container
mounted at that path. On Fedora 44 that is no longer enough. OpenSSL's
default CApath, /etc/pki/tls/certs, is a real directory of hashed symlinks
that update-ca-trust also regenerates, and Fedora 44 dropped the
/etc/pki/tls/cert.pem symlink that is OpenSSL's default CAfile. Both live
outside the mounted directory, so the image's stale copies were used:
consumers that read a bundle file (curl, git, Go) saw the ingress CA, while
anything relying on the default verify paths, Fedora's system Python among
them, did not.

Have setup-certificates recreate the cert.pem symlink and copy all of
/etc/pki (ca-trust, tls, java and rpm-gpg, 4 MB) into the volume, and
mount the volume at /etc/pki in the workshop container and the downloads
init container. The hashed CApath links, the CAfile symlink, the Java
keystore link and the trust store then all come from the same regenerated
copy. The workshop container mount is read-write so a workshop running as
root can still import package signing keys under rpm-gpg; the copied files
are root-owned, so the workshop user cannot alter the trust store either
way.

Verified in a workshop session on a kind cluster with a self-signed CA:
Python's default SSL context verifies the ingress with both the CAfile and
the CApath, as do requests, curl, git, helm and kubectl with system roots.
prune-builds missed several things that build targets leave in the source
tree: the copy of the CLI source staged into the base environment image
context, the packaged runtime subchart tarballs in the operator's
vendored-charts directory, and the Python virtual environments of the
lookup service, secrets manager, session manager and tunnel manager. All of
these are gitignored on this branch, but the ignore rules do not exist on
the older support branches, so after a build they show up there as hundreds
of untracked files when switching branches.

Add them to prune-builds, with the tarballs removed through a new
clean-local-charts target in the operator Makefile that reuses the
LOCAL_SUBCHARTS list package-local-charts is driven by, so the two cannot
drift apart. The upstream vendored tarballs recorded in SHA256SUMS are not
touched, and nor is the scratch directory. Document the target and why it
is worth running before switching to a support branch.
@jorgemoralespou
jorgemoralespou merged commit 01be0e1 into educates:develop Oct 14, 2026
1 check passed
@jorgemoralespou
jorgemoralespou deleted the cherrypick-ca-changes branch October 14, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants