fix: Report certificate expiry for the certManager backend - #752
Merged
Conversation
The certManager backend ignored any expiry information from the certiifcates it deployed. This meant that a Pod with a cert we injected would never be restarted. This commit changes it so it now reports the expiry of certManager created certs to the restart-controller, so it evicts/restarts the Pod in time.
lfrancke
marked this pull request as ready for review
September 9, 2026 09:24
Maleware
self-requested a review
September 9, 2026 12:36
lfrancke
commented
Sep 9, 2026
lfrancke
commented
Sep 9, 2026
lfrancke
commented
Sep 9, 2026
lfrancke
commented
Sep 9, 2026
Maleware
requested changes
Sep 9, 2026
Maleware
left a comment
Maleware
left a comment
Member
There was a problem hiding this comment.
Overall it looks good to me. One thing IMO we have to correct.
Maleware
moved this from Development: Waiting for Review
to Development: In Review
in Stackable Engineering
Maleware
requested changes
Sep 10, 2026
Maleware
left a comment
Maleware
left a comment
Member
There was a problem hiding this comment.
Sorry, I needed to say that.
The overdue-renewal branch fell back to notAfter without checking that notAfter is still in the future. When cert-manager has stopped renewing, the certificate is already expired and that writes a past timestamp into the restarter annotation. commons-operator evicts unconditionally on a past timestamp with no backoff, the replacement Pod is handed the same expired certificate, computes the same past timestamp, and is evicted again. Treat an expired certificate as unusable material instead, which is how this backend already treats a Secret with no tls.crt at all: fail with Unavailable so the kubelet retries, leaving the Pod in ContainerCreating with the reason in an event, and recovering on its own if cert-manager catches up. That means a Pod will not start rather than starting with an expired certificate, which is a behaviour change during a cert-manager outage. With this in place the remaining fallback to notAfter is provably in the future, so it cannot loop.
The renewal point falls back to two thirds through the certificate's validity when the Certificate carries no status.renewalTime. That is expected between our own apply and cert-manager's next reconcile, but it would also happen silently and permanently if our CertificateStatus mirror stopped matching cert-manager's actual status shape, in which case every Pod would be scheduled off a guess with no signal that anything was wrong. Logged at info rather than warn: on a first publish this is normal and not actionable, and the operator already has a history of expected conditions logging as warnings.
lfrancke
moved this from Development: In Review
to Development: Done
in Stackable Engineering
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The certManager backend ignored any expiry information from the certiifcates it deployed. This meant that a Pod with a cert we injected would never be restarted.
This commit changes it so it now reports the expiry of certManager created certs to the restart-controller, so it evicts/restarts the Pod in time.
Definition of Done Checklist
Author
Reviewer
Acceptance
type/deprecationlabel & add to the deprecation scheduletype/experimentallabel & add to the experimental features tracker