autoTls: bound certificate lifetime by the CA that will actually sign it, not the configured caCertificateLifetime

[lfrancke]

get_or_create_k8s_certificate clamps maxCertificateLifetime to (caCertificateLifetime - caCertificateRetirementDuration) / 4.
That is good in general, bad when someone brings their own CA as secret operator never looks at the actual CA that is provided. It just looks at what is in the CR:

    ca:
        secret:
          name: secret-provisioner-tls-ca
          namespace: default
        autoGenerate: false
        caCertificateLifetime: 700d
        caCertificateRetirementDuration: 1d

maxCertificateLifetime = (700 - 1) / 4 = 174 (or 175, not sure, doesn't matter here)

That clamps to 174d . But we have no idea what's really in the CA.

If it has 200 days left:

• above 174 days remaining: works
• 174 down to 139: flip-flops, because jitter puts each cert somewhere in 139-174d and only some of them still fit
• below 139: every mount fails

That is not great.
It only happens when autoGenerate is false, but still....

Scope

• Calculate the clamp by looking at the CA in the Secret that expires last
• Keep it a hard failure where no CA covers now + restartBuffer
• Warning Event and/or condition on the Pod when the cert had to be shortened because of the CA's remaining lifetime?
  • Open question....do we do this elsewhere? Easy to do? I think it'd be good.

Note

This next bit is complicated and I'm not 100% sure it is all correct. I think it makes sense but I have a nagging feeling that something is still off. It is 100% written by hand but I'm not sure if the new steps 3 & 4 are correct to be honest. So many edge cases.

When a pod mounts a TLS volume, NodePublishVolume builds the backend for that SecretClass.
For autoTls that is TlsGenerate::get_or_create_k8s_certificate, which does, in this order:

1. safe_max_cert_lifetime = (caCertificateLifetime - caCertificateRetirementDuration) / 4, from the CR fields
2. max_cert_lifetime = min(maxCertificateLifetime, safe_max_cert_lifetime). This is the clamp.
3. ca::Manager::load_or_create, which reads the CA certificates from the Secret

We need to load the CAs (step 3) before computing the clamp (step 2), and additionally bound it (the max cert lifetime) by the remaining lifetime (not_after - now, minus caCertificateRetirementDuration) of the CA that expires last. This way we never create certificates that can outlive the last expiring CA.

1. ca::Manager::load_or_create, reads the CA certificates from the Secret
2. safe_max_cert_lifetime = (caCertificateLifetime - caCertificateRetirementDuration) / 4, from the CR, unchanged
3. ca_bound = not_after(CA that expires last) - now - caCertificateRetirementDuration, from the loaded certificates. This is new.
4. max_cert_lifetime = min(maxCertificateLifetime, safe_max_cert_lifetime, ca_bound)

Note

Do we even still need safe_max_cert_lifetime? I believe it makes sure that all Pods can still talk to each other even if some already have a new CA and others don't but 🤯

Open questions

• Check the existing warnings whether they are correctly calculated and/or whether we want something new
• Event or Pod condition or both? An Event per mount is noisy on a large StatefulSet, otoh we might want noisy in this case?
• I believe it'd be easy to fold Warn that manually provisioned certificates are about to expire #94 into this. If you agree and it adds less than a day you're welcome to do so. Otherwise skip.
  • I understand that this is underspecified but at least a log line and maybe another condition woudl be good here?

Out of scope

• Validating the SecretClass on admission

Expected effort

1-2 days. If it takes longer, stop and flag it.

[soenkeliebau]

I've read your issue and agree with what you have written and proposed.
I think the suggested new order should fix this error.

One question though, do we want to keep this a hard failure? We could shorten the certificate and warn about it somehow ... its basically the same case as with the autoTls backend, the only difference being that with autoTls at some point a new CA might "appear" and resolve the hard failure, when the ca rotation time arrives.

Regarding your question, if we still need safe_max_cert_lifetime - I believe yes, as you say, the objective of that function is to ensure we never deploy pods that are missing a CA they'd need to trust other pods in the same cluster. I have opened #754 because I think our current implementation has edge cases were this can still happen.