A non-empty wildcard Secret name disables ACME for every tenant but only certifies one ingress controller — child tenants silently serve a fake certificate

[lexfrei]

What happens

A non-empty operator wildcard Secret name disables per-host ACME issuance for every tenant, but supplies the replacement certificate to only one ingress controller — the publishing one. Any child tenant running its own ingress controller (tenant.spec.ingress: true) is left with applications that have no certificate of their own and a controller with no default certificate, so ingress-nginx serves its built-in self-signed fake certificate for every host in that tenant.

There is no error, no fail(), no status condition. The platform reconciles green. The operator finds out from a browser warning.

Two triggers, not one

The trigger is a non-empty _cluster.wildcard-secret-name, and two separate platform keys set it (packages/core/platform/templates/apps.yaml):

• publishing.certificates.wildcardSecretName — an operator-supplied Secret name, passed through directly.
• publishing.certificates.wildcard: true — with solver: dns01 and gateway.enabled: false, the platform sets the same channel value to cozystack-wildcard-tls.

{{- $byoWildcard := .Values.publishing.certificates.wildcardSecretName | default "" }}
{{- $wildcardSecretName := $byoWildcard }}
{{- if and (not $byoWildcard) (eq .Values.publishing.certificates.solver "dns01") .Values.publishing.certificates.wildcard (not .Values.gateway.enabled) }}
{{- $wildcardSecretName = "cozystack-wildcard-tls" }}
{{- end }}
wildcard-secret-name: {{ $wildcardSecretName | quote }}

Everything downstream keys off the resulting value being non-empty, not off which knob produced it, so both keys hit the same trap. The rest of this issue is written in terms of wildcardSecretName, but every statement holds identically for wildcard: true.

Mechanism

Three links, all on main and identically on release-1.5:

1. packages/core/platform/templates/apps.yaml renders wildcard-secret-name into the cluster values channel, so every tenant inherits the value.

2. Every app and system ingress template — dashboard, keycloak, harbor, bucket, linstor-gui — drops the ACME annotation as soon as that value is non-empty. The guard is global, with no namespace condition:

{{- /* Operator wildcard mode: nginx serves the default SSL cert, so skip per-host ACME. */}}
{{- if not $wildcardSecret }}
cert-manager.io/cluster-issuer: {{ $clusterIssuer }}

3. packages/extra/ingress/templates/nginx-ingress.yaml sets default-ssl-certificate only for the publishing controller:

{{- if and $wildcardSecret (eq .Release.Namespace $exposeIngress) }}
extraArgs:
  default-ssl-certificate: "{{ $exposeIngress }}/{{ $wildcardSecret }}"

The skip in (2) is cluster-wide; the replacement in (3) is namespace-local. A child tenant with its own controller falls into the gap: it gets neither.

The adjacent comment in wildcard-certificate.yaml already states the constraint — "child-tenant controllers cannot point --default-ssl-certificate at another namespace" — so the limitation is known. What is missing is that the ACME skip was never gated on the same condition.

Replicating the Secret into the child namespace does not help: that tenant's chart never renders default-ssl-certificate at all, because of the namespace gate in (3).

Reproduction

1. On a cluster with gateway.enabled: false (the default ingress path), create a child tenant with ingress: true, so it runs its own ingress controller, and expose an app in it.
2. Confirm the app serves a valid ACME certificate.
3. Set publishing.certificates.wildcardSecretName on the platform Package to a pre-existing wildcard Secret in the publishing namespace.
4. The child tenant's app now serves the ingress-nginx fake certificate. Its Ingress has lost the cert-manager.io/cluster-issuer annotation, its per-host Certificate is gone, and the child's controller has no default-ssl-certificate argument.

Affected versions

main and release-1.5 carry the same shape.

Possible directions

Either gate the ACME skip on the same condition as the replacement, so only the publishing namespace drops per-host issuance and child controllers keep issuing their own certificates; or render default-ssl-certificate for every tenant controller and give each one a certificate it can actually read from its own namespace.

Whichever way it goes, the failure should not be silent: a tenant that runs its own ingress controller while the platform has told it to stop issuing certificates has no valid TLS, and that state is detectable at render time.

[dosubot]

This is a well-documented issue. The behavior you've described is confirmed in the code and is already tracked as Issue #2820 ("Propagate Operator Wildcard to Per-Tenant Termination Points") [1].

The root cause is exactly as you've outlined — the ACME annotation skip is cluster-wide while the default-ssl-certificate replacement is gated on eq .Release.Namespace $exposeIngress [2]. The code comments explicitly acknowledge the constraint:

"child-tenant controllers cannot point --default-ssl-certificate at another namespace" [3]

Current state of the fix:

• ✅ Operator-supplied wildcard for the root tenant landed (PR feat(platform): operator-provided wildcard certificate for platform and tenant ingress #2819)
• ✅ ACME wildcard issuance on the default ingress-nginx path landed (PR feat(platform): ACME wildcard certificate on the default ingress-nginx path #2988, opt-in)
• ❌ Per-tenant propagation is still open — PR feat(platform, ingress): propagate operator wildcard certificate to per-tenant termination points #2990 addresses this with a cross-namespace Secret replication controller that would synchronize the wildcard Secret into child tenant namespaces [4]

Your point about the silent failure mode is valid and not addressed by the existing tracking issue — there's no render-time validation or status condition surfacing the gap. That might be worth calling out explicitly on #2820 or PR #2990 as an additional acceptance criterion.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.