From c734f5cf79d8c2fd0f1f0b9054755fb7aea3e0dd Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 3 Sep 2026 21:16:13 +0300 Subject: [PATCH 1/2] docs(external-database-exposure): record why exposure is the native Service The proposal's orchestration story pointed at a layer that reconciled expose entries into ServiceExposure objects. That API group was removed in favour of native Service type: LoadBalancer with loadBalancerClass, on the ground that managed-application charts own their Service, so the document named an integration point the platform does not have. Write the decision down where its alternatives can be weighed once: what the removal settled, the three shapes it turned down and where each was argued, and what follows for a design that wanted an object in between. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- ...sure-is-the-native-loadbalancer-service.md | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 design-proposals/external-database-exposure/decisions/0001-external-exposure-is-the-native-loadbalancer-service.md diff --git a/design-proposals/external-database-exposure/decisions/0001-external-exposure-is-the-native-loadbalancer-service.md b/design-proposals/external-database-exposure/decisions/0001-external-exposure-is-the-native-loadbalancer-service.md new file mode 100644 index 0000000..aeac5c4 --- /dev/null +++ b/design-proposals/external-database-exposure/decisions/0001-external-exposure-is-the-native-loadbalancer-service.md @@ -0,0 +1,37 @@ +# 0001. External exposure is the native LoadBalancer Service, not a Cozystack exposure API + +- **Number:** `0001` +- **Date:** `2026-07-16` +- **Status:** Accepted +- **Deciders:** `@kvaps, @lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** the argument in [`cozystack/cozystack#3164`](https://github.com/cozystack/cozystack/issues/3164), which `cozystack/cozystack#3218` names as its rationale, settled by merging [`cozystack/cozystack#3218`](https://github.com/cozystack/cozystack/pull/3218) +- **Implemented in:** [`cozystack/cozystack#3218`](https://github.com/cozystack/cozystack/pull/3218) + +## Context + +Two tenant-facing exposure surfaces were in play at once, one landed and one still pending. The pending one was the structured `expose` proposal ([`cozystack/community#29`](https://github.com/cozystack/community/pull/29)), open at the time and set to replace the chart-level `external` boolean with an additive `expose` list layered on `ServiceExposure` — pendency that [`cozystack/cozystack#3164`](https://github.com/cozystack/cozystack/issues/3164) gives as the reason to settle the question before the proposal extended the exposure half rather than after. The implementation had landed ahead of it: `network.cozystack.io/v1alpha1`, a cluster-scoped `ExposureClass` and a namespaced `ServiceExposure` reconciled by a controller in `cozystack-controller`, merged in [`cozystack/cozystack#3081`](https://github.com/cozystack/cozystack/pull/3081) on 2026-06-30. + +@lllamnyp filed that issue the next day, and the objection in it is what forced the call; the removal followed two weeks later. The vendor-neutral LoadBalancer selection the new group offered is a native Kubernetes field, `Service.spec.loadBalancerClass`. And two API groups over one domain — `network.cozystack.io` for how an application is reachable, beside `sdn.cozystack.io` for who may reach it — is a fault unless the split is written down, which for the exposure surface it never was, the group having skipped the design process the policy surface went through. + +This proposal was written against that group while it existed. Its section 5 handed both the tenant-facing trigger and the per-release `TLSRoute` to the layer that reconciled `expose` entries into `ServiceExposure` objects. + +## Decision + +External exposure is the native Kubernetes primitive, with no Cozystack object in front of it. `cozystack/cozystack#3218` removed the `network.cozystack.io` group; the host ingress, its only consumer, went on rendering `type: LoadBalancer` and gained an optional `publishing.loadBalancerClass` to pick the LoadBalancer controller, leaving the `externalIPs` node-IP default path unchanged. The ground given in the removal is that managed-application charts own their Service, so native `type: LoadBalancer` plus `loadBalancerClass` and an admin-provisioned address pool cover external exposure without a dedicated API group and a controller to reconcile it. The group never reached users: `docs/changelogs/v1.6.0.md` records it as introduced and removed inside one cycle and not part of v1.6.0. + +## Why not the alternatives + +- **Keep `ServiceExposure` as the object charts point at instead of rendering a Service.** This is the kind `cozystack/cozystack#3164` puts its open question to — what it provides that `Service.spec.loadBalancerClass` plus an admin-provisioned pool per class does not — and `cozystack/cozystack#3218` answers it: managed-application charts already render their own Service, so the indirection served one consumer, the host ingress, with no second one in prospect. +- **Keep `ExposureClass` and remove only `ServiceExposure`.** `cozystack/cozystack#3164` grants that this half is defensible on its own, an admin-owned named config object on the StorageClass analogy, and aims its open question elsewhere. The removal took both because the class had no reader left: every reader of the kind lived inside the `serviceexposure` controller that went with it, `Service.spec.loadBalancerClass` names the LoadBalancer controller directly, and the pool behind a class is provisioned by an administrator rather than by an object. +- **Leave both kinds in place and write the missing justification for two API groups.** `cozystack/cozystack#3164`'s second concern is that two tenant-facing networking groups over one domain is a fault unless the split is written down, and that the exposure surface skipped the design process the policy surface went through. A justification can defend a split that buys something; what was left after the first bullet did not. + +## Consequences + +- Nothing mediates between a chart and its external endpoint, so a design that wants an object in between has to earn it rather than assume it. This proposal's section 5 does not ask for one: the per-release `TLSRoute` is rendered by the release's own chart, the shape `packages/apps/harbor/templates/httproute.yaml` uses for an `HTTPRoute` and `packages/system/cozystack-api/templates/api-tlsroute.yaml` for a `TLSRoute`, and the shared engine listener is a `TenantGateway` field specified in [`cozystack/cozystack#3342`](https://github.com/cozystack/cozystack/pull/3342). That split is derived from this decision in the proposal, not argued in `cozystack/cozystack#3218` — the removal thread does not discuss Gateway routing at all. +- The class stops short of the databases. `publishing.loadBalancerClass` reaches charts as `_cluster.load-balancer-class`, and `packages/extra/ingress/templates/nginx-ingress.yaml` is what reads it, so a managed database's LoadBalancer Service carries no class and lands on the cluster's default LoadBalancer implementation. +- The migration cost fell only on clusters tracking `main` that had set `publishing.exposureClass`: provision an address pool, switch to `publishing.loadBalancerClass`, delete any orphaned `cozystack-` pool by hand. No released cluster was affected. + +## Revisit if + +A successor to the structured `expose` model lands and needs an object of its own. The question to answer then is what that object does beyond selecting a class and naming a pool — which is what `cozystack/cozystack#3164` asked, and what the removed shape had no answer for. From a7c0c02750b040fc802e4bce57e67d83c31da940 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 3 Sep 2026 21:16:13 +0300 Subject: [PATCH 2/2] docs(external-database-exposure): orchestrate exposure on the objects that exist Section 5 handed the trigger and the per-release route to a layer that reconciled expose entries into ServiceExposure objects. Reground it on what the tenant Gateway offers: one tls- passthrough listener per tlsPassthroughListeners entry, nothing routed until a TLSRoute attaches by sectionName and names a backend, and the release's own chart rendering that route from values it already reads. Native LoadBalancer Services remain the path for the engines that cannot be SNI-routed. Claims the same reading falsifies go with it: the listeners declare no route kinds rather than restricting them to TLSRoute, declaring an entry withdraws termination from every name its SNI covers, route attachment is confined to the Gateway's own namespace, the listener cap fails the whole render rather than one listener, and the shared listener does not disappear with an engine's last release. The flat-hostname phase gains the second gate that rule puts on it, in each place that describes the phase. Who declares the per-engine entry, the app chart or the tenant chart, is stated as an open question with the cost on each side rather than answered. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- .../external-database-exposure/README.md | 90 +++++++++++-------- ...sure-is-the-native-loadbalancer-service.md | 6 +- 2 files changed, 58 insertions(+), 38 deletions(-) diff --git a/design-proposals/external-database-exposure/README.md b/design-proposals/external-database-exposure/README.md index 3beeb0f..c39d828 100644 --- a/design-proposals/external-database-exposure/README.md +++ b/design-proposals/external-database-exposure/README.md @@ -12,17 +12,21 @@ Today every managed database a tenant exposes externally gets its own `LoadBalan The design ships on the Cilium the platform runs today: passthrough listeners use distinct per-engine subdomain hostnames (`*..`), which avoid the listener-hostname overlap that Cilium isolates correctly only from 1.20 onward. The flat `*.` scheme — shorter `.` connection hostnames — is recorded as the refinement to adopt once the platform is on Cilium 1.20, and the SAN plan below makes that adoption additive rather than client-breaking. -This is the design-proposal artifact required by `cozystack/cozystack#2816`, and it records the decision that issue asks for. The trade-off that issue frames is CNI mesh encryption (datapath lock-in) versus application-level TLS: this proposal chooses **application-level, operator-owned TLS carried through a non-terminating gateway** for the external leg — no edge termination, no second certificate, no private key at the edge, and no dependency on a particular CNI. The other half of that framing — in-cluster (east-west) pod-to-pod encryption, which never leaves the cluster and is necessarily datapath-specific — is recorded and executed separately under `cozystack/cozystack#2977` (PR `cozystack/cozystack#2984`); it complements this proposal rather than competing with it. It is the external-exposure half of epic `cozystack/cozystack#2811`; the certificate/PKI half is covered by the sibling proposal `design-proposals/unified-tls-pki`, on which this one depends for the trust-anchor object. +This is the design-proposal artifact `cozystack/cozystack#2816` (closed) required, and it records the decision that issue asked for. The trade-off it framed is CNI mesh encryption (datapath lock-in) versus application-level TLS: this proposal chooses **application-level, operator-owned TLS carried through a non-terminating gateway** for the external leg — no edge termination, no second certificate, no private key at the edge, and no dependency on a particular CNI. The other half of that framing — in-cluster (east-west) pod-to-pod encryption, which never leaves the cluster and is necessarily datapath-specific — is recorded and executed separately under `cozystack/cozystack#2977` (closed, PR `cozystack/cozystack#2984`); it complements this proposal rather than competing with it. It is the external-exposure half of epic `cozystack/cozystack#2811`; the certificate/PKI half is covered by the sibling proposal `design-proposals/unified-tls-pki`, on which this one depends for the trust-anchor object. ## Scope and related proposals - **Depends on:** `design-proposals/unified-tls-pki` — provides the `.tenant-ca` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion proposal under the same epic, and both are on `main`. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract. The engines reach that point by one path: the extraction controller reads each engine's CA source and writes the canonical `.tenant-ca` — a straight copy with nothing to strip where the operator already maintains a key-free CA object for the chart's sentinel to name (kafka, mariadb), and a strip from the key-bearing Secret every other engine has — rather than any engine publishing that canonical name itself. redis is on the stripping side despite appearances: the operator it patches does publish a key-free object, but the chart's sentinel names the key-bearing cert-manager CA instead, so redis strips exactly like postgres and mongodb. Either way the external client verifies against the one canonical `.tenant-ca`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. -- **Related:** the structured `expose` model — it would replace the chart-level `external` boolean with a structured, additive `expose` list, and lists Gateway/SNI consolidation as the forward-compatible future path. None of it has landed: the proposal (`cozystack/community#29`) was closed, and the `ExposureClass` / `ServiceExposure` implementation (`cozystack/cozystack#3081`) was removed again by `cozystack/cozystack#3218`, which replaced it with the native `Service` field `loadBalancerClass` surfaced as `publishing.loadBalancerClass`. Treat every reference to `expose` below as a design this proposal composes with if it returns, not as a platform mechanism. This proposal and that model meet at the trigger surface: once `expose` lands, "expose this database via SNI-passthrough" is naturally one `expose` entry (an exposure class or scope backed by the tenant Gateway) rather than a new chart toggle, and the per-release route rendering naturally belongs to that orchestration layer. This proposal defines the Gateway-side mechanics either trigger drives; it does not depend on `expose` landing first. -- **Umbrella:** `cozystack/cozystack#2811`. This proposal covers WS4 (`cozystack/cozystack#2815`, SNI exposure) and WS5 (`cozystack/cozystack#2816`, end-to-end TLS). +- **Related:** the structured `expose` model — it would replace the chart-level `external` boolean with a structured, additive `expose` list, and lists Gateway/SNI consolidation as the forward-compatible future path. None of it has landed: the proposal (`cozystack/community#29`) was closed, and the `ExposureClass` / `ServiceExposure` implementation (`cozystack/cozystack#3081`) was removed again by `cozystack/cozystack#3218`, which replaced it with the native `Service` field `loadBalancerClass` surfaced as `publishing.loadBalancerClass`. Treat every reference to `expose` below as a design this proposal composes with if it returns, not as a platform mechanism. Nothing here waits on it: the trigger is a chart value on the database beside `external`, and the per-release route is rendered by the database's own chart against the tenant Gateway (§5), the shape `packages/apps/harbor` already uses for its `HTTPRoute`. A structured exposure model returning later would select this mode in place of that chart value; it would not move the route. +- **Umbrella:** `cozystack/cozystack#2811`. This proposal covers WS4 (`cozystack/cozystack#2815`, SNI exposure) and WS5 (`cozystack/cozystack#2816`, closed, end-to-end TLS). - **Referenced, not designed here:** WS6 east-west / in-cluster CNI encryption (`cozystack/cozystack#2977`, PR `cozystack/cozystack#2984`). It is complementary defense-in-depth for pod-to-pod traffic and is explicitly out of scope (see Non-goals). All repository paths below refer to the `cozystack/cozystack` repository. +## Decisions + +- [0001. External exposure is the native LoadBalancer Service, not a Cozystack exposure API](./decisions/0001-external-exposure-is-the-native-loadbalancer-service.md) — why the removed `ExposureClass` / `ServiceExposure` layer is not the orchestration point. + ## Context ### TLS-passthrough already exists @@ -118,7 +122,7 @@ flowchart TB CA -.->|"verifies"| C3 ``` -A Gateway listener is keyed by the tuple (port, protocol, hostname/SNI), and **multiple `TLSRoute` objects can attach to one listener**, each selecting its backend by its own `spec.hostnames`. SNI-based routing therefore works on any TCP port, not only 443, and — crucially — it does not need a listener per database. The design uses **one passthrough listener per engine type**, on that engine's native port, with a per-engine wildcard hostname: `tls-postgres` on 5432 with hostname `*.postgres.`, `tls-redis` on 6379 with `*.redis.`, `tls-mongos` on 27017 with `*.mongo.` — each `mode: Passthrough`, `AllowedRoutes` limited to `TLSRoute`. Every database release of that engine then attaches a per-release `TLSRoute` carrying `spec.hostnames: [".."]`, and the Gateway SNI-routes each connection to the right backend. All of a tenant's database listeners live on the one tenant Gateway and therefore share its single IP. (Restricting the new listeners to `TLSRoute` is safe where the 443 listeners could not be: the `cilium/cilium#45559` same-port kind-divergence collapse applies per port, and each engine listener is alone on its port.) +A Gateway listener is keyed by the tuple (port, protocol, hostname/SNI), and **multiple `TLSRoute` objects can attach to one listener**, each selecting its backend by its own `spec.hostnames`. SNI-based routing therefore works on any TCP port, not only 443, and — crucially — it does not need a listener per database. The design uses **one passthrough listener per engine type**, on that engine's native port, with a per-engine wildcard hostname: `tls-postgres` on 5432 with hostname `*.postgres.`, `tls-redis` on 6379 with `*.redis.`, `tls-mongos` on 27017 with `*.mongo.` — each `mode: Passthrough`. Every database release of that engine then attaches a per-release `TLSRoute` carrying `spec.hostnames: [".."]`, and the Gateway SNI-routes each connection to the right backend. All of a tenant's database listeners live on the one tenant Gateway and therefore share its single IP. (These listeners declare no route kinds at all, where the 443 listeners declare both kinds together; the two settings answer different constraints of the same Cilium. On the pinned version a listener that spells its kinds out has them applied to every route on the Gateway rather than to the listener that declared them, so naming `TLSRoute` here would reject every `HTTPRoute` the tenant publishes. What an unset field then admits is the implementation's call rather than the spec's, and `cozystack/cozystack#3342` states both halves: Gateway API leaves the protocol-to-kind mapping to the implementation and its conventional table pairs `TLS` with `TCPRoute` as well, so what leaves `TLSRoute` as the whole set here is the pinned Cilium implementing neither `TCPRoute` nor `UDPRoute`.) This keeps listener consumption at **O(engine types)** — at most four or five — rather than O(database instances). A tenant running thirty Postgres releases spends one `tls-postgres` slot, not thirty, so the 64-listener budget (§3) stops being a per-database ceiling and the consolidation goal scales with instance count, not against it. @@ -172,41 +176,54 @@ The ceiling, stated honestly: ### 5. API and controller extension -The trigger and listener synthesis stay in the controller; database charts do not render Gateway plumbing. Database charts already receive the `_cluster` values channel and read parts of it (for example `packages/apps/postgres/templates/db.yaml` reads `_cluster.scheduling`, and mongodb reads `_cluster["cluster-domain"]`), but they do not read the gateway-discovery keys (the gateway-enabled flag and the gateway name) and have no logic to locate the tenant Gateway. Teaching every database chart that discovery dance would duplicate it across five charts and couple application charts to networking topology. The controller already owns listener synthesis; keep it there. - Two objects implement the model, at different cardinalities: one shared listener per engine type, and one `TLSRoute` per database release. -The **listener** is the new API surface. The existing `TLSPassthroughServices []string` field (`api/gateway/v1alpha1/tenantgateway_types.go`) is too weak — a bare service name hardcodes the layer-7 convention of port 443 and hostname `.`. Add one structured field alongside it, leaving the existing field untouched for backward compatibility: +The split between them follows the platform's own convention rather than introducing one: the tenant Gateway's listeners are rendered by the `TenantGateway` controller, and a release's route is rendered by that release's own chart. Listener synthesis belongs to the controller because `Gateway.spec.listeners` has exactly one writer by design — `packages/extra/gateway` renders the `TenantGateway` CR and deliberately renders neither the `Gateway` nor its `Certificate` objects, which is what keeps Helm and the controller from racing on that array. Route rendering belongs to the chart because a chart needs no discovery beyond values it already reads: `_namespace.gateway` names the namespace of the nearest ancestor tenant that owns a Gateway and `_namespace.host` the apex the hostname is built from, both arriving in the release's `cozystack-values` Secret, while the Gateway in that namespace is named `cozystack` for every tenant. Two charts write that Secret and a database chart sees no difference between them: `packages/apps/tenant/templates/namespace.yaml` renders it for every tenant it creates and skips `tenant-root` by name, and `packages/system/cozystack-basics/templates/cozystack-values-secret.yaml` is the sole source for `tenant-root` itself, where `gateway` appears only once `_cluster.gateway-enabled` is set. `packages/apps/harbor/templates/httproute.yaml` is exactly that with an `HTTPRoute`, and `packages/system/cozystack-api/templates/api-tlsroute.yaml` is the same attachment with a `TLSRoute` naming its listener by `sectionName` — reading the namespace from `_cluster.expose-ingress` instead, being a platform component rather than a tenant app. A database chart is that second shape with a tenant app's discovery: `_namespace.gateway` for the namespace, `sectionName: tls-` for the listener. + +The **listener** is the new API surface. The existing `TLSPassthroughServices []string` field (`api/gateway/v1alpha1/tenantgateway_types.go`) is too weak for this design — a bare service name hardcodes the layer-7 convention of port 443 and hostname `.`. `cozystack/cozystack#3342` specifies one structured field alongside it. The two passthrough forms stay independent and neither replaces the other, but the older field does not come through unchanged: the same specification bounds it at 62 entries of at most 249 characters each, refuses a new entry whose name collides with one of its own, and withdraws the HTTPS-terminate listener from every hostname its listeners answer — a behaviour change that reaches clusters which never declare the new field at all. Everything below is the new field's contract, cited from there rather than restated as a second design; the block abridges it, keeping the field names, types and the four list-level markers verbatim, shortening the comments, and omitting the per-field markers the comments describe: ```go -// TLSPassthroughListener declares one shared passthrough listener for a -// database engine type, on its native port, with a wildcard SNI hostname. -// Every release of that engine attaches its own TLSRoute by SNI hostname. +// TLSPassthroughListener declares one layer-4 TLS-passthrough listener +// on the tenant Gateway: mode Passthrough, on a native port, matched by +// SNI. Every release of that engine attaches its own TLSRoute. // -// Validation contract (enforced by CEL / admission): -// - Name: DNS-1123 label; unique within the list; renders as "tls-". -// - Port: 1..65535; unique within the list; must not collide with a -// synthesized layer-7 listener port (443) or another entry. -// - Hostname: optional; a valid DNS wildcard or exact hostname; defaults -// to "*.." when empty (the per-engine subdomain -// scheme; see §1 for why the flat "*." form is gated -// on Cilium 1.20). +// Validation contract (schema and CEL on the CRD, so a bad entry is +// refused on the write rather than stalling the tenant's reconcile): +// - Name: DNS-1123 label, and the sectionName a TLSRoute attaches +// to. The list is a map keyed by name, so names are unique +// by schema; a name must also differ from every +// TLSPassthroughServices entry, since both forms render a +// "tls-" listener. +// - Port: 1..65535, distinct from every other entry, and neither +// 80 nor 443 — the Gateway's own http and TLS-terminate +// listeners own those. +// - Hostname: an exact hostname or a left-most-label wildcard, on +// Gateway API's own Hostname pattern, and within the +// tenant apex (equal to it or a subdomain of it). type TLSPassthroughListener struct { - Name string `json:"name"` // listener suffix -> "tls-" (e.g. "postgres") - Port int32 `json:"port"` // native port (5432/6379/27017) - Hostname string `json:"hostname,omitempty"` // wildcard SNI match, default "*.." + Name string `json:"name"` // renders listener "tls-" + Port int32 `json:"port"` // native port (5432/6379/27017) + Hostname string `json:"hostname"` // SNI match, e.g. "*.postgres." } -// TLSPassthroughListeners renders one Passthrough listener "tls-" on -// .Port per entry, AllowedRoutes restricted to TLSRoute. Independent of the -// layer-7 TLSPassthroughServices field. +// TLSPassthroughListeners renders one "tls-" Passthrough listener +// per entry, independent of the layer-7 TLSPassthroughServices field. // +optional +// +listType=map +// +listMapKey=name +// +kubebuilder:validation:MaxItems=62 TLSPassthroughListeners []TLSPassthroughListener `json:"tlsPassthroughListeners,omitempty"` ``` -The **route** is a standard Gateway API `TLSRoute` (no new type), rendered once per exposed database release: `spec.parentRefs` attaches to the shared `tls-` listener by `sectionName`, `spec.hostnames: [".."]` carries the SNI match, and `spec.rules[].backendRefs` is a standard `BackendRef` (whose embedded `BackendObjectReference` points at the database Service on its native port, cross-namespace via `ReferenceGrant`). `TLSRoute` is GA as `gateway.networking.k8s.io/v1` since Gateway API v1.5.0, but Cilium 1.19.x consumes the experimental-channel CRD — pin to whatever API version the targeted Cilium ships, not the upstream `v1` graduation. +Some rules of that contract have no admission form and fall to the controller instead, so they surface as `Ready=False` on the `TenantGateway` rather than rejecting the write. Hostname overlap is the one that bears on this design: no two passthrough listeners may match the same SNI, counting the port-443 listeners `tlsPassthroughServices` renders and counting wildcards, which is why a listener hostname as broad as `*.` is refused while those defaults are present. It sits in the controller on cost grounds rather than for want of expressiveness — wildcard-aware matching across two lists is a nested CEL scan whose estimated cost pushes the CRD past the apiserver's install-time budget. The per-engine wildcards this design uses stay clear of that rule; the flat `*.` scheme of §1 does not, so that phase is gated on this rule as well as on the Cilium fix. An apex no listener hostname can sit inside is judged the same way, and only while the field is in use: a hostname must be lowercase and within the apex, so an apex carrying upper case — which reaches the `TenantGateway` verbatim from the `namespace.cozystack.io/host` label, normalised by nothing — leaves every entry unsatisfiable. The assembled Gateway's listener total against the Gateway API cap (§3) is judged there too — that one on the finished object rather than on anything declared on a field — as is a repeated entry in the sibling `tlsPassthroughServices` list. One restriction runs the other way, enforced at admission and pinned to the certificate mode rather than to a hostname: the field is refused under `certMode: dns01` or `existingSecret`, where the tenant is served from one wildcard terminate listener that the pinned Cilium cannot keep apart from a passthrough listener under the same apex — so a connection arriving on 443 would reach the database backend. The controller carries its own copy of the admission rules, because the CRD and the controller roll out separately and it cannot assume the spec it reads was checked against the rules it knows. + +The **route** is a standard Gateway API `TLSRoute` (no new type), rendered once per exposed database release: `spec.parentRefs` names the `cozystack` Gateway in the `_namespace.gateway` namespace with `sectionName: tls-`, `spec.hostnames: [".."]` carries the SNI match, and `spec.rules[].backendRefs` points at the database Service on its native port. Attachment to these listeners is narrower than to the port-443 ones: the contract pins their `allowedRoutes.namespaces` to the `TenantGateway`'s own namespace by the `kubernetes.io/metadata.name` label kube-apiserver writes, instead of selecting on `namespace.cozystack.io/gateway`, which every inheriting child tenant namespace carries. So a release renders a route that attaches when it lives in the tenant that owns the Gateway — the case where `_namespace.gateway` names its own namespace — and a release in an inheriting child tenant is out of reach until that attach set widens. Route and backend in one namespace also means no `ReferenceGrant`; a cross-namespace backend would need one. `TLSRoute` is GA as `gateway.networking.k8s.io/v1` since Gateway API v1.5.0, but Cilium 1.19.x consumes the experimental-channel CRD — pin to whatever API version the targeted Cilium ships, not the upstream `v1` graduation (the platform's own route is on `v1alpha2`). + +Per entry the field renders one `mode: Passthrough` listener named `tls-` on the declared `Port` with the entry's `Hostname`, alongside the layer-7 terminate listeners. Nothing is routed until a `TLSRoute` attaches by `sectionName: tls-` and names a backend, and because the listener never terminates TLS the backend — not the Gateway — holds the certificate for the listener's hostname. Declaring the entry does one thing past creating the listener, and for this design it is not a detail: the hostname is reserved against termination, so every name the entry's SNI covers loses its HTTPS-terminate listener and the ACME certificate that came with it, and an `HTTPRoute` claiming such a name is told so in its own status. Under per-engine wildcards that is the whole `*..` subtree, for as long as the entry exists — a reason of its own to settle who owns the entry, since the withdrawal outlives the last release. One consequence of the pinned Cilium is worth stating where the port is declared: the whole Gateway translates into a single Envoy listener whose filter chains match on transport protocol and SNI alone, so a handshake carrying the engine's SNI selects that chain on any port the tenant's load balancer forwards, 443 included. The declared port is where the listener is published, not a boundary on where the backend can be reached; the hostname is the access surface, the backend keeps its own authentication, and the SNI is public. + +What the two objects do not settle is who declares the listener entry. The route's owner follows from the convention above — the database's own chart, once per release, gated on the same `external`-adjacent value that selects this mode. The entry has two candidate owners with different costs, and this proposal leaves the choice open rather than answering it (see Open questions). `cozystack/cozystack#3342` exposes no chart value for the field, so an entry there is reachable only by editing the `TenantGateway` directly — enough to validate the listener, not a path for wiring an engine. -The controller change is to render, per `TLSPassthroughListeners` entry, one Passthrough listener on the supplied `Port` instead of the hardcoded 443, and to attach each per-release `TLSRoute` by `sectionName: tls-` — the same attachment pattern as the existing `api-tlsroute.yaml`, except that many routes share one listener and the Gateway disambiguates them by SNI hostname. Both are populated by the orchestration layer that already knows the tenant Gateway and the database release — not the database chart and not the human. The natural shape of that orchestration is the structured `expose` model, if it returns: an `expose` entry whose class or scope selects SNI-passthrough is the tenant-facing trigger, and the layer that reconciles `expose` entries into `ServiceExposure` objects is the same layer that renders the per-release `TLSRoute` and reference-counts the shared engine listener. Until that model lands, the interim trigger is the engine's `external`-adjacent toggle, with the same rendering responsibility held by the Tenant / HelmRelease orchestration. Either way, the engine-type listener is created on first exposure of that engine and removed once its last release is gone; per-release add/remove only touches the route, never the shared listener. +Everything off the SNI path keeps the exposure primitive the platform ships. `cozystack/cozystack#3218` removed the `network.cozystack.io` group — `ExposureClass` and `ServiceExposure` — on the ground that managed-application charts own their Service, so native `type: LoadBalancer` with `loadBalancerClass` and an admin-provisioned pool cover external exposure without a dedicated API group and controller ([decision 0001](./decisions/0001-external-exposure-is-the-native-loadbalancer-service.md)). Each engine's chart renders that Service itself under `external: true` (`packages/apps/postgres/templates/external-svc.yaml`, `packages/apps/redis/templates/service.yaml`), and the engines the matrix excludes or defers stay there (§4). The class the removal introduced, `publishing.loadBalancerClass`, reaches charts as `_cluster.load-balancer-class` and is read by the host ingress alone (`packages/extra/ingress/templates/nginx-ingress.yaml`), so a database's LoadBalancer Service carries no class and lands on the cluster's default LoadBalancer implementation. Surfacing the class on database Services is a chart change independent of this proposal. ### 6. Trust-anchor and SAN flow @@ -214,7 +231,7 @@ End to end: the chart injects the external hostnames into the operator-issued ce ## User-facing changes -A database gains an `external`-adjacent toggle to select passthrough/SNI mode (expected to become an `expose` entry once the structured-exposure model lands; see §5). Per-engine connection recipes are documented against the subdomain hostnames: `psql "sslnegotiation=direct sslmode=verify-full sslrootcert=ca.crt host=.postgres."` (libpq and server both PG17+), `redis-cli --tls --cacert ca.crt --sni .redis. -h .redis.` (the explicit `--sni` matters: redis-cli does not derive SNI from `-h`; library clients do), `mongosh --tls --tlsCAFile ca.crt --host .mongo.`. Kafka, MariaDB, and non-sharded MongoDB keep today's per-LoadBalancer behavior. +A database gains an `external`-adjacent toggle to select passthrough/SNI mode — a value on the app beside `external`, which its chart also reads to render the route (§5). Per-engine connection recipes are documented against the subdomain hostnames: `psql "sslnegotiation=direct sslmode=verify-full sslrootcert=ca.crt host=.postgres."` (libpq and server both PG17+), `redis-cli --tls --cacert ca.crt --sni .redis. -h .redis.` (the explicit `--sni` matters: redis-cli does not derive SNI from `-h`; library clients do), `mongosh --tls --tlsCAFile ca.crt --host .mongo.`. Kafka, MariaDB, and non-sharded MongoDB keep today's per-LoadBalancer behavior. ## Upgrade and rollback compatibility @@ -230,17 +247,20 @@ Exposing a database externally with TLS explicitly off is not silently corrected - A pre-PG17 (client or server) or non-direct-TLS Postgres client sends no SNI → no route → connection reset/timeout (document the symptom). - Any MariaDB/MySQL client dials a passthrough listener → mutual deadlock (client waits for the server greeting, listener waits for a ClientHello) → timeout; prevented by never rendering a MariaDB listener (matrix exclusion). -- The 64-listener budget is exceeded → the listener is rejected; because listeners are one per engine type this is reached only through child-apex fan-out, and the mitigation is to split that subtree onto its own Gateway (until ListenerSet lands with Cilium 1.20). -- A flat `*.` passthrough listener is created on Cilium <1.20 → hostname overlap with the terminate listeners; routing does not isolate correctly. Prevented by the subdomain default; the flat scheme is gated on the platform reaching Cilium 1.20 (Rollout). +- The assembled listener total exceeds the Gateway API cap of 64 → the controller renders no Gateway at all and the tenant Gateway reports `Ready=False`; it is not the one listener over the line that is refused. Because listeners are one per engine type, the cap is reached through child-apex fan-out rather than database fan-out, and the mitigation is to split that subtree onto its own Gateway until `ListenerSet` lifts the cap. +- A flat `*.` passthrough listener is declared → the controller's overlap rule refuses it while the default `tlsPassthroughServices` listeners hold `api.`, and the tenant Gateway reports `Ready=False` instead of rendering; on the pinned Cilium that hostname would in any case overlap the terminate listeners and routing would not isolate correctly. The per-engine subdomain hostnames every entry declares here avoid both, and the flat scheme is a later phase (Rollout). - An explicit `tls.enabled: false` together with `external: true` → rejected at admission by a ValidatingAdmissionPolicy on the typed kind, evaluated in `cozystack-api`'s admission chain (see Security), not silently overridden; an unset `tls` tri-state auto-enables TLS with `external`. - An operator expects multi-Gateway IP sharing → each Gateway still gets its own IP (expected under the Cilium constraint). -- A database is deleted → its per-release `TLSRoute` is removed; the shared engine-type listener is removed only when its last release is gone, so a single deletion never orphans a listener. +- A release selects passthrough mode before its engine has a listener → the chart renders the `TLSRoute`, no `tls-` section exists to attach to, and the route reports that in its status; nothing serves the hostname until the entry is declared. The reverse ordering of the deletion case, and reachable for the same reason: the route follows the release and the entry does not. +- An engine listener is declared for `*..` → every hostname under that subtree stops being terminated by the Gateway, since a passthrough hostname earns no HTTPS-terminate listener and no certificate. Harmless while the subtree holds only database hostnames, which is what the per-engine scheme buys; publishing an HTTP app under `..` is what to avoid. +- A database in an inheriting child tenant selects passthrough mode → its chart renders the `TLSRoute` and the engine listener exists on the ancestor's Gateway, but the route cannot attach: the native-port listeners admit only the namespace that owns the Gateway, while the port-443 listeners admit the whole inheriting subtree. The refusal lands on the route's status, not on the database. +- A database is deleted → its per-release `TLSRoute` goes with the release, because the release's own chart renders it. The engine listener does not: it is an entry on the `TenantGateway`, so deleting the last release of an engine leaves a listener matching that engine's SNI with nothing to forward to. Which layer removes the entry is the open question below. ## Testing - Helm-template assertions that the certificate SAN includes both `..` and `.` per engine, mirroring the existing TLS test fixtures. -- A controller unit test that a `TLSPassthroughListeners` entry renders a shared listener on the native port with `mode: Passthrough`, the `*..` default hostname, and `AllowedRoutes` restricted to `TLSRoute`, and that two per-release `TLSRoute` objects on that one listener SNI-route to their respective backends. -- An admission test that `tls.enabled: false` with `external: true` is rejected by the typed-kind ValidatingAdmissionPolicy on the aggregated path, while an unset `tls` is admitted and auto-enables; and that the listener validation contract (name/port uniqueness, port range, 443 collision) is enforced. +- A controller unit test that a `tlsPassthroughListeners` entry renders a shared listener on the declared port with `mode: Passthrough`, the entry's hostname, and route attachment confined to the Gateway's own namespace, and that two per-release `TLSRoute` objects on that one listener SNI-route to their respective backends. +- An admission test that `tls.enabled: false` with `external: true` is rejected by the typed-kind ValidatingAdmissionPolicy on the aggregated path, while an unset `tls` is admitted and auto-enables; and that the listener validation contract (name and port uniqueness, port range, the reserved 80 and 443, hostname within the apex, and the refusal under `certMode: dns01` or `existingSecret`) is enforced at admission, while hostname overlap, an apex no listener hostname fits inside, a repeated `tlsPassthroughServices` entry and the assembled listener total surface as `Ready=False`. - An end-to-end test per fitting engine: connect from outside the cluster with SNI and `ca.crt`, and assert that the serial of the presented certificate equals the operator-issued internal certificate — proving reuse, not re-issuance. - A negative test: a client without SNI fails closed. @@ -249,15 +269,15 @@ Exposing a database externally with TLS explicitly off is not silently corrected 1. API field plus controller listener rendering, no engine wired. **Gate:** on the deployed Cilium (1.19.x), stand up a `mode: Passthrough` listener on a native database port (for example 5432) with a per-engine subdomain hostname (`*.postgres.`) alongside the Gateway's terminate listeners, and confirm it SNI-routes to the right backend with exactly one Envoy filter chain. The gate validates the subdomain scheme on the version actually deployed; no part of Phase 1 depends on an unreleased Cilium. 2. Redis (the cleanest fit) behind an opt-in; default-on candidate once the gate and library-client SNI behavior are confirmed in practice. 3. PostgreSQL (direct-TLS) and sharded MongoDB, opt-in. -4. Flat-hostname refinement, gated on the platform's Cilium carrying the isolation fix (see Open questions): add `*.` passthrough listeners, extend per-release routes with the flat hostname, re-run the Phase 1 gate against the overlap case (a passthrough listener sharing `*.` with the terminate listeners, exactly one filter chain). Certificates already carry both SANs, so this phase changes no client. +4. Flat-hostname refinement, gated on two things rather than one (see Open questions): the platform's Cilium carrying the isolation fix, and the controller's own overlap rule, which refuses a `*.` listener hostname against the `api.` the default `tlsPassthroughServices` seeds — lifting that is a change to the controller, not a consequence of the bump. Then add `*.` passthrough listeners, extend per-release routes with the flat hostname, re-run the Phase 1 gate against the overlap case (a passthrough listener sharing `*.` with the terminate listeners, exactly one filter chain). Certificates already carry both SANs, so this phase changes no client. Kafka, MariaDB, and non-sharded MongoDB are explicitly out of this rollout. Each phase ships documentation, a connection recipe, and an end-to-end gate. ## Open questions -- **Ship now on subdomains, or wait and ship flat-only** — the flat `*.` hostnames need a Cilium release carrying the isolation fix `cilium/cilium#44889`: 1.20 (final proposed for late July 2026), or a 1.19.x patch if the open v1.19 backport (`cilium/cilium#46826`) merges first. That leaves a genuine fork: implement the subdomain scheme now on 1.19.x and adopt flat later as the additive phase, or wait for the fixed Cilium and ship flat-only from day one — no dual scheme, no per-engine subdomain legacy to carry. Waiting may well be free in calendar terms, because engine wiring is gated on `unified-tls-pki` convergence anyway (Scope), which lands on a similar horizon. This document specifies the subdomain path as the no-wait default; the fork is a scheduling decision to settle at the next sync, not a design blocker. +- **Ship now on subdomains, or wait and ship flat-only** — the flat `*.` hostnames need a Cilium release carrying the isolation fix `cilium/cilium#44889`: 1.20 (final proposed for late July 2026), or a 1.19.x patch if the open v1.19 backport (`cilium/cilium#46826`) merges first. That leaves a genuine fork: implement the subdomain scheme now on 1.19.x and adopt flat later as the additive phase, or ship flat-only from day one — no dual scheme, no per-engine subdomain legacy to carry. The second branch costs more than waiting: the overlap rule refuses a `*.` listener hostname while the default `tlsPassthroughServices` entries hold `api.`, so flat-only also needs that rule relaxed or those defaults emptied, neither of which a Cilium version delivers. Waiting may well be free in calendar terms, because engine wiring is gated on `unified-tls-pki` convergence anyway (Scope), which lands on a similar horizon. This document specifies the subdomain path as the no-wait default; the fork is a scheduling decision to settle at the next sync, not a design blocker. - **Direct-TLS client/server floor for Postgres** — what fraction of the tenant base predates `sslnegotiation=direct` on either client (libpq PG17+, driver support) or server (PG17+)? Is a per-release attestation gate enough, given there is no graceful downgrade on a passthrough listener? -- **TLSRoute ownership and the `expose` model** — the natural owner of per-release route rendering is whatever layer eventually reconciles `expose` entries, which also answers who reference-counts the shared listener down to zero. If this proposal is implemented before `expose` lands, does the interim Tenant / HelmRelease orchestration carry that logic temporarily, or do we sequence this proposal's engine wiring after `expose`? +- **Who declares the `tlsPassthroughListeners` entry for an engine** — the per-release `TLSRoute` belongs to the database's own chart, the shape `packages/apps/harbor` and `packages/system/cozystack-api` already use (§5); the listener entry does not follow from that. Rendered by the app chart, the first exposed release of an engine adds one element to an array on a `TenantGateway` that `packages/extra/gateway` renders in full, including its `tlsPassthroughServices` list — two Helm releases writing one array, which no template can do on its own and which needs a controller or a patch mechanism under it. Rendered by the tenant chart, ownership is clean and the cost moves: a tenant declares an engine's listener before any release of that engine exists, which opens the port and matches the SNI with nothing to forward the stream to. A routeless passthrough listener is not unheard of — a tenant that owns a Gateway but is not the one the platform's own `TLSRoute`s attach to already carries the three default `tlsPassthroughServices` listeners with none behind them — but those sit on 443 among the terminate listeners, and whether the pinned Cilium programs one alone on a native port is not established anywhere: `TenantGateway` readiness folds in every listener's `Accepted` and `Programmed` conditions, so if it declines, the entry holds the tenant's `Ready` down until its first release arrives. A stored entry also pins the tenant's certificate mode, since the CEL rule that refuses the field under `dns01` or `existingSecret` sits on the `spec` node and stops being suppressed by ratcheting the moment anything on that node changes, and the gateway release retries the refused write without end. Whoever owns the array owns removing its last entry, which settles reference-counting with it. - **64-listener budget accounting** — with the parent listeners, per-child-apex wildcards, and default passthrough services now joined by one listener per database engine type (not per instance), what is the realistic per-tenant ceiling on 1.19.x, and should the controller surface a status condition as the budget nears 64? (ListenerSet dissolves the cap once the platform is on Cilium 1.20.) - **ListenerSet adoption** — Cilium's implementation merged upstream (2026-06-26) and ships in 1.20; Gateway API v1.5.0 carries it in Standard. Do we design this field and its status to be ListenerSet-ready now (listeners as per-namespace objects merged onto a shared Gateway, lifting both the 64 cap and the one-IP-per-tenant ceiling), or revisit after the platform's Cilium bump? This determines whether one-IP-per-tenant is a permanent or temporary ceiling. diff --git a/design-proposals/external-database-exposure/decisions/0001-external-exposure-is-the-native-loadbalancer-service.md b/design-proposals/external-database-exposure/decisions/0001-external-exposure-is-the-native-loadbalancer-service.md index aeac5c4..6d2e604 100644 --- a/design-proposals/external-database-exposure/decisions/0001-external-exposure-is-the-native-loadbalancer-service.md +++ b/design-proposals/external-database-exposure/decisions/0001-external-exposure-is-the-native-loadbalancer-service.md @@ -12,13 +12,13 @@ Two tenant-facing exposure surfaces were in play at once, one landed and one still pending. The pending one was the structured `expose` proposal ([`cozystack/community#29`](https://github.com/cozystack/community/pull/29)), open at the time and set to replace the chart-level `external` boolean with an additive `expose` list layered on `ServiceExposure` — pendency that [`cozystack/cozystack#3164`](https://github.com/cozystack/cozystack/issues/3164) gives as the reason to settle the question before the proposal extended the exposure half rather than after. The implementation had landed ahead of it: `network.cozystack.io/v1alpha1`, a cluster-scoped `ExposureClass` and a namespaced `ServiceExposure` reconciled by a controller in `cozystack-controller`, merged in [`cozystack/cozystack#3081`](https://github.com/cozystack/cozystack/pull/3081) on 2026-06-30. -@lllamnyp filed that issue the next day, and the objection in it is what forced the call; the removal followed two weeks later. The vendor-neutral LoadBalancer selection the new group offered is a native Kubernetes field, `Service.spec.loadBalancerClass`. And two API groups over one domain — `network.cozystack.io` for how an application is reachable, beside `sdn.cozystack.io` for who may reach it — is a fault unless the split is written down, which for the exposure surface it never was, the group having skipped the design process the policy surface went through. +@lllamnyp filed that issue the next day and later approved the removal; the objection in it is what forced the call, and the removal followed two weeks after the filing. The vendor-neutral LoadBalancer selection the new group offered is a native Kubernetes field, `Service.spec.loadBalancerClass`. And two API groups over one domain — `network.cozystack.io` for how an application is reachable, beside `sdn.cozystack.io` for who may reach it — is a fault unless the split is written down, which for the exposure surface it never was, the group having skipped the design process the policy surface went through. This proposal was written against that group while it existed. Its section 5 handed both the tenant-facing trigger and the per-release `TLSRoute` to the layer that reconciled `expose` entries into `ServiceExposure` objects. ## Decision -External exposure is the native Kubernetes primitive, with no Cozystack object in front of it. `cozystack/cozystack#3218` removed the `network.cozystack.io` group; the host ingress, its only consumer, went on rendering `type: LoadBalancer` and gained an optional `publishing.loadBalancerClass` to pick the LoadBalancer controller, leaving the `externalIPs` node-IP default path unchanged. The ground given in the removal is that managed-application charts own their Service, so native `type: LoadBalancer` plus `loadBalancerClass` and an admin-provisioned address pool cover external exposure without a dedicated API group and a controller to reconcile it. The group never reached users: `docs/changelogs/v1.6.0.md` records it as introduced and removed inside one cycle and not part of v1.6.0. +External exposure is the native Kubernetes primitive, with no Cozystack object in front of it. `cozystack/cozystack#3218` removed the `network.cozystack.io` group; the host ingress, its only consumer, went on rendering `type: LoadBalancer` and gained an optional `publishing.loadBalancerClass` to pick the LoadBalancer controller, leaving the `externalIPs` node-IP default path unchanged. The ground given in the removal is that managed-application charts own their Service, so native `type: LoadBalancer` plus `loadBalancerClass` and an admin-provisioned address pool cover external exposure without a dedicated API group and a controller to reconcile it. It reached the v1.6.0 release candidates but not the release: `docs/changelogs/v1.6.0.md` records it as introduced and removed inside one cycle and not part of v1.6.0. ## Why not the alternatives @@ -30,7 +30,7 @@ External exposure is the native Kubernetes primitive, with no Cozystack object i - Nothing mediates between a chart and its external endpoint, so a design that wants an object in between has to earn it rather than assume it. This proposal's section 5 does not ask for one: the per-release `TLSRoute` is rendered by the release's own chart, the shape `packages/apps/harbor/templates/httproute.yaml` uses for an `HTTPRoute` and `packages/system/cozystack-api/templates/api-tlsroute.yaml` for a `TLSRoute`, and the shared engine listener is a `TenantGateway` field specified in [`cozystack/cozystack#3342`](https://github.com/cozystack/cozystack/pull/3342). That split is derived from this decision in the proposal, not argued in `cozystack/cozystack#3218` — the removal thread does not discuss Gateway routing at all. - The class stops short of the databases. `publishing.loadBalancerClass` reaches charts as `_cluster.load-balancer-class`, and `packages/extra/ingress/templates/nginx-ingress.yaml` is what reads it, so a managed database's LoadBalancer Service carries no class and lands on the cluster's default LoadBalancer implementation. -- The migration cost fell only on clusters tracking `main` that had set `publishing.exposureClass`: provision an address pool, switch to `publishing.loadBalancerClass`, delete any orphaned `cozystack-` pool by hand. No released cluster was affected. +- The migration cost fell only on clusters tracking `main` that had set `publishing.exposureClass`: provision an address pool, switch to `publishing.loadBalancerClass`, delete any orphaned `cozystack-` pool by hand. The v1.6.0 release itself never carried the kinds. ## Revisit if