diff --git a/deploy/rustfs-operator/README.md b/deploy/rustfs-operator/README.md index 978fa7c..fc48eaa 100755 --- a/deploy/rustfs-operator/README.md +++ b/deploy/rustfs-operator/README.md @@ -100,9 +100,6 @@ kind: Secret metadata: name: rustfs-rpc-auth namespace: storage - labels: - # Lets Secret updates enqueue this Tenant for prompt revalidation. - rustfs.tenant: rustfs-a type: Opaque stringData: rpc-secret: "replace-with-a-dedicated-rpc-secret" @@ -124,11 +121,11 @@ The operator maps the selected Secret key to `RUSTFS_RPC_SECRET` in every RustFS Pod. Before applying workloads, it verifies that the Secret and selected key exist and that the value is valid UTF-8, non-blank, contains no NUL bytes, and is not the RustFS default credential value (`rustfsadmin`). Keep this value stable -while rotating administrator credentials. The `rustfs.tenant` label is not used -for authorization; it lets updates to an externally managed Secret enqueue the -Tenant for revalidation. A Secret update does not change the environment of -already-running Pods. Coordinated restart and hot reload are outside this -feature. If `spec.rpcSecret` is omitted, the operator does not set +while rotating administrator credentials. Secret updates enqueue every Tenant +whose spec references the Secret, including when multiple Tenants share it. A +Secret update does not change the environment of already-running Pods. +Coordinated restart and hot reload are outside this feature. If `spec.rpcSecret` +is omitted, the operator does not set `RUSTFS_RPC_SECRET`, RustFS resolves it from its own credential configuration, and the operator does not report `RpcAuthReady` for that unmanaged value. @@ -159,7 +156,7 @@ spec: objectLock: true ``` -Policy ConfigMaps and user Secrets must live in the Tenant namespace. `users[].credsSecret.name` selects the credentials Secret; when omitted, the operator falls back to a Secret named after `users[].name` for compatibility with existing manifests. The operator maintains `rustfs.tenant=` on referenced policy ConfigMaps and user Secrets so changes enqueue the owning Tenant; references that do not exist yet are retried. The label triggers reconciliation but does not select the Secret. Provisioned resources are retained when removed from the Tenant spec. +Policy ConfigMaps and user Secrets must live in the Tenant namespace. `users[].credsSecret.name` selects the credentials Secret; when omitted, the operator falls back to a Secret named after `users[].name` for compatibility with existing manifests. The operator maintains `rustfs.tenant=` on referenced policy ConfigMaps. Secret updates are matched against every Tenant spec, so a Secret can be shared by multiple Tenants without a routing label. Legacy routing labels are removed from external Secrets when the operator starts. Provisioned resources are retained when removed from the Tenant spec. ### RBAC Configuration diff --git a/deploy/rustfs-operator/crds/tenant-crd.yaml b/deploy/rustfs-operator/crds/tenant-crd.yaml index 0a1bc1f..8a0115e 100644 --- a/deploy/rustfs-operator/crds/tenant-crd.yaml +++ b/deploy/rustfs-operator/crds/tenant-crd.yaml @@ -1417,9 +1417,8 @@ spec: When configured, the operator maps this key to `RUSTFS_RPC_SECRET` for every RustFS Pod. Keep it stable while rotating `credsSecret`. When omitted, the operator does not set `RUSTFS_RPC_SECRET`; RustFS resolves it from its own - credential configuration. Label an externally managed Secret with - `rustfs.tenant=` so Secret updates enqueue the Tenant for - prompt revalidation. + credential configuration. Secret updates enqueue every Tenant that references + the Secret, so one externally managed Secret can be shared by multiple Tenants. nullable: true properties: key: diff --git a/deploy/rustfs-operator/crds/tenant.yaml b/deploy/rustfs-operator/crds/tenant.yaml index 0a1bc1f..8a0115e 100755 --- a/deploy/rustfs-operator/crds/tenant.yaml +++ b/deploy/rustfs-operator/crds/tenant.yaml @@ -1417,9 +1417,8 @@ spec: When configured, the operator maps this key to `RUSTFS_RPC_SECRET` for every RustFS Pod. Keep it stable while rotating `credsSecret`. When omitted, the operator does not set `RUSTFS_RPC_SECRET`; RustFS resolves it from its own - credential configuration. Label an externally managed Secret with - `rustfs.tenant=` so Secret updates enqueue the Tenant for - prompt revalidation. + credential configuration. Secret updates enqueue every Tenant that references + the Secret, so one externally managed Secret can be shared by multiple Tenants. nullable: true properties: key: diff --git a/docs/operator-user-guide.md b/docs/operator-user-guide.md index 8d3bc3b..5f02bf9 100644 --- a/docs/operator-user-guide.md +++ b/docs/operator-user-guide.md @@ -331,7 +331,7 @@ Credential priority: For production, configure `spec.rpcSecret` so internode RPC authentication does not depend on the administrator credentials. Keep the Secret in the Tenant -namespace and label externally managed Secrets with the Tenant name: +namespace. The Secret does not need a Tenant routing label: ```yaml apiVersion: v1 @@ -339,8 +339,6 @@ kind: Secret metadata: name: rustfs-rpc-auth namespace: storage - labels: - rustfs.tenant: rustfs-a type: Opaque stringData: rpc-secret: "replace-with-a-dedicated-rpc-secret" @@ -358,9 +356,9 @@ spec: ``` The selected value must be valid UTF-8, non-blank, contain no NUL bytes, and -must not be `rustfsadmin`. The `rustfs.tenant` label lets Secret updates enqueue -the Tenant for prompt revalidation; it is not an authorization mechanism. When -the configured Secret passes validation, the Operator reports +must not be `rustfsadmin`. Secret updates enqueue every Tenant whose spec +references that Secret, including when multiple Tenants share it. When the +configured Secret passes validation, the Operator reports `RpcAuthReady=True`. Updating the Secret does not change the environment of already-running Pods; coordinated restart and hot reload are outside this feature. @@ -598,13 +596,13 @@ The operator can create RustFS policies, users, and buckets after the Tenant wor - `spec.users` for regular users. Each user must have at least one direct policy mapping. - `spec.buckets` for buckets and optional object lock. -ConfigMaps and user Secrets must live in the Tenant namespace. The operator maintains `rustfs.tenant=` on referenced policy ConfigMaps and user Secrets so updates enqueue the owning Tenant. References that do not exist yet are retried after they are created. +ConfigMaps and user Secrets must live in the Tenant namespace. The operator maintains `rustfs.tenant=` on referenced policy ConfigMaps. Secret updates are matched against every Tenant spec, so a Secret can be shared by multiple Tenants without a routing label. Legacy routing labels are removed from external Secrets when the operator starts. Policy documents are parsed by RustFS. Use S3 ARN resource patterns such as `arn:aws:s3:::bucket` and `arn:aws:s3:::bucket/*`; for all buckets, use `arn:aws:s3:::*`. A bare `Resource: "*"` is not accepted by RustFS policy parsing. For each `spec.users[]` entry, set `credsSecret.name` to select a user credentials Secret in the Tenant namespace. If `credsSecret` is omitted, the operator reads a Secret with the same name as `user.name`, preserving the behavior of older manifests. An explicit reference is authoritative and does not fall back to the same-name Secret when it is invalid or missing. Each resolved Secret name must be unique within the Tenant; the API rejects duplicate references, and reconciliation also blocks them when an older installed CRD does not enforce the validation. Distinct Secrets must also contain distinct `accesskey` values. Reconciliation validates all user credentials before modifying any RustFS user and rejects every colliding entry. -The Secret must contain `accesskey` and `secretkey`, or the MinIO-compatible keys `CONSOLE_ACCESS_KEY` and `CONSOLE_SECRET_KEY`. If both key formats are present, their values must match. `user.name` remains the declarative and status identity; the Secret's `accesskey` is the actual RustFS user. User access keys must be at least 8 characters and must not contain whitespace, `=`, or `,`; user secret keys must be at least 8 characters. The `rustfs.tenant` label only causes Secret updates to enqueue the Tenant; it does not select which Secret is read. +The Secret must contain `accesskey` and `secretkey`, or the MinIO-compatible keys `CONSOLE_ACCESS_KEY` and `CONSOLE_SECRET_KEY`. If both key formats are present, their values must match. `user.name` remains the declarative and status identity; the Secret's `accesskey` is the actual RustFS user. User access keys must be at least 8 characters and must not contain whitespace, `=`, or `,`; user secret keys must be at least 8 characters. Secret selection and event routing both come from the Tenant spec; no Tenant label is required on the Secret. Updating a user Secret's `secretkey`, or changing `credsSecret.name` to another Secret with the same `accesskey`, rotates that RustFS user's credential. The `accesskey` is immutable after the first successful reconciliation; use a new user entry and Secret when it must change, then migrate clients before removing the old entry. @@ -634,8 +632,6 @@ kind: Secret metadata: name: rustfs-user-app-user namespace: storage - labels: - rustfs.tenant: rustfs-a type: Opaque stringData: accesskey: appuser01 diff --git a/docs/operator-user-guide.zh-CN.md b/docs/operator-user-guide.zh-CN.md index 55cd34e..90890fd 100644 --- a/docs/operator-user-guide.zh-CN.md +++ b/docs/operator-user-guide.zh-CN.md @@ -332,8 +332,7 @@ spec: #### 独立的节点间 RPC Secret 生产环境建议配置 `spec.rpcSecret`，避免节点间 RPC 认证依赖管理员凭据。 -Secret 必须与 Tenant 位于同一 namespace；若 Secret 由外部系统管理，请添加 -Tenant 标签： +Secret 必须与 Tenant 位于同一 namespace，不需要 Tenant 路由标签： ```yaml apiVersion: v1 @@ -341,8 +340,6 @@ kind: Secret metadata: name: rustfs-rpc-auth namespace: storage - labels: - rustfs.tenant: rustfs-a type: Opaque stringData: rpc-secret: "replace-with-a-dedicated-rpc-secret" @@ -360,8 +357,8 @@ spec: ``` 所选值必须是有效 UTF-8、不能为空、不能包含 NUL 字节，并且不能是 -`rustfsadmin`。`rustfs.tenant` 标签仅用于在 Secret 变化时及时触发 Tenant -重新校验，不是授权机制。配置的 Secret 校验通过后，Operator 会报告 +`rustfsadmin`。Secret 更新会触发 spec 中引用它的所有 Tenant，包括多个 Tenant +共享同一个 Secret 的情况。配置的 Secret 校验通过后，Operator 会报告 `RpcAuthReady=True`。更新 Secret 不会改变已运行 Pod 的进程环境；协调重启和 热加载不在此功能范围内。 @@ -598,13 +595,13 @@ Operator 可以在 Tenant workload Ready 后自动创建 RustFS policy、user - `spec.users`：普通用户。每个 user 必须至少直接绑定一个 policy。 - `spec.buckets`：bucket，可选择开启 object lock。 -ConfigMap 和 user Secret 必须位于 Tenant namespace。operator 会在被引用的 policy ConfigMap 和 user Secret 上维护 `rustfs.tenant=` label，使资源变化能够触发 owning Tenant reconcile；尚未创建的引用资源会在创建后重试加 label。 +ConfigMap 和 user Secret 必须位于 Tenant namespace。operator 仍会在被引用的 policy ConfigMap 上维护 `rustfs.tenant=` label。Secret 更新会按所有 Tenant spec 做反向匹配，因此同一个 Secret 可以被多个 Tenant 共享，不需要路由 label。operator 启动时会清理外部 Secret 上遗留的路由 label。 Policy document 由 RustFS 解析。请使用 `arn:aws:s3:::bucket` 和 `arn:aws:s3:::bucket/*` 这类 S3 ARN resource 写法；如需匹配所有 bucket，请使用 `arn:aws:s3:::*`。RustFS policy parser 不接受裸 `Resource: "*"`。 每个 `spec.users[]` 条目都可以通过 `credsSecret.name` 指定 Tenant namespace 中的 user credentials Secret。省略 `credsSecret` 时，Operator 继续读取与 `user.name` 同名的 Secret，以兼容旧版 manifest。显式引用是唯一来源；配置错误或 Secret 不存在时，不会再回退到同名 Secret。同一 Tenant 内解析后的 Secret 名称必须唯一；API 会拒绝重复引用，而在集群仍安装旧版 CRD、尚未启用该校验时，reconcile 也会阻止这些重复配置生效。不同 Secret 中的 `accesskey` 也必须唯一；reconcile 会先校验全部 user credentials，再修改任何 RustFS user，并拒绝所有冲突条目。 -Secret 必须包含 `accesskey` 和 `secretkey`，或者 MinIO 兼容 key：`CONSOLE_ACCESS_KEY` 和 `CONSOLE_SECRET_KEY`。如果两种 key 同时存在，值必须一致。`user.name` 仍是声明和 status 中的逻辑标识，Secret 内的 `accesskey` 才是实际 RustFS user。user access key 至少 8 个字符，且不能包含空白、`=` 或 `,`；user secret key 至少 8 个字符。`rustfs.tenant` label 只负责在 Secret 更新时触发 Tenant reconcile，不参与选择要读取的 Secret。 +Secret 必须包含 `accesskey` 和 `secretkey`，或者 MinIO 兼容 key：`CONSOLE_ACCESS_KEY` 和 `CONSOLE_SECRET_KEY`。如果两种 key 同时存在，值必须一致。`user.name` 仍是声明和 status 中的逻辑标识，Secret 内的 `accesskey` 才是实际 RustFS user。user access key 至少 8 个字符，且不能包含空白、`=` 或 `,`；user secret key 至少 8 个字符。Secret 的选择和事件路由都来自 Tenant spec，不需要在 Secret 上添加 Tenant label。 更新 user Secret 的 `secretkey`，或者把 `credsSecret.name` 切换到具有相同 `accesskey` 的另一个 Secret，都会轮换对应 RustFS user 的凭据。首次成功 provisioning 后，`accesskey` 不可变；如需变更，请新建 user 条目和 Secret，迁移客户端后再移除旧条目。 @@ -634,8 +631,6 @@ kind: Secret metadata: name: rustfs-user-app-user namespace: storage - labels: - rustfs.tenant: rustfs-a type: Opaque stringData: accesskey: appuser01 diff --git a/e2e/src/framework/cert_manager_tls.rs b/e2e/src/framework/cert_manager_tls.rs index acf108e..1ce761c 100644 --- a/e2e/src/framework/cert_manager_tls.rs +++ b/e2e/src/framework/cert_manager_tls.rs @@ -48,7 +48,6 @@ const OPAQUE_SECRET_TYPE: &str = "Opaque"; const REDACTED_FIXTURE_BYTES: &[u8] = b"redacted-test-fixture"; const MANAGED_CERTIFICATE_CASE_SUFFIX: &str = "cert-manager-managed"; const EXTERNAL_SECRET_CASE_SUFFIX: &str = "cert-manager-external"; -const RUSTFS_TENANT_LABEL: &str = "rustfs.tenant"; pub const POSITIVE_CERT_MANAGER_TLS_TIMEOUT: Duration = Duration::from_secs(600); #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -88,10 +87,6 @@ pub fn positive_cert_manager_tls_timeout(config: &E2eConfig) -> Duration { std::cmp::max(config.timeout, POSITIVE_CERT_MANAGER_TLS_TIMEOUT) } -fn tenant_watch_labels(config: &E2eConfig) -> BTreeMap { - BTreeMap::from([(RUSTFS_TENANT_LABEL.to_string(), config.tenant_name.clone())]) -} - pub fn managed_certificate_storage_layout(config: &E2eConfig) -> storage::LocalStorageLayout { positive_tls_storage_layout(config, MANAGED_CERTIFICATE_CASE_SUFFIX) } @@ -1047,13 +1042,7 @@ fn external_tls_secret_manifest( secret_type: &str, data: BTreeMap, ) -> Result { - tls_secret_manifest_with_labels( - &config.test_namespace, - name, - secret_type, - data, - tenant_watch_labels(config), - ) + tls_secret_manifest(&config.test_namespace, name, secret_type, data) } fn tls_secret_manifest( @@ -1062,22 +1051,10 @@ fn tls_secret_manifest( secret_type: &str, data: BTreeMap, ) -> Result { - tls_secret_manifest_with_labels(namespace, name, secret_type, data, BTreeMap::new()) -} - -fn tls_secret_manifest_with_labels( - namespace: &str, - name: &str, - secret_type: &str, - data: BTreeMap, - labels: BTreeMap, -) -> Result { - let labels = (!labels.is_empty()).then_some(labels); let secret = corev1::Secret { metadata: metav1::ObjectMeta { name: Some(name.to_string()), namespace: Some(namespace.to_string()), - labels, ..Default::default() }, type_: Some(secret_type.to_string()), diff --git a/e2e/src/framework/resources.rs b/e2e/src/framework/resources.rs index a1119c3..0a08f83 100644 --- a/e2e/src/framework/resources.rs +++ b/e2e/src/framework/resources.rs @@ -80,15 +80,12 @@ kind: Secret metadata: name: {secret_name} namespace: {namespace} - labels: - rustfs.tenant: {tenant_name} type: Opaque stringData: {key}: {rpc_secret} "#, secret_name = rpc_secret_name(config), namespace = config.test_namespace, - tenant_name = config.tenant_name, key = RPC_SECRET_KEY, rpc_secret = TEST_RPC_SECRET, ) @@ -372,13 +369,13 @@ mod tests { } #[test] - fn rpc_secret_uses_tenant_watch_label() { + fn rpc_secret_does_not_require_tenant_watch_label() { let config = E2eConfig::defaults(); let manifest = rpc_secret_manifest(&config); assert_eq!(rpc_secret_name(&config), "e2e-tenant-rpc-auth"); assert!(manifest.contains("namespace: rustfs-e2e-smoke")); - assert!(manifest.contains("rustfs.tenant: e2e-tenant")); + assert!(!manifest.contains("rustfs.tenant")); assert!(manifest.contains("rpc-secret: test-dedicated-rpc-secret")); } } diff --git a/e2e/tests/cert_manager_tls.rs b/e2e/tests/cert_manager_tls.rs index a4586c6..bfd1dae 100644 --- a/e2e/tests/cert_manager_tls.rs +++ b/e2e/tests/cert_manager_tls.rs @@ -91,32 +91,27 @@ fn external_secret_tenant_manifest_uses_shared_secret_and_rollout_strategy() -> } #[test] -fn external_secret_manifests_carry_tenant_watch_label_for_initial_create_and_rotation() -> Result<()> -{ +fn external_secret_manifests_need_no_tenant_watch_label_for_create_or_rotation() -> Result<()> { let config = tls_e2e::external_secret_case_config(&E2eConfig::defaults()); let initial = tls_e2e::external_tls_secret_manifests(&config)?; - assert_secret_manifest_tenant_watch_label( + assert_secret_manifest_has_no_tenant_watch_label( "initial external TLS Secret", &initial.tls_secret_manifest, - &config.tenant_name, )?; - assert_secret_manifest_tenant_watch_label( + assert_secret_manifest_has_no_tenant_watch_label( "initial external CA Secret", &initial.ca_secret_manifest, - &config.tenant_name, )?; let rotated = tls_e2e::external_tls_secret_manifests(&config)?; - assert_secret_manifest_tenant_watch_label( + assert_secret_manifest_has_no_tenant_watch_label( "rotated external TLS Secret", &rotated.tls_secret_manifest, - &config.tenant_name, )?; - assert_secret_manifest_tenant_watch_label( + assert_secret_manifest_has_no_tenant_watch_label( "rotated external CA Secret", &rotated.ca_secret_manifest, - &config.tenant_name, )?; Ok(()) @@ -1062,21 +1057,17 @@ fn collect_tls_artifacts_on_error(config: &E2eConfig, case_name: &str, result: & } } -fn assert_secret_manifest_tenant_watch_label( - context: &str, - manifest: &str, - expected_tenant: &str, -) -> Result<()> { +fn assert_secret_manifest_has_no_tenant_watch_label(context: &str, manifest: &str) -> Result<()> { let secret: corev1::Secret = serde_yaml_ng::from_str(manifest) .with_context(|| format!("parse {context} Secret manifest"))?; - let labels = secret + let has_tenant_label = secret .metadata .labels .as_ref() - .with_context(|| format!("{context} missing labels"))?; + .is_some_and(|labels| labels.contains_key("rustfs.tenant")); ensure!( - labels.get("rustfs.tenant").map(String::as_str) == Some(expected_tenant), - "{context} should carry rustfs.tenant label for Tenant {expected_tenant}" + !has_tenant_label, + "{context} should not carry a rustfs.tenant routing label" ); Ok(()) } diff --git a/examples/README.md b/examples/README.md index a1c1790..89672db 100755 --- a/examples/README.md +++ b/examples/README.md @@ -143,7 +143,7 @@ kubectl apply -f examples/provisioning-tenant.yaml kubectl wait --for=condition=Ready tenant/provisioning-demo --timeout=10m ``` -The operator maintains `rustfs.tenant=` on referenced policy ConfigMaps and user Secrets so updates enqueue the owning Tenant. +The operator labels referenced policy ConfigMaps. Secret updates are matched against every Tenant spec, so one Secret can be shared without a routing label. --- diff --git a/examples/cert-manager-ca-trust-tenant.yaml b/examples/cert-manager-ca-trust-tenant.yaml index 5fa32c0..d42852b 100644 --- a/examples/cert-manager-ca-trust-tenant.yaml +++ b/examples/cert-manager-ca-trust-tenant.yaml @@ -11,8 +11,8 @@ # - enableInternodeHttps=true requires a CA source. CertificateSecretCa expects # cert-manager to write ca.crt into the TLS Secret. Use SecretRef or SystemCa # only when that matches your trust model. -# - Existing cert-manager Secrets used with manageCertificate=false should carry -# rustfs.tenant= if you want Secret changes to enqueue the Tenant. +# - Existing cert-manager Secrets used with manageCertificate=false need no Tenant +# routing label; Secret changes enqueue every Tenant that references them. apiVersion: rustfs.com/v1alpha1 kind: Tenant metadata: diff --git a/examples/provisioning-tenant.yaml b/examples/provisioning-tenant.yaml index 99bc3c9..ffefc1d 100644 --- a/examples/provisioning-tenant.yaml +++ b/examples/provisioning-tenant.yaml @@ -49,8 +49,6 @@ kind: Secret metadata: name: rustfs-user-app-user namespace: default - labels: - rustfs.tenant: provisioning-demo type: Opaque stringData: accesskey: appuser01 diff --git a/src/legacy_secret_labels.rs b/src/legacy_secret_labels.rs new file mode 100644 index 0000000..3555aa4 --- /dev/null +++ b/src/legacy_secret_labels.rs @@ -0,0 +1,240 @@ +// Copyright 2025 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use crate::types::v1alpha1::tenant::{RUSTFS_TENANT_LABEL, Tenant}; +use k8s_openapi::api::core::v1::Secret; +use k8s_openapi::apimachinery::pkg::apis::meta::v1::ObjectMeta; +use kube::api::{ListParams, Patch, PatchParams}; +use kube::{Api, Client, Resource}; +use serde_json::json; +use tracing::{info, warn}; + +/// Remove the legacy single-Tenant routing label from external Secrets. +/// +/// Tenant-owned Secrets retain the label because it still describes ownership. +/// External Secret events are routed through references in every Tenant spec. +pub(crate) async fn cleanup_external_secret_labels(client: &Client) { + let secrets = match Api::::all(client.clone()) + .list(&ListParams::default().labels(RUSTFS_TENANT_LABEL)) + .await + { + Ok(secrets) => secrets, + Err(error) => { + warn!(%error, "failed to list legacy-labeled Secrets"); + return; + } + }; + + let mut cleaned = 0usize; + for secret in secrets { + if has_tenant_controller_owner(&secret.metadata) { + continue; + } + + let (Some(namespace), Some(name), Some(resource_version)) = ( + secret.metadata.namespace.as_deref(), + secret.metadata.name.as_deref(), + secret.metadata.resource_version.as_deref(), + ) else { + warn!( + secret = ?secret.metadata.name, + namespace = ?secret.metadata.namespace, + "skipping legacy Secret label cleanup because metadata is incomplete" + ); + continue; + }; + + let patch = json!({ + "metadata": { + "resourceVersion": resource_version, + "labels": { + (RUSTFS_TENANT_LABEL): null, + }, + }, + }); + let api = Api::::namespaced(client.clone(), namespace); + match api + .patch(name, &PatchParams::default(), &Patch::Merge(&patch)) + .await + { + Ok(_) => cleaned += 1, + Err(error) => { + warn!( + %error, + namespace, + secret = name, + "failed to remove legacy Tenant routing label from Secret" + ); + } + } + } + + if cleaned > 0 { + info!( + cleaned, + "removed legacy Tenant routing labels from external Secrets" + ); + } +} + +fn has_tenant_controller_owner(metadata: &ObjectMeta) -> bool { + metadata.owner_references.as_ref().is_some_and(|owners| { + owners.iter().any(|owner| { + owner.controller == Some(true) + && owner.api_version == Tenant::api_version(&()) + && owner.kind == Tenant::kind(&()) + }) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use http::{Method, Request, Response, StatusCode}; + use kube::{Client, client::Body}; + use serde_json::Value; + use std::convert::Infallible; + use std::sync::{Arc, Mutex}; + use tower::service_fn; + + #[derive(Debug)] + struct CapturedRequest { + method: Method, + path: String, + body: Option, + } + + #[tokio::test] + async fn cleanup_patches_only_external_secrets_with_resource_version() { + let captured = Arc::new(Mutex::new(Vec::::new())); + let service_capture = captured.clone(); + let service = service_fn(move |request: Request| { + let service_capture = service_capture.clone(); + async move { + let method = request.method().clone(); + let path = request.uri().path().to_string(); + let body = request + .into_body() + .collect_bytes() + .await + .expect("request body should be readable"); + let body = (!body.is_empty()).then(|| { + serde_json::from_slice(&body).expect("request body should be valid JSON") + }); + service_capture + .lock() + .expect("request capture lock should be available") + .push(CapturedRequest { + method: method.clone(), + path: path.clone(), + body, + }); + + let response = match (method.as_str(), path.as_str()) { + ("GET", "/api/v1/secrets") => json_response( + StatusCode::OK, + json!({ + "apiVersion": "v1", + "kind": "SecretList", + "metadata": { "resourceVersion": "50" }, + "items": [ + labeled_secret("external", "42", None), + labeled_secret("owned", "43", Some(tenant_owner_reference())), + ], + }), + ), + ("PATCH", "/api/v1/namespaces/storage/secrets/external") => json_response( + StatusCode::OK, + json!({ + "apiVersion": "v1", + "kind": "Secret", + "metadata": { + "name": "external", + "namespace": "storage", + "resourceVersion": "43" + } + }), + ), + _ => panic!("unexpected Kubernetes request: {method} {path}"), + }; + Ok::<_, Infallible>(response) + } + }); + + cleanup_external_secret_labels(&Client::new(service, "default")).await; + + let captured = captured + .lock() + .expect("request capture lock should be available"); + assert_eq!( + captured + .iter() + .filter(|request| request.method == Method::PATCH) + .count(), + 1 + ); + let patch = captured + .iter() + .find(|request| request.method == Method::PATCH) + .expect("external Secret should be patched"); + assert_eq!(patch.path, "/api/v1/namespaces/storage/secrets/external"); + assert_eq!( + patch.body, + Some(json!({ + "metadata": { + "resourceVersion": "42", + "labels": { "rustfs.tenant": null } + } + })) + ); + } + + fn labeled_secret( + name: &str, + resource_version: &str, + owner: Option, + ) -> Value { + json!({ + "apiVersion": "v1", + "kind": "Secret", + "metadata": { + "name": name, + "namespace": "storage", + "resourceVersion": resource_version, + "labels": { "rustfs.tenant": "tenant-a" }, + "ownerReferences": owner.into_iter().collect::>(), + } + }) + } + + fn tenant_owner_reference() -> Value { + json!({ + "apiVersion": "rustfs.com/v1alpha1", + "kind": "Tenant", + "name": "tenant-a", + "uid": "tenant-uid", + "controller": true, + "blockOwnerDeletion": true + }) + } + + fn json_response(status: StatusCode, value: Value) -> Response { + Response::builder() + .status(status) + .body(Body::from( + serde_json::to_vec(&value).expect("response should serialize"), + )) + .expect("response should build") + } +} diff --git a/src/lib.rs b/src/lib.rs index 1c113ec..b61633b 100755 --- a/src/lib.rs +++ b/src/lib.rs @@ -32,7 +32,7 @@ use k8s_openapi::api::core::v1 as corev1; use k8s_openapi::apimachinery::pkg::apis::meta::v1 as metav1; use kube::core::{ApiResource, DynamicObject, GroupVersionKind}; use kube::runtime::reflector::ObjectRef; -use kube::runtime::{Controller, watcher}; +use kube::runtime::{Controller, reflector::Store, watcher}; use kube::{Api, Client, CustomResourceExt, Resource, api::ListParams}; use kube_leader_election::{ LeaderCallbacks, LeaderElector, LeaderElectorConfig, LeaseLock, SystemClock, @@ -84,6 +84,7 @@ pub fn init_tracing() { mod cluster_dns; mod context; +mod legacy_secret_labels; pub mod metrics; pub mod reconcile; mod status; @@ -197,9 +198,13 @@ async fn run_controller( cancel: CancellationToken, cluster_domain: cluster_dns::ClusterDomain, ) { + legacy_secret_labels::cleanup_external_secret_labels(&client).await; + let tenant_client = Api::::all(client.clone()); let context = Context::new_with_cluster_domain(client.clone(), cluster_domain); - let controller = Controller::new(tenant_client, watcher::Config::default()) + let controller = Controller::new(tenant_client, watcher::Config::default()); + let secret_tenant_store = controller.store(); + let controller = controller .watches( Api::::all(client.clone()), watcher::Config::default(), @@ -208,7 +213,7 @@ async fn run_controller( .watches( Api::::all(client.clone()), watcher::Config::default(), - tenant_refs_for_secret, + move |secret| tenant_refs_for_secret(secret, &secret_tenant_store), ) .owns( Api::::all(client.clone()), @@ -584,12 +589,41 @@ fn cert_manager_certificate_api_resource() -> ApiResource { ) } -fn tenant_refs_for_secret(secret: corev1::Secret) -> Vec> { - tenant_refs_from_metadata( +fn tenant_refs_for_secret( + secret: corev1::Secret, + tenant_store: &Store, +) -> Vec> { + tenant_refs_for_secret_from_tenants(&secret, &tenant_store.state()) +} + +fn tenant_refs_for_secret_from_tenants( + secret: &corev1::Secret, + tenants: &[Arc], +) -> Vec> { + let mut refs = tenant_refs_from_metadata( secret.metadata.namespace.as_deref(), secret.metadata.owner_references.as_deref(), - secret.metadata.labels.as_ref(), - ) + None, + ); + let Some(namespace) = secret.metadata.namespace.as_deref() else { + return refs; + }; + let Some(secret_name) = secret.metadata.name.as_deref() else { + return refs; + }; + + for tenant in tenants { + if tenant.metadata.deletion_timestamp.is_some() + || tenant.metadata.namespace.as_deref() != Some(namespace) + || !tenant.spec.referenced_secret_names().contains(secret_name) + { + continue; + } + + push_unique_tenant_ref(&mut refs, ObjectRef::new(&tenant.name()).within(namespace)); + } + + refs } fn tenant_refs_for_config_map(config_map: corev1::ConfigMap) -> Vec> { @@ -702,6 +736,7 @@ pub async fn crd(file: Option) -> Result<(), Box> #[cfg(test)] mod controller_watch_tests { use super::*; + use crate::types::v1alpha1::tenant::{RpcSecretRef, TenantSpec}; use k8s_openapi::apimachinery::pkg::apis::meta::v1 as metav1; use std::collections::BTreeMap; @@ -728,29 +763,37 @@ mod controller_watch_tests { ..Default::default() }; - let refs = tenant_refs_for_secret(secret); + let refs = tenant_refs_for_secret_from_tenants(&secret, &[]); assert_single_ref(&refs, "tenant-a", "storage"); } #[test] - fn secret_mapper_uses_rustfs_tenant_label_for_external_tenant_secret() { + fn secret_mapper_enqueues_every_tenant_referencing_a_shared_secret() { let secret = corev1::Secret { metadata: metav1::ObjectMeta { - name: Some("rustfs-rpc-auth".to_string()), + name: Some("shared-rpc-auth".to_string()), namespace: Some("storage".to_string()), labels: Some(BTreeMap::from([( "rustfs.tenant".to_string(), - "tenant-b".to_string(), + "tenant-a".to_string(), )])), ..Default::default() }, ..Default::default() }; - - let refs = tenant_refs_for_secret(secret); - - assert_single_ref(&refs, "tenant-b", "storage"); + let tenants = vec![ + tenant_referencing_secret("tenant-a", "storage", "shared-rpc-auth"), + tenant_referencing_secret("tenant-b", "storage", "shared-rpc-auth"), + tenant_referencing_secret("tenant-other-namespace", "other", "shared-rpc-auth"), + tenant_referencing_secret("tenant-other-secret", "storage", "different-secret"), + ]; + + let refs = tenant_refs_for_secret_from_tenants(&secret, &tenants); + + assert_eq!(refs.len(), 2); + assert_eq!(refs[0], ObjectRef::new("tenant-a").within("storage")); + assert_eq!(refs[1], ObjectRef::new("tenant-b").within("storage")); } #[test] @@ -859,6 +902,24 @@ mod controller_watch_tests { } } + fn tenant_referencing_secret(name: &str, namespace: &str, secret_name: &str) -> Arc { + Arc::new(Tenant { + metadata: metav1::ObjectMeta { + name: Some(name.to_string()), + namespace: Some(namespace.to_string()), + ..Default::default() + }, + spec: TenantSpec { + rpc_secret: Some(RpcSecretRef { + name: secret_name.to_string(), + key: "rpc-secret".to_string(), + }), + ..Default::default() + }, + status: None, + }) + } + fn assert_single_ref(refs: &[ObjectRef], name: &str, namespace: &str) { assert_eq!(refs.len(), 1); assert_eq!(refs[0].name, name); diff --git a/src/reconcile.rs b/src/reconcile.rs index 9887a93..9480bb1 100755 --- a/src/reconcile.rs +++ b/src/reconcile.rs @@ -43,9 +43,7 @@ use phases::{ reconcile_services, validate_no_pool_rename, validate_tenant_prerequisites, }; use pool_lifecycle::reconcile_pool_lifecycle; -use reference_labels::{ - MISSING_REFERENCE_REQUEUE_INTERVAL, reconcile_provisioning_reference_labels, -}; +use reference_labels::{MISSING_REFERENCE_REQUEUE_INTERVAL, reconcile_policy_config_map_labels}; #[derive(Snafu, Debug)] pub enum Error { @@ -112,9 +110,9 @@ pub async fn reconcile_rustfs(tenant: Arc, ctx: Arc) -> Result< ) .await?; let action = finalize_tenant_status(&ctx, &latest_tenant, summary, tls_plan).await?; - let reference_labels = - reconcile_provisioning_reference_labels(&ctx, &latest_tenant, &ns).await?; - if reference_labels.has_missing_resources && action == Action::await_change() { + let missing_policy_config_maps = + reconcile_policy_config_map_labels(&ctx, &latest_tenant, &ns).await?; + if missing_policy_config_maps && action == Action::await_change() { return Ok(Action::requeue(MISSING_REFERENCE_REQUEUE_INTERVAL)); } Ok(action) diff --git a/src/reconcile/reference_labels.rs b/src/reconcile/reference_labels.rs index ac7b565..5119070 100644 --- a/src/reconcile/reference_labels.rs +++ b/src/reconcile/reference_labels.rs @@ -12,258 +12,103 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::{Error, object_owned_by_tenant}; +use super::Error; use crate::context::{self, Context}; use crate::types::v1alpha1::tenant::{RUSTFS_TENANT_LABEL, Tenant}; -use k8s_openapi::NamespaceResourceScope; -use k8s_openapi::api::core::v1::{ConfigMap, Secret}; +use k8s_openapi::api::core::v1::ConfigMap; use kube::api::{ListParams, Patch, PatchParams}; -use kube::{Api, Resource, ResourceExt}; -use serde::de::DeserializeOwned; +use kube::{Api, ResourceExt}; use serde_json::json; use std::collections::BTreeSet; -use std::fmt::Debug; use std::time::Duration; use tracing::{debug, info}; pub(super) const MISSING_REFERENCE_REQUEUE_INTERVAL: Duration = Duration::from_secs(60); -#[derive(Default)] -pub(super) struct ReferenceLabelReconcileResult { - pub(super) has_missing_resources: bool, -} - -enum EnsureLabelOutcome { - Present, - Labeled, - Missing, -} - -pub(super) async fn reconcile_provisioning_reference_labels( +pub(super) async fn reconcile_policy_config_map_labels( ctx: &Context, tenant: &Tenant, namespace: &str, -) -> Result { +) -> Result { let tenant_name = tenant.name(); - let desired_user_secrets = tenant - .spec - .users - .iter() - .map(|user| user.credentials_secret_name().to_string()) - .collect::>(); - let referenced_secrets = tenant.spec.referenced_secret_names(); - let desired_policy_config_maps = tenant + let desired_names = tenant .spec .policies .iter() .map(|policy| policy.document.config_map_key_ref.name.clone()) .collect::>(); - - let secret_api = Api::::namespaced(ctx.client.clone(), namespace); - let labeled_secrets = list_labeled_resources(&secret_api, &tenant_name).await?; - let mut result = ReferenceLabelReconcileResult { - has_missing_resources: ensure_resource_labels( - &secret_api, - "Secret", - namespace, - &tenant_name, - &desired_user_secrets, - &labeled_resource_names(&labeled_secrets), - ) - .await?, - }; - remove_stale_secret_labels( - &secret_api, - tenant, - namespace, - &referenced_secrets, - labeled_secrets, - ) - .await?; - - if !desired_policy_config_maps.is_empty() { - let config_map_api = Api::::namespaced(ctx.client.clone(), namespace); - let labeled_config_maps = list_labeled_resources(&config_map_api, &tenant_name).await?; - result.has_missing_resources |= ensure_resource_labels( - &config_map_api, - "ConfigMap", - namespace, - &tenant_name, - &desired_policy_config_maps, - &labeled_resource_names(&labeled_config_maps), - ) - .await?; + if desired_names.is_empty() { + return Ok(false); } - Ok(result) -} - -async fn list_labeled_resources(api: &Api, tenant_name: &str) -> Result, Error> -where - T: Clone + DeserializeOwned + Debug + Resource, - ::DynamicType: Default, -{ + let api = Api::::namespaced(ctx.client.clone(), namespace); let params = ListParams::default().labels(&format!("{RUSTFS_TENANT_LABEL}={tenant_name}")); - api.list(¶ms) + let labeled_names = api + .list(¶ms) .await - .map(|resources| resources.items) - .map_err(|source| context::Error::Kube { source }.into()) -} - -fn labeled_resource_names(resources: &[T]) -> BTreeSet -where - T: ResourceExt, -{ - resources.iter().map(ResourceExt::name_any).collect() -} + .map_err(|source| context::Error::Kube { source })? + .iter() + .map(ResourceExt::name_any) + .collect::>(); -async fn ensure_resource_labels( - api: &Api, - resource_kind: &str, - namespace: &str, - tenant_name: &str, - desired_names: &BTreeSet, - labeled_names: &BTreeSet, -) -> Result -where - T: Clone + DeserializeOwned + Debug + Resource, - ::DynamicType: Default, -{ let mut has_missing_resources = false; - for name in desired_names.difference(labeled_names) { - match ensure_resource_label(api, name, tenant_name).await? { - EnsureLabelOutcome::Present => {} - EnsureLabelOutcome::Labeled => { - info!( - namespace, - tenant = tenant_name, - resource_kind, - resource = name, - "labeled provisioning reference for Tenant watch" - ); - } - EnsureLabelOutcome::Missing => { + for name in desired_names.difference(&labeled_names) { + let config_map = match api.get(name).await { + Ok(config_map) => config_map, + Err(kube::Error::Api(response)) if response.code == 404 => { has_missing_resources = true; debug!( namespace, tenant = tenant_name, - resource_kind, - resource = name, - "provisioning reference is not available for Tenant watch labeling" + config_map = name, + "policy ConfigMap is not available for Tenant watch labeling" ); + continue; } - } - } - Ok(has_missing_resources) -} - -async fn ensure_resource_label( - api: &Api, - resource_name: &str, - tenant_name: &str, -) -> Result -where - T: Clone + DeserializeOwned + Debug + Resource, - ::DynamicType: Default, -{ - let resource = match api.get(resource_name).await { - Ok(resource) => resource, - Err(kube::Error::Api(response)) if response.code == 404 => { - return Ok(EnsureLabelOutcome::Missing); - } - Err(source) => return Err(context::Error::Kube { source }.into()), - }; - if resource - .meta() - .labels - .as_ref() - .and_then(|labels| labels.get(RUSTFS_TENANT_LABEL)) - .is_some_and(|owner| owner == tenant_name) - { - return Ok(EnsureLabelOutcome::Present); - } - - let mut metadata = json!({ - "labels": { - (RUSTFS_TENANT_LABEL): tenant_name, - }, - }); - if let Some(resource_version) = &resource.meta().resource_version { - metadata["resourceVersion"] = json!(resource_version); - } - let patch = json!({ "metadata": metadata }); - api.patch( - resource_name, - &PatchParams::default(), - &Patch::Merge(&patch), - ) - .await - .map_err(|source| context::Error::Kube { source })?; - Ok(EnsureLabelOutcome::Labeled) -} - -async fn remove_stale_secret_labels( - secret_api: &Api, - tenant: &Tenant, - namespace: &str, - referenced_secrets: &BTreeSet, - labeled_secrets: Vec, -) -> Result<(), Error> { - for secret in labeled_secrets { - let name = secret.name_any(); - if referenced_secrets.contains(&name) || object_owned_by_tenant(&secret.metadata, tenant) { - continue; - } + Err(source) => return Err(context::Error::Kube { source }.into()), + }; let mut metadata = json!({ "labels": { - (RUSTFS_TENANT_LABEL): null, + (RUSTFS_TENANT_LABEL): tenant_name, }, }); - if let Some(resource_version) = &secret.metadata.resource_version { + if let Some(resource_version) = &config_map.metadata.resource_version { metadata["resourceVersion"] = json!(resource_version); } let patch = json!({ "metadata": metadata }); - secret_api - .patch(&name, &PatchParams::default(), &Patch::Merge(&patch)) + api.patch(name, &PatchParams::default(), &Patch::Merge(&patch)) .await .map_err(|source| context::Error::Kube { source })?; info!( namespace, - tenant = %tenant.name(), - secret = name, - "removed stale Tenant watch label from provisioning Secret" + tenant = tenant_name, + config_map = name, + "labeled policy ConfigMap for Tenant watch" ); } - Ok(()) + + Ok(has_missing_resources) } #[cfg(test)] mod tests { use super::*; use crate::types::v1alpha1::provisioning::{ - ConfigMapKeyReference, PolicyDocumentSource, ProvisioningPolicy, ProvisioningUser, - UserCredentialsSecretRef, + ConfigMapKeyReference, PolicyDocumentSource, ProvisioningPolicy, }; - use crate::types::v1alpha1::tenant::{RpcSecretRef, TenantSpec}; + use crate::types::v1alpha1::tenant::TenantSpec; use http::{Method, Request, Response, StatusCode}; - use k8s_openapi::apimachinery::pkg::apis::meta::v1::{ObjectMeta, OwnerReference}; + use k8s_openapi::apimachinery::pkg::apis::meta::v1::ObjectMeta; use kube::{Client, client::Body}; use serde_json::Value; use std::convert::Infallible; use std::sync::{Arc, Mutex}; use tower::service_fn; - #[derive(Debug)] - struct CapturedRequest { - method: Method, - path: String, - body: Option, - } - #[tokio::test] - async fn operator_labels_current_references_and_cleans_only_stale_external_secrets() { - let captured = Arc::new(Mutex::new(Vec::::new())); + async fn operator_labels_policy_config_map_with_resource_version() { + let captured = Arc::new(Mutex::new(Vec::<(Method, String, Option)>::new())); let service_capture = captured.clone(); let service = service_fn(move |request: Request| { let service_capture = service_capture.clone(); @@ -281,121 +126,64 @@ mod tests { service_capture .lock() .expect("request capture lock should be available") - .push(CapturedRequest { - method: method.clone(), - path: path.clone(), - body, - }); + .push((method.clone(), path.clone(), body)); let response = match (method.as_str(), path.as_str()) { - ("GET", "/api/v1/namespaces/storage/secrets") => json_response( + ("GET", "/api/v1/namespaces/storage/configmaps") => json_response( StatusCode::OK, json!({ "apiVersion": "v1", - "kind": "SecretList", + "kind": "ConfigMapList", "metadata": { "resourceVersion": "50" }, - "items": [ - labeled_secret("old-user-secret", "42", None), - labeled_secret("shared-secret", "43", None), - labeled_secret( - "owned-secret", - "44", - Some(tenant_owner_reference()), - ), - ], + "items": [], }), ), - ("GET", "/api/v1/namespaces/storage/secrets/new-user-secret") => json_response( + ("GET", "/api/v1/namespaces/storage/configmaps/app-policy") => json_response( StatusCode::OK, json!({ "apiVersion": "v1", - "kind": "Secret", + "kind": "ConfigMap", "metadata": { - "name": "new-user-secret", + "name": "app-policy", "namespace": "storage", "resourceVersion": "7" } }), ), - ("PATCH", "/api/v1/namespaces/storage/secrets/new-user-secret") => { - json_response(StatusCode::OK, labeled_secret("new-user-secret", "8", None)) - } - ("PATCH", "/api/v1/namespaces/storage/secrets/old-user-secret") => { - json_response( - StatusCode::OK, - json!({ - "apiVersion": "v1", - "kind": "Secret", - "metadata": { - "name": "old-user-secret", - "namespace": "storage", - "resourceVersion": "43" - } - }), - ) - } - ("GET", "/api/v1/namespaces/storage/configmaps") => json_response( + ("PATCH", "/api/v1/namespaces/storage/configmaps/app-policy") => json_response( StatusCode::OK, json!({ "apiVersion": "v1", - "kind": "ConfigMapList", - "metadata": { "resourceVersion": "51" }, - "items": [], + "kind": "ConfigMap", + "metadata": { + "name": "app-policy", + "namespace": "storage", + "resourceVersion": "8", + "labels": { "rustfs.tenant": "tenant-a" } + } }), ), - ("GET", "/api/v1/namespaces/storage/configmaps/policy-config") => { - json_response( - StatusCode::OK, - json!({ - "apiVersion": "v1", - "kind": "ConfigMap", - "metadata": { - "name": "policy-config", - "namespace": "storage", - "resourceVersion": "9" - } - }), - ) - } - ("PATCH", "/api/v1/namespaces/storage/configmaps/policy-config") => { - json_response( - StatusCode::OK, - json!({ - "apiVersion": "v1", - "kind": "ConfigMap", - "metadata": { - "name": "policy-config", - "namespace": "storage", - "resourceVersion": "10", - "labels": { "rustfs.tenant": "tenant-a" } - } - }), - ) - } _ => panic!("unexpected Kubernetes request: {method} {path}"), }; Ok::<_, Infallible>(response) } }); let ctx = Context::new(Client::new(service, "default")); - let tenant = tenant_with_user_and_shared_rpc_secret(); - let result = reconcile_provisioning_reference_labels(&ctx, &tenant, "storage") + let missing = reconcile_policy_config_map_labels(&ctx, &tenant(), "storage") .await - .expect("reference labels should reconcile"); + .expect("ConfigMap label should reconcile"); - assert!(!result.has_missing_resources); + assert!(!missing); let captured = captured .lock() .expect("request capture lock should be available"); - let new_secret_patch = captured + let patch = captured .iter() - .find(|request| { - request.method == Method::PATCH && request.path.ends_with("/new-user-secret") - }) - .expect("new user Secret should be labeled"); + .find(|(method, _, _)| method == Method::PATCH) + .expect("policy ConfigMap should be patched"); assert_eq!( - new_secret_patch.body, + patch.2, Some(json!({ "metadata": { "resourceVersion": "7", @@ -403,152 +191,21 @@ mod tests { } })) ); - let stale_secret_patch = captured - .iter() - .find(|request| { - request.method == Method::PATCH && request.path.ends_with("/old-user-secret") - }) - .expect("stale user Secret label should be removed"); - assert_eq!( - stale_secret_patch.body, - Some(json!({ - "metadata": { - "resourceVersion": "42", - "labels": { "rustfs.tenant": null } - } - })) - ); - assert_eq!( - captured - .iter() - .filter(|request| request.method == Method::PATCH) - .count(), - 3, - "shared and Tenant-owned Secrets must retain their labels" - ); - let policy_config_map_patch = captured - .iter() - .find(|request| { - request.method == Method::PATCH && request.path.ends_with("/policy-config") - }) - .expect("policy ConfigMap should be labeled"); - assert_eq!( - policy_config_map_patch.body, - Some(json!({ - "metadata": { - "resourceVersion": "9", - "labels": { "rustfs.tenant": "tenant-a" } - } - })) - ); } - #[tokio::test] - async fn missing_reference_requests_a_follow_up_without_failing_reconcile() { - let service = service_fn(move |request: Request| async move { - let method = request.method().clone(); - let path = request.uri().path().to_string(); - let response = match (method.as_str(), path.as_str()) { - ("GET", "/api/v1/namespaces/storage/secrets") => json_response( - StatusCode::OK, - json!({ - "apiVersion": "v1", - "kind": "SecretList", - "metadata": { "resourceVersion": "1" }, - "items": [], - }), - ), - ("GET", "/api/v1/namespaces/storage/secrets/new-user-secret") => json_response( - StatusCode::NOT_FOUND, - json!({ - "apiVersion": "v1", - "kind": "Status", - "status": "Failure", - "reason": "NotFound", - "code": 404 - }), - ), - _ => panic!("unexpected Kubernetes request: {method} {path}"), - }; - Ok::<_, Infallible>(response) - }); - let ctx = Context::new(Client::new(service, "default")); - let mut tenant = tenant_with_user_and_shared_rpc_secret(); - tenant.spec.rpc_secret = None; - tenant.spec.policies.clear(); - - let result = reconcile_provisioning_reference_labels(&ctx, &tenant, "storage") - .await - .expect("a missing provisioning reference is a pending condition"); - - assert!(result.has_missing_resources); - } - - #[tokio::test] - async fn stale_label_conflict_is_returned_for_controller_retry() { - let service = service_fn(move |_request: Request| async move { - Ok::<_, Infallible>(json_response( - StatusCode::CONFLICT, - json!({ - "apiVersion": "v1", - "kind": "Status", - "status": "Failure", - "reason": "Conflict", - "code": 409 - }), - )) - }); - let secret_api = Api::::namespaced(Client::new(service, "default"), "storage"); - let tenant = tenant_with_user_and_shared_rpc_secret(); - let stale_secret: Secret = - serde_json::from_value(labeled_secret("old-user-secret", "42", None)) - .expect("Secret should deserialize"); - - let error = remove_stale_secret_labels( - &secret_api, - &tenant, - "storage", - &BTreeSet::new(), - vec![stale_secret], - ) - .await - .expect_err("resourceVersion conflicts must be retried by the controller"); - - assert!(matches!( - error, - Error::Context { - source: context::Error::Kube { - source: kube::Error::Api(response), - }, - } if response.code == 409 - )); - } - - fn tenant_with_user_and_shared_rpc_secret() -> Tenant { + fn tenant() -> Tenant { Tenant { metadata: ObjectMeta { name: Some("tenant-a".to_string()), namespace: Some("storage".to_string()), - uid: Some("tenant-uid".to_string()), ..Default::default() }, spec: TenantSpec { - users: vec![ProvisioningUser { - name: "app-user".to_string(), - creds_secret: Some(UserCredentialsSecretRef { - name: "new-user-secret".to_string(), - }), - ..Default::default() - }], - rpc_secret: Some(RpcSecretRef { - name: "shared-secret".to_string(), - key: "rpc-secret".to_string(), - }), policies: vec![ProvisioningPolicy { name: "readwrite".to_string(), document: PolicyDocumentSource { config_map_key_ref: ConfigMapKeyReference { - name: "policy-config".to_string(), + name: "app-policy".to_string(), key: "policy.json".to_string(), }, }, @@ -560,31 +217,6 @@ mod tests { } } - fn tenant_owner_reference() -> OwnerReference { - OwnerReference { - api_version: "rustfs.com/v1alpha1".to_string(), - kind: "Tenant".to_string(), - name: "tenant-a".to_string(), - uid: "tenant-uid".to_string(), - controller: Some(true), - block_owner_deletion: Some(true), - } - } - - fn labeled_secret(name: &str, resource_version: &str, owner: Option) -> Value { - json!({ - "apiVersion": "v1", - "kind": "Secret", - "metadata": { - "name": name, - "namespace": "storage", - "resourceVersion": resource_version, - "labels": { "rustfs.tenant": "tenant-a" }, - "ownerReferences": owner.into_iter().collect::>(), - } - }) - } - fn json_response(status: StatusCode, value: Value) -> Response { Response::builder() .status(status) diff --git a/src/reconcile/tls.rs b/src/reconcile/tls.rs index 5de6d6d..e2b384e 100644 --- a/src/reconcile/tls.rs +++ b/src/reconcile/tls.rs @@ -20,7 +20,7 @@ use crate::types::v1alpha1::status::Reason; use crate::types::v1alpha1::status::certificate::{ CertificateObjectRef, SecretStatusRef, TlsCertificateStatus, TlsServerCertificateStatus, }; -use crate::types::v1alpha1::tenant::Tenant; +use crate::types::v1alpha1::tenant::{RUSTFS_TENANT_LABEL, Tenant}; use crate::types::v1alpha1::tls::{ CaTrustSource, CertManagerIssuerRef, CertManagerTlsConfig, SecretKeyReference, TlsCertificateConfig, TlsConfig, TlsMode, TlsPlan, TlsRotationStrategy, @@ -954,9 +954,11 @@ fn build_cert_manager_certificate( if let Some(private_key) = cert_manager.private_key.as_ref() { spec.insert("privateKey".to_string(), json!(private_key)); } + let mut secret_labels = tenant.common_labels(); + secret_labels.remove(RUSTFS_TENANT_LABEL); spec.insert( "secretTemplate".to_string(), - json!({ "labels": tenant.common_labels() }), + json!({ "labels": secret_labels }), ); let resource = certificate_api_resource(); @@ -2471,7 +2473,7 @@ S2+cuFyHX+xgTPNxiG9zUDrgtXds/63ePISjIADAUvsmI97k96E6jdcgB9MmWdJj certificate .data .pointer("/spec/secretTemplate/labels/rustfs.tenant"), - Some(&serde_json::json!("tenant-a")) + None ); assert_eq!( certificate diff --git a/src/types/v1alpha1/tenant.rs b/src/types/v1alpha1/tenant.rs index 75bd40c..8540636 100755 --- a/src/types/v1alpha1/tenant.rs +++ b/src/types/v1alpha1/tenant.rs @@ -187,9 +187,8 @@ pub struct TenantSpec { /// When configured, the operator maps this key to `RUSTFS_RPC_SECRET` for every /// RustFS Pod. Keep it stable while rotating `credsSecret`. When omitted, the /// operator does not set `RUSTFS_RPC_SECRET`; RustFS resolves it from its own - /// credential configuration. Label an externally managed Secret with - /// `rustfs.tenant=` so Secret updates enqueue the Tenant for - /// prompt revalidation. + /// credential configuration. Secret updates enqueue every Tenant that references + /// the Secret, so one externally managed Secret can be shared by multiple Tenants. #[serde(default, skip_serializing_if = "Option::is_none")] pub rpc_secret: Option,