Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 6 additions & 9 deletions deploy/rustfs-operator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,9 +100,6 @@ kind: Secret
metadata:
name: rustfs-rpc-auth
namespace: storage
labels:
# Lets Secret updates enqueue this Tenant for prompt revalidation.
rustfs.tenant: rustfs-a
type: Opaque
stringData:
rpc-secret: "replace-with-a-dedicated-rpc-secret"
Expand All @@ -124,11 +121,11 @@ The operator maps the selected Secret key to `RUSTFS_RPC_SECRET` in every RustFS
Pod. Before applying workloads, it verifies that the Secret and selected key
exist and that the value is valid UTF-8, non-blank, contains no NUL bytes, and is
not the RustFS default credential value (`rustfsadmin`). Keep this value stable
while rotating administrator credentials. The `rustfs.tenant` label is not used
for authorization; it lets updates to an externally managed Secret enqueue the
Tenant for revalidation. A Secret update does not change the environment of
already-running Pods. Coordinated restart and hot reload are outside this
feature. If `spec.rpcSecret` is omitted, the operator does not set
while rotating administrator credentials. Secret updates enqueue every Tenant
whose spec references the Secret, including when multiple Tenants share it. A
Secret update does not change the environment of already-running Pods.
Coordinated restart and hot reload are outside this feature. If `spec.rpcSecret`
is omitted, the operator does not set
`RUSTFS_RPC_SECRET`, RustFS resolves it from its own credential configuration,
and the operator does not report `RpcAuthReady` for that unmanaged value.

Expand Down Expand Up @@ -159,7 +156,7 @@ spec:
objectLock: true
```

Policy ConfigMaps and user Secrets must live in the Tenant namespace. `users[].credsSecret.name` selects the credentials Secret; when omitted, the operator falls back to a Secret named after `users[].name` for compatibility with existing manifests. The operator maintains `rustfs.tenant=<tenant-name>` on referenced policy ConfigMaps and user Secrets so changes enqueue the owning Tenant; references that do not exist yet are retried. The label triggers reconciliation but does not select the Secret. Provisioned resources are retained when removed from the Tenant spec.
Policy ConfigMaps and user Secrets must live in the Tenant namespace. `users[].credsSecret.name` selects the credentials Secret; when omitted, the operator falls back to a Secret named after `users[].name` for compatibility with existing manifests. The operator maintains `rustfs.tenant=<tenant-name>` on referenced policy ConfigMaps. Secret updates are matched against every Tenant spec, so a Secret can be shared by multiple Tenants without a routing label. Legacy routing labels are removed from external Secrets when the operator starts. Provisioned resources are retained when removed from the Tenant spec.

### RBAC Configuration

Expand Down
5 changes: 2 additions & 3 deletions deploy/rustfs-operator/crds/tenant-crd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1417,9 +1417,8 @@ spec:
When configured, the operator maps this key to `RUSTFS_RPC_SECRET` for every
RustFS Pod. Keep it stable while rotating `credsSecret`. When omitted, the
operator does not set `RUSTFS_RPC_SECRET`; RustFS resolves it from its own
credential configuration. Label an externally managed Secret with
`rustfs.tenant=<Tenant name>` so Secret updates enqueue the Tenant for
prompt revalidation.
credential configuration. Secret updates enqueue every Tenant that references
the Secret, so one externally managed Secret can be shared by multiple Tenants.
nullable: true
properties:
key:
Expand Down
5 changes: 2 additions & 3 deletions deploy/rustfs-operator/crds/tenant.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1417,9 +1417,8 @@ spec:
When configured, the operator maps this key to `RUSTFS_RPC_SECRET` for every
RustFS Pod. Keep it stable while rotating `credsSecret`. When omitted, the
operator does not set `RUSTFS_RPC_SECRET`; RustFS resolves it from its own
credential configuration. Label an externally managed Secret with
`rustfs.tenant=<Tenant name>` so Secret updates enqueue the Tenant for
prompt revalidation.
credential configuration. Secret updates enqueue every Tenant that references
the Secret, so one externally managed Secret can be shared by multiple Tenants.
nullable: true
properties:
key:
Expand Down
16 changes: 6 additions & 10 deletions docs/operator-user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -331,16 +331,14 @@ Credential priority:

For production, configure `spec.rpcSecret` so internode RPC authentication does
not depend on the administrator credentials. Keep the Secret in the Tenant
namespace and label externally managed Secrets with the Tenant name:
namespace. The Secret does not need a Tenant routing label:

```yaml
apiVersion: v1
kind: Secret
metadata:
name: rustfs-rpc-auth
namespace: storage
labels:
rustfs.tenant: rustfs-a
type: Opaque
stringData:
rpc-secret: "replace-with-a-dedicated-rpc-secret"
Expand All @@ -358,9 +356,9 @@ spec:
```

The selected value must be valid UTF-8, non-blank, contain no NUL bytes, and
must not be `rustfsadmin`. The `rustfs.tenant` label lets Secret updates enqueue
the Tenant for prompt revalidation; it is not an authorization mechanism. When
the configured Secret passes validation, the Operator reports
must not be `rustfsadmin`. Secret updates enqueue every Tenant whose spec
references that Secret, including when multiple Tenants share it. When the
configured Secret passes validation, the Operator reports
`RpcAuthReady=True`. Updating the Secret does not change the environment of
already-running Pods; coordinated restart and hot reload are outside this
feature.
Expand Down Expand Up @@ -598,13 +596,13 @@ The operator can create RustFS policies, users, and buckets after the Tenant wor
- `spec.users` for regular users. Each user must have at least one direct policy mapping.
- `spec.buckets` for buckets and optional object lock.

ConfigMaps and user Secrets must live in the Tenant namespace. The operator maintains `rustfs.tenant=<tenant-name>` on referenced policy ConfigMaps and user Secrets so updates enqueue the owning Tenant. References that do not exist yet are retried after they are created.
ConfigMaps and user Secrets must live in the Tenant namespace. The operator maintains `rustfs.tenant=<tenant-name>` on referenced policy ConfigMaps. Secret updates are matched against every Tenant spec, so a Secret can be shared by multiple Tenants without a routing label. Legacy routing labels are removed from external Secrets when the operator starts.

Policy documents are parsed by RustFS. Use S3 ARN resource patterns such as `arn:aws:s3:::bucket` and `arn:aws:s3:::bucket/*`; for all buckets, use `arn:aws:s3:::*`. A bare `Resource: "*"` is not accepted by RustFS policy parsing.

For each `spec.users[]` entry, set `credsSecret.name` to select a user credentials Secret in the Tenant namespace. If `credsSecret` is omitted, the operator reads a Secret with the same name as `user.name`, preserving the behavior of older manifests. An explicit reference is authoritative and does not fall back to the same-name Secret when it is invalid or missing. Each resolved Secret name must be unique within the Tenant; the API rejects duplicate references, and reconciliation also blocks them when an older installed CRD does not enforce the validation. Distinct Secrets must also contain distinct `accesskey` values. Reconciliation validates all user credentials before modifying any RustFS user and rejects every colliding entry.

The Secret must contain `accesskey` and `secretkey`, or the MinIO-compatible keys `CONSOLE_ACCESS_KEY` and `CONSOLE_SECRET_KEY`. If both key formats are present, their values must match. `user.name` remains the declarative and status identity; the Secret's `accesskey` is the actual RustFS user. User access keys must be at least 8 characters and must not contain whitespace, `=`, or `,`; user secret keys must be at least 8 characters. The `rustfs.tenant` label only causes Secret updates to enqueue the Tenant; it does not select which Secret is read.
The Secret must contain `accesskey` and `secretkey`, or the MinIO-compatible keys `CONSOLE_ACCESS_KEY` and `CONSOLE_SECRET_KEY`. If both key formats are present, their values must match. `user.name` remains the declarative and status identity; the Secret's `accesskey` is the actual RustFS user. User access keys must be at least 8 characters and must not contain whitespace, `=`, or `,`; user secret keys must be at least 8 characters. Secret selection and event routing both come from the Tenant spec; no Tenant label is required on the Secret.

Updating a user Secret's `secretkey`, or changing `credsSecret.name` to another Secret with the same `accesskey`, rotates that RustFS user's credential. The `accesskey` is immutable after the first successful reconciliation; use a new user entry and Secret when it must change, then migrate clients before removing the old entry.

Expand Down Expand Up @@ -634,8 +632,6 @@ kind: Secret
metadata:
name: rustfs-user-app-user
namespace: storage
labels:
rustfs.tenant: rustfs-a
type: Opaque
stringData:
accesskey: appuser01
Expand Down
15 changes: 5 additions & 10 deletions docs/operator-user-guide.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -332,17 +332,14 @@ spec:
#### 独立的节点间 RPC Secret

生产环境建议配置 `spec.rpcSecret`,避免节点间 RPC 认证依赖管理员凭据。
Secret 必须与 Tenant 位于同一 namespace;若 Secret 由外部系统管理,请添加
Tenant 标签:
Secret 必须与 Tenant 位于同一 namespace,不需要 Tenant 路由标签:

```yaml
apiVersion: v1
kind: Secret
metadata:
name: rustfs-rpc-auth
namespace: storage
labels:
rustfs.tenant: rustfs-a
type: Opaque
stringData:
rpc-secret: "replace-with-a-dedicated-rpc-secret"
Expand All @@ -360,8 +357,8 @@ spec:
```

所选值必须是有效 UTF-8、不能为空、不能包含 NUL 字节,并且不能是
`rustfsadmin`。`rustfs.tenant` 标签仅用于在 Secret 变化时及时触发 Tenant
重新校验,不是授权机制。配置的 Secret 校验通过后,Operator 会报告
`rustfsadmin`。Secret 更新会触发 spec 中引用它的所有 Tenant,包括多个 Tenant
共享同一个 Secret 的情况。配置的 Secret 校验通过后,Operator 会报告
`RpcAuthReady=True`。更新 Secret 不会改变已运行 Pod 的进程环境;协调重启和
热加载不在此功能范围内。

Expand Down Expand Up @@ -598,13 +595,13 @@ Operator 可以在 Tenant workload Ready 后自动创建 RustFS policy、user
- `spec.users`:普通用户。每个 user 必须至少直接绑定一个 policy。
- `spec.buckets`:bucket,可选择开启 object lock。

ConfigMap 和 user Secret 必须位于 Tenant namespace。operator 会在被引用的 policy ConfigMap 和 user Secret 上维护 `rustfs.tenant=<tenant-name>` label,使资源变化能够触发 owning Tenant reconcile;尚未创建的引用资源会在创建后重试加 label。
ConfigMap 和 user Secret 必须位于 Tenant namespace。operator 仍会在被引用的 policy ConfigMap 上维护 `rustfs.tenant=<tenant-name>` label。Secret 更新会按所有 Tenant spec 做反向匹配,因此同一个 Secret 可以被多个 Tenant 共享,不需要路由 label。operator 启动时会清理外部 Secret 上遗留的路由 label。

Policy document 由 RustFS 解析。请使用 `arn:aws:s3:::bucket` 和 `arn:aws:s3:::bucket/*` 这类 S3 ARN resource 写法;如需匹配所有 bucket,请使用 `arn:aws:s3:::*`。RustFS policy parser 不接受裸 `Resource: "*"`。

每个 `spec.users[]` 条目都可以通过 `credsSecret.name` 指定 Tenant namespace 中的 user credentials Secret。省略 `credsSecret` 时,Operator 继续读取与 `user.name` 同名的 Secret,以兼容旧版 manifest。显式引用是唯一来源;配置错误或 Secret 不存在时,不会再回退到同名 Secret。同一 Tenant 内解析后的 Secret 名称必须唯一;API 会拒绝重复引用,而在集群仍安装旧版 CRD、尚未启用该校验时,reconcile 也会阻止这些重复配置生效。不同 Secret 中的 `accesskey` 也必须唯一;reconcile 会先校验全部 user credentials,再修改任何 RustFS user,并拒绝所有冲突条目。

Secret 必须包含 `accesskey` 和 `secretkey`,或者 MinIO 兼容 key:`CONSOLE_ACCESS_KEY` 和 `CONSOLE_SECRET_KEY`。如果两种 key 同时存在,值必须一致。`user.name` 仍是声明和 status 中的逻辑标识,Secret 内的 `accesskey` 才是实际 RustFS user。user access key 至少 8 个字符,且不能包含空白、`=` 或 `,`;user secret key 至少 8 个字符。`rustfs.tenant` label 只负责在 Secret 更新时触发 Tenant reconcile,不参与选择要读取的 Secret。
Secret 必须包含 `accesskey` 和 `secretkey`,或者 MinIO 兼容 key:`CONSOLE_ACCESS_KEY` 和 `CONSOLE_SECRET_KEY`。如果两种 key 同时存在,值必须一致。`user.name` 仍是声明和 status 中的逻辑标识,Secret 内的 `accesskey` 才是实际 RustFS user。user access key 至少 8 个字符,且不能包含空白、`=` 或 `,`;user secret key 至少 8 个字符。Secret 的选择和事件路由都来自 Tenant spec,不需要在 Secret 上添加 Tenant label

更新 user Secret 的 `secretkey`,或者把 `credsSecret.name` 切换到具有相同 `accesskey` 的另一个 Secret,都会轮换对应 RustFS user 的凭据。首次成功 provisioning 后,`accesskey` 不可变;如需变更,请新建 user 条目和 Secret,迁移客户端后再移除旧条目。

Expand Down Expand Up @@ -634,8 +631,6 @@ kind: Secret
metadata:
name: rustfs-user-app-user
namespace: storage
labels:
rustfs.tenant: rustfs-a
type: Opaque
stringData:
accesskey: appuser01
Expand Down
25 changes: 1 addition & 24 deletions e2e/src/framework/cert_manager_tls.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,6 @@ const OPAQUE_SECRET_TYPE: &str = "Opaque";
const REDACTED_FIXTURE_BYTES: &[u8] = b"redacted-test-fixture";
const MANAGED_CERTIFICATE_CASE_SUFFIX: &str = "cert-manager-managed";
const EXTERNAL_SECRET_CASE_SUFFIX: &str = "cert-manager-external";
const RUSTFS_TENANT_LABEL: &str = "rustfs.tenant";
pub const POSITIVE_CERT_MANAGER_TLS_TIMEOUT: Duration = Duration::from_secs(600);

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
Expand Down Expand Up @@ -88,10 +87,6 @@ pub fn positive_cert_manager_tls_timeout(config: &E2eConfig) -> Duration {
std::cmp::max(config.timeout, POSITIVE_CERT_MANAGER_TLS_TIMEOUT)
}

fn tenant_watch_labels(config: &E2eConfig) -> BTreeMap<String, String> {
BTreeMap::from([(RUSTFS_TENANT_LABEL.to_string(), config.tenant_name.clone())])
}

pub fn managed_certificate_storage_layout(config: &E2eConfig) -> storage::LocalStorageLayout {
positive_tls_storage_layout(config, MANAGED_CERTIFICATE_CASE_SUFFIX)
}
Expand Down Expand Up @@ -1047,13 +1042,7 @@ fn external_tls_secret_manifest(
secret_type: &str,
data: BTreeMap<String, ByteString>,
) -> Result<String> {
tls_secret_manifest_with_labels(
&config.test_namespace,
name,
secret_type,
data,
tenant_watch_labels(config),
)
tls_secret_manifest(&config.test_namespace, name, secret_type, data)
}

fn tls_secret_manifest(
Expand All @@ -1062,22 +1051,10 @@ fn tls_secret_manifest(
secret_type: &str,
data: BTreeMap<String, ByteString>,
) -> Result<String> {
tls_secret_manifest_with_labels(namespace, name, secret_type, data, BTreeMap::new())
}

fn tls_secret_manifest_with_labels(
namespace: &str,
name: &str,
secret_type: &str,
data: BTreeMap<String, ByteString>,
labels: BTreeMap<String, String>,
) -> Result<String> {
let labels = (!labels.is_empty()).then_some(labels);
let secret = corev1::Secret {
metadata: metav1::ObjectMeta {
name: Some(name.to_string()),
namespace: Some(namespace.to_string()),
labels,
..Default::default()
},
type_: Some(secret_type.to_string()),
Expand Down
7 changes: 2 additions & 5 deletions e2e/src/framework/resources.rs
Original file line number Diff line number Diff line change
Expand Up @@ -80,15 +80,12 @@ kind: Secret
metadata:
name: {secret_name}
namespace: {namespace}
labels:
rustfs.tenant: {tenant_name}
type: Opaque
stringData:
{key}: {rpc_secret}
"#,
secret_name = rpc_secret_name(config),
namespace = config.test_namespace,
tenant_name = config.tenant_name,
key = RPC_SECRET_KEY,
rpc_secret = TEST_RPC_SECRET,
)
Expand Down Expand Up @@ -372,13 +369,13 @@ mod tests {
}

#[test]
fn rpc_secret_uses_tenant_watch_label() {
fn rpc_secret_does_not_require_tenant_watch_label() {
let config = E2eConfig::defaults();
let manifest = rpc_secret_manifest(&config);

assert_eq!(rpc_secret_name(&config), "e2e-tenant-rpc-auth");
assert!(manifest.contains("namespace: rustfs-e2e-smoke"));
assert!(manifest.contains("rustfs.tenant: e2e-tenant"));
assert!(!manifest.contains("rustfs.tenant"));
assert!(manifest.contains("rpc-secret: test-dedicated-rpc-secret"));
}
}
29 changes: 10 additions & 19 deletions e2e/tests/cert_manager_tls.rs
Original file line number Diff line number Diff line change
Expand Up @@ -91,32 +91,27 @@ fn external_secret_tenant_manifest_uses_shared_secret_and_rollout_strategy() ->
}

#[test]
fn external_secret_manifests_carry_tenant_watch_label_for_initial_create_and_rotation() -> Result<()>
{
fn external_secret_manifests_need_no_tenant_watch_label_for_create_or_rotation() -> Result<()> {
let config = tls_e2e::external_secret_case_config(&E2eConfig::defaults());

let initial = tls_e2e::external_tls_secret_manifests(&config)?;
assert_secret_manifest_tenant_watch_label(
assert_secret_manifest_has_no_tenant_watch_label(
"initial external TLS Secret",
&initial.tls_secret_manifest,
&config.tenant_name,
)?;
assert_secret_manifest_tenant_watch_label(
assert_secret_manifest_has_no_tenant_watch_label(
"initial external CA Secret",
&initial.ca_secret_manifest,
&config.tenant_name,
)?;

let rotated = tls_e2e::external_tls_secret_manifests(&config)?;
assert_secret_manifest_tenant_watch_label(
assert_secret_manifest_has_no_tenant_watch_label(
"rotated external TLS Secret",
&rotated.tls_secret_manifest,
&config.tenant_name,
)?;
assert_secret_manifest_tenant_watch_label(
assert_secret_manifest_has_no_tenant_watch_label(
"rotated external CA Secret",
&rotated.ca_secret_manifest,
&config.tenant_name,
)?;

Ok(())
Expand Down Expand Up @@ -1062,21 +1057,17 @@ fn collect_tls_artifacts_on_error(config: &E2eConfig, case_name: &str, result: &
}
}

fn assert_secret_manifest_tenant_watch_label(
context: &str,
manifest: &str,
expected_tenant: &str,
) -> Result<()> {
fn assert_secret_manifest_has_no_tenant_watch_label(context: &str, manifest: &str) -> Result<()> {
let secret: corev1::Secret = serde_yaml_ng::from_str(manifest)
.with_context(|| format!("parse {context} Secret manifest"))?;
let labels = secret
let has_tenant_label = secret
.metadata
.labels
.as_ref()
.with_context(|| format!("{context} missing labels"))?;
.is_some_and(|labels| labels.contains_key("rustfs.tenant"));
ensure!(
labels.get("rustfs.tenant").map(String::as_str) == Some(expected_tenant),
"{context} should carry rustfs.tenant label for Tenant {expected_tenant}"
!has_tenant_label,
"{context} should not carry a rustfs.tenant routing label"
);
Ok(())
}
Expand Down
2 changes: 1 addition & 1 deletion examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ kubectl apply -f examples/provisioning-tenant.yaml
kubectl wait --for=condition=Ready tenant/provisioning-demo --timeout=10m
```

The operator maintains `rustfs.tenant=<tenant-name>` on referenced policy ConfigMaps and user Secrets so updates enqueue the owning Tenant.
The operator labels referenced policy ConfigMaps. Secret updates are matched against every Tenant spec, so one Secret can be shared without a routing label.

---

Expand Down
4 changes: 2 additions & 2 deletions examples/cert-manager-ca-trust-tenant.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@
# - enableInternodeHttps=true requires a CA source. CertificateSecretCa expects
# cert-manager to write ca.crt into the TLS Secret. Use SecretRef or SystemCa
# only when that matches your trust model.
# - Existing cert-manager Secrets used with manageCertificate=false should carry
# rustfs.tenant=<tenant-name> if you want Secret changes to enqueue the Tenant.
# - Existing cert-manager Secrets used with manageCertificate=false need no Tenant
# routing label; Secret changes enqueue every Tenant that references them.
apiVersion: rustfs.com/v1alpha1
kind: Tenant
metadata:
Expand Down
Loading
Loading