Skip to content

fix(iac/azure): let the runtime identity write the admin-password secret, and log sync failures as errors - #1309

Merged
IAmJSD merged 4 commits into
LeanerCloud:mainfrom
IAmJSD:fix/issue-199
Oct 26, 2026
Merged

IAmJSD merged 4 commits into
LeanerCloud:mainfrom
IAmJSD:fix/issue-199

Conversation

@IAmJSD

@IAmJSD IAmJSD commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

Root cause

Both Azure compute paths set ADMIN_PASSWORD_SECRET and ADMIN_EMAIL, which arms the admin-password sync (buildAdminPasswordSyncCallback, internal/server/app.go). When the admin changes their password, the sync calls azsecrets.SetSecret. The only Key Vault grant either runtime identity holds is the vault-wide Key Vault Secrets User, which is read-only, so the write gets a 403. The callback logged that at WARN and carried on. The password change had already been committed to the database, so the vault copy silently drifted.

On current main (a14dc8f2) both halves still reproduce:

Fix

I took the issue's option 2 in spirit: an explicit write grant scoped as narrowly as Azure allows.

  • modules/compute/azure/container-apps (admin_password_secret_writer) and modules/compute/azure/aks (workload_admin_password_secret_writer) each add an azurerm_role_assignment of Key Vault Secrets Officer:
    • Scope: <key_vault_id>/secrets/<admin_password_secret_name>, which is that one secret, not the vault. Key Vault RBAC supports secret-level role assignments. No built-in role is set-only, so delete/backup/restore on this one secret come with it, and nothing on any other secret. The vault-wide grant stays read-only.
    • Principal: the identity the app actually authenticates as. That is the Container App UAI (AZURE_CLIENT_ID) or the AKS workload UAI (federated to the pod service account).
    • Count: admin_password_secret_name != "" && admin_email != "", the same condition that arms the Go callback. The grant exists only when something will use it.
    • Plan-time stable: the scope is built from var.key_vault_id plus the secret name rather than the secret's resource_versionless_id. scope is ForceNew, and an apply-time-only value would replace the grant (the #1802 pattern).
    • admin_password_secret_name gains a validation that it must be a bare Key Vault secret name (^[0-9A-Za-z-]{1,127}$), so a path or ID cannot move the scope off the single secret.
  • Bootstrap vs runtime (CLAUDE.md): both are runtime grants in the per-cloud compute modules. The deploy SP already holds Microsoft.Authorization/roleAssignments/write (ci-cd-permissions/locals_data.tf), the same permission it uses for the existing Secrets User, Crypto User and AcrPull assignments. The bootstrap does not need re-applying, and I added no fallback.
  • Fail loudly (internal/server/app.go):
    • A failed PutSecret now logs at ERROR, naming the secret and stating it no longer matches the database password, with a hint to check the runtime identity's write access. Before, this was a WARN.
    • A failed user lookup used to return silently. It now also logs an ERROR, because the callback cannot tell whether the admin's copy just went stale.
    • The password is never logged, and success still logs at INFO.
    • I did not change the HTTP behavior: the password change is already committed, so failing the request would misreport it.

Deploy pause

Azure deploys have been paused since 2026-10-20 (deploy-azure workflows disabled). The Terraform half takes effect only on the next Azure deploy after the pause lifts. The PR is otherwise merge-ready, and merging it deploys nothing. The Go logging change ships with the next image build.

Closes #199

Verification

All of the following is unit, mock and fixture evidence. Nothing was planned or applied against a real subscription, and no real credentials were used.

Go (fail-before/pass-after): new tests in internal/server/app_coverage_test.go:

  • TestBuildAdminPasswordSyncCallback_PutSecretFailureLogsError (ERROR names the secret and the cause, no WARN, no false success line, password not logged)
  • TestBuildAdminPasswordSyncCallback_UserLookupFailureLogsError
  • TestBuildAdminPasswordSyncCallback_QuietPathsLogNoError (success, non-admin and missing user log no ERROR)
# with internal/server/app.go reverted to main:
go test ./internal/server/ -count=1 -run TestBuildAdminPasswordSyncCallback
--- FAIL: TestBuildAdminPasswordSyncCallback_PutSecretFailureLogsError    app_coverage_test.go:146: Expected 1, got 0
--- FAIL: TestBuildAdminPasswordSyncCallback_UserLookupFailureLogsError   app_coverage_test.go:171: Expected 1, got 0
# with the fix: all 9 TestBuildAdminPasswordSyncCallback_* PASS
go test ./internal/server/ -count=1          # ok
go test ./internal/mocks/ -count=1           # ok
go build ./... && go vet ./... && go vet -tags integration ./...   # ok
golangci-lint v2.10.1 run --allow-parallel-runners ./internal/server/   # 0 issues
gocyclo -over 10 internal/server/app.go      # clean

Terraform (mocked terraform test, fail-before/pass-after): new tests/admin_password_writer.tftest.hcl in both modules.

  • Both directions: with sync armed, exactly one grant scoped to <vault>/secrets/admin-password with Secrets Officer. Container-apps also asserts the principal is the app's UAI and that the secret matches the container's ADMIN_PASSWORD_SECRET. AKS asserts the same for the pod spec, and that the vault-wide grant is still Secrets User at vault scope.
  • With no admin email or no secret name there is no grant, and a path-shaped secret name is rejected.
terraform -chdir=terraform/modules/compute/azure/<m> init -backend=false -lockfile=readonly -input=false
terraform -chdir=terraform/modules/compute/azure/<m> test
# container-apps: Success! 4 passed, 0 failed.   aks: Success! 5 passed, 0 failed.
# with main.tf/variables.tf reverted to main: both fail ("Reference to undeclared resource")

Mutation probes, each run against both modules, were all killed: vault-wide scope, dropping the admin_email half of the count gate, removing the name validation, and Secrets User instead of Officer.

Other checks:

CI wiring: a new "Test Azure compute modules" step in the terraform-validate job (azure leg) runs init/fmt/test for every terraform/modules/compute/azure/*/tests. Both modules' .terraform.lock.hcl gain linux_amd64 and darwin_amd64 hashes, following the #1143 precedent, so -lockfile=readonly init works on the Linux runner. They previously carried a single platform's h1:.

Combined-tree verification. This branch was tested together with the other queued branches on a local, never-pushed tree: queue/platform @ b56afb2b = origin/main a14dc8f2 plus these heads, merged in this order:

Branch Tested head
fix/issue-204 76f85a4
fix/issue-438 78fe011
fix/issue-838 120b07f
fix/issue-143 aef1256
fix/issue-349 cfbb551
fix/issue-80 63405f9
fix/issue-199 36b9f8f
fix/issue-41 96fc237
fix/issue-299 6af14ed
fix/issue-324 3ab6729

fix/issue-324 and fix/issue-204 both edit .github/workflows/ci.yml next to the azure-kv-access-policy job. The tree kept #324's new "Run Function URL reader self-tests" step and #204's updated job comment; whichever PR merges second takes that same resolution.

Linux, Go 1.26.9, AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null AWS_EC2_METADATA_DISABLED=true; Docker Postgres via testcontainers. The full suite ran on ed0c7174, which differs from b56afb2b only by one .gitallowed line on fix/issue-324:

  • Passed: go build ./..., go vet ./..., go vet -tags integration ./... and the full go test -race -short -count=1 ./....
  • Static checks: golangci-lint v2.10.1 found 0 issues on every changed package, with and without --build-tags integration. gocyclo -over 10 is silent on the changed non-test files.
  • Integration tests, every changed package that has them, one package at a time (-tags integration -count=1, 60-minute local timeout): internal/api, internal/purchase, internal/scheduler and internal/server all pass.
  • Secrets: the first run caught a bare placeholder account ID in fix/issue-324's new tftest, now allowlisted with an anchored .gitallowed line. Re-run at b56afb2b: git-secrets, using the repo's scripts/setup-git-secrets.sh patterns in a fresh repo with an isolated HOME, found nothing in the changed files, and a planted fake key was flagged. scripts/test-git-secrets-allowlist.sh: 137 PASS, 0 FAIL.
  • Nothing was applied to any cloud.

Gaps and things to eyeball

  • AKS principal binding is not value-asserted. Under mock providers on Terraform 1.10, the cluster's kubelet_identity block (Optional+Computed) plans as an empty list, so aks_acr cannot be evaluated. Any apply run also fails at teardown, so the AKS runs are targeted plans, where principal_id is unknown. The reference is azurerm_user_assigned_identity.workload.principal_id, the same as the read grant next to it. The container-apps test applies and asserts its principal.
  • Role dataActions are from documentation and in-tree usage, not an az role definition list response. As the issue notes, I could not query Azure. "Key Vault Secrets Officer" (Microsoft.KeyVault/vaults/secrets/*) and secret-level scopes are documented by Microsoft.
  • A narrower alternative would be a custom role with only secrets/setSecret/action. That needs a bootstrap-created role definition (roleDefinitions/write is bootstrap-only), and the runtime lookup would fail every Azure deploy until a human re-applies the bootstrap. Scoping the built-in role to one secret seemed the better trade. Happy to switch if the owner prefers the custom role.

Out of scope (noted, not changed)

  • modules/secrets/azure still carries the unused container_app_secrets_user / writeable_secrets_role grant (count = 0 in every environment). Its comment, and known_issues/18_tf_env_azure.md, describe it as the runtime's write path. Removing or rewording it is a separate cleanup.
  • The issue's option 3 (stop setting ADMIN_PASSWORD_SECRET / ADMIN_EMAIL) was not taken, because the startup path also reads ADMIN_PASSWORD_SECRET.

A maintainer, please mirror the issue's triage labels on this PR: type/bug, priority/p3, severity/low, urgency/eventually, impact/internal, effort/s, triaged.

🤖 Generated with Claude Code

…ret, and log sync failures as errors

Both Azure runtime identities held only the vault-wide, read-only "Key Vault
Secrets User", but both deployments set ADMIN_PASSWORD_SECRET and
ADMIN_EMAIL, which arms the admin-password sync. Its SetSecret call 403'd,
the failure was logged at WARN, and the Key Vault copy silently drifted from
the database password. Container Apps, which all three Azure environments
deploy, was affected as well as AKS: the secrets module's
writeable_secrets_role grant is count = 0 everywhere.

- modules/compute/azure/{container-apps,aks}: a "Key Vault Secrets Officer"
  role assignment scoped to <vault>/secrets/<admin secret> only, for the
  identity the app runs as. It exists only when the secret name and admin
  email are both set, the same condition that arms the sync callback. The
  scope is built from plan-time-known inputs because scope is ForceNew. The
  vault-wide grant stays read-only. admin_password_secret_name is now
  validated as a bare secret name so it cannot widen the scope.
- internal/server: a failed sync write, or a failed user lookup that skips
  the sync, now logs at ERROR and names the secret that diverged.
- Mocked terraform tests for both modules, run from a new CI step. The
  module lock files gain linux_amd64 and darwin_amd64 hashes so
  -lockfile=readonly init works on CI.

The deploy SP already holds roleAssignments/write, so the bootstrap does not
need re-applying.

Closes LeanerCloud#199

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@IAmJSD IAmJSD added impact/internal Team-internal only triaged Item has been triaged priority/p3 Polish / idea / may never ship severity/low Minor harm urgency/eventually No deadline effort/s Hours type/bug Defect labels Oct 23, 2026
@coderabbitai

coderabbitai Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

This review includes 10 billable files and costs up to $2.50.

Or wait 59 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
Commits

Reviewing files that changed from the base of the PR and between 2a11540 and dac5bf2.

Files selected for processing: 10
  • .github/workflows/ci.yml
  • internal/server/app.go
  • internal/server/app_coverage_test.go
  • terraform/modules/compute/azure/aks/main.tf
  • terraform/modules/compute/azure/aks/tests/admin_password_writer.tftest.hcl
  • terraform/modules/compute/azure/aks/variables.tf
  • terraform/modules/compute/azure/container-apps/main.tf
  • terraform/modules/compute/azure/container-apps/tests/admin_password_writer.tftest.hcl
  • terraform/modules/compute/azure/container-apps/variables.tf
  • terraform/modules/secrets/azure/main.tf
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Brings in main's CI fixes (LeanerCloud#1236, LeanerCloud#1252) and CI changes since, so this PR gets a fresh CI run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@IAmJSD

IAmJSD commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE 377ea275c2ce2cc3d808182707279c33da447ce5

Blocking:

  • none

Non-blocking:

  • This PR widens a privilege, and the issue explicitly asked for that to be a deliberate decision. It takes option 2 in spirit and rejects option 3 (stop arming the sync). The grant is as narrow as Azure built-ins allow: "Key Vault Secrets Officer" scoped to the single admin-password secret, and only when the sync is armed. Even so, the runtime identity can now delete or overwrite that one secret as well as read it. Overwrite is the intended capability. Delete is new, but it adds little beyond overwrite. Worth an explicit nod from the acting owner, since the issue framed it as a judgment call.
  • terraform/modules/secrets/azure/main.tf:282-299: the writeable_secrets_role comment still describes the count=0 Officer grant as the runtime's write path. Now that the real write path lives in the compute modules, that comment is misleading. The body lists it as out of scope, but a follow-up issue would keep it from being forgotten.

Attacked:

  • Grant scope and wiring:

    • ADMIN_PASSWORD_SECRET is var.admin_password_secret_name in both modules.
    • The env passes coalesce(module.secrets.admin_password_secret_name, ""), which resolves to the literal azurerm_key_vault_secret.admin_password[0].name = "admin-password". So the count is known at plan time, the scope <vault id>/secrets/admin-password is exactly the name azsecrets.SetSecret writes, and the module output gives an implicit dependency on the secret resource, so the assignment can't be created before the secret.
    • The new name validation (^[0-9A-Za-z-]{1,127}$) blocks path traversal of the scope.
    • The deploy SP already holds an unconditioned roleAssignments/write and already assigns Secrets Officer elsewhere (secrets/azure main.tf:276), so the deploy won't 403.
  • Terraform tests: init -backend=false -lockfile=readonly succeeds on linux_amd64 with the newly added h1 hashes, and fmt -check is clean. terraform test passes: container-apps 4/4, aks 5/5, all mock providers. Mutation probes:

    • scope widened to var.key_vault_id in container-apps: sync_armed_grants_write_on_that_secret_only fails;
    • admin_email dropped from the AKS count gate: no_admin_email_means_no_write_grant fails.

    Both files were restored afterwards.

  • Go fail-before: reverse-applying the app.go hunk makes TestBuildAdminPasswordSyncCallback_PutSecretFailureLogsError and ..._UserLookupFailureLogsError FAIL. They pass on the head. The password is never logged, and the HTTP behavior is unchanged.

  • Ran: go build ./...; go vet ./internal/server/ with and without -tags integration; go test -race -short -count=1 ./internal/server/ (ok); go test -race -tags integration -run AdminPassword ./internal/server/ (ok); golangci-lint with and without integration tags (0 issues); gocyclo -over 10 internal/server/app.go (clean); actionlint ci.yml; scripts/check-azure-kv-access-policy.sh and check-azure-role-parity.sh (both OK). Azure deploys are disabled_manually, so the Terraform half lands on the next Azure deploy.

Body-stale:

  • none

Reviewer: rB6. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.

@cristim

cristim commented Oct 26, 2026

Copy link
Copy Markdown
Member

Independent review 1 (cold, fresh context):

VERDICT: REQUEST_CHANGES 377ea27

Findings:

  • medium terraform/modules/secrets/azure/main.tf:97-110 (not in diff, but defeats fix(iac/azure): AKS is read-only on Key Vault, so the admin-password sync write 403s and copies silently diverge #199's outcome on the deployed path) - azurerm_key_vault_secret.admin_password has no lifecycle { ignore_changes = [value] }. azurerm's read of this resource fetches the latest secret version, so once this PR makes the app's SetSecret succeed, the next Azure terraform apply sees value drift and writes a new version with the original var/random_password value. Scenario: admin changes password -> vault updated (this PR) -> next deploy reverts the vault to the old password, silently and with no app log, so the DB and vault copies diverge again, which is exactly the "copies silently diverge" bug fix(iac/azure): AKS is read-only on Key Vault, so the admin-password sync write 403s and copies silently diverge #199 closes. AWS does not have this: aws_secretsmanager_secret_version tracks its own version_id and does not revert. Fix: add ignore_changes = [value] on the Azure admin_password secret (the app owns the value after first boot; seeding still works on create). Alternatively, the owner can explicitly accept a follow-up issue. Either way the PR's "Closes fix(iac/azure): AKS is read-only on Key Vault, so the admin-password sync write 403s and copies silently diverge #199" overstates the fix today.
  • low terraform/modules/compute/azure/aks/main.tf:219 - the AKS writer grant's principal is not covered by any test. Mutation probe: replacing it with a literal foreign GUID left all 5 AKS runs passing. The author discloses this (TF 1.10 mock plan leaves principal_id unknown, and override_during needs 1.11+). It reads correctly by inspection (same azurerm_user_assigned_identity.workload.principal_id as the read grant at :201), so it is non-blocking.
  • nit terraform/modules/secrets/azure/main.tf:281-300 and known_issues/18_tf_env_azure.md still describe the dead container_app_secrets_user (count = 0 everywhere) as the runtime write path. The author flags this as out of scope, but it now contradicts the new grant. A follow-up cleanup is worth filing.

Checked and OK:

  • Project rule (bootstrap vs runtime): both grants live in the compute modules, nothing was added to ci-cd-permissions, and there is no fallback. The deploy SP already holds Microsoft.Authorization/roleAssignments/write at subscription scope (ci-cd-permissions/locals_data.tf:13, role.tf:3), which covers a secret-level scope.
  • Scope: ${var.key_vault_id}/secrets/${name} is the documented ARM scope for per-secret Key Vault RBAC. The vault is RBAC-mode (secrets/azure/main.tf:39 enable_rbac_authorization = true), so a role assignment and not an access policy is correct. The vault-wide grant stays Secrets User (environments/azure/compute.tf:205, aks main.tf:201). The name validation ^[0-9A-Za-z-]{1,127}$ matches Key Vault's secret-name alphabet and blocks path injection into the scope.
  • Role: Key Vault Secrets Officer carries secrets/* dataActions (get/set/delete/backup/restore, plus purge where purge protection is off). It has no keys or certificates rights. Limited to one secret, that is the narrowest built-in role. A custom setSecret-only role would need bootstrap-side roleDefinitions/write, and the PR body's trade-off is sound.
  • Ordering/race: the env passes module.secrets.admin_password_secret_name, an output of azurerm_key_vault_secret.admin_password[0].name, so the grant depends on the secret existing. The write only happens on a user password change and not at startup, so RBAC propagation delay is not a race. The count gate matches the Go arming condition (app.go:986-989). The Go side does not depend on the grant at boot: resolveAdminPassword only reads.
  • Go: the password is never logged. PutSecret's error wraps the azcore ResponseError (URL plus status/body, not the request value), and the tests assert that "newpass" is absent. The lookup-error ERROR logs a user UUID and the secret name only. The HTTP behaviour is unchanged.
  • Identity: the Container App sets AZURE_CLIENT_ID to the same UAI that receives the grant (container-apps main.tf:125). The tftest asserts the principal and that the ADMIN_PASSWORD_SECRET env matches the granted secret.
  • ensureAdminUserWithPassword updates only an empty password_hash, so a reverted vault copy does not reset the DB password on restart. That bounds the medium finding to vault-copy staleness.

Attacked:

  • Scratch copy of 377ea27: terraform fmt -check, init -backend=false -lockfile=readonly and validate pass in aks, container-apps and environments/azure. terraform test (mock providers): aks 5/5 and container-apps 4/4 pass on TF 1.10.5. go test -race -count=1 ./internal/server/ ok.
  • CI at the head SHA is all green. The azure leg's new "Test Azure compute modules" step ran every azure compute module's tests (5, 4, 1, 2, 2 passed).
  • Mutation P1, container-apps scope changed to vault-wide var.key_vault_id: killed (1 failed). Restored byte-identical.
  • Mutation P2, AKS count with the admin_email half dropped: killed (1 failed). Restored byte-identical.
  • Mutation P3, AKS writer principal swapped to a foreign GUID: SURVIVED (5/5 pass), the gap disclosed above. A first attempt using the kubelet identity errored on mock evaluation, so it was inconclusive and was redone. Restored byte-identical.
  • Mutation P4, Go PutSecret failure Errorf changed to Warnf: killed (TestBuildAdminPasswordSyncCallback_PutSecretFailureLogsError). Restored byte-identical.
  • Read GetUserByID/scanUser, ensureAdminUser, the AWS/GCP secret lifecycles for parity, and the deploy SP's custom role scope.
  • No terraform plan/apply against a backend and no cloud calls.

@IAmJSD I confirmed the medium finding at 377ea27: azurerm_key_vault_secret.admin_password (terraform/modules/secrets/azure/main.tf ~97) has no lifecycle { ignore_changes = [value] }. The smallest fix is to add it in this PR. Otherwise reword "Closes #199" and file a follow-up. Azure applies are paused right now, so nothing reverts until they resume.

Bring the branch up to date with main. The only conflicts were the two
Azure compute lock files (aks, container-apps). Main's versions already
carry linux_amd64, linux_arm64, darwin_arm64 and darwin_amd64 hashes
(LeanerCloud#1439) for the same provider versions this branch pins, so main's files
were taken and re-checked with `terraform providers lock` for all four
platforms, which reported no changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Owner decision (D17, acting owner IAmJSD, 2026-10-26): keep the admin-password sync and grant the runtime identity the narrow, single-secret write that this PR adds.

Brought the branch up to date with main via a merge commit. New head: 6b3943831d39bd1c33dd91952af6760e5ba6eda4.

Conflict resolution

  • The only conflicts were terraform/modules/compute/azure/{aks,container-apps}/.terraform.lock.hcl. This branch had only added extra platform hashes there, with no provider version changes. Main's copies (after build(terraform): record linux and darwin provider hashes in every lock file, init env roots with -lockfile=readonly #1439) already carry linux_amd64, linux_arm64, darwin_arm64 and darwin_amd64 hashes for the same versions, so I took main's files and re-ran terraform providers lock for all four platforms in each root. Terraform reported no changes, and the lock files now match main exactly, so they've dropped out of this PR's diff.
  • Everything else (ci.yml, internal/server/app.go, both modules' main.tf/variables.tf/tests) merged cleanly.

Checks on 6b39438 (local)

  • go build ./..., go vet ./..., go vet -tags integration ./..., tests/e2e go vet -tags e2e: pass
  • go test ./terraform/... (including TestLockFilesRecordEveryPlatform): pass
  • golangci-lint v2.10.1 on internal/server with and without --build-tags integration: 0 issues. gocyclo -over 10 on app.go: clean
  • go test -race -short and go test -race -tags integration on internal/server: pass
  • Terraform 1.10.5, init -backend=false -lockfile=readonly + fmt -check + validate + test (mock providers only, no plan or apply against real state): aks 14 passed, container-apps 4 passed
  • actionlint on ci.yml: clean. Secrets scan: clean. scripts/test-git-secrets-allowlist.sh: 183 pass

@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE 6b3943831d39bd1c33dd91952af6760e5ba6eda4

Blocking:

  • none

Non-blocking:

Attacked:

  • Traced the wiring: environments/azure/compute.tf:81/168 passes module.secrets.admin_password_secret_name (azurerm_key_vault_secret.admin_password[0].name = "admin-password", a config literal, so it is plan-time known and the count is plannable). That reference also gives the role assignment an implicit dependency on the secret, so the secret exists before the grant. The container-apps env ADMIN_PASSWORD_SECRET/ADMIN_EMAIL (main.tf:122-123) and AKS (main.tf:769-770) use the same vars as the count gate. The deploy role (ci-cd-permissions/role.tf) has subscription-wide roleAssignments/write with no ABAC condition, so assigning Secrets Officer won't 403 at deploy.
  • Terraform, mocked providers only, init -backend=false -lockfile=readonly: aks 14/14 and container-apps 4/4 pass, and fmt -check is clean. Fail-before: with main's main.tf, both modules fail ("Reference to undeclared resource"). Mutation: setting the container-apps scope back to vault-wide is caught (3 passed, 1 failed).
  • Go: go build ./..., go vet (+-tags integration) on ./internal/server/, go test -race -short -count=1 ./internal/server/ ok, go test -race -tags integration -count=1 ./internal/server/ ok (228s). golangci-lint (both tag sets) found 0 issues, and gocyclo is clean. Fail-before: with main's app.go, PutSecretFailureLogsError and UserLookupFailureLogsError fail. The password is never logged; I checked the format strings.

Body-stale:

Reviewer: rD1. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.

@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: REQUEST_CHANGES 6b3943831d39bd1c33dd91952af6760e5ba6eda4

This supersedes my APPROVE above on the same head. After posting it I read the earlier reviews, as my brief allows, and cristim's cold review at 377ea27 raised a finding I had missed. I checked it again on 6b394383 and it is still open, so I'm changing my verdict.

Blocking:

  • terraform/modules/secrets/azure/main.tf:97-111 (not in the diff, but it defeats fix(iac/azure): AKS is read-only on Key Vault, so the admin-password sync write 403s and copies silently diverge #199 on the deployed path; raised first by cristim's earlier review, which I confirmed on this head). azurerm_key_vault_secret.admin_password has no lifecycle { ignore_changes = [value] }, and the module has no ignore_changes at all. The azurerm provider (~> 3.117) reads the latest secret version on refresh. So the scenario is: an admin changes their password, this PR's grant lets SetSecret succeed, and the vault holds the new value. On the next Azure terraform apply, Terraform sees the value drift and writes a new version with the original var.admin_password or random_password value. The vault and DB copies then silently diverge again, which is exactly what Closes #199 claims to fix, with no app log this time. The AWS module already does this right (modules/secrets/aws/main.tf:236, ignore_changes = [secret_string]). Either fix closes it:
    • (a) Add lifecycle { ignore_changes = [value] } to the Azure admin-password secret, so the app owns the value after first create. The trade-off: changing var.admin_password later no longer rotates the vault copy.
    • (b) Change Closes #199 to Refs #199 and file the follow-up. The acting owner's D17 decision kept the sync and the grant, but as far as I can see it did not address this point.

Non-blocking:

Attacked:

  • Same as my earlier comment on this head: Terraform mock tests (aks 14/14, container-apps 4/4) with fail-before and a scope mutation, Go unit and integration tests on internal/server, lint, and a fail-before for the logging change. I also grepped modules/secrets/{azure,aws,gcp} for ignore_changes to confirm the Azure gap and the AWS precedent.

Body-stale:

Reviewer: rD1. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.

Review of this PR (rD1, raising cristim's earlier cold-review point):
the app's admin-password sync writes the admin-password Key Vault secret
at runtime, but azurerm_key_vault_secret.admin_password had no
ignore_changes, so the next apply would see the synced value as drift and
write the Terraform-side password back. That silently undoes the sync the
owner decided to keep (D17). Add lifecycle { ignore_changes = [value] }.
Trade-off, noted in the comment: changing var.admin_password no longer
rotates the vault copy.

Also drop the second, identical "Test Azure compute modules" CI step
that the main merge brought in (LeanerCloud#1114 added the same step), so Azure CI
runs those module tests once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the REQUEST_CHANGES at 6b394383 (now dac5bf2f):

  • Revert risk: azurerm_key_vault_secret.admin_password now has lifecycle { ignore_changes = [value] }, so a later apply can't overwrite the password the app's sync wrote. That follows from the owner's D17 decision to keep the sync. Trade-off, noted in the code: changing var.admin_password no longer rotates the vault copy; rotate it through the app or az keyvault secret set. Note: the AWS secrets module's admin-password version has no such guard either, which is worth a separate look; it isn't changed here.
  • Duplicate CI step: removed the second identical "Test Azure compute modules" step.

Checks: terraform validate on the secrets module passes, terraform fmt -check passes, actionlint is clean, Trivy config (0.69.3, docker) is clean, and the secrets scan is clean. tflint isn't installed locally; CI runs it.

@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE dac5bf2f6950f272bc23345f8dafd3d7ba13667a

Blocking:

  • none

Non-blocking:

  • terraform/modules/secrets/azure/main.tf: the new lifecycle { ignore_changes = [value] } on azurerm_key_vault_secret.admin_password is a real operator-facing change, and the PR body doesn't mention it. After this, changing var.admin_password no longer rotates the vault copy. The change is right: without it, the next apply would write the Terraform value back over the synced password and undo the sync. The body and known_issues/18_tf_env_azure.md should still say so, because "change the tfvar and apply" stops working as a rotation path. No test pins it.
  • The issue asks the owner to choose between a write grant and option 3 (stop arming the sync). The PR takes the write grant, scoped to the single admin-password secret with the built-in Secrets Officer role. That is the narrowest option without a bootstrap-created custom role. It's a privilege widening the issue explicitly called a deliberate decision, so I'm leaving it visible for the owner rather than treating it as settled.
  • Under the role, count depends on module.secrets.admin_password_secret_name. That comes from the secret's configured name (plan-time known) or null, so there's no unknown-count plan error. depends_on = [module.secrets] in environments/azure/compute.tf ensures the secret exists before the secret-scoped role assignment is created.

Attacked:

  • Least privilege: the scope is exactly <vault id>/secrets/<name>, the vault-wide grant stays Secrets User, and the name validation ^[0-9A-Za-z-]{1,127}$ stops a / or ID from widening the scope. The principal is the identity the app authenticates as (container-app UAI, AKS workload UAI). The count gate matches the Go arming condition (ADMIN_PASSWORD_SECRET and ADMIN_EMAIL both non-empty).
  • Go: the lookup-error path now logs ERROR without the password. A nil user or non-admin still returns quietly. PutSecret failure logs ERROR and no longer prints the success line. The HTTP outcome is unchanged.
  • Fail-before (both restored afterwards): (a) with internal/server/app.go reverted to origin/main, TestBuildAdminPasswordSyncCallback_PutSecretFailureLogsError and _UserLookupFailureLogsError fail; (b) with container-apps main.tf reverted, its terraform test fails ("Reference to undeclared resource").
  • Ran: go build ./...; go vet on internal/server with and without -tags integration; go test -race -short -count=1 and go test -race -tags integration -count=1 on ./internal/server/ (both passed); terraform init -backend=false -lockfile=readonly plus fmt -check plus test in both modules (aks: 14 passed, container-apps: 4 passed; mock providers, no cloud calls); terraform fmt -check -recursive terraform/; terraform validate on modules/secrets/azure. PR CI is green.

Body-stale:

  • "CI wiring: a new 'Test Azure compute modules' step ..." and "Both modules' .terraform.lock.hcl gain linux_amd64 and darwin_amd64 hashes": main already has both the Azure and AWS compute-module test steps and the lock files. The PR's ci.yml change is now only a reorder of those two steps plus a comment, and it changes no lock files.
  • "On current main (a14dc8f2)", the combined-tree table and the fix/issue-204 guard run all describe an earlier base.
  • The secrets-module ignore_changes change (above) is missing from the Fix section.

Reviewer: rE1. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.

@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Local verification at dac5bf2f6950f272bc23345f8dafd3d7ba13667a merged onto main (2baf23d5), together with #1240, #1302, #1309, #1454, #1456, #1463 in that order (combined tree d882e3bf, Linux, Go 1.26.10): go build ./..., go vet with and without -tags integration, the full go test -race -short -count=1 ./..., and go test -race -tags integration -count=1 on every changed package, against Docker Postgres. All pass. CI Success and pre-commit are green on this head, with 1 approving independent review(s) on it. Nothing was run against any cloud.

@IAmJSD
IAmJSD marked this pull request as ready for review October 26, 2026 11:43
@IAmJSD
IAmJSD requested a review from cristim as a code owner October 26, 2026 11:43
@IAmJSD
IAmJSD merged commit ffe19b7 into LeanerCloud:main Oct 26, 2026
41 checks passed
@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Merged by IAmJSD's agent at IAmJSD's request (IAmJSD is acting owner while the maintainers are away), under the bar IAmJSD set: 1 approving independent review(s) on dac5bf2f, green CI, and the combined-tree verification above. Squash-merged at exactly dac5bf2f6950f272bc23345f8dafd3d7ba13667a.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/s Hours impact/internal Team-internal only priority/p3 Polish / idea / may never ship severity/low Minor harm triaged Item has been triaged type/bug Defect urgency/eventually No deadline

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(iac/azure): AKS is read-only on Key Vault, so the admin-password sync write 403s and copies silently diverge

2 participants