[Bug] A Gemini Enterprise project cannot be torn down and reused — the CmekConfig is permanently undeletable once a data store has been attached

[JohnHales]

Bug Description

gemini-stage-0 registers a Discovery Engine CmekConfig and creates a data store against it. Nothing in the blueprint ever deregisters the config, and it cannot be deregistered by hand either: once any data store has been attached to it, DELETE .../cmekConfigs/default_cmek_config fails permanently with

400 FAILED_PRECONDITION: There are currently 1 datastore(s) connected to this CmekConfig.
Please delete these datastores before trying to delete the CmekConfig.

— for a data store that has been deleted and returns 404. The underlying service increments the connected-datastore count on attach and never decrements it on delete.

The consequence for this blueprint is that terraform destroy cannot return a project to a deployable state. A redeploy into the same project inherits the stale CmekConfig, which deploy.sh then adopts (CMEK_US_RESOURCES_KEY="$_CURRENT_KEY"), so the project is no longer representative of a clean deployment.

This is a Google service defect, not a Stellar Engine coding error — but it is a Stellar Engine documentation and lifecycle defect, because the blueprint presents teardown as supported when it is not.

What it is NOT (please do not re-derive this)

We first assumed the cause was ordering: terraform destroy drops the service agent's roles/cloudkms.cryptoKeyEncrypterDecrypter binding on the CMEK key before the data store is gone, and Discovery Engine then cannot complete its internal revoke-cmek work. A two-arm experiment on two empty scratch projects disproves that:

	Key binding during teardown	Result
Run A	removed before deleting the data store	1 datastore(s) connected — stuck
Run B	left in place for the entire lifecycle	1 datastore(s) connected — stuck

Ordering and IAM are not the variable. The only thing both runs share is that a data store was attached to the config and then deleted.

Environment and Deployment Context

• Stellar Engine Version/Commit: main at commit f64ce6cd (re-verified 2026-08-10)
• Deployment Type:
  • US Region Restricted (e.g., Access Policy constraint)
  • FedRAMP Medium
  • FedRAMP High
  • FedRAMP Moderate
  • DoD IL4
  • DoD IL5
  • Stand-alone / Custom
• FAST Stage (if applicable): N/A — this is a blueprint, not a FAST stage
  • Stage 0 (Bootstrap)
  • Stage 1 (Resource Management)
  • Stage 2 (Network Creation)
  • Stage 3 (Security and Audit)
• Affected Component: blueprints/fedramp-high/gemini-enterprise — the teardown path as a whole (trigger: any deployment that creates a data store, i.e. the normal path)
• Terraform Version: 1.12.2 (pinned by deploy.sh via tfenv; the stage declares required_version >= 1.7.4)
• GCP Provider Version: hashicorp/google >= 6.21.0 (stage-0 declared constraint)

Steps to Reproduce

Reproduces on a bare project with no Stellar Engine component involved — no landing zone, no Assured Workloads, no shared VPC:

1. In an empty project, enable discoveryengine and cloudkms, and provision the service agent with gcloud beta services identity create --service=discoveryengine.googleapis.com.
2. Create a KMS key in the us multi-region and grant the service agent roles/cloudkms.cryptoKeyEncrypterDecrypter on it.
3. PATCH .../locations/us/cmekConfigs/default_cmek_config?set_default=true with that key; wait for state: ACTIVE (5-10 minutes on a cold project).
4. POST .../collections/default_collection/dataStores?dataStoreId=<id> with kmsKeyName set.
5. DELETE that data store. Confirm it is gone — GET on it returns 404.
6. DELETE .../cmekConfigs/default_cmek_config.

A full copy-paste script covering both arms, plus a control run that skips the binding removal, is available on request.

Expected Behavior

Deleting the last data store decrements the CmekConfig's connected count, and the config can then be deleted, returning the project to a deployable state.

Actual Behavior

The count never decrements. The CmekConfig is permanently undeletable and the project can never be cleanly redeployed into.

Relevant Logs and Errors

Observed on both arms of the two-arm experiment (2026-07-28), against a data store that had already been deleted and returned 404:

DELETE .../locations/us/cmekConfigs/default_cmek_config
400 FAILED_PRECONDITION: There are currently 1 datastore(s) connected to this CmekConfig.
Please delete these datastores before trying to delete the CmekConfig.

Related service-layer defects observed alongside (same investigation)

1. delete-data-store reports done: true before the deletion has happened. In one run the store still returned HTTP 200 for 40+ minutes afterwards, and the operation handle then 404'd — success reported, work not done, no record left to audit. A second DELETE completed it.
2. revoke-cmek-* / restore-cmek-* operations never report completion. operations.get returns a bare {"name": "…"} indefinitely — no done, no error, no metadata — they do not appear in operations.list, :cancel returns 404, and they hold an exclusive lock that fails every other CmekConfig call with Only one outstanding LongRunningOperation … for ~12 hours. Meanwhile the work completes: a config reached ACTIVE while its restore was still "outstanding". Poll the resource state, never the operation.

Suggested Fix / Mitigation for the blueprint

Google Product Engineering has the counter defect (support case opened 2026-07-28). Independently of their fix, Stellar Engine should:

1. Document that a Gemini Enterprise project is effectively single-use. Do not present terraform destroy as a path back to a deployable state.
2. Recommend project recreation as the reset procedure. Confirmed working 2026-07-28 — gcloud projects delete succeeds on a project in this state, and it is currently the only reliable reset.
3. Consider having the blueprint create the G4G project itself, so teardown/redeploy is a project lifecycle rather than a resource lifecycle.

Additional Context

The counter defect sits in the Discovery Engine service, not in this repository — a Google support case was opened 2026-07-28. It is raised here because the blueprint's lifecycle story depends on it: gemini-stage-0 is presented as something terraform destroy can undo, and it is not. Blocks any workflow that redeploys into an existing project — CI, demo environments, and training deliveries that reset between cohorts.

There is an open support ticket with Google support on fixing the issue. As of 8-11-26, it has been open for a couple of weeks.

A full copy-paste reproduction script for both experiment arms, plus a control run that skips the key-binding removal, is available on request.