Background deletion of an InferenceCluster hangs its serving stack

[negz]

What problem are you facing?

Deleting an InferenceCluster with background propagation, the kubectl delete default, can hang its serving stack and orphan cloud resources.

The InferenceCluster composes a Usage that holds the cluster it provisions until Kubernetes has deleted its ServingStack, so the stack can uninstall while the cluster's API server and kubeconfig still exist:

modelplane/functions/compose-inference-cluster/function/fn.py

Lines 1419 to 1440 in e74545e

	def compose_eks_usage(self) -> None:
	"""Block EKSCluster deletion until the backend is deleted."""
	resource.update(
	self.rsp.desired.resources["usage-eks-by-backend"],
	usagev1beta1.Usage(
	metadata=metav1.ObjectMeta(namespace=_NAMESPACE_SYSTEM),
	spec=usagev1beta1.Spec(
	of=usagev1beta1.Of(
	apiVersion="infrastructure.modelplane.ai/v1alpha1",
	kind="EKSCluster",
	resourceSelector=usagev1beta1.ResourceSelectorModel(matchControllerRef=True),
	),
	by=usagev1beta1.By(
	apiVersion="infrastructure.modelplane.ai/v1alpha1",
	kind="ServingStack",
	resourceSelector=usagev1beta1.ResourceSelector(matchControllerRef=True),
	),
	replayDeletion=True,
	),
	),
	)
	self.rsp.desired.resources["usage-eks-by-backend"].ready = fnv1.READY_TRUE

A Usage by a composite resource only holds until that XR leaves the API, and Crossplane removes an XR's finalizer as soon as it's deleted:

https://github.com/crossplane/crossplane/blob/61fa450c0c14a75bf5632872bd4a5b6945ec5a26/internal/controller/apiextensions/composite/reconciler.go#L629-L645

So with background propagation the ServingStack disappears at once, the Usage releases, and the cluster goes while the stack's Helm releases are still uninstalling. They hang, and a load balancer one of them created can leak its security group and block the VPC from deleting.

ModelDeployments and InferenceGateways have a narrower version of the problem. An InferenceCluster refuses deletion while anything runs on it, but that guard only holds while the ModelReplicas, InferenceGateways and other resources using the cluster exist. Background propagation removes them at once, while their provider-kubernetes Objects are still finalizing on the cluster. So it's a race: if the InferenceCluster goes in that window, the Objects lose the cluster they need to finalize against and wedge until someone removes their finalizers by hand.

Foreground propagation keeps a resource, and anything holding on it, until Kubernetes has deleted everything it composed. The docs ask for --cascade=foreground, but nothing enforces it.

How could Modelplane help solve your problem?

@haarchri suggested a ValidatingAdmissionPolicy that refuses deletes that don't request foreground propagation. His sketch targets ModelDeployments, but the same policy could match InferenceClusters:

kind: ValidatingAdmissionPolicy
apiVersion: admissionregistration.k8s.io/v1
metadata:
  name: forceforeground
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["modelplane.ai"]
      apiVersions: ["v1alpha1"]
      operations:  ["DELETE"]
      resources:   ["modeldeployments"]
  validations:
  - expression: has(request.options.propagationPolicy) && request.options.propagationPolicy == "Foreground"
    reason: 'Invalid'
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: foregroundbinding
spec:
  policyName: "forceforeground"
  matchResources:
    objectSelector:
      matchExpressions:
      - key: "crossplane.io/composite"
        operator: Exists
  validationActions: [Deny]

A Configuration package can only carry XRDs, Compositions, ManagedResourceActivationPolicies and Operations, so the policy would need a separate install step or Crossplane support for packaging it:

https://github.com/crossplane/crossplane-runtime/blob/71f319796d597c2c6cf004cc4bf5f44011a5aeaa/pkg/xpkg/lint.go#L71-L76