feat(nvca): add storage-aware model cache runtime

[balajinvda]

Customer Summary

Adds deterministic model-cache storage selection and read-only reader publication. Existing NVMesh behavior is named roxReadOnly. A new regular-cache rwxReadOnly path reuses one populated RWX claim without a clone or data copy. No external storage provider is enabled by this PR.

TL;DR

• Resolve the exact live nvcf-sc provisioner against the public catalog from feat(nvca): publish storage capability catalog #1334.
• Persist the StorageClass snapshot, catalog decision, required access modes, and required reader-PV mount options before storage side effects.
• Create one immutable ModelCacheBinding per cache identity and make retries use that binding.
• Implement roxReadOnly for the existing NVMesh regular and Helm paths.
• Implement regular-only rwxReadOnly using the populated RWX claim directly.
• Set the workload PVC source and every matching init-container and container mount to read-only.
• Keep Weka, OCI FSS, and OCI Lustre transitions disabled.

Additional Details

This PR is stacked on #1334, which publishes the closed transition contract and storage capability catalog.

For a new model-cache request, NVCA evaluates the workflow gates, reads StorageClass/nvcf-sc, requires Retain, loads the exact provisioner entry, and persists the selection. It then creates or joins an immutable binding before it creates storage resources. Retries, restarts, catalog changes, and feature-gate changes continue from the recorded decision.

Transition behavior

roxReadOnly is restricted to the exact NVMesh provisioner and provider:

1. Populate an RWO writer claim.
2. Retain and transition the populated storage to a ROX reader claim.
3. Apply the binding's persisted reader-PV mount options.
4. Publish the reader claim with read-only volume and container mount intent.

The shipped required options are ro, norecovery, and nouuid. They are persisted in the request selection and ModelCacheBinding, copied defensively, repaired on retry, and validated before publication and cleanup. The legacy nvca-cache-mount-options ConfigMap is used only for annotation-free compatibility requests. The operator cacheMountOptions setting remains additive; any value that negates a provider requirement is ignored.

rwxReadOnly is restricted to regular model cache and requires ReadWriteMany:

1. Populate one RWX writer claim.
2. Validate the exact PVC, PV, CSI handle, completed writer Job, and ownership identity.
3. Return the same claim to workload Pods with read-only Kubernetes mount intent.

This path does not create a reader PVC, rewrite the PV, wait for detach, clone data, or make another copy. Its catalog entry must use an empty readerMountOptions array. It does not claim backend-enforced write denial.

Helm model cache supports only roxReadOnly. Its writer and readers use different namespaces, so rwxReadOnly is rejected by the schema, loader, persisted-selection validator, and runtime dispatcher.

Enablement boundary

The shared-writer path accepts only credential-free Jobs. Current translated writer artifacts contain inputs that it rejects. Provider enablement still requires:

1. Binding-scoped writer input and Secret identity, lifecycle, rotation, and cleanup.
2. Binding-level shared-writer failure state and recovery.
3. Zero-reference retirement and retained-data garbage collection.
4. Functional qualification of the exact CSI, StorageClass, node, and NVCF artifact configuration.

Container cache, CSI installation, performance qualification, and Helm support for non-NVMesh providers are outside this PR.

No third-party dependency changed. NOTICE does not change.

For the Reviewer

Please focus on:

• selection and binding persistence before storage side effects
• immutable required mount-option propagation from catalog to request and binding
• NVMesh retry repair without changing CSI handle, claim UID, ownership, StorageClass, or reclaim policy
• rwxReadOnly same-claim publication with no PV mutation or data copy
• exact writer Job, PVC, PV, CSI, ownership, and lifecycle fencing
• read-only injection into the PVC source and every matching container mount
• fail-closed drift, cleanup, and conflict handling
• Helm and encryption rejection of rwxReadOnly

The complete contract, limitations, rollout, and qualification plan are in docs/dev/sdd-storage-agnostic-cache-architecture.md.

For QA

Local validation completed:

• full pkg/storage, pkg/nvca, pkg/apis/nvca/v2beta1, and internal/miniservice tests with Kubernetes 1.34.1 envtest assets: pass
• go vet for those four packages: pass
• full NVCA Helm lint, executable catalog schema tests, render checks, and mirrored-catalog parity: pass
• license and SPDX check: pass
• git diff --check, ASCII documentation style, and public-safety scans: pass

The tests cover selection, persistence, generated Kubernetes objects, read-only fields, mount-option repair, identity preservation, retry, drift, ownership, cleanup, and RWX no-mutation behavior. They use fake or envtest Kubernetes APIs. They do not mount a real CSI volume or qualify Weka, OCI FSS, OCI Lustre, backend write denial, restart behavior on a live cluster, or performance. Those provider entries remain disabled.

Tickets

Relates to #1326

Checklist

• I am familiar with the Contributing Guidelines.
• I have signed off my commits for Developer Certificate of Origin compliance.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ed3f120b-911b-4509-8046-c630acb03df3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-feat-nvca-storage-runtime-selection.docs.buildwithfern.com/nvcf