Container family: multi-architecture container deployment (port from rift_O4d)

[oshaughnessy-junior]

Ports the container family feature (multi-architecture container deployment) from rift_O4d to master.

Point SINGULARITY_RIFT_IMAGE at a .yaml/.yml manifest describing several images that target different GPU compute capabilities, and let HTCondor pick the right one per matched machine. A plain .sif path or single osdf:// URL keeps the exact legacy single-image behaviour — a manifest is recognised purely by its file extension, and pyyaml is only required when one is actually used.

On rift_O4d the wiring lives in dag_utils_generic.py, part of a backend-neutral rewrite that master does not have. Here the same wiring is applied to dag_utils.py directly; container_manifest.py and the containers/ build kit are ported unchanged.

What is in it

• RIFT/misc/container_manifest.py — manifest parse/validate plus the expression builders: capability ifThenElse image selection, comma-free $$() selective-transfer token, require_gpus capability floor, container-universe container_image selector, capability-defined Requirements clause, CPU-safe single-fallback resolution, and the runtime-selection wrapper script.
• dag_utils.write_ILE_sub_simple / write_CIP_sub — emit the above. The GPU ILE job gets the per-machine selection, the family floor composed (not replaced) with RIFT_REQUIRE_GPUS, and TARGET.<attr> =!= undefined so it never matches a slot where the selection cannot expand. CPU-only CIP collapses to the single CPU-safe fallback image, because a $$() capability expression on a slot with no GPU would hold the job.
• Two opt-in OSG delivery modes, since OSPool pilots read SingularityImage as a literal string: RIFT_CONTAINER_UNIVERSE=1 (universe=container + container_image = $$([...]), schedd-side match-time substitution — the intended OSG mode) and RIFT_CONTAINER_RUNTIME_SELECT=1 (ILE-only wrapper that detects the real GPU at job start and fetches just that image).
• create_event_parameter_pipeline_BasicIteration — enable the osdf transfer credential by inspecting the manifest's image URLs. Without this the existing substring checks only see a .yaml path and every worker job is held for a missing credential. Also, under container universe (which deliberately drops MY.SingularityBindCVMFS) LALAPPS_PATH2CACHE now points at the container's own lal_path2cache rather than a hardcoded /cvmfs path.
• util_RIFT_pseudo_pipe.py — accept singularity_rift_image / singularity_base_exe_dir from the ini (environment still wins), matching how accounting and ile_require_gpus already work.
• containers/ build kit (multi-target build_family.sh + one shared rift_container.def.in template + example manifest) and docs/source/containers.rst.
• test/test_container_manifest.py — 25 tests, all passing. The integration cases inspect the condor commands on the job object returned by write_ILE_sub_simple / write_CIP_sub, so no pool is needed.

No asimov change is required: RIFT/asimov/rift.py on master already exports scheduler: singularity image: and scheduler: environment variables: into the pipeline environment.

Not ported (separate features that live on rift_O4d): containers/survey_scan/ and the GitHub Actions dependency canaries — this branch has no .github/. The containers/README.md says so explicitly rather than leaving dangling links.

Validation

An OSG-style asimov build for S240426s was rebuilt with the ported code and compared against the already-built single-container run in ir1_prod_testing_personal, using that project's own inputs and the pseudo_pipe command line recorded in its asimov.log. Workspace, scripts and captured logs: ~/LVK/IR1/demo_multi_container.

24/24 assertions pass. Generated ILE.sub:

universe = container
container_image = $$([ ifThenElse(TARGET.GPUs_Capability >= 9.0, "osdf:///.../cc90-120_cuda128.sif", "osdf:///.../cc60-90_cuda118.sif") ])
require_gpus = ((DeviceName=!="Tesla K10.G1.8GB")&&...) && (Capability >= 6.0)
requirements = (HAS_SINGULARITY=?=TRUE)&&...&&(TARGET.GPUs_Capability =!= undefined)
use_oauth_services = scitokens

CIP.sub gets the single fallback image, no $$(), no floor, no capability requirement. Every non-container submit command is byte-identical to the reference build.

Open item — please read before a production OSPool campaign

condor_submit -dry-run shows that condor_submit parses container_image before any $$ expansion, and derives the job ad's ContainerImage (the name the image gets in the job scratch dir) as the text after the last /. A $$([...]) selection contains slashes, so that derivation truncates it:

ContainerImage="rift_o4d_cc60-90_cuda118_20260717.sif\") ])"
ContainerImageFullPath="$$([ ifThenElse(TARGET.GPUs_Capability >= 9.0, \"osdf:///...\", \"osdf:///...\") ])"

ContainerImageFullPath and transfer_input_files keep the $$ token and expand correctly at match time; ContainerImage has no $$ left, so nothing repairs it unless the schedd re-derives it after expansion — which I could not verify without a live GPU match. Confirmed on $CondorVersion: 25.11.1.

This is inherited from rift_O4d, not introduced here — the port reproduces rift_O4d's output. It is recorded in docs/source/containers.rst and containers/README.md, with a reproducer (check_condor_expansion.sh) in the demo workspace. RIFT_CONTAINER_RUNTIME_SELECT=1 emits no container-universe attributes at all and is unaffected.

🤖 Generated with Claude Code

[oshaughnessy-junior]

Isolated review of current head 38dc643b04e021419548f0571eedd378852bffbc found two material container-family blockers.

1. [P1] The recommended container-universe selector produces malformed HTCondor deployment metadata. build_container_image_select() emits container_image = $$([ ifThenElse(... URLs with slashes ...) ]). A focused condor_submit -dry-run on HTCondor 25.8.2 reproduces the problem already documented in this PR for 25.11.1:

ContainerImage="rift_container_default.sif\") ])"
ContainerImageSource="$$([ ifThenElse(TARGET.GPUs_Capability >= 8.0, \"osdf"
ContainerImageFullPath="$$([ ifThenElse(...intact expression...) ])"

Because RIFT_CONTAINER_UNIVERSE=1 is documented as the recommended OSG path while post-match re-derivation remains explicitly unverified, this should not be treated as production-clean. Please change the selector encoding/delivery mechanism or validate a real matched job and add a regression that checks the effective image metadata.

2. [P1] cuda_capability_max is ignored by ClassAd/container-universe selection and disagrees with runtime selection. _build_selector() only tests each cuda_capability_min. For a family old=[3,7], new=[8,9], capability 10 emits a ClassAd that selects new, while build_runtime_selection_wrapper() applies max bounds and selects fallback old. This can route a GPU outside every declared support band into an incompatible image, and behavior changes solely with the delivery mode. Include max in every selector (with the documented exclusive boundary), constrain unsupported slots, or reject manifests whose highest band is not open-ended.

Focused results: pytest -q MonteCarloMarginalizeCode/Code/test/test_container_manifest.py -> 25 passed; changed Python files compile; containers/build_family.sh --render-only passes; git diff --check and git merge-tree --write-tree pass. GitHub reports MERGEABLE / CLEAN, but no checks are configured for this branch.

[oshaughnessy-junior]

Update: the container-universe mode was genuinely broken; now fixed and verified live

The open item flagged above was not a dry-run artifact. I submitted real jobs to the IGWN pool to settle it, using two 3 MB .sif images staged on OSDF and keying the selection on TARGET.Memory instead of GPUs_Capability so they match a CPU slot immediately — condor_submit derives ContainerImage identically either way, so the mechanism under test is unchanged.

cluster	form	result
5926098	plain single osdf://…sif	✅ ran, exit 0
5926099	rift_O4d form — $$([ ifThenElse(…, "<full URL A>", "<full URL B>") ])	❌ HELD
5926100	this fix — basenames in container_image + pinned MY.TransferInput	✅ ran, exit 0

The hold, from the execute point:

PREPARE_JOB (prepare-hook) failed (reported status 001):
Unable to download or build singularity image cutest_busybox_20260810.sif") ])

The $$ machinery itself works fine — it was ContainerImage that was already truncated before expansion ever ran:

MATCH_EXP_ContainerImageFullPath = "osdf:///…/cutest_alpine_20260810.sif"   # correct branch, expanded
ContainerImage                   = "cutest_busybox_20260810.sif\") ])"      # no $$ left, never repaired

condor_submit parses container_image before $$ expansion and derives ContainerImage — the name the image gets in the job scratch dir — as the text after the last /. The OSG glidein's PREPARE_JOB prepare-hook reads that attribute, not ContainerImageFullPath.

Fix

1. build_container_image_select() emits basenames, so the selector holds no /, condor_submit's derivation is a no-op, the whole $$ token reaches the job ad, and the schedd expands it at match time (MATCH_EXP_ContainerImage = "cutest_alpine_20260810.sif").
2. The matched image is delivered by the comma-free $$() transfer token, which container universe previously skipped (it assumed container_image would fetch it).
3. MY.TransferInput is pinned to the same list, so condor_submit does not append the basename selector to TransferInput as a bogus extra input file.
4. A family containing an in-place (CVMFS/local) image now raises ContainerManifestError under container universe — such an image can only be named by its full path, which reintroduces the truncation. Stage it at a URL, or use RIFT_CONTAINER_RUNTIME_SELECT=1. The shipped example manifest is updated to all-URL accordingly.

CIP is untouched: CPU-only, already a single plain image, which condor_submit handles correctly.

Generated ILE.sub now:

universe = container
container_image = $$([ ifThenElse(TARGET.GPUs_Capability >= 9.0,
                       "rift_o4d_cc90-120_cuda128_20260717.sif",
                       "rift_o4d_cc60-90_cuda118_20260717.sif") ])
transfer_input_files = …,$$([ (TARGET.GPUs_Capability >= 9.0 ? "osdf:///…cc90-120….sif"
                                                             : "osdf:///…cc60-90….sif") ])
MY.TransferInput     = "<the same list, pinned>"

Tests 26/26; the demo's submit-file assertions 28/28. docs/source/containers.rst, containers/README.md and the example manifest are updated to the corrected contract and the live evidence. Reproducer in ~/LVK/IR1/demo_multi_container (validate_build.sh, live_check/).

Note for rift_O4d: this bug is upstream. The same fix should be applied to dag_utils_generic.py there, otherwise the container-universe mode holds every ILE job on OSPool.

🤖 Generated with Claude Code