container universe: fix container_image selector truncation (holds every ILE job on OSPool)

[oshaughnessy-junior]

Fixes the container-universe mode (RIFT_CONTAINER_UNIVERSE=1), which today holds every GPU worker job on OSPool.

The bug

condor_submit parses container_image before any $$ expansion and derives the job ad's ContainerImage — the name the image will have in the job scratch dir — as the text after the last /. build_container_image_select() puts full osdf:// URLs inside the $$([...]) selector, so it gets cut in half and what survives is not a valid image name:

ContainerImage="rift_o4d_cc60-90_cuda118_20260717.sif\") ])"
ContainerImageSource="$$([ ifThenElse(TARGET.GPUs_Capability >= 9.0, \"osdf"
ContainerImageFullPath="$$([ ifThenElse(TARGET.GPUs_Capability >= 9.0, \"osdf:///...\", \"osdf:///...\") ])"

ContainerImageFullPath keeps the $$ and expands correctly at match time. ContainerImage no longer contains a $$, so nothing repairs it — and the OSG glidein's PREPARE_JOB prepare-hook is what reads it.

Evidence

Three trivial jobs on the IGWN pool, using two 3 MB .sif files staged on OSDF and keyed on TARGET.Memory instead of GPUs_Capability so they match a CPU slot immediately — condor_submit derives ContainerImage identically either way (checked with -dry-run), so the mechanism under test is unchanged.

cluster	form	result
5926098	plain single osdf://…sif	✅ ran, exit 0
5926099	the form this branch currently emits	❌ HELD
5926100	this fix	✅ ran, exit 0

The hold, verbatim from the execute point:

Error from slot1_4@glidein_95_271134742@wn-snel-006.farm.nikhef.nl:
PREPARE_JOB (prepare-hook) failed (reported status 001):
Unable to download or build singularity image cutest_busybox_20260810.sif") ])

The held job's ad shows the $$ machinery working perfectly — only ContainerImage was already broken before expansion ran:

MATCH_EXP_ContainerImageFullPath = "osdf:///…/cutest_alpine_20260810.sif"   # correct branch, expanded
ContainerImage                   = "cutest_busybox_20260810.sif\") ])"      # no $$ left, never repaired

and the fixed job's:

MATCH_EXP_ContainerImage = "cutest_alpine_20260810.sif"
MATCH_EXP_TransferInput  = "osdf:///…/cutest_alpine_20260810.sif"

Confirmed on $CondorVersion: 25.11.1 2026-07-07.

The fix

1. build_container_image_select() emits basenames, so the selector holds no /, condor_submit's derivation is a no-op, the whole $$ token reaches the job ad, and the schedd expands it at match time.
2. The matched image is delivered by the comma-free $$() transfer token, which container universe previously skipped (it assumed container_image would fetch it). Applied in both write_ILE_sub_simple and write_calpilot_sub — CALPILOT runs ILE on a GPU and emits the same selector.
3. MY.TransferInput is pinned to the same list, so condor_submit does not append the basename selector to TransferInput as a bogus extra input file.
4. A family containing an in-place (CVMFS/local) image now raises ContainerManifestError under container universe — such an image can only be named by its full path, which reintroduces the truncation. Stage it at a URL, or use RIFT_CONTAINER_RUNTIME_SELECT=1. The shipped example manifest is updated to all-URL accordingly.

CIP is untouched: CPU-only, already a single plain image, which condor_submit handles correctly.

Before / after on this branch's own code

Rebuilding the S240426s OSG DAG (real inputs from an existing asimov project, same pseudo_pipe command line from its asimov.log) with dag_utils_generic at HEAD~1 vs HEAD:

# before
container_image = $$([ ifThenElse(TARGET.GPUs_Capability >= 9.0, "osdf:///igwn/cit/staging/.../cc90-120_cuda128....sif", "osdf:///igwn/.../cc60-90_cuda118....sif") ])
  -> ContainerImage="rift_o4d_cc60-90_cuda118_20260717.sif\") ])"        # TRUNCATED

# after
container_image = $$([ ifThenElse(TARGET.GPUs_Capability >= 9.0, "rift_o4d_cc90-120_cuda128_20260717.sif", "rift_o4d_cc60-90_cuda118_20260717.sif") ])
transfer_input_files = ...,$$([ (TARGET.GPUs_Capability >= 9.0 ? "osdf:///...cc90-120....sif" : "osdf:///...cc60-90....sif") ])
MY.TransferInput     = "<the same list, pinned>"
  -> ContainerImage="$$([ ifThenElse(...) ])"                            # INTACT

Unit tests 26/26. A 28-assertion submit-file check over the rebuilt DAG passes on HEAD and fails 9 on HEAD~1. Harness and captured logs: ~/LVK/IR1/demo_multi_container (validate_build.sh, check_condor_expansion.sh, live_check/).

Also in this PR

docs/source/containers.rst and containers/README.md never documented the container-universe or runtime-select modes — the docs predate those commits. Added a delivery-mode section covering all three, plus the truncation invariant. Also corrected a stale claim that the capability selection falls back on an undefined capability: it does not (a Requirements clause excludes those slots, and CIP collapses to the single fallback image).

The same fix is in #164, which ports the container-family feature to master.

🤖 Generated with Claude Code

[oshaughnessy-junior]

Focused review at ecc3cd5c found one material documentation issue:

• docs/source/containers.rst:180-187 still says container universe leaves the pilot a literal image URL, emits no match-time transfer token, and lets container_image deliver the image. The implementation now does the opposite: container_image selects a basename, the full URL arrives through the $$() transfer token, and MY.TransferInput is pinned. This directly contradicts the new warning at lines 217-240.
• The same guide still says an undefined GPU capability falls back (lines 256-257) and that OSG validation of expression-valued MY.SingularityImage remains outstanding (lines 264-270). The code excludes undefined capabilities, and this PR documents live OSPool validation of the container-universe path. containers/README.md:214-215 and :236-243 retain the same stale statements.

Please update these touched docs so the public delivery guidance has one consistent contract.

Focused gate otherwise passed: both changed ILE/CALPILOT submit paths were accepted by HTCondor 25.8.2 dry-run with intact basename ContainerImage and URL-bearing TransferInput; the 26 container-manifest tests pass. Both red canaries fail before changed RIFT code runs on the same upstream pygsl-lite==0.1.8 / setuptools spawn(cmd, 1, 1) incompatibility, so they are expected non-blocking canary failures rather than regressions from this delta.