[earakely-scale]

Was stacked on #456 (M4.1, the disk form). #456 has merged, and this PR is now based on main. Merge commit 652e588 brings in main: its tree is main plus this PR's two commits, replayed without conflicts, so the diff is FD-5094 alone.

Linear: FD-5094. Decided by Edgar: compose per run, not by an env-level forms list, since the same env needs disks on a disk-booting sandbox and not on local Docker.

Why

Since #456, every compose of an ubuntu-24.04 computer env also makes each arch's disk: about 15–25 s more per arch on Linux, 1.5–3 min on a Mac, and about 1.6 GB per disk. A run on local Docker never boots them, because a deploy takes the first form its sandbox runs natively, and containers come first.

What changes

• A run composes only the forms its sandbox's deploys boot.
  • agent-env run (and prepare_bundle, dry_run_bundle) composes with Composing.for_run(sandbox_provider, emulate) (forms_of="deploys"). Each compose looks the provider up once, when it needs it (Composing.provider()), so a bundle with no computer env never builds a default sandbox provider it may not load.
  • composer.booted_forms picks the forms: on each link of the sandbox's chain, the form of the first pair that link boots natively, by boot.first_native, the rule the deploy picks its build by too; a chain gets the union of its links' forms.
  • Local Docker therefore composes containers, with or without KVM.
  • A sandbox that boots only disks (none in this repository yet; Modal VMs in M4.2) composes disks. compose itself adds the container each disk pair is made from, unless it's built, so a disk is never skipped for want of a container no one asked for.
  • A reused env's pending pairs are filtered to those forms the same way.
• A compose with no target still builds every form: a put, env build, bundle register and the release.
• Keys don't change, so a later compose that needs a missing form builds it then and reuses everything else. A composed computer records only the forms built, as before: it gets a disk form only when every arch it has has a disk.
• A deploy that boots a form no compose started composes it first, with no error (ComputerEnv.compose_unstarted, called by the computer env provider before it picks a build).
  • This covers a disk its env_params name, or a disk-only sandbox after a local run.
  • It composes on the deploy's sandbox provider, and gate 3 probes what it builds there.
  • Deploys of one env at once compose it once: the others wait on a file lock, read the result again and boot it. A deploy that finds a pair it boots building waits the same way.
  • It doesn't retry failed or skipped pairs of the forms it boots; those stay with env build and the next run's pending. The container a disk is made from is the exception: a compose of the disk builds it again when it isn't built.
  • A cancelled deploy holds the lock until its compose's thread ends, so the next deploy waits for what it builds.
  • Where a local run's fence would refuse the env's record (Config.local_run_refuses_write, the rule check_local_run_write raises by), it composes nothing and the deploy refuses, naming agent-env env build --id <env>.
  • In agent-env run the deploy's compose shows in the run's output. The run sets its progress for each task in a ContextVar (agent_env.utils.progress), which the task's step tasks and asyncio.to_thread carry to the deploy. The deploy says why it composes, then the compose's stage lines; a deploy waiting for another says what it waits for and since when, every HEARTBEAT_S with the time waited, then what it reuses. Outside a run the lines go to the log.
• No public signature changes: Composing gains a field with a default and for_run, pending an optional argument, awaiting_lock an optional waiting callback, and Config local_run_refuses_write.

What a run's deploy of the disk prints, from the KVM box (the run itself composed the containers; arm64 has no emulation there):

[tasks/probe.json] step 1/4 env (deploy_env)
[tasks/probe.json] @local/agentenv-framework/desktop/desktop: composing disk/amd64, disk/arm64 before this deploy (no compose built them for LocalSandboxProvider yet); later deploys reuse them
[tasks/probe.json] @local/agentenv-framework/desktop/desktop: composing container/arm64, disk/amd64, disk/arm64 on LocalSandboxProvider
[tasks/probe.json] @local/agentenv-framework/desktop/desktop: container/arm64: failed at resolve: this machine can't build linux/arm64: its Docker has no emulation for it (0 s)
[tasks/probe.json] @local/agentenv-framework/desktop/desktop: disk/arm64: skipped: its container form didn't build (failed) (0 s)
[tasks/probe.json] @local/agentenv-framework/desktop/desktop: disk/amd64: reusing the disk of the same key, made earlier
[tasks/probe.json] @local/agentenv-framework/desktop/desktop: disk/amd64: probing on LocalSandboxProvider
[tasks/probe.json] @local/agentenv-framework/desktop/desktop: disk/amd64: reused, ready (21 s)
[tasks/probe.json] step 1/4 env done in 28.6s

A second deploy that waits says <env>: waiting for another deploy composing disk/amd64, disk/arm64 (building since HH:MM UTC), then <env>: reusing what another deploy composed: disk/amd64 unverified, disk/arm64 ready.

Tests

• Unit:
  • The forms chosen for local Docker (with and without KVM), a deploy that names the disk, a disk-only sandbox, chains, and a sandbox that boots nothing natively.
  • A put composes every form, and a run's compose only containers, with no disk form in the computer.
  • Three concurrent deploys of the disk an env never composed make each disk once, reuse the containers, and all boot it.
  • A deploy that starts while another deploy's compose has the disk building waits for it and boots it.
  • A run's dry run now says it would compose the containers only.
  • What a run composes is what each link of its sandbox then boots, for local Docker, a disk-only sandbox and a chain; a compose of disks alone builds their containers.
  • The run's progress reaches a deploy's compose through Task.run, with the reason line and the stage lines; a waiting deploy's waiting and reused lines; a cancelled deploy holding the lock; the lock's waiting callback and its repeats.
  • local_run_refuses_write against real store setups, and the deploy both ways: composes where the fence allows the record, refuses with the env build --id hint where it wouldn't.
• Integration (int_test_slow + kvm): disk_boot_test.py first runs the desktop deploying its container, which composes no disk. Then it runs the disk probe task, whose deploy composes the disk (reusing this arch's container record unchanged), probes it at full under KVM, boots it, and resets it; the run's output carries the reason line and the disk's stage lines.

Results

At 24115a5, on a fresh EC2 c8i.4xlarge with nested KVM:

• make unit-test: 12,692 passed and 7 skipped.
• make int-test-fast: 179 passed.
• disk_test.py: 3 passed.
• The slow KVM test passed, in the 427 s the two disk files took together. Its local run composed no disk; then the disk task's deploy composed the disk with this arch's container record unchanged, probed it at full (ready), booted it, probed it, reset it and probed it again.

At 13391a5, Greptile's fix, on a second fresh box: make unit-test 12,693 passed and 7 skipped, and the slow KVM test passed again, in 345 s. The new waiting test fails on 24115a5.

At 3fc487f (the progress lines and the six maintainability fixes), on a third fresh box: make unit-test 12,762 passed and 7 skipped, make int-test-fast 180 passed, and disk_test.py with the slow KVM test 4 passed. At bb700ea (Greptile's cancel and --id fixes), the touched unit files, 305 tests, passed on the same box, and the example above is its agent-env run. At ac9ee5d (the lazy lookup), on a fourth fresh box: make unit-test 12,764 passed and 7 skipped, and the slow KVM test passed.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge.

Summary

Runs compose only the computer forms their sandbox boots. Deploys build missing forms before booting and share that work across concurrent deploys.

• Bundle runs build the computer forms their sandbox deploys can boot.
• A deploy composes a needed computer form before it boots.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Run[Prepare a run] --> Computer{Computer env needs composing?}
  Computer -->|No| Continue[Continue without loading default provider]
  Computer -->|Yes| Provider[Use selected or default provider]
  Provider --> Forms[Compose forms its deploys boot]
  Forms --> Deploy[Deploy]
  Deploy --> Missing{Required form never started?}
  Missing -->|Yes| Lock[Take lock and compose missing form]
  Missing -->|No| Boot[Boot available build]
  Lock --> Boot

Loading

_{Reviews (9) · Last reviewed commit: "fix(computer): a run's compose looks up ..." · Reviewed by Greptile}