[blarghmatey]

What are the relevant tickets?

witan task tk-build-omnigraph-v0-14-0-plus-the-scope-rs-s3-pat-ec78ba (decision: option (e) of tk-decide-how-optimize-cleanup-force-unlock-and-boo-58bab2). Upstream: ModernRelay/omnigraph#1189

Description (What does it do?)

Since 0.14 an inline --target is local-only, so the stock CLI cannot address the data tier's S3 cluster root. With the server stopped there is then no way to converge the catalog on boot, run optimize/cleanup, or force-unlock a retained lock. Upstream #1189 (draft) admits an inline s3:// root for cluster plan/apply/status/force-unlock/upgrade-ledger and for optimize/cleanup/repair/rebuild-full-text-indexes with --graph.

This builds the omnigraph-server image's CLI from v0.14.0 plus that change:

• docker/patches/omnigraph-cli-inline-s3-root.patch is #1189's crates/ diff (commit 32722615) regenerated against the v0.14.0 commit (60bfd19). The PR is based on upstream main, but every crates/ hunk applies to the tag as is (two at line offsets), so nothing needed backporting. Its docs/skills/changelog hunks are left out because none of them touch the build and one targets a page v0.14.0 does not have.
• A cli-patched stage fetches that commit by id, applies the patch and runs cargo build --release --locked -p omnigraph-cli on rust:1.97.1-slim-trixie (the toolchain upstream's rust-toolchain.toml pins, and the runtime's glibc). FROM cli-${OMNIGRAPH_CLI_SOURCE} AS cli picks between it and the release tarball's binary, and the runtime copies omnigraph from cli. The image gets an edu.mit.ol.omnigraph.cli-source label. omnigraph-server still comes from the tarball, and the witan image and witan setup are untouched, so they all keep the stock digests.
• just check-omnigraph-pins now also runs bin/check_omnigraph_cli_patch.py. It requires OMNIGRAPH_CLI_SOURCE=patched exactly when OMNIGRAPH_RELEASE_TAG equals the patch's base tag, a full base commit that the stage actually fetches, and exactly one install of /usr/local/bin/omnigraph in the runtime stage, from cli. The omnigraph-pins job gains setup-uv for it.
• A new omnigraph patched CLI workflow builds the image with the patched CLI and runs bin/qualify_omnigraph_cli_s3.py through the image's own binary against RustFS.
• prek.toml keeps the whitespace fixers off docker/patches/, because stripping a diff's blank context lines stops git apply matching.

I gated it on the pin rather than landing it live: OMNIGRAPH_CLI_SOURCE defaults to release, so at today's 0.13 pin the image is the same as before apart from the new label (cli-source=release), and a default build never touches the Rust stage (checked locally: the plain build log never mentions cli-patched or the rust image, and the image's omnigraph --version is 0.13.0). Once the 0.14 pin bump sets OMNIGRAPH_RELEASE_TAG=v0.14.0, the pin check fails until that same change sets OMNIGRAPH_CLI_SOURCE=patched. The patched CLI therefore ships with the bump and cannot ship next to a server from a different release. If this merges after the bump, the check makes this PR flip the default itself.

The qualification is meant to cover what upstream's e2e did not. It deploys a small cluster to a local root, loads four batches (one fragment and version each), and copies the root to S3 with aws s3 cp. Then, against the copy, with an inline s3:// target:

• cluster plan must see the copy as converged
• cluster apply of a config that adds a graph must complete converged, as a real write
• cluster status --deployment-id and apply --deployment-id --writers-stopped must replay it as lookup-only
• force-unlock releases each retained lock by its exact id
• query must still refuse
• cleanup without --yes must refuse
• optimize must write objects and cleanup --keep 1 --confirm --yes must remove them, with a force-unlock after each

Finally the root is copied back and queried locally to check that every loaded row survived.

Drop the patch, the three OMNIGRAPH_CLI_* args, the stage and the workflow once a release or an immutable edge build carries #1189, or once served maintenance (upstream RFC #1186 / #1177) replaces the direct CLI. The pin check passes on a Dockerfile with none of them.

This PR does not touch omnigraph_server_entrypoint.py, omnigraph_maintenance.py or omnigraph_plan_gate.py. They still address the CLI the 0.13 way, and moving them to an inline s3:// target is tk-rework-the-omnigraph-server-entrypoint-maintenan-4311a3.

How can this be tested?

Locally, on a 12-core WSL2 host that was also running other builds (load average 25 to 35 the whole time, so treat the times as an upper bound rather than what CI or Concourse will see):

• git apply --check of #1189's full patch against v0.14.0: every crates/ hunk applies (planes.rs and targets.rs at line offsets), and only docs/user/operations/upgrade-0.14.md is missing.
• docker build --build-arg OMNIGRAPH_CLI_SOURCE=patched succeeded. The cold cargo build --release --locked -p omnigraph-cli step took 2551s (about 42 minutes), on an empty cargo cache. The same build on the host took 56 minutes, but it ran alongside the docker build. Fetching, git apply and the version check take seconds.
• Image size barely moves. The patched omnigraph is 266,217,704 bytes and the stock v0.14.0 one is 266,341,208, and none of the Rust stage reaches the runtime image. The image is 637 MB, vs. 615 MB for today's 0.13 image, and that difference is the 0.13 to 0.14 binaries, not the patch.
• In the image, omnigraph --version prints omnigraph 0.14.0 and the label is edu.mit.ol.omnigraph.cli-source=patched.
• bin/qualify_omnigraph_cli_s3.py passed against RustFS (the same rustfs/rustfs:1.0.0-beta.12 digest upstream's CI pins), both with the host-built binary and through the image's own binary via the same docker run wrapper the workflow writes. The S3 copy had 48 objects before optimize, 63 after it and 45 after cleanup --keep 1, and all 100 loaded rows read back after the round trip. Every retained lock (apply, optimize, cleanup) was released by its exact id, and the root ended unlocked.
• Run with the stock v0.14.0 binary, the same script stops at the first S3 command with inline access supports local storage only.
• just check-omnigraph-pins passes (omnigraph-server CLI: release (patch base v0.14.0, server v0.13.0)). I ran bin/check_omnigraph_cli_patch.py's problems() against edited copies of the Dockerfile, and it fails each of these: the bump without the flip, the flip without the bump, an unknown selector, a short commit, a fetch of main, the runtime copying from fetch (alone or as well as from cli), a missing runtime ARG, partial pins, a missing patch file, and the pins dropped with the selector left in. It passes a Dockerfile with all of it removed.

A fresh-context code-review pass found that optimize could pass without doing anything and that apply on S3 was only ever a no-op. It also found three ways to get the stock CLI or the wrong commit past the pin check. All of those are fixed in the first commit. The second commit drops --as from optimize/cleanup, which the CLI refuses on those commands. The local run caught that.

In CI the omnigraph patched CLI job passed on 3c58a0d (https://github.com/mitodl/agent-kit/actions/runs/40769678496). The same counts came out (48, 63, 45 objects, 100 rows), and the qualification step took 13s. The cold image build took 30 to 37 minutes on ubuntu-latest (two runs overlapped, both with an empty gha cache). The earlier run on 4693716 failed only on the --as refusal that the second commit fixes.

Not verified:

• Real AWS S3. CI holds no AWS credentials, so RustFS stands in, and qualifying against a copy of the real QA root is left for QA, before the pin bump deploys.
• Recovery of a genuinely outstanding (interrupted) deployment on S3, and repair / rebuild-full-text-indexes on S3.
• How long and how much memory the Rust stage needs on a Concourse worker, and whether oci-build-task's cache keeps the cargo cache mounts between builds there. It only runs there once the bump selects it.

https://claude.ai/code/session_01FGZebE3YVtnu2oBXH1mKNR

[Copilot]

🔵 Needs a closer look

Unsupported --as arguments cause the pinned v0.14.0 CLI to reject cleanup and optimize calls, blocking qualification before maintenance can be exercised.

No open issues 🎉

What changed in this PR

Prepares the omnigraph data-tier image to use a patched CLI for stopped-server S3 operations when the release pin moves to v0.14.0, while keeping the current v0.13 default unchanged.

Changes:

• Adds a pinned source-build stage and upstream inline-S3-root patch.
• Adds CLI selection checks to the existing release-pin validation.
• Adds image-based S3 qualification against RustFS and protects patch files from whitespace rewriting.

File	Description
prek.toml	Excludes vendored patches from whitespace fixers.
justfile	Adds patched-CLI validation to the pin check.
docker/​README.md	Documents patched CLI selection and qualification.
docker/​patches/​omnigraph-cli-inline-s3-root.patch	Vendors upstream S3-root support and tests.
docker/​omnigraph-server.Dockerfile	Adds selectable patched CLI builds and a source label.
bin/​qualify_omnigraph_cli_s3.py	Exercises deployment, maintenance, unlocking, and data preservation on S3.
bin/​check_omnigraph_cli_patch.py	Checks patch inputs and runtime CLI selection against release pins.
.github/​workflows/​witan-tests.yml	Installs uv for the expanded pin check.
.github/​workflows/​omnigraph-cli-patch.yml	Builds the patched image and runs RustFS qualification.

_{Review effort: Balanced}