Repository navigation
bug(agents): agent launcher fails --publish-image on transient registry push timeouts #4884
Description
Related to #3469. Found while running the registry delivery path from #4744.
User Story
I run gator against a remote Kubernetes gateway, publishing the tooling image with scripts/agents/run.sh --publish-image. I directly hit push failures that a retry would have absorbed, so that one slow layer upload doesn't throw away a completed build.
Problem Statement
--publish-image runs one ce push and fails the launch on any error (scripts/agents/image.sh, deliver_agent_image: ce push "$image_tag" || fail "image push failed"). Large layer uploads to Artifact Registry intermittently fail with net/http: timeout awaiting response headers. Each failure aborts the launcher after the image is already built and some layers are already uploaded.
Impact / Why This Matters
The gator tooling image is about 450 MB compressed. Two consecutive --publish-image --build-only runs failed on a different large layer each time. A plain docker push of the same local tag then succeeded on its second attempt. That means rerunning the launcher by hand until it works, or pushing outside the launcher and resolving the digest manually. Doing it by hand also bypasses the launcher's digest check.
Acceptance Criteria
- A transient push failure is retried a bounded number of times before the launcher fails.
- Each retry is logged with the attempt number and the underlying error.
- After a successful retry, the launcher still resolves and prints the digest-pinned
Tooling image:reference. - A persistent failure (for example an authentication error) still fails with an actionable error and no sandbox, provider, or settings changes.
Reproduction Steps
- Configure Docker auth for an Artifact Registry repository the user can push to.
- On an arm64 Mac, run
DOCKER_DEFAULT_PLATFORM=linux/amd64 ./scripts/agents/run.sh --agent gator --gateway <gateway> --publish-image <region>-docker.pkg.dev/<project>/<repo>/gator:<tag> --build-only. - Observe intermittent
failed commit on ref "layer-sha256:…": … net/http: timeout awaiting response headersfollowed byerror: image push failed.
Environment
- OpenShell: launcher from feat(agents): support Kubernetes Gators with reusable tooling images #4744 (
c8afb7321); gateway0.1.6-dev.26+g229382e51 - OS: macOS (arm64), Docker with buildx emulating linux/amd64
- Runtime, deployment, or integration: Kubernetes compute driver on a managed cloud cluster; Artifact Registry
Suggested UX (if applicable)
openshell-agent-launcher: Publishing '…/gator:payload-13-7fa522fae'.
openshell-agent-launcher: push attempt 1/3 failed: timeout awaiting response headers; retrying in 5s
openshell-agent-launcher: Tooling image: …/gator@sha256:e88f5a66…
Logs
failed commit on ref "layer-sha256:eda61e0e…": failed to do request: Put "https://<region>-docker.pkg.dev/artifacts-uploads/…": net/http: timeout awaiting response headers
error: image push failed
Activity