ci: publish groups images for arm64 - #8
Merged
Merged
Conversation
The release workflow builds a single-architecture image, so ghcr.io/agynio/groups is amd64-only. On an arm64 cluster every pod referencing it sits in ImagePullBackOff — which is where the arm64 local bundle VM image currently fails to converge, since users, apps, gateway and agents-orchestrator all depend on groups. Add QEMU emulation and build linux/amd64,linux/arm64 so the published tag is a manifest list covering both. Co-Authored-By: Claude <noreply@anthropic.com>
vitramir
added a commit
to agynio/bundle-vm
that referenced
this pull request
Jul 29, 2026
* fix(ingress): right-size the istiod reservation
The chart reserves 500m CPU and 2Gi for pilot by default, sized for a
large mesh. On this single-node VM that reservation alone was a quarter
of schedulable memory and blocked workloads from scheduling, while pilot
actually runs well under 100Mi.
* fix(deploy): reserve resources for postgres, k8s-runner and metering
These three are deployed outside the platform umbrella chart and declared
no resource requests, leaving them BestEffort — the first pods evicted
under memory pressure, with the database among them.
Requests are sized from measured idle usage; limits leave room to burst.
* feat(bundle): take a per-install bootstrap token from the host
The platform is installed while the image is built, so the bootstrap token
the build used shipped inside every copy of that image: identical on every
machine running a given published version, present on the CDN, and not
rotatable without a rebuild.
The Gateway now reads the token from a Secret rather than from its
Deployment spec, and the host supplies a value per install. The image ships
a placeholder that only the build's own provisioning uses, so a published
image carries nothing usable against a running VM.
* fix: repair the shipped image and make the ingress port configurable
Four defects made the published image unusable, and one design flaw made it
usable only on a single host port.
Gateway bootstrap token. The gateway chart renders CLUSTER_ADMIN_TOKEN from a
literal value and has no clusterAdminTokenSecret support, so reading it from a
Secret left CLUSTER_ADMIN_IDENTITY_ID set with no token — a combination the
gateway rejects at startup by design. It crashlooped, apps-provision never
reached it, and the bake failed after 20 minutes of waiting. The token is a
literal again, and set-bootstrap-token.sh rewrites the Deployment environment,
which is what local-bundle.md specifies ("reads it from its environment like
any other deployment").
Authorization model. The image shipped an authorization-openfga Secret naming a
model OpenFGA does not have, so every permission check failed with
authorization_model_not_found. Nothing looked wrong — pods went Ready and the
Gateway served — until something checked a permission; runner enrollment is one,
so the k8s-runner crashlooped too. install-platform.sh now resolves the model
before finalizing the image and re-runs the idempotent migration if it is
missing.
Ziti sidecar. Unset, the orchestrator fell back to openziti/ziti-tunnel:1.6.15
in code, two majors behind the controller and router this image installs. Pinned
to bootstrap's 2.0.0-pre8 and pre-pulled, since no agent workload runs during the
bake and the first agent on a fresh VM would otherwise fetch it over the network.
groups. Pinned to 0.1.1, the first tag published for arm64 (agynio/groups#8);
the subchart's appVersion still resolves to an amd64-only 0.1.0, which left
every arm64 pod in ImagePullBackOff.
Configurable ingress port. The host port was substituted into everything at
build time, so an image built for 2496 could not serve 2497. The two uses are
now distinct: INGRESS_PORT is an internal constant baked into OpenZiti's
advertised addresses and enrollment JWTs, where it can never change and never
needs to (cluster DNS resolves those names to ClusterIPs); INGRESS_HOST_PORT
survives only in browser-facing URLs, which set-ingress-port.sh repoints on
first boot.
Build failures also print the cluster's own account of themselves now. The
build VM is discarded on failure, so "context deadline exceeded" was all anyone
ever got.
* ci: build and publish the platform image from CI
amd64 builds in GitHub Actions on a v* tag or manual dispatch. arm64 does not:
GitHub offers no arm64 runner with nested virtualization, and without KVM the
bake takes hours rather than minutes. Until such a runner exists arm64 is
released from a maintainer's machine — which is exactly why that path is a
script and not a list of commands in a README.
scripts/release.sh is that one entrypoint, used unchanged by both: fetch base,
build, package, publish to GHCR and the CDN. Every precondition (tools,
credentials, hardware acceleration, free disk) is checked before the bake, so a
missing one fails in the first second instead of forty minutes in.
Two things blocked CI entirely. The base image was resolved from a sibling
checkout's build output, which no CI runner has and which makes "whichever
qcow2 is lying around next door" a build input; it is now pinned by
BASE_IMAGE_VERSION and pulled from GHCR, cached and checksum-verified.
Publishing to the CDN — where agyn local start actually looks — existed only as
a commented s5cmd line in the README, so it depended on someone remembering it;
it is now part of the release, and latest.json is written after the uploads
succeed so latest never names a half-uploaded directory.
CI also runs the checks that need no VM: shell syntax, shellcheck, packer fmt
and validate, and a YAML pass that rejects duplicate top-level keys — the defect
that once silently wiped a values block and shipped a crashlooping service.
* ci: release on push to main as well as on tags
A push to main publishes sha-<commit>: the image is then known to still build,
and the platform build can pin a specific commit's base. A tag publishes the
version users pin, and is the only thing that moves `latest` — sha builds are
not what anyone should be pointed at by default.
Neither runs on pull_request. A release publishes, and a PR is not a release.
* fix(bundle): run one replica of each service in the VM
The chart defaults to two replicas of the gateway, orchestrator and LLM
proxy, which is right for a cluster: leader election gives the
orchestrator failover, and the others share load. This image is a
single-node VM, where a second replica buys none of that and competes for
the same CPU and memory.
It also makes the source-run path usable: DevSpace syncs to one pod, so a
second replica never receives sources and dies at its sync timeout.
* ci: fix shellcheck SC2015 in release.sh
A && B || C is not if-then-else: usage would run when arch was set but version
was empty only by luck of the exit codes. Spelled as an if.
* ci: pin the base image to the first one CI published
BASE_IMAGE_VERSION was 0.1.0, a placeholder that was never published — nothing
could have resolved it. agynio/bundle-vm-base now builds and publishes from
main, so pin the artifact it produced.
amd64 only for the moment. An arm64 build still needs BASE_IMAGE pointed at a
locally built qcow2, until the base is released for arm64 as well.
* fix(fetch-base): read the artifact from the path oras pulls it to
oras records each layer under the path it was pushed with, and the base repo
pushes from artifacts/<arch>/, so a pull recreates that nesting instead of
dropping the files at the output root. fetch-base looked at the root and
reported the artifact as empty — for an artifact it had just downloaded in
full.
Only reachable once a base image existed to pull, which is why it survived
review: there was nothing published to try it against.
* Serve sandbox shells from the VM
agyn sandbox connect opens a WebSocket to the Terminal Proxy from the host, so
the proxy needs a route of its own: terminal.<domain> joins the platform
gateway's host list and gets a VirtualService pointing at the service. No
timeout is set on that route because Istio leaves HTTP request timeouts disabled
by default and rejects an explicit 0s, and a shell holds its socket open for as
long as the user keeps typing. The URL handed back in a ticket carries the
forwarded host port rather than the in-cluster one, which is why
set-ingress-port.sh now rewrites it alongside the chat and media origins: a
stale port there hands out tickets for a socket nobody is serving.
The proxy dials the runner directly instead of over the overlay. Its Ziti
identity has no provisioning path yet, the same reason egress-gateway stays off,
and the only runner in this VM is the in-cluster one, so the overlay would reach
nothing a Service address does not. Tickets are signed with a fixed dev key like
the rest of the secrets in this image. The umbrella pin moves to 0.5.11 because
that is the release carrying the terminal-proxy subchart.
* feat: run the Terminal Proxy on the overlay in the VM
The VM disabled Ziti for the Terminal Proxy because its identity had no
provisioning path. It has one now, so the VM runs the same dial as
production rather than a direct ClusterIP that only works because this
VM's single runner happens to be in-cluster.
Requires agyn-platform 0.5.12: 0.5.11 carries the terminal-proxy chart
that mounts a pre-enrolled identity Secret nothing creates, so enabling
Ziti against it would leave the pod unable to start.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: point the VM at its own terminal WebSocket URL
The VM set terminal-proxy.terminalProxy.websocketUrl, which the umbrella
ignored — it configured the service from a raw env list wired to
platform.externalUrls.terminalProxyWebSocket, which the VM never set. So
the URL handed to clients was the chart's example.com placeholder, and a
sandbox attach dialed a host that is not this machine.
Sets the platform value and repeats it on the subchart. Both are needed:
the umbrella defaults the subchart from a YAML anchor, and anchors
resolve when the chart's own values are parsed, so an override does not
flow into it. The chart fails the render when the two disagree.
Repins to agyn-platform 0.6.2, which carries the Terminal Proxy's
ephemeral identity. Drops runnerAddress, dead now that Ziti is on.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ghcr.io/agynio/groupsis published amd64-only — both0.1.0and0.1.1are single-architecture manifests. On an arm64 cluster, every pod referencing it lands inImagePullBackOff.This currently blocks the arm64 local bundle VM image: the platform bake never converges, and
groupsis not optional there —users,apps,gatewayandagents-orchestratorall hold aGROUPS_ADDRESS/GROUPS_GRPC_TARGETpointing at it.Change
Add
docker/setup-qemu-actionand buildlinux/amd64,linux/arm64in the release workflow, so the published tag is a manifest list covering both. No application code changes.Verification
After the next release,
docker manifest inspect ghcr.io/agynio/groups:<tag>should list bothamd64andarm64entries — today it lists onlyamd64.🤖 Generated with Claude Code