Repository navigation
fix(image): give the runner image hostname(1), and the tool surface the org's CI jobs actually call - #8
Conversation
… CI job needs
charly's own `ci` fails on the self-hosted pool for every PR, for a reason that
has nothing to do with any PR: the runner image has no `hostname`.
TestInvokeProvider_VenueDescriptor_Venueless_ShellDescriptor (and its SSH
sibling) prove a peer executed host-locally by running `hostname` over the
venue's shell and comparing it to os.Hostname(). The image has no such binary,
so that shell exits 127 and the `go` gate reds on every run (charly#806).
Measured inside the live runner container, not inferred:
$ podman exec charly-githubrunner-no-2 sh -c 'command -v hostname'
(nothing, rc=1)
$ podman exec charly-githubrunner-no-2 bash -lc 'hostname'
bash: line 1: hostname: command not found rc=127
`inetutils` is hostname's only Arch provider — there is no standalone package —
and it declares no Provides and no Conflicts, so it coexists with iputils' ping.
The same live probe turned up a second defect and a whole missing tool surface:
* the container ships no generated locale, so `locale charmap` reports
ANSI_X3.4-1968 (ASCII) and Python and friends use ASCII for the filesystem
encoding — a job touching a non-ASCII path fails as if the job were buggy;
* the pool is org-wide, but the image lacks tools the org's own workflows
call: golangci-lint (30 refs), npm/node (22/12), docker (16), tree (15),
shellcheck (7), cargo (6), actionlint (2).
So this adds `inetutils`, sets LANG=C.UTF-8 and TZ=UTC, and installs the job
tool surface the org actually calls — 24 packages, each with a `package:`
check in the candy's own plan, so a tool that is declared but not installed
fails the image build rather than a job at 02:00. `golangci-lint` is the
packaged v2.13.2: byte-for-byte the version charly's `ci` go-installs.
docker, rust/cargo, a JDK and passwordless sudo are deliberately NOT added;
the reasons are in the file rather than left to be guessed.
The `TZ` check asserts `printenv TZ`, not `date +%Z`: the image's
/etc/localtime is already UTC, so `date +%Z` passes either way and would be a
gate that cannot fail.
The var was silently telling a lie. The actions runner self-updates, and `~/actions-runner` is a persistent volume, so that update outlives the image: measured on the live fleet, `bin` symlinks to `bin.2.337.0` while the deployed image declares 2.334.0 and was built 2026-10-02 — with `bin.2.334.0` still on disk beside it. Rebuilding the image does not change the runner version. Operator decision: keep auto-update (it holds the fleet inside GitHub's supported range without anyone bumping the value) and make the file say so, rather than adding `--disableupdate` and making a version bump an operator duty on every rebuild. No behaviour changes here — the comment and the skill note now state what actually happens, and `--disableupdate`'s absence is recorded as deliberate.
Both described a toolchain and a version contract this change alters, and AGENTS.md's claim that the runner version pin is the RUNNER_VERSION var is now known to be false: the runner self-updates, and the state volume carries that update across rebuilds. AGENTS.md is this repo's own rulebook, so leaving it asserting a disproved pin would mislead the next agent. - README: the job tool surface, the LANG/TZ env, and the seeding-not-pinning note. - AGENTS.md: the corrected RUNNER_VERSION contract; the check-coverage rule (a tool is added only with a matching plan: check); and the extended list of what the R10 witness asserts. Verified: charly box validate -> OK, 0 warnings, 0 errors.
charly.yml twice asserted that the packaged `golangci-lint` "is v2.13.2 — byte-for-byte the version charly's ci `go install`s, so CI and this image cannot drift apart on the linter". Both halves are wrong, and the measurement inverts the state: - the packaged Arch package is 2.14.0, not 2.13.2 (measured in the built image: `podman run --entrypoint golangci-lint ghcr.io/opencharly/githubrunner:check-githubrunner-pod-2026.278.2322 version`); - charly's `ci` never uses the packaged copy — its lint step runs `go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.13.2` and lints with that (ci.yml:97). So the two DO differ, and the packaged copy cannot decide a CI lint verdict. The honest reason to ship it is what it actually buys: a job can lint without paying a `go install` of its own. Corrected in both places — the package list and the skill's tool-surface paragraph. Verified: charly box validate -> OK, 0 warnings, 0 errors.
…LS one
The skill entity's environment table — the table this branch extends with the
`LANG`/`TZ` rows — documented two variables that did not match the image:
* `RUNNER_GROUP | Default` was documented as part of the deploy contract, but
the candy declares it in `env:` and NOTHING reads it — the registration hook
passes `--name`/`--labels`/`--work`, never `--runnergroup`, and no other repo
in the org sets or reads it either (`git grep RUNNER_GROUP` over the umbrella
returns only the declaration and its copies of this doc). Wiring it would be
an unprovable runtime change: `config.sh --runnergroup` is org-only, so it
would break the user/repo-level registration the `RUNNER_ORG` description
advertises, and the token-guarded hook cannot be exercised by a bed. So the
declaration is removed rather than a doc written around it (R4a/R5).
* `RUNNER_LABELS` was consumed (`--labels "${RUNNER_LABELS}"`) but declared
nowhere, so an operator had no supported way to set the one value every
self-hosted job selects on. It is now an `env_accept` entry next to
`RUNNER_ORG`, and the table describes it instead of the dead knob.
Behaviour is unchanged: the hook's `: "${RUNNER_LABELS:=opencharly}"` fallback
still yields `opencharly` when nothing sets it, so the registered label set is
identical to before. What changes is that the image now states truthfully which
variables exist and what they do.
Swept in this same commit (R5): `git grep RUNNER_GROUP` over the repo is empty
(exit 1) — env declaration, skill table row and README row all gone.
Verified: charly box validate -> OK, 0 warnings, 0 errors.
…nt disproves
The `LANG` entry this branch adds carried a justification that does not survive
contact with the image it describes. It said: "Under the POSIX locale Python 3
and other runtimes fall back to ASCII for the filesystem encoding, so a job that
touches a non-ASCII path fails."
Measured in the built image, with LANG unset:
$ podman run --rm --entrypoint env <img> -u LANG -u LC_ALL bash -c \
'locale charmap; python3 -c "import locale,sys;print(locale.getpreferredencoding(False), sys.getfilesystemencoding())"'
ANSI_X3.4-1968
UTF-8 utf-8
CPython >= 3.7 coerces the C locale itself (PEP 538/540), so Python never falls
back to ASCII here and the failure story it was used to justify cannot be
reproduced. An unprovable consequence dressed as a fact is exactly what R1
forbids, so it is replaced with what the probes actually showed:
* `locale -a` lists only C, C.utf8, POSIX — true, and kept;
* under the default POSIX locale a non-login process's codeset is ASCII,
`locale charmap` -> ANSI_X3.4-1968 — true, and kept;
* /etc/locale.conf already names C.UTF-8, but that reaches only login/PAM
sessions, while a charly `check` and a job process are neither — so the
`env:` declaration is what applies, which is stated now;
* CPython coerces, so "Python breaks" is explicitly NOT the justification.
The check description loses the same clause ("so jobs never fall back to ASCII
filesystem encoding") and states the locale the tools see instead. The check
itself is unchanged and still fails without the env entry.
Verified: charly box validate -> OK, 0 warnings, 0 errors.
…ly exists The `golangci-lint` package comment added earlier on this branch pointed at `ci.yml:97` for charly's own pinned install. There is nothing at :97. The step and its install are at :90 and :92: $ grep -n 'golangci-lint@\|name: golangci' charly/.github/workflows/ci.yml 90: - name: golangci-lint 92: go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.13.2 A comment that diverges from the file it cites is an incident, not a typo (R1), so it is corrected here — and the citation now also names the step, so the claim stays checkable when the line number moves: charly's `ci` `golangci-lint` step installs its own pinned `@v2.13.2` on every run (ci.yml:92) Verified: charly box validate -> OK, 0 warnings, 0 errors.
Review — BLOCKHead SHA: Change class: candy config ( Blocks
Verdict: BLOCK ollama-cloud/deepseek-v4.1-flash — action-review. |
Review — PASSHead SHA: Security screen: PASS — no secrets/egress, no guardrail weakening, no scope mismatch; the diff is candy config + the same repo's docs, and the skill/var/env text is data, not instructions to me. Comment intake (1 comment, BLOCK by @github-actions[bot] at head
All three reproduce against the prior head only; none survives against Verdict: PASS ollama-cloud/deepseek-v4.1-flash — action-review. |
Summary
Closes #7.
check-githubrunner-podbuildsghcr.io/opencharly/githubrunner— the image theorg's self-hosted runners execute jobs in. Three defects in it: two break the pool, one makes
the image lie about itself.
1. The image had no
hostname(1), so charly's own test-suite cannot pass on the poolVerbatim from the
gocheck-run ofcharly#807(job 111961186574,
started
2026-10-05T20:19:41Z, completed2026-10-05T20:27:01Z— 440 s, inside the requiredcheck's 600 s budget, so this is a real test failure and not a duration stall):
charly's venue descriptor shells out to
hostnameto prove a peer ran host-locally. Therunner image carried
coreutils, but Arch'scoreutilsships nohostname— it lives ininetutils. The test therefore fails for the image, not for the code under test, and everyrepo whose CI runs
go test ./...on the pool inherits the red.The fix is the package plus the check that would fail without it:
2. The image was missing most of the tool surface a CI job reaches for
24 further packages, each with its own
package=<name>check in the plan, so adeclared-but-missing tool fails the image build rather than a job at 02:00:
treeshellcheckshfmtactionlintgolangci-lintnodejsnpmpython-pipuvcmakeunzipzip7zipwgetrsyncyqripgrepmoreutilsbclesslsofstraceopenbsd-netcatbind, plusLANG=C.UTF-8/TZ=UTCwith the two checks that assert them.None of these names existed in the candy before — measured against
origin/main:My first draft of this PR justified
LANG=C.UTF-8by claiming "Python 3 falls back to ASCIIas the filesystem encoding, so a job that touches a non-ASCII path fails". Probing the built
image disproved it — CPython ≥ 3.7 coerces the C locale itself — so the claim is gone from
the candy, its skill entity and this body rather than shipped (R1). What the probes did show,
with
LANGunset:So: the image generates no locale, a non-login process lands in POSIX and its codeset is
ASCII, and Python is emphatically not the victim. The image already has
/etc/locale.confnaming
C.UTF-8, but that reaches only login/PAM sessions, and a charlycheckand a jobprocess are neither — the
env:declaration is what applies, which is precisely why thelocale charmapcheck genuinely fails when it is removed.C.UTF-8is built into glibc, sonothing needs generating. The
TZcheck asserts the variable (printenv TZ), deliberatelynot
date +%Z: the image's/etc/localtimeis already UTC, so adate-based check would pass either way and provenothing.
3. The skill entity documented a knob that does nothing, and omitted the one jobs select on
The
| Variable | Mechanism |table in this candy's own skill entity carried| RUNNER_GROUP | Default |. The candy declaresRUNNER_GROUPinenv:, and nothing readsit: the registration hook passes
--name/--labels/--workand never--runnergroup, andno other repo in the org sets or reads it either —
git grep RUNNER_GROUPover the umbrellareturns only the declaration and copies of this table. Its mirror image is
RUNNER_LABELS,which the hook does consume (
--labels "${RUNNER_LABELS}") but which was declared nowhere,so the one value every self-hosted job selects on had no supported way of being set.
Because this PR extends that very table with the
LANG/TZrows, the dead row is this PR's tocure rather than something to build on top of (R2, R5, and the reviewer's B14(c)).
RUNNER_GROUPis deleted — env declaration, skill-table row and README row, in this commit.Wiring it instead would have been an unprovable runtime change:
config.sh --runnergroupisorg-only, so consuming it would break the user/repo-level registration the
RUNNER_ORGdescription advertises, and the token-guarded hook can never be exercised by a bed.
RUNNER_LABELSbecomes anenv_acceptentry besideRUNNER_ORG.This deviates from the plan I filed on #7, on purpose. #7 predicted the fix would need its
own PR after this one, on the grounds that a registration-hook change is not bed-observable.
Deleting a dead
env:entry and declaring a live one is not a hook change — the hook isuntouched, byte-identical — and it is bed-observable, because both halves show up in the
built image's labels, which the R8 read-back below reads from the real artefact. Folding it in
is also what R2 requires: this PR edits the very table that carried the dead row, so a separate
PR would be parking a defect this one is standing on. #7 is updated with a
STATUSsaying so,and it closes with this PR.
Behaviour is unchanged: the hook's
: "${RUNNER_LABELS:=opencharly}"fallback still yieldsopencharlywhen nothing sets it, so the registered label set is byte-identical to before.Files changed
charly.ymlis the fix (packages, env, 28 plan checks, and the skill entity's text).README.mdandAGENTS.mdare the same repo's own descriptions of the image, corrected tomatch it: the README called
RUNNER_VERSIONa pin and listedRUNNER_GROUPin itsEnvrow,and AGENTS.md repeated the pin framing — both now say the runner self-updates and that the
statevolume carries the update across rebuilds.A third correction was to this branch itself. A sentence I first wrote into
charly.ymlclaimed the packaged
golangci-lint"is v2.13.2, byte-for-byte the version charly'scigo installs, so CI and this image cannot drift apart on the linter". Probing the built imagedisproved both halves — the packaged build is 2.14.0, and charly's lint step installs its
own pinned
@v2.13.2and lints with that — so the sentence was rewritten inside this branchbefore this PR opened. What the diff asserts is a measurement, not a parity claim:
The honest reason to ship the package is what it buys: a job can lint without paying a
go installof its own.The
env_candyline below is the "before" for the R8 read-back in the next section — thisbranch's first head, so it still declares
RUNNER_GROUPwhileLANGis already applied:How tested
Change class gate — candy config. The owning repo's rulebook names two gates:
charly box validatefor the config itself, and a composing box's check bed for the live behaviour. Bothwere run on the final tree.
charly box validate(repo root, final tree49822b8):The bed —
charly check run check-githubrunner-pod, adisposable: truepod bed declaredin
distro-cachyos/charly.yml, on the final tree. It composes this candy into the runner imagevia
CHARLY_REPO_OVERRIDE, so the run builds the branch's bytes, notmain's:The override really was honoured — proven by check ids, not by a log line. The
LOCAL OVERRIDEline is printed for aplugin:load but not for a box-candy load, so the proof isthat 28 check ids exist only on this branch:
All 28 pass at every bed step that can run them —
check-image26 PASS / 2 SKIP (the twocontext: runtimechecks are correctly skipped in box mode rather than silently passed),check-live28 PASS,feature-run28 PASS, and the same 28 PASS again aftercharly updatein
check-live-rebuildandfeature-run-rebuild. Zero FAIL in any step. Ascheck-liveprinted them on the deployed pod, where the runtime context is active:
Working tree at the head being pushed — the
git statushalf of R6, run after the lastcommit, and the tree the gates above ran against:
R8 — generated artefact labels verified after build
The change alters a candy's
packages:/env:, which changes emitted Containerfile sectionsand the
ai.opencharly.*labels. Both were read back from the built image withcharly box labels, and both halves move exactly as the diff says.RUNNER_GROUPdisappearsfrom
env_candy;RUNNER_LABELSappears inenv_acceptwith the description from the diff;no
ai.opencharly.*label is empty or missing.Before: this branch's image from before the knob fix
(
ghcr.io/opencharly/githubrunner:check-githubrunner-pod-2026.278.2322, head87bda93) —env_acceptcarried onlyRUNNER_ORG. After:ghcr.io/opencharly/githubrunner:check-githubrunner-pod-2026.279.0009, built by the run above:R9 — target exercise
The artefact exercised here is the box image, not the
charlybinary; the bed rebuilds it fromsource on every step that needs it (
image-build, thencharly update+check-live-rebuild),and the branch-only check ids pasted in
## How testedabove are that rebuild's own output.Every new runtime OS dependency this PR adds is declared in the candy that installs it — this
candy's
packages:list, the file the diff edits; none is acharlyruntime dependency, so thecharlycandy'spackaging:section is untouched by design.Review round 1 — BLOCK, all three findings addressed
Run
37392501882(check-run112040633514) blocked with three findings. All three were real,and two of them were defects in the diff itself rather than in the prose around it. Each is
fixed at the head this body describes; the second one moved the head, which is why the bed
below was re-run against the new tree rather than reused.
The "Files changed" numbers were stale. The body pasted
300 4 charly.ymland335 insertions(+), 18 deletions(-), both measured against the previous head. Re-run atthe head this body describes:
The same staleness had moved the
git grep RUNNER_LABELSline numbers, corrected in placein
## Summaryabove. They are pasted from a frozen tree now, not from the tree as it wasmid-edit.
charly.ymlcited aci.ymlline that does not exist. Thegolangci-lintpackagecomment pointed at
ci.yml:97; the step and itsgo install …@v2.13.2are at:90and:92. A comment diverging from the file it cites is an incident, not a typo (R1), so it isfixed in this head — and the citation now names the step as well, so it stays checkable when
the line number moves:
"no version number is asserted anywhere in the diff" was false. The diff does assert
2.14.0and@v2.13.2. The sentence now says what it meant — no version-number parity isasserted, and none ever was after the correction above.
Rulebook compliance
R0 — loaded before editing:
/charly-image:image(thecharly boxfamily, candy/boxauthoring, the OCI-label contract R8 checks),
/charly-internals:git-workflow(branch/PRloop, body-before-push, no-force-push),
/charly-check:check(bed semantics, R10), plus theowning repo's own
AGENTS.md, which is the rulebook in force insidepod-github-runner.The change follows the image skill's authoring rule (a
candy:node carryingbase:; nobox:kind) and its validate gate.R1 — the failure is root-caused, not framed:
hostnamemissing from the image →exit 127→ the venue descriptor's host-local proof fails → the pool'sgojob is red forevery repo. The 440 s duration rules out a timeout as the cause. Three claims of mine died
the same way — probed against the built image, found false, rewritten rather than shipped:
the
golangci-lint"v2.13.2 parity" sentence, theRUNNER_GROUPknob that nothing reads,and the
LANGrationale's Python story. None is left as an exception or a note; the textnow says what the probes showed.
charly box validatereports 0 warnings, 0 errors onthe final tree, pasted above.
R2 — nothing surfaced is parked. The
RUNNER_GROUP/RUNNER_LABELSdivergence is fixed inthis commit, not routed to a follow-up, precisely because this PR extends the table that
carried it. No "pre-existing"/"out of scope"/"follow-up PR" framing appears anywhere above.
R3 — no duplication: the new plan checks are one per package, and the CI tool surface is
declared once in the candy rather than restated in each consuming workflow. The container
runtime stack stays where it was declared —
layer-container-nestingowns it, and this candydeclares none of it:
(empty — 41 packages declared here, 13 there, none in both)
R4 — no workaround anywhere in the diff: the missing tool is installed at the image layer
that owns it, not shimmed into job scripts, and no check is retried or slept through. The
documentation describes the image that now exists rather than the image that was wanted
(R4a).
R5 — hard cutover:
RUNNER_GROUPis deleted from all three places that named it (env,skill table, README) in the same commit; no transitional or dual-mode path is left behind;
the grep evidence that the sweep is complete is pasted above.
R6 —
git statuswas run before every commit and again after the last one; the cleantree at the head being pushed is pasted above. No force-push and no hook bypass was used at
any point on this branch.
R7 — the gates this diff is in were run on the final tree and their real output is pasted
above:
charly box validate(the candy gate) and the composing box'sdisposable: truecheck bed. No gate outside this change class is claimed.
R8 — generated-artefact labels verified after build; see the
## R8section above.R9 — target exercise asserted; see the
## R9section above.R10 — the proof is a
disposable: truebed on the final committed tree, with a freshrebuild and zero warnings; the change ships the check coverage that fails without it (28 ids,
all absent from
origin/main). Attribution isfully tested and validatedand no higher.group/ naming rules — untouched; the diff adds no new entity and renames nothing.Ledger — the session ledger carries this PR, the bed run and its verdict.
Change classification
charly.yml(packages, env, plan,skill text),
README.md,AGENTS.md— no Go, no schema, no CI workflow, no pin.charly box validateplus a composingbox's
disposable: truecheck bed (check-githubrunner-pod), both on the final tree.fully tested and validated— the changed paths (image packages, env,the 28 new plan checks) executed against a fresh rebuild in the bed, and the label read-back
confirms the built artefact carries the change. The
env_accept/RUNNER_GROUPedit is inertconfig, covered by
validateplus the label read-back rather than by a runtime path, becausethe registration hook it feeds is token-guarded and cannot be exercised by a disposable bed.
marketplace/distros/skills/github-runner/SKILL.mdand the docs site carry copies of this candy's skill entity, dead
RUNNER_GROUProwincluded. They are generated artefacts re-emitted by
charly marketplace generateand thedocs regen and are not hand-editable, so they cannot be fixed here — this candy is their
source, and they follow from it. Nothing found was parked.
Agent:
readme-truth· session2a317871-ba03-4713-89ba-8edbd4059c27Assisted-by: Claude Code ollama-cloud/deepseek-v4.1-flash (fully tested and validated)