From d29c03a90c9809616f69c4cae3eddb4f7cdec858 Mon Sep 17 00:00:00 2001 From: Eric Moore Date: Wed, 7 Oct 2026 21:06:02 -0500 Subject: [PATCH 1/3] fix(save): the LRU prune survives an early break; a failed save is loud (#3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prune piped `find | sort -n` into a `while` loop that breaks once the budget is met. On a large target/ that leaves sort still writing: it takes SIGPIPE (or, on the runner, EPIPE: "sort: write failed: 'standard output': Broken pipe", exit 2), pipefail fails the pipeline, and set -e kills the step before tar/push. Callers' continue-on-error hid it, so every save that took the prune path published nothing. - scripts/prune.sh: cc_prune_lru writes the sorted list to a temp file and reads it, so there is no writer for the break to kill. Same LRU semantics; the report now shows before→after instead of a subshell value. - save/action.yml: inputs reach the push script through env (no ${{ }} splices). Each stage sets STAGE; an EXIT trap turns any failure into a ::warning:: (fail-open, exit 0) or, with the new `strict: "true"` input, an ::error:: that fails the step. New outputs `published`, `reason`, `key`. A failure after the push keeps published=true. `nothing to cache` is a warning too. _cc_timeout's kill of an exited watchdog no longer trips set -e and masks the 124. - selftest: strict save, and assert `published`. - README: the outputs, strict, and why to gate on `published`. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016kQb87339uTdFWivk3j2hF --- .github/workflows/selftest.yml | 8 ++ README.md | 44 +++++++++++ save/action.yml | 129 +++++++++++++++++++++++++-------- scripts/prune.sh | 42 +++++++++++ 4 files changed, 194 insertions(+), 29 deletions(-) create mode 100644 scripts/prune.sh diff --git a/.github/workflows/selftest.yml b/.github/workflows/selftest.yml index 94502fc..b5187cc 100644 --- a/.github/workflows/selftest.yml +++ b/.github/workflows/selftest.yml @@ -43,6 +43,7 @@ jobs: cp ~/.cargo/registry/cache/blob.bin ~/ciriscache_blob.before - name: Save + id: save uses: ./save with: package: ghcr.io/cirisai/ciriscache-selftest @@ -52,6 +53,13 @@ jobs: alias-keys: selftest-${{ matrix.os }}-${{ github.run_id }}-${{ github.run_attempt }}-alias paths: "~/.cargo/registry/cache target/deps" token: ${{ secrets.GITHUB_TOKEN }} + strict: "true" # a failed save must red the selftest + + - name: Assert published + shell: bash + run: | + test "${{ steps.save.outputs.published }}" = "true" \ + || { echo "save did not publish: ${{ steps.save.outputs.reason }}"; exit 1; } - name: Wipe local copies shell: bash diff --git a/README.md b/README.md index 6d13243..b4a47cc 100644 --- a/README.md +++ b/README.md @@ -81,6 +81,50 @@ re-upload): An alias that already exists is moved to the new manifest (correct for a rolling `phead`). A retag failure is non-fatal — the primary save still counts. +## Did the save publish? (`published`, `strict`) + +A save is fail-open by default: if any stage fails (oras login, the +`max-size-mb` prune, tar, oras push or its 20-minute timeout) the step emits a +`::warning::` naming the stage, sets `published=false` and `reason`, and exits +0 so the build stays green. A retag failure after the push is also a warning, +and `published` stays `true` (the primary key is up). + +| output | value | +|---|---| +| `published` | `true` only when the blob was pushed under `key` | +| `reason` | why not, e.g. `prune failed (exit 2)`, `oras push failed (exit 124)`, `nothing to cache`; empty when published | +| `key` | the normalized key the save targeted | + +**Check `published`, not the step's outcome.** Fail-open means a failed save +does not fail the job, and neither does a caller's `continue-on-error: true`. +Before CIRISCache#3 a save could fail on every run for weeks behind +`continue-on-error` with nothing but a hidden red step to show for it. You no +longer need `continue-on-error` on `save`; if you keep it, the warning is still +emitted. Where a missing blob must red the build (a release that downstream +repos exact-pin), set `strict: "true"`: the failure becomes an `::error::` and +fails the step. Or gate a later step on the output: + +```yaml + - uses: CIRISAI/CIRISCache/save@v1 + id: save + with: + key: ${{ env.CACHE_KEY }} + token: ${{ secrets.GITHUB_TOKEN }} + - if: steps.save.outputs.published != 'true' + run: | + echo "::error::cache not published: ${{ steps.save.outputs.reason }}" + exit 1 +``` + +## Tests + +`bash tests/prune_test.sh` and `bash tests/save_step_test.sh` (bash, coreutils, +GNU find; no registry) run on every PR via `.github/workflows/test.yml`. The +first checks the LRU prune under the step's `bash -e -o pipefail`, with SIGPIPE +both default and ignored as on the runner. The second runs the save step's +script verbatim with a stub `oras` and checks every failure path's annotation +and outputs. `selftest.yml` is the live GHCR round-trip. + ## Why not R2 / S3 / sccache-remote? Those work, but each adds an external account + credential = another point of diff --git a/save/action.yml b/save/action.yml index a5a8467..37ad39e 100644 --- a/save/action.yml +++ b/save/action.yml @@ -42,6 +42,8 @@ inputs: re-enters with a fresh mtime — mild steady-state churn in exchange for a bounded blob. Empty (default) = disabled. Never touches ~/.cargo (registry/git are lockfile-keyed, not generational). + Implemented in scripts/prune.sh; a prune failure is reported like any + other failed stage (see `published`). default: "" token: description: > @@ -50,6 +52,30 @@ inputs: — the repo must be granted write access to the package once (see docs/DESIGN.md). No PAT / org secret required. required: true + strict: + description: > + "true" = a failed save is an `::error::` and fails the step. Default + "false" = fail-open: a failed save is an `::warning::`, the step exits 0 + and `published` is "false". Either way the reason is in `reason`. + default: "false" + +outputs: + published: + description: > + "true" only when the blob was pushed under `key` (alias retags are + best-effort and do not change it). "false" on any failure or when there + was nothing to cache. Gate on THIS, not on the step's outcome: under the + default fail-open (`strict: "false"`) or a caller's `continue-on-error`, + a failed save does not fail the job. + value: ${{ steps.push.outputs.published }} + reason: + description: > + Why `published` is "false" (e.g. `prune failed (exit 2)`, `oras push + failed (exit 124)`, `nothing to cache`); empty when published. + value: ${{ steps.push.outputs.reason }} + key: + description: "The (normalized) cache key the save targets." + value: ${{ steps.k.outputs.key }} runs: using: composite @@ -72,18 +98,55 @@ runs: echo "CIRISCache key: $KEY" - id: push shell: bash + # Inputs arrive as env, not `${{ }}` splices, so the script below is + # plain bash: tests/save_step_test.sh runs it verbatim with a stub oras. env: CIRISCACHE_TOKEN: ${{ inputs.token }} + CC_REF: ${{ steps.k.outputs.ref }} + CC_ACTOR: ${{ github.actor }} + CC_PATHS: ${{ inputs.paths }} + CC_ALIAS_KEYS: ${{ inputs.alias-keys }} + CC_MAX_SIZE_MB: ${{ inputs.max-size-mb }} + CC_STRICT: ${{ inputs.strict }} run: | set -euo pipefail + # A failed save must be LOUD (CIRISCache#3). Every stage below sets + # STAGE; the EXIT trap turns any non-zero exit into an annotation that + # names the stage, plus `published=false` / `reason=…` outputs the + # caller can gate on. Fail-open by default (warning, exit 0): a save is + # an optimisation and must not red a build. `strict: "true"` makes it + # an error that fails the step. Callers that still wrap save in + # `continue-on-error: true` get the same annotation — the old silent + # path (a red step hidden as success) is gone. + STAGE="setup"; PUBLISHED="false" + _cc_report() { + local rc=$? + trap - EXIT + [ "$rc" -eq 0 ] && return 0 + local reason="$STAGE failed (exit $rc)" + if [ "$PUBLISHED" = "true" ]; then + # The blob is already up; only a best-effort tail step failed. + echo "::warning::CIRISCache: $reason after $CC_REF was published (non-fatal)" + exit 0 + fi + echo "published=false" >> "$GITHUB_OUTPUT" + echo "reason=$reason" >> "$GITHUB_OUTPUT" + if [ "${CC_STRICT:-false}" = "true" ]; then + echo "::error::CIRISCache: save NOT published for $CC_REF — $reason" + exit "$rc" + fi + echo "::warning::CIRISCache: save NOT published for $CC_REF — $reason (fail-open; check the 'published' output)" + exit 0 + } + trap _cc_report EXIT # Portable step-internal timeout (CIRISPersist#458 — fail-open save). A # HUNG `oras push`/`oras tag` (dead GHCR / stalled upload) would otherwise # freeze the step until the JOB timeout, delaying (and on a tag run, # orphaning) the release. `timeout(1)` isn't portable (absent on macOS # runners; Windows `timeout.exe` differs), so use a bash watchdog: run in # the background, SIGTERM after N seconds, return the real rc (124 on - # watchdog fire). The caller wraps save in `continue-on-error: true`, so a - # timed-out push is non-fatal. Watchdog stdout → /dev/null so it can't + # watchdog fire). A timed-out push lands in the EXIT trap above as + # `oras push failed (exit 124)`. Watchdog stdout → /dev/null so it can't # hold a `$(…)` capture pipe open. _cc_timeout() { local secs="$1"; shift @@ -93,15 +156,19 @@ runs: local wdpid=$! local rc=0 wait "$cmdpid" 2>/dev/null || rc=$? - kill -TERM "$wdpid" 2>/dev/null + # `|| true`: once the watchdog has fired it has exited, and a kill of + # a dead pid must not trip `set -e` and mask the 124. + kill -TERM "$wdpid" 2>/dev/null || true wait "$wdpid" 2>/dev/null || true [ "$rc" -ge 128 ] && return 124 return "$rc" } - REF="${{ steps.k.outputs.ref }}" - printf '%s' "${CIRISCACHE_TOKEN}" | oras login ghcr.io -u "${{ github.actor }}" --password-stdin + REF="$CC_REF" + STAGE="oras login" + printf '%s' "${CIRISCACHE_TOKEN}" | oras login ghcr.io -u "$CC_ACTOR" --password-stdin + STAGE="path expansion" EXP="" - for p in ${{ inputs.paths }}; do + for p in $CC_PATHS; do p="${p/#\~/$HOME}" case "$p" in /*) ;; *) p="$GITHUB_WORKSPACE/$p";; esac # Windows MSYS tar: normalize D:\a\... -> /d/a/... so there's no @@ -113,7 +180,12 @@ runs: [ "${RUNNER_OS:-}" = "Windows" ] && p="$(cygpath -u "$p")" if [ -e "$p" ]; then EXP="$EXP $p"; else echo "CIRISCache: skip missing $p"; fi done - if [ -z "$EXP" ]; then echo "CIRISCache: nothing to cache"; exit 0; fi + if [ -z "$EXP" ]; then + echo "published=false" >> "$GITHUB_OUTPUT" + echo "reason=nothing to cache" >> "$GITHUB_OUTPUT" + echo "::warning::CIRISCache: nothing to cache for $REF (no path in 'paths' exists) — not published" + exit 0 + fi # -P keeps absolute paths so restore lands them back in place. On Windows # use a POSIX archive path + --force-local (belt-and-suspenders); members # are already cygpath'd above. oras is a native Windows binary, so it gets @@ -121,33 +193,31 @@ runs: TMP="${RUNNER_TEMP}"; FL="" if [ "${RUNNER_OS:-}" = "Windows" ]; then TMP="$(cygpath -u "${RUNNER_TEMP}")"; FL="--force-local"; fi # ── LRU prune (see max-size-mb input) ───────────────────────────── - if [ -n "${{ inputs.max-size-mb }}" ] && [ -d "$GITHUB_WORKSPACE/target" ] \ + # The prune lives in scripts/prune.sh (cc_prune_lru) so + # tests/prune_test.sh can run it under this step's exact shell + # options. It reads its oldest-first list from a temp FILE, not a + # pipe: an early `break` out of `sort | while` SIGPIPEs sort, which + # under pipefail killed this step before tar/push (CIRISCache#3). + if [ -n "$CC_MAX_SIZE_MB" ] && [ -d "$GITHUB_WORKSPACE/target" ] \ && find --version >/dev/null 2>&1; then - BUDGET_KB=$(( ${{ inputs.max-size-mb }} * 1024 )) - T="$GITHUB_WORKSPACE/target" - USED_KB=$(du -sk "$T" | cut -f1) - if [ "$USED_KB" -gt "$BUDGET_KB" ]; then - echo "CIRISCache: target/ ${USED_KB}KB over ${BUDGET_KB}KB budget — LRU prune" - # Oldest-mtime first; delete until within budget. %T@ (GNU find) - # + a size column so we can stop exactly at the line. - find "$T" -type f -printf '%T@ %k %p\n' 2>/dev/null | sort -n | \ - while read -r _mt kb path; do - [ "$USED_KB" -le "$BUDGET_KB" ] && break - rm -f -- "$path" 2>/dev/null || continue - USED_KB=$(( USED_KB - kb )) - done - # The subshell above cannot export USED_KB; re-measure honestly. - AFTER_KB=$(du -sk "$T" | cut -f1) - echo "CIRISCache: pruned target/ ${USED_KB}KB→${AFTER_KB}KB (budget ${BUDGET_KB}KB)" - else - echo "CIRISCache: target/ ${USED_KB}KB within ${BUDGET_KB}KB budget — no prune" - fi - elif [ -n "${{ inputs.max-size-mb }}" ]; then + STAGE="prune" + AP="$GITHUB_ACTION_PATH" + [ "${RUNNER_OS:-}" = "Windows" ] && AP="$(cygpath -u "$AP")" + # shellcheck source=../scripts/prune.sh + . "$AP/../scripts/prune.sh" + TMPDIR="$TMP" cc_prune_lru "$GITHUB_WORKSPACE/target" $(( CC_MAX_SIZE_MB * 1024 )) + elif [ -n "$CC_MAX_SIZE_MB" ]; then echo "CIRISCache: max-size-mb set but GNU find unavailable — prune skipped (fail-open)" fi + STAGE="tar" + # shellcheck disable=SC2086 # $FL and $EXP are word lists by design tar $FL -czPf "$TMP/cache.tar.gz" $EXP cd "$TMP" + STAGE="oras push" _cc_timeout 1200 oras push "$REF" cache.tar.gz:application/gzip + PUBLISHED="true" + echo "published=true" >> "$GITHUB_OUTPUT" + echo "reason=" >> "$GITHUB_OUTPUT" echo "CIRISCache: saved $REF ($(du -h cache.tar.gz | cut -f1))" # Registry-side retag: point each alias key at the SAME manifest we just # pushed — no re-tar / re-gzip / re-upload of the (byte-identical) @@ -156,8 +226,9 @@ runs: # `phead`). CIRISPersist#458: the double ~9-min save was two full # tar+push cycles of identical content; this collapses it to push-once + # alias. A retag failure is non-fatal — the primary $REF is already saved. + STAGE="alias retag" PKG="${REF%:*}" - for ALIAS in ${{ inputs.alias-keys }}; do + for ALIAS in $CC_ALIAS_KEYS; do ALIAS=$(printf '%s' "$ALIAS" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9_.-' '-' | cut -c1-128) [ -z "$ALIAS" ] && continue if _cc_timeout 60 oras tag "$REF" "$ALIAS"; then diff --git a/scripts/prune.sh b/scripts/prune.sh new file mode 100644 index 0000000..ebb18a1 --- /dev/null +++ b/scripts/prune.sh @@ -0,0 +1,42 @@ +# shellcheck shell=bash +# CIRISCache LRU prune — sourced by save/action.yml and tests/prune_test.sh. +# +# cc_prune_lru DIR BUDGET_KB +# Delete the OLDEST-mtime files under DIR until its `du -sk` total is within +# BUDGET_KB (see the save action's `max-size-mb` input). Needs GNU find +# (`-printf %T@`); the caller checks for it. Returns non-zero on failure. +# +# Why the sorted list goes through a FILE and not a pipe (CIRISCache#3): the +# loop `break`s as soon as the budget is met, long before the list is +# exhausted on a large target/. In `find | sort | while …; break` the reader +# then exits with sort still writing, so sort takes SIGPIPE — or, on the +# Actions runner, where SIGPIPE is ignored, EPIPE: "sort: write failed: +# 'standard output': Broken pipe", exit 2. Under `set -o pipefail` that fails +# the pipeline, `set -e` kills the step before tar/push, and a caller's +# `continue-on-error: true` hid it: nothing was published for every save that +# took this path. Reading from a file has no writer to break. +cc_prune_lru() { + local dir="$1" budget_kb="$2" used_kb before_kb after_kb lru _mt kb path + used_kb=$(du -sk "$dir" | cut -f1) + if [ "$used_kb" -le "$budget_kb" ]; then + echo "CIRISCache: target/ ${used_kb}KB within ${budget_kb}KB budget — no prune" + return 0 + fi + before_kb=$used_kb + echo "CIRISCache: target/ ${used_kb}KB over ${budget_kb}KB budget — LRU prune" + lru=$(mktemp "${TMPDIR:-/tmp}/ciriscache-lru.XXXXXX") + # Oldest-mtime first; %T@ (GNU find) + a size column so we can stop exactly + # at the line. Both stages run to completion before the loop reads a line. + find "$dir" -type f -printf '%T@ %k %p\n' 2>/dev/null | sort -n > "$lru" \ + || { rm -f -- "$lru"; echo "CIRISCache: LRU listing of $dir failed"; return 1; } + while read -r _mt kb path; do + [ "$used_kb" -le "$budget_kb" ] && break + rm -f -- "$path" 2>/dev/null || continue + used_kb=$(( used_kb - kb )) + done < "$lru" + rm -f -- "$lru" + # The running total is an estimate (%k vs du's directory blocks); report + # the honest re-measure. + after_kb=$(du -sk "$dir" | cut -f1) + echo "CIRISCache: pruned target/ ${before_kb}KB→${after_kb}KB (budget ${budget_kb}KB)" +} From 950724c881ef4ae981314cdcb238378b4451406f Mon Sep 17 00:00:00 2001 From: Eric Moore Date: Wed, 7 Oct 2026 21:06:02 -0500 Subject: [PATCH 2/3] test(save): prune and save-step regression tests, run on every PR (#3) - tests/prune_test.sh: a 3000-file target/ over budget with controlled mtimes, pruned under `bash -e -o pipefail` with SIGPIPE default and ignored. Asserts exit 0, the step reaches tar, no broken pipe, the kept set is exactly the newest files within budget with minimal deletion, and the loop broke early with >64 KiB of list unread. Also checks save/action.yml sources and calls cc_prune_lru. - tests/save_step_test.sh: runs the push step's script verbatim with a stub oras: good save through the prune, login/prune/push failures, strict mode, nothing to cache, retag failure, and a failure after the push. Asserts exit code, annotation, and the published/reason outputs. - .github/workflows/test.yml runs both. Mutation: restoring the pipe-into-while form fails prune_test in both SIGPIPE modes (exit 141 and exit 2 with the runner's "Broken pipe" line). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016kQb87339uTdFWivk3j2hF --- .github/workflows/test.yml | 24 ++++++++ tests/prune_test.sh | 109 ++++++++++++++++++++++++++++++++++++ tests/save_step_test.sh | 111 +++++++++++++++++++++++++++++++++++++ 3 files changed, 244 insertions(+) create mode 100644 .github/workflows/test.yml create mode 100755 tests/prune_test.sh create mode 100755 tests/save_step_test.sh diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 0000000..c0bd31c --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,24 @@ +name: test + +# Shell-level tests of the save action (CIRISCache#3): the LRU prune under +# the step's exact shell options, and every failure path of the push step +# with a stub oras. No registry, no token — runs on every PR. + +on: + push: + branches: [main] + pull_request: + workflow_dispatch: + +permissions: + contents: read + +jobs: + shell: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: prune_test + run: bash tests/prune_test.sh + - name: save_step_test + run: bash tests/save_step_test.sh diff --git a/tests/prune_test.sh b/tests/prune_test.sh new file mode 100755 index 0000000..5e5688d --- /dev/null +++ b/tests/prune_test.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# Regression test for CIRISCache#3: the LRU prune must survive an early +# `break` under the save step's shell options (`bash -e -o pipefail`). +# +# Builds a target/ over budget with controlled mtimes, runs cc_prune_lru, and +# asserts: (a) exit 0, (b) the kept set is exactly the newest-mtime files and +# the total is within budget, (c) the loop broke EARLY with more than a pipe +# buffer of list still unread — the condition that killed `sort | while`. +# Runs twice: SIGPIPE default (sort dies by signal, 141) and SIGPIPE ignored +# (sort gets EPIPE, exit 2 — what the Actions runner showed). +# +# PRUNE_SH overrides the script under test (used to mutation-check this test +# against the old pipe-into-while form). Needs bash + coreutils + GNU find. +set -euo pipefail + +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +PRUNE_SH="${PRUNE_SH:-$ROOT/scripts/prune.sh}" +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT + +N=3000 # files of 4 KiB each: ~12 MB +BUDGET_KB=8192 # ~1/3 of the files must go; the rest stays unread +BASE=1700000000 +FAIL=0 + +fail() { echo "FAIL: $*"; FAIL=1; } + +# Built once; each case gets a `cp -a` copy (mtimes preserved). +make_template() { + local t="$WORK/template" i f + mkdir -p "$t/deps" + for i in $(seq 1 "$N"); do + # Long names, like real rlibs, so the unread list is well past 64 KiB. + f="$t/deps/libsome_crate_with_a_reasonably_long_name_$(printf %05d "$i")-0123456789abcdef.rlib" + head -c 4096 /dev/zero > "$f" + touch -d "@$(( BASE + i ))" "$f" # file i is the i-th oldest + done +} +make_template + +make_fixture() { rm -rf "$1"; cp -a "$WORK/template" "$1"; } + +run_case() { + local label="$1" sigpipe="$2" t="$WORK/target" out rc=0 + make_fixture "$t" + local before_kb; before_kb=$(du -sk "$t" | cut -f1) + [ "$before_kb" -gt "$BUDGET_KB" ] || { fail "$label: fixture not over budget"; return; } + + # Exactly the save step's options; `trap '' PIPE` reproduces the runner. + out=$(bash -e -o pipefail -c " + $sigpipe + . '$PRUNE_SH' + cc_prune_lru '$t' $BUDGET_KB + echo REACHED-TAR + " 2>&1) || rc=$? + echo "--- $label: exit $rc" + while IFS= read -r line; do echo " $line"; done <<<"$out" + + # (a) the step survives and reaches tar. + [ "$rc" -eq 0 ] || fail "$label: prune exited $rc (want 0)" + grep -q REACHED-TAR <<<"$out" || fail "$label: step died before tar" + grep -q 'Broken pipe' <<<"$out" && fail "$label: sort hit a broken pipe" + + # (b) within budget, and the survivors are exactly the newest files: a + # contiguous run of indices ending at N. + local after_kb kept oldest_kept + after_kb=$(du -sk "$t" | cut -f1) + [ "$after_kb" -le "$BUDGET_KB" ] || fail "$label: ${after_kb}KB still over ${BUDGET_KB}KB" + kept=$(find "$t" -type f | wc -l) + # awk reads to EOF: `sort | head -1` would SIGPIPE under pipefail — the + # very bug this file tests. + oldest_kept=$(find "$t" -type f -printf '%f\n' \ + | awk -F_ '{ n = substr($NF, 1, 5) + 0; if (m == "" || n < m) m = n } END { print m }') + [ $(( N - oldest_kept + 1 )) -eq "$kept" ] \ + || fail "$label: kept $kept files but oldest kept is #$oldest_kept (not the newest $kept)" + # Minimal deletion: putting back the newest deleted file (4 KiB) would + # exceed the budget, so the loop did not delete past the line. + [ $(( after_kb + 4 )) -gt "$BUDGET_KB" ] \ + || fail "$label: over-pruned (${after_kb}KB + 4KB still within ${BUDGET_KB}KB)" + + # (c) the early-break path ran: files were left unread, and their list + # lines exceed a 64 KiB pipe buffer, so a piped sort would still be + # writing when the reader quit. + [ "$kept" -gt 0 ] && [ "$kept" -lt "$N" ] || fail "$label: no early break (kept $kept of $N)" + local unread_bytes + unread_bytes=$(find "$t" -type f -printf '%T@ %k %p\n' | wc -c) + [ "$unread_bytes" -gt 65536 ] \ + || fail "$label: unread list ${unread_bytes}B fits a pipe buffer; fixture too small to catch SIGPIPE" + echo " kept $kept/$N (oldest kept #$oldest_kept), ${before_kb}KB→${after_kb}KB, unread list ${unread_bytes}B" +} + +run_case "SIGPIPE default" "" +run_case "SIGPIPE ignored (runner)" "trap '' PIPE" + +# Within budget: no prune, no deletion. +t="$WORK/small"; rm -rf "$t"; mkdir -p "$t"; head -c 4096 /dev/zero > "$t/a" +out=$(bash -e -o pipefail -c ". '$PRUNE_SH'; cc_prune_lru '$t' $BUDGET_KB" 2>&1) \ + || fail "within-budget: exited non-zero" +grep -q 'no prune' <<<"$out" && [ -f "$t/a" ] || fail "within-budget: pruned or wrong message: $out" + +# The action must actually call this function (a test of a function the +# action no longer uses would pass forever). Comments stripped; captured to +# a variable because `grep | grep -q` can SIGPIPE the first grep. +code=$(grep -v '^[[:space:]]*#' "$ROOT/save/action.yml") +grep -q 'scripts/prune\.sh' <<<"$code" || fail "save/action.yml does not source scripts/prune.sh" +grep -q 'cc_prune_lru ' <<<"$code" || fail "save/action.yml does not call cc_prune_lru" + +if [ "$FAIL" -ne 0 ]; then echo "prune_test: FAILED"; exit 1; fi +echo "prune_test: OK" diff --git a/tests/save_step_test.sh b/tests/save_step_test.sh new file mode 100755 index 0000000..0145187 --- /dev/null +++ b/tests/save_step_test.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +# The save step's failure paths must be LOUD (CIRISCache#3). Extracts the +# `push` step's script from save/action.yml verbatim, runs it under the +# runner's shell (`bash -e -o pipefail`) with a stub `oras`, and asserts the +# exit code, the annotation, and the `published` / `reason` outputs for: a +# good save through the prune, each failing stage (login, prune, push, a +# push timeout is the same path), strict mode, nothing to cache, and a retag +# failure after the blob is up. +set -euo pipefail + +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +WORK=$(mktemp -d) +trap 'chmod -R u+rwx "$WORK" 2>/dev/null; rm -rf "$WORK"' EXIT +FAIL=0 +fail() { echo "FAIL: $*"; FAIL=1; } + +# The run block of `- id: push`: the lines after `run: |` indented past it. +awk ' + /^ - id: push$/ { inpush = 1; next } + inpush && /^ - / { exit } + inpush && /^ run: \|$/ { inrun = 1; next } + inrun { if ($0 == "") { print ""; next } if (substr($0, 1, 8) != " ") exit; print substr($0, 9) } +' "$ROOT/save/action.yml" > "$WORK/step.sh" +[ -s "$WORK/step.sh" ] || { echo "FAIL: could not extract the push step"; exit 1; } +# shellcheck disable=SC2016 # a literal ${{, not an expansion +grep -q '\${{' "$WORK/step.sh" && { echo "FAIL: push step splices \${{ }} into the script"; exit 1; } + +# Stub oras: logs each call; ORAS_FAIL=login|push|tag makes that verb exit 1. +mkdir -p "$WORK/bin" +cat > "$WORK/bin/oras" <<'EOF' +#!/usr/bin/env bash +echo "oras $*" >> "$ORAS_LOG" +[ "$1" = login ] && cat > /dev/null +[ "${ORAS_FAIL:-}" = "$1" ] && { echo "stub oras: $1 failed" >&2; exit 1; } +exit 0 +EOF +chmod +x "$WORK/bin/oras" + +# run_step NAME [VAR=VALUE ...] — fresh workspace with a target/ over a 1 MB +# budget (400 x 4 KiB, oldest first), then the step. Sets RC, OUT, OUTPUTS. +run_step() { + local name="$1"; shift + local ws="$WORK/$name" + mkdir -p "$ws/target/deps" "$ws/tmp" + local i + for i in $(seq 1 400); do + head -c 4096 /dev/zero > "$ws/target/deps/f$i" + touch -d "@$(( 1700000000 + i ))" "$ws/target/deps/f$i" + done + : > "$ws/gh_output"; : > "$ws/oras.log" + RC=0 + OUT=$(cd "$ws" && env PATH="$WORK/bin:$PATH" HOME="$ws" \ + GITHUB_WORKSPACE="$ws" GITHUB_OUTPUT="$ws/gh_output" RUNNER_TEMP="$ws/tmp" \ + GITHUB_ACTION_PATH="$ROOT/save" RUNNER_OS=Linux ORAS_LOG="$ws/oras.log" \ + CIRISCACHE_TOKEN=tok CC_REF="ghcr.io/x/y:k1" CC_ACTOR=bot CC_PATHS="target" \ + CC_ALIAS_KEYS="" CC_MAX_SIZE_MB="1" CC_STRICT="false" "$@" \ + bash -e -o pipefail "$WORK/step.sh" 2>&1) || RC=$? + OUTPUTS=$(cat "$ws/gh_output") + echo "--- $name: exit $RC" + while IFS= read -r line; do echo " $line"; done <<<"$OUT" +} +# The last value written for KEY (GITHUB_OUTPUT is last-wins). +output() { awk -F= -v k="$1" '$1 == k { v = substr($0, length(k) + 2) } END { print v }' <<<"$OUTPUTS"; } +expect() { # expect NAME RC PUBLISHED REASON-SUBSTRING ANNOTATION-REGEX + [ "$RC" -eq "$2" ] || fail "$1: exit $RC, want $2" + [ "$(output published)" = "$3" ] || fail "$1: published='$(output published)', want '$3'" + case "$(output reason)" in *"$4"*) ;; *) fail "$1: reason='$(output reason)', want *$4*";; esac + if [ -n "$5" ]; then grep -Eq "$5" <<<"$OUT" || fail "$1: no annotation matching /$5/"; fi +} + +# A good save through the over-budget prune, with SIGPIPE ignored as on the +# runner: pruned, pushed, published. +(trap '' PIPE; run_step ok_ignored_sigpipe; declare -p RC OUT OUTPUTS > "$WORK/ok.env") +# shellcheck disable=SC1091 +. "$WORK/ok.env" +expect ok_ignored_sigpipe 0 true "" "" +grep -q 'LRU prune' <<<"$OUT" || fail "ok_ignored_sigpipe: prune did not run" +grep -q 'oras push ghcr.io/x/y:k1' "$WORK/ok_ignored_sigpipe/oras.log" || fail "ok_ignored_sigpipe: no push" +[ "$(du -sk "$WORK/ok_ignored_sigpipe/target" | cut -f1)" -le 1024 ] || fail "ok_ignored_sigpipe: target/ over budget" +grep -q '::' <<<"$OUT" && fail "ok_ignored_sigpipe: unexpected annotation" + +run_step login_fails ORAS_FAIL=login +expect login_fails 0 false "oras login failed (exit 1)" '^::warning::CIRISCache: save NOT published for ghcr.io/x/y:k1' + +run_step push_fails ORAS_FAIL=push +expect push_fails 0 false "oras push failed (exit 1)" '^::warning::CIRISCache: save NOT published' + +run_step push_fails_strict ORAS_FAIL=push CC_STRICT=true +expect push_fails_strict 1 false "oras push failed (exit 1)" '^::error::CIRISCache: save NOT published' + +# Prune failure: an unreadable directory makes du/find fail. +mkdir -p "$WORK/prune_fails/target/locked"; chmod 000 "$WORK/prune_fails/target/locked" +run_step prune_fails +expect prune_fails 0 false "prune failed" '^::warning::CIRISCache: save NOT published' +grep -q 'oras push' "$WORK/prune_fails/oras.log" && fail "prune_fails: pushed after a failed prune" + +run_step nothing_to_cache CC_PATHS="no-such-dir" +expect nothing_to_cache 0 false "nothing to cache" '^::warning::CIRISCache: nothing to cache' + +# A retag failure after the push: still published, warning only. +run_step tag_fails ORAS_FAIL=tag CC_ALIAS_KEYS="k1-alias" +expect tag_fails 0 true "" '^::warning::CIRISCache: alias retag failed' + +# A non-conditional failure AFTER the push (here the alias-key normalization, +# via a `tr` that fails): the trap must not overwrite published=true. +mkdir -p "$WORK/badtr"; printf '#!/bin/sh\nexit 1\n' > "$WORK/badtr/tr"; chmod +x "$WORK/badtr/tr" +run_step post_publish_fails CC_ALIAS_KEYS="k1-alias" PATH="$WORK/badtr:$WORK/bin:$PATH" +expect post_publish_fails 0 true "" '^::warning::CIRISCache: alias retag failed \(exit 1\) after ghcr.io/x/y:k1 was published' + +if [ "$FAIL" -ne 0 ]; then echo "save_step_test: FAILED"; exit 1; fi +echo "save_step_test: OK" From ff17ca7e84c209d35549c66c7517b153e09e62bb Mon Sep 17 00:00:00 2001 From: Eric Moore Date: Wed, 7 Oct 2026 21:06:49 -0500 Subject: [PATCH 3/3] test(selftest): set max-size-mb so the live save sources scripts/prune.sh (#3) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016kQb87339uTdFWivk3j2hF --- .github/workflows/selftest.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/selftest.yml b/.github/workflows/selftest.yml index b5187cc..0b03631 100644 --- a/.github/workflows/selftest.yml +++ b/.github/workflows/selftest.yml @@ -54,6 +54,10 @@ jobs: paths: "~/.cargo/registry/cache target/deps" token: ${{ secrets.GITHUB_TOKEN }} strict: "true" # a failed save must red the selftest + # Within budget, so nothing is pruned, but the prune branch still + # sources scripts/prune.sh via $GITHUB_ACTION_PATH on every OS that + # has GNU find (CIRISCache#3). + max-size-mb: "1024" - name: Assert published shell: bash