Skip to content

fix(ci): converge workflow and retry contracts #194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rldyourmnd merged 1 commit into from
Aug 26, 2026
+24 −10
Merged

fix(ci): converge workflow and retry contracts #194

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/codeql.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
actions: read
contents: read
security-events: write
uses: NDDev-OpenNetwork/ci-workflows/.github/workflows/public-codeql.yml@b50364e2a415267688c1d845cea6866cdb5e53d6 # 0.1.4
uses: NDDev-OpenNetwork/ci-workflows/.github/workflows/public-codeql.yml@409817cf743e76383c84e30c72edf781d73b71a1 # 0.1.11
with:
# Public repository: `pull_request` runs untrusted fork code. Name the
# hosted runner explicitly — the reusable's default belongs to the pinned
Expand Down
6 changes: 3 additions & 3 deletions docs/external-downloads.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@ appears outside this inventory.
| Surface | Classification | Resilience and authority |
| --- | --- | --- |
| `internal/imagebuild/artifacts.go`, `config/golden-image*.yaml` | image materialization | Exact HTTPS host/path and SHA identities; size bounds; transient transport/read and HTTP 408/429/5xx receive at most three attempts. Verified bytes are baked once into the immutable worker image. |
| `actions/tool-cache/tool-cache.sh` | per-job standalone tool | Trust-scoped immutable RustFS object first; size and SHA reverified; unavailable, missing, incomplete or corrupt cache falls back to exact upstream with five bounded attempts and emits `nddev_tool_cache_event`. |
| `scripts/install-benchmark-toolchain.sh` | representative benchmark fallback | Exact preinstalled version exits without download. Hosted/cold fallback uses fixed URLs, SHA-256 and five bounded attempts; this script is benchmark evidence, not a normal fleet start hook. |
| `actions/tool-cache/tool-cache.sh` | per-job standalone tool | Trust-scoped immutable RustFS object first; size and SHA reverified; unavailable, missing, incomplete or corrupt cache falls back to exact upstream with three total attempts and emits `nddev_tool_cache_event`. |
| `scripts/install-benchmark-toolchain.sh` | representative benchmark fallback | Exact preinstalled version exits without download. Hosted/cold fallback uses fixed URLs, SHA-256 and three total attempts; this script is benchmark evidence, not a normal fleet start hook. |
| `.github/workflows/ci.yml` | public self-CI | GitHub-hosted `setup-go`; private fleet consumers instead use the baked toolchain paths published by `ci-workflows`. |
| `actions/package-cache/package-cache.sh`, `scripts/configure-sccache.sh` | private package/cache data | Authenticated repository-scoped S3-compatible cache traffic; never an executable download authority. Cache failure degrades to the package manager or upstream path. |
| `internal/garmproviderincus/provider/{incus.go,specs.go,admission.go,cache_delivery.go}` | VPC-local runner bootstrap | One-use instance identity and pinned runner metadata from the fleet gateway; install-script fetches use five bounded attempts. No arbitrary external tool origin is accepted. |
| `internal/garmproviderincus/provider/{incus.go,specs.go,admission.go,cache_delivery.go}` | VPC-local runner bootstrap | One-use instance identity and pinned runner metadata from the fleet gateway; install-script fetches use three total attempts. No arbitrary external tool origin is accepted. |
| `scripts/build-garm-nddev.sh` | reviewed source build | Fetches the exact reviewed upstream commit and applies the checked-in derivative patch set; not executed in a job start hook. |
| `internal/imagebuild/assets/{provision.sh,container-provision.sh}` | local image provisioning | Talks only to the guest-local Incus socket or installs already-downloaded, verified image inputs. |
| `internal/imagebuild/assets/{smoke.sh,smoke-integration.sh}` | reachability smoke | HTTP requests assert egress, service-container and metadata isolation. Returned bytes are never executed or promoted. |
Expand Down
26 changes: 20 additions & 6 deletions internal/schedulerrecovery/command.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,12 @@ import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"os/exec"
"path/filepath"
"strings"
"syscall"
"time"
)

Expand Down Expand Up @@ -85,12 +87,24 @@ func (executor CommandExecutor) run(ctx context.Context, argv []string, attempt
}
bounded, cancel := context.WithTimeout(ctx, executor.Config.Timeout)
defer cancel()
command := exec.CommandContext(bounded, argv[0], argv[1:]...)
command.Env = append(command.Environ(),
"GHA_SCHEDULER_RECOVERY_ATTEMPT="+attempt.ID,
"GHA_SCHEDULER_RECOVERY_STUCK="+strings.Join(attempt.Stuck, ","),
)
output, err := command.Output()
var output []byte
var err error
for executionAttempt := 0; executionAttempt < 3; executionAttempt++ {
command := exec.CommandContext(bounded, argv[0], argv[1:]...)
command.Env = append(command.Environ(),
"GHA_SCHEDULER_RECOVERY_ATTEMPT="+attempt.ID,
"GHA_SCHEDULER_RECOVERY_STUCK="+strings.Join(attempt.Stuck, ","),
)
output, err = command.Output()
if !errors.Is(err, syscall.ETXTBSY) {
break
}
select {
case <-bounded.Done():
break
case <-time.After(5 * time.Millisecond):
}
}
if bounded.Err() != nil {
return nil, fmt.Errorf("command timed out: %w", bounded.Err())
}
Expand Down