Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,11 @@ lcov.info
# Dart pub tool dir, regenerated by `dart pub get`; one global rule for the whole workspace (root + members).
**/.dart_tool/
services/ws-server/storage/
# Per-scenario credential file written by `et-cli generate-deployment` beside its mise.toml / compose.yaml.
# Derived from the scenario input, so regenerating a deployment rewrites it; it is the one place the password
# lives, which is what keeps it out of the generated files that ARE committed under verification/.
**/secrets.env
# No rule for the per-scenario credential file, deliberately.
# Under verification/ it is committed like every other generated artifact: the password is derived from an
# input this repository already carries, so the file reproduces from public material rather than keeping a
# secret, and the drift check has to be able to read it. A deployment generated anywhere else gets its own
# `.gitignore` written beside it by `et-cli`, which is what keeps a real credential out of a real repository.
**/.git/
**/Dockerfile*
README.md
9 changes: 5 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,8 @@ coverage-python.xml
# Dart pub tool dir, regenerated by `dart pub get`; one global rule for the whole workspace (root + members).
**/.dart_tool/
services/ws-server/storage/
# Per-scenario credential file written by `et-cli generate-deployment` beside its mise.toml / compose.yaml.
# Derived from the scenario input, so regenerating a deployment rewrites it; it is the one place the password
# lives, which is what keeps it out of the generated files that ARE committed under verification/.
**/secrets.env
# No rule for the per-scenario credential file, deliberately.
# Under verification/ it is committed like every other generated artifact: the password is derived from an
# input this repository already carries, so the file reproduces from public material rather than keeping a
# secret, and the drift check has to be able to read it. A deployment generated anywhere else gets its own
# `.gitignore` written beside it by `et-cli`, which is what keeps a real credential out of a real repository.
117 changes: 106 additions & 11 deletions .mise/config.maint.toml
Original file line number Diff line number Diff line change
Expand Up @@ -611,32 +611,127 @@ docker run --rm --platform linux/amd64 -v "$mnt" et-mise-linux mise run oci-buil
"""
shell = "{{ vars.task_shell }}"

# Verify every mise-tools GHCR store is public, so unauthenticated users can pull-mise-tools.
# Publish the hub and runner images to GHCR as multi-arch manifests (maintainer-only).
# A generated k3s deployment names these images rather than building them, so the cluster pulls
# ghcr.io/edge-toolkit/core/et-<service>:latest and only the scenario image -- the one carrying that
# deployment's module set -- is still built by whoever deploys it. Written ONLY from a maintainer's machine,
# for the same reason the tool stores are: no workflow holds packages:write, so a compromised workflow cannot
# replace what a cluster then runs.
#
# Auth is the one-time `docker login ghcr.io` the tool-store tasks above set up; docker pushes here rather
# than crane because buildx is what assembles the manifest list, and it uploads as part of the build.
#
# Each image's runtime stage carries `org.opencontainers.image.source`. GHCR reads that label to attach the
# package to the repository, which is what gives the package page its README and source link and lets it
# inherit the repository's access rather than standing on its own; a package pushed without it arrives
# unlinked and stays that way until a later push supplies one.
#
# Both platforms come from one build by default, so an arm64 node and an amd64 node resolve the same tag. That
# needs a `docker-container` builder -- the default `docker` driver builds a single platform and fails the
# moment two are asked for -- and binfmt registered for the foreign architecture. The foreign leg is emulated,
# and ws-web-runner links V8, so expect that one to take hours rather than minutes; pass image names to publish
# a subset rather than rebuilding all four.
#
# `IMAGE_PLATFORMS` narrows that list for a host that cannot emulate the other architecture. On Apple silicon
# the emulator decides whether it can at all: under Rosetta the x86 build works, while plain QEMU cannot run
# rustc and dies before compiling anything, with
#
# qemu: uncaught target signal 11 (Segmentation fault) - core dumped
# error: process didn't exit successfully: `.../rustc -vV` (signal: 11, SIGSEGV: invalid memory reference)
#
# (observed on a macOS arm64 host running Rancher Desktop with `useRosetta: false`, building ws-server for
# linux/amd64).
#
# Which is why each architecture is pushed under its own `:latest-<arch>` tag and `:latest` is then assembled
# from those with `docker buildx imagetools create`. A push of a tag replaces it rather than adding to it, so
# building straight to `:latest` would mean the two halves could only ever come from one host in one run --
# the amd64 push would drop the arm64 image. Assembling instead lets a host publish the architecture it can
# build and leaves the other half of `:latest` intact, and the assembly reads the registry rather than this
# run, so it picks up an architecture a different machine published months earlier. The cost is one extra tag
# per architecture, which shares every blob with the image it points at.
[tasks.push-service-images]
description = "Build and push the hub and runner images to GHCR for linux/amd64 + linux/arm64 (maintainer-only)"
# docker is a host daemon rather than a mise tool, and these images install nothing through mise.
run = """
images="${usage_image:-ws-server ws-web-runner ws-wasi-runner ws-pyo3-runner}"
every_platform="linux/amd64,linux/arm64"
platforms="${IMAGE_PLATFORMS:-$every_platform}"
tag=latest
builder=et-service-images
if ! docker buildx inspect "$builder" >/dev/null 2>&1; then
docker buildx create --name "$builder" --driver docker-container --bootstrap >/dev/null
fi
for name in $images; do
dockerfile="services/$name/Dockerfile"
if [ ! -f "$dockerfile" ]; then
echo "push-service-images: no Dockerfile at $dockerfile" >&2
exit 1
fi
repo="ghcr.io/edge-toolkit/core/et-$name"
for platform in ${platforms//,/ }; do
arch="${platform##*/}"
echo "push-service-images: $dockerfile -> $repo:$tag-$arch"
args="--builder $builder --platform $platform --file $dockerfile --tag $repo:$tag-$arch --push"
# $args is a word-split flag list by design; do not quote it.
# shellcheck disable=SC2086
docker buildx build $args .
done
# Rebuild the multi-arch tag from whichever per-architecture tags the registry now holds.
# Every architecture is probed, not just the ones this run built, so a host that can only produce one of
# them still leaves the tag naming both once the other host has published its half.
#
# Mixing revisions across architectures is the accepted cost of that, not an oversight. This is the whole
# reason the per-architecture tags exist: no machine here can build both halves. Apple silicon cannot
# produce the amd64 images at all -- QEMU's x86 emulation cannot run rustc, which dies on `rustc -vV` with
# the SIGSEGV quoted above -- so the two halves are published from different hosts, days apart, and each
# run can only see what the registry already holds. Revision-qualifying the source tags, or withholding
# `latest` until one run has built both, would mean `latest` could never be assembled at all until a single
# host can build both architectures. Until that is true, a `latest` whose halves may differ in revision is
# strictly better than no `latest`, and the per-architecture tags stay available for pinning an exact pair.
# Do not "fix" this by gating on a single revision; that trade has been made deliberately.
sources=""
for platform in ${every_platform//,/ }; do
candidate="$repo:$tag-${platform##*/}"
if docker buildx imagetools inspect "$candidate" >/dev/null 2>&1; then
sources="$sources $candidate"
fi
done
echo "push-service-images: $repo:$tag <-$sources"
# $sources is a word-split list of image references by design; do not quote it.
# shellcheck disable=SC2086
docker buildx imagetools create --tag "$repo:$tag" $sources
Comment thread
jayvdb marked this conversation as resolved.
done
"""
shell = "{{ vars.task_shell }}"
usage = 'arg "[image]..." var=#true help="Service directory names to publish (default: the hub and every runner)"'

# Verify every GHCR package is public, so unauthenticated users can pull a tool store or a service image.
# GHCR has NO API to change package visibility -- the packages REST surface is read/delete/restore only,
# and the web UI's toggle posts a session-cookie form -- so flipping a store public is a one-time manual
# and the web UI's toggle posts a session-cookie form -- so flipping a package public is a one-time manual
# click. This task automates everything around that click: it lists the org's container packages, prints
# each store's visibility, and fails with the exact settings page URL for any still-private store (flip it
# under Danger Zone -> Change visibility -> Public). Visibility survives later pushes, so each store needs
# this exactly once, right after its first `push-mise-tools*` publish. The listing call needs gh auth with
# the read:packages scope -- grant it once with `gh auth refresh -s read:packages`.
[tasks.check-mise-tools-visibility]
# each one's visibility, and fails with the exact settings page URL for any still private (flip it under
# Danger Zone -> Change visibility -> Public). Visibility survives later pushes, so each package needs this
# exactly once, right after its first publish -- a private service image is what leaves a pod in
# ImagePullBackOff on a cluster holding no credential. The listing call needs gh auth with the read:packages
# scope -- grant it once with `gh auth refresh -s read:packages`.
[tasks.check-ghcr-visibility]
depends = ["_setup-store-tools"]
description = "Verify every mise-tools GHCR store is public; print settings URLs for any still private"
description = "Verify every GHCR package is public; print settings URLs for any still private"
run = """
gh="$(mise which gh)"
jaq="$(mise which jaq)"
json=$("$gh" api "/orgs/edge-toolkit/packages?package_type=container" --paginate)
filter='.[] | select(.name | startswith("core/mise-tools/"))'
filter='.[] | select(.name | startswith("core/"))'
report='.name + ": " + .visibility'
still_private='select(.visibility != "public") | .name | @uri'
echo "$json" | "$jaq" -r "$filter | $report"
if [ -z "$(echo "$json" | "$jaq" -r "$filter | .name")" ]; then
echo "no mise-tools packages exist yet; publish one with push-mise-tools first" >&2
echo "no packages exist yet; publish one first" >&2
exit 1
fi
private=$(echo "$json" | "$jaq" -r "$filter | $still_private")
if [ -z "$private" ]; then
echo "all mise-tools stores are public"
echo "all packages are public"
exit 0
fi
echo "still private -- make each public at its settings page (Danger Zone -> Change visibility):" >&2
Expand Down
14 changes: 14 additions & 0 deletions .mise/config.toml
Original file line number Diff line number Diff line change
Expand Up @@ -439,6 +439,20 @@ url = "https://github.com/edge-toolkit/core/releases/download/augeas-v1/1.14.1-x
# LLVM 22.1.8, which is the profraw-10 format minicov 0.3.8 emits. Bump this and minicov together, never one
# alone, and check `rustc +<nightly> --version --verbose` reports an LLVM the minicov release names.
rust_nightly = "nightly-2026-08-05"
# The rustup host triple for this machine's arch, used as both the toolchain spec and CARGO_BUILD_TARGET.
# x64 keeps gnullvm: llvm-mingw supplies a full compiler + CRT from mise, so nothing depends on a system MSVC
# install, which is what lets the Nano Server image build at all. arm64 has no such option -- llvm-mingw
# publishes ucrt-x86_64 only, and every other native prerequisite the gnullvm lane leans on (conda:m2-make,
# conda:m2-gnupg, conda:libclang) is x86_64-only in its channel too. The windows-11-arm runner does ship
# Visual Studio with the arm64 MSVC toolset, so msvc is the one triple that can build there today; it is also
# rustup's own host default on that arch, so this asks for exactly the toolchain rustup would pick unaided.
#
# Defined here rather than beside the Windows tools that read it, and that placement is load-bearing.
# `lockfile_platforms` makes any lockfile-writing command resolve the `windows-x64` tools from every host, so
# `config.windows.toml`'s `[tools]` get rendered on macOS and Linux -- where that file is not an active config
# and its own `[vars]` are therefore absent from the context. Declaring it there failed every `mise upgrade`
# that had work to do, with `Field win_rust_host is not defined`; this file is active everywhere.
win_rust_host = "{% if arch() == 'arm64' %}aarch64-pc-windows-msvc{% else %}x86_64-pc-windows-gnullvm{% endif %}"
# Tier-`a_` prefix marks leaf values referenced by later vars.
# a_* vars are referenced by later vars (face1_src), and mise renders [vars] in alphabetical order; `a_` sorts
# before every non-prefixed var name so the reference resolves. Shared GitHub release tag for the HF-asset
Expand Down
14 changes: 3 additions & 11 deletions .mise/config.windows.toml
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,8 @@ url = "https://github.com/edge-toolkit/core/releases/download/gnupg-w32-v1/2.5.2
# error: toolchain 'stable-x86_64-pc-windows-gnullvm' may not be able to run on this system
# note: add the `--force-non-host` flag to install the toolchain anyway
#
# so the x64 spelling cannot simply be reused on arm64 -- see win_rust_host in [vars] for which triple each
# arch picks and why. CARGO_BUILD_TARGET below tracks the same var, so host and target stay in step.
# so the x64 spelling cannot simply be reused on arm64, which is why the triple comes from `win_rust_host`
# rather than a literal. CARGO_BUILD_TARGET below tracks the same var, so host and target stay in step.
[[tools.rust]]
components = "clippy,rust-analyzer"
profile = "minimal"
Expand Down Expand Up @@ -206,14 +206,6 @@ gradle_bin = '{{ vars.mise_installs }}\gradle\9.7.0\gradle-9.7.0\bin'
# conda:libclang's binary dir, for the LIBCLANG_PATH below.
# Pinned to the [tools] version above -- bump both together (conftest's version_drift enforces).
win_libclang = '{{ vars.mise_installs }}\conda-libclang\22.1.8\Library\bin'
# The rustup host triple for this machine's arch, used as both the toolchain spec and CARGO_BUILD_TARGET.
# x64 keeps gnullvm: llvm-mingw supplies a full compiler + CRT from mise, so nothing depends on a system MSVC
# install, which is what lets the Nano Server image build at all. arm64 has no such option -- llvm-mingw
# publishes ucrt-x86_64 only, and every other native prerequisite the gnullvm lane leans on (conda:m2-make,
# conda:m2-gnupg, conda:libclang) is x86_64-only in its channel too. The windows-11-arm runner does ship
# Visual Studio with the arm64 MSVC toolset, so msvc is the one triple that can build there today; it is also
# rustup's own host default on that arch, so this asks for exactly the toolchain rustup would pick unaided.
win_rust_host = "{% if arch() == 'arm64' %}aarch64-pc-windows-msvc{% else %}x86_64-pc-windows-gnullvm{% endif %}"
# The flat dependency farm inside npm:oxlint-tsgolint's mise install, for the OXLINT_TSGOLINT_PATH entry below.
# mise's embedded aube hoists a tool's dependencies rather than nesting them: `<install>\node_modules\<pkg>`
# is a symlink to `.mise\<pkg>@<version>\node_modules\<pkg>`, and that package dir holds only its own files
Expand Down Expand Up @@ -268,7 +260,7 @@ MISE_BASH_PATH = "{{ vars.winsh }}"
# both because only cmd.exe has the limit and because oxlint hard-fails on a set-but-invalid value, which an
# off-Windows `{% if %}` empty-string render would be. Inert when js isn't in MISE_ENV; just an unused path.
OXLINT_TSGOLINT_PATH = '{{ vars.tsgolint_pkg }}\@oxlint-tsgolint\win32-x64\tsgolint.exe'
# Build for the host's own target -- the LLVM mingw one on x64 (see win_rust_host above for the arm64 case).
# Build for the host's own target -- the LLVM mingw one on x64, and msvc on arm64, per `win_rust_host`.
# gnullvm, not gnu: llvm-mingw ships compiler-rt + libunwind, not the GCC runtime (libgcc/libgcc_eh) the gnu
# target's link line demands. The linker is llvm-mingw's clang and the raw-dylib import tool is its
# llvm-dlltool, both given as absolute paths (see win_llvm_bin above) so the cargo: source-build subprocess
Expand Down
2 changes: 1 addition & 1 deletion .mise/mise.js.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 6 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -413,9 +413,17 @@ ignore = [
# Unmaintained upstream at jethrogb/rust-cexpr.
# Pulled by bindgen (build-time only via aws-lc-sys / ort-sys).
"cexpr",
# Archived upstream at rust-lang/cfg-if, and the newer release the scanner points at is archived too.
# 1.0.4 is current, and the crate is a leaf of most of the graph; there is nothing to move to.
"cfg-if",
# Unmaintained upstream at dajoha/color-print.
# Pulled by deno_node via color-print.
"color-print-proc-macro",
# Upstream 9999years/command-error has been quiet for a year, which is what trips the age heuristic.
# The scanner also notes its `process-wrap ^8` against a current 10. 0.8.0 is the latest release, so there
# is no upgrade to take, and this is a direct dependency the repo adopts deliberately -- a lint rule of its
# own holds every `Command` to it.
"command-error",
# Archived upstream at matklad/countme.
"countme",
# Unmaintained upstream at denoland/deno_native_certs.
Expand Down
2 changes: 2 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@
ARG BASE_IMAGE=ubuntu:24.04
FROM ${BASE_IMAGE} AS build-minimal

LABEL org.opencontainers.image.source="https://github.com/edge-toolkit/core"

# Universal prereqs a typical dev box already has; everything else is mise's job.
# gcc, g++, libc6-dev and make are the C/C++ toolchain rustc links through (`cc`)
# and that C/C++ `-sys` crates build with (make for build scripts that shell out
Expand Down
1 change: 1 addition & 0 deletions Dockerfile.nanoserver
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ RUN curl -fsSL -o git.zip %GIT_URL% && `
# (VC++ DLLs, mise.exe, TEMP) and the gnu-host rust flip live here too,
# since every later stage builds on it.
FROM mcr.microsoft.com/windows/nanoserver:${WINDOWS_VERSION} AS build-minimal
LABEL org.opencontainers.image.source="https://github.com/edge-toolkit/core"
COPY --from=vcruntime C:\Windows\System32\vcruntime140.dll C:\Windows\System32\vcruntime140.dll
COPY --from=vcruntime C:\Windows\System32\vcruntime140_1.dll C:\Windows\System32\vcruntime140_1.dll
COPY --from=vcruntime C:\Windows\System32\msvcp140.dll C:\Windows\System32\msvcp140.dll
Expand Down
1 change: 1 addition & 0 deletions Dockerfile.windows
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ ARG WINDOWS_VERSION=ltsc2022
# stages mise.exe + the gh_token from the build context, trusts the project
# config, and runs `mise install` against the always-loaded `[tools]` set.
FROM mcr.microsoft.com/windows/servercore:${WINDOWS_VERSION} AS build-minimal
LABEL org.opencontainers.image.source="https://github.com/edge-toolkit/core"
SHELL ["cmd", "/S", "/C"]
# DL3066 wants a numeric user-id, which Windows containers have no equivalent of.
# Identities are SIDs behind well-known account names; ContainerAdministrator is the canonical one.
Expand Down
Loading
Loading