Skip to content

image: make the model layer identical across releases - #7

Merged
reoring merged 1 commit into
mainfrom
work/20261119/stable-model-layer
Nov 19, 2026
Merged

reoring merged 1 commit into
mainfrom
work/20261119/stable-model-layer

Conversation

@reoring

@reoring reoring commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Why

Comparing the published amd64 images 0.4.1 and 0.4.2: the 2,954 MiB model layer has the same content, ownership and permissions in both, and differs only in file mtimes (all 21 entries). Its digest therefore changes on every release, and every version update makes nodes download ~2.9 GiB again for a 3 MiB server change. On Craftsman the update to 0.4.2 never left ContainerCreating before Knative reported ProgressDeadlineExceeded (APT-1342).

What

  • Model built in its own stage and exported as one COPY --link layer with normalised permissions and timestamps (epoch 0, including symlinks).
  • The model manifest is verified against a pinned SHA-256, so a moved upstream tag fails the build instead of silently changing the layer.
  • BuildKit image, Dockerfile frontend and gzip level are pinned so the layer serialisation is fixed on a cold build. No dependence on a warm CI cache.

Unchanged: model, Ollama base digest, non-root runtime, read-only /models, amd64 + arm64 publication.

Verification (local, amd64)

  • Two builds in separate disposable BuildKit instances, --no-cache, no imported cache: the model layer digest is identical in both (sha256:7e6226a6…5a19, 3,098,479,606 bytes); the four base layers are unchanged.
  • Second image run with --network none: /healthz 200, ollama list shows gemma3:4b, /readyz goes 503 → 200 after warm-up, UID 65532, /models not writable.
  • arm64 was not built or run locally.

Effect and limits

  • The first release with this change downloads a new model layer once. After that, a server-only release reuses it on any node that already has it.
  • A node that has never run llm still downloads the whole image (~6.4 GiB).
  • embedding has the same defect; inference and vision look structurally the same (not compared). Not changed here.

Do not release yet

Hold the tag until the slow image pull on the Craftsman nodes (1–3.4 MB/s) is understood; a release now would hit the same deadline there.

Every release rebuilt the 2.9 GiB model layer with new file mtimes, so
its digest changed although the model did not, and every version update
made nodes download it again. On Craftsman that made an update miss the
Knative progress deadline (APT-1342).

Build the model in its own stage, normalise permissions and timestamps,
and copy the tree as one layer; verify the model manifest digest so a
moved upstream tag fails the build; pin BuildKit and gzip level so the
layer serialisation is fixed even on a cold build.
@reoring
reoring merged commit 1d699d1 into main Nov 19, 2026
1 check passed
@reoring
reoring deleted the work/20261119/stable-model-layer branch November 19, 2026 05:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant