Repository navigation
image: make the model layer identical across releases - #7
Merged
Merged
Conversation
Every release rebuilt the 2.9 GiB model layer with new file mtimes, so its digest changed although the model did not, and every version update made nodes download it again. On Craftsman that made an update miss the Knative progress deadline (APT-1342). Build the model in its own stage, normalise permissions and timestamps, and copy the tree as one layer; verify the model manifest digest so a moved upstream tag fails the build; pin BuildKit and gzip level so the layer serialisation is fixed even on a cold build.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Comparing the published amd64 images
0.4.1and0.4.2: the 2,954 MiB model layer has the same content, ownership and permissions in both, and differs only in file mtimes (all 21 entries). Its digest therefore changes on every release, and every version update makes nodes download ~2.9 GiB again for a 3 MiB server change. On Craftsman the update to 0.4.2 never leftContainerCreatingbefore Knative reportedProgressDeadlineExceeded(APT-1342).What
COPY --linklayer with normalised permissions and timestamps (epoch 0, including symlinks).Unchanged: model, Ollama base digest, non-root runtime, read-only
/models, amd64 + arm64 publication.Verification (local, amd64)
--no-cache, no imported cache: the model layer digest is identical in both (sha256:7e6226a6…5a19, 3,098,479,606 bytes); the four base layers are unchanged.--network none:/healthz200,ollama listshowsgemma3:4b,/readyzgoes 503 → 200 after warm-up, UID 65532,/modelsnot writable.Effect and limits
embeddinghas the same defect;inferenceandvisionlook structurally the same (not compared). Not changed here.Do not release yet
Hold the tag until the slow image pull on the Craftsman nodes (1–3.4 MB/s) is understood; a release now would hit the same deadline there.