docs(tutorials): add Lab 12: Serve Models from a KitOps ModelKit on HAMi - #655
docs(tutorials): add Lab 12: Serve Models from a KitOps ModelKit on HAMi#655rudrakshkarpe wants to merge 5 commits into
Conversation
…HAMi Add a new intermediate lab that replaces the runtime Hugging Face model download with a KitOps ModelKit pulled from an OCI registry. A kitunpacker initContainer unpacks the ModelKit into a shared volume, and a custom SGLang image (with an optional co-resident vLLM example) serves the model from that local path — no Hugging Face download in the main container — while HAMi enforces nvidia.com/gpumem and nvidia.com/gpucores caps. This is the model-supply-chain counterpart to Lab 11 (SGLang) and Lab 6 (vLLM). All commands and outputs are captured from a verification run on a kind cluster backed by a single NVIDIA H100 80GB, using a public Jozu Hub ModelKit. Addresses Project-HAMi#561 Co-authored-by: Shivay Lamba <19529592+shivaylamba@users.noreply.github.com> Signed-off-by: rudrakshkarpe <rudraksh.karpe@gmail.com>
…idebar Add the labs/hami-kitops entry to sidebars-tutorials.js so the new lab appears in the Labs category navigation. Co-authored-by: Shivay Lamba <19529592+shivaylamba@users.noreply.github.com> Signed-off-by: rudrakshkarpe <rudraksh.karpe@gmail.com>
|
@rudrakshkarpe: The label(s) DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: rudrakshkarpe The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
✅ Deploy Preview for project-hami ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
📝 WalkthroughWalkthroughAdds Lab 12 in English and Chinese covering KitOps ModelKit inspection, initContainer unpacking, local SGLang/vLLM serving, HAMi resource limits, Kubernetes deployment, inference testing, troubleshooting, and cleanup. Registers the tutorial in the Labs sidebar. ChangesKitOps ModelKit serving lab
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant Kubernetes
participant kitunpacker
participant ModelKitRegistry
participant ModelVolume
participant SGLang
participant Client
Kubernetes->>kitunpacker: Start initContainer
kitunpacker->>ModelKitRegistry: Pull ModelKit
kitunpacker->>ModelVolume: Unpack and validate model files
Kubernetes->>SGLang: Start serving container
SGLang->>ModelVolume: Load local model directory
Client->>SGLang: Request OpenAI-compatible completion
SGLang-->>Client: Return completion response
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Three references to Lab 11 linked to ./hami-sglang, which does not exist on master yet, so `docusaurus build` failed its broken-link check. Make them plain-text references so this lab is self-contained and independently buildable; they can be upgraded to internal links once Lab 11 has merged. Co-authored-by: Shivay Lamba <19529592+shivaylamba@users.noreply.github.com> Signed-off-by: rudrakshkarpe <rudraksh.karpe@gmail.com>
Reword the lab so KitOps + Jozu Hub is presented as a first-class, registry-native way to package and deliver a model, rather than as a replacement for a Hugging Face download. Drops the "replace" / "instead of Hugging Face" / "no HF download" framing throughout (prose, script comments, log lines, and the verification table) in favour of describing how the model is packaged as a ModelKit, stored on Jozu Hub, pulled by the initContainer, and served from a local path. Neutralises incidental "HF-format" wording to "safetensors layout". Log strings and their shown output are kept in sync. Co-authored-by: Shivay Lamba <19529592+shivaylamba@users.noreply.github.com> Signed-off-by: rudrakshkarpe <rudraksh.karpe@gmail.com>
mesutoezdil
left a comment
mesutoezdil
left a comment
There was a problem hiding this comment.
solid lab, the initcontainer + flatten trick is neat and i checked the kit inspect digest against the live registry, it matches (sha256:df4629f6...) so the captured output is real. two things before this is mergeable: the format:check box is ticked but npx prettier --check tutorials/labs/hami-kitops.md fails on the kitfile block, and there is no zh mirror while the other 10 labs all have one under i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/. rest is inline. build passes locally and no broken links.
| description: "Package a model as a KitOps ModelKit, pull it from Jozu Hub with an initContainer, and serve it locally with SGLang (and optionally vLLM) on HAMi GPU shares." | ||
| sidebar_label: "Lab 12: KitOps ModelKit Inference" | ||
| lab: | ||
| level: Intermediate |
There was a problem hiding this comment.
this asks the reader to write 4 files and build 3 images, and the other 60 min labs (volcano, kueue) are Advanced, so imo this is Advanced too
There was a problem hiding this comment.
That makes sense given the amount of setup involved. I have moved it to Advanced in both the frontmatter and sidebar.
| Public ModelKit used throughout: | ||
|
|
||
| ```plaintext | ||
| jozu.ml/jonathangamer202002/qwen3-4b-instruct:latest |
There was a problem hiding this comment.
the whole lab hangs off one personal account and a mutable tag, so pls pin by digest and ideally mirror the modelkit under a project owned repo, otherwise the lab breaks the day that account is cleaned up
There was a problem hiding this comment.
Thanks for checking the digest. I have pinned the ModelKit to the verified digest so the lab no longer depends on latest. Is there a preferred HAMi-owned namespace on Jozu Hub that we should use for the mirror? I would be happy to update the reference once that location is confirmed.
| Scale down any large competing GPU workloads so the ModelKit + 4B model can fit: | ||
|
|
||
| ```bash | ||
| kubectl -n sglang scale deploy/sglang-qwen3-17b --replicas=0 2>/dev/null || true |
There was a problem hiding this comment.
this scales a deploy that only exists if the reader did lab 11, and || true hides that, so a generic "free up gpu shares" line would age better?
There was a problem hiding this comment.
Yes, the Lab 11 deployment was too specific here. I replaced it with general guidance for checking current GPU usage and freeing enough HAMi capacity.
| FROM alpine:3.20 | ||
|
|
||
| RUN apk add --no-cache bash coreutils findutils ca-certificates curl tar \ | ||
| && curl -fsSL "https://github.com/jozu-ai/kitops/releases/latest/download/kitops-linux-x86_64.tar.gz" -o /tmp/kit.tgz \ |
There was a problem hiding this comment.
base image is pinned to alpine 3.20 but the kit cli is releases/latest, so pin the kitops version too or the init image quietly changes between builds?
There was a problem hiding this comment.
Nice catch. KitOps is now pinned to v1.11.0, so rebuilding the init image will not silently pull a different release.
|
|
||
| # simple cross-pod lock (mkdir is atomic) so two engines sharing one cache | ||
| # don't race to write the same files. | ||
| if ! mkdir "${LOCK}" 2>/dev/null; then |
There was a problem hiding this comment.
with emptyDir each pod gets its own volume so this lock never sees a peer, as step 8 says itself, worth a note that it only matters on a shared pvc
There was a problem hiding this comment.
You are right. I clarified that this lock only coordinates Pods using the same shared PVC. With the default emptyDir, every Pod has its own volume and lock.
| `sglang/Dockerfile`: | ||
|
|
||
| ```dockerfile | ||
| FROM lmsysorg/sglang:latest |
There was a problem hiding this comment.
:latest here and on vllm/vllm-openai at line 322 would be the only ones under tutorials/, and it clashes with the verified: 2026-07-23 claim since the base can change under you
There was a problem hiding this comment.
Makes sense. I pinned SGLang to v0.5.7 and vLLM to v0.23.0 so the documented setup stays reproducible.
| volumes: | ||
| - name: modelkit | ||
| emptyDir: | ||
| sizeLimit: 20Gi |
There was a problem hiding this comment.
fyi the model layer is ~7.5 GiB and it lands in KITOPS_HOME and then in .raw-qwen3 before the mv, so peak is ~15 GiB, worth saying so nobody trims this to 10Gi?
There was a problem hiding this comment.
That is helpful context. I added the roughly 15 GiB peak unpack usage and explained why the example keeps the volume at 20 GiB or larger.
| version: "1.0" | ||
| authors: | ||
| - HAMi KubeCon Demo | ||
| description: > |
There was a problem hiding this comment.
prettier --check fails only on this yaml block (folded description + trailing blank line), so npm run format fixes it and unblocks docs-health ci..
There was a problem hiding this comment.
Thanks for flagging this. I ran the repository formatter and confirmed that npm run format:check now passes.
| license: Apache-2.0 | ||
| description: Qwen3 4B instruct, safetensors (Qwen3ForCausalLM) | ||
| docs: | ||
| - path: ./README.md |
There was a problem hiding this comment.
kit resolves kitfile paths against the pack context and the steps only tell you to place ./qwen3, so kit pack . trips on the missing ./README.md?
There was a problem hiding this comment.
You are right, the example did not create that file. I removed the unnecessary docs entry, so the shown kit pack . flow is now self-contained.
| }, | ||
| { | ||
| type: "doc", | ||
| id: "labs/hami-kitops", |
Co-authored-by: Shivay Lamba <shivaylamba2000@gmail.com> Signed-off-by: rudrakshkarpe <rudraksh.karpe@gmail.com>
|
Thanks for going through the lab so carefully, and for validating the ModelKit digest. I have worked through the inline notes, added the Chinese version, and rerun the formatting and build checks. This PR will stay behind #654, and I will rebase it once that PR lands. I also pinned the current ModelKit by digest. If there is a preferred HAMi-owned Jozu Hub namespace, I would be glad to move the example there. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tutorials/labs/hami-kitops.md`:
- Around line 188-204: Update the kit unpacker scripts so ready() requires a
validated completion marker containing the exact MODELKIT_REF, rather than
treating partial or stale files as ready. Publish unpacked output atomically
only after validation completes, while preserving the shared-PVC mkdir lock and
waiting behavior. Apply the same lock, marker, reference checks, and atomic
publication flow in tutorials/labs/hami-kitops.md lines 188-204 and 230-240, and
mirror it in
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md
lines 188-204 and 230-240.
- Around line 147-151: Verify the downloaded KitOps archive against the official
release checksum or signature before extraction and execution in both Dockerfile
snippets: tutorials/labs/hami-kitops.md lines 147-151 and
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md
lines 147-151. Add the release verification artifact download and validation
after curl and before tar, preserving the existing extraction and kit version
steps only after verification succeeds.
- Around line 210-236: Update the cleanup traps around the kitunpacker flow in
tutorials/labs/hami-kitops.md (lines 210-236) and
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md
(lines 210-236): add an EXIT trap before kit login that removes both RAW and
KITOPS_HOME, retain explicit INT and TERM traps that exit with failure status,
and remove or avoid relying on the success-only cleanup after kit unpack. Ensure
cleanup runs when kit login or kit unpack fails.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 54eac392-4570-4c33-9690-59cd33bb10b9
📒 Files selected for processing (3)
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.mdsidebars-tutorials.jstutorials/labs/hami-kitops.md
| RUN apk add --no-cache bash coreutils findutils ca-certificates curl tar \ | ||
| && curl -fsSL "https://github.com/kitops-ml/kitops/releases/download/${KITOPS_VERSION}/kitops-linux-x86_64.tar.gz" -o /tmp/kit.tgz \ | ||
| && tar -xzf /tmp/kit.tgz -C /usr/local/bin kit \ | ||
| && rm -f /tmp/kit.tgz \ | ||
| && kit version |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Verify the KitOps release artifact before extracting it.
Both Dockerfile snippets download and execute kit from an external release without checksum/signature verification. Add verification of the downloaded archive using the release checksums/signature in both language versions before the Dockerfile extracts and runs kit.
tutorials/labs/hami-kitops.md#L147-L151i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md#L147-L151
📍 Affects 2 files
tutorials/labs/hami-kitops.md#L147-L151(this comment)i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md#L147-L151
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tutorials/labs/hami-kitops.md` around lines 147 - 151, Verify the downloaded
KitOps archive against the official release checksum or signature before
extraction and execution in both Dockerfile snippets:
tutorials/labs/hami-kitops.md lines 147-151 and
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md
lines 147-151. Add the release verification artifact download and validation
after curl and before tar, preserving the existing extraction and kit version
steps only after verification succeeds.
| ready() { [ -f "${DEST}/config.json" ] && ls "${DEST}"/*.safetensors >/dev/null 2>&1; } | ||
|
|
||
| echo "[kitunpacker] ref=${MODELKIT_REF} -> ${DEST}" | ||
|
|
||
| if ready; then | ||
| echo "[kitunpacker] model already present, skipping unpack" | ||
| exit 0 | ||
| fi | ||
|
|
||
| # This lock only coordinates Pods when they mount the same shared PVC. With | ||
| # the emptyDir used in this lab, every Pod has an isolated volume and lock. | ||
| # mkdir is atomic, so Pods sharing a PVC do not race to write the same files. | ||
| if ! mkdir "${LOCK}" 2>/dev/null; then | ||
| echo "[kitunpacker] another unpack in progress, waiting for it to finish..." | ||
| i=0 | ||
| while [ "${i}" -lt 360 ]; do | ||
| ready && { echo "[kitunpacker] model became ready"; exit 0; } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Make shared-PVC publication atomic and reference-aware.
Both scripts can treat a partially written or stale model directory as ready. Publish only after the full unpack/validation completes, and compare a completion marker containing the exact MODELKIT_REF.
tutorials/labs/hami-kitops.md#L188-L204,L230-L240: fixready()and publish the validated model atomically.i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md#L188-L204,L230-L240: mirror the same lock, marker, and reference checks.
📍 Affects 2 files
tutorials/labs/hami-kitops.md#L188-L204(this comment)i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md#L188-L204
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tutorials/labs/hami-kitops.md` around lines 188 - 204, Update the kit
unpacker scripts so ready() requires a validated completion marker containing
the exact MODELKIT_REF, rather than treating partial or stale files as ready.
Publish unpacked output atomically only after validation completes, while
preserving the shared-PVC mkdir lock and waiting behavior. Apply the same lock,
marker, reference checks, and atomic publication flow in
tutorials/labs/hami-kitops.md lines 188-204 and 230-240, and mirror it in
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md
lines 188-204 and 230-240.
| # shellcheck disable=SC2064 | ||
| trap "rmdir '${LOCK}' 2>/dev/null || true" EXIT INT TERM | ||
|
|
||
| # optional login for private registries (public Jozu Hub needs none) | ||
| if [ -n "${REGISTRY_URL:-}" ] && [ -n "${USERNAME:-}" ] && [ -n "${PASSWORD:-}" ]; then | ||
| echo "[kitunpacker] logging in to ${REGISTRY_URL} as ${USERNAME}" | ||
| echo "${PASSWORD}" | kit login "${REGISTRY_URL}" -u "${USERNAME}" --password-stdin | ||
| fi | ||
|
|
||
| rm -rf "${RAW}"; mkdir -p "${RAW}" | ||
| echo "[kitunpacker] pulling + unpacking model layers from registry..." | ||
| kit unpack --filter model "${MODELKIT_REF}" -d "${RAW}" | ||
|
|
||
| # Flatten: ModelKits may store the .safetensors shards in a model/ subdir while | ||
| # config.json / *.index.json / tokenizer sit one level up. vLLM/transformers | ||
| # need them all in one directory, so collect everything into DEST. | ||
| SRC_CFG="$(find "${RAW}" -name config.json | head -1)" | ||
| [ -n "${SRC_CFG}" ] || { echo "[kitunpacker] config.json not found after unpack" >&2; exit 1; } | ||
| SRC="$(dirname "${SRC_CFG}")" | ||
|
|
||
| mkdir -p "${DEST}" | ||
| # all weight shards, wherever they live under the unpacked tree | ||
| find "${SRC}" -name '*.safetensors' -exec mv -f {} "${DEST}/" \; | ||
| # all top-level metadata files (config, index, tokenizer, vocab, generation cfg) | ||
| find "${SRC}" -maxdepth 1 -type f -exec mv -f {} "${DEST}/" \; | ||
|
|
||
| rm -rf "${RAW}" "${KITOPS_HOME}" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## Locate files"
fd -a 'hami-kitops\.md$' . || true
echo "## Inspect shell script around cleanup in both files"
for f in tutorials/labs/hami-kitops.md i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md; do
if [ -f "$f" ]; then
echo "### $f lines 1-260"
sed -n '1,260p' "$f" | cat -n
fi
done
echo "## Search trap/kube command pattern nearby in both files"
for f in tutorials/labs/hami-kitops.md i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md; do
if [ -f "$f" ]; then
echo "### $f lock/cleanup references"
rg -n "trap|LOCK|KITOPS|cleanup|rm -rf|k.+" "$f" -C 3 || true
fi
done
echo "## Doc mentions KITOPS_HOME"
rg -n "KITOPS_HOME|kitops" README.md tutorials docs i18n -S || true
echo "## Behavior probe: POSIX shell trap cleanup after failed command between trap and tail"
bash - <<'BASH'
trap 'echo cleanup; exit 130' EXIT INT TERM
trap 'echo signal-trap; exit 130' INT TERM
sleep 0.5 & PID=$!
kill -2 "$PID"
wait "$PID" || true
echo "remaining"
BASHRepository: Project-HAMi/website
Length of output: 50380
🌐 Web query:
KitOps KITOPS_HOME credentials cache storage location documentation
💡 Result:
In KitOps, the credentials cache is stored in a file named credentials.json located within the base configuration directory [1][2]. By default, this base directory is determined by the system platform, but it can be overridden using the KITOPS_HOME environment variable or the --config command-line flag [3][4]. The storage location of the credentials file is calculated as follows: [KITOPS_BASE_DIR]/credentials.json [1][2] Precedence for determining [KITOPS_BASE_DIR]: 1. --config flag (if provided during CLI execution) [3][4] 2. KITOPS_HOME environment variable (if set) [3][4] 3. Platform-specific default directory [3][2]: - Linux: $XDG_DATA_HOME/kitops (falls back to ~/.local/share/kitops) [3][1] - MacOS: ~/Library/Caches/kitops [3][1] - Windows: %LOCALAPPDATA%\kitops [3][1] Note that while the credentials file is stored at the root of this configuration directory, other KitOps data, such as cached model images, are stored in subdirectories (e.g., [KITOPS_BASE_DIR]/storage/ and [KITOPS_BASE_DIR]/cache/) [1][5][2].
Citations:
- 1: https://github.com/kitops-ml/kitops/blob/v1.15.0/pkg/lib/constants/consts.go
- 2: https://pkg.go.dev/github.com/kitops-ml/kitops/pkg/lib/constants
- 3: https://kitops.org/docs/cli/installation/
- 4: Respect KITOPS_HOME in library config path kitops-ml/kitops#1189
- 5: https://kitops.org/docs/cli/cli-reference/
Clean KitOps credentials and cache on every termination path.
KITOPS_HOME is mounted to the volume at /models/.kitcache, but it is only removed after the successful kit login / kit unpack flow completes. If login or unpacking fails after credentials were written, rm -rf "${RAW}" "${KITOPS_HOME}" is skipped. Add an EXIT cleanup trap before kit login that removes both temp and data dirs, and keep signal traps explicit so termination from kit login or kit unpack still exits with a failure status. Mirror this in i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md.
📍 Affects 2 files
tutorials/labs/hami-kitops.md#L210-L236(this comment)i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md#L210-L236
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tutorials/labs/hami-kitops.md` around lines 210 - 236, Update the cleanup
traps around the kitunpacker flow in tutorials/labs/hami-kitops.md (lines
210-236) and
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-kitops.md
(lines 210-236): add an EXIT trap before kit login that removes both RAW and
KITOPS_HOME, retain explicit INT and TERM traps that exit with failure status,
and remove or avoid relying on the success-only cleanup after kit unpack. Ensure
cleanup runs when kit login or kit unpack fails.
Source: MCP tools
rudrakshkarpe
changed the title
2 participants
What type of PR is this?
/kind documentation
What this PR does / why we need it:
Adds Lab 12: Serve Models from a KitOps ModelKit on HAMi, a new advanced tutorial that packages a model as a KitOps ModelKit (a versioned OCI artifact) and downloads it from an OCI registry (Jozu Hub) into the Pod, then serves it locally on HAMi GPU shares. It shows the enterprise, registry-native model supply chain requested in the issue: package the model once, version and store it on Jozu Hub, and pull it into the workload as an OCI artifact.
The lab walks through:
kit inspect --remote)kitunpackerinit image and a custom SGLang serve imagekitops-initinitContainer unpacks the ModelKit into a shared volume and the main container serves it from a local pathnvidia.com/gpumem/nvidia.com/gpucorescaps and verifying them in-PodKitfilefor packaging your own ModelKitThe public ModelKit is pinned by digest, KitOps is pinned to v1.11.0, and the SGLang and vLLM base images use versioned tags. Manifests are inline and self-contained. Commands and outputs are captured from a verification run on a kind + NVIDIA H100 80GB cluster (
verified: 2026-07-23).Also adds the sidebar entry and a Chinese translation under
i18n/zh/. The overview card is generated automatically from frontmatter.Which issue(s) this PR fixes:
Addresses #561
Checklist:
npm run lintandnpm run format:checkpassnpm run buildsucceeds for bothenandzhgit commit -s)Summary by CodeRabbit
New Features
Documentation