diff --git a/content/blog/kagent-part-1-local-ai-agent-kubernetes.md b/content/blog/kagent-part-1-local-ai-agent-kubernetes.md new file mode 100644 index 000000000..f987c5de2 --- /dev/null +++ b/content/blog/kagent-part-1-local-ai-agent-kubernetes.md @@ -0,0 +1,646 @@ +--- +title: "kagent Part 1: Building a Local, Kubernetes-Native AI Agent with Human-in-the-Loop Approval" +seoTitle: "kagent Tutorial: Build a Local AI Agent for Kubernetes with Ollama" +seoDescription: "A hands-on lab building a kagent AI agent on a local kind cluster with Ollama: read-only and write-capable agents, human-in-the-loop approval gates, and a practical guide for common issues." +datePublished: 2026-09-08T10:00:00.000Z +slug: kagent-part-1-local-ai-agent-kubernetes +author: prianshu-mukherjee +draft: false +cover: /img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-approval-pending.png +tags: ["kagent", "kubernetes", "ai-agents", "human-in-the-loop"] +--- + +A chatbot can explain Kubernetes to you. An agent can decide what to inspect next, pick a tool, read the result, and act on it. Which means the question is no longer "can a model talk about my cluster?" but "can it operate on my cluster in a way I can actually trust?" A write-capable agent can make a change unless the system explicitly stops it, that's the boundary this lab is built around. + +kagent is a Kubernetes-native framework for building exactly that. It gives you a runtime, a set of Kubernetes CRDs like `Agent` and `ModelConfig`, and MCP-backed tool integrations that let a model reason about a live cluster and call real tools against it, not just describe what it would do, but actually do it. + +Once a model can call tools, the design question stops being "is the answer good?" and becomes "what is this thing actually allowed to do, and who signs off before it does it?" That's what this lab is about. + +**kagent vs. k8sgpt, briefly:** k8sgpt runs fixed analyzers against your cluster, collects structured findings, and has a model explain them. There's no loop where the model chooses what to do next. kagent runs an actual agent loop, the model decides which tool to call, reads the result, and decides whether to call another tool or answer the user. That's materially different, which is why least-privilege tooling and approval gates matter so much here. + +This is Part 1 of a short series. In this one, we build a fully local kagent stack running on a single laptop: kind cluster, kagent, Ollama serving a small model in-cluster, a read-only agent, and a write-capable agent gated behind human approval. No cloud API key, no external LLM dependency, nothing that leaves your machine. Budget about 45 to 60 minutes hands-on if you're following along. + +## What you'll build + +By the end of this lab you'll have, all running locally: + +- A kind cluster with kagent installed +- Ollama serving `qwen2.5:1.5b` as an in-cluster model service +- A **read-only** agent that can inspect cluster state but cannot change anything +- A **write-capable** agent whose destructive actions pause for your explicit approval +In other words: + +- You interact through the kagent dashboard. +- The agent decides which tool to call. +- The tool server talks to the Kubernetes API. +- Model inference happens locally, through Ollama. +- Write operations pause for your approval before they execute. +![Architecture diagram: a kind node with a user, the kagent controller/UI, Ollama, the Kubernetes MCP tool server, and an approval gate before any write-capable tool call](/img/blog/kagent-part-1-local-ai-agent-kubernetes/architecture-diagram.jpg) + +**Prerequisites:** Docker, `kind`, `kubectl`, and Helm, plus enough memory to run a small local model alongside the kagent stack. A laptop with 16GB RAM is comfortable. Keep the model small: this walkthrough uses `qwen2.5:1.5b`. + +Record your host before starting so the timings have useful context: + +```bash +system_profiler SPHardwareDataType | grep -E "Chip|Total Number of Cores|Memory:" +docker info --format 'Docker: {{.NCPU}} CPUs, {{.MemTotal}} bytes' +``` + +The measurements reported here came from an Apple M4 with 10 cores and 16 GB RAM, with Docker allocated 10 CPUs and 8,321,515,520 bytes (about 7.75 GiB). + +This is a deliberately patient lab on CPU inference. In one run, the read-only pod-listing answer generated 1,183 tokens at 2-4 tokens/sec, which took roughly 5-10 minutes; the approval interaction generated about 144 tokens and took about a minute. Across Step 7 and the complete Step 9 workflow, budget roughly 15-40 minutes of waiting for model output. A healthy cluster can look idle while Ollama is working. + +Clone the lab repo before you start, every step below references files inside it: + +```bash +git clone https://github.com/Prianshu-git/Kagent-demo +cd Kagent-demo +``` + +```yaml +# 00-cluster/kind-config.yaml +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +name: kagent-security-lab +nodes: + - role: control-plane +``` + +--- + +## Step 1: Create the cluster + +Start with a clean kind cluster: + +```bash +kind create cluster --name kagent-security-lab --config 00-cluster/kind-config.yaml +kubectl cluster-info --context kind-kagent-security-lab +``` + +kind's config file doesn't reliably set the cluster name on every version - passing `--name` explicitly guarantees the context comes up as `kind-kagent-security-lab`, which every command later in this lab assumes. + +This creates the local Kubernetes environment that will host kagent and Ollama. A healthy cluster should show the control plane and core Kubernetes components up. + +**Checkpoint:** run `kubectl get nodes` and confirm one node in `Ready` status. + +--- + +## Step 2: Install kagent + +kagent uses a two-step Helm install: CRDs first, then the app itself. + +```bash +helm install kagent-crds oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \ + --namespace kagent \ + --create-namespace \ + --version 0.9.12 + +helm install kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent \ + --namespace kagent \ + --set providers.default=ollama \ + --version 0.9.12 + +# give the deployments a moment to create their pods before waiting on them: +# running `kubectl wait` immediately after `helm install` can fail with +# "no matching resources found" if the pods don't exist yet +sleep 15 +kubectl wait --for=condition=ready pod --all -n kagent --timeout=180s +``` + +Version pinning matters because kagent changes frequently. This lab uses kagent `0.9.12`. + +```bash +kubectl get pods -n kagent -o wide +``` + +On a fresh cluster, the kagent controller may log transient failures before Postgres is ready. That's normal. Give it a moment to converge, then validate the pod state. The system recovers on its own. + +**Checkpoint:** every pod in the `kagent` namespace is `Running`. + +--- + +## Step 3: Deploy Ollama in the cluster + +The lab defines the Ollama deployment in `01-local-llm/ollama-deployment.yaml`. + +```bash +kubectl apply -f 01-local-llm/ollama-deployment.yaml +kubectl wait --for=condition=ready pod -l app=ollama -n ollama --timeout=120s +``` + +Validate the Service and endpoints: + +```bash +kubectl get svc -n ollama +kubectl get endpoints -n ollama +``` + +```text +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +ollama ClusterIP 10.96.147.225 80/TCP 153m +``` + +```text +NAME ENDPOINTS AGE +ollama 10.244.0.30:11434 153m +``` + +The Service has a real endpoint. That's your confirmation the in-cluster model service is reachable from the rest of Kubernetes. + +**Checkpoint:** the Service has an endpoint IP address. + +--- + +## Step 4: Pull a local model + +This lab uses `qwen2.5:1.5b`, a 1.5-billion-parameter model optimized for CPU inference. + +```bash +kubectl exec -n ollama deploy/ollama -- ollama pull qwen2.5:1.5b +kubectl exec -n ollama deploy/ollama -- ollama list +``` + +Terminal output from pulling the model: + +```text +pulling manifest +pulling 183715c43589: 48% ▕████████ ▏ 471 MB/986 MB 2.5 MB/s 3m26s +pulling 183715c43589: 72% ▕█████████████ ▏ 713 MB/986 MB 1.1 MB/s 4m17s +pulling 183715c43589: 94% ▕████████████████ ▏ 928 MB/986 MB 16 KB/s 58m29s +pulling 183715c43589: 100% ▕█████████████████ ▏ 985 MB/986 MB 1.8 MB/s 0s +verifying sha256 digest +writing manifest +success +``` + +After the pull completes, check what models are available: + +```text +NAME ID SIZE MODIFIED +qwen2.5:1.5b 65ec06548149 986 MB About an hour ago +llama3.2:latest a80c4f17acd5 2.0 GB 14 hours ago +llama3.2:3b a80c4f17acd5 2.0 GB 15 hours ago +``` + +**Why `qwen2.5:1.5b`?** The immediate reason is resource pressure, not a claim that a model with half the parameters should be an order of magnitude faster. This deployment is capped at `2 vCPU / 4Gi`, and it also has another `ModelConfig` available for `llama3.2:3b`; keep one model resident at a time with `ollama stop` when comparing them. In a clean direct `ollama run --verbose` check at these limits, the Qwen run generated at 0.38 tokens/sec. The Llama check timed out and the pod was subsequently OOM-killed, so there is not a trustworthy Llama tokens/sec number to publish from that run. Give Ollama enough memory and CPU before drawing a model-quality or model-speed conclusion. + +**Checkpoint:** `ollama list` shows the model downloaded and ready. + +--- + +## Step 5: Connect kagent to the local model + +The model config lives in `01-local-llm/modelconfig.yaml`: + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: ModelConfig +metadata: + name: local-model-config + namespace: kagent +spec: + model: qwen2.5:1.5b + provider: Ollama + ollama: + host: http://ollama.ollama.svc.cluster.local +``` + +Apply it: + +```bash +kubectl apply -f 01-local-llm/modelconfig.yaml +kubectl get modelconfig -n kagent -o wide +``` + +```text +NAME PROVIDER MODEL +default-model-config Ollama llama3.2:3b +local-model-config Ollama qwen2.5:1.5b +``` + +(`default-model-config` stays on `llama3.2:3b` here. This lab never uses it, since every agent below points explicitly at `local-model-config`.) + +Before moving to the agent layer, validate the model directly: + +```bash +kubectl exec -n ollama deploy/ollama -- ollama run qwen2.5:1.5b "reply with the single word: ready" +``` + +```text +ready +``` + +That proves the model is reachable and generating before any agent starts making tool calls. + +**Checkpoint:** the model responds with "ready". + +--- + +## Step 6: Access the kagent dashboard + +Before you open the UI, forward the dashboard service to your machine: + +```bash +kubectl port-forward -n kagent service/kagent-ui 8082:8080 +``` + +Leave that running in its own terminal. The dashboard is now at **http://localhost:8082**. Every remaining step in this lab uses that URL. + +--- + +## Step 7: Build your first agent (read-only) + +The first agent is intentionally narrow. It's defined in `02-first-agent/agent.yaml`: + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: Agent +metadata: + name: local-k8s-agent + namespace: kagent +spec: + type: Declarative + declarative: + modelConfig: local-model-config + tools: + - type: McpServer + mcpServer: + apiGroup: kagent.dev + kind: RemoteMCPServer + name: kagent-tool-server + toolNames: + - k8s_get_resources + - k8s_get_available_api_resources + - k8s_describe_resource + - k8s_get_pod_logs +``` + +Every tool this agent has access to is read-only. It can inspect cluster state, but it cannot mutate anything. This is one of the clearest, cheapest ways to establish a secure-by-default agent posture: don't grant a tool the agent doesn't need for the job it's doing. + +Deploy it: + +```bash +kubectl apply -f 02-first-agent/agent.yaml +kubectl get agent -n kagent +``` + +![kagent's Agent Details panel for local-k8s-agent, showing its four read-only tools and description: "Read-only Kubernetes inspection agent, running entirely against an in-cluster local model. No write access at this stage."](/img/blog/kagent-part-1-local-ai-agent-kubernetes/read-only-agent-details.png) + +Notice the agent's own description confirms its scope before you even ask it anything. No write tools are listed, because none are attached. + +Open the kagent dashboard at `http://localhost:8082`, select `local-k8s-agent`, and ask: + +> What pods are running in the kagent namespace? + +That's the simplest possible end-to-end validation: the agent calls `k8s_get_resources` with appropriate filters, reads the response, and answers based on what it finds. The answer should match what `kubectl get pods -n kagent` shows you directly. + +![local-k8s-agent answering "What pods are running in the kagent namespace?" with an expanded k8s_get_resources tool call and a table of 20 pods](/img/blog/kagent-part-1-local-ai-agent-kubernetes/read-only-agent-query.png) + +*(This particular cluster has extra agents from other work running alongside the lab. On a fresh cluster you'll see just `local-k8s-agent` and `local-hitl-agent` here, and possibly the core kagent components.)* + +**Checkpoint:** the agent's answer reflects the actual cluster state. + +--- + +## Step 8: Add a write-capable agent behind approval gates + +Now we reach the real security boundary: write operations. + +`03-human-in-the-loop/hitl-agent.yaml` enables destructive tools, but marks them for approval: + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: Agent +metadata: + name: local-hitl-agent + namespace: kagent +spec: + type: Declarative + declarative: + modelConfig: local-model-config + tools: + - type: McpServer + mcpServer: + apiGroup: kagent.dev + kind: RemoteMCPServer + name: kagent-tool-server + toolNames: + - k8s_get_resources + - k8s_describe_resource + - k8s_get_pod_logs + - k8s_get_events + - k8s_get_resource_yaml + - k8s_apply_manifest + - k8s_delete_resource + - k8s_patch_resource + requireApproval: + - k8s_apply_manifest + - k8s_delete_resource + - k8s_patch_resource +``` + +The `requireApproval` list is the whole story here. It's the difference between "the model can propose a change" and "the model can make a change." Everything in that list pauses for human approval before it executes. + +Deploy it: + +```bash +kubectl apply -f 03-human-in-the-loop/hitl-agent.yaml +kubectl get agent -n kagent local-hitl-agent -o wide +``` + +```text +NAME TYPE RUNTIME READY ACCEPTED +local-hitl-agent Declarative python True True +``` + +![kagent's Agent Details panel for local-hitl-agent, showing k8s_apply_manifest and k8s_delete_resource each tagged "Requires approval before execution"](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-agent-tools.png) + +The `requireApproval` YAML above isn't just declared, it's visibly enforced in the UI: every write-capable tool on this agent is flagged before you've asked it to do anything. + +--- + +## Step 9: Walk through the human-in-the-loop workflow + +Open the kagent dashboard and select `local-hitl-agent`. This is a four-part sequence. Do them in order, since each one demonstrates a different piece of the approval boundary. + +### 9.1: Read without approval + +Ask: + +> List all pods in the kagent namespace. + +This executes immediately. It's a read operation, so it's never gated. Only the tools in `requireApproval` pause. + +### 9.2: Approve a write + +Ask: + +> Create a ConfigMap called test-config in the default namespace with the key message set to hello. + +The agent proposes the write and the action pauses in the UI waiting for you. + +![kagent HITL approval screen showing a pending ConfigMap creation, with the full manifest visible and Approve/Reject buttons](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-approval-pending.png) + +Approve it. Then verify it landed: + +```bash +kubectl get configmap test-config -n default -o yaml +``` + +This is the critical point of the whole lab: the model proposed the action, but the human approval gate is the actual boundary between a suggestion and a real mutation. + +### 9.3: Reject a delete + +Ask: + +> Delete the ConfigMap test-config in the default namespace. + +Again it stops at the approval gate. This time, type a reason into the box and click **Reject** instead of Approve: + +> Resource still in use + +![Rejection reason being entered for the pending delete request, with Reject and Cancel buttons visible](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-rejection-reason.png) + +![kagent HITL screen after the delete is rejected, showing a "Rejected" status and the agent confirming the ConfigMap remains in the default namespace](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-rejection-confirmed.png) + +The agent understood the request, proposed the call, and then backed off cleanly when you said no. It didn't retry, argue, or find another way to delete the resource. That's the behavior you actually want from a tool with delete access. + +Verify the resource is untouched: + +```bash +kubectl get configmap test-config -n default +``` + +**A nice extra behavior worth showing:** after backing off, the agent offered to check whether anything was actually depending on `test-config`, since the rejection reason I gave it was "Resource still in use." I said yes, and it came back with a small structured choice instead of guessing what I meant: + +![Agent asking a follow-up question with three quick-action options: check pods and deployments, force delete, or do nothing](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-dependency-check-prompt.png) + +I picked **"Check pods and deployments for references to test-config."** The agent called `k8s_get_resource_yaml` and `k8s_get_resources` against the `default` namespace and reported back: + +![Agent's result after checking pods and deployments, reporting that neither nginx-smoke nor pg-smoke references test-config and no deployments exist in the namespace](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-dependency-check-result.png) + +`test-config` isn't actually referenced by anything in the namespace. The "still in use" reason I gave was just a convenient excuse to test a rejection, not a real dependency. The agent's own investigation surfaced that: no pods or deployments pointed at it, so as far as the cluster is concerned it's safe to delete whenever I actually want to. This is a small but telling moment. The agent didn't just accept the rejection and stop, it offered a concrete next step for resolving *why* the resource was flagged as in use, then went and checked rather than taking my word for it. + +### 9.4: Use an ambiguous prompt + +Ask: + +> Set up a namespace for my application. + +This is intentionally vague, with no namespace name or other parameters. A well-behaved agent should ask a clarifying question rather than guess. + +The agent will ask: + +> What should the namespace be called? + +This is why agentic systems aren't just "LLM with tools." Sometimes the correct action is to stop and ask. Proceeding on a guess would make things worse, not better. + +--- + +## Understanding performance: token speed and resource limits + +You probably noticed each interaction took a while. The measurements below describe this particular pod's resource ceiling, not an immutable property of local models: the Ollama deployment was capped at 2 vCPU and 4Gi of memory, and its last restart was `OOMKilled` while testing the larger model. The other floor is structural: an agent interaction needs several model passes, so even a better-resourced runtime still has more work to do than a single chat completion. + +You can watch it happen directly: + +```bash +kubectl logs -n ollama deploy/ollama --tail=50 +``` + +For a reproducible direct comparison, stop the previous model and use the same short prompt for each model: + +```bash +kubectl exec -n ollama deploy/ollama -- ollama stop qwen2.5:1.5b +kubectl exec -n ollama deploy/ollama -- ollama run --verbose qwen2.5:1.5b \ + 'Explain Kubernetes pods in one concise sentence.' +kubectl exec -n ollama deploy/ollama -- ollama stop llama3.2:3b +kubectl exec -n ollama deploy/ollama -- ollama run --verbose llama3.2:3b \ + 'Explain Kubernetes pods in one concise sentence.' +``` + +At the `2 vCPU / 4Gi` limits used for this run, Qwen reported: + +```text +eval count: 26 token(s) +eval duration: 1m8.797653711s +eval rate: 0.38 tokens/s +``` + +The corresponding Llama run timed out before Ollama returned usable verbose metrics, and the pod's last state was `OOMKilled` with exit code 137. Do not turn that failed run into a speed ratio. To test whether more capacity changes the result, apply the following setting, wait for the replacement pod to be `Ready`, and run the same sequence again: + +```bash +kubectl set resources deployment/ollama -n ollama \ + --limits=cpu=6,memory=8Gi --requests=cpu=2,memory=4Gi +``` + +Record the actual limits and both `eval rate` values with the result. On a Docker allocation below 8Gi, this pod may remain Pending alongside the kagent stack. + +During an earlier agent run, Ollama's generation timings looked like this: + +```text +slot print_timing: id 0 | task 96 | n_gen = 100, tg = 1.92 t/s, tg_3s = 1.94 t/s +slot print_timing: id 0 | task 96 | n_gen = 110, tg = 2.00 t/s, tg_3s = 3.33 t/s +slot print_timing: id 0 | task 96 | n_gen = 127, tg = 2.18 t/s, tg_3s = 5.01 t/s +slot print_timing: id 0 | task 96 | n_gen = 140, tg = 2.15 t/s, tg_3s = 1.97 t/s +``` + +The agent loop multiplies that cost, because a single interaction involves several full passes through the model: reasoning about the question, selecting a tool call, reading the tool result, reasoning about that result, deciding on the next action, and generating the final answer. Each of those is a separate pass through the model. More tool steps mean more passes, which means slower overall. + +Fully local AI is a real, workable option. It is not low-latency under a constrained local pod, and resource limits are a variable you can tune before changing models. If you're building on this, keep prompts short, keep the tool list narrow, keep one model resident at a time, give Ollama enough RAM and CPU, and reach for a GPU-backed node if you have one. + +--- + +## What this lab actually proves + +Strip away the specific commands and this lab demonstrated one thing: a local AI agent can operate inside a real Kubernetes environment, with a real approval boundary, without depending on a hosted model or a cloud key. + +The architecture is explicit: + +- The model runs inside the cluster through Ollama. +- Agent logic is defined declaratively in kagent CRDs. +- Tools are exposed through a dedicated tool server, not called directly. +- Tool access is narrowed to the smallest set of operations each agent actually needs. +- Write operations require explicit human approval before execution. +Two guardrails did all the work here: + +1. **Least-privilege tool selection.** The read-only agent literally cannot mutate anything. +2. **Human approval for writes.** The write-capable agent can propose but not execute alone. +Neither is exotic. They're the minimum viable safety controls for any agentic Kubernetes workflow that's allowed to touch cluster state. + +--- + +## Notes on model selection and behavior + +One thing worth flagging as you experiment: smaller models like `qwen2.5:1.5b` are optimized for speed over reasoning depth. They're excellent at structured tool calling, which is what agents need most, but they can occasionally reach for the wrong tool entirely. + +Here's a real example from this lab. Asked "how many namespaces are currently in my cluster," `local-k8s-agent` called `k8s_get_available_api_resources`, a tool that lists API resource *types*, not namespaces, and then confidently answered "There are currently 51 namespaces in your cluster." A kind cluster running kagent and Ollama has something like seven. The model didn't hallucinate a number out of nowhere; it grabbed the wrong tool and then reported that tool's item count as a namespace count. + +![local-k8s-agent incorrectly answering a namespace count by calling k8s_get_available_api_resources instead of a namespace-listing tool](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hallucination-wrong-tool.png) + +That failure mode sits one layer upstream of tool output: the tools themselves return ground truth, but nothing guarantees the model calls the *right* tool for the question. That's exactly why read-only scoping and approval gates matter. They bound what a wrong tool choice, or a wrong action, can actually do to your cluster. + +If you want to compare a larger model, rerun the direct benchmark after giving Ollama enough memory and CPU; this run did not produce a trustworthy `llama3.2:3b` rate. The architecture stays exactly the same either way. + +--- + +## Current cluster status + +By the time you finish, your lab should look roughly like this: + +> The pod ages below (13h, 14h, 17h) are from a long-running dev cluster, not a fresh run of this lab. If you're following along on a clean cluster, expect ages in minutes. You'll also only see `local-k8s-agent` and `local-hitl-agent` alongside the core kagent components. The extra `*-agent` pods here (`cilium-*`, `istio-agent`, `kgateway-agent`, and so on) are from other work on this particular cluster and aren't part of this lab. + +```text +NAME READY STATUS RESTARTS AGE +kagent-controller-99b4bb79d-cm5jn 1/1 Running 0 13h +kagent-grafana-mcp-678857cd56-s55kt 1/1 Running 0 17h +kagent-kmcp-controller-manager-76bb479b6-h2zq9 1/1 Running 13 17h +kagent-postgresql-85766c5f8c-vfjbr 1/1 Running 0 17h +kagent-querydoc-65cdb65878-h9bx7 1/1 Running 0 17h +kagent-tools-7548fb9ffd-r54kh 1/1 Running 0 13h +kagent-ui-75bd88cc5c-2wl2k 1/1 Running 0 13h +local-hitl-agent-6497c985f4-phjdc 1/1 Running 0 5m +local-k8s-agent-65d9f49888-qgjjg 1/1 Running 0 5m +``` + +Your core agents: + +```text +NAME TYPE RUNTIME READY ACCEPTED +local-hitl-agent Declarative python True True +local-k8s-agent Declarative python True True +``` + +Your model config: + +```text +NAME PROVIDER MODEL +default-model-config Ollama llama3.2:3b +local-model-config Ollama qwen2.5:1.5b +``` + +--- + +## Troubleshooting: what you might hit along the way + +None of these are unusual for a local, multi-component stack. They're worth knowing about before you hit them. + +### Model-name mismatch in default config + +If you see: + +```text +model 'llama3.2' not found (status code: 404) +``` + +...it usually means `default-model-config` is pointing at a model name that doesn't match what's actually being served. Fix it directly: + +```bash +kubectl patch modelconfig default-model-config -n kagent --type merge -p '{"spec":{"model":"qwen2.5:1.5b","ollama":{"host":"http://ollama.ollama.svc.cluster.local"}}}' +``` + +Re-check: + +```bash +kubectl get modelconfig -n kagent -o wide +``` + +The model itself can be perfectly healthy while the agent is still broken, because the config is pointing at the wrong value. Local AI stacks are still software stacks. They fail like software. + +### Startup race with the database + +On a fresh cluster, the kagent controller can start logging failures before Postgres is actually ready. It looks like a broken install. It isn't. The system recovers on its own once the database comes up. Give it a minute, then check pod state rather than reacting to the first error line you see: + +```bash +kubectl get pods -n kagent -o wide +``` + +### Scheduling pressure in kind + +The Ollama pod can hit memory pressure if the node is already busy running the rest of the kagent stack. The fix is to right-size the request for a small model rather than assuming a large, GPU-style resource request. This whole lab is designed to run comfortably on a laptop-sized node. + +### kind image cache mismatch + +Even if an image already exists on your host Docker daemon, the kind node needs it loaded into its own container runtime separately. Check directly on the control-plane node: + +```bash +docker exec kagent-security-lab-control-plane crictl images | grep -i ollama +``` + +If that comes back empty, pull and load it explicitly: + +```bash +docker pull ollama/ollama:latest +kind load docker-image ollama/ollama:latest --name kagent-security-lab +``` + +Then reapply the Deployment and let the pod recreate. + +These four are good reminders that AI infrastructure is still infrastructure. It needs the same checks as any other cluster workload: readiness, scheduling, image propagation, dependency ordering. + +--- + +## Cleanup + +When you're done, remove the local cluster entirely: + +```bash +kind delete cluster --name kagent-security-lab +``` + +Or, if you just want to clean up the test ConfigMap from the HITL workflow: + +```bash +kubectl delete configmap test-config -n default --ignore-not-found +``` + +--- + +## Final takeaway + +The big lesson here isn't that local AI is instant, or that securing an agentic workflow is trivial. It's that local, secure, Kubernetes-native agent workloads are genuinely possible. But they're real systems, not a clever prompt with a couple of tools bolted on. They need a model runtime, a tool surface, a structured agent loop, an approval boundary, and an honest understanding of where the performance and operational bottlenecks actually live. + +That's the real question this lab was built around: not "can AI manage Kubernetes?" but "how do we make that capability useful, observable, and safe enough to run near real infrastructure?" + +**Part 2** picks up exactly where this leaves off. Least-privilege tools and a human approval gate are a solid starting point, but they're not the whole security story for an agent allowed anywhere near a real cluster. Next up: scoping agents with **RBAC and ClusterRoles**, routing and controlling agent traffic through **agentgateway**, and getting real **metrics and observability** into what these agents are actually doing. + +Repository: [`Prianshu-git/Kagent-demo`](https://github.com/Prianshu-git/Kagent-demo) \ No newline at end of file diff --git a/lib/_blog-feed-data.js b/lib/_blog-feed-data.js index 62be5f702..69f5184af 100644 --- a/lib/_blog-feed-data.js +++ b/lib/_blog-feed-data.js @@ -36,6 +36,18 @@ export const FEED_POSTS = [ "vllm" ] }, + { + "slug": "kagent-part-1-local-ai-agent-kubernetes", + "title": "kagent Part 1: Building a Local, Kubernetes-Native AI Agent with Human-in-the-Loop Approval", + "description": "A hands-on lab building a kagent AI agent on a local kind cluster with Ollama: read-only and write-capable agents, human-in-the-loop approval gates, and a practical guide for common issues.", + "datePublished": "2026-08-18T10:00:00.000Z", + "cover": "/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-approval-pending.png", + "tags": [ + "kagent", + "kubernetes", + "ai-agents" + ] + }, { "slug": "local-llm-glossary", "title": "The Local LLM Glossary: Every Term, Flag, and Number in Plain English", @@ -343,17 +355,5 @@ export const FEED_POSTS = [ "datePublished": "2026-04-29T05:56:22.335Z", "cover": "https://cloudmate-test.s3.us-east-1.amazonaws.com/res%2Fhashnode%2Fimage%2Fupload%2Fv1777443605504%2F64d466df-ca7e-4b49-b46d-a2c3177667b6.png", "tags": [] - }, - { - "slug": "day-5-docker-compose-how-docker-actually-gets-used", - "title": "Day 5: Docker Compose - How Docker Actually Gets Used", - "description": "", - "datePublished": "2026-04-28T13:38:38.532Z", - "cover": "/img/blog/day-5-docker-compose-how-docker-actually-gets-used/621a6671-ce9f-4f66-bd2d-dd827035c5fd.png", - "tags": [ - "docker", - "docker-compose", - "docker-images" - ] } ]; diff --git a/public/_redirects b/public/_redirects index eacd72736..42a274147 100644 --- a/public/_redirects +++ b/public/_redirects @@ -114,6 +114,7 @@ /blog/iptables-demo /iptables-demo 301! /blog/istio-service-mesh /istio-service-mesh 301! /blog/k8sgpt-tutorial-when-kubernetes-meets-ai /k8sgpt-tutorial-when-kubernetes-meets-ai 301! +/blog/kagent-part-1-local-ai-agent-kubernetes /kagent-part-1-local-ai-agent-kubernetes 301! /blog/keptn-getting-started /keptn-getting-started 301! /blog/ksctl-making-kubernetes-easy-across-clouds /ksctl-making-kubernetes-easy-across-clouds 301! /blog/kube-proxy-deep-dive /kube-proxy-deep-dive 301! @@ -342,6 +343,7 @@ /iptables-demo /blog/iptables-demo 200! /istio-service-mesh /blog/istio-service-mesh 200! /k8sgpt-tutorial-when-kubernetes-meets-ai /blog/k8sgpt-tutorial-when-kubernetes-meets-ai 200! +/kagent-part-1-local-ai-agent-kubernetes /blog/kagent-part-1-local-ai-agent-kubernetes 200! /keptn-getting-started /blog/keptn-getting-started 200! /ksctl-making-kubernetes-easy-across-clouds /blog/ksctl-making-kubernetes-easy-across-clouds 200! /kube-proxy-deep-dive /blog/kube-proxy-deep-dive 200! diff --git a/public/_worker.js b/public/_worker.js index 995282a62..c425a26bc 100644 --- a/public/_worker.js +++ b/public/_worker.js @@ -799,7 +799,7 @@ async function handleNewsletterApi(request, env) { const KUBESIMPLIFY_ROUTES = new Set(['/about', '/workshops', '/partnerships', '/resources', '/products', '/learn', '/privacy']); const KUBESIMPLIFY_PREFIXES = ['/products/', '/learn/']; -const BLOG_SLUGS = new Set(["10-things-you-might-not-know-about-k9s","12-practical-grep-command-examples-in-linux","a-beginners-guide-to-dualbooting-windows-with-ubuntu-part-1","a-beginners-guide-to-dualbooting-windows-with-ubuntu-part-2","a-complete-walk-through-of-devops","a-kubeconfig-for-gke-that-doesnt-need-gcloud","a-simple-way-to-structure-your-terraform-code","a-simplified-guide-to-yaml","about-my-pdf-editor-project","an-overview-of-gitops-and-argocd","announcing-buildsafe","api-response-in-go","arkade","automate-repetitive-tasks-shell-scripting","automated-github-releases-with-github-actions-and-conventional-commits","avoid-overspending-with-kubecost","aws-elastic-cloud-compute","bake-your-container-images-with-bake","become-a-hashicorp-certified-terraform-associate-preparation-guide","best-devops-tools-2025","bonsai-27b-rtx-pro-6000-dgx-spark","breaking-down-docker","building-a-zero-cve-strategy","building-apigateway-with-lambda-using-pulumi","certified-kubernetes-security-specialist-cks-2022-exam-guide","cicd-pipeline-github-actions-with-aws-ecs","ckad-exam-april-2022","claude-code-leak-what-the-source-actually-teaches","clawspark-your-private-openclaw-ai-assistant-that-never-phones-home","cloud-computing","cloud-native-buildpacks-concepts","confidential-containers-running-on-kubernetes","container-and-kubernetes-security","controlling-mcp-tools-with-agentgateway-on-kubernetes","coolify","creating-multi-node-kubernetes-cluster-locally","day-1-the-local-llm-revolution-why-your-desk-just-became-the-new-datacenter","day-1-what-actually-happens-when-you-type-docker-run","day-2-anatomy-of-an-llm-inference-request-from-prompt-to-answer-step-by-step","day-2-your-images-are-a-supply-chain-and-it-s-probably-broken","day-3-stop-writing-dockerfiles-from-scratch","day-3-the-dgx-spark-unpacked-gb10-unified-memory-sm-121-and-the-one-reason-this-hardware-exists","day-4-breaking-isolation-on-purpose-volumes-networks-and-the-real-world","day-4-quantization-demystified-bf16-fp8-nvfp4-mxfp4-int4-gguf-and-why-it-all-matters","day-5-docker-compose-how-docker-actually-gets-used","day-5-local-llm-inference-engines-wrappers-and-what-to-pick","day-6-run-an-llm-on-your-laptop-with-docker","day-7-ship-it-and-what-comes-next","deploy-a-maven-project-on-a-tomcat-server-using-jenkins-and-aws","deploy-a-simple-server-using-aws-terraform","deploying-java-application-using-docker-and-kubernetes-devops-project","devin-outposts-on-kubernetes","ditch-the-overheating-laptop-supercharge-your-docker-workflow-with-docker-offload","diy-how-to-build-a-kubernetes-policy-engine","docker-captain-journey","docker-mcp-catalog","docker-networking-demystified","dynamic-mig-in-kubernetes-with-hami","embed-http-servers-in-wasm-with-rust-and-csharp","enhancing-runtime-security-with-falco-my-hands-on-experience","ephemeral-pull-request-environment-using-vcluster","essential-linux-commands-for-devops","event-driven-architecture-simplified-monolith-to-microservices","everything-you-need-to-know-about-docker-compose","everything-you-need-to-know-about-the-linux-ls-command","exploiting-metasploitable2-using-msfconsole-kali-linux-lab","firewall-a-networks-gatekeeper","four-pillars-of-observability-in-kubernetes","get-good-at-git","getting-started-with-kind-creating-a-multi-node-local-kubernetes-cluster","getting-started-with-ko-a-fast-container-image-builder-for-your-go-applications","getting-started-with-kyverno","git-and-github-a-beginners-guide","github-actions-101-what-are-github-actions-and-how-to-use-them-a-beginners-guide","gitops-demystified","ha-kubernetes","how-a-kubernetes-service-actually-works-and-all-5-types-you-need","how-get-started-with-hashicorp-vault","how-kubernetes-endpointslices-actually-work-and-why-endpoints-had-to-die","how-to-backup-kubernetes-with-kasten-community-edition","how-to-change-directory-in-shell-scripts","how-to-install-a-kubernetes-cluster-with-kubeadm-containerd-and-cilium-a-hands-on-guide","how-to-setup-your-ftp-server-in-linux","implementing-kubernetes-network-policies-a-comprehensive-guide","important-concepts-of-operating-systems","ing-switch-119-annotations-gateway-api-traefik-impact-ratings","ing-switch-migrate-from-ingress-nginx-to-traefik-or-gateway-api-in-minutes-not-days","installing-prometheus-with-selinux","introducing-kiac-kubernetes-in-apple-containers","introducing-unikraft-lightweight-virtualization-using-unikernels","introduction-of-jenkins-pipeline","introduction-to-cicd-and-cicd-pipeline","introduction-to-cri","introduction-to-developer-platforms-with-gimlet","introduction-to-helm","introduction-to-jenkins","introduction-to-kubernetes","introduction-to-terraform","iptables-demo","istio-service-mesh","k8sgpt-tutorial-when-kubernetes-meets-ai","keptn-getting-started","ksctl-making-kubernetes-easy-across-clouds","kube-proxy-deep-dive","kube-scheduler-deep-dive","kubecon-cloudnativecon-north-america-2024-recap-themes-innovations-and-community-spirit","kubecon-cloudnativecon-rejekts-and-wasm-io-wrap-up-a-leap-into-the-future-with-webassembly-ai-and-sustainable-cloud-practices","kubectl-run-nginx-inside","kubeflow-machine-learning-on-kubernetes-part-1","kubeflow-notebooks-ml-experimentation-made-easier-part-2","kubeflow-pipelines-orchestrating-machine-learning-workflows-part-3","kubernetes-125-dockerd","kubernetes-126","kubernetes-access-control-with-authentication-authorization-admission-control","kubernetes-adoption-key-challenges-in-migrating-to-kubernetes","kubernetes-backup-using-cloudcasa","kubernetes-containerd-setup","kubernetes-crio","kubernetes-management-with-rust-a-dive-into-generic-client-go-controller-abstractions-and-crd-macros-with-kubers","kubernetes-on-apple-macbooks-m-series","kubernetes-scheduling-the-complete-guide","kubernetes-v133-key-features-updates-and-what-you-need-to-know","kubernetes-v135-whats-new-whats-changing-and-what-you-should-know","kubesimplify-a-journey-to-remember","kubesimplify-at-wasmio-and-kubecon-eu-2024","kyverno-and-cosign","kyverno-cli","lets-learn-terraform","lets-simplify-golang-part-1","lets-simplify-golang-part-2","lets-simplify-golang-part-3","lets-talk-about-ansible","linux-boot-process-simplified","linux-system-directories-explained","llm-costs-and-observability-with-agentgateway-on-kubernetes","local-llm-glossary","managing-contexts-in-kubernetes-with-plugins","managing-your-operating-system-with-package-managers","mastering-kubernetes-costs-from-monitoring-to-automation","microservices","mlxcel-rust-native-inference-engine-tested-on-m1-max","moving-code-between-git-repositories-with-copybara","multi-stage-docker-build","multi-tenancy-in-2025-and-beyond","my-first-international-conference-open-source-summit-2022","my-journey-to-kubestronaut-on-kubernetes-10th-birthday","my-kubecon-euvirtual-experience","my-schedule-for-kubecon-cloudnativecon-eu-2022","navigating-through-cncf-landscape","nemotron-3-5-lightning-on-dgx-spark","nemotron3-on-dgx-spark","networking-fundamentals-for-devops","nexus-repository-manager-what-is-it-and-how-to-configure-it-on-a-digital-ocean-droplet","nudgebee-ai-sre-copilot-hands-on","nvcf-is-now-open-source-inside-nvidia-s-gpu-function-platform","operating-systems-101-essential-knowledge-for-devopssre-engineers","optimizing-kubernetes-costs-balancing-spot-and-on-demand-instances-with-topology-spread-constraints","optimizing-scalability-a-deep-dive-into-load-testing-with-locust-on-eks","package-managers-demystified","perform-crud-operations-on-kubernetes-using-golang","platform-engineering-demystified-navigating-the-basics","pods-in-kubernetes","practical-guide-to-kubernetes-api","progressive-rollouts-with-argo-cd-rollouts","prometheus-explained","pure-cilium-a-guide-for-local-load-balancing-and-bgp","quick-bites-of-fluxcd-health-assessment","qwen3-8-27b-on-dgx-spark","rancher-desktop-evolution","ready-for-wasm-day-2023","running-a-big-llm-across-multiple-gpus-with-vllm","running-qwen3-8-flash-next-on-dgx-spark-and-rtx-pro-6000","sharing-gpus-in-kubernetes-with-hami","simplified-introduction-to-bacalhau","slicing-gpus-in-kubernetes-with-nvidia-mig","speeding-up-using-microk8s","ssh-into-your-dgx-spark-from-anywhere-in-the-world-using-tailscale","starting-your-devops-journey-as-a-windows-user","statefulsets","supply-chain-security-using-slsa-part-1-fundamentals","supply-chain-security-using-slsa-part-2-the-framework","terraform-best-practices","testing-docker-ais-gordon-how-smart-is-it","the-complete-guide-to-the-dd-command-in-linux","the-secret-gems-behind-building-container-images-enter-buildkit-and-docker-buildx","the-ultimate-guide-to-audit-logging-in-kubernetes-from-setup-to-analysis","the-webassembly-course","tutorial-build-a-cloud-cost-monitoring-system-with-terraform-ansible-and-komiser","understanding-docker-desktop-all-in-one-platform-for-containers","understanding-etcd-in-kubernetes-a-beginners-guide","understanding-how-containers-work-behind-the-scenes","understanding-the-architecture-of-kubernetes-a-beginners-guide","understanding-the-ins-and-outs-of-git-using-github","wandler-local-openai-compatible-inference-transformersjs-webgpu","what-is-reproducibility-and-why-does-it-matter","what-is-shell-scripting","why-are-network-policies-in-kubernetes-so-hard-to-understand","why-devops-case-study","wtf-is-linux-shell-command-substitution","yours-kindly-drone","zero-trust-istio-sidecar-vs-ambient"]); +const BLOG_SLUGS = new Set(["10-things-you-might-not-know-about-k9s","12-practical-grep-command-examples-in-linux","a-beginners-guide-to-dualbooting-windows-with-ubuntu-part-1","a-beginners-guide-to-dualbooting-windows-with-ubuntu-part-2","a-complete-walk-through-of-devops","a-kubeconfig-for-gke-that-doesnt-need-gcloud","a-simple-way-to-structure-your-terraform-code","a-simplified-guide-to-yaml","about-my-pdf-editor-project","an-overview-of-gitops-and-argocd","announcing-buildsafe","api-response-in-go","arkade","automate-repetitive-tasks-shell-scripting","automated-github-releases-with-github-actions-and-conventional-commits","avoid-overspending-with-kubecost","aws-elastic-cloud-compute","bake-your-container-images-with-bake","become-a-hashicorp-certified-terraform-associate-preparation-guide","best-devops-tools-2025","bonsai-27b-rtx-pro-6000-dgx-spark","breaking-down-docker","building-a-zero-cve-strategy","building-apigateway-with-lambda-using-pulumi","certified-kubernetes-security-specialist-cks-2022-exam-guide","cicd-pipeline-github-actions-with-aws-ecs","ckad-exam-april-2022","claude-code-leak-what-the-source-actually-teaches","clawspark-your-private-openclaw-ai-assistant-that-never-phones-home","cloud-computing","cloud-native-buildpacks-concepts","confidential-containers-running-on-kubernetes","container-and-kubernetes-security","controlling-mcp-tools-with-agentgateway-on-kubernetes","coolify","creating-multi-node-kubernetes-cluster-locally","day-1-the-local-llm-revolution-why-your-desk-just-became-the-new-datacenter","day-1-what-actually-happens-when-you-type-docker-run","day-2-anatomy-of-an-llm-inference-request-from-prompt-to-answer-step-by-step","day-2-your-images-are-a-supply-chain-and-it-s-probably-broken","day-3-stop-writing-dockerfiles-from-scratch","day-3-the-dgx-spark-unpacked-gb10-unified-memory-sm-121-and-the-one-reason-this-hardware-exists","day-4-breaking-isolation-on-purpose-volumes-networks-and-the-real-world","day-4-quantization-demystified-bf16-fp8-nvfp4-mxfp4-int4-gguf-and-why-it-all-matters","day-5-docker-compose-how-docker-actually-gets-used","day-5-local-llm-inference-engines-wrappers-and-what-to-pick","day-6-run-an-llm-on-your-laptop-with-docker","day-7-ship-it-and-what-comes-next","deploy-a-maven-project-on-a-tomcat-server-using-jenkins-and-aws","deploy-a-simple-server-using-aws-terraform","deploying-java-application-using-docker-and-kubernetes-devops-project","devin-outposts-on-kubernetes","ditch-the-overheating-laptop-supercharge-your-docker-workflow-with-docker-offload","diy-how-to-build-a-kubernetes-policy-engine","docker-captain-journey","docker-mcp-catalog","docker-networking-demystified","dynamic-mig-in-kubernetes-with-hami","embed-http-servers-in-wasm-with-rust-and-csharp","enhancing-runtime-security-with-falco-my-hands-on-experience","ephemeral-pull-request-environment-using-vcluster","essential-linux-commands-for-devops","event-driven-architecture-simplified-monolith-to-microservices","everything-you-need-to-know-about-docker-compose","everything-you-need-to-know-about-the-linux-ls-command","exploiting-metasploitable2-using-msfconsole-kali-linux-lab","firewall-a-networks-gatekeeper","four-pillars-of-observability-in-kubernetes","get-good-at-git","getting-started-with-kind-creating-a-multi-node-local-kubernetes-cluster","getting-started-with-ko-a-fast-container-image-builder-for-your-go-applications","getting-started-with-kyverno","git-and-github-a-beginners-guide","github-actions-101-what-are-github-actions-and-how-to-use-them-a-beginners-guide","gitops-demystified","ha-kubernetes","how-a-kubernetes-service-actually-works-and-all-5-types-you-need","how-get-started-with-hashicorp-vault","how-kubernetes-endpointslices-actually-work-and-why-endpoints-had-to-die","how-to-backup-kubernetes-with-kasten-community-edition","how-to-change-directory-in-shell-scripts","how-to-install-a-kubernetes-cluster-with-kubeadm-containerd-and-cilium-a-hands-on-guide","how-to-setup-your-ftp-server-in-linux","implementing-kubernetes-network-policies-a-comprehensive-guide","important-concepts-of-operating-systems","ing-switch-119-annotations-gateway-api-traefik-impact-ratings","ing-switch-migrate-from-ingress-nginx-to-traefik-or-gateway-api-in-minutes-not-days","installing-prometheus-with-selinux","introducing-kiac-kubernetes-in-apple-containers","introducing-unikraft-lightweight-virtualization-using-unikernels","introduction-of-jenkins-pipeline","introduction-to-cicd-and-cicd-pipeline","introduction-to-cri","introduction-to-developer-platforms-with-gimlet","introduction-to-helm","introduction-to-jenkins","introduction-to-kubernetes","introduction-to-terraform","iptables-demo","istio-service-mesh","k8sgpt-tutorial-when-kubernetes-meets-ai","kagent-part-1-local-ai-agent-kubernetes","keptn-getting-started","ksctl-making-kubernetes-easy-across-clouds","kube-proxy-deep-dive","kube-scheduler-deep-dive","kubecon-cloudnativecon-north-america-2024-recap-themes-innovations-and-community-spirit","kubecon-cloudnativecon-rejekts-and-wasm-io-wrap-up-a-leap-into-the-future-with-webassembly-ai-and-sustainable-cloud-practices","kubectl-run-nginx-inside","kubeflow-machine-learning-on-kubernetes-part-1","kubeflow-notebooks-ml-experimentation-made-easier-part-2","kubeflow-pipelines-orchestrating-machine-learning-workflows-part-3","kubernetes-125-dockerd","kubernetes-126","kubernetes-access-control-with-authentication-authorization-admission-control","kubernetes-adoption-key-challenges-in-migrating-to-kubernetes","kubernetes-backup-using-cloudcasa","kubernetes-containerd-setup","kubernetes-crio","kubernetes-management-with-rust-a-dive-into-generic-client-go-controller-abstractions-and-crd-macros-with-kubers","kubernetes-on-apple-macbooks-m-series","kubernetes-scheduling-the-complete-guide","kubernetes-v133-key-features-updates-and-what-you-need-to-know","kubernetes-v135-whats-new-whats-changing-and-what-you-should-know","kubesimplify-a-journey-to-remember","kubesimplify-at-wasmio-and-kubecon-eu-2024","kyverno-and-cosign","kyverno-cli","lets-learn-terraform","lets-simplify-golang-part-1","lets-simplify-golang-part-2","lets-simplify-golang-part-3","lets-talk-about-ansible","linux-boot-process-simplified","linux-system-directories-explained","llm-costs-and-observability-with-agentgateway-on-kubernetes","local-llm-glossary","managing-contexts-in-kubernetes-with-plugins","managing-your-operating-system-with-package-managers","mastering-kubernetes-costs-from-monitoring-to-automation","microservices","mlxcel-rust-native-inference-engine-tested-on-m1-max","moving-code-between-git-repositories-with-copybara","multi-stage-docker-build","multi-tenancy-in-2025-and-beyond","my-first-international-conference-open-source-summit-2022","my-journey-to-kubestronaut-on-kubernetes-10th-birthday","my-kubecon-euvirtual-experience","my-schedule-for-kubecon-cloudnativecon-eu-2022","navigating-through-cncf-landscape","nemotron-3-5-lightning-on-dgx-spark","nemotron3-on-dgx-spark","networking-fundamentals-for-devops","nexus-repository-manager-what-is-it-and-how-to-configure-it-on-a-digital-ocean-droplet","nudgebee-ai-sre-copilot-hands-on","nvcf-is-now-open-source-inside-nvidia-s-gpu-function-platform","operating-systems-101-essential-knowledge-for-devopssre-engineers","optimizing-kubernetes-costs-balancing-spot-and-on-demand-instances-with-topology-spread-constraints","optimizing-scalability-a-deep-dive-into-load-testing-with-locust-on-eks","package-managers-demystified","perform-crud-operations-on-kubernetes-using-golang","platform-engineering-demystified-navigating-the-basics","pods-in-kubernetes","practical-guide-to-kubernetes-api","progressive-rollouts-with-argo-cd-rollouts","prometheus-explained","pure-cilium-a-guide-for-local-load-balancing-and-bgp","quick-bites-of-fluxcd-health-assessment","qwen3-8-27b-on-dgx-spark","rancher-desktop-evolution","ready-for-wasm-day-2023","running-a-big-llm-across-multiple-gpus-with-vllm","running-qwen3-8-flash-next-on-dgx-spark-and-rtx-pro-6000","sharing-gpus-in-kubernetes-with-hami","simplified-introduction-to-bacalhau","slicing-gpus-in-kubernetes-with-nvidia-mig","speeding-up-using-microk8s","ssh-into-your-dgx-spark-from-anywhere-in-the-world-using-tailscale","starting-your-devops-journey-as-a-windows-user","statefulsets","supply-chain-security-using-slsa-part-1-fundamentals","supply-chain-security-using-slsa-part-2-the-framework","terraform-best-practices","testing-docker-ais-gordon-how-smart-is-it","the-complete-guide-to-the-dd-command-in-linux","the-secret-gems-behind-building-container-images-enter-buildkit-and-docker-buildx","the-ultimate-guide-to-audit-logging-in-kubernetes-from-setup-to-analysis","the-webassembly-course","tutorial-build-a-cloud-cost-monitoring-system-with-terraform-ansible-and-komiser","understanding-docker-desktop-all-in-one-platform-for-containers","understanding-etcd-in-kubernetes-a-beginners-guide","understanding-how-containers-work-behind-the-scenes","understanding-the-architecture-of-kubernetes-a-beginners-guide","understanding-the-ins-and-outs-of-git-using-github","wandler-local-openai-compatible-inference-transformersjs-webgpu","what-is-reproducibility-and-why-does-it-matter","what-is-shell-scripting","why-are-network-policies-in-kubernetes-so-hard-to-understand","why-devops-case-study","wtf-is-linux-shell-command-substitution","yours-kindly-drone","zero-trust-istio-sidecar-vs-ambient"]); export default { async fetch(request, env) { diff --git a/public/atom.xml b/public/atom.xml index 90b9ec676..945a050ca 100644 --- a/public/atom.xml +++ b/public/atom.xml @@ -5,7 +5,7 @@ https://blog.kubesimplify.com/ - 2026-09-01T11:10:35.934Z + 2026-09-02T10:41:19.326Z Kubesimplify hello@kubesimplify.com @@ -49,6 +49,18 @@ + + kagent Part 1: Building a Local, Kubernetes-Native AI Agent with Human-in-the-Loop Approval + + https://blog.kubesimplify.com/kagent-part-1-local-ai-agent-kubernetes + 2026-08-18T10:00:00.000Z + 2026-08-18T10:00:00.000Z + A hands-on lab building a kagent AI agent on a local kind cluster with Ollama: read-only and write-capable agents, human-in-the-loop approval gates, and a practical guide for common issues. + + + + + The Local LLM Glossary: Every Term, Flag, and Number in Plain English diff --git a/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/architecture-diagram.jpg b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/architecture-diagram.jpg new file mode 100644 index 000000000..b83642430 Binary files /dev/null and b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/architecture-diagram.jpg differ diff --git a/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hallucination-wrong-tool.png b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hallucination-wrong-tool.png new file mode 100644 index 000000000..923933857 Binary files /dev/null and b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hallucination-wrong-tool.png differ diff --git a/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-agent-tools.png b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-agent-tools.png new file mode 100644 index 000000000..37167b22f Binary files /dev/null and b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-agent-tools.png differ diff --git a/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-approval-confirmed.png b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-approval-confirmed.png new file mode 100644 index 000000000..7970256f9 Binary files /dev/null and b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-approval-confirmed.png differ diff --git a/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-approval-pending.png b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-approval-pending.png new file mode 100644 index 000000000..9aec27d4b Binary files /dev/null and b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-approval-pending.png differ diff --git a/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-dependency-check-prompt.png b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-dependency-check-prompt.png new file mode 100644 index 000000000..42286d2e1 Binary files /dev/null and b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-dependency-check-prompt.png differ diff --git a/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-dependency-check-result.png b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-dependency-check-result.png new file mode 100644 index 000000000..8f7b7cb3b Binary files /dev/null and b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-dependency-check-result.png differ diff --git a/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-rejection-confirmed.png b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-rejection-confirmed.png new file mode 100644 index 000000000..dab92dc61 Binary files /dev/null and b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-rejection-confirmed.png differ diff --git a/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-rejection-reason.png b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-rejection-reason.png new file mode 100644 index 000000000..f9f3858ef Binary files /dev/null and b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-rejection-reason.png differ diff --git a/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/read-only-agent-details.png b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/read-only-agent-details.png new file mode 100644 index 000000000..830714947 Binary files /dev/null and b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/read-only-agent-details.png differ diff --git a/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/read-only-agent-query.png b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/read-only-agent-query.png new file mode 100644 index 000000000..d2c9f7ef3 Binary files /dev/null and b/public/img/blog/kagent-part-1-local-ai-agent-kubernetes/read-only-agent-query.png differ diff --git a/public/llms-full.txt b/public/llms-full.txt index bf7a9a900..0ddbe3e7e 100644 --- a/public/llms-full.txt +++ b/public/llms-full.txt @@ -1653,6 +1653,616 @@ The scripts, recipes and raw benchmark output are in the repo if you want to rep --- +# kagent Part 1: Building a Local, Kubernetes-Native AI Agent with Human-in-the-Loop Approval + +- Canonical: https://blog.kubesimplify.com/kagent-part-1-local-ai-agent-kubernetes +- Published: 2026-08-18 +- Summary: A hands-on lab building a kagent AI agent on a local kind cluster with Ollama: read-only and write-capable agents, human-in-the-loop approval gates, and a practical guide for common issues. + +A chatbot can explain Kubernetes to you. An agent can decide what to inspect next, pick a tool, read the result, and act on it. Which means the question is no longer "can a model talk about my cluster?" but "can it operate on my cluster in a way I can actually trust?" A write-capable agent can make a change unless the system explicitly stops it. That's the boundary this lab is built around. + +kagent is a Kubernetes-native framework for building exactly that. It gives you a runtime, a set of Kubernetes CRDs like `Agent` and `ModelConfig`, and MCP-backed tool integrations that let a model reason about a live cluster and call real tools against it: not just describe what it would do, but actually do it. + +Once a model can call tools, the design question stops being "is the answer good?" and becomes "what is this thing actually allowed to do, and who signs off before it does it?" That's what this lab is about. + +**kagent vs. k8sgpt, briefly:** k8sgpt runs fixed analyzers against your cluster, collects structured findings, and has a model explain them. There's no loop where the model chooses what to do next. kagent runs an actual agent loop: the model decides which tool to call, reads the result, and decides whether to call another tool or answer the user. That's materially different, which is why least-privilege tooling and approval gates matter so much here. + +This is Part 1 of a short series. In this one, we build a fully local kagent stack running on a single laptop: kind cluster, kagent, Ollama serving a small model in-cluster, a read-only agent, and a write-capable agent gated behind human approval. No cloud API key, no external LLM dependency, nothing that leaves your machine. Budget about 45 to 60 minutes hands-on if you're following along. + +## What you'll build + +By the end of this lab you'll have, all running locally: + +- A kind cluster with kagent installed +- Ollama serving `qwen2.5:1.5b` as an in-cluster model service +- A **read-only** agent that can inspect cluster state but cannot change anything +- A **write-capable** agent whose destructive actions pause for your explicit approval +In other words: + +- You interact through the kagent dashboard. +- The agent decides which tool to call. +- The tool server talks to the Kubernetes API. +- Model inference happens locally, through Ollama. +- Write operations pause for your approval before they execute. +![Architecture diagram: a kind node with a user, the kagent controller/UI, Ollama, the Kubernetes MCP tool server, and an approval gate before any write-capable tool call](/img/blog/kagent-part-1-local-ai-agent-kubernetes/architecture-diagram.png) + +**Prerequisites:** Docker, `kind`, `kubectl`, and Helm, plus enough memory to run a small local model alongside the kagent stack. A laptop with 16GB RAM is comfortable. Keep the model small: this walkthrough uses `qwen2.5:1.5b`. + +Clone the lab repo before you start: every step below references files inside it. + +```bash +git clone https://github.com/Prianshu-git/Kagent-demo +cd Kagent-demo +``` + +```yaml +# 00-cluster/kind-config.yaml +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +name: kagent-security-lab +nodes: + - role: control-plane +``` + +--- + +## Step 1: Create the cluster + +Start with a clean kind cluster: + +```bash +kind create cluster --name kagent-security-lab --config 00-cluster/kind-config.yaml +kubectl cluster-info --context kind-kagent-security-lab +``` + +kind's config file doesn't reliably set the cluster name on every version. Passing `--name` explicitly guarantees the context comes up as `kind-kagent-security-lab`, which every command later in this lab assumes. + +This creates the local Kubernetes environment that will host kagent and Ollama. A healthy cluster should show the control plane and core Kubernetes components up. + +**Checkpoint:** run `kubectl get nodes` and confirm one node in `Ready` status. + +--- + +## Step 2: Install kagent + +kagent uses a two-step Helm install: CRDs first, then the app itself. + +```bash +helm install kagent-crds oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \ + --namespace kagent \ + --create-namespace \ + --version 0.9.12 + +helm install kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent \ + --namespace kagent \ + --set providers.default=ollama \ + --version 0.9.12 + +# give the deployments a moment to create their pods before waiting on them. +# Running `kubectl wait` immediately after `helm install` can fail with +# "no matching resources found" if the pods don't exist yet +sleep 15 +kubectl wait --for=condition=ready pod --all -n kagent --timeout=180s +``` + +Version pinning matters because kagent changes frequently. This lab uses kagent `0.9.12`. + +```bash +kubectl get pods -n kagent -o wide +``` + +On a fresh cluster, the kagent controller may log transient failures before Postgres is ready. That's normal. Give it a moment to converge, then validate the pod state. The system recovers on its own. + +**Checkpoint:** every pod in the `kagent` namespace is `Running`. + +--- + +## Step 3: Deploy Ollama in the cluster + +The lab defines the Ollama deployment in `01-local-llm/ollama-deployment.yaml`. + +```bash +kubectl apply -f 01-local-llm/ollama-deployment.yaml +kubectl wait --for=condition=ready pod -l app=ollama -n ollama --timeout=120s +``` + +Validate the Service and endpoints: + +```bash +kubectl get svc -n ollama +kubectl get endpoints -n ollama +``` + +```text +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +ollama ClusterIP 10.96.147.225 80/TCP 153m +``` + +```text +NAME ENDPOINTS AGE +ollama 10.244.0.30:11434 153m +``` + +The Service has a real endpoint. That's your confirmation the in-cluster model service is reachable from the rest of Kubernetes. + +**Checkpoint:** the Service has an endpoint IP address. + +--- + +## Step 4: Pull a local model + +This lab uses `qwen2.5:1.5b`, a 1.5-billion-parameter model optimized for CPU inference. + +```bash +kubectl exec -n ollama deploy/ollama -- ollama pull qwen2.5:1.5b +kubectl exec -n ollama deploy/ollama -- ollama list +``` + +Terminal output from pulling the model: + +```text +pulling manifest +pulling 183715c43589: 48% ▕████████ ▏ 471 MB/986 MB 2.5 MB/s 3m26s +pulling 183715c43589: 72% ▕█████████████ ▏ 713 MB/986 MB 1.1 MB/s 4m17s +pulling 183715c43589: 94% ▕████████████████ ▏ 928 MB/986 MB 16 KB/s 58m29s +pulling 183715c43589: 100% ▕█████████████████ ▏ 985 MB/986 MB 1.8 MB/s 0s +verifying sha256 digest +writing manifest +success +``` + +After the pull completes, check what models are available: + +```text +NAME ID SIZE MODIFIED +qwen2.5:1.5b 65ec06548149 986 MB About an hour ago +llama3.2:latest a80c4f17acd5 2.0 GB 14 hours ago +llama3.2:3b a80c4f17acd5 2.0 GB 15 hours ago +``` + +**Why `qwen2.5:1.5b`?** It's significantly smaller than `llama3.2:3b` (986 MB vs. 2.0 GB) and, on the CPU-only setup used for this lab, generated tokens roughly 8x to 16x faster in practice: 2 to 4 tokens/sec versus 0.25 tokens/sec for `llama3.2:3b` (pod limits: 2 vCPU / 4Gi memory). The tradeoff is slightly lower reasoning capability, but for structured tool calling, which is what agents actually need, it's more than adequate. I switched models partway through building this lab; inference had become painfully slow with `llama3.2:3b` on CPU-only compute. + +**Checkpoint:** `ollama list` shows the model downloaded and ready. + +--- + +## Step 5: Connect kagent to the local model + +The model config lives in `01-local-llm/modelconfig.yaml`: + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: ModelConfig +metadata: + name: local-model-config + namespace: kagent +spec: + model: qwen2.5:1.5b + provider: Ollama + ollama: + host: http://ollama.ollama.svc.cluster.local +``` + +Apply it: + +```bash +kubectl apply -f 01-local-llm/modelconfig.yaml +kubectl get modelconfig -n kagent -o wide +``` + +```text +NAME PROVIDER MODEL +default-model-config Ollama llama3.2:3b +local-model-config Ollama qwen2.5:1.5b +``` + +(`default-model-config` stays on `llama3.2:3b` here; this lab never uses it, since every agent below points explicitly at `local-model-config`.) + +Before moving to the agent layer, validate the model directly: + +```bash +kubectl exec -n ollama deploy/ollama -- ollama run qwen2.5:1.5b "reply with the single word: ready" +``` + +```text +ready +``` + +That proves the model is reachable and generating before any agent starts making tool calls. + +**Checkpoint:** the model responds with "ready". + +--- + +## Step 6: Access the kagent dashboard + +Before you open the UI, forward the dashboard service to your machine: + +```bash +kubectl port-forward -n kagent service/kagent-ui 8082:8080 +``` + +Leave that running in its own terminal. The dashboard is now at **http://localhost:8082**. Every remaining step in this lab uses that URL. + +--- + +## Step 7: Build your first agent (read-only) + +The first agent is intentionally narrow. It's defined in `02-first-agent/agent.yaml`: + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: Agent +metadata: + name: local-k8s-agent + namespace: kagent +spec: + type: Declarative + declarative: + modelConfig: local-model-config + tools: + - type: McpServer + mcpServer: + apiGroup: kagent.dev + kind: RemoteMCPServer + name: kagent-tool-server + toolNames: + - k8s_get_resources + - k8s_get_available_api_resources + - k8s_describe_resource + - k8s_get_pod_logs +``` + +Every tool this agent has access to is read-only. It can inspect cluster state, but it cannot mutate anything. This is one of the clearest, cheapest ways to establish a secure-by-default agent posture: don't grant a tool the agent doesn't need for the job it's doing. + +Deploy it: + +```bash +kubectl apply -f 02-first-agent/agent.yaml +kubectl get agent -n kagent +``` + +![kagent's Agent Details panel for local-k8s-agent, showing its four read-only tools and description: "Read-only Kubernetes inspection agent, running entirely against an in-cluster local model. No write access at this stage."](/img/blog/kagent-part-1-local-ai-agent-kubernetes/read-only-agent-details.png) + +Notice the agent's own description confirms its scope before you even ask it anything: no write tools are listed, because none are attached. + +Open the kagent dashboard at `http://localhost:8082`, select `local-k8s-agent`, and ask: + +> What pods are running in the kagent namespace? + +That's the simplest possible end-to-end validation: the agent calls `k8s_get_resources` with appropriate filters, reads the response, and answers based on what it finds. The answer should match what `kubectl get pods -n kagent` shows you directly. + +![local-k8s-agent answering "What pods are running in the kagent namespace?" with an expanded k8s_get_resources tool call and a table of 20 pods](/img/blog/kagent-part-1-local-ai-agent-kubernetes/read-only-agent-query.png) + +*(This particular cluster has extra agents from other work running alongside the lab; on a fresh cluster you'll see just `local-k8s-agent` and `local-hitl-agent` here, and possibly the core kagent components.)* + +**Checkpoint:** the agent's answer reflects the actual cluster state. + +--- + +## Step 8: Add a write-capable agent behind approval gates + +Now we reach the real security boundary: write operations. + +`03-human-in-the-loop/hitl-agent.yaml` enables destructive tools, but marks them for approval: + +```yaml +apiVersion: kagent.dev/v1alpha2 +kind: Agent +metadata: + name: local-hitl-agent + namespace: kagent +spec: + type: Declarative + declarative: + modelConfig: local-model-config + tools: + - type: McpServer + mcpServer: + apiGroup: kagent.dev + kind: RemoteMCPServer + name: kagent-tool-server + toolNames: + - k8s_get_resources + - k8s_describe_resource + - k8s_get_pod_logs + - k8s_get_events + - k8s_get_resource_yaml + - k8s_apply_manifest + - k8s_delete_resource + - k8s_patch_resource + requireApproval: + - k8s_apply_manifest + - k8s_delete_resource + - k8s_patch_resource +``` + +The `requireApproval` list is the whole story here. It's the difference between "the model can propose a change" and "the model can make a change." Everything in that list pauses for human approval before it executes. + +Deploy it: + +```bash +kubectl apply -f 03-human-in-the-loop/hitl-agent.yaml +kubectl get agent -n kagent local-hitl-agent -o wide +``` + +```text +NAME TYPE RUNTIME READY ACCEPTED +local-hitl-agent Declarative python True True +``` + +![kagent's Agent Details panel for local-hitl-agent, showing k8s_apply_manifest and k8s_delete_resource each tagged "Requires approval before execution"](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-agent-tools.png) + +The `requireApproval` YAML above isn't just declared, it's visibly enforced in the UI: every write-capable tool on this agent is flagged before you've asked it to do anything. + +--- + +## Step 9: Walk through the human-in-the-loop workflow + +Open the kagent dashboard and select `local-hitl-agent`. This is a four-part sequence: do them in order, since each one demonstrates a different piece of the approval boundary. + +### 9.1: Read without approval + +Ask: + +> List all pods in the kagent namespace. + +This executes immediately. It's a read operation, so it's never gated; only the tools in `requireApproval` pause. + +### 9.2: Approve a write + +Ask: + +> Create a ConfigMap called test-config in the default namespace with the key message set to hello. + +The agent proposes the write and the action pauses in the UI waiting for you. + +![kagent HITL approval screen showing a pending ConfigMap creation, with the full manifest visible and Approve/Reject buttons](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-approval-pending.png) + +Approve it. + +![kagent HITL screen after approval, showing "Approved" status and the agent confirming the ConfigMap was successfully created](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-approval-confirmed.png) + +Then verify it landed: + +```bash +kubectl get configmap test-config -n default -o yaml +``` + +This is the critical point of the whole lab: the model proposed the action, but the human approval gate is the actual boundary between a suggestion and a real mutation. + +### 9.3: Reject a delete + +Ask: + +> Delete the ConfigMap test-config in the default namespace. + +Again it stops at the approval gate. This time, type a reason into the box and click **Reject** instead of Approve: + +> Resource still in use + +![Rejection reason being entered for the pending delete request, with Reject and Cancel buttons visible](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-rejection-reason.png) + +![kagent HITL screen after the delete is rejected, showing a "Rejected" status and the agent confirming the ConfigMap remains in the default namespace](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-rejection-confirmed.png) + +The agent understood the request, proposed the call, and then backed off cleanly when you said no. It didn't retry, argue, or find another way to delete the resource. That's the behavior you actually want from a tool with delete access. + +Verify the resource is untouched: + +```bash +kubectl get configmap test-config -n default +``` + +**A nice extra behavior worth showing:** after backing off, the agent offered to check whether anything was actually depending on `test-config`, since the rejection reason I gave it was "Resource still in use." I said yes, and it came back with a small structured choice instead of guessing what I meant: + +![Agent asking a follow-up question with three quick-action options: check pods and deployments, force delete, or do nothing](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-dependency-check-prompt.png) + +I picked **"Check pods and deployments for references to test-config."** The agent called `k8s_get_resource_yaml` and `k8s_get_resources` against the `default` namespace and reported back: + +![Agent's result after checking pods and deployments, reporting that neither nginx-smoke nor pg-smoke references test-config and no deployments exist in the namespace](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hitl-dependency-check-result.png) + +Turns out `test-config` wasn't actually referenced by anything in the namespace. The "still in use" reason I gave was just a convenient excuse to test a rejection, not a real dependency. The agent's own investigation surfaced that: no pods or deployments pointed at it, so as far as the cluster is concerned it's safe to delete whenever I actually want to. This is a small but telling moment: the agent didn't just accept the rejection and stop, it offered a concrete next step for resolving *why* the resource was flagged as in use, then went and checked rather than taking my word for it. + +### 9.4: Use an ambiguous prompt + +Ask: + +> Set up a namespace for my application. + +This is intentionally vague: no namespace name, no other parameters. A well-behaved agent should ask a clarifying question rather than guess. + +The agent will ask: + +> What should the namespace be called? + +This is why agentic systems aren't just "LLM with tools." Sometimes the correct action is to stop and ask. Proceeding on a guess would make things worse, not better. + +--- + +## Understanding performance: token speed and why qwen2.5:1.5b wins + +You probably noticed each interaction took a while. That's not a bug: it's an honest, real tradeoff of local CPU inference. + +You can watch it happen directly: + +```bash +kubectl logs -n ollama deploy/ollama --tail=50 +``` + +During this lab's actual execution, Ollama's generation timings looked like this: + +```text +slot print_timing: id 0 | task 96 | n_gen = 100, tg = 1.92 t/s, tg_3s = 1.94 t/s +slot print_timing: id 0 | task 96 | n_gen = 110, tg = 2.00 t/s, tg_3s = 3.33 t/s +slot print_timing: id 0 | task 96 | n_gen = 127, tg = 2.18 t/s, tg_3s = 5.01 t/s +slot print_timing: id 0 | task 96 | n_gen = 140, tg = 2.15 t/s, tg_3s = 1.97 t/s +``` + +That's roughly 2 to 4 tokens per second with `qwen2.5:1.5b` on CPU, compared to 0.25 tokens per second with `llama3.2:3b` on the same hardware. That's roughly an 8x to 16x difference, and it's immediately noticeable in practice. + +The agent loop multiplies that cost, because a single interaction involves several full passes through the model: reasoning about the question, selecting a tool call, reading the tool result, reasoning about that result, deciding on the next action, and generating the final answer. Each of those is a separate pass through the model: more tool steps means more passes, means slower overall. + +Fully local AI is a real, workable option. It's just not a low-latency option on a CPU-only laptop. If you're building on this, keep prompts short, keep the tool list narrow, keep the model small, give it enough RAM and CPU, and reach for a GPU-backed node if you have one. + +--- + +## What this lab actually proves + +Strip away the specific commands and this lab demonstrated one thing: a local AI agent can operate inside a real Kubernetes environment, with a real approval boundary, without depending on a hosted model or a cloud key. + +The architecture is explicit: + +- The model runs inside the cluster through Ollama. +- Agent logic is defined declaratively in kagent CRDs. +- Tools are exposed through a dedicated tool server, not called directly. +- Tool access is narrowed to the smallest set of operations each agent actually needs. +- Write operations require explicit human approval before execution. +Two guardrails did all the work here: + +1. **Least-privilege tool selection.** The read-only agent literally cannot mutate anything. +2. **Human approval for writes.** The write-capable agent can propose but not execute alone. +Neither is exotic. They're the minimum viable safety controls for any agentic Kubernetes workflow that's allowed to touch cluster state. + +--- + +## Notes on model selection and behavior + +One thing worth flagging as you experiment: smaller models like `qwen2.5:1.5b` are optimized for speed over reasoning depth. They're excellent at structured tool calling, which is what agents need most, but they can occasionally reach for the wrong tool entirely. + +Here's a real example from this lab. Asked "how many namespaces are currently in my cluster," `local-k8s-agent` called `k8s_get_available_api_resources`, a tool that lists API resource *types*, not namespaces, and then confidently answered "There are currently 51 namespaces in your cluster." A kind cluster running kagent and Ollama has something like seven. The model didn't hallucinate a number out of nowhere; it grabbed the wrong tool and then reported that tool's item count as a namespace count. + +![local-k8s-agent incorrectly answering a namespace count by calling k8s_get_available_api_resources instead of a namespace-listing tool](/img/blog/kagent-part-1-local-ai-agent-kubernetes/hallucination-wrong-tool.png) + +That failure mode sits one layer upstream of tool output: the tools themselves return ground truth, but nothing guarantees the model calls the *right* tool for the question. That's exactly why read-only scoping and approval gates matter: they bound what a wrong tool choice, or a wrong action, can actually do to your cluster. + +If you need deeper reasoning at the cost of latency, switch back to `llama3.2:3b`. If you need speed for simple operations, `qwen2.5:1.5b` is hard to beat. The architecture stays exactly the same either way. + +--- + +## Current cluster status + +By the time you finish, your lab should look roughly like this: + +> The pod ages below (13h, 14h, 17h) are from a long-running dev cluster, not a fresh run of this lab. If you're following along on a clean cluster, expect ages in minutes. You'll also only see `local-k8s-agent` and `local-hitl-agent` alongside the core kagent components; the extra `*-agent` pods here (`cilium-*`, `istio-agent`, `kgateway-agent`, and so on) are from other work on this particular cluster and aren't part of this lab. + +```text +NAME READY STATUS RESTARTS AGE +kagent-controller-99b4bb79d-cm5jn 1/1 Running 0 13h +kagent-grafana-mcp-678857cd56-s55kt 1/1 Running 0 17h +kagent-kmcp-controller-manager-76bb479b6-h2zq9 1/1 Running 13 17h +kagent-postgresql-85766c5f8c-vfjbr 1/1 Running 0 17h +kagent-querydoc-65cdb65878-h9bx7 1/1 Running 0 17h +kagent-tools-7548fb9ffd-r54kh 1/1 Running 0 13h +kagent-ui-75bd88cc5c-2wl2k 1/1 Running 0 13h +local-hitl-agent-6497c985f4-phjdc 1/1 Running 0 5m +local-k8s-agent-65d9f49888-qgjjg 1/1 Running 0 5m +``` + +Your core agents: + +```text +NAME TYPE RUNTIME READY ACCEPTED +local-hitl-agent Declarative python True True +local-k8s-agent Declarative python True True +``` + +Your model config: + +```text +NAME PROVIDER MODEL +default-model-config Ollama llama3.2:3b +local-model-config Ollama qwen2.5:1.5b +``` + +--- + +## Troubleshooting: what you might hit along the way + +None of these are unusual for a local, multi-component stack. They're worth knowing about before you hit them. + +### Model-name mismatch in default config + +If you see: + +```text +model 'llama3.2' not found (status code: 404) +``` + +...it usually means `default-model-config` is pointing at a model name that doesn't match what's actually being served. Fix it directly: + +```bash +kubectl patch modelconfig default-model-config -n kagent --type merge -p '{"spec":{"model":"qwen2.5:1.5b","ollama":{"host":"http://ollama.ollama.svc.cluster.local"}}}' +``` + +Re-check: + +```bash +kubectl get modelconfig -n kagent -o wide +``` + +The model itself can be perfectly healthy while the agent is still broken, because the config is pointing at the wrong value. Local AI stacks are still software stacks. They fail like software. + +### Startup race with the database + +On a fresh cluster, the kagent controller can start logging failures before Postgres is actually ready. It looks like a broken install. It isn't: the system recovers on its own once the database comes up. Give it a minute, then check pod state rather than reacting to the first error line you see: + +```bash +kubectl get pods -n kagent -o wide +``` + +### Scheduling pressure in kind + +The Ollama pod can hit memory pressure if the node is already busy running the rest of the kagent stack. The fix is to right-size the request for a small model rather than assuming a large, GPU-style resource request. This whole lab is designed to run comfortably on a laptop-sized node. + +### kind image cache mismatch + +Even if an image already exists on your host Docker daemon, the kind node needs it loaded into its own container runtime separately. Check directly on the control-plane node: + +```bash +docker exec kagent-security-lab-control-plane crictl images | grep -i ollama +``` + +If that comes back empty, pull and load it explicitly: + +```bash +docker pull ollama/ollama:latest +kind load docker-image ollama/ollama:latest --name kagent-security-lab +``` + +Then reapply the Deployment and let the pod recreate. + +These four are good reminders that AI infrastructure is still infrastructure. It needs the same checks as any other cluster workload: readiness, scheduling, image propagation, dependency ordering. + +--- + +## Cleanup + +When you're done, remove the local cluster entirely: + +```bash +kind delete cluster --name kagent-security-lab +``` + +Or, if you just want to clean up the test ConfigMap from the HITL workflow: + +```bash +kubectl delete configmap test-config -n default --ignore-not-found +``` + +--- + +## Final takeaway + +The big lesson here isn't that local AI is instant, or that securing an agentic workflow is trivial. It's that local, secure, Kubernetes-native agent workloads are genuinely possible. But they're real systems, not a clever prompt with a couple of tools bolted on. They need a model runtime, a tool surface, a structured agent loop, an approval boundary, and an honest understanding of where the performance and operational bottlenecks actually live. + +That's the real question this lab was built around: not "can AI manage Kubernetes?" but "how do we make that capability useful, observable, and safe enough to run near real infrastructure?" + +**Part 2** picks up exactly where this leaves off. Least-privilege tools and a human approval gate are a solid starting point, but they're not the whole security story for an agent allowed anywhere near a real cluster. Next up: scoping agents with **RBAC and ClusterRoles**, routing and controlling agent traffic through **agentgateway**, and getting real **metrics and observability** into what these agents are actually doing. + +Repository: [`Prianshu-git/Kagent-demo`](https://github.com/Prianshu-git/Kagent-demo) + +--- + # The Local LLM Glossary: Every Term, Flag, and Number in Plain English - Canonical: https://blog.kubesimplify.com/local-llm-glossary diff --git a/public/llms.txt b/public/llms.txt index cc9d8bef5..da81fef1b 100644 --- a/public/llms.txt +++ b/public/llms.txt @@ -4,7 +4,7 @@ ## About -Kubesimplify is a community-driven publication on cloud-native technologies, with 201 in-depth technical articles by 63 practitioner authors. We cover Kubernetes (kubelet internals, scheduling, networking, operators), container runtimes (containerd, CRI-O, Docker), GitOps (Argo CD, Flux), service meshes, observability, AI/ML infrastructure on Kubernetes, GPU workloads, platform engineering, and the broader CNCF ecosystem. +Kubesimplify is a community-driven publication on cloud-native technologies, with 202 in-depth technical articles by 63 practitioner authors. We cover Kubernetes (kubelet internals, scheduling, networking, operators), container runtimes (containerd, CRI-O, Docker), GitOps (Argo CD, Flux), service meshes, observability, AI/ML infrastructure on Kubernetes, GPU workloads, platform engineering, and the broader CNCF ecosystem. Authoritative, practitioner-written, citation-friendly. Articles include code examples, diagrams, and references. @@ -34,11 +34,12 @@ Authoritative, practitioner-written, citation-friendly. Articles include code ex - Cloud Native Security: https://blog.kubesimplify.com/hub/security (network policies, Falco, Kyverno, SLSA supply-chain) - Linux Fundamentals: https://blog.kubesimplify.com/hub/linux (shell, sysadmin, networking primitives) -## Recent posts (most recent 30 of 201) +## Recent posts (most recent 30 of 202) - [Running a big LLM across multiple GPUs with vLLM](https://blog.kubesimplify.com/running-a-big-llm-across-multiple-gpus-with-vllm) (2026-09-01). A runbook for serving a model too big for one GPU: download to serving in seven steps, with every vLLM flag, startup log line and real error explained, plus tensor, pipeline and expert parallelism benchmarked head to head on a 235B model across four RTX PRO 6000 cards. - [Zero Trust in Practice: Migrating from Istio Sidecar to Ambient Mode](https://blog.kubesimplify.com/zero-trust-istio-sidecar-vs-ambient) (2026-08-31). A hands-on comparison of Istio sidecar and ambient mode for zero-trust service mesh. Same app, same policy, two architectures proven step by step on a local cluster. - [Running Qwen3.8-Flash-Next on a DGX Spark and RTX PRO 6000](https://blog.kubesimplify.com/running-qwen3-8-flash-next-on-dgx-spark-and-rtx-pro-6000) (2026-08-27) +- [kagent Part 1: Building a Local, Kubernetes-Native AI Agent with Human-in-the-Loop Approval](https://blog.kubesimplify.com/kagent-part-1-local-ai-agent-kubernetes) (2026-08-18). A hands-on lab building a kagent AI agent on a local kind cluster with Ollama: read-only and write-capable agents, human-in-the-loop approval gates, and a practical guide for common issues. - [The Local LLM Glossary: Every Term, Flag, and Number in Plain English](https://blog.kubesimplify.com/local-llm-glossary) (2026-08-18). Plain-English definitions for every term you hit in local LLM posts: prefill and decode, tokens per second, FP8 and NVFP4, Q4_K_M, KV cache, YaRN, Gated DeltaNet, speculative decoding, and every vLLM, llama.cpp, and Ollama flag worth knowing. - [Running Qwen3.8-27B on DGX Spark](https://blog.kubesimplify.com/qwen3-8-27b-on-dgx-spark) (2026-08-17). Qwen3.8-27B on DGX Spark with llama.cpp, Ollama, vLLM, and SGLang: the recipes, the tokens per second I measured, MTP speculative decoding, and the sharp edges I hit along the way. - [I Ran an AI SRE Copilot on My Own Hardware. Here Is What It Actually Does.](https://blog.kubesimplify.com/nudgebee-ai-sre-copilot-hands-on) (2026-08-17). Running NudgeBee v1.4.0 end to end - a self-hosted AIOps platform behind AI-SRE, AI-FinOps, AI-K8sOps, and agentic automation - on a Mac, a kiac cluster, and a DGX Spark. @@ -65,11 +66,10 @@ Authoritative, practitioner-written, citation-friendly. Articles include code ex - [Day 7: Ship It - and What Comes Next](https://blog.kubesimplify.com/day-7-ship-it-and-what-comes-next) (2026-05-04). Your container runs as root and has 18 CVEs. A Docker Captain's guide to hardening, Scout policies, DHI, Sandboxes, and what comes after Docker. - [Day 6: Run an LLM on Your Laptop - With Docker](https://blog.kubesimplify.com/day-6-run-an-llm-on-your-laptop-with-docker) (2026-04-30). \"Pull AI models from Docker Hub, run them locally with GPU acceleration, and build an AI-powered app - [A Kubeconfig for GKE That Doesn't Need gcloud](https://blog.kubesimplify.com/a-kubeconfig-for-gke-that-doesnt-need-gcloud) (2026-04-29) -- [Day 5: Docker Compose - How Docker Actually Gets Used](https://blog.kubesimplify.com/day-5-docker-compose-how-docker-actually-gets-used) (2026-04-28) ## Topics covered (auto-derived from tags) -- kubernetes (101 articles): https://blog.kubesimplify.com/tag/kubernetes +- kubernetes (102 articles): https://blog.kubesimplify.com/tag/kubernetes - devops (71 articles): https://blog.kubesimplify.com/tag/devops - docker (31 articles): https://blog.kubesimplify.com/tag/docker - k8s (27 articles): https://blog.kubesimplify.com/tag/k8s @@ -91,9 +91,9 @@ Authoritative, practitioner-written, citation-friendly. Articles include code ex - github (8 articles): https://blog.kubesimplify.com/tag/github - terraform (8 articles): https://blog.kubesimplify.com/tag/terraform - gpu (7 articles): https://blog.kubesimplify.com/tag/gpu +- ai-agents (7 articles): https://blog.kubesimplify.com/tag/ai-agents - docker-images (7 articles): https://blog.kubesimplify.com/tag/docker-images - kubesimplify (7 articles): https://blog.kubesimplify.com/tag/kubesimplify -- linux-basics (7 articles): https://blog.kubesimplify.com/tag/linux-basics ## Top contributors diff --git a/public/rss.xml b/public/rss.xml index beb1920cb..150628f41 100644 --- a/public/rss.xml +++ b/public/rss.xml @@ -32,6 +32,14 @@ aillmvllmllamacppgpu + + kagent Part 1: Building a Local, Kubernetes-Native AI Agent with Human-in-the-Loop Approval + https://blog.kubesimplify.com/kagent-part-1-local-ai-agent-kubernetes + https://blog.kubesimplify.com/kagent-part-1-local-ai-agent-kubernetes + Tue, 18 Aug 2026 10:00:00 GMT + A hands-on lab building a kagent AI agent on a local kind cluster with Ollama: read-only and write-capable agents, human-in-the-loop approval gates, and a practical guide for common issues. + kagentkubernetesai-agentshuman-in-the-loop + The Local LLM Glossary: Every Term, Flag, and Number in Plain English https://blog.kubesimplify.com/local-llm-glossary diff --git a/vercel.json b/vercel.json index 7670b27c6..5570a1b27 100644 --- a/vercel.json +++ b/vercel.json @@ -561,6 +561,11 @@ "destination": "https://blog.kubesimplify.com/k8sgpt-tutorial-when-kubernetes-meets-ai", "permanent": true }, + { + "source": "/blog/kagent-part-1-local-ai-agent-kubernetes", + "destination": "https://blog.kubesimplify.com/kagent-part-1-local-ai-agent-kubernetes", + "permanent": true + }, { "source": "/blog/keptn-getting-started", "destination": "https://blog.kubesimplify.com/keptn-getting-started", @@ -2172,6 +2177,17 @@ } ] }, + { + "source": "/kagent-part-1-local-ai-agent-kubernetes", + "destination": "https://blog.kubesimplify.com/kagent-part-1-local-ai-agent-kubernetes", + "permanent": true, + "has": [ + { + "type": "host", + "value": "kubesimplify.com" + } + ] + }, { "source": "/keptn-getting-started", "destination": "https://blog.kubesimplify.com/keptn-getting-started",