diff --git a/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/01-workflow.md b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/01-workflow.md new file mode 100644 index 0000000000..8f4ad47d04 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/01-workflow.md @@ -0,0 +1,53 @@ +--- +title: Understand the workflow +description: Map the manager, local model workers, validator, and shop to their Arm VM and sandbox boundaries. +weight: 2 +layout: learningpathall +--- + +## Follow the request from plan to preview + +Use Axion Agent Studio to add a discount and shipping rule to a prepared shop. Hosted Gemini plans the work, and local Gemma workers implement and test it on Arm-based Google Axion CPUs. + +Google Kubernetes Engine (GKE) schedules the workloads, and Agent Sandbox manages the isolated worker environments. This Learning Path extends [Run agent-generated Python in GKE Agent Sandbox on Google Axion](/learning-paths/servers-and-cloud-computing/gke-agent-sandbox/) with a multi-role application workflow. You start with an existing compatible cluster. + +The prepared repository contains a working storefront, catalog, and checkout. You submit a request to change its pricing policy and display. The manager provides the relevant source files and role instructions to each worker. + +![Technical deployment diagram showing hosted Gemini outside GKE, an N4A manager, three separate Gemma sandbox Pods sharing one C4A VM, and an N4A validator. The manager coordinates all service calls and enables the shop only after validation passes.#center](images/architecture-technical.svg "Deployment boundaries and numbered workflow stages. VM capacity is distinct from Pod requests and limits.") + +[Open the full-size architecture diagram (SVG)](../images/architecture-technical.svg) · [Download the high-resolution PNG](../images/architecture-technical.png) + +1. The **manager** calls hosted Gemini through Vertex AI to extract a supported policy and a plan for three roles. +2. The **Backend Engineer** and **Frontend Engineer** run concurrently. Each calls its own local Gemma server. The backend produces `pricing.py`. The frontend produces four constrained fragments that the worker assembles into the existing `checkout.html`. +3. The manager combines the non-overlapping files. The **Quality Assurance (QA) Engineer** then generates `qa_tests.py` against this combined candidate. +4. The **validator** executes protected pricing checks and the agent-written tests in a separate gVisor sandbox. The manager can request one bounded repair round if validation fails. +5. After the checks pass, the manager requests a hosted Gemini summary and then enables the updated shop. If the summary is unavailable, a passed validation still permits the preview. The deterministic checks control the preview gate; the summary does not approve the code. + +The manager serves the dashboard and checkout HTML. For a validated basket, it forwards the quote request to the validator, which executes the accepted pricing code. Each run creates candidate files and a preview URL, not a new public application deployment. + +## Separate the infrastructure layers + +A **virtual machine (VM)** is a GKE worker node with CPU and memory. A **Pod** is a Kubernetes workload scheduled onto a node. Here, a **Sandbox resource** manages a worker Pod's lifecycle, and the Pod selects the **gVisor runtime** to isolate execution. The sandbox uses its underlying VM's CPU; it does not obtain a separate model API compute service. + +Each engineer's Pod contains a worker HTTP service and a `llama-server` process. The model endpoint listens on Pod loopback. The manager calls the worker service, which sends inference requests to that local endpoint. All three roles use the same model weights with different instructions and output contracts. + +| Workload | Recorded VM placement | What runs there | +| --- | --- | --- | +| Manager, dashboard, and shop serving | One `n4a-standard-2` regular node | Python coordination, API calls, status, HTML serving | +| Backend, frontend, and QA workers | One shared `c4a-standard-16` sandbox node | Three local Gemma servers and worker services | +| Independent validator | One `n4a-standard-8` sandbox node | Protected checks, generated tests, accepted pricing code | +| GKE system workloads | One `c4a-standard-2` regular node | Cluster support workloads | + +The recorded environment has four VMs with 28 vCPUs in total. The three engineer cards share one 16-vCPU VM. The GKE control plane is managed separately. + +C4A hosts the generation work in this tested arrangement. N4A hosts coordination and validation. These are placement choices demonstrated by this application, not a measured claim that either machine family is optimal for every workload. + +## Identify what the Arm CPUs do + +Axion CPUs run local model inference, coordination, file assembly, tests, and shop serving. Hosted Gemini inference runs outside these VMs. The demo uses no GPU for the local Gemma workers: their llama.cpp arguments include `--n-gpu-layers 0`. + +The application restricts the task to percentage discounts, subtotal thresholds, shipping charges, and the existing checkout display. Login, payments, database migrations, new quote fields, and arbitrary UI redesigns are outside this contract. That boundary makes the generated result practical to verify. + +## What you have learned + +You can distinguish hosted planning from local inference and map each workload to its VM and isolation boundary. Next, verify that your environment provides those resources and the required Sandbox API. diff --git a/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/02-prepare.md b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/02-prepare.md new file mode 100644 index 0000000000..78f21661c9 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/02-prepare.md @@ -0,0 +1,85 @@ +--- +title: Prepare your environment +description: Check the cluster API, Arm node pools, permissions, and source bundle before deploying the workflow. +weight: 3 +layout: learningpathall +--- + +## Confirm the cluster contract + +Allow approximately two hours for the exercises, plus time for cluster provisioning and large image downloads. + +Use a dedicated learning environment. Running VM nodes, disks, Artifact Registry storage, and hosted Gemini requests can incur charges even when the dashboard is idle. Namespace CPU quotas do not cap your cloud bill. + +The supplied manifests target the recorded GKE Agent Sandbox API: `agents.x-k8s.io/v1beta1`, with `spec.operatingMode` and `spec.podTemplate`. The recorded cluster uses GKE `1.36.4-gke.1391002`, Container-Optimized OS on `arm64`, and containerd `2.2.7`. + +The [Agent Sandbox v1.0.9 resource definition](https://github.com/kubernetes-sigs/agent-sandbox/blob/v1.0.9/k8s/crds/agents.x-k8s.io_sandboxes.yaml) includes these fields. Follow [Google's Agent Sandbox installation guidance](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/how-install-agent-sandbox) for your cluster, then verify its installed custom resource definition (CRD) and use server-side dry-run later. Matching field names do not establish that the full workflow runs correctly on another controller release. If the required schema is absent, agree a compatible setup with the cluster administrator before continuing. + +Use Bash for the commands in this Learning Path. Replace the values in this block with your own existing environment, then keep the terminal open: + +```bash +export PROJECT_ID="your-project-id" +export REGION="your-registry-region" +export CLUSTER_NAME="your-cluster-name" +export CLUSTER_LOCATION="your-cluster-region-or-zone" +export REPOSITORY="your-existing-docker-repository" +export REGISTRY="$REGION-docker.pkg.dev/$PROJECT_ID/$REPOSITORY" +export TAG="lp-review-1" +export NAMESPACE="axion-agent-studio" + +gcloud container clusters get-credentials "$CLUSTER_NAME" \ + --location "$CLUSTER_LOCATION" --project "$PROJECT_ID" +kubectl config current-context +kubectl get runtimeclass gvisor +kubectl explain sandbox.spec --api-version=agents.x-k8s.io/v1beta1 +kubectl get nodes -L kubernetes.io/arch,node.kubernetes.io/instance-type,studio.axion.dev/pool +``` + +Confirm that the context is the intended cluster. The Sandbox schema must expose `operatingMode` and `podTemplate`, and the `gvisor` RuntimeClass must exist. + +The node pools must have the following **custom labels**. These labels are distinct from the GKE node-pool names. Ask the administrator to configure labels on the node pools so replacements retain them. + +| Custom `studio.axion.dev/pool` label | Capacity used in the recorded demo | Runtime | +| --- | --- | --- | +| `n4a-regular` | One N4A node, 2 vCPUs | Regular container runtime | +| `c4a-sandbox` | One C4A node, 16 vCPUs | gVisor | +| `n4a-sandbox` | One N4A node, 8 vCPUs | gVisor | + +All three pools must report `kubernetes.io/arch=arm64`. Sandbox nodes must report `sandbox.gke.io/runtime=gvisor`. Keep separate capacity for system workloads. Verify the node's allocatable resources rather than assuming its advertised capacity is entirely available to Pods. + +## Arrange access before building + +An administrator must allow creation and deletion of a dedicated Namespace, which is a cluster-scoped resource. Your deployment identity needs permission to read the cluster and create the tutorial's namespaced resources: Sandbox objects, Services, ServiceAccounts, Roles, RoleBindings, NetworkPolicies, and a ResourceQuota. + +For dashboard node details, an administrator also needs to apply the generated cluster-scoped observer role and binding. These permissions are limited to the named tutorial nodes. + +Your image builder needs Artifact Registry write access. The node service account needs read access to the image repository and the GKE node permissions required by your organization. + +The manager uses Workload Identity Federation for GKE to call Vertex AI without a downloaded service-account key. The cluster and regular node pool must have it enabled. Ask an IAM administrator to grant the manager's Kubernetes service account the appropriate model access. The recorded setup uses `roles/aiplatform.user` with this principal shape: + +```text +principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/axion-agent-studio/sa/studio-manager +``` + +Replace both project identifiers when arranging the binding. The workers and validator do not need this hosted-model permission. Follow [Google's workload authentication guide](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) for the IAM configuration. Confirm that the Vertex AI and IAM Credentials APIs and the selected Gemini model are available to your project. + +The manifests assume enforced NetworkPolicy and the GKE Dataplane V2 metadata route. They permit the trusted manager to reach its worker services, the validator, DNS, the verified Kubernetes API endpoint, the metadata endpoint, and public HTTPS. The HTTPS rule is not a hostname allowlist. The sandbox workloads do not receive general internet egress or an automatically mounted Kubernetes token. + +## Companion source release pending + +This draft is available for documentation review. The companion application source is not included in this pull request because its public release and licensing still need owner approval. A public source location and immutable revision or checksum must be added before this Learning Path can be followed end to end. + +The remaining commands describe the intended workflow after that source release is available. After you obtain the approved source, set the path to its extracted directory. All subsequent application commands run from that directory: + +```bash +export LP_WORKDIR="$HOME/axion-lp-review/axion-agent-studio" +cd "$LP_WORKDIR" +python3.12 -m venv .venv +source .venv/bin/activate +python -m pip install -r requirements-dev.txt +python -m pytest tests -q +``` + +Local tests exercise contracts, validation logic, and application behavior using controlled inputs. They do not prove live Gemini access, model quality, or GKE isolation. Review the source release license and notices before using the application. + +At this point you have checked the cluster boundary and prepared the application source. Next, build the arm64 images and generate manifests for your own environment. diff --git a/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/03-configure.md b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/03-configure.md new file mode 100644 index 0000000000..b8d9f26d59 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/03-configure.md @@ -0,0 +1,165 @@ +--- +title: Configure the local model and images +description: Pin the recorded Gemma configuration, build arm64 application images, and render portable manifests. +weight: 4 +layout: learningpathall +--- + +## Select the recorded model configuration + +The successful recorded run uses Gemma 4 E4B IT in GGUF `Q4_0` format. Use the same model for all three roles first. A smaller model changes the quality and timing experiment; it is not a drop-in validation of this workflow. + +| Setting | Recorded value | +| --- | --- | +| Model file | `gemma-4-E4B-it-Q4_0.gguf` | +| File size | 4,590,807,392 bytes | +| llama.cpp runtime | Build `b11580`, revision beginning `4413a192` | +| Inference threads / batch threads | 4 / 4 per model server | +| Context size | 8,192 tokens | +| Parallel inference slots | 1 per model server | +| GPU layers | 0 | +| Hosted planner | `gemini-3.8-flash`, through Vertex AI | + +The model file's SHA-256 is: + +```text +a555b900214b477d8880e7832e0b8925e139b0159640036b09fe472b6f2097f2 +``` + +Review the [model repository and its terms](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/tree/b8093469224f83f5c38f691eb906c380e9e63114). The included downloader selects an immutable repository revision and verifies both size and checksum. It downloads about 4.6 GB: + +```bash +cd "$LP_WORKDIR" +python -m worker.download_model --model e4b +export MODEL_FILE="$LP_WORKDIR/models/gemma-4-E4B-it-Q4_0.gguf" +export GEMINI_MODEL="gemini-3.8-flash" +``` + +If your organization supplies the same file through an approved channel, point `MODEL_FILE` at that file instead. Do not silently substitute different weights. Check [Gemini model availability](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-8-flash) for your project and selected location before deploying. + +## Build the arm64 images + +Authenticate Docker to your own registry and build with a Buildx builder that supports the target platform. The application Dockerfiles resolve arm64 wheels in a builder stage and copy them into the arm64 runtime. + +```bash +cd "$LP_WORKDIR" +mkdir -p .build +gcloud auth configure-docker "$REGION-docker.pkg.dev" +docker buildx inspect --bootstrap + +docker buildx build --platform linux/arm64 \ + -f worker/Dockerfile -t "$REGISTRY/worker:$TAG" \ + --metadata-file .build/worker.json --push . +docker buildx build --platform linux/arm64 \ + -f infra/Dockerfile.studio -t "$REGISTRY/studio:$TAG" \ + --metadata-file .build/studio.json --push . +``` + +Package the verified model as an init-container image. Stage only the verified model file into a dedicated build context: + +```bash +python tools/stage_model.py "$MODEL_FILE" +docker buildx build --platform linux/arm64 \ + -f "$LP_WORKDIR/infra/Dockerfile.model-local" \ + --build-arg "BASE_IMAGE=$REGISTRY/worker:$TAG" \ + --build-arg "MODEL_FILENAME=gemma-4-E4B-it-Q4_0.gguf" \ + -t "$REGISTRY/model:$TAG" --metadata-file .build/model.json \ + --push .build/model +``` + +The renderer uses this recorded llama.cpp image digest: + +```text +ghcr.io/ggml-org/llama.cpp@sha256:21397f7329ff6fd5705b0bcedcb7e2ce10b5ce97cc12e6564278ac7133523ea6 +``` + +The digest pins the runtime image, while the model checksum pins the weights. Keep build output and image digests with your review evidence. Before a public release, rebuild and verify all application dependencies and record their resulting image digests. + +## Describe your own network + +Obtain the primary node, Pod, and Service ranges from the cluster administrator. The supplied renderer requires non-overlapping private IPv4 ranges within RFC 1918 address space (`10.0.0.0/8`, `172.16.0.0/12`, or `192.168.0.0/16`), with prefix lengths from `/8` to `/30`. The commands here configure one range for each category. The renderer rejects other network arrangements. Use your cluster's actual ranges; do not substitute broader ranges to pass configuration checks. + +For the API rule, use the actual destination behind the in-cluster `default/kubernetes` Service after address translation, expressed as one IPv4 `/32`. This can differ from the external address in your kubeconfig. Do not use the virtual Service IP. + +Inspect the Service's backend addresses, and have the administrator confirm which address applies to your cluster's data plane: + +```bash +kubectl -n default get endpointslice \ + -l kubernetes.io/service-name=kubernetes -o yaml +``` + +If your cluster exposes that backend only through a legacy Endpoints object, inspect `kubectl -n default get endpoints kubernetes -o yaml`. The renderer accepts one verified backend address for this recorded network arrangement. A cluster requiring several API backend addresses needs a reviewed manifest change before deployment. The review renderer checks this distinction to avoid opening its API rule to a whole network. + +To populate the dashboard's verified VM labels and node statistics, capture only the tutorial nodes. This creates local configuration data; review the names, machine types, and internal addresses before using it: + +```bash +kubectl get nodes \ + -l 'studio.axion.dev/pool in (n4a-regular,c4a-sandbox,n4a-sandbox)' \ + -o json > nodes.local.json +python - <<'PYNODES' +import json +from pathlib import Path +nodes = json.loads(Path("nodes.local.json").read_text())["items"] +if not 1 <= len(nodes) <= 8: + raise SystemExit("Expected one to eight explicitly selected tutorial nodes") +addresses = {} +for node in nodes: + name = node["metadata"]["name"] + labels = node["metadata"].get("labels", {}) + ips = [a["address"] for a in node["status"]["addresses"] + if a["type"] == "InternalIP" and ":" not in a["address"]] + if labels.get("kubernetes.io/arch") != "arm64" or len(ips) != 1: + raise SystemExit(f"Review architecture and primary IPv4 for {name}") + addresses[name] = ips[0] + print(name, labels.get("node.kubernetes.io/instance-type"), ips[0]) +Path("node-addresses.local.json").write_text(json.dumps(addresses, indent=2) + "\n") +PYNODES +``` + +The renderer confines node-stat requests to this explicit mapping and your declared node subnet ranges. An administrator must approve the generated ClusterRole and ClusterRoleBinding, which grant `get` on `nodes` and `nodes/stats` for these node names. The node-stat response can include other Pods on the approved node; the application filters its own workloads, but that namespace filtering is not enforced by the `nodes/stats` RBAC permission. Node addresses can change after node replacement; regenerate and re-review the mapping then. Do not publish this local file. + +You can omit `--node-addresses` if your administrator does not permit this access, but verified VM details and some worker metrics will be unavailable in the dashboard. In that case use `kubectl` for placement evidence and report missing metrics rather than treating them as zero. + +Set these values to your verified ranges: + +```bash +export KUBE_API_CIDR="your-control-plane-ip/32" +export NODE_CIDR="your-node-subnet-cidr" +export POD_CIDR="your-pod-range-cidr" +export SERVICE_CIDR="your-service-range-cidr" + +python tools/configure.py \ + --project "$PROJECT_ID" --registry "$REGISTRY" --tag "$TAG" \ + --kube-api-cidr "$KUBE_API_CIDR" --node-cidr "$NODE_CIDR" \ + --pod-cidr "$POD_CIDR" --service-cidr "$SERVICE_CIDR" \ + --node-addresses node-addresses.local.json \ + --gemini-model "$GEMINI_MODEL" --output config.local.json +``` + +Pin the three application images to the digests written by Buildx, then render the manifests. Run this from the same source directory, where `.build/*.json` contains the successful build metadata: + +```bash +python - <<'PYCONFIG' +import json, os, re +from pathlib import Path +path = Path("config.local.json") +config = json.loads(path.read_text()) +for name in ("worker", "studio", "model"): + metadata = json.loads(Path(f".build/{name}.json").read_text()) + digest = metadata["containerimage.digest"] + if not re.fullmatch(r"sha256:[0-9a-f]{64}", digest): + raise SystemExit(f"Invalid image digest for {name}") + config[f"{name}_image"] = f"{os.environ['REGISTRY']}/{name}@{digest}" +path.write_text(json.dumps(config, indent=2) + "\n") +PYCONFIG +python infra/render.py --config config.local.json --require-digests \ + --phase full --output generated/full.json +``` + +The configuration selects C4A for all three engineers, N4A for the manager, and N4A for validation. It does not restore a previous recording or inject a successful result. Review the generated JSON before applying it. Do not commit your local configuration, network addresses, or credentials to a public repository. + +Each engineer Pod requests 4.25 CPUs and 8.5 GiB of memory across its worker and model containers. Its combined limits are 6.5 CPUs and 13 GiB. Kubernetes schedules using requests: the three CPU requests total 12.75, while their 19.5 total CPU limit permits bursts but does not create more than the shared node's capacity. + +## What you have configured + +You have selected the recorded model settings, built and pinned your own arm64 images, and generated manifests using your cluster network. Next, check those manifests against the live API before deployment. diff --git a/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/04-deploy.md b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/04-deploy.md new file mode 100644 index 0000000000..014344f4bf --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/04-deploy.md @@ -0,0 +1,87 @@ +--- +title: Deploy and inspect the workers +description: Validate the manifests against the installed API, deploy the workflow, and verify its actual placement and readiness. +weight: 5 +layout: learningpathall +--- + +## Validate the manifests before applying them + +Continue in the same terminal and source directory. Check that the namespace is unused before a new installation. If it already contains another deployment, choose a dedicated learning cluster or coordinate with its owner; do not overwrite it. + +```bash +cd "$LP_WORKDIR" +kubectl get namespace "$NAMESPACE" +``` + +For a fresh environment, the namespace lookup must report `NotFound`. Create this dedicated namespace before dry-running namespaced resources; a dry-run namespace creation alone does not make the namespace available to subsequent objects: + +```bash +kubectl create namespace "$NAMESPACE" +kubectl apply --dry-run=server -f generated/full.json +``` + +If server-side dry-run rejects the Sandbox schema, stop. Resolve API compatibility before creating resources. A successful render proves JSON construction, while server-side dry-run checks admission against the actual cluster. + +After dry-run succeeds, have an identity authorized for the namespaced resources and the narrowly scoped node observer role apply the workload set: + +```bash +kubectl apply -f generated/full.json +kubectl -n "$NAMESPACE" get sandboxes.agents.x-k8s.io +kubectl -n "$NAMESPACE" get pods -o wide +kubectl -n "$NAMESPACE" rollout status deployment/studio-manager --timeout=300s +``` + +Model images are large. Follow the Pod events while the init container copies model weights into its temporary volume. Allow time for image pulls and model loading; do not interpret an initializing worker as an inference failure. + +```bash +kubectl -n "$NAMESPACE" get events --sort-by=.metadata.creationTimestamp +kubectl -n "$NAMESPACE" get pods \ + -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName,RUNTIME:.spec.runtimeClassName,PHASE:.status.phase' +``` + +Confirm three worker Sandboxes and a separate validator Sandbox. Each must select `gvisor`. Match their node names to the node list from the previous page: backend, frontend, and QA share the C4A node; the manager and validator use their respective N4A nodes. + +Inspect an actual worker rather than relying only on diagram labels: + +```bash +export BACKEND_POD="$(kubectl -n "$NAMESPACE" get pod -l component=worker,role=backend -o jsonpath='{.items[0].metadata.name}')" +kubectl -n "$NAMESPACE" get pod "$BACKEND_POD" \ + -o jsonpath='{.spec.containers[?(@.name=="llama")].args}{"\n"}' +kubectl -n "$NAMESPACE" logs "$BACKEND_POD" -c llama --tail=50 +``` + +Check the model name, four inference threads, context size, and zero GPU layers. Do not paste access tokens or private endpoint details into public logs. + +## Open the dashboard through the manager + +Keep your deployment terminal open so its environment variables remain available. Open a second terminal solely for port-forwarding, set the same namespace there, and forward the regular-pool manager service. The service is internal; the tutorial does not create a public load balancer or ingress. + +```bash +export NAMESPACE="axion-agent-studio" +kubectl -n "$NAMESPACE" port-forward service/studio-manager 8090:8080 --address 127.0.0.1 +``` + +Leave the port-forward running. In a browser, open [Axion Agent Studio](http://127.0.0.1:8090/?view=booth). Return to your original deployment terminal for the remaining commands and inspect the live configuration: + +```bash +curl --fail --silent --show-error http://127.0.0.1:8090/api/config | python3 -m json.tool +``` + +The response must contain `live_available: true` before you submit work. This preflight check confirms service health and the presence of hosted-model configuration. It does not make a Gemini request or prove that credentials and model access work; the planning stage of your first live run checks that access. The separate `/health` endpoint only confirms that the manager process is responding. + +Resolve missing workers, missing hosted-model configuration, or an unreachable validator before continuing. + +## Troubleshoot deployment issues + +| Symptom | Check | +| --- | --- | +| Sandbox admission fails | Installed API version and fields, controller availability, server-side dry-run details | +| Pod remains Pending | Arm labels, gVisor runtime and taints, allocatable CPU/memory, ResourceQuota | +| Image pull fails | Registry path, architecture, node identity's repository access | +| Worker never becomes ready | Init-container events, model checksum and size, llama container logs | +| Hosted planning returns 403 | Manager's Workload Identity principal, model permission, project and location | +| CPU metrics are unavailable | Metrics API and observer permissions; absence of a sample is not zero CPU | +| Browser connection is refused | Port-forward process, local port, manager Pod and Service endpoints | + +Once readiness passes, you have connected the dashboard to the live services. Next, submit a request and inspect the result. diff --git a/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/05-run.md b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/05-run.md new file mode 100644 index 0000000000..062b939705 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/05-run.md @@ -0,0 +1,79 @@ +--- +title: Submit and verify a checkout change +description: Run the supported business request and compare the validated checkout with its original behavior. +weight: 6 +layout: learningpathall +--- + +## Inspect the starting shop + +Open the dashboard's shop preview before starting a run. Change basket quantities and note the original subtotal, shipping, and total. Return to the overview. The prepared shop is the code context for the agents; the prompt does not provide a new repository. + +Paste this request into the request field and select **Run live agents**: + +```text +Apply a 15% discount when the original cart subtotal is $80 or more. Give free shipping when the original subtotal is $120 or more; otherwise charge $7.99 for nonempty carts. Keep empty-cart totals zero. Calculate monetary values in integer cents, rounding the discount down. + +Show the original subtotal, Savings, promotion status, delivery charge and final total using the backend quote. Update these values when basket quantities change, preserving the existing checkout interface and quote integration. + +Verify one cent below, exactly at and one cent above both thresholds, empty carts, multiple products and quantities, and rounding. +``` + +Watch the architecture highlight the active stages. The backend and frontend run together. QA starts after both builds, because it needs their combined result. Validation follows QA. The manager remains responsible for dispatch and status throughout the run, but it does not run the local Gemma models itself. + +Only one workflow runs at a time in this demo. A busy service rejects another submission rather than creating an unbounded queue. + +## Check the result before opening the preview + +Look for a passed run, zero failed checks, and an updated shop link. Inspect the plan and generated-file evidence. The expected outputs are `pricing.py`, the assembled `checkout.html`, and `qa_tests.py`. + +For this policy, the independent validator constructs these monetary checks: + +| Original subtotal | Discount | Shipping | Final total | +| --- | --- | --- | --- | +| Empty cart | $0.00 | $0.00 | $0.00 | +| $79.99 | $0.00 | $7.99 | $87.98 | +| $80.00 | $12.00 | $7.99 | $75.99 | +| $80.01 | $12.00 | $7.99 | $76.00 | +| $119.99 | $17.99 | $7.99 | $109.99 | +| $120.00 | $18.00 | $0.00 | $102.00 | +| $120.01 | $18.00 | $0.00 | $102.01 | + +Both thresholds use the **original** subtotal. The discount calculation uses integer division: `subtotal_cents * 15 // 100`. This rounds down to whole cents. Boundary checks use synthetic catalog items, so you do not need to find a real shop basket worth exactly $79.99. + +The protected oracle is maintained separately from generated pricing code. For the supplied prompt, the recorded run reports 14 passing result groups: 12 monetary cases, one basic checkout HTML contract check, and one supplemental QA test group. That QA group contained two generated test methods in the recorded run. Future generated tests can differ. + +The HTML check verifies a limited contract; it is not an automated browser interaction or accessibility test. Manually confirm the checkout behavior next. + +## Compare the updated shop + +Open the updated shop through the dashboard. Confirm the Savings row, promotion status, delivery charge, and final total. Change quantities so that a basket crosses each threshold. Empty the basket and check that all monetary values return to zero. + +For an original subtotal of $121.00, the expected discount is $18.15, shipping is free, and the final total is $102.85. In the recorded demonstration, the original shop total for that basket was $128.99. + +Use the updated link inside the dashboard's shop view: the preview HTML communicates with its parent through a constrained message interface. Opening the raw HTML route alone is not the complete interactive application. + +## Inspect the run evidence + +Copy the value of `run` from the dashboard URL, then set it in your original deployment terminal. Keep the port-forward terminal running. These calls save evidence for the existing run and do not start another inference request: + +```bash +cd "$LP_WORKDIR" +export RUN_ID="your-run-id" +mkdir -p review-evidence +curl --fail --silent --show-error \ + "http://127.0.0.1:8090/api/runs/$RUN_ID" -o review-evidence/run.json +curl --fail --silent --show-error \ + "http://127.0.0.1:8090/api/runs/$RUN_ID/artifacts" -o review-evidence/artifacts.json +curl --fail --silent --show-error \ + "http://127.0.0.1:8090/api/runs/$RUN_ID/metrics" -o review-evidence/metrics.json +python3 -m json.tool review-evidence/run.json +``` + +Confirm `mode` is `live`, `status` is `passed`, `tests.failed` is zero, and `preview_url` is populated. Inspect the policy values: 15, 8000, 12000, and 799. A generated summary alone is not success evidence. + +If the request is rejected as unsupported, keep it within the existing pricing and display contract. If checks still fail after the repair round, the updated preview stays unavailable. Inspect the recorded failure instead of presenting it as a successful change. + +## What you have verified + +You have checked the generated policy against independent monetary expectations and tested the updated checkout manually. Next, interpret the CPU evidence without confusing workload limits with VM capacity. diff --git a/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/06-measure.md b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/06-measure.md new file mode 100644 index 0000000000..4c793e3cbf --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/06-measure.md @@ -0,0 +1,61 @@ +--- +title: Interpret CPU usage and isolation +description: Read workload CPU samples accurately and identify what the sandbox and validation boundaries do and do not establish. +weight: 7 +layout: learningpathall +--- + +## Read workload CPU usage correctly + +The dashboard presents manager, backend, frontend, and QA workload measurements. These are not four separate VM measurements. The three engineer workloads share one C4A VM, while the manager runs on N4A. + +A core measurement is CPU time consumed over the sample interval. For example, 3.37 cores means the workload consumed CPU time at an average rate equivalent to 3.37 fully occupied vCPUs during that interval. + +The engineer percentage uses the **combined Pod CPU limit**, 6.5 CPUs: + +```text +CPU percentage = measured workload cores / 6.5 × 100 +3.37 cores / 6.5 × 100 = approximately 51.8% +``` + +The denominator includes a 6-CPU model-server limit and a 0.5-CPU worker-service limit. It is not the 16-vCPU machine size and is not the 4.25-CPU scheduling request. + +The inference server is configured with four threads. Four busy inference threads alone correspond to about 61.5% of the combined 6.5-CPU Pod limit. More quota does not automatically create more threads or make token generation scale linearly. + +A lower frontend percentage does not imply that its model is remote or absent. Its constrained fragment task can finish sooner, while model loading, prompt processing, token generation, and waiting create different CPU patterns. Memory bandwidth, sampling windows, and contention also affect utilization. Inspect the stage and timestamps with the measurement. + +| Quantity | Meaning in this demo | +| --- | --- | +| VM vCPUs | Virtual CPU capacity provisioned for the node; some is reserved for system workloads | +| Pod CPU request | CPU capacity Kubernetes uses when placing the Pod; this does not mean dedicated cores | +| Pod CPU limit | Sum of the configured CPU limits across this Pod's workload containers | +| Inference threads | llama.cpp's configured CPU worker parallelism | +| Sampled cores | Observed CPU consumption averaged over a time interval | +| CPU percentage | Sampled cores divided by the declared workload limit, multiplied by 100 | + +For example, 3.9 sampled cores against a 6.5-CPU limit produces 60%. That is 60% of the agent's configured limit, not 60% of the shared 16-vCPU VM. A CPU limit is a ceiling, not a reservation: the three 6.5-CPU limits do not create 19.5 vCPUs on the 16-vCPU node. + +Use the current metrics endpoint for an idle-versus-active comparison: + +```bash +curl --fail --silent --show-error \ + http://127.0.0.1:8090/api/metrics/current | python3 -m json.tool +``` + +Run-specific history remains a record of that run after it finishes. Refreshing the browser does not rerun the models. Check the source and sample time; a missing or stale sample must not be presented as current utilization. + +## Explain the isolation and test boundaries + +Workers and validation run under gVisor, as non-root users, with dropped capabilities and a read-only container root filesystem. Writable temporary volumes are limited. Worker Sandboxes do not receive Kubernetes service-account tokens, and NetworkPolicies restrict their network access. + +The manager handles generated source as data. The separate validator executes it with bounded subprocess and test controls. Protected acceptance checks are not supplied by the model, so a worker cannot pass simply by rewriting those expected results. + +These measures reduce exposure to generated code. They do not establish a complete production security review, correctness for every possible input, or isolation between mutually hostile end users. The demo has one active workflow and no production user authentication layer. + +## Record the run results + +Save the actual node placement, image digests, model checksum, inference settings, policy, tests, and sample timestamps with your run. The saved October 21 run completed in 130.21 seconds with no repair round. That is one observation, not a latency guarantee or comparative benchmark. + +This workflow demonstrates local model execution and application validation on Arm. It does not measure a C4A-versus-N4A speedup, x86 comparison, throughput under concurrent users, or cost savings. Those claims require a separate controlled experiment with comparable quality and completion criteria. + +You have now traced a request through local CPU inference, independent checks, and an updated application. Remove the tutorial resources when you finish collecting evidence. diff --git a/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/07-cleanup.md b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/07-cleanup.md new file mode 100644 index 0000000000..8ca70768a2 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/07-cleanup.md @@ -0,0 +1,40 @@ +--- +title: Save evidence and clean up +description: Export the review results and remove only the resources created for this Learning Path. +weight: 8 +layout: learningpathall +--- + +## Preserve what you need for review + +Export the run, artifacts, and metrics using the commands in [Inspect the run evidence](../05-run/#inspect-the-run-evidence) before deleting workloads. Record any manual checkout checks alongside them. Review evidence for project identifiers, endpoint addresses, account names, and private code before sharing it publicly. + +The manager stores run data in an `emptyDir` volume. The validator keeps accepted candidates in process memory. The model init container copies weights into another temporary volume. A Pod replacement can lose this state; an old dashboard record does not guarantee its candidate quote service is still available. Rerun the workflow if the validated candidate has disappeared. + +## Remove the tutorial workloads + +Return to the terminal holding the port-forward and press **Ctrl+C**. In the deployment terminal, confirm the context and review the namespace contents: + +```bash +cd "$LP_WORKDIR" +kubectl config current-context +kubectl -n "$NAMESPACE" get pods,services,sandboxes.agents.x-k8s.io +``` + +For the dedicated installation created by this tutorial, have the same authorized operator or administrator remove the manifest set, including its cluster-scoped node observer role and binding: + +```bash +kubectl delete -f generated/full.json +``` + +This manifest includes the namespace, so deletion removes everything inside it. Use this command only for the dedicated tutorial installation you checked earlier. If another team has added resources there, coordinate cleanup before deleting the namespace. + +Deleting Pods or a namespace does **not** stop charges for a fixed-size GKE node pool. Have the cluster owner scale down or remove dedicated tutorial pools according to your organization's procedure. Do not stop individual managed VMs in Compute Engine as a substitute for changing their GKE node pool. + +The cluster, shared VPC, registry, and node pools were prerequisites and are not deleted by the application manifests. Remove only tutorial-specific image versions if you no longer need them. If the manager's IAM grant was created solely for this exercise, have its administrator remove that specific principal binding. Keep shared registry, node, and IAM configuration intact. + +## Review the outcome + +You deployed a bounded coding-agent workflow, used local Gemma inference on C4A for three roles, validated generated behavior on N4A, and served the accepted shop through an N4A manager. You also separated VM capacity from Pod quotas and measured usage. + +Keep your run evidence with the configuration and image digests you used. Record any failed or skipped checks so another developer can understand what your run verified. diff --git a/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/_index.md b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/_index.md new file mode 100644 index 0000000000..1e237a97b4 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/_index.md @@ -0,0 +1,64 @@ +--- +title: Run AI coding agents with GKE Agent Sandbox on Google Axion +description: Use hosted Gemini planning and local Gemma inference on Arm-based Axion CPUs to implement, test, and preview an e-commerce checkout change. +lastmod: 2026-11-06 +minutes_to_complete: 120 +who_is_this_for: This is an advanced topic for developers familiar with Python, containers, and Kubernetes who want to run and validate a bounded coding-agent workflow on Arm-based cloud infrastructure. +learning_objectives: + - Deploy a manager, three local model workers, and an isolated validator on an existing compatible GKE cluster. + - Submit a checkout change and inspect the plan, generated files, and independent test results. + - Verify Arm VM placement and distinguish workload CPU usage from VM capacity. + - Compare the original and validated checkout, then remove the tutorial workloads. +prerequisites: + - An existing GKE Standard cluster with Workload Identity Federation, enforced NetworkPolicy, and the compatible Agent Sandbox API described in Prepare your environment + - Available C4A and N4A node capacity, Artifact Registry access, and a Google Cloud project with billing and Vertex AI model access + - A Linux or macOS workstation with Bash, Python 3.12, Google Cloud CLI, the GKE authentication plugin, kubectl, and Docker Buildx + - Permission to deploy namespaced Kubernetes resources and an administrator who can configure node pools and the required IAM binding + - Access to the companion application source after its public release and licensing are approved, plus permission to use the model and application code +author: Rani Chowdary Mandepudi +generate_summary_faq: true +rerun_summary: false +rerun_faqs: false +skilllevels: Advanced +subjects: Containers and Virtualization +armips: + - Neoverse +platforms: + - Google Axion +tools_software_languages: + - Google Kubernetes Engine + - gVisor + - Python + - Gemma + - llama.cpp +operatingsystems: + - Linux + - macOS +further_reading: + - resource: + title: Install Agent Sandbox on GKE + link: https://docs.cloud.google.com/kubernetes-engine/docs/how-to/how-install-agent-sandbox + type: documentation + - resource: + title: GKE Sandbox isolation model + link: https://docs.cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods + type: documentation + - resource: + title: Arm workloads on GKE + link: https://docs.cloud.google.com/kubernetes-engine/docs/concepts/arm-on-gke + type: documentation + - resource: + title: Authenticate to Google Cloud APIs from GKE workloads + link: https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity + type: documentation + - resource: + title: Gemma E4B GGUF model files at the recorded revision + link: https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/tree/b8093469224f83f5c38f691eb906c380e9e63114 + type: website +weight: 1 +layout: learningpathall +learning_path_main_page: "yes" +draft: true +cascade: + draft: true +--- diff --git a/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/_next-steps.md b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/_next-steps.md new file mode 100644 index 0000000000..727b395ddd --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # The weight controls the order of the pages. _index.md always has weight 1. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/images/architecture-technical.png b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/images/architecture-technical.png new file mode 100644 index 0000000000..0459867958 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/images/architecture-technical.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/images/architecture-technical.svg b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/images/architecture-technical.svg new file mode 100644 index 0000000000..06b851811e --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/axion-agent-studio/images/architecture-technical.svg @@ -0,0 +1,200 @@ + +Axion Agent Studio — deployment and execution architecture +A browser submits a request to a regular N4A manager Pod. Hosted Gemini plans and summarizes work through Vertex AI. The manager dispatches to three gVisor sandbox Pods on one shared C4A virtual machine, each containing a worker API and local Gemma model server. Backend and frontend build in parallel, then QA generates tests. A separate N4A validator executes protected checks and generated tests, and serves accepted pricing calculations. Only passed validation enables the shop preview served by the manager. A separate C4A system node is shown. VM sizes describe provisioned capacity, not Pod limits. + + + + + + + + + +DEPLOYMENT + EXECUTION FLOW +Axion Agent Studio +HOSTED PLANNING +LOCAL CPU EXECUTION +INDEPENDENT VALIDATION + + +01 +Developer / browser +Submit request · inspect evidence +Local access through kubectl port-forward + + +02 +Gemini · Vertex AI +Plan tasks · summarize results + +Hosted inference · outside workload VMs + + +GKE STANDARD +Google Axion · Arm workload nodes + +4 VMs · 28 provisioned vCPUs + + + + + + + + + + + + + + + +VM 01 · N4A +n4a-standard-2 · 2 vCPUs + +Agent manager + +FASTAPI · REGULAR POD + +Coordinate +Plan · dispatch · merge +Prepared shop repository +Run control +One active request +One bounded repair round +Observability +Status · CPU · memory +VM placement + test evidence + + + +05 +Validated shop +Enabled after checks pass +HTML served by manager +Pricing runs in the validator + + + + + + + + + + + + + + + +VM 02 · C4A +c4a-standard-16 · 16 vCPUs shared by all three engineers + +LOCAL CPU MODELS + + + + + +03 Tasks + scoped code ⇄ changes / tests · HTTP :8081 + + +Backend Engineer +gVisor sandbox Pod + +Worker API · :8081 + + +llama.cpp · local Gemma +E4B · Q4_0 · CPU inference +pricing.py +Builds pricing and delivery logic + + +Frontend Engineer +gVisor sandbox Pod + +Worker API · :8081 + + +llama.cpp · local Gemma +E4B · Q4_0 · CPU inference +checkout.html +Updates constrained checkout fragments + + +QA Engineer +gVisor sandbox Pod + +Worker API · :8081 + + +llama.cpp · local Gemma +E4B · Q4_0 · CPU inference +qa_tests.py +Generates pricing unit tests +Backend + frontend in parallel +QA follows both builds +Candidate / test results / accepted pricing quotes + + + + + + + + + + + + + + + +VM 03 · N4A +n4a-standard-8 · 8 vCPUs + + +04 +Independent validator +Protected checks + generated tests · gVisor Pod +Executes candidate Python and accepted pricing quotes + + + + + + + + + + + + + + + + +VM 04 · C4A +c4a-standard-2 +2 vCPUs +GKE system workloads +Separate from agent work + + +Request + dashboard / shop response + + +HTTPS · Workload Identity Federation +CPU capacity belongs to VMs. Each engineer Pod requests 4.25 CPUs, has a 6.5-CPU limit, and runs 4 inference threads. + + +VM boundary + +gVisor sandbox Pod + +Service calls + responses +Gemini plans. Arm executes. Independent checks gate the preview. +Recorded deployment topology · GKE control plane managed separately · No local GPU inference + \ No newline at end of file