tungbq · tungbq · Aug 30, 2026 · Aug 30, 2026
@@ -0,0 +1,21 @@
+name: Verify microservices-orchestration
+
+on:
+  push:
+    branches: ['main']
+    paths:
+      - 'projects/microservices-orchestration/**'
+  pull_request:
+    branches: ['main']
+    paths:
+      - 'projects/microservices-orchestration/**'
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - name: Build, deploy to kind, and verify the order flow
+        run: |
+          cd projects/microservices-orchestration
+          chmod +x demo_project.sh
+          ./demo_project.sh
@@ -37,7 +37,7 @@ This is the **third** repo of my DevOps trio repositories: [**tungbq/devops-basi
 | 16  | CI/CD pipeline for microservices on Kubernetes | [#71](https://github.com/tungbq/devops-project/issues/71)                                             | `CI/CD` `Kubernetes` `Microservices`                   | 🚧 Planned  |
 | 17  | Configuration Management with Ansible          | [#14](https://github.com/tungbq/devops-project/issues/14)                                             | `Ansible` `Configuration Management`                   | 🚧 Planned  |
 | 18  | 3 tier app with k8s                            | [#9](https://github.com/tungbq/devops-project/issues/9)                                               | `Kubernetes` `Microservices`                           | 🚧 Planned  |
-| 19  | Microservices Orchestration                    | [#8](https://github.com/tungbq/devops-project/issues/8)                                               | `Kubernetes` `Microservices`                           | 🚧 Planned  |
+| 19  | Microservices Orchestration                    | [microservices-orchestration](./projects/microservices-orchestration/)                               | `Kubernetes` `Microservices` `Scaling`                 | ✔️ Done     |
 | 20  | Deploy Kubernetes using Kubespray              | [#70](https://github.com/tungbq/devops-project/issues/70)                                             | `Kubernetes` `Kubespray`                               | 🚧 Planned  |
 | 21  | Deploy a static website to AWS S3              | [#6](https://github.com/tungbq/devops-project/issues/6)                                               | `AWS` `S3` `Static Website`                            | 🚧 Planned  |
 

@@ -0,0 +1,148 @@
+# Project: Microservices Orchestration
+
+Build a small microservices-based application and use Kubernetes to orchestrate it — service discovery, config management, health probes, and scaling.
+
+## Overview
+
+### Introduction
+
+- Tech stack: `Python` (FastAPI), `Docker`, `Kubernetes` (via [kind](https://kind.sigs.k8s.io/))
+- Three services, each independently deployable and independently scalable:
+  - **order-service** — the only externally-exposed service; accepts an order, then calls the other two.
+  - **inventory-service** — reserves stock for an item.
+  - **notification-service** — "sends" a confirmation (logs it in-memory; this is a demo, not a real notifier).
+- The point of this project isn't the business logic (it's deliberately trivial) — it's what Kubernetes does *around* the three services: **service discovery** (order-service never hardcodes an IP), **config as data** (a ConfigMap, not code, tells order-service where its dependencies live), **health probes** (readiness/liveness), and **horizontal scaling** (`kubectl scale` and a real HPA).
+- To gain a basic understanding of Kubernetes, you could visit: [**Kubernetes**](https://kubernetes.io/) and [**devops-basics/kubernetes**](https://github.com/tungbq/devops-basics)
+
+### Architecture
+
+```
+                     ┌─────────────────────┐
+   client ──POST──▶  │   order-service      │  (NodePort, replicas: 2)
+                     │   (the only one       │
+                     │    exposed outside)   │
+                     └─────────┬─────────────┘
+                     ConfigMap  │  (INVENTORY_URL, NOTIFICATION_URL —
+                     injects ──▶│   K8s DNS short names, not IPs)
+                               ┌┴──────────────┐
+                     ┌─────────┴───┐    ┌───────┴──────────┐
+                     │ inventory-   │    │ notification-    │
+                     │ service      │    │ service           │
+                     │ (ClusterIP)  │    │ (ClusterIP)       │
+                     └──────────────┘    └───────────────────┘
+```
+
+### Prerequisite
+
+- Basic knowledge about Docker and Kubernetes (Deployments, Services, ConfigMaps)
+- Tools: `docker`, `curl`. `kind` and `kubectl` are auto-installed by the demo script if missing (see [`demo_project.sh`](./demo_project.sh)).
+
+## 1-Run the whole thing
+
+The fastest way to see it work end-to-end (build → local kind cluster → deploy → verify → tear down):
+
+```bash
+cd projects/microservices-orchestration
+./demo_project.sh
+```
+
+This is the exact script [`.github/workflows/verify-microservices-orchestration.yml`](../../.github/workflows/verify-microservices-orchestration.yml) runs on every push/PR that touches this project — verified in CI, not just locally.
+
+## 2-Run it step by step (to actually poke around)
+
+### 2.1-Build the images
+
+```bash
+docker build -t inventory-service:local ./inventory-service
+docker build -t notification-service:local ./notification-service
+docker build -t order-service:local ./order-service
+```
+
+### 2.2-Create a local cluster and load the images into it
+
+`kind` clusters can't `docker pull` your locally-built images — there's no registry — so they're loaded directly into the cluster's node instead.
+
+```bash
+kind create cluster --name microservices-demo --config kind-config.yaml
+kind load docker-image inventory-service:local notification-service:local order-service:local --name microservices-demo
+```
+
+### 2.3-Deploy
+
+```bash
+kubectl apply -f k8s/namespace.yaml
+kubectl apply -f k8s/configmap.yaml
+kubectl apply -f k8s/inventory-service.yaml
+kubectl apply -f k8s/notification-service.yaml
+kubectl apply -f k8s/order-service.yaml
+```
+
+### 2.4-Place an order
+
+`kind-config.yaml` maps the cluster's NodePort 30080 to `localhost:8080`:
+
+```bash
+curl -X POST http://localhost:8080/orders \
+  -H "Content-Type: application/json" \
+  -d '{"customer": "you@example.com", "item": "widget", "quantity": 2}'
+```
+
+The response's `"pod"` fields show three *different* pod names — proof the request actually crossed process/network boundaries via Kubernetes Services, not an in-process function call:
+
+```json
+{
+  "pod": "order-service-...",
+  "orderId": "...",
+  "status": "confirmed",
+  "reservation": { "pod": "inventory-service-...", "item": "widget", "reserved": 2, "remaining": 8 },
+  "notification": { "pod": "notification-service-...", "to": "you@example.com", "...": "..." }
+}
+```
+
+## 3-Scaling
+
+### 3.1-Manual scaling
+
+```bash
+kubectl -n microservices-demo scale deployment order-service --replicas=4
+kubectl -n microservices-demo get pods -l app=order-service
+```
+
+Fire a few orders and watch the `"pod"` field in the response rotate across replicas — the Service load-balances across whichever pods are `Ready`:
+
+```bash
+for i in $(seq 1 6); do
+  curl -s -X POST http://localhost:8080/orders \
+    -H "Content-Type: application/json" \
+    -d '{"customer":"x@test.com","item":"widget","quantity":1}' \
+    | grep -o '"pod":"[a-z0-9-]*"' | head -1
+done
+```
+
+### 3.2-Autoscaling (HPA)
+
+`kind` doesn't ship [metrics-server](https://github.com/kubernetes-sigs/metrics-server) by default — the HPA (`k8s/order-service-hpa.yaml`) exists in the cluster either way, it just has no CPU metric to scale on until metrics-server is installed. `kind`'s kubelet also uses a self-signed cert metrics-server rejects by default, hence the `--kubelet-insecure-tls` patch:
+
+```bash
+kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
+kubectl -n kube-system patch deployment metrics-server --type=json \
+  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
+kubectl -n kube-system rollout status deployment/metrics-server
+
+kubectl apply -f k8s/order-service-hpa.yaml
+kubectl -n microservices-demo get hpa order-service --watch
+```
+
+Verified working: `kubectl top pods` and `kubectl describe hpa order-service` both report real CPU numbers a few seconds after metrics-server comes up (not "unknown"). Generating enough load to actually trigger a scale-up (sustained >50% CPU for a few minutes) is left as an exercise — a simple loop of curl requests in a `while true` won't generate meaningful CPU load against an app this trivial; a proper load generator (`hey`, `k6`, or similar) is the honest way to do it.
+
+## 4-Tear down
+
+```bash
+kind delete cluster --name microservices-demo
+```
+
+## What this project deliberately leaves out
+
+- **A real database.** Stock/notifications are in-memory and reset on pod restart — adding Postgres here would teach StatefulSets/PVCs, which is a different (and equally valid) lesson from "how do three stateless services find and talk to each other."
+- **A service mesh (Istio/Linkerd).** This repo already has [aks-istio-application](../aks-istio-application/) and [aks-nginx-with-istio](../aks-nginx-with-istio/) for that. This project is the "plain Kubernetes" baseline those build on top of.
+- **A real message queue.** order-service calls notification-service synchronously and just swallows the error if it fails (see the code comment in `order-service/app.py`) — a real system would use an outbox pattern or a queue. That's a distinct lesson from orchestration basics.
@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+# Builds all three services, spins up a local kind cluster, deploys the
+# manifests, and verifies the whole order -> inventory -> notification
+# chain actually works end-to-end. Safe to re-run — always starts from a
+# fresh cluster. Used both for local hands-on runs and by
+# .github/workflows/verify-microservices-orchestration.yml.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "$SCRIPT_DIR"
+
+CLUSTER_NAME="microservices-demo"
+NS="microservices-demo"
+KIND_VERSION="v0.30.0"
+KUBECTL_VERSION="v1.34.0"
+
+install_if_missing() {
+  local bin="$1" url="$2"
+  if command -v "$bin" >/dev/null 2>&1; then
+    return
+  fi
+  echo "Installing $bin..."
+  arch="$(uname -m)"
+  case "$arch" in
+    x86_64) arch="amd64" ;;
+    aarch64|arm64) arch="arm64" ;;
+    *) echo "Unsupported architecture: $arch" >&2; exit 1 ;;
+  esac
+  curl -sL "${url//ARCH/$arch}" -o "/tmp/$bin"
+  chmod +x "/tmp/$bin"
+  sudo mv "/tmp/$bin" "/usr/local/bin/$bin"
+}
+
+install_if_missing kind "https://kind.sigs.k8s.io/dl/${KIND_VERSION}/kind-linux-ARCH"
+install_if_missing kubectl "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/ARCH/kubectl"
+
+echo "==> Building images"
+docker build -t inventory-service:local ./inventory-service
+docker build -t notification-service:local ./notification-service
+docker build -t order-service:local ./order-service
+
+echo "==> Creating kind cluster ($CLUSTER_NAME)"
+kind delete cluster --name "$CLUSTER_NAME" >/dev/null 2>&1 || true
+kind create cluster --name "$CLUSTER_NAME" --config kind-config.yaml --wait 90s
+
+echo "==> Loading images into the cluster (no registry needed for a local demo)"
+kind load docker-image inventory-service:local notification-service:local order-service:local --name "$CLUSTER_NAME"
+
+echo "==> Applying manifests"
+kubectl apply -f k8s/namespace.yaml
+kubectl apply -f k8s/configmap.yaml
+kubectl apply -f k8s/inventory-service.yaml
+kubectl apply -f k8s/notification-service.yaml
+kubectl apply -f k8s/order-service.yaml
+
+echo "==> Waiting for rollouts"
+kubectl -n "$NS" rollout status deployment/inventory-service --timeout=120s
+kubectl -n "$NS" rollout status deployment/notification-service --timeout=120s
+kubectl -n "$NS" rollout status deployment/order-service --timeout=120s
+
+echo "==> Verifying the orchestrated order flow via the NodePort (localhost:8080)"
+for i in $(seq 1 15); do
+  if curl -sf http://localhost:8080/healthz >/dev/null 2>&1; then break; fi
+  sleep 2
+done
+
+response=$(curl -sf -X POST http://localhost:8080/orders \
+  -H "Content-Type: application/json" \
+  -d '{"customer": "demo@example.com", "item": "widget", "quantity": 2}')
+echo "$response"
+
+status=$(echo "$response" | grep -o '"status":"[a-z_]*"')
+if [[ "$status" != '"status":"confirmed"' ]]; then
+  echo "FAIL: order was not confirmed — got: $status" >&2
+  exit 1
+fi
+echo "==> Order confirmed — order-service reached inventory-service and notification-service via K8s service discovery."
+
+echo "==> Cleaning up"
+kind delete cluster --name "$CLUSTER_NAME"
+
+echo "==> Done."
@@ -0,0 +1,18 @@
+# Use the official Python image from the Docker Hub
+FROM python:3.12-slim
+
+# Set the working directory in the container
+WORKDIR /app
+
+# Install dependencies first so this layer is cached across source changes
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy the current directory contents into the container at /app
+COPY . /app
+
+# Expose the port the app runs on
+EXPOSE 8000
+
+# Command to run the FastAPI app
+CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
@@ -0,0 +1,45 @@
+import os
+import socket
+
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel
+
+app = FastAPI(title="inventory-service")
+
+POD_NAME = os.environ.get("POD_NAME", socket.gethostname())
+
+# In-memory stock — a demo service, not a real inventory system. Reset on
+# every pod restart on purpose, so scaling replicas visibly shows each pod
+# starting from the same seed stock (see the repo's HPA/scaling walkthrough).
+STOCK = {"widget": 10, "gadget": 5, "gizmo": 0}
+
+
+class ReserveRequest(BaseModel):
+    item: str
+    quantity: int = 1
+
+
+@app.get("/healthz")
+def healthz():
+    return {"status": "ok"}
+
+
+@app.get("/readyz")
+def readyz():
+    return {"status": "ready"}
+
+
+@app.get("/stock")
+def stock():
+    return {"pod": POD_NAME, "stock": STOCK}
+
+
+@app.post("/reserve")
+def reserve(req: ReserveRequest):
+    available = STOCK.get(req.item)
+    if available is None:
+        raise HTTPException(status_code=404, detail=f"unknown item '{req.item}'")
+    if available < req.quantity:
+        raise HTTPException(status_code=409, detail=f"insufficient stock for '{req.item}'")
+    STOCK[req.item] -= req.quantity
+    return {"pod": POD_NAME, "item": req.item, "reserved": req.quantity, "remaining": STOCK[req.item]}
@@ -0,0 +1,2 @@
+fastapi>=0.115
+uvicorn[standard]>=0.30
@@ -0,0 +1,11 @@
+# order-service's downstream URLs, kept out of the Deployment spec so they
+# can be changed (e.g. pointed at a different namespace) without rebuilding
+# or re-templating the Deployment itself.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: order-service-config
+  namespace: microservices-demo
+data:
+  INVENTORY_URL: "http://inventory-service:8000"
+  NOTIFICATION_URL: "http://notification-service:8000"
@@ -0,0 +1,48 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: inventory-service
+  namespace: microservices-demo
+  labels:
+    app: inventory-service
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: inventory-service
+  template:
+    metadata:
+      labels:
+        app: inventory-service
+    spec:
+      containers:
+        - name: inventory-service
+          image: inventory-service:local
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 8000
+          env:
+            - name: POD_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.name
+          readinessProbe:
+            httpGet: { path: /readyz, port: 8000 }
+            initialDelaySeconds: 2
+            periodSeconds: 5
+          livenessProbe:
+            httpGet: { path: /healthz, port: 8000 }
+            initialDelaySeconds: 5
+            periodSeconds: 10
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: inventory-service
+  namespace: microservices-demo
+spec:
+  selector:
+    app: inventory-service
+  ports:
+    - port: 8000
+      targetPort: 8000
@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: microservices-demo