Kotlin GitOps DevSecOps Platform

A portfolio-grade DevSecOps platform built around a deliberately simple Kotlin Spring Boot API. The business logic stays small on purpose — the value is in the surrounding infrastructure: containerization, CI/CD, security scanning, observability, Kubernetes, Helm, TLS, encrypted secrets, and GitOps.

Repository: github.com/Teapotka/devsecops
Container image: ghcr.io/teapotka/kotlin-gitops-devsecops-platform:latest

Architecture overview

Platform layers

flowchart TB
    subgraph dev["Developer workflow"]
        Code["Kotlin / Spring Boot API"]
        Git["Git push to main"]
    end

    subgraph cicd["GitHub Actions"]
        CI["CI: test + build"]
        Docker["Docker Publish → GHCR"]
        Sec["Security: Trivy + Snyk"]
        Sonar["SonarCloud + JaCoCo"]
    end

    subgraph runtime["Runtime targets"]
        Compose["Docker Compose\n(local full stack)"]
        K8s["Kubernetes (kind)\n+ Helm + Argo CD"]
    end

    subgraph data["Data layer"]
        PG[(PostgreSQL)]
        Redis[(Redis cache)]
    end

    subgraph obs["Observability"]
        Prom["Prometheus"]
        Loki["Loki"]
        Tempo["Tempo"]
        Grafana["Grafana"]
        OTel["OTel Collector"]
    end

    Code --> Git
    Git --> CI & Docker & Sec & Sonar
    Docker --> Compose & K8s
    Compose & K8s --> PG & Redis
    Compose & K8s --> OTel --> Tempo
    Compose & K8s --> Prom
    Compose --> Loki
    Prom & Loki & Tempo --> Grafana

Application request flow

sequenceDiagram
    participant Client
    participant Controller
    participant Service
    participant Cache as Redis cache
    participant Repo as JPA repository
    participant DB as PostgreSQL

    Client->>Controller: HTTP request
    Controller->>Service: validated DTO
  alt stats endpoint (GET /api/stats/orders)
        Service->>Cache: lookup order-stats
        alt cache hit
            Cache-->>Service: cached stats
        else cache miss
            Service->>Repo: aggregate queries
            Repo->>DB: SQL
            DB-->>Service: results
            Service->>Cache: store stats
        end
    else user/order CRUD
        Service->>Repo: persist / query
        Repo->>DB: SQL
        Service->>Cache: invalidate stats (on writes)
    end
    Service-->>Controller: DTO response
    Controller-->>Client: JSON

Observability correlation

flowchart LR
    App["Spring Boot API\n(JSON logs + Micrometer)"]
    Agent["OTel Java Agent"]
    Collector["OTel Collector"]
    Tempo["Tempo"]
    Loki["Loki"]
    Prom["Prometheus"]
    Grafana["Grafana\n(logs ↔ traces ↔ metrics)"]

    App -->|OTLP traces| Agent --> Collector --> Tempo
    App -->|/actuator/prometheus| Prom
    App -->|Docker Loki driver| Loki
    Tempo & Loki & Prom --> Grafana

GitOps deployment loop

flowchart LR
    Dev["Developer"] -->|push| Repo["GitHub repo\n(Helm chart + encrypted secrets)"]
    Repo -->|webhook / poll| Argo["Argo CD"]
    Argo -->|helm template + apply| Cluster["kind cluster\ndevsecops namespace"]
    GHCR["GHCR image"] -->|pull| Cluster

Technology stack

Layer	Technologies
Language / runtime	Kotlin 2.2, Java 21, Spring Boot 4.0.6
API	Spring WebMVC, Bean Validation, global exception handler
Persistence	Spring Data JPA, Hibernate, Flyway, PostgreSQL 17
Cache	Spring Cache, Spring Data Redis 7.4
Metrics	Spring Actuator, Micrometer, Prometheus
Logging	SLF4J, Logstash JSON encoder (non-local profiles)
Tracing	OpenTelemetry Java agent, OTel Collector, Grafana Tempo
Containers	Docker multi-stage build, Docker Compose
CI/CD	GitHub Actions, GHCR
Security	Trivy, Snyk, SonarCloud, JaCoCo
Orchestration	Kubernetes (kind), Helm 3
Ingress / TLS	ingress-nginx, cert-manager (self-signed for local)
Secrets	SOPS + age
GitOps	Argo CD

API reference

Business endpoints expose only users, orders, and aggregated stats.

Users

Method	Path	Description
`POST`	`/api/users`	Create user
`GET`	`/api/users`	List users
`GET`	`/api/users/{id}`	Get user by ID
`DELETE`	`/api/users/{id}`	Delete user

Orders

Method	Path	Description
`POST`	`/api/orders`	Create order for a user
`GET`	`/api/orders`	List orders
`GET`	`/api/orders/{id}`	Get order by ID
`GET`	`/api/orders/user/{userId}`	List orders for a user
`PATCH`	`/api/orders/{id}/status`	Update order status
`DELETE`	`/api/orders/{id}`	Delete order

Order statuses: CREATED, PAID, CANCELLED, SHIPPED.

Stats (Redis-backed)

Method	Path	Description
`GET`	`/api/stats/orders`	Aggregated order statistics (`cached: true/false` in response)
`DELETE`	`/api/stats/cache`	Manually clear stats cache

Actuator

Method	Path	Description
`GET`	`/actuator/health`	Health (used by Docker/K8s probes)
`GET`	`/actuator/prometheus`	Prometheus scrape endpoint
`GET`	`/actuator/metrics`	Application metrics
`GET`	`/actuator/info`	Build metadata

Build phases

Work progressed in deliberate phases: establish a correct API, prove it with tests, then wrap it in progressively more production-like infrastructure.

Phase	Topic	Status	What was built
1	Backend API foundation	✅ Done	Spring Boot app, Flyway migrations (`user_accounts`, `orders`), JPA entities, DTOs, services, controllers, validation, global error handler, Actuator + Prometheus
2	Clean API & real observability signals	✅ Done	Removed demo/app-info controllers; structured SLF4J logging in services and exception handler; JSON logs via `logstash-logback-encoder` for container profiles
3	Test suite	✅ Done	13 test classes — MockK unit tests, `@WebMvcTest` controllers, Testcontainers integration tests, JaCoCo coverage
4	Redis cache & stats	✅ Done	`CacheConfig`, `StatsService`, `StatsController`; cache invalidation on order create/update/delete
5	Docker	✅ Done	Multi-stage `Dockerfile` (Gradle build → JRE runtime, non-root user), `.dockerignore`, `docker-compose.yml` for app + Postgres + Redis
6	CI/CD foundation	✅ Done	`ci.yml` (test + build), `docker-publish.yml` (build & push to GHCR on `main`)
7	Security scanning	✅ Done	`security.yml` (Trivy fs + image, Snyk deps), `sonarcloud.yml` (SonarCloud + JaCoCo)
8	Local observability	✅ Done	Prometheus, Grafana (provisioned datasources + dashboard), Loki; Loki Docker logging driver on the `app` service
9	Distributed tracing	✅ Done	OTel Java agent in image, OTel Collector, Tempo; Grafana trace-to-log correlation
10	Kubernetes manifests	✅ Done	`k8s/` namespace, app, Postgres (PVC), Redis; tested on kind cluster `devsecops`
11	Helm chart	✅ Done	`helm/devsecops-api/` — parameterized deployment for app, Postgres, Redis, ingress, secrets toggle
12	Ingress & TLS	✅ Done	ingress-nginx on kind, `api.localhost`, cert-manager self-signed `ClusterIssuer` + `Certificate`
13	Secrets management	✅ Done	SOPS + age; `k8s/secrets/devsecops-secret.enc.yaml` in Git; plain `secret.yaml` manifests removed
14	GitOps	✅ Done	Argo CD `Application` → `helm/devsecops-api` on `main`, automated sync with prune/self-heal

Design decisions and efficiency

Keep the domain small, invest in the platform

A complex order-management system would obscure the DevOps story. Two entities and a stats endpoint are enough to exercise validation, relationships, aggregation queries, caching, logging, and tracing — without becoming a second project.

DTOs and Actuator separation

Controllers return DTOs, never JPA entities. Health, metrics, and infra checks use Actuator, not fake /api/health routes. This mirrors how production Spring services are structured.

Redis cache with explicit invalidation

GET /api/stats/orders runs several aggregate queries (counts by status, revenue sum). Caching the result in Redis avoids repeated full-table scans under read load.

Efficiency pattern:

Read path: cache-aside — check Redis first, compute on miss, store result.
Write path: invalidate on POST, PATCH, and DELETE orders so stats never serve stale data.
Trade-off: slightly more write latency (one cache delete) in exchange for predictable read performance.

Multi-stage Docker build

The Dockerfile separates compile and runtime:

Build stage — Gradle with layer cache mount (--mount=type=cache) speeds rebuilds.
Runtime stage — slim JRE-only image, non-root appuser, OTel agent pre-downloaded.

Result: smaller attack surface, faster CI image builds (GHA cache-from / cache-to), and tracing enabled without runtime downloads.

Loki Docker logging driver (Option B)

Instead of Promtail sidecars, the Compose app service uses the Loki Docker logging driver with JSON pipeline stages. This keeps the logging path simple for local demos:

App emits structured JSON (level, message, traceId).
Driver pushes directly to Loki.
Grafana derives trace links from traceId in log lines.

kind over managed cloud

Local kind cluster with ingress-ready node labels and host port mappings (80, 443) provides a zero-cost Kubernetes environment that behaves like a real cluster — without cloud billing or network complexity.

Helm as the deployment contract

Raw k8s/ manifests prove the resources work; the Helm chart parameterizes image, resources, ingress, and feature flags (postgres.enabled, secrets.create: false). Argo CD deploys the chart, not hand-applied YAML — one source of truth.

SOPS for secrets in Git

Database passwords must not live as plaintext in Git. SOPS + age encrypts Kubernetes Secret manifests at rest in the repository. Only .enc.yaml files are committed; decryption keys stay in .secrets/ (gitignored).

GitOps pull model

Argo CD watches main and reconciles cluster state. Deployments are not pushed via kubectl from CI — CI builds and publishes the image; GitOps publishes the desired state. This separates build automation from release automation.

CI parallelism

Four independent workflows run on push:

flowchart TD
    Push["git push"] --> CI["CI\n./gradlew test + build"]
    Push --> Docker["Docker Publish\nbootJar + push GHCR"]
    Push --> Sec["Security\nTrivy fs + image, Snyk"]
    Push --> Sonar["SonarCloud\nJaCoCo coverage report"]

Failures are isolated per concern; Docker publish does not block security informational scans (image Trivy uses exit-code: 0 for reporting).

Repository layout

devsecops/
├── src/main/kotlin/com/demo/devsecops/   # Application code
├── src/test/kotlin/                       # Unit, web, integration tests
├── Dockerfile / docker-compose.yml
├── helm/devsecops-api/                    # Helm chart (GitOps target)
├── k8s/                                   # Raw manifests + cert-manager + encrypted secrets
├── kind/cluster-config.yaml               # kind cluster with ingress ports
├── gitops/argocd/application.yaml         # Argo CD Application
├── observability/                         # Prometheus, Grafana, Loki, Tempo, OTel Collector
└── .github/workflows/                     # ci, docker-publish, security, sonarcloud

Quick start

Prerequisites

JDK 21
Docker Desktop (or Docker Engine + Compose)
Optional for Kubernetes path: kind, kubectl, helm, argocd CLI, sops, age

Local development (IDE)

Start dependencies only:

docker compose up -d postgres redis

Run the app (plain text logs):

./gradlew bootRun --args='--spring.profiles.active=local'

Service	URL
API	http://localhost:8080
PostgreSQL	localhost:5433
Redis	localhost:6379

Default credentials (local demo only): devsecops_user / devsecops_password, database devsecops.

Smoke test

curl -s http://localhost:8080/actuator/health | jq .
curl -s -X POST http://localhost:8080/api/users \
  -H 'Content-Type: application/json' \
  -d '{"email":"demo@example.com","password":"secret123"}'

Docker Compose (full stack)

Runs the API plus the entire observability stack:

docker compose up --build

Service	Port	Purpose
app	8080	Spring Boot API
postgres	5433	Database
redis	6379	Cache
prometheus	9090	Metrics
grafana	3000	Dashboards (admin / admin)
loki	3100	Log aggregation
tempo	3200	Trace storage
otel-collector	4317, 4318	OTLP receiver

After startup:

Open Grafana → Explore → Loki → query {service="devsecops-api"}.
Generate traffic with user/order API calls; confirm JSON logs and trace IDs.
Open Tempo datasource → search traces for devsecops-api.

Kubernetes with kind

Create a cluster with ingress port mappings:

kind create cluster --name devsecops --config kind/cluster-config.yaml

Apply raw manifests (without Helm):

# Decrypt secrets first (see SOPS section), then:
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/secrets/
kubectl apply -f k8s/postgres/
kubectl apply -f k8s/redis/
kubectl apply -f k8s/app/
kubectl apply -f k8s/cert-manager/

kubectl get pods -n devsecops
kubectl port-forward svc/devsecops-api 8080:8080 -n devsecops

Production note: the bundled Postgres and Redis are single-pod demos. In production, use managed databases or operators.

Helm

helm upgrade --install devsecops-api ./helm/devsecops-api \
  -n devsecops --create-namespace

Key values in helm/devsecops-api/values.yaml:

image.repository / image.tag — GHCR image
ingress.enabled — api.localhost with TLS
secrets.create: false — expects pre-provisioned SOPS-decrypted secret in cluster

helm upgrade devsecops-api ./helm/devsecops-api -n devsecops
helm uninstall devsecops-api -n devsecops

Ingress and TLS

Install ingress-nginx on kind (control-plane node labeled ingress-ready=true).
Install cert-manager.
Apply k8s/cert-manager/cluster-issuer.yaml and certificate.yaml.

kubectl get ingress -n devsecops
kubectl get certificate -n devsecops
curl -k https://api.localhost/actuator/health

TLS uses a self-signed issuer for local development. For a public VPS, swap the ClusterIssuer to Let's Encrypt (Phase 15).

Secrets management (SOPS)

Encrypted secrets are stored at k8s/secrets/devsecops-secret.enc.yaml. Configuration: .sops.yaml (age public key).

# One-time: generate age key (keep out of Git)
mkdir -p .secrets
age-keygen -o .secrets/age-key.txt

# Decrypt for local apply
export SOPS_AGE_KEY_FILE=.secrets/age-key.txt
sops --decrypt k8s/secrets/devsecops-secret.enc.yaml > k8s/secrets/devsecops-secret.dec.yaml

kubectl apply -f k8s/secrets/devsecops-secret.dec.yaml

Git ignores .secrets/, *.age-key.txt, and *.dec.yaml. Never commit decrypted secrets.

GitOps with Argo CD

gitops/argocd/application.yaml defines an Argo CD Application that:

Tracks main on this repository
Deploys helm/devsecops-api into namespace devsecops
Enables automated sync with prune and self-heal

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl apply -f gitops/argocd/application.yaml

kubectl get applications -n argocd

Argo CD requires the Helm chart and encrypted secrets to exist on the remote main branch. After git push, the Application should report Synced / Healthy.

CI/CD and security

Workflow	Trigger	Purpose
`ci.yml`	push, PR	`./gradlew clean test` + `build` (Redis service for cache tests)
`docker-publish.yml`	push to `main`, PR, manual	Build JAR, build image, push to GHCR
`security.yml`	push, PR, manual	Trivy filesystem + image scans (SARIF → GitHub Security), Snyk dependency scan
`sonarcloud.yml`	push, PR, manual	Tests, JaCoCo report, SonarCloud analysis

Required GitHub secrets: SONAR_TOKEN, SNYK_TOKEN.

Optional repository variables: SONAR_ORGANIZATION, SONAR_PROJECT_KEY.

Observability

Signal	Source	Backend	UI
Metrics	Micrometer → `/actuator/prometheus`	Prometheus	Grafana dashboard `devsecops-api`
Logs	JSON stdout → Loki Docker driver	Loki	Grafana Explore
Traces	OTel Java agent → Collector	Tempo	Grafana Explore

Grafana datasources are provisioned with trace-to-log and trace-to-metrics links — click a trace ID in Loki to jump to Tempo, and vice versa.

Demo traffic for observability screenshots:

# Create user → order → status change → intentional 404
curl -s -X POST localhost:8080/api/users -H 'Content-Type: application/json' \
  -d '{"email":"obs@example.com","password":"secret123"}'
# ... use returned IDs for order calls
curl -s localhost:8080/api/orders/00000000-0000-0000-0000-000000000000

Testing

./gradlew test                  # full suite
./gradlew test jacocoTestReport # with coverage report

Layer	Examples	Tooling
Unit	`UserAccountServiceTest`, `OrderServiceTest`, `StatsServiceTest`	MockK
Web	`UserAccountControllerTest`, `OrderControllerTest`	`@WebMvcTest`, MockMvc
Persistence	`UserAccountRepositoryTest`, `OrderRepositoryTest`	`@DataJpaTest`, Testcontainers
Integration	`ApiIntegrationTest`, `OrderIntegrationTest`	`@SpringBootTest`, Testcontainers PostgreSQL

CI runs ./gradlew clean test before every build. SonarCloud consumes the JaCoCo XML report from build/reports/jacoco/test/jacocoTestReport.xml.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
gitops/argocd		gitops/argocd
gradle/wrapper		gradle/wrapper
helm/devsecops-api		helm/devsecops-api
k8s		k8s
kind		kind
observability		observability
src		src
.dockerignore		.dockerignore
.gitattributes		.gitattributes
.gitignore		.gitignore
.sops.yaml		.sops.yaml
Dockerfile		Dockerfile
README.md		README.md
build.gradle.kts		build.gradle.kts
docker-compose.yml		docker-compose.yml
gradlew		gradlew
gradlew.bat		gradlew.bat
settings.gradle.kts		settings.gradle.kts

Latest commit

History

Folders and files

Repository files navigation