A demand-gated, multi-objective scheduler that decides, every 30 seconds, how many replicas of each microservice run on each cluster of a geo-distributed Kubernetes federation — weighing latency, capacity, network proximity and carbon intensity, and placing replicas only where the demand is.
DMOS started as an MSc thesis at the University of Catania and is the subject of the IEEE NCA 2026 paper DMOS: Demand-Gated Multi-Objective Replica Placement for Geo-Distributed Multi-Cluster Systems (publications).
A central control loop, running on an operator machine, reads each
cluster's Prometheus (Cilium Hubble L7 metrics, node resources,
ping_exporter RTTs) and a carbon-intensity source, and every cycle:
-
Budget — sizes the frontend from the demand entering each cluster:
N = Σᵢ ⌈λᵢⁱⁿ / capacity⌉, within per-cluster caps. -
Placement (Level 1) — scores every eligible cluster and splits N in proportion to the score (largest remainder, capacity-aware). In the published design demand is a multiplicative gate, not one more weight:
S_i = G(d_i) · [ ω_lat·Φ_lat + ω_cap·Φ_cap + ω_net·Φ_net + ω_car·Φ_car ] G(d) = (ε + (1−ε)·d)^g d_i = cluster's share of the entering demandA cluster without demand cannot attract replicas whatever its other qualities; among those with demand, the weights decide.
-
Control (Level 2) — a trend predictor anticipates the scale-up and a PD controller adds a ±1 correction per cluster; rate limits, cooldowns, a dead zone and churn budgets keep the system stable.
-
Co-scaling — the seven stateless backends follow the frontend on each cluster in proportion to capacity and call fan-out.
How demand enters the score, the weights, the gate and the start-up policy
are a named score model in config/weights.yaml (nca is the
configuration described in the NCA 2026 paper; gate_g0/g1/g3,
additive_dem50, locality, *_first are the variants of the journal
extension), selected with DMOS_SCORE_MODEL. The NCA campaign itself ran
the arm l1l2 (gate g = 2, weights 0.25/0.20/0.25/0.30; see
the publication page). Every run
records its effective configuration and a config_id.
┌──────────────────────── operator machine ─────────────────────────┐
│ python -m src.dmos_main │
│ budget N ─► L1 score + proportional allocation ─► L2 predictor/PD │
│ ─► co-scaling of the backends :9090 /metrics :8081 /config│
└──────────┬──────────────────────────────┬─────────────────────────┘
PromQL│ kube-API (scale)
▼ ▼
┌──────────────── 6 × K3s clusters on Proxmox, Cilium ClusterMesh ────────────────┐
│ ms-cluster-01 EU-DE (HIGH) ms-cluster-02 EU-FR (MED) ms-cluster-03 EU-PL (LOW)│
│ ms-cluster-04 US-VA (HIGH) ms-cluster-05 SG (LOW) ms-cluster-06 NO (MED)│
│ each: Cilium (Hubble L7, Ingress :30080) + kube-prometheus-stack (:32090) │
│ + ping_exporter + Online Boutique; tc netem 0–200 ms per cluster │
└──────────────────────────────────────────────────────────────────────────────────┘
▲ k6 load generator (dedicated VM), per-cluster ingress weights
Details: docs/dmos-architecture.md.
git clone <repo-url> dmos && cd dmos
python -m venv venv && source venv/Scripts/activate # Linux/macOS: venv/bin/activate
pip install -r requirements.txt- Testbed — build the clusters and the DMOS add-ons, write
config/local.yaml(gitignored), render.cache/, apply netem: docs/setup-guide.md. - Check —
bash scripts/validate-setup.shandbash experiments/preflight_runs.sh. - Run DMOS —
DMOS_SCORE_MODEL=nca python -m src.dmos_main(from the repository root). One cycle every 30 s; the first 120 s are an observation window (nothing is scaled). - Run experiments — every test is named
dmos-<arm>__<regime>(python experiments/run_naming.py list):bash experiments/run_one.sh l1l2 conruns one cell,bash experiments/run_matrix_session.shone repetition of the NCA core (off / l2 / l1l2 × 6 regimes). See experiments/RUNBOOK.md. - Analyse —
python experiments/paper_generate_all.py(docs/METRICS-AND-OUTPUTS.md).
Carbon intensity is replayed from hourly Electricity Maps traces of 2025
that you download into data/carbon_traces/ with your own access (they are
not distributed with the repository; see
their README). Without them DMOS falls back
to built-in hourly profiles.
src/
dmos_main.py orchestrator: control loop, L2 application, co-scaling, HTTP endpoints
level1/ score functions, score models, scheduler (L0 + scores), allocation
level2/ trend predictor, PD controller
metrics/ per-cluster Prometheus client, carbon client
k8s/ multi-cluster Kubernetes client
utils/ configuration loader, provenance (effective config, config_id), logger
config/ weights.yaml (score models, legacy profiles), services.yaml,
carbon.yaml, local.yaml.example, Helm values
data/carbon_traces/ where the Electricity Maps 2025 hourly traces go (not distributed; README)
deployments/ manifests DMOS expects in each cluster (+ clustermesh-overflow/ pilot)
scripts/ setup, rendering of .cache/, validation, netem, carbon trace download
experiments/ k6 workloads, run harness (run_one.sh, lib/run_common.sh),
provenance tools, collector, analysis pipeline (paper_*.py)
tests/ pytest suite (no cluster needed)
docs/ documentation (index below), publications/
Design
- Architecture and design — the reference: every mechanism and constant, with the code that implements it
- Start-up: observation window and two-phase score
- Measuring demand: the ingress rate
- Network proximity (Φ_net) and the emulated geography
- Backend co-scaling
Configuration and setup
- Setup guide
- Configuration reference — every key and environment variable
- Ingress architecture, Hubble L7 visibility, Cilium network policies, ClusterMesh notes
- Troubleshooting, Integration guide
Experiments and results
- Reproducibility guide — from
git cloneto the figures - Experiment runbook
- Run provenance — what each run records and how to audit it
- Metrics and outputs — run files, metric definitions, figures and tables
- Load-testing scenarios
- Capacity testing — frontend, gRPC backends, throttle check
Publications — docs/publications/: one page per paper with the code version, configuration, figure → script mapping and citation.
Historical documents (earlier testbeds, superseded designs) are kept in docs/_archive/.
pytest tests/ -v # from the repository rootThe suite covers the score components and score models, configuration
loading, the carbon and Prometheus clients (trace lookup per region), the
allocation, the test naming (run_naming.py) and the provenance tools
(effective configuration, testbed capture, run manifests).
It needs no cluster; cluster-side checks are in scripts/validate-setup.sh
and experiments/preflight_runs.sh.
If you build on DMOS, please cite the NCA 2026 paper (BibTeX):
A. Marchese, M. Santanocito, O. Tomarchio. DMOS: Demand-Gated Multi-Objective Replica Placement for Geo-Distributed Multi-Cluster Systems. IEEE International Symposium on Network Computing and Applications (NCA), 2026.
DMOS originates from the MSc thesis: M. Santanocito, Distributed Multi-Objective Scheduler for Kubernetes Multi-Cluster (DMOS), University of Catania, 2026.
The code is released under the license in LICENSE. The carbon
intensity data used by the experiments come from Electricity Maps and are
not distributed with the repository; publications that use them must cite
Electricity Maps (see data/carbon_traces/README.md).