A Kubernetes distribution in a single static binary of about 5.5 MB, written in Rust with no libc and no third-party crates.
kr is the API server, datastore, scheduler, controllers, kubelet, container runtime, pod network, service proxy and cluster DNS in one executable. It is built to idle in about 14 MB of memory (6 MB for an agent), start in well under a second, and run unmodified kubectl and standard workloads. It passes the Kubernetes v1.37 conformance suite (460 of 460 specs on a two-node cluster, 458 of 460 on one node; see Conformance).
$ sudo kr server &
$ export KUBECONFIG=/var/tmp/srv/admin.kubeconfig
$ kubectl create deployment web --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 --replicas=3 -- /agnhost netexec
$ kubectl get pods
Requirements: Linux 5.10 or newer on x86_64 or aarch64, cgroup v2, overlayfs, and root for the full feature set (pod networking, cgroup limits). The root nftables Service datapath also needs the br_netfilter module loaded with net.bridge.bridge-nf-call-iptables=1 (and net.bridge.bridge-nf-call-ip6tables=1 for IPv6), so replies from same-node pods reach conntrack; most distributions and container runtimes load it. A rootless mode exists with reduced networking. Building needs a Rust nightly toolchain.
cargo build --release # produces target/release/kr (about 5.5 MB)sudo target/release/kr server --data /var/lib/kr & # API on :6443; the node registers itself
export KUBECONFIG=/var/lib/kr/admin.kubeconfig
kubectl get nodes
kubectl run hello --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 -- netexec
kubectl get pods -o wideThe server is also the first node: it runs the kubelet and the service proxy in-process.
# on the server
kr token create --ttl 24h --data /var/lib/kr
# on the new machine
sudo kr agent --server https://SERVER:6443 --token TOKENAgents get a pod subnet from the server and reach each other over VXLAN (default), host routes (KR_NET_BACKEND=host-gw) or encrypted WireGuard tunnels (KR_NET_BACKEND=wireguard, needs the wireguard kernel module).
kr server --cluster-init # first server, Raft-replicated store
kr server --server https://FIRST:6443 --token-file F # additional serverskr ctr pull docker.io/library/busybox
kr ctr run --memory 64Mi --cpus 500m docker.io/library/busybox -- echo hellokr server and kr agent read /etc/kr/config.yaml (or --config FILE, KR_CONFIG) when it exists, like k3s's config.yaml. Keys are the flag names without dashes; config.yaml.d/*.yaml next to it are merged over it in lexical order (a key replaces the earlier value, key+ appends). Flags override the file, the file overrides the KR_* environment variables.
# /etc/kr/config.yaml
tls-san: [kube.example.com, 203.0.113.7] # extra names/addresses in the API server certificate
node-name: cp-1
node-label: ["topology.kubernetes.io/zone=a"]
node-taint: ["dedicated=control-plane:NoSchedule"]
node-ip: 10.0.0.4
node-external-ip: 203.0.113.7
disable: [servicelb, metrics-server]
cluster-init: true| Flag (config key) | Env | Effect |
|---|---|---|
--tls-san NAME (repeatable) |
KR_TLS_SAN |
added to the API server certificate; the certificate is issued again at start when the set changed (server only) |
--node-name |
KR_NODE_NAME |
node name (default: hostname) |
--node-label K=V, --node-taint K=V:Effect (repeatable) |
KR_NODE_LABELS, KR_NODE_TAINTS (comma separated) |
set when the node registers; later starts leave edits made with kubectl alone unless the configured set changed |
--node-ip, --node-external-ip |
KR_NODE_IP, KR_NODE_EXTERNAL_IP |
InternalIP (also the address servers advertise) and ExternalIP |
--disable NAME (repeatable, or comma separated) |
KR_DISABLE |
servicelb, metrics-server, coredns, local-storage, network-policy, helm-controller; traefik is accepted and ignored. Cluster DNS and network policy run on every node, so set coredns / network-policy on every node |
More keys (flag or config key; each sets the KR_* variable shown, the flag wins over the file, the file over the variable):
| Key | Env | Value |
|---|---|---|
cluster-cidr, service-cidr |
KR_POD_CIDR, KR_SERVICE_CIDR |
A.B.0.0/16; in a v4,v6 pair the IPv4 entry is used |
cluster-domain |
KR_CLUSTER_DOMAIN |
DNS domain (same on every node) |
flannel-backend |
KR_NET_BACKEND |
vxlan, host-gw, wireguard-native (= wireguard); other values stop the start |
proxy-mode |
KR_PROXY |
userspace, nftables |
max-pods |
KR_MAX_PODS |
default 110 |
eviction-hard |
KR_EVICTION_MEMORY_AVAILABLE |
memory.available<500Mi; other signals ignored, percentages refused |
image-gc-high-threshold, image-gc-low-threshold, image-minimum-gc-age |
KR_IMAGE_GC_HIGH, _LOW, _MIN_AGE |
percent, percent, 2m / seconds |
node-eviction-rate, secondary-node-eviction-rate, unhealthy-zone-threshold, large-cluster-size-threshold |
KR_NODE_EVICTION_RATE, KR_SECONDARY_NODE_EVICTION_RATE, KR_UNHEALTHY_ZONE_THRESHOLD, KR_LARGE_CLUSTER_SIZE_THRESHOLD |
numbers, as for kube-controller-manager |
private-registry |
KR_REGISTRIES_YAML |
path of a k3s-format registries.yaml |
embedded-registry |
server only, boolean: nodes share pulled images peer to peer (port 5001) for the registries under mirrors: in registries.yaml; agents follow (ADR-067) |
|
kubelet-root-dir |
KR_KUBELET_DIR |
default /var/lib/kubelet |
data-dir, server, token, token-file |
(--data, --server, --token, --token-file; KR_TOKEN) |
as in k3s; data-dir is an alias of data |
disable-network-policy, disable-helm-controller |
KR_DISABLE |
boolean config keys, same as disable: [...] |
Variables with no k3s counterpart (KR_WIREGUARD_PORT, KR_POD_FAMILIES, KR_PREFER_IPV6, KR_DEVICE_*, KR_RAFT_ELECTION_MS, KR_OFFLOAD_*, KR_TLS_RESUMPTION, ...) stay environment-only.
| Variable | Effect |
|---|---|
KR_REGISTRY_MIRRORS |
registry.k8s.io=http://host:5000,docker.io=https://cache:5000 pull-through mirrors |
KR_NET_BACKEND |
vxlan (default), host-gw or wireguard |
KR_ROOT |
where images, pod state and IP leases live |
Measured on one 32-core x86_64 Linux machine, each node in a privileged Docker container, with cached images (pause and agnhost). Memory is the resident set size (RSS) of the main kr process; "PSS" is memory divided between processes that share pages.
| x86_64 | aarch64 | |
|---|---|---|
kr binary (stripped, opt-level=z, LTO, panic=abort) |
5.46 MB | 4.34 MB |
| Third-party crates | 0 | 0 |
| Rust source (13 workspace crates) | about 126,000 lines |
For scale: the binary contains the TLS 1.3 stack (and a TLS 1.2 client), X.509, the HTTP/1.1, HTTP/2, gRPC and WebSocket stacks, gzip and zstd, the Raft and storage engines, the OCI runtime, the API server with its generated protobuf tables, and the controllers.
| State | kr server |
kr agent |
|---|---|---|
| Idle, no pods | 14.0 MB RSS | 6.2 MB RSS |
| 10 pods | 15.3 MB | 7.8 MB |
| 50 pods | 19.0 MB | 11.8 MB |
| 100 pods | 23.5 MB | 16.1 MB |
| 200 pods | 32.9 MB | 24.7 MB |
The table is the main process; kr also forks a few small helpers: four offload workers (about 0.5 MB each) and an fsync helper (about 1.4 MB) on a server, and two offload workers on an agent.
The watch history (the recent writes kept so a watch can resume) is limited to 32 MiB, so memory under sustained writes levels off instead of growing: 62,000 updates of a 1 KB object (with the 200 pods above still running) leave the server at about 57 MB, the same as after 40,000.
Each pod also has a sandbox holder process that keeps its namespaces alive: about 0.65 MB RSS, of which only 80 KB is private (the rest is shared with the main process). Each running container has a shim: about 1.9 MB RSS, again only about 0.1 MB private, so 200 shims add about 23 MB (PSS) in total.
| Operation | Result |
|---|---|
kr server to node Ready (server and one agent) |
about 2 s |
100 pause pods, cached image, on one agent |
about 8.6 s to all Running |
200 pause pods, cached image, on one agent |
about 19 s to all Running |
Adopting 200 pods after a kill -9 of the agent |
about 2 s |
GET /version, new TLS 1.3 connection each time |
p50 1.9 ms, p99 3.0 ms |
GET /api/v1/namespaces/default/pods (200 pods, 857 KB) |
p50 8.2 ms, p99 25 ms |
| New TLS connections under 40 parallel clients | about 630 per second (client-bound), no failures |
Numbers will vary with hardware; the commands to reproduce them are in tools/test/ (two-node.sh, soak.sh, conformance.sh) and the measurements above used /proc/<pid>/status (VmRSS) and smaps_rollup (Pss).
Control plane
- Embedded registry mirror like k3s (
--embedded-registry): nodes pull images from each other before the registry, so an image is downloaded once per cluster and pulls keep working while the registry is down - Certificate and key rotation without restarts:
kr pki rotate(leaf certificates,--service-account-key,--ca prepare|activate|finish) is picked up by the running servers; agents follow the CA bundle and renew by themselves - Helm controller:
HelmChart/HelmChartConfig(helm.cattle.io/v1) like k3s, run as arancher/klipper-helmJob (--helm-job-imageto change the image,--disable helm-controllerto turn off); drop HelmChart YAML into<data dir>/manifests, which also deletes what a removed file created - Kubernetes API v1.37: core, apps, batch, autoscaling, policy, networking, storage, RBAC, certificates, coordination, discovery, flowcontrol, scheduling, resource (DRA), events and more, served as JSON and protobuf, with watches, chunked lists, Tables and aggregated discovery
- Embedded datastore (append-only log with MVCC revisions, compaction, optional Raft replication for 3 or more servers) instead of etcd
- Server-side apply with managed fields, strategic merge and JSON patch, field validation, defaulting
- CustomResourceDefinitions with structural schemas, CEL validation, defaulting, pruning, selectable fields, status and scale subresources, and conversion (None and webhook)
- Admission: validating and mutating webhooks, ValidatingAdmissionPolicy and MutatingAdmissionPolicy (CEL), Pod Security admission, ResourceQuota, LimitRange, priority classes, ServiceAccount
- Authentication: client certificates, bootstrap tokens, signed ServiceAccount tokens (TokenRequest, projected, audience-bound), TokenReview, impersonation
- Authorization: RBAC, Node authorizer, SubjectAccessReview and the self-service review APIs
- API aggregation: APIServices backed by a Service, with front-proxy authentication (watches are relayed as streams)
- API priority and fairness: FlowSchemas classify requests into priority levels with concurrency limits, per-flow fair queues and 429 +
Retry-Afterwhen full - Pod certificates (
PodCertificateRequestand projectedpodCertificatevolumes) and ClusterTrustBundles
Controllers and scheduler
- Deployment (rolling update with proportional scaling, rollback, history), ReplicaSet, ReplicationController, StatefulSet, DaemonSet, Job (success policies, indexed jobs, backoff per index, pod failure policy), CronJob
- Endpoints, EndpointSlice and mirroring, namespace and garbage collection (orphan and foreground), PV and PVC provisioning with a built-in local provisioner, PodDisruptionBudget, eviction
- Scheduler with resource fit, node selectors and affinity, taints and tolerations, topology spread, extended resources and priority preemption
Node
- OCI image pull (registry v2 with mirrors), overlayfs rootfs, containers in user, mount, PID, network, IPC, UTS and cgroup namespaces, seccomp, capability dropping, cgroup v2 limits, in-place pod resize
- Probes (HTTP, TCP, exec, gRPC), lifecycle hooks, init containers and sidecars, ephemeral containers,
exec,attach,logs,port-forwardover WebSocket and SPDY - Volumes: emptyDir, hostPath, configMap, secret, downwardAPI, projected (token, configMap, secret, ClusterTrustBundle, pod certificate), PVC, image volumes
- Memory-pressure eviction, node leases, node authorization and certificate rotation
Networking
- Built-in bridge and veth pod network with host-local IPAM, no CNI plugins
- Cross-node VXLAN or host-gw, NAT through nftables over netlink (no
iptables/nftbinaries needed) - IPv4, IPv6 or dual stack (
KR_POD_FAMILIES=4,6or4,6, default4,6on the server): pods get an address in each configured family (10.88.N.0/24,fd00:88:0:N::/64per node; Services are DNAT rules in the kernel when root with the built-in network (KR_PROXY=userspaceopts out;KR_PROXY=nftalso with an external CNI; ADR-054); the IPv4 pod and Service networks areKR_POD_CIDR/KR_SERVICE_CIDR, anyA.B.0.0/16, set before the first start and the same on every node, k3s's--cluster-cidrand--service-cidr), Services get ClusterIPs from10.96.0.0/16and/orfd00:96::/112(ipFamilies,ipFamilyPolicy), with AAAA records, per-family EndpointSlices and VXLAN across nodes. Hosts may be IPv4, IPv6 or both (join, Raft, kubelet and VXLAN run over an IPv6 underlay), and rootless NAT handles both - Service proxy for ClusterIP, NodePort and hostPort (with ClientIP affinity), in-process cluster DNS (
cluster.local, pod and service records, ExternalName)
Every crate is #![no_std] with alloc; only the final kr binary provides _start, the panic handler, the allocator and the syscall layer (inline assembly). There is no libc and no external crate.
| Crate | Role |
|---|---|
kr-sys |
raw syscalls for x86_64 and aarch64, namespaces, cgroups, netlink primitives |
kr-rt |
process start-up, allocator, single-threaded epoll executor for async/await, worker processes for CPU-bound work |
kr-fmt |
JSON, YAML subset, protobuf wire format, base64, quantities |
kr-crypto |
SHA-2, HMAC, X25519, Ed25519, ECDSA P-256, RSA, AES-GCM, ChaCha20-Poly1305, X.509, TLS 1.3 client and server with session resumption, TLS 1.2 client (opt-in) |
kr-net |
TCP, HTTP/1.1 client and server, HTTP/2 with HPACK, gRPC (unary and streaming, client and server), WebSocket, SPDY, DNS client and server, OCI registry client, S3 client (SigV4), netlink (incl. WireGuard), nftables, userspace NAT for rootless pods |
kr-archive |
gzip (inflate), zstd and tar for image layers |
kr-store |
embedded MVCC datastore with write-ahead log, watch and snapshots |
kr-raft |
Raft for replicated servers |
kr-apiserver |
REST (JSON and protobuf), discovery, OpenAPI, authn/authz (RBAC, node authorizer), admission (incl. Pod Security, quota, CEL validating policies), CRDs with conversion, server-side apply, webhooks, aggregation, API priority and fairness, audit (log and webhook), encryption at rest, metrics, Raft cluster join and forwarding |
kr-controllers |
built-in controllers (workloads, GC, namespaces, nodes, HPA, PVC and attach/detach, CSR signing, quota, service LB, storage version migration), the Helm controller, the manifests directory, the scheduler |
kr-runtime |
OCI image pull and store, container runtime with seccomp, AppArmor/SELinux, device cgroup filter and OCI hooks, image GC, pod network |
kr-kubelet |
pod lifecycle, probes (incl. gRPC), volumes and CSI, device plugins, pod resources API, eviction, stats, exec/attach/port-forward, CNI, network policy, egress, DNS, service proxy (userspace and nftables), adoption of running pods |
kr |
the binary: server, agent, ctr, token, pki/certificate, snapshot, secrets-encrypt |
Design points:
- One thread. The runtime is a cooperative epoll executor, so there are no locks and no data races; clone3, setns and friends are used safely because of it. (Decision records:
docs/DECISIONS.md.) - The kubelet calls the runtime in-process; there is no CRI, no containerd, no runc and no separate daemons.
- The Kubernetes protobuf schema tables are generated from the upstream
.protofiles bytools/gen_k8s_proto_schema.pyand compiled into the binary.
Results of the CNCF conformance suite (Sonobuoy, non-disruptive-conformance, parallel), Kubernetes v1.37 e2e image:
| Cluster | Result |
|---|---|
| 2 nodes (server and agent) | 460 of 460 |
| 1 node | 458 of 460 in the latest full run: the other 2 specs need a second node and cannot pass on one |
Reproduce with tools/test/conformance.sh (see docs/CONFORMANCE_README.md):
tools/test/conformance.sh run full # one node, in a throwaway container
KR_NODES=2 tools/test/conformance.sh run full # server plus agent
tools/test/conformance.sh run focus 'Aggregator.*\[Conformance\]' # a subset
tools/test/conformance.sh results /tmp/out # failures by area- Only ECDSA P-256 keys for pod certificates; RSA is verify-only.
- Not certified by the CNCF: the suite passes, but the certification process has not been run.
- The full list is in
docs/KNOWN_ISSUES.md.
tools/test/unit.sh # every library crate (the kr binary itself has no unit tests)
cargo test --workspace
tools/test/aarch64.sh # cross-build and smoke test under qemu-user
tools/test/two-node.sh up # server kr1 and agent kr2 in Docker
tools/test/ha.sh # three-server Raft cluster
tools/test/rootless-egress.sh # rootless pod and ctr egress: download, upload, DNS, ping (run as a normal user)
tools/test/rootless-egress-v6.sh # the same over IPv6 in a throwaway container
tools/test/dualstack.sh # dual-stack pods, Services and DNS on a two-node cluster
tools/test/aggregated-watch.sh # a watch through an aggregated API (extension server in a pod)
tools/test/families.sh 6 # pod network families: 4, 6 or 4,6 on a two-node cluster
tools/test/families-change.sh # restart an existing cluster with other families
tools/test/rootless-families.sh 6 # rootless kr ctr and pods per familyDocuments: docs/PLAN.md (design and budgets), docs/DECISIONS.md (architecture decision records), docs/KNOWN_ISSUES.md, docs/CONFORMANCE_KNOWN_ISSUES.md.
License: Apache-2.0.