Skip to content
apiplantPublic

About

A Kubernetes distribution in a single static 3MB binary, written in Rust

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

393 Commits

Folders and files

Repository files navigation

kr

A Kubernetes distribution in a single static binary of about 5.5 MB, written in Rust with no libc and no third-party crates.

kr is the API server, datastore, scheduler, controllers, kubelet, container runtime, pod network, service proxy and cluster DNS in one executable. It is built to idle in about 14 MB of memory (6 MB for an agent), start in well under a second, and run unmodified kubectl and standard workloads. It passes the Kubernetes v1.37 conformance suite (460 of 460 specs on a two-node cluster, 458 of 460 on one node; see Conformance).

$ sudo kr server &
$ export KUBECONFIG=/var/tmp/srv/admin.kubeconfig
$ kubectl create deployment web --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 --replicas=3 -- /agnhost netexec
$ kubectl get pods

Contents

Quick start

Requirements: Linux 5.10 or newer on x86_64 or aarch64, cgroup v2, overlayfs, and root for the full feature set (pod networking, cgroup limits). The root nftables Service datapath also needs the br_netfilter module loaded with net.bridge.bridge-nf-call-iptables=1 (and net.bridge.bridge-nf-call-ip6tables=1 for IPv6), so replies from same-node pods reach conntrack; most distributions and container runtimes load it. A rootless mode exists with reduced networking. Building needs a Rust nightly toolchain.

cargo build --release          # produces target/release/kr (about 5.5 MB)

Single node

sudo target/release/kr server --data /var/lib/kr &      # API on :6443; the node registers itself
export KUBECONFIG=/var/lib/kr/admin.kubeconfig
kubectl get nodes
kubectl run hello --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 -- netexec
kubectl get pods -o wide

The server is also the first node: it runs the kubelet and the service proxy in-process.

Add a node

# on the server
kr token create --ttl 24h --data /var/lib/kr

# on the new machine
sudo kr agent --server https://SERVER:6443 --token TOKEN

Agents get a pod subnet from the server and reach each other over VXLAN (default), host routes (KR_NET_BACKEND=host-gw) or encrypted WireGuard tunnels (KR_NET_BACKEND=wireguard, needs the wireguard kernel module).

High availability

kr server --cluster-init                                  # first server, Raft-replicated store
kr server --server https://FIRST:6443 --token-file F     # additional servers

Running containers without Kubernetes

kr ctr pull docker.io/library/busybox
kr ctr run --memory 64Mi --cpus 500m docker.io/library/busybox -- echo hello

Configuration file and install options

kr server and kr agent read /etc/kr/config.yaml (or --config FILE, KR_CONFIG) when it exists, like k3s's config.yaml. Keys are the flag names without dashes; config.yaml.d/*.yaml next to it are merged over it in lexical order (a key replaces the earlier value, key+ appends). Flags override the file, the file overrides the KR_* environment variables.

# /etc/kr/config.yaml
tls-san: [kube.example.com, 203.0.113.7]   # extra names/addresses in the API server certificate
node-name: cp-1
node-label: ["topology.kubernetes.io/zone=a"]
node-taint: ["dedicated=control-plane:NoSchedule"]
node-ip: 10.0.0.4
node-external-ip: 203.0.113.7
disable: [servicelb, metrics-server]
cluster-init: true
Flag (config key) Env Effect
--tls-san NAME (repeatable) KR_TLS_SAN added to the API server certificate; the certificate is issued again at start when the set changed (server only)
--node-name KR_NODE_NAME node name (default: hostname)
--node-label K=V, --node-taint K=V:Effect (repeatable) KR_NODE_LABELS, KR_NODE_TAINTS (comma separated) set when the node registers; later starts leave edits made with kubectl alone unless the configured set changed
--node-ip, --node-external-ip KR_NODE_IP, KR_NODE_EXTERNAL_IP InternalIP (also the address servers advertise) and ExternalIP
--disable NAME (repeatable, or comma separated) KR_DISABLE servicelb, metrics-server, coredns, local-storage, network-policy, helm-controller; traefik is accepted and ignored. Cluster DNS and network policy run on every node, so set coredns / network-policy on every node

More keys (flag or config key; each sets the KR_* variable shown, the flag wins over the file, the file over the variable):

Key Env Value
cluster-cidr, service-cidr KR_POD_CIDR, KR_SERVICE_CIDR A.B.0.0/16; in a v4,v6 pair the IPv4 entry is used
cluster-domain KR_CLUSTER_DOMAIN DNS domain (same on every node)
flannel-backend KR_NET_BACKEND vxlan, host-gw, wireguard-native (= wireguard); other values stop the start
proxy-mode KR_PROXY userspace, nftables
max-pods KR_MAX_PODS default 110
eviction-hard KR_EVICTION_MEMORY_AVAILABLE memory.available<500Mi; other signals ignored, percentages refused
image-gc-high-threshold, image-gc-low-threshold, image-minimum-gc-age KR_IMAGE_GC_HIGH, _LOW, _MIN_AGE percent, percent, 2m / seconds
node-eviction-rate, secondary-node-eviction-rate, unhealthy-zone-threshold, large-cluster-size-threshold KR_NODE_EVICTION_RATE, KR_SECONDARY_NODE_EVICTION_RATE, KR_UNHEALTHY_ZONE_THRESHOLD, KR_LARGE_CLUSTER_SIZE_THRESHOLD numbers, as for kube-controller-manager
private-registry KR_REGISTRIES_YAML path of a k3s-format registries.yaml
embedded-registry server only, boolean: nodes share pulled images peer to peer (port 5001) for the registries under mirrors: in registries.yaml; agents follow (ADR-067)
kubelet-root-dir KR_KUBELET_DIR default /var/lib/kubelet
data-dir, server, token, token-file (--data, --server, --token, --token-file; KR_TOKEN) as in k3s; data-dir is an alias of data
disable-network-policy, disable-helm-controller KR_DISABLE boolean config keys, same as disable: [...]

Variables with no k3s counterpart (KR_WIREGUARD_PORT, KR_POD_FAMILIES, KR_PREFER_IPV6, KR_DEVICE_*, KR_RAFT_ELECTION_MS, KR_OFFLOAD_*, KR_TLS_RESUMPTION, ...) stay environment-only.

Useful environment variables

Variable Effect
KR_REGISTRY_MIRRORS registry.k8s.io=http://host:5000,docker.io=https://cache:5000 pull-through mirrors
KR_NET_BACKEND vxlan (default), host-gw or wireguard
KR_ROOT where images, pod state and IP leases live

Benchmarks

Measured on one 32-core x86_64 Linux machine, each node in a privileged Docker container, with cached images (pause and agnhost). Memory is the resident set size (RSS) of the main kr process; "PSS" is memory divided between processes that share pages.

Size

x86_64 aarch64
kr binary (stripped, opt-level=z, LTO, panic=abort) 5.46 MB 4.34 MB
Third-party crates 0 0
Rust source (13 workspace crates) about 126,000 lines

For scale: the binary contains the TLS 1.3 stack (and a TLS 1.2 client), X.509, the HTTP/1.1, HTTP/2, gRPC and WebSocket stacks, gzip and zstd, the Raft and storage engines, the OCI runtime, the API server with its generated protobuf tables, and the controllers.

Memory

State kr server kr agent
Idle, no pods 14.0 MB RSS 6.2 MB RSS
10 pods 15.3 MB 7.8 MB
50 pods 19.0 MB 11.8 MB
100 pods 23.5 MB 16.1 MB
200 pods 32.9 MB 24.7 MB

The table is the main process; kr also forks a few small helpers: four offload workers (about 0.5 MB each) and an fsync helper (about 1.4 MB) on a server, and two offload workers on an agent.

The watch history (the recent writes kept so a watch can resume) is limited to 32 MiB, so memory under sustained writes levels off instead of growing: 62,000 updates of a 1 KB object (with the 200 pods above still running) leave the server at about 57 MB, the same as after 40,000.

Each pod also has a sandbox holder process that keeps its namespaces alive: about 0.65 MB RSS, of which only 80 KB is private (the rest is shared with the main process). Each running container has a shim: about 1.9 MB RSS, again only about 0.1 MB private, so 200 shims add about 23 MB (PSS) in total.

Latency and throughput

Operation Result
kr server to node Ready (server and one agent) about 2 s
100 pause pods, cached image, on one agent about 8.6 s to all Running
200 pause pods, cached image, on one agent about 19 s to all Running
Adopting 200 pods after a kill -9 of the agent about 2 s
GET /version, new TLS 1.3 connection each time p50 1.9 ms, p99 3.0 ms
GET /api/v1/namespaces/default/pods (200 pods, 857 KB) p50 8.2 ms, p99 25 ms
New TLS connections under 40 parallel clients about 630 per second (client-bound), no failures

Numbers will vary with hardware; the commands to reproduce them are in tools/test/ (two-node.sh, soak.sh, conformance.sh) and the measurements above used /proc/<pid>/status (VmRSS) and smaps_rollup (Pss).

Features

Control plane

  • Embedded registry mirror like k3s (--embedded-registry): nodes pull images from each other before the registry, so an image is downloaded once per cluster and pulls keep working while the registry is down
  • Certificate and key rotation without restarts: kr pki rotate (leaf certificates, --service-account-key, --ca prepare|activate|finish) is picked up by the running servers; agents follow the CA bundle and renew by themselves
  • Helm controller: HelmChart / HelmChartConfig (helm.cattle.io/v1) like k3s, run as a rancher/klipper-helm Job (--helm-job-image to change the image, --disable helm-controller to turn off); drop HelmChart YAML into <data dir>/manifests, which also deletes what a removed file created
  • Kubernetes API v1.37: core, apps, batch, autoscaling, policy, networking, storage, RBAC, certificates, coordination, discovery, flowcontrol, scheduling, resource (DRA), events and more, served as JSON and protobuf, with watches, chunked lists, Tables and aggregated discovery
  • Embedded datastore (append-only log with MVCC revisions, compaction, optional Raft replication for 3 or more servers) instead of etcd
  • Server-side apply with managed fields, strategic merge and JSON patch, field validation, defaulting
  • CustomResourceDefinitions with structural schemas, CEL validation, defaulting, pruning, selectable fields, status and scale subresources, and conversion (None and webhook)
  • Admission: validating and mutating webhooks, ValidatingAdmissionPolicy and MutatingAdmissionPolicy (CEL), Pod Security admission, ResourceQuota, LimitRange, priority classes, ServiceAccount
  • Authentication: client certificates, bootstrap tokens, signed ServiceAccount tokens (TokenRequest, projected, audience-bound), TokenReview, impersonation
  • Authorization: RBAC, Node authorizer, SubjectAccessReview and the self-service review APIs
  • API aggregation: APIServices backed by a Service, with front-proxy authentication (watches are relayed as streams)
  • API priority and fairness: FlowSchemas classify requests into priority levels with concurrency limits, per-flow fair queues and 429 + Retry-After when full
  • Pod certificates (PodCertificateRequest and projected podCertificate volumes) and ClusterTrustBundles

Controllers and scheduler

  • Deployment (rolling update with proportional scaling, rollback, history), ReplicaSet, ReplicationController, StatefulSet, DaemonSet, Job (success policies, indexed jobs, backoff per index, pod failure policy), CronJob
  • Endpoints, EndpointSlice and mirroring, namespace and garbage collection (orphan and foreground), PV and PVC provisioning with a built-in local provisioner, PodDisruptionBudget, eviction
  • Scheduler with resource fit, node selectors and affinity, taints and tolerations, topology spread, extended resources and priority preemption

Node

  • OCI image pull (registry v2 with mirrors), overlayfs rootfs, containers in user, mount, PID, network, IPC, UTS and cgroup namespaces, seccomp, capability dropping, cgroup v2 limits, in-place pod resize
  • Probes (HTTP, TCP, exec, gRPC), lifecycle hooks, init containers and sidecars, ephemeral containers, exec, attach, logs, port-forward over WebSocket and SPDY
  • Volumes: emptyDir, hostPath, configMap, secret, downwardAPI, projected (token, configMap, secret, ClusterTrustBundle, pod certificate), PVC, image volumes
  • Memory-pressure eviction, node leases, node authorization and certificate rotation

Networking

  • Built-in bridge and veth pod network with host-local IPAM, no CNI plugins
  • Cross-node VXLAN or host-gw, NAT through nftables over netlink (no iptables/nft binaries needed)
  • IPv4, IPv6 or dual stack (KR_POD_FAMILIES=4, 6 or 4,6, default 4,6 on the server): pods get an address in each configured family (10.88.N.0/24, fd00:88:0:N::/64 per node; Services are DNAT rules in the kernel when root with the built-in network (KR_PROXY=userspace opts out; KR_PROXY=nft also with an external CNI; ADR-054); the IPv4 pod and Service networks are KR_POD_CIDR / KR_SERVICE_CIDR, any A.B.0.0/16, set before the first start and the same on every node, k3s's --cluster-cidr and --service-cidr), Services get ClusterIPs from 10.96.0.0/16 and/or fd00:96::/112 (ipFamilies, ipFamilyPolicy), with AAAA records, per-family EndpointSlices and VXLAN across nodes. Hosts may be IPv4, IPv6 or both (join, Raft, kubelet and VXLAN run over an IPv6 underlay), and rootless NAT handles both
  • Service proxy for ClusterIP, NodePort and hostPort (with ClientIP affinity), in-process cluster DNS (cluster.local, pod and service records, ExternalName)

How it is built

Every crate is #![no_std] with alloc; only the final kr binary provides _start, the panic handler, the allocator and the syscall layer (inline assembly). There is no libc and no external crate.

Crate Role
kr-sys raw syscalls for x86_64 and aarch64, namespaces, cgroups, netlink primitives
kr-rt process start-up, allocator, single-threaded epoll executor for async/await, worker processes for CPU-bound work
kr-fmt JSON, YAML subset, protobuf wire format, base64, quantities
kr-crypto SHA-2, HMAC, X25519, Ed25519, ECDSA P-256, RSA, AES-GCM, ChaCha20-Poly1305, X.509, TLS 1.3 client and server with session resumption, TLS 1.2 client (opt-in)
kr-net TCP, HTTP/1.1 client and server, HTTP/2 with HPACK, gRPC (unary and streaming, client and server), WebSocket, SPDY, DNS client and server, OCI registry client, S3 client (SigV4), netlink (incl. WireGuard), nftables, userspace NAT for rootless pods
kr-archive gzip (inflate), zstd and tar for image layers
kr-store embedded MVCC datastore with write-ahead log, watch and snapshots
kr-raft Raft for replicated servers
kr-apiserver REST (JSON and protobuf), discovery, OpenAPI, authn/authz (RBAC, node authorizer), admission (incl. Pod Security, quota, CEL validating policies), CRDs with conversion, server-side apply, webhooks, aggregation, API priority and fairness, audit (log and webhook), encryption at rest, metrics, Raft cluster join and forwarding
kr-controllers built-in controllers (workloads, GC, namespaces, nodes, HPA, PVC and attach/detach, CSR signing, quota, service LB, storage version migration), the Helm controller, the manifests directory, the scheduler
kr-runtime OCI image pull and store, container runtime with seccomp, AppArmor/SELinux, device cgroup filter and OCI hooks, image GC, pod network
kr-kubelet pod lifecycle, probes (incl. gRPC), volumes and CSI, device plugins, pod resources API, eviction, stats, exec/attach/port-forward, CNI, network policy, egress, DNS, service proxy (userspace and nftables), adoption of running pods
kr the binary: server, agent, ctr, token, pki/certificate, snapshot, secrets-encrypt

Design points:

  • One thread. The runtime is a cooperative epoll executor, so there are no locks and no data races; clone3, setns and friends are used safely because of it. (Decision records: docs/DECISIONS.md.)
  • The kubelet calls the runtime in-process; there is no CRI, no containerd, no runc and no separate daemons.
  • The Kubernetes protobuf schema tables are generated from the upstream .proto files by tools/gen_k8s_proto_schema.py and compiled into the binary.

Conformance

Results of the CNCF conformance suite (Sonobuoy, non-disruptive-conformance, parallel), Kubernetes v1.37 e2e image:

Cluster Result
2 nodes (server and agent) 460 of 460
1 node 458 of 460 in the latest full run: the other 2 specs need a second node and cannot pass on one

Reproduce with tools/test/conformance.sh (see docs/CONFORMANCE_README.md):

tools/test/conformance.sh run full               # one node, in a throwaway container
KR_NODES=2 tools/test/conformance.sh run full    # server plus agent
tools/test/conformance.sh run focus 'Aggregator.*\[Conformance\]'   # a subset
tools/test/conformance.sh results /tmp/out       # failures by area

Limitations

  • Only ECDSA P-256 keys for pod certificates; RSA is verify-only.
  • Not certified by the CNCF: the suite passes, but the certification process has not been run.
  • The full list is in docs/KNOWN_ISSUES.md.

Development

tools/test/unit.sh            # every library crate (the kr binary itself has no unit tests)
cargo test --workspace
tools/test/aarch64.sh         # cross-build and smoke test under qemu-user
tools/test/two-node.sh up     # server kr1 and agent kr2 in Docker
tools/test/ha.sh              # three-server Raft cluster
tools/test/rootless-egress.sh # rootless pod and ctr egress: download, upload, DNS, ping (run as a normal user)
tools/test/rootless-egress-v6.sh # the same over IPv6 in a throwaway container
tools/test/dualstack.sh       # dual-stack pods, Services and DNS on a two-node cluster
tools/test/aggregated-watch.sh   # a watch through an aggregated API (extension server in a pod)
tools/test/families.sh 6        # pod network families: 4, 6 or 4,6 on a two-node cluster
tools/test/families-change.sh   # restart an existing cluster with other families
tools/test/rootless-families.sh 6   # rootless kr ctr and pods per family

Documents: docs/PLAN.md (design and budgets), docs/DECISIONS.md (architecture decision records), docs/KNOWN_ISSUES.md, docs/CONFORMANCE_KNOWN_ISSUES.md.

License: Apache-2.0.

About

A Kubernetes distribution in a single static 3MB binary, written in Rust

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages