diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 0000000..b05d0df --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,101 @@ +name: release + +on: + push: + tags: ['v*'] + +permissions: + contents: write + +concurrency: + group: release-${{ github.ref }} + cancel-in-progress: false + +jobs: + guard: + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v5 + with: + fetch-depth: 0 + + - name: Refuse a tag that is not on main + run: | + git fetch --no-tags --quiet origin main + if ! git merge-base --is-ancestor "${GITHUB_SHA}" origin/main; then + echo "${GITHUB_REF_NAME} points at a commit that is not on main" + exit 1 + fi + + build: + needs: guard + strategy: + fail-fast: true + matrix: + include: + - runner: ubuntu-24.04 + arch: amd64 + - runner: ubuntu-24.04-arm + arch: arm64 + runs-on: ${{ matrix.runner }} + steps: + - uses: actions/checkout@v5 + + - name: install libseccomp headers + run: | + sudo apt-get update + sudo apt-get install -y --no-install-recommends libseccomp-dev pkg-config + + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + + - run: cargo test --lib + + - name: Build the release archive + run: make dist VERSION="${GITHUB_REF_NAME}" + + - uses: actions/upload-artifact@v4 + with: + name: dist-${{ matrix.arch }} + path: dist/*.tar.gz + if-no-files-found: error + + publish: + needs: build + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v5 + + - uses: actions/download-artifact@v4 + with: + pattern: dist-* + merge-multiple: true + path: dist + + - name: Checksum every archive together + run: cd dist && sha256sum *.tar.gz > SHA256SUMS + + - name: Take the release notes from the changelog + run: | + version="${GITHUB_REF_NAME#v}" + awk -v v="$version" ' + index($0, "## [" v "]") == 1 { inside = 1; next } + inside && /^## / { exit } + inside { print } + ' CHANGELOG.md | sed '/./,$!d' > notes.md + if [ -s notes.md ]; then + echo "notes=--notes-file notes.md" >> "$GITHUB_ENV" + else + echo "CHANGELOG.md has no section for $version, generating notes from commits" + echo "notes=--generate-notes" >> "$GITHUB_ENV" + fi + + - name: Publish the release + env: + GH_TOKEN: ${{ github.token }} + run: | + gh release create "${GITHUB_REF_NAME}" \ + --title "${GITHUB_REF_NAME}" \ + --verify-tag \ + $notes \ + dist/*.tar.gz dist/SHA256SUMS diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..61cf06b --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,89 @@ +# Changelog + +The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project follows +[semantic versioning](https://semver.org/spec/v2.0.0.html). Before 1.0 the CLI, the annotations and +the on-disk state may change between minor versions. + +A release ships one binary, `mars`, which is the whole runtime: it takes a filesystem bundle and a +`config.json` and turns it into an isolated process. + +To cut a release, rename `Unreleased` below to the version and the date, commit, then push the +matching `v` tag. The release workflow takes its notes from the section named after the tag, so a +tag with no section of its own falls back to a bare list of commits. + +Nothing is tagged yet, so everything below sits under `Unreleased`. + +## [Unreleased] + +Pre-release, and deliberately not production software: this exists to build a model of the layer +Docker hides, not to displace `runc`. Read "Before production" in +[the usage guide](https://marstack-labs.github.io/marstack-container/) before running it anywhere +that matters. + +### Added + +- **A complete OCI runtime.** `create`, `start`, `state`, `kill`, `delete`, `exec`, `list`, `ps`, + `pause`, `resume`, `events`, `update`, `spec`, `features`, all five lifecycle hooks, and a console + socket over `SCM_RIGHTS`. 26 of the OCI validation suite pass, next to `runc` 1.5.1's 22 on the + same host. +- **Isolation written by hand.** mount, pid, uts, ipc, user, cgroup, net and time namespaces, + `pivot_root`, the standard mounts and device nodes. The three-level fork chain is forced by the + kernel: `unshare(CLONE_NEWPID)` does not move the caller into the new namespace, and `uid_map` has + to be written from outside it by a privileged process. +- **A cgroup v2 driver written against `cgroupfs`,** covering `memory`, `cpu`, `pids`, `cpuset` and + `io`. Delegating this to a crate would delegate away the point of the project. +- **OverlayFS rootfs assembly** as a documented extension rather than a spec feature. Three + `dev.mars.overlay.*` annotations, so the `config.json` stays valid for any other runtime, which + will ignore them. +- **Hardening:** capabilities, seccomp, `no_new_privs`, read-only rootfs, `maskedPaths` and + `readonlyPaths`, sysctls, rlimits, `oomScoreAdj`, and user namespaces with a `newuidmap` fallback. +- **A Docker drop-in.** `docker run --runtime=mars` covering `run`, `run -it`, `exec`, `stop` and + `--memory`, with `TRACE=1` to log how Docker calls a runtime it has never seen. +- **OTLP trace export with no background thread.** The OpenTelemetry SDK exports on one, which a + process that forks or calls `setns` must not have, so the exporter is around 120 lines that build + OTLP/HTTP JSON and write one POST. +- **An integration suite of 128 assertions** that read kernel state — `/proc`, `/proc/mounts`, + `/sys/fs/cgroup`, `ip -o link`, wait statuses — rather than trusting what the runtime reports + about itself. +- **Nine production failure modes reproduced** with the evidence read out of the kernel, in + [`docs/failure-modes.md`](docs/failure-modes.md). Among them: an OOM kill does not reliably + produce exit 137, because the kernel picks its victim by badness score and PID 1 often survives. +- **A kernel attack surface benchmark,** `scripts/hap-bench.sh`, counting the distinct host kernel + functions a runtime traverses while it holds root. 1542 for `mars` against `crun`'s 2030 and + `runc`'s 2361 on a plain start, and 494 against 889 and 1134 on `exec`. Method, the three + measurement faults that produced wrong answers first, and the limits are in + [`docs/attack-surface.md`](docs/attack-surface.md). + +### Fixed + +- **A standard seccomp profile no longer refuses to load.** `libseccomp` returns `EACCES` for a rule + whose action equals the filter's default action, and the profiles Podman and CRI-O ship deny by + default and then spell out denials for privileged syscalls — so 59 rules were redundant by + construction and each one was fatal. `EACCES` is now tolerated only when the rule's verdict equals + the default, which cannot change what the filter permits. +- **The integration suite resolves the runtime path before changing directory.** It had never passed + in CI: 96 of 118 assertions failed on a relative `MARS` that could not survive the `cd` into a + bundle. +- **The OOM test denies swap, so the kill actually happens.** It set `memory.limit` but not + `memory.swap`, and the `awk` loop writes each page once and never touches it again — so on a host + with swap the kernel satisfied `memory.max` by spilling cold pages and killed nothing. The test + hung for nineteen minutes in CI before it was noticed. + +### Changed + +- Renamed from `mars-container-runtime` to `marstack-container`, and moved to the `MarStack-Labs` + organisation alongside `marstack-cloud`, `marstack-secrets` and `marstack-access`. GitHub redirects + the old paths, but the `Cargo.toml` `repository` field was already stale and now points at the new + one. + +### Not implemented + +Rootless without any privilege. The user namespace machinery works and is tested, but `mars` still +expects to be started with privilege; a fully rootless run also needs a delegated cgroup under +`user.slice`, `fuse-overlayfs` or `userxattr` for whiteouts, and `slirp4netns` for networking. + +### Out of scope + +Image pulling from registries, CNI networking, checkpoint and restore, CRI, cgroup v1, the systemd +cgroup driver, SELinux and AppArmor labels (parsed and ignored rather than silently claimed), and +`SCMP_ACT_NOTIFY`. diff --git a/CODEOWNERS b/CODEOWNERS new file mode 100644 index 0000000..36934bc --- /dev/null +++ b/CODEOWNERS @@ -0,0 +1 @@ +* @umars28 diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..e8cf5f0 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,48 @@ +# Code of conduct + +## The short version + +Be decent. Argue about the code, not the person writing it. + +## What is expected + +- Assume the other person is competent and acting in good faith, especially when they are wrong. +- Criticise the change, the design, or the claim. Not the author. +- Say what you mean plainly. Being direct is not being unkind; being vague to seem polite wastes + everyone's time and usually reads worse anyway. +- When you are wrong, say so and move on. No performance of contrition required. +- English is not everyone's first language. Read past the phrasing to the point. + +## What is not + +- Harassment, insults, or personal attacks, public or private. +- Sexualised language or imagery, or unwelcome attention of any kind. +- Publishing anyone's private information without permission. +- Sustained disruption of discussion, or bad-faith argument to exhaust rather than persuade. +- Encouraging or excusing any of the above. + +## Reporting + +Report anything that concerns you to **usabirin26@dev1.idserve.net**. Reports are read by the +maintainer and kept confidential. You will get a reply. + +If your report concerns the maintainer, say so — you are entitled to raise it with GitHub Support +instead, and that is a reasonable thing to do rather than a hostile one. + +## Enforcement + +The maintainer decides what happens: a private word, a public correction, removing a comment, or a +ban from the project. Which one depends on severity and on whether it keeps happening, not on how +useful the person's contributions are. + +Enforcement will be applied to the maintainer on the same terms. + +## Scope + +Issues, pull requests, discussions, commit messages, and any space where someone is representing +this project. + +## Attribution + +Written for this project rather than adapted from a template, but it owes the obvious debt to the +[Contributor Covenant](https://www.contributor-covenant.org/). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..1d26550 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,107 @@ +# Contributing + +Thanks for taking the time. This is pre-1.0 and not production software, so the shape of things can +still change — if you are about to spend real effort, open an issue first and we can agree on the +approach before you write it. + +This project exists to understand the layer Docker hides, which changes what a finished change looks +like. The bar is not "it works". It is "here is the kernel state that proves it works, and here is +the assertion that reads it". + +## Getting set up + +Rust 1.85 or newer, `libseccomp`, and a Linux kernel with a pure cgroup v2 hierarchy. Nothing else. + +Linux only. It needs namespaces, cgroups and `libseccomp`, and it does not build on macOS. A +[Lima](https://lima-vm.io) VM definition is included so the environment is reproducible. + +```sh +git clone git@github.com:MarStack-Labs/marstack-container.git +cd marstack-container + +limactl start --name=mars-dev ./lima/mars-dev.yaml +limactl shell mars-dev + +make preflight # check the host before blaming the build +make check # the gate: fmt, clippy with -D warnings, unit tests +make integration # 128 assertions against kernel state, needs root +``` + +`make preflight` first, always. Most of what goes wrong is the host rather than the code, and the +one that stops people is **a VPS that is itself a container** — OpenVZ, LXC and most budget plans +share the provider's kernel, which blocks `pivot_root` and cgroup delegation. No amount of `sudo` +changes that. + +## What a change looks like + +**An assertion reads the kernel, not the runtime.** The integration suite checks `/proc`, +`/proc/mounts`, `/sys/fs/cgroup`, `ip -o link` and wait statuses. A test that asserts on what `mars` +printed about itself proves only that `mars` is self-consistent. + +**A change to behaviour comes with a test that fails without it.** The check that works: break the +rule you just wrote — invert the condition, delete the guard, reverse the ordering — and see whether +a test goes red. If none does, the test describes the code rather than holding it to anything. + +That is not a slogan here. Three separate measurement faults in `scripts/hap-bench.sh` each produced +a plausible, wrong number before an assertion caught it, and the numbers were wrong in the flattering +direction every time. Two integration bugs had been hiding each other for months: a relative runtime +path made 96 assertions fail identically, and fixing that revealed an OOM test that hung for +nineteen minutes because it never denied swap. + +**A test that cannot fail is worse than no test.** If a runtime exits non-zero, or a process never +appears where it should, say so and discard the run rather than recording a number from it. + +**Refuse rather than guess.** When a `config.json` is ambiguous or the host is not what was expected, +say no and say which check refused. A silent fallback becomes someone else's incident. + +**Error messages are for the person reading them at 3am.** Say what happened and what they can do +about it. `rootfs /tmp/x/rootfs does not exist or is not a directory` names the path; that is the +minimum. + +**No comments.** Explain a trap in the commit body or in `docs/`, where it is read by someone +deciding whether to trust this thing rather than only by someone already inside the file. + +## Things that will be turned down + +Not because they are bad ideas, but because they are decisions this project has already made. Each +is argued in the README under "Scope": + +- Image pulling from registries — containerd's job; the runtime is called after the bundle exists +- CNI networking — the namespace is created, populating it belongs to a plugin +- CRI — the kubelet interface sits a layer above an OCI runtime +- cgroup v1 or the systemd cgroup driver — a second driver doubles the surface for no insight +- Checkpoint and restore — a project of its own + +A dependency is also a decision. The cgroup driver and the OTLP exporter are hand-written on +purpose: one because delegating it would delegate away the point, the other because the SDK runs a +background thread and a process that forks must not have one. Adding a dependency needs a sentence +on the work it removes, and that sentence has to survive comparison with the standard library. + +## Commits + +Conventional commits — `feat:`, `fix:`, `docs:`, `test:`, `build:`, `perf:`, `refactor:`, `chore:`, +`style:` — with an optional scope like `fix(seccomp):`. Subject in the imperative, under 72 +characters. + +The body says **why**, and names any trap a future reader would otherwise hit. If you found a bug +while writing the change, say how you found it; that is often more useful than the fix. If your first +diagnosis was wrong, say that too — the seccomp fix in this repository has a commit body that records +a wrong reading, because the wrong reading was the plausible one. + +Keep unrelated changes in separate commits. + +## Documentation + +Anything that changes what a release contains goes in [CHANGELOG.md](CHANGELOG.md) under +`Unreleased`. + +A change to a security property changes [SECURITY.md](SECURITY.md) in the same commit — including +when the change makes the document *less* flattering. A silently-dropped guarantee is worse than a +documented gap. + +A phase writeup in `docs/` is where reasoning lives. If you learned something from the kernel that +took a day to work out, that day is worth more written down than the diff is. + +## Reporting a vulnerability + +Do not open a public issue. See [SECURITY.md](SECURITY.md). diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..f392494 --- /dev/null +++ b/Makefile @@ -0,0 +1,58 @@ +VERSION ?= 0.0.1-dev +PREFIX ?= /usr/local + +ARCH := $(shell uname -m | sed -e s/x86_64/amd64/ -e s/aarch64/arm64/) +SHA256 := $(shell command -v sha256sum >/dev/null 2>&1 && echo "sha256sum" || echo "shasum -a 256") + +.PHONY: build install uninstall dist test integration validation bench \ + fmt fmt-check clippy preflight check run clean + +build: + cargo build --release + +install: build + install -d $(DESTDIR)$(PREFIX)/bin + install -m 0755 target/release/mars $(DESTDIR)$(PREFIX)/bin/mars + +uninstall: + rm -f $(DESTDIR)$(PREFIX)/bin/mars + +dist: build + rm -rf dist && mkdir -p dist + cp target/release/mars dist/mars + tar -czf dist/mars_$(VERSION)_linux_$(ARCH).tar.gz -C dist mars -C .. LICENSE README.md + rm dist/mars + cd dist && $(SHA256) *.tar.gz > SHA256SUMS + +test: + cargo test --lib + +integration: build + cargo build + sudo -E ./tests/run-integration.sh + +validation: install + sudo -E ./scripts/run-validation.sh + +bench: install + sudo -E ./scripts/hap-bench.sh + +fmt: + cargo fmt + +fmt-check: + cargo fmt --check + +clippy: + cargo clippy --all-targets -- -D warnings + +preflight: + ./scripts/preflight.sh + +check: fmt-check clippy test + +run: build + sudo -E target/release/mars --help + +clean: + rm -rf target dist diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..65f0980 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,53 @@ +# Security policy + +## Supported versions + +None. `marstack-container` is pre-release, has no tagged releases, and is not production software — +that is its stated purpose, not modesty about the code. It exists to build a model of the runtime +layer. Do not run it anywhere that matters. + +## Reporting a vulnerability + +Open a private security advisory through GitHub on this repository. Please do not open a public +issue for a vulnerability. + +Include what you did, what happened, and what you expected. A reproduction against a local bundle is +the most useful thing you can send: a `config.json` and the command you ran beats a description. + +Findings that let a process reach the host filesystem outside its rootfs, keep a capability the spec +dropped, escape the cgroup it was placed in, execute a syscall the seccomp profile denies, or run as +a uid the user namespace did not map are the highest severity in this project. + +## Where the privilege actually is + +A runtime holds `CAP_SYS_ADMIN` for the whole of container creation and then exits. Everything +dangerous it can do, it does in that window, which is why the size of that window is measured rather +than assumed: [`docs/attack-surface.md`](docs/attack-surface.md) counts the distinct host kernel +functions traversed while privileged, next to `runc` and `crun`, and states what the number cannot +tell you. + +The isolation and hardening decisions themselves are argued in +[`docs/01-isolation.md`](docs/01-isolation.md) and +[`docs/05-hardening.md`](docs/05-hardening.md) — including the orderings the kernel enforces, which +broke twice before they were written down. + +## What is not covered + +- **Rootless without privilege is unfinished.** The user namespace machinery works and is tested, + but the runtime still expects to be started with privilege. Treat any claim of rootless safety as + unproven here. +- **SELinux and AppArmor labels are parsed and ignored,** not applied. A bundle asking for a label + gets no error and no label. This is recorded rather than hidden because a silently-dropped label + is worse than a refused one. +- **`SCMP_ACT_NOTIFY` is refused,** since it needs a listener process to receive the notification + descriptor. +- **No CNI networking.** The network namespace is created and left empty. +- **The attack surface figures are one architecture, one kernel, and a guest kernel** rather than + bare metal. The comparison between runtimes holds because all three meet identical conditions; the + absolute numbers do not travel. + +## What runs on every commit + +`cargo clippy` with warnings as errors, the unit tests, an integration suite of 128 assertions that +read kernel state rather than the runtime's own claims, and the OCI validation suite against both +`mars` and `runc` for comparison.