Skip to content

feat(kubernetes): add sidecar supervisor topology#2076

Open
TaylorMutch wants to merge 22 commits into
mainfrom
feat/kubernetes-sidecar-topology-v2
Open

feat(kubernetes): add sidecar supervisor topology#2076
TaylorMutch wants to merge 22 commits into
mainfrom
feat/kubernetes-sidecar-topology-v2

Conversation

@TaylorMutch

@TaylorMutch TaylorMutch commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds an experimental Kubernetes sidecar supervisor topology after the combined-topology base from #2074. The default combined topology remains unchanged; sidecar is opt-in and may change as we continue validating the permission model, pod layout, and configuration surface across Kubernetes runtimes.

The goal of sidecar is to move pod-level network enforcement, gateway authentication, gateway session handling, and SSH relay ownership out of the agent container and into a dedicated network sidecar. The agent container runs as the resolved sandbox UID/GID with runAsNonRoot, allowPrivilegeEscalation: false, and all Linux capabilities dropped.

Outcome

  • Keeps combined as the default topology and full OpenShell enforcement path.
  • Adds topology = "sidecar" under [openshell.drivers.kubernetes] and nests sidecar-only settings under [openshell.drivers.kubernetes.sidecar]; Helm renders the same shape from supervisor.topology and supervisor.sidecar.*.
  • Splits sidecar-mode pods into a root network init container, a non-root network supervisor sidecar, and a low-privilege agent container.
  • Enables shareProcessNamespace: true for sidecar pods so the network sidecar can resolve workload process and binary identity through /proc.
  • Keeps process/binary-aware network policy enabled by default in sidecar mode. Setting supervisor.sidecar.processBinaryAwareNetworkPolicy=false drops the sidecar SYS_PTRACE capability and downgrades sidecar network policy to endpoint/L7 enforcement without matching policy.binaries.
  • Moves gateway credentials and gateway callback/session ownership into the network sidecar. The agent container receives policy and provider environment state over the sidecar control socket instead of mounting gateway authentication material.
  • Adds sidecar Helm overlays, Skaffold profiles, docs, and a branch Kubernetes E2E matrix entry for the sidecar topology.
  • Adds deploy/helm/openshell/ci/values-sidecar-kata.yaml as the local/CI-dev overlay used for Kata validation.

Enforcement Model

combined remains the complete OpenShell sandbox contract: network policy, filesystem policy, process controls, supervisor privilege drop, supervisor identity mount isolation, gateway relay, SSH, exec, and file sync all run through the existing agent-container supervisor path.

sidecar is network-focused but no longer disables all process-side enforcement:

  • Network endpoint and L7 policy are enforced by the network sidecar.
  • Process/binary-aware network policy is enforced by default through the shared PID namespace and sidecar /proc inspection.
  • The process supervisor in the agent container still launches the workload and applies Landlock filesystem policy and child seccomp filters where the kernel/runtime supports them.
  • The process supervisor does not perform root-to-sandbox privilege dropping in sidecar mode because Kubernetes starts the agent container directly as the sandbox UID/GID.
  • Supervisor identity mount isolation is not applied in the agent container because gateway credentials are not mounted there in the default sidecar path.

Runtime Validation Status

  • GKE Autopilot: sidecar mode has been validated on Autopilot clusters created with gcloud container clusters create-auto --workload-policies=allow-net-admin; the Google Cloud CLI reference lists allow-net-admin as the supported Autopilot workload policy: https://docs.cloud.google.com/sdk/gcloud/reference/container/clusters/create-auto#--workload-policies
  • Kata Containers: sidecar mode has been validated on our Kata-enabled Kubernetes cluster using branch-built gateway/supervisor images and deploy/helm/openshell/ci/values-sidecar-kata.yaml. This validation passed without using the supervisor sideload init container.
  • gVisor: this PR does not claim gVisor support. The sidecar topology still depends on pod-local network rule setup, which was previously observed to fail under gVisor.

Related Issue

References #1827, #981, #899, #1305.

Related PRs: #1973, #2074, #2016.

Changes

  • Include prerequisite sandbox UID/GID support: numeric sandbox identities, configurable Kubernetes sandbox UID/GID values, and OpenShift SCC annotation fallback.
  • Add the experimental sidecar topology and sidecar-specific configuration.
  • Rename Kubernetes driver supervisor_topology to topology and move sidecar settings under openshell.drivers.kubernetes.sidecar.
  • Render sidecar-mode sandbox pods with a network init container, non-root supervisor sidecar, shared PID namespace, and unprivileged agent container.
  • Add sidecar control-socket bootstrap for policy/provider environment state so the agent container does not need gateway credentials.
  • Add process-supervisor network-only behavior for sidecar mode while preserving workload launch, Landlock/seccomp application, and SSH/session relay behavior.
  • Add a sidecar flag to disable process/binary-aware network policy and drop the sidecar SYS_PTRACE capability when users accept the downgrade.
  • Add IPv4-only fallback handling for sidecar network setup so Kata clusters without IPv6 support can start sidecar pods.
  • Add sidecar and sidecar-Kata Helm overlays plus Skaffold profile support.
  • Add a sidecar topology row to the branch Kubernetes E2E matrix.
  • Document topology choice, architecture diagrams, permission model, RuntimeClass validation status, and sidecar tradeoffs.

Testing

  • git diff --check origin/main..feat/kubernetes-sidecar-topology-v2
  • cargo check -p openshell-driver-kubernetes -p openshell-sandbox -p openshell-supervisor-process -p openshell-supervisor-network
  • cargo test -p openshell-driver-kubernetes --lib
  • cargo test -p openshell-supervisor-process --lib
  • mise run helm:test
  • markdownlint-cli2 docs/kubernetes/topology.mdx docs/kubernetes/setup.mdx
  • ruby -e 'require "yaml"; YAML.load_file(".github/workflows/branch-e2e.yml"); puts "ok"'
  • mise run rust:lint
  • CI container: mise run rust:format:check && mise run rust:lint
  • CI container: cargo test -p openshell-supervisor-process
  • CARGO_BUILD_JOBS=1 mise run pre-commit
  • GitHub OpenShell / Branch Checks passing on this branch, including Linux amd64 and arm64 Rust jobs.
  • Prior validation from feat(kubernetes): add sidecar and proxy-pod topology configurations #2016: HELM_K3S_LB_HOST_PORT=18080 mise run e2e:kubernetes:sidecar
  • Sidecar topology validated on GKE Autopilot using --workload-policies=allow-net-admin.
  • Sidecar topology validated on a Kata-enabled Kubernetes cluster with branch-built images and deploy/helm/openshell/ci/values-sidecar-kata.yaml; validation passed without the supervisor sideload init container.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@copy-pr-bot

copy-pr-bot Bot commented Jun 30, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

@TaylorMutch TaylorMutch added the test:e2e Requires end-to-end coverage label Jun 30, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/2076 is at {"messa while the PR head is 6276de3. A maintainer needs to comment /ok to test 6276de38ac92b65b0b24a68b8d09a5eed06f50d7 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

@TaylorMutch TaylorMutch marked this pull request as ready for review June 30, 2026 22:46
@TaylorMutch TaylorMutch force-pushed the feat/kubernetes-sidecar-topology-v2 branch 2 times, most recently from 4f981c2 to 8c9ae53 Compare July 2, 2026 21:12
@mrunalp

mrunalp commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Some comments with help of agent

1. Central trust boundary — the workload can use the sandbox's gateway identity .
Client mTLS and the SA token render 0440 with fsGroup=sandbox_gid, the process leaf skips supervisor-identity isolation in network-only, and the 127.0.0.1:18080 forward is always open. Since the workload child also runs with sandbox_gid, it can read those credentials and reach the gateway directly.

2. nftables fence only covers TCP/UDP.
The output chain is policy accept and only rejects TCP/UDP from non-proxy_uid UIDs. ICMP (e.g. unprivileged ping sockets) and other L4 protocols fall through. CAP_NET_RAW is dropped so raw sockets aren't available, but this is a narrower egress boundary than combined mode's per-netns setup. Low severity — suggest a brief comment in generate_sidecar_bypass_ruleset noting the intentional scope.

3. processEnforcement=full does not restore network binary-identity.
supervisor_sidecar_env hard-codes NETWORK_BINARY_IDENTITY=relaxed regardless of process_enforcement, so full restores agent-container Landlock/privilege-drop but network policy stays endpoint/L7-scoped. The docs are technically accurate ("combined-mode process/filesystem controls"), but a reader could reasonably infer full ≈ combined. A one-line note that network policy remains endpoint-scoped in sidecar mode regardless of processEnforcement would prevent that.

4. Doc drift in the debug skill.
debug-openshell-cluster/SKILL.md says the sidecar runs "with primary GID 0" to read the token, but the implementation uses runAsGroup: sandbox_gid + fsGroup: sandbox_gid (which re-owns the projected volume to sandbox_gid). Please update to match.

@TaylorMutch TaylorMutch force-pushed the feat/kubernetes-sidecar-topology-v2 branch 2 times, most recently from e00a00b to ed858eb Compare July 6, 2026 18:52
@TaylorMutch

Copy link
Copy Markdown
Collaborator Author
  1. Central trust boundary — the workload can use the sandbox's gateway identity .
    Client mTLS and the SA token render 0440 with fsGroup=sandbox_gid, the process leaf skips supervisor-identity isolation in network-only, and the 127.0.0.1:18080 forward is always open. Since the workload child also runs with sandbox_gid, it can read those credentials and reach the gateway directly.

Addressed this by moving the supervisor and sandbox to run with different GIDs.

Updated the other areas with clarifying comments.

@mrunalp

mrunalp commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Deployed this branch on an OpenShift 4.22 cluster (K8s 1.35) with supervisor.topology=sidecar and TLS + mTLS enabled (default PKI init job). The sidecar pod renders correctly, but openshell-network-init is stuck in CrashLoopBackOff:

Error:   × failed to create sidecar directory /etc/openshell-tls/proxy/client
  ╰─▶ Permission denied (os error 13)
$ oc -n openshell get pod test-sidecar
NAME           READY   STATUS                  RESTARTS
test-sidecar   0/2     Init:CrashLoopBackOff   6

Root cause

In run_network_init (crates/openshell-sandbox/src/main.rs) the ordering is:

  1. prepare_sidecar_directory(sidecar_tls_dir /* /etc/openshell-tls/proxy */, proxy_uid, proxy_gid, 0o755) — this chowns the TLS dir to proxy_uid (e.g. 1337), mode 0755.
  2. copy_sidecar_client_tls_if_present(...)prepare_sidecar_directory(dest_dir /* …/proxy/client */, …)create_dir_all.

Step 2 runs as root with capabilities.drop: ["ALL"] (the init container only adds NET_ADMIN, NET_RAW, CHOWN, FOWNER, so it has no CAP_DAC_OVERRIDE). The parent dir is now owned by uid 1337, mode 0755, so root is neither the owner nor has group-write → EACCES. Root can no longer bypass DAC checks because DAC_OVERRIDE was dropped.

Note the state dir was already changed to 0o2775 (setgid + group-write), but the TLS dir is still 0o755, and it's the one that gets a subdirectory created in it right after being chowned away from root.

@TaylorMutch

Copy link
Copy Markdown
Collaborator Author

@mrunalp I reproduced locally and pushed a fix. Please have a look when you get a chance!

Add the Kubernetes sidecar supervisor topology, its Helm/Skaffold configuration, topology documentation, and sidecar e2e matrix coverage. Skip root-only sandbox identity rewriting when process enforcement is network-only so the low-permission sidecar process container can start successfully.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@TaylorMutch TaylorMutch force-pushed the feat/kubernetes-sidecar-topology-v2 branch from 3d80066 to 254cdbe Compare July 7, 2026 04:01
# staging directory before the image build job runs.

FROM scratch AS supervisor
FROM alpine:3.22 AS supervisor

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making a note that this might break the podman implementation which expects a scratch container. I might be wrong here though. I would expect E2E tests to shake this out.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e2e tests seem to pass with this change; open to other options if this is a blocker

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have to test. It's going to depend more on the output of the build than the base. Basically the way the podman mount works is it mounts the entire filesystem of the image to the mount point inside the instantiated container runtime and the supervisor binary needs to live at the root of that.

Comment thread deploy/helm/openshell/templates/gateway-config.yaml Outdated
Comment thread e2e/with-kube-gateway.sh Outdated
Comment on lines +401 to +410
case "${selected_engine}" in
docker|podman)
export CONTAINER_ENGINE="${selected_engine}"
return
;;
*)
echo "ERROR: CONTAINER_ENGINE=${CONTAINER_ENGINE} is invalid; expected docker or podman" >&2
exit 2
;;
esac

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is there a docker branch here?

@drew

drew commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

This draft intentionally includes the prerequisite UID/GID commits from #1973 for now. We expect to rebase those out after that work lands separately.

Looks like you've rebased now since that's been merged?

Comment on lines +32 to +76
/// Supervisor pod/runtime topology. Kubernetes sidecar mode sets this to
/// `"sidecar"`; the default combined supervisor path omits it.
pub const SUPERVISOR_TOPOLOGY: &str = "OPENSHELL_SUPERVISOR_TOPOLOGY";

/// Network enforcement backend selected by the compute driver.
pub const NETWORK_ENFORCEMENT_MODE: &str = "OPENSHELL_NETWORK_ENFORCEMENT_MODE";

/// Process enforcement mode selected by the compute driver.
///
/// The default when unset is `"full"`, where the process supervisor enforces
/// filesystem/process policy before spawning workloads. Kubernetes sidecar
/// topology sets this to `"network-only"` so the process wrapper can run as
/// the sandbox UID without Linux capabilities while preserving SSH/session
/// behavior.
pub const PROCESS_ENFORCEMENT_MODE: &str = "OPENSHELL_PROCESS_ENFORCEMENT_MODE";

/// Whether network policy evaluation must bind requests to the peer binary.
///
/// The default when unset is `"required"`. Kubernetes sidecar experiments may
/// set this to `"relaxed"` to enforce endpoint and L7 policy without per-binary
/// `/proc` identity binding.
pub const NETWORK_BINARY_IDENTITY: &str = "OPENSHELL_NETWORK_BINARY_IDENTITY";

/// File written by the network supervisor when sidecar networking is ready.
pub const SUPERVISOR_READY_FILE: &str = "OPENSHELL_SUPERVISOR_READY_FILE";

/// File written by the process supervisor with the workload entrypoint PID and
/// read by the network sidecar for process/binary-bound network policy checks.
pub const ENTRYPOINT_PID_FILE: &str = "OPENSHELL_ENTRYPOINT_PID_FILE";

/// Local protobuf policy snapshot written by the network sidecar and read by
/// the process-only supervisor in Kubernetes sidecar topology.
pub const SIDECAR_POLICY_SNAPSHOT_FILE: &str = "OPENSHELL_SIDECAR_POLICY_SNAPSHOT_FILE";

/// Local provider environment snapshot written by the network sidecar and read
/// by the process-only supervisor in Kubernetes sidecar topology.
pub const SIDECAR_PROVIDER_ENV_SNAPSHOT_FILE: &str = "OPENSHELL_SIDECAR_PROVIDER_ENV_SNAPSHOT_FILE";

/// Optional TLS server name override used when connecting to the gateway.
pub const GATEWAY_TLS_SERVER_NAME: &str = "OPENSHELL_GATEWAY_TLS_SERVER_NAME";

/// Directory where the network supervisor writes the proxy CA files consumed
/// by workload child processes.
pub const PROXY_TLS_DIR: &str = "OPENSHELL_PROXY_TLS_DIR";

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do these constants need to be in core?

@TaylorMutch TaylorMutch Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They don't; I'll clean this up. It looks like this sandbox_env is all the configuration that's used by the supervisor, and the config for this isn't located in the driver. Not sure how best to clean this up.

Comment on lines +488 to +492
install_sidecar_iptables_legacy_bypass_rules(proxy_uid).map_err(|iptables_error| {
miette::miette!(
"sidecar nft ruleset load failed: {nft_error}; sidecar iptables-legacy fallback failed: {iptables_error}"
)
})

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(from agent review)

When nftables loading fails on a dual-stack pod, this fallback reports success after installing only IPv4 iptables-legacy rules. IPv6 TCP/UDP traffic remains unrestricted, allowing workloads to bypass the sidecar proxy and network policy entirely. Install equivalent ip6tables-legacy rules or fail closed when IPv6 cannot be fenced.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@TaylorMutch TaylorMutch requested a review from drew July 7, 2026 17:51
Run nftables setup as individual commands so optional conntrack and log expressions can fail without rolling back required table, chain, and reject rules.

Signed-off-by: Seth Jennings <sjenning@redhat.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@mrunalp

mrunalp commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Discussed with @TaylorMutch:

Suggestion: keep Landlock (and seccomp) in sidecar network-only mode — they don't need privilege

In sidecar topology the process leaf runs network-only, which currently disables Landlock, seccomp, and filesystem policy along with the genuinely privileged controls:

// crates/openshell-supervisor-process/src/process.rs:591
let prepared_sandbox = if enforcement_mode.enforces_process_controls() {
    Some(sandbox::linux::prepare(policy, workdir)?)   // Landlock ruleset prep
} else { None };

enforces_process_controls() is false for sidecar, so prepare() is skipped and no Landlock ruleset is built — same in the ssh.rs spawn paths. But most of what's gated here doesn't actually require privilege, so we're relaxing more than we need to.

Landlock is unprivileged by design

landlock_create_ruleset / add_rule / restrict_self need no capabilities — Landlock only ever removes access, so any process may self-restrict. The single precondition for restrict_self is no_new_privs, and we already set that unconditionally:

  • harden_child_process()set_no_new_privs() (sandbox/linux/seccomp.rs:126) runs in the child pre-exec outside the enforces_process_controls() gate (e.g. ssh.rs:1266), so it fires in sidecar mode too.

So the kernel prerequisite for Landlock is already satisfied in the capless agent container. It's gated off only incidentally:

  1. Bundlingenforces_process_controls() lumps Landlock with drop_privileges (setuid/setgid — needs root) and the supervisor-identity mount isolation (needs CAP_SYS_ADMIN/userns). Landlock got swept up with controls that genuinely need privilege.
  2. "Prepare as root"prepare() opens Landlock PathFds "as root before drop_privileges" (sandbox/linux/mod.rs:15,21) so it can open mode-700 paths. That's an optimization for the root→user transition, not a Landlock requirement.

What it would take

  • Ungate Landlock (and seccomp) from enforces_process_controls() so they apply even when we're not dropping privileges.
  • Open the path FDs as the sandbox UID in the sidecar path. Paths that UID can't open are ones the workload can't reach anyway — skip them / treat open-failure as already-denied. "Open as root" is unnecessary when the workload is that UID.
  • Keep the existing kernel-availability probe (log_sandbox_readiness) and degrade gracefully where Landlock/ABI is unavailable.

Seccomp is the same: filters install unprivileged under no_new_privs, so the syscall hardening can also run in the sidecar agent.

What legitimately stays out — and why it doesn't matter here

  • Privilege dropping (root → sandbox uid): needs root, but in sidecar mode Kubernetes already starts the container as runAsUser=sandbox_uid, runAsNonRoot: true. Nothing to drop.
  • Supervisor-identity mount isolation (hiding the SA token): needs CAP_SYS_ADMIN. Moot in sidecar mode — the credential redesign already keeps gateway creds out of the agent container (snapshot brokering).

Net effect

Re-enabling Landlock + seccomp shrinks the sidecar's "relaxed filesystem/process controls" tradeoff down to essentially just "network binary-identity is relaxed." The agent container stays capless and non-root, but the workload regains filesystem and syscall confinement.

Caveat: kernel availability

Landlock must be compiled in (CONFIG_SECURITY_LANDLOCK) and present in the active LSM list. RHEL 9 / RHCOS historically shipped without it enabled (added in later minors), so on OpenShift it may be unavailable even after ungating — in which case the availability probe should simply no-op it (as it does today), while it engages on upstream/newer kernels (k3d, GKE, etc.). Worth ungating regardless: it applies wherever the kernel supports it and degrades cleanly where it doesn't. Quick check on a target node: cat /sys/kernel/security/lsm should list landlock.

Render sidecar pods with a shared process namespace, keep binary-aware network policy enabled, and move Kubernetes sidecar settings under the nested sidecar config table.

Also apply unprivileged Landlock/seccomp setup in NetworkOnly supervisor mode so sidecar topology keeps sandbox child hardening without privileged process setup.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Coordinate sidecar policy and provider bootstrap over a local Unix socket so the process leaf no longer reads policy/provider snapshot files.

Report entrypoint startup through the control channel and keep gateway credentials confined to the network sidecar.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@TaylorMutch

Copy link
Copy Markdown
Collaborator Author

@mrunalp addressed in cd0477f - thank you for calling it out!

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants