Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions client/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ apiVersion: v2
name: client
description: A unified Helm chart for tracebloc on AKS, EKS, bare-metal, and OpenShift
type: application
version: 1.9.95
appVersion: "1.9.95"
version: 1.9.96
appVersion: "1.9.96"
keywords:
- tracebloc
- kubernetes
Expand Down
3 changes: 2 additions & 1 deletion client/templates/jobs-manager-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -663,7 +663,8 @@ spec:
{{- if and (default dict .Values.egressProxy).enabled (default dict .Values.egressProxy).routeWorkloads }}
# client-runtime#102: when routing is enabled, jobs-manager injects
# HTTPS_PROXY=<gateway> into training pods (and suppresses raw HTTP_PROXY_HOST).
# Inert until egressProxy.routeWorkloads=true.
# routeWorkloads defaults true (RFC-0003 D6), so this renders by default;
# set egressProxy.routeWorkloads=false to opt a fleet out of gateway routing.
- name: EGRESS_PROXY_URL
value: "http://egress-proxy-service:{{ (default dict .Values.egressProxy).port | default 3128 }}"
{{- end }}
Expand Down
11 changes: 6 additions & 5 deletions client/templates/network-policy-training.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -61,11 +61,12 @@ spec:
protocol: UDP
- port: 53
protocol: TCP
{{- /* Rule 2 (external HTTPS) is the egress hole SECURITY §8.2 closes. Gated on
networkPolicy.training.allowExternalHttps via `dig` with a default of TRUE:
an absent key (helm upgrade --reuse-values from a release predating it)
keeps the rule — old behaviour — so only an explicit `false` drops it, once
an operator has verified the egress gateway on that cluster (#102). */}}
{{- /* Rule 2 (external HTTPS) is the egress hole SECURITY §8.2 closes. The
shipped default is now `false` (values.yaml — RFC-0003 D6 deny-by-default),
so a fresh render drops this rule. The `dig` default of TRUE here is the
legacy fallback ONLY: an absent key (helm upgrade --reuse-values from a
release predating the key) keeps the rule — old permissive behaviour — and
an operator can still opt a fleet back out with an explicit `true` (#102). */}}
{{- if dig "allowExternalHttps" true .Values.networkPolicy.training }}
# 2. External HTTPS — everything NOT in the cluster's pod/service CIDRs.
# Training pods reach the backend / Azure Service Bus / App Insights directly.
Expand Down
16 changes: 15 additions & 1 deletion client/tests/egress_enforcement_check_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,21 @@ set:
clientId: "test-id"
clientPassword: "test"
tests:
- it: does NOT render by default (lockdown off — allowExternalHttps defaults true)
- it: renders by default (lockdown on — allowExternalHttps defaults false)
asserts:
- hasDocuments:
count: 1
Comment thread
cursor[bot] marked this conversation as resolved.

# Opt-out coverage: with lockdown now the default, an operator who sets
# allowExternalHttps=true has re-opened direct egress, so the seal-check has
# nothing to verify and must NOT render. This is the assertion the inverted
# default-render case above used to carry (Bugbot on this PR) — without it a
# regression that still emits the Job after opt-out would stay green.
- it: does NOT render when an operator opts back out (allowExternalHttps=true)
set:
networkPolicy:
training:
allowExternalHttps: true
asserts:
- hasDocuments:
count: 0
Expand Down
20 changes: 10 additions & 10 deletions client/tests/egress_proxy_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -152,22 +152,22 @@ tests:

# --- Step 1: routing training pods through the gateway ---

- it: jobs-manager gets EGRESS_PROXY_URL only when routeWorkloads is enabled
- it: jobs-manager gets EGRESS_PROXY_URL by default (routeWorkloads defaults true)
template: templates/jobs-manager-deployment.yaml
documentIndex: 0
set:
egressProxy:
routeWorkloads: true
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: EGRESS_PROXY_URL
value: "http://egress-proxy-service:3128"

- it: jobs-manager does NOT get EGRESS_PROXY_URL by default (routeWorkloads false)
- it: jobs-manager omits EGRESS_PROXY_URL when routeWorkloads is disabled
template: templates/jobs-manager-deployment.yaml
documentIndex: 0
set:
egressProxy:
routeWorkloads: false
asserts:
- notContains:
path: spec.template.spec.containers[0].env
Expand Down Expand Up @@ -197,10 +197,10 @@ tests:

# --- Step 3: the lockdown — drop the external 0.0.0.0/0:443 rule ---

- it: keeps the external 443 egress rule by default (allowExternalHttps true)
- it: drops the external 443 egress rule by default (allowExternalHttps defaults false)
template: templates/network-policy-training.yaml
asserts:
- contains:
- notContains:
path: spec.egress
content:
to:
Expand All @@ -214,14 +214,14 @@ tests:
- port: 443
protocol: TCP

- it: drops the external 443 rule but keeps the gateway path when allowExternalHttps is false
- it: renders the external 443 rule and keeps the gateway path when allowExternalHttps is opted back to true
template: templates/network-policy-training.yaml
set:
networkPolicy:
training:
allowExternalHttps: false
allowExternalHttps: true
asserts:
- notContains:
- contains:
path: spec.egress
content:
to:
Expand Down
6 changes: 3 additions & 3 deletions client/tests/network_policy_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ tests:
networkPolicy:
training:
enabled: true
allowExternalHttps: true # pin the default — this asserts the rule-2 hole
allowExternalHttps: true # opt back out (default is now false) — asserts the rule-2 hole still renders on true
dnsNamespace: kube-system
dnsSelector:
k8s-app: kube-dns
Expand Down Expand Up @@ -266,8 +266,8 @@ tests:
port: 8888
protocol: TCP
# locked-down egress is exactly 4 rules: DNS + MySQL + requests-proxy +
# egress-proxy. NOTE: `egressProxy.enabled` DEFAULTS TRUE (values.yaml —
# only `routeWorkloads` defaults false), so the gateway's permit rule
# egress-proxy. NOTE: `egressProxy.enabled` and `routeWorkloads` both
# DEFAULT TRUE (values.yaml — RFC-0003 D6), so the gateway's permit rule
# renders here even though this case doesn't set it. The direct length
# assertion makes the shape unmissable and catches a stray appended rule
# (Saqlain review on #537).
Expand Down
8 changes: 4 additions & 4 deletions client/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -414,8 +414,8 @@
},
"allowExternalHttps": {
"type": "boolean",
"default": true,
"description": "When false, drop the 0.0.0.0/0:443 egress rule so training pods reach only DNS, MySQL, requests-proxy and the egress gateway (SECURITY \u00a78.2 / client-runtime#102). Default true keeps existing behaviour; flip per-fleet after verifying the egress gateway works (G2)."
"default": false,
"description": "When false (the default, RFC-0003 D6), drop the 0.0.0.0/0:443 egress rule so training pods reach only DNS, MySQL, requests-proxy and the egress gateway (SECURITY \u00a78.2 / client-runtime#102). Set true to opt a fleet back out to direct egress. An absent key on a `--reuse-values` upgrade from a release predating it keeps the old permissive behaviour."
},
"enforcementProbeHost": {
"type": "string",
Expand Down Expand Up @@ -683,8 +683,8 @@
},
"routeWorkloads": {
"type": "boolean",
"default": false,
"description": "Route training-pod outbound HTTPS through the gateway (jobs-manager injects HTTPS_PROXY). Default false \u2014 enable per-fleet, verify a run, then drop the direct egress rule (networkPolicy.training.allowExternalHttps=false)."
"default": true,
"description": "Route training-pod outbound HTTPS through the gateway (jobs-manager injects HTTPS_PROXY). Default true (RFC-0003 D6, deny-by-default); set false to opt a fleet out of gateway routing."
},
"port": {
"type": "integer",
Expand Down
21 changes: 12 additions & 9 deletions client/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -370,11 +370,12 @@ networkPolicy:
# Egress lockdown (SECURITY §8.2 / client-runtime#102). When false, the
# training NetworkPolicy DROPS the 0.0.0.0/0:443 rule, so training pods can
# reach only DNS, in-cluster MySQL, the requests-proxy, and the egress
# gateway. Default true keeps the fleet unchanged; flip OFF per-fleet AFTER
# verifying the egress gateway works on that cluster (G2). The template
# gateway. Shipped default is now FALSE — deny-by-default egress (RFC-0003 D6 /
# client-runtime#199): new installs are locked down, external HTTPS goes through
# the gateway allowlist. Set true only to opt a fleet back out. The template
# nil-guards this key, so a `helm upgrade --reuse-values` from a release
# predating it keeps the old behaviour (rule present).
allowExternalHttps: true
allowExternalHttps: false
# client-runtime#104: enforcement check, run via `helm test <release>` after
# flipping the lockdown (allowExternalHttps=false). A training-labelled pod opens a
# direct TCP connection to this host's :443 and RETRIES until it is blocked (PASS) or
Expand Down Expand Up @@ -480,10 +481,11 @@ sealCheck:
# so the training NetworkPolicy never selects it (it keeps its own egress).
egressProxy:
enabled: true
# Route training-pod outbound HTTPS through the gateway (Step 1 of #102). Default
# FALSE so the gateway ships inert; flip per-fleet to true, verify a real training
# run, THEN set networkPolicy.training.allowExternalHttps=false to drop direct egress.
routeWorkloads: false
# Route training-pod outbound HTTPS through the gateway (Step 1 of #102). Shipped
# default is now TRUE — deny-by-default egress (RFC-0003 D6 / client-runtime#199):
# training egress flows through the gateway allowlist, paired with
# networkPolicy.training.allowExternalHttps=false. Set false only to opt back out.
routeWorkloads: true
# squid image, pinned by multi-arch (amd64+arm64) index digest — tracebloc pins
# all images by digest. `tag` stays for readability; the digest is authoritative.
# ubuntu/squid:6.6-24.04_beta (Ubuntu 24.04 LTS base), resolved 2026-06-10.
Expand Down Expand Up @@ -594,8 +596,9 @@ egressProxy:
# DaemonSet on every customer node spooling to disk with nothing it can deliver —
# filling toward the 1 GiB cap per node for no benefit, which is precisely the
# "telemetry must never be the reason a node fills" risk D7 exists to bound.
# Same posture as `egressProxy.routeWorkloads`: the component ships inert and is
# flipped per fleet once its prerequisite is in place.
# Ships inert and is flipped per fleet once its prerequisite is in place — the
# posture `egressProxy.routeWorkloads` had before RFC-0003 D6 flipped it on by
# default.
#
# CLASS A ONLY (D12). The receiver is scoped to the control-plane containers this
# chart owns; training and ingestion pods are Class B and stay out until
Expand Down
21 changes: 12 additions & 9 deletions docs/SEAL-CHECK.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,9 @@ these):
(the storage check's ServiceAccount/RBAC) deliberately do not — counting
them would inflate the suite.
- A check that does not *render* (its gating values turned it off, or its
preconditions are not declared — e.g. the egress-enforcement probe before
the lockdown is flipped) is **not part of the suite on that cluster**, and
the values that gated it away say why.
preconditions are not declared — e.g. the egress-enforcement probe when an
operator has opted a fleet back out with `allowExternalHttps=true`) is **not
part of the suite on that cluster**, and the values that gated it away say why.
- Log lines are human-oriented and not part of the contract; the machine
contract today is *labels + Job exit status*. (A structured verdict is the
CLI's job — cli#393.)
Expand Down Expand Up @@ -155,19 +155,22 @@ on the live cluster when the corresponding check runs.*

| Guarantee | k3d local (k3s) | EKS | AKS | OpenShift | bare metal |
|---|---|---|---|---|---|
| Training egress blocked (NetworkPolicy) | **Substrate verified; full-probe run pending** — k3s enforces egress NetworkPolicy (k3d v5.8.3 / k3s v1.33.6+k3s1, 2026-07-30; see §8.4 Status), full-chart `egress-enforcement` probe run not yet recorded | **Substrate verified; full-probe run pending** — dev fleet `tb-client-dev-templates` runs the VPC CNI netpol agent (v1.2.7, `--enable-network-policy=true`, mode `standard`, 2026-08-24; see EKS Status below), full-chart `egress-enforcement` probe not recorded (per-fleet lockdown held — client-runtime#199). Other EKS CNIs (Calico / Cilium) — **verified** by `egress-enforcement` once the lockdown is flipped | Conditional on CNI (Azure NPM / Calico) — **verified** by `egress-enforcement` once the lockdown is flipped | OVN-Kubernetes enforces by default — still **verified** by `egress-enforcement` | Conditional on CNI (Flannel alone does not enforce) — **verified** by `egress-enforcement` |
| Training egress blocked (NetworkPolicy) | **Substrate verified; full-probe run pending** — k3s enforces egress NetworkPolicy (k3d v5.8.3 / k3s v1.33.6+k3s1, 2026-07-30; see §8.4 Status), full-chart `egress-enforcement` probe run not yet recorded | **Substrate verified; full-probe run pending** — dev fleet `tb-client-dev-templates` runs the VPC CNI netpol agent (v1.2.7, `--enable-network-policy=true`, mode `standard`, 2026-08-24; see EKS Status below), full-chart `egress-enforcement` probe run recording still in progress across fleets (client-runtime#199; deny-by-default is the chart default as of 1.9.96). Other EKS CNIs (Calico / Cilium) — **verified** by `egress-enforcement` (renders by default as of 1.9.96; opt-out with `allowExternalHttps=true`) | Conditional on CNI (Azure NPM / Calico) — **verified** by `egress-enforcement` (renders by default as of 1.9.96; opt-out with `allowExternalHttps=true`) | OVN-Kubernetes enforces by default — still **verified** by `egress-enforcement` | Conditional on CNI (Flannel alone does not enforce) — **verified** by `egress-enforcement` |
| Backend reachability (required egress) | **Verified** by `backend-reachability` | **Verified** | **Verified** | **Verified** | **Verified** |
| Storage on the declared class, bound | **Verified** by `storage-assertions` | **Verified** | **Verified** | **Verified** (PV scan degraded if `clusterScope=false`) | **Verified** |
| No unmanaged hostPath backing (dynamic mode) | **Verified** once the Option C flip lands (today's installer still declares hostPath mode → sub-check SKIPs, honestly) | **Verified** | **Verified** | **Verified** with `clusterScope=true`; partial (name check + explicit WARNING) otherwise | n/a — hostPath *is* the declared model (SKIP) |
| Nothing under `~/.tracebloc` on the host (post-Option-C) | Not observable in-cluster — CLI/installer-side check (see follow-ups) | n/a | n/a | n/a | n/a |

Two lockdown caveats the suite states rather than hides:

- `egress-enforcement` only *renders* after the per-fleet lockdown flip
(`allowExternalHttps=false` — the RFC-0003 §8.1 rollout). Until that flip,
training-pod outbound :443 is deliberately open and there is no
enforcement to verify — the environment is **not sealed for egress** and
nothing here claims it is.
- `egress-enforcement` renders **by default** as of chart 1.9.96
(`allowExternalHttps=false` is the shipped default — RFC-0003 D6), so a fresh
install seals training-pod outbound :443 and `helm test` runs this check. An
operator who opts a fleet back out (`allowExternalHttps=true`) re-opens direct
:443; the hook then does not render and there is no enforcement to verify —
that fleet is **not sealed for egress** until the lockdown is restored. (Charts
`≥ 1.7.0` and `< 1.9.96` shipped permissive, so on those the hook renders only
after an explicit flip.)
- A rendered check that fails means the environment is **unsealed** for that
guarantee until fixed — e.g. a CNI that does not enforce NetworkPolicy
fails `egress-enforcement` with remediation hints, exactly so the lockdown
Expand Down
Loading
Loading