Skip to content

fix(deploy): verify Dragonfly Registry access after rollouts - #160

Draft
hky1999 wants to merge 3 commits into
inclusionAI:developfrom
hky1999:hky1999/registry-connectivity
Draft

hky1999 wants to merge 3 commits into
inclusionAI:developfrom
hky1999:hky1999/registry-connectivity

Conversation

@hky1999

@hky1999 hky1999 commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Registry relocation can block Dragonfly image preparation when ingress relies only on CIDRs. Add a companion Helm release that verifies authenticated Nydus reads from every configured seed without owning Registry storage, resources or placement.

Behavior:

  • NetworkPolicy creation is opt-in and requires an explicit review of existing BuildKit/direct-client ingress. Its namespace and Pod selectors are ANDed; empty or invalid port lists fail rendering.
  • Any nonempty duplicate-free seed list is supported, with matching name-scoped exec RBAC. All configured seeds must be Ready and pass before stable success is recorded.
  • Curl transfer exit codes are checked independently of HTTP status. Blob redirects are bounded; cross-origin hops permanently remove Authorization, Proxy-Authorization and Cookie, reject HTTPS downgrades, and retain final digest/size validation. Each hop is decoded separately.
  • HTTPS CONNECT uses an explicitly configured Dragonfly signing CA; direct Registry/token/object-storage trust is configured separately. CONNECT response headers are suppressed. These settings do not use SDK AKERNEL_VERIFY_TLS.
  • Proxy probes require operator verification of credential-safe logging. Unmodified Dragonfly v1.3.0 INFO request dumps expose headers; direct-only checks or suspension are available until redaction/request-log suppression is verified with dummy credentials.
  • Response/bootstrap limits are 1 MiB, decompression is bounded to 2 MiB, and concurrency is two under the default 256 MiB Job limit. The Job owns state.json; Helm does not initialize/reset it.

Validation:

  • 36 checker/chart/real-curl regressions, Helm lint and deployment syntax checks passed, covering three-/five-seed profiles, truncated HTTP 200, HTTPS CONNECT, redirects, credential isolation and peak RSS.
  • Actual Kubernetes exec against all five production seeds passed direct authenticated manifest/bootstrap reads on the updated implementation.
  • Isolated Dragonfly v1.3.0 HTTPS tests read the real 8,704-byte retained bootstrap. Registry CA alone failed; the Dragonfly CA succeeded. A dummy token appeared twice in INFO console/file logs and zero times at WARN. No production credential was used in these proxy tests.
  • Actual Kubernetes server-side reapply of an empty state ConfigMap preserved Job-written state. This does not claim a complete Helm 4 upgrade test.
  • Earlier disposable rollout/Pod-recreation trigger tests passed. Temporary fixtures were cleaned up. Proxy cache-hit validation is not evidence of a fresh origin fetch.

Deployment status: this review updates code, not production runtime images. The existing namespace/Pod ingress rule remains installed. The newly added production checker CronJob is suspended because current Dragonfly INFO logging is not credential-safe; no Registry or Dragonfly rollout was performed.

@iamWHTWD iamWHTWD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the chart and checker at 317ba889. The local test suite passed. The inline comments cover first-install Registry access, probe reliability, HTTPS and credential handling, and Helm upgrade behavior.

podSelector:
matchLabels:
{{- toYaml .Values.registry.podSelector | nindent 6 }}
policyTypes: [Ingress]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve existing Registry ingress on first install

If this is the first ingress NetworkPolicy selecting these Registry Pods, installing the chart makes them ingress-isolated. Existing clients such as BuildKit or direct push/pull clients can lose access while all five seed probes still pass. Could we make policy creation opt-in and require operators to verify that their existing clients are covered before enabling it? Please also reject an empty registry.ports list. It renders as ports: null, which allows matching seeds to reach every destination port.

Kubernetes ingress isolation · Ingress rule API

Comment thread deploy/registry-connectivity/check.py Outdated
config = "".join(f"{key} = {json.dumps(value)}\n" for key, value in options.items())
config += "".join(f"header = {json.dumps(key + ': ' + value)}\n" for key, value in (headers or {}).items())
payload = config.encode()
command = ["/bin/sh", "-c", 'head -c "$1" | curl --config - --silent --show-error --include --write-out "\\n__AKERNEL_HTTP_STATUS__%{http_code}\\n" | base64', "registry-check", str(len(payload))]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Fail the check when curl fails

The pipeline reports base64's exit status and the checker only reads curl's HTTP status. A local response that sent HTTP 200 with a larger Content-Length than its body made curl exit 18, while the pipeline exited 0 and the checker decoded 200. If the received bytes match the expected digest, this can record a failed transfer as a successful probe. Please propagate and validate curl's exit status and add a regression test for this case.

Comment thread deploy/registry-connectivity/check.py Outdated
if status != 200:
raise CheckFailed("authenticated bootstrap read failed")
result = bootstrap_contents(body, descriptor)
status, _, body = http.get(url, auth, proxy=cfg.get("seedProxy", "http://127.0.0.1:4001"))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Account for Dragonfly's proxy certificate in the HTTPS probe

For an HTTPS Registry URL, curl uses CONNECT. Dragonfly terminates that TLS connection and presents its own certificate, which is self-signed when no proxy CA is configured. The README's HTTPS example only mentions trusting the Registry and token service CAs, so a healthy Registry can still fail this check. Please document and verify the proxy CA requirement with an HTTPS probe.

Dragonfly certificate handling

raise ValueError("missing token")
except (ValueError, TypeError) as error:
raise CheckFailed("invalid pull token response") from error
auth = {"Authorization": "Bearer " + token, "Accept": "application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json"}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep the pull token out of seed logs

This bearer header is also sent through the Dragonfly proxy for the bootstrap request. Dragonfly v1.3.0 has INFO-level forwarding logs that include the full request, so the token can appear in seed logs even when the checker's own logs are redacted. Please require redaction or disable the relevant request logging, and verify the behavior with a dummy credential before promising that tokens stay out of logs.

Dragonfly request logging

name: {{ .Release.Name }}-state
namespace: {{ .Release.Namespace }}
data:
state.json: '{}'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Let the Job own state.json

With Helm 4 server-side apply, Helm initially owns this field. Once a successful Job patches it, a later upgrade can conflict when Helm tries to restore {}. Could the chart create the ConfigMap without this key and let the Job write it? The checker already handles a missing data field.

Kubernetes field management

connection.update(timeout=1)
while connection.peek_stdout():
chunk = connection.read_stdout()
size += len(chunk)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Fit the allowed probe size within the Job's memory limit

This buffers base64 output before joining and decoding it. In a local simulation, five concurrent 20 MiB responses reached about 494 MiB peak RSS. The chart caps the container at 256 MiB even though each response can be up to 32 MiB. Please align the response cap, concurrency, and memory limit so an allowed probe cannot OOM the checker.

rcyw commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Reviewed head 317ba8894a43bf9645173d4e681de02e009b75e0 against develop@db99c935f85b5935b9ce79e46860ab6ff5bc8948.

The companion-release design is useful: keeping Registry Deployment/PVC ownership separate, ANDing namespace and Pod selectors, restricting exec to configured Pod names, and advancing success state only after stable verification are sensible boundaries. The existing inline review tracks other ingress, transport, credential and operational concerns. I reproduced two additional compatibility issues to address alongside those threads before merging.

R1 — [P2] Support the configured seed count rather than requiring exactly five

Both the chart and fingerprint() reject any seed count other than five. AKernel's default Aliyun seed pool size is three, and that size is passed through to seedClient.replicas. The new deployment entry point recommends this guard for private Registry/Dragonfly deployments, but it cannot be used with that normal profile.

Reproduction:

  • Set dragonfly.seedNames to three distinct names: Helm rendering fails with exactly five distinct Dragonfly seedNames are required.
  • Call the real checker with the same three configured names and all three Pods Ready: it still fails with all five configured Dragonfly seeds must be present and Ready.
  • The existing five-seed tests pass.

Please accept a nonempty, duplicate-free configured list, require every listed seed to be present/Ready and to pass verification, and generate the same name-scoped exec RBAC from that list. Five can remain an example/default profile; installing a connectivity guard should not require expanding an existing three-seed deployment. Add positive three-/five-seed cases and negative missing, unready and duplicate-name coverage.

R2 — [P2] Handle valid Registry blob redirects

The curl invocation does not follow redirects, and the bootstrap read treats a non-200 response as a failed check. Blob redirects are part of the Distribution API, and the OCI specification allows redirects while restricting Authorization forwarding across host boundaries. A healthy redirecting/object-storage-backed Registry can therefore produce permanently failed guard Jobs.

I exercised the actual SeedHTTP.get() and verify_seed() against loopback HTTP fixtures, replacing only Kubernetes exec transport with a local subprocess running the same head | curl | base64 command:

  • The direct-200 control passed, including manifest/bootstrap validation and a proxy-form HTTP request.
  • Changing the blob response to a same-origin 307 with a readable Location produced authenticated bootstrap read failed.
  • A bounded curl --location control downloaded exactly the expected layer bytes and SHA-256.

Please support bounded redirects with an explicit credential-forwarding policy and retain final digest/size validation. Also parse the final response body correctly: simply adding --location is insufficient with --include, because the current decoder skips CONNECT/100 blocks but not intermediate redirect headers. Add direct/proxy redirect, loop/limit, and cross-origin credential-isolation regressions; do not enable unrestricted credential forwarding.

Validation and scope

The existing 18 checker/chart tests, make deploy-script-check, and Helm lint with complete fixture values all passed locally (Python 3.12.3, PyYAML 6.0.3, Helm 3.22.0). The additional reproduction had one passing direct-200 control and three failing assertions corresponding to the two findings above. CI run 42772539960 has 11 successful jobs and publication skipped, but its current tests do not cover these compatibility branches.

These were isolated local fixtures, not a live CNI/Kubernetes-exec or production rollout qualification. The reported five-seed production results do not cover a three-seed topology or a redirecting Registry. Please rerun the affected tests and CI on the resulting head after the fixes.

Non-blocking follow-ups: add build/minimal-start coverage for the checker image and pinned dependencies, and distinguish proxy cache-hit verification from evidence of a fresh origin fetch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants