fix(network): fail closed when credential placeholders cannot be rewritten

[TonyLuo-NV]

Summary

Makes the sandbox egress proxy fail closed when it cannot rewrite a credential placeholder, instead of forwarding the literal openshell:resolve:env:<NAME> token (or its provider-alias form) to the upstream. Closes two fail-open paths that leak the reserved placeholder on the wire and cause upstream auth failures when internal state is degraded.

Related Issue

Fixes #2161 — [DGX Spark] gateway never rewrites openshell:resolve:env: credential placeholder on sandbox egress — literal placeholder sent on the wire.

Full chain for reviewers:

• Root-cause issue (this repo): [DGX Spark] gateway never rewrites openshell:resolve:env: credential placeholder on sandbox egress — literal placeholder sent on the wire (works on DGX Station, same versions) #2161
• This PR: fail-closed hardening for both placeholder-leak paths
• Downstream bug report: [DGX Spark][Sandbox] managed MCP unusable at runtime — OpenShell never resolves the openshell:resolve:env: credential placeholder (sends it literally); mcp status stays all-green NemoClaw#6379 — managed MCP unusable on a DGX Spark host while mcp status stayed all-green
• Downstream diagnostic PR: fix(mcp): probe wire-level credential resolution in mcp status and add (#6379) NemoClaw#6382 — wire-level credential-resolution probe in mcp status / mcp add

The invariant enforced here: a live credential placeholder must never leave the proxy unresolved. Degraded internal state fails closed; explicit policy opt-outs (tls: skip) keep working.

Changes

Leak G3 — resolver-None fail-open (crates/openshell-core/src/secrets.rs)

rewrite_http_header_block took an early return that forwarded the header block verbatim whenever resolver was None. The existing fail-closed marker scan runs only after that early return (on the resolved path), so a header block carrying a live placeholder was passed through untouched. It now scans the header region for the reserved markers (openshell:resolve:env: prefix and the OPENSHELL-RESOLVE-ENV- alias marker, raw and percent-decoded) even when there is no resolver, and returns UnresolvedPlaceholderError { location: "header" } if one is present. The scan window mirrors the resolved-path scan (header_end + 256). Marker-free traffic still passes through unchanged, so credential-free endpoints are unaffected.

Leak G2 — tls_state == None raw-tunnel fallback (crates/openshell-supervisor-network/src/proxy.rs)

When TLS was detected on a CONNECT but tls_state was None (ephemeral CA generation or CA file write failed at startup, see run.rs), handle_tcp_connection emitted a Low-severity event and fell back to a raw copy_bidirectional tunnel — silently bypassing credential rewrite. Inside the proxy handler tls_state is None only on CA-init failure: the mode != Proxy case never starts this handler, and tls: skip is handled by an earlier early return. The handler now refuses the connection with a 503 Service Unavailable JSON error and a High-severity Denied/Blocked denial event instead of tunneling. The explicit tls: skip opt-out and all other modes are untouched.

Severity escalation (crates/openshell-supervisor-network/src/run.rs)

The two startup CA-failure logs ("Failed to write CA files…" and "Failed to generate ephemeral CA…") are raised from Medium to High, since a disabled TLS termination now fails closed on TLS-bearing connections.

Testing

All run locally with the sandbox HTTP proxy unset (the sandbox HTTP_PROXY=127.0.0.1:8118 intercepts loopback connections and makes the inference-routing socket tests fail spuriously; they pass with it unset and are unrelated to this change).

• cargo test -p openshell-core secrets:: → 58 passed, 0 failed. New unit tests: placeholder in request line / header value / provider-alias form with resolver=None all fail closed; the pre-existing marker-free resolver=None passthrough test still passes.
• cargo test -p openshell-supervisor-network → 812 lib tests + integration passed, 0 failed. The relay integration test (relay_request_without_resolver_fails_closed_on_placeholder) now asserts the request is rejected before any byte reaches upstream; a new test_json_error_response_503_tls_termination_unavailable locks the G2 fail-closed response contract.
• cargo fmt --all -- --check → clean.
• cargo clippy -p openshell-core -p openshell-supervisor-network --all-targets -- -D warnings → clean.

A true connection-level test of the G2 branch was not added: handle_tcp_connection requires a real TcpStream with a peeked TLS ClientHello and no existing test drives it, so the observable fail-closed response contract is locked via the build_json_error_response test instead.

Two pre-existing tests that documented the vulnerable behavior were updated to assert the fail-closed outcome:

• secrets.rs: no_resolver_passes_through_without_scanning → split into request-line / header-value / alias-marker fail-closed tests.

• l7/rest.rs: relay_request_without_resolver_leaks_placeholders → relay_request_without_resolver_fails_closed_on_placeholder.

• Unit tests added/updated

• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

Signed-off-by: Tony Luo xialuo@nvidia.com

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[TonyLuo-NV]

I have read the DCO document and I hereby sign the DCO.

[johntmyers]

gator-agent

PR Review Status

Validation: this PR is project-valid because it is a concentrated security hardening fix for linked issue #2161, which documents a reproducible sandbox egress credential-placeholder leak path.
Head SHA: 1c567f94a73301139c6306d822f9f09688ebcf56

Review findings:

• crates/openshell-supervisor-network/src/proxy.rs:1039 sends HTTP/1.1 200 Connection Established before TLS detection, but the new failure branch writes the intended 503 later at crates/openshell-supervisor-network/src/proxy.rs:1247. For normal CONNECT clients, the first observable response is still success; the JSON 503 is sent inside the tunnel and will likely surface as a TLS handshake/protocol error instead of the documented tls_termination_unavailable response. It also logs an allowed CONNECT before the denial. Please make CA-init failure a pre-200 Connection Established decision for non-tls: skip CONNECTs, and add a connection-level test that asserts the first bytes returned to the client are the 503 when tls_state is unavailable.

Docs: no Fern docs update is required for this code-only degraded-state hardening.

Next state: gator:in-review