fix(aws): stop credential_process recursion in AWS CLI child processes (IGA-3789)

[highb]

Fixes the credential_process recursion reported in IGA-3789.

The bug

cone aws setup writes credential_process = cone aws credentials "<profile>" into ~/.aws/config. cone aws credentials then shells out to the AWS CLI (cmd/cone/aws.go) at two sites — sso get-role-credentials and sso login — and neither set cmd.Env. The child aws inherited AWS_PROFILE and HOME, resolved a cone-managed profile, and ran that profile's credential_process, which re-invoked cone, which shelled out to aws again. Unbounded. The reporter killed it at 685 processes.

The recursion needs the child to be able to select a cone-managed profile, which is why it does not happen for everyone:

how the profile is selected	recurses?
AWS_PROFILE exported (inherited by the child)	yes
cone profile installed as [default]	yes
aws --profile X only, nothing exported	no — the flag is not inherited

That last row is why "running the same aws sso get-role-credentials by hand works fine" was in the report.

The fix

aws sso get-role-credentials authenticates with the SSO bearer token passed on the command line — it needs nothing from ~/.aws. Its child environment now drops AWS_PROFILE/AWS_DEFAULT_PROFILE and pins AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE at os.DevNull, so there is no profile left — named or [default] — that could re-enter cone.

aws sso login --sso-session cone-sso still has to read the [sso-session cone-sso] block out of the real config, so only the profile selectors are dropped there. It is not itself a recursion vector, but an inherited AWS_PROFILE has no business reaching it either.

No new dependency; the AWS SDK migration suggested on the ticket is a larger change and is not attempted here.

Verification

Reproduced and fixed on two platforms, both making real calls to the public AWS SSO OIDC endpoint. Counts are identical on Linux/aarch64 with AWS CLI 2.35.24 (the version on the ticket) and on macOS/arm64 with AWS CLI 2.36.6 (the reported platform):

=== RUN   TestIGA3789_RecursionChannelExists
    credential_process invocations with an inherited environment: 5   <- depth cap; unbounded without it
--- PASS
=== RUN   TestIGA3789_FixedChildEnvDoesNotRecurse
    getRoleCredentials: err=... (UnauthorizedException) ... Session token not found or invalid
    credential_process invocations: 0
--- PASS
=== RUN   TestIGA3789_SSOLoginDoesNotRecurse
    credential_process invocations: 0
--- PASS

The live UnauthorizedException is what proves the call reached AWS rather than failing locally; the test asserts on it, so a run that never got that far cannot pass. The ssoLogin case reaches RegisterClient and fails there on the placeholder start URL — getting that far confirms the [sso-session cone-sso] block is still readable with the profile selectors dropped.

The os.DevNull pinning does not break a legitimate credential vend. Tested against a real AWS Identity Center account and permission set with a valid cached SSO token: aws sso get-role-credentials run under exactly the child environment the fix constructs returned the same accessKeyId / secretAccessKey / sessionToken / expiration set as a control run under a normal environment, and those credentials then authenticated against sts get-caller-identity.

The regression tests were mutation-checked, since a test asserting a child "cannot recurse" passes trivially if it is wired up wrong. Removing both cmd.Env assignments fails all four assertions; removing only the os.DevNull overrides and keeping the profile stripping fails exactly the two config-file assertions — so the [default]-profile vector has its own coverage.

go test ./... and -race are green, and golangci-lint reports the same 12 issues as the pre-PR baseline — this PR adds none.

Tests

• cmd/cone/aws_child_env_test.go — runs in normal CI. Installs a fake aws on PATH that dumps the environment it was handed, then asserts the child of getRoleCredentials carries no profile selectors and has both config paths pinned, that ssoLogin keeps a readable config, and that an override cannot be shadowed by an inherited value.
• cmd/cone/aws_recursion_repro_test.go — the end-to-end harness above, behind //go:build reprocredrecursion, excluded from default builds and CI because it needs a real AWS CLI and live network. Run with go test -tags reprocredrecursion -run TestIGA3789 -v ./cmd/cone/.... Its recursion is bounded by a depth counter in the child environment and a per-call context timeout, and HOME is redirected to a temp dir.

One trap worth knowing if you touch that harness: botocore resolves the web-identity provider ahead of the custom-process provider, so on any host exporting AWS_ROLE_ARN + AWS_WEB_IDENTITY_TOKEN_FILE (Kubernetes IRSA, many CI runners) credential resolution succeeds before credential_process is ever consulted, and the harness records zero invocations no matter how recursive the configuration is. scrubAWSEnv clears the whole provider set for this reason; weakening it turns the test into a silent false negative. The test fails with a message pointing at that scrub if the count comes back zero.

No CI path lints the gated file. golangci-lint run --build-tags reprocredrecursion flags three issues in it (gosec G101 on the shell-script constant, G306 on the 0o700 script write, one errcheck) — all artifacts of it being a test harness, but it will keep drifting unlinted.

Known limitation

Pinning AWS_CONFIG_FILE at os.DevNull discards more than profiles: ca_bundle, proxy, endpoint_url, use_fips_endpoint and similar settings in ~/.aws/config also stop reaching the child aws. The environment-variable forms (AWS_CA_BUNDLE, HTTPS_PROXY, …) still pass through, since awsChildEnv only strips the two profile keys, so the exposure is config-file-only. A user behind a TLS-inspecting proxy who sets ca_bundle in ~/.aws/config rather than in the environment would regress. If that turns out to matter, the fix is to write a minimal temp config carrying those settings instead of using /dev/null.

Not verified

• AWS CLI v1.
• cone aws setup end to end against a live tenant: no cone-managed profile was installed on either test machine, so the full setup → AWS_PROFILE=<cone-profile> → aws sts get-caller-identity path — the exact customer trigger — has not been walked. The credential-vend half of it is covered by the real-account test above.
• A related report of "threads and memory" exhaustion from another customer is plausibly the same root cause but was not independently confirmed.

[linear-code]

IGA-3789

[highb]

Independent verification on macOS + AWS CLI 2.36.6

Re-ran this on the reported platform, against a different AWS CLI version than the one in the PR body, and mutation-tested the regression coverage. Everything the PR listed as unverified is now closed except the live-tenant cone aws setup walk. Summary below; the PR description has been updated with the same results.

The fix's own regression risk is disproven

This was the important one — the PR argued from the call's flags, rather than observing, that a legitimate vend still works with AWS_CONFIG_FILE pinned at os.DevNull. It does. Against a real AWS Identity Center account and permission set with a valid cached SSO token:

• control, normal environment → ["accessKeyId","expiration","secretAccessKey","sessionToken"]
• under test, the exact child environment the fix builds (env -u AWS_PROFILE -u AWS_DEFAULT_PROFILE AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null) → same key set

and the credentials vended by the second run then authenticated against sts get-caller-identity, returning the expected assumed-role identity. So the pinning does not break the happy path on macOS / CLI 2.36.6.

The harness reproduces identically on macOS

go test -tags reprocredrecursion -run TestIGA3789 -v ./cmd/cone/... on darwin/arm64, AWS CLI 2.36.6 — same counts as the Linux/2.35.24 run:

--- PASS: TestIGA3789_RecursionChannelExists (9.85s)
    credential_process invocations with an inherited environment: 5
--- PASS: TestIGA3789_FixedChildEnvDoesNotRecurse (0.45s)
    getRoleCredentials: err=... (UnauthorizedException) ... Session token not found or invalid
--- PASS: TestIGA3789_SSOLoginDoesNotRecurse (5.52s)

Worth noting what the third case actually proves: ssoLogin got as far as RegisterClient and failed there with InvalidRequestException: Invalid start url provided. Reaching RegisterClient at all means the [sso-session cone-sso] block was still read after AWS_PROFILE was stripped — which is the thing that could have broken by dropping the profile selectors from that call site.

Also relevant to the earlier false negative: this machine exports no AWS_* variables at all, so there was no ambient web-identity provider to short-circuit the chain.

The regression tests actually gate the fix

A test asserting a child "cannot recurse" passes trivially if it is wired up wrong, so I mutated the fix out. Note that simply reverting aws.go is not a valid mutation — it fails to build on undefined: awsChildEnv and tells you nothing. Keeping awsChildEnv defined and removing only its call sites:

mutation	result
both cmd.Env assignments removed	all 4 assertions fail — inherited AWS_PROFILE, inherited AWS_DEFAULT_PROFILE, both … = "" want "/dev/null"
only the os.DevNull overrides removed, profile stripping kept	exactly the 2 config-file assertions fail

The second one is the useful result: the [default]-profile vector has independent coverage rather than riding on the profile stripping.

Housekeeping

• go test ./... and -race: green. make build on darwin/arm64: fine.
• golangci-lint run reports 12 issues, identical to the pre-PR baseline — this PR adds none.
• No CI path lints the gated file. golangci-lint run --build-tags reprocredrecursion flags 3 more in it (gosec G101 on the shell-script constant, G306 on the 0o700 write, one errcheck). All benign for a test harness, but it will drift.

One thing I'd still flag

AWS_CONFIG_FILE=/dev/null discards more than profiles — ca_bundle, proxy, endpoint_url, use_fips_endpoint in ~/.aws/config also stop reaching the child. The env-var forms still pass through (awsChildEnv only strips the two profile keys), so it is config-file-only, but a user behind a TLS-inspecting proxy setting ca_bundle in the config file would regress. Neither test machine had any of those set, so a green result here does not clear that case. Cheap de-risk if we think it is real: write a minimal temp config carrying those settings instead of /dev/null.

Still open

cone aws setup → AWS_PROFILE=<cone-profile> → aws sts get-caller-identity against a live tenant. Neither test machine had a cone-managed profile installed, so the exact customer trigger has not been walked end to end, though its credential-vend half is covered above. Also untested: AWS CLI v1.