Add EKS operator for commands in existing Pods - #72542
Open
AlejandroMorgante wants to merge 3 commits into
Open
Conversation
Allow Dags to execute commands in externally managed EKS Pods without making Airflow responsible for their lifecycle. Generated-by: Codex (GPT-5)
EKS kubeconfig authentication runs under sh and can receive log output before its credential data. Bash-only parsing caused both existing and exec Pod operations to reject otherwise valid credentials.
AlejandroMorgante
force-pushed
the
add-eks-pod-exec-operator
branch
from
September 4, 2026 20:24
e104f5c to
b43ca2b
Compare
Accurate defaults and permission requirements help Dag authors configure the operator without granting unnecessary access or relying on ambiguous fallback behavior.
Contributor
Author
Contributor
Author
|
@SameerMesiah97, could you please review this when you have a chance? This is the follow-up to #71244, adding the EKS integration for executing commands in existing Pods. Thank you! |
AlejandroMorgante
marked this pull request as draft
September 6, 2026 00:29
AlejandroMorgante
marked this pull request as ready for review
September 6, 2026 00:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Add
EksPodExecOperatorso Dags can execute commands in running containers of existing Amazon EKS Pods without Airflow creating, restarting, or deleting them.This supports long-lived workloads managed by another platform, where Airflow should own the command execution but not the Pod lifecycle. Reusing an already-running Pod also avoids paying its startup cost for every task.
The operator uses the AWS connection to generate a temporary EKS kubeconfig, then delegates command execution, log streaming, exit-code handling, and bounded optional XCom output to
KubernetesPodExecOperator. This keeps EKS authentication in the Amazon provider while reusing the Kubernetes behavior introduced in #71244.The change includes unit coverage, documentation, and an EKS system-test path that creates and deletes the external Pod outside the operator.
The live test also exposed an existing shell-portability issue in the EKS kubeconfig credential parser. The regression fix protects both
EksPodOperatorandEksPodExecOperatorwhen credential discovery writes an informational line before the token output.Testing:
breeze run --skip-image-upgrade-check pytest providers/amazon/tests/unit/amazon/aws/operators/test_eks.py -q --tb=short(66 passed)breeze run --skip-image-upgrade-check pytest providers/amazon/tests/unit/amazon/aws/hooks/test_eks.py::TestEksHook -q(7 passed)breeze run --skip-image-upgrade-check mypy providers/amazon/src/airflow/providers/amazon/aws/hooks/eks.py providers/amazon/src/airflow/providers/amazon/aws/operators/eks.py providers/amazon/tests/system/amazon/aws/example_eks_with_nodegroups.py(no issues)System-test import and collection (
1 test collected)Live AWS end-to-end system test using
--forward-credentials: create the EKS cluster and nodegroup, create an externally managed Pod, execute and validate the command output, delete the Pod, and tear down the AWS resources (1 passedin 15:37)Apache Magpie self-review (
pr-management-code-review, dry-run): completed with no code-level findings.Was generative AI tooling used to co-author this PR?
Generated-by: Codex (GPT-5) following the guidelines