Skip to content

Add EKS operator for commands in existing Pods - #72542

Open
AlejandroMorgante wants to merge 3 commits into
apache:mainfrom
AlejandroMorgante:add-eks-pod-exec-operator
Open

Add EKS operator for commands in existing Pods#72542
AlejandroMorgante wants to merge 3 commits into
apache:mainfrom
AlejandroMorgante:add-eks-pod-exec-operator

Conversation

@AlejandroMorgante

@AlejandroMorgante AlejandroMorgante commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Add EksPodExecOperator so Dags can execute commands in running containers of existing Amazon EKS Pods without Airflow creating, restarting, or deleting them.

This supports long-lived workloads managed by another platform, where Airflow should own the command execution but not the Pod lifecycle. Reusing an already-running Pod also avoids paying its startup cost for every task.

The operator uses the AWS connection to generate a temporary EKS kubeconfig, then delegates command execution, log streaming, exit-code handling, and bounded optional XCom output to KubernetesPodExecOperator. This keeps EKS authentication in the Amazon provider while reusing the Kubernetes behavior introduced in #71244.

The change includes unit coverage, documentation, and an EKS system-test path that creates and deletes the external Pod outside the operator.

The live test also exposed an existing shell-portability issue in the EKS kubeconfig credential parser. The regression fix protects both EksPodOperator and EksPodExecOperator when credential discovery writes an informational line before the token output.

Testing:

  • breeze run --skip-image-upgrade-check pytest providers/amazon/tests/unit/amazon/aws/operators/test_eks.py -q --tb=short (66 passed)

  • breeze run --skip-image-upgrade-check pytest providers/amazon/tests/unit/amazon/aws/hooks/test_eks.py::TestEksHook -q (7 passed)

  • breeze run --skip-image-upgrade-check mypy providers/amazon/src/airflow/providers/amazon/aws/hooks/eks.py providers/amazon/src/airflow/providers/amazon/aws/operators/eks.py providers/amazon/tests/system/amazon/aws/example_eks_with_nodegroups.py (no issues)

  • System-test import and collection (1 test collected)

  • Live AWS end-to-end system test using --forward-credentials: create the EKS cluster and nodegroup, create an externally managed Pod, execute and validate the command output, delete the Pod, and tear down the AWS resources (1 passed in 15:37)

  • Apache Magpie self-review (pr-management-code-review, dry-run): completed with no code-level findings.


Was generative AI tooling used to co-author this PR?
  • Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines

Allow Dags to execute commands in externally managed EKS Pods without making Airflow responsible for their lifecycle.

Generated-by: Codex (GPT-5)
EKS kubeconfig authentication runs under sh and can receive log output before its credential data. Bash-only parsing caused both existing and exec Pod operations to reject otherwise valid credentials.
@AlejandroMorgante
AlejandroMorgante force-pushed the add-eks-pod-exec-operator branch from e104f5c to b43ca2b Compare September 4, 2026 20:24
Accurate defaults and permission requirements help Dag authors configure the operator without granting unnecessary access or relying on ambiguous fallback behavior.
@AlejandroMorgante

Copy link
Copy Markdown
Contributor Author

I also validated the operator with a real end-to-end run against an existing Pod in an EKS cluster. The task connected to the running Pod, executed the command, streamed its logs, and completed successfully in approximately 20 seconds. The Pod remained running and its lifecycle was not managed by Airflow.

This confirms the intended use case without provisioning a new Pod for each command.

image

@AlejandroMorgante

Copy link
Copy Markdown
Contributor Author

@SameerMesiah97, could you please review this when you have a chance? This is the follow-up to #71244, adding the EKS integration for executing commands in existing Pods. Thank you!

@AlejandroMorgante
AlejandroMorgante marked this pull request as draft September 6, 2026 00:29
@AlejandroMorgante
AlejandroMorgante marked this pull request as ready for review September 6, 2026 00:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant