Skip to content

Task SDK: Fix executor_config aliasing across tasks sharing default_args - #72556

Open
bujjibabukatta wants to merge 1 commit into
apache:mainfrom
bujjibabukatta:fix/#72544
Open

Task SDK: Fix executor_config aliasing across tasks sharing default_args#72556
bujjibabukatta wants to merge 1 commit into
apache:mainfrom
bujjibabukatta:fix/#72544

Conversation

@bujjibabukatta

Copy link
Copy Markdown
Contributor

Summary

executor_config set in a shared default_args dict was being shared by reference across tasks and DAGs, not copied. So if one task mutated its executor_config at runtime (a normal thing to do with the Kubernetes executor), it could silently change the config for other tasks using the same default_args.

Closes #72544.

Root Cause

Every place executor_config gets passed around does a shallow copy of the outer dict, but nobody ever copied the executor_config dict itself:

  • BaseOperator.__init__ just did executor_config or {} — no copy.
  • TaskInstance then copied it straight from the task object again, so even after a fix in one place, it still ends up as the same shared dict at runtime.

Callback fields like on_failure_callback already avoid this by rebuilding a fresh list each time. executor_config never got the same treatment.

Fix

In BaseOperator.__init__, executor_config is now wrapped in copy.copy() before being assigned, so it no longer just falls back to executor_config or {} directly. The same change was made in TaskInstance, where task.executor_config is now copied instead of assigned as-is. Together, these ensure every task and task instance gets its own independent executor_config dict instead of sharing one.

This is a shallow copy, consistent with how callback fields are already handled. If executor_config ever holds nested mutable objects (like a V1Pod), those inner parts would still be shared by reference — open to switching to a deep copy if reviewers think that's needed.

Was generative AI tooling used ?

  • Yes - Claude

Generated-by: Claude following the guidelines

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

executor_config from shared default_args is aliased (not copied) across tasks and DAGs

1 participant