Let BashOperator push multiple XComs, including on failure

[dheerajturaga]

A Bash task can only produce one XCom today: the last line written to stdout. Anything richer means serialising to JSON on a single line and parsing it back with output_processor, which drags shell authors into quote-escaping for even simple key/value pairs. The KubernetesPodOperator sidecar has the same JSON contract, and every example of it in this repo writes a JSON literal rather than a shell variable — which is telling.

Separately, the task runner only pushes XCom on the success path (_push_xcom_if_needed), so a failing command cannot hand any diagnostic data to downstream tasks or to the UI — precisely when that data is most wanted.

What this adds

When do_xcom_push is set, BashOperator exposes a directory as $AIRFLOW_XCOM_DIR plus an xcom push helper on PATH, and reads the directory back in a finally block — so entries are pushed whether the command succeeded, failed, or skipped.

Two rules govern what becomes an XCom:

• a file's name is the key, and its content (one trailing newline stripped) is the value;
• a .json suffix parses the content and drops the suffix from the key.

echo "$rows" > "$AIRFLOW_XCOM_DIR/row_count"   # no escaping, whatever $rows contains
generate_report | xcom push report             # stdin, for large or multiline values
xcom push --json summary '{"rows": 42}'        # same as writing summary.json

Because the filename carries the key, there is no format to escape — the value can contain quotes, spaces, or newlines and still round-trips. The helper is sugar over the documented directory contract, so scripts that reset PATH or exec into a container can just redirect into $AIRFLOW_XCOM_DIR.

If the script writes a return_value entry and the command succeeds, that value is returned from execute() rather than the stdout last line, so it flows through the runner's normal return-value push and composes with multiple_outputs=True.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

🤖 Generated with Claude Code

https://claude.ai/code/session_01SPiZuBWUwhduTb3JdNYzsN