[Spark 4] Streaming runner proof of concept, unbounded sources, windowed GBK and stateful ParDo on transformWithState - #39576

Draft

tkaymak wants to merge 21 commits into

apache:masterfrom

tkaymak:spark4-streaming-poc

Commits on Jul 31, 2026

[Spark 4] Add streaming dispatch seam and lifecycle hooks
tkaymak
committed
4ca982e
[Spark 4] Add DataSourceV2 micro-batch source for Beam unbounded sources
tkaymak
committed
0131dbf
[Spark 4] Bridge Beam state and timers onto transformWithState
tkaymak
committed
3ffc438
[Spark 4] Wire up the streaming registry and evaluation context
tkaymak
committed
09be88c
[Spark 4] Add streaming test scaffolding and Ignore'd translator skeletons
tkaymak
committed
e2f6b0e
[Spark 4] Add the three real streaming translators
tkaymak
committed
3ec0d65
[Spark 4] Enable the streaming end to end tests
tkaymak
committed
1e15750
[Spark 4] Register Spark's streaming internals with Kryo
tkaymak
committed
5783125
[Spark 4] Reformat the shared base to satisfy spotless
tkaymak
committed
c8de94f
[Spark 4] Assert the chained pipeline against Spark's query progress
tkaymak
committed
d9639f6

Commits on Aug 4, 2026

[Spark 4] Satisfy ErrorProne on the streaming sources
tkaymak
committed
fed52a1

Commits on Aug 19, 2026

Commits on Aug 20, 2026

Commits on Aug 21, 2026