From 00a572f75adecd9421c821d58498ba249defd083 Mon Sep 17 00:00:00 2001 From: Yi Hu Date: Wed, 26 Aug 2026 16:40:52 -0400 Subject: [PATCH 1/4] Update Beam website roadmap pages * Update overall roadmap highlights: - Add section for Extended Metadata and SDK-level CDC support - Highlight SchemaTransform API, Beam YAML, and Managed I/O under cross-language transforms - Update supported runtime versions for Python (3.10-3.14) and Java (11, 17, 21, 25) * Modernize SDK connector roadmaps: - Java: Add Extended Metadata/CDC, OpenTelemetry trace context propagation (#33176), and lakehouse format ecosystem (Iceberg, Delta Lake, Managed I/O); remove stale connector items - Python: Reframe around Java cross-language SchemaTransforms, auto-generated transform wrappers, and native Beam ML / AI ecosystem - Go: Add portable schemas/logical types (#39684), auto cross-language wrappers, native utility transforms (GroupIntoBatches, Watch), and first-class I/Os - Multi-SDK: Mark legacy roadmap as stale * Update SDK roadmaps - Add link to SDK language version compatibility matrix - Add last updated timestamp * Refresh runner roadmaps: - Flink: Document Flink 2.x support and DataStream API batch migration - Spark: Transition from Spark 3 to Spark 4 and Structured Streaming improvements - Dataflow & Prism: Update GitHub issue tracker links and resources * Mark Euphoria DSL as deprecated and scheduled for removal --- website/www/site/content/en/roadmap/_index.md | 40 +++++++++++-------- .../content/en/roadmap/connectors-go-sdk.md | 11 ++--- .../content/en/roadmap/connectors-java-sdk.md | 38 ++++++++++++------ .../en/roadmap/connectors-multi-sdk.md | 4 +- .../en/roadmap/connectors-python-sdk.md | 25 ++++++++---- .../content/en/roadmap/dataflow-runner.md | 9 +++-- .../www/site/content/en/roadmap/euphoria.md | 2 + .../site/content/en/roadmap/flink-runner.md | 20 ++++++++-- website/www/site/content/en/roadmap/go-sdk.md | 2 + .../www/site/content/en/roadmap/java-sdk.md | 9 +++-- .../site/content/en/roadmap/prism-runner.md | 2 + .../www/site/content/en/roadmap/python-sdk.md | 14 +++---- .../site/content/en/roadmap/spark-runner.md | 18 ++++++--- 13 files changed, 126 insertions(+), 68 deletions(-) diff --git a/website/www/site/content/en/roadmap/_index.md b/website/www/site/content/en/roadmap/_index.md index e764898d3f84..a9676da1ee5c 100644 --- a/website/www/site/content/en/roadmap/_index.md +++ b/website/www/site/content/en/roadmap/_index.md @@ -27,10 +27,6 @@ The major components of Beam each have their own roadmap which you can find via the menu. Below are some highlights for the project as a whole. -## Beam 3 - -Beam 3 is the planned first major version upgrade. See https://s.apache.org/beam3-milestones for details. - ## Portability Framework Portability is the primary Beam vision: running pipelines authored with _any SDK_ @@ -41,27 +37,48 @@ runners. See the details on the [Portability Roadmap](/roadmap/portability/) -## Cross-language transforms +### Portable schemas + +Schemas allow SDKs and runners to understand +the structure of user data and unlock relational optimization possibilities. +Portable schemas enable compatibility between rows in Python and Java. +A particularly interesting use case is the combination of SQL (implemented in Java) +with the Python SDK via Beam's cross-language support. +Learn more about portable schemas from this [design doc](https://s.apache.org/beam-schemas). + +### Cross-language transforms As a benefit of the portability effort, we are able to utilize Beam transforms across SDKs. Examples include using Java connectors and Beam SQL from Python or Go pipelines or Beam TFX transforms from Java and Go. + +The [SchemaTransform API](/documentation/sdks/python-custom-multi-language-pipelines-guide) is now the preferred way to author cross-language transforms, leveraging Beam Schemas for configuration and data exchange. Implementing SchemaTransforms is also required to make transforms available to [Beam YAML](/documentation/sdks/yaml) and the [Managed I/O](/documentation/io/managed-io) API. For details see [Roadmap for multi-SDK efforts](/roadmap/connectors-multi-sdk/). +## Extended Metadata and SDK level CDC (Change Data Capture) support + +Beam is introducing **Extended Metadata** to make element values extensible. This provides native, cross-SDK support for: + +* **Change Data Capture (CDC)**: First-class change operations (`INSERT`, `UPDATE`, `DELETE`) natively consumed by connectors like Iceberg, Delta Lake, and Spanner. +* **Lifecycle & Observability**: Allow SDK level Pipeline drain support (currently as a Dataflow runner feature). +* **DoFn API Evolution**: Fluent `OutputBuilder` APIs allowing transforms to selectively set metadata without combinatorial method overloads. + +Learn more from the [Beam Element Extended Metadata Design Doc](https://s.apache.org/beam-element-extended-metadata). + ## Go SDK The Go SDK is not actively being developed beyond bugfixes due to lack of contributors. If you'd like to help move this forward again, see the [Go SDK's Roadmap](/roadmap/go-sdk) ## Python 3 support -As of Apache Beam 2.69.0, we support python version from 3.9 uptil Python 3.13. Supporting Python 3.14 is in our roadmap. +As of Apache Beam 2.75.0, we support python version from 3.10 uptil Python 3.14. Supporting new Python 3 minor version is in our roadmap. See details on the [Python SDK's Roadmap](/roadmap/python-sdk/#python-3-support). ## Java support -As of Beam 2.69.0, we support Java 8, 11, 17, 21, 25. Java 8 support is deprecated and scheduled for removal in Beam 3.0.0. +As of Beam 2.75.0, we support Java 11, 17, 21, 25. Certain modules requires Java 17+ due to upstream dependency constraints. See details on the [Java SDK's Roadmap](/roadmap/java-sdk). ## SQL @@ -70,12 +87,3 @@ Beam's SQL module is rapidly maturing to allow users to author batch and streaming pipelines using only SQL, but also to allow Beam Java developers to use SQL in components of their pipeline for added efficiency. See the [Beam SQL Roadmap](/roadmap/sql/) - -## Portable schemas - -Schemas allow SDKs and runners to understand -the structure of user data and unlock relational optimization possibilities. -Portable schemas enable compatibility between rows in Python and Java. -A particularly interesting use case is the combination of SQL (implemented in Java) -with the Python SDK via Beam's cross-language support. -Learn more about portable schemas from this [presentation](https://s.apache.org/portable-schemas-seattle). diff --git a/website/www/site/content/en/roadmap/connectors-go-sdk.md b/website/www/site/content/en/roadmap/connectors-go-sdk.md index 2b1692d533eb..799c2f84a986 100644 --- a/website/www/site/content/en/roadmap/connectors-go-sdk.md +++ b/website/www/site/content/en/roadmap/connectors-go-sdk.md @@ -17,11 +17,8 @@ limitations under the License. Roadmap for connectors developed using Go SDK. -* Go SDK plans to utilize currently available Java and Python connectors +_Last updated on Aug 2026._ + +Go SDK plans to utilize currently available Java and Python connectors through cross-language transforms feature. - * KafkaIO via Java - DONE - * BigQuery via Java - In Progress - * Beam SQL via Java -* The Go SDK supports SplittableDoFns for bounded pipelines, so scalable bounded pipelines are possible. - * The textio package supports [ReadSdf](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio#ReadSdf) and [ReadAllSdf](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio#ReadAllSdf) for efficient batch text reads. - * A general FileIO will be produced to simplify adding new file based connectors. +* Support auto cross-language transform wrappers ([doc](https://s.apache.org/autogen-wrappers)), currently supported by Python. diff --git a/website/www/site/content/en/roadmap/connectors-java-sdk.md b/website/www/site/content/en/roadmap/connectors-java-sdk.md index c9b517e9079c..3c282eb427e5 100644 --- a/website/www/site/content/en/roadmap/connectors-java-sdk.md +++ b/website/www/site/content/en/roadmap/connectors-java-sdk.md @@ -17,20 +17,34 @@ limitations under the License. Roadmap for connectors developed using Java SDK. -# Couchbase +_Last updated on Aug 2026._ -Couchbase is a NoSQL document-oriented database. See -[Issue 18381](https://github.com/apache/beam/issues/18381) for more details on the -planned Beam connector for Couchbase. +# Extended Metadata and SDK level CDC (Change Data Capture) support -# InfluxDB +Beam is introducing **Extended Metadata** to make element values extensible. The support is first being added to Java SDK, and then will be propagated to other SDKs. This provides native, cross-SDK support for: -InfluxDB is a database for fast and highly available storage and retrieval -of time series data. See [BEAM-2546](https://issues.apache.org/jira/browse/BEAM-2546) for -more details on the planned Beam connector for InfluxDB. +* **Change Data Capture (CDC)**: First-class change operations (`INSERT`, `UPDATE`, `DELETE`) natively consumed by connectors like Iceberg, Delta Lake, and Spanner. +* **Lifecycle & Observability**: Allow SDK level Pipeline drain support (currently as a Dataflow runner feature). +* **DoFn API Evolution**: Fluent `OutputBuilder` APIs allowing transforms to selectively set metadata without combinatorial method overloads. -# Memcached +Learn more from the [Beam Element Extended Metadata Design Doc](https://s.apache.org/beam-element-extended-metadata). -Memcached is a distributed memory caching system. See -[BEAM-1678](https://issues.apache.org/jira/browse/BEAM-1678) for more details on the -planned Beam connector for Memcached. +# OpenTelemetry Support + +Distributed tracing is essential for diagnosing latency bottlenecks and tracking record provenance across streaming pipelines. Beam is adding native [OpenTelemetry](https://opentelemetry.io/) support across the Java SDK and connectors ([#33176](https://github.com/apache/beam/issues/33176)): + +* **Per-Element Context Propagation**: Leveraging the Extended Metadata framework to propagate W3C Trace Context headers (`traceparent` and `tracestate`) on each element across transform stages, shuffles, and worker boundaries. +* **Connector Header Propagation**: Integrating trace context injection and extraction into key streaming I/Os, including both reads and writes in `KafkaIO` and `PubSubIO`, as well as trace generation for `SpannerIO` change streams. +* **Runner Integration & Log Correlation**: Enabling end-to-end trace propagation through runners (such as Dataflow streaming runner), including stitching OpenTelemetry traces directly with worker logs for unified observability. + +# IO Connector Ecosystem + +We are actively expanding the Beam I/O connector portfolio with a focus on modern data lakehouse formats, native Change Data Capture (CDC), and unified access patterns: + +* **First-Class Lakehouse Connectors**: + * **[Apache Iceberg](/documentation/io/built-in/iceberg/)**: Full support for batch and streaming reads, streaming appends, dynamic destinations, and changelog CDC reading. Available natively in Java, via cross-language transforms in Python, in [Beam YAML](/documentation/sdks/yaml/), and via the [Managed I/O](/documentation/io/managed-io/) API. + * **Delta Lake**: Native read capabilities ([#38551](https://github.com/apache/beam/issues/38551)) and batch changelog (CDC) reading ([#39492](https://github.com/apache/beam/issues/39492)), exposed through core transforms and the [Managed I/O](/documentation/io/managed-io/) API. +* **Managed I/O Expansion**: Exposing more storage systems through the [Managed I/O API](/documentation/io/managed-io/), which standardizes connector configurations using Beam Schemas, provides frictionless cross-language access (Java, Python, YAML), and enables runner-level optimizations. +* **Native CDC Integration**: Integrating database change streams (such as Spanner change streams and Debezium-based sources) directly into lakehouse sinks using Beam's native CDC `ValueKind` metadata. + +See the full list of available connectors in the [Beam Connectors Overview](/documentation/io/connectors/). diff --git a/website/www/site/content/en/roadmap/connectors-multi-sdk.md b/website/www/site/content/en/roadmap/connectors-multi-sdk.md index 3a404b22becf..552d1029b610 100644 --- a/website/www/site/content/en/roadmap/connectors-multi-sdk.md +++ b/website/www/site/content/en/roadmap/connectors-multi-sdk.md @@ -17,6 +17,8 @@ limitations under the License. Connector-related efforts that will benefit multiple SDKs. +_This roadmap is stale. Last updated on May 2020._ + # Splittable DoFn Splittable DoFn is the next generation sources framework for Beam that will replace current frameworks for developing bounded and unbounded sources. @@ -25,8 +27,6 @@ efforts. See [Beam portability framework roadmap](/roadmap/portability/) for mor # Cross-language transforms -_Last updated on May 2020._ - As an added benefit of Beam portability effort, we are able to utilize Beam transforms across SDKs. This has many benefits. * Connector sharing across SDKs. For example, diff --git a/website/www/site/content/en/roadmap/connectors-python-sdk.md b/website/www/site/content/en/roadmap/connectors-python-sdk.md index 66ca4c956fff..d7640e63727e 100644 --- a/website/www/site/content/en/roadmap/connectors-python-sdk.md +++ b/website/www/site/content/en/roadmap/connectors-python-sdk.md @@ -17,13 +17,22 @@ limitations under the License. Roadmap for connectors developed using Python SDK. -# Kafka +_Last updated on Aug 2026._ -An Apache Kafka connectors for Python SDK that is fully developed using -Splittable DoFn API is planned. This is partially blocked till -Splittable DoFn work related to portability framework is finalized. -See [BEAM-3788](https://issues.apache.org/jira/browse/BEAM-3788) for more details. +# Cross-Language Connectors and Auto-Generated Wrappers + +Rather than reimplementing storage connectors natively from scratch in Python, the preferred approach for Beam data connectors is leveraging mature Java I/O connectors via Beam's cross-language (xlang) framework: + +* **Preferred Cross-Language Path**: Core I/O connectors (such as Kafka, Apache Iceberg, Delta Lake, Snowflake, and JDBC) are authored as [SchemaTransforms](/documentation/sdks/python-custom-multi-language-pipelines-guide) in Java and exposed to Python pipelines. Python users can invoke them seamlessly or configure them via the simplified [Managed I/O](/documentation/io/managed-io) API. +* **Auto-Generated Transform Wrappers**: The Python SDK natively supports generating wrappers for external transforms ([design doc](https://s.apache.org/autogen-wrappers)). By discovering SchemaTransforms registered in expansion services, the Python SDK automatically provides idiomatic, typed Python wrappers and documentation without requiring developers to manually write and maintain boilerplate code. + +# AI and Machine Learning (Beam ML) + +Machine learning and artificial intelligence represent the primary focus area for native Python transform and connector development: + +* **[RunInference](/documentation/ml/overview/)**: High-throughput, production-grade model inference supporting leading ML frameworks (including PyTorch, TensorFlow, TensorRT, ONNX, and Hugging Face) in both batch and streaming pipelines. +* **Vector Database Connectors**: Native integrations and configurations for vector databases (such as Qdrant, Pinecone, and Vertex AI Vector Search), supporting large-scale embedding generation and Retrieval-Augmented Generation (RAG) workflows. +* **LLM & Multi-Model Workflows**: Native patterns for model evaluation, multi-model ensemble pipelines, and orchestration with modern AI systems. + +Learn more about Beam's machine learning capabilities in the [Beam ML Documentation](/documentation/ml/overview/). -# Parquet -A Python connector for Parquet file format is currently in development. -See [BEAM-4444](https://issues.apache.org/jira/browse/BEAM-4444) for more details. diff --git a/website/www/site/content/en/roadmap/dataflow-runner.md b/website/www/site/content/en/roadmap/dataflow-runner.md index 1c096255029a..4f259ae11b61 100644 --- a/website/www/site/content/en/roadmap/dataflow-runner.md +++ b/website/www/site/content/en/roadmap/dataflow-runner.md @@ -17,7 +17,10 @@ limitations under the License. # Google Cloud Dataflow Runner Roadmap -This roadmap is in progress. In the meantime, here are available resources: +Please check GitHub Issue tracker for recent developments on this topic: - - [Runner documentation](/documentation/runners/dataflow) - - Issues: [runner-dataflow](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Arunner-dataflow) + - Issues: [runner-dataflow](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Adataflow) + +Available resources: + +- [Runner documentation](/documentation/runners/dataflow) diff --git a/website/www/site/content/en/roadmap/euphoria.md b/website/www/site/content/en/roadmap/euphoria.md index c3c0accff9d9..f1ca5866a0aa 100644 --- a/website/www/site/content/en/roadmap/euphoria.md +++ b/website/www/site/content/en/roadmap/euphoria.md @@ -17,6 +17,8 @@ limitations under the License. # Euphoria API +**Note** Euphoria is deprecated and scheduled for removal in a future release. + Easy to use Java 8 DSL for the Beam Java SDK. Provides a high-level abstraction of Beam transformations, which is both easy to read and write. Can be used as a complement to existing Beam pipelines (convertible back and forth). You can have a glimpse of the API at [WordCount example](/documentation/sdks/java/euphoria/#wordcount-example). - Issues: [dsl-euphoria](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Adsl-euphoria) / [BEAM-3900](https://issues.apache.org/jira/browse/BEAM-3900) diff --git a/website/www/site/content/en/roadmap/flink-runner.md b/website/www/site/content/en/roadmap/flink-runner.md index fbf0f99f8070..710ea99dcb66 100644 --- a/website/www/site/content/en/roadmap/flink-runner.md +++ b/website/www/site/content/en/roadmap/flink-runner.md @@ -17,7 +17,21 @@ limitations under the License. # Apache Flink Runner Roadmap -This roadmap is in progress. In the meantime, here are available resources: +_Last updated on Aug 2026._ - - [Runner documentation](/documentation/runners/flink) - - Issues: [runner-flink](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Arunner-flink) +## Flink versions + +Beam aims to support latest Flink minor versions. See [Flink Runner Support Table](/documentation/runners/flink#flink-versions-supported-by-beam-releases) +for current coverage. + +### Flink 2.x Support + +One of the major changes in Flink 2 is the removal of Flink DataSet API. As a result Beam Flink 2 runners switched to use DataStream API for batch mode. One area of interest is to improve the batch support of DataStream API. + +## Available resources + +Please check GitHub Issue tracker for recent developments on this topic: + + - Issues: [runner-flink](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Aflink) + +- [Runner documentation](/documentation/runners/flink) diff --git a/website/www/site/content/en/roadmap/go-sdk.md b/website/www/site/content/en/roadmap/go-sdk.md index eb16b7936f73..6aa33dabf0e0 100644 --- a/website/www/site/content/en/roadmap/go-sdk.md +++ b/website/www/site/content/en/roadmap/go-sdk.md @@ -17,6 +17,8 @@ limitations under the License. # Go SDK Roadmap +_Last updated on Nov 2021._ + The Go SDK is [fully released as of v2.33.0](/blog/go-sdk-release/). The Go SDK the first SDK purely on the [Beam Portability Framework](/roadmap/portability/) diff --git a/website/www/site/content/en/roadmap/java-sdk.md b/website/www/site/content/en/roadmap/java-sdk.md index d40dabec20fd..9b469ec28e3d 100644 --- a/website/www/site/content/en/roadmap/java-sdk.md +++ b/website/www/site/content/en/roadmap/java-sdk.md @@ -17,9 +17,10 @@ limitations under the License. # Java SDK Roadmap -## Next Java LTS version support (Java 25) +_Last updated on Aug 2026._ -Work to support the next LTS release of Java is in progress. For more details -about the scope and info on the various tasks please see the GitHub Issue. +## Next Java LTS version support -- GitHub: [#35627](https://github.com/apache/beam/issues/35627) +Beam aims to support new Java LTS versions in a timely manner. + +Work to support the next LTS release of Java is done as of Beam 2.69.0. For more details, see [Java Version Compatibility matrix](/documentation/sdks/java/#java-version-compatibility). diff --git a/website/www/site/content/en/roadmap/prism-runner.md b/website/www/site/content/en/roadmap/prism-runner.md index 7f741b94d2ae..7f4ae9940c8d 100644 --- a/website/www/site/content/en/roadmap/prism-runner.md +++ b/website/www/site/content/en/roadmap/prism-runner.md @@ -17,6 +17,8 @@ limitations under the License. # Apache Beam Prism Runner Roadmap +_Last updated on Aug 2026._ + The goal for the Prism runner is to provide a good default onboarding experience for Apache Beam. * Prism should be able to execute any Beam pipeline that can execute on a local machine. diff --git a/website/www/site/content/en/roadmap/python-sdk.md b/website/www/site/content/en/roadmap/python-sdk.md index 36a5cf3df4dd..5ab5ee62fe25 100644 --- a/website/www/site/content/en/roadmap/python-sdk.md +++ b/website/www/site/content/en/roadmap/python-sdk.md @@ -17,20 +17,18 @@ limitations under the License. # Python SDK Roadmap -## Python 3 Support +_Last updated on Aug 2026._ -Apache Beam 2.64.0 and higher support Python 3.9, 3.10, 3.11 and 3.12. We're continuing to [improve](https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050) the experience for Python 3 users and add support for Python 3.x minor versions: +## Python version Support +Apache Beam 2.73.0 and higher support Python 3.10, 3.11, 3.12, 3.13 and 3.14. Beam aims to support new Python minor versions in a timely manner: - - [Kanban Board](https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail) - - [Python 3 Conversion Quick Start Guide](https://docs.google.com/document/d/1s1BJVCY65LB_SYK1SU1u7NbZiFANoq-nEYaEvzRbYlA) - - [Tracking Issue](https://issues.apache.org/jira/browse/BEAM-1251) - - [Original Proposal](https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE) + - [Python Version Compatibility matrix](/documentation/sdks/python/#python-version-compatibility) Contributions and feedback are welcome! -If you are interested in helping, you can select an unassigned issue on the Kanban board and assign it to yourself. If you cannot assign the issue to yourself, comment on the issue. When submitting a new PR, please tag [@aaltay](https://github.com/aaltay), and [@tvalentyn](https://github.com/tvalentyn). +If you are interested in helping, you can select an unassigned issue on the Kanban board and assign it to yourself. If you cannot assign the issue to yourself, comment on the issue. When submitting a new PR, please tag [@damccorm](https://github.com/damccorm), and [@tvalentyn](https://github.com/tvalentyn). -To report a Python 3 related issue, create a subtask in [BEAM-1251](https://issues.apache.org/jira/browse/BEAM-1251) and cc: [~altay] and [~tvalentyn] in a JIRA comment. The best way to help us identify and investigate the issue is with a minimal pipeline that reproduces the issue. +To report a Python related issue, create a GitHub Issue in [Python label](https://github.com/apache/beam/issues?q=is%3Aissue%20state%3Aopen%20label%3Apython) and cc: [~damccorm] and [~tvalentyn] in a JIRA comment. The best way to help us identify and investigate the issue is with a minimal pipeline that reproduces the issue. You can also discuss encountered issues on user@ or dev@ mailing lists as appropriate. diff --git a/website/www/site/content/en/roadmap/spark-runner.md b/website/www/site/content/en/roadmap/spark-runner.md index 06f4dce8aca0..e710ebf78db7 100644 --- a/website/www/site/content/en/roadmap/spark-runner.md +++ b/website/www/site/content/en/roadmap/spark-runner.md @@ -17,10 +17,18 @@ limitations under the License. # Apache Spark Runner Roadmap -## Spark 3 +_Last updated on Aug 2026._ -Support for Spark 3 in Beam's Spark runner is ongoing. For info on the various -tasks please refer to the JIRA ticket. +## Spark 4 -- JIRA: [BEAM-7093](https://issues.apache.org/jira/browse/BEAM-7093) -- Contact: [Ismaël Mejía](mailto:iemejia@apache.org) +Support for Spark 4 in Beam's Spark runner is ongoing. + +As of Beam 2.74.0, Spark 4 is supported by Classic Spark runner in batch mode. + +As od Spark 4, DStream API is deprecated and community interest is around Structured Streaming. We are working on improving the Structured Streaming support in Beam's Spark runner, this includes +- Complete streaming support for Spark 4 based on Structured Streaming +- Build Spark 4 portable runner based on Structured Streaming + +For info on the various tasks please refer to the GitHub Issue. + +- Issues: [runner-spark](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Aspark) From a516c266553fd7e0334b169e44feda5ca50d113d Mon Sep 17 00:00:00 2001 From: Yi Hu Date: Thu, 27 Aug 2026 13:25:47 -0400 Subject: [PATCH 2/4] Consolidate individual roadmaps, simplify and less specific --- .../content/en/roadmap/connectors-go-sdk.md | 24 ----- .../content/en/roadmap/connectors-java-sdk.md | 50 ---------- .../en/roadmap/connectors-multi-sdk.md | 92 ++++--------------- .../en/roadmap/connectors-python-sdk.md | 38 -------- website/www/site/content/en/roadmap/go-sdk.md | 12 ++- .../www/site/content/en/roadmap/python-sdk.md | 25 ++++- .../partials/section-menu/en/roadmap.html | 16 ++-- 7 files changed, 54 insertions(+), 203 deletions(-) delete mode 100644 website/www/site/content/en/roadmap/connectors-go-sdk.md delete mode 100644 website/www/site/content/en/roadmap/connectors-java-sdk.md delete mode 100644 website/www/site/content/en/roadmap/connectors-python-sdk.md diff --git a/website/www/site/content/en/roadmap/connectors-go-sdk.md b/website/www/site/content/en/roadmap/connectors-go-sdk.md deleted file mode 100644 index 799c2f84a986..000000000000 --- a/website/www/site/content/en/roadmap/connectors-go-sdk.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: "Connectors - Go SDK" ---- - - -Roadmap for connectors developed using Go SDK. - -_Last updated on Aug 2026._ - -Go SDK plans to utilize currently available Java and Python connectors -through cross-language transforms feature. -* Support auto cross-language transform wrappers ([doc](https://s.apache.org/autogen-wrappers)), currently supported by Python. diff --git a/website/www/site/content/en/roadmap/connectors-java-sdk.md b/website/www/site/content/en/roadmap/connectors-java-sdk.md deleted file mode 100644 index 3c282eb427e5..000000000000 --- a/website/www/site/content/en/roadmap/connectors-java-sdk.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -title: "Connectors - Java SDK" ---- - - -Roadmap for connectors developed using Java SDK. - -_Last updated on Aug 2026._ - -# Extended Metadata and SDK level CDC (Change Data Capture) support - -Beam is introducing **Extended Metadata** to make element values extensible. The support is first being added to Java SDK, and then will be propagated to other SDKs. This provides native, cross-SDK support for: - -* **Change Data Capture (CDC)**: First-class change operations (`INSERT`, `UPDATE`, `DELETE`) natively consumed by connectors like Iceberg, Delta Lake, and Spanner. -* **Lifecycle & Observability**: Allow SDK level Pipeline drain support (currently as a Dataflow runner feature). -* **DoFn API Evolution**: Fluent `OutputBuilder` APIs allowing transforms to selectively set metadata without combinatorial method overloads. - -Learn more from the [Beam Element Extended Metadata Design Doc](https://s.apache.org/beam-element-extended-metadata). - -# OpenTelemetry Support - -Distributed tracing is essential for diagnosing latency bottlenecks and tracking record provenance across streaming pipelines. Beam is adding native [OpenTelemetry](https://opentelemetry.io/) support across the Java SDK and connectors ([#33176](https://github.com/apache/beam/issues/33176)): - -* **Per-Element Context Propagation**: Leveraging the Extended Metadata framework to propagate W3C Trace Context headers (`traceparent` and `tracestate`) on each element across transform stages, shuffles, and worker boundaries. -* **Connector Header Propagation**: Integrating trace context injection and extraction into key streaming I/Os, including both reads and writes in `KafkaIO` and `PubSubIO`, as well as trace generation for `SpannerIO` change streams. -* **Runner Integration & Log Correlation**: Enabling end-to-end trace propagation through runners (such as Dataflow streaming runner), including stitching OpenTelemetry traces directly with worker logs for unified observability. - -# IO Connector Ecosystem - -We are actively expanding the Beam I/O connector portfolio with a focus on modern data lakehouse formats, native Change Data Capture (CDC), and unified access patterns: - -* **First-Class Lakehouse Connectors**: - * **[Apache Iceberg](/documentation/io/built-in/iceberg/)**: Full support for batch and streaming reads, streaming appends, dynamic destinations, and changelog CDC reading. Available natively in Java, via cross-language transforms in Python, in [Beam YAML](/documentation/sdks/yaml/), and via the [Managed I/O](/documentation/io/managed-io/) API. - * **Delta Lake**: Native read capabilities ([#38551](https://github.com/apache/beam/issues/38551)) and batch changelog (CDC) reading ([#39492](https://github.com/apache/beam/issues/39492)), exposed through core transforms and the [Managed I/O](/documentation/io/managed-io/) API. -* **Managed I/O Expansion**: Exposing more storage systems through the [Managed I/O API](/documentation/io/managed-io/), which standardizes connector configurations using Beam Schemas, provides frictionless cross-language access (Java, Python, YAML), and enables runner-level optimizations. -* **Native CDC Integration**: Integrating database change streams (such as Spanner change streams and Debezium-based sources) directly into lakehouse sinks using Beam's native CDC `ValueKind` metadata. - -See the full list of available connectors in the [Beam Connectors Overview](/documentation/io/connectors/). diff --git a/website/www/site/content/en/roadmap/connectors-multi-sdk.md b/website/www/site/content/en/roadmap/connectors-multi-sdk.md index 552d1029b610..c78768c4ad44 100644 --- a/website/www/site/content/en/roadmap/connectors-multi-sdk.md +++ b/website/www/site/content/en/roadmap/connectors-multi-sdk.md @@ -1,5 +1,7 @@ --- title: "Multi-SDK Connector Efforts" +aliases: + - /roadmap/connectors-java-sdk/ --- -Connector-related efforts that will benefit multiple SDKs. +Roadmap for connectors and cross-cutting efforts that benefit multiple SDKs. -_This roadmap is stale. Last updated on May 2020._ +_Last updated on Aug 2026._ -# Splittable DoFn -Splittable DoFn is the next generation sources framework for Beam that will -replace current frameworks for developing bounded and unbounded sources. -Splittable DoFn is being developed along side current Beam portability -efforts. See [Beam portability framework roadmap](/roadmap/portability/) for more details. +# Extended Metadata and SDK level CDC (Change Data Capture) support -# Cross-language transforms +Beam is introducing **Extended Metadata** to make element values extensible across SDKs. This provides native support for Change Data Capture (CDC) operations (`INSERT`, `UPDATE`, `DELETE`) consumed by lakehouse and database connectors, SDK-level pipeline drain, and fluent `OutputBuilder` APIs. -As an added benefit of Beam portability effort, we are able to utilize Beam transforms across SDKs. This has many benefits. +Learn more from the [Beam Element Extended Metadata Design Doc](https://s.apache.org/beam-element-extended-metadata). -* Connector sharing across SDKs. For example, - + Beam pipelines written using Python and Go SDKs will be able to utilize the vast selection of connectors that are currently implemented for Java SDK. - + Java SDK will be able to utilize connectors for systems that only offer a Python API. - + Go SDK, will be able to utilize connectors currently available for Java and Python SDKs. -* Ease of developing and maintaining Beam transforms - in general, with cross-language transforms, Beam transform authors will be able to implement new Beam transforms using a -language of choice and utilize these transforms from other languages reducing the maintenance and support overheads. -* [Beam SQL](/documentation/dsls/sql/overview/), that is currently only available to Java SDK, will become available to Python and Go SDKs. -* [Beam TFX transforms](https://www.tensorflow.org/tfx/transform/get_started), that are currently only available to Beam Python SDK pipelines will become available to Java and Go SDKs. +# OpenTelemetry Support -## Completed and Ongoing Efforts +Beam is adding native [OpenTelemetry](https://opentelemetry.io/) support across SDKs, runners, and connectors ([#33176](https://github.com/apache/beam/issues/33176)) for distributed tracing. This leverages Extended Metadata to propagate W3C Trace Context (`traceparent` and `tracestate`) across transform stages and worker boundaries, integrates trace header propagation into streaming I/Os (such as Kafka, Pub/Sub, and Spanner change streams), and enables runner-level trace and log correlation. -Many efforts related to cross-language transforms are currently in flux. Some of the completed and ongoing efforts are given below. +# IO Connector Ecosystem -### Cross-language transforms API and expansion service +Beam is actively expanding the Beam I/O connector portfolio with a focus on modern data lakehouse formats, native Change Data Capture (CDC), and unified access patterns: -Work related to developing/updating the cross-language transforms API for Java/Python/Go SDKs and work related to cross-language transform expansion services. +* **First-Class Lakehouse Connectors**: + * **[Apache Iceberg](/documentation/io/built-in/iceberg/)**: Full support for batch and streaming reads, streaming appends, dynamic destinations, and changelog CDC reading. Available natively in Java, via cross-language transforms in Python, in [Beam YAML](/documentation/sdks/yaml/), and via the [Managed I/O](/documentation/io/managed-io/) API. + * **Delta Lake**: Native read capabilities ([#38551](https://github.com/apache/beam/issues/38551)) and batch changelog (CDC) reading ([#39492](https://github.com/apache/beam/issues/39492)), exposed through core transforms and the [Managed I/O](/documentation/io/managed-io/) API. +* **Managed I/O Expansion**: Exposing more storage systems through the [Managed I/O API](/documentation/io/managed-io/), which standardizes connector configurations using Beam Schemas, provides frictionless cross-language access (Java, Python, YAML), and enables runner-level optimizations. +* **Native CDC Integration**: Integrating database change streams (such as Spanner change streams and Debezium-based sources) directly into lakehouse sinks using Beam's native CDC `ValueKind` metadata. -* Basic API for Java SDK - completed -* Basic API for Python SDK - completed -* Basic API for Go SDK - In progress -* Basic cross-language transform expansion service for Java and Python SDKs - completed -* Artifact staging - mostly completed - [email thread](https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E), [doc](https://docs.google.com/document/d/1XaiNekAY2sptuQRIXpjGAyaYdSc-wlJ-VKjl04c8N48/edit#heading=h.900gc947qrw8) - -### Support for Flink runner - -Work related to making cross-language transforms available for Flink runner. - -* Basic support for executing cross-language transforms on portable Flink runner - completed - -### Support for Dataflow runner - -Work related to making cross-language transforms available for Dataflow runner. - -* Basic support for executing cross-language transforms on Dataflow runner - + This work requires updates to Dataflow service's job submission and job execution logic. This is currently being developed at Google. - -### Support for Direct runner - -Work related to making cross-language transforms available on Direct runner - -* Basic support for executing cross-language transforms on Pyton Direct runner - completed -* Basic support for executing cross-language transforms on Java Direct runner - Not started - -### Connector/transform support - -Ongoing and planned work related to making existing connectors/transforms available to other SDKs through the cross-language transforms framework. - -* Java JdbcIO - completed - [BEAM-10135](https://issues.apache.org/jira/browse/BEAM-10135), [BEAM-10136](https://issues.apache.org/jira/browse/BEAM-10136) -* Java KafkaIO - completed - [BEAM-7029](https://issues.apache.org/jira/browse/BEAM-7029) -* Java KinesisIO - completed - [BEAM-10137](https://issues.apache.org/jira/browse/BEAM-10137), [BEAM-10138](https://issues.apache.org/jira/browse/BEAM-10138) -* Java PubSubIO - In progress - [BEAM-7738](https://issues.apache.org/jira/browse/BEAM-7738) -* Java SnowflakeIO - completed - [BEAM-9897](https://issues.apache.org/jira/browse/BEAM-9897), [BEAM-9898](https://issues.apache.org/jira/browse/BEAM-9898) -* Java SpannerIO - In progress - [BEAM-10139](https://issues.apache.org/jira/browse/BEAM-10139), [BEAM-10140](https://issues.apache.org/jira/browse/BEAM-10140) -* Java SQL - completed - [BEAM-8603](https://issues.apache.org/jira/browse/BEAM-8603) - -### Portable Beam schema - -Portable Beam schema support will provide a generalized mechanism for serializing and transferring data across language boundaries which will be extremely useful for pipelines that employ cross-language transforms. - -* Make row coder a standard coder and implement in python - completed - [BEAM-7886](https://issues.apache.org/jira/browse/BEAM-7886) - -### Integration/Performance testing - -* Add an integration test suite for cross-language transforms on Flink runner - In progress - [BEAM-6683](https://issues.apache.org/jira/browse/BEAM-6683) - -### Documentation - -Work related to adding documenting on cross-language transforms to Beam Website. - -* Document cross-language transforms API for Java/Python - Not started -* Document API for making existing transforms available as cross-language transforms for Java/Python - Not started +See the full list of available connectors in the [Beam Connectors Overview](/documentation/io/connectors/). diff --git a/website/www/site/content/en/roadmap/connectors-python-sdk.md b/website/www/site/content/en/roadmap/connectors-python-sdk.md deleted file mode 100644 index d7640e63727e..000000000000 --- a/website/www/site/content/en/roadmap/connectors-python-sdk.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: "Connectors - Python SDK" ---- - - -Roadmap for connectors developed using Python SDK. - -_Last updated on Aug 2026._ - -# Cross-Language Connectors and Auto-Generated Wrappers - -Rather than reimplementing storage connectors natively from scratch in Python, the preferred approach for Beam data connectors is leveraging mature Java I/O connectors via Beam's cross-language (xlang) framework: - -* **Preferred Cross-Language Path**: Core I/O connectors (such as Kafka, Apache Iceberg, Delta Lake, Snowflake, and JDBC) are authored as [SchemaTransforms](/documentation/sdks/python-custom-multi-language-pipelines-guide) in Java and exposed to Python pipelines. Python users can invoke them seamlessly or configure them via the simplified [Managed I/O](/documentation/io/managed-io) API. -* **Auto-Generated Transform Wrappers**: The Python SDK natively supports generating wrappers for external transforms ([design doc](https://s.apache.org/autogen-wrappers)). By discovering SchemaTransforms registered in expansion services, the Python SDK automatically provides idiomatic, typed Python wrappers and documentation without requiring developers to manually write and maintain boilerplate code. - -# AI and Machine Learning (Beam ML) - -Machine learning and artificial intelligence represent the primary focus area for native Python transform and connector development: - -* **[RunInference](/documentation/ml/overview/)**: High-throughput, production-grade model inference supporting leading ML frameworks (including PyTorch, TensorFlow, TensorRT, ONNX, and Hugging Face) in both batch and streaming pipelines. -* **Vector Database Connectors**: Native integrations and configurations for vector databases (such as Qdrant, Pinecone, and Vertex AI Vector Search), supporting large-scale embedding generation and Retrieval-Augmented Generation (RAG) workflows. -* **LLM & Multi-Model Workflows**: Native patterns for model evaluation, multi-model ensemble pipelines, and orchestration with modern AI systems. - -Learn more about Beam's machine learning capabilities in the [Beam ML Documentation](/documentation/ml/overview/). - diff --git a/website/www/site/content/en/roadmap/go-sdk.md b/website/www/site/content/en/roadmap/go-sdk.md index 6aa33dabf0e0..a1512119da23 100644 --- a/website/www/site/content/en/roadmap/go-sdk.md +++ b/website/www/site/content/en/roadmap/go-sdk.md @@ -1,5 +1,7 @@ --- title: "Go SDK Roadmap" +aliases: + - /roadmap/connectors-go-sdk/ ---