diff --git a/website/www/site/content/en/roadmap/_index.md b/website/www/site/content/en/roadmap/_index.md index e764898d3f84..a9676da1ee5c 100644 --- a/website/www/site/content/en/roadmap/_index.md +++ b/website/www/site/content/en/roadmap/_index.md @@ -27,10 +27,6 @@ The major components of Beam each have their own roadmap which you can find via the menu. Below are some highlights for the project as a whole. -## Beam 3 - -Beam 3 is the planned first major version upgrade. See https://s.apache.org/beam3-milestones for details. - ## Portability Framework Portability is the primary Beam vision: running pipelines authored with _any SDK_ @@ -41,27 +37,48 @@ runners. See the details on the [Portability Roadmap](/roadmap/portability/) -## Cross-language transforms +### Portable schemas + +Schemas allow SDKs and runners to understand +the structure of user data and unlock relational optimization possibilities. +Portable schemas enable compatibility between rows in Python and Java. +A particularly interesting use case is the combination of SQL (implemented in Java) +with the Python SDK via Beam's cross-language support. +Learn more about portable schemas from this [design doc](https://s.apache.org/beam-schemas). + +### Cross-language transforms As a benefit of the portability effort, we are able to utilize Beam transforms across SDKs. Examples include using Java connectors and Beam SQL from Python or Go pipelines or Beam TFX transforms from Java and Go. + +The [SchemaTransform API](/documentation/sdks/python-custom-multi-language-pipelines-guide) is now the preferred way to author cross-language transforms, leveraging Beam Schemas for configuration and data exchange. Implementing SchemaTransforms is also required to make transforms available to [Beam YAML](/documentation/sdks/yaml) and the [Managed I/O](/documentation/io/managed-io) API. For details see [Roadmap for multi-SDK efforts](/roadmap/connectors-multi-sdk/). +## Extended Metadata and SDK level CDC (Change Data Capture) support + +Beam is introducing **Extended Metadata** to make element values extensible. This provides native, cross-SDK support for: + +* **Change Data Capture (CDC)**: First-class change operations (`INSERT`, `UPDATE`, `DELETE`) natively consumed by connectors like Iceberg, Delta Lake, and Spanner. +* **Lifecycle & Observability**: Allow SDK level Pipeline drain support (currently as a Dataflow runner feature). +* **DoFn API Evolution**: Fluent `OutputBuilder` APIs allowing transforms to selectively set metadata without combinatorial method overloads. + +Learn more from the [Beam Element Extended Metadata Design Doc](https://s.apache.org/beam-element-extended-metadata). + ## Go SDK The Go SDK is not actively being developed beyond bugfixes due to lack of contributors. If you'd like to help move this forward again, see the [Go SDK's Roadmap](/roadmap/go-sdk) ## Python 3 support -As of Apache Beam 2.69.0, we support python version from 3.9 uptil Python 3.13. Supporting Python 3.14 is in our roadmap. +As of Apache Beam 2.75.0, we support python version from 3.10 uptil Python 3.14. Supporting new Python 3 minor version is in our roadmap. See details on the [Python SDK's Roadmap](/roadmap/python-sdk/#python-3-support). ## Java support -As of Beam 2.69.0, we support Java 8, 11, 17, 21, 25. Java 8 support is deprecated and scheduled for removal in Beam 3.0.0. +As of Beam 2.75.0, we support Java 11, 17, 21, 25. Certain modules requires Java 17+ due to upstream dependency constraints. See details on the [Java SDK's Roadmap](/roadmap/java-sdk). ## SQL @@ -70,12 +87,3 @@ Beam's SQL module is rapidly maturing to allow users to author batch and streaming pipelines using only SQL, but also to allow Beam Java developers to use SQL in components of their pipeline for added efficiency. See the [Beam SQL Roadmap](/roadmap/sql/) - -## Portable schemas - -Schemas allow SDKs and runners to understand -the structure of user data and unlock relational optimization possibilities. -Portable schemas enable compatibility between rows in Python and Java. -A particularly interesting use case is the combination of SQL (implemented in Java) -with the Python SDK via Beam's cross-language support. -Learn more about portable schemas from this [presentation](https://s.apache.org/portable-schemas-seattle). diff --git a/website/www/site/content/en/roadmap/connectors-go-sdk.md b/website/www/site/content/en/roadmap/connectors-go-sdk.md deleted file mode 100644 index 2b1692d533eb..000000000000 --- a/website/www/site/content/en/roadmap/connectors-go-sdk.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -title: "Connectors - Go SDK" ---- - - -Roadmap for connectors developed using Go SDK. - -* Go SDK plans to utilize currently available Java and Python connectors -through cross-language transforms feature. - * KafkaIO via Java - DONE - * BigQuery via Java - In Progress - * Beam SQL via Java -* The Go SDK supports SplittableDoFns for bounded pipelines, so scalable bounded pipelines are possible. - * The textio package supports [ReadSdf](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio#ReadSdf) and [ReadAllSdf](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio#ReadAllSdf) for efficient batch text reads. - * A general FileIO will be produced to simplify adding new file based connectors. diff --git a/website/www/site/content/en/roadmap/connectors-java-sdk.md b/website/www/site/content/en/roadmap/connectors-java-sdk.md deleted file mode 100644 index c9b517e9079c..000000000000 --- a/website/www/site/content/en/roadmap/connectors-java-sdk.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -title: "Connectors - Java SDK" ---- - - -Roadmap for connectors developed using Java SDK. - -# Couchbase - -Couchbase is a NoSQL document-oriented database. See -[Issue 18381](https://github.com/apache/beam/issues/18381) for more details on the -planned Beam connector for Couchbase. - -# InfluxDB - -InfluxDB is a database for fast and highly available storage and retrieval -of time series data. See [BEAM-2546](https://issues.apache.org/jira/browse/BEAM-2546) for -more details on the planned Beam connector for InfluxDB. - -# Memcached - -Memcached is a distributed memory caching system. See -[BEAM-1678](https://issues.apache.org/jira/browse/BEAM-1678) for more details on the -planned Beam connector for Memcached. diff --git a/website/www/site/content/en/roadmap/connectors-multi-sdk.md b/website/www/site/content/en/roadmap/connectors-multi-sdk.md index 3a404b22becf..c78768c4ad44 100644 --- a/website/www/site/content/en/roadmap/connectors-multi-sdk.md +++ b/website/www/site/content/en/roadmap/connectors-multi-sdk.md @@ -1,5 +1,7 @@ --- title: "Multi-SDK Connector Efforts" +aliases: + - /roadmap/connectors-java-sdk/ --- -Connector-related efforts that will benefit multiple SDKs. +Roadmap for connectors and cross-cutting efforts that benefit multiple SDKs. -# Splittable DoFn -Splittable DoFn is the next generation sources framework for Beam that will -replace current frameworks for developing bounded and unbounded sources. -Splittable DoFn is being developed along side current Beam portability -efforts. See [Beam portability framework roadmap](/roadmap/portability/) for more details. +_Last updated on Aug 2026._ -# Cross-language transforms +# Extended Metadata and SDK level CDC (Change Data Capture) support -_Last updated on May 2020._ +Beam is introducing **Extended Metadata** to make element values extensible across SDKs. This provides native support for Change Data Capture (CDC) operations (`INSERT`, `UPDATE`, `DELETE`) consumed by lakehouse and database connectors, SDK-level pipeline drain, and fluent `OutputBuilder` APIs. -As an added benefit of Beam portability effort, we are able to utilize Beam transforms across SDKs. This has many benefits. +Learn more from the [Beam Element Extended Metadata Design Doc](https://s.apache.org/beam-element-extended-metadata). -* Connector sharing across SDKs. For example, - + Beam pipelines written using Python and Go SDKs will be able to utilize the vast selection of connectors that are currently implemented for Java SDK. - + Java SDK will be able to utilize connectors for systems that only offer a Python API. - + Go SDK, will be able to utilize connectors currently available for Java and Python SDKs. -* Ease of developing and maintaining Beam transforms - in general, with cross-language transforms, Beam transform authors will be able to implement new Beam transforms using a -language of choice and utilize these transforms from other languages reducing the maintenance and support overheads. -* [Beam SQL](/documentation/dsls/sql/overview/), that is currently only available to Java SDK, will become available to Python and Go SDKs. -* [Beam TFX transforms](https://www.tensorflow.org/tfx/transform/get_started), that are currently only available to Beam Python SDK pipelines will become available to Java and Go SDKs. +# OpenTelemetry Support -## Completed and Ongoing Efforts +Beam is adding native [OpenTelemetry](https://opentelemetry.io/) support across SDKs, runners, and connectors ([#33176](https://github.com/apache/beam/issues/33176)) for distributed tracing. This leverages Extended Metadata to propagate W3C Trace Context (`traceparent` and `tracestate`) across transform stages and worker boundaries, integrates trace header propagation into streaming I/Os (such as Kafka, Pub/Sub, and Spanner change streams), and enables runner-level trace and log correlation. -Many efforts related to cross-language transforms are currently in flux. Some of the completed and ongoing efforts are given below. +# IO Connector Ecosystem -### Cross-language transforms API and expansion service +Beam is actively expanding the Beam I/O connector portfolio with a focus on modern data lakehouse formats, native Change Data Capture (CDC), and unified access patterns: -Work related to developing/updating the cross-language transforms API for Java/Python/Go SDKs and work related to cross-language transform expansion services. +* **First-Class Lakehouse Connectors**: + * **[Apache Iceberg](/documentation/io/built-in/iceberg/)**: Full support for batch and streaming reads, streaming appends, dynamic destinations, and changelog CDC reading. Available natively in Java, via cross-language transforms in Python, in [Beam YAML](/documentation/sdks/yaml/), and via the [Managed I/O](/documentation/io/managed-io/) API. + * **Delta Lake**: Native read capabilities ([#38551](https://github.com/apache/beam/issues/38551)) and batch changelog (CDC) reading ([#39492](https://github.com/apache/beam/issues/39492)), exposed through core transforms and the [Managed I/O](/documentation/io/managed-io/) API. +* **Managed I/O Expansion**: Exposing more storage systems through the [Managed I/O API](/documentation/io/managed-io/), which standardizes connector configurations using Beam Schemas, provides frictionless cross-language access (Java, Python, YAML), and enables runner-level optimizations. +* **Native CDC Integration**: Integrating database change streams (such as Spanner change streams and Debezium-based sources) directly into lakehouse sinks using Beam's native CDC `ValueKind` metadata. -* Basic API for Java SDK - completed -* Basic API for Python SDK - completed -* Basic API for Go SDK - In progress -* Basic cross-language transform expansion service for Java and Python SDKs - completed -* Artifact staging - mostly completed - [email thread](https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E), [doc](https://docs.google.com/document/d/1XaiNekAY2sptuQRIXpjGAyaYdSc-wlJ-VKjl04c8N48/edit#heading=h.900gc947qrw8) - -### Support for Flink runner - -Work related to making cross-language transforms available for Flink runner. - -* Basic support for executing cross-language transforms on portable Flink runner - completed - -### Support for Dataflow runner - -Work related to making cross-language transforms available for Dataflow runner. - -* Basic support for executing cross-language transforms on Dataflow runner - + This work requires updates to Dataflow service's job submission and job execution logic. This is currently being developed at Google. - -### Support for Direct runner - -Work related to making cross-language transforms available on Direct runner - -* Basic support for executing cross-language transforms on Pyton Direct runner - completed -* Basic support for executing cross-language transforms on Java Direct runner - Not started - -### Connector/transform support - -Ongoing and planned work related to making existing connectors/transforms available to other SDKs through the cross-language transforms framework. - -* Java JdbcIO - completed - [BEAM-10135](https://issues.apache.org/jira/browse/BEAM-10135), [BEAM-10136](https://issues.apache.org/jira/browse/BEAM-10136) -* Java KafkaIO - completed - [BEAM-7029](https://issues.apache.org/jira/browse/BEAM-7029) -* Java KinesisIO - completed - [BEAM-10137](https://issues.apache.org/jira/browse/BEAM-10137), [BEAM-10138](https://issues.apache.org/jira/browse/BEAM-10138) -* Java PubSubIO - In progress - [BEAM-7738](https://issues.apache.org/jira/browse/BEAM-7738) -* Java SnowflakeIO - completed - [BEAM-9897](https://issues.apache.org/jira/browse/BEAM-9897), [BEAM-9898](https://issues.apache.org/jira/browse/BEAM-9898) -* Java SpannerIO - In progress - [BEAM-10139](https://issues.apache.org/jira/browse/BEAM-10139), [BEAM-10140](https://issues.apache.org/jira/browse/BEAM-10140) -* Java SQL - completed - [BEAM-8603](https://issues.apache.org/jira/browse/BEAM-8603) - -### Portable Beam schema - -Portable Beam schema support will provide a generalized mechanism for serializing and transferring data across language boundaries which will be extremely useful for pipelines that employ cross-language transforms. - -* Make row coder a standard coder and implement in python - completed - [BEAM-7886](https://issues.apache.org/jira/browse/BEAM-7886) - -### Integration/Performance testing - -* Add an integration test suite for cross-language transforms on Flink runner - In progress - [BEAM-6683](https://issues.apache.org/jira/browse/BEAM-6683) - -### Documentation - -Work related to adding documenting on cross-language transforms to Beam Website. - -* Document cross-language transforms API for Java/Python - Not started -* Document API for making existing transforms available as cross-language transforms for Java/Python - Not started +See the full list of available connectors in the [Beam Connectors Overview](/documentation/io/connectors/). diff --git a/website/www/site/content/en/roadmap/connectors-python-sdk.md b/website/www/site/content/en/roadmap/connectors-python-sdk.md deleted file mode 100644 index 66ca4c956fff..000000000000 --- a/website/www/site/content/en/roadmap/connectors-python-sdk.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: "Connectors - Python SDK" ---- - - -Roadmap for connectors developed using Python SDK. - -# Kafka - -An Apache Kafka connectors for Python SDK that is fully developed using -Splittable DoFn API is planned. This is partially blocked till -Splittable DoFn work related to portability framework is finalized. -See [BEAM-3788](https://issues.apache.org/jira/browse/BEAM-3788) for more details. - -# Parquet -A Python connector for Parquet file format is currently in development. -See [BEAM-4444](https://issues.apache.org/jira/browse/BEAM-4444) for more details. diff --git a/website/www/site/content/en/roadmap/dataflow-runner.md b/website/www/site/content/en/roadmap/dataflow-runner.md index 1c096255029a..4f259ae11b61 100644 --- a/website/www/site/content/en/roadmap/dataflow-runner.md +++ b/website/www/site/content/en/roadmap/dataflow-runner.md @@ -17,7 +17,10 @@ limitations under the License. # Google Cloud Dataflow Runner Roadmap -This roadmap is in progress. In the meantime, here are available resources: +Please check GitHub Issue tracker for recent developments on this topic: - - [Runner documentation](/documentation/runners/dataflow) - - Issues: [runner-dataflow](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Arunner-dataflow) + - Issues: [runner-dataflow](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Adataflow) + +Available resources: + +- [Runner documentation](/documentation/runners/dataflow) diff --git a/website/www/site/content/en/roadmap/euphoria.md b/website/www/site/content/en/roadmap/euphoria.md index c3c0accff9d9..f1ca5866a0aa 100644 --- a/website/www/site/content/en/roadmap/euphoria.md +++ b/website/www/site/content/en/roadmap/euphoria.md @@ -17,6 +17,8 @@ limitations under the License. # Euphoria API +**Note** Euphoria is deprecated and scheduled for removal in a future release. + Easy to use Java 8 DSL for the Beam Java SDK. Provides a high-level abstraction of Beam transformations, which is both easy to read and write. Can be used as a complement to existing Beam pipelines (convertible back and forth). You can have a glimpse of the API at [WordCount example](/documentation/sdks/java/euphoria/#wordcount-example). - Issues: [dsl-euphoria](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Adsl-euphoria) / [BEAM-3900](https://issues.apache.org/jira/browse/BEAM-3900) diff --git a/website/www/site/content/en/roadmap/flink-runner.md b/website/www/site/content/en/roadmap/flink-runner.md index fbf0f99f8070..de0b0015a133 100644 --- a/website/www/site/content/en/roadmap/flink-runner.md +++ b/website/www/site/content/en/roadmap/flink-runner.md @@ -17,7 +17,21 @@ limitations under the License. # Apache Flink Runner Roadmap -This roadmap is in progress. In the meantime, here are available resources: +_Last updated on Aug 2026._ - - [Runner documentation](/documentation/runners/flink) - - Issues: [runner-flink](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Arunner-flink) +## Flink versions + +Beam aims to support latest Flink minor versions. See [Flink Runner Support Table](/documentation/runners/flink#flink-version-compatibility) +for current coverage. + +### Flink 2.x Support + +One of the major changes in Flink 2 is the removal of Flink DataSet API. As a result Beam Flink 2 runners switched to use DataStream API for batch mode. One area of interest is to improve the batch support of DataStream API. + +## Available resources + +Please check GitHub Issue tracker for recent developments on this topic: + + - Issues: [runner-flink](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Aflink) + +- [Runner documentation](/documentation/runners/flink) diff --git a/website/www/site/content/en/roadmap/go-sdk.md b/website/www/site/content/en/roadmap/go-sdk.md index eb16b7936f73..39bded291eb0 100644 --- a/website/www/site/content/en/roadmap/go-sdk.md +++ b/website/www/site/content/en/roadmap/go-sdk.md @@ -1,5 +1,7 @@ --- title: "Go SDK Roadmap" +aliases: + - /roadmap/connectors-go-sdk/ ---