Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 24 additions & 16 deletions website/www/site/content/en/roadmap/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,6 @@ The major components of Beam each have their own roadmap which you can find
via the menu.
Below are some highlights for the project as a whole.

## Beam 3

Beam 3 is the planned first major version upgrade. See https://s.apache.org/beam3-milestones for details.

## Portability Framework

Portability is the primary Beam vision: running pipelines authored with _any SDK_
Expand All @@ -41,27 +37,48 @@ runners.

See the details on the [Portability Roadmap](/roadmap/portability/)

## Cross-language transforms
### Portable schemas

Schemas allow SDKs and runners to understand
the structure of user data and unlock relational optimization possibilities.
Portable schemas enable compatibility between rows in Python and Java.
A particularly interesting use case is the combination of SQL (implemented in Java)
with the Python SDK via Beam's cross-language support.
Learn more about portable schemas from this [design doc](https://s.apache.org/beam-schemas).

### Cross-language transforms

As a benefit of the portability effort, we are able to utilize Beam transforms across SDKs.
Examples include using Java connectors and Beam SQL from Python or Go pipelines
or Beam TFX transforms from Java and Go.

The [SchemaTransform API](/documentation/sdks/python-custom-multi-language-pipelines-guide) is now the preferred way to author cross-language transforms, leveraging Beam Schemas for configuration and data exchange. Implementing SchemaTransforms is also required to make transforms available to [Beam YAML](/documentation/sdks/yaml) and the [Managed I/O](/documentation/io/managed-io) API.
For details see [Roadmap for multi-SDK efforts](/roadmap/connectors-multi-sdk/).

## Extended Metadata and SDK level CDC (Change Data Capture) support

Beam is introducing **Extended Metadata** to make element values extensible. This provides native, cross-SDK support for:

* **Change Data Capture (CDC)**: First-class change operations (`INSERT`, `UPDATE`, `DELETE`) natively consumed by connectors like Iceberg, Delta Lake, and Spanner.
* **Lifecycle & Observability**: Allow SDK level Pipeline drain support (currently as a Dataflow runner feature).
* **DoFn API Evolution**: Fluent `OutputBuilder` APIs allowing transforms to selectively set metadata without combinatorial method overloads.

Learn more from the [Beam Element Extended Metadata Design Doc](https://s.apache.org/beam-element-extended-metadata).

## Go SDK

The Go SDK is not actively being developed beyond bugfixes due to lack of contributors. If you'd like to help move this forward again, see the [Go SDK's Roadmap](/roadmap/go-sdk)

## Python 3 support

As of Apache Beam 2.69.0, we support python version from 3.9 uptil Python 3.13. Supporting Python 3.14 is in our roadmap.
As of Apache Beam 2.75.0, we support python version from 3.10 uptil Python 3.14. Supporting new Python 3 minor version is in our roadmap.

See details on
the [Python SDK's Roadmap](/roadmap/python-sdk/#python-3-support).

## Java support

As of Beam 2.69.0, we support Java 8, 11, 17, 21, 25. Java 8 support is deprecated and scheduled for removal in Beam 3.0.0.
As of Beam 2.75.0, we support Java 11, 17, 21, 25. Certain modules requires Java 17+ due to upstream dependency constraints.
See details on the [Java SDK's Roadmap](/roadmap/java-sdk).

## SQL
Expand All @@ -70,12 +87,3 @@ Beam's SQL module is rapidly maturing to allow users to author batch and
streaming pipelines using only SQL, but also to allow Beam Java developers
to use SQL in components of their pipeline for added efficiency. See the
[Beam SQL Roadmap](/roadmap/sql/)

## Portable schemas

Schemas allow SDKs and runners to understand
the structure of user data and unlock relational optimization possibilities.
Portable schemas enable compatibility between rows in Python and Java.
A particularly interesting use case is the combination of SQL (implemented in Java)
with the Python SDK via Beam's cross-language support.
Learn more about portable schemas from this [presentation](https://s.apache.org/portable-schemas-seattle).
27 changes: 0 additions & 27 deletions website/www/site/content/en/roadmap/connectors-go-sdk.md

This file was deleted.

36 changes: 0 additions & 36 deletions website/www/site/content/en/roadmap/connectors-java-sdk.md

This file was deleted.

92 changes: 17 additions & 75 deletions website/www/site/content/en/roadmap/connectors-multi-sdk.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
---
title: "Multi-SDK Connector Efforts"
aliases:
- /roadmap/connectors-java-sdk/
---
<!--
Licensed under the Apache License, Version 2.0 (the "License");
Expand All @@ -15,88 +17,28 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

Connector-related efforts that will benefit multiple SDKs.
Roadmap for connectors and cross-cutting efforts that benefit multiple SDKs.

# Splittable DoFn
Splittable DoFn is the next generation sources framework for Beam that will
replace current frameworks for developing bounded and unbounded sources.
Splittable DoFn is being developed along side current Beam portability
efforts. See [Beam portability framework roadmap](/roadmap/portability/) for more details.
_Last updated on Aug 2026._

# Cross-language transforms
# Extended Metadata and SDK level CDC (Change Data Capture) support

_Last updated on May 2020._
Beam is introducing **Extended Metadata** to make element values extensible across SDKs. This provides native support for Change Data Capture (CDC) operations (`INSERT`, `UPDATE`, `DELETE`) consumed by lakehouse and database connectors, SDK-level pipeline drain, and fluent `OutputBuilder` APIs.

As an added benefit of Beam portability effort, we are able to utilize Beam transforms across SDKs. This has many benefits.
Learn more from the [Beam Element Extended Metadata Design Doc](https://s.apache.org/beam-element-extended-metadata).

* Connector sharing across SDKs. For example,
+ Beam pipelines written using Python and Go SDKs will be able to utilize the vast selection of connectors that are currently implemented for Java SDK.
+ Java SDK will be able to utilize connectors for systems that only offer a Python API.
+ Go SDK, will be able to utilize connectors currently available for Java and Python SDKs.
* Ease of developing and maintaining Beam transforms - in general, with cross-language transforms, Beam transform authors will be able to implement new Beam transforms using a
language of choice and utilize these transforms from other languages reducing the maintenance and support overheads.
* [Beam SQL](/documentation/dsls/sql/overview/), that is currently only available to Java SDK, will become available to Python and Go SDKs.
* [Beam TFX transforms](https://www.tensorflow.org/tfx/transform/get_started), that are currently only available to Beam Python SDK pipelines will become available to Java and Go SDKs.
# OpenTelemetry Support

## Completed and Ongoing Efforts
Beam is adding native [OpenTelemetry](https://opentelemetry.io/) support across SDKs, runners, and connectors ([#33176](https://github.com/apache/beam/issues/33176)) for distributed tracing. This leverages Extended Metadata to propagate W3C Trace Context (`traceparent` and `tracestate`) across transform stages and worker boundaries, integrates trace header propagation into streaming I/Os (such as Kafka, Pub/Sub, and Spanner change streams), and enables runner-level trace and log correlation.

Many efforts related to cross-language transforms are currently in flux. Some of the completed and ongoing efforts are given below.
# IO Connector Ecosystem

### Cross-language transforms API and expansion service
Beam is actively expanding the Beam I/O connector portfolio with a focus on modern data lakehouse formats, native Change Data Capture (CDC), and unified access patterns:

Work related to developing/updating the cross-language transforms API for Java/Python/Go SDKs and work related to cross-language transform expansion services.
* **First-Class Lakehouse Connectors**:
* **[Apache Iceberg](/documentation/io/built-in/iceberg/)**: Full support for batch and streaming reads, streaming appends, dynamic destinations, and changelog CDC reading. Available natively in Java, via cross-language transforms in Python, in [Beam YAML](/documentation/sdks/yaml/), and via the [Managed I/O](/documentation/io/managed-io/) API.
* **Delta Lake**: Native read capabilities ([#38551](https://github.com/apache/beam/issues/38551)) and batch changelog (CDC) reading ([#39492](https://github.com/apache/beam/issues/39492)), exposed through core transforms and the [Managed I/O](/documentation/io/managed-io/) API.
* **Managed I/O Expansion**: Exposing more storage systems through the [Managed I/O API](/documentation/io/managed-io/), which standardizes connector configurations using Beam Schemas, provides frictionless cross-language access (Java, Python, YAML), and enables runner-level optimizations.
* **Native CDC Integration**: Integrating database change streams (such as Spanner change streams and Debezium-based sources) directly into lakehouse sinks using Beam's native CDC `ValueKind` metadata.

* Basic API for Java SDK - completed
* Basic API for Python SDK - completed
* Basic API for Go SDK - In progress
* Basic cross-language transform expansion service for Java and Python SDKs - completed
* Artifact staging - mostly completed - [email thread](https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E), [doc](https://docs.google.com/document/d/1XaiNekAY2sptuQRIXpjGAyaYdSc-wlJ-VKjl04c8N48/edit#heading=h.900gc947qrw8)

### Support for Flink runner

Work related to making cross-language transforms available for Flink runner.

* Basic support for executing cross-language transforms on portable Flink runner - completed

### Support for Dataflow runner

Work related to making cross-language transforms available for Dataflow runner.

* Basic support for executing cross-language transforms on Dataflow runner
+ This work requires updates to Dataflow service's job submission and job execution logic. This is currently being developed at Google.

### Support for Direct runner

Work related to making cross-language transforms available on Direct runner

* Basic support for executing cross-language transforms on Pyton Direct runner - completed
* Basic support for executing cross-language transforms on Java Direct runner - Not started

### Connector/transform support

Ongoing and planned work related to making existing connectors/transforms available to other SDKs through the cross-language transforms framework.

* Java JdbcIO - completed - [BEAM-10135](https://issues.apache.org/jira/browse/BEAM-10135), [BEAM-10136](https://issues.apache.org/jira/browse/BEAM-10136)
* Java KafkaIO - completed - [BEAM-7029](https://issues.apache.org/jira/browse/BEAM-7029)
* Java KinesisIO - completed - [BEAM-10137](https://issues.apache.org/jira/browse/BEAM-10137), [BEAM-10138](https://issues.apache.org/jira/browse/BEAM-10138)
* Java PubSubIO - In progress - [BEAM-7738](https://issues.apache.org/jira/browse/BEAM-7738)
* Java SnowflakeIO - completed - [BEAM-9897](https://issues.apache.org/jira/browse/BEAM-9897), [BEAM-9898](https://issues.apache.org/jira/browse/BEAM-9898)
* Java SpannerIO - In progress - [BEAM-10139](https://issues.apache.org/jira/browse/BEAM-10139), [BEAM-10140](https://issues.apache.org/jira/browse/BEAM-10140)
* Java SQL - completed - [BEAM-8603](https://issues.apache.org/jira/browse/BEAM-8603)

### Portable Beam schema

Portable Beam schema support will provide a generalized mechanism for serializing and transferring data across language boundaries which will be extremely useful for pipelines that employ cross-language transforms.

* Make row coder a standard coder and implement in python - completed - [BEAM-7886](https://issues.apache.org/jira/browse/BEAM-7886)

### Integration/Performance testing

* Add an integration test suite for cross-language transforms on Flink runner - In progress - [BEAM-6683](https://issues.apache.org/jira/browse/BEAM-6683)

### Documentation

Work related to adding documenting on cross-language transforms to Beam Website.

* Document cross-language transforms API for Java/Python - Not started
* Document API for making existing transforms available as cross-language transforms for Java/Python - Not started
See the full list of available connectors in the [Beam Connectors Overview](/documentation/io/connectors/).
29 changes: 0 additions & 29 deletions website/www/site/content/en/roadmap/connectors-python-sdk.md

This file was deleted.

9 changes: 6 additions & 3 deletions website/www/site/content/en/roadmap/dataflow-runner.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,10 @@ limitations under the License.

# Google Cloud Dataflow Runner Roadmap

This roadmap is in progress. In the meantime, here are available resources:
Please check GitHub Issue tracker for recent developments on this topic:

- [Runner documentation](/documentation/runners/dataflow)
- Issues: [runner-dataflow](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Arunner-dataflow)
- Issues: [runner-dataflow](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Adataflow)

Available resources:

- [Runner documentation](/documentation/runners/dataflow)
2 changes: 2 additions & 0 deletions website/www/site/content/en/roadmap/euphoria.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ limitations under the License.

# Euphoria API

**Note** Euphoria is deprecated and scheduled for removal in a future release.

Easy to use Java 8 DSL for the Beam Java SDK. Provides a high-level abstraction of Beam transformations, which is both easy to read and write. Can be used as a complement to existing Beam pipelines (convertible back and forth). You can have a glimpse of the API at [WordCount example](/documentation/sdks/java/euphoria/#wordcount-example).

- Issues: [dsl-euphoria](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Adsl-euphoria) / [BEAM-3900](https://issues.apache.org/jira/browse/BEAM-3900)
Expand Down
20 changes: 17 additions & 3 deletions website/www/site/content/en/roadmap/flink-runner.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,21 @@ limitations under the License.

# Apache Flink Runner Roadmap

This roadmap is in progress. In the meantime, here are available resources:
_Last updated on Aug 2026._

- [Runner documentation](/documentation/runners/flink)
- Issues: [runner-flink](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Arunner-flink)
## Flink versions

Beam aims to support latest Flink minor versions. See [Flink Runner Support Table](/documentation/runners/flink#flink-version-compatibility)
for current coverage.

### Flink 2.x Support

One of the major changes in Flink 2 is the removal of Flink DataSet API. As a result Beam Flink 2 runners switched to use DataStream API for batch mode. One area of interest is to improve the batch support of DataStream API.

## Available resources

Please check GitHub Issue tracker for recent developments on this topic:

- Issues: [runner-flink](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Aflink)

- [Runner documentation](/documentation/runners/flink)
Loading
Loading