Skip to content

[lake/hudi][docs] Add Hudi tiering service documentation #3549

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
luoyuxia merged 2 commits into from
Jul 22, 2026
Merged

[lake/hudi][docs] Add Hudi tiering service documentation #3549

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1309ea3
[lake/hudi][docs] Add Hudi tiering service documentation
fhan688 Jun 30, 2026
3e47148
[docs] Address review comments for Hudi tiering
fhan688 Jul 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1798,8 +1798,8 @@ public class ConfigOptions {
.enumType(DataLakeFormat.class)
.noDefaultValue()
.withDescription(
"The data lake format of the table specifies the tiered Lakehouse storage format. Currently, supported formats are `paimon`, `iceberg`, and `lance`. "
+ "In the future, more kinds of data lake format will be supported, such as DeltaLake or Hudi. "
"The data lake format of the table specifies the tiered Lakehouse storage format. Currently, supported formats are `paimon`, `iceberg`, `hudi`, and `lance`. "
+ "In the future, more kinds of data lake formats will be supported, such as DeltaLake. "
+ "Once the `table.datalake.format` property is configured, Fluss adopts the key encoding and bucketing strategy used by the corresponding data lake format. "
+ "This ensures consistency in key encoding and bucketing, enabling seamless **Union Read** functionality across Fluss and Lakehouse. "
+ "The `table.datalake.format` can be pre-defined before enabling `table.datalake.enabled`. This allows the data lake feature to be dynamically enabled on the table without requiring table recreation. "
Expand Down Expand Up @@ -2415,8 +2415,8 @@ public class ConfigOptions {
.enumType(DataLakeFormat.class)
.noDefaultValue()
.withDescription(
"The datalake format used by Fluss as lakehouse storage. Currently, supported formats are Paimon, Iceberg, and Lance. "
+ "In the future, more kinds of data lake format will be supported, such as DeltaLake or Hudi.");
"The datalake format used by Fluss as lakehouse storage. Currently, supported formats are Paimon, Iceberg, Hudi, and Lance. "
+ "In the future, more kinds of data lake format will be supported, such as DeltaLake.");

// ------------------------------------------------------------------------
// ConfigOptions for tiering service
Expand Down
2 changes: 1 addition & 1 deletion website/docs/engine-flink/options.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ See more details about [ALTER TABLE ... SET](engine-flink/ddl.md#set-properties)
| table.kv.standby-replica.enabled | Boolean | (None) | Whether to enable standby replicas for primary key tables. Standby replicas maintain recent KV snapshots for fast leader promotion. Automatically set to `true` by the coordinator during table creation for new PK tables. Tables created before this option was introduced are treated as disabled. Can be dynamically enabled via `ALTER TABLE SET ('table.kv.standby-replica.enabled' = 'true')`. |
| table.log.tiered.local-segments | Integer | 2 | The number of log segments to retain in local for each table when log tiered storage is enabled. It must be greater that 0. The default is 2. |
| table.datalake.enabled | Boolean | false | Whether enable lakehouse storage for the table. Disabled by default. When this option is set to ture and the datalake tiering service is up, the table will be tiered and compacted into datalake format stored on lakehouse storage. |
| table.datalake.format | Enum | (None) | The data lake format of the table specifies the tiered Lakehouse storage format. Currently, supported formats are `paimon`, `iceberg`, and `lance`. In the future, more kinds of data lake format will be supported, such as DeltaLake or Hudi. Once the `table.datalake.format` property is configured, Fluss adopts the key encoding and bucketing strategy used by the corresponding data lake format. This ensures consistency in key encoding and bucketing, enabling seamless **Union Read** functionality across Fluss and Lakehouse. The `table.datalake.format` can be pre-defined before enabling `table.datalake.enabled`. This allows the data lake feature to be dynamically enabled on the table without requiring table recreation. If `table.datalake.format` is not explicitly set during table creation, the table will default to the format specified by the `datalake.format` configuration in the Fluss cluster. |
| table.datalake.format | Enum | (None) | The data lake format of the table specifies the tiered Lakehouse storage format. Currently, supported formats are `paimon`, `iceberg`, `hudi`, and `lance`. In the future, more kinds of data lake format will be supported, such as DeltaLake. Once the `table.datalake.format` property is configured, Fluss adopts the key encoding and bucketing strategy used by the corresponding data lake format. This ensures consistency in key encoding and bucketing, enabling seamless **Union Read** functionality across Fluss and Lakehouse. The `table.datalake.format` can be pre-defined before enabling `table.datalake.enabled`. This allows the data lake feature to be dynamically enabled on the table without requiring table recreation. If `table.datalake.format` is not explicitly set during table creation, the table will default to the format specified by the `datalake.format` configuration in the Fluss cluster. |
| table.datalake.freshness | Duration | 3min | It defines the maximum amount of time that the datalake table's content should lag behind updates to the Fluss table. Based on this target freshness, the Fluss service automatically moves data from the Fluss table and updates to the datalake table, so that the data in the datalake table is kept up to date within this target. If the data does not need to be as fresh, you can specify a longer target freshness time to reduce costs. |
| table.datalake.auto-compaction | Boolean | false | If true, compaction will be triggered automatically when tiering service writes to the datalake. It is disabled by default. |
| table.datalake.auto-expire-snapshot | Boolean | false | If true, snapshot expiration will be triggered automatically when tiering service commits to the datalake. It is disabled by default. |
Expand Down
2 changes: 1 addition & 1 deletion website/docs/engine-spark/reads.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ ORDER BY order_id;
When a table has the configuration `table.datalake.enabled = 'true'`, its data exists in two layers:

- **Fresh data** is retained in Fluss (real-time layer, sub-second freshness)
- **Historical data** is tiered to the lake storage (e.g., Paimon, Iceberg)
- **Historical data** is tiered to the lake storage (e.g., Paimon, Iceberg, Hudi)

Fluss Spark connector supports **union read** that combines both layers to provide a complete, up-to-date view of the data. This allows Fluss to store only a small portion of the dataset in the cluster (reducing costs), while the lake serves as the source of complete historical data.

Expand Down
60 changes: 55 additions & 5 deletions website/docs/install-deploy/deploying-streaming-lakehouse.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Configure lakehouse settings in `server.yaml` on all Fluss servers (CoordinatorS
Fluss follows a simple convention for these keys:

- `datalake.enabled: true` explicitly enables lakehouse capability for the cluster. If it is left unset, configuring `datalake.format` alone also enables it; whenever `datalake.enabled` is `true`, `datalake.format` must also be set.
- `datalake.format` selects the lake format for the cluster (`paimon`, `iceberg`, or `lance`).
- `datalake.format` selects the lake format for the cluster (`paimon`, `iceberg`, `hudi`, or `lance`).
- Format-specific options use the `datalake.<format>.*` prefix. Fluss strips this prefix and passes the remaining keys straight to the corresponding lake catalog/client — for example `datalake.paimon.metastore` becomes `metastore` and `datalake.iceberg.type` becomes `type`. Any option supported by the lake catalog can be set this way.
- Only configure options for the format selected by `datalake.format`.

Expand Down Expand Up @@ -71,6 +71,25 @@ datalake.iceberg.warehouse: s3://bucket/iceberg
datalake.iceberg.io-impl: org.apache.iceberg.aws.s3.S3FileIO
```

</TabItem>
<TabItem value="hudi" label="Hudi">

```yaml title="server.yaml"
datalake.enabled: true
datalake.format: hudi
datalake.hudi.mode: dfs
datalake.hudi.catalog.path: /path/to/hudi
```

For Hive Metastore catalog:
```yaml title="server.yaml"
datalake.enabled: true
datalake.format: hudi
datalake.hudi.mode: hms
datalake.hudi.catalog.path: hdfs:///warehouse/hudi
datalake.hudi.hadoop.hive.metastore.uris: thrift://<hive-metastore-host>:9083
```

</TabItem>
<TabItem value="lance" label="Lance">

Expand Down Expand Up @@ -122,6 +141,16 @@ Add JARs to `${FLUSS_HOME}/plugins/<format>/` based on your configuration:
| Iceberg with S3 | [`iceberg-aws-<version>.jar`, `iceberg-aws-bundle-<version>.jar`](https://iceberg.apache.org/docs/1.10.1/aws/), matching your Iceberg version |
| Iceberg JDBC catalog | PostgreSQL/MySQL JDBC driver |

</TabItem>
<TabItem value="hudi" label="Hudi">

| Scenario | Required JAR |
|----------|--------------|
| Hudi lake connector | [fluss-lake-hudi-$FLUSS_VERSION$.jar](https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-hudi/$FLUSS_VERSION$/fluss-lake-hudi-$FLUSS_VERSION$.jar) |

@XuQianJin-Stars XuQianJin-Stars Jul 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

URL base style The Hudi connector JAR link uses a hard-coded https://repo1.maven.org/maven2/.../fluss-lake-hudi/$FLUSS_VERSION$/..., which is consistent with the sibling Paimon/Iceberg/Lance rows in the same file (lines ~149, ~158, ~167). No change needed; just noting it is internally consistent. (hudi.md itself uses the $FLUSS_MAVEN_REPO_URL$ variable, which is the other documented convention — also fine.)

| Hudi Flink bundle | [hudi-flink1.20-bundle-1.1.0.jar](https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-flink1.20-bundle/1.1.0/hudi-flink1.20-bundle-1.1.0.jar), matching the Flink version used by the Hudi integration |
| Hudi Flink dependencies | `flink-core`, `flink-table-common`, `flink-table-api-java`, and `flink-table-runtime`, matching the Hudi Flink bundle |
| Hudi Hive Metastore catalog | Hive and Hadoop dependencies required by your environment |

</TabItem>
<TabItem value="lance" label="Lance">

Expand Down Expand Up @@ -160,6 +189,14 @@ Add the following to `${FLINK_HOME}/lib`:
- Hadoop client JARs — export `HADOOP_CLASSPATH`, download the pre-bundled [`hadoop-apache-3.3.5-2.jar`](https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar), or install a full Hadoop package (see [Iceberg Hadoop Dependencies](../streaming-lakehouse/datalake-formats/iceberg.md#prerequisites-hadoop-dependencies))
- JDBC driver (if using JDBC catalog)

</TabItem>
<TabItem value="hudi" label="Hudi">

- [fluss-flink-1.20-$FLUSS_VERSION$.jar](https://repo1.maven.org/maven2/org/apache/fluss/fluss-flink-1.20/$FLUSS_VERSION$/fluss-flink-1.20-$FLUSS_VERSION$.jar)
- [fluss-lake-hudi-$FLUSS_VERSION$.jar](https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-hudi/$FLUSS_VERSION$/fluss-lake-hudi-$FLUSS_VERSION$.jar)
- [hudi-flink1.20-bundle-1.1.0.jar](https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-flink1.20-bundle/1.1.0/hudi-flink1.20-bundle-1.1.0.jar), matching your Flink runtime
- Hadoop, Hive, or filesystem dependencies required by your Hudi catalog and table storage

</TabItem>
<TabItem value="lance" label="Lance">

Expand Down Expand Up @@ -199,6 +236,17 @@ ${FLINK_HOME}/bin/flink run /path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \
--datalake.iceberg.warehouse "s3://bucket/iceberg"
```

</TabItem>
<TabItem value="hudi" label="Hudi">

```shell
${FLINK_HOME}/bin/flink run /path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \
--fluss.bootstrap.servers localhost:9123 \
--datalake.format hudi \
--datalake.hudi.mode dfs \
--datalake.hudi.catalog.path /tmp/hudi
```

</TabItem>
<TabItem value="lance" label="Lance">

Expand Down Expand Up @@ -237,14 +285,16 @@ CREATE TABLE my_table (
## Verification

1. Check the Tiering Service job is running in Flink Web UI
2. After the freshness interval, query the lake table:
2. After the freshness interval, query the table:

```sql title="Flink SQL"
-- Lake-only query
SELECT * FROM my_table$lake;

-- Union Read (real-time + historical)
SELECT * FROM my_table;

-- Lake-only query for formats exposed through the Fluss lake catalog
SELECT * FROM my_table$lake;
```

For Hudi lake-only reads, use Hudi's native catalog or another Hudi-compatible engine. The Fluss catalog `$lake` suffix currently exposes Paimon and Iceberg lake tables.

See [Union Read](../streaming-lakehouse/union-read.md) for more details.
3 changes: 2 additions & 1 deletion website/docs/install-deploy/overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ We have listed them in the table below the figure.
<td>
<li>[Paimon](../streaming-lakehouse/datalake-formats/paimon.md)</li>
<li>[Iceberg](../streaming-lakehouse/datalake-formats/iceberg.md)</li>
<li>[Hudi](../streaming-lakehouse/datalake-formats/hudi.md)</li>
<li>[Lance](../streaming-lakehouse/datalake-formats/lance.md)</li>
</td>
</tr>
Expand All @@ -147,4 +148,4 @@ Fluss can be deployed in three different ways:
- [Docker run / Docker Compose](install-deploy/deploying-with-docker.md)

**NOTE**:
- Local Cluster is for testing purpose only.
- Local Cluster is for testing purpose only.
Loading
Loading