[lake/hudi][docs] Add Hudi tiering service documentation by fhan688 · Pull Request #3549 · apache/fluss

fhan688 · 2026-06-30T12:19:48Z

Purpose

Linked issue: #3514

This PR adds user-facing documentation for using Fluss Tiering Service with Apache Hudi.

The Hudi lake integration has already been implemented in the fluss-lake-hudi module, but the website did
not yet provide a dedicated guide that explains how to configure Hudi as lakehouse storage, start the tiering
service, create Hudi-backed Fluss tables, and read tiered data.

Brief change log

Add a new Hudi integration guide under streaming-lakehouse/integrate-data-lakes/formats.
Document Hudi dependencies, version compatibility, DFS/HMS catalog configuration, server-side plugin JARs,
and Flink tiering service startup.
Describe the Fluss-to-Hudi table mapping based on the implementation:
- primary-key Fluss tables are mapped to Hudi Merge-On-Read tables;
- log tables are mapped to Hudi Copy-On-Write tables;
- Hudi record key, bucket index, partition path fields, and Fluss system columns are managed consistently
  with the code.
Document Hudi table properties, union read behavior, direct Hudi reads through native Hudi catalog, type
mapping, auto compaction, commit metadata, and current limitations.
Update existing documentation/config descriptions to list Hudi as a supported lakehouse format.

Tests

git diff --check
mvn -q -pl fluss-common -am -DskipTests -DskipITs -Dcheckstyle.skip=true compile

Website build was not run locally because website/node_modules is not installed in the current workspace.

API and Format

No API or storage format changes.

This PR only adds documentation and updates option descriptions to reflect that Hudi is now a supported
lakehouse format.

Documentation

This PR introduces the Hudi Tiering Service integration guide and updates existing lakehouse storage
documentation to include Hudi.

Copilot

Pull request overview

Adds a dedicated, user-facing Apache Hudi integration guide to the Fluss website and updates existing configuration/option descriptions to include Hudi as a supported lakehouse (datalake) format.

Changes:

Added a comprehensive Hudi Tiering Service guide (dependencies, configuration, tiering job startup, table mapping, reads, and limitations).
Updated tiered storage and lakehouse docs to list Hudi among supported formats.
Updated configuration/option descriptions (website + ConfigOptions) to reflect Hudi support.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
website/docs/streaming-lakehouse/integrate-data-lakes/formats/hudi.md	New end-to-end documentation for configuring and using Hudi as Fluss lakehouse storage.
website/docs/maintenance/tiered-storage/overview.md	Updates supported lakehouse formats list to include Hudi.
website/docs/maintenance/tiered-storage/lakehouse-storage.md	Updates lakehouse storage description to include Hudi as supported.
website/docs/maintenance/configuration.md	Updates `datalake.format` documentation to include Hudi.
website/docs/engine-flink/options.md	Updates `table.datalake.format` documentation to include `hudi`.
fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java	Updates option descriptions to include `hudi` in supported datalake formats.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

luoyuxia · 2026-07-06T03:32:07Z

@fhan688 Could you please rebase main branch since we already reorganize doc for lake integration

fhan688 · 2026-07-09T03:20:56Z

please help review, thanks! @XuQianJin-Stars

XuQianJin-Stars · 2026-07-18T03:09:42Z

-                            "The data lake format of the table specifies the tiered Lakehouse storage format. Currently, supported formats are `paimon`, `iceberg`, and `lance`. "
-                                    + "In the future, more kinds of data lake format will be supported, such as DeltaLake or Hudi. "
+                            "The data lake format of the table specifies the tiered Lakehouse storage format. Currently, supported formats are `paimon`, `iceberg`, `hudi`, and `lance`. "
+                                    + "In the future, more kinds of data lake format will be supported, such as DeltaLake. "


"more kinds of data lake format will be supported". Since "kinds" is plural, "format" should be "formats":

... more kinds of data lake format will be supported ... ^^^^^^^ -> formats

XuQianJin-Stars · 2026-07-18T03:12:08Z

+Fluss supports the Hudi catalog modes implemented by the connector:
+
+- `dfs`: Hudi DFS catalog. Tables are stored under `${catalog.path}/${database_name}/${table_name}`.
+- `hms`: Hudi Hive Metastore catalog. Configure Hive Metastore access through Hudi/Hadoop options, for example `hive.conf.dir` or `hive.metastore.uris`.


Hive Metastore prose vs example key form (under "Fluss supports the Hudi catalog modes implemented by the connector:", the hms bullet) The prose reads:
Configure Hive Metastore access through Hudi/Hadoop options, for example hive.conf.dir or hive.metastore.uris.
while the example directly above uses the prefixed key datalake.hudi.hive.metastore.uris. A reader may be confused about which literal key to put in server.yaml. Suggest aligning the prose to the prefixed form actually used in the example:
Configure Hive Metastore access through Hudi/Hadoop options, for example datalake.hudi.hive.conf.dir or datalake.hudi.hive.metastore.uris.

XuQianJin-Stars · 2026-07-18T03:13:04Z

+
+Conceptually, Fluss creates a Hudi table with properties equivalent to:
+
+```sql title="Hudi table properties"


cluster/server options use datalake.hudi.* (stripped by Fluss before passing to the catalog), while table-level Fluss options use hudi.* (stripped before passing to Hudi). This distinction is implicit and easy to miss. A one-line note such as "Cluster options use the datalake.hudi. prefix; table-level Hudi options use the hudi. prefix — both are stripped before being forwarded" would prevent misconfiguration. Minor, but worth it given Hudi has many options.

XuQianJin-Stars · 2026-07-18T03:15:38Z

+
+| Scenario | Required JAR |
+|----------|--------------|
+| Hudi lake connector | [fluss-lake-hudi-$FLUSS_VERSION$.jar](https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-hudi/$FLUSS_VERSION$/fluss-lake-hudi-$FLUSS_VERSION$.jar) |


URL base style The Hudi connector JAR link uses a hard-coded https://repo1.maven.org/maven2/.../fluss-lake-hudi/$FLUSS_VERSION$/..., which is consistent with the sibling Paimon/Iceberg/Lance rows in the same file (lines ~149, ~158, ~167). No change needed; just noting it is internally consistent. (hudi.md itself uses the $FLUSS_MAVEN_REPO_URL$ variable, which is the other documented convention — also fine.)

luoyuxia

Left two inline comments on the server-side Hudi plugin dependencies and the Hive Metastore URI configuration examples.

luoyuxia · 2026-07-18T03:32:54Z

+Put the following JARs into `${FLUSS_HOME}/plugins/hudi/` before starting the Fluss servers:
+
+- [fluss-lake-hudi-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-hudi/$FLUSS_VERSION$/fluss-lake-hudi-$FLUSS_VERSION$.jar)
+- The Hudi Flink bundle matching your runtime. For Flink 1.20, use [hudi-flink1.20-bundle-1.1.0.jar](https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-flink1.20-bundle/1.1.0/hudi-flink1.20-bundle-1.1.0.jar).


Could we also include the Flink core and Table API/runtime JARs required by the server-side plugin? fluss-lake-hudi directly imports classes such as org.apache.flink.configuration.Configuration, org.apache.flink.table.catalog.Catalog, and DataTypes, while its flink-core, flink-table-common, flink-table-api-java, and flink-table-runtime dependencies are declared as provided. hudi-flink1.20-bundle-1.1.0.jar does not contain these classes because it assumes an existing Flink runtime.

From my codex, but the concern make sense to.

luoyuxia · 2026-07-18T03:32:54Z

+datalake.format: hudi
+datalake.hudi.mode: hms
+datalake.hudi.catalog.path: hdfs:///warehouse/hudi
+datalake.hudi.hive.metastore.uris: thrift://<hive-metastore-host>:9083


This option will not reach HiveConf in Hudi 1.1.0. Fluss first strips datalake.hudi., producing hive.metastore.uris; HadoopConfigurations.getHadoopConf then copies only options prefixed with hadoop. and strips that prefix. Consequently METASTOREURIS remains empty and HoodieHiveCatalog rejects the embedded metastore. Please use datalake.hudi.hadoop.hive.metastore.uris here and in the tiering/deployment examples. The native Hudi catalog example should use hadoop.hive.metastore.uris. Alternatively, hive.conf.dir can point to a directory containing hive-site.xml.

From my codex, but not sure about it.

luoyuxia

@fhan688 Thanks for the pr. LGTM overall. Only left minor comments. PTAL

luoyuxia · 2026-07-22T02:33:28Z

@fhan688 Could you please help rebase main branch again?

luoyuxia · 2026-07-22T02:38:29Z

@fhan688 Could you please help rebase main branch again?

luoyuxia requested a review from Copilot July 2, 2026 11:43

Copilot started reviewing on behalf of luoyuxia July 2, 2026 11:44 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

fhan688 force-pushed the Introduce-document-for-tiering-service-to-hudi branch from 0d0b790 to b9d6272 Compare July 8, 2026 04:03

XuQianJin-Stars reviewed Jul 18, 2026

View reviewed changes

luoyuxia reviewed Jul 18, 2026

View reviewed changes

luoyuxia approved these changes Jul 20, 2026

View reviewed changes

fhan688 closed this Jul 20, 2026

fhan688 reopened this Jul 20, 2026

fhan688 added 2 commits July 22, 2026 14:56

[lake/hudi][docs] Add Hudi tiering service documentation

1309ea3

[docs] Address review comments for Hudi tiering

3e47148

fhan688 force-pushed the Introduce-document-for-tiering-service-to-hudi branch from d78fa18 to 3e47148 Compare July 22, 2026 07:02

luoyuxia merged commit 1ca13b4 into apache:main Jul 22, 2026
19 checks passed


		Conceptually, Fluss creates a Hudi table with properties equivalent to:

		```sql title="Hudi table properties"

Uh oh!

Conversation

fhan688 commented Jun 30, 2026

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

luoyuxia commented Jul 6, 2026

Uh oh!

fhan688 commented Jul 9, 2026

Uh oh!

XuQianJin-Stars Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

XuQianJin-Stars Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

XuQianJin-Stars Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

XuQianJin-Stars Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

luoyuxia left a comment

Choose a reason for hiding this comment

Uh oh!

luoyuxia Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luoyuxia Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

luoyuxia Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

luoyuxia Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

luoyuxia left a comment

Choose a reason for hiding this comment

Uh oh!

luoyuxia commented Jul 22, 2026

Uh oh!

luoyuxia commented Jul 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

luoyuxia Jul 18, 2026 •

edited

Loading