feat(storage): Add HDFS support via opendal services-hdfs-native

[mixermt]

Which issue does this PR close?

• Closes Support tables which live on HDFS #2440.
• Part of HDFS support in iceberg-rust #1130.

This revives #1131's successor #2441 by @jordepic (closed by the stale bot after a first review round), rebased onto the current Storage-trait layout and updated for opendal 0.58, where services-hdfs-native changed behavior in ways that required design changes (details below).

What changes are included in this PR?

Adds an opendal-hdfs-native cargo feature to iceberg-storage-opendal, with OpenDalStorageFactory::HdfsNative / OpenDalStorage::HdfsNative variants and hdfs:// routing in OpenDalResolvingStorage. The backend uses OpenDAL's services-hdfs-native (pure-Rust HDFS RPC via hdfs-native — no JNI/libhdfs). The feature is experimental and not part of opendal-all, matching opendal-oss/opendal-azdls.

NameNode resolution (differs from #2441, forced by opendal 0.58 where name_node is mandatory and its comma-split list is the HA mechanism):

1. hdfs.name-node property when set — a single endpoint or a comma-separated list for HA failover (new HDFS_NAME_NODE / HDFS_HADOOP_CONF_PREFIX constants in iceberg::io).
2. Otherwise the path authority (hdfs://host:port/path).
3. Authority-less paths without the property are rejected with a pointed error (opendal 0.58 can no longer defer to fs.defaultFS).

hadoop.-prefixed properties pass through to the HDFS client config, overriding $HADOOP_CONF_DIR values (mirroring the hadoop. catalog-property convention of the Java integrations); hdfs-native still loads core-site.xml/hdfs-site.xml from $HADOOP_CONF_DIR/$HADOOP_HOME for everything else, and Kerberos works via libgssapi_krb5 (runtime dlopen). Operators are cached per effective NameNode since each holds live RPC connections.

Relative paths are returned opendal-style without a leading / — opendal::Deleter::delete (used by delete_stream) rejects leading slashes, which an integration test caught.

Test infrastructure: single-node HDFS docker fixture (apache/hadoop:3.5.0, host networking — required because hdfs-native dials DataNodes by their registered IP, unroutable on a bridge). The DataNode healthcheck gates on NameNode registration so --wait means writable. Tests are #[ignore]d (host networking is Linux-only); CI opts in via cargo nextest --run-ignored=only -E 'test(file_io_hdfs)'.

Are these changes tested?

• 24 unit tests across config parsing, path parsing (authority/port/authority-less/wrong-scheme), NameNode precedence, operator caching, relativize, and scheme resolution; 3 config tests in iceberg::io.
• 12 integration tests against the docker fixture covering exists/read/write/overwrite/metadata/range reads/streaming writes/delete/delete_prefix/delete_stream (+empty), resolving storage over hdfs://, and the HA flow (logical authority in the path + hdfs.name-node property). All 12 verified green from a cold cluster locally; the suite runs in CI on Linux.
• make check, full-workspace --all-features lib tests, and the existing s3/gcs/resolving integration suites pass; public-api.txt regenerated for both crates.

AI Disclosure

Developed with AI assistance (Claude Code): drafting code/tests/fixtures starting from #2441, and cross-checking the design against the vendored opendal 0.58.1 / hdfs-native 0.14.5 sources. I reviewed the implementation and ran all verification locally. Areas worth reviewer attention: the NameNode-resolution semantics above (opendal's synthetic-nameservice behavior constrains what hdfs://<nameservice> paths can do without the property), and the Windows --all-features build of hdfs-native, which I could only verify via CI.

🤖 Generated with Claude Code

[blackmwk]

I found two correctness issues that need to be addressed before merge: malformed hdfs: URLs can panic, and bulk deletion can conflate NameNodes running on different ports. Details and suggested regression coverage are inline. Please also resolve the two existing review threads.

---

This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg Rust maintainer. After you've addressed the points above and pushed an update, an Apache Iceberg Rust maintainer — a real person — will take the next look at the PR. The findings cite the project's review criteria; if you think one of them is mis-applied, please reply on the PR and a maintainer will weigh in.

More on how Apache Iceberg Rust handles maintainer review: CONTRIBUTING.md.

[blackmwk]

Thanks @mixermt for this pr. I think the variant should be named to Hdfs-native rather than hdfs. Hdfs should be reserved for non native hdfs support.