test(file_index): verify Rust reads of Java-generated indexed tables

[liwuhen]

Purpose

Rust currently has two complementary forms of FileIndex coverage:

• Java golden payload tests that validate index encoding and predicate evaluation.
• Rust write, commit, reload, and query tests that validate the internal Rust
  read/write path.

However, there is no end-to-end test that opens a table written and committed
through Apache Paimon Java's normal table-writing path and then reads and queries
that table through Rust.

Without this coverage, the combined compatibility of the FileIndex payload,
DataFileMeta, manifest metadata, and embedded or sidecar index paths is not
directly protected by a cross-language regression test.

This change adds reproducible Java-generated table fixtures and verifies that
Rust can load and use their FileIndexes through the normal scan and read path.

The coverage includes:

• Bitmap and Bloom Filter indexes.
• Embedded and .index sidecar storage.
• Duplicate values, NULL, matching predicates, and non-matching predicates.
• FileIndex reads enabled and disabled.
• Row-level residual filtering after FileIndex evaluation.
• An observable data-file skip that cannot be attributed to manifest statistics.

This is test coverage only. It does not indicate a known interoperability bug
and does not add new index types or writing modes.

Brief change log

• Add four committed tables generated by Apache Paimon Java:
  • bitmap_embedded
  • bitmap_sidecar
  • bloom_filter_embedded
  • bloom_filter_sidecar
• Generate the fixtures through Java's normal catalog, append writer, commit,
  snapshot, and manifest paths.
• Record the exact Apache Paimon Java commit, table options, input rows, and
  expected query results.
• Retain the Java fixture generator and a regeneration script.
• Add a Rust integration test that:
  • Loads the committed table schema and metadata.
  • Plans and executes queries through the normal table read path.
  • Compares results with file-index.read.enabled=true and false.
  • Verifies duplicate and NULL values.
  • Verifies row-level residual filtering.
  • Checks the expected embedded or sidecar metadata representation.
  • Uses the existing FileIOProvider observation pattern to prove that
    FileIndex can skip opening the Parquet data file.
• Gate the filesystem-backed integration test on the storage-fs feature and
  exclude Windows, consistently with existing filesystem fixture tests.

The fixtures were generated from Apache Paimon Java commit
1d368b4a5932f8221fd28e2555001abdb8fb12ee (2.2-SNAPSHOT).

The fixture rows, in physical order, are:

• (1, "keep")
• (1, "drop")
• (NULL, "null-id")
• (3, "three")

The observable skip predicate is id = 2. It lies inside the data file's
[1, 3] min/max range, so manifest statistics retain the file. With FileIndex
reads enabled, Rust returns an empty result without opening the Parquet file.
With FileIndex reads disabled, Rust opens the Parquet file and row-level
filtering produces the same empty result.

Tests

• Integration test reads_java_written_file_indexes_and_prunes_data_files
  (file_index_java_fixture_test.rs):
  • Covers Bitmap and Bloom Filter indexes in embedded and sidecar forms.
  • Verifies that id = 1 returns both duplicate rows.
  • Verifies that id IS NULL returns the null row.
  • Verifies that id = 1 AND payload = 'keep' retains row-level residual
    filtering.
  • Verifies that id = 2 returns no rows with FileIndex reads both enabled and
    disabled.
  • Verifies that manifest data statistics do not prune the id = 2 query.
  • Verifies that the data file is not opened when FileIndex reads are enabled.
  • Verifies that the data file is opened when FileIndex reads are disabled.
• The Java fixture generator completed successfully for all four tables.
• Commands run locally:
  • cargo test --locked -p paimon --test file_index_java_fixture_test
  • cargo test --locked -p paimon --no-default-features --test file_index_java_fixture_test --no-run
  • cargo clippy --locked -p paimon --test file_index_java_fixture_test -- -D warnings
  • cargo fmt --all -- --check
  • bash -n crates/paimon/testdata/file_index/regenerate.sh
  • git diff --check

API and Format

No public API or persisted data format changes.

This change only adds cross-language compatibility fixtures, fixture-generation
support, and integration-test coverage. Production FileIndex readers, writers,
metadata handling, and path resolution are not modified.

The scope does not include:

• Rust-written tables queried by Java.
• Range-bitmap writing.
• Primary-key or KV writing.
• Copy-on-write or data-evolution rewrites.
• Backfilling indexes for existing files.
• Performance benchmarks.
• New production metrics or public observability APIs.

Documentation

Add fixture documentation describing:

• The exact Apache Paimon Java commit and version.
• The generator source and regeneration command.
• Bitmap and Bloom Filter configuration.
• Embedded and sidecar thresholds.
• Fixture rows and expected query results.
• The rationale for the observable FileIndex skip predicate.