GH-3780: Decouple ParquetReadOptions from hadoop-mapreduce-client-core

[jerolba]

Rationale for this change

Reading the Parquet read configuration via ParquetReadOptions previously referenced
ParquetInputFormat.getFilter(...) and statically imported its constants. ParquetInputFormat extends
org.apache.hadoop.mapreduce.lib.input.FileInputFormat, so using the read configuration forced the JVM
to initialize ParquetInputFormat and therefore FileInputFormat and its whole
org.apache.hadoop.mapreduce.* transitive dependency graph, even though only plain String/boolean
config properties were needed. The new ParquetInputProperties (constants) and ParquetInputFilters
(filter resolution) centralise the ParquetConfiguration-based read configuration so that read-only
consumers no longer transitively pull in the Hadoop mapreduce dependency.

What changes are included in this PR?

New files

• parquet-hadoop/src/main/java/org/apache/parquet/conf/ParquetInputProperties.java
  • Declares all Parquet read-configuration constants previously on ParquetInputFormat:
    READ_SUPPORT_CLASS, UNBOUND_RECORD_FILTER, STRICT_TYPE_CHECKING, FILTER_PREDICATE,
    RECORD_FILTERING_ENABLED, STATS_FILTERING_ENABLED, DICTIONARY_FILTERING_ENABLED,
    COLUMN_INDEX_FILTERING_ENABLED, PAGE_VERIFY_CHECKSUM_ENABLED, BLOOM_FILTERING_ENABLED,
    OFF_HEAP_DECRYPT_BUFFER_ENABLED, HADOOP_VECTORED_IO_ENABLED, HADOOP_VECTORED_IO_DEFAULT.
  • Constants only; no org.apache.hadoop.mapreduce dependency.
• parquet-hadoop/src/main/java/org/apache/parquet/conf/ParquetInputFilters.java
  • ParquetConfiguration-based filter resolution: getFilter(ParquetConfiguration),
    getUnboundRecordFilter(ParquetConfiguration) and getFilterPredicate(ParquetConfiguration).
  • No org.apache.hadoop.mapreduce dependency.

Removed

• parquet-hadoop/src/main/java/org/apache/parquet/ParquetInputConfiguration.java (intermediate
  class, split into the two classes above).

Modified

• parquet-hadoop/.../ParquetReadOptions.java
  • Static imports now resolve to ParquetInputProperties, and the getFilter(...) call to
    ParquetInputFilters instead of ParquetInputFormat. ParquetReadOptions no longer references
    ParquetInputFormat.
• parquet-hadoop/.../hadoop/InternalParquetRecordReader.java
  • Static imports of RECORD_FILTERING_ENABLED / STRICT_TYPE_CHECKING now from
    ParquetInputProperties.
• parquet-hadoop/.../hadoop/ParquetInputFormat.java
  • Constants are now @Deprecated, delegating to ParquetInputProperties.*.
  • Kept the legacy Hadoop Configuration-based getFilter(Configuration) /
    getUnboundRecordFilter(Configuration) overloads (used by the MapReduce read path), delegating by
    wrapping into HadoopParquetConfiguration. Internal callers use ParquetInputFilters /
    ParquetInputProperties directly.
• Tests updated accordingly (DeprecatedInputFormatTest, TestParquetFileWriter,
  TestInputOutputFormat, TestInputFormatColumnProjection, TestDataPageChecksums,
  TestColumnChunkPageWriteStore, TestPropertiesDrivenEncryption).

Are these changes tested?

No change in behavior, just refactoring code location. Code compilation and existing tests validate the change.

Are there any user-facing changes?

No. Fully backward and binary compatible: the ParquetInputFormat constants and methods are retained
(deprecated) and delegate to the new class; constant string/boolean values are unchanged, so any
previously-serialised configuration still works.

Closes #3780

[dossett]

+1 (non-binding), this seems delightful!

It would be really nice to have a test that proves a file can be read without the hadoop libraries in the classpath or otherwise accessible. I think the easiest way to do that would be to create a new maven module that excludes hadoop-mapreduce-client-core entirely and write a unit test in that module that shows ParquetReadOptions.builder().build() works. That seems like a lot for a single unit test, but it would provide a solid guarantee.

[jerolba]

I agree that it would be nice to have it tested, but it will add a lot of complexity to the project just for a test setup.

I've tested this change in Carpet removing hadoop-mapreduce-client-core dependency and, based on the code path used by the library, it works. The change doesn't ensure that usages of other parquet-hadoop classes will not require the dependency.

Unfortunately, we are far from removing the dependency of hadoop-common. I think that it will require a complete rewrite of the parquet-hadoop module.

[dossett]

I was curious how complex the test would be, and it doesn't seem too bad. What do you think?

https://github.com/dossett/parquet-java/pull/1/changes

(that's a hypothetical PR against this branch)