[C++][Parquet] Make compression adaptive with V2 data pages

Describe the enhancement requested

In BuildDataPageV2, we enable compression as soon as the compressed size is smaller than the uncompressed size.

However, this adds overhead when reading even when the compression gain is very small. We could probably be slightly smarter and make the threshold configurable.

Component(s)

C++, Parquet

Hi 👋
While reviewing this section of the code, I noticed that a page is currently marked as compressed as soon as the compressed size is smaller than the uncompressed one:

if (compressor_temp_buffer_->size() < values->size()) {
    page_is_compressed = true;
}

This means even a very small compression gain (e.g., 1%) would still trigger compression, which could add overhead during reads due to decompression cost.

Would it make sense to introduce a configurable threshold (for example, require at least a 5–10% reduction before setting page_is_compressed = true)?

If so, what ratio or configuration mechanism would you prefer — a fixed constant, a build option, or a runtime setting?

Hi 👋
Since there was no feedback yet regarding the threshold ratio, I’ve gone ahead with a configurable approach via
WriterProperties::compression_threshold().

Default: 0.0f (preserves current behavior — always compress if smaller)

Configurable: users can set, for example, 0.05f (5% reduction) to avoid negligible compression gains.

This makes the behavior flexible while maintaining backward compatibility.

Please let me know if you’d prefer a different default or configuration style (e.g., compile-time constant or environment flag).

@harshkumar-2005

@harshkumar-2005 Perhaps we should use the same kind as vocabulary as in the IPC options:

arrow/cpp/src/arrow/ipc/options.h

Lines 72 to 85 in e044907

    
           /// \brief Minimum space savings percentage required for compression to be applied 
        
           /// 
        
           /// Space savings is calculated as (1.0 - compressed_size / uncompressed_size). 
        
           /// 
        
           /// For example, if min_space_savings = 0.1, a 100-byte body buffer won't undergo 
        
           /// compression if its expected compressed size exceeds 90 bytes. If this option is 
        
           /// unset, compression will be used indiscriminately. If no codec was supplied, this 
        
           /// option is ignored. 
        
           /// 
        
           /// Values outside of the range [0,1] are handled as errors. 
        
           /// 
        
           /// Note that enabling this option may result in unreadable data for Arrow C++ versions 
        
           /// prior to 12.0.0. 
        
           std::optional<double> min_space_savings;

Also cc @wgtmac @adamreeve

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

Working on it

Issue resolved by pull request 52580
#52580

apacheGH-47752: [C++][Python][Parquet] Add minimum compression saving…

Navigation Menu

Repository navigation

[C++][Parquet] Make compression adaptive with V2 data pages #47752

Describe the enhancement requested

Component(s)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	/// \brief Minimum space savings percentage required for compression to be applied
	///
	/// Space savings is calculated as (1.0 - compressed_size / uncompressed_size).
	///
	/// For example, if min_space_savings = 0.1, a 100-byte body buffer won't undergo
	/// compression if its expected compressed size exceeds 90 bytes. If this option is
	/// unset, compression will be used indiscriminately. If no codec was supplied, this
	/// option is ignored.
	///
	/// Values outside of the range [0,1] are handled as errors.
	///
	/// Note that enabling this option may result in unreadable data for Arrow C++ versions
	/// prior to 12.0.0.
	std::optional<double> min_space_savings;

[C++][Parquet] Make compression adaptive with V2 data pages #47752

Description

Describe the enhancement requested

Component(s)

Activity

harshkumar-2005 commented on Oct 23, 2025

harshkumar-2005 commented on Oct 24, 2025

pitrou commented on Oct 24, 2025

github-actions commented on Oct 25, 2026

MashrafeeAryan commented on Oct 25, 2026

rok commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions