fix(transform): apply truncate to binary values - #3120
Open
jaideeppyne wants to merge 1 commit into
Open
Conversation
The spec lists binary as a valid source type for truncate[W], and Transform::result_type accepts it, but TransformFunction for Truncate rejects it in both paths: transform_literal has no Binary arm, and transform handles DataType::Binary but not DataType::LargeBinary, which is what schema_to_arrow_schema produces for an Iceberg binary column. Closes apache#3119
jaideeppyne
force-pushed
the
fix-truncate-binary
branch
from
August 31, 2026 13:40
b41907c to
e5aaa87
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
What changes are included in this PR?
The spec lists
binaryunder the source types fortruncate[W], and the Truncate Transform Details table givesv.subarray(0, L).Transform::result_typealready acceptsPrimitiveType::Binary, so a binary truncate partition field is legal, but neither half ofTransformFunction for Truncatecould actually handle one:transform_literalhad noPrimitiveLiteral::Binaryarm, soTransform::projectandstrict_projectreturnedFeatureUnsupportedfor every operator.InclusiveProjectioninscan/cache.rspropagates that, so a scan with a filter on such a column errors instead of pruning.transformhandledDataType::Binarybut notDataType::LargeBinary, andschema_to_arrow_schemamaps IcebergbinarytoLargeBinary. So the array path failed on the exact arrow type this crate produces for a binary column.truncate_binaryalready existed and was correct, it just was not reachable from either entry point. I added the two arms.fixedstays rejected, since the spec does not list it as a truncate source type, and the literal arm is guarded on the datum type so aFixeddatum (also backed byPrimitiveLiteral::Binary) still errors.Are these changes tested?
Yes, unit tests in
truncate.rs: array truncation forBinaryArrayandLargeBinaryArray, literal truncation (longer than width, shorter than width, empty), inclusive projection for</<=/>/>=/=/STARTS WITH/IN, strict projection for!=, and thatfixedis still rejected. Reverting just the two source arms and keeping the tests fails the 4 new tests and leaves the 16 pre-existing truncate tests passing. Fullcargo test -p icebergis green (1617 tests), fmt and clippy clean.I found this with a differential harness over
bucket[N],truncate[W],year/month/day/houron a 781-value corpus (spec test vectors plus randoms and boundary values across every source type), running the same inputs through pyiceberg 0.11.1, iceberg-go and this crate's literal and array paths. Of 9333 comparisons, 49 were binary truncate literals where pyiceberg and iceberg-go return a value and this crate errored. TheLargeBinaryhalf is not visible that way, I hit it after checking whatschema_to_arrow_schemaactually emits forbinary.Not covered:
BinaryView, which no path in this crate currently produces, and whichbucketdoes not handle either.AI Disclosure
Claude Code wrote the differential harness and drafted the patch and tests. I checked the spec clauses, verified every claim above against real test output, and reviewed the final diff.