fix(variant): convert shredded timestamps to the column's time unit - #882
jackylee-ch wants to merge 2 commits into
Conversation
ShreddedValue::Timestamp always carries microseconds, but timestamp_array wrote that integer into an Arrow array whose unit follows the declared precision, and timestamp_value_at read it back unconverted. Outside precision 4..=6 both sides were off by 1000x. Precision 7..=9 maps to a NANOS leaf, one of the four timestamp forms parquet-format's shredded-value table permits, so a conforming file written by another engine was also mis-read. Convert on both sides instead of reinterpreting: the write scales micros to the leaf unit, rejecting a micros value that cannot be expressed in i64 nanoseconds, and the read keys off the array's own TimeUnit so a foreign file decodes correctly regardless of the local option.
|
Reviewed head b75b8b9. Requirement fit: SUPPORTED. The seven focused shredded-timestamp tests pass, but two follow-ups remain. Integration/rebase follow-up: main commit d05cb03 (#884) added |
… precision-zero tests and old-writer compat fixture
2 participants
ShreddedValue::Timestampis always micros, but the shredded leaf's Arrow unit follows the declared precision (MILLIS <4, MICROS 4..=6, NANOS >6). Both sides passed the integer through unchanged, so a TIMESTAMP leaf outside 4..=6 was 1000x off. A NANOS leaf is spec-conforming, so foreign files were mis-read too.Write now scales micros to the leaf unit, flooring into MILLIS, and errors on micros beyond i64 nanoseconds (~1677..2262) instead of wrapping. Read keys off the array's TimeUnit, so a foreign file decodes from its own schema.
Disclosure: files paimon-rust already wrote outside 4..=6 hold micros under a MILLIS/NANOS annotation. They read back correctly only because the reader repeated the mistake; after this change they decode 1000x wrong, and nothing in the file tells them apart. Only an explicit
variant.shreddingSchemaproduces such a leaf — inference picks precision 6.