GH-3735: Use unsigned UTF-8 byte order for Variant object field keys

[peterxcli]

Rationale for this change

The Variant specification requires an object's field ids to be sorted by the unsigned byte order of the field names' UTF-8 encoding, so that a reader can binary search them. VariantBuilder sorted those fields with String.compareTo, which orders UTF-16 code units, and Variant.getFieldByKey searched them with the same comparison.

The two orders agree on every name in the Basic Multilingual Plane, so they only diverge for a name holding a character above U+FFFF. String.compareTo places a leading high surrogate (0xD800 to 0xDBFF) below code points in U+E000 to U+FFFF, while UTF-8 byte order places it above. An object that parquet-java writes with such a name therefore carries its field ids in an order the specification forbids, and a reader that follows the specification can miss fields when it binary searches. The gap runs the other way too: parquet-java could fail to find a name above U+FFFF in an object written by another implementation.

This work started from #3736 by @rayokota. It also carries over the fallback that lets a reader still find names in objects written in the old order, which came from the equivalent Spark fix in apache/spark#58239 and the discussion there.

What changes are included in this PR?

VariantUtil.compareKeys(String, String) orders two field names the way their UTF-8 encodings compare as unsigned bytes, without encoding either one. UTF-8 byte order is exactly code point order, so the method walks the UTF-16 code units and adjusts for surrogates: a surrogate always encodes a code point above U+FFFF, so U+D800 to U+DFFF rank above U+E000 to U+FFFF, and the latter shift down to fill the gap they leave. VariantBuilder.FieldEntry.compareTo sorts an object's fields through this method, so the sort allocates nothing.

Variant.getFieldByKey runs its usual String.compareTo binary search whenever the lookup key holds no code unit at or above U+D800. Such a key compares the same way under both orders, so a single search finds it in an object written either way.

A lookup key that does hold such a code unit goes to a separate method, which searches in UTF-8 byte order and then searches again in the UTF-16 order that older versions wrote, so their objects stay readable. Giving it its own method keeps the common search small. Folding both attempts into getFieldByKey cost ordinary lookups about 15%, because the larger method inlines worse.

Are these changes tested?

Four tests in TestVariantObjectBuilder cover the fix.

• testObjectKeysSortedByUtf8ByteOrder builds an object whose names are U+FFFF (EF BF BF) and U+10000 (F0 90 80 80), appends them in reverse, and asserts the encoded order is U+FFFF then U+10000. The old comparison put them the other way around.
• testLargeObjectBinarySearchWithSupplementaryKey builds an object of 42 fields, above BINARY_SEARCH_THRESHOLD, mixing ASCII names with U+FFFF and U+10000, and asserts getFieldByKey finds both through the binary search.
• testLegacyUtf16OrderedObjectLookup rewrites a canonical object's id and offset lists into the old UTF-16 order, then asserts getFieldByKey still finds the ASCII, U+FFFF, and U+10000 names, and still returns null for a name the object does not hold.
• testCompareKeysMatchesUtf8ByteOrder checks compareKeys against Arrays.compareUnsigned on the two names' UTF-8 bytes, over names covering every UTF-8 length, both sides of the surrogate range, prefixes, and 200 random names. It leaves out unpaired surrogates, which have no UTF-8 encoding because Java's encoder substitutes ? for them, as the method documents.

All 184 parquet-variant tests pass, along with the parquet-avro variant read and write suites.

I also measured lookups on an object of 256 fields, alternating JVM runs on Zulu 21 and Apple silicon and taking the best of seven rounds per run, in ns/op.

Lookup	master	this PR
ASCII name, present	44-46	48
ASCII name, absent	21-22	22-23
Name above U+FFFF	37-40	41-42

The ASCII rows run the same binary search as before, and the small difference is the single scan of the lookup key for a code unit at or above U+D800. The last row now searches in UTF-8 order, and searches a second time when the object uses the old order.

Are there any user-facing changes?

An object written from now on orders a field name above U+FFFF as the specification requires. getFieldByKey still finds such names in objects already written in the UTF-16 order.

Closes #3735

[divjotarora]

One comment about potential perf issues, non-blocking as obviously correctness is more important. Also, great test coverage!

[divjotarora]

IIUC VariantUtil.encodeKey(midKey) means we now do an allocation on every iteration, correct? Can we measure the impact of this? Would it be better to cache the encoded keys, similar to getMetadataKeyCached?

[wgtmac]

Or do something like this?

boolean needsUtf8 = containsCodeUnitAtLeast(key, Character.MIN_SURROGATE);
byte[] keyBytes = needsUtf8 ? VariantUtil.encodeKey(key) : null;

// binary-search loop:
int cmp = needsUtf8
    ? VariantUtil.compareKeys(VariantUtil.encodeKey(midKey), keyBytes)
    : midKey.compareTo(key);