Describe the bug
pyarrow.compute.utf8_normalize returns decomposed text for every form. With form="NFC" the
output is the NFD form (base letter plus combining mark), whether the input was precomposed or
decomposed. unicodedata.normalize from the standard library gives the expected result.
pyarrow 25.0.1, Python 3.13.9, macOS 26.6 (arm64), wheel from PyPI.
import pyarrow as pa, pyarrow.compute as pc, unicodedata
s = "héllo" # precomposed e-acute, U+00E9
a = pa.array([s], pa.string())
for form in ("NFC", "NFD", "NFKC", "NFKD"):
out = pc.utf8_normalize(a, form=form).to_pylist()[0]
exp = unicodedata.normalize(form, s)
print(form, [hex(ord(c)) for c in out], "expected", [hex(ord(c)) for c in exp])
Output:
NFC ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68', '0xe9', '0x6c', '0x6c', '0x6f']
NFD ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f']
NFKC ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68', '0xe9', '0x6c', '0x6c', '0x6f']
NFKD ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f']
The same happens when the input is already decomposed ("héllo"): NFC and NFKC return it
unchanged instead of composing to U+00E9.
Expected behavior
form="NFC" and form="NFKC" compose, as unicodedata.normalize does; NFD and NFKD are correct
today.
Component(s)
Python, C++
Describe the bug
pyarrow.compute.utf8_normalizereturns decomposed text for everyform. Withform="NFC"theoutput is the NFD form (base letter plus combining mark), whether the input was precomposed or
decomposed.
unicodedata.normalizefrom the standard library gives the expected result.pyarrow 25.0.1, Python 3.13.9, macOS 26.6 (arm64), wheel from PyPI.
Output:
The same happens when the input is already decomposed (
"héllo"): NFC and NFKC return itunchanged instead of composing to U+00E9.
Expected behavior
form="NFC"andform="NFKC"compose, asunicodedata.normalizedoes; NFD and NFKD are correcttoday.
Component(s)
Python, C++