Repository navigation
[C++][Python] run_end_decode and to_pandas fail on run-end encoded arrays with dictionary values #55489
Description
Describe the bug, including details regarding any error messages, version, and platform.
pc.run_end_decode and Table.to_pandas() both fail on a run-end encoded array whose values are dictionary-encoded:
import pyarrow as pa
import pyarrow.compute as pc
dict_array = pa.array(["a", "a", "b", "b"]).dictionary_encode()
runs = pc.run_end_encode(dict_array.indices)
ree = pa.RunEndEncodedArray.from_arrays(
runs.run_ends, pa.DictionaryArray.from_arrays(runs.values, dict_array.dictionary)
)
pc.run_end_decode(ree)
# ArrowNotImplementedError: Function 'run_end_decode' has no kernel matching input types
# (run_end_encoded<run_ends: int32, values: dictionary<values=string, indices=int32, ordered=0>>)
pa.table({"c": ree}).to_pandas()
# Same error. The pandas conversion decodes the runs with run_end_decode.The IPC format allows this type, and Arrow servers send it. For example, Deephaven sends run_end_encoded<dictionary<string>> for a column with long runs of a few distinct values. A client that reads such a stream can't convert it to pandas or decode it.
The expected result of run_end_decode is a dictionary array with the same dictionary, as if the runs of indices were decoded:
indices = pc.run_end_decode(pa.RunEndEncodedArray.from_arrays(ree.run_ends, ree.values.indices))
pa.DictionaryArray.from_arrays(indices, ree.values.dictionary)#40659 added run-end encoded support to to_pandas, but not for dictionary values.
Version: pyarrow 26.0.1, pandas 3.1.1, Python 3.15, macOS.
Component(s)
C++, Python
Activity