GH-50398 [Python] Use more CPython Limited APIs in python/pyarrow#50409
GH-50398 [Python] Use more CPython Limited APIs in python/pyarrow#50409mroeschke wants to merge 2 commits into
Conversation
|
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or See also: |
mroeschke
changed the title
|
|
There was a problem hiding this comment.
Pull request overview
This PR refactors PyArrow’s C/Cython extension code to reduce reliance on CPython internal struct-field access and macro APIs, moving toward wider use of CPython Limited API-compatible functions (as part of the long-term goal of supporting abi3 wheels).
Changes:
- Introduces
internal::PyObject_StdStringTypeName()(based onPyType_GetName) and switches multiple error paths to use it instead of readingPy_TYPE(obj)->tp_name. - Replaces several CPython macros / struct accesses with function-based APIs (e.g.,
PyFloat_AsDouble,PyTuple_GetItem,PyList_SetItem,PyBytes_Size/AsString,PyType_GetSlot,PyType_GetFlags). - Switches the
MonthDayNanostruct-sequence type initialization to a cachedPyStructSequence_NewType()pattern.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| python/pyarrow/src/arrow/python/udf.cc | Use limited-API helper for Python type names in error messages. |
| python/pyarrow/src/arrow/python/python_to_arrow.cc | Replace macros with function forms for float/tuple/list APIs. |
| python/pyarrow/src/arrow/python/pyarrow.cc | Use limited-API helper for type names in unwrap error messages. |
| python/pyarrow/src/arrow/python/numpy_to_arrow.cc | Replace bytes/tuple macros with function APIs. |
| python/pyarrow/src/arrow/python/io.cc | Use limited-API helper for type names in IO error messages. |
| python/pyarrow/src/arrow/python/inference.cc | Use limited-API helper for dict-key type name error messages. |
| python/pyarrow/src/arrow/python/helpers.h | Declare the new PyObject_StdStringTypeName() helper. |
| python/pyarrow/src/arrow/python/helpers.cc | Implement type-name helper; switch to PyType_GetSlot / PyType_GetFlags. |
| python/pyarrow/src/arrow/python/extension_type.cc | Use limited-API helper for extension instance type name in ToString(). |
| python/pyarrow/src/arrow/python/decimal.cc | Use limited-API helper for type names in decimal conversion errors. |
| python/pyarrow/src/arrow/python/datetime.cc | Switch MonthDayNano struct-sequence init to cached PyStructSequence_NewType. |
| python/pyarrow/src/arrow/python/common.h | Include helpers and replace bytes macro access; improve type-name errors. |
| python/pyarrow/src/arrow/python/benchmark.cc | Replace list access macro with function form. |
| python/pyarrow/src/arrow/python/arrow_to_pandas.cc | Replace list-set macro with PyList_SetItem. |
| python/pyarrow/io.pxi | Replace PyBytes_AS_STRING usage with PyBytes_AsString in Cython. |
|
There are CI failures due to an unexpected deprecation warning, see python/cpython#124182 (comment) for a potential solution. |
| } | ||
|
|
||
| std::string PyObject_StdStringTypeName(PyObject* obj) { | ||
| // Once Python 3.10 is dropped, this can use PyType_GetName(Py_TYPE(obj)) (added in 3.11). |
There was a problem hiding this comment.
To be fair we are just about to release 25.0.0, Release Candidate already created and being voted, which is the last release which will support Python 3.10. We could just drop 3.10 from main any time now.
github-actions
Bot
added
awaiting changes
There was a problem hiding this comment.
We should probably remove this file instead. Nobody ever runs.
| bytes = PyByteArray_AS_STRING(obj); | ||
| size = PyByteArray_GET_SIZE(obj); |
There was a problem hiding this comment.
Why not replace these as well? Or are they part of the Limited API?
| bytes = PyBytes_AsString(obj); | ||
| size = PyBytes_Size(obj); |
There was a problem hiding this comment.
Need to check for errors here, at least as a debug assertion (since those functions shouldn't fail on a bytes object).
| } | ||
|
|
||
| const int32_t length = static_cast<int32_t>(PyBytes_GET_SIZE(utf8_obj.obj())); | ||
| const int32_t length = static_cast<int32_t>(PyBytes_Size(utf8_obj.obj())); |
| } | ||
| PyArray_Descr* sub_dtype = | ||
| reinterpret_cast<PyArray_Descr*>(PyTuple_GET_ITEM(tup, 0)); | ||
| reinterpret_cast<PyArray_Descr*>(PyTuple_GetItem(tup, 0)); |
| PyList_SET_ITEM(bytes_field_names_.obj(), i, bytes); | ||
| PyList_SET_ITEM(unicode_field_names_.obj(), i, unicode); | ||
| PyList_SetItem(bytes_field_names_.obj(), i, bytes); | ||
| PyList_SetItem(unicode_field_names_.obj(), i, unicode); |
Rationale for this change
What changes are included in this PR?
This is an agent generated PR that swaps implementations to use CPython Limited APIs and required less reworks.
Py_TYPE(obj)->tp_namestruct-field reads with a new limited-API helper (PyObject_StdStringTypeName, built onPyType_GetName)tp_as_number->nb_intslot access withPyType_GetSlot(..., Py_nb_int)PyType_HasFeaturemacro withPyType_GetFlags(...)bit testsPyTypeObject + PyStructSequence_InitType2pattern with a cachedPyStructSequence_NewTypeAre these changes tested?
Yes
Are there any user-facing changes?
No