[jules-ch]

Rationale for this change

See #55443, conversion to pyarrow for masked or datetime values are slow because of scanning and validity bitmap creation.

What changes are included in this PR?

• Update ValuesToBitmap and MaskToBitmap, letting compiler vectorize null_count computation and setting bitmap for null values.

import timeit

import numpy as np
import pyarrow as pa

n = 25_920_000
values = np.random.rand(n)
dates = np.arange(n).astype("datetime64[s]")
mask = np.zeros(n, dtype=bool)

for label, fn in {
    "float64 (zero-copy)     ": lambda: pa.array(values),
    "float64 mask=           ": lambda: pa.array(values, mask=mask),
    "float64 from_pandas=True": lambda: pa.array(values, from_pandas=True),
    "datetime64              ": lambda: pa.array(dates),
}.items():
    t = min(timeit.repeat(fn, number=1, repeat=5))
    print(f"{label} {t * 1e3:.1f} ms")

pyarrow 26.0.1
float64 (zero-copy)      0.0 ms
float64 mask=            34.0 ms
float64 from_pandas=True 45.1 ms
datetime64               45.7 ms

pyarrow 26.0.2.dev1+gb025b2107
float64 (zero-copy)      0.0 ms
float64 mask=            3.2 ms
float64 from_pandas=True 6.6 ms
datetime64               6.6 ms

Are these changes tested?

Are there any user-facing changes?

This PR includes breaking changes to public APIs. (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)

This PR contains a "Critical Fix". (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

• Human
• AI

Reviewed before submission by:

• Human
• AI
• Not reviewed

• GitHub Issue: [Python] [Perf] Speed up null bitmap creation in NumPy to Arrow conversion #55443

[github-actions]

Thanks for opening a pull request!

This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format.

If this is not a minor PR, could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

After updating the title, you can mark the pull request as ready for review.

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #55443 has been automatically assigned in GitHub to PR creator.