What is the problem the feature request solves?
Native RLike documents that is_match hand-rolls a loop over the input string array instead of calling Arrow's regexp_is_match(_scalar), which recompiles the pattern on every batch (see arrow-string regexp.rs). #5102 (and follow-up PRs) rewrote that loop (e.g. iter/map/collect). The rationale lives in docs, but nothing in native/spark-expr/benches/ measures it, so future changes to this hot path can evaporate without a baseline.
Describe the potential solution
Add native/spark-expr/benches/rlike.rs alongside the existing regexp_extract.rs Criterion bench, and register it in native/spark-expr/Cargo.toml ([[bench]], harness = false).
Suggested coverage (mirroring review discussion):
- Scalar pattern, Utf8 subject array (common path through
is_match)
- Mix of matches / non-matches / nulls
- Optionally Dictionary(Int32, Utf8), since that is the reachable
dictionary shape in Comet today
This does not need to land in the string-layout / dictionary fix PR;
it is follow-up so the rewritten loop has a measurable baseline.
Additional context
Requested in review on the RLike string-layout PR: the null-check removal is likely lost in noise next to Regex::is_match, so this is not a merge blocker, but worth tracking rather than dropping.
Reference: native/spark-expr/benches/regexp_extract.rs
Related PR: #5215 (review)
What is the problem the feature request solves?
Native RLike documents that
is_matchhand-rolls a loop over the input string array instead of calling Arrow'sregexp_is_match(_scalar), which recompiles the pattern on every batch (see arrow-stringregexp.rs). #5102 (and follow-up PRs) rewrote that loop (e.g.iter/map/collect). The rationale lives in docs, but nothing innative/spark-expr/benches/measures it, so future changes to this hot path can evaporate without a baseline.Describe the potential solution
Add
native/spark-expr/benches/rlike.rsalongside the existingregexp_extract.rsCriterion bench, and register it innative/spark-expr/Cargo.toml([[bench]],harness = false).Suggested coverage (mirroring review discussion):
is_match)dictionary shape in Comet today
This does not need to land in the string-layout / dictionary fix PR;
it is follow-up so the rewritten loop has a measurable baseline.
Additional context
Requested in review on the RLike string-layout PR: the null-check removal is likely lost in noise next to
Regex::is_match, so this is not a merge blocker, but worth tracking rather than dropping.Reference:
native/spark-expr/benches/regexp_extract.rsRelated PR: #5215 (review)