Skip to content

Close the remaining native Array performance gaps (#505) - #540

Merged
ciaranra merged 4 commits into
devfrom
perf-505-remaining
Aug 17, 2026
Merged

Close the remaining native Array performance gaps (#505)#540
ciaranra merged 4 commits into
devfrom
perf-505-remaining

Conversation

@ciaranra

Copy link
Copy Markdown
Member

What

Fixes the remaining #505 items at the layer that owns each cost: fancy (integer-list) reads, boolean mask reads, array(list) construction, and astype widening. flatten() and f64->f32 were profiled, found already at NumPy speed on this host, and left untouched with tightened guards.

Performance (release build, 1e6 elements, best-of-5, PECOS/NumPy ratio)

operation before after
fancy read (every-7th list) 11x 0.52x
boolean mask read (~50%) 14x 1.30x
array(int_list) 14x 0.37x
array(float_list) 32x 0.35x
astype u8->i64 2.0x 1.07x
flatten() 1000x1000 1.06x unchanged (already copy-bound: 31% memmove)

Root causes (per-item perf profiles in the implementation report): per-element Python protocol dispatch during index/list parsing, per-element coordinate-vector allocation in selection, and a redundant full validation scan before provably-safe widening casts. Fixes: single-pass native index normalization, logical C-order offset gathering, native mask compaction, preallocated typed buffers for homogeneous built-in lists, and a range-subset bypass (source integer range contained in target integer range, computed in i128) that skips validation only when overflow is impossible -- narrowing still validates (300 -> uint8 still raises naming the value).

Guard ceilings tightened, never loosened

test_performance_guard.py ceilings drop to ~10x the new medians (fancy 200ms->20ms, mask 300ms->50ms), with new guards for both list constructors and f64->f32; 15/15 pass on release.

Verification (independent of the implementation run)

  • Full pecos-rslib suite: 2,103 pass; the 63-config bit-exact decode-equivalence fixture suite passes on the rewritten indexing paths.
  • Adversarial randomized fuzz vs the NumPy oracle: 2,700 comparisons over fancy reads (negative/repeated/empty/out-of-range), masks (densities, wrong-length raises, 2D Fortran layout), and casts -- zero divergence.
  • List-construction dtype inference spot-checked against NumPy: homogeneous lists all match; the mixed-type divergence found during review ([True, 1] -> bool) reproduces identically on dev and is filed as Mixed-type list construction infers dtype from the first element instead of promoting like NumPy #539, not introduced here.
  • Cold cargo clippy -p pecos-rslib --all-targets -- -D warnings, cargo fmt --check, unshielded pre-commit run --all-files: clean.

Implementation by Codex from a profile-first task packet; root causes, semantics, and benchmarks independently verified.

@ciaranra ciaranra self-assigned this Aug 17, 2026
@ciaranra
ciaranra merged commit 4f4efa6 into dev Aug 17, 2026
42 checks passed
@ciaranra
ciaranra deleted the perf-505-remaining branch August 17, 2026 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant