Skip to content

Lower scalar PQL selection into shared row-aggregate IR - #280

Open
Victsm wants to merge 2 commits into
codex/shared-numeric-aggregation-20261110from
codex/shared-scalar-ir-20261110
Open

Victsm wants to merge 2 commits into
codex/shared-numeric-aggregation-20261110from
codex/shared-scalar-ir-20261110

Conversation

@Victsm

@Victsm Victsm commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Keep the existing PQL compiler format, but lower compatible single-hop scalar aggregates into common row-selection/reduction operations. Carry filters, limits and explicit reduction policies through the same resolver, and share typed/grouped reduction primitives. Numeric limits retain their historical meaning; this does not invent a row cap for SUM/COUNT or change specialized list/LP/forecasting outputs.

Stack slice 2/5. Targets the preceding stack branch; merge in dependency order.

Change map

#280 (comment)

Testing

Release native build; 45 focused compiler/numeric/native forecasting tests passed at this checkpoint. The combined stack additionally passed 221 Rust TT/custom-TT/E2E tests (3 existing ignores).

Dependency order: #279 → #280 → #281 → #282 → #283.

@Victsm

Victsm commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

Change map

User-requested aggregation consolidation, slice 2/5.

Observed requirement and smallest sufficient change: Keep the existing PQL compiler format, but lower compatible single-hop scalar aggregates into common row-selection/reduction operations. Carry filters, limits and explicit reduction policies through the same resolver, and share typed/grouped reduction primitives. Numeric limits retain their historical meaning; this does not invent a row cap for SUM/COUNT or change specialized list/LP/forecasting outputs.

Focused verification / stopping condition: Release native build; 45 focused compiler/numeric/native forecasting tests passed at this checkpoint. The combined stack additionally passed 221 Rust TT/custom-TT/E2E tests (3 existing ignores). Stop at this slice’s verified behavior; fitting redesign, new temporal/LP capabilities and live sweeps are separate.

File Function / scope What changed Why
packages/diskgraph/diskgraph-core/src/accumulators.rs update_running_mean / WelfordAccumulator Extract the ordered running-mean update Reuse arithmetic while preserving caller-specific admission/variance policy
packages/diskgraph/diskgraph-core/src/tt_builder.rs DGTTPlan / RowAggregate / normalization Represent frames, grouped reductions and PQL selection/value policies Converge execution IR while keeping existing PQL compiler JSON compatible
packages/diskgraph/diskgraph-pybind/src/tt_builder.rs Python TT builder serialization Keep the established PQL plan format and inline enum representation Existing compiler/forecasting readers remain compatible after the larger shared IR variant
packages/diskgraph/diskgraph-train-table/src/lib.rs Component integration Connect the shared plan/execution contract Preserve the existing caller boundary
packages/diskgraph/diskgraph-train-table/src/ttgen_accumulator.rs Numeric batch accumulation / value hashing Share accumulation and typed grouping hash policy Retain full equality checks and frontend reduction behavior
packages/diskgraph/diskgraph-train-table/src/ttgen_executor.rs execute_numeric_group / execute_aggregate / stages Share numeric batch evaluation and ordered accumulation Keep precision, empty/NULL and fallback policies explicit
packages/diskgraph/diskgraph-train-table/src/ttgen_resolve.rs resolve_op / resolve_scalar_aggregate / paths and filters Resolve both frontend operations through shared handles and filter metadata Preserve PQL semantics and reuse traversal rather than reinterpret query policy
packages/diskgraph/diskgraph-train-table/src/ttgen_row_aggregate.rs reduce_rows / group_rows / merge_window_bounds Share typed reductions, nested grouping and window-bound union Avoid a second semantic implementation of these scalar operations
packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs Focused regression coverage Verify numerical, clock, identity, lifecycle or route behavior of this slice Catch wrong answers and unintended fallback/reconstruction, not merely successful calls

Verified repair — 2026-11-11

Head: 3ea9810c9bb89fd495664e9745bae8e5a492901d. Carries the repaired parent and adapts its fallback call to this slice’s shared executor signature.

Validation: repaired combined stack passed 146 Rust TT integration tests and 579 Python native/conformance tests with a freshly rebuilt extension. Public Python suite: 1,951 passed; optional-native import checks: two modules skipped as expected without DiskGraph. Generated artifact manifest and 403-component notice checks passed. No live sweeps or merges.

@copy-pr-bot

copy-pr-bot Bot commented Nov 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@Victsm
Victsm force-pushed the codex/shared-scalar-ir-20261110 branch from e27f36a to 3ea9810 Compare November 11, 2026 03:47
@Victsm
Victsm had a problem deploying to gha-kumo-workflow-tests November 11, 2026 03:47 — with GitHub Actions Error
@Victsm
Victsm deployed to gha-kumo-workflow-tests November 11, 2026 03:47 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
gha-kumo-workflow-tests — 3ea9810c Deployed Nov 11, 2026 by Victsm via import #2045
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant