Fix integer t.all() totals truncating to 0 on DataFusion - #307
Open
hussainsultan wants to merge 1 commit into
Open
Fix integer t.all() totals truncating to 0 on DataFusion#307hussainsultan wants to merge 1 commit into
hussainsultan wants to merge 1 commit into
Conversation
agg.mutate(share=lambda t: t.total / t.all(t.total)) routes t.all through attach_windowed_totals, whose __bsl_totals__<name> column kept the measure's integer dtype. xorq's DataFusion executes ibis truediv on two int64 operands as integer division, so every share came back 0.0 on the memtable → canonical-backend path — while the identical calc-measure spelling was correct (its path already casts via _float_total) and the duckdb-compiled SQL carried the cast. Apply the same integer→float64 policy where the totals columns are created: attach_windowed_totals casts the windowed total, and attach_calc_totals casts calc-of-calc totals, so an int-valued calc cannot reintroduce the truncation downstream. test_totals_semantics.py drops the duckdb pin (the inline spelling now asserts SUM_SHARE on the shared memtable fixture) and gains a dedicated regression test covering integer sum and count measures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hussainsultan
force-pushed
the
fix/int-totals-truncation
branch
from
August 24, 2026 06:16
a04c332 to
1de4f0c
Compare
hussainsultan
marked this pull request as ready for review
August 24, 2026 13:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
returned 0.0 for every share on the memtable → canonical-backend (xorq/DataFusion) path when the measure is integer-typed — silent-wrong, while the identical calc-measure spelling (
with_measures(share=...)) was correct and the duckdb-compiled SQL carried a float cast.Cause
attach_windowed_totals(calc_compiler.py) builds the lifted__bsl_totals__<name>column asagg_expr.over(window())with the measure's original dtype. For an integer SUM/COUNT measure that column stays int64, and xorq's DataFusion executes ibis truediv on two int64 operands as integer division (60 / 160 → 0), even though ibis types the result float64:The calc-measure path already guards against exactly this via
_float_totalinMeasureScope.all("with two integer operands some engines do integer division, and 30/160 came back as 0"); the inline lift missed the same policy.Fix
Apply the established integer→float64 cast where the totals columns are created:
attach_windowed_totalscasts the windowed total (_float_total, integer dtypes only — non-numeric totals untouched),attach_calc_totalscasts calc-of-calc totals the same way, so an int-valued calc can't reintroduce the truncation downstream.Casting the denominator makes the division int/float64, which DataFusion executes correctly (verified:
0.375 / 0.625).Tests
test_integer_ratio_via_aggregate_mutate_is_a_ratiocovers integer sum and count measures through the inline lift on the memtable fixture.SUM_SHAREon the shared memtable → DataFusion path directly.🤖 Generated with Claude Code