Skip to content

Support SAMPLE, DERIVATIVE and INTEGRAL table functions - #19227

Open
alpass163 wants to merge 4 commits into
apache:masterfrom
alpass163:cyb/integral_sample_derivative_tz_function
Open

alpass163 wants to merge 4 commits into
apache:masterfrom
alpass163:cyb/integral_sample_derivative_tz_function

Conversation

@alpass163

Copy link
Copy Markdown
Contributor

Adds three built-in relational functions.

SAMPLE(DATA => table, N => n)

Table function that performs reservoir sampling over whole rows, with n rows
per partition. The first n rows fill the reservoir; each following row replaces
a uniformly chosen slot with probability n / (rows seen so far). All input
columns are kept, partition columns first, and the sampled rows are emitted in
input order.

DERIVATIVE(DATA => table PARTITION BY ... ORDER BY time, [unit => duration])

Table function that emits one output row per pair of adjacent points in ORDER
BY time within each partition. The output time is the time of the second point,
and each calculation column c outputs (c2 - c1) / (t2 - t1) * unit. unit
defaults to 1s in the current timestamp precision; a null on either side only
makes that column null for the pair.

INTEGRAL(value [, time_unit [, time]])

Aggregate function that computes the trapezoidal integral of value over
time, divided by time_unit. time_unit defaults to 1s and time defaults
to the time column of the table. Ordered and naive accumulators are provided for
both grouped and ungrouped aggregation; the naive ones support intermediate
states so partial results can be merged in distributed aggregation.

Bug fixes

Fixes an existing planning bug for table functions that must run once per
partition (mergeChildrenForSingleProcessor in TableDistributedPlanGenerator):

  • When the processor had several children, they were merged with the ordering
    common to all children, or by plain concatenation when there was none. The
    merged stream was therefore not ordered by the GROUP clause (partition keys
    followed by the in-partition ordering), and rows of one partition could arrive
    non-consecutively when a partition spans several children, e.g. PARTITION BY
    on non-device columns or a device spanning regions. Such a partition was then
    treated as several partitions. The merged input now uses the ordering of the
    GroupNode and inserts an explicit SortNode when the children do not already
    provide it.
  • canSplitTableFunctionProcessor only excluded FFT from per-child splitting.
    M4, FORECAST, PATTERN_MATCH, CLASSIFY, LOWPASS, HIGHPASS, XCORR, SESSION,
    VARIATION and CAPACITY could be cloned for every child, so one logical
    partition could be handled by several processors with separate state. They are
    now listed in NON_SPLITTABLE_TABLE_FUNCTIONS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant