Skip to content

feat: Python table functions on Snowflake - #1610

Merged
kvlonge merged 6 commits into
mainfrom
feat/python-table-functions
Nov 20, 2026
Merged

kvlonge merged 6 commits into
mainfrom
feat/python-table-functions

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Why

Per-group work such as smoothing a series is hard in SQL, and projects had no Python table functions, so it lived outside the project.

Changes

  • functions/python/<name>.py may define one @table_fn(...) function with returns as a column-to-type map. It gets one whole partition as a data frame and returns rows.
  • polars by default; a pd.DataFrame annotation picks pandas, and [defaults] function_dataframe = "pandas" changes the project default.
  • Calls need OVER (PARTITION BY ...) in models, functions and SQL tests, or compilation fails. Snowflake renders TABLE(fn(args) OVER (...)); analysis and SQL tests treat the clause as part of the call.
  • Snowflake deploys a generated vectorized handler that names inputs, converts the frame, checks returned columns (missing, extra, repeated) and adds pandas (plus polars, pyarrow).
  • A polars function with a fractional NUMBER/DECIMAL/DEC/NUMERIC argument fails compilation.
  • The frame joins the fingerprint only when set, so existing functions keep their identity. Other adapters still reject Python table functions.
  • Docs: new section in functions.mdx; fixed runtime_version (required), the adapter table, and the testing reference's claim that models can't depend on table functions.

Goldens: config-unknown-default-key lists the new allowed key; the macro-expansion oracle gains the call's partition flag.

DuckShop: n/a; DuckDB has no Python table functions.

Verification

  • Fail-first tests for parsing, detection, both errors, the setting, fingerprint, OVER handling, Snowflake DDL and the generated handler.
  • Integration: Snowflake projects compile both frames and the default and reject the errors.
  • Live Snowflake: a polars and a pandas function passed TEST(mode table_fn), and a wrong-columns function failed naming them; schema dropped.
  • make test (only stale-local-version checks fail) and make check-ci on the final head.

Add @table_fn(...) for whole-partition Python table functions. Each call
receives one partition as a data frame, polars by default or pandas from a
pd.DataFrame annotation or the [defaults] function_dataframe setting, and
returns a data frame matching its declared returns columns.

Calls must use OVER (PARTITION BY ...), checked at compile time in models,
function bodies and SQL tests. Polars functions reject fractional NUMBER
arguments. Snowflake deploys a vectorized end_partition handler that checks
the returned columns.
Comments before OVER and inside its clause no longer hide the partition.
The generated Snowflake handler prefixes its locals so any entry point name
works, and names repeated returned columns. DEC counts as a fixed-point
type for polars arguments. The docs state how the frame annotation is read.
The partition check, analysis, SQL-test markers and Snowflake rendering
now skip each dialect's own line comments, such as Snowflake's //, before
OVER and inside it.
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 20, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 20, 2026
@kvlonge
kvlonge enabled auto-merge November 20, 2026 22:16
@kvlonge
kvlonge added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 6352fa6 Nov 20, 2026
36 checks passed
@kvlonge
kvlonge deleted the feat/python-table-functions branch November 20, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant