Skip to content

handle missing fields in SQL as null - #7295

Open
mccanne wants to merge 1 commit into
mainfrom
sql-nullish
Open

handle missing fields in SQL as null#7295
mccanne wants to merge 1 commit into
mainfrom
sql-nullish

Conversation

@mccanne

@mccanne mccanne commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

This commit improves SQL semantics in the face of missing fields by turning them into nulls and adding a nullish coalescing variant of the . operator to propagate nulls in a path expression. This operator is inserted by the semantic pass and is not user visible. It is displayed in DAG output as "??.".

To make this work, we fixed a problem in the optimizer where the flags in field.Chain weren't being propagated.

This also fixed a problem in agg functions like count which are supposed to ignore nulls. The change to change-star.yaml reflects this, where this test is now has a SQL-compatible result.

Closes #5984

This commit improves SQL semantics in the face of missing fields
by turning them into nulls and adding a nullish coalescing variant
of the . operator to propagate nulls in a path expression.  This operator
is inserted by the semantic pass and is not user visible.  It is displayed
in DAG output as "??.".

To make this work, we fixed a problem in the optimizer where the flags
in field.Chain weren't being propagated.

This also fixed a problem in agg functions like count which are supposed
to ignore nulls.  The change to change-star.yaml reflects this, where this
test is now has a SQL-compatible result.

Closes #5984
@philrz
philrz requested a review from a team September 10, 2026 15:55
@philrz

philrz commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

A run of this branch at 7c021ad on the current sqllogic-ztests triggered 124 new similar failures in the "groupby" set, such as groupby/slt_good_0/q2177.yaml. Here's a simplified, self-contained repro:

$ super -version &&
  super -c "SELECT * FROM (VALUES (1,2)) AS T(x,y) GROUP BY T.x, T.y"

Version: v0.3.0-379-g7c021ad3e

column "x" must appear in GROUP BY clause at line 1, column 8:
SELECT * FROM (VALUES (1,2)) AS T(x,y) GROUP BY T.x, T.y
       ~
column "y" must appear in GROUP BY clause at line 1, column 8:
SELECT * FROM (VALUES (1,2)) AS T(x,y) GROUP BY T.x, T.y
       ~

Whereas that works ok on current tip of main.

$ super -version &&
  super -c "SELECT * FROM (VALUES (1,2)) AS T(x,y) GROUP BY T.x, T.y"

Version: v0.3.0-378-g2338849bb

{x:1,y:2}

I had Claude come up with the simplified repro, so it also gave its take on root cause and possible fixes, and that's available in a Gist if it helps at all.

@philrz

philrz commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Another finding along the way that's mentioned in that Gist is that on this branch the null type is disappearing from unions when I do SELECT *.

$ echo '
    {col0:22::(int64|null),col1:6::(int64|null),col2:8::(int64|null)}
    {col0:28::(int64|null),col1:57::(int64|null),col2:45::(int64|null)}
    {col0:82::(int64|null),col1:44::(int64|null),col2:71::(int64|null)}' > tab1.sup &&
  super -version &&
  super -c "SELECT * FROM tab1.sup"

Version: v0.3.0-379-g7c021ad3e

{col0:22,col1:6,col2:8}
{col0:28,col1:57,col2:45}
{col0:82,col1:44,col2:71}

Whereas they're maintained on tip of main.

$ super -version &&
  super -c "SELECT * FROM tab1.sup"

Version: v0.3.0-378-g2338849bb

{col0:22::(int64|null),col1:6::(int64|null),col2:8::(int64|null)}
{col0:28::(int64|null),col1:57::(int64|null),col2:45::(int64|null)}
{col0:82::(int64|null),col1:44::(int64|null),col2:71::(int64|null)}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQL: NULL values absent from JOIN output

2 participants