feat: ddl statements + catalog by clintropolis · Pull Request #19830 · apache/druid

clintropolis · 2026-07-30T17:45:53Z

Description

This PR adds DDL statement support to Druid so that CREATE and ALTER statements can be used to manage the catalog contents. This functionality requires WRITE permissions to the datasource (same as the backing catalog APIs), and is gated behind a new runtime property druid.sql.planner.enableCatalogDdl which is false by default. This PR includes support for defining the logical schema (columns list in catalog), aggregate projections (projections property in catalog), base table projection (baseTable property in catalog), clustering (clusterKeys in catalog), time partitioning (segmentGranularity in catalog), and any other properties (sealed, targetSegmentRows) with generic property setting syntax.

These operations are purely metadata operations, and so DROP TABLE has been omitted since in my mind it has the most room for confusion (a DROP TABLE only cleared metadata and didn't do anything else sounds kind of confusing to me). Dropping columns and projections from a table felt more easily explainable that these do not modify any existing data and instead stage the schema for subsequent ingestion (and future work should wire this catalog stuff better into compaction/reindexing so that we can also frame it such that compaction/reindexing will begin to eventually converge on the updated schema).

These all go through the regular query path, and return 0 rows on successful operation. Some follow-up work is needed to improve web-console syntax highlighting and behaviors, and since there is no drop table statement it would still probably be nice to eventually add a catalog management ui, but the statements at least already work as-is through the current query interface.

Some examples:

create table:

CREATE TABLE "wikipedia" (
  "channel" VARCHAR,
  "__time" TIMESTAMP,
  "page" VARCHAR,
  "namespace" VARCHAR,
  "user" VARCHAR,
  "comment" VARCHAR,
  "added" BIGINT,
  "delta" BIGINT
)
PARTITIONED BY DAY
CLUSTERED BY "channel"

alter table to add column:

ALTER TABLE "wikipedia" ADD COLUMN "deleted" BIGINT

alter table to add projection:

ALTER TABLE "wikipedia" ADD PROJECTION "channel_sums" AS (
  SELECT 
    TIME_FLOOR("__time", 'PT1H'),
    "channel",
    SUM("added") as "sum_added",
    SUM("delta") as "sum_delta",
    SUM("deleted") as "sum_deleted"
  GROUP BY 1,2
)

alter table set property:

ALTER TABLE "wikipedia" SET PROPERTIES (sealed = true)

alter table set 'base table' projection to create clustered segments:

ALTER TABLE "wikipedia" ADD PROJECTION __base AS (
  SELECT
    "channel",
    "__time",
    "page",
    "namespace,"
    "user",
    "comment",
    "added",
    "delta",
    "deleted"
  CLUSTERED BY "channel"
)

create table with base table and projection definitions:

CREATE TABLE "wikipedia" (
  "channel" VARCHAR,
  "__time" TIMESTAMP,
  "page" VARCHAR,
  "namespace" VARCHAR,
  "user" VARCHAR,
  "comment" VARCHAR,
  "added" BIGINT,
  "delta" BIGINT,
  "deleted" BIGINT,
  PROJECTION __base AS (
    SELECT
      "channel",
      "__time",
      "page",
      "namespace",
      "user",
      "comment",
      "added",
      "delta",
      "deleted"
    CLUSTERED BY "channel"
  ),
  PROJECTION "channel_sums" AS (
    SELECT 
      TIME_FLOOR("__time", 'PT1H'),
      "channel",
      SUM("added") as "sum_added",
      SUM("delta") as "sum_delta",
      SUM("deleted") as "sum_deleted"
    GROUP BY 1,2
  ),
  PROJECTION "channel_page_max" AS (
    SELECT 
      TIME_FLOOR("__time", 'PT1H'),
      "channel",
      "page",
      MAX("added") as "sum_added",
      MAX("delta") as "sum_delta",
      MAX("deleted") as "sum_deleted"
    GROUP BY 1,2,3
  )
)
PARTITIONED BY DAY
CLUSTERED BY channel
SEALED

FrankChen021

Severity	Findings
P0	0
P1	2
P2	3
P3	0
Total	5

Reviewed 42 of 42 changed files.

This is an automated review by Codex GPT-5.6-Sol

FrankChen021 · 2026-07-31T12:33:50Z

+    @Override
+    protected void execute(CatalogTableWriter writer)
+    {
+      writer.updateProperties(tableId, properties);


[P1] Revalidate the complete table after property edits

This uses the property-only edit endpoint, whose transaction loads and validates properties without columns, so DatasourceDefn.validate(ResolvedTable) and its cross-field checks never run. For example, after defining a DAY projection, SET PROPERTIES can change segmentGranularity to PT1H even though full validation rejects a projection coarser than its segments; it can likewise clear sealed while __base remains. The catalog then contains an invalid specification and subsequent ingestion fails. Load and validate the complete revised TableSpec inside the Coordinator transaction.

FrankChen021 · 2026-07-31T12:33:50Z

+    @Override
+    protected void execute(CatalogTableWriter writer)
+    {
+      writer.updateColumns(tableId, Collections.singletonList(column));


[P1] Enforce column operation predicates atomically

UpdateColumns appends a column when its name is absent, so ALTER TABLE t ALTER COLUMN typo SET DATA TYPE BIGINT silently adds typo instead of rejecting the nonexistent target. ADD COLUMN has the inverse predicate checked by a separate Broker read, allowing concurrent ADDs to both pass and the later merge to overwrite the first type. Add/alter existence semantics need dedicated checks inside the Coordinator's column-update transaction.

FrankChen021 · 2026-07-31T12:33:50Z

+        engine,
+        sql,
+        query,
+        CONTEXT,


[P2] Preserve statement context when planning projections

The nested planner receives only the hard-coded CONTEXT and discards the enclosing statement context, even though SET clauses are explicitly supported before DDL. For example, SET sqlTimeZone = 'America/Los_Angeles' followed by a projection using TIME_FLOOR stores a UTC expression, so the equivalent query under the same context plans differently and cannot match the projection. Merge relevant outer PlannerContext values before applying the deterministic overrides.

FrankChen021 · 2026-07-31T12:33:50Z

+            )
+        );
+      }
+      final VirtualColumn virtualColumn = planned.getVirtualColumn(selected.get(i));


[P2] Handle direct aliases in base projections

ScanQuery deduplicates its column list, while outputNames retains every SELECT item. A valid declared layout [id, copy] with SELECT id, id AS copy therefore has two outputs but only one selected entry, and the second iteration throws IndexOutOfBoundsException; a direct alias also has no virtual column to materialize copy. Preserve the select-to-source mapping or reject this form with a user-facing validation error.

FrankChen021 · 2026-07-31T12:33:50Z

+    }
+    name.unparse(writer, leftPrec, rightPrec);
+
+    final SqlWriter.Frame frame = writer.startList("(", ")");


[P2] Do not unparse omitted columns as empty parentheses

The grammar permits CREATE TABLE tbl PARTITIONED BY DAY with no parenthesized element list, but unparse always emits CREATE TABLE tbl () PARTITIONED BY DAY. Empty parentheses cannot be parsed because AddDruidTableElement is mandatory once '(' is present, so a valid AST does not round-trip. Omit the frame when both lists are empty or teach the grammar to accept ().

jtuglu1 · 2026-08-06T15:26:56Z

cc @maytasm

…ments

FrankChen021

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Reviewed 44 of 44 changed files. Found two current-head correctness issues around projection type compatibility and atomic __base updates.

This is an automated review by Codex GPT-5.6-Luna(max)

FrankChen021 · 2026-08-07T13:09:10Z

+    }
+
+    final String granularity = table.stringProperty(SEGMENT_GRANULARITY_PROPERTY);
+    DataSchema.validateProjections(


[P1] ALTER COLUMN can leave projections type-incompatible

Projection validation checks referenced-column presence but does not reconcile projection dimension and aggregator types with current column definitions. A table with x BIGINT and SUM(x) can accept ALTER COLUMN x SET DATA TYPE VARCHAR, leaving an incompatible projection that later fails during initialization and can break ingestion. Validate compatibility or reject incompatible column edits before commit.

FrankChen021 · 2026-08-07T13:09:10Z

+
+      if (BASE_PROJECTION_NAME.equals(projectionName)) {
+        // The base table is a property of the table, not one of its projections, so it is set rather than appended.
+        if (existing.spec().properties().get(DatasourceDefn.BASE_TABLE_PROPERTY) != null) {


[P2] Base projection existence check is not atomic

The __base branch performs the IF NOT EXISTS or duplicate check from a prior table read, then calls updateProperties without repeating that predicate inside the Coordinator transaction. Concurrent ADD PROJECTION __base statements can both observe absence and commit, with the later update overwriting the earlier projection; the non-IF form can also succeed instead of reporting a duplicate. Move the check into the atomic update.

…ments

FrankChen021

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Reviewed 44 of 44 changed files.

This is an automated review by Codex GPT-5.6-Luna(max)

FrankChen021 · 2026-08-12T13:19:51Z

+    if (baseTable == null) {
+      throw CatalogException.badRequest("A base table layout is required");
+    }
+    return catalog.tables().updateProperties(id, table -> {


[P1] Base-table edits are not actually serialized

updateProperties performs an unlocked read followed by an update of the properties blob, with no row lock or version predicate. Concurrent ADD [IF NOT EXISTS] PROJECTION __base calls can both observe no layout and let the latter overwrite the former; a concurrent column edit can also leave __base referencing removed or retyped columns. Use row locking or compare-and-set updates and validate against the committed version.

FrankChen021 · 2026-08-12T13:19:51Z

+        requireSealed(Boolean.TRUE.equals(existing.spec().properties().get(DatasourceDefn.SEALED_PROPERTY)));
+        writer.setBaseTable(
+            tableId,
+            translateBaseTable(handlerContext, tableId.name(), columns, alterTable.getProjection()),


[P2] IF NOT EXISTS still plans an existing base layout

The handler translates and plans the supplied base-table query before the Coordinator evaluates ifNotExists. If __base already exists, ALTER TABLE ... ADD IF NOT EXISTS PROJECTION __base AS (...) should be a no-op, but an invalid or missing column in the supplied body can still make it fail. Short-circuit known existing layouts before translation while retaining the server-side race check.

…ments

FrankChen021

Severity	Findings
P0	0
P1	2
P2	0
P3	0
Total	2

Reviewed 45 of 45 changed files.

This is an automated review by Codex GPT-5.6-Luna(max)

FrankChen021 · 2026-08-13T12:28:16Z

-                    .bind(COLUMNS_COL, JacksonUtils.toBytes(jsonMapper, revised.columns()))
+                    .bind(blobColumn, JacksonUtils.toBytes(jsonMapper, blob.apply(revised)))
                    .bind(UPDATE_TIME_COL, updateTime)
+                    .bind(OLD_VERSION_PARAM, existing.updateTime())


[P1] CAS version can repeat

The new compare-and-set update uses System.currentTimeMillis() as the replacement version, while the predicate compares the row against existing.updateTime(). Two edits in the same millisecond can therefore reuse the same version: both transactions may pass the old-version predicate, overwrite one another, and publish indistinguishable cache versions. Generate a strictly increasing version atomically (or use an equivalent database-side mechanism) before relying on this CAS.

FrankChen021 · 2026-08-13T12:28:16Z

+  {
+    String path = tablePath(TABLE_EDIT_PATH, tableId);
+    FutureUtils.getUnchecked(
+        serviceClient.asyncRequest(


[P1] Retries can replay catalog mutations

This new mutating POST goes through a client configured with StandardRetryPolicy.maxAttempts(6). If the Coordinator commits the DDL but the response is lost, a retryable channel error or 5xx can replay the non-idempotent edit (and the analogous create request), producing conflicts or duplicate effects while the SQL caller sees failure. Disable retries for writes or add request idempotency/deduplication.

producing conflicts or duplicate effects
I don't think this can really happen with the compare and set stuff, so mainly the risk is a confusing error message? Re: idempotency, the SQL supports IF NOT EXISTS/IF EXISTS, so i don't think we want it to be implicitly idempotent.

Also this retrying isn't really new in this PR, createTable was already going through it prior to this PR.. maybe in a follow-up we could consider tightening up retry on write operations, since I think all it really gives us is a more accurate error message?

…ments

FrankChen021

I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no new issues found.

Reviewed 45 of 45 changed files.

Validation: git diff --check on the current 45-file PR diff; tests and builds not run.

This is an automated review by Codex GPT-5.6-Luna(max)

capistrant

This is a very cool addition to Druid, thank you for all the work and iteration that has gone into the PR. I reviewed test coverage and docs extensively to understand the features. I support the overall design and impl decisions spec'd out. I appreciate the detailed doc and the coverage that spans UTs and Embedded tests to provide confidence in the fact that the feature does what is spec'd out in the PR description and docs.

The many rounds with @FrankChen021 seem to have also worked through a lot of potential gotchas and have contributed to confidence in bugs being smoked out in this process versus in live testing after merge.

I'm approving with some nits I found reading through all the test files and javadocs. I also took into consideration that this is an experimental feature defaulted to off that is part of an experimental extension. release notes should stress this rawness and solicit bug reports/feedback from any early adopters.

capistrant · 2026-08-24T14:44:59Z

+  @SuppressWarnings("unchecked")
+  private DatasourceProjectionMetadata projection(int index)
+  {
+    return ((List<DatasourceProjectionMetadata>) WRITER.calls.get(0).spec.properties().get("projections")).get(index);
+  }
+
+  private String projectionsJson() throws Exception
+  {
+    return queryFramework().queryJsonMapper()
+                           .writeValueAsString(WRITER.calls.get(0).spec.properties().get("projections"));
+  }


nit: don't interleave these in the tests

capistrant · 2026-08-24T14:55:41Z

+  private List<DatasourceProjectionMetadata> projectionsOf(String tableName)
+  {
+    return TestHelper.JSON_MAPPER.convertValue(
+        client.readTable(TableId.datasource(tableName))
+              .spec().properties().get(DatasourceDefn.PROJECTIONS_KEYS_PROPERTY),
+        new TypeReference<List<DatasourceProjectionMetadata>>() {}
+    );
+  }


nit: another interleaved private that could be moved out of test area

capistrant · 2026-08-24T14:57:20Z

 written into a defined column of the table is consistent with that columns definition, minimizing errors where unexpected
 data is written into a particular column of the table.

+### SQL DDL


nice doc, ty for adding it up front. greatly helped set stage for review

capistrant · 2026-08-24T15:06:39Z

+  private String columnType(String tableName, String columnName) throws CatalogException
+  {
+    return catalog.tables().read(TableId.datasource(tableName)).spec().columns().stream()
+                  .filter(c -> columnName.equals(c.name()))
+                  .findFirst()
+                  .orElseThrow(() -> new AssertionError("No column [" + columnName + "]"))
+                  .dataType();
+  }


nit: don't interleave with tests here. sorry to be repeating so many of these 😅

capistrant · 2026-08-24T15:14:49Z

  }

+  /**
+   * The Druid type of a column, which for {@code __time} is always {@link ColumnType#LONG} whatever was declared.


The end reads confusing to me. are you meaning "regardless of what was declared" or something in that direction

…ments

clintropolis added 3 commits July 30, 2026 00:41

feat: ddl create and alter statements

fbc2cbf

feat: ddl + projections

ae9c072

feat: ddl + base table projection

b0617ff

clintropolis added the Area - SQL label Jul 30, 2026

github-actions Bot added Area - Documentation Area - Querying labels Jul 30, 2026

fix

c71e15f

github-advanced-security Bot found potential problems Jul 30, 2026

View reviewed changes

fixes

00ccd65

FrankChen021 reviewed Jul 31, 2026

View reviewed changes

clintropolis added 4 commits August 6, 2026 13:42

fixes

aac737a

Merge remote-tracking branch 'upstream/master' into catalog-ddl-state…

2b89172

…ments

Merge remote-tracking branch 'upstream/master' into catalog-ddl-state…

9fb88d7

…ments

fix

19fc09a

FrankChen021 reviewed Aug 7, 2026

View reviewed changes

clintropolis added 3 commits August 10, 2026 23:57

Merge remote-tracking branch 'upstream/master' into catalog-ddl-state…

62d0d7c

…ments

fixes

957ad33

Merge remote-tracking branch 'upstream/master' into catalog-ddl-state…

1dbc471

…ments

FrankChen021 reviewed Aug 12, 2026

View reviewed changes

clintropolis added 2 commits August 12, 2026 21:40

Merge remote-tracking branch 'upstream/master' into catalog-ddl-state…

93a677c

…ments

fix

68d530e

FrankChen021 reviewed Aug 13, 2026

View reviewed changes

clintropolis added 3 commits August 21, 2026 18:17

Merge remote-tracking branch 'upstream/master' into catalog-ddl-state…

b60a480

…ments

fix test soft conflict

7d6155e

edge case

3fd5d7b

FrankChen021 reviewed Aug 24, 2026

View reviewed changes

capistrant approved these changes Aug 24, 2026

View reviewed changes

clintropolis added 2 commits August 24, 2026 22:39

Merge remote-tracking branch 'upstream/master' into catalog-ddl-state…

e21b985

…ments

more test, adjustments

3e8505c

Uh oh!

Conversation

clintropolis commented Jul 30, 2026

Description

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jtuglu1 commented Aug 6, 2026

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

capistrant left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants