Native CREATE TABLE / DROP TABLE for DataLakeCatalog; optionally prune on DROP

[zvonand]

Cherry-picked from ClickHouse#98670.

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Supports CREATE TABLE, CREATE TABLE … AS source, and DROP TABLE for DataLakeCatalog; DROP TABLE can request catalog-side data purge via the new data_lake_delete_data_on_drop setting

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[zvonand]

RelEasy — resolution kept with warnings

The AI resolution was pushed, but a post-resolution check is still failing on it. Nothing was rolled back — please review the point(s) below before merging and either fix them or dismiss them as false positives.

• src/Core/SettingsChangesHistory.cpp: 1 unauthorized setting row(s) added during resolution: data_lake_delete_data_on_drop — these names are not in the source PR's diff for this file. This registry is append-only: only rows the source PR itself adds may land in the port. Re-resolve and drop the extras (likely context lines from "ours" that got swept in alongside a real edit). (still failing after 2 correction pass(es))

[github-actions]

Workflow [PR], commit [7809d87]

[zvonand]

@blau-ai

[blau-ai]

CI triage for #2305 (head 73509df)

Verdict: the red checks split into three PR-caused functional failures (Iceberg native CREATE/DROP + prune-on-drop don't behave correctly) and a set of infra/env and pre-existing/flaky failures that are not this PR's fault. Build, Fast test, unit tests, and all the non-CAS stateless shards are green.

🔴 PR-caused — need a fix before merge

1. delete_data_on_drop doesn't actually delete the data (assert 5 == 0, deterministic — failed on retry too)

• test_storage_iceberg_no_spark/test_drop_delete_data_on_drop.py::test_drop_honours_query_level_delete_data_on_drop[local-1] (3/8) — new test added by this PR
• test_storage_iceberg_multistorage/test.py::test_delete_data_on_drop_removes_external_files (8/8)
• test_storage_iceberg_multistorage/test.py::test_delete_data_on_drop_removes_same_bucket_external_files (8/8)

All three DROP a table with data-purge requested and then find the data files still on disk (assert _files_left(...) == 0 → assert 5 == 0). The [delete_data_on_drop=0] variant passes, so the plumbing that is failing is specifically the "purge requested → delete files" path. The new test's own docstring points at the mechanism:

StorageObjectStorage::drop runs in a background thread where the query context is already gone. A query-level data_lake_delete_data_on_drop reaches it only because IStorage::prepareForDrop captures it while the DROP TABLE query is still running.

Where to look: the capture in IStorage::prepareForDrop and whether the captured flag actually reaches the object-storage/local iceberg drop path in StorageObjectStorage::drop / InterpreterDropQuery.cpp. Since the pure-local config also fails, this looks like a genuine porting gap rather than a catalog-backend quirk.

2. Native CREATE TABLE sets the namespace location = the table location (deterministic)

• test_database_iceberg/test.py::test_create_table_namespace_location (8/8) — assertions added/modified by this PR
• test_database_iceberg/test.py::test_create_table_with_engine_namespace_location (8/8)

Both fail with ns_location == table_location, both equal s3://warehouse-rest/data/test_ns_location_xxxx/first. Expected: namespace location = …/test_ns_location_xxxx, table location = …/test_ns_location_xxxx/first. So on native CREATE, the namespace's default location property is being written as the table's directory (with the table-name segment appended) instead of the namespace base. The test comment spells out why that matters:

The namespace default location must point at the namespace base, not at the first table's directory, or later tables created without an explicit location would land under that first table.

Where to look: the table-location / namespace-location construction on the create path (ICatalog/Common constructTableLocation, RestCatalog/S3TablesCatalog create-namespace-or-table logic). The gtest gtest_construct_table_location.cpp in this PR is the right place to add a case that pins ns_location != table_location.

3. Regression iceberg_2 — export-partition loses partitions (data loss, deterministic)

• /iceberg/export partition/ice catalog/replicated merge tree/sanity/export multiple partitions in one ALTER — Expected 4 rows at destination, got 1
• /iceberg/export partition/ice catalog/replicated merge tree/concurrent writes/multi-statement ALTER commits each partition as its own snapshot — same assert_destination_row_count failure (verification.py:146)

Only 1 of 4 rows survived at the destination Iceberg table. Both failing scenarios are specifically about accumulating multiple partitions across multiple snapshots, which is exactly the append/snapshot-commit code this PR refactors (src/Storages/ObjectStorage/DataLakes/Iceberg/* — IcebergWrites, MetadataGenerator, Mutations). The signature ("only one partition's rows remain") is what you'd get if a new snapshot doesn't chain onto its parent / doesn't carry over the previously-committed manifests. This is a deterministic, semantically-specific failure, not a timing flake — please check the parent-snapshot-id / existing-manifest carry-over in the refactored commit path.

🟠 Ambiguous — re-run, then decide

Integration 4/8 — ERROR: Test execution was interrupted (exit status: 2)
The ClickHouse server on the node went down mid-SELECT * FROM an Iceberg table during test_storage_iceberg_with_spark/test_schema_inference.py::test_schema_inference[s3-1-True] (ATTEMPT_TO_READ_AFTER_EOF), after which every later test in the shard failed with Connection refused (server was gone). I could not find any crash marker (no SIGSEGV / sanitizer / LOGICAL_ERROR / assertion) in the reachable Actions log — the actual per-container clickhouse-server.log is locked inside a ~203 MB logs.tar.gz I can't extract here. It died in the Iceberg read path this PR touches, so it could be PR-related, but a plain OOM/infra kill would look identical and leave no stack. Suggested next step: re-run this shard; if it reproduces, pull logs.tar.gz for that node and grep the server log for Received signal/AddressSanitizer.

🟢 Not PR-related

Infra / no outbound git on the runner — all fail with fatal: could not read Username for 'https://github.com': No such device or address; nothing to fix in the PR (safe to re-run once runners have git egress):

• Source upload (submodule checkout)
• SQLLogic test (clone gregrahn/sqllogictest)
• SQLStorm test (clone ClickHouse/SQLStorm)
• Integration 6/8 test_git_import (clones githubtraining/hellogitworld)

CAS-storage backend, do_not_block_pipeline_on_failure — unrelated to this PR (this PR touches DataLake/Iceberg, not replication/DataPartsExchange/CAS):

• Stateless (amd_binary, cas s3) 02265_column_ttl — reproducible but the error is CAS-specific (Source localhost did not prove it still holds the manifest … by relink, DataPartsExchange.cpp). Pre-existing CAS issue.
• Stateless (amd_asan_ubsan, cas s3, 2/2) 02014_storage_merge_order_by — flaky (passed 3/3 on rerun, S3 Connection refused); Some queries hung — same shard.

Summary table

Check	Class	PR-caused?
Integration 8/8 — test_database_iceberg namespace location (×2)	native CREATE location bug	Yes
Integration 3/8 + 8/8 — delete_data_on_drop (×3)	prune-on-drop no-op	Yes
Regression iceberg_2 — export partition (×2)	Iceberg snapshot/manifest data loss	Yes (very likely)
Integration 4/8 — server down mid Iceberg SELECT	crash cause unproven	Maybe — re-run
Source upload / SQLLogic / SQLStorm / 6/8 git_import	no git egress on runner	No (infra)
Stateless cas-s3 02265_column_ttl / 02014_… / hung	CAS backend, non-blocking	No (pre-existing/flaky)

I can dig into the C++ for fixes to (1)/(2)/(3) and open a blau/* PR, or push directly to releasy/port/pr-98670-2f540f — just say which you'd prefer. (I haven't pushed anything.)

Evidence: praktika result_pr.json, Actions job logs for runs under 33663741558, and the iceberg_2 regression raw.log. I can't build/run ClickHouse in this container, so the root-cause hypotheses above are from static reading of the diff + tests + CI logs; CI on your fix is the real confirmation.

[blau-ai]

Follow-up: root cause pinned for the two integration-test groups

Static read of the diff (I can't run the server, so treat as strong hypotheses — CI on a fix confirms):

Group B — namespace location == table location → src/Databases/DataLake/RestCatalog.cpp:1696-1698

const String location = metadata_content->getValue<String>("location");  // FULL table location
createNamespaceIfNotExists(namespace_name, location);                     // ...used as the NAMESPACE location

RestCatalog::createTable feeds the new table's location (…/<ns>/first) straight in as the namespace's location property, so ns_location == table_location. Upstream ClickHouse#98670 creates the namespace with the namespace base (…/<ns>, i.e. table location minus the trailing table segment). The merge-commit message on this port notes it dropped the explicit catalog->createNamespaceIfNotExists(...) calls in DatabaseDataLake::createTable / IcebergMetadata::createInitial because antalya's ICatalog lacks that virtual — and let RestCatalog::createTable create the namespace with the wrong location instead. Same latent bug applies to GlueCatalog::createTable.

• Fix: strip the trailing /<table_name> segment before createNamespaceIfNotExists (pass an empty location when it can't be derived — the function already treats empty as "no default"), or restore upstream's explicit namespace-base derivation in DatabaseDataLake::createTable (it already has base_location, DatabaseDataLake.cpp:951-968). Add a ns_location != table_location case to gtest_construct_table_location.cpp.

Group A — delete_data_on_drop purges nothing → src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp:1609-1612

void IcebergMetadata::drop(bool delete_data)
{
    if (!delete_data)
        return;               // always taken -> nothing purged

The delete logic below (collectReachableFiles + external-files + base wipe, lines 1621-1657) is correctly ported but never runs because delete_data arrives false. The [local-1] case sets the real setting (data_lake_delete_data_on_drop=1, not an alias) and still gets false, so the fault is the capture/delivery plumbing, not a setting rename. The path is: DROP → InterpreterDropQuery::prepareForDrop (InterpreterDropQuery.cpp:355) → StorageObjectStorage::prepareForDrop captures into the atomic<optional<bool>> delete_data_on_drop member (StorageObjectStorage.cpp:853-858) → background StorageObjectStorage::drop() reads it (:865-866) → configuration->drop(delete_data). The captured optional is arriving empty (value_or(false)).

• Most likely culprit: prepareForDrop not reaching the real storage instance — note src/Storages/StorageProxy.h:87 explicitly does not forward prepareForDrop, so if the Iceberg table sits behind a proxy the atomic stays nullopt on the object whose drop() later runs. That single point would explain all three Group-A failures. Verify prepareForDrop is invoked on/forwarded to the same StorageObjectStorage that runs the background drop().

Happy to implement both fixes (Group A/B) and take a shot at Group 3 (export-partition snapshot carry-over) — say the word and whether you want a blau/* PR or a direct commit to releasy/port/pr-98670-2f540f.