Support aggregate function states in Parquet and Iceberg by zvonand · Pull Request #2301 · Altinity/ClickHouse

zvonand · 2026-09-02T06:53:48Z

Closes #2206

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Support aggregate function states in Parquet and Iceberg

CI/CD Options

Exclude tests:

Regression jobs to run:

zvonand · 2026-09-02T06:54:03Z

@codex review

chatgpt-codex-connector · 2026-09-02T06:54:23Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-02T07:16:26.909845Z	`090c16a`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

github-actions · 2026-09-02T06:55:07Z

Workflow [PR], commit [ea636f0]

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 090c16ac1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Parquet and Iceberg have no aggregate-state type, so neither can describe an `AggregateFunction` or `SimpleAggregateFunction` column with its schema alone. This adds a ClickHouse-only annotation next to the data - the `clickhouse.column_types` key of a parquet file's footer, and a `clickhouse.type` key on an Iceberg schema field - naming the ClickHouse type so it can be rebuilt on read. An `AggregateFunction` state is stored as an opaque `BYTE_ARRAY` / `binary` holding the bytes the `-State` combinator produces; a `SimpleAggregateFunction(f, T)` is stored as an ordinary value of `T`. Other query engines ignore the key and see a plain binary or `T`-typed column. The recorded name always carries the state version, which pins the serialized layout: `getName` drops a zero version, so a versioned function pinned to version 0 would otherwise be rebuilt with the default version and its bytes misread. `getNameForAnnotation` keeps it. Two experimental settings gate the feature, both off by default: `allow_experimental_aggregate_function_states_in_parquet` for writing such a column and for reconstructing one during parquet schema inference, and `allow_experimental_aggregate_function_states_in_iceberg` for `CREATE TABLE` and for honouring a `clickhouse.type` key that names an `AggregateFunction`. Honouring the annotation lets the file or the table metadata, rather than the query, choose the deserializer the stored bytes are handed to, so a refused annotation is rejected rather than read as `String` - reading states as strings would be a wrong result, not an error. `SimpleAggregateFunction` needs no opt-in on read, holding ordinary values with no state deserializer involved. The annotation is checked against the type the schema derives on its own before it is honoured, so a stale or crafted one cannot re-type a column to anything of the same nesting shape. The parquet reader checks strictly, requiring a type its own writer maps to what the file holds; Iceberg checks the nesting structure, which is all its type system allows. The Iceberg gate is read from the context of the query that parses the schema rather than from one captured at `ATTACH`, so a reading query can opt in at all; `IcebergSchemaProcessor::addIcebergTableSchema` publishes a schema-id only once every field has parsed and drops what it wrote on failure, so a refused read can be retried with the setting on. The metadata prefetcher asks for the schema-id with `SchemaParsing::Skip`, since it has no query to carry the gate and must not publish a schema that later queries would be served. The parquet schema cache is keyed by both settings, so a schema inferred with a gate open is not served to a query that has it closed. `ALTER TABLE ... EXPORT PART` and `EXPORT PARTITION` from an `AggregatingMergeTree` table into an Iceberg table carry both gates: `EXPORT PARTITION` records them in its ZooKeeper manifest, so every replica executing the task applies the values the `ALTER` gave instead of its own profile, and the destination is resolved under those settings. A manifest without the fields, written before they existed, reads them as closed. Iceberg records no bounds for aggregate states, whose extremes cannot be compared, and parquet min/max statistics over serialized states are never used for pruning. Adds `04673_parquet_aggregate_function_state`, integration tests for the export paths and for round-tripping states through Spark, and unit tests for the annotation matching, the name parsing, the schema processor and the metadata generator. Documents the feature in `Parquet.md` and `iceberg.md`.

zvonand · 2026-09-03T16:09:21Z

@blau-ai

blau-ai · 2026-09-03T16:19:51Z

CI triage

Verdict: 4 failing checks — 0 caused by this PR. All are pre-existing flakiness or CI/infra noise; none touch the Parquet / Iceberg / aggregate-function-state code paths this PR adds. The feature's own regression suites are green (see bottom).

Note: the workflow isn't finished yet — the Integration tests (amd_asan_ubsan, db disk, old analyzer, N/8) shards are still pending. This triage covers the checks that are currently red.

1. Stateless tests (amd_debug, distributed plan, s3 storage, parallel) — not PR-related (flaky)

Failing test: 03338_http_compression_profile_events. The runner's own diagnosis:

Step 1: Re-running with the same randomized settings (budget: 60s)...
  Runs: 34, Failed: 0, Passed: 34, Other: 0
Result: All reruns passed. The failure is not reproducible (likely a transient issue).
...
Error on processing query: Timeout exceeded while receiving data from server.
Waited for 300 seconds ... (query: SELECT count() FROM system.query_log WHERE ...)

A one-off system.query_log flush/read timeout in an HTTP-compression test — unrelated to this PR. Next step: safe to re-run.

2. Stateless tests (amd_binary, cas storage, parallel) — not PR-related (flaky, CAS infra)

Failing test: 02265_column_ttl (16/33 reruns failed → runner labels it flaky; suite carries do_not_block_pipeline_on_failure). Root cause:

Code: 210. DB::Exception: Source localhost did not prove it still holds the manifest it
offered for part 20100101_0_0_1 by relink; the relink is abandoned and the fetch will be
retried later. (NETWORK_ERROR)
  at DataPartsExchange.cpp:1549  DB::DataPartsExchange::Fetcher::relinkPartToDisk(...)
  at DataPartsExchange.cpp:944   DB::DataPartsExchange::Fetcher::fetchSelectedPart(...)

This is the CAS (content-addressable-storage) part-fetch/relink verification path in DataPartsExchange.cpp, a file this PR does not modify. It only shows up in the cas/cas s3 storage configs and is a known timing-dependent flake there. Next step: safe to re-run.

3. Stateless tests (amd_binary, cas s3 storage, parallel) — not PR-related (2 flaky)

02265_column_ttl — same CAS relink NETWORK_ERROR as Update README.md #2 (18/19 reruns failed here; CAS-storage flake in DataPartsExchange.cpp).
04003_deterministic_filter_chain_partition_pruning_key_condition_explain — 6/6 reruns passed (not reproducible). Failure was an S3 write blip:

WriteBufferFromS3: S3Exception ... e.code() = 111, Connection refused ...
  bucket test, key cas_s3/cas/ns/stream/.../_log/...zst

A transient MinIO/S3 "connection refused" — infra, not the PR. Next step: safe to re-run.

4. RegressionTestsRelease / Swarms / swarms — not PR-related (racy regression test)

Failing scenario: /swarms/feature/node failure. It stops a swarm node in the middle of a long SELECT from an Iceberg table and asserts on the cancellation message. The query was cancelled, but with a different (equally valid) message than the test's allow-list expects:

File "swarms/tests/node_failure.py", line 112, in run_long_query
  assert any(m in result.output for m in expected_messages) ...
Expected one of ['DB::Exception: Query was cancelled.'] in output, but got:
Code: 394. DB::Exception: ... Query '...' is killed in pending state. (QUERY_WAS_CANCELLED)

The test's own docstring acknowledges this path is "inherently racy: stopping a swarm node mid-query can surface either a graceful cancellation or an abrupt connection drop, and both are valid." Here the query returned QUERY_WAS_CANCELLED (code 394, "killed in pending state") — a correct cancellation whose wording simply isn't in the expected_messages allow-list. This test lives in the external Altinity/clickhouse-regression repo (node_failure.py, commit 302ae29), not in this PR, and the job was already on job.retry: 1.

This is a gap in the regression test's message allow-list, not a ClickHouse bug. Suggested fix (in clickhouse-regression, not here): add the pending-cancellation variant to the allow-list, e.g. include "Query was cancelled" (substring) or "is killed in pending state" / gate on exit code 394 (QUERY_WAS_CANCELLED) instead of the exact message. The other 11 swarms features (incl. all the s3/s3Cluster/iceberg/icebergS3Cluster union scenarios) passed. Next step for this PR: safe to re-run; the allow-list fix belongs in the regression repo.

Health check — the PR's own coverage is green

The suites that actually exercise this change all pass:

Regression: Iceberg (1) & (2) ✅, S3Export (part) ✅, s3_export_partition ✅
Unit tests (asan_ubsan) ✅ (0 failed / 14452) — covers the new gtests added in this PR
Fast test ✅ (0 failed), and the new 04673_parquet_aggregate_function_state stateless test is not among any failures

Nothing in the red checks points at the Parquet aggregate-function-state / Iceberg export code. My recommendation is to re-run the four failed jobs (or let the flaky-rerun mechanism settle them) and wait for the still-pending integration shards; I don't see a code change needed on this PR to address them.

_{🤖 Automated CI triage by @blau-ai. Report data from praktika S3 + the swarms TestFlows report; I can't build/run ClickHouse, so correctness is confirmed by CI, not locally.}

zvonand added the antalya-26.6 label Sep 2, 2026

chatgpt-codex-connector Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread src/Processors/Formats/Impl/Parquet/Write.cpp Outdated

Comment thread src/Processors/Formats/Impl/Parquet/SchemaConverter.cpp Outdated

Comment thread src/Storages/ObjectStorage/DataLakes/Iceberg/DataFileStatistics.cpp

This comment was marked as outdated.

Sign in to view

zvonand force-pushed the feature/antalya-26.6/aggregate-function-states-in-parquet-iceberg branch from bac8c04 to d5099b8 Compare September 3, 2026 10:09

zvonand force-pushed the feature/antalya-26.6/aggregate-function-states-in-parquet-iceberg branch from d5099b8 to ea636f0 Compare September 3, 2026 10:46

Navigation Menu

Uh oh!

Support aggregate function states in Parquet and Iceberg - #2301

Support aggregate function states in Parquet and Iceberg#2301
zvonand wants to merge 1 commit into
antalya-26.6from
feature/antalya-26.6/aggregate-function-states-in-parquet-iceberg

zvonand commented Sep 2, 2026 •

edited

Loading

Uh oh!

zvonand commented Sep 2, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

zvonand commented Sep 3, 2026

Uh oh!

blau-ai commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

zvonand commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

zvonand commented Sep 2, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

github-actions Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

zvonand commented Sep 3, 2026

Uh oh!

blau-ai commented Sep 3, 2026

CI triage

1. Stateless tests (amd_debug, distributed plan, s3 storage, parallel) — not PR-related (flaky)

2. Stateless tests (amd_binary, cas storage, parallel) — not PR-related (flaky, CAS infra)

3. Stateless tests (amd_binary, cas s3 storage, parallel) — not PR-related (2 flaky)

4. RegressionTestsRelease / Swarms / swarms — not PR-related (racy regression test)

Health check — the PR's own coverage is green

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zvonand commented Sep 2, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 2, 2026 •

edited

Loading

github-actions Bot commented Sep 2, 2026 •

edited

Loading