diff --git a/docs/en/engines/table-engines/integrations/iceberg.md b/docs/en/engines/table-engines/integrations/iceberg.md index 19b87ea5c1a9..c515d279ff0f 100644 --- a/docs/en/engines/table-engines/integrations/iceberg.md +++ b/docs/en/engines/table-engines/integrations/iceberg.md @@ -106,6 +106,72 @@ The following table shows how Iceberg data types are mapped to ClickHouse data t | `map` | `Map` | | `struct` | `Tuple` | +### Aggregate function states {#aggregate-function-states} + +Iceberg has no aggregate-state type, so ClickHouse stores the two aggregate-state types as ordinary +Iceberg values and records the ClickHouse type name in a `clickhouse.type` key on the schema field: + +| ClickHouse type | Iceberg type | Stored as | +|---|---|---| +| `AggregateFunction(f, T...)` | `binary` | The serialized state, the same bytes `f` writes with the `-State` combinator | +| `SimpleAggregateFunction(f, T)` | whatever `T` maps to | An ordinary value of type `T` | + +Other query engines ignore the key and see a plain `binary` (or `T`-typed) column. + +Creating such a table requires +[`allow_experimental_aggregate_function_states_in_iceberg`](/operations/settings/settings#allow_experimental_aggregate_function_states_in_iceberg), +and so does every query that reads a column whose `clickhouse.type` names an `AggregateFunction` - an +`INSERT` or an `ALTER TABLE ... EXPORT PART` as much as a `SELECT`. A state is an opaque blob handed +to the deserializer of the aggregate function the table's metadata names, so with the setting enabled +it is that metadata, not the query, which chooses the deserializer; keep it disabled for tables from +untrusted sources, where such a field is then rejected rather than read as `String`. A +`SimpleAggregateFunction` column is not gated on read, holding ordinary values of its storage type. + +The setting is read from the query that parses the schema, so a session `SET` or a `SETTINGS` clause +on that query supplies it. The `iceberg*` table functions build a fresh storage object per query, so +for them it takes effect per query; the table engine parses the schema once and keeps it, so the +query that first touches the table after `ATTACH` decides, and that outcome holds - including for +queries that do not set it - until `DETACH TABLE` or a server restart. + +Writing additionally requires +[`allow_experimental_aggregate_function_states_in_parquet`](/operations/settings/settings#allow_experimental_aggregate_function_states_in_parquet), +since the data files are written by the ordinary Parquet writer. An object storage table engine +freezes its format settings at `CREATE TABLE` - the server settings plus that query's `SETTINGS` +clause, session settings ignored - so for `INSERT` the Parquet setting belongs there and has no +effect if given on the `INSERT` instead: + +```sql +CREATE TABLE agg (k UInt32, u AggregateFunction(uniq, UInt64), s SimpleAggregateFunction(sum, UInt64)) +ENGINE = IcebergLocal('/path/to/table/') +PARTITION BY k +SETTINGS allow_experimental_aggregate_function_states_in_iceberg = 1, + allow_experimental_aggregate_function_states_in_parquet = 1; + +-- The states merge exactly as they do in an AggregatingMergeTree table. +SELECT k, uniqMerge(u), sum(s) FROM agg GROUP BY k +SETTINGS allow_experimental_aggregate_function_states_in_iceberg = 1; +``` + +`ALTER TABLE ... EXPORT PART` and `ALTER TABLE ... EXPORT PARTITION` from an `AggregatingMergeTree` +table into such an Iceberg table work as well, so a partition of pre-aggregated states can be moved +into the lake without finalizing it. Both take the settings off the `ALTER` query itself: + +```sql +ALTER TABLE mt EXPORT PART 'all_1_1_0' TO TABLE agg +SETTINGS allow_experimental_aggregate_function_states_in_parquet = 1; + +-- EXPORT PARTITION, which only ReplicatedMergeTree implements, records both values in its manifest, +-- so every replica executing the task applies them in place of its own profile. +ALTER TABLE rmt EXPORT PARTITION ID '1' TO TABLE agg +SETTINGS allow_experimental_aggregate_function_states_in_parquet = 1, + allow_experimental_aggregate_function_states_in_iceberg = 1; +``` + +The Parquet footer of each data file carries the same information, so `DESCRIBE file('data.parquet')` +reports the aggregate types too; that path goes through Parquet schema inference and needs the +Parquet setting rather than the Iceberg one. See +[aggregate function states in Parquet](/interfaces/formats/Parquet#aggregate-function-states). + ## Schema evolution {#schema-evolution} ClickHouse supports reading Iceberg tables whose schema has evolved over time. This includes tables where columns have been added, removed, or reordered, as well as columns changed from required to nullable. Additionally, the following type casts are supported: diff --git a/docs/en/interfaces/formats/Parquet/Parquet.md b/docs/en/interfaces/formats/Parquet/Parquet.md index db31fc0dab80..b125a8339baf 100644 --- a/docs/en/interfaces/formats/Parquet/Parquet.md +++ b/docs/en/interfaces/formats/Parquet/Parquet.md @@ -95,6 +95,41 @@ On write, top-level columns of type `Point`, `LineString`, `Polygon`, `MultiLine Geometry columns must appear at the root of the schema or nested inside `Tuple` (`struct`); nesting them inside `Array` or `Map` is not supported. `Nullable` is not supported for geo columns either. +## Aggregate function states {#aggregate-function-states} + +Parquet has no aggregate-state type, so ClickHouse writes an [`AggregateFunction`](/sql-reference/data-types/aggregatefunction.md) column as a plain `BYTE_ARRAY` holding the serialized state - the same bytes the `-State` combinator produces - with no `STRING`/`UTF8` logical type, since a state is arbitrary binary data rather than text. A [`SimpleAggregateFunction(f, T)`](/sql-reference/data-types/simpleaggregatefunction.md) column is written as an ordinary value of type `T`. + +Neither type can be recovered from the Parquet schema alone: every `AggregateFunction` state is just a binary column, and a `SimpleAggregateFunction` is indistinguishable from its storage type. ClickHouse therefore records the type names in a `clickhouse.column_types` key in the file-level Parquet metadata, as a JSON object mapping column name to ClickHouse type name. The recorded name includes the state version, which is what pins the serialized layout across server versions. + +Both writing an `AggregateFunction` column and reconstructing one from that metadata on read are gated by [`allow_experimental_aggregate_function_states_in_parquet`](/operations/settings/settings#allow_experimental_aggregate_function_states_in_parquet), which is disabled by default. A state is an opaque blob passed to the deserializer of whichever aggregate function the file names, so with the setting enabled it is the file, not the query, choosing that deserializer; keep it disabled for files from untrusted sources, where such a file is then rejected rather than read as `String`. With it disabled, writing is refused with `UNKNOWN_TYPE`, exactly as in versions that did not support states in Parquet at all. `SimpleAggregateFunction` is gated in neither direction: it is stored as an ordinary value of its storage type and has always been written that way. + +With the setting enabled the states round-trip without any hint: + +```sql +SET allow_experimental_aggregate_function_states_in_parquet = 1; + +INSERT INTO FUNCTION file('states.parquet') +SELECT k, uniqState(v) AS u, sumSimpleState(v) AS s FROM source GROUP BY k; + +DESCRIBE file('states.parquet'); +-- k UInt64 +-- u AggregateFunction(uniq, UInt64) +-- s SimpleAggregateFunction(sum, UInt64) + +SELECT k, uniqMerge(u), sum(s) FROM file('states.parquet') GROUP BY k; +``` + +An explicit structure overrides the recorded metadata and is unaffected by the setting, so a state column can also be read as `String` to get the raw serialized bytes: + +```sql +SELECT uniqMerge(CAST(u AS AggregateFunction(uniq, UInt64))) +FROM file('states.parquet', Parquet, 'u String'); +``` + +Reading fails rather than guessing if the recorded type does not describe the data actually in the file. Min/max statistics are never used for pruning a state column, because bounds over serialized states are meaningless. + +Other query engines see a plain binary (or `T`-typed) column and ignore the metadata key. The same mechanism backs [aggregate-state support in Iceberg tables](/engines/table-engines/integrations/iceberg#aggregate-function-states), whose data files are Parquet. + ## Example usage {#example-usage} ### Inserting data {#inserting-data} diff --git a/src/Core/Settings.cpp b/src/Core/Settings.cpp index 5d0f3cc34cc2..53c7afa7cd2c 100644 --- a/src/Core/Settings.cpp +++ b/src/Core/Settings.cpp @@ -7427,6 +7427,34 @@ Query Iceberg table using the specific snapshot id. )", 0) \ DECLARE(Bool, allow_experimental_geo_types_in_iceberg, false, R"( Allow parsing Iceberg `geometry` and `geography` field types as ClickHouse `Geometry` (Variant) type. +)", 0) \ + DECLARE(Bool, allow_experimental_aggregate_function_states_in_iceberg, false, R"( +Allow `AggregateFunction` and `SimpleAggregateFunction` columns in Iceberg tables, both when creating +a table and when reading one. Writing additionally requires +[`allow_experimental_aggregate_function_states_in_parquet`](/operations/settings/settings#allow_experimental_aggregate_function_states_in_parquet), +since the data files are Parquet. + +An `AggregateFunction` state is stored as an opaque `binary` value with its ClickHouse type name in +the schema field's `clickhouse.type` key. Honouring that key means the table's metadata, not the +query, chooses the deserializer the stored bytes are handed to, so keep this disabled for tables from +untrusted sources; a field recording such a type is then rejected rather than read as `String`. +`SimpleAggregateFunction` is not gated on read, holding ordinary values of its storage type. + +The setting is read from the query that parses the table's schema. [More about aggregate function +states in Iceberg](/engines/table-engines/integrations/iceberg#aggregate-function-states). +)", 0) \ + DECLARE(Bool, allow_experimental_aggregate_function_states_in_parquet, false, R"( +Allow `AggregateFunction` states in Parquet files, both when writing and when inferring a schema. +While disabled, writing such a column is refused with `UNKNOWN_TYPE`. + +A state is written as an opaque `BYTE_ARRAY`, with the ClickHouse type name - including the state +version, which pins the serialized layout - recorded in the `clickhouse.column_types` file metadata +key. Reconstructing the type from that key means the file, not the query, chooses the deserializer +the stored bytes are handed to, so keep this disabled for files from untrusted sources; a file +recording such a type is then rejected rather than read as `String`. + +Neither direction affects an explicitly given structure, nor `SimpleAggregateFunction`. [More about +aggregate function states in Parquet](/interfaces/formats/Parquet#aggregate-function-states). )", 0) \ DECLARE(Bool, show_data_lake_catalogs_in_system_tables, false, R"( Enables showing data lake catalogs in system tables. diff --git a/src/Core/SettingsChangesHistory.cpp b/src/Core/SettingsChangesHistory.cpp index 008ba54f43d2..4a0f13b023a8 100644 --- a/src/Core/SettingsChangesHistory.cpp +++ b/src/Core/SettingsChangesHistory.cpp @@ -42,6 +42,8 @@ const VersionToSettingsChangesMap & getSettingsChangesHistory() addSettingsChanges(settings_changes_history, "26.6.2.20001.altinityantalya", { {"use_puffin_files_cache", false, true, "Enables cache of parsed Puffin file content such as deletion vectors."}, + {"allow_experimental_aggregate_function_states_in_iceberg", false, false, "New setting gating aggregate function states in Iceberg tables, on creation and on read. Disabled by default, so such a column keeps being refused with `SUPPORT_IS_DISABLED` as in versions without the feature."}, + {"allow_experimental_aggregate_function_states_in_parquet", false, false, "New setting gating `AggregateFunction` states in Parquet files, on write and in schema inference. Disabled by default, so writing such a column keeps throwing `UNKNOWN_TYPE` as in versions without the feature."}, }); addSettingsChanges(settings_changes_history, "26.6", diff --git a/src/DataTypes/DataTypeAggregateFunction.cpp b/src/DataTypes/DataTypeAggregateFunction.cpp index d512b8654052..2cac2040492e 100644 --- a/src/DataTypes/DataTypeAggregateFunction.cpp +++ b/src/DataTypes/DataTypeAggregateFunction.cpp @@ -5,10 +5,20 @@ #include #include +#include #include #include #include +#include +#include +#include +#include +#include +#include +#include +#include +#include #include #include #include @@ -20,7 +30,9 @@ #include #include +#include #include +#include #include #include @@ -54,13 +66,19 @@ String DataTypeAggregateFunction::getFunctionName() const String DataTypeAggregateFunction::doGetName() const { - return getNameImpl(true); + return getNameImpl(true, false); } String DataTypeAggregateFunction::getNameWithoutVersion() const { - return getNameImpl(false); + return getNameImpl(false, false); +} + + +String DataTypeAggregateFunction::getNameForAnnotation() const +{ + return getNameImpl(true, true); } @@ -91,14 +109,16 @@ void DataTypeAggregateFunction::updateVersionFromRevision(size_t revision, bool setVersion(function->getVersionFromRevision(revision), if_empty); } -String DataTypeAggregateFunction::getNameImpl(bool with_version) const +String DataTypeAggregateFunction::getNameImpl(bool with_version, bool always_emit_version) const { WriteBufferFromOwnString stream; stream << "AggregateFunction("; /// If aggregate function does not support versioning its version is 0 and is not printed. + /// always_emit_version keeps an explicit 0 for a versioned function, which getName() would drop, + /// making it indistinguishable from the default version. Non-versioned functions still omit it. auto data_type_version = getVersion(); - if (with_version && data_type_version) + if (with_version && (data_type_version || (always_emit_version && isVersioned()))) stream << data_type_version << ", "; stream << function->getName(); @@ -478,4 +498,229 @@ bool hasAggregateFunctionType(const DataTypePtr & type) return result; } +bool astHasAggregateFunctionType(const ASTPtr & ast) +{ + /// `ParserDataType` represents a type name with arguments as `ASTDataType` and a bare one as + /// `ASTIdentifier`. The comparison is exact because `AggregateFunction` is registered + /// case-sensitively and has no alias. + std::string_view name; + if (const auto * data_type = ast->as()) + name = data_type->name; + else if (const auto * identifier = ast->as()) + name = identifier->name(); + + if (name == "AggregateFunction") + return true; + + for (const auto & child : ast->children) + if (astHasAggregateFunctionType(child)) + return true; + + return false; +} + +bool needsClickHouseTypeAnnotation(const DataTypePtr & type) +{ + auto result = false; + auto check = [&](const IDataType & t) + { + if (WhichDataType(t).isAggregateFunction()) + { + result = true; + return; + } + /// `SimpleAggregateFunction(f, T)` is `T` carrying an `IDataTypeCustomName`. + result |= typeid_cast(t.getCustomName()) != nullptr; + }; + + check(*type); + type->forEachChild(check); + return result; +} + +String getClickHouseTypeAnnotationName(const DataTypePtr & type) +{ + /// `SimpleAggregateFunction(f, T)` has no version and its `T` cannot hold an `AggregateFunction`, + /// so getName() already round-trips it. + if (type->getCustomName()) + return type->getName(); + + switch (type->getTypeId()) + { + case TypeIndex::AggregateFunction: + return assert_cast(*type).getNameForAnnotation(); + case TypeIndex::Array: + return "Array(" + getClickHouseTypeAnnotationName(assert_cast(*type).getNestedType()) + ")"; + case TypeIndex::Map: + { + const auto & map_type = assert_cast(*type); + return "Map(" + getClickHouseTypeAnnotationName(map_type.getKeyType()) + ", " + + getClickHouseTypeAnnotationName(map_type.getValueType()) + ")"; + } + case TypeIndex::Tuple: + { + const auto & tuple_type = assert_cast(*type); + const auto & elements = tuple_type.getElements(); + const auto & names = tuple_type.getElementNames(); + WriteBufferFromOwnString stream; + stream << "Tuple("; + for (size_t i = 0; i < elements.size(); ++i) + { + if (i) + stream << ", "; + if (tuple_type.hasExplicitNames()) + stream << backQuoteIfNeed(names[i]) << ' '; + stream << getClickHouseTypeAnnotationName(elements[i]); + } + stream << ")"; + return stream.str(); + } + default: + return type->getName(); + } +} + +namespace +{ + +/// Neither format's schema inference derives a `LowCardinality`, and either side may carry a +/// `Nullable` the other does not (an OPTIONAL parquet leaf always derives as one). Neither says +/// anything about the values, so both wrappers come off before the types are compared. +DataTypePtr removeNullableAndLowCardinality(const DataTypePtr & type) +{ + return removeNullable(removeLowCardinality(type)); +} + +bool isFixedStringOfSize(const DataTypePtr & type, size_t size) +{ + const auto * fixed_string = typeid_cast(type.get()); + return fixed_string && fixed_string->getN() == size; +} + +bool isDateTime64WithScale(const DataTypePtr & type, UInt32 scale) +{ + const auto * date_time64 = typeid_cast(type.get()); + return date_time64 && date_time64->getScale() == scale; +} + +/// True when `derived`, the type a parquet schema alone reads as, is one this server's parquet writer +/// can produce for a column of type `annotated`. Both arguments are free of `Nullable` and +/// `LowCardinality`. +/// +/// A directed relation rather than an equality: the writer's mapping in `preparePrimitiveColumn` is +/// not injective, since parquet has no logical type for several ClickHouse types and a few +/// `output_format_parquet_*` settings, not recorded in the file, pick between representations. Each +/// case below mirrors one of those. Where the mapping is unclear the check is deliberately generous: +/// a pair wrongly accepted only honours an annotation that would have been honoured anyway, while a +/// pair wrongly rejected refuses a file that reads fine. +bool parquetWriterCouldProduce(const DataTypePtr & annotated, const DataTypePtr & derived) +{ + if (annotated->equals(*derived)) + return true; + + /// Parquet timestamps come in milli-, micro- and nanoseconds only, so a scale in between is + /// written scaled up to the next unit and reads back with that unit's scale. + auto next_timestamp_unit = [](UInt32 scale) -> UInt32 + { + if (scale <= 3) + return 3; + if (scale <= 6) + return 6; + return 9; + }; + + switch (annotated->getTypeId()) + { + /// Parquet has no 16-bit date, so `Date` is written as a DATE, deriving as `Date32` - or, with + /// `output_format_parquet_date_as_uint16`, as a UINT_16. + case TypeIndex::Date: + return WhichDataType(derived).isDate32() || WhichDataType(derived).isUInt16(); + /// No second-resolution timestamp, so `DateTime` is written as TIMESTAMP_MILLIS, deriving as + /// `DateTime64(3)` - or, with `output_format_parquet_datetime_as_uint32`, as a UINT_32. + case TypeIndex::DateTime: + return isDateTime64WithScale(derived, 3) || WhichDataType(derived).isUInt32(); + case TypeIndex::DateTime64: + return isDateTime64WithScale( + derived, next_timestamp_unit(assert_cast(*annotated).getScale())); + /// TIME is written with the timestamp units and derives as `DateTime64`, like TIMESTAMP. + /// Second-resolution `Time` is written as micros. + case TypeIndex::Time: + return isDateTime64WithScale(derived, 6); + case TypeIndex::Time64: + return isDateTime64WithScale( + derived, assert_cast(*annotated).getScale() <= 6 ? 6 : 9); + /// An enum is written as an ENUM byte array, deriving as `String` - or, with + /// `output_format_parquet_enum_as_byte_array` off, as its underlying integer. + case TypeIndex::Enum8: + return isString(derived) || WhichDataType(derived).isInt8(); + case TypeIndex::Enum16: + return isString(derived) || WhichDataType(derived).isInt16(); + /// No logical type for these, so they are written as raw bytes: `IPv4` as a plain UINT_32, + /// the rest as a FIXED_LEN_BYTE_ARRAY of their width. + case TypeIndex::IPv4: + return WhichDataType(derived).isUInt32(); + case TypeIndex::IPv6: + case TypeIndex::UInt128: + case TypeIndex::Int128: + return isFixedStringOfSize(derived, 16); + case TypeIndex::UInt256: + case TypeIndex::Int256: + return isFixedStringOfSize(derived, 32); + /// With `output_format_parquet_fixed_string_as_fixed_byte_array` off a `FixedString` is + /// written as a plain BYTE_ARRAY, which derives as `String`. + case TypeIndex::FixedString: + return isString(derived); + /// A JSON column read with `input_format_parquet_enable_json_parsing` off derives as `String`. + case TypeIndex::Object: + return isString(derived); + default: + return false; + } +} + +} + +bool annotatedTypeMatchesDerived(const DataTypePtr & annotated_type, const DataTypePtr & derived_type, bool strict) +{ + auto annotated = removeNullableAndLowCardinality(annotated_type); + auto derived = removeNullableAndLowCardinality(derived_type); + + if (annotated->getTypeId() == TypeIndex::AggregateFunction) + return isString(derived); + + switch (annotated->getTypeId()) + { + case TypeIndex::Array: + { + const auto * derived_array = typeid_cast(derived.get()); + if (!derived_array) + return false; + return annotatedTypeMatchesDerived( + assert_cast(*annotated).getNestedType(), derived_array->getNestedType(), strict); + } + case TypeIndex::Tuple: + { + const auto & annotated_tuple = assert_cast(*annotated); + const auto * derived_tuple = typeid_cast(derived.get()); + if (!derived_tuple || derived_tuple->getElements().size() != annotated_tuple.getElements().size()) + return false; + for (size_t i = 0; i < annotated_tuple.getElements().size(); ++i) + if (!annotatedTypeMatchesDerived(annotated_tuple.getElement(i), derived_tuple->getElement(i), strict)) + return false; + return true; + } + case TypeIndex::Map: + { + const auto & annotated_map = assert_cast(*annotated); + const auto * derived_map = typeid_cast(derived.get()); + if (!derived_map) + return false; + return annotatedTypeMatchesDerived(annotated_map.getKeyType(), derived_map->getKeyType(), strict) + && annotatedTypeMatchesDerived(annotated_map.getValueType(), derived_map->getValueType(), strict); + } + default: + return !strict || parquetWriterCouldProduce(annotated, derived); + } +} + } diff --git a/src/DataTypes/DataTypeAggregateFunction.h b/src/DataTypes/DataTypeAggregateFunction.h index c9eef738e8db..884be6c2157c 100644 --- a/src/DataTypes/DataTypeAggregateFunction.h +++ b/src/DataTypes/DataTypeAggregateFunction.h @@ -3,6 +3,7 @@ #include #include #include +#include namespace DB @@ -24,7 +25,7 @@ class DataTypeAggregateFunction final : public IDataType Array parameters; mutable std::optional version; - String getNameImpl(bool with_version) const; + String getNameImpl(bool with_version, bool always_emit_version) const; public: static constexpr bool is_parametric = true; @@ -39,6 +40,9 @@ class DataTypeAggregateFunction final : public IDataType String doGetName() const override; String getNameWithoutVersion() const; + /// Like getName(), but keeps an explicit 0 version, which getName() drops. Used by + /// getClickHouseTypeAnnotationName(). + String getNameForAnnotation() const; const char * getFamilyName() const override { return "AggregateFunction"; } TypeIndex getTypeId() const override { return TypeIndex::AggregateFunction; } @@ -94,4 +98,33 @@ void setVersionToAggregateFunctions(DataTypePtr & type, bool if_empty, std::opti /// Checks type of any nested type is DataTypeAggregateFunction. bool hasAggregateFunctionType(const DataTypePtr & type); +/// Same as `hasAggregateFunctionType`, but for a parsed type name instead of a resolved type, so that +/// a name that came from a file can be gated before `DataTypeFactory` looks the aggregate function up. +/// `SimpleAggregateFunction` does not count: it holds an ordinary value, not a serialized state. +bool astHasAggregateFunctionType(const ASTPtr & ast); + +/// True when the name of `type` cannot be recovered from a Parquet or Iceberg schema alone, so it has +/// to be recorded next to the data: `AggregateFunction` or `SimpleAggregateFunction` anywhere inside. +bool needsClickHouseTypeAnnotation(const DataTypePtr & type); + +/// The type name recorded in the Parquet `clickhouse.column_types` / Iceberg `clickhouse.type` +/// annotation. Same as `type->getName()`, except that every `AggregateFunction` state, nested ones +/// included, keeps its version: states are serialized with exactly `getVersion()`, and getName() drops +/// a zero version, so a state pinned to version 0 would be rebuilt with the default one and misread. +String getClickHouseTypeAnnotationName(const DataTypePtr & type); + +/// Checks that the annotated type name describes the same physical data as `derived`, the type the +/// Parquet or Iceberg schema derives on its own. `Nullable` and `LowCardinality` are ignored at every +/// level: neither format derives a `LowCardinality`, and either side may carry a `Nullable` the other +/// does not. +/// +/// An `AggregateFunction` position must sit over a `String`. Every other position is checked only as +/// strictly as the format allows, so that a stale or crafted annotation cannot re-type a column to +/// anything of the same nesting shape: +/// - Lenient (the default, and all Iceberg can do): only the nesting structure has to match, since +/// Iceberg cannot express most ClickHouse types (`UInt64` is stored as `long`, derives as `Int64`). +/// - Strict (the Parquet reader): the annotated type must also be one the parquet writer maps to what +/// the file holds. That mapping is not injective, so it is a directed relation, not an equality. +bool annotatedTypeMatchesDerived(const DataTypePtr & annotated, const DataTypePtr & derived, bool strict = false); + } diff --git a/src/DataTypes/tests/gtest_annotated_type_matches_derived.cpp b/src/DataTypes/tests/gtest_annotated_type_matches_derived.cpp new file mode 100644 index 000000000000..d20658a2b9a9 --- /dev/null +++ b/src/DataTypes/tests/gtest_annotated_type_matches_derived.cpp @@ -0,0 +1,168 @@ +#include + +#include +#include +#include + +using namespace DB; + +/// `annotatedTypeMatchesDerived` decides whether a type name recorded next to the data may be +/// honoured for a column whose schema derives a different type. + +namespace +{ + +DataTypePtr typeFromString(const String & name) +{ + tryRegisterAggregateFunctions(); + return DataTypeFactory::instance().get(name); +} + +bool matches(const String & annotated, const String & derived) +{ + return annotatedTypeMatchesDerived(typeFromString(annotated), typeFromString(derived)); +} + +/// The mode the parquet reader uses: the annotation must also name a type the parquet writer maps to +/// what the file holds. +bool matchesStrictly(const String & annotated, const String & derived) +{ + return annotatedTypeMatchesDerived(typeFromString(annotated), typeFromString(derived), /*strict=*/ true); +} + +} + +TEST(AnnotatedTypeMatchesDerived, AggregateFunctionNeedsAStringPosition) +{ + /// A serialized state derives as `String`, or `Nullable(String)` for an optional Iceberg field. + EXPECT_TRUE(matches("AggregateFunction(uniq, UInt64)", "String")); + EXPECT_TRUE(matches("AggregateFunction(uniq, UInt64)", "Nullable(String)")); + + /// Anything else means the annotation does not describe these bytes. + EXPECT_FALSE(matches("AggregateFunction(uniq, UInt64)", "Int64")); + EXPECT_FALSE(matches("AggregateFunction(uniq, UInt64)", "FixedString(16)")); + EXPECT_FALSE(matches("AggregateFunction(uniq, UInt64)", "Array(String)")); + + /// Strictness elsewhere does not loosen an `AggregateFunction` position, and this pair is the + /// check the parquet reader makes for a state column. + EXPECT_TRUE(matchesStrictly("AggregateFunction(uniq, UInt64)", "String")); + EXPECT_FALSE(matchesStrictly("AggregateFunction(uniq, UInt64)", "Int64")); +} + +TEST(AnnotatedTypeMatchesDerived, ContainersAreMatchedElementwise) +{ + EXPECT_TRUE(matches("Array(AggregateFunction(uniq, UInt64))", "Array(String)")); + EXPECT_TRUE(matches("Array(AggregateFunction(uniq, UInt64))", "Array(Nullable(String))")); + EXPECT_FALSE(matches("Array(AggregateFunction(uniq, UInt64))", "Array(Int64)")); + + EXPECT_TRUE(matches("Map(String, AggregateFunction(uniq, UInt64))", "Map(String, String)")); + EXPECT_FALSE(matches("Map(String, AggregateFunction(uniq, UInt64))", "Map(String, Int64)")); + + /// Tuple elements are matched by position, so a state in one element does not excuse another. + EXPECT_TRUE(matches("Tuple(a AggregateFunction(uniq, UInt64), b UInt64)", "Tuple(a String, b Int64)")); + EXPECT_FALSE(matches("Tuple(a AggregateFunction(uniq, UInt64), b UInt64)", "Tuple(a Int64, b String)")); +} + +TEST(AnnotatedTypeMatchesDerived, TheNestingStructureItselfMustMatch) +{ + EXPECT_FALSE(matches("Array(AggregateFunction(uniq, UInt64))", "String")); + EXPECT_FALSE(matches("Map(String, AggregateFunction(uniq, UInt64))", "Array(Tuple(String, String))")); + EXPECT_FALSE(matches("Tuple(a AggregateFunction(uniq, UInt64), b UInt64)", "String")); + /// A tuple that lost or gained an element is not the same column either. + EXPECT_FALSE(matches("Tuple(a AggregateFunction(uniq, UInt64), b UInt64)", "Tuple(a String)")); +} + +TEST(AnnotatedTypeMatchesDerived, PositionsThatDoNotHoldStatesAreLenient) +{ + /// `SimpleAggregateFunction` holds an ordinary value, so the annotation is expected to disagree + /// with the derived type: Iceberg stores `UInt64` as `long`, which reads back as `Int64`. + EXPECT_TRUE(matches("SimpleAggregateFunction(sum, UInt64)", "Int64")); + EXPECT_TRUE(matches("SimpleAggregateFunction(anyLast, Nullable(String))", "Nullable(String)")); + EXPECT_TRUE(matches("Array(SimpleAggregateFunction(sum, UInt64))", "Array(Int64)")); +} + +TEST(AnnotatedTypeMatchesDerived, StrictModeRefusesAnAnnotationThatRetypesTheColumn) +{ + /// The reason strict mode exists: `castColumn` converts an integer to a timestamp silently, so a + /// lenient check would let an annotation decide what a column of numbers means, with no way to + /// opt out. Parquet writes `UInt64` as UINT_64 and `DateTime64(9)` as TIMESTAMP(NANOS), neither + /// of which derives as `Int64`. + EXPECT_FALSE(matchesStrictly("SimpleAggregateFunction(anyLast, DateTime64(9))", "Int64")); + EXPECT_FALSE(matchesStrictly("SimpleAggregateFunction(sum, UInt64)", "Int64")); + /// Lenient mode still accepts both - it has to, see PositionsThatDoNotHoldStatesAreLenient. + EXPECT_TRUE(matches("SimpleAggregateFunction(anyLast, DateTime64(9))", "Int64")); + + /// Same for the other reinterpretations parquet round-trips too exactly to excuse. + EXPECT_FALSE(matchesStrictly("SimpleAggregateFunction(anyLast, Date)", "Int32")); + EXPECT_FALSE(matchesStrictly("SimpleAggregateFunction(anyLast, IPv4)", "Int32")); + EXPECT_FALSE(matchesStrictly("SimpleAggregateFunction(anyLast, Float64)", "Int64")); + EXPECT_FALSE(matchesStrictly("SimpleAggregateFunction(anyLast, String)", "FixedString(16)")); + EXPECT_FALSE(matchesStrictly("SimpleAggregateFunction(anyLast, Decimal(18, 4))", "Decimal(18, 2)")); + /// A scale parquet cannot have is rounded up to the next unit, not down and not to any other. + EXPECT_FALSE(matchesStrictly("SimpleAggregateFunction(anyLast, DateTime64(4))", "DateTime64(3)")); + EXPECT_FALSE(matchesStrictly("SimpleAggregateFunction(anyLast, DateTime64(4))", "DateTime64(9)")); + + /// A state in one position does not excuse a re-typing in another. + EXPECT_FALSE(matchesStrictly( + "Tuple(a AggregateFunction(uniq, UInt64), b DateTime64(9))", "Tuple(a String, b Int64)")); + EXPECT_FALSE(matchesStrictly("Array(SimpleAggregateFunction(sum, UInt64))", "Array(Int64)")); + EXPECT_FALSE(matchesStrictly( + "Map(String, SimpleAggregateFunction(anyLast, DateTime64(9)))", "Map(String, Int64)")); +} + +TEST(AnnotatedTypeMatchesDerived, StrictModeAcceptsWhatTheParquetWriterProduces) +{ + /// Every pair below is a type the parquet writer can be handed and the type the reader derives + /// from what it wrote; 04673_parquet_aggregate_function_state.sh round-trips them through the real + /// writer. The mapping is not injective, so this is a directed relation: `Date` may derive as + /// `Date32`, but a `Date32` annotation over a UINT_16 may not. + + /// Types parquet stores exactly. + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(sum, UInt64)", "UInt64")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, String)", "String")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, UUID)", "UUID")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Decimal(18, 4))", "Decimal(18, 4)")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, FixedString(16))", "FixedString(16)")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Date32)", "Date32")); + + /// The annotation and the derived type do not carry `Nullable` the same way, and it says nothing + /// about the values, so it is ignored on both sides. + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(sum, UInt64)", "Nullable(UInt64)")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Nullable(String))", "String")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Nullable(String))", "Nullable(String)")); + /// And at every level, not just the top: a list element is written OPTIONAL by default. + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Array(UInt64))", "Array(Nullable(UInt64))")); + EXPECT_TRUE(matchesStrictly( + "SimpleAggregateFunction(anyLast, Map(String, UInt64))", "Map(String, Nullable(UInt64))")); + EXPECT_TRUE(matchesStrictly( + "Tuple(a AggregateFunction(uniq, UInt64), b UInt64)", "Tuple(a Nullable(String), b Nullable(UInt64))")); + + /// A `LowCardinality` column is written through its dictionary type, and no parquet schema + /// derives back as `LowCardinality`. + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, LowCardinality(String))", "String")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, LowCardinality(String))", "Nullable(String)")); + EXPECT_TRUE(matchesStrictly( + "SimpleAggregateFunction(anyLast, LowCardinality(Nullable(String)))", "Nullable(String)")); + + /// Types parquet has no logical type for, written as the nearest thing it does have. + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Date)", "Date32")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Date)", "UInt16")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, DateTime)", "DateTime64(3, 'UTC')")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, DateTime)", "UInt32")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, DateTime64(9))", "DateTime64(9, 'UTC')")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, DateTime64(0))", "DateTime64(3, 'UTC')")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, DateTime64(4))", "DateTime64(6, 'UTC')")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, DateTime64(7))", "DateTime64(9, 'UTC')")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Time)", "DateTime64(6, 'UTC')")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Time64(3))", "DateTime64(6, 'UTC')")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Time64(9))", "DateTime64(9, 'UTC')")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Enum8('a' = 1))", "String")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Enum8('a' = 1))", "Int8")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Enum16('a' = 1))", "String")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Enum16('a' = 1))", "Int16")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, IPv4)", "UInt32")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, IPv6)", "FixedString(16)")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, Int128)", "FixedString(16)")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, UInt256)", "FixedString(32)")); + EXPECT_TRUE(matchesStrictly("SimpleAggregateFunction(anyLast, FixedString(16))", "String")); +} diff --git a/src/DataTypes/tests/gtest_ast_has_aggregate_function_type.cpp b/src/DataTypes/tests/gtest_ast_has_aggregate_function_type.cpp new file mode 100644 index 000000000000..ac509b86947e --- /dev/null +++ b/src/DataTypes/tests/gtest_ast_has_aggregate_function_type.cpp @@ -0,0 +1,63 @@ +#include + +#include +#include +#include +#include + +using namespace DB; + +/// `astHasAggregateFunctionType` decides whether a type name recorded next to the data denotes an +/// aggregate state, from the name alone: resolving it into a type is exactly what the gate around it +/// is there to prevent, so nothing here goes through `DataTypeFactory`. + +namespace +{ + +bool hasState(const String & type_name) +{ + ParserDataType parser; + ASTPtr ast = parseQuery( + parser, type_name.data(), type_name.data() + type_name.size(), "data type", + /*max_query_size=*/ 0, DBMS_DEFAULT_MAX_PARSER_DEPTH, DBMS_DEFAULT_MAX_PARSER_BACKTRACKS); + return astHasAggregateFunctionType(ast); +} + +} + +TEST(AstHasAggregateFunctionType, AStateIsFoundWhereverItIsNested) +{ + EXPECT_TRUE(hasState("AggregateFunction(uniq, UInt64)")); + EXPECT_TRUE(hasState("AggregateFunction(0, sumMap, Array(UInt8), Array(UInt32))")); + EXPECT_TRUE(hasState("AggregateFunction(quantiles(0.5, 0.9), UInt64)")); + EXPECT_TRUE(hasState("Array(AggregateFunction(uniq, UInt64))")); + EXPECT_TRUE(hasState("Map(String, AggregateFunction(uniq, UInt64))")); + EXPECT_TRUE(hasState("Tuple(a UInt64, b Array(AggregateFunction(uniq, UInt64)))")); +} + +TEST(AstHasAggregateFunctionType, SimpleAggregateFunctionIsNotAState) +{ + /// It is an ordinary value of its storage type, deserialized by that type, so it is not gated. + EXPECT_FALSE(hasState("SimpleAggregateFunction(sum, UInt64)")); + EXPECT_FALSE(hasState("Array(SimpleAggregateFunction(max, String))")); + EXPECT_FALSE(hasState("Map(String, SimpleAggregateFunction(any, Nullable(UInt64)))")); +} + +TEST(AstHasAggregateFunctionType, OrdinaryTypesAreNotStates) +{ + EXPECT_FALSE(hasState("UInt64")); + EXPECT_FALSE(hasState("String")); + EXPECT_FALSE(hasState("Tuple(a UInt64, b Array(Nullable(String)))")); + + /// Only a type name counts, not a name that merely spells the same in another position. + EXPECT_FALSE(hasState("Enum8('AggregateFunction' = 1)")); + EXPECT_FALSE(hasState("Tuple(`AggregateFunction` String)")); +} + +TEST(AstHasAggregateFunctionType, TheNameIsAnsweredWithoutResolvingIt) +{ + /// A file can name an aggregate function this build does not have - precisely a name to refuse + /// rather than look up. + EXPECT_TRUE(hasState("AggregateFunction(no_such_aggregate_function, UInt64)")); + EXPECT_TRUE(hasState("Array(AggregateFunction(no_such_aggregate_function, UInt64))")); +} diff --git a/src/Formats/FormatFactory.cpp b/src/Formats/FormatFactory.cpp index d6c30363c0d2..aecee3036ba9 100644 --- a/src/Formats/FormatFactory.cpp +++ b/src/Formats/FormatFactory.cpp @@ -52,6 +52,7 @@ FORMAT_FACTORY_SETTINGS(DECLARE_FORMAT_EXTERN, INITIALIZE_SETTING_EXTERN) extern const SettingsAggregateFunctionInputFormat aggregate_function_input_format; extern const SettingsBool allow_special_serialization_kinds_in_output_formats; extern const SettingsBool allow_experimental_nullable_tuple_type; + extern const SettingsBool allow_experimental_aggregate_function_states_in_parquet; extern SettingsGeoJSONUnsupportedGeometryHandling input_format_geojson_unsupported_geometry_handling; extern SettingsBool input_format_parallel_parsing; @@ -252,6 +253,7 @@ FormatSettings getFormatSettings(const ContextPtr & context, const Settings & se format_settings.parquet.verify_checksums = settings[Setting::input_format_parquet_verify_checksums]; format_settings.parquet.local_time_as_utc = settings[Setting::input_format_parquet_local_time_as_utc]; format_settings.parquet.allow_geoparquet_parser = settings[Setting::input_format_parquet_allow_geoparquet_parser]; + format_settings.parquet.allow_aggregate_function_states = settings[Setting::allow_experimental_aggregate_function_states_in_parquet]; format_settings.parquet.write_geometadata = settings[Setting::output_format_parquet_geometadata]; if (auto memory_limit = total_memory_tracker.getHardLimit(); memory_limit > 0) { diff --git a/src/Formats/FormatSettings.h b/src/Formats/FormatSettings.h index 745898c0c751..4233e292fa2e 100644 --- a/src/Formats/FormatSettings.h +++ b/src/Formats/FormatSettings.h @@ -386,6 +386,7 @@ struct FormatSettings double bloom_filter_bits_per_value = 10.5; size_t bloom_filter_flush_threshold_bytes = 1024 * 1024 * 128; bool allow_geoparquet_parser = true; + bool allow_aggregate_function_states = false; bool write_geometadata = true; size_t max_dictionary_size = 1024 * 1024; } parquet{}; diff --git a/src/Processors/Formats/Impl/Parquet/Decoding.cpp b/src/Processors/Formats/Impl/Parquet/Decoding.cpp index 3a0e3b296a24..12ed98b79ff5 100644 --- a/src/Processors/Formats/Impl/Parquet/Decoding.cpp +++ b/src/Processors/Formats/Impl/Parquet/Decoding.cpp @@ -1,8 +1,12 @@ #include #include +#include +#include #include +#include #include +#include #include #include @@ -1713,6 +1717,51 @@ void Int96Converter::convertColumn(std::span data, size_t num_values } } +void AggregateFunctionStateConverter::convertColumn(std::span chars, const UInt64 * offsets, size_t separator_bytes, size_t num_values, IColumn & col) const +{ + auto & column = assert_cast(col); + column.set(function, version); + + /// A dictionary page earlier in the same column chunk arrives via insertRangeFrom(), which shares + /// ownership by setting `src`; while `src` is set the destructor frees nothing, so the states + /// allocated below would leak. + column.ensureOwnership(); + + auto & data = column.getData(); + data.reserve(data.size() + num_values); + + Arena & arena = column.createOrGetArena(); + const size_t size_of_state = function->sizeOfData(); + const size_t align_of_state = function->alignOfData(); + + chassert(chars.size() >= offsets[num_values - 1]); + for (ssize_t i = 0; i < ssize_t(num_values); ++i) + { + const char * ptr = chars.data() + offsets[i - 1]; + const size_t length = offsets[i] - offsets[i - 1] - separator_bytes; + ReadBufferFromMemory in(ptr, length); + + AggregateDataPtr place = arena.alignedAlloc(size_of_state, align_of_state); + function->create(place); + try + { + function->deserialize(place, in, version, &arena); + if (!in.eof()) + throw Exception( + ErrorCodes::INCORRECT_DATA, + "Aggregate function state for `{}` has {} trailing byte(s) after deserialization", + function->getName(), + in.available()); + } + catch (...) + { + function->destroy(place); + throw; + } + data.push_back(place); + } +} + void GeoConverter::convertColumn(std::span chars, const UInt64 * offsets, size_t separator_bytes, size_t num_values, IColumn & col) const { col.reserve(col.size() + num_values); diff --git a/src/Processors/Formats/Impl/Parquet/Decoding.h b/src/Processors/Formats/Impl/Parquet/Decoding.h index c67ec63075fa..0d79e419b3ec 100644 --- a/src/Processors/Formats/Impl/Parquet/Decoding.h +++ b/src/Processors/Formats/Impl/Parquet/Decoding.h @@ -3,6 +3,7 @@ #include #include #include +#include namespace DB::ErrorCodes { @@ -320,6 +321,24 @@ struct GeoConverter : public StringConverter void convertColumn(std::span chars, const UInt64 * offsets, size_t separator_bytes, size_t num_values, IColumn & col) const override; }; +/// Input physical type: BYTE_ARRAY holding serialized aggregate-function states, as written by +/// convertAggregateFunctionColumnToString() in PrepareForWrite.cpp. +/// Output column type: ColumnAggregateFunction. +/// +/// Not castColumn(String -> AggregateFunction), which goes through +/// SerializationAggregateFunction::deserializeWholeText and so depends on the session setting +/// `aggregate_function_input_format`. +struct AggregateFunctionStateConverter : public StringConverter +{ + AggregateFunctionPtr function; + size_t version; + + AggregateFunctionStateConverter(AggregateFunctionPtr function_, size_t version_) + : function(std::move(function_)), version(version_) {} + + void convertColumn(std::span chars, const UInt64 * offsets, size_t separator_bytes, size_t num_values, IColumn & col) const override; +}; + void decodeRepOrDefLevels(parq::Encoding::type encoding, UInt8 max, size_t num_values, std::span data, PaddedPODArray & out); diff --git a/src/Processors/Formats/Impl/Parquet/PrepareForWrite.cpp b/src/Processors/Formats/Impl/Parquet/PrepareForWrite.cpp index 7d657c4548dd..f863dd305c94 100644 --- a/src/Processors/Formats/Impl/Parquet/PrepareForWrite.cpp +++ b/src/Processors/Formats/Impl/Parquet/PrepareForWrite.cpp @@ -25,6 +25,10 @@ #include #include #include +#include +#include +#include +#include /// This file deals with schema conversion and with repetition and definition levels. @@ -810,6 +814,34 @@ void validateIcebergFieldIds( } } +/// Parquet has no aggregate-state type, so an `AggregateFunction` column is written as an opaque +/// BYTE_ARRAY of serialized states. writeFileFooter() records the ClickHouse type name, which +/// includes the state version, in the file's key-value metadata. +void convertAggregateFunctionColumnToString(ColumnPtr & column, DataTypePtr & type) +{ + const auto & aggregate_type = assert_cast(*type); + const auto & function = aggregate_type.getFunction(); + const size_t version = aggregate_type.getVersion(); + const auto & states = assert_cast(*column).getData(); + + auto result = ColumnString::create(); + auto & chars = result->getChars(); + auto & offsets = result->getOffsets(); + offsets.reserve(states.size()); + { + WriteBufferFromVector buffer(chars); + for (const auto * state : states) + { + function->serialize(state, buffer, version); + offsets.push_back(buffer.count()); + } + buffer.finalize(); + } + + column = std::move(result); + type = std::make_shared(); +} + void prepareColumnRecursive( ColumnPtr column, DataTypePtr type, const std::string & name, const WriteOptions & options, ColumnChunkWriteStates & states, SchemaElements & schemas, const std::optional> & column_field_ids) @@ -818,6 +850,29 @@ void prepareColumnRecursive( /// parquet dictionary-encoding. column = column->convertToFullColumnIfReplicated()->convertToFullColumnIfSparse()->convertToFullColumnIfConst(); + if (type->getTypeId() == TypeIndex::AggregateFunction) + { + /// The states and the type name recorded next to them are a ClickHouse-only convention no + /// other reader can interpret, so writing them is opt-in. Without the setting, keep throwing + /// `UNKNOWN_TYPE` the way `preparePrimitiveColumn` used to. + if (!options.allow_aggregate_function_states) + throw Exception( + ErrorCodes::UNKNOWN_TYPE, + "Internal type '{}' of column '{}' is not supported for conversion into Parquet data format. " + "Enable setting allow_experimental_aggregate_function_states_in_parquet to write the " + "serialized states", + type->getFamilyName(), name); + + convertAggregateFunctionColumnToString(column, type); + /// Serialized states are arbitrary bytes, not text: no STRING/UTF8 logical type, even though + /// `output_format_parquet_string_as_string` is on by default. + WriteOptions binary_options = options; + binary_options.output_string_as_string = false; + preparePrimitiveColumn( + column, type, name, binary_options, states, schemas, lookupLeafFieldId(column_field_ids, name)); + return; + } + switch (type->getTypeId()) { case TypeIndex::Nullable: prepareColumnNullable(column, type, name, options, states, schemas, column_field_ids); break; diff --git a/src/Processors/Formats/Impl/Parquet/SchemaConverter.cpp b/src/Processors/Formats/Impl/Parquet/SchemaConverter.cpp index e365f096d7cf..6d2b9a2dc35d 100644 --- a/src/Processors/Formats/Impl/Parquet/SchemaConverter.cpp +++ b/src/Processors/Formats/Impl/Parquet/SchemaConverter.cpp @@ -18,6 +18,11 @@ #include #include #include +#include +#include +#include +#include +#include #include @@ -53,6 +58,42 @@ SchemaConverter::SchemaConverter( } } } + + /// Type names recorded by the ClickHouse writer for columns the parquet schema cannot describe on + /// its own (aggregate states); see writeFileFooter() in Write.cpp. Only the JSON is parsed here - + /// resolving a name is inferSchema()'s job, per column and after the aggregate-state opt-in, so + /// that a bad annotation breaks at most the column it belongs to. With an explicit structure + /// (sample_block != null) the annotation is ignored, so don't even parse the JSON. + for (const auto & kv : file_metadata.key_value_metadata) + { + if (sample_block) + break; + + if (kv.key != clickhouse_column_types_key) + continue; + + try + { + Poco::JSON::Parser parser; + const auto object = parser.parse(kv.value).extract(); + for (const auto & name : object->getNames()) + clickhouse_column_type_names[name] = object->getValue(name); + } + catch (Exception & e) + { + e.addMessage("while parsing the `{}` key-value metadata of the parquet file", clickhouse_column_types_key); + throw; + } + catch (const Poco::Exception & e) + { + throw Exception( + ErrorCodes::INCORRECT_DATA, + "Cannot parse the `{}` key-value metadata of the parquet file: {}", + clickhouse_column_types_key, + e.displayText()); + } + break; + } } void SchemaConverter::checkHasColumns() @@ -118,6 +159,52 @@ void SchemaConverter::prepareForReading() } } +DataTypePtr SchemaConverter::resolveAnnotatedType(const String & column_name, const String & type_name) const +{ + ASTPtr ast; + try + { + ParserDataType parser; + ast = parseQuery( + parser, type_name.data(), type_name.data() + type_name.size(), "data type", + /*max_query_size=*/ 0, options.format.max_parser_depth, DBMS_DEFAULT_MAX_PARSER_BACKTRACKS); + } + catch (Exception & e) + { + throw Exception( + ErrorCodes::INCORRECT_DATA, + "Parquet file records ClickHouse type {} for column {} in its `{}` metadata, which is not a " + "valid type name: {}", + type_name, column_name, clickhouse_column_types_key, e.message()); + } + + /// The file, not the query, names the aggregate function whose deserializer is handed the state + /// bytes, hence the opt-in. Ask the parsed name rather than the resolved type, since resolving it + /// is what looks the aggregate function up. Reject rather than fall back to the inferred type: + /// reading states as strings would be a wrong result, not an error. + if (astHasAggregateFunctionType(ast) && !options.format.parquet.allow_aggregate_function_states) + throw Exception( + ErrorCodes::INCORRECT_DATA, + "Parquet file records ClickHouse type {} for column {} in its `{}` metadata. Inferring " + "aggregate function states from parquet metadata is disabled: enable setting " + "allow_experimental_aggregate_function_states_in_parquet to honour the " + "recorded type, or pass the structure explicitly", + type_name, column_name, clickhouse_column_types_key); + + try + { + return DataTypeFactory::instance().get(ast); + } + catch (Exception & e) + { + throw Exception( + ErrorCodes::INCORRECT_DATA, + "Parquet file records ClickHouse type {} for column {} in its `{}` metadata, which this " + "server cannot resolve: {}", + type_name, column_name, clickhouse_column_types_key, e.message()); + } +} + NamesAndTypesList SchemaConverter::inferSchema() { chassert(!sample_block); @@ -132,7 +219,41 @@ NamesAndTypesList SchemaConverter::inferSchema() if (node.output_idx.has_value()) { const OutputColumnInfo & col = output_columns.at(node.output_idx.value()); - res.emplace_back(col.name, col.output_type); + auto it = clickhouse_column_type_names.find(col.name); + if (it == clickhouse_column_type_names.end()) + { + res.emplace_back(col.name, col.output_type); + continue; + } + + DataTypePtr annotated_type; + try + { + annotated_type = resolveAnnotatedType(col.name, it->second); + + /// Strict: this reader knows the writer that recorded the annotation, so it can + /// require the annotated type to be one that writer maps to what the column holds. + /// Anything else is stale or crafted, and would re-type the column to whatever it + /// names - with no opt-out, `SimpleAggregateFunction` being honoured ungated. + if (!annotatedTypeMatchesDerived(annotated_type, col.output_type, /*strict=*/ true)) + throw Exception( + ErrorCodes::INCORRECT_DATA, + "Parquet file records ClickHouse type {} for column {} in its `{}` metadata, but the " + "parquet schema for that column reads as {}", + annotated_type->getName(), col.name, clickhouse_column_types_key, col.output_type->getName()); + } + catch (Exception & e) + { + /// An annotation this server won't honour concerns one column, so let the file's + /// other columns still be inferred, the same way processSubtreePrimitive() skips a + /// column of a parquet type it can't read. + if (options.format.parquet.skip_columns_with_unsupported_types_in_schema_inference + && (e.code() == ErrorCodes::INCORRECT_DATA || e.code() == ErrorCodes::NOT_IMPLEMENTED)) + continue; + throw; + } + + res.emplace_back(col.name, annotated_type); } } return res; @@ -418,10 +539,11 @@ bool SchemaConverter::processSubtreePrimitive(TraversalNode & node) } /// GeoParquet types like Point or Polygon can't be inside Nullable. - /// Geometry (Variant) is also not Nullable-compatible. + /// Geometry (Variant) is also not Nullable-compatible, and neither is AggregateFunction. if (typeid_cast(inferred_type.get()) || typeid_cast(inferred_type.get()) - || typeid_cast(inferred_type.get())) + || typeid_cast(inferred_type.get()) + || typeid_cast(inferred_type.get())) { output_nullable = false; output_nullable_if_not_json = false; @@ -1009,6 +1131,23 @@ void SchemaConverter::processPrimitiveColumn( } } + /// Aggregate states are written as opaque BYTE_ARRAY by convertAggregateFunctionColumnToString(). + if (const auto * aggregate_type = typeid_cast(type_hint.get())) + { + if (type != parq::Type::BYTE_ARRAY) + throw Exception( + ErrorCodes::TYPE_MISMATCH, + "Column is requested as {} but its parquet physical type is {}, not BYTE_ARRAY", + type_hint->getName(), thriftToString(type)); + + out_inferred_type = type_hint; + /// Min/max over serialized states is meaningless, so it must not be used for pruning. + out_decoder.allow_stats = false; + out_decoder.string_converter = std::make_shared( + aggregate_type->getFunction(), aggregate_type->getVersion()); + return; + } + /// GeoParquet. /// Spec says "Geometry columns MUST be at the root of the schema", but we allow them to be /// nested in tuples etc, why not. (Though nesting in arrays/maps probably currently wouldn't diff --git a/src/Processors/Formats/Impl/Parquet/SchemaConverter.h b/src/Processors/Formats/Impl/Parquet/SchemaConverter.h index f5bd40089f7d..743bf772192d 100644 --- a/src/Processors/Formats/Impl/Parquet/SchemaConverter.h +++ b/src/Processors/Formats/Impl/Parquet/SchemaConverter.h @@ -44,6 +44,13 @@ struct SchemaConverter /// The key is the parquet column name, without ColumnMapper. std::unordered_map geo_columns; + /// Type names recorded by the ClickHouse writer under the `clickhouse.column_types` key-value + /// metadata entry, keyed by top-level column name. Kept as raw strings: the file, not the query, + /// controls them, so a name becomes a type only in inferSchema(), only for a column that ends up + /// in the schema, and only after the aggregate-state opt-in has been checked. Empty when a sample + /// block is given, in which case the requested type wins. + std::unordered_map clickhouse_column_type_names; + SchemaConverter(const parq::FileMetaData &, const ReadOptions &, const Block *); void prepareForReading(); @@ -128,6 +135,11 @@ struct SchemaConverter void checkHasColumns(); + /// Turns the type name the writer recorded for `column_name` into a type, applying the + /// aggregate-state opt-in before the name reaches `DataTypeFactory`. Every failure is reported as + /// `INCORRECT_DATA` naming the column, so inferSchema() can skip just that column. + DataTypePtr resolveAnnotatedType(const String & column_name, const String & type_name) const; + void processSubtree(TraversalNode & node); /// These functions are used by processSubtree for different kinds of SchemaElement. diff --git a/src/Processors/Formats/Impl/Parquet/ThriftUtil.h b/src/Processors/Formats/Impl/Parquet/ThriftUtil.h index 81292cb53dce..7219483a37ff 100644 --- a/src/Processors/Formats/Impl/Parquet/ThriftUtil.h +++ b/src/Processors/Formats/Impl/Parquet/ThriftUtil.h @@ -13,6 +13,11 @@ namespace DB::Parquet /// Namespace with structs generated from parquet.thrift namespace parq = parquet::format; +/// Key-value metadata key under which ClickHouse records the names of column types that cannot be +/// recovered from the parquet schema alone (aggregate states). The value is a JSON object mapping +/// top-level column name to ClickHouse type name. +constexpr const char * clickhouse_column_types_key = "clickhouse.column_types"; + /// All templates are explicitly instantiated, feel free to add more types. /// Returns number of bytes written. diff --git a/src/Processors/Formats/Impl/Parquet/Write.cpp b/src/Processors/Formats/Impl/Parquet/Write.cpp index a808c549abf1..4b75d6d42c60 100644 --- a/src/Processors/Formats/Impl/Parquet/Write.cpp +++ b/src/Processors/Formats/Impl/Parquet/Write.cpp @@ -27,6 +27,8 @@ #include #include #include +#include +#include #if USE_SNAPPY #include @@ -1573,6 +1575,34 @@ void writeFileFooter(FileWriteState & file, } } + /// Record the names of types the parquet schema cannot describe on its own (aggregate states), + /// so that schema inference can reconstruct them. + { + Poco::JSON::Object::Ptr column_types = new Poco::JSON::Object; + bool any = false; + for (const auto & [column_name, type] : header.getNamesAndTypesList()) + { + if (!needsClickHouseTypeAnnotation(type)) + continue; + column_types->set(column_name, getClickHouseTypeAnnotationName(type)); + any = true; + } + + if (any) + { + std::ostringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM + oss; + Poco::JSON::Stringifier::stringify(column_types, oss); + + parquet::format::KeyValue key_value; + key_value.__set_key(clickhouse_column_types_key); + key_value.__set_value(oss.str()); + + meta.key_value_metadata.push_back(std::move(key_value)); + meta.__isset.key_value_metadata = true; + } + } + size_t footer_size = serializeThriftStruct(meta, out); if (footer_size > INT32_MAX) diff --git a/src/Processors/Formats/Impl/Parquet/Write.h b/src/Processors/Formats/Impl/Parquet/Write.h index e7d657819bb2..4fa8bbb86105 100644 --- a/src/Processors/Formats/Impl/Parquet/Write.h +++ b/src/Processors/Formats/Impl/Parquet/Write.h @@ -25,6 +25,8 @@ struct WriteOptions bool output_datetime_as_uint32 = false; bool output_date_as_uint16 = false; bool output_enum_as_byte_array = false; + /// See `allow_experimental_aggregate_function_states_in_parquet`. + bool allow_aggregate_function_states = false; /// Note: the meaning of some compression methods here is different from /// wrapReadBufferWithCompressionMethod: diff --git a/src/Processors/Formats/Impl/ParquetBlockOutputFormat.cpp b/src/Processors/Formats/Impl/ParquetBlockOutputFormat.cpp index 161da8da9aa4..a2655a2299df 100644 --- a/src/Processors/Formats/Impl/ParquetBlockOutputFormat.cpp +++ b/src/Processors/Formats/Impl/ParquetBlockOutputFormat.cpp @@ -49,6 +49,7 @@ ParquetBlockOutputFormat::ParquetBlockOutputFormat(WriteBuffer & out_, SharedHea options.output_datetime_as_uint32 = format_settings.parquet.output_datetime_as_uint32; options.output_date_as_uint16 = format_settings.parquet.output_date_as_uint16; options.output_enum_as_byte_array = format_settings.parquet.output_enum_as_byte_array; + options.allow_aggregate_function_states = format_settings.parquet.allow_aggregate_function_states; options.data_page_size = format_settings.parquet.data_page_size; options.write_batch_size = format_settings.parquet.write_batch_size; options.write_page_index = format_settings.parquet.write_page_index; diff --git a/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp b/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp index 6a238834caec..f1febe5bdbec 100644 --- a/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp +++ b/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp @@ -597,10 +597,16 @@ void registerParquetSchemaReader(FormatFactory & factory) "Parquet", [](const FormatSettings & settings) { + /// Both aggregate-state settings change the inferred schema, not just how the file is + /// read: whether a recorded `AggregateFunction` annotation is honoured, and whether a + /// refused column is dropped or makes inference throw. return fmt::format( - "schema_inference_make_columns_nullable={};enable_json_parsing={}", + "schema_inference_make_columns_nullable={};enable_json_parsing={};" + "allow_aggregate_function_states={};skip_columns_with_unsupported_types_in_schema_inference={}", settings.schema_inference_make_columns_nullable, - settings.parquet.enable_json_parsing); + settings.parquet.enable_json_parsing, + settings.parquet.allow_aggregate_function_states, + settings.parquet.skip_columns_with_unsupported_types_in_schema_inference); }); } diff --git a/src/Storages/ExportReplicatedMergeTreePartitionManifest.h b/src/Storages/ExportReplicatedMergeTreePartitionManifest.h index f592c46d9c22..35a5321d5e3d 100644 --- a/src/Storages/ExportReplicatedMergeTreePartitionManifest.h +++ b/src/Storages/ExportReplicatedMergeTreePartitionManifest.h @@ -241,6 +241,11 @@ struct ExportReplicatedMergeTreePartitionManifest String filename_pattern; bool write_full_path_in_iceberg_metadata = false; bool allow_lossy_cast = false; + /// The two experimental gates for aggregate function states. Both are off by default, so + /// carrying the initiator's choice here is what lets `ALTER TABLE ... EXPORT PARTITION ... + /// SETTINGS ...` enable the feature for one export rather than server-wide. + bool allow_aggregate_function_states_in_parquet = false; + bool allow_aggregate_function_states_in_iceberg = false; String iceberg_metadata_json; /// Optional because of backwards compatibility @@ -289,6 +294,8 @@ struct ExportReplicatedMergeTreePartitionManifest json.set("task_timeout_seconds", task_timeout_seconds); json.set("write_full_path_in_iceberg_metadata", write_full_path_in_iceberg_metadata); json.set("allow_lossy_cast", allow_lossy_cast); + json.set("allow_aggregate_function_states_in_parquet", allow_aggregate_function_states_in_parquet); + json.set("allow_aggregate_function_states_in_iceberg", allow_aggregate_function_states_in_iceberg); if (parquet_compression_method) json.set("parquet_compression_method", *parquet_compression_method); if (output_format_compression_level) @@ -370,6 +377,18 @@ struct ExportReplicatedMergeTreePartitionManifest /// on upgrade. New tasks always persist the initiator's actual choice. manifest.allow_lossy_cast = json->has("allow_lossy_cast") ? json->getValue("allow_lossy_cast") : true; + /// Absent for tasks created before these fields existed, which were scheduled by a version + /// that could not carry the gates at all: read them as the gates' own default, closed. + if (json->has("allow_aggregate_function_states_in_parquet")) + { + manifest.allow_aggregate_function_states_in_parquet = json->getValue("allow_aggregate_function_states_in_parquet"); + } + + if (json->has("allow_aggregate_function_states_in_iceberg")) + { + manifest.allow_aggregate_function_states_in_iceberg = json->getValue("allow_aggregate_function_states_in_iceberg"); + } + /// Left unset (nullopt) for tasks created before these fields existed - such tasks were /// always scheduled under the old, strict column-matching check (a mismatch could never /// reach scheduling in the first place), so callers should treat an absent value as diff --git a/src/Storages/MergeTree/ExportPartitionTaskScheduler.cpp b/src/Storages/MergeTree/ExportPartitionTaskScheduler.cpp index de271bc01ec7..b6a6f047e315 100644 --- a/src/Storages/MergeTree/ExportPartitionTaskScheduler.cpp +++ b/src/Storages/MergeTree/ExportPartitionTaskScheduler.cpp @@ -137,12 +137,23 @@ std::optional ExportPartitionTaskScheduler::run() const auto & manifest = entry.manifest; const auto key = entry.getCompositeKey(); - const auto database = storage.getContext()->resolveDatabase(manifest.destination_database); + + /// Resolving the destination is part of executing the task, so it runs under the task's + /// settings rather than this replica's own profile. It matters for a destination in a data + /// lake catalog database: `DatabaseDataLake::tryGetTable` rebuilds the table from the catalog + /// on every lookup and always asks for its schema, and `IcebergSchemaProcessor::getFieldType` + /// reads the aggregate-state gate from the context of that parse. With the raw server context + /// the parse throws `SUPPORT_IS_DISABLED`, the catalog turns that into "no such table", and + /// the task is skipped here regardless of how the scheduling `ALTER` was written. An ordinary + /// table engine is already attached, so either context resolves it the same way. + const auto destination_resolution_context = ExportPartitionUtils::getContextCopyWithTaskSettings(storage.getContext(), manifest); + + const auto database = destination_resolution_context->resolveDatabase(manifest.destination_database); const auto & table = manifest.destination_table; const auto destination_storage_id = StorageID(QualifiedTableName {database, table}); - const auto destination_storage = DatabaseCatalog::instance().tryGetTable(destination_storage_id, storage.getContext()); + const auto destination_storage = DatabaseCatalog::instance().tryGetTable(destination_storage_id, destination_resolution_context); if (!destination_storage) { diff --git a/src/Storages/MergeTree/ExportPartitionUtils.cpp b/src/Storages/MergeTree/ExportPartitionUtils.cpp index be172a927aa8..a6320a3925de 100644 --- a/src/Storages/MergeTree/ExportPartitionUtils.cpp +++ b/src/Storages/MergeTree/ExportPartitionUtils.cpp @@ -257,6 +257,16 @@ namespace ExportPartitionUtils /// schema drifts to a lossy target between scheduling and execution. context_copy->setSetting("export_merge_tree_part_allow_lossy_cast", manifest.allow_lossy_cast); + /// Reapply the initiator's aggregate-state opt-ins. The data file this task writes is + /// Parquet, so without the first one the writer refuses an `AggregateFunction` column with + /// `UNKNOWN_TYPE`. The second one matters when the destination lives in a data lake catalog + /// database: resolving it re-parses the destination's Iceberg schema, whose `clickhouse.type` + /// annotation is honoured only while that setting is on. + context_copy->setSetting( + "allow_experimental_aggregate_function_states_in_parquet", manifest.allow_aggregate_function_states_in_parquet); + context_copy->setSetting( + "allow_experimental_aggregate_function_states_in_iceberg", manifest.allow_aggregate_function_states_in_iceberg); + if (manifest.iceberg_partition_timezone) { context_copy->setSetting("iceberg_partition_timezone", *manifest.iceberg_partition_timezone); diff --git a/src/Storages/MergeTree/tests/gtest_export_partition_ordering.cpp b/src/Storages/MergeTree/tests/gtest_export_partition_ordering.cpp index 49c86a51e66d..2c305733fc8a 100644 --- a/src/Storages/MergeTree/tests/gtest_export_partition_ordering.cpp +++ b/src/Storages/MergeTree/tests/gtest_export_partition_ordering.cpp @@ -15,6 +15,8 @@ namespace Setting { extern const SettingsMergeTreePartExportSchemaMatchMode export_merge_tree_part_schema_match_mode; extern const SettingsBool export_merge_tree_part_ignore_extra_source_columns; + extern const SettingsBool allow_experimental_aggregate_function_states_in_parquet; + extern const SettingsBool allow_experimental_aggregate_function_states_in_iceberg; } namespace @@ -221,4 +223,42 @@ TEST_F(ExportPartitionManifestBackCompatTest, IgnoreExtraSourceColumnsAppliedToW } } +TEST_F(ExportPartitionManifestBackCompatTest, MissingAggregateFunctionStateGatesParseAsDisabled) +{ + auto manifest = makeValidManifest(); + manifest.allow_aggregate_function_states_in_parquet = true; + manifest.allow_aggregate_function_states_in_iceberg = true; + + Poco::JSON::Parser parser; + auto json = parser.parse(manifest.toJsonString()).extract(); + json->remove("allow_aggregate_function_states_in_parquet"); + json->remove("allow_aggregate_function_states_in_iceberg"); + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM + oss.exceptions(std::ios::failbit); + Poco::JSON::Stringifier::stringify(json, oss); + + auto parsed = ExportReplicatedMergeTreePartitionManifest::fromJsonString(oss.str()); + EXPECT_FALSE(parsed.allow_aggregate_function_states_in_parquet); + EXPECT_FALSE(parsed.allow_aggregate_function_states_in_iceberg); +} + +TEST_F(ExportPartitionManifestBackCompatTest, AggregateFunctionStateGatesAppliedToWorkerContextForEveryValue) +{ + for (const bool value : {false, true}) + { + auto manifest = makeValidManifest(); + manifest.allow_aggregate_function_states_in_parquet = value; + manifest.allow_aggregate_function_states_in_iceberg = value; + + auto worker_context = ExportPartitionUtils::getContextCopyWithTaskSettings(getContext().context, manifest); + + EXPECT_EQ( + worker_context->getSettingsRef()[Setting::allow_experimental_aggregate_function_states_in_parquet].value, + value) << "value=" << value; + EXPECT_EQ( + worker_context->getSettingsRef()[Setting::allow_experimental_aggregate_function_states_in_iceberg].value, + value) << "value=" << value; + } +} + } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h index cd7c9f29d7e3..910b2c962d98 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h @@ -165,6 +165,9 @@ DEFINE_ICEBERG_FIELD_ALIAS(ref_min_snapshots_to_keep, min-snapshots-to-keep); DEFINE_ICEBERG_FIELD_ALIAS(ref_max_snapshot_age_ms, max-snapshot-age-ms); DEFINE_ICEBERG_FIELD_ALIAS(ref_max_ref_age_ms, max-ref-age-ms); DEFINE_ICEBERG_FIELD_ALIAS(clickhouse_export_partition_transaction_id, clickhouse.export-partition-transaction-id); +/// Per-field key holding the ClickHouse type name for a column whose type the Iceberg field type +/// alone cannot express (aggregate states). +DEFINE_ICEBERG_FIELD_ALIAS(clickhouse_type, clickhouse.type); /// These are compound fields like `data_file.file_path`, we use prefix 'c_' to distinguish them. DEFINE_ICEBERG_FIELD_COMPOUND(data_file, file_path); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, file_format); diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/DataFileStatistics.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/DataFileStatistics.cpp index 4e6879496faf..3ac80ff979e0 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/DataFileStatistics.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/DataFileStatistics.cpp @@ -1,6 +1,7 @@ #include #include +#include #include #include @@ -28,6 +29,21 @@ static Range getExtremeRangeFromColumn(const ColumnPtr & column) return Range(min_val, true, max_val, true); } +/// `ColumnAggregateFunction::getExtremes` reports a serialized state, and comparing two such +/// `Field`s throws. Iceberg records no bounds for aggregate states anyway. +static bool supportsExtremeRange(const IColumn & column) +{ + if (checkAndGetColumn(&column)) + return false; + + bool result = true; + column.forEachSubcolumnRecursively([&](const IColumn & subcolumn) + { + result &= checkAndGetColumn(&subcolumn) == nullptr; + }); + return result; +} + void DataFileStatistics::update(const Chunk & chunk) { if (!chunk.hasRows()) @@ -37,9 +53,12 @@ void DataFileStatistics::update(const Chunk & chunk) { column_sizes.resize(num_columns, 0); null_counts.resize(num_columns, 0); + track_ranges.resize(num_columns); for (size_t i = 0; i < num_columns; ++i) { - ranges.push_back(getExtremeRangeFromColumn(chunk.getColumns()[i])); + const auto & col = chunk.getColumns()[i]; + track_ranges[i] = supportsExtremeRange(*col); + ranges.push_back(track_ranges[i] ? getExtremeRangeFromColumn(col) : Range::createWholeUniverse()); } } @@ -54,7 +73,8 @@ void DataFileStatistics::update(const Chunk & chunk) for (UInt8 v : nullable_col->getNullMapData()) null_counts[i] += v; } - ranges[i] = uniteRanges(ranges[i], getExtremeRangeFromColumn(col)); + if (track_ranges[i]) + ranges[i] = uniteRanges(ranges[i], getExtremeRangeFromColumn(col)); } } @@ -68,6 +88,7 @@ void DataFileStatistics::merge(const DataFileStatistics & other) column_sizes = other.column_sizes; null_counts = other.null_counts; ranges = other.ranges; + track_ranges = other.track_ranges; return; } @@ -76,7 +97,8 @@ void DataFileStatistics::merge(const DataFileStatistics & other) { column_sizes[i] += other.column_sizes[i]; null_counts[i] += other.null_counts[i]; - ranges[i] = uniteRanges(ranges[i], other.ranges[i]); + if (track_ranges[i]) + ranges[i] = uniteRanges(ranges[i], other.ranges[i]); } } @@ -115,6 +137,11 @@ std::vector> DataFileStatistics::getLowerBounds() const std::vector> result; for (size_t i = 0; i < ranges.size(); ++i) { + /// Untracked columns (aggregate states) carry a whole-universe range whose infinite bounds + /// cannot be dumped. Emitting them would fail the all-or-nothing canWriteStatistics() check + /// and drop bounds for every column in the file; omit them so the rest still prune. + if (!track_ranges[i]) + continue; result.push_back({field_ids[i], ranges[i].left}); } return result; @@ -125,6 +152,8 @@ std::vector> DataFileStatistics::getUpperBounds() const std::vector> result; for (size_t i = 0; i < ranges.size(); ++i) { + if (!track_ranges[i]) + continue; result.push_back({field_ids[i], ranges[i].right}); } return result; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/DataFileStatistics.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/DataFileStatistics.h index 12278b7fcd56..c313c25e2e29 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/DataFileStatistics.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/DataFileStatistics.h @@ -40,6 +40,8 @@ class DataFileStatistics std::vector column_sizes; std::vector null_counts; std::vector ranges; + /// False for columns whose extremes cannot be compared, see supportsExtremeRange(). + std::vector track_ranges; }; using DataFileStatisticsPtr = std::shared_ptr; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp index 24930b88462a..e75357715626 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp @@ -1,5 +1,6 @@ #include #include +#include #include #include #include "config.h" @@ -143,6 +144,7 @@ extern const SettingsString iceberg_metadata_compression_method; extern const SettingsBool allow_insert_into_iceberg; extern const SettingsBool allow_experimental_iceberg_compaction; extern const SettingsBool allow_experimental_geo_types_in_iceberg; +extern const SettingsBool allow_experimental_aggregate_function_states_in_iceberg; extern const SettingsBool allow_iceberg_remove_orphan_files; extern const SettingsBool allow_experimental_expire_snapshots; extern const SettingsBool iceberg_delete_data_on_drop; @@ -264,7 +266,8 @@ Iceberg::PersistentTableComponents IcebergMetadata::initializePersistentTableCom }; } -std::pair IcebergMetadata::getRelevantState(const ContextPtr & context, bool force_fetch_latest_metadata) const +std::pair IcebergMetadata::getRelevantState( + const ContextPtr & context, bool force_fetch_latest_metadata, SchemaParsing schema_parsing) const { const auto [metadata_version, metadata_file_path, compression_method] = getLatestOrExplicitMetadataFileAndVersion( object_storage, @@ -276,7 +279,7 @@ std::pair IcebergMetadata::getReleva persistent_components.table_uuid, persistent_components.metadata_compression_method, force_fetch_latest_metadata); - return getState(context, metadata_file_path, metadata_version); + return getState(context, metadata_file_path, metadata_version, schema_parsing); } IcebergMetadata::IcebergMetadata( @@ -337,8 +340,12 @@ void IcebergMetadata::backgroundMetadataPrefetcherThread() /// first, we fetch the latest metadata version and cache it; /// as a part of the same method, we download metadata.json of the latest metadata version /// and after parsing it, we fetch manifest lists, parse and cache them + /// + /// Nothing warmed here needs the columns' ClickHouse types, so the schema is left unparsed: + /// `IcebergSchemaProcessor::getFieldType` reads its gates from the query, and this task has + /// none. See `SchemaParsing`. auto ctx = Context::createCopy(Context::getGlobalContextInstance()); - auto [actual_data_snapshot, actual_table_state_snapshot] = getRelevantState(ctx, true); + auto [actual_data_snapshot, actual_table_state_snapshot] = getRelevantState(ctx, true, SchemaParsing::Skip); if (actual_data_snapshot) { for (const auto & entry : actual_data_snapshot->manifest_list_entries) @@ -370,14 +377,23 @@ Int32 IcebergMetadata::parseTableSchema( const Poco::JSON::Object::Ptr & metadata_object, IcebergSchemaProcessor & schema_processor, ContextPtr context_, - LoggerPtr metadata_logger) + LoggerPtr metadata_logger, + SchemaParsing schema_parsing) { const auto format_version = metadata_object->getValue(f_format_version); + /// Which schema the metadata points at is read from the metadata alone; turning its fields into + /// ClickHouse types is what `SchemaParsing::Skip` leaves out. + auto add_schema = [&](const Poco::JSON::Object::Ptr & schema) + { + if (schema_parsing == SchemaParsing::Parse) + schema_processor.addIcebergTableSchema(schema, context_); + }; + if (format_version == 2) { auto [schema, current_schema_id] = parseTableSchemaV2Method(metadata_object); - schema_processor.addIcebergTableSchema(schema, context_); + add_schema(schema); return current_schema_id; } else @@ -385,7 +401,7 @@ Int32 IcebergMetadata::parseTableSchema( try { auto [schema, current_schema_id] = parseTableSchemaV1Method(metadata_object); - schema_processor.addIcebergTableSchema(schema, context_); + add_schema(schema); return current_schema_id; } catch (const Exception & first_error) @@ -395,7 +411,7 @@ Int32 IcebergMetadata::parseTableSchema( try { auto [schema, current_schema_id] = parseTableSchemaV2Method(metadata_object); - schema_processor.addIcebergTableSchema(schema, context_); + add_schema(schema); LOG_WARNING( metadata_logger, "Iceberg table schema was parsed using v2 specification, but it was impossible to parse it using v1 " @@ -422,15 +438,22 @@ static Poco::JSON::Object::Ptr traverseMetadataAndFindNecessarySnapshotObject( Poco::JSON::Object::Ptr metadata_object, Int64 snapshot_id, IcebergSchemaProcessorPtr schema_processor, - ContextPtr local_context) + ContextPtr local_context, + IcebergMetadata::SchemaParsing schema_parsing) { if (!metadata_object->has(f_snapshots)) throw Exception(ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, "No snapshot set found in metadata for iceberg file"); - auto schemas = metadata_object->get(f_schemas).extract(); - for (UInt32 j = 0; j < schemas->size(); ++j) + /// Locating a snapshot needs no ClickHouse types, so `SchemaParsing::Skip` leaves the schemas of + /// the other metadata versions unparsed as well. The snapshot-to-schema-id registrations below + /// stay: they are pairs of ids, with no field type behind them. + if (schema_parsing == IcebergMetadata::SchemaParsing::Parse) { - auto schema = schemas->getObject(j); - schema_processor->addIcebergTableSchema(schema, local_context); + auto schemas = metadata_object->get(f_schemas).extract(); + for (UInt32 j = 0; j < schemas->size(); ++j) + { + auto schema = schemas->getObject(j); + schema_processor->addIcebergTableSchema(schema, local_context); + } } Poco::JSON::Object::Ptr current_snapshot = nullptr; auto snapshots = metadata_object->get(f_snapshots).extract(); @@ -495,14 +518,15 @@ IcebergDataSnapshotPtr IcebergMetadata::createIcebergDataSnapshotFromSnapshotJSO total_equality_deletes); } -IcebergDataSnapshotPtr -IcebergMetadata::getIcebergDataSnapshot(Poco::JSON::Object::Ptr metadata_object, Int64 snapshot_id, ContextPtr local_context) const +IcebergDataSnapshotPtr IcebergMetadata::getIcebergDataSnapshot( + Poco::JSON::Object::Ptr metadata_object, Int64 snapshot_id, ContextPtr local_context, SchemaParsing schema_parsing) const { auto object = traverseMetadataAndFindNecessarySnapshotObject( metadata_object, snapshot_id, persistent_components.schema_processor, - local_context); + local_context, + schema_parsing); if (!object) throw Exception(ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, "No snapshot found for id `{}`", snapshot_id); @@ -536,7 +560,7 @@ bool IcebergMetadata::optimize( } std::pair -IcebergMetadata::getStateImpl(const ContextPtr & local_context, Poco::JSON::Object::Ptr metadata_object) const +IcebergMetadata::getStateImpl(const ContextPtr & local_context, Poco::JSON::Object::Ptr metadata_object, SchemaParsing schema_parsing) const { std::optional manifest_list_file; @@ -577,18 +601,18 @@ IcebergMetadata::getStateImpl(const ContextPtr & local_context, Poco::JSON::Obje ErrorCodes::BAD_ARGUMENTS, "No snapshot found in snapshot log before requested timestamp for iceberg table {}", persistent_components.table_path); - auto data_snapshot = getIcebergDataSnapshot(metadata_object, *current_snapshot_id, local_context); + auto data_snapshot = getIcebergDataSnapshot(metadata_object, *current_snapshot_id, local_context, schema_parsing); return {data_snapshot, static_cast(data_snapshot->schema_id_on_snapshot_commit)}; } else if (snapshot_id_changed) { Int64 current_snapshot_id = local_context->getSettingsRef()[Setting::iceberg_snapshot_id]; - auto data_snapshot = getIcebergDataSnapshot(metadata_object, current_snapshot_id, local_context); + auto data_snapshot = getIcebergDataSnapshot(metadata_object, current_snapshot_id, local_context, schema_parsing); return {data_snapshot, static_cast(data_snapshot->schema_id_on_snapshot_commit)}; } else { - auto schema_id = parseTableSchema(metadata_object, *persistent_components.schema_processor, local_context, log); + auto schema_id = parseTableSchema(metadata_object, *persistent_components.schema_processor, local_context, log, schema_parsing); if (!metadata_object->has(f_current_snapshot_id)) { return {nullptr, schema_id}; @@ -600,13 +624,14 @@ IcebergMetadata::getStateImpl(const ContextPtr & local_context, Poco::JSON::Obje { return {nullptr, schema_id}; } - auto data_snapshot = getIcebergDataSnapshot(metadata_object, current_snapshot_id, local_context); + auto data_snapshot = getIcebergDataSnapshot(metadata_object, current_snapshot_id, local_context, schema_parsing); return {data_snapshot, schema_id}; } } std::pair -IcebergMetadata::getState(const ContextPtr & local_context, const String & metadata_path, Int32 metadata_version) const +IcebergMetadata::getState( + const ContextPtr & local_context, const String & metadata_path, Int32 metadata_version, SchemaParsing schema_parsing) const { IcebergDataSnapshotPtr data_snapshot; TableStateSnapshot table_state_snapshot; @@ -628,7 +653,7 @@ IcebergMetadata::getState(const ContextPtr & local_context, const String & metad /// queries. Downstream parsers determine the version they need from the Avro metadata of /// each manifest list / manifest file, so we do not update the shared cached value here. - std::tie(data_snapshot, table_state_snapshot.schema_id) = getStateImpl(local_context, metadata_object); + std::tie(data_snapshot, table_state_snapshot.schema_id) = getStateImpl(local_context, metadata_object, schema_parsing); table_state_snapshot.snapshot_id = data_snapshot ? std::optional{data_snapshot->snapshot_id} : std::nullopt; table_state_snapshot.metadata_version = metadata_version; table_state_snapshot.metadata_file_path = metadata_path; @@ -797,6 +822,24 @@ void IcebergMetadata::checkAlterIsPossible(const AlterCommands & commands) } } +/// Called from every DDL path that can introduce an aggregate-state column, so that the CREATE TABLE +/// check cannot be bypassed with a later ALTER. +static void checkAggregateFunctionStatesAllowed(const String & column_name, const DataTypePtr & type, const ContextPtr & context) +{ + if (!type || !needsClickHouseTypeAnnotation(type)) + return; + if (context->getSettingsRef()[Setting::allow_experimental_aggregate_function_states_in_iceberg]) + return; + + throw Exception( + ErrorCodes::SUPPORT_IS_DISABLED, + "Column '{}' has type {}, which Iceberg cannot express natively: the aggregate state is stored as " + "binary and its ClickHouse type is recorded in the schema. To allow this, enable setting " + "allow_experimental_aggregate_function_states_in_iceberg", + column_name, + type->getName()); +} + void IcebergMetadata::alter( const AlterCommands & params, ContextPtr context, @@ -811,6 +854,9 @@ void IcebergMetadata::alter( "To allow its usage, enable setting allow_insert_into_iceberg"); } + for (const auto & command : params) + checkAggregateFunctionStatesAllowed(command.column_name, command.data_type, context); + Iceberg::alter(params, context, storage_id, object_storage, data_lake_settings, persistent_components, write_format, catalog); } @@ -897,6 +943,9 @@ void IcebergMetadata::createInitial( ErrorCodes::TABLE_ALREADY_EXISTS, "Iceberg table with path {} already exists", configuration_ptr->getPathForRead().path); } + for (const auto & column : columns->getAll()) + checkAggregateFunctionStatesAllowed(column.name, column.type, local_context); + String location_path = configuration_ptr->getRawPath().path; if (location_path.find("://") == String::npos && !location_path.starts_with('/')) location_path = "/" + location_path; @@ -959,7 +1008,7 @@ Iceberg::IcebergDataSnapshotPtr IcebergMetadata::getRelevantDataSnapshotFromTabl if (!table_state_snapshot.snapshot_id.has_value()) return nullptr; Poco::JSON::Object::Ptr snapshot_object = traverseMetadataAndFindNecessarySnapshotObject( - metadata_object, *table_state_snapshot.snapshot_id, persistent_components.schema_processor, local_context); + metadata_object, *table_state_snapshot.snapshot_id, persistent_components.schema_processor, local_context, SchemaParsing::Parse); return createIcebergDataSnapshotFromSnapshotJSON(snapshot_object, *table_state_snapshot.snapshot_id, local_context); } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h index b28d476438da..7ced7561e15a 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h @@ -100,11 +100,24 @@ class IcebergMetadata : public IDataLakeMetadata std::shared_ptr getInitialSchemaByPath(ContextPtr local_context, ObjectInfoPtr object_info) const override; std::shared_ptr getSchemaTransformer(ContextPtr local_context, ObjectInfoPtr object_info) const override; + /// Whether looking up the table state also turns the table's Iceberg schema into ClickHouse + /// types and publishes them in the shared `IcebergSchemaProcessor`. + enum class SchemaParsing + { + /// The caller goes on to read the data files and needs their ClickHouse types. + Parse, + /// The caller needs the snapshot's list of files and nothing else. Parsing on its behalf would + /// apply the gates `IcebergSchemaProcessor::getFieldType` reads from the query, which such a + /// caller has none of, and publish the outcome into a cache served to every later query. + Skip, + }; + static Int32 parseTableSchema( const Poco::JSON::Object::Ptr & metadata_object, Iceberg::IcebergSchemaProcessor & schema_processor, ContextPtr context_, - LoggerPtr metadata_logger); + LoggerPtr metadata_logger, + SchemaParsing schema_parsing = SchemaParsing::Parse); bool supportsUpdate() const override { return true; } bool supportsWrites() const override { return true; } @@ -114,8 +127,10 @@ class IcebergMetadata : public IDataLakeMetadata IcebergHistory getHistory(ContextPtr local_context) const; - std::pair - getRelevantState(const ContextPtr & context, bool force_fetch_latest_metadata = false) const; + std::pair getRelevantState( + const ContextPtr & context, + bool force_fetch_latest_metadata = false, + SchemaParsing schema_parsing = SchemaParsing::Parse) const; /// Returns file records contributed by a single manifest list entry of `data_snapshot`. IcebergFiles getFilesForManifest( @@ -233,14 +248,14 @@ class IcebergMetadata : public IDataLakeMetadata ContextPtr context_, LoggerPtr log); - Iceberg::IcebergDataSnapshotPtr - getIcebergDataSnapshot(Poco::JSON::Object::Ptr metadata_object, Int64 snapshot_id, ContextPtr local_context) const; + Iceberg::IcebergDataSnapshotPtr getIcebergDataSnapshot( + Poco::JSON::Object::Ptr metadata_object, Int64 snapshot_id, ContextPtr local_context, SchemaParsing schema_parsing) const; Iceberg::IcebergDataSnapshotPtr createIcebergDataSnapshotFromSnapshotJSON(Poco::JSON::Object::Ptr snapshot_object, Int64 snapshot_id, ContextPtr local_context) const; std::pair - getStateImpl(const ContextPtr & local_context, Poco::JSON::Object::Ptr metadata_object) const; + getStateImpl(const ContextPtr & local_context, Poco::JSON::Object::Ptr metadata_object, SchemaParsing schema_parsing) const; std::pair - getState(const ContextPtr & local_context, const String & metadata_path, Int32 metadata_version) const; + getState(const ContextPtr & local_context, const String & metadata_path, Int32 metadata_version, SchemaParsing schema_parsing) const; Iceberg::IcebergDataSnapshotPtr getRelevantDataSnapshotFromTableStateSnapshot(Iceberg::TableStateSnapshot table_state_snapshot, ContextPtr local_context) const; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/MetadataGenerator.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/MetadataGenerator.cpp index 55ee1c99baf3..5a962956852a 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/MetadataGenerator.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/MetadataGenerator.cpp @@ -1,4 +1,6 @@ #include +#include +#include #include #include #include @@ -246,6 +248,16 @@ Poco::JSON::Object::Ptr MetadataGenerator::getCurrentSchema() const return current_schema; } +/// The annotation is part of a field's identity: any two `AggregateFunction` states share the same +/// Iceberg type (`binary`), and only the annotation tells them apart. +static bool clickHouseTypeAnnotationMatches(const Poco::JSON::Object::Ptr & field, const DataTypePtr & type) +{ + const bool expected = needsClickHouseTypeAnnotation(type); + if (expected != field->has(Iceberg::f_clickhouse_type)) + return false; + return !expected || field->getValue(Iceberg::f_clickhouse_type) == getClickHouseTypeAnnotationName(type); +} + bool MetadataGenerator::isAddColumnApplied(const String & column_name, DataTypePtr type) const { auto current_schema = findCurrentSchema(); @@ -264,7 +276,8 @@ bool MetadataGenerator::isAddColumnApplied(const String & column_name, DataTypeP /// The stored descriptor was produced from a lower `last-column-id` than the one we /// just used, so the ids of nested elements differ even for the very same type. return field->getValue(Iceberg::f_required) == expected_type.second - && icebergTypesEqualIgnoringIds(field->get(Iceberg::f_type), expected_type.first); + && icebergTypesEqualIgnoringIds(field->get(Iceberg::f_type), expected_type.first) + && clickHouseTypeAnnotationMatches(field, type); } return false; } @@ -319,7 +332,8 @@ bool MetadataGenerator::isModifyColumnApplied(const String & column_name, DataTy if (field->getValue(Iceberg::f_name) != column_name) continue; return field->getValue(Iceberg::f_required) == expected_type.second - && icebergTypesEqualIgnoringIds(field->get(Iceberg::f_type), expected_type.first); + && icebergTypesEqualIgnoringIds(field->get(Iceberg::f_type), expected_type.first) + && clickHouseTypeAnnotationMatches(field, type); } return false; } @@ -587,6 +601,8 @@ void MetadataGenerator::generateAddColumnMetadata(const String & column_name, Da new_field->set(Iceberg::f_name, column_name); new_field->set(Iceberg::f_required, new_type.second); new_field->set(Iceberg::f_type, new_type.first); + if (needsClickHouseTypeAnnotation(type)) + new_field->set(Iceberg::f_clickhouse_type, getClickHouseTypeAnnotationName(type)); metadata_object->set(Iceberg::f_last_column_id, last_column_id + 1); @@ -613,18 +629,32 @@ bool MetadataGenerator::generateModifyColumnMetadata(const String & column_name, && icebergTypesEqualIgnoringIds(current_field->get(Iceberg::f_type), new_type.first)) { auto existing_iceberg_type = current_field->get(Iceberg::f_type); - if (existing_iceberg_type.isString()) + if (!existing_iceberg_type.isString()) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Cannot MODIFY COLUMN '{}': the requested and existing types both map to the same " + "Iceberg complex type, and the change cannot be recorded in the Iceberg schema", + column_name); + + /// A `clickhouse.type` annotation spells the existing type out exactly; without it the + /// type has to be derived from the Iceberg type string. + DataTypePtr reconstructed_ch_type; + if (current_field->has(Iceberg::f_clickhouse_type)) { - auto reconstructed_ch_type = Iceberg::IcebergSchemaProcessor::getSimpleType( + reconstructed_ch_type + = DataTypeFactory::instance().get(current_field->getValue(Iceberg::f_clickhouse_type)); + } + else + { + reconstructed_ch_type = Iceberg::IcebergSchemaProcessor::getSimpleType( existing_iceberg_type.extract(), context, context->getSettingsRef()[Setting::allow_experimental_geo_types_in_iceberg]); if (!current_field->getValue(Iceberg::f_required) && reconstructed_ch_type->canBeInsideNullable()) reconstructed_ch_type = makeNullable(reconstructed_ch_type); + } - if (reconstructed_ch_type->equals(*type)) - return false; - + if (!reconstructed_ch_type->equals(*type)) throw Exception( ErrorCodes::BAD_ARGUMENTS, "Cannot MODIFY COLUMN '{}' from {} to {}: both map to the same Iceberg type '{}' " @@ -633,13 +663,15 @@ bool MetadataGenerator::generateModifyColumnMetadata(const String & column_name, reconstructed_ch_type->getName(), type->getName(), existing_iceberg_type.extract()); - } - throw Exception( - ErrorCodes::BAD_ARGUMENTS, - "Cannot MODIFY COLUMN '{}': the requested and existing types both map to the same " - "Iceberg complex type, and the change cannot be recorded in the Iceberg schema", - column_name); + /// `IDataType::equals` compares only the type id for numeric and string types, so + /// `UInt64` and `SimpleAggregateFunction(sum, UInt64)` compare equal; the annotation + /// has to be compared on its own. + if (clickHouseTypeAnnotationMatches(current_field, type)) + return false; + + /// Only the annotation differs, and it is stored in the schema field itself, so the + /// change is representable: fall through and write the new schema. } if (!checkValidSchemaEvolution(current_field->get(Iceberg::f_type), new_type.first)) @@ -656,6 +688,10 @@ bool MetadataGenerator::generateModifyColumnMetadata(const String & column_name, current_field->set(Iceberg::f_type, new_type.first); current_field->set(Iceberg::f_required, new_type.second); + if (needsClickHouseTypeAnnotation(type)) + current_field->set(Iceberg::f_clickhouse_type, getClickHouseTypeAnnotationName(type)); + else + current_field->remove(Iceberg::f_clickhouse_type); metadata_object->set(Iceberg::f_current_schema_id, next_schema_id); current_schema->set(Iceberg::f_schema_id, next_schema_id); diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp index 723ef8b2d00e..54192740e6c0 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp @@ -1,4 +1,5 @@ #include +#include #include #include #include @@ -22,6 +23,7 @@ #include #include +#include #include #include #include @@ -51,10 +53,12 @@ namespace ErrorCodes extern const int LOGICAL_ERROR; extern const int BAD_ARGUMENTS; extern const int ICEBERG_SPECIFICATION_VIOLATION; +extern const int SUPPORT_IS_DISABLED; } namespace Setting { +extern const SettingsBool allow_experimental_aggregate_function_states_in_iceberg; extern const SettingsTimezone iceberg_timezone_for_timestamptz; } @@ -325,9 +329,10 @@ void IcebergSchemaProcessor::addIcebergTableSchema(Poco::JSON::Object::Ptr schem std::lock_guard lock(mutex); Int32 schema_id = schema_ptr->getValue(f_schema_id); - current_schema_id = schema_id; if (iceberg_table_schemas_by_ids.contains(schema_id)) { + /// A schema-id is published below only once all of its fields have been parsed, and dropped + /// again if parsing throws, so a schema-id present here is never half-done. chassert(clickhouse_table_schemas_by_ids.contains(schema_id)); std::unordered_map type_mapping; if (allow_geo_parser) @@ -341,27 +346,48 @@ void IcebergSchemaProcessor::addIcebergTableSchema(Poco::JSON::Object::Ptr schem ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, "Iceberg schema with schema-id {} is bound to two different schemas across metadata versions", schema_id); + return; } - else - { - iceberg_table_schemas_by_ids[schema_id] = schema_ptr; - auto fields = schema_ptr->get(f_fields).extract(); - auto clickhouse_schema = std::make_shared(); - String current_full_name{}; - for (size_t i = 0; i != fields->size(); ++i) + + current_schema_id = schema_id; + /// Parsing a field throws on metadata this build cannot turn into a ClickHouse type: an unknown + /// Iceberg type, a contradictory `clickhouse.type` annotation, or a type gated behind a setting + /// the query did not enable. Everything written below is keyed by `schema_id`, so drop all of it + /// unless the whole schema parsed - otherwise the branch above would skip the retry. + bool parsed = false; + const auto rollback_unless_parsed = make_scope_guard( + [&]() TSA_NO_THREAD_SAFETY_ANALYSIS { - auto field = fields->getObject(static_cast(i)); - auto name = field->getValue(f_name); - bool required = field->getValue(f_required); - current_full_name = name; - auto type = getFieldType(field, f_type, context_, required, current_full_name, true); - clickhouse_schema->push_back(NameAndTypePair{name, type}); - clickhouse_types_by_source_ids[{schema_id, field->getValue(f_id)}] = NameAndTypePair{current_full_name, type}; - clickhouse_ids_by_source_names[{schema_id, current_full_name}] = field->getValue(f_id); - } - clickhouse_table_schemas_by_ids[schema_id] = clickhouse_schema; + /// NOTE: the exclusive lock taken above is still held here, but TSA cannot see that through a lambda. + current_schema_id = std::nullopt; + if (parsed) + return; + iceberg_table_schemas_by_ids.erase(schema_id); + clickhouse_table_schemas_by_ids.erase(schema_id); + std::erase_if(clickhouse_types_by_source_ids, [schema_id](const auto & entry) { return entry.first.first == schema_id; }); + std::erase_if(clickhouse_ids_by_source_names, [schema_id](const auto & entry) { return entry.first.first == schema_id; }); + }); + + auto fields = schema_ptr->get(f_fields).extract(); + auto clickhouse_schema = std::make_shared(); + String current_full_name{}; + for (size_t i = 0; i != fields->size(); ++i) + { + auto field = fields->getObject(static_cast(i)); + auto name = field->getValue(f_name); + bool required = field->getValue(f_required); + current_full_name = name; + auto type = getFieldType(field, f_type, context_, required, current_full_name, true); + clickhouse_schema->push_back(NameAndTypePair{name, type}); + clickhouse_types_by_source_ids[{schema_id, field->getValue(f_id)}] = NameAndTypePair{current_full_name, type}; + clickhouse_ids_by_source_names[{schema_id, current_full_name}] = field->getValue(f_id); } - current_schema_id = std::nullopt; + + /// Publish the two schema maps together, so a schema-id in one means the schema is fully parsed + /// and present in the other as well. + iceberg_table_schemas_by_ids[schema_id] = schema_ptr; + clickhouse_table_schemas_by_ids[schema_id] = clickhouse_schema; + parsed = true; } NameAndTypePair IcebergSchemaProcessor::getFieldCharacteristics(Int32 schema_version, Int32 source_id) const @@ -544,6 +570,57 @@ DataTypePtr IcebergSchemaProcessor::getFieldType( bool required, String & current_full_name, bool is_subfield_of_root) +{ + auto derived_type = getDerivedFieldType(field, type_key, context_, required, current_full_name, is_subfield_of_root); + + /// A field may carry a `clickhouse.type` key recording a type the Iceberg field type alone cannot + /// express; it is written by createEmptyMetadataFile() and the ALTER paths in MetadataGenerator. + if (!field->has(f_clickhouse_type)) + return derived_type; + + const String annotated_name = field->getValue(f_clickhouse_type); + auto annotated_type = DataTypeFactory::instance().get(annotated_name); + + /// The table's metadata, not the query, names the aggregate function whose deserializer is handed + /// the state bytes, hence the opt-in. Reject rather than fall back to the derived type: reading + /// states as strings would be a wrong result, not an error. `SimpleAggregateFunction` needs no + /// opt-in, holding ordinary values with no state deserializer involved. + /// The setting is read from the context of the call, the way `getSimpleType` above reads + /// `iceberg_timezone_for_timestamptz`, so that the reading query can opt in at all: a table engine + /// constructs its schema processor at `ATTACH`. + if (hasAggregateFunctionType(annotated_type) + && !context_->getSettingsRef()[Setting::allow_experimental_aggregate_function_states_in_iceberg]) + throw Exception( + ErrorCodes::SUPPORT_IS_DISABLED, + "Iceberg field '{}' records ClickHouse type {} in its `{}` key. Reading aggregate function " + "states from an Iceberg table is disabled: enable setting " + "allow_experimental_aggregate_function_states_in_iceberg to honour the recorded type", + field->has(f_name) ? field->getValue(f_name) : type_key, + annotated_name, + f_clickhouse_type); + + /// A stale or hand-edited annotation must not reinterpret the stored bytes as an unrelated type. + if (!annotatedTypeMatchesDerived(annotated_type, derived_type)) + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Iceberg field '{}' records ClickHouse type {} in its `{}` key, but the field itself reads as {}", + field->has(f_name) ? field->getValue(f_name) : type_key, + annotated_name, + f_clickhouse_type, + derived_type->getName()); + + /// The annotation spells out the full type, Nullable included, so it needs no makeNullable() + /// adjustment of the kind getDerivedFieldType() applies. + return annotated_type; +} + +DataTypePtr IcebergSchemaProcessor::getDerivedFieldType( + const Poco::JSON::Object::Ptr & field, + const String & type_key, + ContextPtr context_, + bool required, + String & current_full_name, + bool is_subfield_of_root) { if (field->isObject(type_key)) return getComplexTypeFromObject(field->getObject(type_key), current_full_name, context_, is_subfield_of_root); diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.h index 8533614fe799..36ff767d1014 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.h @@ -128,6 +128,15 @@ class IcebergSchemaProcessor : private WithContext String & current_full_name = default_link, bool is_subfield_of_root = false); + /// The type as the Iceberg field type alone describes it, before any `clickhouse.type` annotation. + DataTypePtr getDerivedFieldType( + const Poco::JSON::Object::Ptr & field, + const String & type_key, + ContextPtr context_, + bool required, + String & current_full_name = default_link, + bool is_subfield_of_root = false); + bool allowPrimitiveTypeConversion(const String & old_type, const String & new_type); const Node * getDefaultNodeForField(const Poco::JSON::Object::Ptr & field); diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp index 48f088cbe96e..8b39d89369da 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -554,6 +555,10 @@ std::pair getIcebergType(DataTypePtr type, Int32 & ite throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unsupported type for iceberg {}", type->getName()); case TypeIndex::String: return {"string", true}; + case TypeIndex::AggregateFunction: + /// Iceberg has no aggregate-state type, so the ClickHouse type name goes into the + /// field's `clickhouse.type` key. + return {Iceberg::f_binary, true}; case TypeIndex::UUID: return {"uuid", true}; case TypeIndex::Decimal32: @@ -1039,6 +1044,8 @@ std::pair createEmptyMetadataFile( auto type = getIcebergType(column.type, iter); field->set(Iceberg::f_required, type.second); field->set(Iceberg::f_type, type.first); + if (needsClickHouseTypeAnnotation(column.type)) + field->set(Iceberg::f_clickhouse_type, getClickHouseTypeAnnotationName(column.type)); column_name_to_source_id[column.name] = iter_for_initial_columns; schema_fields->add(field); } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_metadata_generator.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_metadata_generator.cpp index 1f2c5bec0cb3..7d1e740c9f74 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_metadata_generator.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_metadata_generator.cpp @@ -6,6 +6,8 @@ #include #include +#include +#include #include #include #include @@ -388,6 +390,39 @@ Poco::Dynamic::Var findCurrentFieldType(const Poco::JSON::Object::Ptr & metadata return {}; } +/// Metadata whose current schema holds one field carrying a `clickhouse.type` annotation. +Poco::JSON::Object::Ptr makeMetadataWithAnnotatedField( + const String & name, const Poco::Dynamic::Var & iceberg_type, bool required, const String & annotation) +{ + auto metadata = makeMetadataWithField(name, iceberg_type, required); + metadata->getArray(f_schemas)->getObject(0)->getArray(f_fields)->getObject(0)->set(f_clickhouse_type, annotation); + return metadata; +} + +/// The `clickhouse.type` annotation recorded for `name` in the schema `current-schema-id` points at. +std::optional findCurrentFieldAnnotation(const Poco::JSON::Object::Ptr & metadata, const String & name) +{ + auto current_schema_id = metadata->getValue(f_current_schema_id); + auto schemas = metadata->getArray(f_schemas); + for (UInt32 i = 0; i < schemas->size(); ++i) + { + auto schema = schemas->getObject(i); + if (schema->getValue(f_schema_id) != current_schema_id) + continue; + auto fields = schema->getArray(f_fields); + for (UInt32 j = 0; j < fields->size(); ++j) + { + auto field = fields->getObject(j); + if (field->getValue(f_name) != name) + continue; + if (!field->has(f_clickhouse_type)) + return std::nullopt; + return field->getValue(f_clickhouse_type); + } + } + return std::nullopt; +} + void expectModifyRejected( const Poco::JSON::Object::Ptr & metadata, const String & column, const DataTypePtr & requested_type) { @@ -491,4 +526,122 @@ TEST(IcebergMetadataGenerator, ModifyColumnWideningRecordsTheNewTypeInANewSchema EXPECT_EQ(stored_type.extract(), "long"); } +/// The tests below cover the changes only the `clickhouse.type` annotation records: to +/// `IDataType::equals`, `UInt64` and `SimpleAggregateFunction(sum, UInt64)` are the same type. + +TEST(IcebergMetadataGenerator, ModifyColumnToTheSameSimpleAggregateFunctionAddsNoSchema) +{ + tryRegisterAggregateFunctions(); + auto metadata + = makeMetadataWithAnnotatedField("s", "long", /* required */ true, "SimpleAggregateFunction(sum, UInt64)"); + const auto before = readSchemaState(metadata); + + EXPECT_FALSE(MetadataGenerator(metadata).generateModifyColumnMetadata( + "s", DataTypeFactory::instance().get("SimpleAggregateFunction(sum, UInt64)"), getContext().context)); + expectSchemaUnchanged(metadata, before); +} + +TEST(IcebergMetadataGenerator, ModifyColumnChangingTheSimpleAggregateFunctionRewritesTheAnnotation) +{ + tryRegisterAggregateFunctions(); + auto metadata + = makeMetadataWithAnnotatedField("s", "long", /* required */ true, "SimpleAggregateFunction(sum, UInt64)"); + const auto before = readSchemaState(metadata); + + EXPECT_TRUE(MetadataGenerator(metadata).generateModifyColumnMetadata( + "s", DataTypeFactory::instance().get("SimpleAggregateFunction(max, UInt64)"), getContext().context)); + + const auto after = readSchemaState(metadata); + EXPECT_EQ(after.schema_count, before.schema_count + 1); + EXPECT_NE(after.current_schema_id, before.current_schema_id); + EXPECT_EQ(findCurrentFieldAnnotation(metadata, "s"), "SimpleAggregateFunction(max, UInt64)"); +} + +TEST(IcebergMetadataGenerator, ModifyColumnAddingASimpleAggregateFunctionRecordsTheAnnotation) +{ + tryRegisterAggregateFunctions(); + /// `Int64` rather than `UInt64` because `long` reads back as `Int64`; only the annotation is new. + auto metadata = makeMetadataWithField("s", "long", /* required */ true); + ASSERT_EQ(findCurrentFieldAnnotation(metadata, "s"), std::nullopt); + + EXPECT_TRUE(MetadataGenerator(metadata).generateModifyColumnMetadata( + "s", DataTypeFactory::instance().get("SimpleAggregateFunction(sum, Int64)"), getContext().context)); + + EXPECT_EQ(findCurrentFieldAnnotation(metadata, "s"), "SimpleAggregateFunction(sum, Int64)"); +} + +TEST(IcebergMetadataGenerator, ModifyColumnDroppingASimpleAggregateFunctionRemovesTheAnnotation) +{ + tryRegisterAggregateFunctions(); + auto metadata + = makeMetadataWithAnnotatedField("s", "long", /* required */ true, "SimpleAggregateFunction(sum, UInt64)"); + + EXPECT_TRUE(MetadataGenerator(metadata).generateModifyColumnMetadata( + "s", std::make_shared(), getContext().context)); + + EXPECT_EQ(findCurrentFieldAnnotation(metadata, "s"), std::nullopt); +} + +TEST(IcebergMetadataGenerator, ModifyColumnRejectsChangingTheAggregateFunctionOfAState) +{ + tryRegisterAggregateFunctions(); + /// States serialized by `uniq` cannot be reinterpreted as `sum` states, so this MODIFY has to be + /// rejected rather than recorded by rewriting the annotation. + auto metadata + = makeMetadataWithAnnotatedField("u", f_binary, /* required */ true, "AggregateFunction(uniq, UInt64)"); + expectModifyRejected(metadata, "u", DataTypeFactory::instance().get("AggregateFunction(sum, UInt64)")); + EXPECT_EQ(findCurrentFieldAnnotation(metadata, "u"), "AggregateFunction(uniq, UInt64)"); +} + +/// A retry after a "commit state unknown" failure has to recognise its own aggregate-state column, +/// of which the annotation is the only record. + +TEST(IcebergMetadataGenerator, AddColumnRecordsAndDetectsTheClickHouseTypeAnnotation) +{ + tryRegisterAggregateFunctions(); + auto metadata = makeMetadataWithGap(); + MetadataGenerator gen(metadata); + + /// Iceberg only allows adding optional columns, hence the Nullable state type. + auto type = DataTypeFactory::instance().get("SimpleAggregateFunction(anyLast, Nullable(String))"); + EXPECT_FALSE(gen.isAddColumnApplied("extra", type)); + + /// Emulate the commit that the catalog applied while reporting a failure. + gen.generateAddColumnMetadata("extra", type); + EXPECT_EQ(findCurrentFieldAnnotation(metadata, "extra"), "SimpleAggregateFunction(anyLast, Nullable(String))"); + EXPECT_TRUE(gen.isAddColumnApplied("extra", type)); +} + +TEST(IcebergMetadataGenerator, AddColumnAppliedComparesTheClickHouseTypeAnnotation) +{ + tryRegisterAggregateFunctions(); + { + auto metadata = makeMetadataWithAnnotatedField( + "s", "string", /* required */ false, "SimpleAggregateFunction(anyLast, Nullable(String))"); + MetadataGenerator gen(metadata); + + EXPECT_TRUE(gen.isAddColumnApplied("s", DataTypeFactory::instance().get("SimpleAggregateFunction(anyLast, Nullable(String))"))); + /// A different aggregate function over the same storage type is a different column. + EXPECT_FALSE(gen.isAddColumnApplied("s", DataTypeFactory::instance().get("SimpleAggregateFunction(any, Nullable(String))"))); + /// So is the plain storage type, which would lose the aggregate function entirely. + EXPECT_FALSE(gen.isAddColumnApplied("s", makeNullable(std::make_shared()))); + } + { + auto metadata = makeMetadataWithAnnotatedField("u", f_binary, /* required */ true, "AggregateFunction(uniq, UInt64)"); + MetadataGenerator gen(metadata); + + EXPECT_TRUE(gen.isAddColumnApplied("u", DataTypeFactory::instance().get("AggregateFunction(uniq, UInt64)"))); + EXPECT_FALSE(gen.isAddColumnApplied("u", DataTypeFactory::instance().get("AggregateFunction(uniqExact, UInt64)"))); + } + { + /// A plain column committed under the same name is not the aggregate-state column the ALTER + /// asked for. + auto metadata = makeMetadataWithField("s", "string", /* required */ false); + MetadataGenerator gen(metadata); + + EXPECT_TRUE(gen.isAddColumnApplied("s", makeNullable(std::make_shared()))); + EXPECT_FALSE(gen.isAddColumnApplied("s", DataTypeFactory::instance().get("SimpleAggregateFunction(anyLast, Nullable(String))"))); + } +} + #endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_schema_processor.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_schema_processor.cpp index e13a421eda1f..152fe21b0299 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_schema_processor.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_schema_processor.cpp @@ -1,15 +1,28 @@ #include #include +#include #include #include #include +#include + +#include "config.h" +#if USE_AVRO +#include +#endif #include #include using namespace DB::Iceberg; +namespace DB::ErrorCodes +{ +extern const int ICEBERG_SPECIFICATION_VIOLATION; +extern const int SUPPORT_IS_DISABLED; +} + namespace { Poco::JSON::Object::Ptr parseSchema(const std::string & json) @@ -17,6 +30,32 @@ Poco::JSON::Object::Ptr parseSchema(const std::string & json) Poco::JSON::Parser parser; return parser.parse(json).extract(); } + +/// The gate is read from the context the schema processor is called with, not one captured at +/// construction, so a test sets it on a copy of the global context and passes that copy in. +DB::ContextMutablePtr contextWithAggregateFunctionStates(bool allow) +{ + auto context = DB::Context::createCopy(getContext().context); + context->setSetting("allow_experimental_aggregate_function_states_in_iceberg", DB::Field(allow)); + return context; +} + +/// An annotation that does not describe the field it sits on must be rejected as malformed metadata, +/// and by that check rather than by some unrelated failure. +void expectAnnotatedSchemaRejected(const Poco::JSON::Object::Ptr & schema) +{ + auto context = contextWithAggregateFunctionStates(true); + IcebergSchemaProcessor processor(context); + try + { + processor.addIcebergTableSchema(schema, context); + FAIL() << "The annotation does not describe the field type and must be rejected"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION) << e.message(); + } +} } TEST(IcebergSchemaProcessor, GetSimpleTypeBoolean) @@ -369,3 +408,232 @@ TEST(IcebergSchemaProcessor, GetSimpleTypeDecimalSignOnlyScaleThrows) { EXPECT_THROW(IcebergSchemaProcessor::getSimpleType("decimal(20,+)", getContext().context), DB::Exception); } + +/// An optional field derives as `Nullable(String)`, and `AggregateFunction` cannot be inside +/// `Nullable`, so the annotation is returned exactly as it stands. +TEST(IcebergSchemaProcessor, AnnotatedAggregateFunctionFieldIsHonoured) +{ + tryRegisterAggregateFunctions(); + auto schema = parseSchema( + R"json({"schema-id":0,"fields":[{"id":1,"name":"u","required":false,"type":"binary","clickhouse.type":"AggregateFunction(uniq, UInt64)"}]})json"); + auto context = contextWithAggregateFunctionStates(true); + IcebergSchemaProcessor processor(context); + processor.addIcebergTableSchema(schema, context); + + auto columns = processor.getClickhouseTableSchemaById(0); + ASSERT_EQ(columns->size(), 1u); + EXPECT_EQ(columns->front().name, "u"); + EXPECT_EQ(columns->front().type->getName(), "AggregateFunction(uniq, UInt64)"); +} + +/// Without the opt-in the same annotation must be refused, rather than fall back to the derived +/// `Nullable(String)`: reading the states as strings would be a wrong result, not an error. +TEST(IcebergSchemaProcessor, AnnotatedAggregateFunctionFieldIsRefusedWithoutTheSetting) +{ + tryRegisterAggregateFunctions(); + auto schema = parseSchema( + R"json({"schema-id":0,"fields":[{"id":1,"name":"u","required":false,"type":"binary","clickhouse.type":"AggregateFunction(uniq, UInt64)"}]})json"); + auto context = contextWithAggregateFunctionStates(false); + IcebergSchemaProcessor processor(context); + try + { + processor.addIcebergTableSchema(schema, context); + FAIL() << "The annotation names an aggregate function and must not be honoured without the setting"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::SUPPORT_IS_DISABLED) << e.message(); + } +} + +/// `SimpleAggregateFunction` stays ungated: the field holds ordinary values and no state +/// deserializer is involved. +TEST(IcebergSchemaProcessor, AnnotatedSimpleAggregateFunctionFieldNeedsNoSetting) +{ + tryRegisterAggregateFunctions(); + auto schema = parseSchema( + R"json({"schema-id":0,"fields":[{"id":1,"name":"s","required":true,"type":"long","clickhouse.type":"SimpleAggregateFunction(sum, Int64)"}]})json"); + auto context = contextWithAggregateFunctionStates(false); + IcebergSchemaProcessor processor(context); + processor.addIcebergTableSchema(schema, context); + + auto columns = processor.getClickhouseTableSchemaById(0); + ASSERT_EQ(columns->size(), 1u); + EXPECT_EQ(columns->front().name, "s"); + EXPECT_EQ(columns->front().type->getName(), "SimpleAggregateFunction(sum, Int64)"); +} + +/// The value in force is the one on the context of the call, not one frozen into the processor. A +/// table engine builds its processor at `ATTACH`, so only a per-call read lets a query opt in at all. +TEST(IcebergSchemaProcessor, AnnotatedAggregateFunctionFieldFollowsTheCallingContext) +{ + tryRegisterAggregateFunctions(); + IcebergSchemaProcessor processor(getContext().context); + + auto allowed_context = contextWithAggregateFunctionStates(true); + processor.addIcebergTableSchema( + parseSchema( + R"json({"schema-id":0,"fields":[{"id":1,"name":"u","required":false,"type":"binary","clickhouse.type":"AggregateFunction(uniq, UInt64)"}]})json"), + allowed_context); + auto columns = processor.getClickhouseTableSchemaById(0); + ASSERT_EQ(columns->size(), 1u); + EXPECT_EQ(columns->front().type->getName(), "AggregateFunction(uniq, UInt64)"); + + auto refused_context = contextWithAggregateFunctionStates(false); + try + { + processor.addIcebergTableSchema( + parseSchema( + R"json({"schema-id":1,"fields":[{"id":2,"name":"v","required":false,"type":"binary","clickhouse.type":"AggregateFunction(uniq, UInt64)"}]})json"), + refused_context); + FAIL() << "The setting is off on the context of this call and the annotation must be refused"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::SUPPORT_IS_DISABLED) << e.message(); + } +} + +/// A stale or hand-edited annotation must be rejected: a `long` field holds ordinary integers, not +/// serialized states. +TEST(IcebergSchemaProcessor, AnnotationNotMatchingTheFieldTypeIsRejected) +{ + tryRegisterAggregateFunctions(); + expectAnnotatedSchemaRejected(parseSchema( + R"json({"schema-id":0,"fields":[{"id":1,"name":"u","required":true,"type":"long","clickhouse.type":"AggregateFunction(uniq, UInt64)"}]})json")); +} + +/// `addIcebergTableSchema` publishes a schema-id only once all of its fields have parsed, and drops +/// whatever it wrote if parsing throws. Otherwise the next call for that schema-id would take the +/// "already added" branch and hand out a schema that was never built. The retries below reuse one +/// processor, as a table engine does - it builds its processor once, at `ATTACH`. + +/// The aggregate-state gate is the everyday way in: a read is refused, the user enables the setting, +/// and the retry runs against the processor the refused read left behind. +TEST(IcebergSchemaProcessor, RetryAfterRefusedAggregateFunctionStateSchemaSucceeds) +{ + tryRegisterAggregateFunctions(); + const std::string json + = R"json({"schema-id":0,"fields":[{"id":1,"name":"u","required":false,"type":"binary","clickhouse.type":"AggregateFunction(uniq, UInt64)"}]})json"; + IcebergSchemaProcessor processor(getContext().context); + + auto refused_context = contextWithAggregateFunctionStates(false); + try + { + processor.addIcebergTableSchema(parseSchema(json), refused_context); + FAIL() << "The annotation names an aggregate function and must be refused without the setting"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::SUPPORT_IS_DISABLED) << e.message(); + } + + /// Nothing of the refused schema may survive the throw. + EXPECT_FALSE(processor.hasClickhouseTableSchemaById(0)); + EXPECT_FALSE(processor.tryGetFieldCharacteristics(0, 1).has_value()); + EXPECT_FALSE(processor.tryGetColumnIDByName(0, "u").has_value()); + + auto allowed_context = contextWithAggregateFunctionStates(true); + ASSERT_NO_THROW(processor.addIcebergTableSchema(parseSchema(json), allowed_context)); + auto columns = processor.getClickhouseTableSchemaById(0); + ASSERT_EQ(columns->size(), 1u); + EXPECT_EQ(columns->front().name, "u"); + EXPECT_EQ(columns->front().type->getName(), "AggregateFunction(uniq, UInt64)"); +} + +/// The same invariant without an aggregate state: an Iceberg type string the parser rejects. The +/// first field does parse, so this also pins that the per-field characteristics recorded before the +/// throw are dropped rather than left to shadow the retry, which renames that field. +TEST(IcebergSchemaProcessor, RetryAfterUnparsableTypeUnderTheSameSchemaIdSucceeds) +{ + IcebergSchemaProcessor processor(getContext().context); + EXPECT_THROW( + processor.addIcebergTableSchema( + parseSchema( + R"json({"schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"decimal(20,)"}]})json"), + getContext().context), + DB::Exception); + + EXPECT_FALSE(processor.hasClickhouseTableSchemaById(0)); + EXPECT_FALSE(processor.tryGetFieldCharacteristics(0, 1).has_value()); + EXPECT_FALSE(processor.tryGetColumnIDByName(0, "a").has_value()); + + ASSERT_NO_THROW(processor.addIcebergTableSchema( + parseSchema( + R"json({"schema-id":0,"fields":[{"id":1,"name":"a2","required":false,"type":"int"},{"id":2,"name":"b","required":false,"type":"decimal(20,0)"}]})json"), + getContext().context)); + + auto columns = processor.getClickhouseTableSchemaById(0); + ASSERT_EQ(columns->size(), 2u); + EXPECT_EQ(columns->front().name, "a2"); + auto field = processor.getFieldCharacteristics(0, 1); + EXPECT_EQ(field.name, "a2"); + EXPECT_EQ(field.type->getName(), "Nullable(Int32)"); + EXPECT_FALSE(processor.tryGetColumnIDByName(0, "a").has_value()); +} + +#if USE_AVRO +/// `IcebergMetadata::backgroundMetadataPrefetcherThread` warms the metadata files cache on a timer, +/// with no query behind it, so it asks for the schema-id with `SchemaParsing::Skip`. That must neither +/// apply the gates `IcebergSchemaProcessor::getFieldType` reads from the query - a background task +/// carries none, so an `AggregateFunction` column would be refused on every period - nor publish a +/// parsed schema, which the processor would then serve to queries that never enabled the setting. +TEST(IcebergSchemaProcessor, SkippingTheSchemaNeitherAppliesTheGateNorPublishesTheSchema) +{ + tryRegisterAggregateFunctions(); + Poco::JSON::Parser parser; + auto metadata = parser + .parse( + R"json({"format-version":2,"current-schema-id":0,"schemas":[{"schema-id":0,)json" + R"json("fields":[{"id":1,"name":"u","required":false,"type":"binary",)json" + R"json("clickhouse.type":"AggregateFunction(uniq, UInt64)"}]}]})json") + .extract(); + + auto log = getLogger("IcebergSchemaProcessorTest"); + auto context = contextWithAggregateFunctionStates(false); + DB::Iceberg::IcebergSchemaProcessor processor(context); + + /// A query without the setting is refused, which is what the gate is for. + try + { + DB::IcebergMetadata::parseTableSchema( + metadata, processor, context, log, DB::IcebergMetadata::SchemaParsing::Parse); + FAIL() << "The annotation names an aggregate function and must be refused without the setting"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::SUPPORT_IS_DISABLED) << e.message(); + } + + /// The prefetcher, asking for the schema-id alone, is not. + Int32 schema_id = -1; + ASSERT_NO_THROW( + schema_id = DB::IcebergMetadata::parseTableSchema( + metadata, processor, context, log, DB::IcebergMetadata::SchemaParsing::Skip)); + EXPECT_EQ(schema_id, 0); + + /// And it published nothing, so the gate still decides for the queries that follow. + EXPECT_FALSE(processor.hasClickhouseTableSchemaById(0)); + EXPECT_FALSE(processor.tryGetFieldCharacteristics(0, 1).has_value()); + EXPECT_FALSE(processor.tryGetColumnIDByName(0, "u").has_value()); + + try + { + DB::IcebergMetadata::parseTableSchema( + metadata, processor, context, log, DB::IcebergMetadata::SchemaParsing::Parse); + FAIL() << "Skipping the schema must not have made the annotation acceptable without the setting"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::SUPPORT_IS_DISABLED) << e.message(); + } + + auto allowed_context = contextWithAggregateFunctionStates(true); + ASSERT_NO_THROW(DB::IcebergMetadata::parseTableSchema( + metadata, processor, allowed_context, log, DB::IcebergMetadata::SchemaParsing::Parse)); + auto columns = processor.getClickhouseTableSchemaById(0); + ASSERT_EQ(columns->size(), 1u); + EXPECT_EQ(columns->front().name, "u"); + EXPECT_EQ(columns->front().type->getName(), "AggregateFunction(uniq, UInt64)"); +} +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_type_mapping.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_type_mapping.cpp index 0b51a126abc8..fc239ce5f5cf 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_type_mapping.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_type_mapping.cpp @@ -5,6 +5,9 @@ #include #include +#include +#include +#include #include #include #include @@ -164,4 +167,73 @@ TEST(IcebergTypeMapping, NullableDecimalPrecisionAboveSpecLimitIsRejected) expectIcebergTypeRejected(makeNullable(createDecimal(76, 1))); } +TEST(IcebergTypeMapping, AggregateFunctionMapsToBinary) +{ + tryRegisterAggregateFunctions(); + auto type = DataTypeFactory::instance().get("AggregateFunction(uniq, UInt64)"); + Int32 iter = 0; + auto [iceberg_type, required] = getIcebergType(type, iter); + ASSERT_TRUE(iceberg_type.isString()); + EXPECT_EQ(iceberg_type.extract(), "binary"); + EXPECT_TRUE(required); +} + +TEST(IcebergTypeMapping, AggregateStatesNeedATypeAnnotation) +{ + tryRegisterAggregateFunctions(); + /// Both are indistinguishable from a plain binary/long column in the Iceberg schema, so the + /// ClickHouse type name has to be recorded alongside it. + EXPECT_TRUE(needsClickHouseTypeAnnotation(DataTypeFactory::instance().get("AggregateFunction(uniq, UInt64)"))); + EXPECT_TRUE(needsClickHouseTypeAnnotation(DataTypeFactory::instance().get("SimpleAggregateFunction(sum, UInt64)"))); + EXPECT_TRUE(needsClickHouseTypeAnnotation(DataTypeFactory::instance().get("Array(AggregateFunction(uniq, UInt64))"))); + EXPECT_TRUE( + needsClickHouseTypeAnnotation(DataTypeFactory::instance().get("SimpleAggregateFunction(anyLast, Nullable(String))"))); + + EXPECT_FALSE(needsClickHouseTypeAnnotation(std::make_shared())); + EXPECT_FALSE(needsClickHouseTypeAnnotation(DataTypeFactory::instance().get("Array(Nullable(String))"))); + EXPECT_FALSE(needsClickHouseTypeAnnotation(DataTypeFactory::instance().get("Tuple(a UInt32, b String)"))); +} + +TEST(IcebergTypeMapping, AnnotationNamePinsAggregateStateVersion) +{ + tryRegisterAggregateFunctions(); + + /// `sumMap` is versioned (default 1) and serializes version 0 differently. getName() drops the 0, + /// so the annotation has to spell it out or inference would rebuild the type with the default + /// version and misread the bytes. + auto v0 = DataTypeFactory::instance().get("AggregateFunction(0, sumMap, Array(UInt8), Array(UInt8))"); + EXPECT_EQ(v0->getName(), "AggregateFunction(sumMap, Array(UInt8), Array(UInt8))"); + EXPECT_EQ(getClickHouseTypeAnnotationName(v0), "AggregateFunction(0, sumMap, Array(UInt8), Array(UInt8))"); + /// The recorded name must round-trip back to the exact same version. + EXPECT_EQ( + assert_cast( + *DataTypeFactory::instance().get(getClickHouseTypeAnnotationName(v0))) + .getVersion(), + 0u); + + /// The version pinned inside a container is preserved too. + EXPECT_EQ( + getClickHouseTypeAnnotationName( + DataTypeFactory::instance().get("Array(AggregateFunction(0, sumMap, Array(UInt8), Array(UInt8)))")), + "Array(AggregateFunction(0, sumMap, Array(UInt8), Array(UInt8)))"); + + /// Every other case is identical to getName(): a versioned state keeps its explicit or default + /// version, and a non-versioned one (`groupBitmap`, `uniq`, `sum`) gains no spurious "0,". + for (const auto * name : + {"AggregateFunction(sumMap, Array(UInt8), Array(UInt8))", + "AggregateFunction(1, sumMap, Array(UInt8), Array(UInt8))", + "AggregateFunction(1, groupBitmap, UInt64)", + "AggregateFunction(groupBitmap, UInt64)", + "AggregateFunction(uniq, UInt64)", + "AggregateFunction(sum, UInt64)", + "Array(AggregateFunction(uniq, UInt64))", + "SimpleAggregateFunction(sum, UInt64)", + "SimpleAggregateFunction(anyLast, Nullable(String))", + "Tuple(a UInt32, b String)"}) + { + auto type = DataTypeFactory::instance().get(name); + EXPECT_EQ(getClickHouseTypeAnnotationName(type), type->getName()) << name; + } +} + #endif diff --git a/src/Storages/StorageReplicatedMergeTree.cpp b/src/Storages/StorageReplicatedMergeTree.cpp index 880dae5cd904..52dc05f540c5 100644 --- a/src/Storages/StorageReplicatedMergeTree.cpp +++ b/src/Storages/StorageReplicatedMergeTree.cpp @@ -249,6 +249,8 @@ namespace Setting extern const SettingsUInt64 iceberg_insert_max_bytes_in_data_file; extern const SettingsUInt64 iceberg_insert_max_rows_in_data_file; extern const SettingsTimezone iceberg_partition_timezone; + extern const SettingsBool allow_experimental_aggregate_function_states_in_parquet; + extern const SettingsBool allow_experimental_aggregate_function_states_in_iceberg; } @@ -8763,6 +8765,10 @@ void StorageReplicatedMergeTree::exportPartitionToTable(const PartitionCommand & manifest.filename_pattern = query_context->getSettingsRef()[Setting::export_merge_tree_part_filename_pattern].value; manifest.write_full_path_in_iceberg_metadata = query_context->getSettingsRef()[Setting::write_full_path_in_iceberg_metadata]; manifest.allow_lossy_cast = query_context->getSettingsRef()[Setting::export_merge_tree_part_allow_lossy_cast]; + manifest.allow_aggregate_function_states_in_parquet + = query_context->getSettingsRef()[Setting::allow_experimental_aggregate_function_states_in_parquet]; + manifest.allow_aggregate_function_states_in_iceberg + = query_context->getSettingsRef()[Setting::allow_experimental_aggregate_function_states_in_iceberg]; manifest.iceberg_partition_timezone = query_context->getSettingsRef()[Setting::iceberg_partition_timezone].toString(); manifest.schema_match_mode = query_context->getSettingsRef()[Setting::export_merge_tree_part_schema_match_mode].value; manifest.ignore_extra_source_columns = query_context->getSettingsRef()[Setting::export_merge_tree_part_ignore_extra_source_columns].value; diff --git a/tests/integration/helpers/export_partition_helpers.py b/tests/integration/helpers/export_partition_helpers.py index 04c9cb244757..5c54ed1f5078 100644 --- a/tests/integration/helpers/export_partition_helpers.py +++ b/tests/integration/helpers/export_partition_helpers.py @@ -158,12 +158,16 @@ def make_mt( columns, partition_by, order_by="tuple()", + engine="MergeTree()", ): - """Create a MergeTree table with block-number settings.""" + """Create a MergeTree table with block-number settings. + + *engine* allows a MergeTree variant, e.g. AggregatingMergeTree(). + """ node.query( f""" CREATE TABLE {name} ({columns}) - ENGINE = MergeTree() + ENGINE = {engine} PARTITION BY {partition_by} ORDER BY {order_by} SETTINGS {_BLOCK_SETTINGS} @@ -179,21 +183,26 @@ def make_iceberg_s3( url=None, s3_retry_attempts=3, if_not_exists=False, + extra_settings="", ): """Create an IcebergS3 table at a MinIO prefix. *url* defaults to ``http://minio1:9001/root/data/{name}/``. + *extra_settings* is appended to the SETTINGS clause. """ if url is None: url = f"http://minio1:9001/root/data/{name}/" ine = "IF NOT EXISTS " if if_not_exists else "" pclause = f"PARTITION BY {partition_by}" if partition_by else "" + settings = f"s3_retry_attempts = {s3_retry_attempts}" + if extra_settings: + settings += ", " + extra_settings node.query( f""" CREATE TABLE {ine}{name} ({columns}) ENGINE = IcebergS3('{url}', '{MINIO_USER}', '{MINIO_PASS}') {pclause} - SETTINGS s3_retry_attempts = {s3_retry_attempts} + SETTINGS {settings} """ ) diff --git a/tests/integration/test_export_merge_tree_part_to_iceberg/test.py b/tests/integration/test_export_merge_tree_part_to_iceberg/test.py index 053dec967036..c500b1fefb1e 100644 --- a/tests/integration/test_export_merge_tree_part_to_iceberg/test.py +++ b/tests/integration/test_export_merge_tree_part_to_iceberg/test.py @@ -16,6 +16,7 @@ test_export_part_multi_column_partition_key_success – composite (a, b, c) partition key round-trips test_export_part_partition_key_mismatch_variants_are_rejected (parametrized) – partition key column reordering, cardinality mismatches, and transform-expression reordering between src/dst are all rejected synchronously + test_export_part_aggregate_function_states – AggregateFunction / SimpleAggregateFunction states survive the export """ import logging @@ -1405,3 +1406,83 @@ def test_export_part_tuple_subcolumn_partition_key_iceberg_rejected(cluster): node.query(f"DROP TABLE IF EXISTS {mt} SYNC") node.query(f"DROP TABLE IF EXISTS {iceberg}") + + +def test_export_part_aggregate_function_states(cluster): + """ + Export a part holding pre-aggregated states into an Iceberg table. + + What this adds over test_storage_iceberg_with_spark/test_aggregate_function_states.py, which + reaches the same writer through a plain INSERT, is the export path's own pre-flight schema check: + verifyExportSchemaCastable() -> canBeSafelyCast() has to accept aggregate-state columns, or the + export is rejected synchronously and never scheduled. + """ + node = cluster.instances["node1"] + sfx = unique_suffix() + mt = f"mt_agg_states_{sfx}" + iceberg = f"iceberg_agg_states_{sfx}" + + # `k` is Int32 rather than UInt32 because Iceberg maps both to `int`, so a UInt32 source would + # additionally need export_merge_tree_part_allow_lossy_cast - a separate concern. + columns = ( + "k Int32, u AggregateFunction(uniq, UInt64), s SimpleAggregateFunction(sum, UInt64)" + ) + + make_mt(node, mt, columns, "k", order_by="k", engine="AggregatingMergeTree()") + # The Iceberg setting gates honouring the recorded `AggregateFunction` type as well as the DDL, + # and it is read from the query that parses the Iceberg schema. The CREATE below only needs it + # for the DDL; the export and the reads that follow pass it themselves. + make_iceberg_s3( + node, + iceberg, + columns, + "k", + extra_settings="allow_experimental_aggregate_function_states_in_iceberg = 1", + ) + + # A single INSERT leaves each partition with exactly one part, so no background merge can + # rewrite the states out from under the export. This one writes MergeTree, not Parquet, so it + # needs no Parquet setting. + node.query( + f"INSERT INTO {mt} " + f"SELECT toInt32(number % 2), uniqState(toUInt64(number % 23)), sumSimpleState(number) " + f"FROM numbers(200) GROUP BY number % 2" + ) + + for partition_id in ("0", "1"): + part = get_part(node, mt, partition_id) + # The exported data file is Parquet, written by a background task from the settings + # snapshotted off this ALTER query, so the Parquet gate has to be opened here. The export + # reads the destination's Iceberg schema, both in its pre-flight check and in that background + # task, so the Iceberg gate belongs here too. + export_part( + node, + mt, + part, + iceberg, + extra_settings=( + "allow_experimental_aggregate_function_states_in_parquet = 1, " + "allow_experimental_aggregate_function_states_in_iceberg = 1" + ), + ) + wait_for_export_part(node, mt, part) + assert_part_log(node, mt, part) + + exported = node.query( + f"SELECT k, uniqMerge(u), sum(s) FROM {iceberg} GROUP BY k ORDER BY k " + f"SETTINGS allow_experimental_aggregate_function_states_in_iceberg = 1" + ) + assert exported == node.query( + f"SELECT k, uniqMerge(u), sum(s) FROM {mt} GROUP BY k ORDER BY k" + ), f"Exported states do not match the source table:\n{exported}" + + total = node.query( + f"SELECT uniqMerge(u), sum(s) FROM {iceberg} " + f"SETTINGS allow_experimental_aggregate_function_states_in_iceberg = 1" + ) + assert total == node.query( + "SELECT uniq(toUInt64(number % 23)), sum(number) FROM numbers(200)" + ), f"Merging every exported state does not match the original rows:\n{total}" + + node.query(f"DROP TABLE IF EXISTS {mt} SYNC") + node.query(f"DROP TABLE IF EXISTS {iceberg}") diff --git a/tests/integration/test_export_replicated_mt_partition_to_iceberg/test.py b/tests/integration/test_export_replicated_mt_partition_to_iceberg/test.py index 4c746bc26607..dac13665be6b 100644 --- a/tests/integration/test_export_replicated_mt_partition_to_iceberg/test.py +++ b/tests/integration/test_export_replicated_mt_partition_to_iceberg/test.py @@ -3020,3 +3020,64 @@ def test_export_partition_multicolumn_identity_metadata_matches_data(cluster): f"WHERE event_date = '2024-03-05' AND retention = 30" ).strip()) assert filtered == 3, f"Partition-filtered read expected 3 rows, got {filtered}" + + +def test_export_partition_aggregate_function_states(cluster): + """ + Export a partition holding pre-aggregated states into an Iceberg table. + + Unlike EXPORT PART, whose background task is handed the full settings snapshot of the ALTER, + EXPORT PARTITION rebuilds the task settings on every replica from a fixed whitelist persisted in + the ZooKeeper manifest. Both aggregate-state gates are on that whitelist; without the Parquet one + the background writer refuses the AggregateFunction column with UNKNOWN_TYPE and the task never + reaches COMPLETED. + """ + node = cluster.instances["replica1"] + + uid = unique_suffix() + mt_table = f"mt_agg_states_{uid}" + iceberg_table = f"iceberg_agg_states_{uid}" + + # `k` is Int32 rather than UInt32 because Iceberg maps both to `int`, so a UInt32 source would + # additionally need export_merge_tree_part_allow_lossy_cast - a separate concern. + columns = ( + "k Int32, u AggregateFunction(uniq, UInt64), s SimpleAggregateFunction(sum, UInt64)" + ) + + make_rmt(node, mt_table, columns, "k", replica_name="replica1", order_by="k") + # The Iceberg setting gates the DDL as well as honouring the recorded `AggregateFunction` type. + make_iceberg_s3( + node, + iceberg_table, + columns, + partition_by="k", + extra_settings="allow_experimental_aggregate_function_states_in_iceberg = 1", + ) + + # One INSERT leaves each partition with a single part, so no merge can rewrite the states out + # from under the export. It writes MergeTree, not Parquet, so it needs no Parquet setting. + node.query( + f"INSERT INTO {mt_table} " + f"SELECT toInt32(number % 2), uniqState(toUInt64(number % 23)), sumSimpleState(number) " + f"FROM numbers(200) GROUP BY number % 2" + ) + + # Both gates are given here and nowhere else: the ALTER records them in the manifest, and the + # replica executing the task applies them in place of its own profile, in which both are off. + node.query( + f"ALTER TABLE {mt_table} EXPORT PARTITION ID '0' TO TABLE {iceberg_table}", + settings={ + "allow_insert_into_iceberg": 1, + "allow_experimental_aggregate_function_states_in_parquet": 1, + "allow_experimental_aggregate_function_states_in_iceberg": 1, + }, + ) + wait_for_export_status(node, mt_table, iceberg_table, "0", "COMPLETED") + + exported = node.query( + f"SELECT k, uniqMerge(u), sum(s) FROM {iceberg_table} GROUP BY k ORDER BY k " + f"SETTINGS allow_experimental_aggregate_function_states_in_iceberg = 1" + ) + assert exported == node.query( + f"SELECT k, uniqMerge(u), sum(s) FROM {mt_table} WHERE k = 0 GROUP BY k ORDER BY k" + ), f"Exported states do not match the source partition:\n{exported}" diff --git a/tests/integration/test_storage_iceberg_with_spark/test_aggregate_function_states.py b/tests/integration/test_storage_iceberg_with_spark/test_aggregate_function_states.py new file mode 100644 index 000000000000..12d49dac61ee --- /dev/null +++ b/tests/integration/test_storage_iceberg_with_spark/test_aggregate_function_states.py @@ -0,0 +1,266 @@ +import glob +import json +import os +import re + +import pytest + +from helpers.iceberg_utils import ( + create_iceberg_table, + default_download_directory, + get_creation_expression, + get_uuid_str, +) + +from helpers.test_tools import TSV + + +SETTING = "allow_experimental_aggregate_function_states_in_iceberg" + +STATE_SETTINGS = {SETTING: 1} +# Reading a column whose `clickhouse.type` names an `AggregateFunction` is gated by the same setting, +# because the table's metadata - not the query - picks the deserializer handed the state bytes. The +# setting is read from the query that parses the Iceberg schema, so every query that touches such a +# table carries it here: a SELECT, but an INSERT too. A table function parses the schema afresh each +# time, so for it the setting acts per query; a table engine parses it once and keeps the result for +# the lifetime of the storage object, so passing it everywhere keeps these tests independent of which +# query happens to parse the schema first. +WRITE_SETTINGS = {"allow_insert_into_iceberg": 1} +# Iceberg data files are Parquet, and writing an `AggregateFunction` state to Parquet is opt-in of +# its own; there is no internal override for the Iceberg path. An object storage table engine freezes +# its format settings at CREATE TABLE - global server settings plus the SETTINGS clause of that +# query, session settings deliberately ignored (registerStorageObjectStorage.cpp) - so the Parquet +# setting belongs in the CREATE, not on the INSERT. +PARQUET_STATE_SETTINGS = ["allow_experimental_aggregate_function_states_in_parquet = 1"] + +# `k` is Int32 and not UInt32 because Iceberg maps both to `int`. +SCHEMA = "(k Int32, u AggregateFunction(uniq, UInt64), s SimpleAggregateFunction(sum, UInt64))" + +# The same, with a signed storage type for the SimpleAggregateFunction: Iceberg's vectorized Arrow +# reader rejects the UINT_64 converted type ClickHouse writes for UInt64, which is a general +# interoperability gap for unsigned types, unrelated to aggregate states. +SPARK_SCHEMA = "(k Int32, u AggregateFunction(uniq, UInt64), s SimpleAggregateFunction(sum, Int64))" + +# Where the iceberg warehouse is mounted, both inside the ClickHouse container and - after +# default_download_directory() - on the host running the test and the Spark session. +WAREHOUSE = "/var/lib/clickhouse/user_files/iceberg_data/default" + + +def insert_states(instance, table_name, sum_expression="number"): + """Two groups of pre-aggregated states, the same shape as an AggregatingMergeTree part.""" + instance.query( + f""" + INSERT INTO {table_name} (k, u, s) + SELECT toInt32(number % 2), uniqState(toUInt64(number % 23)), sumSimpleState({sum_expression}) + FROM numbers(200) + GROUP BY number % 2 + """, + settings={**WRITE_SETTINGS, **STATE_SETTINGS}, + ) + + +def latest_metadata(table_name): + """The newest metadata JSON of a table already fetched by default_download_directory().""" + files = glob.glob(os.path.join(WAREHOUSE, table_name, "metadata", "v*.metadata.json")) + assert files, f"No metadata JSON downloaded for {table_name}" + # Sort by version rather than by name, so that v10 does not come before v2. + newest = max(files, key=lambda path: int(re.fullmatch(r"v(\d+)\.metadata\.json", os.path.basename(path)).group(1))) + with open(newest) as f: + return json.load(f) + + +def fields_by_name(metadata): + schema_id = metadata["current-schema-id"] + for schema in metadata["schemas"]: + if schema["schema-id"] == schema_id: + return {field["name"]: field for field in schema["fields"]} + raise AssertionError(f"Schema {schema_id} not found in {metadata['schemas']}") + + +# Every code path these tests exercise sits above the object storage layer, so one storage type is +# enough; the per-storage write paths are covered by the other tests in this directory. +@pytest.mark.parametrize("storage_type", ["s3"]) +def test_aggregate_states_require_setting(started_cluster_iceberg_with_spark, storage_type): + """Every DDL path that can introduce an aggregate state into an Iceberg table is gated.""" + instance = started_cluster_iceberg_with_spark.instances["node1"] + TABLE_NAME = "test_agg_states_setting_" + storage_type + "_" + get_uuid_str() + + creation_expression = get_creation_expression( + storage_type, TABLE_NAME, started_cluster_iceberg_with_spark, SCHEMA, format_version=2 + ) + error = instance.query_and_get_error(creation_expression) + assert SETTING in error, f"CREATE TABLE error does not name the setting: {error}" + + create_iceberg_table( + storage_type, + instance, + TABLE_NAME, + started_cluster_iceberg_with_spark, + SCHEMA, + format_version=2, + settings=STATE_SETTINGS, + additional_settings=PARQUET_STATE_SETTINGS, + ) + + # A plain table plus ADD COLUMN would otherwise bypass the CREATE TABLE check. Iceberg only + # allows adding optional columns, hence the Nullable state type. + error = instance.query_and_get_error( + f"ALTER TABLE {TABLE_NAME} ADD COLUMN extra SimpleAggregateFunction(anyLast, Nullable(String))", + settings=WRITE_SETTINGS, + ) + assert SETTING in error, f"ADD COLUMN error does not name the setting: {error}" + + error = instance.query_and_get_error( + f"ALTER TABLE {TABLE_NAME} MODIFY COLUMN k SimpleAggregateFunction(anyLast, Nullable(String))", + settings=WRITE_SETTINGS, + ) + assert SETTING in error, f"MODIFY COLUMN error does not name the setting: {error}" + + instance.query( + f"ALTER TABLE {TABLE_NAME} ADD COLUMN extra SimpleAggregateFunction(anyLast, Nullable(String))", + settings={**WRITE_SETTINGS, **STATE_SETTINGS}, + ) + + # Reading the recorded `AggregateFunction` type needs the setting on the query, the same way the + # INSERT inside insert_states() does; the Parquet gate the INSERT also passes through is opened by + # the CREATE above. + insert_states(instance, TABLE_NAME) + assert instance.query(f"SELECT count() FROM {TABLE_NAME}", settings=STATE_SETTINGS) == "2\n" + + # Without it the recorded type is refused rather than read as `String`. The refusal is asserted + # through the table function, which parses the schema afresh on every query; the table engine + # above has already parsed it and keeps the parsed schema for the lifetime of its storage object. + table_function_expr = get_creation_expression( + storage_type, + TABLE_NAME, + started_cluster_iceberg_with_spark, + table_function=True, + ) + error = instance.query_and_get_error(f"SELECT count() FROM {table_function_expr}") + assert SETTING in error, f"Read error does not name the setting: {error}" + + assert ( + instance.query(f"SELECT count() FROM {table_function_expr}", settings=STATE_SETTINGS) + == "2\n" + ) + + +@pytest.mark.parametrize("storage_type", ["s3"]) +def test_aggregate_states_round_trip(started_cluster_iceberg_with_spark, storage_type): + """States written by ClickHouse merge back to exactly what the source data produces.""" + instance = started_cluster_iceberg_with_spark.instances["node1"] + TABLE_NAME = "test_agg_states_round_trip_" + storage_type + "_" + get_uuid_str() + + create_iceberg_table( + storage_type, + instance, + TABLE_NAME, + started_cluster_iceberg_with_spark, + SCHEMA, + format_version=2, + partition_by="k", + settings=STATE_SETTINGS, + additional_settings=PARQUET_STATE_SETTINGS, + ) + insert_states(instance, TABLE_NAME) + + # The table function reads the schema out of the Iceberg metadata rather than out of the + # CREATE TABLE definition, so this is what proves the annotation round-tripped. + table_function_expr = get_creation_expression( + storage_type, + TABLE_NAME, + started_cluster_iceberg_with_spark, + table_function=True, + ) + assert instance.query( + f"DESCRIBE {table_function_expr} FORMAT TSV", + settings={"print_pretty_type_names": 0, **STATE_SETTINGS}, + ) == TSV( + [ + ["k", "Int32"], + ["u", "AggregateFunction(uniq, UInt64)"], + ["s", "SimpleAggregateFunction(sum, UInt64)"], + ] + ) + + assert instance.query( + f"SELECT k, uniqMerge(u), sum(s) FROM {TABLE_NAME} GROUP BY k ORDER BY k", + settings=STATE_SETTINGS, + ) == instance.query( + "SELECT toInt32(number % 2) AS k, uniq(toUInt64(number % 23)), sum(number)" + " FROM numbers(200) GROUP BY k ORDER BY k" + ) + + assert instance.query( + f"SELECT uniqMerge(u), sum(s) FROM {TABLE_NAME}", settings=STATE_SETTINGS + ) == instance.query( + "SELECT uniq(toUInt64(number % 23)), sum(number) FROM numbers(200)" + ) + + # Min/max bounds over serialized states are meaningless, so none are recorded for `u`: a + # filter over the state column must still see every row rather than prune some away. + assert instance.query( + f"SELECT count() FROM {TABLE_NAME} WHERE finalizeAggregation(u) > 0", + settings=STATE_SETTINGS, + ) == "2\n" + + default_download_directory( + started_cluster_iceberg_with_spark, + storage_type, + f"{WAREHOUSE}/{TABLE_NAME}/", + f"{WAREHOUSE}/{TABLE_NAME}/", + ) + + fields = fields_by_name(latest_metadata(TABLE_NAME)) + assert fields["k"]["type"] == "int" + assert "clickhouse.type" not in fields["k"] + assert fields["u"]["type"] == "binary" + assert fields["u"]["clickhouse.type"] == "AggregateFunction(uniq, UInt64)" + assert fields["s"]["type"] == "long" + assert fields["s"]["clickhouse.type"] == "SimpleAggregateFunction(sum, UInt64)" + + +@pytest.mark.parametrize("storage_type", ["s3"]) +def test_aggregate_states_read_by_spark(started_cluster_iceberg_with_spark, storage_type): + """Spark ignores the `clickhouse.type` key and sees plain binary / bigint columns.""" + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + TABLE_NAME = "test_agg_states_spark_" + storage_type + "_" + get_uuid_str() + + create_iceberg_table( + storage_type, + instance, + TABLE_NAME, + started_cluster_iceberg_with_spark, + SPARK_SCHEMA, + format_version=2, + settings=STATE_SETTINGS, + additional_settings=PARQUET_STATE_SETTINGS, + ) + insert_states(instance, TABLE_NAME, "toInt64(number)") + + default_download_directory( + started_cluster_iceberg_with_spark, + storage_type, + f"{WAREHOUSE}/{TABLE_NAME}/", + f"{WAREHOUSE}/{TABLE_NAME}/", + ) + + # CREATE TABLE wrote v0, the single INSERT wrote v1. + with open(f"{WAREHOUSE}/{TABLE_NAME}/metadata/version-hint.text", "wb") as f: + f.write(b"1") + + df = spark.read.format("iceberg").load(f"{WAREHOUSE}/{TABLE_NAME}") + assert dict(df.dtypes) == {"k": "int", "u": "binary", "s": "bigint"} + + spark_rows = {row["k"]: (bytes(row["u"]).hex().upper(), row["s"]) for row in df.collect()} + assert len(spark_rows) == 2 + + # Spark sees byte for byte the states ClickHouse wrote, with no reinterpretation. + expected = instance.query( + f"SELECT k, hex(u), s FROM {TABLE_NAME} ORDER BY k FORMAT TSV", settings=STATE_SETTINGS + ).strip() + assert expected != "" + for line in expected.split("\n"): + k, state_hex, sum_value = line.split("\t") + assert spark_rows[int(k)] == (state_hex, int(sum_value)) diff --git a/tests/queries/0_stateless/02902_topKGeneric_deserialization_memory.sql b/tests/queries/0_stateless/02902_topKGeneric_deserialization_memory.sql index 3228810e0baa..ca97d74199fb 100644 --- a/tests/queries/0_stateless/02902_topKGeneric_deserialization_memory.sql +++ b/tests/queries/0_stateless/02902_topKGeneric_deserialization_memory.sql @@ -1,9 +1,13 @@ --- Tags: no-fasttest - -- https://github.com/ClickHouse/ClickHouse/issues/49706 --- Using format Parquet for convenience so it errors out without output (but still deserializes the output) --- Without the fix this would OOM the client when deserializing the state +-- The code under test runs in the client: it receives the `AggregateFunction` column over the +-- native protocol and deserializes the state. Reading a `topKResample` state must allocate only +-- what the state actually holds, never the `reserved` capacity implied by the parameters of +-- `topKResample` - that would be ~3M counters for each of the 6528 resample buckets, and the +-- client would run out of memory. +-- The output is dumped to `/dev/null` because the state itself is binary and uninteresting here. +-- `FORMAT Null` cannot be used instead: for it the server does not send the data to the client +-- at all (`null_format` in `executeQuery`), so nothing would be deserialized. SELECT topKResampleState(1048576, 257, 65536, 10)(toString(number), number) FROM numbers(3) -FORMAT Parquet; -- { clientError UNKNOWN_TYPE } +INTO OUTFILE '/dev/null' TRUNCATE FORMAT RowBinary; diff --git a/tests/queries/0_stateless/04673_parquet_aggregate_function_state.reference b/tests/queries/0_stateless/04673_parquet_aggregate_function_state.reference new file mode 100644 index 000000000000..4944df02dbde --- /dev/null +++ b/tests/queries/0_stateless/04673_parquet_aggregate_function_state.reference @@ -0,0 +1,92 @@ +-- without the setting writing a state is refused, as it was before Parquet supported states +1 +-- a state nested in an Array is refused just the same, the recursion reaching the leaf first +1 +-- a SimpleAggregateFunction is an ordinary value of its storage type, so it needs no setting +s SimpleAggregateFunction(sum, UInt64) +-- without the setting the recorded state type is refused, not silently read as String +1 +-- schema inference recovers the aggregate types from the file +k UInt8 +u AggregateFunction(uniq, UInt8) +s SimpleAggregateFunction(sum, UInt64) +-- states round-trip: merged per group +0 17 1683 +1 17 1617 +2 17 1650 +-- merging across all groups matches the source too +1 1 +-- an explicit structure overrides the recorded type - and needs no setting, being the query's own +17 +17 +-- a column chunk holding both a dictionary page and a plain page +['PLAIN','RLE_DICTIONARY'] +500 1 +-- a state nested in an Array +a Array(AggregateFunction(uniq, UInt64)) +[10,10] +-- a state pinned to a non-default version keeps that version +AggregateFunction(0, sumMap, Array(UInt8), Array(UInt32)) +1 +-- a refused state column can be skipped instead of refusing the whole file +k UInt8 +s SimpleAggregateFunction(sum, UInt64) +3 +-- an annotation naming an aggregate function this server does not have breaks only its column +1 +k UInt8 +s SimpleAggregateFunction(sum, UInt64) +3 +-- every annotated type this writer can produce still reads back, values intact +num SimpleAggregateFunction(sum, UInt64) +str SimpleAggregateFunction(anyLast, String) +nullable SimpleAggregateFunction(anyLast, Nullable(String)) +low_cardinality SimpleAggregateFunction(anyLast, LowCardinality(String)) +arr SimpleAggregateFunction(anyLast, Array(UInt64)) +m SimpleAggregateFunction(anyLast, Map(String, UInt64)) +d SimpleAggregateFunction(anyLast, Date) +dt SimpleAggregateFunction(anyLast, DateTime(\'UTC\')) +dt64 SimpleAggregateFunction(anyLast, DateTime64(4, \'UTC\')) +e SimpleAggregateFunction(anyLast, Enum8(\'a\' = 1, \'b\' = 2)) +ip4 SimpleAggregateFunction(anyLast, IPv4) +ip6 SimpleAggregateFunction(anyLast, IPv6) +i128 SimpleAggregateFunction(anyLast, Int128) +fs SimpleAggregateFunction(anyLast, FixedString(16)) +uu SimpleAggregateFunction(anyLast, UUID) +dec SimpleAggregateFunction(anyLast, Decimal(18, 4)) +state AggregateFunction(uniq, UInt64) +tup Tuple(AggregateFunction(uniq, UInt64), UInt64) +Row 1: +────── +num: 3 +str: abc +nullable: xyz +low_cardinality: lc +arr: [1,2] +m: {'k':7} +d: 2020-01-02 +dt: 2020-01-02 03:04:05 +dt64: 2020-01-02 03:04:05.1234 +e: b +ip4: 1.2.3.4 +ip6: ::1 +i128: 170141183460469231731687303715884105727 +hex(fs): 66697865640000000000000000000000 +uu: 00000000-0000-0000-0000-000000000001 +dec: 1.25 +uniq_state: 3 +uniq_in_tuple: 3 +plain_in_tuple: 7 +-- the same file reads back with the nullability the reader is told to infer +1 +1 +-- a recorded type that re-reads the stored bytes as something else is refused +1 +3 +-- a schema inferred with the setting on is not served to a query that has it off +k UInt8 +u AggregateFunction(uniq, UInt8) +1 +-- nor is a schema inferred with the refused column skipped served to a query that is not skipping +k UInt8 +1 diff --git a/tests/queries/0_stateless/04673_parquet_aggregate_function_state.sh b/tests/queries/0_stateless/04673_parquet_aggregate_function_state.sh new file mode 100755 index 000000000000..1f9651bfbd93 --- /dev/null +++ b/tests/queries/0_stateless/04673_parquet_aggregate_function_state.sh @@ -0,0 +1,287 @@ +#!/usr/bin/env bash +# Tags: no-fasttest + +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. "$CUR_DIR"/../shell_config.sh + +SUFFIX="${CLICKHOUSE_DATABASE}_${RANDOM}" +FILE="agg_state_${SUFFIX}.parquet" +NESTED="agg_state_nested_${SUFFIX}.parquet" +MIXED="agg_state_mixed_${SUFFIX}.parquet" +VERSIONED="agg_state_versioned_${SUFFIX}.parquet" +REFUSED="agg_state_refused_${SUFFIX}.parquet" +REFUSED_NESTED="agg_state_refused_nested_${SUFFIX}.parquet" +SIMPLE="agg_state_simple_${SUFFIX}.parquet" +UNKNOWN="agg_state_unknown_${SUFFIX}.parquet" +MATRIX="agg_state_matrix_${SUFFIX}.parquet" +RETYPED_SOURCE="agg_state_retyped_source_${SUFFIX}.parquet" +RETYPED="agg_state_retyped_${SUFFIX}.parquet" +CACHED="agg_state_cached_${SUFFIX}.parquet" +CACHED_SKIP="agg_state_cached_skip_${SUFFIX}.parquet" + +cleanup() +{ + rm -f "${USER_FILES_PATH}/${FILE}" "${USER_FILES_PATH}/${NESTED}" "${USER_FILES_PATH}/${MIXED}" \ + "${USER_FILES_PATH}/${VERSIONED}" "${USER_FILES_PATH}/${REFUSED}" \ + "${USER_FILES_PATH}/${REFUSED_NESTED}" "${USER_FILES_PATH}/${SIMPLE}" \ + "${USER_FILES_PATH}/${UNKNOWN}" "${USER_FILES_PATH}/${MATRIX}" \ + "${USER_FILES_PATH}/${RETYPED_SOURCE}" "${USER_FILES_PATH}/${RETYPED}" \ + "${USER_FILES_PATH}/${CACHED}" "${USER_FILES_PATH}/${CACHED_SKIP}" +} +trap cleanup EXIT + +# The serialized states, and the type names recorded next to them, are a ClickHouse-only convention, +# and an `AggregateFunction` state is deserialized by whichever aggregate function the file's own +# metadata names, so both writing states and inferring them from a file are opt-in. +STATES="--allow_experimental_aggregate_function_states_in_parquet=1" +# A recorded type this server will not honour concerns one column, so it is skippable like a column +# of any other unsupported type. +SKIP="--input_format_parquet_skip_columns_with_unsupported_types_in_schema_inference=1" +# Several blocks below infer the same file again under a different value of the settings above. Those +# settings are part of the schema cache key, so the schema would be inferred afresh anyway; disabling +# the cache keeps those blocks independent of cache behaviour altogether. The two blocks at the end, +# which are about the cache key itself, deliberately do not use this. +NO_CACHE="--schema_inference_use_cache_for_file=0" + +echo '-- without the setting writing a state is refused, as it was before Parquet supported states' +if ${CLICKHOUSE_CLIENT} --query " + INSERT INTO FUNCTION file('${REFUSED}', Parquet) SELECT uniqState(number) AS u FROM numbers(10) +" 2>&1 | grep -q "allow_experimental_aggregate_function_states_in_parquet" +then + echo 1 +fi + +echo '-- a state nested in an Array is refused just the same, the recursion reaching the leaf first' +if ${CLICKHOUSE_CLIENT} --query " + INSERT INTO FUNCTION file('${REFUSED_NESTED}', Parquet) SELECT [uniqState(number)] AS a FROM numbers(10) +" 2>&1 | grep -q "allow_experimental_aggregate_function_states_in_parquet" +then + echo 1 +fi + +echo '-- a SimpleAggregateFunction is an ordinary value of its storage type, so it needs no setting' +${CLICKHOUSE_CLIENT} --query " + INSERT INTO FUNCTION file('${SIMPLE}', Parquet) SELECT sumSimpleState(number) AS s FROM numbers(10) +" +${CLICKHOUSE_CLIENT} --query "DESC file('${SIMPLE}', Parquet)" + +${CLICKHOUSE_CLIENT} ${STATES} --query " + INSERT INTO FUNCTION file('${FILE}', Parquet) + SELECT + number % 3 AS k, + uniqState(toUInt8(number % 17)) AS u, + sumSimpleState(number) AS s + FROM numbers(100) + GROUP BY k +" + +echo '-- without the setting the recorded state type is refused, not silently read as String' +# Whether the server forwards its own log record for the exception depends on the logger +# configuration, so the setting name can be printed more than once - assert that it is named at all +# instead of how many lines name it. +if ${CLICKHOUSE_CLIENT} --query "DESC file('${FILE}', Parquet)" 2>&1 \ + | grep -q "allow_experimental_aggregate_function_states_in_parquet" +then + echo 1 +fi + +echo '-- schema inference recovers the aggregate types from the file' +${CLICKHOUSE_CLIENT} ${STATES} --query "DESC file('${FILE}', Parquet)" + +echo '-- states round-trip: merged per group' +${CLICKHOUSE_CLIENT} ${STATES} --query " + SELECT k, uniqMerge(u), sum(s) FROM file('${FILE}', Parquet) GROUP BY k ORDER BY k +" + +echo '-- merging across all groups matches the source too' +${CLICKHOUSE_CLIENT} ${STATES} --query " + SELECT uniqMerge(u) = (SELECT uniq(toUInt8(number % 17)) FROM numbers(100)) AS uniq_matches, + sum(s) = (SELECT sum(number) FROM numbers(100)) AS sum_matches + FROM file('${FILE}', Parquet) +" + +echo '-- an explicit structure overrides the recorded type - and needs no setting, being the query'\''s own' +${CLICKHOUSE_CLIENT} --query " + SELECT uniqMerge(CAST(u AS AggregateFunction(uniq, UInt8))) + FROM file('${FILE}', Parquet, 'u String') +" +${CLICKHOUSE_CLIENT} --query " + SELECT uniqMerge(u) FROM file('${FILE}', Parquet, 'u AggregateFunction(uniq, UInt8)') +" + +echo '-- a column chunk holding both a dictionary page and a plain page' +# A dictionary budget that fits only some of the states makes the writer emit a dictionary page and +# then fall back to plain within the same column chunk, so the reader mixes Dictionary::index(), +# which shares state ownership via ColumnAggregateFunction::src, with the state converter, which +# allocates its own states. The encodings are asserted so this keeps covering the mixed case. +${CLICKHOUSE_CLIENT} ${STATES} --query " + INSERT INTO FUNCTION file('${MIXED}', Parquet) + SELECT uniqState(number) AS u FROM numbers(2000) + GROUP BY number % 500 + SETTINGS output_format_parquet_max_dictionary_size = 20000, output_format_parquet_row_group_size = 100000 +" +${CLICKHOUSE_CLIENT} --query " + SELECT arraySort(tupleElement(arrayJoin(columns), 'encodings')) FROM file('${MIXED}', ParquetMetadata) +" +${CLICKHOUSE_CLIENT} ${STATES} --query " + SELECT count(), uniqMerge(u) = (SELECT uniq(number) FROM numbers(2000)) AS matches + FROM file('${MIXED}', Parquet) +" + +echo '-- a state nested in an Array' +${CLICKHOUSE_CLIENT} ${STATES} --query " + INSERT INTO FUNCTION file('${NESTED}', Parquet) + SELECT [uniqState(number), uniqState(number + 100)] AS a FROM numbers(10) +" +${CLICKHOUSE_CLIENT} ${STATES} --query "DESC file('${NESTED}', Parquet)" +${CLICKHOUSE_CLIENT} ${STATES} --query "SELECT arrayMap(x -> finalizeAggregation(x), a) FROM file('${NESTED}', Parquet)" + +echo '-- a state pinned to a non-default version keeps that version' +# `sumMap` is versioned, and version 0 serializes the values as they are while version 1 promotes +# them to a wider type, so reading a version-0 state as the default version 1 fails outright. +# getName() drops the pinned 0, so the annotation is the only place that can carry it. +${CLICKHOUSE_CLIENT} ${STATES} --query " + INSERT INTO FUNCTION file('${VERSIONED}', Parquet) + SELECT CAST(sumMapState([number % 3], [toUInt32(number)]) AS AggregateFunction(0, sumMap, Array(UInt8), Array(UInt32))) AS m + FROM numbers(100) +" +grep -ao 'AggregateFunction(0, sumMap[^"]*' "${USER_FILES_PATH}/${VERSIONED}" +${CLICKHOUSE_CLIENT} ${STATES} --query " + SELECT sumMapMerge(m) = (SELECT sumMap([number % 3], [toUInt32(number)]) FROM numbers(100)) AS matches + FROM file('${VERSIONED}', Parquet) +" + +echo '-- a refused state column can be skipped instead of refusing the whole file' +# The schema is inferred as a whole, before the query prunes the columns it does not read, so by +# default one refused annotation still refuses a query that never touches that column. Skipping the +# column is the way out that does not enable the state deserializer: the rest of the file reads. +${CLICKHOUSE_CLIENT} ${SKIP} ${NO_CACHE} --query "DESC file('${FILE}', Parquet)" +${CLICKHOUSE_CLIENT} ${SKIP} ${NO_CACHE} --query "SELECT sum(k) FROM file('${FILE}', Parquet)" + +echo '-- an annotation naming an aggregate function this server does not have breaks only its column' +# Such a file is what a newer ClickHouse, or a corrupted footer, produces. It is made here by +# overwriting the recorded function name in place with one no build has - the replacement is the same +# length, so every offset in the file, the thrift footer included, stays valid. +python3 -c " +import sys +data = open(sys.argv[1], 'rb').read() +open(sys.argv[2], 'wb').write(data.replace(b'AggregateFunction(uniq,', b'AggregateFunction(zzzz,')) +" "${USER_FILES_PATH}/${FILE}" "${USER_FILES_PATH}/${UNKNOWN}" +if ${CLICKHOUSE_CLIENT} ${STATES} ${NO_CACHE} --query "DESC file('${UNKNOWN}', Parquet)" 2>&1 \ + | grep -q "for column u" +then + echo 1 +fi +${CLICKHOUSE_CLIENT} ${STATES} ${SKIP} ${NO_CACHE} --query "DESC file('${UNKNOWN}', Parquet)" +${CLICKHOUSE_CLIENT} ${STATES} ${SKIP} ${NO_CACHE} --query "SELECT sum(k) FROM file('${UNKNOWN}', Parquet)" + +echo '-- every annotated type this writer can produce still reads back, values intact' +# The recorded type is matched strictly against the parquet schema, and parquet does not round-trip +# every ClickHouse type: `Date` becomes a DATE that reads back as `Date32`, `DateTime` a +# TIMESTAMP_MILLIS that reads back as `DateTime64(3)`, `Enum8` an ENUM byte array that reads back as +# `String`, `IPv4` a plain UINT_32, `IPv6` and `Int128` untyped fixed byte arrays, `LowCardinality(T)` a +# plain `T`, and any leaf may gain or lose a `Nullable`. So the matching has to allow exactly the +# mappings the writer performs - which is what this block pins down, by going through the real writer. +${CLICKHOUSE_CLIENT} ${STATES} --query " + INSERT INTO FUNCTION file('${MATRIX}', Parquet) + SELECT + sumSimpleState(number) AS num, + anyLastSimpleState('abc') AS str, + anyLastSimpleState(CAST('xyz', 'Nullable(String)')) AS nullable, + CAST(anyLastSimpleState(toLowCardinality('lc')), + 'SimpleAggregateFunction(anyLast, LowCardinality(String))') AS low_cardinality, + anyLastSimpleState([toUInt64(1), toUInt64(2)]) AS arr, + anyLastSimpleState(map('k', toUInt64(7))) AS m, + anyLastSimpleState(toDate('2020-01-02')) AS d, + anyLastSimpleState(toDateTime('2020-01-02 03:04:05', 'UTC')) AS dt, + anyLastSimpleState(toDateTime64('2020-01-02 03:04:05.1234', 4, 'UTC')) AS dt64, + anyLastSimpleState(CAST('b', 'Enum8(''a'' = 1, ''b'' = 2)')) AS e, + anyLastSimpleState(toIPv4('1.2.3.4')) AS ip4, + anyLastSimpleState(toIPv6('::1')) AS ip6, + anyLastSimpleState(toInt128('170141183460469231731687303715884105727')) AS i128, + anyLastSimpleState(CAST('fixed', 'FixedString(16)')) AS fs, + anyLastSimpleState(toUUID('00000000-0000-0000-0000-000000000001')) AS uu, + anyLastSimpleState(toDecimal64('1.25', 4)) AS dec, + uniqState(number) AS state, + tuple(uniqState(number), toUInt64(7)) AS tup + FROM numbers(3) +" +${CLICKHOUSE_CLIENT} ${STATES} --query "DESC file('${MATRIX}', Parquet)" +${CLICKHOUSE_CLIENT} ${STATES} --query " + SELECT num, str, nullable, low_cardinality, arr, m, d, dt, dt64, e, ip4, ip6, toString(i128) AS i128, hex(fs), uu, dec, + finalizeAggregation(state) AS uniq_state, finalizeAggregation(tup.1) AS uniq_in_tuple, tup.2 AS plain_in_tuple + FROM file('${MATRIX}', Parquet) + FORMAT Vertical +" + +echo '-- the same file reads back with the nullability the reader is told to infer' +# `schema_inference_make_columns_nullable` decides whether a leaf derives as Nullable, and the +# recorded type says nothing about it, so neither setting may refuse the file. +${CLICKHOUSE_CLIENT} ${STATES} ${NO_CACHE} --schema_inference_make_columns_nullable=1 --query " + SELECT count() FROM file('${MATRIX}', Parquet) +" +${CLICKHOUSE_CLIENT} ${STATES} ${NO_CACHE} --schema_inference_make_columns_nullable=0 --query " + SELECT count() FROM file('${MATRIX}', Parquet) +" + +echo '-- a recorded type that re-reads the stored bytes as something else is refused' +# `SimpleAggregateFunction` is honoured without a setting, so a stale or hand-crafted annotation is +# the one thing standing between an INT64 column and being read as nanosecond timestamps - castColumn +# converts the integers happily and silently, and the values a SELECT returns are then not the values +# in the file. The recorded name is overwritten in place with one of exactly the same length, so every +# offset in the file, the thrift footer included, stays valid. +${CLICKHOUSE_CLIENT} --query " + INSERT INTO FUNCTION file('${RETYPED_SOURCE}', Parquet) + SELECT toUInt8(number) AS k, sumWithOverflowSimpleState(toInt64(number)) AS v FROM numbers(3) GROUP BY k +" +python3 -c " +import sys +old = b'SimpleAggregateFunction(sumWithOverflow, Int64)' +new = b'SimpleAggregateFunction(anyLast, DateTime64(9))' +assert len(old) == len(new) +data = open(sys.argv[1], 'rb').read() +assert old in data +open(sys.argv[2], 'wb').write(data.replace(old, new)) +" "${USER_FILES_PATH}/${RETYPED_SOURCE}" "${USER_FILES_PATH}/${RETYPED}" +if ${CLICKHOUSE_CLIENT} ${NO_CACHE} --query "DESC file('${RETYPED}', Parquet)" 2>&1 \ + | grep -q "reads as Int64" +then + echo 1 +fi +${CLICKHOUSE_CLIENT} ${SKIP} ${NO_CACHE} --query "SELECT sum(k) FROM file('${RETYPED}', Parquet)" + +echo '-- a schema inferred with the setting on is not served to a query that has it off' +# The inferred schema outlives the query that filled the cache, so the cache key has to name the +# settings that pick the schema. Otherwise a single query with the gate open would hand every later +# query, of any user, a schema carrying `AggregateFunction`, running the state deserializer the file +# names with the gate closed. Unlike the blocks above this one needs the cache, so it must not +# disable it, and it uses a file no earlier block has inferred under other settings. +${CLICKHOUSE_CLIENT} ${STATES} --query " + INSERT INTO FUNCTION file('${CACHED}', Parquet) + SELECT toUInt8(number % 3) AS k, uniqState(toUInt8(number)) AS u FROM numbers(30) GROUP BY k +" +# An entry is invalidated when the file is at least as new as the entry itself, and both timestamps +# have one-second granularity, so a file written and inferred within the same second is never served +# from the cache. Dating the file back makes the entry usable without waiting for a second to pass. +touch -t 200001010000 "${USER_FILES_PATH}/${CACHED}" +${CLICKHOUSE_CLIENT} ${STATES} --query "DESC file('${CACHED}', Parquet)" +if ${CLICKHOUSE_CLIENT} --query "DESC file('${CACHED}', Parquet)" 2>&1 \ + | grep -q "allow_experimental_aggregate_function_states_in_parquet" +then + echo 1 +fi + +echo '-- nor is a schema inferred with the refused column skipped served to a query that is not skipping' +# The skip setting picks the schema too: with it the refused column is dropped from the schema, and +# without it inference throws instead, so neither answer may be cached for the other. +${CLICKHOUSE_CLIENT} ${STATES} --query " + INSERT INTO FUNCTION file('${CACHED_SKIP}', Parquet) + SELECT toUInt8(number % 3) AS k, uniqState(toUInt8(number)) AS u FROM numbers(30) GROUP BY k +" +touch -t 200001010000 "${USER_FILES_PATH}/${CACHED_SKIP}" +${CLICKHOUSE_CLIENT} ${SKIP} --query "DESC file('${CACHED_SKIP}', Parquet)" +if ${CLICKHOUSE_CLIENT} --query "DESC file('${CACHED_SKIP}', Parquet)" 2>&1 \ + | grep -q "allow_experimental_aggregate_function_states_in_parquet" +then + echo 1 +fi