[yyanyy]

Adds a v4 vector(E, D) primitive type for fixed-length dense numeric vectors. Non-null vectors contain exactly D non-null finite elements of the allowed type E. Avro and ORC map vectors to arrays with reader and writer enforcement, while Parquet uses VECTOR(D) following apache/parquet-format#624. Schema JSON uses vector(<E>, <D>), and Appendix D defines JSON value serialization.

Issue: #18297

Test plan:

• git diff --check origin/main...HEAD
• Parsed format/spec.md using Python-Markdown with the tables extension.
• make lint could not install its Python dependencies because the package index was unreachable; docs CI is required.

---

AI Disclosure

• Model: GPT-5.6
• Platform/Tool: Codex
• Human Oversight: partially reviewed
• Prompt Summary: Generated the vector type spec wording and mapping/serialization table updates from the author's proposal and review feedback.

[danielcweeks]

You might want to add something to the effect of "a list of type E and length D is not equivalent to a vector(E,D)"

[stevenzwu]

+1.

I want to add two more points

• "No type promotion is defined" reads as unspecified, not forbidden.
• The unknown exception is the v3+ rule that unknown promotes to any type. Without it, this sentence contradicts that table.

A vector must not be promoted to another type. No type other than unknown may be promoted to vector. The element type and dimension must not change. A list of element type E is not a vector(E, D), even when every value has length D.

I would also add more details along with the upper and lower bounds

Equality of vector values is element-wise, using the equality rules of element type. A vector must not be an identifier field. No sort order is defined for a vector value. Writers must not write lower or upper bounds for a vector field.

[PFischbeck]

Concerning lower/upper bounds, we should also adjust the table in the "Field Statistics" section to mention vectors are excluded.

[pvary]

If we don't have stats for vectors, then we need to rely on other means to identify int8/uint8 etc.

[RussellSpitzer]

What stats would let you know the type width? I know we have some things around variable length types but with int float and friends I'm not sure they would make sense

[yyanyy]

thanks everyone,

No sort order is defined for a vector value

From some offline conversation there was a feedback that we might not need to artificially remove the sort order requirement, since vectors can in deed be sort naturally. While I don't have a concrete use case now, I'm thinking to not mention that this is not permitted to introduce unnecessary complication?

stats

Updated the stats table to mention most of the stats (like upper/lower bound), apart from the most basic ones (row count, null count) do not fit for vector

identify int8/uint8 etc

Following the discussion about narrower integers, I looked into how Parquet handles them. Its smallest integer physical type is INT32, while int8/uint8 are logical annotations that constrain the value range. These annotations do not themselves guarantee compact storage; the physical representation, encoding, and compression also matter.
Iceberg can already represent these values as int, and an engine can inspect the Parquet annotation to discover the narrower range. However, without a corresponding Iceberg schema contract, that range is not guaranteed across files and future writes. We therefore indeed miss an opportunity for engines to consistently use smaller representations for vector elements, although individual readers could still optimize particular files.
Custom table properties or column documentation might provide an engine-specific workaround, but would not establish an interoperable contract. As Ryan mentioned during the sync that many file format are currently rethinking about supporting narrower numeric types, I lean toward addressing narrower Iceberg element types in a follow-up proposal (I think during the sync we talked about potentially having Iceberg v5 to add them, after a more proper definition from file formats).
Regarding statistics, I think we should distinguish a column whose existing values happen to fit int8 from a column declared to be int8. Statistics can establish the former and support optional allocation optimizations, but cannot constrain future values. Using them as a persistent width declaration would conflate those two cases and might not be ideal IMO. A standardized schema contract seems preferable, but that belongs with the broader discussion of narrower integer support.

[pvary]

Thanks for digging into this! I think adding int8 and float16 is a relatively small spec change, see #18485.

It gives us a safe, future-proof way to store these values, and AI workloads rely heavily on these smaller types. Parquet already defines a representation for both (INT(8, true) on int32, and FLOAT16), so we can reuse it rather than wait for the file formats. If we go this way, the width becomes part of the schema, engines know the size up front and can plan their internal representation and query plan accordingly.

[stevenzwu]

identity still allows every primitive except geometry and geography. We probably should exclude vector there.

[pvary]

What do we think about storing smaller precision int/uint/float values?
The current types are very wide, and don't match types used on real world AI workloads.

[harsha-simhadri]

+1, vectors used in search are predominantly 8-bit integer types, half- and single-precision floats. Double and long are rarely used.

[RussellSpitzer]

That would have to be a subsequent proposal since we don't have such types defined in iceberg yet.

[Tagar]

➕1 to support 8-bit byte data type as well.

[yyanyy]

Please see some additional thoughts/summary from the sync meeting about the smaller precision types #18416 (comment) . I also added an item in next meeting (10/22)'s agenda to give people a chance to talk about the lower bit width types again in case people have extra feedback beyond last sync. Thanks!

[Kurtiscwright]

D must be positive

The Parquet format discussion and I believe the community sync both mentioned zero being an acceptable value. If that's so, I think this should read.

D must be positive or zero

What do you think?

P.S. Nitpick: I think Vector should be below #version-3 types on the table.

[PFischbeck]

The Parquet format discussion and I believe the community sync both mentioned zero being an acceptable value.

Are you sure? The current Parquet proposal mentions "greater than zero", and I'm not so sure about the benefit of a zero-length vector.

[stevenzwu]

@Kurtiscwright Iceberg community sync talked about float/double element values can be 0, -0, +0. Here, D is the number of dimensions. As @PFischbeck said, the vector size should be "greater than zero".

+1 to Kurtis's suggestion of positioning this new v4 type after the v3 geo types.

[stevenzwu]

Avro stores both list and vector as array. The file type alone does not say which one it is.

Iceberg marks a list by writing element-id on the Avro array. That is the list element's field id. A vector has no element field id, so its array must not have element-id. A writer that adds one makes the file look like a list.

Suggested note on the Avro row:

Array elements must use the Avro mapping for E and must not be null. The array must not have element-id. Readers and writers enforce dimension D and the element rules from the Iceberg schema. An array of length D is not a vector.

[PFischbeck]

Would it make sense to add an Avro logical type for vector?

[PFischbeck]

In the community sync, we said it might make sense to ask the Avro community about this. I started a discussion thread on the Avro dev mailing list: https://lists.apache.org/thread.html/ds0gm18b8cghotv4xtsmgo2ybghht60m

[stevenzwu]

@PFischbeck thanks for starting the Avro dev thread. There is no replies so far. As agreed in the sync, Avro discussion shouldn't be a blocker.

[stevenzwu]

+1.

I want to add two more points

• "No type promotion is defined" reads as unspecified, not forbidden.
• The unknown exception is the v3+ rule that unknown promotes to any type. Without it, this sentence contradicts that table.

A vector must not be promoted to another type. No type other than unknown may be promoted to vector. The element type and dimension must not change. A list of element type E is not a vector(E, D), even when every value has length D.

I would also add more details along with the upper and lower bounds

Equality of vector values is element-wise, using the equality rules of element type. A vector must not be an identifier field. No sort order is defined for a vector value. Writers must not write lower or upper bounds for a vector field.

[pvary]

Do we want to allow -0, +0?

[PFischbeck]

Are you suggesting we might only allow one representation of the number zero? If yes, do you have use-cases in mind?

[pvary]

I did some research, and found that some AI hardware has some issues with these values. That said, I agree with the things discussed on the sync, and we should support these values as most other vector storage systems support it.

[yyanyy]

I'm thinking to not mention -0/+0 explicitly here (with the updated writeup) since I think they continue to follow the definition here, and specifying mentioning them mostly because we talked about them without any real use case might feel a bit weird (especially with the AI hardware issue being mentioned)

[PFischbeck]

Thanks for working on this!

• We should (probably) disallow vector default values, i.e., only allow null as default
• We should disallow vector in all partition transforms (in expressions-spec.md)

[PFischbeck]

Are you suggesting we might only allow one representation of the number zero? If yes, do you have use-cases in mind?

[PFischbeck]

Concerning lower/upper bounds, we should also adjust the table in the "Field Statistics" section to mention vectors are excluded.

[PFischbeck]

What would this be used for?

[RussellSpitzer]

Are we allowing defaults? If so that would be one use case but this would be a pretty inefficient storage since I assume folks would want to default to all 0 or something

[yyanyy]

good call on not allowing non-null default, updated the spec for it (and put an item in next sync meeting in case people have extra thoughts). Though I'll probably not remove the [1.0, 2.0, 3.0] representation here, since this is just about json serialization demonstration, which might go beyond the default values' use case (e.g. for defining an expression per expression spec)

[PFischbeck]

I'm not too familiar with ORC, but does it make sense to also add new ORC type attributes to mark it as vector?

[RussellSpitzer]

Honestly orc is a zombie in our ecosystem at the moment so whatever we do that is minimally sufficient is probably fine.

[yyanyy]

updated the ORC to say not supported

[PFischbeck]

Would it make sense to add an Avro logical type for vector?

[RussellSpitzer]

Thanks for working on this!

• We should (probably) disallow vector default values, i.e., only allow null as default
• We should disallow vector in all partition transforms (in expressions-spec.md)

I think these are reasonable. For the purposes of the spec, while this is a primitive it basically should follow the same rules as list imho

[stevenzwu]

@PFischbeck thanks for starting the Avro dev thread. There is no replies so far. As agreed in the sync, Avro discussion shouldn't be a blocker.

[stevenzwu]

It seems that my original comment on this line got buried deep by other reply comments. Restarting a new thread.

Classifying vector as a primitive exposes two inconsistencies:

Identity: In this spec, identity accepts any primitive except geometry and geography, so adding vector silently makes it a valid source type. Rather than extending the exclusion list, could we use an explicit allowlist of supported primitive types, as bucket already does? format/expressions-spec.md uses the same broad rule for identity and should be aligned.

Bucket: No change is needed in this spec: bucket already has an explicit allowlist that does not include vector. However, format/expressions-spec.md defines bucket as accepting any primitive except geometry, geography, boolean, float, and double. That admits vector(int, D) and vector(long, D), while Appendix B defines no vector hash. The expressions spec should use the same allowlist as this spec.

format/expressions-spec.md is outside this diff, but both specs should remain consistent.

[stevenzwu]

I created a separate PR to address the partition transform source types: #18545

This thread is the same as the partition source types that @PFischbeck mentioned in his top-level comment.

[yyanyy]

thanks Steven, I think this raises another question that's probably worth mentioning in the community sync - do we consider vector a primitive type? Apart from the exclusion list here, there's other places where the spec mentioned about primitive type, e.g. test IN (constant set) that can allow "any primitive" but it probably doesn't make sense for vector to support this. Shall we define which types are primitive types, and exclude vector from that list?

[stevenzwu]

yeah, you are right. if the vector is not a primitive type, it is auto excluded for identity anyway.