[rok]

Rationale for this change

Discussion on the ML led to a decision to split the logical annotation proposal from the original proposal, which also aimed to provide a physical representation. This PR proposes the logical annotation only, to address the need for a logical vector type.

Naming: the annotation is called FINITE_VECTOR (previously VECTOR in this PR) so that the name carries the contract: fixed length, non-null and finite elements. This leaves VECTOR available should a future proposal introduce a more general vector type or a dedicated physical representation.

---

This adds a FINITE_VECTOR logical type stored as a canonical 3-level LIST structure. This is a contract on top of LIST, not a new physical representation:

1. Each non-null vector contains exactly num_elements elements.
2. Elements are required (non-null) and finite (no NaN, +Inf or -Inf).
3. Elements are numeric or boolean Parquet primitives.

<vector-repetition> group <name> (FINITE_VECTOR(<num_elements>)) {
  repeated group list {
    required <element-type> element;
  }
}

• Element types: BOOLEAN, INT32/INT64 (plain, INT or DECIMAL), FLOAT, DOUBLE, FLOAT16 and FLBA DECIMAL. No nested elements.
• Writers must enforce the element count, non-null and finite guarantees. Readers may rely on them without validating.
• Levels, pages, encodings, statistics and bloom filters are those of the element column in an ordinary LIST. No statistics are defined for whole vectors.
• Writers set ConvertedType.LIST alongside LogicalType.FINITE_VECTOR. Readers that fall back to the converted type read an ordinary LIST; readers that ignore both annotations see an unannotated group with a single repeated child group.

Trade-offs:

• Repetition/definition levels are still written. Reader and writer efficiency is left to the implementations.
• Adding new float types (e.g. FP8, BF16) requires adding a primitive Parquet type. Packed FLBA(num_elements * type_width) storage would decouple this, at the cost of element-level encodings.

Out of scope:

• Vector indexes, quantisation and normalisation guarantees. Those need their own annotations rather than parameters on FiniteVectorType.
• A more efficient physical type for storing fixed-sized lists may be proposed in the future and this logical type may be able to annotate it. This is intentionally left out of this proposal.

What changes are included in this PR?

Specifies the FINITE_VECTOR logical type, its LIST representation, writer/reader requirements and element-level requirements.

Do these changes have PoC implementations?

• parquet-cpp / pyarrow: GH-55347: [C++][Parquet][Python] Support the proposed FINITE_VECTOR logical type arrow#55348
• parquet-java: GH-3877: Add FINITE_VECTOR logical type support parquet-java#3876

[HuaHuaY]

Sorry I haven't followed VECTOR type progress in a while. Why was Option B mentioned in the email from May eliminated? I see that only Option A and Option C1 are left in the email from September.

[rok]

Option B mentioned in the email from May eliminated? I see that only Option A and Option C1 are left in the email from September.

Thanks for the question @HuaHuaY ! Option B requires changes to physical layout of data - either via new repetition level or another mechanism. As logical and physical changes are somewhat independent and should likely both be done, we decided to propose them separately. So in short Option B is not eliminated, only delayed until logical type for vectors is settled. Apologies this was not clear from my communication.

[alamb]

The proposal seems pretty good to me -- I think there are some places where we could reduce its size and make it more concise, but the high level idea looks good

[divjotarora]

Small wording comments, but overall LGTM. Thanks for driving this @rok!

[PFischbeck]

Overall looks good to me!

[rok]

Hey @JigaoLuo thanks for reviewing! Discussion on points you raise would better fit the discusson on the ML (1, 2). I'll still try to reply here, but if you want to push this further the dev@parquet ML is the platform you should use.

there is other metadata that would make vector data more self-describing. It could live in Parquet or in Iceberg:

1. Distance function. See the reasoning from Harsha and the related discussion on the Iceberg side: https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?disco=AAACDqu3oWI&pli=1

I am not an embeddings or vector DB expert, but wouldn't distance metric always an operation acting on pairs of vectors? It feels more like an application level concern and parquet already has a file-level metadata key-value metadata where such information could be stored. Or am I missing something here that could not be captured there?

2. Lineage: which embedding model or API produced the vectors.

Agreed that this is important information, but I'm not sure it's something that should live in the type. It would make parquet more self-contained, but it would require agreement on model identity between embedding producers which is out of reach for parquet. If you feel strongly about this, please sketch out how you imagine this would look like.

[JigaoLuo]

Hi @rok Thanks for sharing these. I'll go through the two discussions.

It feels more like an application level concern and parquet already has a file-level metadata key-value metadata where such information could be stored.

On point 1, I think this is a good place for it.

[alamb]

I think this looks good to me. I left some suggestions to cut it down more, but I don't think any other changes are needed (other than what @pitrou has suggested)

[alamb]

🧑‍🍳 👌

[wgtmac]

Just curious if BYTE_ARRAY annotated with DECIMAL should be supported.

[Fokko]

For completeness, it could be added, but I'd leave it out. Every supported element type is fixed width. Any decimal precision can already be stored as FIXED_LEN_BYTE_ARRAY, so excluding BYTE_ARRAY loses nothing.

[rok]

Fixed width allows for some optimizations that variable width doesn't. I believe allowing variable-width types would make writers more complex as they'd have to enable or disable these optimizations at write time.
If it turns out there's a need for this we can revisit later too!

[Fokko]

For completeness, it could be added, but I'd leave it out. Every supported element type is fixed width. Any decimal precision can already be stored as FIXED_LEN_BYTE_ARRAY, so excluding BYTE_ARRAY loses nothing.