Repository navigation
API, Core: Add VectorType for fixed-length vectors - #18676
Open
PFischbeck wants to merge 5 commits into
Open
PFischbeck wants to merge 5 commits into
PFischbeck wants to merge 5 commits into
Conversation
Add a VectorType primitive parameterized by an element type (int, long, float, double) and a dimension (>= 1), serialized as vector(<element-type>, <dimension>) e.g. vector(int, 768). Gate it to format version 4, reject it as an identity partition source or identifier field, and allow only null defaults. Generated-by: Cursor Grok 4.6 Co-authored-by: Cursor <cursoragent@cursor.com>
Generated-by: Cursor Grok 4.6 Co-authored-by: Cursor <cursoragent@cursor.com>
Vector columns keep value and null counts, but a bound field of type vector has no Parquet representation. V4 manifests store content stats in Parquet, so those columns must not include lower, upper, or tight bounds. Generated-by: Cursor Grok 4.7 Co-authored-by: Cursor <cursoragent@cursor.com>
Document that vector columns are excluded from lower and upper bounds. tracksTightBounds stays the variant and geo check, because its only caller already requires hasBounds. Generated-by: Cursor Grok 4.7 Co-authored-by: Cursor <cursoragent@cursor.com>
Spotless formats a one-sentence javadoc as a single line. Generated-by: Cursor Grok 4.7 Co-authored-by: Cursor <cursoragent@cursor.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a VectorType primitive parameterized by an element type (int, long, float, double) and a dimension (>= 1), serialized as
vector(<element-type>, <dimension>), e.g.vector(int, 768). It's gated to format version 4, rejected as an identity partition source, and allows only null defaults.Reading and writing to data files is not covered in this PR.
Design discussion: https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?usp=sharing
Spec change PR: #18416
The API design choice to treat vector element types as a separate enum is for future compatibility. As of now, all supported element types are also Iceberg primitive types. But there are discussions that we might extend this list in the future, even to types like
float16that might only be supported as element type, not Iceberg primitive type.I'm open for suggestions here (such as only allowing the fixed set of primitive types). My main concern then would be that this is a one-way road to not allowing for other non-Iceberg element types in the future without breaking API changes.
AI disclaimer: Cursor Grok 4.6 was used to generate most of the code (with several iterations and feedback loops). All parts were human-reviewed.