Skip to content

Core: Support metrics with nested field in AppendBenchmark and FileGenerationUtil - #17391

Open
dramaticlly wants to merge 2 commits into
apache:mainfrom
dramaticlly:complexFileGenUtils
Open

Core: Support metrics with nested field in AppendBenchmark and FileGenerationUtil#17391
dramaticlly wants to merge 2 commits into
apache:mainfrom
dramaticlly:complexFileGenUtils

Conversation

@dramaticlly

@dramaticlly dramaticlly commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Adding schema support for nested fields in AppendBenchmark and FileGenerationUtil. Previous it will only handle the primitive type but real production workload often involves complex type.

fast numFiles Score (ms/op)
true (FastAppend) 50,000 519.3 ± 676.1
true (FastAppend) 100,000 876.3 ± 3311.6
false (MergeAppend) 50,000 484.0 ± 247.3
false (MergeAppend) 100,000 739.2 ± 395.2

@github-actions github-actions Bot added the core label Jul 28, 2026
@dramaticlly dramaticlly changed the title Core: Add nested field traversal in AppendBenchmark and FileGenerationUtil Core: Support nested fielld in AppendBenchmark and FileGenerationUtil Jul 28, 2026
@dramaticlly dramaticlly changed the title Core: Support nested fielld in AppendBenchmark and FileGenerationUtil Core: Support metrics with nested field in AppendBenchmark and FileGenerationUtil Jul 28, 2026
List<Types.NestedField> leaves = Lists.newArrayList();
TypeUtil.visit(
schema,
new TypeUtil.SchemaVisitor<Void>() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have struct / list / map here, but how about variant? With v3 adoption growing, a schema containing variant will throw rather than skip. Let's at least consider adding

@Override
public Void variant(Types.VariantType v) {
  return null;
}

@dramaticlly dramaticlly Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think current behavior for leafFields will throw UnsupportedOperationException if running into a V3 table with variant type. To me, it seems reasonable as exception serve as a reminder to add variant support instead of silently missing metrics or bound. There's also change like #15629 to add dedicated benchmark for variant.

Prefer to keep the existing scope and revisit v3+ type later in follow up PR

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a variand benchmark, we need reviews of it!


@Test
public void testBoundsForNestedTypes() {
MetricsConfig metricsConfig = MetricsConfig.getDefault();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we only use MetricsConfig.getDefault(), but the new key code path for a nested field is never exercised under non-default modes (none, counts, truncate).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, parametrized on all metrics mode for better coverage

@dramaticlly

Copy link
Copy Markdown
Contributor Author

@huaxingao @nastra if you want to take a look as well :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants