Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions full-path-realtime/quotes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ BigQuery deliberately uses automatic active-window selection: every eligible act
ClickHouse reference observation is selected and the report records `requested_count: null`.
Snowflake and Redshift freeze their accepted pairwise counts in their command notebooks.

## Measurements behind Part 6

The [Part 6 measurement index](part6-measurements/README.md) connects the final article visual numbers to the exact measurements, SQL, calculation definitions, and reproduction instructions. It includes the writer and reader resource packages, CPU-core scaling, and the separate horizontal-scaling experiment.

## Accepted evidence map

| System | Implementation and run notes | Accepted results | Cost and charts |
Expand Down
84 changes: 40 additions & 44 deletions full-path-realtime/quotes/global/visualizations/README.md
Original file line number Diff line number Diff line change
@@ -1,46 +1,42 @@
# Global real-time benchmark visualizations

This directory is the provider-neutral synthesis layer for the quotes benchmark.
It does not replace provider-specific renderers or matched cost artifacts.

## Contracts

- Query latency charts reread original runner JSONL. Each provider is plotted at
its own observed `raw_rows`; iteration numbers are never joined across systems.
- The 100B row cap is inclusive. Aggregate lines use a provider-local centered
seven-observation median; drill-down lines use five observations.
- The Snowflake aggregate series retains the accepted pairwise Tukey upper-fence
policy. No outlier policy is invented for ClickHouse, BigQuery, or drill-downs.
- MV lag means persisted materialized-view refresh lag. ClickHouse is zero by
ingest-time incremental-MV design. BigQuery's default direct-query delta
reconciliation is query-answer behavior and is not this persisted-lag metric.
- Complete-ingest fresh-data-path costs are read unchanged from the accepted
Snowflake, BigQuery, and Redshift pairwise summaries. The renderer verifies
their ClickHouse source identity, keeps BigQuery Capacity and On-demand as
alternatives, and charges Redshift's shared writer-plus-MSK path once in each
SUPER or Typed counterfactual.
- Full-path cost-performance values are read unchanged from the accepted
Snowflake, BigQuery, and Redshift summaries. Each provider or read-layout
alternative is normalized to ClickHouse within its own pairwise matched
window; this chart does not introduce a cross-provider iteration match. Its
score bars use a disclosed log-relative scale with no artificial width floor.
- The absolute cost-versus-runtime map reads `total_cost` and `runtime_sec` from
those same summaries. It intentionally accepts their different pairwise
query windows, and uses inverted logarithmic axes so faster is right and
lower cost is up.

Run the commands in `_commands.txt`. Charts, SVGs, source CSVs, and provenance
summaries are written to `quotes/global/results/charts/`.

`manifest.json` declares the exact accepted label set for every global chart.
Each renderer validates that set case-insensitively before drawing, so a
missing provider, pricing alternative, or Redshift layout fails closed.
Generated summaries use repository-relative paths and source hashes.

The command file defines `CHART_BACKGROUND_COLOR` once (default `#161614`) and
always produces both standard and true slide-wide variants. Wide variants use
larger presentation type and expand the plot into the available 16:8.5 canvas.
The global latency charts intentionally omit the pairwise runtime/cost evidence
strip because each provider retains its own accepted comparison window. The
absolute cost-versus-runtime quadrant also has a wide layout so its inverted
logarithmic axes, markers, and labels remain legible on a webinar slide.
This directory combines the accepted ClickHouse Cloud, Snowflake, BigQuery, Redshift Serverless, and Databricks Serverless SQL results. Start with the [quotes methodology and evidence map](../../README.md) or the [Part 6 measurement index](../../part6-measurements/README.md).

## Results and evidence

| Measurement | Downloadable values | Calculations and source paths |
| --- | --- | --- |
| Aggregate-query latency | [CSV](../results/charts/aggregate_query_latency_all_systems_data.csv) | [Summary](../results/charts/aggregate_query_latency_all_systems_summary.json) |
| Drill-down query latency | [CSV](../results/charts/drilldown_query_latency_all_systems_data.csv) | [Summary](../results/charts/drilldown_query_latency_all_systems_summary.json) |
| Pre-aggregation lag | [Observations](../results/charts/mv_lag_all_systems_data.csv), [display curve](../results/charts/mv_lag_all_systems_curve.csv) | [Summary](../results/charts/mv_lag_all_systems_summary.json) |
| Complete preparation cost | [CSV](../results/charts/ingest_fresh_path_cost_all_systems_data.csv) | [Summary](../results/charts/ingest_fresh_path_cost_all_systems_summary.json) |
| End-to-end performance per dollar | [CSV](../results/charts/full_path_cost_performance_all_systems_data.csv) | [Summary](../results/charts/full_path_cost_performance_all_systems_summary.json) |
| Total cost versus accumulated query runtime | [CSV](../results/charts/full_path_cost_vs_query_runtime_all_systems_data.csv) | [Summary](../results/charts/full_path_cost_vs_query_runtime_all_systems_summary.json) |

[manifest.json](manifest.json) identifies the accepted provider inputs and required labels for each chart. It includes Databricks' September full-run integration, its accepted query observations, and its preparation and full-path summaries. Provider guides retain the matching and pricing details.

## Measurement definitions

**Query latency.** Charts read the original runner observations and plot each provider at its own observed row progress. The presentation cap is inclusive at 100 billion rows. Aggregate lines use a provider-local centered seven-observation median; drill-down lines use five observations. Snowflake's aggregate series retains its accepted pairwise Tukey upper-fence policy. No new outlier policy is applied to the other providers or workloads.

**Pre-aggregation lag.** The common chart measures the lag between incoming raw data and maintained summaries. ClickHouse's baseline is zero because its incremental materialized view processes the same inserted blocks as its raw table. BigQuery's default direct-query reconciliation produces current answers separately from its summary-refresh lag. Databricks records refresh age and a row-watermark gap rather than the same time-lag measure; the manifest keeps it out of this common lag chart and points to the run-specific [freshness evidence](../../databricks/SEPTEMBER_INTEGRATION.md).

**Preparation cost.** Complete-ingest values come from accepted Snowflake, BigQuery, Redshift, and Databricks summaries. ClickHouse source identities are checked before combining them. BigQuery Capacity and On-demand are alternative prices. Each Redshift SUPER or typed read-layout comparison includes the shared writer-plus-MSK path once.

**End-to-end performance per dollar.** Each provider or pricing/layout alternative is normalized to ClickHouse within its own accepted pairwise query window. The score is `(preparation cost + matched query cost) × accumulated query runtime`; lower is better. The score chart uses `log10(relative score)`, with ClickHouse's 1× reference at the origin. The global chart combines accepted pairwise ratios.

**Cost versus runtime.** The map reads the accepted total cost and accumulated runtime from those same summaries. The axes are logarithmic and inverted: faster is right and lower cost is up. Each provider retains its accepted query window; the source summary records the inputs. Query cost uses the accepted normalized allocation.

## Reproduce the charts

From the repository root:

```sh
bash full-path-realtime/quotes/global/visualizations/_commands.txt
```

The command notebook uses `uv` by default, supports overriding `UV`, and writes PNG, SVG, source CSVs, and JSON summaries to [global/results/charts/](../results/charts/). Rebuild provider cost summaries first when their inputs change; follow the [documented reproduction order](../../README.md#reproduce-reconciliation-cost-and-charts).

Every renderer checks the exact required provider/alternative label set declared in the manifest before drawing. Generated summaries use repository-relative source paths and hashes. The standard and true 5156×2900 slide-wide variants use the same accepted inputs; wide filenames end in `_wide`.

The command file defines `CHART_BACKGROUND_COLOR` once, defaulting to `#161614`. `COSTS_ONLY=1` preserves latency and freshness outputs; `WIDE_ONLY=1` generates only wide variants. The global latency charts omit the pairwise runtime/cost strip because the providers retain their own accepted comparison windows.
102 changes: 102 additions & 0 deletions full-path-realtime/quotes/part6-measurements/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# Measurements behind CostBench Part 6

This guide links the quantitative visuals in [Why ClickHouse Cloud wins at real-time analytics](https://clickhouse.com/blog/why-clickhouse-cloud-wins-at-real-time-analytics) to their measurements, queries, calculations, and reproduction instructions. Visual numbers follow the final 01–25 article sequence.

## Find a visual

| Visuals | Measurements | Start here |
| --- | --- | --- |
| 02, 03, 25 | End-to-end score, preparation and query cost, accumulated query runtime | [End-to-end performance per dollar](#end-to-end-performance-per-dollar) |
| 04 | Pre-aggregation freshness | [Pre-aggregation freshness](#pre-aggregation-freshness) |
| 05 | Cost of preparing incoming data | [Preparation cost](#preparation-cost) |
| 09–12 | Writer throughput, CPU work, compression, row reduction, and background merges | [Writer preparation measurements](#writer-preparation-measurements) |
| 13 | Accumulated query cost and runtime | [Query cost and runtime](#query-cost-and-runtime) |
| 14, 22 | Aggregate-query latency and current summaries | [Query latency and freshness](#query-latency-and-freshness) |
| 16, 17 | Reader throughput, latency, CPU use, and query memory | [Reader execution measurements](#reader-execution-measurements) |
| 18 | Query throughput across CPU-core settings | [CPU-core scaling](#cpu-core-scaling) |
| 19 | Query-side and end-to-end performance-per-dollar scores | [Query-side and end-to-end scores](#query-side-and-end-to-end-scores) |
| 21 | One query distributed across 1–100 nodes | [Horizontal query scaling](#horizontal-query-scaling) |

## End-to-end performance per dollar

Start with the [quotes benchmark methodology and accepted evidence map](../README.md) and the [global chart guide](../global/visualizations/README.md). The score combines preparation cost and matched query cost, then multiplies by accumulated query runtime. Each provider comparison retains its accepted inputs and matched query window.

- [Score values and provider source paths](../global/results/charts/full_path_cost_performance_all_systems_summary.json)
- [Downloadable score values](../global/results/charts/full_path_cost_performance_all_systems_data.csv)
- [Cost-versus-runtime values and sources](../global/results/charts/full_path_cost_vs_query_runtime_all_systems_summary.json)
- [Downloadable cost-versus-runtime values](../global/results/charts/full_path_cost_vs_query_runtime_all_systems_data.csv)
- [Chart commands](../global/visualizations/_commands.txt)

The individual comparison guides retain the preparation, query-cost, and runtime calculations: [Snowflake](../snowflake/README.md), [Databricks](../databricks/SEPTEMBER_INTEGRATION.md), [BigQuery](../bigquery/README.md), and [Redshift Serverless](../redshift-serverless/README.md). Part 6 uses BigQuery Capacity and the Redshift SUPER read path for its main comparison.

## Pre-aggregation freshness

The [global chart guide](../global/visualizations/README.md) describes the common pre-aggregation-lag chart. Its [measurements](../global/results/charts/mv_lag_all_systems_data.csv), [displayed curve](../global/results/charts/mv_lag_all_systems_curve.csv), and [source summary](../global/results/charts/mv_lag_all_systems_summary.json) are published together.

ClickHouse's raw data and incremental summaries advance from the same inserted blocks; its baseline comes from the [table and materialized-view definitions](../clickhouse-cloud/create.sql). Databricks records refresh age and a row-watermark gap separately; see its [accepted integration](../databricks/SEPTEMBER_INTEGRATION.md) and [run evidence](../databricks/results/serverless_baseline_full_20260918T170453Z/integration/). The [global manifest](../global/visualizations/manifest.json) records that difference explicitly.

## Preparation cost

- [Downloadable preparation-cost values](../global/results/charts/ingest_fresh_path_cost_all_systems_data.csv)
- [Accepted provider inputs and calculations](../global/results/charts/ingest_fresh_path_cost_all_systems_summary.json)
- [Provider configurations, pricing, and reproduction order](../README.md)

These totals cover complete ingestion and its preparation work. BigQuery Capacity and On-demand are alternative prices; each Redshift read-layout comparison includes the same writer-plus-MSK path once.

## Writer preparation measurements

Use the [writer preparation README](../clickhouse-cloud/results_t2/utilization/writer/preparation-metrics/README.md) for visuals 09–12. It maps each chart to extraction SQL, source CSVs, formulas, and exact chart payloads. It covers processing throughput, CPU work by ingestion stage, raw-data compression, summary-row reduction, background consolidation, and the service's resource band.

The measurements come from the two-node, four-vCPU ingest service during the original continuous-ingestion run. The package contains commands to regenerate and check every published value from the committed exports. Filenames retain the earlier 08a–08d identifiers.

## Query cost and runtime

The accumulated query curves retain the accepted observations and pricing for each comparison. Start with the provider-specific chart guides: [Snowflake](../snowflake/visualizations/README.md), [Databricks](../databricks/visualizations/README.md), [BigQuery](../bigquery/visualizations/README.md), and [Redshift Serverless](../redshift-serverless/visualizations/README.md).

The [global manifest](../global/visualizations/manifest.json) and [end-to-end summaries](#end-to-end-performance-per-dollar) connect those inputs to the comparison totals. Query cost uses the accepted normalized allocation; accumulated runtime is the sum of the selected query durations.

## Query latency and freshness

Visuals 14 and 22 use the A4 full-history aggregate-query latency through 100 billion quotes alongside the systems' summary freshness. The exact query is in [ClickHouse's aggregate SQL](../clickhouse-cloud/queries_mv.sql); corresponding queries and observations are listed in the [global manifest](../global/visualizations/manifest.json).

- [Aggregate-query measurements](../global/results/charts/aggregate_query_latency_all_systems_data.csv)
- [Latency definitions and source paths](../global/results/charts/aggregate_query_latency_all_systems_summary.json)
- [Original ClickHouse runner observations](../clickhouse-cloud/results_t2/mv/dashboard_20260808T065559Z.jsonl)
- [Pre-aggregation freshness](#pre-aggregation-freshness)

## Reader execution measurements

Use the [reader resource and display README](../clickhouse-cloud/results_t2/utilization/reader/query-metrics/query-resources/README.md) for visuals 16 and 17. It explains the current throughput and latency displays, consumed CPU work, per-query memory peaks, and effective aggregation throughput, with source exports and commands for exact offline reproduction.

The [original query-metrics README](../clickhouse-cloud/results_t2/utilization/reader/query-metrics/README.md) links the extraction SQL, full query-log exports, individual-query measurements, and original runner files. These measurements describe the 16-vCPU reader while fresh quotes were being ingested. Filenames retain the earlier 11a/11b identifiers.

## CPU-core scaling

Visual 18 uses the [cache-controlled CPU scaling experiment](../clickhouse-cloud/experiments/reader-thread-scaling/2026-10-11-cache-modes/README.md): the original D1/D2 queries, fixed data, and six measured core settings on one 59-vCPU / 236-GiB node.

The README links the [findings](../clickhouse-cloud/experiments/reader-thread-scaling/2026-10-11-cache-modes/REPORT.md), [combined measurements](../clickhouse-cloud/experiments/reader-thread-scaling/2026-10-11-cache-modes/derived/combined-workload.csv), exact display payload, cache checks, and offline/live reproduction instructions. It is a separate experiment from the continuous-ingestion reader measurements. Filenames retain the earlier 11c identifier.

## Query-side and end-to-end scores

Visual 19 brings together two CostBench scopes:

- [Query-side benchmark](../../../query-side-only/README.md): query performance per dollar over already-prepared data, with each provider's recorded runtimes and pricing.
- [End-to-end quotes benchmark](../README.md): continuous ingestion, preparation, freshness, and query execution together, using the accepted [score summaries](#end-to-end-performance-per-dollar).

Each score retains the measurements and calculations of its own study.

## Horizontal query scaling

Visual 21 uses a separate UK property-price aggregation experiment in [ClickHouse/examples](https://github.com/ClickHouse/examples/tree/3b6da4b0d0607d4debe5783021ffd2019058637e/blog-examples/ParallelReplicasBench). The fixed one-trillion-row query groups by county, town, and district and calculates counts, sums, and averages. Seven configurations use 1, 3, 10, 20, 40, 80, and 100 nodes, each with 89 CPU cores.

- [Recorded results and per-run statistics](https://github.com/ClickHouse/examples/tree/3b6da4b0d0607d4debe5783021ffd2019058637e/blog-examples/ParallelReplicasBench/results/2025-08-30_horiz_uk_t1_n1-3-10-20-40-80-100_c89_r10)
- [Configuration and selected-run summary](https://github.com/ClickHouse/examples/blob/3b6da4b0d0607d4debe5783021ffd2019058637e/blog-examples/ParallelReplicasBench/results/2025-08-30_horiz_uk_t1_n1-3-10-20-40-80-100_c89_r10/summary_matrix-nodes-1_3_10_20_40_80_100_cores-89_2025-08-30_165307.json)
- [Exact aggregation SQL](https://github.com/ClickHouse/examples/blob/3b6da4b0d0607d4debe5783021ffd2019058637e/blog-examples/ParallelReplicasBench/datasets/uk_property_prices/aggregation.sql)
- [Dataset loading instructions](https://github.com/ClickHouse/examples/blob/3b6da4b0d0607d4debe5783021ffd2019058637e/blog-examples/ParallelReplicasBench/datasets/uk_property_prices/README.md)
- [Benchmark runner and reproduction](https://github.com/ClickHouse/examples/blob/3b6da4b0d0607d4debe5783021ffd2019058637e/blog-examples/ParallelReplicasBench/README.md)

At each setting, the visual uses the fastest recorded run and the initiator's global `query_bytes_per_sec / 1e9` for decimal GB/s. The simpler 100-billion-row, 1.45-TB/s demonstration in the folded note is documented separately in the [parallel-replicas article](https://clickhouse.com/blog/clickhouse-parallel-replicas).

## Reproducing results

The [quotes benchmark README](../README.md#reproduce-reconciliation-cost-and-charts) gives the order for reconciliation, cost calculation, and chart generation. Each measurement package above includes its own offline derivation and validation commands. Published exports allow the displayed calculations to be regenerated without access to the original live services. Repeating the stock-quotes workload requires a [licensed dataset](../README.md#data-source-and-licensing).
Loading