Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
622acfd
feat(plan): time project discovery, rules and the warehouse connection
kvlonge Nov 17, 2026
de14f00
perf(plan): list each schema's tables and views at the same time
kvlonge Nov 17, 2026
ab3ca11
perf(plan): read cursor bounds in batched UNION ALL statements
kvlonge Nov 17, 2026
0d1c97d
fix(plan): rebind sources only on new column evidence on case-sensiti…
kvlonge Nov 17, 2026
fa9e899
perf(plan): look up migration origins without rescanning every finger…
kvlonge Nov 17, 2026
a3081d9
test(perf): guard live plans of the 10K project and give it real plan…
kvlonge Nov 17, 2026
40b0124
fix(plan): compare inspected source column types as the warehouse sto…
kvlonge Nov 17, 2026
20b0782
test(plan): rebind the quoted-case session case on a retyped source c…
kvlonge Nov 17, 2026
ca9468b
perf(plan): analyse readers again only when a source's new columns ca…
kvlonge Nov 18, 2026
ddb023a
feat(plan): let a target plan from its synced copy of warehouse state
kvlonge Nov 18, 2026
a023a2c
test(perf): bound the 10K live plan by the location's publish, not th…
kvlonge Nov 18, 2026
b6c6fea
feat(python-nodes): depend on a job or a tag
kvlonge Nov 18, 2026
fe5add4
perf(compile): verify a model's typed SQL macro markers once per process
kvlonge Nov 18, 2026
02e5b53
perf(plan): read target columns for the selection and its upstream only
kvlonge Nov 18, 2026
a700dab
chore(fensu): record the intentional mirrors the plan work touches
kvlonge Nov 18, 2026
79301ec
Merge remote-tracking branch 'origin/main' into perf/plan-speed-main
kvlonge Nov 18, 2026
382755b
refactor(graph): publish set dependencies from the project selection
kvlonge Nov 18, 2026
f13893c
test(goldens): list plan_from among a target's allowed keys
kvlonge Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions benchmarks/duckshop-10k/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,13 @@ run `python benchmarks/duckshop-10k/generate.py --update-checksum`.

| | Count |
|---|---|
| Models | 10,282 |
| Staging, intermediate, marts and snapshots | 6,149 (59.8%), 1,961 (19.1%), 2,162 (21.0%), 10 |
| Models | 10,840 |
| Staging, intermediate, marts and snapshots | 6,149 (56.7%), 2,519 (23.2%), 2,162 (19.9%), 10 |
| Incremental models | 583, 112 of them with a capped automatic start and 112 renamed |
| Python export assets | 433 in 241 modules sharing one helper module |
| Sources | 2,466 table sources and one expression source |
| Unit tests and scenarios | 288 and 2 |
| Dependency depth | 11 at most; 168 models at depth 7 or more |
| Dependency depth | 11 at most; 170 models at depth 7 or more |

Most models read one to four upstreams; a few dozen wide marts, unions and scorecards read ten or
more. Staging deduplicates change logs with a change-log macro, types three platform layouts
Expand All @@ -51,6 +53,13 @@ clickstream events in microbatches and builds customer identity, value and segme
budget variance chain. Marts give every instance, team and region its reports, and publish facts and
dimensions with enforced contracts.

A large project also carries planning workload the layers above barely touch, so the generator adds
it in the proportions such projects have: an incrementally kept history of every third timestamped
intermediate table, one in five of them capping how far ahead its automatic start may sit and one in
five renamed from an older name with `migrate_from`, and a Python export asset for every fifth
published mart, a few per module, all sharing one helper module. A live plan reads a cursor bound for
each incremental model, plans each rename and discovers each Python node.

The models that use `HASH`, `TIMESTAMP_LTZ` or `CURRENT_TIMESTAMP()` are tagged `known_divergence`,
because their results depend on the engine or the session.

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/duckshop-10k/checksum.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
generator 3 files 10874 sha256 5f3d30b82ebf52f2eb869494f52f77d1699f9a08743febd6ee9ca0cb22ed0611
generator 4 files 11674 sha256 d0012261f902e87a546a04bd8c6552c8a050f9cec6753b5acccc53584bce9d37
2 changes: 1 addition & 1 deletion benchmarks/duckshop-10k/duckshop10k/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@
regenerate checksum.txt, whenever a change alters any generated file.
"""

GENERATOR_VERSION = "3"
GENERATOR_VERSION = "4"
SEED = 20261024
PROJECT_NAME = "duckshop_10k"
2 changes: 2 additions & 0 deletions benchmarks/duckshop-10k/duckshop10k/project.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from duckshop10k.rules_gen import build_rules
from duckshop10k.staging import StagedTable, stage_collector, stage_instance
from duckshop10k.tests_gen import build_scenarios, build_tests
from duckshop10k.workload import build_workload
from duckshop10k.world import Instance, instances, storefront_instances


Expand Down Expand Up @@ -89,6 +90,7 @@ def build(profile_name: str = "full") -> dict[str, str]:
build_extras(project, intermediate)
build_tests(project, staging.tables, intermediate)
build_scenarios(project, staging.tables)
build_workload(project)
build_rules(project)
_static_files(project)
project.write("seeds/source_instances.csv", instances_seed(systems))
Expand Down
153 changes: 153 additions & 0 deletions benchmarks/duckshop-10k/duckshop10k/workload.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
"""Planning workload a large project carries: incremental history, renamed models and exports.

A live plan of a real project reads a cursor bound for every incremental model, plans a
migration for every renamed model and discovers every Python node. The layers above build
few of each, so this module adds them in proportions such projects have: about one model in
twenty kept incrementally, a hundred of those renamed from an older name, and a few hundred
Python exports of published marts, spread over many modules that share helper code.
"""

from __future__ import annotations

from duckshop10k.core import Column, Model, Project

#: Every this-many timestamped intermediate table gets an incrementally kept copy.
INCREMENT_STRIDE = 3
#: One incremental model in this many caps how far ahead its automatic start may sit.
MAX_AHEAD_EVERY = 5
#: One incremental model in this many was renamed from an older name.
RENAMED_EVERY = 5
#: Every this-many published mart gets a Python export asset.
EXPORT_STRIDE = 5
#: Export assets per Python module, so discovery walks many modules.
EXPORTS_PER_MODULE = 6
_TIMESTAMP = "TIMESTAMP_NTZ"
_INCREMENT_FOLDER = "models/intermediate/increments"
_EXPORT_PACKAGE = "python/exports"
_SHARED_MODULE = "python/shared/export_feed.py"

SHARED_EXPORT_HELPERS = '''"""What every export asset records about the relation it sent."""

from sqlbuild.assets import AssetContext
from sqlbuild.refs import SqlResourceRef


def feed_metadata(ctx: AssetContext, reference: SqlResourceRef, feed: str) -> dict[str, object]:
"""The relation an export read, its row count and the partner feed it went to."""
relation = ctx.relation(reference)
rows = ctx.query(f"SELECT COUNT(*) FROM {relation}").fetchone()[0]
return {"feed": feed, "relation": str(relation), "rows": rows, "format": feed_format(feed)}


def feed_format(feed: str) -> str:
"""Partner feeds named after a region take CSV; every other feed takes Parquet."""
return "csv" if feed.endswith(("_eu", "_us", "_apac", "_mea", "_latam")) else "parquet"
'''


def build_workload(project: Project) -> None:
"""Add the incremental, renamed and Python export nodes to the assembled project."""

increments: list[Model] = _increments(project)
_rename(increments)
_exports(project)


def _timestamp_column(model: Model) -> Column | None:
return next((column for column in model.columns if column.type == _TIMESTAMP), None)


def _is_table(model: Model) -> bool:
return not any(line.startswith("materialized") for line in model.config)


def _increments(project: Project) -> list[Model]:
candidates: list[Model] = sorted(
(
model
for model in project.models.values()
if model.layer == "intermediate"
and _is_table(model)
and _timestamp_column(model) is not None
),
key=lambda model: model.name,
)
added: list[Model] = []
for position, upstream in enumerate(candidates[::INCREMENT_STRIDE]):
cursor: Column = next(column for column in upstream.columns if column.type == _TIMESTAMP)
config: list[str] = [
"materialized incremental",
"incremental_strategy append",
f"cursor {cursor.name}",
"cursor_type timestamp",
"cursor_grain day",
f"cursor_inputs (\n {upstream.name} {cursor.name},\n )",
]
if position % MAX_AHEAD_EVERY == 0:
config.append("cursor_start_max_ahead 1d")
columns: list[Column] = [
Column(column.name, column.type, audits=("not_null",) * (column == cursor))
for column in upstream.columns
]
select: str = ",\n ".join(f"s.{column.name}" for column in columns)
added.append(
project.add(
Model(
name=f"{upstream.name}_history",
folder=f"{_INCREMENT_FOLDER}/{upstream.folder.rsplit('/', 1)[-1]}",
layer="intermediate",
description=(
f"Every row of {upstream.name} kept incrementally by {cursor.name}, so "
"earlier days stay queryable after the source rewrites them"
),
columns=columns,
body=(
"WITH source_rows AS (\n SELECT *\n"
f' FROM __ref("{upstream.name}")\n),\n\n'
f"final AS (\n SELECT\n {select}\n FROM source_rows AS s\n)\n\n"
"SELECT * FROM final"
),
config=config,
refs=[upstream.name],
)
)
)
return added


def _rename(increments: list[Model]) -> None:
for model in increments[::RENAMED_EVERY]:
model.config.append(f"migrate_from {model.name.removesuffix('_history')}_archive")


def _exports(project: Project) -> None:
marts: list[Model] = sorted(
(model for model in project.models.values() if model.layer == "marts" and _is_table(model)),
key=lambda model: (model.folder, model.name),
)
by_folder: dict[str, list[Model]] = {}
for model in marts[::EXPORT_STRIDE]:
by_folder.setdefault(model.folder.rsplit("/", 1)[-1], []).append(model)
project.write(_SHARED_MODULE, SHARED_EXPORT_HELPERS)
for folder, models in sorted(by_folder.items()):
for start in range(0, len(models), EXPORTS_PER_MODULE):
chunk: list[Model] = models[start : start + EXPORTS_PER_MODULE]
module: str = f"{folder}_{start // EXPORTS_PER_MODULE + 1:02d}"
project.write(f"{_EXPORT_PACKAGE}/{module}.py", _export_module(folder, chunk))


def _export_module(folder: str, models: list[Model]) -> str:
assets: list[str] = [
f'@asset(depends_on=model("{model.name}"))\n'
f"def export_{model.name}(ctx: AssetContext):\n"
f' """Send {model.name} to the {folder} partner feed."""\n'
f' metadata = feed_metadata(ctx, model("{model.name}"), "{folder}")\n'
" return ctx.result(metadata=metadata, materialized=True)\n"
for model in models
]
return (
f'"""Exports of the {folder} marts to partner feeds."""\n\n'
"from sqlbuild.assets import AssetContext, asset\n"
"from sqlbuild.refs import model\n\n"
"from python.shared.export_feed import feed_metadata\n\n\n" + "\n\n".join(assets)
)
27 changes: 22 additions & 5 deletions benchmarks/duckshop-10k/tests/test_generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,15 @@ def test_given_full_profile_when_building_then_matches_recorded_checksum() -> No
assert checksum_line(full()) == (HERE / "checksum.txt").read_text()


def test_given_full_profile_when_measuring_then_has_about_ten_thousand_models_in_real_mix() -> None:
def test_given_full_profile_when_measuring_then_has_about_eleven_thousand_models_in_real_mix() -> (
None
):
measured = shape(full())

assert 9_900 <= measured.models <= 10_300
assert 58 <= measured.layer_percent("staging") <= 62
assert 18 <= measured.layer_percent("intermediate") <= 21
assert 19 <= measured.layer_percent("marts") + measured.layer_percent("snapshots") <= 23
assert 10_500 <= measured.models <= 11_000
assert 55 <= measured.layer_percent("staging") <= 59
assert 21 <= measured.layer_percent("intermediate") <= 25
assert 18 <= measured.layer_percent("marts") + measured.layer_percent("snapshots") <= 22


def test_given_full_profile_when_measuring_then_fan_in_and_depth_match_the_profile() -> None:
Expand Down Expand Up @@ -78,6 +80,9 @@ def test_given_full_profile_when_measuring_then_samples_a_few_hundred_unit_tests
(r"TRY_PARSE_JSON", 100),
(r"\bFILTER\(", 10),
(r"incremental_mode microbatch", 20),
(r"materialized incremental", 500),
(r"cursor_start_max_ahead", 100),
(r"migrate_from", 100),
(r"materialized snapshot", 5),
(r"__udf\(", 2),
(r"__table_fn\(", 1),
Expand All @@ -93,6 +98,18 @@ def test_given_full_profile_when_scanning_then_uses_each_construct(
assert sum(1 for text in models(full()).values() if pattern.search(text)) >= minimum_models


def test_given_full_profile_when_scanning_then_writes_a_few_hundred_python_assets() -> None:
files = full()
modules = [path for path in files if path.startswith("python/exports/")]
assets = sum(
files[path].count("\n@asset(") + files[path].startswith("@asset(") for path in modules
)

assert 350 <= assets <= 500
assert len(modules) >= 40
assert all("from python.shared.export_feed import" in files[path] for path in modules)


def test_given_known_divergence_constructs_when_scanning_then_models_are_tagged() -> None:
for path, text in models(full()).items():
if re.search(r"\bHASH\(|TIMESTAMP_LTZ|CURRENT_TIMESTAMP\(\)", text.split(");", 1)[1]):
Expand Down
2 changes: 2 additions & 0 deletions crates/sqlbuild-analysis/src/analysis_route/constants.rs
Original file line number Diff line number Diff line change
Expand Up @@ -490,3 +490,5 @@ pub(crate) const CLICKHOUSE_DECIMAL_WIDTHS: [(&str, u32); 4] = [
("Decimal128", 38),
("Decimal256", 76),
];
/// Snowflake's precision of a fixed-point type written without one.
pub(crate) const SNOWFLAKE_DEFAULT_PRECISION: i64 = 38;
1 change: 1 addition & 0 deletions crates/sqlbuild-analysis/src/analysis_route/main/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ pub mod native_gap;
pub(crate) mod native_outcomes;
pub mod native_revalidation;
pub mod query_expression_ranges;
pub mod same_column_type;
pub mod schema_column;
pub mod skipped_analysis;
pub mod unapplied_nullability_rules;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
//! Whether a declared column type and a warehouse-listed one name the type the engine stores.

use crate::analysis_route::constants::{SNOWFLAKE_DEFAULT_PRECISION, SNOWFLAKE_DIALECT};
use crate::analysis_route::main::analysis_route::analysis_route;
use crate::analysis_route::models::AnalysisRoute;
use crate::type_system::main::same_type::same_type;
use crate::type_system::models::{NormalizedType, PythonInteger, TypeFamily};

/// Whether `declared` and `listed` are one stored type in `dialect`, its defaults and synonyms met.
#[must_use]
pub fn same_column_type(dialect: Option<&str>, declared: &str, listed: &str) -> bool {
if declared.trim().eq_ignore_ascii_case(listed.trim()) {
return true;
}
let AnalysisRoute::Native(native) = analysis_route(dialect) else {
return false;
};
let (Some(declared), Some(listed)) = (
native.normalize_type(declared),
native.normalize_type(listed),
) else {
return false;
};
let (declared, listed): (NormalizedType, NormalizedType) = match native.name() {
SNOWFLAKE_DIALECT => (snowflake_defaults(declared), snowflake_defaults(listed)),
_ => (declared, listed),
};
native.same_type(&declared, &listed) || same_type(&declared, &listed, native.name())
}

/// A Snowflake fixed-point type written without precision as the `NUMBER(38,0)` it stores.
fn snowflake_defaults(normalized: NormalizedType) -> NormalizedType {
if normalized.family != TypeFamily::Decimal || normalized.precision.is_some() {
return normalized;
}
let precision: PythonInteger = PythonInteger::from(SNOWFLAKE_DEFAULT_PRECISION);
let scale: PythonInteger = normalized
.scale
.clone()
.unwrap_or_else(|| PythonInteger::from(0));
NormalizedType {
normalized_name: format!("DECIMAL({precision},{scale})"),
family: TypeFamily::Decimal,
precision: Some(precision),
scale: Some(scale),
length: None,
}
}
2 changes: 2 additions & 0 deletions crates/sqlbuild-analysis/src/analysis_route/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ mod test_native_outcome;
mod test_native_revalidation;
#[path = "tests/test_postgres_dialect.rs"]
mod test_postgres_dialect;
#[path = "tests/test_same_column_type.rs"]
mod test_same_column_type;
#[path = "tests/test_type_spelling.rs"]
mod test_type_spelling;
#[path = "tests/test_types.rs"]
Expand Down
Loading
Loading