Skip to content

Commit 1fb37f7

Browse files
authored
feat(plan): plan from synced state, depend on jobs or tags, and faster plans (#1580)
* feat(plan): time project discovery, rules and the warehouse connection `sqb plan --timings`, and the timings a plan document carries, left discovering the project, evaluating the configured rules and opening the warehouse connection inside Other. Record them as their own steps, "Discovering the project", "Evaluating rules" and "Connecting to the warehouse", so Other is only time no step covers. A location's plan from its synced state uses the project it already discovered, compiled and checked, and opens no connection, so those steps do not appear there. * perf(plan): list each schema's tables and views at the same time Planning listed every schema with SHOW TABLES and then SHOW VIEWS, one after the other. A schema-scoped reader can now hand the catalog its listing as independent reads and combine their results, and Snowflake does so, so the catalog runs every schema's TABLES and VIEWS reads together within the adapter's inspection concurrency. Builds keep the listing path that records schema listing versions. The schema existence check behind a failed SHOW is remembered per plan; callers asking while it runs now wait and share its answer, so two failed reads of a missing schema check it once. * perf(plan): read cursor bounds in batched UNION ALL statements A live plan read every incremental model's cursor bounds with its own statement, so a project with hundreds of cursor models ran hundreds of round trips, eight at a time on Snowflake. Read up to 50 physical cursor reads in one UNION ALL statement, one row per read tagged with its position, at the adapter's inspection concurrency. A read without a minimum repeats its maximum, so no branch casts a bare NULL. When a batch fails, its reads run their own statements, so one unreadable relation is still the only one reported unavailable. A single read keeps its single statement. The eligible-maximum reads stay one statement each: they filter rows rather than read metadata, and batched they ran one after another inside a single statement. * fix(plan): rebind sources only on new column evidence on case-sensitive dialects On Snowflake without QUOTED_IDENTIFIERS_IGNORE_CASE, and on Postgres, every source with inspected warehouse columns counted as changed, so planning analysed every selected model downstream of a source again, even when the warehouse columns matched the declared ones. The blanket rule came with physical rebinding, where warehouse names are bound with their exact case: a column stored as lowercase "id" is not the unquoted id a model selects. Compare each listed column's bound identity (quoted names exactly, unquoted names folded the way the dialect folds them) with the compile-time shape instead, so that case still rebinds and rejects the reader, and matching shapes add no work. * perf(plan): look up migration origins without rescanning every fingerprint A plan with declared model migrations read the newest fingerprint of each origin by rebuilding a lookup key from every model fingerprint in the state schema, twice per migration, and resolved each origin name by walking every project model. On a 10K-model project with about a hundred `migrate_from` models that was close to five seconds of a location's warm preview. Read each schema's fingerprint keys once, when the schema's fingerprints are inspected, index them by built relation name and by node name, and resolve origin names through one name index per plan. Lookups choose the same fingerprint as before: the candidates keep their stored order, so ties resolve the same way. The 10K preview guard now also bounds the warm preview at 10 seconds. * test(perf): guard live plans of the 10K project and give it real planning workload The 10K guards timed compile and a location's preview from its synced state, but nothing timed a live `sqb plan`, and the fixture had a few dozen incremental models, five Python files and no migrations. A live plan that analysed every model again against inspected source columns, and read each cursor bound with its own statement, went unnoticed. Each preview case now also plans the project in-process against an offline Snowflake warehouse loaded from the same synced catalog, one source in fifty carrying an undeclared column. It bounds the models analysed again against source columns, the warehouse statements sent (cursor bounds read in full batches) and the live plan's time as a multiple of the warm preview. The fake warehouse answers the column description of expression sources. Generator 4 adds an append-incremental history of every third timestamped intermediate table (583 models, 112 capping their automatic start, 112 renamed with migrate_from) and a Python export asset for every fifth mart (433 assets in 241 modules sharing one helper module). * fix(plan): compare inspected source column types as the warehouse stores them A source declared `VARCHAR` lists as `VARCHAR(16777216)` on Snowflake, and one declared `INTEGER` or `NUMBER` lists as `NUMBER(38,0)`, so every such source counted as changed and planning analysed every selected model reading it again. Compare a declared type with the listed one through the analysis dialect's engine types instead of their text: synonyms, default string lengths, a timestamp precision left unstated, and Snowflake's bare NUMBER or DECIMAL as NUMBER(38,0) all match. A shorter VARCHAR or a different scale still counts as a change. Dialects without a native analysis module keep the exact text comparison. The 10K live-plan guard's offline warehouse now lists source columns with Snowflake's own spellings. * test(plan): rebind the quoted-case session case on a retyped source column Physical rebinding now analyses a source's readers again only when its listed columns differ from the declared ones, so the session-setting case listed the declared column unchanged and saw nothing to rebind; the plan still stopped on the compile-time B002. List the column with a different length so the case rebinds and checks the setting again. * perf(plan): analyse readers again only when a source's new columns can change them When a source's warehouse columns differed from its declared ones, planning analysed every selected model reading it again, and every model downstream of those. Warehouses routinely carry columns a source does not declare, so on a real project this meant re-analysing about 1,600 models on every plan, almost all of them unchanged. When the listing only adds columns, every declared column present with the type it stores, analyse a reader again only if the new columns can change its result: it projects the source's columns through a star, directly or through CTEs, subqueries and set operations; it binds an unqualified name in a scope where the source sits beside another relation; it names one of the added columns anywhere; or it uses a construct the walk does not follow, or SQL the parser cannot read. Everything else keeps its compile-time analysis. A re-analysed model passes on only the difference its own new shape makes, so its readers are judged the same way, and a model whose shape did not change stops the cascade. A missing or retyped column still re-analyses every reader. On a 10K-model fixture with one source in fifty carrying an extra column, re-analysis drops from 519 models to 52; the guard's expected range follows. * feat(plan): let a target plan from its synced copy of warehouse state Every plan lists and describes the warehouse's schemas live, which is most of a plan's time on a large project, while `sqb dev` already keeps a synced copy of the catalog in the project's control store. A target can now set `plan_from = "synced_state"`, and `--plan-from live|synced_state` on `sqb plan` and `sqb build` overrides it for one invocation. The default stays `live`. Planning from synced state reads the target's mirrored catalog from the control store, then checks it with one INFORMATION_SCHEMA.TABLES read per database: the listing it returns is used as is, so created and dropped tables are seen, and columns come from the copy except for tables whose LAST_DDL is later than the copy observed them (less five minutes for clock skew), which are described again. With more than 128 such tables the schema's columns are read live. Cursor bounds and incremental state are still read live. When the project has no control store, no location syncs the target for the project, the copy is empty or the check fails, the plan reads live and says why on stderr. Otherwise it reports how long the check took and how many tables it described again. Planning never writes to the control store. * test(perf): bound the 10K live plan by the location's publish, not the warm preview The live plan was bounded at six times the warm preview. Once the warm preview got faster, live plans that had not changed came out at 5.3-6.4 times it on a loaded machine, so the ratio failed for reasons that have nothing to do with live planning. Bound the live plan by the location's publish instead, which compiles the same project on the same machine. Measured at 1.22-1.40 across six runs, quiet and loaded; the limit is 1.8. The warm preview keeps its 10 second limit. * feat(python-nodes): depend on a job or a tag Tasks, assets and checks can declare depends_on=job("orders_job") or tag("finance"), each with an optional exclude= of ordinary selectors. Compile resolves them into edges to the set's end nodes: its models, seeds, sources, loaders, tasks and assets, less the exclude, that nothing else in the set reads. Planning, selector expansion, --changed, sqb dag and job group waits then treat them like named dependencies, and a model named and reached through a set is one edge. Compile fails for an unknown job or tag, a set its exclude empties, a loader declaring one, a code location prefix (reserved), a set node reading the dependant's output (naming the nearest reader and the exclude that breaks the cycle), and a job group the new edges would make wait in a loop, as groups wait job by job today; that error names the nodes closing the loop and suggests moving them to another job. Each node's identity records the end nodes it resolved, and the plan lists what it now also waits on, or no longer waits on, since its last build, in text and as wait_changes in --json. Published Python node definitions carry set_dependencies for the UI's lineage to use. * perf(compile): verify a model's typed SQL macro markers once per process A plan of an unchanged project still compiled in 6-7 s with every model's analysis answered by the compile cache: the cache covers analysis, not the per-model work around it. The largest piece was building each model's analysis text, which wraps typed SQL macro calls in markers and parses the whole model to check each marker lands where it parses, up to once per marked call. Every compile did this twice per model, once for the analysis request and once for the compiled model. Keep the verified marker set per (dialect, SQL, calls) for the process, a bounded memo the second read and later compiles in long-lived processes reuse. On a 2,280-model project, compiled-model assembly drops from 0.49 s to 0.05 s per compile. * perf(plan): read target columns for the selection and its upstream only A plan read the warehouse columns of every built model in the project, whatever it selected: on a 2,400-model Snowflake project that is one SHOW COLUMNS IN SCHEMA of 131K rows, measured at 3-29 s, even for a one-model plan. Read columns only for the relations of the selection and what it reads, the models whose changes the plan and its stale warnings detect; an unnarrowed plan still reads them all. Snowflake reads up to 200 relations with concurrent SHOW COLUMNS IN TABLE (about 8 ms a relation at the inspection concurrency, 200 in 1.7 s) instead of from 33, so a small selection no longer lists the schema; the sequential direct path keeps 32. INFORMATION_SCHEMA.COLUMNS with an IN list is not an option there: 7.8 s for 30 tables. Parsing the listing's type JSON is memoised by text: a 131K-row schema holds 135 distinct texts, 0.28 s down to 0.02 s. * chore(fensu): record the intentional mirrors the plan work touches The Python node decorators, two-field key readers and the development location read each touch an existing near-miss cluster; each keeps its own record type or SQL, with the shared work already in one owner. * refactor(graph): publish set dependencies from the project selection Moves the set_dependencies publication into ProjectSelection so the Python extension crate stays within its code-line cap after merging main. * test(goldens): list plan_from among a target's allowed keys
1 parent 4decdcc commit 1fb37f7

178 files changed

Lines changed: 6529 additions & 449 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎benchmarks/duckshop-10k/README.md‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,13 @@ run `python benchmarks/duckshop-10k/generate.py --update-checksum`.
3535

3636
| | Count |
3737
|---|---|
38-
| Models | 10,282 |
39-
| Staging, intermediate, marts and snapshots | 6,149 (59.8%), 1,961 (19.1%), 2,162 (21.0%), 10 |
38+
| Models | 10,840 |
39+
| Staging, intermediate, marts and snapshots | 6,149 (56.7%), 2,519 (23.2%), 2,162 (19.9%), 10 |
40+
| Incremental models | 583, 112 of them with a capped automatic start and 112 renamed |
41+
| Python export assets | 433 in 241 modules sharing one helper module |
4042
| Sources | 2,466 table sources and one expression source |
4143
| Unit tests and scenarios | 288 and 2 |
42-
| Dependency depth | 11 at most; 168 models at depth 7 or more |
44+
| Dependency depth | 11 at most; 170 models at depth 7 or more |
4345

4446
Most models read one to four upstreams; a few dozen wide marts, unions and scorecards read ten or
4547
more. Staging deduplicates change logs with a change-log macro, types three platform layouts
@@ -51,6 +53,13 @@ clickstream events in microbatches and builds customer identity, value and segme
5153
budget variance chain. Marts give every instance, team and region its reports, and publish facts and
5254
dimensions with enforced contracts.
5355

56+
A large project also carries planning workload the layers above barely touch, so the generator adds
57+
it in the proportions such projects have: an incrementally kept history of every third timestamped
58+
intermediate table, one in five of them capping how far ahead its automatic start may sit and one in
59+
five renamed from an older name with `migrate_from`, and a Python export asset for every fifth
60+
published mart, a few per module, all sharing one helper module. A live plan reads a cursor bound for
61+
each incremental model, plans each rename and discovers each Python node.
62+
5463
The models that use `HASH`, `TIMESTAMP_LTZ` or `CURRENT_TIMESTAMP()` are tagged `known_divergence`,
5564
because their results depend on the engine or the session.
5665

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
generator 3 files 10874 sha256 5f3d30b82ebf52f2eb869494f52f77d1699f9a08743febd6ee9ca0cb22ed0611
1+
generator 4 files 11674 sha256 d0012261f902e87a546a04bd8c6552c8a050f9cec6753b5acccc53584bce9d37

‎benchmarks/duckshop-10k/duckshop10k/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,6 @@
44
regenerate checksum.txt, whenever a change alters any generated file.
55
"""
66

7-
GENERATOR_VERSION = "3"
7+
GENERATOR_VERSION = "4"
88
SEED = 20261024
99
PROJECT_NAME = "duckshop_10k"

‎benchmarks/duckshop-10k/duckshop10k/project.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
from duckshop10k.rules_gen import build_rules
1414
from duckshop10k.staging import StagedTable, stage_collector, stage_instance
1515
from duckshop10k.tests_gen import build_scenarios, build_tests
16+
from duckshop10k.workload import build_workload
1617
from duckshop10k.world import Instance, instances, storefront_instances
1718

1819

@@ -89,6 +90,7 @@ def build(profile_name: str = "full") -> dict[str, str]:
8990
build_extras(project, intermediate)
9091
build_tests(project, staging.tables, intermediate)
9192
build_scenarios(project, staging.tables)
93+
build_workload(project)
9294
build_rules(project)
9395
_static_files(project)
9496
project.write("seeds/source_instances.csv", instances_seed(systems))
Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
"""Planning workload a large project carries: incremental history, renamed models and exports.
2+
3+
A live plan of a real project reads a cursor bound for every incremental model, plans a
4+
migration for every renamed model and discovers every Python node. The layers above build
5+
few of each, so this module adds them in proportions such projects have: about one model in
6+
twenty kept incrementally, a hundred of those renamed from an older name, and a few hundred
7+
Python exports of published marts, spread over many modules that share helper code.
8+
"""
9+
10+
from __future__ import annotations
11+
12+
from duckshop10k.core import Column, Model, Project
13+
14+
#: Every this-many timestamped intermediate table gets an incrementally kept copy.
15+
INCREMENT_STRIDE = 3
16+
#: One incremental model in this many caps how far ahead its automatic start may sit.
17+
MAX_AHEAD_EVERY = 5
18+
#: One incremental model in this many was renamed from an older name.
19+
RENAMED_EVERY = 5
20+
#: Every this-many published mart gets a Python export asset.
21+
EXPORT_STRIDE = 5
22+
#: Export assets per Python module, so discovery walks many modules.
23+
EXPORTS_PER_MODULE = 6
24+
_TIMESTAMP = "TIMESTAMP_NTZ"
25+
_INCREMENT_FOLDER = "models/intermediate/increments"
26+
_EXPORT_PACKAGE = "python/exports"
27+
_SHARED_MODULE = "python/shared/export_feed.py"
28+
29+
SHARED_EXPORT_HELPERS = '''"""What every export asset records about the relation it sent."""
30+
31+
from sqlbuild.assets import AssetContext
32+
from sqlbuild.refs import SqlResourceRef
33+
34+
35+
def feed_metadata(ctx: AssetContext, reference: SqlResourceRef, feed: str) -> dict[str, object]:
36+
"""The relation an export read, its row count and the partner feed it went to."""
37+
relation = ctx.relation(reference)
38+
rows = ctx.query(f"SELECT COUNT(*) FROM {relation}").fetchone()[0]
39+
return {"feed": feed, "relation": str(relation), "rows": rows, "format": feed_format(feed)}
40+
41+
42+
def feed_format(feed: str) -> str:
43+
"""Partner feeds named after a region take CSV; every other feed takes Parquet."""
44+
return "csv" if feed.endswith(("_eu", "_us", "_apac", "_mea", "_latam")) else "parquet"
45+
'''
46+
47+
48+
def build_workload(project: Project) -> None:
49+
"""Add the incremental, renamed and Python export nodes to the assembled project."""
50+
51+
increments: list[Model] = _increments(project)
52+
_rename(increments)
53+
_exports(project)
54+
55+
56+
def _timestamp_column(model: Model) -> Column | None:
57+
return next((column for column in model.columns if column.type == _TIMESTAMP), None)
58+
59+
60+
def _is_table(model: Model) -> bool:
61+
return not any(line.startswith("materialized") for line in model.config)
62+
63+
64+
def _increments(project: Project) -> list[Model]:
65+
candidates: list[Model] = sorted(
66+
(
67+
model
68+
for model in project.models.values()
69+
if model.layer == "intermediate"
70+
and _is_table(model)
71+
and _timestamp_column(model) is not None
72+
),
73+
key=lambda model: model.name,
74+
)
75+
added: list[Model] = []
76+
for position, upstream in enumerate(candidates[::INCREMENT_STRIDE]):
77+
cursor: Column = next(column for column in upstream.columns if column.type == _TIMESTAMP)
78+
config: list[str] = [
79+
"materialized incremental",
80+
"incremental_strategy append",
81+
f"cursor {cursor.name}",
82+
"cursor_type timestamp",
83+
"cursor_grain day",
84+
f"cursor_inputs (\n {upstream.name} {cursor.name},\n )",
85+
]
86+
if position % MAX_AHEAD_EVERY == 0:
87+
config.append("cursor_start_max_ahead 1d")
88+
columns: list[Column] = [
89+
Column(column.name, column.type, audits=("not_null",) * (column == cursor))
90+
for column in upstream.columns
91+
]
92+
select: str = ",\n ".join(f"s.{column.name}" for column in columns)
93+
added.append(
94+
project.add(
95+
Model(
96+
name=f"{upstream.name}_history",
97+
folder=f"{_INCREMENT_FOLDER}/{upstream.folder.rsplit('/', 1)[-1]}",
98+
layer="intermediate",
99+
description=(
100+
f"Every row of {upstream.name} kept incrementally by {cursor.name}, so "
101+
"earlier days stay queryable after the source rewrites them"
102+
),
103+
columns=columns,
104+
body=(
105+
"WITH source_rows AS (\n SELECT *\n"
106+
f' FROM __ref("{upstream.name}")\n),\n\n'
107+
f"final AS (\n SELECT\n {select}\n FROM source_rows AS s\n)\n\n"
108+
"SELECT * FROM final"
109+
),
110+
config=config,
111+
refs=[upstream.name],
112+
)
113+
)
114+
)
115+
return added
116+
117+
118+
def _rename(increments: list[Model]) -> None:
119+
for model in increments[::RENAMED_EVERY]:
120+
model.config.append(f"migrate_from {model.name.removesuffix('_history')}_archive")
121+
122+
123+
def _exports(project: Project) -> None:
124+
marts: list[Model] = sorted(
125+
(model for model in project.models.values() if model.layer == "marts" and _is_table(model)),
126+
key=lambda model: (model.folder, model.name),
127+
)
128+
by_folder: dict[str, list[Model]] = {}
129+
for model in marts[::EXPORT_STRIDE]:
130+
by_folder.setdefault(model.folder.rsplit("/", 1)[-1], []).append(model)
131+
project.write(_SHARED_MODULE, SHARED_EXPORT_HELPERS)
132+
for folder, models in sorted(by_folder.items()):
133+
for start in range(0, len(models), EXPORTS_PER_MODULE):
134+
chunk: list[Model] = models[start : start + EXPORTS_PER_MODULE]
135+
module: str = f"{folder}_{start // EXPORTS_PER_MODULE + 1:02d}"
136+
project.write(f"{_EXPORT_PACKAGE}/{module}.py", _export_module(folder, chunk))
137+
138+
139+
def _export_module(folder: str, models: list[Model]) -> str:
140+
assets: list[str] = [
141+
f'@asset(depends_on=model("{model.name}"))\n'
142+
f"def export_{model.name}(ctx: AssetContext):\n"
143+
f' """Send {model.name} to the {folder} partner feed."""\n'
144+
f' metadata = feed_metadata(ctx, model("{model.name}"), "{folder}")\n'
145+
" return ctx.result(metadata=metadata, materialized=True)\n"
146+
for model in models
147+
]
148+
return (
149+
f'"""Exports of the {folder} marts to partner feeds."""\n\n'
150+
"from sqlbuild.assets import AssetContext, asset\n"
151+
"from sqlbuild.refs import model\n\n"
152+
"from python.shared.export_feed import feed_metadata\n\n\n" + "\n\n".join(assets)
153+
)

‎benchmarks/duckshop-10k/tests/test_generator.py‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -39,13 +39,15 @@ def test_given_full_profile_when_building_then_matches_recorded_checksum() -> No
3939
assert checksum_line(full()) == (HERE / "checksum.txt").read_text()
4040

4141

42-
def test_given_full_profile_when_measuring_then_has_about_ten_thousand_models_in_real_mix() -> None:
42+
def test_given_full_profile_when_measuring_then_has_about_eleven_thousand_models_in_real_mix() -> (
43+
None
44+
):
4345
measured = shape(full())
4446

45-
assert 9_900 <= measured.models <= 10_300
46-
assert 58 <= measured.layer_percent("staging") <= 62
47-
assert 18 <= measured.layer_percent("intermediate") <= 21
48-
assert 19 <= measured.layer_percent("marts") + measured.layer_percent("snapshots") <= 23
47+
assert 10_500 <= measured.models <= 11_000
48+
assert 55 <= measured.layer_percent("staging") <= 59
49+
assert 21 <= measured.layer_percent("intermediate") <= 25
50+
assert 18 <= measured.layer_percent("marts") + measured.layer_percent("snapshots") <= 22
4951

5052

5153
def test_given_full_profile_when_measuring_then_fan_in_and_depth_match_the_profile() -> None:
@@ -78,6 +80,9 @@ def test_given_full_profile_when_measuring_then_samples_a_few_hundred_unit_tests
7880
(r"TRY_PARSE_JSON", 100),
7981
(r"\bFILTER\(", 10),
8082
(r"incremental_mode microbatch", 20),
83+
(r"materialized incremental", 500),
84+
(r"cursor_start_max_ahead", 100),
85+
(r"migrate_from", 100),
8186
(r"materialized snapshot", 5),
8287
(r"__udf\(", 2),
8388
(r"__table_fn\(", 1),
@@ -93,6 +98,18 @@ def test_given_full_profile_when_scanning_then_uses_each_construct(
9398
assert sum(1 for text in models(full()).values() if pattern.search(text)) >= minimum_models
9499

95100

101+
def test_given_full_profile_when_scanning_then_writes_a_few_hundred_python_assets() -> None:
102+
files = full()
103+
modules = [path for path in files if path.startswith("python/exports/")]
104+
assets = sum(
105+
files[path].count("\n@asset(") + files[path].startswith("@asset(") for path in modules
106+
)
107+
108+
assert 350 <= assets <= 500
109+
assert len(modules) >= 40
110+
assert all("from python.shared.export_feed import" in files[path] for path in modules)
111+
112+
96113
def test_given_known_divergence_constructs_when_scanning_then_models_are_tagged() -> None:
97114
for path, text in models(full()).items():
98115
if re.search(r"\bHASH\(|TIMESTAMP_LTZ|CURRENT_TIMESTAMP\(\)", text.split(");", 1)[1]):

‎crates/sqlbuild-analysis/src/analysis_route/constants.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -490,3 +490,5 @@ pub(crate) const CLICKHOUSE_DECIMAL_WIDTHS: [(&str, u32); 4] = [
490490
("Decimal128", 38),
491491
("Decimal256", 76),
492492
];
493+
/// Snowflake's precision of a fixed-point type written without one.
494+
pub(crate) const SNOWFLAKE_DEFAULT_PRECISION: i64 = 38;

‎crates/sqlbuild-analysis/src/analysis_route/main/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ pub mod native_gap;
55
pub(crate) mod native_outcomes;
66
pub mod native_revalidation;
77
pub mod query_expression_ranges;
8+
pub mod same_column_type;
89
pub mod schema_column;
910
pub mod skipped_analysis;
1011
pub mod unapplied_nullability_rules;
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
//! Whether a declared column type and a warehouse-listed one name the type the engine stores.
2+
3+
use crate::analysis_route::constants::{SNOWFLAKE_DEFAULT_PRECISION, SNOWFLAKE_DIALECT};
4+
use crate::analysis_route::main::analysis_route::analysis_route;
5+
use crate::analysis_route::models::AnalysisRoute;
6+
use crate::type_system::main::same_type::same_type;
7+
use crate::type_system::models::{NormalizedType, PythonInteger, TypeFamily};
8+
9+
/// Whether `declared` and `listed` are one stored type in `dialect`, its defaults and synonyms met.
10+
#[must_use]
11+
pub fn same_column_type(dialect: Option<&str>, declared: &str, listed: &str) -> bool {
12+
if declared.trim().eq_ignore_ascii_case(listed.trim()) {
13+
return true;
14+
}
15+
let AnalysisRoute::Native(native) = analysis_route(dialect) else {
16+
return false;
17+
};
18+
let (Some(declared), Some(listed)) = (
19+
native.normalize_type(declared),
20+
native.normalize_type(listed),
21+
) else {
22+
return false;
23+
};
24+
let (declared, listed): (NormalizedType, NormalizedType) = match native.name() {
25+
SNOWFLAKE_DIALECT => (snowflake_defaults(declared), snowflake_defaults(listed)),
26+
_ => (declared, listed),
27+
};
28+
native.same_type(&declared, &listed) || same_type(&declared, &listed, native.name())
29+
}
30+
31+
/// A Snowflake fixed-point type written without precision as the `NUMBER(38,0)` it stores.
32+
fn snowflake_defaults(normalized: NormalizedType) -> NormalizedType {
33+
if normalized.family != TypeFamily::Decimal || normalized.precision.is_some() {
34+
return normalized;
35+
}
36+
let precision: PythonInteger = PythonInteger::from(SNOWFLAKE_DEFAULT_PRECISION);
37+
let scale: PythonInteger = normalized
38+
.scale
39+
.clone()
40+
.unwrap_or_else(|| PythonInteger::from(0));
41+
NormalizedType {
42+
normalized_name: format!("DECIMAL({precision},{scale})"),
43+
family: TypeFamily::Decimal,
44+
precision: Some(precision),
45+
scale: Some(scale),
46+
length: None,
47+
}
48+
}

‎crates/sqlbuild-analysis/src/analysis_route/tests.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@ mod test_native_outcome;
2424
mod test_native_revalidation;
2525
#[path = "tests/test_postgres_dialect.rs"]
2626
mod test_postgres_dialect;
27+
#[path = "tests/test_same_column_type.rs"]
28+
mod test_same_column_type;
2729
#[path = "tests/test_type_spelling.rs"]
2830
mod test_type_spelling;
2931
#[path = "tests/test_types.rs"]

0 commit comments

Comments
 (0)