Skip to content

Commit ca787ea

Browse files
authored
refactor: move the output layer of the Python bindings into sqlbuild-python-output (#1541)
sqlbuild-python is still over the crate size limit. The output layer (build progress reporting, build result output and its documents, output capture and invocation records, cost telemetry, compute log storage and the Recall session and its stores) moves into a new crate above sqlbuild-python-planner and beside sqlbuild-python-compile. The Recall query command and its rendering stay in sqlbuild-python with the other commands, as do event exporting and the cost report, which reach code that stays there. Each moved area's registration stays too. Old paths are deleted and callers updated; there are no re-exports. main/ entries hold the bodies they used to forward to. The compute log storage registers its own class, so its area has an entry, and the Recall session takes the release version from its sqlbuild-python callers, since CARGO_PKG_VERSION is the fixed workspace version outside that crate. The sqlbuild-python cap drops to its new size, XSB072/XSB073 budgets are re-keyed with unchanged totals, and the native field fallback list follows the moved files.
1 parent 172c09a commit ca787ea

201 files changed

Lines changed: 6082 additions & 5049 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 21 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ members = [
2828
"crates/sqlbuild-python-project",
2929
"crates/sqlbuild-python-planner",
3030
"crates/sqlbuild-python-compile",
31+
"crates/sqlbuild-python-output",
3132
"crates/sqlbuild-python",
3233
]
3334
resolver = "2"
@@ -65,9 +66,10 @@ order = [
6566
"sqlbuild-python-project",
6667
"sqlbuild-python-planner",
6768
"sqlbuild-python-compile",
69+
"sqlbuild-python-output",
6870
"sqlbuild-python",
6971
]
70-
python-boundary = ["sqlbuild-python-base", "sqlbuild-python-front", "sqlbuild-python-project", "sqlbuild-python-planner", "sqlbuild-python-compile", "sqlbuild-python"]
72+
python-boundary = ["sqlbuild-python-base", "sqlbuild-python-front", "sqlbuild-python-project", "sqlbuild-python-planner", "sqlbuild-python-compile", "sqlbuild-python-output", "sqlbuild-python"]
7173
polyglot-floor = "sqlbuild-analysis"
7274

7375
[workspace.package]
@@ -192,6 +194,7 @@ sqlbuild-python-front = { path = "crates/sqlbuild-python-front" }
192194
sqlbuild-python-project = { path = "crates/sqlbuild-python-project" }
193195
sqlbuild-python-planner = { path = "crates/sqlbuild-python-planner" }
194196
sqlbuild-python-compile = { path = "crates/sqlbuild-python-compile" }
197+
sqlbuild-python-output = { path = "crates/sqlbuild-python-output" }
195198
sqlbuild-pysource = { path = "crates/sqlbuild-pysource" }
196199
sqlbuild-recall = { path = "crates/sqlbuild-recall" }
197200
sqlbuild-scopes = { path = "crates/sqlbuild-scopes" }

‎crates/README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,14 +33,15 @@ tested and benchmarked with `cargo test` alone.
3333
| `sqlbuild-python-project` | The compiled project as PyO3 bindings, on top of `sqlbuild-python-front`: the project host and the Python views of compiled models, inputs and resources, the compiled-project snapshot, analysis classes and the analysis session, the project graph, references, lineage, fingerprints, contracts and semantic shapes, audit rendering, and the planner's identities, scope, selection, phases, fixture strategy and SQL test planning. |
3434
| `sqlbuild-python-planner` | The plan layer as PyO3 bindings, on top of `sqlbuild-python-project`: execution plans and their entries, planner phases, change detection, migrations, reference and cursor resolution, scenarios, strategy and warehouse inspection, plus freshness, consumption, microbatches, observations, lifecycle events and plan output. |
3535
| `sqlbuild-python-compile` | The compile layer as PyO3 bindings, on top of `sqlbuild-python-project` and beside `sqlbuild-python-planner`: the compile command and driver, input loops and compile-input attachment, project assembly, analysis and the analysis session, Rules, lineage, semantic checks, compile deltas and outputs, the DAG artifact, manifest, frontier, offline scope facts, SQL test glue and auditing. |
36-
| `sqlbuild-python` | The `_native` module: its Python classes and functions and the conversions from Python objects, on top of `sqlbuild-python-compile`, `sqlbuild-python-planner`, `sqlbuild-python-project`, `sqlbuild-python-front` and `sqlbuild-python-base`. |
36+
| `sqlbuild-python-output` | The output layer as PyO3 bindings, on top of `sqlbuild-python-planner` and beside `sqlbuild-python-compile`: build progress reporting, build result output and its documents, output capture and invocation records, cost telemetry, compute log storage and the Recall session and its stores. |
37+
| `sqlbuild-python` | The `_native` module: its Python classes and functions and the conversions from Python objects, on top of `sqlbuild-python-output`, `sqlbuild-python-compile`, `sqlbuild-python-planner`, `sqlbuild-python-project`, `sqlbuild-python-front` and `sqlbuild-python-base`. |
3738

3839
## Layering
3940

4041
Dependencies point one way, from the top of this graph to the bottom:
4142

4243
```text
43-
sqlbuild-python -> sqlbuild-python-compile -> sqlbuild-python-planner -> sqlbuild-python-project -> sqlbuild-python-front -> sqlbuild-python-base
44+
sqlbuild-python -> sqlbuild-python-output -> sqlbuild-python-compile -> sqlbuild-python-planner -> sqlbuild-python-project -> sqlbuild-python-front -> sqlbuild-python-base
4445
-> sqlbuild-project -> sqlbuild-refactor -> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-attachments
4546
-> sqlbuild-model-config -> sqlbuild-scopes -> sqlbuild-render -> sqlbuild-discovery -> sqlbuild-host
4647
-> sqlbuild-config -> sqlbuild-sqltext -> sqlbuild-cache -> sqlbuild-pysource -> sqlbuild-state -> sqlbuild-cursor -> sqlbuild-core
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
[package]
2+
name = "sqlbuild-python-output"
3+
description = "The output layer of the sqlbuild._native bindings: build progress, result output, output capture, cost telemetry, compute logs and the Recall session."
4+
version.workspace = true
5+
edition.workspace = true
6+
license.workspace = true
7+
publish.workspace = true
8+
rust-version.workspace = true
9+
10+
[dependencies]
11+
pyo3.workspace = true
12+
serde_json.workspace = true
13+
sqlbuild-analysis.workspace = true
14+
sqlbuild-cache.workspace = true
15+
sqlbuild-cli.workspace = true
16+
sqlbuild-config.workspace = true
17+
sqlbuild-core.workspace = true
18+
sqlbuild-host.workspace = true
19+
sqlbuild-orchestration.workspace = true
20+
sqlbuild-python-base.workspace = true
21+
sqlbuild-python-front.workspace = true
22+
sqlbuild-python-project.workspace = true
23+
sqlbuild-python-planner.workspace = true
24+
sqlbuild-recall.workspace = true
25+
26+
[lints]
27+
workspace = true
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
//! Aggregate live state and plan-ordered result rows of standalone concurrent audits.
2+
3+
use std::collections::HashMap;
4+
5+
use pyo3::prelude::{Bound, Py, PyAny, PyResult};
6+
use sqlbuild_core::nested_progress::main::audit_resource_id::audit_resource_id;
7+
8+
use crate::build_progress::_helpers::result_text::optional;
9+
use crate::build_progress::_helpers::result_text::text;
10+
11+
/// The identity an audit plan entry or result reports to the progress projector.
12+
pub(crate) fn resource_id(value: &Bound<'_, PyAny>, name_field: &str) -> PyResult<String> {
13+
let kind: Option<String> = match optional(value, "attached_target_kind")? {
14+
Some(kind) => Some(kind),
15+
None => Some(text(value, "attachment_kind")?),
16+
};
17+
Ok(audit_resource_id(
18+
&text(value, name_field)?,
19+
kind.as_deref().unwrap_or(""),
20+
(
21+
optional(value, "attached_target_name")?.as_deref(),
22+
optional(value, "attached_column_name")?.as_deref(),
23+
),
24+
))
25+
}
26+
27+
/// Mutable reporter state; only held briefly and never across Python calls.
28+
#[derive(Debug, Default)]
29+
pub(crate) struct ReporterState {
30+
pub(crate) completed_by_index: HashMap<usize, Option<Py<PyAny>>>,
31+
pub(crate) next_index: usize,
32+
pub(crate) running: usize,
33+
pub(crate) completed: usize,
34+
pub(crate) current_group: Option<String>,
35+
pub(crate) closed: bool,
36+
pub(crate) cursor_hidden: bool,
37+
pub(crate) on_result: Option<Py<PyAny>>,
38+
}
Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
//! Live progress rows, completion rows and runtime diagnostics of `sqb build`.
2+
3+
use std::collections::HashMap;
4+
5+
use pyo3::prelude::{Bound, Py, PyAny, PyAnyMethods, PyResult};
6+
use sqlbuild_core::build_progress::models::{SchedulerStateText, StatementText, TestResultText};
7+
8+
use crate::build_progress::_helpers::result_text::optional;
9+
use crate::build_progress::_helpers::result_text::text;
10+
use sqlbuild_python_base::records::main::fields::field::field as native_field;
11+
12+
pub(crate) const RUNTIME_LOGGER: &str = "sqlbuild.executor.runtime";
13+
pub(crate) const PASS_OUTCOME: &str = "pass";
14+
pub(crate) const SCHEDULER_DIAGNOSTIC_LIMIT: usize = 50;
15+
pub(crate) const QUERY_DIAGNOSTIC_LIMIT: usize = 25;
16+
17+
/// Which executor result type a completion reports.
18+
#[derive(Clone, Copy, PartialEq, Eq)]
19+
pub(crate) enum ResultKind {
20+
Model,
21+
Seed,
22+
Function,
23+
Load,
24+
PythonNode,
25+
SqlTest,
26+
Other,
27+
}
28+
29+
pub(crate) fn result_kind(result: &Bound<'_, PyAny>) -> PyResult<ResultKind> {
30+
for (field, kind) in [
31+
("model_name", ResultKind::Model),
32+
("seed_name", ResultKind::Seed),
33+
("function_name", ResultKind::Function),
34+
("loader_name", ResultKind::Load),
35+
("node_name", ResultKind::PythonNode),
36+
("test_name", ResultKind::SqlTest),
37+
] {
38+
if result.hasattr(field)? {
39+
return Ok(kind);
40+
}
41+
}
42+
Ok(ResultKind::Other)
43+
}
44+
45+
fn sql_test_resource_id(result: &Bound<'_, PyAny>) -> PyResult<String> {
46+
let name: String = text(result, "test_name")?;
47+
let path: Bound<'_, PyAny> = native_field(result, "source_path")?;
48+
let block: Bound<'_, PyAny> = native_field(result, "block_index")?;
49+
let case: Bound<'_, PyAny> = native_field(result, "case_name")?;
50+
if path.is_none() || block.is_none() || case.is_none() {
51+
return Ok(format!("sql_test:{name}"));
52+
}
53+
Ok(format!(
54+
"sql_test:{}:{}:{}",
55+
path.call_method0("as_posix")?.str()?,
56+
block.str()?,
57+
case.str()?
58+
))
59+
}
60+
61+
/// The resource name and projector identity of a completion.
62+
pub(crate) fn resource_identity(result: &Bound<'_, PyAny>) -> PyResult<Option<(String, String)>> {
63+
Ok(match result_kind(result)? {
64+
ResultKind::Model => {
65+
let name: String = text(result, "model_name")?;
66+
Some((name.clone(), format!("model:{name}")))
67+
}
68+
ResultKind::Seed => {
69+
let name: String = text(result, "seed_name")?;
70+
Some((name.clone(), format!("seed:{name}")))
71+
}
72+
ResultKind::Function => {
73+
let name: String = text(result, "function_name")?;
74+
let kind: String = text(result, "function_kind")?;
75+
Some((name.clone(), format!("{kind}:{name}")))
76+
}
77+
ResultKind::Load => {
78+
let name: String = text(result, "source_name")?;
79+
Some((name.clone(), format!("source:{name}")))
80+
}
81+
ResultKind::PythonNode => {
82+
let name: String = text(result, "node_name")?;
83+
let kind: String = text(result, "kind")?;
84+
Some((name.clone(), format!("{kind}:{name}")))
85+
}
86+
ResultKind::SqlTest => Some((text(result, "test_name")?, sql_test_resource_id(result)?)),
87+
ResultKind::Other => None,
88+
})
89+
}
90+
91+
/// Mutable callback state; only held briefly and never across Python calls.
92+
#[derive(Debug, Default)]
93+
pub(crate) struct CallbackState {
94+
pub(crate) counter: usize,
95+
pub(crate) node_name: String,
96+
pub(crate) node_kind: String,
97+
pub(crate) sub_message: String,
98+
pub(crate) frame: usize,
99+
pub(crate) line_active: bool,
100+
pub(crate) cursor_hidden: bool,
101+
pub(crate) stop_event: Option<Py<PyAny>>,
102+
pub(crate) thread: Option<Py<PyAny>>,
103+
pub(crate) tests_by_model: HashMap<String, Vec<TestResultText>>,
104+
pub(crate) scheduler_count: usize,
105+
pub(crate) scheduler_omitted: usize,
106+
pub(crate) last_scheduler: Option<SchedulerStateText>,
107+
pub(crate) query_counts: HashMap<String, usize>,
108+
pub(crate) query_omitted: usize,
109+
pub(crate) last_query: Option<StatementText>,
110+
}
111+
112+
pub(crate) fn scheduler_state(state: &Bound<'_, PyAny>) -> PyResult<SchedulerStateText> {
113+
Ok(SchedulerStateText {
114+
running: native_field(state, "running")?.extract()?,
115+
ready: native_field(state, "ready")?.extract()?,
116+
waiting: native_field(state, "waiting")?.extract()?,
117+
limit: native_field(state, "limit")?.extract()?,
118+
aborted: native_field(state, "aborted")?.extract()?,
119+
})
120+
}
121+
122+
pub(crate) fn statement(telemetry: &Bound<'_, PyAny>) -> PyResult<StatementText> {
123+
Ok(StatementText {
124+
query_id: optional(telemetry, "query_id")?,
125+
status: text(telemetry, "status")?,
126+
elapsed_seconds: native_field(telemetry, "elapsed_seconds")?.extract()?,
127+
resource_type: text(telemetry, "resource_type")?,
128+
resource_name: text(telemetry, "resource_name")?,
129+
phase: text(telemetry, "phase")?,
130+
})
131+
}
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
//! The `sqb build` completion footer.
2+
3+
use crate::build_progress::main::format_build_footer_with::format_build_footer_with;
4+
use pyo3::prelude::{Bound, PyAny, PyResult};
5+
use pyo3::pyfunction;
6+
7+
use crate::build_progress::_helpers::result_text::build_text;
8+
9+
/// Render the build completion line, warnings, failures and skipped rollup.
10+
#[pyfunction]
11+
#[pyo3(signature = (*, result, elapsed, use_color, python_node_results = None))]
12+
pub(crate) fn format_build_footer(
13+
result: &Bound<'_, PyAny>,
14+
elapsed: f64,
15+
use_color: bool,
16+
python_node_results: Option<&Bound<'_, PyAny>>,
17+
) -> PyResult<String> {
18+
format_build_footer_with(
19+
&build_text(result)?,
20+
(elapsed, use_color),
21+
python_node_results,
22+
Vec::new(),
23+
)
24+
}

crates/sqlbuild-python/src/bindings/_helpers/build_progress/getters.rs renamed to crates/sqlbuild-python-output/src/build_progress/_helpers/getters.rs

Lines changed: 3 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -3,24 +3,10 @@
33
use pyo3::IntoPyObjectExt;
44
use pyo3::prelude::{Bound, PyAny, PyResult, Python};
55

6-
use crate::bindings::_helpers::build_progress::build_callbacks::BuildProgressCallbacks;
6+
use crate::build_progress::models::BuildProgressCallbacks;
77
use sqlbuild_python_base::build_progress::models::TransientLineCoordinator;
8-
use sqlbuild_python_base::records::main::registry::class_of::class_of;
9-
use sqlbuild_python_base::records::models::GetterEntry;
108

11-
/// The getter readers of this module's native pyclasses.
12-
pub(crate) const GETTERS: &[GetterEntry] = &[
13-
GetterEntry {
14-
class: class_of::<BuildProgressCallbacks>,
15-
read: read_build_progress_callbacks,
16-
},
17-
GetterEntry {
18-
class: class_of::<TransientLineCoordinator>,
19-
read: read_transient_line_coordinator,
20-
},
21-
];
22-
23-
fn read_build_progress_callbacks<'py>(
9+
pub(crate) fn read_build_progress_callbacks<'py>(
2410
value: &Bound<'py, PyAny>,
2511
name: &str,
2612
) -> Option<PyResult<Bound<'py, PyAny>>> {
@@ -37,7 +23,7 @@ fn read_build_progress_callbacks<'py>(
3723
})
3824
}
3925

40-
fn read_transient_line_coordinator<'py>(
26+
pub(crate) fn read_transient_line_coordinator<'py>(
4127
value: &Bound<'py, PyAny>,
4228
name: &str,
4329
) -> Option<PyResult<Bound<'py, PyAny>>> {
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
pub(crate) mod audit_progress;
2+
pub(crate) mod build_callbacks;
3+
pub(crate) mod footer;
4+
pub(crate) mod getters;
5+
pub(crate) mod nested_progress;
6+
pub(crate) mod plan_facts;
7+
pub(crate) mod progress_reporters;
8+
pub(crate) mod progress_text;
9+
pub(crate) mod result_text;

0 commit comments

Comments
 (0)