Skip to content

Commit cf1da94

Browse files
committed
wip: replay the output-layer move (#1541) on the large-project tree
Moves the output layer from the bindings crate into the new sqlbuild-python-output crate, as the slice did, keeping the large-project tree's behaviour. The tree's registration bodies match the slice's base, so the slice's registrations carry over unchanged; main's statement-report and setup-context imports, which the tree does not have, are not carried over. Validation: cargo check --workspace --all-targets passes with no warnings, the layout matches the slice, and every line the tree added to a moved file is still present. Nothing is built or tested yet. fensu reports 349 faults, unchanged from the previous slice.
1 parent 4e8afee commit cf1da94

204 files changed

Lines changed: 6096 additions & 5166 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 21 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ members = [
2727
"crates/sqlbuild-python-project",
2828
"crates/sqlbuild-python-planner",
2929
"crates/sqlbuild-python-compile",
30+
"crates/sqlbuild-python-output",
3031
"crates/sqlbuild-python",
3132
]
3233
resolver = "2"
@@ -63,9 +64,10 @@ order = [
6364
"sqlbuild-python-project",
6465
"sqlbuild-python-planner",
6566
"sqlbuild-python-compile",
67+
"sqlbuild-python-output",
6668
"sqlbuild-python",
6769
]
68-
python-boundary = ["sqlbuild-python-base", "sqlbuild-python-front", "sqlbuild-python-project", "sqlbuild-python-planner", "sqlbuild-python-compile", "sqlbuild-python"]
70+
python-boundary = ["sqlbuild-python-base", "sqlbuild-python-front", "sqlbuild-python-project", "sqlbuild-python-planner", "sqlbuild-python-compile", "sqlbuild-python-output", "sqlbuild-python"]
6971
polyglot-floor = "sqlbuild-analysis"
7072

7173
[workspace.package]
@@ -189,6 +191,7 @@ sqlbuild-python-front = { path = "crates/sqlbuild-python-front" }
189191
sqlbuild-python-project = { path = "crates/sqlbuild-python-project" }
190192
sqlbuild-python-planner = { path = "crates/sqlbuild-python-planner" }
191193
sqlbuild-python-compile = { path = "crates/sqlbuild-python-compile" }
194+
sqlbuild-python-output = { path = "crates/sqlbuild-python-output" }
192195
sqlbuild-pysource = { path = "crates/sqlbuild-pysource" }
193196
sqlbuild-recall = { path = "crates/sqlbuild-recall" }
194197
sqlbuild-scopes = { path = "crates/sqlbuild-scopes" }

‎crates/README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,14 +32,15 @@ tested and benchmarked with `cargo test` alone.
3232
| `sqlbuild-python-project` | The compiled project as PyO3 bindings, on top of `sqlbuild-python-front`: the project host and the Python views of compiled models, inputs and resources, the compiled-project snapshot, analysis classes and the analysis session, the project graph, references, lineage, fingerprints, contracts and semantic shapes, audit rendering, and the planner's identities, scope, selection, phases, fixture strategy and SQL test planning. |
3333
| `sqlbuild-python-planner` | The plan layer as PyO3 bindings, on top of `sqlbuild-python-project`: execution plans and their entries, planner phases, change detection, migrations, reference and cursor resolution, scenarios, strategy and warehouse inspection, plus freshness, consumption, microbatches, observations, lifecycle events and plan output. |
3434
| `sqlbuild-python-compile` | The compile layer as PyO3 bindings, on top of `sqlbuild-python-project` and beside `sqlbuild-python-planner`: the compile command and driver, input loops and compile-input attachment, project assembly, analysis and the analysis session, Rules, lineage, semantic checks, compile deltas and outputs, the DAG artifact, manifest, frontier, offline scope facts, SQL test glue and auditing. |
35-
| `sqlbuild-python` | The `_native` module: its Python classes and functions and the conversions from Python objects, on top of `sqlbuild-python-compile`, `sqlbuild-python-planner`, `sqlbuild-python-project`, `sqlbuild-python-front` and `sqlbuild-python-base`. |
35+
| `sqlbuild-python-output` | The output layer as PyO3 bindings, on top of `sqlbuild-python-planner` and beside `sqlbuild-python-compile`: build progress reporting, build result output and its documents, output capture and invocation records, cost telemetry, compute log storage and the Recall session and its stores. |
36+
| `sqlbuild-python` | The `_native` module: its Python classes and functions and the conversions from Python objects, on top of `sqlbuild-python-output`, `sqlbuild-python-compile`, `sqlbuild-python-planner`, `sqlbuild-python-project`, `sqlbuild-python-front` and `sqlbuild-python-base`. |
3637

3738
## Layering
3839

3940
Dependencies point one way, from the top of this graph to the bottom:
4041

4142
```text
42-
sqlbuild-python -> sqlbuild-python-compile -> sqlbuild-python-planner -> sqlbuild-python-project -> sqlbuild-python-front -> sqlbuild-python-base
43+
sqlbuild-python -> sqlbuild-python-output -> sqlbuild-python-compile -> sqlbuild-python-planner -> sqlbuild-python-project -> sqlbuild-python-front -> sqlbuild-python-base
4344
-> sqlbuild-project -> sqlbuild-refactor -> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-attachments
4445
-> sqlbuild-model-config -> sqlbuild-scopes -> sqlbuild-render -> sqlbuild-discovery -> sqlbuild-host
4546
-> sqlbuild-config -> sqlbuild-sqltext -> sqlbuild-cache -> sqlbuild-pysource -> sqlbuild-state -> sqlbuild-cursor -> sqlbuild-core
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
[package]
2+
name = "sqlbuild-python-output"
3+
description = "The output layer of the sqlbuild._native bindings: build progress, result output, output capture, cost telemetry, compute logs and the Recall session."
4+
version.workspace = true
5+
edition.workspace = true
6+
license.workspace = true
7+
publish.workspace = true
8+
rust-version.workspace = true
9+
10+
[dependencies]
11+
pyo3.workspace = true
12+
serde_json.workspace = true
13+
sqlbuild-analysis.workspace = true
14+
sqlbuild-cache.workspace = true
15+
sqlbuild-cli.workspace = true
16+
sqlbuild-config.workspace = true
17+
sqlbuild-core.workspace = true
18+
sqlbuild-host.workspace = true
19+
sqlbuild-orchestration.workspace = true
20+
sqlbuild-python-base.workspace = true
21+
sqlbuild-python-front.workspace = true
22+
sqlbuild-python-project.workspace = true
23+
sqlbuild-python-planner.workspace = true
24+
sqlbuild-recall.workspace = true
25+
26+
[lints]
27+
workspace = true
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
//! Aggregate live state and plan-ordered result rows of standalone concurrent audits.
2+
3+
use std::collections::HashMap;
4+
5+
use pyo3::prelude::{Bound, Py, PyAny, PyResult};
6+
use sqlbuild_core::nested_progress::main::audit_resource_id::audit_resource_id;
7+
8+
use crate::build_progress::_helpers::result_text::optional;
9+
use crate::build_progress::_helpers::result_text::text;
10+
11+
/// The identity an audit plan entry or result reports to the progress projector.
12+
pub(crate) fn resource_id(value: &Bound<'_, PyAny>, name_field: &str) -> PyResult<String> {
13+
let kind: Option<String> = match optional(value, "attached_target_kind")? {
14+
Some(kind) => Some(kind),
15+
None => Some(text(value, "attachment_kind")?),
16+
};
17+
Ok(audit_resource_id(
18+
&text(value, name_field)?,
19+
kind.as_deref().unwrap_or(""),
20+
(
21+
optional(value, "attached_target_name")?.as_deref(),
22+
optional(value, "attached_column_name")?.as_deref(),
23+
),
24+
))
25+
}
26+
27+
/// Mutable reporter state; only held briefly and never across Python calls.
28+
#[derive(Debug, Default)]
29+
pub(crate) struct ReporterState {
30+
pub(crate) completed_by_index: HashMap<usize, Option<Py<PyAny>>>,
31+
pub(crate) next_index: usize,
32+
pub(crate) running: usize,
33+
pub(crate) completed: usize,
34+
pub(crate) current_group: Option<String>,
35+
pub(crate) closed: bool,
36+
pub(crate) cursor_hidden: bool,
37+
pub(crate) on_result: Option<Py<PyAny>>,
38+
}
Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
//! Live progress rows, completion rows and runtime diagnostics of `sqb build`.
2+
3+
use std::collections::HashMap;
4+
5+
use pyo3::prelude::{Bound, Py, PyAny, PyAnyMethods, PyResult};
6+
use sqlbuild_core::build_progress::models::{SchedulerStateText, StatementText, TestResultText};
7+
8+
use crate::build_progress::_helpers::result_text::optional;
9+
use crate::build_progress::_helpers::result_text::text;
10+
use sqlbuild_python_base::records::main::fields::field::field as native_field;
11+
12+
pub(crate) const RUNTIME_LOGGER: &str = "sqlbuild.executor.runtime";
13+
pub(crate) const PASS_OUTCOME: &str = "pass";
14+
pub(crate) const SCHEDULER_DIAGNOSTIC_LIMIT: usize = 50;
15+
pub(crate) const QUERY_DIAGNOSTIC_LIMIT: usize = 25;
16+
17+
/// Which executor result type a completion reports.
18+
#[derive(Clone, Copy, PartialEq, Eq)]
19+
pub(crate) enum ResultKind {
20+
Model,
21+
Seed,
22+
Function,
23+
Load,
24+
PythonNode,
25+
SqlTest,
26+
Other,
27+
}
28+
29+
pub(crate) fn result_kind(result: &Bound<'_, PyAny>) -> PyResult<ResultKind> {
30+
for (field, kind) in [
31+
("model_name", ResultKind::Model),
32+
("seed_name", ResultKind::Seed),
33+
("function_name", ResultKind::Function),
34+
("loader_name", ResultKind::Load),
35+
("node_name", ResultKind::PythonNode),
36+
("test_name", ResultKind::SqlTest),
37+
] {
38+
if result.hasattr(field)? {
39+
return Ok(kind);
40+
}
41+
}
42+
Ok(ResultKind::Other)
43+
}
44+
45+
fn sql_test_resource_id(result: &Bound<'_, PyAny>) -> PyResult<String> {
46+
let name: String = text(result, "test_name")?;
47+
let path: Bound<'_, PyAny> = native_field(result, "source_path")?;
48+
let block: Bound<'_, PyAny> = native_field(result, "block_index")?;
49+
let case: Bound<'_, PyAny> = native_field(result, "case_name")?;
50+
if path.is_none() || block.is_none() || case.is_none() {
51+
return Ok(format!("sql_test:{name}"));
52+
}
53+
Ok(format!(
54+
"sql_test:{}:{}:{}",
55+
path.call_method0("as_posix")?.str()?,
56+
block.str()?,
57+
case.str()?
58+
))
59+
}
60+
61+
/// The resource name and projector identity of a completion.
62+
pub(crate) fn resource_identity(result: &Bound<'_, PyAny>) -> PyResult<Option<(String, String)>> {
63+
Ok(match result_kind(result)? {
64+
ResultKind::Model => {
65+
let name: String = text(result, "model_name")?;
66+
Some((name.clone(), format!("model:{name}")))
67+
}
68+
ResultKind::Seed => {
69+
let name: String = text(result, "seed_name")?;
70+
Some((name.clone(), format!("seed:{name}")))
71+
}
72+
ResultKind::Function => {
73+
let name: String = text(result, "function_name")?;
74+
let kind: String = text(result, "function_kind")?;
75+
Some((name.clone(), format!("{kind}:{name}")))
76+
}
77+
ResultKind::Load => {
78+
let name: String = text(result, "source_name")?;
79+
Some((name.clone(), format!("source:{name}")))
80+
}
81+
ResultKind::PythonNode => {
82+
let name: String = text(result, "node_name")?;
83+
let kind: String = text(result, "kind")?;
84+
Some((name.clone(), format!("{kind}:{name}")))
85+
}
86+
ResultKind::SqlTest => Some((text(result, "test_name")?, sql_test_resource_id(result)?)),
87+
ResultKind::Other => None,
88+
})
89+
}
90+
91+
/// Mutable callback state; only held briefly and never across Python calls.
92+
#[derive(Debug, Default)]
93+
pub(crate) struct CallbackState {
94+
pub(crate) counter: usize,
95+
pub(crate) node_name: String,
96+
pub(crate) node_kind: String,
97+
pub(crate) sub_message: String,
98+
pub(crate) frame: usize,
99+
pub(crate) line_active: bool,
100+
pub(crate) cursor_hidden: bool,
101+
pub(crate) stop_event: Option<Py<PyAny>>,
102+
pub(crate) thread: Option<Py<PyAny>>,
103+
pub(crate) tests_by_model: HashMap<String, Vec<TestResultText>>,
104+
pub(crate) scheduler_count: usize,
105+
pub(crate) scheduler_omitted: usize,
106+
pub(crate) last_scheduler: Option<SchedulerStateText>,
107+
pub(crate) query_counts: HashMap<String, usize>,
108+
pub(crate) query_omitted: usize,
109+
pub(crate) last_query: Option<StatementText>,
110+
}
111+
112+
pub(crate) fn scheduler_state(state: &Bound<'_, PyAny>) -> PyResult<SchedulerStateText> {
113+
Ok(SchedulerStateText {
114+
running: native_field(state, "running")?.extract()?,
115+
ready: native_field(state, "ready")?.extract()?,
116+
waiting: native_field(state, "waiting")?.extract()?,
117+
limit: native_field(state, "limit")?.extract()?,
118+
aborted: native_field(state, "aborted")?.extract()?,
119+
})
120+
}
121+
122+
pub(crate) fn statement(telemetry: &Bound<'_, PyAny>) -> PyResult<StatementText> {
123+
Ok(StatementText {
124+
query_id: optional(telemetry, "query_id")?,
125+
status: text(telemetry, "status")?,
126+
elapsed_seconds: native_field(telemetry, "elapsed_seconds")?.extract()?,
127+
resource_type: text(telemetry, "resource_type")?,
128+
resource_name: text(telemetry, "resource_name")?,
129+
phase: text(telemetry, "phase")?,
130+
})
131+
}
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
//! The `sqb build` completion footer.
2+
3+
use crate::build_progress::main::format_build_footer_with::format_build_footer_with;
4+
use pyo3::prelude::{Bound, PyAny, PyResult};
5+
use pyo3::pyfunction;
6+
7+
use crate::build_progress::_helpers::result_text::build_text;
8+
9+
/// Render the build completion line, warnings, failures and skipped rollup.
10+
#[pyfunction]
11+
#[pyo3(signature = (*, result, elapsed, use_color, python_node_results = None))]
12+
pub(crate) fn format_build_footer(
13+
result: &Bound<'_, PyAny>,
14+
elapsed: f64,
15+
use_color: bool,
16+
python_node_results: Option<&Bound<'_, PyAny>>,
17+
) -> PyResult<String> {
18+
format_build_footer_with(
19+
&build_text(result)?,
20+
(elapsed, use_color),
21+
python_node_results,
22+
Vec::new(),
23+
)
24+
}

crates/sqlbuild-python/src/bindings/_helpers/build_progress/getters.rs renamed to crates/sqlbuild-python-output/src/build_progress/_helpers/getters.rs

Lines changed: 3 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -3,24 +3,10 @@
33
use pyo3::IntoPyObjectExt;
44
use pyo3::prelude::{Bound, PyAny, PyResult, Python};
55

6-
use crate::bindings::_helpers::build_progress::build_callbacks::BuildProgressCallbacks;
6+
use crate::build_progress::models::BuildProgressCallbacks;
77
use sqlbuild_python_base::build_progress::models::TransientLineCoordinator;
8-
use sqlbuild_python_base::records::main::registry::class_of::class_of;
9-
use sqlbuild_python_base::records::models::GetterEntry;
108

11-
/// The getter readers of this module's native pyclasses.
12-
pub(crate) const GETTERS: &[GetterEntry] = &[
13-
GetterEntry {
14-
class: class_of::<BuildProgressCallbacks>,
15-
read: read_build_progress_callbacks,
16-
},
17-
GetterEntry {
18-
class: class_of::<TransientLineCoordinator>,
19-
read: read_transient_line_coordinator,
20-
},
21-
];
22-
23-
fn read_build_progress_callbacks<'py>(
9+
pub(crate) fn read_build_progress_callbacks<'py>(
2410
value: &Bound<'py, PyAny>,
2511
name: &str,
2612
) -> Option<PyResult<Bound<'py, PyAny>>> {
@@ -37,7 +23,7 @@ fn read_build_progress_callbacks<'py>(
3723
})
3824
}
3925

40-
fn read_transient_line_coordinator<'py>(
26+
pub(crate) fn read_transient_line_coordinator<'py>(
4127
value: &Bound<'py, PyAny>,
4228
name: &str,
4329
) -> Option<PyResult<Bound<'py, PyAny>>> {
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
pub(crate) mod audit_progress;
2+
pub(crate) mod build_callbacks;
3+
pub(crate) mod footer;
4+
pub(crate) mod getters;
5+
pub(crate) mod nested_progress;
6+
pub(crate) mod plan_facts;
7+
pub(crate) mod progress_reporters;
8+
pub(crate) mod progress_text;
9+
pub(crate) mod result_text;

0 commit comments

Comments
 (0)