Skip to content

Commit 41e183e

Browse files
authored
feat: store Recall history in ClickHouse and add replacement sinks (#1163)
* feat: store Recall history in ClickHouse A [recall] or targets.<name>.recall connection on the clickhouse adapter now sends history to MergeTree tables in an existing ClickHouse database through the adapter's connection. Recall creates and additively upgrades its tables and views there, prints the DDL when the user may not, and batches RowBinary inserts with deterministic deduplication tokens. Lookups before each insert keep the first copy of a key, record conflicting reuse and refresh artifacts older than a week. Table TTLs follow the configured retention; a store with another retention is reported once a day with the statements that would change it. sqb recall query, schema and output read the ClickHouse store. Also fix the ClickHouse view definitions so aggregate aliases no longer shadow the columns they filter on, and take queued records when a heartbeat falls due before a batch fills. * perf: read ClickHouse Recall views with FINAL so filters reach the tables Every Recall table the views read is now a ReplacingMergeTree ordered by its key, and the views read it with FINAL instead of deduplicating with LIMIT BY. Filters on invocation_id now reach the primary key: on a store of about 3,000 invocations and 2.2 million events, a per-invocation artifact lookup reads 185 thousand rows instead of 507 thousand, and sqb recall output drops from about 2.8 s to 0.5 s. * refactor: share Recall timestamp conversions with the ClickHouse store * fix: report a ClickHouse Recall store still being created as not recorded yet A reader that ran while the first writer was still creating the views failed with an unknown-table query error; it now gets the coded store error that says no history is recorded yet. * feat: let a sink taking a RecallWriter decide what Recall stores A @lifecycle_event_sink or @command_output_sink with a parameter annotated sqlbuild.recall.RecallWriter replaces Recall's default for its stream: Recall stores only what it publishes. Discovery allows one replacement per stream and reports D015 naming every path:function otherwise, rejects a sink with two writers or with providers, and reserves the provider name recall_writer (D012). Replacements are not run as ordinary sinks; other sinks still see every record. Lifecycle events reach the replacement as they are published, filtered by its event kinds and minimum severity. Output chunks reach it on the delivery thread as schema version 2 CommandOutputRecords carrying the exact bytes in data; record.replace() keeps the ID and sequence. An unpublished record is counted as filtered, a raising sink as failed, never falling back to the unfiltered record and never changing the exit code or output. The invocation row now waits for discovery so filter_active records whether a replacement ran. * fix: withhold the launch record when a Recall replacement sink is the only sink * fix: let a SELECT and INSERT user write a Recall store created from the printed DDL * fix: count only the records of a failed ClickHouse table insert as failed * fix: keep referenced SQL text available in the ClickHouse artifacts view until its TTL * fix: suppress output capture only on the Recall delivery thread, not for sqb recall readers * fix: hold Recall output until discovery decides whether a replacement sink filters it * fix: name the command in the first Recall heartbeat so a run killed during discovery is listed * fix: run the Recall lifecycle replacement sink on the delivery thread within the shutdown budget * fix: add missing Recall columns to SQLite stores at the same schema version and write only stored columns * fix: keep writing a ClickHouse Recall store whose user may not add new optional columns * fix: hand lifecycle events to the Recall replacement sink in publication order * test: keep ClickHouse Recall output past the local store's default limit * docs: regenerate the bundled docs skill for ClickHouse Recall * test: approve the ClickHouse Recall reader as a raw statement boundary
1 parent 2c22eb5 commit 41e183e

117 files changed

Lines changed: 7094 additions & 292 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎crates/sqlbuild-core/src/command_output/constants.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,8 @@ pub const MIN_SHUTDOWN_TIMEOUT_SECONDS: f64 = 0.0;
99
pub const COMMAND_OUTPUT_RECORD_TYPE: &str = "command_output";
1010
pub const COMMAND_OUTPUT_LOSS_RECORD_TYPE: &str = "command_output_loss";
1111
pub const CURRENT_SCHEMA_VERSION: i64 = 1;
12+
/// The schema of records a Recall replacement sink receives: version 1 plus the exact `data` bytes.
13+
pub const RECALL_REPLACEMENT_SCHEMA_VERSION: i64 = 2;
1214
pub const MAX_INVOCATION_CONTEXT_BYTES: usize = 16 * 1024;
1315
pub const MAX_INVOCATION_CONTEXT_DEPTH: usize = 8;
1416
pub const INVOCATION_CONTEXT_ENV: &str = "SQLBUILD_INVOCATION_CONTEXT_JSON";
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
//! `ansi_free_text`: output bytes as the text a command-output record's `message` holds.
2+
3+
use crate::command_output::_helpers::text::strip_ansi;
4+
5+
/// `data` without terminal control sequences, decoded as UTF-8 with invalid bytes replaced.
6+
#[must_use]
7+
pub fn ansi_free_text(data: &[u8]) -> String {
8+
String::from_utf8_lossy(&strip_ansi(data)).into_owned()
9+
}

‎crates/sqlbuild-core/src/command_output/main/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
pub mod ansi_free_text;
12
pub mod append_output;
23
pub mod chunk_output_text;
34
pub mod close_output;

‎crates/sqlbuild-core/src/command_output/main/validate_output_record.rs‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
33
use crate::command_output::constants::{
44
COMMAND_OUTPUT_LOSS_RECORD_TYPE, COMMAND_OUTPUT_RECORD_TYPE, CURRENT_SCHEMA_VERSION,
5+
RECALL_REPLACEMENT_SCHEMA_VERSION,
56
};
67
use crate::command_output::models::RecordShape;
78

@@ -35,6 +36,9 @@ pub fn validate_output_record(shape: &RecordShape) -> Result<(), String> {
3536
if !shape.message_is_text {
3637
return Err("message must be a string".to_owned());
3738
}
39+
if !shape.data_is_bytes {
40+
return Err("data must be bytes".to_owned());
41+
}
3842
let chunk_index = at_least("chunk_index", shape.chunk_index, 0)?;
3943
let chunk_count = at_least("chunk_count", shape.chunk_count, 1)?;
4044
let dropped = at_least("dropped_records", shape.dropped_records, 0)?;
@@ -54,8 +58,13 @@ pub fn validate_output_record(shape: &RecordShape) -> Result<(), String> {
5458
Some(COMMAND_OUTPUT_RECORD_TYPE | COMMAND_OUTPUT_LOSS_RECORD_TYPE) => {}
5559
_ => return Err("record_type is unknown".to_owned()),
5660
}
57-
if shape.schema_version != Some(CURRENT_SCHEMA_VERSION) {
58-
return Err(format!("schema_version must be {CURRENT_SCHEMA_VERSION}"));
61+
if !matches!(
62+
shape.schema_version,
63+
Some(CURRENT_SCHEMA_VERSION | RECALL_REPLACEMENT_SCHEMA_VERSION)
64+
) {
65+
return Err(format!(
66+
"schema_version must be {CURRENT_SCHEMA_VERSION} or {RECALL_REPLACEMENT_SCHEMA_VERSION}"
67+
));
5968
}
6069
Ok(())
6170
}

‎crates/sqlbuild-core/src/command_output/models.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,7 @@ pub struct RecordShape {
116116
pub occurred_at_is_utc: bool,
117117
pub stream_is_known: bool,
118118
pub message_is_text: bool,
119+
pub data_is_bytes: bool,
119120
pub chunk_index: Option<i64>,
120121
pub chunk_count: Option<i64>,
121122
pub dropped_records: Option<i64>,

‎crates/sqlbuild-python/src/bindings/_helpers/cli/entry.rs‎

Lines changed: 37 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
//! The `sqb` entry: invocation scopes, parsing, observed dispatch and the error projection.
22
3+
use std::sync::Arc;
34
use std::time::Instant;
45

56
use pyo3::exceptions::PySystemExit;
@@ -16,7 +17,9 @@ use crate::bindings::_helpers::build_progress::transient_lines::shared_transient
1617
use crate::bindings::_helpers::cli::argv::parse_cli_invocation;
1718
use crate::bindings::_helpers::cli::dispatch::{command_facts, dispatch_command};
1819
use crate::bindings::_helpers::cli::launch_state::offer_launch_record;
19-
use crate::bindings::_helpers::cli::observability::cli_observability_scope;
20+
use crate::bindings::_helpers::cli::observability::{
21+
CliObservabilityScope, cli_observability_scope,
22+
};
2023
use crate::bindings::_helpers::compile_command::report::format_compile_error_json;
2124
use crate::bindings::_helpers::compile_driver::python_calls::{
2225
call_with, imported, keywords, within,
@@ -397,12 +400,27 @@ fn lifecycle_event<'py>(
397400
)
398401
}
399402

400-
/// Subscribe the invocation's Recall session to every lifecycle fact and SQL artifact.
401-
fn subscribe_recall(dispatcher: &Bound<'_, PyAny>, recall: &InvocationRecall) -> PyResult<()> {
403+
/// Install the discovered Recall replacement sinks, release the context row, and subscribe the session to every lifecycle fact and SQL artifact.
404+
fn subscribe_recall(
405+
(scope, dispatcher): (&Bound<'_, CliObservabilityScope>, &Bound<'_, PyAny>),
406+
recall: &InvocationRecall,
407+
) -> PyResult<()> {
402408
let Some(session) = recall.session.clone() else {
403409
return Ok(());
404410
};
405-
let sink = Bound::new(dispatcher.py(), RecallLifecycleSink { session })?;
411+
let py: Python<'_> = dispatcher.py();
412+
let (lifecycle, output) = scope.get().recall_sinks(py)?;
413+
let filter_active: bool = recall
414+
.replacements
415+
.configure(&session, (&lifecycle, &output))?;
416+
session.publish_context(filter_active);
417+
let sink = Bound::new(
418+
py,
419+
RecallLifecycleSink {
420+
session,
421+
replacements: Arc::clone(&recall.replacements),
422+
},
423+
)?;
406424
dispatcher.call_method1("subscribe_history", (sink,))?;
407425
Ok(())
408426
}
@@ -432,6 +450,9 @@ fn observed_dispatch<'py>(
432450
) -> PyResult<Bound<'py, PyAny>> {
433451
let py: Python<'py> = invocation.py();
434452
if !policy.observed {
453+
if let Some(session) = &invocation.recall.session {
454+
session.publish_context(false);
455+
}
435456
return dispatch_command(&invocation.args, invocation.handlers.clone());
436457
}
437458
let command: String = invocation.command_name();
@@ -441,20 +462,20 @@ fn observed_dispatch<'py>(
441462
.unwrap_or(u64::MAX)
442463
.into_bound_py_any(py)
443464
};
444-
let scope: Bound<'py, PyAny> = Bound::new(
465+
let observability: Bound<'py, CliObservabilityScope> = Bound::new(
445466
py,
446467
cli_observability_scope(
447468
invocation.args.clone().unbind(),
448469
invocation.project_dir()?.unbind(),
449470
),
450-
)?
451-
.into_any();
471+
)?;
472+
let scope: Bound<'py, PyAny> = observability.clone().into_any();
452473
within(&scope, |dispatcher| {
453474
let publish = |event: Bound<'py, PyAny>| -> PyResult<()> {
454475
dispatcher.call_method1("publish_lifecycle", (event,))?;
455476
Ok(())
456477
};
457-
subscribe_recall(dispatcher, &invocation.recall)?;
478+
subscribe_recall((&observability, dispatcher), &invocation.recall)?;
458479
let payload: Bound<'py, PyDict> = PyDict::new(py);
459480
payload.set_item("command", &command)?;
460481
add_invocation_context(invocation, &payload)?;
@@ -551,7 +572,14 @@ fn parsed_invocation<'py>(
551572
InvocationRecall::default()
552573
};
553574
let recall_sink: Bound<'py, PyAny> = match recall.session.clone() {
554-
Some(session) => Bound::new(py, RecallLifecycleSink { session })?.into_any(),
575+
Some(session) => Bound::new(
576+
py,
577+
RecallLifecycleSink {
578+
session,
579+
replacements: Arc::clone(&recall.replacements),
580+
},
581+
)?
582+
.into_any(),
555583
None => py.None().into_bound(py),
556584
};
557585
let closure_recall: InvocationRecall = recall.clone();

‎crates/sqlbuild-python/src/bindings/_helpers/cli/observability.rs‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,8 @@ struct Observed {
150150
exporter_scope: Option<Py<PyAny>>,
151151
unsubscribe_exporters: Option<Py<PyAny>>,
152152
unsubscribe_outcome: Option<Py<PyAny>>,
153+
/// The discovered Recall lifecycle and command-output replacement sinks, each `(sink, writer_parameter)` or `None`.
154+
recall_sinks: Option<(Py<PyAny>, Py<PyAny>)>,
153155
/// Context managers entered, innermost last.
154156
entered: Vec<Py<PyAny>>,
155157
}
@@ -332,6 +334,14 @@ impl Observed {
332334
&host::entry(py, UserCodeHost::DiscoverRuntimeExtensions)?,
333335
&[("project_dir", project_dir.clone())],
334336
)?;
337+
let lifecycle_replacement: Bound<'_, PyAny> =
338+
native_field(&extensions, "recall_lifecycle_sink")?;
339+
let output_replacement: Bound<'_, PyAny> =
340+
native_field(&extensions, "recall_output_sink")?;
341+
if !lifecycle_replacement.is_none() || !output_replacement.is_none() {
342+
withhold_launch_record(py)?;
343+
}
344+
self.recall_sinks = Some((lifecycle_replacement.unbind(), output_replacement.unbind()));
335345
let event_exporters: Bound<'_, PyAny> = native_field(&extensions, "event_exporters")?;
336346
let sinks: Bound<'_, PyAny> = native_field(&extensions, "command_output_sinks")?;
337347
command_output_sinks = sinks.is_truthy()?;
@@ -432,6 +442,23 @@ pub(crate) struct CliObservabilityScope {
432442
}
433443

434444
impl CliObservabilityScope {
445+
/// The Recall lifecycle and command-output replacement sinks discovery found, each `(sink, writer_parameter)` or `None`.
446+
pub(crate) fn recall_sinks<'py>(
447+
&self,
448+
py: Python<'py>,
449+
) -> PyResult<(Bound<'py, PyAny>, Bound<'py, PyAny>)> {
450+
let observed = self.locked()?;
451+
Ok(
452+
match observed
453+
.as_ref()
454+
.and_then(|observed| observed.recall_sinks.as_ref())
455+
{
456+
Some((lifecycle, output)) => (lifecycle.bind(py).clone(), output.bind(py).clone()),
457+
None => (py.None().into_bound(py), py.None().into_bound(py)),
458+
},
459+
)
460+
}
461+
435462
fn locked(&self) -> PyResult<MutexGuard<'_, Option<Observed>>> {
436463
self.observed
437464
.lock()
@@ -510,6 +537,7 @@ impl CliObservabilityScope {
510537
exporter_scope: None,
511538
unsubscribe_exporters: None,
512539
unsubscribe_outcome: None,
540+
recall_sinks: None,
513541
entered: Vec::new(),
514542
};
515543
if let Err(error) = observed.install(py, (args, self.project_dir.bind(py)), &terminal_index)

‎crates/sqlbuild-python/src/bindings/_helpers/compile_records/constants.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1191,6 +1191,8 @@ const R_DISCOVERED_RUNTIME_EXTENSIONS: RecordSpec = RecordSpec {
11911191
FieldSpec::new("event_exporters", FieldDefault::EmptyTuple),
11921192
FieldSpec::new("command_output_sinks", FieldDefault::EmptyTuple),
11931193
FieldSpec::new("lifecycle_shutdown_timeout_seconds", FieldDefault::None),
1194+
FieldSpec::new("recall_lifecycle_sink", FieldDefault::None),
1195+
FieldSpec::new("recall_output_sink", FieldDefault::None),
11941196
],
11951197
};
11961198
const R_COMPILER_DIAGNOSTIC: RecordSpec = RecordSpec {

‎crates/sqlbuild-python/src/bindings/_helpers/discovery/imported_providers.rs‎

Lines changed: 52 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,15 @@ use crate::bindings::_helpers::discovery::imported_modules::{
1515
};
1616
use crate::bindings::_helpers::native_host::host;
1717
use crate::bindings::_helpers::native_host::user_code::UserCodeHost;
18+
use crate::bindings::_helpers::recall::writer::RecallWriter;
1819
use crate::bindings::_helpers::records::fields::field as native_field;
1920
use crate::bindings::_helpers::records::fields::mapping_get_or;
2021
use crate::bindings::_helpers::records::ids::{ErrorId, RecordId};
2122

2223
const PROVIDER_ERROR: ErrorId = ErrorId::ProviderDiscovery;
2324
const SINK_ERROR: ErrorId = ErrorId::EventExporterDiscovery;
25+
/// The provider name kept free so a sink's `RecallWriter` parameter can never be mistaken for a provider.
26+
const RESERVED_PROVIDER_NAME: &str = "recall_writer";
2427
const PROVIDER_INPUT_ERROR: UserCodeHost = UserCodeHost::ProviderInputError;
2528
const VALIDATION_ERROR: (&str, &str) = ("pydantic", "ValidationError");
2629
const SINK_DEFINITION: UserCodeHost = UserCodeHost::GetLifecycleEventSinkDefinition;
@@ -157,6 +160,17 @@ pub(crate) fn discover_provider_classes<'py>(
157160
let relative_path: Bound<'py, PyAny> = project.relative(&file_path)?;
158161
let name: Bound<'py, PyAny> = provider_name(&provider_class, &relative_path)?;
159162
let name_text: String = name.extract()?;
163+
if name_text == RESERVED_PROVIDER_NAME {
164+
return Err(discovery_error(
165+
py,
166+
PROVIDER_ERROR,
167+
format!(
168+
"Provider name '{RESERVED_PROVIDER_NAME}' in {} is reserved for the \
169+
RecallWriter of Recall replacement sinks; rename the provider",
170+
project.relative_to(&file_path)?
171+
),
172+
));
173+
}
160174
if let Some((_, existing)) = seen.iter().find(|(listed, _)| *listed == name_text) {
161175
return Err(discovery_error(
162176
py,
@@ -267,6 +281,33 @@ pub(crate) fn discover_event_exporter_declarations<'py>(
267281
tuple_of(py, declarations)
268282
}
269283

284+
/// Each parameter of `function` with its annotation, and whether that annotation is `RecallWriter`.
285+
fn annotated_parameters<'py>(
286+
function: &Bound<'py, PyAny>,
287+
) -> PyResult<Vec<(String, Bound<'py, PyAny>, bool)>> {
288+
let py: Python<'py> = function.py();
289+
let writer = py.get_type::<RecallWriter>();
290+
let mut parameters: Vec<(String, Bound<'py, PyAny>, bool)> = Vec::new();
291+
for pair in shim(py, UserCodeHost::ParameterAnnotations)?
292+
.call1((function,))?
293+
.try_iter()?
294+
{
295+
let (parameter, annotation): (String, Bound<'py, PyAny>) = pair?.extract()?;
296+
let is_writer: bool = annotation.is(writer.as_any());
297+
parameters.push((parameter, annotation, is_writer));
298+
}
299+
Ok(parameters)
300+
}
301+
302+
/// Whether `declaration`'s function takes a `RecallWriter`, making it a Recall replacement sink.
303+
fn takes_recall_writer(declaration: &Bound<'_, PyAny>) -> PyResult<bool> {
304+
Ok(
305+
annotated_parameters(&native_field(declaration, "function")?)?
306+
.iter()
307+
.any(|(_, _, is_writer)| *is_writer),
308+
)
309+
}
310+
270311
/// Provider parameters after the event, each naming a known provider of the right class.
271312
fn exporter_provider_usages<'py>(
272313
declaration: &Bound<'py, PyAny>,
@@ -280,13 +321,10 @@ fn exporter_provider_usages<'py>(
280321
.str()?
281322
.extract()?;
282323
let empty: Bound<'py, PyAny> = imported(py, "inspect", "Parameter")?.getattr("empty")?;
283-
let annotations: Bound<'py, PyAny> =
284-
shim(py, UserCodeHost::ParameterAnnotations)?.call1((&function,))?;
285-
for (index, pair) in annotations.try_iter()?.enumerate() {
286-
if index == 0 {
324+
for (parameter, annotation, is_writer) in annotated_parameters(&function)?.into_iter().skip(1) {
325+
if is_writer {
287326
continue;
288327
}
289-
let (parameter, annotation): (String, Bound<'py, PyAny>) = pair?.extract()?;
290328
let Some(provider) = providers.get_item(&parameter)? else {
291329
return Err(discovery_error(
292330
py,
@@ -344,7 +382,7 @@ pub(crate) fn bind_event_exporter_declarations<'py>(
344382
tuple_of(py, bound)
345383
}
346384

347-
/// Validated lifecycle-event sink declarations under sinks/, bound to providers.
385+
/// Validated lifecycle-event sink declarations under sinks/, bound to providers; Recall replacement sinks are left out.
348386
#[pyfunction]
349387
#[pyo3(signature = (*, project_dir, providers = None))]
350388
pub(crate) fn discover_event_exporter_functions<'py>(
@@ -358,10 +396,15 @@ pub(crate) fn discover_event_exporter_functions<'py>(
358396
if let Some(cached) = project.cached_extensions()? {
359397
return cached.get_item(1);
360398
}
361-
let declarations: Bound<'py, PyAny> =
362-
discover_event_exporter_declarations(project_dir.clone())?;
399+
let mut ordinary: Vec<Bound<'py, PyAny>> = Vec::new();
400+
for declaration in discover_event_exporter_declarations(project_dir.clone())?.try_iter()? {
401+
let declaration: Bound<'py, PyAny> = declaration?;
402+
if !takes_recall_writer(&declaration)? {
403+
ordinary.push(declaration);
404+
}
405+
}
363406
bind_event_exporter_declarations(
364-
declarations,
407+
tuple_of(py, ordinary)?,
365408
providers.map_or_else(|| tuple_of(py, Vec::new()), Ok)?,
366409
project_dir,
367410
)

‎crates/sqlbuild-python/src/bindings/_helpers/native_host/adapter.rs‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,11 @@ pub(crate) enum AdapterHost {
4444
NormalizeRelationType,
4545
NormalizeType,
4646
SafeCastNullability,
47+
ConnectRecallStore,
48+
RunRecallQuery,
49+
RunRecallCommand,
50+
InsertRecallRows,
51+
CloseRecallStore,
4752
}
4853

4954
/// Every adapter entry, in `AdapterHost` order.
@@ -243,6 +248,31 @@ pub(crate) const ENTRIES: &[HostEntry] = &[
243248
kind: HostKind::Function,
244249
edge: HostEdge::Adapter,
245250
},
251+
HostEntry {
252+
name: "connect_recall_store",
253+
kind: HostKind::Function,
254+
edge: HostEdge::Adapter,
255+
},
256+
HostEntry {
257+
name: "run_recall_query",
258+
kind: HostKind::Function,
259+
edge: HostEdge::Adapter,
260+
},
261+
HostEntry {
262+
name: "run_recall_command",
263+
kind: HostKind::Function,
264+
edge: HostEdge::Adapter,
265+
},
266+
HostEntry {
267+
name: "insert_recall_rows",
268+
kind: HostKind::Function,
269+
edge: HostEdge::Adapter,
270+
},
271+
HostEntry {
272+
name: "close_recall_store",
273+
kind: HostKind::Function,
274+
edge: HostEdge::Adapter,
275+
},
246276
];
247277

248278
impl AdapterHost {

0 commit comments

Comments
 (0)