You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 41e183e
Browse filesBrowse the repository at this point in the historyBrowse files
feat: store Recall history in ClickHouse and add replacement sinks (#1163)
* feat: store Recall history in ClickHouse
A [recall] or targets.<name>.recall connection on the clickhouse adapter now
sends history to MergeTree tables in an existing ClickHouse database through
the adapter's connection. Recall creates and additively upgrades its tables and
views there, prints the DDL when the user may not, and batches RowBinary
inserts with deterministic deduplication tokens. Lookups before each insert
keep the first copy of a key, record conflicting reuse and refresh artifacts
older than a week. Table TTLs follow the configured retention; a store with
another retention is reported once a day with the statements that would
change it. sqb recall query, schema and output read the ClickHouse store.
Also fix the ClickHouse view definitions so aggregate aliases no longer
shadow the columns they filter on, and take queued records when a heartbeat
falls due before a batch fills.
* perf: read ClickHouse Recall views with FINAL so filters reach the tables
Every Recall table the views read is now a ReplacingMergeTree ordered by its
key, and the views read it with FINAL instead of deduplicating with LIMIT BY.
Filters on invocation_id now reach the primary key: on a store of about 3,000
invocations and 2.2 million events, a per-invocation artifact lookup reads
185 thousand rows instead of 507 thousand, and sqb recall output drops from
about 2.8 s to 0.5 s.
* refactor: share Recall timestamp conversions with the ClickHouse store
* fix: report a ClickHouse Recall store still being created as not recorded yet
A reader that ran while the first writer was still creating the views failed
with an unknown-table query error; it now gets the coded store error that
says no history is recorded yet.
* feat: let a sink taking a RecallWriter decide what Recall stores
A @lifecycle_event_sink or @command_output_sink with a parameter annotated
sqlbuild.recall.RecallWriter replaces Recall's default for its stream: Recall
stores only what it publishes. Discovery allows one replacement per stream and
reports D015 naming every path:function otherwise, rejects a sink with two
writers or with providers, and reserves the provider name recall_writer
(D012). Replacements are not run as ordinary sinks; other sinks still see
every record.
Lifecycle events reach the replacement as they are published, filtered by its
event kinds and minimum severity. Output chunks reach it on the delivery
thread as schema version 2 CommandOutputRecords carrying the exact bytes in
data; record.replace() keeps the ID and sequence. An unpublished record is
counted as filtered, a raising sink as failed, never falling back to the
unfiltered record and never changing the exit code or output. The invocation
row now waits for discovery so filter_active records whether a replacement
ran.
* fix: withhold the launch record when a Recall replacement sink is the only sink
* fix: let a SELECT and INSERT user write a Recall store created from the printed DDL
* fix: count only the records of a failed ClickHouse table insert as failed
* fix: keep referenced SQL text available in the ClickHouse artifacts view until its TTL
* fix: suppress output capture only on the Recall delivery thread, not for sqb recall readers
* fix: hold Recall output until discovery decides whether a replacement sink filters it
* fix: name the command in the first Recall heartbeat so a run killed during discovery is listed
* fix: run the Recall lifecycle replacement sink on the delivery thread within the shutdown budget
* fix: add missing Recall columns to SQLite stores at the same schema version and write only stored columns
* fix: keep writing a ClickHouse Recall store whose user may not add new optional columns
* fix: hand lifecycle events to the Recall replacement sink in publication order
* test: keep ClickHouse Recall output past the local store's default limit
* docs: regenerate the bundled docs skill for ClickHouse Recall
* test: approve the ClickHouse Recall reader as a raw statement boundary
0 commit comments