diff --git a/.github/demos/plan-load.gif b/.github/demos/plan-load.gif index a8b209f61..4accf6009 100644 Binary files a/.github/demos/plan-load.gif and b/.github/demos/plan-load.gif differ diff --git a/README.md b/README.md index e2deb5a71..afe427d6e 100644 --- a/README.md +++ b/README.md @@ -19,26 +19,29 @@ Benchmarks

-Load Postgres, files, Snowflake, BigQuery, Google Sheets and Python sources into your warehouse. -SQLLoad is Rust and Arrow end to end, and when your warehouse reads the files itself, rows never -pass through SQLLoad, so there's no loader bottleneck as your data grows. It's fast, too: faster -than ingestr and dlt in every [benchmark](https://sqlload.io/docs/benchmarks/) we publish. +SQLLoad loads databases, files, APIs and Kafka into your warehouse. `sqlload plan` shows every +table and how it loads before anything moves, and `plan -v` adds each new table's column types and +the run's memory budget. Decimals keep their precision, a value a column can't hold exactly fails +its table, naming the column and the value, and each run has one memory budget. Tables with a +cursor column carry on from the warehouse's own maximum, so there's no state to drift. ```bash pip install sqlload # or: uv tool install sqlload ```

- sqlload plan lists the two tables it will create from Postgres, their column types in DuckDB, and how each loads; sqlload load then creates them and loads 1,050 rows + In the playground project, sqlload plan lists the two tables it will create in DuckDB and how each loads from CSV files; sqlload load creates them and loads 1,050 rows, reporting the slowest table and peak memory; with cursor_column misspelled as updated, the next load refuses raw.orders before loading it: did you mean updated_at?

-`sqlload plan` shows the tables it will create from Postgres, each column's DuckDB type and how -each table loads. `sqlload load` then creates and fills them. +In the [playground](https://sqlload.io/docs/cli/playground/) project, `sqlload plan` shows the +tables it will create and how each loads, and `sqlload load` creates and fills them. With +`cursor_column` misspelled, the next load refuses the orders table before touching it, and says +which column was meant. ## Quick start ```bash -sqlload plan # every table, column type and route, before anything is created +sqlload plan # every table and how it loads, before anything is created (-v: column types) sqlload apply # create the tables sqlload load # load them ``` @@ -56,7 +59,26 @@ Moving from dlt? [Coming from dlt](https://sqlload.io/docs/guides/from-dlt/) map write dispositions, incremental loading, `rest_api` and `filesystem`, and keeps the table and column names dlt gave your tables. -## Speed +## Why SQLLoad + +- **Exact types.** A source's `decimal(12,2)` loads as `DECIMAL(12,2)` (`NUMBER(12,2)` on + Snowflake), timestamps keep their + microseconds, and a pinned type narrower than the live column fails `plan` rather than cutting + values. See [contracts](https://sqlload.io/docs/concepts/contracts/). +- **Loud failures.** `plan` names each problem it can find, such as a misspelled column, a + narrowing type or a budget too small, and its fix, before anything is created. A commit whose + [checks](https://sqlload.io/docs/concepts/checks/) fail publishes none of its rows, every commit + lands whole even if a load is killed halfway, and in a warehouse (except a ClickHouse `Ordinary` database) a full refresh swaps in at once, + so readers see the old table or the new one, never a mix. +- **Bounded memory.** A run has one [memory budget](https://sqlload.io/docs/cli/load/#memory-budget), + shared out between its tables. A budget too small for even one table is refused before anything + loads, and near the limit SQLLoad stops a table, or the run, and rolls back what it was + writing, rather than waiting for the operating system to kill it. +- **No hidden state.** Tables with a `cursor_column` carry on from the + [warehouse's own maximum](https://sqlload.io/docs/concepts/cursors/) of the cursor column, so a + table you fix or reload by hand never disagrees with a state file. + +## Speed and memory Faster, in a fraction of the memory. The same data loaded by each tool on one 4-vCPU, 8 GB machine; lower is better: @@ -77,18 +99,6 @@ peak memory 131 MiB for SQLLoad, 495 MiB for ingestr and 341 MiB for dlt. See the [benchmarks](https://sqlload.io/docs/benchmarks/) for every scenario, the method, the versions and how to run them yourself. -## Why SQLLoad - -- **See the plan before any data moves.** `sqlload plan` shows every table, column type, write - strategy and route before anything is created or loaded, so there are no surprises in your - warehouse. -- **Crash-safe loads.** Every commit lands whole, even if a load is killed halfway. A full refresh - swaps in one go, so your dashboards see the old table or the new one, never a mix. -- **Postgres change data capture.** A `postgres_cdc` source copies your tables once, then applies - every insert, update and delete from a logical replication slot, in whole source transactions. - `sqlload load --follow` keeps streaming. See - [Postgres CDC](https://sqlload.io/docs/sources/postgres-cdc/). - ## What it loads | System | Source | Target | @@ -98,8 +108,8 @@ versions and how to run them yourself. | BigQuery (beta) | ✓ | ✓ | | Databricks (beta) | ✓ | ✓ | | ClickHouse | | ✓ | -| DuckDB | | ✓ | -| MotherDuck (beta) | | ✓ | +| DuckDB | ✓ | ✓ | +| MotherDuck (beta) | ✓ | ✓ | | ODBC | ✓ SQL Server, MySQL, OpenEdge | | | Files | ✓ Parquet, CSV, JSON, Excel, XML | ✓ Parquet, CSV | | Google Sheets | ✓ | | diff --git a/packages/sqlload/README.md b/packages/sqlload/README.md index 40a451508..afe427d6e 100644 --- a/packages/sqlload/README.md +++ b/packages/sqlload/README.md @@ -19,26 +19,29 @@ Benchmarks

-Load Postgres, files, Snowflake, BigQuery, Google Sheets and Python sources into your warehouse. -SQLLoad is Rust and Arrow end to end, and when your warehouse reads the files itself, rows never -pass through SQLLoad, so there's no loader bottleneck as your data grows. It's fast, too: faster -than ingestr and dlt in every [benchmark](https://sqlload.io/docs/benchmarks/) we publish. +SQLLoad loads databases, files, APIs and Kafka into your warehouse. `sqlload plan` shows every +table and how it loads before anything moves, and `plan -v` adds each new table's column types and +the run's memory budget. Decimals keep their precision, a value a column can't hold exactly fails +its table, naming the column and the value, and each run has one memory budget. Tables with a +cursor column carry on from the warehouse's own maximum, so there's no state to drift. ```bash -uv tool install sqlload # or: pip install sqlload +pip install sqlload # or: uv tool install sqlload ```

- sqlload plan lists the two tables it will create from Postgres, their column types in DuckDB, and how each loads; sqlload load then creates them and loads 1,050 rows + In the playground project, sqlload plan lists the two tables it will create in DuckDB and how each loads from CSV files; sqlload load creates them and loads 1,050 rows, reporting the slowest table and peak memory; with cursor_column misspelled as updated, the next load refuses raw.orders before loading it: did you mean updated_at?

-`sqlload plan` shows the tables it will create from Postgres, each column's DuckDB type and how -each table loads. `sqlload load` then creates and fills them. +In the [playground](https://sqlload.io/docs/cli/playground/) project, `sqlload plan` shows the +tables it will create and how each loads, and `sqlload load` creates and fills them. With +`cursor_column` misspelled, the next load refuses the orders table before touching it, and says +which column was meant. ## Quick start ```bash -sqlload plan # every table, column type and route, before anything is created +sqlload plan # every table and how it loads, before anything is created (-v: column types) sqlload apply # create the tables sqlload load # load them ``` @@ -52,7 +55,30 @@ configuration file: sqlload copy orders/ duckdb://shop.duckdb#raw.orders ``` -## Speed +Moving from dlt? [Coming from dlt](https://sqlload.io/docs/guides/from-dlt/) maps sources, +write dispositions, incremental loading, `rest_api` and `filesystem`, and keeps the table and column +names dlt gave your tables. + +## Why SQLLoad + +- **Exact types.** A source's `decimal(12,2)` loads as `DECIMAL(12,2)` (`NUMBER(12,2)` on + Snowflake), timestamps keep their + microseconds, and a pinned type narrower than the live column fails `plan` rather than cutting + values. See [contracts](https://sqlload.io/docs/concepts/contracts/). +- **Loud failures.** `plan` names each problem it can find, such as a misspelled column, a + narrowing type or a budget too small, and its fix, before anything is created. A commit whose + [checks](https://sqlload.io/docs/concepts/checks/) fail publishes none of its rows, every commit + lands whole even if a load is killed halfway, and in a warehouse (except a ClickHouse `Ordinary` database) a full refresh swaps in at once, + so readers see the old table or the new one, never a mix. +- **Bounded memory.** A run has one [memory budget](https://sqlload.io/docs/cli/load/#memory-budget), + shared out between its tables. A budget too small for even one table is refused before anything + loads, and near the limit SQLLoad stops a table, or the run, and rolls back what it was + writing, rather than waiting for the operating system to kill it. +- **No hidden state.** Tables with a `cursor_column` carry on from the + [warehouse's own maximum](https://sqlload.io/docs/concepts/cursors/) of the cursor column, so a + table you fix or reload by hand never disagrees with a state file. + +## Speed and memory Faster, in a fraction of the memory. The same data loaded by each tool on one 4-vCPU, 8 GB machine; lower is better: @@ -73,37 +99,22 @@ peak memory 131 MiB for SQLLoad, 495 MiB for ingestr and 341 MiB for dlt. See the [benchmarks](https://sqlload.io/docs/benchmarks/) for every scenario, the method, the versions and how to run them yourself. -## Why SQLLoad - -- **See the plan before any data moves.** `sqlload plan` shows every table, column type, write - strategy and route before anything is created or loaded, so there are no surprises in your - warehouse. -- **Crash-safe loads.** Every commit lands whole, even if a load is killed halfway. A full refresh - swaps in one go, so your dashboards see the old table or the new one, never a mix. -- **Postgres change data capture.** A `postgres_cdc` source copies your tables once, then applies - every insert, update and delete from a logical replication slot, in whole source transactions. - `sqlload load --follow` keeps streaming. See - [Postgres CDC](https://sqlload.io/docs/sources/postgres-cdc/). - ## What it loads | System | Source | Target | |---|---|---| -| Postgres | ✓ (incl. CDC) | ✓ | +| Postgres | ✓ (incl. [CDC](https://sqlload.io/docs/sources/postgres-cdc/)) | ✓ | | Snowflake | ✓ | ✓ | -| BigQuery | ✓ | preview | -| Databricks | | ✓ | -| ClickHouse | | preview | -| DuckDB | | ✓ | -| MotherDuck | | beta² | -| Files | ✓ Parquet, CSV, JSON (S3, local folder, SFTP) | ✓ Parquet, CSV (S3, local folder) | +| BigQuery (beta) | ✓ | ✓ | +| Databricks (beta) | ✓ | ✓ | +| ClickHouse | | ✓ | +| DuckDB | ✓ | ✓ | +| MotherDuck (beta) | ✓ | ✓ | +| ODBC | ✓ SQL Server, MySQL, OpenEdge | | +| Files | ✓ Parquet, CSV, JSON, Excel, XML | ✓ Parquet, CSV | | Google Sheets | ✓ | | -| HTTP APIs and Python | ✓ | | -| ODBC databases¹ | ✓ | | - -¹ In the published wheels from 0.39.0; before that, in builds with the `odbc` feature. - -² From 0.34.0, through a separate DuckDB 1.5 process: install `sqlload[motherduck]`. +| APIs, Python | ✓ | | +| Kafka | ✓ to Snowflake, Databricks, BigQuery | | See [Supported systems](https://sqlload.io/docs/supported-systems/) for every combination and write strategy. diff --git a/website/recordings/.gitignore b/website/recordings/.gitignore new file mode 100644 index 000000000..77f75d7e9 --- /dev/null +++ b/website/recordings/.gitignore @@ -0,0 +1,2 @@ +out/ +scratch/ diff --git a/website/recordings/README.md b/website/recordings/README.md new file mode 100644 index 000000000..eb92f31a9 --- /dev/null +++ b/website/recordings/README.md @@ -0,0 +1,35 @@ +# CLI recordings + +Terminal recordings of `sqlload` for the README, rendered with +[VHS](https://github.com/charmbracelet/vhs) from the `.tape` scripts in `tapes/`. The tapes and +scripts are the source; the rendered GIF, MP4 and PNG files go to `out/`, which is not committed. + +| Demo | Shows | +|------|-------| +| `plan-load` | In the playground project, `sqlload plan`, then `sqlload load`, then a load that stops on a misspelled `cursor_column`. | + +The README uses `plan-load`, copied to `.github/demos/plan-load.gif`. It runs locally on DuckDB, +with the project `sqlload playground` writes. + +## Requirements + +- `vhs`, `ttyd` and `ffmpeg` on `PATH`, plus a Chromium or Chrome that VHS can find (it downloads + one if none is on `PATH`). +- DejaVu Sans Mono, the font the tapes use. +- `sqlload` on `PATH`, or `SQLLOAD_BIN` pointing at a directory that contains the `sqlload` to + record. Record with the latest release, so the output matches what users see. + +## Record + +```bash +cd website/recordings +./record.sh plan-load # out/plan-load.gif, out/plan-load.mp4 and out/plan-load.png +cp out/plan-load.gif ../../.github/demos/ +``` + +`record.sh` runs `prepare.sh ` to build a clean project under `scratch//`, then renders +`tapes/.tape`. Shared terminal settings and the colour theme, which matches the website's +code blocks, are in `tapes/common.tape`. + +Look at every frame before publishing a recording: output such as timings and memory changes +between runs. diff --git a/website/recordings/prepare.sh b/website/recordings/prepare.sh new file mode 100755 index 000000000..067c0f558 --- /dev/null +++ b/website/recordings/prepare.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# Build a clean scratch project for one demo. Usage: prepare.sh +set -euo pipefail +demo="$1" +here="$(cd "$(dirname "$0")" && pwd)" +dir="$here/scratch/$demo" +rm -rf "$dir" +mkdir -p "$dir" +cd "$dir" +case "$demo" in + plan-load) + # The playground project, and a copy of its settings with cursor_column misspelled, which the + # tape swaps in after the first load. + sqlload playground shop >/dev/null + awk '{print} /^unique_key = \["order_id"\]$/ {print "cursor_column = \"updated\""}' shop/sqlload.toml > typo.toml + grep -q '^cursor_column = "updated"' typo.toml || { echo "prepare.sh: no unique_key line to add the typo after" >&2; exit 1; } ;; + *) + echo "unknown demo: $demo" >&2 + exit 2 ;; +esac diff --git a/website/recordings/record.sh b/website/recordings/record.sh new file mode 100755 index 000000000..dc59bc058 --- /dev/null +++ b/website/recordings/record.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# Record one demo to out/.gif, out/.mp4 and out/.png. Usage: record.sh +# Uses the sqlload on PATH; set SQLLOAD_BIN to a directory containing another sqlload to use that instead. +set -euo pipefail +demo="$1" +here="$(cd "$(dirname "$0")" && pwd)" +if [[ -n "${SQLLOAD_BIN:-}" ]]; then + export PATH="$SQLLOAD_BIN:$PATH" +fi +"$here/prepare.sh" "$demo" +mkdir -p "$here/out" +cd "$here" +# Headless Chromium's sandbox is unavailable in many containers and CI runners. +VHS_NO_SANDBOX="${VHS_NO_SANDBOX:-true}" vhs "tapes/$demo.tape" diff --git a/website/recordings/tapes/common.tape b/website/recordings/tapes/common.tape new file mode 100644 index 000000000..29e88bfca --- /dev/null +++ b/website/recordings/tapes/common.tape @@ -0,0 +1,13 @@ +Set Shell bash +Set FontFamily "DejaVu Sans Mono" +Set FontSize 18 +Set LineHeight 1.0 +Set Width 1200 +Set Height 520 +Set Padding 28 +Set Margin 24 +Set MarginFill "#0d0f12" +Set BorderRadius 10 +Set WindowBar Colorful +Set TypingSpeed 40ms +Set Theme { "name": "SQLLoad", "black": "#5c606b", "red": "#ff5760", "green": "#79c98a", "yellow": "#e3b341", "blue": "#5aa9e6", "magenta": "#c79bf0", "cyan": "#4dd6c4", "white": "#c9cdd4", "brightBlack": "#8a909c", "brightRed": "#ff5760", "brightGreen": "#79c98a", "brightYellow": "#e3b341", "brightBlue": "#5aa9e6", "brightMagenta": "#c79bf0", "brightCyan": "#4dd6c4", "brightWhite": "#e9ebef", "background": "#111418", "foreground": "#c9cdd4", "selection": "#2a2f37", "cursor": "#e9ebef" } diff --git a/website/recordings/tapes/plan-load.tape b/website/recordings/tapes/plan-load.tape new file mode 100644 index 000000000..854f41568 --- /dev/null +++ b/website/recordings/tapes/plan-load.tape @@ -0,0 +1,36 @@ +Source tapes/common.tape +Output out/plan-load.gif +Output out/plan-load.mp4 +Hide +Type "cd scratch/plan-load/shop && unset NO_COLOR && export PS1='$ ' && clear" +Enter +Sleep 500ms +Show +Type "sqlload plan" +Sleep 300ms +Enter +Sleep 3.5s +Hide +Type "clear" +Enter +Sleep 200ms +Show +Type "sqlload load" +Sleep 300ms +Enter +Sleep 3.5s +Hide +Type "cp ../typo.toml sqlload.toml && clear" +Enter +Sleep 200ms +Show +Type "grep cursor_column sqlload.toml" +Sleep 200ms +Enter +Sleep 800ms +Type "sqlload load" +Sleep 300ms +Enter +Sleep 3s +Screenshot out/plan-load.png +Sleep 1s diff --git a/website/src/content/docs/docs/cli/load.mdx b/website/src/content/docs/docs/cli/load.mdx index e2d4b8ce4..6894b7233 100644 --- a/website/src/content/docs/docs/cli/load.mdx +++ b/website/src/content/docs/docs/cli/load.mdx @@ -159,11 +159,11 @@ tables of one run: the table that uses too much stands out. On other systems, `p ## Memory budget -A run stays under one memory budget, and `plan` shows it first, with where it came +A run stays under one memory budget, and `plan -v` shows it first, with where it came from and how it's shared: ```console -$ sqlload plan --memory-limit 1200MB +$ sqlload plan -v --memory-limit 1200MB Memory 1.1 GiB budget (--memory-limit); 39.8 MiB held before loading; DuckDB 488 MiB (memory_limit 244 MiB, 2 threads); 154 MiB per table, 4 at once Objects (2) sqlload apply