Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified .github/demos/plan-load.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
56 changes: 33 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,26 +19,29 @@
<a href="https://sqlload.io/docs/benchmarks/">Benchmarks</a>
</p>

Load Postgres, files, Snowflake, BigQuery, Google Sheets and Python sources into your warehouse.
SQLLoad is Rust and Arrow end to end, and when your warehouse reads the files itself, rows never
pass through SQLLoad, so there's no loader bottleneck as your data grows. It's fast, too: faster
than ingestr and dlt in every [benchmark](https://sqlload.io/docs/benchmarks/) we publish.
SQLLoad loads databases, files, APIs and Kafka into your warehouse. `sqlload plan` shows every
table and how it loads before anything moves, and `plan -v` adds each new table's column types and
the run's memory budget. Decimals keep their precision, a value a column can't hold exactly fails
its table, naming the column and the value, and each run has one memory budget. Tables with a
cursor column carry on from the warehouse's own maximum, so there's no state to drift.

```bash
pip install sqlload # or: uv tool install sqlload
```

<p align="center">
<img src="https://raw.githubusercontent.com/sqlstack-io/sqlload/main/.github/demos/plan-load.gif" alt="sqlload plan lists the two tables it will create from Postgres, their column types in DuckDB, and how each loads; sqlload load then creates them and loads 1,050 rows" width="100%">
<img src="https://raw.githubusercontent.com/sqlstack-io/sqlload/main/.github/demos/plan-load.gif" alt="In the playground project, sqlload plan lists the two tables it will create in DuckDB and how each loads from CSV files; sqlload load creates them and loads 1,050 rows, reporting the slowest table and peak memory; with cursor_column misspelled as updated, the next load refuses raw.orders before loading it: did you mean updated_at?" width="100%">
</p>

`sqlload plan` shows the tables it will create from Postgres, each column's DuckDB type and how
each table loads. `sqlload load` then creates and fills them.
In the [playground](https://sqlload.io/docs/cli/playground/) project, `sqlload plan` shows the
tables it will create and how each loads, and `sqlload load` creates and fills them. With
`cursor_column` misspelled, the next load refuses the orders table before touching it, and says
which column was meant.

## Quick start

```bash
sqlload plan # every table, column type and route, before anything is created
sqlload plan # every table and how it loads, before anything is created (-v: column types)
sqlload apply # create the tables
sqlload load # load them
```
Expand All @@ -56,7 +59,26 @@ Moving from dlt? [Coming from dlt](https://sqlload.io/docs/guides/from-dlt/) map
write dispositions, incremental loading, `rest_api` and `filesystem`, and keeps the table and column
names dlt gave your tables.

## Speed
## Why SQLLoad

- **Exact types.** A source's `decimal(12,2)` loads as `DECIMAL(12,2)` (`NUMBER(12,2)` on
Snowflake), timestamps keep their
microseconds, and a pinned type narrower than the live column fails `plan` rather than cutting
values. See [contracts](https://sqlload.io/docs/concepts/contracts/).
- **Loud failures.** `plan` names each problem it can find, such as a misspelled column, a
narrowing type or a budget too small, and its fix, before anything is created. A commit whose
[checks](https://sqlload.io/docs/concepts/checks/) fail publishes none of its rows, every commit
lands whole even if a load is killed halfway, and in a warehouse (except a ClickHouse `Ordinary` database) a full refresh swaps in at once,
so readers see the old table or the new one, never a mix.
- **Bounded memory.** A run has one [memory budget](https://sqlload.io/docs/cli/load/#memory-budget),
shared out between its tables. A budget too small for even one table is refused before anything
loads, and near the limit SQLLoad stops a table, or the run, and rolls back what it was
writing, rather than waiting for the operating system to kill it.
- **No hidden state.** Tables with a `cursor_column` carry on from the
[warehouse's own maximum](https://sqlload.io/docs/concepts/cursors/) of the cursor column, so a
table you fix or reload by hand never disagrees with a state file.

## Speed and memory

Faster, in a fraction of the memory. The same data loaded by each tool on one 4-vCPU, 8 GB machine;
lower is better:
Expand All @@ -77,18 +99,6 @@ peak memory 131 MiB for SQLLoad, 495 MiB for ingestr and 341 MiB for dlt.
See the [benchmarks](https://sqlload.io/docs/benchmarks/) for every scenario, the method, the
versions and how to run them yourself.

## Why SQLLoad

- **See the plan before any data moves.** `sqlload plan` shows every table, column type, write
strategy and route before anything is created or loaded, so there are no surprises in your
warehouse.
- **Crash-safe loads.** Every commit lands whole, even if a load is killed halfway. A full refresh
swaps in one go, so your dashboards see the old table or the new one, never a mix.
- **Postgres change data capture.** A `postgres_cdc` source copies your tables once, then applies
every insert, update and delete from a logical replication slot, in whole source transactions.
`sqlload load --follow` keeps streaming. See
[Postgres CDC](https://sqlload.io/docs/sources/postgres-cdc/).

## What it loads

| System | Source | Target |
Expand All @@ -98,8 +108,8 @@ versions and how to run them yourself.
| BigQuery (beta) | ✓ | ✓ |
| Databricks (beta) | ✓ | ✓ |
| ClickHouse | | ✓ |
| DuckDB | | ✓ |
| MotherDuck (beta) | | ✓ |
| DuckDB | ✓ | ✓ |
| MotherDuck (beta) | ✓ | ✓ |
| ODBC | ✓ SQL Server, MySQL, OpenEdge | |
| Files | ✓ Parquet, CSV, JSON, Excel, XML | ✓ Parquet, CSV |
| Google Sheets | ✓ | |
Expand Down
81 changes: 46 additions & 35 deletions packages/sqlload/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,26 +19,29 @@
<a href="https://sqlload.io/docs/benchmarks/">Benchmarks</a>
</p>

Load Postgres, files, Snowflake, BigQuery, Google Sheets and Python sources into your warehouse.
SQLLoad is Rust and Arrow end to end, and when your warehouse reads the files itself, rows never
pass through SQLLoad, so there's no loader bottleneck as your data grows. It's fast, too: faster
than ingestr and dlt in every [benchmark](https://sqlload.io/docs/benchmarks/) we publish.
SQLLoad loads databases, files, APIs and Kafka into your warehouse. `sqlload plan` shows every
table and how it loads before anything moves, and `plan -v` adds each new table's column types and
the run's memory budget. Decimals keep their precision, a value a column can't hold exactly fails
its table, naming the column and the value, and each run has one memory budget. Tables with a
cursor column carry on from the warehouse's own maximum, so there's no state to drift.

```bash
uv tool install sqlload # or: pip install sqlload
pip install sqlload # or: uv tool install sqlload
```

<p align="center">
<img src="https://raw.githubusercontent.com/sqlstack-io/sqlload/main/.github/demos/plan-load.gif" alt="sqlload plan lists the two tables it will create from Postgres, their column types in DuckDB, and how each loads; sqlload load then creates them and loads 1,050 rows" width="100%">
<img src="https://raw.githubusercontent.com/sqlstack-io/sqlload/main/.github/demos/plan-load.gif" alt="In the playground project, sqlload plan lists the two tables it will create in DuckDB and how each loads from CSV files; sqlload load creates them and loads 1,050 rows, reporting the slowest table and peak memory; with cursor_column misspelled as updated, the next load refuses raw.orders before loading it: did you mean updated_at?" width="100%">
</p>

`sqlload plan` shows the tables it will create from Postgres, each column's DuckDB type and how
each table loads. `sqlload load` then creates and fills them.
In the [playground](https://sqlload.io/docs/cli/playground/) project, `sqlload plan` shows the
tables it will create and how each loads, and `sqlload load` creates and fills them. With
`cursor_column` misspelled, the next load refuses the orders table before touching it, and says
which column was meant.

## Quick start

```bash
sqlload plan # every table, column type and route, before anything is created
sqlload plan # every table and how it loads, before anything is created (-v: column types)
sqlload apply # create the tables
sqlload load # load them
```
Expand All @@ -52,7 +55,30 @@ configuration file:
sqlload copy orders/ duckdb://shop.duckdb#raw.orders
```

## Speed
Moving from dlt? [Coming from dlt](https://sqlload.io/docs/guides/from-dlt/) maps sources,
write dispositions, incremental loading, `rest_api` and `filesystem`, and keeps the table and column
names dlt gave your tables.

## Why SQLLoad

- **Exact types.** A source's `decimal(12,2)` loads as `DECIMAL(12,2)` (`NUMBER(12,2)` on
Snowflake), timestamps keep their
microseconds, and a pinned type narrower than the live column fails `plan` rather than cutting
values. See [contracts](https://sqlload.io/docs/concepts/contracts/).
- **Loud failures.** `plan` names each problem it can find, such as a misspelled column, a
narrowing type or a budget too small, and its fix, before anything is created. A commit whose
[checks](https://sqlload.io/docs/concepts/checks/) fail publishes none of its rows, every commit
lands whole even if a load is killed halfway, and in a warehouse (except a ClickHouse `Ordinary` database) a full refresh swaps in at once,
so readers see the old table or the new one, never a mix.
- **Bounded memory.** A run has one [memory budget](https://sqlload.io/docs/cli/load/#memory-budget),
shared out between its tables. A budget too small for even one table is refused before anything
loads, and near the limit SQLLoad stops a table, or the run, and rolls back what it was
writing, rather than waiting for the operating system to kill it.
- **No hidden state.** Tables with a `cursor_column` carry on from the
[warehouse's own maximum](https://sqlload.io/docs/concepts/cursors/) of the cursor column, so a
table you fix or reload by hand never disagrees with a state file.

## Speed and memory

Faster, in a fraction of the memory. The same data loaded by each tool on one 4-vCPU, 8 GB machine;
lower is better:
Expand All @@ -73,37 +99,22 @@ peak memory 131 MiB for SQLLoad, 495 MiB for ingestr and 341 MiB for dlt.
See the [benchmarks](https://sqlload.io/docs/benchmarks/) for every scenario, the method, the
versions and how to run them yourself.

## Why SQLLoad

- **See the plan before any data moves.** `sqlload plan` shows every table, column type, write
strategy and route before anything is created or loaded, so there are no surprises in your
warehouse.
- **Crash-safe loads.** Every commit lands whole, even if a load is killed halfway. A full refresh
swaps in one go, so your dashboards see the old table or the new one, never a mix.
- **Postgres change data capture.** A `postgres_cdc` source copies your tables once, then applies
every insert, update and delete from a logical replication slot, in whole source transactions.
`sqlload load --follow` keeps streaming. See
[Postgres CDC](https://sqlload.io/docs/sources/postgres-cdc/).

## What it loads

| System | Source | Target |
|---|---|---|
| Postgres | ✓ (incl. CDC) | ✓ |
| Postgres | ✓ (incl. [CDC](https://sqlload.io/docs/sources/postgres-cdc/)) | ✓ |
| Snowflake | ✓ | ✓ |
| BigQuery | ✓ | preview |
| Databricks | | ✓ |
| ClickHouse | | preview |
| DuckDB | | ✓ |
| MotherDuck | | beta² |
| Files | ✓ Parquet, CSV, JSON (S3, local folder, SFTP) | ✓ Parquet, CSV (S3, local folder) |
| BigQuery (beta) | ✓ | ✓ |
| Databricks (beta) | ✓ | ✓ |
| ClickHouse | | ✓ |
| DuckDB | ✓ | ✓ |
| MotherDuck (beta) | ✓ | ✓ |
| ODBC | ✓ SQL Server, MySQL, OpenEdge | |
| Files | ✓ Parquet, CSV, JSON, Excel, XML | ✓ Parquet, CSV |
| Google Sheets | ✓ | |
| HTTP APIs and Python | ✓ | |
| ODBC databases¹ | ✓ | |

¹ In the published wheels from 0.39.0; before that, in builds with the `odbc` feature.

² From 0.34.0, through a separate DuckDB 1.5 process: install `sqlload[motherduck]`.
| APIs, Python | ✓ | |
| Kafka | ✓ to Snowflake, Databricks, BigQuery | |

See [Supported systems](https://sqlload.io/docs/supported-systems/) for every combination and
write strategy.
Expand Down
2 changes: 2 additions & 0 deletions website/recordings/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
out/
scratch/
35 changes: 35 additions & 0 deletions website/recordings/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# CLI recordings

Terminal recordings of `sqlload` for the README, rendered with
[VHS](https://github.com/charmbracelet/vhs) from the `.tape` scripts in `tapes/`. The tapes and
scripts are the source; the rendered GIF, MP4 and PNG files go to `out/`, which is not committed.

| Demo | Shows |
|------|-------|
| `plan-load` | In the playground project, `sqlload plan`, then `sqlload load`, then a load that stops on a misspelled `cursor_column`. |

The README uses `plan-load`, copied to `.github/demos/plan-load.gif`. It runs locally on DuckDB,
with the project `sqlload playground` writes.

## Requirements

- `vhs`, `ttyd` and `ffmpeg` on `PATH`, plus a Chromium or Chrome that VHS can find (it downloads
one if none is on `PATH`).
- DejaVu Sans Mono, the font the tapes use.
- `sqlload` on `PATH`, or `SQLLOAD_BIN` pointing at a directory that contains the `sqlload` to
record. Record with the latest release, so the output matches what users see.

## Record

```bash
cd website/recordings
./record.sh plan-load # out/plan-load.gif, out/plan-load.mp4 and out/plan-load.png
cp out/plan-load.gif ../../.github/demos/
```

`record.sh` runs `prepare.sh <demo>` to build a clean project under `scratch/<demo>/`, then renders
`tapes/<demo>.tape`. Shared terminal settings and the colour theme, which matches the website's
code blocks, are in `tapes/common.tape`.

Look at every frame before publishing a recording: output such as timings and memory changes
between runs.
20 changes: 20 additions & 0 deletions website/recordings/prepare.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
# Build a clean scratch project for one demo. Usage: prepare.sh <demo>
set -euo pipefail
demo="$1"
here="$(cd "$(dirname "$0")" && pwd)"
dir="$here/scratch/$demo"
rm -rf "$dir"
mkdir -p "$dir"
cd "$dir"
case "$demo" in
plan-load)
# The playground project, and a copy of its settings with cursor_column misspelled, which the
# tape swaps in after the first load.
sqlload playground shop >/dev/null
awk '{print} /^unique_key = \["order_id"\]$/ {print "cursor_column = \"updated\""}' shop/sqlload.toml > typo.toml
grep -q '^cursor_column = "updated"' typo.toml || { echo "prepare.sh: no unique_key line to add the typo after" >&2; exit 1; } ;;
*)
echo "unknown demo: $demo" >&2
exit 2 ;;
esac
14 changes: 14 additions & 0 deletions website/recordings/record.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
# Record one demo to out/<demo>.gif, out/<demo>.mp4 and out/<demo>.png. Usage: record.sh <demo>
# Uses the sqlload on PATH; set SQLLOAD_BIN to a directory containing another sqlload to use that instead.
set -euo pipefail
demo="$1"
here="$(cd "$(dirname "$0")" && pwd)"
if [[ -n "${SQLLOAD_BIN:-}" ]]; then
export PATH="$SQLLOAD_BIN:$PATH"
fi
"$here/prepare.sh" "$demo"
mkdir -p "$here/out"
cd "$here"
# Headless Chromium's sandbox is unavailable in many containers and CI runners.
VHS_NO_SANDBOX="${VHS_NO_SANDBOX:-true}" vhs "tapes/$demo.tape"
13 changes: 13 additions & 0 deletions website/recordings/tapes/common.tape
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
Set Shell bash
Set FontFamily "DejaVu Sans Mono"
Set FontSize 18
Set LineHeight 1.0
Set Width 1200
Set Height 520
Set Padding 28
Set Margin 24
Set MarginFill "#0d0f12"
Set BorderRadius 10
Set WindowBar Colorful
Set TypingSpeed 40ms
Set Theme { "name": "SQLLoad", "black": "#5c606b", "red": "#ff5760", "green": "#79c98a", "yellow": "#e3b341", "blue": "#5aa9e6", "magenta": "#c79bf0", "cyan": "#4dd6c4", "white": "#c9cdd4", "brightBlack": "#8a909c", "brightRed": "#ff5760", "brightGreen": "#79c98a", "brightYellow": "#e3b341", "brightBlue": "#5aa9e6", "brightMagenta": "#c79bf0", "brightCyan": "#4dd6c4", "brightWhite": "#e9ebef", "background": "#111418", "foreground": "#c9cdd4", "selection": "#2a2f37", "cursor": "#e9ebef" }
36 changes: 36 additions & 0 deletions website/recordings/tapes/plan-load.tape
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
Source tapes/common.tape
Output out/plan-load.gif
Output out/plan-load.mp4
Hide
Type "cd scratch/plan-load/shop && unset NO_COLOR && export PS1='$ ' && clear"
Enter
Sleep 500ms
Show
Type "sqlload plan"
Sleep 300ms
Enter
Sleep 3.5s
Hide
Type "clear"
Enter
Sleep 200ms
Show
Type "sqlload load"
Sleep 300ms
Enter
Sleep 3.5s
Hide
Type "cp ../typo.toml sqlload.toml && clear"
Enter
Sleep 200ms
Show
Type "grep cursor_column sqlload.toml"
Sleep 200ms
Enter
Sleep 800ms
Type "sqlload load"
Sleep 300ms
Enter
Sleep 3s
Screenshot out/plan-load.png
Sleep 1s
4 changes: 2 additions & 2 deletions website/src/content/docs/docs/cli/load.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -159,11 +159,11 @@ tables of one run: the table that uses too much stands out. On other systems, `p

## Memory budget

A run stays under one memory budget, and `plan` shows it first, with where it came
A run stays under one memory budget, and `plan -v` shows it first, with where it came
from and how it's shared:

```console
$ sqlload plan --memory-limit 1200MB
$ sqlload plan -v --memory-limit 1200MB
Memory 1.1 GiB budget (--memory-limit); 39.8 MiB held before loading; DuckDB 488 MiB (memory_limit 244 MiB, 2 threads); 154 MiB per table, 4 at once

Objects (2) sqlload apply
Expand Down
Loading