Context
Compressed data is the norm outside toy examples: log archives, data dumps, and curl | gzip pipelines all ship .gz. Today users must decompress first:
zcat huge.csv.gz | sql-pipe 'SELECT ...' # extra step, loses file-as-table ergonomics
Zig stdlib provides std.compress.flate/gzip — zero new dependencies.
Proposed behavior
- File argument ending in
.gz (or starting with magic bytes 1f 8b) is decompressed transparently before the format loader.
- Stdin with gzip magic bytes is decompressed the same way (sniff first 2 bytes of the buffered stream).
- Format detection uses the inner extension:
data.csv.gz → CSV.
- Table name derived from the inner name:
data.csv.gz → table data.
sql-pipe data.csv.gz 'SELECT COUNT(*) FROM data'
curl -s https://example.com/dump.ndjson.gz | sql-pipe 'SELECT * FROM t'
Scope (v1)
- gzip only (magic
1f 8b + .gz suffix).
- Streaming decompress into the existing buffered reader — no full in-memory inflation.
Excluded (YAGNI)
- zstd, xz, bzip2, zip archives
- Compressed output
- Multi-member gzip concatenation beyond what std.compress handles
Acceptance criteria
Context
Compressed data is the norm outside toy examples: log archives, data dumps, and
curl | gzippipelines all ship.gz. Today users must decompress first:Zig stdlib provides
std.compress.flate/gzip— zero new dependencies.Proposed behavior
.gz(or starting with magic bytes1f 8b) is decompressed transparently before the format loader.data.csv.gz→ CSV.data.csv.gz→ tabledata.Scope (v1)
1f 8b+.gzsuffix).Excluded (YAGNI)
Acceptance criteria
sql-pipe data.csv.gz 'SELECT ...'works withoutzcatdata.ndjson.gzloads as NDJSON, tabledata