Skip to content

Feature: transparent gzip input (.csv.gz files and stdin) #220

Description

@vmvarela

Context

Compressed data is the norm outside toy examples: log archives, data dumps, and curl | gzip pipelines all ship .gz. Today users must decompress first:

zcat huge.csv.gz | sql-pipe 'SELECT ...'   # extra step, loses file-as-table ergonomics

Zig stdlib provides std.compress.flate/gzip — zero new dependencies.

Proposed behavior

  • File argument ending in .gz (or starting with magic bytes 1f 8b) is decompressed transparently before the format loader.
  • Stdin with gzip magic bytes is decompressed the same way (sniff first 2 bytes of the buffered stream).
  • Format detection uses the inner extension: data.csv.gz → CSV.
  • Table name derived from the inner name: data.csv.gz → table data.
sql-pipe data.csv.gz 'SELECT COUNT(*) FROM data'
curl -s https://example.com/dump.ndjson.gz | sql-pipe 'SELECT * FROM t'

Scope (v1)

  • gzip only (magic 1f 8b + .gz suffix).
  • Streaming decompress into the existing buffered reader — no full in-memory inflation.

Excluded (YAGNI)

  • zstd, xz, bzip2, zip archives
  • Compressed output
  • Multi-member gzip concatenation beyond what std.compress handles

Acceptance criteria

  • sql-pipe data.csv.gz 'SELECT ...' works without zcat
  • Gzipped stdin detected by magic bytes
  • data.ndjson.gz loads as NDJSON, table data
  • Clear error on corrupt gzip stream
  • Non-gzip input path unchanged (no sniffing overhead beyond 2 bytes)
  • Integration test with a gzipped fixture

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority:mediumShould be done soonsize:sSmall — 1 to 4 hoursstatus:readyRefined and ready for sprint selectiontype:featureNew functionality

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions