VGI for Rust

Add your own functions and tables to DuckDB with Rust and Apache Arrow.
Built by 🚜 Query.Farm

A VGI worker is a small Rust program that DuckDB talks to over Apache Arrow IPC. It can expose scalar / table / aggregate functions and whole catalogs (schemas, tables, views) that behave like native DuckDB objects. DuckDB launches your worker for you when a query needs it — you never run a server by hand.

This repo is the Rust worker SDK (vgi). It is byte-for-byte wire-compatible with the canonical Python SDK, so a Rust worker drops in behind the same ATTACH ... (TYPE vgi). Built on vgi-rpc; stock arrow-rs 59.x, MSRV 1.97.

HTTP producer batching and worker-visible response budgets are covered in docs/response-budgets.md.

Window evaluation retains decoded Arrow partitions in each dispatcher's LRU cache, avoiding repeated storage reads and IPC decoding for successive frames. The cache defaults to 256 entries and 256 MiB of accounted Arrow buffers, schemas, and filter masks. Set VGI_WINDOW_CACHE_MAX_BYTES before starting the worker to change the byte budget; 0 disables retention. Oversized or evicted partitions remain available through shared FunctionStorage. Window destruction removes the local partition, and aggregate destruction clears the local execution. Other worker processes have independent caches. On admission and partition destruction, a worker checks the small shared schema records to evict copies whose destructor ran elsewhere; cache hits do not read storage. Idle workers can retain copies until their next admission or eviction, subject to the per-worker budget. This bounds retained cache data, not total worker RSS or buffers still held by active requests. Retained capacity can exceed logical payload size: the benchmarked decoder gives a 128 MiB string partition a 256 MiB backing allocation before metadata. That partition exceeds the default budget; VGI_WINDOW_CACHE_MAX_BYTES=402653184 (384 MiB) allows it to be retained.

Why a worker instead of a C++ extension?

Traditional DuckDB extension	VGI worker
Written in C/C++, compiled and linked against DuckDB	Written in Rust, one standalone binary
Must be rebuilt for each DuckDB version	Version independent
Complex build / signing / release cycle	cargo build, ship the binary
Runs in-process	Process isolation

Reach for it when you want to: call REST APIs from SQL, run ML inference, expose an external database / API / filesystem as a queryable catalog, or ship domain-specific functions to your team as a single binary.

Your first worker

1. Create a project and add the dependencies:

# Cargo.toml
[dependencies]
vgi = "0.44"
vgi-rpc = "0.31"
arrow-array = "60"
arrow-schema = "60"

2. Write a function and serve it:

// src/main.rs
use std::sync::Arc;

use arrow_array::{cast::AsArray, ArrayRef, RecordBatch, StringArray};
use arrow_schema::DataType;
use vgi::{ArgumentMonotonicity, ArgSpec, FunctionMetadata, ProcessParams, ScalarFunction, Worker};
use vgi_rpc::{Result, RpcError};

/// `upper_case(s)` — uppercase a string column.
struct UpperCase;

impl ScalarFunction for UpperCase {
    fn name(&self) -> &str {
        "upper_case"
    }

    fn metadata(&self) -> FunctionMetadata {
        FunctionMetadata {
            description: "Convert string values to uppercase".into(),
            return_type: Some(DataType::Utf8),
            argument_monotonicity: Some(vec![ArgumentMonotonicity::Unknown]),
            ..Default::default()
        }
    }

    fn argument_specs(&self) -> Vec<ArgSpec> {
        vec![ArgSpec::column("value", 0, "varchar", "String to uppercase")]
    }

    fn process(&self, params: &ProcessParams, batch: &RecordBatch) -> Result<RecordBatch> {
        let col = batch.column(0).as_string::<i32>();
        let upper: StringArray = col.iter().map(|v| v.map(str::to_uppercase)).collect();
        let out: ArrayRef = Arc::new(upper);
        RecordBatch::try_new(params.output_schema.clone(), vec![out])
            .map_err(|e| RpcError::runtime_error(e.to_string()))
    }
}

fn main() {
    let mut worker = Worker::new();
    worker.register_scalar(UpperCase);
    worker.run(); // serves stdio (default), --unix <path>, or --http
}

argument_monotonicity is optional and scalar-only. A present vector has one entry per ordered argument declaration. Fixed, defaulted, and constant arguments each occupy one slot; a vararg declaration occupies one slot even when a call expands it. Named call syntax does not change this order.

3. Build it (cargo build --release), then call it from a DuckDB engine that has the vgi extension. The vgi extension currently ships with Query Farm's Haybarn DuckDB distribution, which starts with no install via uvx haybarn-cli. From your project directory:

-- Haybarn ships the `vgi` extension. DuckDB LAUNCHES the worker for you;
-- LOCATION is the command it runs, and the alias 'demo' is what you
-- qualify functions with in SQL.
ATTACH 'demo' (TYPE vgi, LOCATION './target/release/my-worker');

SELECT demo.main.upper_case(name) FROM (VALUES ('alice'), ('bob')) t(name);
-- ALICE
-- BOB

LOCATION gotcha: the path is resolved relative to DuckDB's working directory, not your project. If the worker isn't found, use an absolute path.

Iterating

Change your Rust, rebuild, and re-attach — DuckDB pools the worker per attachment, so DETACH demo; ATTACH 'demo' (...) (or a fresh session) picks up a new build.

Troubleshooting

• ATTACH can't find the worker — LOCATION is relative to DuckDB's working directory; use an absolute path.
• Catalog Error: ... does not exist — qualify with the attach alias (demo.main.upper_case) or run USE demo;.
• Runtime / type errors — errors returned from process (and bind-time argument_specs type checks) surface directly in DuckDB's error message.
• Telling bad input from a bug — every error carries a gRPC-style code (vgi_rpc.error_code). The SDK codes its own: bad arguments and type-bound failures are INVALID_ARGUMENT, unknown functions / tables / schemas are NOT_FOUND, writes to a read-only catalog are FAILED_PRECONDITION, unsupported operations are UNIMPLEMENTED; anything else is UNKNOWN. Use vgi::errors::{invalid_argument, not_found, failed_precondition, unimplemented} to code your own worker's errors the same way.

Function types

Type	Trait	SQL pattern	Use case
Scalar	ScalarFunction	SELECT f(col) FROM t	Per-row transforms (1:1)
Table	TableFunction	SELECT * FROM f(args)	Generate / scan data
Table-In-Out	TableInOutFunction	SELECT * FROM f((SELECT …))	Streaming transforms
Table-Buffering	TableBufferingFunction	SELECT * FROM f((SELECT …))	Aggregate-then-emit (sink → combine → source)
Aggregate	AggregateFunction	SELECT f(col) … GROUP BY …	Grouped / window / streaming aggregates

Each trait is small: name, metadata, argument_specs, an on_bind to resolve the output schema, and process (or the buffering / aggregate lifecycle methods). See filter ownership and typed pruning for safe external-data scans using FilterHandling::Sdk and partial-column candidate masks.

Projection & filter pushdown, ORDER BY / TABLESAMPLE hints, settings, secrets, bearer auth, and a cross-process state store are handled for you.

Beyond functions: full catalogs

Worker::set_catalog exposes a complete catalog — schemas, function-backed tables, views, and macros — with constraints, column statistics, time travel (AT), and secondary catalogs attachable by name:

ATTACH 'external_db' (TYPE vgi, LOCATION './my-catalog-worker');

SELECT * FROM external_db.main.users;            -- a function-backed table
SELECT * FROM external_db.analytics.daily_view;  -- a view
SELECT external_db.main.transform(col) FROM t;   -- a function

Transports

Worker::run picks the transport from argv: stdio (default), Unix socket (--unix <path>, the launcher contract), or HTTP (--http, Arrow-IPC over HTTP with AEAD-sealed stateless stream tokens and optional bearer auth).

Every transport hosts vgi.v2, vgi_rpc.Reflection.v1, and any protocols the worker adds with Worker::hosted_protocols; HTTP additionally hosts vgi_rpc.Identity.v1 when the worker sets Worker::resolve_token and/or Worker::mint_grant; with a grant key (--grant-key / VGI_RPC_GRANT_KEYS) it mints sealed grants and accepts them back as bearer credentials. With a grant source and a signing key (VGI_SIGNING_KEY), HTTP also hosts vgi.attach_tickets.v1: seal_attach turns a user's ATTACH (secret options included) into a vgia1. ticket that a runner holding the user's grant redeems with the single attach option vgi_attach_ticket. See docs/hosted-protocols.md.

Over HTTP the worker seals the attach_opaque_data and transaction_opaque_data values it hands the client (XChaCha20-Poly1305 under VGI_SIGNING_KEY / Worker::signing_key, or a key generated at startup when neither is set), binds each to its caller and each transaction to its attach, and refuses anything that does not open with the one error <field> not recognized. Secret attach options never enter either value on any transport. See vgi::opaque.

Protocol overview

VGI uses vgi-rpc, an Apache-Arrow-IPC RPC framework, for all DuckDB ↔ worker communication. You don't write to this directly — the traits handle it — but here's what happens per query:

DuckDB (client)                      VGI worker
  │──── bind(request) ─────────────▶ │  function name, args, input schema
  │◀─── BindResponse ───────────────  │  output schema (your on_bind)
  │──── init(request) ─────────────▶ │  start the processing stream
  │◀─── stream header ──────────────  │  execution_id, max_workers
  │──── process(batch) ────────────▶ │
  │◀─── output batch ───────────────  │  your process(batch)
  │──── [stream close] ────────────▶ │

Workspace layout

crate	published	summary
vgi/	✅ crates.io · docs.rs	The worker SDK: function models, declarative catalogs, wire dispatch, transports.
vgi-example-worker/	—	A fixture worker registering every function kind and full catalogs; drives the integration suite. publish = false.

Read vgi-example-worker/src/ for a working example of every trait — scalar, table, table-in-out, buffering, aggregate, and catalog-backed tables/views.

Testing your own worker

Enable vgi-client's optional testing feature to test your worker through attach, bind, and scan. TestWorker launches your binary; ScanResult compares typed schemas and row multisets, preserving duplicates and NULLs. Typed filter fixtures and check_pushdown compare a pushed scan with an independent reference, including projection and filter-only columns.

See Testing your worker for setup, expected-row and pushdown tests, boundary fixtures, and CI commands. The external-data example includes runnable subprocess tests. DuckDB remains useful as an independent SQL reference for conformance.

Running the SDK's integration suite (contributors)

The full behavioral suite is the canonical VGI C++ integration suite (test/sql/integration/* in the vgi extension repo), which drives DuckDB's unittest binary against the example worker. It passes across all three transports (8176 assertions on subprocess, 7774 on HTTP, 0 failures):

cargo build --release
scripts/run_tests.sh            # subprocess transport, full in-scope suite
LAUNCH=1 scripts/run_tests.sh   # launcher (Unix socket) transport
scripts/run_http_tests.sh       # HTTP transport

cargo fmt / clippy / build / doc run in CI.

For native iroh:// / httpi:// clients and bridge-ready workers, see Iroh operations.

Development

vgi depends on the published vgi-rpc from crates.io. To develop against an unreleased vgi-rpc checkout, add an uncommitted patch to the root Cargo.toml:

[patch.crates-io]
vgi-rpc = { path = "../vgi-rpc-rust/vgi-rpc" }

cargo build --workspace
cargo clippy --workspace --all-targets --all-features -- -D warnings
# every feature of vgi / vgi-client on its own (cargo install cargo-hack)
cargo hack clippy -p vgi -p vgi-client -p vgi-example-worker --each-feature --all-targets -- -D warnings
cargo test --doc -p vgi
cargo fmt --all

License

Query Farm Source-Available License v1.0 — see LICENSE. Copyright © 2025, 2026 Query Farm LLC.