Skip to content

About

Read-only JavaScript/TypeScript reader for Icechunk repositories, designed for use with zarrita.

Topics

Resources

Stars

28 stars

Watchers

2 watching

Forks

Repository files navigation

icechunk-js

npm version Icechunk Zarrita.js License: MIT

Read-only JavaScript/TypeScript reader for Icechunk repositories, designed for use with zarrita.

  • Pure TypeScript, works in browsers and Node.js 18+
  • Icechunk v1 and v2 format auto-detection
  • All chunk payload types: inline, native, and virtual

Getting Started

npm install icechunk-js

Basic Usage with zarrita

import { IcechunkStore } from "icechunk-js";
import { open, get } from "zarrita";

// Open a store from a URL
const store = await IcechunkStore.open("https://bucket.s3.amazonaws.com/repo");

// Open an array and read data
const array = await open(store.resolve("/temperature"), { kind: "array" });
const data = await get(array, [0, 0, null]);

For Development

npm install

npm run dev

npm run test

npm run typecheck

To regenerate FlatBuffers TypeScript after syncing schemas, run npm run generate:fbs. The pinned flatc compiler is downloaded automatically if not already available. The version is set in scripts/ensure-flatc.sh.

API

IcechunkStore

The main class for zarrita integration. Implements zarrita's AsyncReadable interface with both get() and getRange() (needed for sharded arrays).

Concurrent chunk reads are range-coalesced by default: reads issued in the same microtask turn that target the same backing object — one virtual-chunk file, or one native chunk object — are sorted, deduplicated and merged into one Range request per contiguous run, then sliced apart so each read gets exactly its bytes. (A zarr.get normally issues all its chunk reads in one turn; a createQueue concurrency limit, or chunks waiting on different manifests, split them into several batches.) Neighbours up to maxGap apart (default 256 KiB; the gap bytes are fetched and discarded) merge into requests of at most maxBytes (default 16 MiB) whose gap bytes stay within maxWasteRatio (default 1) times the bytes delivered. A failed merged request is retried one chunk at a time, and aborting one read never cancels the others. Tune it with coalesce: { maxGap, maxBytes, maxWasteRatio, window } (window > 0 batches on a setTimeout timer, which browsers throttle in background tabs), turn it off with coalesce: false, or read store.rangeCoalescer.stats to see requests issued and bytes fetched/discarded. Each IcechunkStore.open builds its own coalescer; use resolve() to derive sub-stores that share one.

zarrita's own withRangeCoalescing (zarrita >= 0.7) can be passed as withRangeCoalescing to replace the built-in wrapper.

Note: zarrita's wrapper merges abort signals with AbortSignal.any. If one read in a merged batch is aborted, other reads in the same batch may also reject. With it, avoid sharing an AbortController across requests that must cancel independently.

import { IcechunkStore } from "icechunk-js";
import { withRangeCoalescing } from "zarrita"; // optional

// Open from a URL (default: branch "main")
const store = await IcechunkStore.open("https://example.com/repo", {
  branch: "main",
  // tag: 'v1.0',
  // snapshot: 'ABC123...',
  // formatVersion: 'v1',     // skip format auto-detection for v1 repos
  // maxManifestCacheSize: 50, // LRU cache size (default: 100)
  // coalesce: { maxGap: 256 << 10, maxBytes: 16 << 20, maxWasteRatio: 1, window: 0 }, // or false
  // withRangeCoalescing, // zarrita's wrapper instead of the built-in one
  // signal: abortController.signal, // cancel initialization
  // validateChecksums: true,  // integrity headers for virtual chunks
  // azureAccount: 'myaccount', // required for az:// virtual chunks
});

// Open from an existing ReadSession
const store = await IcechunkStore.open(session);

// Open from a custom Storage backend
const store = await IcechunkStore.open(myStorage, { branch: "main" });

Store methods

// Scope to a subpath (shares the same session and cache)
const scoped = store.resolve("group/subgroup");

// Browse the hierarchy
const children = store.listChildren("/"); // direct children of root
const allNodes = store.listNodes(); // all nodes in the snapshot
const node = store.getNode("/temperature"); // single node by path
const meta = store.getMetadata("/temperature"); // parsed zarr.json

// Access the underlying session for advanced operations
const session = store.session;

Virtual chunk authentication

For private datasets with virtual chunks (S3, GCS, Azure), provide a fetchClient that handles authentication:

import type { FetchClient } from "icechunk-js";

const fetchClient: FetchClient = {
  async fetch(url, init) {
    // URL rewriting, pre-signing, or header injection happens here.
    // icechunk-js has already translated s3:// → https:// and built
    // Range headers in `init` (plus If-Match when validateChecksums is on).
    const signedUrl = await presign(url);
    return globalThis.fetch(signedUrl, {
      ...init,
      headers: { ...init?.headers, Authorization: `Bearer ${token}` },
    });
  },
};

const store = await IcechunkStore.open("https://example.com/repo", {
  fetchClient,
});

Cloud storage URLs in virtual chunk references are automatically translated:

  • s3://bucket/key → https://bucket.s3.amazonaws.com/key
  • gs://bucket/key (or gcs://) → https://storage.googleapis.com/bucket/key
  • az://container/path (or azure://) → https://{azureAccount}.blob.core.windows.net/container/path
  • abfs://container@account.dfs.core.windows.net/path → https://account.blob.core.windows.net/container/path

For S3, addressing follows the repo's virtual-chunk-container config (region, endpoint, path-style), mirroring the Rust implementation. Buckets whose name contains a dot must use path-style addressing; when the container config records a region (repos written by Icechunk do), reads go straight to that regional endpoint — which serves CORS and works in both Node and the browser, with no extra configuration.

If no region is known, dotted-name buckets fall back to the global endpoint, whose region redirect (for buckets outside us-east-1) is resolved automatically in Node but not in browsers — the cross-origin redirect carries no CORS headers. Supply a fetchClient that routes to the regional endpoint or a CORS-enabled proxy in that case.

Repository

For direct access to branches, tags, and checkouts.

Note: Over plain HTTP, listBranches() and listTags() only work reliably with v2 repos, which embed refs in the top-level repo file. For v1 repos, direct checkoutBranch() / checkoutTag() can work when the target ref still lives at the legacy ref.json path, but versioned ref filenames still require listPrefix() discovery, which HttpStorage does not provide. Use a listing-capable storage backend for full v1 branch/tag support.

import { Repository, HttpStorage } from "icechunk-js";

// Replace with the root URL of a real Icechunk repository.
const storage = new HttpStorage("https://example.com/repo");

// Auto-detect format (default)
const repo = await Repository.open({ storage });

// Or with format version hint (skips /repo request for v1 stores)
// const repo = await Repository.open({ storage, formatVersion: 'v1' });

// List branches and tags (v2 repos, or storage backends that support listing)
const branches = await repo.listBranches();
const tags = await repo.listTags();

// Checkout to get a ReadSession
const session = await repo.checkoutBranch("main");
// or: repo.checkoutTag('v1.0')
// or: repo.checkoutSnapshot('ABCDEFGHIJKLMNOP')

Walking commit history

for await (const entry of repo.walkHistory(session)) {
  console.log(entry.id, entry.message, entry.flushedAt, entry.metadata);
}

ReadSession

Low-level access to nodes, chunks, and snapshot metadata.

// Snapshot info
const snapshotId = session.getSnapshotId();
const parentId = session.getParentSnapshotId(); // null for root
const message = session.getMessage();
const timestamp = session.getFlushedAt();
const metadata = session.getSnapshotMetadata();

// Navigate the hierarchy
const nodes = session.listNodes();
const children = session.listChildren("/group");
const node = session.getNode("/array");

// Get Zarr metadata and chunks
const zarrMeta = session.getMetadata("/array");
const chunk = await session.getChunk("/array", [0, 0, 0]);

// Transaction log (what changed in this snapshot)
const txLog = await session.loadTransactionLog();
if (txLog) {
  console.log("New arrays:", txLog.newArrays.length);
  console.log("Updated chunks:", txLog.updatedChunks.length);
}

HttpStorage

HTTP/HTTPS storage backend using the Fetch API. Works in Node.js 18+ and browsers.

import { HttpStorage } from "icechunk-js";

const storage = new HttpStorage("https://bucket.s3.amazonaws.com/repo", {
  headers: { Authorization: "Bearer token" },
  credentials: "include",
  cache: "no-store",
});

Custom Storage

Implement the Storage interface for other backends:

import type { Storage, ByteRange, RequestOptions } from 'icechunk-js';

class MyStorage implements Storage {
  async getObject(path: string, range?: ByteRange, options?: RequestOptions): Promise<Uint8Array> { ... }
  async exists(path: string, options?: RequestOptions): Promise<boolean> { ... }
  async *listPrefix(prefix: string): AsyncIterable<string> { ... }
}

License

MIT

About

Read-only JavaScript/TypeScript reader for Icechunk repositories, designed for use with zarrita.

Topics

Resources

Stars

28 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages