Read-only JavaScript/TypeScript reader for Icechunk repositories, designed for use with zarrita.
- Pure TypeScript, works in browsers and Node.js 18+
- Icechunk v1 and v2 format auto-detection
- All chunk payload types: inline, native, and virtual
npm install icechunk-jsimport { IcechunkStore } from "icechunk-js";
import { open, get } from "zarrita";
// Open a store from a URL
const store = await IcechunkStore.open("https://bucket.s3.amazonaws.com/repo");
// Open an array and read data
const array = await open(store.resolve("/temperature"), { kind: "array" });
const data = await get(array, [0, 0, null]);npm install
npm run dev
npm run test
npm run typecheckTo regenerate FlatBuffers TypeScript after syncing schemas, run
npm run generate:fbs. The pinned flatc compiler is downloaded automatically
if not already available. The version is set in scripts/ensure-flatc.sh.
The main class for zarrita integration. Implements zarrita's AsyncReadable
interface with both get() and getRange() (needed for sharded arrays).
Concurrent chunk reads are range-coalesced by default: reads issued in the same
microtask turn that target the same backing object — one virtual-chunk file, or
one native chunk object — are sorted, deduplicated and merged into one Range
request per contiguous run, then sliced apart so each read gets exactly its
bytes. (A zarr.get normally issues all its chunk reads in one turn; a
createQueue concurrency limit, or chunks waiting on different manifests,
split them into several batches.) Neighbours up to maxGap apart (default
256 KiB; the gap bytes are fetched and discarded) merge into requests of at
most maxBytes (default 16 MiB) whose gap bytes stay within maxWasteRatio
(default 1) times the bytes delivered. A failed merged request is retried one
chunk at a time, and aborting one read never cancels the others. Tune it with
coalesce: { maxGap, maxBytes, maxWasteRatio, window } (window > 0 batches
on a setTimeout timer, which browsers throttle in background tabs), turn it
off with coalesce: false, or read store.rangeCoalescer.stats to see
requests issued and bytes fetched/discarded. Each IcechunkStore.open builds
its own coalescer; use resolve() to derive sub-stores that share one.
zarrita's own withRangeCoalescing (zarrita >= 0.7) can be passed as
withRangeCoalescing to replace the built-in wrapper.
Note: zarrita's wrapper merges abort signals with
AbortSignal.any. If one read in a merged batch is aborted, other reads in the same batch may also reject. With it, avoid sharing anAbortControlleracross requests that must cancel independently.
import { IcechunkStore } from "icechunk-js";
import { withRangeCoalescing } from "zarrita"; // optional
// Open from a URL (default: branch "main")
const store = await IcechunkStore.open("https://example.com/repo", {
branch: "main",
// tag: 'v1.0',
// snapshot: 'ABC123...',
// formatVersion: 'v1', // skip format auto-detection for v1 repos
// maxManifestCacheSize: 50, // LRU cache size (default: 100)
// coalesce: { maxGap: 256 << 10, maxBytes: 16 << 20, maxWasteRatio: 1, window: 0 }, // or false
// withRangeCoalescing, // zarrita's wrapper instead of the built-in one
// signal: abortController.signal, // cancel initialization
// validateChecksums: true, // integrity headers for virtual chunks
// azureAccount: 'myaccount', // required for az:// virtual chunks
});
// Open from an existing ReadSession
const store = await IcechunkStore.open(session);
// Open from a custom Storage backend
const store = await IcechunkStore.open(myStorage, { branch: "main" });// Scope to a subpath (shares the same session and cache)
const scoped = store.resolve("group/subgroup");
// Browse the hierarchy
const children = store.listChildren("/"); // direct children of root
const allNodes = store.listNodes(); // all nodes in the snapshot
const node = store.getNode("/temperature"); // single node by path
const meta = store.getMetadata("/temperature"); // parsed zarr.json
// Access the underlying session for advanced operations
const session = store.session;For private datasets with virtual chunks (S3, GCS, Azure), provide a
fetchClient that handles authentication:
import type { FetchClient } from "icechunk-js";
const fetchClient: FetchClient = {
async fetch(url, init) {
// URL rewriting, pre-signing, or header injection happens here.
// icechunk-js has already translated s3:// → https:// and built
// Range headers in `init` (plus If-Match when validateChecksums is on).
const signedUrl = await presign(url);
return globalThis.fetch(signedUrl, {
...init,
headers: { ...init?.headers, Authorization: `Bearer ${token}` },
});
},
};
const store = await IcechunkStore.open("https://example.com/repo", {
fetchClient,
});Cloud storage URLs in virtual chunk references are automatically translated:
s3://bucket/key→https://bucket.s3.amazonaws.com/keygs://bucket/key(orgcs://) →https://storage.googleapis.com/bucket/keyaz://container/path(orazure://) →https://{azureAccount}.blob.core.windows.net/container/pathabfs://container@account.dfs.core.windows.net/path→https://account.blob.core.windows.net/container/path
For S3, addressing follows the repo's virtual-chunk-container config (region, endpoint, path-style), mirroring the Rust implementation. Buckets whose name contains a dot must use path-style addressing; when the container config records a region (repos written by Icechunk do), reads go straight to that regional endpoint — which serves CORS and works in both Node and the browser, with no extra configuration.
If no region is known, dotted-name buckets fall back to the global endpoint,
whose region redirect (for buckets outside us-east-1) is resolved
automatically in Node but not in browsers — the cross-origin redirect carries no
CORS headers. Supply a fetchClient that routes
to the regional endpoint or a CORS-enabled proxy in that case.
For direct access to branches, tags, and checkouts.
Note: Over plain HTTP,
listBranches()andlistTags()only work reliably with v2 repos, which embed refs in the top-levelrepofile. For v1 repos, directcheckoutBranch()/checkoutTag()can work when the target ref still lives at the legacyref.jsonpath, but versioned ref filenames still requirelistPrefix()discovery, whichHttpStoragedoes not provide. Use a listing-capable storage backend for full v1 branch/tag support.
import { Repository, HttpStorage } from "icechunk-js";
// Replace with the root URL of a real Icechunk repository.
const storage = new HttpStorage("https://example.com/repo");
// Auto-detect format (default)
const repo = await Repository.open({ storage });
// Or with format version hint (skips /repo request for v1 stores)
// const repo = await Repository.open({ storage, formatVersion: 'v1' });
// List branches and tags (v2 repos, or storage backends that support listing)
const branches = await repo.listBranches();
const tags = await repo.listTags();
// Checkout to get a ReadSession
const session = await repo.checkoutBranch("main");
// or: repo.checkoutTag('v1.0')
// or: repo.checkoutSnapshot('ABCDEFGHIJKLMNOP')for await (const entry of repo.walkHistory(session)) {
console.log(entry.id, entry.message, entry.flushedAt, entry.metadata);
}Low-level access to nodes, chunks, and snapshot metadata.
// Snapshot info
const snapshotId = session.getSnapshotId();
const parentId = session.getParentSnapshotId(); // null for root
const message = session.getMessage();
const timestamp = session.getFlushedAt();
const metadata = session.getSnapshotMetadata();
// Navigate the hierarchy
const nodes = session.listNodes();
const children = session.listChildren("/group");
const node = session.getNode("/array");
// Get Zarr metadata and chunks
const zarrMeta = session.getMetadata("/array");
const chunk = await session.getChunk("/array", [0, 0, 0]);
// Transaction log (what changed in this snapshot)
const txLog = await session.loadTransactionLog();
if (txLog) {
console.log("New arrays:", txLog.newArrays.length);
console.log("Updated chunks:", txLog.updatedChunks.length);
}HTTP/HTTPS storage backend using the Fetch API. Works in Node.js 18+ and browsers.
import { HttpStorage } from "icechunk-js";
const storage = new HttpStorage("https://bucket.s3.amazonaws.com/repo", {
headers: { Authorization: "Bearer token" },
credentials: "include",
cache: "no-store",
});Implement the Storage interface for other backends:
import type { Storage, ByteRange, RequestOptions } from 'icechunk-js';
class MyStorage implements Storage {
async getObject(path: string, range?: ByteRange, options?: RequestOptions): Promise<Uint8Array> { ... }
async exists(path: string, options?: RequestOptions): Promise<boolean> { ... }
async *listPrefix(prefix: string): AsyncIterable<string> { ... }
}MIT