[prrao87]

Why

This fork starts at lance-graph 3014793. We developed lance-csr separately, on top of that same commit, and the changes are too broad to replay as separate diffs: dependency upgrades, removed subsystems, a new execution path and much wider Cypher support. This PR replaces the tree with lance-csr as its first release, 0.1.0, which we then plan to publish to PyPI as lance-csr and maintain from here.

What changed

Dependencies and scope

• Upgraded to Lance 13, pylance 13, DataFusion 54, Arrow 58 and PyO3 0.28. lance-graph's Lance 1.0 reader panics on data written by current pylance.
• Removed Delta Lake and Unity Catalog support, the connector layer, and the legacy application code.
• The Python package is now lance_csr (distribution lance-csr). The Rust crates keep their lance-graph* names for now.

Engine

• CypherEngine holds Lance snapshots instead of copies. You choose per table: pass lance.dataset(...) to keep a table on disk (read lazily, and refreshable), or .to_table() to load it into memory.
• refresh() swaps in new Lance versions atomically, and execute_batches() streams results.
• Lance inputs were silently capped at 10,000 rows; they now read in full.

CSR traversal (use_csr=True)

• A CSR adjacency index runs as DataFusion extension operators. It replaces relationship joins for single hops and for every hop of variable-length paths, directed or undirected.
• Target node rows are fetched by row position instead of a hash join. Target filters run inside the expansion, unused columns are pruned, and a cycle-closing edge is checked inside the CSR.
• refresh() updates the CSR incrementally (appended edges plus tombstones) instead of rebuilding it.
• Any hop the CSR can't express falls back to a join in the same query.

Planning

• A pattern starts at its most selective node.
• DataFusion gets distinct counts for id and endpoint columns, so it sizes joins correctly.
• A continued pattern starts from whichever bound end has the smaller fan-out.

Cypher, now enough for all 14 LDBC Interactive complex queries:

• [:A|B] alternation, also inside variable-length paths (with optional endpoint labels per relationship type), and multi-label nodes;
• chains of WITH, including WITH DISTINCT, WITH ... WHERE, and nodes carried through WITH;
• OPTIONAL MATCH, EXISTS { ... } and pattern predicates;
• shortestPath and allShortestPaths, as a breadth-first search over the CSR;
• CASE, arithmetic, list literals and indexing, range, size, coalesce, toString, date_part, to_epoch_ms, and collect(DISTINCT ...).

Correctness fixes to behavior inherited from lance-graph, each with a regression test:

• Undirected patterns dropped every incoming edge.
• Multi-label nodes scanned only their first label.
• Hops inside variable-length paths could be silently dropped.
• camelCase aggregate aliases failed to plan.
• WITH didn't end variable scope.
• collect() kept nulls.
• Overflowing numeric literals panicked, and LIMIT -1 wrapped around.

Packaging and wheels

• Version 0.1.0 throughout, and a NOTICE crediting lance-graph.
• A wheel pipeline builds two wheels. The Linux x86-64 wheel targets glibc 2.28 via zig, so it runs on RHEL 8, Ubuntu 20.04 and newer. The Apple Silicon wheel is built natively on GitHub's macos-14 runners; lance-graph never shipped one. An sdist covers other platforms.
• Every wheel is installed into clean environments with Rust removed from PATH, on CPython 3.12, 3.13 and 3.14 (the minimum supported version is 3.12). There it runs a smoke test (in-memory and on-disk tables, CSR traversal, shortestPath, refresh()) and the full Python test suite.
• A published release vX.Y.Z publishes to PyPI after every build and test passes. It uses trusted publishing through a protected pypi environment, and checks the tag against the package version. Pull requests and manual dry runs build and test without publishing.

Validation

All local checks were run on macOS (Apple M5) from this branch.

• Rust: cargo test --all passes 582 tests; cargo clippy --all-targets --all-features and cargo fmt --check are clean.
• Python: 280 tests pass against a release wheel built from this branch; ruff and pyright are clean.
• Benchmarks, run from the same wheel, in all three modes (lazy CSR, in-memory CSR, in-memory joins), two passes:
  • the 30-query LDBC SF1 suite and all 14 LDBC Interactive complex queries match their expected results;
  • timings are within -3.4% to +1.0% (geometric mean per mode) of the pre-port run.
• The wheel smoke test passes locally against the macOS arm64 wheel, with Rust removed from PATH. The workflow YAML parses.
• CI runs the Rust tests, lint and format checks across the whole workspace, the Python tests, and the wheel build and test matrix. The publish job only runs on a release.

🤖 Generated with Claude Code
Extensive benchmarking runs and testing by @prrao87