Skip to content

refactor(python): wrap the compiled classes in typed, documented python classes - #71

Merged
prrao87 merged 1 commit into
mainfrom
refactor/python-api-wrappers
Oct 28, 2026
Merged

prrao87 merged 1 commit into
mainfrom
refactor/python-api-wrappers

Conversation

@prrao87

@prrao87 prrao87 commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Why

The public lance_csr classes were the compiled PyO3 classes, re-exported. Their docstrings lived in Rust /// comments, and the API reference page was written by hand, so every Python API change meant re-checking that page line by line against the bindings. Generating the page instead would have meant importing the compiled extension during the docs build, which puts a Rust build into both the GitHub Actions check and the Cloudflare Pages build.

This follows the pattern LanceDB, Polars and pylance use. The public classes become thin Python classes with type hints and NumPy docstrings, each holding a private compiled object, and mkdocstrings reads them statically. The docs build stays Markdown-only (a strict build takes about 1.3 seconds), and the reference can't drift from the signatures.

What changed

The public API is unchanged: same names, signatures, behaviour and exceptions.

  • Every public class is a typed, documented Python class that wraps the compiled one in lance_csr._internal. Builders still return new objects, and methods that return a config, such as CypherEngine.config(), return the public GraphConfig.
  • ExecutionStrategy, SqlDialect and DistanceMetric are IntEnums with the same member names and integer values.
  • Passing the wrong class, such as a string where a GraphConfig is expected, raises TypeError naming the parameter.
  • GraphConfig() still has no public constructor and raises TypeError pointing at GraphConfig.builder().
  • Deprecation warnings from with_node_label and with_relationship still point at the caller's line.
  • The Python-facing docstrings move from the Rust bindings into the Python classes. While moving them, each was checked against the code, which fixed drift such as GraphConfigBuilder.build() documenting RuntimeError when it raises ValueError.
  • The API reference is now "Python API reference", generated by mkdocstrings. Its prose sections (type checking, datasets, exceptions, timeouts, path values) stay hand-written.
  • Rust users get a "Rust API reference" nav entry pointing to docs.rs, and the Rust page links both crates on crates.io and docs.rs.
  • Docstring examples run as doctests in make test and in Python CI. The docs workflow also runs when the Python package changes.
  • The contributor guides now say to update docstrings rather than a hand-written page.

Per-call cost of the wrapper, measured in one process against the compiled object directly: 13 to 86 ns per call. A one-row Cypher query on a 100-row table takes about 0.9 ms in the same debug build, so the wrapper is within noise there.

Validation

  • make test: 1,590 passed, including new tests that check each wrapper against the compiled class (members, signatures, enum values), that every public method is documented and typed, and the wrapper seams (builders, TypeError, enum pass-through, deprecation warning location).
  • make lint: ruff, format and Pyright pass, including the strict public-API typing check, unchanged.
  • cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings (and on the bindings crate), cargo test --workspace: pass.
  • zensical build --strict passes without the compiled extension on the path; every docstring cross-reference and page link resolves. Checked the page in light and dark mode and at phone width.
  • docs/check_snippets.py (every page's Python examples) and uvx typos@1.26.0 .: pass.

Benchmarks

Measured with the CONTRIBUTING.md harness: release wheels from main (57a3326) and this branch (99a145f), both suites, memory and lazy modes, alternating builds and running each twice. Every run passed (30 queries in graph-benchmark-ldbc at b649bc1, 14 in graph-benchmark-ldbc-complex at f3e63fd), and each runner loaded the built wheel, not the pinned release. Apple M5, 10 cores, 24 GB, macOS (Darwin 25.6.0).

Suite Mode Geomean branch/main Gap between the two main runs
graph-benchmark-ldbc memory +0.7% -0.1% (largest query 6%)
graph-benchmark-ldbc lazy +1.5% -0.1% (largest query 6%)
graph-benchmark-ldbc-complex memory +0.5% -1.6% (largest query 4%)
graph-benchmark-ldbc-complex lazy +0.7% +1.3% (largest query 6%)

Every change is well within the 15 to 20% run-to-run range the guide describes, so this shows no measurable slowdown. The largest single-query change, Q30 in graph-benchmark-ldbc lazy (+8.7%), goes the other way in memory (-5.7%).

graph-benchmark-ldbc, `memory` mode
Query main (ms) branch (ms) Change
Q1 0.61 0.61 +0.5%
Q2 0.52 0.55 +5.1%
Q3 0.62 0.66 +6.5%
Q4 0.49 0.50 +1.5%
Q5 0.77 0.80 +5.0%
Q6 0.42 0.42 +1.5%
Q7 1.02 1.03 +0.9%
Q8 0.62 0.63 +1.4%
Q9 0.81 0.81 +0.6%
Q10 0.71 0.72 +0.5%
Q11 1.55 1.53 -1.0%
Q12 0.74 0.75 +0.9%
Q13 0.79 0.80 +0.9%
Q14 0.53 0.52 -0.9%
Q15 0.56 0.57 +2.1%
Q16 0.64 0.65 +0.7%
Q17 0.91 0.90 -0.6%
Q18 0.69 0.69 +0.4%
Q19 0.93 0.93 -0.2%
Q20 1.95 1.97 +0.7%
Q21 0.76 0.75 -1.3%
Q22 0.71 0.71 +0.7%
Q23 0.72 0.73 +0.5%
Q24 0.68 0.69 +0.4%
Q25 0.79 0.79 +0.1%
Q26 0.73 0.74 +0.4%
Q27 0.87 0.86 -0.1%
Q28 1.40 1.41 +1.1%
Q29 0.69 0.68 -1.1%
Q30 43.89 41.37 -5.7%

Geometric mean of branch/main ratios: 1.007 (+0.7%). Between the two main runs: 0.999 (-0.1%), largest single-query gap 6%.

graph-benchmark-ldbc, `lazy` mode
Query main (ms) branch (ms) Change
Q1 1.17 1.17 +0.2%
Q2 1.40 1.45 +3.4%
Q3 0.80 0.81 +1.0%
Q4 0.78 0.79 +1.7%
Q5 2.14 2.23 +4.5%
Q6 0.73 0.74 +1.7%
Q7 1.09 1.10 +0.7%
Q8 1.76 1.79 +1.9%
Q9 1.36 1.36 -0.3%
Q10 0.94 0.94 +0.4%
Q11 2.00 2.01 +0.3%
Q12 6.89 6.84 -0.7%
Q13 1.06 1.08 +1.6%
Q14 0.69 0.69 +0.1%
Q15 0.76 0.78 +1.9%
Q16 1.09 1.12 +2.7%
Q17 1.40 1.42 +1.5%
Q18 1.15 1.15 -0.6%
Q19 1.37 1.38 +1.3%
Q20 16.60 16.77 +1.0%
Q21 0.83 0.84 +1.7%
Q22 0.92 0.94 +2.3%
Q23 1.22 1.25 +2.1%
Q24 1.09 1.07 -1.8%
Q25 1.20 1.19 -0.9%
Q26 1.06 1.11 +5.0%
Q27 1.32 1.33 +1.2%
Q28 2.39 2.47 +3.3%
Q29 1.03 1.04 +0.9%
Q30 42.40 46.10 +8.7%

Geometric mean of branch/main ratios: 1.015 (+1.5%). Between the two main runs: 0.999 (-0.1%), largest single-query gap 6%.

graph-benchmark-ldbc-complex, `memory` mode
Query main (ms) branch (ms) Change
Q1 26.27 26.63 +1.4%
Q2 2.91 2.97 +2.0%
Q3 61.30 64.37 +5.0%
Q4 6.43 6.62 +3.1%
Q5 78.62 80.71 +2.7%
Q6 25.78 25.88 +0.4%
Q7 21.12 21.01 -0.5%
Q8 2.07 2.06 -0.5%
Q9 42.62 42.47 -0.4%
Q10 104.56 104.94 +0.4%
Q11 3.81 3.73 -2.0%
Q12 22.45 21.86 -2.7%
Q13 1.12 1.12 -0.0%
Q14 49.00 48.27 -1.5%

Geometric mean of branch/main ratios: 1.005 (+0.5%). Between the two main runs: 0.984 (-1.6%), largest single-query gap 4%.

graph-benchmark-ldbc-complex, `lazy` mode
Query main (ms) branch (ms) Change
Q1 31.67 32.32 +2.0%
Q2 26.53 26.29 -0.9%
Q3 411.08 414.50 +0.8%
Q4 13.67 13.74 +0.5%
Q5 109.51 111.79 +2.1%
Q6 27.17 27.50 +1.2%
Q7 33.33 33.54 +0.6%
Q8 3.09 3.17 +2.5%
Q9 818.37 844.38 +3.2%
Q10 137.58 136.61 -0.7%
Q11 6.10 6.15 +0.8%
Q12 28.58 27.96 -2.2%
Q13 1.49 1.48 -0.6%
Q14 48.37 48.70 +0.7%

Geometric mean of branch/main ratios: 1.007 (+0.7%). Between the two main runs: 1.013 (+1.3%), largest single-query gap 6%.

🤖 Generated with Claude Code

The public lance_csr classes are now pure-Python wrappers around the
private compiled module, carrying the type hints and NumPy docstrings.
The Python API reference is generated from them with mkdocstrings,
read statically, so the docs builds don't need the compiled extension.

The public API is unchanged. Enums become IntEnums with the same names
and values; a wrong argument class raises TypeError naming the
parameter. Python-facing docstrings move out of the Rust bindings, and
the Rust API reference on docs.rs is linked from the docs.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying lance-csr with  Cloudflare Pages  Cloudflare Pages

Latest commit: 99a145f
Status: ✅  Deploy successful!
Preview URL: https://939951d0.lance-csr.pages.dev
Branch Preview URL: https://refactor-python-api-wrappers.lance-csr.pages.dev

View logs

@prrao87
prrao87 marked this pull request as ready for review October 28, 2026 15:33
@prrao87
prrao87 merged commit 3324055 into main Oct 28, 2026
8 checks passed
@prrao87
prrao87 deleted the refactor/python-api-wrappers branch October 28, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant