feat: Add Lance optional dependencies

[haoxu0]

Summary

Adds an installable lance extra for the native Lance read/write paths:

• pylance>=12.0.0
• lance-namespace>=0.11.1
• includes the extra in feast[ci]
• documents pip install 'feast[lance]'
• adds a packaging metadata regression test

This is split from #6943 so the data-source implementation and packaging change can be reviewed independently.

Verification

• pytest sdk/python/tests/unit/infra/data_sources/test_lance_dependencies.py -q
• ruff check sdk/python/tests/unit/infra/data_sources/test_lance_dependencies.py
• ruff format --check sdk/python/tests/unit/infra/data_sources/test_lance_dependencies.py
• pip install --dry-run '.[lance]'

Related: #6943, #6945

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.92%. Comparing base (f754323) to head (da563e1).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6958      +/-   ##
==========================================
+ Coverage   49.44%   49.92%   +0.48%     
==========================================
  Files         443      443              
  Lines       55511    55511              
  Branches     8096     8096              
==========================================
+ Hits        27445    27714     +269     
+ Misses      26168    25874     -294     
- Partials     1898     1923      +25     

Flag	Coverage Δ
go-feature-server	30.58% <ø> (ø)
python-unit	51.35% <ø> (+0.52%)	⬆️
see 6 files with indirect coverage changes

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f754323...da563e1. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ntkathole]

Need to run make lock-python-dependencies-all and pixi lock

[haoxu0]

Thanks — you are right on both counts, and I want to be straight that I cannot run either command from my environment. Here is what I was able to establish, in case it saves you time.

pixi lock is required, and I initially thought it would not be. I assumed a new PyPI extra could not affect it, since none of the three [tool.pixi.feature.*.pypi-dependencies] entries references ci or lance — they use aws/gcp/..., duckdb/delta/... and ray/..., so nothing in the resolved package set changes. That reasoning was incomplete: feast is an editable path dependency, so pixi.lock records its full requires_dist, including this line:

- feast[aws,azure,cassandra,...,iceberg,image,k8s,mcp,milvus,...] ; extra == 'ci'

which is the pre-change expansion and does not contain lance. So the lock does need regenerating, but the change should be confined to that one block: the ci list gains lance, plus two new ; extra == 'lance' entries for pylance and lance-namespace. No resolution change, because no feature pulls those extras. I deliberately did not hand-edit the file, since a hand edit that does not match the generator is worse than none.

For the requirements, only py3.{10,11,12}-ci-requirements.txt are affected. lance is referenced solely by the ci extra, and those are the only requirement files containing torch:

py3.10-ci-requirements.txt                    torch=2
py3.10-minimal-requirements.txt               torch=0
py3.10-minimal-sdist-requirements.txt         torch=0
py3.10-minimal-sdist-requirements-build.txt   torch=0
py3.10-requirements.txt                       torch=0

I confirmed the non-ci files are unaffected by regenerating py3.10-requirements.txt on this branch: zero pylance/lance-namespace lines appear.

One thing worth deciding before the refresh runs. I regenerated py3.10-requirements.txt from master's unmodified pyproject.toml as a control, and it already differs from the committed file by 1551 lines and 14 version pins — fastapi 0.141.1 → 0.142.2, mypy 2.3.1 → 2.4.0, plus markupsafe, mmh3, pyjwt, python-dotenv, pytz, charset-normalizer, librt, ast-serialize and a new opentelemetry-api transitive. That is upstream drift since the last refresh, not anything this PR introduces, but it means make lock-python-dependencies-all run today will bundle those bumps into this PR whoever runs it. Happy either way — I just did not want to hand you a dependency PR carrying a dozen unrelated upgrades without flagging it.

Why I cannot run it. pixi run needs conda.anaconda.org and the --torch-backend cpu compile needs download.pytorch.org, which then fans out to download-r2.pytorch.org; all three are blocked by my network egress policy, and add_cpu_torch_hashes.py invokes --torch-backend cpu a second time, so there is no way around the PyTorch index. Everything else resolves fine, so this is specifically the conda and PyTorch hosts.

Could you run make lock-python-dependencies-all and pixi lock on this branch, or would you prefer I close this and have someone fold the extra into a refresh you were going to do anyway? I am glad to do any other part of it.

[haoxu0]

Follow-up: I tried running both commands and make lock-python-dependencies-all cannot complete on master, independently of this PR.

ValueError: Missing CPU hashes for torch==2.14.1+cpu
make: *** [lock-python-dependencies-all] Error 1

add_cpu_torch_hashes.py looks each torch entry up by appending +cpu to its version, but a --universal --torch-backend cpu resolve emits two pins per package, and the second is already +cpu:

torch==2.14.0 ; sys_platform == 'darwin'
torch==2.14.0+cpu ; sys_platform != 'darwin'

so it asks for 2.14.0+cpu+cpu and raises. That shape is in all three committed py3.*-ci-requirements.txt today, so it is not something this branch introduced. The script and its tests landed together in #6188 and the requirements were last regenerated before that, and the existing test feeds in a bare torch==2.13.0, which is why it went unnoticed.

Fix is up as #6959, with regression tests for the split shape and a check against all three committed files. Once that merges I will rerun the lock target here and push the result.

One note for whoever regenerates: the target starts with rm -rf sdk/python/requirements/*, so a failed run leaves 14 of the 15 files deleted. Worth a git checkout -- sdk/python/requirements/ afterwards.

pixi lock itself works and is unaffected.

[haoxu0]

pixi lock is done and pushed as df2b0ec.

The change is confined to the one block I expected, because feast is an editable path dependency and so the lock records its full requires_dist including every extra's expansion:

+  - pylance>=12.0.0 ; extra == 'lance'
+  - lance-namespace>=0.11.1 ; extra == 'lance'
-  - feast[...,k8s,mcp,...] ; extra == 'ci'
+  - feast[...,k8s,lance,mcp,...] ; extra == 'ci'

3 insertions, 1 deletion. Nothing resolves differently — the 1377 pypi and 211 conda artifact URLs are byte-identical to before, which is expected since none of the three pixi features consumes the ci or lance extra.

make lock-python-dependencies-all is still blocked on #6959. I will rerun it and push the requirements here once that merges.

[haoxu0]

Updated in afe2961:

• synced the branch with current master and resolved the mcp-server/lance extra overlap
• ran pixi lock; it reports the lockfile is up to date
• regenerated all three requirements files affected by the ci extra: py3.10-ci-requirements.txt, py3.11-ci-requirements.txt, and py3.12-ci-requirements.txt
• each now contains locked lance-namespace==0.11.1 and pylance==13.0.0 entries with hashes

I also retried make lock-python-dependencies-all. With the #6959 fix applied locally it gets past the CPU torch hash issue, but the target then fails in the unrelated minimal-sdist build step because pybuild-deps==0.5.0 calls pip-tools with an incompatible OutputWriter(generate_hashes=...) API. Since lance is only included through the ci extra, the 12 non-CI requirement files are unaffected; I restored those rather than committing unrelated resolver drift.

Local verification after the final master sync:

• dependency test: 1 passed
• Ruff: passed
• pixi lock: already up to date
• git diff --check origin/master: passed

[ntkathole]

Please rebase

[haoxu0]

Please rebase

done