[nivle1661]

Summary

Read Parquet files and folders below a Unity Catalog Volume through the existing streaming source reader. This adds source-path support; it does not change snapshot loading.

• Use fresh path/size/mtime identities for every physical part, detecting same-size rewrites and rejecting changes during reads. Hidden temporary entries follow the same rules during versioning and ingestion.
• Share read-mode and empty-source handling with SQL. Enforce the connector catalog/schema and require a child path below the Volume root; explain the training/custom-source-only introspection limit.
• Keep the databricks extra data-only. Vendor MLflow dependencies belong to mlflow, verified through an actual setuptools metadata build and wheel merge.

Change map

Maintained change map

Testing Done

• 67 provider/backend/Volume/source tests and 69 packaging tests passed.
• Independent review verified all Vedaant review points, including targeted same-size/chunked rewrites and concurrent-read rejection.
• Lint, import sorting, generated lock consistency and diff checks passed. Managed workflow 42938850179 passed all four Databricks engine phases and cleanup.

[nivle1661]

Change map

Requirement: read scoped Unity Catalog Volume Parquet inputs through the existing streaming reader. Rewrites previously reused constant or size-only versions, empty inputs differed from SQL sources, and the data-only extra installed MLflow vendor clients.

File	Function / scope	What changed	Why
packages/kumo-databricks/README.md	Volume source contract	Document scoped Parquet inputs, metadata versioning and introspection limits.	Describe source reading accurately; snapshot restoration is unchanged.
packages/kumo-databricks/src/kumo_databricks/backend.py	Volume reader admission and ingestion	Enforce connector catalog/schema and below-root paths; use shared read_table and empty-source errors; version every physical part with fresh metadata.	Avoid stale training rows/indexes and preserve SQL behavior.
packages/kumo-databricks/src/kumo_databricks/volume.py	Files API metadata adapter	Expose millisecond modification times and a fresh all-part identity.	Detect same-size and non-max-part rewrites, including changes during reads.
packages/kumo-databricks/tests/fakes.py	Provider metadata test double	Track logical clock/mtime and Files API metadata in neutral test resources.	Exercise the actual metadata contract rather than constant versions.
packages/kumo-databricks/tests/test_backend.py	Volume and SQL regressions	Test rewrites, concurrent mutations, hidden entries, empty inputs, path scope and explicit introspection restrictions.	Reject stale or unsupported sources without opening a client unnecessarily.
packages/kumo-engine/pyproject.toml	Optional dependency boundary	Keep databricks data-only; request vendor MLflow clients through the mlflow extra.	Avoid adding agent/cloud SDK dependencies to data-only installs.
test/scripts/test_engine_wheel.py	Real wheel metadata regression	Build authored setuptools metadata, run the real wheel merger and inspect its extras; isolate the existing setup mock.	Test shipped dependency metadata instead of rebuilding requirements in the test.
uv.lock	Generated workspace metadata	Regenerate the optional-dependency metadata with uv 0.13.7.	Keep the lock consistent with the authored extras.

Verification: 67 provider/backend/Volume/source tests and 69 packaging regressions passed. Independent review checked every in-scope Vedaant numbered/smaller request. Managed Databricks workflow 42938850179 passed all four engine phases and cleanup. Lock metadata was regenerated with uv 0.13.7.

Stopping condition: metadata changes invalidate source versions, mid-read changes discard staging, empty/unsupported/out-of-scope inputs fail clearly, and actual built wheel extras preserve the data-only boundary. No snapshot-load behavior or native platform UI is claimed.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: NVIDIA/kumo-engine
• Branch: egu/databricks-volume-restore
• File: .coderabbit.yaml
• Commit: 1a44210
• Layers: Repository YAML
• Review profile: QUIET
• Plan: Enterprise
• Run ID: 644f5773-f401-4c21-876e-c62ba78320cc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• Trigger review

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

Failed checks: 3 inconclusive

Check name	Status	Explanation	Resolution
Change Map Design Review		The change map and linked problem are unavailable, so their design rationale cannot be reviewed. The PR description links to a comment but does not include its contents. The repository diff shows Volu…	Provide the full conversation comment beginning with <!-- kumo-pr-change-map --> and the linked problem or release requirements.
Pr Size		The PR base is not available. The diff against origin/main has 96 changed files, but the diff against the merged integration branch has 8. The PR metadata does not identify which branch is the base.	Provide the provider-reported PR base revision or the authoritative list of changed files.
Public Resource Access		Anonymous access to 206 changed public URLs returned HTTP 200. The new package archive locations in packages/kumo-engine/test/snowflake_notebook/dependency-asset.json:5 and graph-assets.json:18 us…	Provide the test account and evidence of anonymous access to both package archives, or provide public package sources for them.

Full details: Change Map Design Review

Explanation

The change map and linked problem are unavailable, so their design rationale cannot be reviewed. The PR description links to a comment but does not include its contents. The repository diff shows Volume identity changes in packages/kumo-databricks/src/kumo_databricks/volume.py; it cannot establish what the comment claims.

Resolution

Provide the full conversation comment beginning with <!-- kumo-pr-change-map --> and the linked problem or release requirements.

Full details: Public Resource Access

Explanation

Anonymous access to 206 changed public URLs returned HTTP 200. The new package archive locations in packages/kumo-engine/test/snowflake_notebook/dependency-asset.json:5 and graph-assets.json:18 use an account-relative Snowflake stage. The account is not supplied, so anonymous access to those locations could not be verified.

Resolution

Provide the test account and evidence of anonymous access to both package archives, or provide public package sources for them.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

[Vedaant-J]

The title says "restore Databricks snapshots", but snapshot restore (GraphSnapshot.load_from_connector, _snapshot_io) isn't touched. What the PR adds is reading a /Volumes/<catalog>/<schema>/<volume>/... file or folder as a Databricks source through the existing streaming reader. Please retitle. Reusing the shared reader is the right approach (parallel ranged reads, cleanup on failure, no new Python pass). The version handling needs fixing before merge:

1. A single Volume file always gets version "-1". volume_filesystem drops last_modified, so _source_version returns the constant SourceVersion("-1"). I rewrote a file once with the same size and once with different rows; the version stayed the same both times. Because "-1" isn't None, TrainingTable._table_id gives the same tt-… id after a rewrite, and _store skips the upload because _SUCCESS exists. The artifact root keeps the old rows while the new model's lineage points at them. The same constant goes into Graph.snapshot's data_version, so an old index is reused. The SDK exposes DirectoryEntry.last_modified / GetMetadataResponse.last_modified; please put mtimes into the listing (or pass an object_identity from get_metadata) and add a test that rewrites a file and expects a new version.
2. Folder versions hash only path and size, but are labelled listing, which source_changes treats as comparable. A same-size rewrite reports unchanged, and a concurrent same-size rewrite isn't caught by the changed-during-read check (its mtime is None). Same fix as 1.
3. Empty inputs. The SQL route raises PermanentSourceError(... no rows), but the Volume route accepts a zero-row Parquet file. An empty folder raises a bare IndexError (with dtypes) or ValueError("a staged table requires at least one path…"). Calling read_table(reader, '', dest, columns=dtypes, mode=self.mode, sql=sql, version=data_version) and sharing the existing empty-table tail fixes this, and removes the hand-written mode logic and the second require_no_sql.
4. Volume paths only half work. get_columns, get_sample and get_table_config still treat the path as a SQL table and fail with "does not match the connector catalog". Either route those too, or document that Volume paths are for training tables and custom sources only, with a clear error.
5. Connector scope is bypassed. A connector set to catalog a / schema b reads /Volumes/other/..., and a bare volume root lists the whole volume recursively. Unity Catalog still enforces permissions, but it's inconsistent with the SQL path and undocumented.
6. Please move the mlflow-skinny[databricks] dependency to its own PR or the mlflow extra. It pulls databricks-agents, tiktoken, Azure/GCS clients and boto3 into every data-only kumo-engine[databricks] install (uv.lock +263). The new wheel test rebuilds the requirements by hand, so it mostly tests marker handling rather than the real build.

Smaller:

• test_volume_parquet_ingest_preserves_training_rows compares "-1" with "-1", so it passes trivially. test_invalid_volume_coordinate_never_opens_a_client exercises the existing SQL-name check, not new code.
• data_version lists hidden files but prepare doesn't, so an in-progress _temporary/ write makes them disagree and the source reports changed.
• The README says only same-size rewrites go undetected; for single files every rewrite does.
• Unrelated churn in the faiss test and test_backend.py.
• The test catalog and volume names match the live harness; neutral fake names would be safer in a public repo.

[nivle1661]

1. File and folder versions now include fresh millisecond modification metadata. Chunked files include every physical part's identity, so same-size and non-max-part rewrites with changed metadata change the version. Mid-read mutations fail and discard staging.
2. Versioning and ingestion use the same hidden-file filtering.
3. Volume ingestion now uses the shared read_table path and common empty-source handling, including zero-row Parquet and empty folders.
4. Documented the training-table/custom-source restriction and added clear errors for unsupported Volume introspection methods.
5. Enforced the connector's catalog/schema and required a file or folder below the Volume root before opening a client.
6. Moved the vendor MLflow dependencies out of the data-only databricks extra into mlflow. The packaging regression builds authored setuptools metadata and runs the actual wheel merger instead of reconstructing requirements.