Skip to content

[python] Add LeRobot Dataset v3 import - #9446

Draft
XiaoHongbo-Hope wants to merge 11 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/lerobot-import
Draft

[python] Add LeRobot Dataset v3 import#9446
XiaoHongbo-Hope wants to merge 11 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/lerobot-import

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Purpose

Add a one-time image-based LeRobot v3 importer:

LeRobot Dataset v3 -> Arrow batches -> Paimon table

snapshot_id = conn.load_from_lerobot(
    "robot_data",
    "/path/to/lerobot_dataset",
)
  • Supports local directories, FileIO URIs such as OSS through source_options, and Hugging Face repo_id.
  • Derives the schema from meta/info.json, preserves feature names and Episode boundaries, writes Arrow batches, and creates one snapshot.
  • Maps scalars to scalar types, vectors to VECTOR, higher-rank tensors to nested ARRAY, and compressed images to BLOB.
  • Reuses the HDF5 strict existing-table schema validation and append semantics.
  • Keeps LeRobot optional through pypaimon[lerobot] and rejects v2.x clearly.

This PR supports image features only. Video features are rejected with a clear error; packed video import will be a separate follow-up on #9473. This PR does not depend on #9473.

Tests

  • Official LeRobot API creates a two-Episode v3 image dataset in a temporary directory without network access.
  • Covers metadata-derived schema, scalar/vector/tensor values, ordering, image BLOB round trip, append, snapshot, incompatible schema, OSS reads, missing dependency, v2 rejection, and video rejection.
  • multimodal_lerobot_test.py: 7 passed
  • multimodal_hdf5_test.py: 24 passed
  • Flake8, py_compile, and git diff --check: passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant