Skip to content

Use datalad-fuse's remfile backend when available - #1957

Draft
CodyCBakerPhD wants to merge 24 commits into
claude/validate-stream-via-datalad-fusefrom
claude/datalad-fuse-remfile
Draft

CodyCBakerPhD wants to merge 24 commits into
claude/validate-stream-via-datalad-fusefrom
claude/datalad-fuse-remfile

Conversation

@CodyCBakerPhD

Copy link
Copy Markdown
Contributor

Stacked on #1956. This tries the development version of datalad-fuse from datalad/datalad-fuse#131, which adds pluggable backends, including remfile for HDF5-based files.

Changes

Locally, the datalad-fuse and validate stream tests pass with both datalad-fuse 0.5.1 (where the remfile test is skipped) and #131 plus remfile.

Once #131 is released, the CI install can switch to that release and the datalad extra's lower bound can be raised.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr


Generated by Claude Code

claude added 11 commits October 6, 2026 20:46
Add `dandi.support.datalad_fuse.AnnexedReadableFile`, a `Readable` for a
locked annexed file whose content is not present locally, streamed with
the adapter of datalad-fuse (no FUSE mount), which asks git-annex where
the content is.  It is available with the new `datalad` extra.

Cache NWB metadata of locked annexed files, streamed or local, under
their git-annex key paired with their path, with fscacher 0.5.0's
`annex_key_fingerprint`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
Add a `stream` policy to `--missing-file-content`: the content of
annexed files that is not present locally (as in a DataLad clone of a
Dandiset) is streamed with datalad-fuse's adapter, so that pynwb and
nwbinspector run on it without it being downloaded.  pynwb validation
results are cached under the files' git-annex keys.

This is the validation integration of #1933, on top of the datalad-fuse
readable rather than a git-only reader of the git-annex branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
annex_fingerprint is not specific to NWB files, and living in
dandi.support.datalad_fuse it needs no deferred import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
…am-via-datalad-fuse

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
As documented in datalad/datalad-fuse#138.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
With the development version of datalad-fuse from datalad/datalad-fuse#131,
use its RemoteFilesystemAdapter, which reads HDF5-based files such as NWB
with remfile if installed (fsspec otherwise), and look up S3 exports for
content without a registered URL.  The released FsspecAdapter is still
used with datalad-fuse 0.5.1.

Test it in a new datalad-dev CI job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
fscacher imports joblib, which imports numpy, so importing it with the
module made the CLI (via dandi.validate in the stacked PR) import numpy,
failing test_no_heavy_imports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
…am-via-datalad-fuse

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
…talad-fuse-remfile

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
@codecov

codecov Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 10.81081% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.10%. Comparing base (a65f6ee) to head (bb56977).

Files with missing lines Patch % Lines
dandi/support/tests/test_datalad_fuse.py 9.52% 19 Missing ⚠️
dandi/support/datalad_fuse.py 12.50% 14 Missing ⚠️
Additional details and impacted files
@@                             Coverage Diff                             @@
##           claude/validate-stream-via-datalad-fuse    #1957      +/-   ##
===========================================================================
- Coverage                                    91.13%   88.10%   -3.04%     
===========================================================================
  Files                                           94       94              
  Lines                                        14806    14837      +31     
===========================================================================
- Hits                                         13494    13072     -422     
- Misses                                        1312     1765     +453     
Flag Coverage Δ
unittests 88.10% <10.81%> (-3.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

claude added 8 commits October 6, 2026 22:32
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
…am-via-datalad-fuse

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
…talad-fuse-remfile

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
h5py holds its global lock while reading from a Python file object, and
the read waits for another thread to fetch the data over HTTP (fsspec's
I/O thread, or an HTTP server in the same process as in the tests).  A
garbage collection in that thread could then run the finalizer of an
unreferenced h5py-backed object (e.g., hdmf's HDF5IO.__del__), which
waits for the same lock: a deadlock, as seen in CI in
test_validate_stream, ended by pytest-timeout with a segfault.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
This reverts commit d18843d.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
…talad-fuse-remfile

Resolve the import conflict, and check the type of the file object
wrapped by the one that pauses garbage collection in the remfile test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
@CodyCBakerPhD
CodyCBakerPhD added this pull request to stack #1959 October 12, 2026 21:30
@CodyCBakerPhD CodyCBakerPhD added cmd-validate enhancement New feature or request patch Increment the patch version when merged labels Oct 12, 2026
…talad-fuse-remfile

Bring in the rebased base branch.  The conflicting files are identical in
the old and the rebased base, so they keep this branch's versions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
With datalad/datalad-fuse#131 and remfile installed, remfile reads HDF5
files, with its own read-ahead (chunks growing up to 100 MiB) that
datalad-fuse does not let us configure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cmd-validate enhancement New feature or request patch Increment the patch version when merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants