fix(download): sanitize '~'-prefixed URL components and migrate existing '~bci' ghosts - #1181
Open
mamagarobonomon wants to merge 2 commits into
Open
mamagarobonomon wants to merge 2 commits into
mamagarobonomon wants to merge 2 commits into
Conversation
…ing ghosts URLs like https://lampx.tugraz.at/~bci/database/... mirror the server's '~bci' user component verbatim into the cache root via MNE's _url_to_local_path, creating a literal '~bci' directory. Tilde-prefixed path components are shell-expansion and tooling landmines, and the layout duplicates data across MOABB versions that used different base URLs. Two changes: - _sanitize_path: add '~' to the translated character set, so '~bci' becomes '-bci' in every destination path. - data_dl: look up the legacy file at the RAW (unsanitized) mirrored path. With the legacy lookup sanitized through the same table, the migration branch could never fire for these URLs and existing '~bci' files would be orphaned into full re-downloads; the raw lookup finds them and the existing replace() migrates them in-place. Behavior verified: existing ghost migrates with no network; fresh downloads land directly in the sanitized path.
Collaborator
|
Good catch! Seems good for me. |
bruAristimunha
approved these changes
Sep 18, 2026
bruAristimunha
enabled auto-merge (squash)
September 18, 2026 18:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
BNCI datasets are served from
https://lampx.tugraz.at/~bci/database/.... MNE's_url_to_local_pathmirrors the URL path verbatim into the cache root (stripping only the leading/), so the server's~bciuser component becomes a literal local directory named~bci:This bites in three ways:
~as "home" can misbehave around a directory literally named~bci.bnci-horizon-2020.eu/database/data-sets/, current ones mirrorlampx.tugraz.at/~bci/database/— the same files land in two trees and neither is found by the other layout.data_dlcomputes bothdestinationandlegacy_destinationthrough the same_sanitize_path, so they are identical whenever the URL contains no characters from the sanitize table — and~was not in the table.Fix (two lines + tests)
_sanitize_path: add~to the translated character set →~bcibecomes-bciin every destination.data_dl: the legacy lookup now uses the raw (unsanitized) mirrored path, so pre-fix~bcifiles are found and the existinglegacy_destination.replace(destination)branch migrates them in-place instead of orphaning them into a full re-download.Verification
~bcighost file is migrated to the-bcipath with no network access (pooch short-circuits on the matching known-hash).~bciis never created.~URLs are unaffected (raw and sanitized paths coincide); the zenodo normalization in_normalize_destinationis untouched.moabb/tests/test_download.py(no download mark, no network).Context
Discovered while regenerating BNCI 2015-013 benchmarks: a 3.2 GB
~bci/ghost duplicated an existing NEMAR cache byte-for-byte (md5-verified) because the layout mismatch prevented any cache hit. This affects every current BNCI fetch on lampx URLs.