Expand validation docs for dandi-cli#1822 (companion JSONL, grouping, VisiData) - #235
yarikoptic wants to merge 10 commits into
Conversation
✅ Deploy Preview for dandi-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
@yarikoptic Thank you for setting up the deploy preview, that helps considerably Do you have any guess as to how hard it might be to do something like that for the DANDI Archive? (if only the web front end since I imagine backend would be tough) EDIT: I see
Can you look into this? My original thought was to have such things on the DANDI-CLI testing suite since that is closer to where code changes would break such things |
|
|
||
| On a dataset with actual errors (missing README, wrong file extension): | ||
|
|
||
| ```console |
There was a problem hiding this comment.
@yarikoptic Remind me, are these lines of code tested through a doctest of some kind?
There was a problem hiding this comment.
No — not doctested. This particular block (--min-severity ERROR) is still a hand-abbreviated paraphrase of bids_invalid_errors_only.txt (22 lines in a different order than the 3 shown here), since that file is a flat list with no built-in truncation.
For the three blocks where the backing file is already truncated by the tool itself (--max-per-group/--summary output), I just pushed f9d5914 switching them to literal pymdownx.snippets includes of the committed files in docs/examples/validation/ — so those are now guaranteed accurate at build time (mkdocs build fails on a bad include path), which is the closest thing to a doctest we get without a custom test harness. Details in my reply on the sibling thread. This block would need the example command changed to something the tool itself truncates (e.g. add --max-per-group) before it can be embedded the same way — left as a follow-up rather than done here.
Generated by Claude Code
…rsist validation logs)
=== Do not change lines below ===
{
"chain": [],
"cmd": "yolo -v /home/yoh/proj/dandi/dandi-cli-enh-validators:/home/yoh/proj/dandi/dandi-cli-enh-validators:ro -- 'In /home/yoh/proj/dandi/dandi-cli-enh-validators which is submitted as dandi/dandi-cli#1822 we significantly improved validation interfacing -- we serialize validation outputs and store so we could reload and potentially review with different filtering or use external tools like visidata to navigate. I would like here to improve our https://docs.dandiarchive.org/user-guide-sharing/validating-files/ section with improved documentation, reflecting the state of that PR. We should demonstrate that we store companion validation files during upload so they could be re-reviewed/analyzed. We should show basic use of visidata to quickly review them. Could use bids-examples repo and some sample dandisets (should be sufficiently small) to show how e.g. to compose multiple validation files. Ideally we should script production of example outputs, and/or store/share validation output example for easier access. Do research how other projects using mkdocs produce similar demo walkthroughs, and what we have done so far in this repo. Do research, build plan for content and also implementation details.'",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
…rsist validation logs)
=== Do not change lines below ===
{
"chain": [],
"cmd": "yolo -v /home/yoh/proj/dandi/dandi-cli-enh-validators:/home/yoh/proj/dandi/dandi-cli-enh-validators:ro -- --resume",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
- Fix asciinema-player cast file path for use_directory_urls (../ -> ../../) - Rename "Validating BIDS Files" -> "Validating BIDS Datasets" - Longer pauses after demo-say narration (2s -> 4s) for readability - Add pauses after typed VisiData commands (go-col-regex, search) - Use clean "validation-demo $" prompt instead of leaking absolute paths - Add record.sh driver for headless asciinema recording via Xvfb - Re-record visidata-demo.cast with all improvements Co-Authored-By: Claude Code 2.1.92 / Claude Opus 4.6 <noreply@anthropic.com>
0015ae8 to
de1b5a3
Compare
we already had this for long time -- it is a frontend and backend IIRC working against staging S3, eg from dandi/dandi-archive#2771 
|
kabilar
left a comment
kabilar
left a comment
There was a problem hiding this comment.
Thanks Yarik. Since Cody has already reviewed and this is a big pull request, I'll defer to his review.
✅ Deploy Preview for dandi-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
- Align pipe-table columns in validating-files.md (VisiData key bindings, result schema, origin schema, severity levels) so the raw markdown reads as a table, not just the rendered HTML. - Replace three hand-typed console blocks (-g severity, -g severity -g id, --summary) with pymdownx.snippets includes of the real generated files under docs/examples/validation/, so their content is guaranteed byte-for-byte accurate instead of a manually-retyped (and, as found in review, already slightly drifted) paraphrase. mkdocs build already runs on every push/PR (build-docs.yaml), so a bad include path now fails CI. - Add a weekly check-validation-examples workflow that reruns scripts/generate-validation-examples.sh against the latest dandi-cli/nwbinspector/bids-validator and opens an issue if the regenerated output differs from what's committed, addressing the "how do we keep these in sync" concern from review. It will start failing/succeeding meaningfully once dandi/dandi-cli#1822 lands in a release, since the script needs its new validate flags. The two remaining hand-typed blocks (single-file NWB validate, and the valid-BIDS eeg_cbm example) are left as-is: the former documents single-file validation for which no example file is generated, and the latter's backing file (bids_eeg_cbm.txt) is produced by the script but was never committed -- flagged in the PR thread rather than fabricated here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUsi4ZwCnmLfN9mndgZhF9
Not just on a weekly schedule -- a PR that edits docs/examples/validation/**, the generator script, or this workflow now triggers the same regeneration+diff, and the job fails outright on drift (not just on schedule). Issue creation stays schedule-only, since a red PR check is already the right signal there and we don't want an issue opened for every such PR. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUsi4ZwCnmLfN9mndgZhF9
The previous version ran generate-validation-examples.sh unconditionally. Released dandi-cli doesn't have --grouping/--summary/--load yet (dandi/ dandi-cli#1822 isn't out), and the script's run_validate() swallows dandi's "unrecognized arguments" errors (2>/dev/null || true), so those calls silently produced empty output -- which then diffed as "everything deleted" against the committed examples and failed the job. That's what just turned PR #235 red: the check's own missing dependency, not stale examples. Now it probes `dandi validate --help` for --grouping first and skips the regeneration/diff/fail steps entirely (with a ::warning::) when the installed dandi-cli doesn't support it yet, so it goes quiet instead of false-failing until the dependency ships. Also print the first 100 lines of the diff directly in the job log (not just inside the schedule-only issue body), so a failing PR/push run is diagnosable without digging. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUsi4ZwCnmLfN9mndgZhF9
Makes the regenerated docs/examples/validation/ downloadable from the run, so a real drift finding (this run just found bids-validator-deno 2.4.1 -> 3.0.1 / BIDS schema 1.2.1 -> 1.2.7 since April) can be pulled straight into a commit instead of re-copied out of the job log. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUsi4ZwCnmLfN9mndgZhF9
git diff (and git diff --quiet) silently ignore untracked files, so the previous version never noticed when the script regenerates a file that was never committed in the first place -- which is exactly the state bids_eeg_cbm.txt, bids_eeg_cbm_summary.txt, bids_invalid_eeg_cbm_summary.txt and bids_000027_converted.txt are in right now (flagged in review, still unresolved). `git add -N` stages them with no content so the diff step sees them as an addition instead of nothing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUsi4ZwCnmLfN9mndgZhF9
|
Root cause, confirmed, not a flake: the new What's blocking a fix from me: regenerating requires either the real Suggested next step: @yarikoptic, you already have the local tooling for this — the script's own Everything else pushed to this branch — table formatting, 3 hand-typed blocks converted to Generated by Claude Code |
This branch has not been deployed
5 participants
Summary
Major expansion of the Validating Files documentation to cover the new validation features from dandi/dandi-cli#1822.
validating-files.mdfrom ~53 to ~360 lines with new sections:dandi validateusage with tabbed output format examples (text/JSON/YAML/JSONL)--min-severity,--ignore) and grouping (-g severity -g id) with real output--output,--load, automatic JSONL companion files)dandi validateruns on Dandiset 000027 (NWB) and bids-examples/bids-error-examples (BIDS)record.shvia Xvfb + xdotool)New files
docs/examples/validation/*.txt,jsonl,yaml,jsondocs/examples/validation/visidata-demo.castscripts/generate-validation-examples.shscripts/visidata-demo/{demo.sh,record.sh,dot_visidatarc}Dependencies
--load,--grouping,--summary,--format json_lines, companion JSONL) are from that PRTest plan
mkdocs buildsucceeds with no errors🤖 Generated with Claude Code
Extra TODOs for humans: