Please submit pointers to your validator output formats

[yarikoptic]

Please checkout README in https://github.com/con/validation for the motivations etc.

• BIDS (attn @effigies)
  • Interaction of BIDS and HED for errors bids-standard/bids-validator#157 (comment) for overall schema/description
  • validator version: deno: Report/record bids validator + hed validator in the bids-validator output bids-standard/bids-validator#10
  • standard versions: How to get schema version? bids-standard/bids-validator#118
• NWB (attn @rly; @bendichter for nwb-inspector)
  • HDMF Error objects described at https://github.com/hdmf-dev/hdmf/blob/dev/src/hdmf/validate/errors.py#L19
  • standard versions: [Feature]: Include version of NWB standard used in validation errors NeurodataWithoutBorders/pynwb#2062
• HED (attn @VisLab)
  • described below
  • BIDS/HED validation records discussion: Interaction of BIDS and HED for errors bids-standard/bids-validator#157 (comment)
• Zarr -- TODO
• OME-Zarr -- TODO
• SPARC's SDS (attn @tgbugs)
  • below
  • https://github.com/SciCrunch/sparc-curation/blob/master/sparcur/schemas.py
• PsychDS (attn @bleonar5)
  • originally inspired by BIDS validator but likely is not a fork of it: https://github.com/psych-ds/psychds-validator
  • ...
• ...

Generic higher level (domain agnostic?) specifications

• SARIF (brought up by @effigies): Static Analysis Results Interchange Format (SARIF)

[rly]

The NWB validator can be called from the command line and as a Python function.

The CLI is described here. When validation passes, the output looks like:

Validating /Users/rly/Documents/NWB/scratch/test.nwb against cached namespace information using namespace 'core'.
 - no errors found.

with exit code 0.

When it fails, it looks like:

Validating /Users/rly/Documents/NWB/scratch/test.nwb against cached namespace information using namespace 'core'.
 - found the following errors:
SpatialSeries/data (processing/behavior/position/series_0/data): incorrect shape - expected '[[None], [None, 1], [None, 2], [None, 3]]', got '(27206, 6)'

with exit code 1.

The Python function is described here. Using the paths keyword argument returns a tuple (list of errors, exit code). Using the io keyword argument returns a list of errors.

[yarikoptic]

FWIW, here is where we are mapping pynwb.validate output into our ValidationResult: https://github.com/dandi/dandi-cli/blob/HEAD/dandi/pynwb_utils.py#L359 so it looks like

                ValidationResult(
                    origin=ValidationOrigin(
                        name="pynwb",
                        version=pynwb.__version__,
                    ),
                    severity=Severity.WARNING,
                    id=f"pywnb.{error_output}",
                    scope=Scope.FILE,
                    path=Path(path),
                    message="Failed to validate.",
                )

but that is quite suboptimal since there is only error_output string without better decomposition into the "path" within the file etc. For nwb inspector outputs we seems to get some path within asset (file) and map it as well: https://github.com/dandi/dandi-cli/blob/HEAD/dandi/files/bases.py#L546

[rly]

Thanks @yarikoptic . The pynwb validator errors are parseable. I tried to address this in dandi/dandi-cli#1513

[VisLab]

The HED Python validator represents each issue as a dictionary and the error list is a list of these dictionaries. The error code is used to select a template for composing the error message from the dictionary. Most of our calling functions then get a printable issue string for these. The issue list could trivially be output as JSON but it maximizing usefulness would take some thought.

I guess the first step for us would be to define a JSON schema for the format of the issue list (which we currently don't have). We have a lot of fields, some of which correspond to yours and some don't.

[yarikoptic]

Thank you @VisLab , of special interest would be those which don't ATM have anything corresponding. Would you be so kind to create a list/table of them?

[VisLab]

I went over the hed-python code. There are single issue and context keys. The actual issue objects are nested a tree structure so that the context that applies to multiple issues can be easily printed once. This tree could be flattened.

Single issue keys

Key	Description
code	External HED error code from the specification-- to be mapped to web-page explanation
message	Formatted message after parameters are filled in Ex:'Invalid character "x08" at index 7'
severity	Numerical code 1 == error 10 is warning
index_in_tag	Position of start of problem in tag
index_in_tag_end	Position of end of problem in tag
source_tag	Pointer to an object with a lot of info including link to object in schema if available

Context keys:

Key	Description
ec_title	overall title for error report
ec_filename	file name this error applies to
ec_sidecarColumnName	sidecar column name
ec_sidecarKeyName	name of a categorical value
ec_row	row number in a tsv file
e_column	column number in tsv file
ec_line	line number in the file for which error is reported
ec_HedString	HED String in which tag appears
ec_section	Section of the HED schema in which error appears (for Schema validation)
ec_schema_tag	Schema tag in which error appears (for Schema validation)
ec_attribute	Schema attribute for which error appears (For schema validation) }

Sample tree structure of the issues:

{'children': [], 
('ec_sidecarColumnName', 'defs'): {
    'children': [], 
	('ec_sidecarKeyName', 'def1'): {
	    'children': [], 
		('ec_HedString', '(Definition/Apple, Definition/Banana, (Blue))'): {
		    'children': [
				{  'code': 'DEFINITION_INVALID', 
				   'message': "Too many tags found in definition for Apple.  Expected 1, found: ['Definition/Banana']", 
				   'severity': 1, 
				   'ec_sidecarColumnName': 'defs', 
				   'ec_sidecarKeyName': 'def1', 
				   'ec_HedString': <hed.models.hed_string.HedString object at 0x000002AFFF59B2E0>
				}, 
				{'code': 'TAG_GROUP_ERROR', 
				    'message': "Multiple top level tags found in a single group.  First one found: Definition/Apple. Remainder:['Definition/Banana']  Problem spans string indexes: 1, 17", 
					'severity': 1, 
					'source_tag': <hed.models.hed_tag.HedTag object at 0x000002AFFF4A2520>, 
					'ec_filename': '', 
					'ec_sidecarColumnName': 'defs', 
					'ec_sidecarKeyName': 'def1', 
					'ec_HedString': <hed.models.hed_string.HedString object at 0x000002AFFF54B7F0>, 
					'char_index': 1, 'char_index_end': 17
					}
				]
				}
				}, ...

[tgbugs]

The sparc validator is a bit weird in that except in cases where something breaks due to a bug in the validator itself we return the export as is or remove the malformed data (with a note in the validator that it has been removed). The validator output is embedded as errors sections within objects and a summary is lifted out, however that is not currently described in the linked schema below because we don't validate that part of the export, so I will updated the schema so that it is at least visible.

https://github.com/SciCrunch/sparc-curation/blob/master/sparcur/schemas.py

    "path_error_report": {
      "#/inputs/dataset_description_file": {
        "error_count": 6,
        "messages": [
          "'description' is a required property",
          "'name' is a required property",
          "'protocol_url_or_doi' is a required property"
        ]
      },
      "#/inputs/manifest_file/-1/checksums/-1": {
        "error_count": 1,
        "messages": [
          "{'type': 'checksum', ... 44 bytes later ... e3531d7671eab8911'} is not valid under any of the given schemas"
        ]
      },
      "#/inputs/submission_file/submission": {
        "error_count": 1,
        "messages": [
          "{'consortium_data_st ... 69 bytes later ... er': 'U19NS130608'} is not valid under any of the given schemas"
        ]
      },
      "#/meta/award_number": {
        "error_count": 1,
        "messages": [
          "'U19NS130608' does not match '^(OT2OD|OT3OD|U18|TR|U01)'"
        ]
      },
      "#/meta/techniques/-1": {
        "error_count": 2,
        "messages": [
          "'RNA -seq' is not a 'iri'",
          "'single-cell RNA sequencing' is not a 'iri'"
        ]
      }
    }

[candleindark]

{'children': [], 
('ec_sidecarColumnName', 'defs'): {
    'children': [], 
	('ec_sidecarKeyName', 'def1'): {
	    'children': [], 
		('ec_HedString', '(Definition/Apple, Definition/Banana, (Blue))'): {
		    'children': [
				{  'code': 'DEFINITION_INVALID', 
				   'message': "Too many tags found in definition for Apple.  Expected 1, found: ['Definition/Banana']", 
				   'severity': 1, 
				   'ec_sidecarColumnName': 'defs', 
				   'ec_sidecarKeyName': 'def1', 
				   'ec_HedString': <hed.models.hed_string.HedString object at 0x000002AFFF59B2E0>
				}, 
				{'code': 'TAG_GROUP_ERROR', 
				    'message': "Multiple top level tags found in a single group.  First one found: Definition/Apple. Remainder:['Definition/Banana']  Problem spans string indexes: 1, 17", 
					'severity': 1, 
					'source_tag': <hed.models.hed_tag.HedTag object at 0x000002AFFF4A2520>, 
					'ec_filename': '', 
					'ec_sidecarColumnName': 'defs', 
					'ec_sidecarKeyName': 'def1', 
					'ec_HedString': <hed.models.hed_string.HedString object at 0x000002AFFF54B7F0>, 
					'char_index': 1, 'char_index_end': 17
					}
				]
				}
				}, ...

@VisLab I am not sure if I understand this correctly and need some confirmation/clarification. The following are statements that need to be confirmed or corrected and questions that need to be answered.

1. Individual errors, each represented as a dictionary, are always contained within a list which is the value of a "children" key.
2. The location of a list of errors within the nested structure specifies the context of the errors.
3. Why does each error have keys such as 'ec_sidecarColumnName' and 'ec_sidecarKeyName' which are context keys, and some of these context keys are a context key in the path of the location of the error while others are not?

[VisLab]

1. Individual errors, each represented as a dictionary, are always contained within a list which is the value of a "children" key.

The basic format for a HED issue is a simple dictionary whose fields vary depending on the issue, but code, message, and severity are always there and for the purposes of integration, this might be sufficient. Here is an example (from above).

{
   'code': 'TAG_GROUP_ERROR', 
   'message': "Multiple top level tags found in a single group.  First one found: Definition/Apple. Remainder:['Definition/Banana'] Problem spans string indexes: 1, 17", 
   'severity': 1, 
   'source_tag': <hed.models.hed_tag.HedTag object at 0x000002AFFF4A2520>, 
   'ec_filename': '', 
   'ec_sidecarColumnName': 'defs', 
   'ec_sidecarKeyName': 'def1', 
   'ec_HedString': <hed.models.hed_string.HedString object at 0x000002AFFF54B7F0>, 
   'char_index': 1, 'char_index_end': 17
}

2. The location of a list of errors within the nested structure specifies the context of the errors.
3. Why does each error have keys such as 'ec_sidecarColumnName' and 'ec_sidecarKeyName' which are context keys, and some of these context keys are a context key in the path of the location of the error while others are not?

Alll of the validators take an optional ErrorHandler object. If this is used, then when validating a sidecar, for example, the ErrorHandler pushes the sidecar context before starting the validation and pops it when the validation has completed. This allows all of the errors generated from that sidecar to be grouped. The messages can be augmented by sidecar context information such as the column key for which the sidecar error occurs, etc.

The get_printable_issue_string takes care of that so that the error messages come out nicely formatted with all of the errors for that sidecar under a single title.

We are open to producing an "Extracted" version which just has a list of simple dictionaries and all of the context information is in the error message itself if this would be useful for Dandi.

[candleindark]

@VisLab Thanks for the detailed explanation.

We are open to producing an "Extracted" version which just has a list of simple dictionaries and all of the context information is in the error message itself if this would be useful for Dandi.

An "extracted" version would be nice. To make the extracted version accessible to machines as well as to humans, I think the context information of the error should be included as well as the human readable message.

The jsonschema Python package provides an elaborate representation for validation errors. May be we can adopt some of its features? 🙂

[VisLab]

The jsonschema Python package provides an elaborate representation for validation errors. May be we can adopt some of its features? 🙂

This is a good idea, but maybe not exactly that. I'd like to see a code, severity, message being required. The context being flexible.

[candleindark]

The jsonschema Python package provides an elaborate representation for validation errors. May be we can adopt some of its features? 🙂

This is a good idea, but maybe not exactly that. I'd like to see a code, severity, message being required. The context being flexible.

Oh, I mean to use the ValidationError from jsonschema only as a reference. In fact, the context attribute in the ValidationError doesn't have the same meaning as context in HED.

What I like from jsonschema is that individual ValidatorError objects are yielded from iter_errors, and each object contains all the info about an the represented error.

[VisLab]

Okay I added an iter_errors generator to the develop branch of hed-python. See: hed-standard/hed-python#1053

[dstansby]

For OME-Zarr, https://github.com/ome-zarr-models/ome-zarr-models-py is (I think) the only software that exists to do full validation of OME-Zarr data. We don't output any machine readable validation information at the moment.

It's definitely something we'd be open to exploring, so if anyone has any pointers or wants to open an issue to request this feature that would be great!

[yarikoptic]

Dear Standards folks. Following up on the

• [Feature]: Include version of NWB standard used in validation errors NeurodataWithoutBorders/pynwb#2062
• How to get schema version? bids-standard/bids-validator#118

and others, I am adding in original description pointers to issues or solutions on how/where versions of the standards are listed in the output of the validators. Also I think it is worth while to list also names and versions of validator software itself -- this way, when arriving to a validation output file it would be clear what validator (and version) was used and against which standards (versions) it was validated against. ATM in dandi-cli we need to duplicate the effort of sensing versions of the standards and the tools.