Skip to content

PROF-driven SHACL + schema validation for CGMES 3.0 + NCP 2.5 with code scanning integration - #397

Merged
HarisVranaj merged 2 commits into
entsoe:cgmes-3.0_ncp-2.5_tc-2.0from
Haigutus:feat/prof-driven-shacl-validation-2.5
Sep 11, 2026
Merged

HarisVranaj merged 2 commits into
entsoe:cgmes-3.0_ncp-2.5_tc-2.0from
Haigutus:feat/prof-driven-shacl-validation-2.5

Conversation

@Haigutus

@Haigutus Haigutus commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Counterpart of the 2.4-branch PR #387 (cgmes-3.0_ncp-2.4_tc-1.1, CGMES 3.0 + NCP 2.4 — stays as is) for cgmes-3.0_ncp-2.5_tc-2.0, adapted to the NCP 2.5 line of the application-profiles-library main. Implements #381 on this branch. Fork CI run: https://github.com/Haigutus/relicapgrid/actions/runs/34101124661 (green, four SARIF categories uploaded).

What it adds:

  • buildScripts/prof_map.py — maps each instance file's declared profile (md:Model.profile / dcterms:conformsTo) to its SHACL files via the DX-PROF descriptors. For NCP 2.5 it also resolves the role/validation aggregates: their owl:imports closure is resolved by basename in the checkout, and the shapes they mark sh:deactivated are honoured.
  • buildScripts/validate_instances.py — validates every Instance/**/*.xml (127 mapped, ~2.5 min): SHACL in cross-file-resolving groups (per area + boundary + CGM assembly; NC per area with Grid context) with per-dataset constraints (sh:minCount/maxCount/sh:closed, ClassCount/PrefixDeclaration, AllowedClasses) evaluated per instance file; a shapes-independent schema-conformance layer; a Model.DependentOn/requires resolvability check. NC additionally runs a dependency-closure comparison (each file over itself + its transitive dcterms:requires), reported as a per-rule diff in the run summary — informational, not uploaded.
  • .github/workflows/shacl_validation.yml — one APL checkout (main carries CGMES 3.0 and NCP 2.5), four code-scanning categories (shacl-/schema- × cgmes-3.0/ncp-2.5), full sh:ValidationReports + summary as run artifact.
  • buildScripts/schemas/ — the NCP 2.5-dev export schema (triplets format, provenance in SOURCE.json), vendored until a triplets release ships it; triplets pinned to 0.2.0.

Differences from #387: single APL checkout; NC shape sets come from the PROF role/validation aggregates (2.4 had none); the closure comparison run; the vendored schema; and the branch-specific policies below.

Policies encoded (each documented at its constant in validate_instances.py):

  • Boundary datasets declare no application profile → validated as CoreEquipment-EU/3.0 until Boundary dataset headers declare no application profile (cgmes-3.0_ncp-2.5_tc-2.0) #395 is fixed (BOUNDARY_PROFILE_FIX).
  • Instance/Jotunheim/GridSituation/cimxml/ duplicates Instance/Jotunheim/NetworkCode/ (same dataset UUIDs) → skipped; Dataset_version_dependency/ files validated alone, never in a union frame.
  • Skipped with reason in every summary: referenceData (no application profile), Svedala_FAP_PV.xml (Provenance/1.0 has no PROF).

Issues found by this run:

Notes for review:

  • Closure vs type split differ only in ContingencyEquipment.Equipment-valueType/sh:in (314) and AssessedElement.ConductingEquipment-valueType/sh:in (316): SSH continues EQ objects as <cim:Equipment rdf:about>, so on the area union the target carries types {ACLineSegment, Equipment} and the APL ( assoc rdf:type ) sh:in check fails per SHACL semantics; under the declared requires closure it does not. Left in the code-scanning result for now; the summary table shows the diff.
  • SARIF uploads need security-events: write (in the workflow); alerts appear under Security → Code scanning after the first run on the branch, tool triplets-shacl.

kristjan.vilgo added 2 commits September 7, 2026 11:32
…ode scanning)

Port of the cgmes-3.0_ncp-2.4_tc-1.1 pipeline (entsoe#387) to this branch, adapted to
the application-profiles-library main line:

- buildScripts/prof_map.py: besides the role/constraints shape files, resolve
  the NCP 2.5 role/validation aggregates — owl:imports closures resolved by
  basename in the checkout (SHACL/** incl. Validation/) — and expose the
  shapes they mark sh:deactivated.
- buildScripts/validate_instances.py: one APL checkout (main) serves both
  releases; NC shape sets are the validation aggregates plus the AllComplex
  aggregate for the union pass; APL deactivations honoured by dropping those
  shapes' findings (triplets 0.2.0 evaluates them); per-dataset findings are
  the counting constraints, data-graph counts (ClassCount, PrefixDeclaration)
  and class membership of shapes the per-file pass re-runs; superseded dataset
  versions (Dataset_version_dependency/) never join a union frame; boundary
  dcat:Dataset headers (no profile declared) validated as CoreEquipment-EU/3.0
  (BOUNDARY_PROFILE_FIX, documented hardcode); Jotunheim/GridSituation/cimxml
  skipped as duplicate of Jotunheim/NetworkCode.
- Dependency-closure comparison run for NC: reference checks scoped to each
  file + its transitive dcterms:requires closure, per-rule diff against the
  type split in summary.md, closure-ncp-2.5.sarif as run artifact only.
- buildScripts/schemas/: vendored NCP 2.5-dev export schema (triplets format,
  provenance in SOURCE.json) passed in as rdf_map until a triplets release
  ships it; triplets pinned to 0.2.0, pandas 2.3.3 excluded.
- .github/workflows/shacl_validation.yml: single APL checkout, categories
  shacl-/schema-{cgmes-3.0,ncp-2.5}.

Local run: 127 files mapped, ~2.5 min.
@Haigutus

Haigutus commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator Author

Snippet of NCP issues report. I am not sure if lang tags should be requiered.

image

@HarisVranaj
HarisVranaj merged commit f09bb34 into entsoe:cgmes-3.0_ncp-2.5_tc-2.0 Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants