Repository navigation
IDS conformance: status against buildingSMART's implementer test cases, and what is left #831
Description
Summary
I ran all of buildingSMART's IDS implementer test cases (Documentation/ImplementersDocumentation/TestCases, 307 cases that aren't invalid-) through IDSSpecifications on main (8479a7c), and again with the open IDS PRs applied together. This issue is the overview, so the individual issues stay focused. Happy to share the runner (a vitest file) if it would help as a regression suite.
A case agrees when its expected result (pass- / fail-) matches: a fail- case must have at least one applicable item failing, a pass- case none.
| Folder | Cases | main: agree | main: throw | with the PRs: agree | with the PRs: throw |
|---|---|---|---|---|---|
| entity | 27 | 20 | 2 | 23 | 0 |
| property | 76 | 51 | 19 | 67 | 0 |
| attribute | 45 | 35 | 0 | 35 | 0 |
| restriction | 23 | 18 | 0 | 19 | 0 |
| ids | 11 | 9 | 0 | 9 | 0 |
| classification | 27 | 17 | 0 | 17 | 0 |
| partof | 33 | 17 | 0 | 17 | 0 |
| material | 29 | 24 | 0 | 24 | 0 |
| tolerance | 36 | 22 | 0 | 22 | 0 |
| total | 307 | 213 | 21 | 233 | 0 |
"With the PRs" is engine_components #814, #816, #818, #820, #822, #824 and #828 merged together, plus ThatOpen/engine_fragment#324 for the importer. No case that agrees on main stops agreeing.
One caveat: 60 of the agreeing pass- cases agree only because nothing was applicable, including every classification and partof pass- case. So the agreement figure flatters those folders.
Already filed
- Applicability facets combined with OR: IDS: applicability facets are combined with OR, so a specification tests elements that match only one of them #813 / fix(ids): require every applicability facet to match #814
- Pattern restrictions and length limits: IDS: pattern restrictions match substrings, and length limits overwrite each other #815 / fix(ids): match patterns against the whole value and apply every length limit #816
- XML entities on load and export: IDS: XML entities are not decoded on load, and exported values are not escaped #817 / fix(ids): decode XML entities on load and escape values on export #818
- Entity predefinedType throws, type lookup: IDS: an Entity facet with predefinedType throws on entities that have no PredefinedType attribute #819 / fix(ids): handle entities without a PredefinedType attribute #820
- Optional and prohibited property requirements: IDS: optional and prohibited property requirements give inverted results #821 / fix(ids): apply optional and prohibited cardinality to the property facet as a whole #822
- Type objects' own property sets: IDS: properties on a type object itself (IfcWallType.HasPropertySets) are not checked #823 / fix(ids): check a type object's own property sets in IDSProperty #824
- Property facet throws on properties without data; list, bounded, enumerated and table values: IDS: property facet throws on properties without data (19 official test cases), and list, bounded, enumerated and table values are not checked #827 / fix(ids): don't crash on properties without data; match any value of list, bounded, enumerated and table properties #828, with IfcImporter leaves out list, bounded, enumerated and table properties, so property sets point at items with no data engine_fragment#323 / IfcPropertiesProcessor.getProperties is broken #324
- Attribute facet as applicability selects nothing: IDS: attribute facet used as applicability selects nothing #829
- Classification and PartOf facets are stubs: IDS: Classification and PartOf facets are stubs, so specifications using them pass without checking anything #830
What is left (74 cases with the PRs applied)
| Cause | Cases | Notes |
|---|---|---|
| PartOf facet not implemented | 16 | #830 |
| Numeric comparison tolerance | 14 | Reals must be compared with the tolerance in ImplementersDocumentation/tolerance.md (ϵ = 1.0e-6); values are compared exactly. All tolerance disagreements. |
| Applicability on classes the importer doesn't keep as items | 11 | IfcSurfaceStyleRefraction, IfcSurfaceStyleRendering, IfcTask, IfcTaskTime, IfcPerson, IfcClassification, IfcPresentationLayerWithStyle, IfcRelConnectsPathElements. 31 cases select nothing for this reason; 11 of them are fail- cases that read as passes. Mostly the attribute and restriction folders. |
| Classification facet not implemented | 10 | #830 |
| Unit conversion | 6 | IDS values are in its nominated standard units; files in millimetres are compared without conversion. |
| Material facet | 5 | optional and prohibited cardinality, a constituent set without data, inheritance from the type |
| IFC2X3 type mapping | 4 | e.g. an IfcAirTerminal in an IFC2X3 file, which is an IfcFlowTerminal with a type |
| Specification cardinality | 2 | minOccurs / maxOccurs on a specification aren't read on load, and export always writes minOccurs="1" maxOccurs="unbounded". A required specification with nothing applicable, and a prohibited one whose applicability matches, should fail. |
| Material property sets | 2 | IfcMaterialProperties / IfcExtendedMaterialProperties (the importer's includeMaterialProperties is off by default) |
| Attribute details | 2 | an optional attribute that is empty must fail; GlobalId is compared as a string |
| Several patterns in one restriction | 1 | two xs:pattern values mean either may match; the importer reads .value from the array and gets undefined |
| Predefined property sets | 1 | e.g. IfcDoorLiningProperties |
I can file any of these separately, and implement the ones you'd like help with; the tolerance, unit conversion, specification cardinality and multi-pattern items look self-contained.
Activity