feat(dr-egeria): Promise classification support, lineage-graph for_lineage, JSON Dictionary parsing fix

config/config.json

@@ -41,7 +41,7 @@
     ],
     "console_logging_level": "ERROR",
     "enable_logging": true,
-    "file_logging_level": "WARN",
+    "file_logging_level": "WARNING",
     "log_directory": "/Users/dwolfson/localGit/egeria-v5-3/egeria-python/logs",
     "logging_console_format": "<green>{time:YYYY-MM-DD HH:mm:ss.SSS}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>",
     "logging_file_format": "{time:YYYY-MM-DD HH:mm:ss} | {level} | {module}-{function}:{line}-{message}-{extra}"

docs/dr_egeria_manual.md

@@ -248,7 +248,7 @@ Dr.Egeria organizes its commands into "families," each corresponding to a specif
 - **Governance Officer**: Manage governance definitions, policies, and responsibilities.
 - **Action Author**: Define governance action process flows — reusable single-step action types and multi-step processes — and wire them to the engines and elements that execute them, without writing code (e.g., `Create Governance Action Process`, `Create Governance Action Process Step`, `Link First Process Step`, `Link Next Process Step`, `Link Action to Action Executor`, `Link Action to Target`).
 - **Curation**: Apply classifications and relationships to *existing* Referenceable elements after the fact — this family creates no elements of its own, it curates ones created elsewhere. Every command names a `Target Element` (Reference Name) and either a classification level/status or a second element to relate to.
-  - **Classifications** (`Classify`/`Reclassify`/`Declassify`, or `Classify`/`Update`/`Declassify` for the three multi-value ones): Impact, Confidence, Confidentiality, Criticality, Retention, Ownership, Digital Resource Origin, Zone Membership, Security Tags, Data Scope, Governance Expectations, Governance Measurements, Known Duplicate, Consolidated Duplicate, and the 0438 naming-standards markers Class Word, Modifier, Prime Word (added 2026-08-09 once Egeria PR #9166 shipped the backing `glossary-manager` endpoints — routed through `GlossaryManager`, not `ClassificationExplorer`, unlike every other classification here). Also, added 2026-08-21 once an `omvs_audit.py` gap-closure pass shipped their backing REST endpoints (all confirmed against a live 6.2-SNAPSHOT server): the 10 governance-point classifications (0435: Control/Verification/Enforcement/Execution Point, Policy Administration/Decision/Enforcement/Information/Management/Retrieval Point — routed through `GovernanceOfficer`), 6 metamodel markers (0463: Incomplete, ObjectIdentifier, ReferenceData, MobileResource, InstanceMetadata, MetamodelInstance — routed through `ClassificationExplorer`), ProjectKind (routed through `ProjectManager`), CollectionKind (routed through `CollectionManager`), and a Data Sharing Agreement retrofit pair for classifying/declassifying an already-existing Agreement (routed through `DigitalBusiness` — `Create Data Sharing Agreement`, Digital Products family, already sets this at creation time via `initialClassifications`).
+  - **Classifications** (`Classify`/`Reclassify`/`Declassify`, or `Classify`/`Update`/`Declassify` for the three multi-value ones): Impact, Confidence, Confidentiality, Criticality, Retention, Ownership, Digital Resource Origin, Zone Membership, Security Tags, Data Scope, Governance Expectations, Governance Measurements, Known Duplicate, Consolidated Duplicate, and the 0438 naming-standards markers Class Word, Modifier, Prime Word (added 2026-08-09 once Egeria PR #9166 shipped the backing `glossary-manager` endpoints — routed through `GlossaryManager`, not `ClassificationExplorer`, unlike every other classification here). Also, added 2026-08-21 once an `omvs_audit.py` gap-closure pass shipped their backing REST endpoints (all confirmed against a live 6.2-SNAPSHOT server): the 10 governance-point classifications (0435: Control/Verification/Enforcement/Execution Point, Policy Administration/Decision/Enforcement/Information/Management/Retrieval Point — routed through `GovernanceOfficer`), 6 metamodel markers (0463: Incomplete, ObjectIdentifier, ReferenceData, MobileResource, InstanceMetadata, MetamodelInstance — routed through `ClassificationExplorer`), ProjectKind (routed through `ProjectManager`), CollectionKind (routed through `CollectionManager`), and a Data Sharing Agreement retrofit pair for classifying/declassifying an already-existing Agreement (routed through `DigitalBusiness` — `Create Data Sharing Agreement`, Digital Products family, already sets this at creation time via `initialClassifications`). Also Promise (0010, added 2026-09-30 — `Classify`/`Declassify Promise`, routed through `ClassificationExplorer`): marks an element as a placeholder for a real-world resource that has not yet been delivered, with Deployment Status, Start/Due/Last Review/Completion Time and Additional Properties. A promised element is only visible to lineage requests, so `Declassify Promise` looks up its Target Element and sends the clear request with `forLineage=true` automatically.
   - **Relationships** (`Link`/`Unlink`, or `Attach`/`Update`/`Detach` for Search Keyword): Search Keyword, Semantic Assignment, Semantic Definition, Link Element To Scope (`ScopedBy`), Link Resource To Element (`ResourceList`), Link More Information, Peer Duplicate, Consolidated Duplicate Link.
   - Investigation (0130 ProjectKind, routed through `ProjectManager`) and NamingStandardsVocabulary (0438, routed through `GlossaryManager`) — new Data Standards types from Egeria PR #9300, added 2026-09-08. **Not yet verified against a live server**: the type isn't deployed there yet (`validate_compact_specs` fails its OM_TYPE-against-live-server check for both until the server picks up the new archive), and this checkout's `.http` ground truth files haven't been updated for this PR either — re-verify (including `validate_compact_specs`) once both are current.
   - No classification in this family remains genuinely unimplemented as of 2026-08-21 — Policy Management Point (previously blocked on a missing pyegeria SDK method) is now registered along with the 9 sibling governance-point classifications above.

md_processing/data/compact_commands/commands_curation_compact.json

@@ -587,7 +587,7 @@
       "default_value": "",
       "valid_values": [],
       "existing_element": "",
-      "description": "Due date/time for a Meeting, ToDo, or Review person action.",
+      "description": "Due date/time - for a Meeting, ToDo, or Review person action, or for delivery of the resource described by a Promise.",
       "generated": false,
       "style": "Simple",
       "user_specified": true,
@@ -2842,6 +2842,112 @@
       "max_cardinality": 1,
       "level": "Domain",
       "Journal Entry": ""
+    },
+    "Deployment Status": {
+      "variable_name": "deployment_status",
+      "inUpdate": true,
+      "attr_labels": "",
+      "examples": "",
+      "default_value": "",
+      "valid_values": [
+        "PROPOSED",
+        "UNDER_DEVELOPMENT",
+        "DEVELOPMENT_COMPLETE",
+        "APPROVED_FOR_DEPLOYMENT",
+        "REJECTED",
+        "STANDBY",
+        "ACTIVE",
+        "DISABLED",
+        "FAILED",
+        "OTHER"
+      ],
+      "existing_element": "",
+      "description": "Progress toward delivering the promised real-world digital resource/artifact - one of an enumerated set of values (DeploymentStatus).",
+      "generated": false,
+      "style": "Valid Value",
+      "user_specified": true,
+      "unique": false,
+      "input_required": false,
+      "min_cardinality": 0,
+      "max_cardinality": 1,
+      "level": "Basic",
+      "Journal Entry": "Added 2026-09-30 -- Promise classification (0010), Classification Explorer setElementAsPromise."
+    },
+    "User Defined Deployment Status": {
+      "variable_name": "user_defined_deployment_status",
+      "inUpdate": true,
+      "attr_labels": "",
+      "examples": "",
+      "default_value": "",
+      "valid_values": [],
+      "existing_element": "",
+      "description": "Locally defined deployment status - used when Deployment Status is OTHER.",
+      "generated": false,
+      "style": "Simple",
+      "user_specified": true,
+      "unique": false,
+      "input_required": false,
+      "min_cardinality": 0,
+      "max_cardinality": 1,
+      "level": "Domain",
+      "Journal Entry": "Added 2026-09-30 -- Promise classification (0010), Classification Explorer setElementAsPromise."
+    },
+    "Start Time": {
+      "variable_name": "start_time",
+      "inUpdate": true,
+      "attr_labels": "",
+      "examples": "",
+      "default_value": "",
+      "valid_values": [],
+      "existing_element": "",
+      "description": "Date/time (ISO 8601) that work started on delivering the promised resource.",
+      "generated": false,
+      "style": "Simple",
+      "user_specified": true,
+      "unique": false,
+      "input_required": false,
+      "min_cardinality": 0,
+      "max_cardinality": 1,
+      "level": "Domain",
+      "Journal Entry": "Added 2026-09-30 -- Promise classification (0010), Classification Explorer setElementAsPromise."
+    },
+    "Last Review Time": {
+      "variable_name": "last_review_time",
+      "inUpdate": true,
+      "attr_labels": "",
+      "examples": "",
+      "default_value": "",
+      "valid_values": [],
+      "existing_element": "",
+      "description": "Date/time (ISO 8601) that the promise was last reviewed.",
+      "generated": false,
+      "style": "Simple",
+      "user_specified": true,
+      "unique": false,
+      "input_required": false,
+      "min_cardinality": 0,
+      "max_cardinality": 1,
+      "level": "Domain",
+      "Journal Entry": "Added 2026-09-30 -- Promise classification (0010), Classification Explorer setElementAsPromise."
+    },
+    "Completion Time": {
+      "variable_name": "completion_time",
+      "inUpdate": true,
+      "attr_labels": "",
+      "examples": "",
+      "default_value": "",
+      "valid_values": [],
+      "existing_element": "",
+      "description": "Date/time (ISO 8601) that the promised resource was delivered.",
+      "generated": false,
+      "style": "Simple",
+      "user_specified": true,
+      "unique": false,
+      "input_required": false,
+      "min_cardinality": 0,
+      "max_cardinality": 1,
+      "level": "Domain",
+      "Journal Entry": "Added 2026-09-30 -- Promise classification (0010), Classification Explorer setElementAsPromise."
     }
   },
   "bundles": {
@@ -3316,6 +3422,19 @@
         "Target Element",
         "Additional Properties"
       ]
+    },
+    "Promise Base": {
+      "inherits": "Request Base",
+      "own_attributes": [
+        "Target Element",
+        "Deployment Status",
+        "User Defined Deployment Status",
+        "Start Time",
+        "Due Time",
+        "Last Review Time",
+        "Completion Time",
+        "Additional Properties"
+      ]
     }
   },
   "commands": {
@@ -5370,6 +5489,44 @@
       "bundle": "Declassify Base",
       "custom_attributes": [],
       "Journal Entry": ""
+    },
+    "Classify Promise": {
+      "display_name": "promise",
+      "qn_prefix": "Promise",
+      "alternate_names": [],
+      "family": "Curation",
+      "description": "Classify an existing element as a Promise - a placeholder for a real-world digital resource/artifact that has not yet been delivered. Once classified, the element is only returned to lineage requests (For Lineage = true).",
+      "verb": "Classify",
+      "upsert": false,
+      "attach": false,
+      "level": "Basic",
+      "find_method": "",
+      "find_constraints": "",
+      "extra_find": "",
+      "extra_constraints": "",
+      "OM_TYPE": "Promise",
+      "bundle": "Promise Base",
+      "custom_attributes": [],
+      "Journal Entry": "Added 2026-09-30 -- Promise classification (0010), Classification Explorer setElementAsPromise."
+    },
+    "Declassify Promise": {
+      "display_name": "promise",
+      "qn_prefix": "Promise",
+      "alternate_names": [],
+      "family": "Curation",
+      "description": "Remove the Promise classification from an element - typically because the promised resource has now been delivered. The Target Element is looked up with For Lineage = true, since a promised element is otherwise invisible.",
+      "verb": "Declassify",
+      "upsert": false,
+      "attach": false,
+      "level": "Basic",
+      "find_method": "",
+      "find_constraints": "",
+      "extra_find": "",
+      "extra_constraints": "",
+      "OM_TYPE": "Promise",
+      "bundle": "Declassify Base",
+      "custom_attributes": [],
+      "Journal Entry": "Added 2026-09-30 -- Promise classification (0010), Classification Explorer setElementAsPromise."
     }
   }
 }

md_processing/v2/curation.py

@@ -13,8 +13,11 @@
 Policy Administration/Policy Decision/Policy Enforcement/Policy Information/
 Policy Management/Policy Retrieval Point), 6 metamodel/classification-
 explorer markers (0463: Incomplete, ObjectIdentifier, ReferenceData,
-MobileResource, InstanceMetadata, MetamodelInstance), and ProjectKind (0130)/
-CollectionKind (0021).
+MobileResource, InstanceMetadata, MetamodelInstance), ProjectKind (0130)/
+CollectionKind (0021), and Promise (0010 -- a placeholder for a not-yet-delivered
+resource; the promised element is only visible to forLineage=true requests, so
+Declassify Promise resolves its Target Element and sends its clear request with
+forLineage=true, see ClassificationSpec.declassify_for_lineage).
 
 CurationLinkProcessor -- Link/Unlink/Attach/Detach relationship commands
 for: Semantic Assignment, Semantic Definition, Scoped By, Peer Duplicate,
@@ -88,6 +91,10 @@ class ClassificationSpec:
     update_method: Optional[str] = None
     level_field: Optional[str] = None  # e.g. "severityLevel" -- reads "Level Identifier"
     fields: Dict[str, str] = dc_field(default_factory=dict)  # attribute name -> property name
+    # True when the classified element is hidden from normal queries (e.g. Promise), so
+    # Declassify must both resolve its Target Element and send the clear request with
+    # forLineage=true -- Egeria rejects/cannot see it otherwise.
+    declassify_for_lineage: bool = False
 
 
 # Shared governance-classification fields (Impact/Confidence/Confidentiality/Criticality/Retention)
@@ -236,6 +243,17 @@ class ClassificationSpec:
     "NamingStandardsVocabulary": ClassificationSpec(
         "_async_set_glossary_as_naming_standards_vocabulary", "_async_clear_glossary_as_naming_standards_vocabulary",
         "NamingStandardsVocabularyProperties"),
+    # Promise (0010) -- added 2026-09-30 once ClassificationExplorer's setElementAsPromise/
+    # clearElementAsPromise shipped upstream. A promised element is only returned to
+    # lineage requests, hence declassify_for_lineage. Not yet verified against a live server.
+    "Promise": ClassificationSpec(
+        "_async_set_element_as_promise", "_async_clear_element_as_promise", "PromiseProperties",
+        fields={"Deployment Status": "deploymentStatus",
+                "User Defined Deployment Status": "userDefinedDeploymentStatus",
+                "Start Time": "startTime", "Due Time": "dueTime",
+                "Last Review Time": "lastReviewTime", "Completion Time": "completionTime",
+                "Additional Properties": "additionalProperties"},
+        declassify_for_lineage=True),
 }
 
 # OM_TYPEs in CLASSIFICATION_METHODS whose set/clear methods live on a client other than
@@ -301,6 +319,10 @@ def supports_target_element_lookup(self) -> bool:
     async def fetch_as_is(self) -> Optional[Dict[str, Any]]:
         return None
 
+    def lookup_for_lineage(self) -> bool:
+        class_spec = CLASSIFICATION_METHODS.get(self.get_command_spec().get("OM_TYPE"))
+        return bool(class_spec and class_spec.declassify_for_lineage and self.command.verb == "Declassify")
+
     async def apply_changes(self) -> str:
         verb = self.command.verb
         object_type = self.canonical_object_type or self.command.object_type
@@ -319,6 +341,8 @@ async def apply_changes(self) -> str:
 
         if verb == "Declassify":
             body = {"class": "DeleteClassificationRequestBody", **_audit_fields(attributes)}
+            if class_spec.declassify_for_lineage:
+                body["forLineage"] = True
             await getattr(client, class_spec.clear_method)(element_guid, body)
             logger.success(f"Declassified {object_type} on {element_guid}")
             return f"\n\n## {verb} {object_type}\n\nRemoved {om_type} classification from {element_guid}."

md_processing/v2/processors.py

@@ -1389,6 +1389,12 @@ async def fetch_element(self, guid: str, _max_timeout_retries: int = 2) -> Optio
 
         return None
 
+    def lookup_for_lineage(self) -> bool:
+        """Whether reference lookups for this command should include elements only visible to
+        lineage requests (forLineage=true) -- e.g. Declassify Promise, whose target is hidden
+        from normal queries once promised. Overridden per processor; default False."""
+        return False
+
     async def resolve_element_guid(self, name_or_guid: str, tech_type: Optional[str] = None) -> Optional[str]:
         """
         Resolves a name or GUID to a GUID using various strategies.
@@ -1478,7 +1484,11 @@ def _cache_if_not_found(key: tuple, value: Any) -> None:
             unsupported_type_warnings = self.context.setdefault("_unsupported_lookup_types_warned", set())
 
             # Pass 1: Try WITH type constraint (fastest, avoids ambiguity)
-            pass1_key = (name_or_guid, tech_type or None)
+            for_lineage = self.lookup_for_lineage()
+            # Lineage lookups see a different element set, so they get their own cache keys.
+            lineage_key = ("forLineage",) if for_lineage else ()
+            lineage_kwargs = {"for_lineage": True} if for_lineage else {}
+            pass1_key = (name_or_guid, tech_type or None, *lineage_key)
             cached1 = query_cache.get(pass1_key, _NOT_CACHED)
             if cached1 is _AMBIGUOUS:
                 logger.debug(f"resolve_element_guid: cache hit (ambiguous, Pass 1) for {pass1_key!r}")
@@ -1489,7 +1499,7 @@ def _cache_if_not_found(key: tuple, value: Any) -> None:
                 res = cached1
             else:
                 try:
-                    res = await self.client.__async_get_guid__(qualified_name=name_or_guid, display_name=name_or_guid, property_name="displayName", tech_type=tech_type or None)
+                    res = await self.client.__async_get_guid__(qualified_name=name_or_guid, display_name=name_or_guid, property_name="displayName", tech_type=tech_type or None, **lineage_kwargs)
                     _cache_if_not_found(pass1_key, res)
                 except PyegeriaException as e:
                     # Catch multiple matches error
@@ -1517,7 +1527,7 @@ def _cache_if_not_found(key: tuple, value: Any) -> None:
             # Pass 2: If no result (or if type was invalid), try WITHOUT type constraint
             is_not_found = not res or (isinstance(res, str) and (res.startswith("No ") or " found" in res))
             if is_not_found and tech_type:
-                pass2_key = (name_or_guid, None)
+                pass2_key = (name_or_guid, None, *lineage_key)
                 cached2 = query_cache.get(pass2_key, _NOT_CACHED)
                 if cached2 is _AMBIGUOUS:
                     logger.debug(f"resolve_element_guid: cache hit (ambiguous, Pass 2) for {pass2_key!r}")
@@ -1528,7 +1538,7 @@ def _cache_if_not_found(key: tuple, value: Any) -> None:
                     res = cached2
                 else:
                     try:
-                        res = await self.client.__async_get_guid__(qualified_name=name_or_guid, display_name=name_or_guid, property_name="displayName")
+                        res = await self.client.__async_get_guid__(qualified_name=name_or_guid, display_name=name_or_guid, property_name="displayName", **lineage_kwargs)
                         _cache_if_not_found(pass2_key, res)
                     except PyegeriaException as e:
                         if "Multiple elements found" in str(e):