Drug mapping enhancements + Dependabot fixes

docs/development/plugins.md

@@ -76,12 +76,18 @@ The concept-embedding row and the free-text-embedding row look similar but are n
 
 ### Assumptions this table is making
 
-- **Vocabulary only, so far.** Every example above reads OMOP *vocabulary* i.e. operations over concepts, concept relationships, concept embeddings. Nothing about `context.cdm_engine` restricts a plugin to vocabulary tables specifically, however. These tables are available over the same CDM connection groundworkers itself uses, and a plugin *could* query clinical/person-level tables (`condition_occurrence`, `drug_exposure`, etc.) through it just as easily, subject to whatever environmental and procedural access controls actually govern that data. 
-- **Plugin graph access is TODO.** `PluginContext` does not expose graph traversal at this time, which means that a plugin that wants graph traversal would have to depend on `omop-graph` itself and build its own adapter, reusing `context.cdm_engine` for the connection but not groundworkers' own shared instance. Recommend not doing it that way, however, as it would be easier to expose the core graph the same way `vector_store` and `embedding_backend_factory` already are. 
+- **Vocabulary only, so far.** Every example above reads OMOP *vocabulary* i.e. operations over concepts, concept relationships, concept embeddings, or the host's shared grounding cascade. Nothing about `context.cdm_engine` restricts a plugin to vocabulary tables specifically, however. These tables are available over the same CDM connection groundworkers itself uses, and a plugin *could* query clinical/person-level tables (`condition_occurrence`, `drug_exposure`, etc.) through it just as easily, subject to whatever environmental and procedural access controls actually govern that data.
+- **Shared mapping access.** `PluginContext.mapping_context` is an optional,
+  read-only capability façade over the host's mapping and grounding policy.
+  Use its operations when a plugin needs candidate bundles, text grounding, or
+  concept context; the underlying vocabulary, graph, and service objects are
+  deliberately not exposed. Do not construct a second `omop-graph` or
+  vocabulary adapter from `context.cdm_engine`. The field is `None` only when
+  the host did not compose services for the plugin context.
 
 ## Build and register a new plugin
 
-`build()` receives `PluginContext`. The context provides the resolved CDM database and engine, vector store, shared lazy model backend factories, and a deliberately narrow resolver for independent named OA resources. Return `None` when a prerequisite is unavailable. Groundworkers then keeps serving its core capabilities and reports why the plugin was not activated.
+`build()` receives `PluginContext`. The context provides the resolved CDM database and engine, vector store, shared lazy model backend factories, the optional read-only `mapping_context`, and a deliberately narrow resolver for independent named OA resources. Return `None` when a prerequisite is unavailable. Groundworkers then keeps serving its core capabilities and reports why the plugin was not activated.
 
 `register()` adds MCP tools, prompts, or resources against the state returned by `build()`. The shared MCP registration wrapper supplies the same safe error translation used by core tools. Code-defined prompts can continue to derive their arguments from the callable signature. A plugin publishing data-defined prompts can pass explicit string metadata with `PromptArgument`; this keeps dynamic pack fields out of Python closure signatures:
 

docs/services/vocab.md

@@ -48,11 +48,29 @@ vocab.search_normalized(
     vocabulary_id: str | None = None,
     standard_only: bool = False,
     limit: int = 20,
+    allow_unindexed_scan: bool = False,
 ) -> list[ConceptMatch]
 ```
 
 Normalized search: lowercases and strips punctuation from both the query and concept names before matching. Catches common surface-form differences (abbreviations, spacing, case) that exact search misses without the false-positive risk of full-text search.
 
+#### Why this one needs a selective filter
+
+Comparison is on a *computed* expression, so an ordinary index on `concept_name` cannot serve it. Without a selective filter the planner scans every concept. Measured against a 1.7M-concept CDM:
+
+| Call | Time |
+| --- | --- |
+| unconstrained | 40s |
+| `domain='Drug'` | 25s |
+| `vocabulary_id='MedDRA'` | 0.4s |
+
+So the call is **refused** unless one of the following holds:
+
+- `vocabulary_id` or `parent_ids` is given. Note that `domain` does **not** count — it looks like a narrowing filter but the largest domains hold millions of concepts, which is why `domain='Drug'` still took 25s.
+- `allow_unindexed_scan=True`, which accepts the scan deliberately.
+
+It raises rather than warning because the unindexed query does not fail — it succeeds slowly, so a warning gets absorbed into a caller that merely looks sluggish.
+
 ### `search_fulltext`
 
 ```python

src/groundworkers/app.py

@@ -15,6 +15,7 @@
 from groundworkers.plugins import (
     GroundworkersPlugin,
     GroundworkersPluginReadiness,
+    MappingContext,
     PluginConfigResolver,
     PluginContext,
     PluginReadinessResult,
@@ -52,8 +53,12 @@ class Adapters:
     # _build_plugin_context) so a plugin reuses the one backend connection
     # instead of opening a second one. None when the corresponding backend
     # is unconfigured, same gating as the adapters above.
-    embedding_backend_factory: Callable[[], Any] | None = field(default=None, repr=False)
-    embedding_model_backend_factory: Callable[[], Any] | None = field(default=None, repr=False)
+    embedding_backend_factory: Callable[[], Any] | None = field(
+        default=None, repr=False
+    )
+    embedding_model_backend_factory: Callable[[], Any] | None = field(
+        default=None, repr=False
+    )
     chat_backend_factory: Callable[[], Any] | None = field(default=None, repr=False)
 
 
@@ -104,7 +109,9 @@ def get_model_backend() -> ModelBackend:
         nonlocal shared_model_backend
         if shared_model_backend is None:
             if resolved_model is None:
-                raise RuntimeError("No embedding model is configured for Groundworkers.")
+                raise RuntimeError(
+                    "No embedding model is configured for Groundworkers."
+                )
             from omop_llm import build_model_backend_from_resolved
 
             shared_model_backend = build_model_backend_from_resolved(resolved_model)
@@ -180,9 +187,14 @@ def get_chat_backend() -> ModelBackend:
 def build_services(config: AppConfig, adapters: Adapters) -> Services:
     services = Services()
     assisted_classifier = None
-    if adapters.llm is not None and config.groundworkers.source_planning_llm_assisted_enabled:
+    if (
+        adapters.llm is not None
+        and config.groundworkers.source_planning_llm_assisted_enabled
+    ):
         assisted_classifier = AssistedColumnRoleClassifier(adapters.llm)
-    services.source_planning = SourcePlanningService(assisted_classifier=assisted_classifier)
+    services.source_planning = SourcePlanningService(
+        assisted_classifier=assisted_classifier
+    )
     if adapters.omop_graph is not None:
         # cdm is optional; only the classified-edge traversals
         # (concept_associations / concept_extended_inheritance) require it.
@@ -206,7 +218,27 @@ def build_services(config: AppConfig, adapters: Adapters) -> Services:
     return services
 
 
-def _build_plugin_context(config: AppConfig, adapters: Adapters) -> PluginContext:
+def build_mapping_context(
+    config: AppConfig,
+    *,
+    adapters: Adapters | None = None,
+    services: Services | None = None,
+) -> MappingContext:
+    """Build the mapping capabilities shared by hosted and standalone plugins."""
+
+    shared_adapters = adapters or build_adapters(config)
+    shared_services = services or build_services(config, shared_adapters)
+    return MappingContext(
+        mapping_service=shared_services.mapping,
+        grounding_service=shared_services.grounding,
+    )
+
+
+def _build_plugin_context(
+    config: AppConfig,
+    adapters: Adapters,
+    services: Services | None = None,
+) -> PluginContext:
     """Assemble the resolved handles every plugin is given.
 
     `cdm_database`/`vector_store` come straight off `AppConfig`, already
@@ -223,6 +255,11 @@ def _build_plugin_context(config: AppConfig, adapters: Adapters) -> PluginContex
         embedding_backend_factory=adapters.embedding_backend_factory,
         embedding_model_backend_factory=adapters.embedding_model_backend_factory,
         chat_backend_factory=adapters.chat_backend_factory,
+        mapping_context=(
+            build_mapping_context(config, adapters=adapters, services=services)
+            if services is not None
+            else None
+        ),
     )
 
 
@@ -292,7 +329,8 @@ def verify_plugin_readiness(
             summary="This plugin does not expose readiness verification.",
         )
     adapters = build_adapters(config)
-    context = _build_plugin_context(config, adapters)
+    services = build_services(config, adapters)
+    context = _build_plugin_context(config, adapters, services)
     state, issue = _build_plugin(config, context, plugin)
     if state is None:
         return PluginReadinessResult(
@@ -342,13 +380,14 @@ def load_plugin_readiness(
 
 def build_application(config: AppConfig) -> GroundworkersApp:
     adapters = build_adapters(config)
-    context = _build_plugin_context(config, adapters)
+    services = build_services(config, adapters)
+    context = _build_plugin_context(config, adapters, services)
     plugin_definitions = tuple(discover_plugins())
     plugins, plugin_issues = _build_plugins(config, context, plugin_definitions)
     return GroundworkersApp(
         config=config,
         adapters=adapters,
-        services=build_services(config, adapters),
+        services=services,
         plugins=plugins,
         plugin_definitions=plugin_definitions,
         plugin_issues=plugin_issues,

src/groundworkers/application/setup/databases.py

@@ -111,6 +111,9 @@ def resolve_database_targets(
     cdm = resolver.resolve_database(groundworkers.cdm_db)
     if not isinstance(cdm, ResolvedCDMDatabase):
         return ()
+    cdm_schema = cdm.schema_name or (
+        "main" if cdm.connection.url.startswith("sqlite") else "public"
+    )
     embedding_target: DatabaseTarget | None = None
     expected_embedding_model_name: str | None = None
     embedding_safe_url: str | None = None
@@ -146,7 +149,7 @@ def resolve_database_targets(
             database_entry_name=cdm.name,
             connection_name=cdm.connection.name,
             safe_url=cdm.connection.safe_url,
-            cdm_schema=cdm.schema_name or "main",
+            cdm_schema=cdm_schema,
             vocabulary_schema=cdm.vocab_schema,
             connection_url=cdm.connection.url,
             role="cdm",
@@ -157,7 +160,7 @@ def resolve_database_targets(
             database_entry_name=cdm.name,
             connection_name=cdm.connection.name,
             safe_url=cdm.connection.safe_url,
-            cdm_schema=cdm.schema_name or "main",
+            cdm_schema=cdm_schema,
             vocabulary_schema=cdm.vocab_schema,
             connection_url=cdm.connection.url,
             role="graph",
@@ -168,7 +171,7 @@ def resolve_database_targets(
             database_entry_name=cdm.name,
             connection_name=cdm.connection.name,
             safe_url=cdm.connection.safe_url,
-            cdm_schema=cdm.schema_name or "main",
+            cdm_schema=cdm_schema,
             vocabulary_schema=cdm.vocab_schema,
             connection_url=cdm.connection.url,
             role="groundworkers",
@@ -478,9 +481,7 @@ def _groundworkers_embedding_model_diagnostics(
                         "Groundworkers grounding model could not be checked because the omop-emb model registry table is missing.",
                     ),
                 )
-            rows = _embedding_registry_rows(
-                connection, schema=target.embedding_schema
-            )
+            rows = _embedding_registry_rows(connection, schema=target.embedding_schema)
     except Exception as exc:
         # Broad except: reported as a redacted warning.
         failure = classify_connection_error(exc)
@@ -903,10 +904,7 @@ def _column_has_null_values(
     column = quote_identifier(connection, column_name)
     return bool(
         connection.execute(
-            text(
-                f"SELECT EXISTS (SELECT 1 FROM {qualified} "
-                f"WHERE {column} IS NULL)"
-            )
+            text(f"SELECT EXISTS (SELECT 1 FROM {qualified} WHERE {column} IS NULL)")
         ).scalar()
     )
 

src/groundworkers/application/setup/performance_maintenance.py

@@ -38,7 +38,11 @@ def build_performance_commands(
 ) -> tuple[Command, ...]:
     """Build safe, out-of-process commands for the selected indexes."""
 
-    environment = () if config_path is None else (("OA_CONFIG_PATH", str(Path(config_path).expanduser())),)
+    environment = (
+        ()
+        if config_path is None
+        else (("OA_CONFIG_PATH", str(Path(config_path).expanduser())),)
+    )
     commands: list[Command] = []
     selected = set(actions)
     if PerformanceRemediation.TRIGRAM_INDEXES in selected:
@@ -55,7 +59,9 @@ def build_performance_commands(
         )
     if PerformanceRemediation.EMBEDDING_INDEX in selected:
         if not embedding_model:
-            raise ValueError("A registered embedding model is required to build its index.")
+            raise ValueError(
+                "A registered embedding model is required to build its index."
+            )
         commands.append(
             Command(
                 argv=(
@@ -114,7 +120,11 @@ def populate_trigram_indexes(snapshot: ConfigurationSnapshot) -> None:
             connection.execute(text("CREATE EXTENSION IF NOT EXISTS pg_trgm"))
             for table, column, index_name in (
                 ("concept", "concept_name", "idx_concept_lower_name_trgm"),
-                ("concept_synonym", "concept_synonym_name", "idx_concept_synonym_lower_name_trgm"),
+                (
+                    "concept_synonym",
+                    "concept_synonym_name",
+                    "idx_concept_synonym_lower_name_trgm",
+                ),
             ):
                 connection.execute(
                     text(

src/groundworkers/application/setup/runtime_setup.py

@@ -498,4 +498,3 @@ def _optional_str(value: object) -> str | None:
 
 def _effective_tool(stack: StackConfig, name: str):
     return stack.tools.get(name)
-

src/groundworkers/base/sql.py

@@ -2,9 +2,30 @@
 
 from __future__ import annotations
 
+from enum import StrEnum
+
+from sqlalchemy import func
 from sqlalchemy.engine import Connection, Engine
 
-__all__ = ["effective_schema", "quote_identifier"]
+__all__ = [
+    "STOP_PHRASE_SQL_PATTERN",
+    "NormalizationProfile",
+    "effective_schema",
+    "normalization_expression",
+    "quote_identifier",
+]
+
+
+class NormalizationProfile(StrEnum):
+    """Supported normalization contracts shared by Python and SQL callers."""
+
+    VERBATIM = "verbatim"
+    AGGRESSIVE = "aggressive"
+    DRUG_NAME = "drug_name"
+
+
+#: Stop phrases stripped before comparison, as a PostgreSQL regex.
+STOP_PHRASE_SQL_PATTERN = r"\m(?:nos|nec|nfs|unspecified|unknown|other|w/?o|w/)\M"
 
 
 def quote_identifier(bind: Engine | Connection, name: str) -> str:
@@ -28,3 +49,24 @@ def effective_schema(bind: Engine | Connection) -> str | None:
     comes from, so reflection and generated SQL cannot disagree.
     """
     return bind.get_execution_options().get("schema_translate_map", {}).get(None)
+
+
+def normalization_expression(
+    column,
+    *,
+    profile: str | NormalizationProfile,
+    remove_stop_phrases: bool,
+):
+    """Build the SQL expression used for normalized comparison."""
+    profile = NormalizationProfile(profile)
+    space = " "
+    globally = "g"
+    expr = func.lower(column)
+    if remove_stop_phrases:
+        expr = func.regexp_replace(expr, STOP_PHRASE_SQL_PATTERN, space, globally)
+    expr = func.regexp_replace(expr, r"[^a-z0-9]+", space, globally)
+    if profile == "drug_name":
+        expr = func.replace(expr, " extended release ", space)
+        expr = func.replace(expr, " modified release ", space)
+    expr = func.regexp_replace(expr, r"\s+", space, globally)
+    return func.btrim(expr)