Split catalog_selection into a validation module and a database-reads module

AGENTS.md

@@ -104,7 +104,7 @@ To add or change a rule, read `docs/engineering/skills/rules-registry.md`.
 
 ### Data source isolation
 
-- Checked: `api.catalog.extractor-import-isolation`, `api.metadata.read-modules-isolated`, `api.reports.v1-persistence-only`, `api.stage12.comparison-tables-isolated`, `api.stage12.retirement-owned`, `api.v2.single-orm-layer`
+- Checked: `api.catalog.extractor-import-isolation`, `api.catalog.selection-validation-no-database`, `api.metadata.read-modules-isolated`, `api.reports.v1-persistence-only`, `api.stage12.comparison-tables-isolated`, `api.stage12.retirement-owned`, `api.v2.single-orm-layer`
 
 ### Migration and schema
 

changelog.d/4176.changed.md

@@ -0,0 +1 @@
+Split the v2 catalog version selection into `catalog_selection_validation`, which loads no database code, and `catalog_selection_reads`, which holds the one catalog query, and stop the `policyengine_api.data.v2.catalog` package importing its extraction module when it loads. Importing a v2 validators, transformations or types module no longer loads SQLAlchemy unless that module imports the table models itself.

docs/engineering/skills/v2-code-organization.md

@@ -173,6 +173,13 @@ route -> service -> database session + database connectors -> SQLModel tables
                    -> types
 ```
 
+Catalog version validation, its error types and `SelectedCatalog` live in
+`policyengine_api/data/v2/catalog/catalog_selection_validation.py`, which
+loads no database code, so validators, transformations, types and routes may
+import it (rule `api.catalog.selection-validation-no-database`). The catalog
+query itself, `select_catalog`, is in `catalog_selection_reads.py`; the
+policies and metadata read connectors call it.
+
 Database connectors may consume service-layer input types when inserting or
 updating rows. They must not import route modules. Validators and
 transformations may consume types and already-loaded database model instances,

docs/generated/rules.md

@@ -8,11 +8,11 @@ Every rule that governs `policyengine-api` is listed here with how it is enforce
 
 | Tier | enforced | ratcheted | advisory | doc-only | Total |
 | --- | ---: | ---: | ---: | ---: | ---: |
-| mechanical | 60 | 0 | 1 | 12 | 73 |
+| mechanical | 61 | 0 | 1 | 12 | 74 |
 | metric | 1 | 0 | 0 | 0 | 1 |
 | judgment | 0 | 0 | 0 | 1 | 1 |
 
-**Blocking:** 61 of 75 rules.
+**Blocking:** 62 of 76 rules.
 
 ## Gaps
 
@@ -200,6 +200,7 @@ flowchart LR
 | Rule | Statement | Tier | Status | Blocking | Enforced by | CI | Proof |
 | --- | --- | --- | --- | --- | --- | --- | --- |
 | `api.catalog.extractor-import-isolation` | The catalog extractor never imports PolicyEngine core, country packages or v1 metadata directly. | mechanical | enforced | yes | pytest `tests/unit/v2/test_catalog_extraction.py` | `pr.yml:test` | **unproven** |
+| `api.catalog.selection-validation-no-database` | Importing catalog_selection_validation (catalog version validation, its error types and SelectedCatalog) loads no SQLAlchemy, SQLModel, v2 table model or v2 database module; the catalog query lives in catalog_selection_reads. | mechanical | enforced | yes | pytest `tests/unit/v2/test_catalog_selection_modules.py` | `pr.yml:test` | yes |
 | `api.metadata.read-modules-isolated` | v2 metadata read modules never import PolicyEngine packages or v1 metadata sources. | mechanical | enforced | yes | pytest `tests/unit/v2/test_metadata_service.py` | `pr.yml:test` | **unproven** |
 | `api.reports.v1-persistence-only` | The existing report route and service import only v1 persistence. | mechanical | enforced | yes | pytest `tests/unit/v2/test_report_v1_runtime_selection.py` | `pr.yml:test` | **unproven** |
 | `api.stage12.comparison-tables-isolated` | Serving routes and services never read the Stage 12 comparison tables, and production tables never depend on them. | mechanical | enforced | yes | pytest `tests/unit/v2/test_stage12_comparison_isolation.py` | `pr.yml:test` | **unproven** |
@@ -211,6 +212,8 @@ flowchart LR
   r_api_catalog_extractor_import_isolation("api.catalog.extractor-import-isolation (enforced)")
   e_pytest_tests_unit_v2_test_catalog_extraction_py["pytest tests/unit/v2/test_catalog_extraction.py"]
   j_CI_pr_yml_test["CI pr.yml:test"]
+  r_api_catalog_selection_validation_no_database("api.catalog.selection-validation-no-database (enforced)")
+  e_pytest_tests_unit_v2_test_catalog_selection_modules_py["pytest tests/unit/v2/test_catalog_selection_modules.py"]
   r_api_metadata_read_modules_isolated("api.metadata.read-modules-isolated (enforced)")
   e_pytest_tests_unit_v2_test_metadata_service_py["pytest tests/unit/v2/test_metadata_service.py"]
   r_api_reports_v1_persistence_only("api.reports.v1-persistence-only (enforced)")
@@ -222,6 +225,8 @@ flowchart LR
   e_pytest_tests_unit_v2_test_model_persistence_py["pytest tests/unit/v2/test_model_persistence.py"]
   r_api_catalog_extractor_import_isolation --> e_pytest_tests_unit_v2_test_catalog_extraction_py
   e_pytest_tests_unit_v2_test_catalog_extraction_py --> j_CI_pr_yml_test
+  r_api_catalog_selection_validation_no_database --> e_pytest_tests_unit_v2_test_catalog_selection_modules_py
+  e_pytest_tests_unit_v2_test_catalog_selection_modules_py --> j_CI_pr_yml_test
   r_api_metadata_read_modules_isolated --> e_pytest_tests_unit_v2_test_metadata_service_py
   e_pytest_tests_unit_v2_test_metadata_service_py --> j_CI_pr_yml_test
   r_api_reports_v1_persistence_only --> e_pytest_tests_unit_v2_test_report_v1_runtime_selection_py

policyengine_api/data/v2/catalog/__init__.py

@@ -1,15 +1,7 @@
-"""PolicyEngine.py-derived v2 reference catalog extraction and publication."""
+"""PolicyEngine.py-derived v2 reference catalog extraction and publication.
 
-from policyengine_api.data.v2.catalog.extraction import (
-    CatalogExtractionError,
-    extract_catalog,
-    extract_installed_catalog,
-)
-from policyengine_api.data.v2.catalog.records import NormalizedCatalog
-
-__all__ = [
-    "CatalogExtractionError",
-    "NormalizedCatalog",
-    "extract_catalog",
-    "extract_installed_catalog",
-]
+This package imports nothing when it loads, so importing one of its modules
+(such as ``catalog_selection_validation``) loads only what that module
+imports. Import names from their modules, for example
+``policyengine_api.data.v2.catalog.extraction.extract_catalog``.
+"""

policyengine_api/data/v2/catalog/catalog_selection_reads.py

@@ -0,0 +1,76 @@
+"""The database read that selects one v2 catalog by PolicyEngine.py version.
+
+Version validation, the error types and ``SelectedCatalog`` are in
+``catalog_selection_validation``, which loads no database code when imported.
+"""
+
+from __future__ import annotations
+
+from sqlalchemy.exc import SQLAlchemyError
+from sqlmodel import Session, select
+
+from policyengine_api.data.v2.catalog.catalog_selection_validation import (
+    SUPPORTED_V2_COUNTRY_IDS,
+    MetadataCatalogNotPublishedError,
+    MetadataCatalogUnavailableError,
+    MetadataCatalogVersionNotFoundError,
+    SelectedCatalog,
+    UnsupportedPreviewCountryError,
+    validate_policyengine_version,
+)
+from policyengine_api.data.v2.models import (
+    TaxBenefitModel,
+    TaxBenefitModelVersion,
+)
+
+
+def select_catalog(
+    session: Session,
+    *,
+    country_id: str,
+    running_policyengine_version: str,
+    policyengine_version: str | None = None,
+) -> SelectedCatalog:
+    """Select one country catalog using the requested or running package version."""
+    if country_id not in SUPPORTED_V2_COUNTRY_IDS:
+        raise UnsupportedPreviewCountryError(country_id)
+    explicit_version = policyengine_version is not None
+    selected_version = (
+        validate_policyengine_version(policyengine_version)
+        if explicit_version
+        else running_policyengine_version
+    )
+    try:
+        row = session.exec(
+            select(TaxBenefitModel, TaxBenefitModelVersion)
+            .join(
+                TaxBenefitModelVersion,
+                TaxBenefitModelVersion.model_id == TaxBenefitModel.id,
+            )
+            .where(
+                TaxBenefitModel.name == f"policyengine-{country_id}",
+                TaxBenefitModelVersion.version == selected_version,
+            )
+        ).one_or_none()
+    except SQLAlchemyError as error:
+        msg = "the v2 metadata catalog cannot be queried"
+        raise MetadataCatalogUnavailableError(msg) from error
+
+    if row is None:
+        if explicit_version:
+            msg = (
+                f"PolicyEngine.py {selected_version} is not published for {country_id}"
+            )
+            raise MetadataCatalogVersionNotFoundError(msg)
+        msg = (
+            f"the running PolicyEngine.py {selected_version} catalog "
+            f"is absent for {country_id}"
+        )
+        raise MetadataCatalogNotPublishedError(msg)
+    model, model_version = row
+    return SelectedCatalog(
+        country_id=country_id,
+        policyengine_version=selected_version,
+        model=model,
+        model_version=model_version,
+    )

policyengine_api/data/v2/catalog/catalog_selection.py → policyengine_api/data/v2/catalog/catalog_selection_validation.py

@@ -1,19 +1,26 @@
-"""PolicyEngine.py catalog version selection for v2 resource reads."""
+"""Version validation, error types and the result type for v2 catalog selection.
+
+Importing this module loads no SQLAlchemy, SQLModel, table model or database
+module (rule ``api.catalog.selection-validation-no-database``), so v2
+validators, transformations, types and routes can use it. The one database
+read, ``select_catalog``, is in ``catalog_selection_reads``.
+"""
 
 from __future__ import annotations
 
 from dataclasses import dataclass
+from typing import TYPE_CHECKING
 
 from packaging.version import InvalidVersion, Version
-from sqlalchemy.exc import SQLAlchemyError
-from sqlmodel import Session, select
 
-from policyengine_api.data.v2.models import (
-    TaxBenefitModel,
-    TaxBenefitModelVersion,
-)
 from policyengine_api.supported_countries import SUPPORTED_COUNTRY_IDS
 
+if TYPE_CHECKING:
+    from policyengine_api.data.v2.models import (
+        TaxBenefitModel,
+        TaxBenefitModelVersion,
+    )
+
 SUPPORTED_V2_COUNTRY_IDS = frozenset(SUPPORTED_COUNTRY_IDS)
 MAXIMUM_POLICYENGINE_VERSION_LENGTH = 128
 
@@ -72,55 +79,3 @@ def validate_policyengine_version(value: str) -> str:
         msg = "policyengine_version must be a canonical non-placeholder version"
         raise InvalidPolicyEngineVersionError(msg)
     return value
-
-
-def select_catalog(
-    session: Session,
-    *,
-    country_id: str,
-    running_policyengine_version: str,
-    policyengine_version: str | None = None,
-) -> SelectedCatalog:
-    """Select one country catalog using the requested or running package version."""
-    if country_id not in SUPPORTED_V2_COUNTRY_IDS:
-        raise UnsupportedPreviewCountryError(country_id)
-    explicit_version = policyengine_version is not None
-    selected_version = (
-        validate_policyengine_version(policyengine_version)
-        if explicit_version
-        else running_policyengine_version
-    )
-    try:
-        row = session.exec(
-            select(TaxBenefitModel, TaxBenefitModelVersion)
-            .join(
-                TaxBenefitModelVersion,
-                TaxBenefitModelVersion.model_id == TaxBenefitModel.id,
-            )
-            .where(
-                TaxBenefitModel.name == f"policyengine-{country_id}",
-                TaxBenefitModelVersion.version == selected_version,
-            )
-        ).one_or_none()
-    except SQLAlchemyError as error:
-        msg = "the v2 metadata catalog cannot be queried"
-        raise MetadataCatalogUnavailableError(msg) from error
-
-    if row is None:
-        if explicit_version:
-            msg = (
-                f"PolicyEngine.py {selected_version} is not published for {country_id}"
-            )
-            raise MetadataCatalogVersionNotFoundError(msg)
-        msg = (
-            f"the running PolicyEngine.py {selected_version} catalog "
-            f"is absent for {country_id}"
-        )
-        raise MetadataCatalogNotPublishedError(msg)
-    model, model_version = row
-    return SelectedCatalog(
-        country_id=country_id,
-        policyengine_version=selected_version,
-        model=model,
-        model_version=model_version,
-    )

policyengine_api/fastapi_routes/v2/metadata/common.py

@@ -11,7 +11,7 @@
 from starlette.responses import Response
 from starlette.status import HTTP_200_OK
 
-from policyengine_api.data.v2.catalog.catalog_selection import (
+from policyengine_api.data.v2.catalog.catalog_selection_validation import (
     InvalidPolicyEngineVersionError,
     MetadataCatalogUnavailableError,
     MetadataCatalogVersionNotFoundError,

policyengine_api/fastapi_routes/v2/policies/routes.py

@@ -9,7 +9,7 @@
 from sqlalchemy.exc import SQLAlchemyError
 from starlette.responses import JSONResponse
 
-from policyengine_api.data.v2.catalog.catalog_selection import (
+from policyengine_api.data.v2.catalog.catalog_selection_validation import (
     MetadataCatalogUnavailableError,
     MetadataCatalogVersionNotFoundError,
 )

policyengine_api/query_parameters.py

@@ -26,7 +26,7 @@
 )
 
 from policyengine_api.constants import LOCAL_AREA_BREAKDOWN_REGIONS
-from policyengine_api.data.v2.catalog.catalog_selection import (
+from policyengine_api.data.v2.catalog.catalog_selection_validation import (
     validate_policyengine_version,
 )
 from policyengine_api.services.economic_impact_request import LocalAreaBreakdown

policyengine_api/routes/household_routes.py

@@ -15,7 +15,9 @@
 
 from policyengine_api.constants import COUNTRY_PACKAGE_VERSIONS, POLICYENGINE_VERSION
 from policyengine_api.data.v1_models import Household
-from policyengine_api.data.v2.catalog.catalog_selection import SUPPORTED_V2_COUNTRY_IDS
+from policyengine_api.data.v2.catalog.catalog_selection_validation import (
+    SUPPORTED_V2_COUNTRY_IDS,
+)
 from policyengine_api.extensions import cache
 from policyengine_api.gcp_logging import logger
 from policyengine_api.migration_flags import (

policyengine_api/routes/policy_routes.py

@@ -7,7 +7,7 @@
 from werkzeug.exceptions import BadRequest, NotFound
 
 from policyengine_api.data.v1_models import Policy
-from policyengine_api.data.v2.catalog.catalog_selection import (
+from policyengine_api.data.v2.catalog.catalog_selection_validation import (
     SUPPORTED_V2_COUNTRY_IDS,
 )
 from policyengine_api.gcp_logging import logger

policyengine_api/services/policy_mirroring.py

@@ -10,7 +10,7 @@
 
 from sqlalchemy.exc import SQLAlchemyError
 
-from policyengine_api.data.v2.catalog.catalog_selection import (
+from policyengine_api.data.v2.catalog.catalog_selection_validation import (
     MetadataCatalogNotPublishedError,
     MetadataCatalogUnavailableError,
     MetadataCatalogVersionNotFoundError,

policyengine_api/services/v2/metadata/database_connectors/reads.py

@@ -6,10 +6,10 @@
 
 from sqlalchemy.exc import SQLAlchemyError
 
-from policyengine_api.data.v2.catalog.catalog_selection import (
+from policyengine_api.data.v2.catalog.catalog_selection_reads import select_catalog
+from policyengine_api.data.v2.catalog.catalog_selection_validation import (
     MetadataCatalogUnavailableError,
     SelectedCatalog,
-    select_catalog,
 )
 
 if TYPE_CHECKING:
@@ -25,7 +25,7 @@ def read_metadata_catalog(
 ) -> SelectedCatalog:
     """Return the country's catalog for policyengine_version, else the running version.
 
-    Raise the catalog_selection errors for an unsupported country, an
+    Raise the catalog_selection_validation errors for an unsupported country, an
     unpublished version or an unreachable database.
     """
     return select_catalog(

policyengine_api/services/v2/metadata/selected_catalogs.py

@@ -6,7 +6,9 @@
 from typing import TYPE_CHECKING
 
 if TYPE_CHECKING:
-    from policyengine_api.data.v2.catalog.catalog_selection import SelectedCatalog
+    from policyengine_api.data.v2.catalog.catalog_selection_validation import (
+        SelectedCatalog,
+    )
 
 CatalogSelectionKey = tuple[str, str | None]
 """A country ID and the requested PolicyEngine.py version.

policyengine_api/services/v2/metadata/services.py

@@ -4,7 +4,7 @@
 
 from typing import TYPE_CHECKING
 
-from policyengine_api.data.v2.catalog.catalog_selection import (
+from policyengine_api.data.v2.catalog.catalog_selection_validation import (
     SelectedCatalog,
     validate_policyengine_version,
 )

policyengine_api/services/v2/metadata/transformations.py

@@ -37,7 +37,9 @@
 )
 
 if TYPE_CHECKING:
-    from policyengine_api.data.v2.catalog.catalog_selection import SelectedCatalog
+    from policyengine_api.data.v2.catalog.catalog_selection_validation import (
+        SelectedCatalog,
+    )
     from policyengine_api.data.v2.models import (
         Dataset,
         Parameter,

policyengine_api/services/v2/metadata/validators.py

@@ -4,12 +4,14 @@
 
 from typing import TYPE_CHECKING, TypeVar
 
-from policyengine_api.data.v2.catalog.catalog_selection import (
+from policyengine_api.data.v2.catalog.catalog_selection_validation import (
     MetadataCatalogUnavailableError,
 )
 
 if TYPE_CHECKING:
-    from policyengine_api.data.v2.catalog.catalog_selection import SelectedCatalog
+    from policyengine_api.data.v2.catalog.catalog_selection_validation import (
+        SelectedCatalog,
+    )
     from policyengine_api.data.v2.models import Dataset, Region