Adopt three-tier test layout with real-Postgres integration tier

.claude/rules/backend/code-conventions.md

@@ -56,8 +56,11 @@ Python/FastAPI conventions for the CortexDJ backend.
 
 ## Testing
 
-- Tests live flat under `backend/tests/` — fast, DB-free, provider-free. `conftest.py` sets a dummy `OPENAI_API_KEY` so importing `brain_agent` never needs a real key; tests that invoke the agent use `agent.override` with pydantic-ai `TestModel`.
-- Real-LLM evals live in `tests/evals/`, marked `@pytest.mark.eval` and opt-in via `pytest -m eval`; the default run excludes them through `addopts = "-m 'not eval'"`.
+- Three tiers under `backend/tests/`: `unit/<area>/` (fast, DB-free, provider-free — the default `pytest` run), `integration/` (real Postgres + HTTP, opt-in via `-m integration`), and `evals/` (real-LLM, `@pytest.mark.eval`, opt-in via `-m eval`). The default run excludes both opt-in tiers through `addopts = "-m 'not eval and not integration'"`; test dirs are packages so basenames can repeat across tiers.
+- Async tests use anyio's pytest plugin: `@pytest.mark.anyio` plus the `anyio_backend` fixture in the root conftest. No pytest-asyncio, no raw `asyncio.run()` wrappers.
+- The integration tier builds schema from the shipped Alembic migrations and rolls back an outer transaction per test. Isolation holds only because the app's sole commit point is the session dependency (`dependencies/db.py`) — services and model classmethods `flush()`, never `commit()`. The tier refuses database names not ending in `_test` and applies its markers structurally in `tests/integration/conftest.py` (don't add per-file `pytestmark`). Local setup: `backend/scripts/create-test-db.sh`, then `POSTGRES_DB=cortexdj_test uv run --directory backend pytest -m integration`.
+- Shared builders live in `tests/factories.py` (`make_*` = unpersisted, `create_*` = persisted via flush); agent-deps mocks in `tests/fakes.py`.
+- Root `conftest.py` sets a dummy `OPENAI_API_KEY` so importing `brain_agent` never needs a real key; tests that invoke the agent use `agent.override` with pydantic-ai `TestModel`.
 - CI's test job enforces the coverage floor; don't run coverage locally unless you're investigating a CI failure.
 
 ## Dependencies

.github/workflows/ci.yml

@@ -57,11 +57,28 @@ jobs:
   test:
     name: Test
     runs-on: ubuntu-latest
+    services:
+      # Same image as local docker compose; backs the integration tier.
+      postgres:
+        image: pgvector/pgvector:pg17
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: postgres
+          POSTGRES_DB: cortexdj_test
+        ports:
+          - "5432:5432"
+        options: >-
+          --health-cmd "pg_isready -U postgres"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
     env:
-      # Settings require these at import; the tests never connect to a DB.
+      # The integration tier connects to the service container above and
+      # refuses to run unless the DB name contains "test" (it runs alembic
+      # upgrade/downgrade). The unit tier stays DB-free.
       POSTGRES_USER: postgres
       POSTGRES_PASSWORD: postgres
-      POSTGRES_DB: cortexdj
+      POSTGRES_DB: cortexdj_test
       POSTGRES_HOST: localhost
       POSTGRES_PORT: "5432"
       # Dummy provider key: tests use TestModel/monkeypatch, never a real call.
@@ -85,7 +102,12 @@ jobs:
 
       - name: Run tests + coverage
         working-directory: ./backend
-        # Coverage floor at 40% (baseline ~43%); ratchet up as coverage grows.
+        # Coverage floor at 40% (unit tier alone measures ~43%; combined with
+        # the integration tier, ~53%); ratchet up as coverage grows.
+        # The explicit -m "not eval" OVERRIDES addopts' 'not integration'
+        # (the last -m wins), which is what selects the integration tier
+        # here — don't remove it as "redundant" or the tier silently
+        # deselects while CI stays green.
         run: uv run pytest -m "not eval" --cov=cortexdj --cov-report=term-missing --cov-report=xml --cov-fail-under=40
 
       - name: Upload coverage report

.gitignore

@@ -65,6 +65,7 @@ Thumbs.db
 
 # Testing
 .coverage
+coverage.xml
 htmlcov/
 .pytest_cache/
 .hypothesis/

DEVELOPMENT.md

@@ -47,14 +47,27 @@ uv run --directory backend pre-commit run --all-files
 
 ## Tests
 
+The suite has three tiers under `backend/tests/`:
+
+- **`unit/<area>/`** — fast, DB-free, provider-free. This is the default `pytest` run.
+- **`integration/`** — real Postgres + HTTP (`httpx` against the app). Schema comes from the shipped Alembic migrations; each test rolls back an outer transaction, so tests never pollute each other. Opt-in via `-m integration`.
+- **`evals/`** — real-LLM `brain_agent` evals. Opt-in via `-m eval`; use as a nightly safety net on `main` or manual spot-checks, not on every PR. See `.claude/rules/backend/pydantic-ai.md`.
+
 ```bash
-uv run --directory backend pytest              # unit tests (eval suite excluded)
+uv run --directory backend pytest              # unit tier (integration + eval excluded)
 uv run --directory backend pytest -v           # verbose output
-uv run --directory backend pytest tests/test_preprocessing.py  # single file
+uv run --directory backend pytest tests/unit/ml/test_preprocessing.py  # single file
 uv run --directory backend pytest -m eval      # real-model brain_agent eval suite (opt-in)
 ```
 
-The `eval` marker gates tests that call the real OpenAI API via `brain_agent` — the default `pytest` invocation excludes them via `addopts = "-m 'not eval'"`. Use them as a nightly safety net on `main` or manual spot-checks, not on every PR. See `.claude/rules/backend/pydantic-ai.md` and `backend/tests/evals/` for the suite layout.
+To run the integration tier locally, create the test database once, then point `POSTGRES_DB` at it (the env var overrides the `.env` value; host/port still come from `.env`):
+
+```bash
+./backend/scripts/create-test-db.sh
+POSTGRES_DB=cortexdj_test uv run --directory backend pytest -m integration
+```
+
+The tier refuses to run unless the database name ends with `_test`, because it runs `alembic upgrade`/`downgrade` against whatever database the env points at. Shared test-data builders live in `backend/tests/factories.py` (`make_*` = unpersisted, `create_*` = persisted via flush).
 
 ## Database migrations
 

backend/pyproject.toml

@@ -34,6 +34,7 @@ dependencies = [
 
 [dependency-groups]
 dev = [
+  "anyio>=4.14.2",
   "mypy>=2.3.0",
   "pre-commit>=4.6.0",
   "pydantic-evals>=2.11.0",
@@ -43,12 +44,15 @@ dev = [
 ]
 
 [tool.pytest.ini_options]
+testpaths = ["tests"]
 markers = [
-    "main: marks tests for main demo examples",
-    "additional: marks tests for additional demo examples",
+    "integration: real-DB/HTTP tests; needs Postgres (CI service or `docker compose up -d postgres`)",
     "eval: marks tests that call a real LLM (opt-in; run with `pytest -m eval`)",
 ]
-addopts = "-m 'not eval'"
+# Default run = the fast, DB-free unit tier; `integration` and `eval` are
+# opt-in via -m. Test dirs are packages (an __init__.py each) so the same
+# basename can recur across tiers.
+addopts = "-m 'not eval and not integration'"
 
 [tool.mypy]
 strict = true

backend/scripts/create-test-db.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+# Create the integration-test database (idempotent). The integration tier
+# refuses to run unless POSTGRES_DB ends with "_test", because it runs
+# `alembic upgrade`/`downgrade` against whatever database the env points at.
+set -euo pipefail
+
+docker compose up -d postgres
+
+echo "Waiting for Postgres to become ready..."
+until docker compose exec postgres pg_isready -U postgres -q; do
+  sleep 1
+done
+
+docker compose exec postgres psql -U postgres -tc \
+  "SELECT 1 FROM pg_database WHERE datname = 'cortexdj_test'" | grep -q 1 ||
+  docker compose exec postgres createdb -U postgres cortexdj_test
+
+echo "cortexdj_test is ready. Run the tier with:"
+echo "  POSTGRES_DB=cortexdj_test uv run --directory backend pytest -m integration"

backend/src/cortexdj/dependencies/db.py

@@ -35,6 +35,14 @@ def get_postgres_url(prefix: str) -> str:
 
 
 async def _get_async_sqlalchemy_session_dependency() -> AsyncGenerator[AsyncSession, None]:
+    """Yield a session that owns its transaction: commit on success, rollback on error.
+
+    This is the single commit point for request-scoped sessions (and for
+    background work via ``get_async_sqlalchemy_session``). Services and model
+    classmethods must only ``flush()``, never ``commit()`` — the integration
+    test tier overrides this dependency and rolls back an outer transaction
+    per test, which only isolates tests if no other code commits.
+    """
     async with AsyncSessionMaker() as session:
         try:
             yield session

backend/src/cortexdj/migrations/env.py

@@ -9,7 +9,11 @@
 config.set_main_option("sqlalchemy.url", get_postgres_url("postgresql+psycopg"))
 
 if config.config_file_name is not None:
-    fileConfig(config.config_file_name)
+    # Keep existing loggers alive: migrations also run in-process (the
+    # integration test tier calls `command.upgrade`), and fileConfig's
+    # default disable_existing_loggers=True would silence every logger
+    # created before this point.
+    fileConfig(config.config_file_name, disable_existing_loggers=False)
 
 from cortexdj.models import Base  # noqa: F401, E402
 

backend/src/cortexdj/scripts/build_track_index.py

@@ -209,6 +209,8 @@ async def _main_async() -> int:
 
     MISS_LOG_PATH.unlink(missing_ok=True)
 
+    # Raw script-owned session (not the commit-owning dependency), so the
+    # explicit commits below are intentional.
     async with AsyncSessionMaker() as db:
         from cortexdj.services.spotify import get_user_spotify_client
 

backend/src/cortexdj/scripts/seed_sessions.py

@@ -193,8 +193,6 @@ async def seed_participant(
 
             seeded += 1
 
-        await db.commit()
-
     return seeded
 
 

backend/src/cortexdj/services/title_generator.py

@@ -51,7 +51,6 @@ async def generate_thread_title(
             title = title.strip().strip('"').strip("'")
             async with get_async_sqlalchemy_session() as db:
                 await Thread.update_title(db, thread_id, agent_type, title)
-                await db.commit()
             logger.info(f"Generated title for thread {thread_id}: {title}")
 
     except Exception:
@@ -60,7 +59,6 @@ async def generate_thread_title(
         try:
             async with get_async_sqlalchemy_session() as db:
                 await Thread.update_title(db, thread_id, agent_type, fallback)
-                await db.commit()
             logger.info(f"Used fallback title for thread {thread_id}: {fallback}")
         except Exception:
             logger.exception(f"Failed to save fallback title for thread {thread_id}")

backend/tests/conftest.py

@@ -1,8 +1,20 @@
 import os
 
+import pytest
+
 # ``OpenAIResponsesModel`` constructs its provider client eagerly at module
 # import time, which requires ``OPENAI_API_KEY``. Any test that imports
 # ``brain_agent`` — directly or transitively — would otherwise fail at
 # collection. Tests that actually invoke the model use ``agent.override``
 # with ``TestModel``; this dummy value never reaches a real API call.
 os.environ.setdefault("OPENAI_API_KEY", "sk-test-deterministic-no-real-calls")
+
+# Deliberately no POSTGRES_* defaults here: settings load from the repo-root
+# ``.env`` (port 5433), and real env vars take precedence over dotenv — a
+# setdefault would silently override ``.env`` and break local integration runs.
+
+
+@pytest.fixture
+def anyio_backend() -> str:
+    """Run ``@pytest.mark.anyio`` async tests on the asyncio backend."""
+    return "asyncio"

backend/tests/evals/test_brain_agent_evals.py

@@ -21,15 +21,14 @@
 agent is allowed to chain tools as long as the critical one fires.
 """
 
-import asyncio
 from dataclasses import dataclass, field
 
 import pytest
 from pydantic_evals import Case, Dataset
 from pydantic_evals.evaluators import Evaluator, EvaluatorContext
 
 from cortexdj.agents.brain_agent import brain_agent
-from tests.evals.conftest import fake_spotify_client, make_fake_deps
+from tests.fakes import fake_spotify_client, make_fake_deps
 
 
 @dataclass
@@ -159,8 +158,9 @@ def evaluate(self, ctx: EvaluatorContext[BrainAgentInput, BrainAgentOutput, None
 
 
 @pytest.mark.eval
-def test_brain_agent_tool_routing() -> None:
-    report = asyncio.run(_dataset.evaluate(_run_brain_agent))
+@pytest.mark.anyio
+async def test_brain_agent_tool_routing() -> None:
+    report = await _dataset.evaluate(_run_brain_agent)
 
     execution_failures = [f.name for f in report.failures]
     assert not execution_failures, f"Eval execution errors: {execution_failures}"

backend/tests/factories.py

@@ -0,0 +1,106 @@
+"""Test-data builders shared across test tiers.
+
+``make_*`` returns an unpersisted ORM instance (unit tier). ``create_*``
+persists via the given session — add, flush, refresh — for the integration
+tier, where the surrounding fixture rolls an outer transaction back after
+each test. Builders flush and never commit, same as services: the session's
+owner holds the only commit.
+"""
+
+from datetime import UTC, datetime
+from typing import Any
+from uuid import uuid4
+
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from cortexdj.models.eeg_segment import EegSegment
+from cortexdj.models.session import Session
+from cortexdj.models.thread import Thread
+from cortexdj.models.track_audio_embedding import EMBEDDING_DIM, TrackAudioEmbedding
+from cortexdj.schemas.agent_type import AgentType
+
+
+def make_session(**overrides: Any) -> Session:
+    data: dict[str, Any] = {
+        "id": str(uuid4()),
+        "participant_id": "P01",
+        "dataset_source": "deap",
+        "recorded_at": datetime(2024, 1, 1, tzinfo=UTC),
+        "duration_seconds": 60.0,
+    }
+    data.update(overrides)
+    return Session(**data)
+
+
+async def create_session(db: AsyncSession, **overrides: Any) -> Session:
+    session = make_session(**overrides)
+    db.add(session)
+    await db.flush()
+    await db.refresh(session)
+    return session
+
+
+def make_eeg_segment(session_id: str, **overrides: Any) -> EegSegment:
+    data: dict[str, Any] = {
+        "id": str(uuid4()),
+        "session_id": session_id,
+        "segment_index": 0,
+        "start_time": 0.0,
+        "end_time": 4.0,
+        "arousal_score": 0.7,
+        "valence_score": 0.6,
+        "dominant_state": "excited",
+        "band_powers": {"alpha": 1.0, "beta": 0.5, "theta": 0.3, "delta": 0.2, "gamma": 0.1},
+    }
+    data.update(overrides)
+    return EegSegment(**data)
+
+
+async def create_eeg_segment(db: AsyncSession, session_id: str, **overrides: Any) -> EegSegment:
+    segment = make_eeg_segment(session_id, **overrides)
+    db.add(segment)
+    await db.flush()
+    await db.refresh(segment)
+    return segment
+
+
+def make_track_audio_embedding(**overrides: Any) -> TrackAudioEmbedding:
+    # Default is a unit vector, not zeros — cosine distance against a zero
+    # vector is undefined, which would poison similarity-ordering tests.
+    embedding = [0.0] * EMBEDDING_DIM
+    embedding[0] = 1.0
+    data: dict[str, Any] = {
+        "spotify_id": f"sp-{uuid4().hex[:16]}",
+        "title": "Test Track",
+        "artist": "Test Artist",
+        "source": "user_library",
+        "embedding": embedding,
+    }
+    data.update(overrides)
+    return TrackAudioEmbedding(**data)
+
+
+async def create_track_audio_embedding(db: AsyncSession, **overrides: Any) -> TrackAudioEmbedding:
+    row = make_track_audio_embedding(**overrides)
+    db.add(row)
+    await db.flush()
+    await db.refresh(row)
+    return row
+
+
+def make_thread(**overrides: Any) -> Thread:
+    data: dict[str, Any] = {
+        "thread_id": f"t-{uuid4().hex[:8]}",
+        "agent_type": AgentType.CHAT.value,
+        "title": None,
+    }
+    data.update(overrides)
+    return Thread(**data)
+
+
+async def create_thread(db: AsyncSession, **overrides: Any) -> Thread:
+    thread = make_thread(**overrides)
+    db.add(thread)
+    await db.flush()
+    await db.refresh(thread)
+    return thread

backend/tests/evals/conftest.py → backend/tests/fakes.py

@@ -1,9 +1,8 @@
-"""Fixtures for brain_agent evals.
+"""Fake dependency builders for agent tests (unit tier and evals).
 
-Provides ``make_fake_deps`` — a constructor for ``AgentDeps`` instances
-that don't hit the real database, Spotify, or EEG model. Used by both
-the deterministic ``prepare_tools`` tests (TestModel-backed) and the
-real-model ``@pytest.mark.eval`` tests.
+``make_fake_deps`` constructs ``AgentDeps`` that never hit the real
+database, Spotify, or EEG model. Kept separate from ``factories.py``:
+these build mocks around agent wiring, not persistable ORM rows.
 """
 
 from typing import Any