Skip to content

Celery worker entrypoint doesn't load .env; connects to SQLite with module schemas #94

Description

@antosubash

Summary

The documented worker command — uv run celery -A scripts.run_worker:celery worker -l info — doesn't load host/.env, so Settings() falls back to the SQLite default and the model __table_args__ keep their per-module schema qualifiers (datasets.datasets_dataset etc.). When a task tries to execute against a real Postgres database (because the broker is reachable but the worker's settings disagree with the web process), every query fails with either:

  • sqlite3.OperationalError: unknown function: now() (when the worker is actually using SQLite), or
  • psycopg2.errors.UndefinedTable: relation "datasets.datasets_dataset" does not exist (when the worker is wired to Postgres but the model still has the schema-qualified table name from _default_provider).

Repro

  1. Set SM_DATABASE_URL=postgresql+asyncpg://... in host/.env (which uvicorn picks up automatically through pydantic-settings via cwd).
  2. Start the worker the way host/Makefile's `worker` target does: `uv run celery -A scripts.run_worker:celery worker -l info`.
  3. Enqueue a real task (e.g. `lacowiki_datasets.convert_dataset`).
  4. Worker fails to write `background_tasks_task_execution` rows because either the env var is missing or the schema provider doesn't match the dev migration.

The dev-only helper host/scripts/reset_admin_dev.py works around this with:

import simple_module_db.base as _smdb_base
_smdb_base._default_provider = lambda: _smdb_base.DatabaseProvider.SQLITE

…which is itself a sign the seam is too sharp for normal app code.

Expected

scripts/run_worker.py (or the build_celery factory it calls) should:

  1. Load .env from the host directory before instantiating BackgroundTasksSettings/Settings.
  2. Apply the same provider/schema convention the web process applies, so model __table_args__ line up with the actual database the worker reaches.

Workaround

Wrap the worker entrypoint:

# scripts/run_worker.py
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(Path(__file__).resolve().parent.parent / ".env")

import simple_module_db.base as _smdb_base
_smdb_base._default_provider = lambda: _smdb_base.DatabaseProvider.SQLITE  # match dev migrations

from background_tasks.celery_app import build_celery
from background_tasks.settings import BackgroundTasksSettings
celery = build_celery(BackgroundTasksSettings())

Acceptance

  • A fresh sm new --preset full host's worker command runs against the same database the web process uses, with no extra wiring in scripts/.

Activity

  1. antosubash commented on May 1, 2026

    @antosubash
    OwnerAuthor

    Closing — addressed by commit ade3e24 (PR #98). scripts/run_worker.py now calls load_dotenv_into_environ(host/.env) before instantiating settings, and schema-per-module is driven by SM_SCHEMA_PER_MODULE (see also #68) so the worker and web process stay in lockstep against the same Postgres DB.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions