perf(api): Stop replaying migrations, and give tests their own database

[khvn26]

Thanks for submitting a PR! Please check the boxes below:

• I have read the Contributing Guide.
• I have added information to docs/ if required so people know about the feature.
• I have filled in the "Changes" section below.
• I have filled in the "How did you test this code" section below.

Changes

Creating a test database runs the whole ~600 migration history, and pytest-django does it once per xdist worker, so a cold ten-worker run paid for it ten times. Every migration test then replayed the history from zero to reach the state it tests, and migrated forward again on teardown — the 128 of them accounted for half the suite's CPU on their own, from 2.6% of the tests. ClickHouse was the same story from a different angle: Django replays all ~600 migrations against it to build the migration state for the three tables it owns, 22 seconds of it, on every session — including a run of one Postgres-only test. Profiling that showed 28s in render_multiple and no SQL statement over 0.2ms.

PostgreSQL copies a database in about a tenth of a second, so api/tests/migration_snapshots.py caches each migration state as a template database and clones it instead. States are keyed on the length of the migration plan prefix they correspond to, which makes them nest: reaching a deeper state clones the nearest cached ancestor and replays only the migrations in between, so a session converges on the cost of one migration run however many migration tests it has. Template names embed a digest of the migration files, so editing a migration — or switching to a branch with different ones — builds a new template rather than handing anyone a stale schema. That is --reuse-db without the footgun, and it makes the --ci database bootstrap unnecessary. ClickHouse gets the same treatment through the copy its backend already knows how to do.

Test data is throwaway, which makes a Docker volume the wrong place for it: on macOS its I/O is virtualised, and it made the runtime a function of how busy the host's disk was — 258s against 137s on identical code. Tests now run against test-db, a separate RAM-backed server tuned for a workload that clones databases and never needs to survive a crash. Nothing in it is worth keeping, so docker compose restart test-db is always a safe reset, and an empty server just rebuilds its templates. The db service make serve uses is left alone, with its durability intact.

The rest is smaller change of the same character:

• The default password hasher deliberately costs 345ms a call, and fixtures like admin_user hash on nearly every test. Tests use the cheapest hasher instead; nothing depends on the algorithm.
• Coverage roughly doubles the runtime and was on by default. It is now make test-coverage, which CI uses.
• -vvvv in addopts spent seconds formatting a line per test — 8s on collection alone — and buried failures in the noise.
• make test recreated every container and booted Django twice just to check the databases were up, about 12s before pytest started. It now probes the sockets and only calls Compose when something is actually down.
• A bare make test still fans out with xdist; make test opts=... is taken to be a focused run and stays in-process, because each worker costs a Python start, a Django setup and a database clone.

	before	after
Full suite	1241s	~155s
Focused run, p95	~24s of fixed cost before a test ran	5.4s

4937 passed, 23 skipped, unchanged from before.

One thing this surfaces

Three enqueue_membership_refresh tests in tests/unit/segment_membership are latently flaky, roughly one run in five. get_tasks_to_process filters on scheduled_for < NOW(), but delay() stamps scheduled_for from the client's clock, and the test database's clock runs about 0.08ms behind the host's — so a task enqueued and polled in the same breath is briefly invisible to the poller. The suite used to be slow enough that this never mattered. Nothing here causes it and I have deliberately not papered over it; the durable fix belongs in flagsmith-common, where the database should stamp scheduled_for rather than the client.

Left on the table

Each ClickHouse test currently drags Postgres down to TransactionTestCase semantics, because Django only wraps a test in transactions when every database it declares supports them. That costs a full flush and a post_migrate rebuild of every permission, about 2.5s a test and ~60s of the suite. fake_transaction on the ClickHouse alias fixes it, but it also changes on_commit behaviour and breaks tests that count queued tasks, so it is a behaviour change rather than an optimisation and wants its own PR.

How did you test this code?

The suite is the test: same 4937 passed, 23 skipped as before, over six consecutive full runs plus several from a cold (empty) test server, which exercises the template-building path rather than the clone path.

To check the caching is honest rather than just fast, migration tests were run both with templates already built and against a freshly restarted test-db, and -k migrat was confirmed to still select all 128 tests — the migrator_factory override keeps the fixture's name so django_test_migrations still marks them.

Timings were taken on an otherwise idle machine, against origin/main as the control under the same conditions. Focused-run figures are the wall time of make test opts='<file>' over 40 randomly sampled test files, and separately over 20 hand-picked selectors chosen to include the awkward cases (a single test, a migration test, a ClickHouse-heavy directory, a bare -k expression).

To sanity-check the new database:

make test                                  # whole suite, fans out
make test opts='tests/unit/segments -q'    # focused, in-process
docker compose restart test-db && make test  # cold: rebuilds every template

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
docs	Ignored	Preview	Aug 26, 2026 2:52pm
flagsmith-frontend-preview	Ignored	Preview	Aug 26, 2026 2:52pm
flagsmith-frontend-staging	Ignored	Preview	Aug 26, 2026 2:52pm

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.64%. Comparing base (bed0b2f) to head (a7061ce).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8378      +/-   ##
==========================================
- Coverage   98.80%   98.64%   -0.17%     
==========================================
  Files        1619     1622       +3     
  Lines       65682    66082     +400     
==========================================
+ Hits        64898    65187     +289     
- Misses        784      895     +111     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.