Skip to content

Commit 13e1701

Browse files
committed
Run the suite in randomized order in CI and document the rule
A file order that happens to work is not evidence of isolation; only running with --randomize catches a test that leans on another file's side effects. Fixing the leaks this uncovered is only durable if the check keeps running, so add a CI step alongside the canonical run and write the underlying rule into AGENTS.md.
1 parent 5f96fdb commit 13e1701

2 files changed

Lines changed: 7 additions & 0 deletions

File tree

.github/workflows/ci.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,3 +41,9 @@ jobs:
4141
# inference package (vendor/) is out of scope for this repo's CI.
4242
- name: Test
4343
run: bun run test
44+
45+
# Catches tests that only pass because of the default file order (shared
46+
# module-level state, an unrestored global mock, a leaked env var). The
47+
# seed is fixed so a failure here reproduces locally with the same flag.
48+
- name: Test (randomized order)
49+
run: bun test ./src ./tests ./evals --randomize --seed 424242

AGENTS.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ When refactoring replaces an old path, delete the old one. No back-compat shims,
3333
- Add or update tests with every behavior change.
3434
- Bug fixes start with a failing test that reproduces the bug. Do not start by patching.
3535
- `tests/unit/` shared unit tests and helpers · co-located `src/**/*.test.ts` for module logic · `tests/fixtures/` fixture repos · `tests/integration/` reactor/permission harness. Planned: `tests/e2e/` (fixture-repo runs).
36+
- A test must not depend on another file having run, or on the default file order. It must pass under `bun test ./src ./tests ./evals --randomize`. If a test mutates module-level state or calls `mock.module`, it must restore that state itself (`afterEach`/`afterAll`), not rely on the process happening to reset it. When capturing a module's real exports to restore later, shallow-copy them at capture time (`{ ...(await import(path)) }`) — Bun mutates the live namespace object in place when the module is mocked, so holding a bare reference to it silently turns into the mocked exports.
3637

3738
## Build & Validation
3839

0 commit comments

Comments
 (0)