Skip to content

About

Diagnose order-dependent Vitest failures with Redis state evidence, sequence reduction, and controlled replay.

Topics

Resources

Contributing

Security policy

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

Test Autopsy: local Vitest and Redis test pollution diagnosis. Difficult failures, readable evidence.

A passing test. A failing suite. An explanation you can replay.

Current CI status Node.js 22.12 or later Vitest 4.1.11 MIT license

Quick start · Demo evidence · How it works · Verification · Documentation · Türkçe

Test Autopsy investigates order-dependent Vitest failures and the Redis state left behind by earlier tests. It reduces a known failing file order, records state changes and tracked write call sites, then tests whether restoring suspect keys changes the victim's outcome.

v0.1.0 · Experimental developer preview. Built for one Vitest Node project, serial test files, ioredis, and a dedicated Redis namespace. The evidence below comes from automated tests and a planted fixture; independent real-project adoption and time savings have not been established.

Why Test Autopsy?

A login test passes alone and fails after a user-blocking test. The assertion tells you what failed. It does not tell you which earlier file left the blocking state behind.

Test Autopsy connects a reduced, verified file order, the observed Redis changes, and a controlled restore experiment in one inspectable report. When verified, minimality means that removing any one retained predecessor makes the victim pass; it is not a global minimum.

Qualify the failure

Require the victim to pass alone and fail with the same fingerprint in the supplied order. Keep unresolved execution separate.

Understand the evidence →

Reduce the sequence

Remove unnecessary predecessor files while preserving their order and the victim failure. Report the limits of minimality.

Inspect the reduction method →

Trace and intervene

Connect changed Redis keys with tracked write call sites. Restore candidate keys before the victim file and compare outcomes.

Connect your Redis client →

Inspect and replay

Keep readable reports and a replay manifest containing the reduced order, victim identity, Redis prefix, and test-file hashes.

Explore commands and artifacts →

Predecessor files retain their tests. In the victim file, only the selected test runs with its applicable setup and hooks; other test cases are skipped. The reducer operates on files.

The captured example

Captured planted fixture: the order shrinks from three files to polluter plus victim. One candidate Redis key, fourteen trials, and two of two passing restore-intervention trials.

The included fixture leaves autopsy:demo:blocked:user:42 behind. The recorded investigation retains the polluter and victim files; restoring that key makes both intervention trials pass. These are results from this planted fixture, not a real-world success-rate benchmark.

Read the captured evidence · Inspect the example tests · Understand the evaluation limits

Quick start

Use Node.js 22.12+ and npm 11+; CI uses npm 11.17.0. If your Node installation bundles npm 10, follow the npm upgrade guidance. The documented distribution is this source repository.

git clone https://github.com/manyZXZ/test-autopsy.git
cd test-autopsy
npm ci
npm run build
docker compose up -d

The Compose service starts a disposable Redis on port 16379. Set the endpoint and run the planted example:

macOS / Linux
export AUTOPSY_REDIS_URL=redis://127.0.0.1:16379
npm run demo
Windows PowerShell
$env:AUTOPSY_REDIS_URL = 'redis://127.0.0.1:16379'
npm run demo

The demo verifies the reduced sequence, restore intervention, and restored baseline. Open examples/redis-pollution/.test-autopsy/report.html to inspect the result. Run docker compose down when finished. If Docker is unavailable, use your own dedicated Redis and set its actual endpoint.

Integrate your own project — client setup, configuration, diagnosis, and replay

2. Prepare an isolated Redis instance

Point both your tests and Test Autopsy at the same dedicated test Redis. Give this investigation its own namespace, such as autopsy:myapp:. No other process should read or write that namespace while a diagnosis is running.

POSIX shell:

export AUTOPSY_REDIS_URL=redis://127.0.0.1:6379/15

PowerShell:

$env:AUTOPSY_REDIS_URL = 'redis://127.0.0.1:6379/15'

The database number is only an example. A dedicated instance is preferable to sharing a database with unrelated workloads. The prefix is the actual snapshot and restore boundary.

Initialize any expected baseline fixtures before starting Test Autopsy. The tool snapshots existing state; it does not know which values your application considers correct.

3. Install the built package in the project being investigated

From the Test Autopsy repository:

npm pack

Then, from your application repository, install the generated archive using its actual path:

npm install --save-dev /absolute/path/to/manyzxz-test-autopsy-0.1.0.tgz

Use a path appropriate to your operating system. Keep the resulting dependency change local until you decide to adopt the tool in that project.

4. Track your test Redis client

Wrap the ioredis client used by your application tests:

import Redis from 'ioredis';
import { trackRedis } from '@manyzxz/test-autopsy/vitest';

export const redis = trackRedis(
  new Redis(process.env.AUTOPSY_REDIS_URL!),
);

Your application must use the same client instance. Wrapping an unused client cannot attribute writes made through a different connection. The instrumentation records commands, keys, and source stacks; it does not log command values. The CLI injects its Vitest runner while preserving your setup files; no additional setup hook is required.

State snapshots can still show a changed key when a write came from an untracked client, but that change has no reliable tracked write call site. See integration and attribution.

5. Describe the failing order

Create autopsy.config.json in your application repository:

{
  "root": ".",
  "vitestConfig": "vitest.config.ts",
  "files": [
    "test/harmless.test.ts",
    "test/user-block.test.ts",
    "test/login.test.ts"
  ],
  "victim": {
    "file": "test/login.test.ts",
    "name": "authentication > permits an active user"
  },
  "redis": {
    "urlEnv": "AUTOPSY_REDIS_URL",
    "keyPrefix": "autopsy:myapp:",
    "allowRemote": false,
    "maxKeys": 1000,
    "maxBytes": 10485760
  },
  "repetitions": 2,
  "maxTrials": 60,
  "timeoutMs": 30000,
  "outputDir": ".test-autopsy"
}

Replace file paths and the full victim name with values from your own suite. This is an explicit sequence, not a glob search. The victim file belongs at the end of the sequence. The numbers above are example settings, not performance guarantees.

6. Run the investigation

From the application repository:

npx --no-install test-autopsy doctor --config autopsy.config.json
npx --no-install test-autopsy diagnose --config autopsy.config.json

doctor validates the configured environment. diagnose executes trusted project code repeatedly and restores only the configured Redis scope between trials.

A confirmed diagnosis exits with code 1, so the shell can surface a reproduced problem. This is expected behavior; it is not itself a tool crash.

The output directory contains report.html, report.md, report.json, and replay.json. Open the HTML report in a browser or read the Markdown report directly. Before sharing artifacts, review key names, test names, source stacks, and error messages for sensitive information.

7. Replay and verify a fix

Use the replay file emitted by the diagnosis:

npx --no-install test-autopsy replay .test-autopsy/replay.json --config autopsy.config.json

Prepare the same clean Redis baseline again before replay. Replay refuses changed test files, victim identity, or Redis prefix; it is not a full environment snapshot. An application-code fix can be checked against unchanged tests. If your fix changes a test file, run a fresh diagnosis to create new metadata. Verify the original order and ordinary suite as well.

Reports you can review

Designed summary of the captured demo: polluter and victim retained, 1-minimal sequence, two passing restore trials, and HTML, Markdown, JSON artifacts.

report.html

A self-contained visual report with ordered files, Redis evidence, source call sites, and the trial matrix.

Download the captured HTML example →

report.md

A readable account of the investigation for local review and a carefully sanitized issue or discussion.

Read the curated example →

report.json

Structured trial outcomes, candidate keys, tracked mutations, limitations, and intervention results.

Inspect the report types →

replay.json

The retained order, exact victim, prefix, test-file hashes, and failure fingerprint for a guarded rerun.

Read the replay reference →
Open the real HTML report preview

Captured evidence report from the included planted Redis fixture

The screenshot shows the actual report renderer. Machine-specific paths were removed from the downloadable example.

How it works

Capture the baseline, qualify the victim, confirm the failure, reduce predecessor files, restore suspect keys, and save reports and replay metadata.

Each trial starts from the captured Redis baseline. The restore boundary is the configured physical key prefix; the tool does not call FLUSHDB or FLUSHALL to reset a trial. A restoration intervention happens immediately before the victim file.

Read the architecture · Read the methodology

Verification

Verified baseline cc1c8d5: fifty unit checks plus thirty-nine real Redis integration checks, eighty-nine distinct checks total, and six successful CI jobs. Counts are not coverage percentages.

This is a fixed verification snapshot of commit cc1c8d5: 50 unit tests + 39 real Redis integration tests, with all six jobs successful in the recorded CI run. The badge at the top links to current CI status. These counts are not coverage percentages or a measure of real-world diagnosis accuracy.

The checks exercise reduction, namespace isolation, restoration, replay guards, output escaping, timeouts, cancellation, client tracking, environment pinning, and clean package installation. See validation for commands and evidence boundaries.

Requirements

Component Supported starting point
Node.js 22.12.0 or later
npm 11 or later; CI and package installation use 11.17.0
Module format TypeScript/JavaScript; Test Autopsy itself is ESM
Vitest 4.1.11; peer range ~4.1.11
Redis client for attribution ioredis 5; development dependency pinned to 5.11.1
Redis environment Standalone, dedicated test instance and exclusive namespace
Vitest configuration One project, Node environment, no concurrent test cases

Tracked clients must use awaited individual commands on a standalone TCP/TLS connection. Pipelines, transactions, auto-pipelining, Unix sockets, Sentinel, custom connectors, and Redis Cluster are outside this preview. See configuration and limitations.

Reading a result correctly

Result Meaning Next action
diagnosed A repeatable order-dependent victim failure was retained through the investigation. Inspect the retained sequence, suspects, and intervention evidence.
not-reproduced Every confirmation trial of the configured order passed. Verify the order, baseline, and failing test identity.
fails-alone The victim failed without predecessor files. Establish a clean passing baseline first.
inconclusive Execution or consistency checks did not support a reliable conclusion. Inspect trial details and resolve the missing condition.

An intervention marked supported means the victim passed in repeated trials after the selected suspect keys were restored before its file. This supports a Redis-state explanation in the tested conditions. It does not prove that each restored key is causal, that a single test is solely responsible, or that every future execution will behave the same way.

Minimality, exit codes, and cancellation

Minimality has a precise scope

When reduction.minimal is true, the result is 1-minimal at the file level under the trial oracle: every individual removal of a retained predecessor was verified to pass with the victim file retained. It is not a mathematical guarantee of the shortest sequence. If the budget runs out or unresolved trials prevent the guarantee, the report must not present a certified minimum.

Exit codes

Code Meaning
0 Successful non-failing operation, no reproduced diagnosis, or replay passed.
1 Confirmed diagnosis or replay reproduced the failure.
2 Invalid configuration, failed prerequisites, victim fails alone, or inconclusive investigation.

Cancellation exits with 130 after an attempt to restore the captured state.

Do not classify every nonzero exit as “pollution detected.” See CLI reference for command-specific behavior.

Documentation

01 · Start an investigation

Configure exact file order, the victim, and a dedicated Redis namespace.

Configuration · CLI reference

02 · Understand the evidence

Read the failure oracle, reduction guarantees, state model, and intervention semantics.

Methodology · Architecture

03 · Resolve the unexpected

Diagnose missing victims, changing outcomes, timeouts, and retained private artifacts.

Troubleshooting · Privacy and execution

04 · Evaluate and contribute

Reproduce the known fixture, inspect the limits, and help test real project failures.

Validation · Roadmap · Contributing

Türkçe başlangıç rehberi → · Captured example · Visual asset maintenance

Scope and execution

Use Test Autopsy when you know a failing order and a victim that can pass alone, Redis is a plausible interaction source, and you can give the investigation exclusive access to a disposable namespace.

Supported boundaries and limits of a finding
  • Redis is the reset scope. Files, SQL databases, external services, environment changes, and other side effects are not restored by the tool.
  • The baseline is supplied by you. Pre-existing polluted data is faithfully captured as baseline data.
  • A file is the unit of reduction. Tests inside a file are not rearranged or independently minimized.
  • Intervention happens before the victim file. Earlier tests or setup inside that file can modify state afterward.
  • Tracking is explicit. Unwrapped clients can contribute state differences without source attribution.
  • Serial execution is required. Concurrent tests, background workers, and shared namespaces make attribution or reproduction ambiguous.
  • Snapshotting changes timing. Time-sensitive failures and expiring keys require particular care.
  • Repetition is evidence, not certainty. A small number of repeated trials cannot establish a statistical reliability bound.
  • Replay is local metadata. It does not package your code, credentials, services, fixtures, or toolchain.
  • An observed change is not a proven defect. Some state changes are intentional. Read the intervention and source context together.

These are product boundaries for the preview, not claims that wider flaky-test diagnosis is solved.

What is read, restored, and retained locally

Test Autopsy runs locally. It has no hosted dashboard, telemetry service, model API, or automatic artifact upload.

Redis payloads are kept in memory and a private temporary local snapshot to restore the baseline. The baseline file is deleted after successful restoration; worker logs and diagnostic work files remain locally for troubleshooting. A restore failure retains the snapshot for recovery, and forced termination can leave it behind. Public reports use state fingerprints instead of serialized values. Key names, source stacks, and private worker logs may contain sensitive information. Review artifacts before publishing.

The test runner executes your project, its configuration, and its dependencies. It is not a sandbox. Use trusted repositories and dedicated development services. More detail: privacy and security.

Development

npm ci
npm run check
npm run package:smoke
npm run test:integration

package:smoke builds and packs the package, installs it into a temporary project, verifies the CLI and public exports, and checks that diagnostic work files are excluded from the archive. The CI workflow defines Node.js 22/24 jobs on Windows and Linux, plus Linux integration jobs using Redis 7.4.9; the badge links to their current status.

Integration tests require a reachable dedicated Redis instance at redis://127.0.0.1:16379, or the URL provided by AUTOPSY_INTEGRATION_REDIS_URL. The included planted example runs with npm run demo after setting AUTOPSY_REDIS_URL; see validation for prerequisites and interpretation.

For contribution expectations, reproducible bug reports, and review criteria, see CONTRIBUTING.md. Security-sensitive reports belong in the process described in SECURITY.md.

README graphics are editable repository assets. Run npm run readme:assets to regenerate the light/dark SVG set; see visual maintenance.

Prior art and positioning

Test pollution and order-dependent testing are established topics. This project does not claim to invent them.

Test Autopsy's particular focus is connecting a reduced Vitest file order with scoped Redis state evidence and a repeatable restore intervention. These references provide context; they are not an exhaustive market survey or an endorsement of this tool.

Contribute a real-world reproducer

Contribute a reproducible failure. Open the contributor guide to share a small real-world test pollution example.

The most useful early contribution is a small real-world reproducer with a known failure, the manual investigation, and evidence of whether Test Autopsy reaches a useful finding. New stores and runners should follow demonstrated usefulness and clear reset semantics.

License

MIT © 2026 Ataberk Koç. See LICENSE.

About

Diagnose order-dependent Vitest failures with Redis state evidence, sequence reduction, and controlled replay.

Topics

Resources

Contributing

Security policy

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages