testfold

Test runner designed for AI agent workflows. Runs Jest/Playwright suites and generates structured reports optimized for LLM consumption.

Why

When AI agents (Claude Code, Cursor, etc.) run tests, they face issues:

• Token waste - reading verbose logs with ANSI escape codes, duplicate stack traces, framework boilerplate
• Parsing overhead - different output formats from Jest vs Playwright vs other frameworks
• Multiple commands - running npm run test:unit, npm run test:e2e separately
• Noise - agents read full console output when they only need "what failed and why"

What it does

• Runs all test suites with single npm test command
• Generates summary.json - structured pass/fail data agents can parse instantly
• Creates per-test markdown failure reports - clean error + stack trace, no ANSI codes
• Supports Jest, Playwright, and custom parser modules
• Graceful error recovery - partial results extracted from failed runs

Output

summary.json                    # Read first: {"success": false, "failed": 2, "failedTests": [...], "errors": [...]}
test-results/
  timing.json                   # Slowest tests sorted by duration
  unit-timing.txt               # Per-suite timing (top slowest tests, file grouping)
  test-summary.log              # ANSI-free summary table
  unit.json                     # Raw Jest/Playwright results
  unit.log                      # Full command output
  failures/
    unit/
      01-user-service-login.md  # Individual failure report
      02-auth-middleware.md

Agent reads summary.json (50 tokens) instead of scrolling through 2000 lines of test output. For performance analysis, check timing.json to identify slow tests.

Setup

npm install testfold

Create testfold.config.ts:

import type { Config } from 'testfold';

export default {
  artifactsDir: './test-results',
  testsDir: './tests', // For path prefix resolution
  reporters: ['console', 'json', 'markdown-failures', 'timing', 'text'],
  suites: [
    { name: 'Unit', type: 'jest', command: 'npx jest --json', resultFile: 'unit.json' },
    { name: 'E2E', type: 'playwright', command: 'npx playwright test', resultFile: 'results.json' },
  ],
} satisfies Config;

Usage

testfold                              # Run all suites
testfold unit integration             # Run specific suites
testfold -e staging                   # Use staging environment
testfold -r json                      # Override reporters
testfold -r console,json,timing-text  # Multiple reporters (comma-separated)
testfold -r console -r json           # Multiple reporters (repeated flag)
testfold -r ./my-reporter.ts          # Custom reporter from file
testfold -g "auth"                    # Filter tests by name pattern
testfold --grep "login" unit          # grep maps to --testNamePattern (Jest) or --grep (Playwright)
testfold --grep-invert "slow"         # Exclude tests matching pattern
testfold -f auth.test.ts              # Run specific test file
testfold -f auth.test.ts -f api.test.ts # Run multiple test files
testfold -- --testNamePattern="auth"  # Pass args to test framework
testfold -- user                      # Path prefix resolution: "user" -> tests/unit/user.test.ts
testfold --dry-run                    # Preview commands without executing

Reporters

Reporter	Output
console	Terminal output with colors, hierarchy, agent-friendly failure sections, JSON summary line
json	summary.json with structured data, failedTests[], errors[]
markdown-failures	Per-test failure reports in failures/
timing	timing.json with slowest tests
timing-text	Per-suite .txt files with top slowest tests and file grouping
text	Plain text output for CI (no ANSI, no markdown)
summary-log	ANSI-free test-summary.log with summary table
Custom path	./my-reporter.ts — load Reporter from file

Environment Support

Load environment-specific .env files:

export default {
  suites: [
    {
      name: 'E2E',
      type: 'playwright',
      command: 'npx playwright test',
      resultFile: 'results.json',
      environments: {
        staging: {
          baseUrl: 'https://staging.example.com',
        },
        production: {
          envFile: '.env.prod',
          urlExtractor: (content) => content.match(/APP_URL=(.+)/)?.[1],
        },
      },
    },
  ],
} satisfies Config;

Run with: testfold -e staging

Custom Parsers

For test frameworks beyond Jest/Playwright:

// testfold.config.ts
export default {
  suites: [
    {
      name: 'Custom',
      type: 'custom',
      command: 'my-test-runner',
      resultFile: 'results.json',
      parser: './parsers/my-parser.ts', // Path to custom parser
    },
  ],
} satisfies Config;

// parsers/my-parser.ts
import type { Parser, ParseResult } from 'testfold';

export default class MyParser implements Parser {
  async parse(jsonPath: string, logPath?: string): Promise<ParseResult> {
    // Parse your test framework's output
    return { passed: 10, failed: 0, skipped: 0, duration: 1000, success: true, failures: [] };
  }
}

Custom Reporters

Load reporters from file paths:

import type { Reporter } from 'testfold';

export default class SlackReporter implements Reporter {
  onStart(suites) {
    /* notify start */
  }
  onSuiteComplete(suite, result) {
    /* notify per suite */
  }
  async onComplete(results) {
    /* send summary */
  }
}

Use via CLI: testfold -r ./reporters/slack.ts or in config reporters: ['console', './reporters/slack.ts'].

Guard Hooks

Hooks can return a GuardResult to fail a suite even when all tests pass (e.g., leak detection):

import type { Config, GuardResult } from 'testfold';

export default {
  hooks: {
    beforeSuite: async (suite) => {
      const stats = await getResourceStats();
      return { ok: true }; // or { ok: false, error: 'Resource leak detected' }
    },
    afterSuite: async (suite, result) => {
      const leaks = await checkForLeaks();
      if (leaks.length > 0) return { ok: false, error: `${leaks.length} leaks found` };
    },
  },
  // ...
} satisfies Config;

Returning void or undefined is treated as success (backward compatible).

File Filters

--file (-f) values are passed to Jest and Playwright as positional test-path filters, which the frameworks treat as regular expressions searched in each test file's path; Playwright also accepts a :line suffix. The frameworks combine several filters as a union, so testfold checks each requested filter after the run:

• A filter must select at least one test file that one of the selected suites ran. A filter that matches a file in any selected suite is satisfied; it does not have to match every suite.
• Each filter that selected no test file in any selected suite is recorded as an infrastructure failure named Unmatched File Filter: <filter>, attached to the first selected suite. The run fails with exit code 2, even when other filters matched and their tests passed.
• The check applies only when every selected suite ran and reported the test files it executed. When a suite was skipped by fail-fast, stopped by a beforeSuite guard, timed out, produced no result file, or uses a custom parser that reports neither testFiles nor a complete testResults list, the check is skipped because the filter may belong to that suite; such a run already fails or depends on that suite's own result.

With --file, reporters receive each suite's result after all selected suites finish, so a suite is never shown as passed before the check.

Custom parsers can return testFiles (the test files the framework ran) in their ParseResult to take part in the check.

Path Prefix Resolution

Pass partial test file names and testfold resolves them to full paths:

testfold unit -- auth              # Resolves to tests/unit/auth.test.ts (if unique match)
testfold unit -- user-service      # Resolves to tests/unit/user-service.test.ts

Configure the search directory:

export default {
  testsDir: './tests', // Default: './tests'
  // ...
} satisfies Config;

Graceful Error Recovery

Even when tests crash or timeout, testfold attempts to parse partial results. Non-zero exit codes don't prevent result extraction.

Per-Suite Artifact Cleanup

Running a single suite only cleans that suite's artifacts, preserving results from previous runs of other suites.

Agent-Friendly Output

When tests fail, the console reporter outputs:

• Consolidated failures — all failing tests across suites in one section
• Re-run instructions — exact CLI commands to re-run just the failed tests
• Agent instructions block — structured === AGENT INSTRUCTIONS === section with error patterns and suggested actions
• JSON summary line — TESTFOLD_RESULT:{json} as the last line (always, success or failure)
• Coverage detection — prints path to coverage/ directory when present
• Real-time progress — emits [SuiteName] … N tests to stderr during long-running suites

Semantic exit codes: 0 = pass, 1 = test failures, 2 = infrastructure error, 3 = timeout.

Use --dry-run to preview resolved commands without execution.

Development and releases

Node.js 22.18.0 and newer are supported. Local development is pinned to Node.js 24.20.0 through .nvmrc and .node-version, while packageManager pins npm 11.19.1. Install the locked dependencies and run the complete local gate:

npm ci --no-audit --no-fund
npm run verify

npm run verify checks types, lint, unit and integration behavior, GitHub workflow contracts, and the exact npm candidate in an isolated consumer. Publishing is performed from an accepted GitHub Release asset through npm trusted publishing; it does not require an NPM_TOKEN repository secret.

See CONTRIBUTING.md for development expectations and docs/RELEASE.md for the release procedure.

Docs

• Architecture
• Development
• Testing
• Release runbook
• Changelog
• Security policy

License

MIT