Skip to content

A test run that hits the 120 s limit scores worse than --skip-tests, leaves vitest running, and may score stale coverage #106

Closed

Description

@serge-ivo

Problem

On a pnpm monorepo, npx vitest run --reporter=json --coverage for one workspace project hit the fixed limit (tool run exitCode: null, durationMs: 120078). The testing check reported an error "app vitest command failed" although no test failed. Score and grade depend on machine speed, not code.

Where

  • src/runners/testing.ts:581 — run(cmd, root.cwd, 120_000, …), hardcoded.
  • src/runners/exec.ts:131-146 — a timeout (e.code === "ETIMEDOUT", e.signal === "SIGTERM") is recorded as a generic status: "failed"; nothing marks it as a timeout.
  • testing.ts:505 → command-failed → :906-911 error issue → :946-951 execution 0/20. --skip-tests gets 10/20 (:953): timing out is penalised more than not running.
  • execSync timeout kills only the /bin/sh child; grandchildren (npx → vitest → workers) keep running. Verified locally with execSync('sh -c "sleep …" | cat', { timeout: 1000 }): both processes survive the throw.
  • testing.ts:587 reads coverage/coverage-summary.json from disk regardless of run outcome — after a timeout this can be a previous run's file (inferred from code; not reproduced).

What to do (cheapest first)

  1. exec.ts: on ETIMEDOUT record status: "timeout", timedOut: true, timeoutMs on the ToolRun. Every runner benefits.
  2. testing.ts: per-project run status timeout (label "timed out after N s"); a warning issue test-run-timeout, not an error; execution scored as for --skip-tests (10/20); do not read coverage for a root whose run timed out. The check-level status stays the normal derivation — no new check status.
  3. Configurable timeout: .vcqa.json checks.testing.settings.timeoutMs (the checks.<name>.settings shape from Analyzer settings schema: pass per-analyzer config through scans and record effective settings #30) and --test-timeout <ms>; default unchanged. Record the effective value in details.
  4. Kill the whole process tree on timeout (spawn detached + process.kill(-pid), or spawnSync with an explicit tree kill).

Alternatives rejected

  • Exclude testing from the composite on timeout. Discards 60 static points (pyramid/pairing/quality, :933-971) and weight 13, and makes slow tests a way to raise the score.
  • New check-level status timeout. Consumers recognise only skipped/unavailable (github-app metric-history.ts:252, app repoMetricHistory.ts:398) and would plot it as a real score.

Acceptance criteria

  • Fixture with a test that sleeps past a 2 s configured timeout: ToolRun timedOut: true, project status timeout, a warning (not error), execution 10/20, no coverage read for that root.
  • Same fixture with a larger timeout passes.
  • No process from the timed-out run is alive after scan() returns.
  • Docs PR on vibecodeqa/vibecodeqa: --test-timeout in docs/docs/reference.md, settings.timeoutMs in docs/docs/configuration.md.

🤖 Filed with Claude Code

Activity

  1. serge-ivo commented on Oct 5, 2026

    @serge-ivo
    ContributorAuthor

    Delivered in #110 (branch fix/106-test-timeout, commits 62701d7 and 6fbb074). It covers all four steps:

    • timeout recorded as its own status in exec.ts
    • a test-run-timeout warning, with execution scored the same as --skip-tests and no coverage read for the timed-out project
    • --test-timeout and checks.testing.settings.timeoutMs
    • the whole process tree stopped through src/runners/exec-tree.ts

    Verified: pnpm build is clean; pnpm lint has 0 errors (62 warnings, same as main); pnpm test passes 746/746. The 2 s timeout fixture in src/runners/test-timeout.test.ts covers every acceptance criterion, including that no pid is alive after scan() returns.
    Docs: vibecodeqa/vibecodeqa#58. Merge it after this ships in a release.

  2. added a commit that references this issue on Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions