Skip to content

Dependency-bound checks score a checkout without installed dependencies instead of reporting unavailable #100

Open

Description

@serge-ivo

Problem

When a project's dependencies aren't installed, four checks still produce a score, even though the tool either didn't run against the project or ran without its types, config or plugins:

  • types: npx tsc … 2>&1 || true (src/runners/types-check.ts:95), scored as 100 - 5 * parsed errors (:105-106). Output with no error TS lines (npx resolving the wrong package, a crash) scores 100. A real compiler run without @types and dependencies scores near 0 from module-not-found errors. Which one you get depends on the environment. This is inferred from the code, not reproduced.
  • lint: npx eslint … 2>/dev/null || true (src/runners/lint.ts:89). A project config whose plugins aren't installed crashes, and the empty output parses as zero issues.
  • performance / knip: npx knip --reporter json 2>/dev/null || true (src/runners/performance.ts:489).
  • testing: tests can't run. Under --skip-tests the check still gets 10 of 20 execution points as partial credit (src/runners/testing.ts:950-953).

Together that's 28 of 100 weight points (lint 5, types 6, testing 13, performance 4 in @vibecodeqa/schema CHECK_META) measured differently depending on whether node_modules exists. This is the cli#92 defect class: a tool that didn't run must not read as a pass. It affects every scan of a fresh checkout, a forgotten install, and the planned dependency-free server scan (ops docs/25-architecture.md §6).

The mechanism to fix it already exists. unavailableResult (src/runners/toolchain.ts:54) is used for a missing Dart SDK and is excluded from the composite.

Scope

  • Add a dependency probe in toolchain.ts, per project directory: are the project's dependencies resolvable (e.g. node_modules present for the package that declares them)? Cache it like hasDartSdk.
  • When the probe fails, these report unavailable with reason "dependencies not installed — run <pm> install": types (TS targets); lint when it would delegate to a project ESLint config; the knip part of performance; and testing execution and coverage (see the open question).
  • Unchanged: zero-config Biome (lint.ts:145), which is deliberately ephemeral and needs no project dependencies.

Open question (owner)

  • testing: should the whole check be unavailable, or should only the execution and coverage parts be dropped, keeping the test-pyramid analysis scored?

Acceptance criteria

  • A TypeScript and ESLint fixture with no node_modules reports types, lint and knip as unavailable with the install reason, and no npx download is attempted for them.
  • The same fixture with dependencies installed scores as it does today.
  • A zero-config Biome fixture still scores.

Related

cli#26 (unmasking real exit codes is the general fix; this is its dependency-shaped special case). #107 (not-run checks must not show as 100 in deltas; land together). A --commit <sha> historical scan was considered and parked; it would depend on this.


🤖 Filed with Claude Code

Activity

  1. serge-ivo commented on Oct 5, 2026

    @serge-ivo
    ContributorAuthor

    Partly delivered in #111 (branch fix/100-deps-unavailable, commit 0e1e33e). When dependencies are not installed, types, lint (project ESLint) and knip/dead-code now report unavailable with "dependencies not installed — run <pm> install", and no npx is attempted. Zero-config Biome is unchanged.

    Verified: pnpm build is clean; pnpm lint has 0 errors (62 warnings, same as main); pnpm test passes 749/749. That includes new src/runners/toolchain.test.ts cases for each acceptance criterion and monorepo cases.

    Still open:

    • testing: not touched, pending the owner's answer to the open question. Should the whole check be unavailable, or only execution and coverage?
    • Project-configured Biome (npx biome check) isn't gated, because only ESLint is named here. It probably has the same exposure.

    This issue stays open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions