Skip to content

Per-check deltas treat the 100/A placeholder of skipped/unavailable checks as a real score — removing a tool reads as an improvement #107

Closed

Description

@serge-ivo

Problem

Since #52, a check that did not run (disabled, not applicable, Pro key missing, tool missing) is written with a placeholder score: 100, grade: "A" (src/core.ts:326-327; rationale at src/runners/toolchain.ts:51-55). The composite excludes it (score.ts:19) and the check tables render it as skipped (cli.ts:213-220, :644-646; pr-comment.ts:113). The per-check deltas do not:

  • delta.ts:52-54, :117-122 and trend.ts:34-41 subtract raw scores, then pr-comment.ts:92-96, cli.ts:187-191 and report/pages.ts:1004 print them.
  • Dart SDK installed between scans → PR comment "⚠️ lint: 100 → 72 (-28)".
  • Dart SDK removed (or a Pro key dropped) → "✅ lint: 64 → 100 (+36)": uninstalling a tool is reported as a quality improvement.
  • analyzer-snapshot.ts:9 writes score: 100 into meta.analyzerSnapshots for not-run checks, although AnalyzerSnapshot.score is optional (schema/src/report-schema.ts:52).

What to do

  1. delta.ts / trend.ts: when either side's status is skipped or unavailable (fall back to details.skipped/unavailable/comingSoon for older reports, as github-app/src/metric-history.ts:250-260 does), emit no numeric delta; report a status transition instead ("lint: unavailable → 72").
  2. Renderers (pr-comment.ts, cli.ts, report/pages.ts) list status transitions separately from score changes.
  3. analyzer-snapshot.ts: omit score for not-run checks.
  4. Schema: CheckResult.status is declared in schema 0.6.0 (Reports don't say which commit, branch, CI run or producer they describe — declare scan provenance in VibeReport.meta (0.6.0) schema#6) — the CLI already writes it (core.ts:347); today it survives only via .passthrough().

Alternatives rejected

  • score: null / grade: null. Breaking: CheckResultSchema requires z.number() and GradeSchema (report-schema.ts:32-39); consumers pin ^0.5.0; mcp does arithmetic on it (mcp/src/index.ts:415); stored history already holds 100s. github-app#9 already settled "read status, not score". Revisit only for a v2 report contract.

Acceptance criteria

  • Delta between a report with lint unavailable and one where it scored 72 shows no "100 → 72" line in markdown, PR comment or HTML, and does show the status transition.
  • The reverse direction shows no "+N" improvement.
  • meta.analyzerSnapshots has no score for skipped/unavailable checks.

Related

#52 (introduced the placeholder), github-app#9 (consumer-side fix). Siblings: vibecodeqa/mcp#8, vibecodeqa/app#78.


🤖 Filed with Claude Code

Activity

  1. serge-ivo commented on Oct 5, 2026

    @serge-ivo
    ContributorAuthor

    Delivered in #115 (fix/107-not-run-deltas, ec86546). Steps 1–3:

    • a status transition instead of a numeric delta when either side did not run, with the details.* fallback for older reports
    • transitions listed separately in the PR comment, CLI markdown and HTML
    • no score in analyzer snapshots for not-run checks

    Step 4 (CheckResult.status in the schema) waits for schema#6 0.6.0.

    Verified: pnpm build is clean; pnpm lint has 0 errors (62 warnings, same as main); pnpm test passes 751/751. The new tests in delta, pr-comment, report/pages and analyzer-snapshot cover both directions, and they fail with the fix removed.

    #100's unavailable change (#111) should land together with this. This stays open until #115 merges.

  2. added a commit that references this issue on Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions