Skip to content

feat: remove the release-blocker concept - #13

Merged
feng-shiplight merged 3 commits into
mainfrom
husky
Aug 10, 2026
Merged

feat: remove the release-blocker concept#13
feng-shiplight merged 3 commits into
mainfrom
husky

Conversation

@feng-shiplight

Copy link
Copy Markdown
Contributor

Summary

Removes the "Release blockers" metric end to end. It was structurally always 0: the blocked gap category it counted could only be produced by a standalone-word match on "blocked" in a narrow text blob (evidence type/path/url/command, residual risk, proof-gap next step), and its other trigger — assessmentStatus === "blocked" — was unreachable, since deriveExpectationAssessment only ever returns NOT COVERED / MANUAL / IMPLICIT / COVERED / PARTIAL. Nothing in the pipeline writes the word, so the tile read as "we checked and found none" while never having checked.

Measured against the two projects consuming this engine: 0 of 326 real gap records across shipyard (54 targets, 297 records) and monots (4 targets, 29 records) would have been classified blocked.

It also had teeth where it should not have:

  • rollupStatusFor set an entire project's status to BLOCKED off that keyword match.
  • riskIndicatorsFor added gapCounts.blocked to releaseRiskCounts.blockers — the same records counted twice, so one blocked gap rendered as "2 blockers".

Removed: the blocked gap category and both classifier branches, the blocked-gaps analytics metric, ReleaseRiskSummary.blockers, WorkspaceReleaseRiskCounts.blockers, WorkspaceAttentionCounts.blocked, the BLOCKED rollup status, the "Blocked" owner risk badge, the dashboard tile, and the feature-table column. The task status blocked is a separate lifecycle vocabulary and is untouched.

Regression found in review and fixed. Deleting the status.includes("BLOCK") branch from riskBadgeFor looked safe because the structured path can never produce that status — but the markdown-fallback path passes the Result cell verbatim from the scanned project, so a row reading BLOCKED | HIGH | None badged as Covered. The root cause was broader than the missing branch: an evidenceConfidence === "HIGH" claim promoted any unrecognized status to Covered, and the audited project writes that text. Confidence now settles only a row that states no result of its own; a stated non-passing result maps to Gap.

Breaking change / versioning

Exported surface narrowed, so the packages that changed are bumped per CONTRIBUTING.md and the fada5bd precedent:

  • @shiplightai/quality-map 0.1.0 → 0.2.0 (GAP_CATEGORIES, GapCategory, the schema enum)
  • @shiplightai/quality-core 0.2.0 → 0.3.0 (removed read-model fields and badge member)

quality-tools stays at 0.3.2 by design: it re-exports GAP_CATEGORIES, but moving it outside ^0.3.0 requires migrating the agent-skill pin in the same change, and scripts/check-quality-skill.sh npx-fetches the published package — so that pin cannot move before a release exists. Flagging for the release step.

Impact on consumers is bounded: accepted_gaps: ["blocked"] still parses and becomes an inert no-op (validate.ts checks array-of-strings only, not membership), so no stored artifact breaks. The one real consumer is shipyard's z.enum(GAP_CATEGORIES) at apps/web/src/app/api/quality-center/feature/quality-map/schema.ts, and narrowing it matches that file's stated purpose.

Guards added

  • Pin the literal published gap-category vocabulary.
  • Assert gapCategoryOrder covers GAP_CATEGORIES — an omission there silently drops a category from every gap record and is not a type error.
  • Pin the riskBadgeFor ladder over the statuses markdown reports actually carry.

Test plan

  • pnpm build clean
  • pnpm typecheck clean (5 projects)
  • pnpm test — 382 passing, 69 files (6 new)
  • pnpm --filter @shiplightai/quality-tools check:size — packed −0.04%, unpacked −0.01%
  • pnpm --filter @shiplightai/quality-ui check:size — packed −0.16%, unpacked −0.23%
  • scripts/check-quality-skill.sh passes
  • Five-lens local review (correctness, security, performance, test coverage, conventions); all MEDIUM+ findings resolved
  • docs/assets/quality-explorer-overview.png still shows the removed tile — needs a fresh screenshot (caption already updated)

🤖 Generated with Claude Code

feng-shiplight and others added 3 commits August 10, 2026 12:28
The "Release blockers" tile on the Explorer overview was structurally
always 0. The `blocked` gap category it counted could only be produced by
a standalone-word match on "blocked" in a narrow text blob (evidence
type/path/url/command, residual risk, proof-gap next step), and its other
trigger — `assessmentStatus === "blocked"` — was unreachable, because
`deriveExpectationAssessment` only ever returns NOT COVERED / MANUAL /
IMPLICIT / COVERED / PARTIAL. Nothing in the pipeline writes the word, so
the tile read as "we checked and found none" while never having checked.

Measured against the two projects that consume this engine: 0 of 326 real
gap records across shipyard (54 targets) and monots (4 targets) would have
been classified `blocked`. The category never fired anywhere.

It also had teeth where it should not have. `rollupStatusFor` set an
entire project's status to BLOCKED off that keyword match, and
`riskIndicatorsFor` added `gapCounts.blocked` to
`releaseRiskCounts.blockers` — the same records counted twice, so one
blocked gap rendered as "2 blockers".

Removed end to end: the `blocked` gap category and its classifier
branches, the `blocked-gaps` analytics metric, `ReleaseRiskSummary.blockers`,
`WorkspaceReleaseRiskCounts.blockers`, `WorkspaceAttentionCounts.blocked`,
the BLOCKED rollup status, the "Blocked" owner risk badge, the dashboard
tile, and the feature-table column. The task status `blocked` is a separate
lifecycle vocabulary and is untouched.

Fix a regression this removal introduced. Deleting the `status.includes("BLOCK")`
branch from `riskBadgeFor` looked safe because the structured path can never
produce that status — but the markdown-fallback path passes the Result cell
verbatim from the scanned project, so a row reading `BLOCKED | HIGH | None`
badged as Covered. The root cause was broader than the missing branch: a
`evidenceConfidence === "HIGH"` claim promoted any unrecognized status to
Covered, and the audited project writes that text. Confidence now settles
only a row that states no result of its own, and a stated non-passing result
maps to Gap.

Bump the packages whose exported surface narrowed, matching the precedent in
fada5bd: quality-map 0.1.0 -> 0.2.0 (GAP_CATEGORIES, GapCategory, the schema
enum) and quality-core 0.2.0 -> 0.3.0 (the removed read-model fields and badge
member). quality-tools stays at 0.3.2: it re-exports GAP_CATEGORIES, but
moving it outside ^0.3.0 requires migrating the agent-skill pin in the same
change, and scripts/check-quality-skill.sh npx-fetches the published package
— so that pin cannot move before a release exists.

Add the guards this change showed were missing: pin the literal published
gap-category vocabulary, assert `gapCategoryOrder` covers `GAP_CATEGORIES`
(an omission there silently drops a category from every gap record, and is
not a type error), and pin the `riskBadgeFor` ladder over the statuses
markdown reports actually carry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The review gate called out to ShiplightAI/internal-tools/claude-review@v1,
a private action. That has been failing since sometime today with
"Unable to resolve action `shiplightai/internal-tools`, not found", while
the action, its v1 tag (d9bb901f2388), this workflow file, and both
repos' Actions policies are all unchanged from this morning's passing
run — so the break is an org-side access change this repo cannot see or
fix.

A public repo depending on a private action is the wrong shape regardless
of that outage. Outside contributors cannot read the logic that gates
their PR, it never runs for fork PRs because secrets are withheld there,
and it fails in exactly this way when org settings move.

Nothing in the shared action was secret: it is a checkout plus
anthropics/claude-code-action plus a prompt, and the OAuth token is
supplied by this repo. Both wrapped actions stay pinned at the same SHAs
they had inside the composite. Verified the rendered prompt is
byte-identical to the action's template with this repo's review-focus
substituted in.

The cost is that shared prompt edits no longer propagate here
automatically, which is the right trade for the one public repo in the
set — its review logic should be readable by the people it reviews.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The inlined review job went green in 15s without posting a review. The
run log shows why: claude_code_oauth_token resolved to "", and
claude-code-action skips every step and exits 0 when it has no token.

The secret is genuinely absent — repos/ShiplightAI/quality/actions/secrets
and .../actions/organization-secrets both return total_count 0 — though
reviews were posted from it earlier today, so it disappeared along with
access to the private action.

A gate that reports success without reviewing anything is the quiet
failure CONTRIBUTING.md warns about: "a value that is dropped rather than
rejected". Check the token before the run so a missing secret is a red
check with an actionable message instead of a silent pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@feng-shiplight
feng-shiplight merged commit b4b1dc1 into main Aug 10, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant