feat: remove the release-blocker concept - #13
Merged
Merged
Conversation
The "Release blockers" tile on the Explorer overview was structurally
always 0. The `blocked` gap category it counted could only be produced by
a standalone-word match on "blocked" in a narrow text blob (evidence
type/path/url/command, residual risk, proof-gap next step), and its other
trigger — `assessmentStatus === "blocked"` — was unreachable, because
`deriveExpectationAssessment` only ever returns NOT COVERED / MANUAL /
IMPLICIT / COVERED / PARTIAL. Nothing in the pipeline writes the word, so
the tile read as "we checked and found none" while never having checked.
Measured against the two projects that consume this engine: 0 of 326 real
gap records across shipyard (54 targets) and monots (4 targets) would have
been classified `blocked`. The category never fired anywhere.
It also had teeth where it should not have. `rollupStatusFor` set an
entire project's status to BLOCKED off that keyword match, and
`riskIndicatorsFor` added `gapCounts.blocked` to
`releaseRiskCounts.blockers` — the same records counted twice, so one
blocked gap rendered as "2 blockers".
Removed end to end: the `blocked` gap category and its classifier
branches, the `blocked-gaps` analytics metric, `ReleaseRiskSummary.blockers`,
`WorkspaceReleaseRiskCounts.blockers`, `WorkspaceAttentionCounts.blocked`,
the BLOCKED rollup status, the "Blocked" owner risk badge, the dashboard
tile, and the feature-table column. The task status `blocked` is a separate
lifecycle vocabulary and is untouched.
Fix a regression this removal introduced. Deleting the `status.includes("BLOCK")`
branch from `riskBadgeFor` looked safe because the structured path can never
produce that status — but the markdown-fallback path passes the Result cell
verbatim from the scanned project, so a row reading `BLOCKED | HIGH | None`
badged as Covered. The root cause was broader than the missing branch: a
`evidenceConfidence === "HIGH"` claim promoted any unrecognized status to
Covered, and the audited project writes that text. Confidence now settles
only a row that states no result of its own, and a stated non-passing result
maps to Gap.
Bump the packages whose exported surface narrowed, matching the precedent in
fada5bd: quality-map 0.1.0 -> 0.2.0 (GAP_CATEGORIES, GapCategory, the schema
enum) and quality-core 0.2.0 -> 0.3.0 (the removed read-model fields and badge
member). quality-tools stays at 0.3.2: it re-exports GAP_CATEGORIES, but
moving it outside ^0.3.0 requires migrating the agent-skill pin in the same
change, and scripts/check-quality-skill.sh npx-fetches the published package
— so that pin cannot move before a release exists.
Add the guards this change showed were missing: pin the literal published
gap-category vocabulary, assert `gapCategoryOrder` covers `GAP_CATEGORIES`
(an omission there silently drops a category from every gap record, and is
not a type error), and pin the `riskBadgeFor` ladder over the statuses
markdown reports actually carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The review gate called out to ShiplightAI/internal-tools/claude-review@v1, a private action. That has been failing since sometime today with "Unable to resolve action `shiplightai/internal-tools`, not found", while the action, its v1 tag (d9bb901f2388), this workflow file, and both repos' Actions policies are all unchanged from this morning's passing run — so the break is an org-side access change this repo cannot see or fix. A public repo depending on a private action is the wrong shape regardless of that outage. Outside contributors cannot read the logic that gates their PR, it never runs for fork PRs because secrets are withheld there, and it fails in exactly this way when org settings move. Nothing in the shared action was secret: it is a checkout plus anthropics/claude-code-action plus a prompt, and the OAuth token is supplied by this repo. Both wrapped actions stay pinned at the same SHAs they had inside the composite. Verified the rendered prompt is byte-identical to the action's template with this repo's review-focus substituted in. The cost is that shared prompt edits no longer propagate here automatically, which is the right trade for the one public repo in the set — its review logic should be readable by the people it reviews. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The inlined review job went green in 15s without posting a review. The run log shows why: claude_code_oauth_token resolved to "", and claude-code-action skips every step and exits 0 when it has no token. The secret is genuinely absent — repos/ShiplightAI/quality/actions/secrets and .../actions/organization-secrets both return total_count 0 — though reviews were posted from it earlier today, so it disappeared along with access to the private action. A gate that reports success without reviewing anything is the quiet failure CONTRIBUTING.md warns about: "a value that is dropped rather than rejected". Check the token before the run so a missing secret is a red check with an actionable message instead of a silent pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the "Release blockers" metric end to end. It was structurally always 0: the
blockedgap category it counted could only be produced by a standalone-word match on "blocked" in a narrow text blob (evidence type/path/url/command, residual risk, proof-gap next step), and its other trigger —assessmentStatus === "blocked"— was unreachable, sincederiveExpectationAssessmentonly ever returns NOT COVERED / MANUAL / IMPLICIT / COVERED / PARTIAL. Nothing in the pipeline writes the word, so the tile read as "we checked and found none" while never having checked.Measured against the two projects consuming this engine: 0 of 326 real gap records across shipyard (54 targets, 297 records) and monots (4 targets, 29 records) would have been classified
blocked.It also had teeth where it should not have:
rollupStatusForset an entire project's status toBLOCKEDoff that keyword match.riskIndicatorsForaddedgapCounts.blockedtoreleaseRiskCounts.blockers— the same records counted twice, so one blocked gap rendered as "2 blockers".Removed: the
blockedgap category and both classifier branches, theblocked-gapsanalytics metric,ReleaseRiskSummary.blockers,WorkspaceReleaseRiskCounts.blockers,WorkspaceAttentionCounts.blocked, theBLOCKEDrollup status, the"Blocked"owner risk badge, the dashboard tile, and the feature-table column. The task statusblockedis a separate lifecycle vocabulary and is untouched.Regression found in review and fixed. Deleting the
status.includes("BLOCK")branch fromriskBadgeForlooked safe because the structured path can never produce that status — but the markdown-fallback path passes the Result cell verbatim from the scanned project, so a row readingBLOCKED | HIGH | Nonebadged as Covered. The root cause was broader than the missing branch: anevidenceConfidence === "HIGH"claim promoted any unrecognized status to Covered, and the audited project writes that text. Confidence now settles only a row that states no result of its own; a stated non-passing result maps toGap.Breaking change / versioning
Exported surface narrowed, so the packages that changed are bumped per
CONTRIBUTING.mdand thefada5bdprecedent:@shiplightai/quality-map0.1.0 → 0.2.0 (GAP_CATEGORIES,GapCategory, the schema enum)@shiplightai/quality-core0.2.0 → 0.3.0 (removed read-model fields and badge member)quality-toolsstays at 0.3.2 by design: it re-exportsGAP_CATEGORIES, but moving it outside^0.3.0requires migrating the agent-skill pin in the same change, andscripts/check-quality-skill.shnpx-fetches the published package — so that pin cannot move before a release exists. Flagging for the release step.Impact on consumers is bounded:
accepted_gaps: ["blocked"]still parses and becomes an inert no-op (validate.tschecks array-of-strings only, not membership), so no stored artifact breaks. The one real consumer is shipyard'sz.enum(GAP_CATEGORIES)atapps/web/src/app/api/quality-center/feature/quality-map/schema.ts, and narrowing it matches that file's stated purpose.Guards added
gapCategoryOrdercoversGAP_CATEGORIES— an omission there silently drops a category from every gap record and is not a type error.riskBadgeForladder over the statuses markdown reports actually carry.Test plan
pnpm buildcleanpnpm typecheckclean (5 projects)pnpm test— 382 passing, 69 files (6 new)pnpm --filter @shiplightai/quality-tools check:size— packed −0.04%, unpacked −0.01%pnpm --filter @shiplightai/quality-ui check:size— packed −0.16%, unpacked −0.23%scripts/check-quality-skill.shpassesdocs/assets/quality-explorer-overview.pngstill shows the removed tile — needs a fresh screenshot (caption already updated)🤖 Generated with Claude Code