Let a review comment declare several BCQuality articles - #805
Merged
Conversation
A single expected comment can exercise more than one article, most often when it pairs a positive requirement with an explicit false-positive boundary (synthetic__error-handling-errorinfo-actionable-boundary-01, synthetic__events-ishandled-reset-boundary-01). The singular article field forced such findings to drop one of the two, and routing the overflow through metadata.articles is a dead end: the disjointness validator rejects it as soon as another comment in the same entry legitimately owns that article. Replace ReviewComment.article with articles: list[ArticleId] and migrate the 17 existing annotations. Set extra=forbid on the model so a leftover singular article key fails loudly instead of being silently ignored, which would otherwise turn a stale annotation into invisible coverage loss. Also refresh the notebook outputs, whose committed run still reported 130 entries.
Sun Haoran (haoranpb)
approved these changes
Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ReviewComment.articlewas singular, so an expected comment could be attributed to exactly one BCQuality article. That does not match how the gold answers are actually written: a number of them pair a positive requirement with an explicit false-positive boundary in one comment body, and those are two separate articles in BCQuality.This is deliberate, not incidental — the instance ids literally carry
-boundary-. As the article corpus grows, one finding mapping to several articles becomes more common, not less.Routing the second article through
metadata.articleslooked like an escape hatch, but it is a dead end. It holds only while no other comment in the entry owns that article:Case 2 is ordinary — two comments in one entry citing the same article — so the workaround breaks in normal use.
Change
ReviewComment.article: ArticleId | None->ReviewComment.articles: list[ArticleId].dataset/codereview.jsonlto the list form. No compatibility alias is kept; the field is days old and the dataset is fully in-repo, so carrying a shim would just be debt.extra="forbid"onReviewComment. Pydantic ignores unknown keys by default, so without this a leftover"article": "..."would be silently dropped — a stale annotation would become invisible coverage loss instead of an error. Verified no comment payload in the dataset carries any other extra key, and both programmatic construction sites (collection/collect_codereview.py,evaluate/review_parsing.py) pass explicit keyword arguments.declared_articles()and the disjointness validator now flatten per-comment lists;metadata.articleskeeps its existing false-positive-guard role and the disjointness rule is unchanged.9 articles covered across 17/144 annotated entries; 241 of 250 articles have zero coverage.Validation
articlekey is now rejected rather than ignored.ruff format,ruff checkclean;pre-commit run --files notebooks/code-review-coverage.ipynbfully green;ty checkreports only the pre-existingsrc/bcbench/redteam.py:150unused-ignore warning.pytest -q: 740 passed, 2 skipped.