Skip to content

Post the changes in the PR comment, and say why one didn't post - #214

Merged
reuvenharrison merged 6 commits into
mainfrom
pr-comment-lists-changes
Nov 15, 2026
Merged

reuvenharrison merged 6 commits into
mainfrom
pr-comment-lists-changes

Conversation

@reuvenharrison

Copy link
Copy Markdown
Contributor

Two changes to what the breaking and changelog actions put on a pull request.

The comment lists the changes

It used to be a link and nothing else, so a reviewer had to open it to learn what changed, after the inline annotations had already told them. Now:

⚠️ 7 breaking changes in this pull request

severity change endpoint
error api-removed-without-deprecation GET /pets1
error response-success-status-removed POST /pets2
...

2 more in the review.

See them side by side, with the old and new spec

Five rows at most, then a count of the rest: a changelog can run to thousands of changes and a GitHub comment is capped at 65,536 characters.

The rows come from a --format json render that replaces the --fail-on exit-code probe the has-changes decision used. So the count in the heading and the rows under it read the same comparison, the decision stays format-proof (the probe existed because the default render follows any format in .oasdiff.yaml), and the number of oasdiff runs per job is unchanged. The render goes to a file rather than a variable, since a large changelog on a command line hits ARG_MAX, which is the regression #118 and #119 fixed elsewhere in this file.

A comment that doesn't post says what to change

Both causes read the same before: a job without pull-requests: write, and a pull request from a fork, where GitHub makes GITHUB_TOKEN read-only and no permission helps. Only the first is actionable, so they are now reported differently:

::warning::oasdiff: couldn't post the review link as a PR comment (HTTP 403). Add 'permissions: pull-requests: write' to the job that runs this action. The link is in the job summary meanwhile.

oasdiff: this pull request comes from a fork, where GitHub makes the token read-only, so the review link is in the job summary instead of a comment.

The fork case is a plain line, not a warning, because nothing is misconfigured. The delivery mode the action reports separates them too (comment_fork against comment_failed), so a run that could have posted and a run that never could are no longer counted together.

Tests

Three jobs, stubbing oasdiff and curl to assert on the exact comment body the action would post: the rows and the cap on breaking, the same on changelog so the two entrypoints cannot drift, and the fork-against-permission split with its two delivery modes. Verified locally against the real oasdiff binary as well as the stubs; shellcheck -s sh and actionlint clean.

🤖 Generated with Claude Code

reuvenharrison and others added 3 commits November 15, 2026 17:15
The free review comment was a bare link: a reviewer had to open it to find
out what changed, having already read the same changes in the inline
annotations. It now lists the changes, severity, check id and endpoint, up
to five rows and a count of the rest, so the comment answers the question
on the pull request. The data comes from a json render that replaces the
exit-code probe the has-changes decision used, so the comment and the
count read the same comparison and the number of oasdiff runs is unchanged.

A comment that could not be posted reported the same way whether the job
was missing pull-requests: write or the pull request came from a fork,
where GitHub makes the token read-only and no permission helps. The two are
now separate: the first is a warning naming the permission to add, the
second a plain line, and they are distinguished in the delivery mode the
action reports.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both Pro jobs signalled "there are changes" by exiting 1 on the --fail-on
probe, which the json render replaced, so the Pro branch never ran.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Severest first, so the five rows that stay open are the ones that decide
whether the pull request can merge rather than the first five the
comparison happened to produce. The remainder collapses into a details
block, up to fifty rows, so a reviewer gets every change without leaving
the pull request and the comment stays short. Beyond fifty, the count goes
in a line and the review carries the rest.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@reuvenharrison

Copy link
Copy Markdown
Contributor Author

Reshaped the comment after comparing it with the free Codecov comment, which solves the same problem: verdict first, a table of the rows that need attention, the bulk collapsed, the full report one link away.

Two changes from that reading:

  • Severest first. The five rows that stay open are now the ones that decide whether the pull request can merge, not the first five the comparison produced. A changelog where 40 of 45 changes are info used to bury the two errors.
  • The rest collapses instead of being dropped. Up to fifty rows go in a <details> block, so a reviewer sees every change without leaving the pull request. Past fifty, the count goes in a line and the review carries the rest.

Rendered at 1, 3, 8, 51 and 60 changes; the three test jobs now pin the open-row count, the collapsed block and the severity order.

The free comment lists the changes and the Pro comment did not, so the paid
surface showed less than the free one. The Pro comment now carries the same
count, table and link, with the gate state above them, which is the part
only Pro has. Its test asserts the body rather than only the log line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five rows reading new-optional-request-property on the same endpoint told
a reviewer nothing about which property was added. Each row now carries the
change's own sentence, which names the property, followed by the check id
linked to its catalog page, the shape the Pro review comment already uses.
A pipe in the text is escaped so it cannot break the table.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Property names and paths come from the spec, which the author of a pull
request controls. A name containing a backtick closed the code span the
change text opens, so the rest of it rendered as markdown: a mention, a
link or HTML, posted under the action's name. Every character markdown
reads is now written as an entity, which GitHub prints literally, and the
endpoint cell uses a code element instead of a backtick span.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@reuvenharrison
reuvenharrison merged commit d97f53f into main Nov 15, 2026
41 checks passed
@reuvenharrison
reuvenharrison deleted the pr-comment-lists-changes branch November 15, 2026 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant