Skip to content

AI review: give reviewers the failing step of each CI job, ignore Trivy failures - #18

Merged
ryanmelt merged 2 commits into
mainfrom
ai-review-ci-job-logs
Oct 11, 2026
Merged

ryanmelt merged 2 commits into
mainfrom
ai-review-ci-job-logs

Conversation

@ryanmelt

@ryanmelt ryanmelt commented Oct 11, 2026 •

Copy link
Copy Markdown
Member

Problem

The AI reviewers report that they can't see the logs of failed CI jobs. In the four recent cosmos AI Review runs checked (#3902, #3996, #3998, #4015), every failed job reached the agents as just (log unavailable). There are two problems in ai-review/ai_review_gate.sh:

  1. The download fails in Actions, and the error was thrown away. gh api repos/.../actions/jobs/<id>/logs sent its stderr to /dev/null. The same call works locally with a user token, and the gate job's token has Actions: read, so the cause is still unknown.
  2. The wrong part of the log was kept. Even a successful download kept the last 150 lines of the job, and those are harden-runner's post-job cleanup. In the Ruby unit-test job checked, the rspec failure summary sits about 30 lines above ##[error]Process completed with exit code 1., followed by about 170 lines of cleanup.

Changes

  • Cut the log at the first Post job cleanup. line, end at the last ##[error] line, and keep the LOG_LINES lines before it. Also strip the byte-order mark, timestamps and colour codes, and cut lines at 1,000 characters.
  • A failed download now produces a ::warning:: with gh's error, and the agents see (log unavailable: <error>).
  • Two new tests: one for the trimming, one for the error reporting. A third test covers the Trivy filter below.

Follow-up

This makes the download failure visible but doesn't fix it. The next review of a PR with a failing job will show the error in the "Wait for CI and collect failures" step, and the cause can be fixed from that.

Ignore Trivy failures

A Trivy failure means a base image or dependency has a known vulnerability. A fix to the PR can't address that, but the reviewers still spent turns on it, and it counted toward the CI fix rounds. The gate now skips a failed job when every failed step has "trivy" in its name, and adds a ::notice:: to the gate log. For example, cosmos's Trivy Scan fails on its Run Trivy on image … steps.

  • A job that also failed somewhere else (for example the image build in Trivy Scan) is still reported, with its full log.
  • A run whose only failed jobs were ignored doesn't fall back to the "workflow failure" entry.
  • On an AI review fix commit, a Trivy-only failure counts as passing CI, so it doesn't start another fix round.
  • Build UBI's Trivy step is unaffected: it uses continue-on-error and reports through a commit status, which the gate doesn't collect.

Testing

  • python3 -m unittest discover -s tests: 89 tests pass
  • ruff check and ruff format: clean; shellcheck: clean
  • Ran the new trimming against the real log of job 114567821523 (cosmos #3902): the output contains the full rspec failure summary and none of the post-job cleanup

🤖 Generated with Claude Code

ryanmelt and others added 2 commits October 11, 2026 16:27
The reviewers have never received a failed job's log: every download in
the gate failed in Actions with its error sent to /dev/null, so they saw
only "(log unavailable)". Even a successful download kept the last 150
lines of the job, which are harden-runner's post-job cleanup rather than
the failure.

Keep the lines leading up to the last error before the post steps, cap
long lines, and report a failed download as a warning with its error so
the cause shows in the gate log.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A Trivy failure means a base image or dependency has a known
vulnerability, which a fix to the PR cannot address, so the reviewers
spent turns on it and it counted toward the CI fix rounds. Skip a
failed job whose only failed steps are Trivy scans; a job that also
failed elsewhere, such as an image build, is still reported.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ryanmelt ryanmelt changed the title Give the AI reviewers the failing step of each failed CI job AI review: give reviewers the failing step of each CI job, ignore Trivy failures Oct 11, 2026
@ryanmelt
ryanmelt merged commit 9123f54 into main Oct 11, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant