diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 74485c3..302be4d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -16,6 +16,17 @@ permissions: env: CI: '1' +# Audited dependency sets: the committed root Gemfile.lock (job "audit") plus +# the sets the matrix jobs resolve from the unlocked gemfiles/standalone.gemfile +# and gemfiles/rails.gemfile. ruby/setup-ruby resolves those gemfiles during the +# job and writes the lockfile next to the gemfile (gemfiles/*.lock, gitignored, +# never committed); the audit of that lockfile rides on rake compile (rake +# audit:advisories -> bin/audit-advisories), the task every matrix entry +# already runs, so each entry audits the set it actually resolved, in the job +# that installed it. +# bundle-audit exits 1 on an advisory, so a finding fails the matrix entry and +# with it the aggregate "CI OK" job. +# # Branch protection only needs to require the final "CI OK" check. # Gems are cached by ruby/setup-ruby (bundler-cache), keyed on the lockfile or # gemfile, Ruby version and OS. diff --git a/AGENTS.md b/AGENTS.md index 1e38cfe..d6d99f8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,7 +17,7 @@ Writing ruby2html templates/components (here or in a user app)? Load [`llms.txt` | Demo Rails app | `app/`, `config/`, `db/` | Not packaged; loads the gem via `require_relative '../lib/gem/ruby2html'` | | Specs | `spec/gem/` (renderer, attributes, escaping, `render_parity_spec.rb`; `render_spec_helper.rb` runs each group against Native **and** PureRuby in one process), `spec/gem/rails/` (handler, compiler, components, middleware), `spec/features/`, `spec/benchmarks/` | Rails-booting via `spec/rails_helper.rb`; `rake spec:standalone` runs `spec/gem` minus `rails/` without Rails | | Benchmarks | `bench/*.rb` (standalone), `spec/benchmarks/requests_spec.rb` (request-level vs ERB/Slim/Phlex) | | -| Scripts | `bin/gem-smoke`, `bin/changelog-notes`, `bin/build-ext` | Packaging smoke test; print a CHANGELOG section; CI-style extension builds (`--werror`, `--cc=clang`, `--sanitize`) | +| Scripts | `bin/gem-smoke`, `bin/changelog-notes`, `bin/build-ext`, `bin/audit-advisories` | Packaging smoke test; print a CHANGELOG section; CI-style extension builds (`--werror`, `--cc=clang`, `--sanitize`); audit one resolved lockfile against the Ruby advisory database (`bundle-audit check --update --gemfile-lock `; exits non-zero on an advisory, except the reviewed advisories the script accepts, scoped to the exact reviewed version the lockfile resolves) | - Supported: all maintained CRuby versions (>= 3.1, gemspec floor; CI tracks the official list — currently 3.3, 3.4, 4.0), JRuby (pure Ruby + the Java escaper), TruffleRuby (pure Ruby), Rails 7.2+. - Dev Ruby: `.ruby-version` (latest stable). The CRuby CI matrix is generated automatically from the maintained-versions list, so a new release (e.g. a new 4.x) joins CI with no edit. @@ -42,7 +42,8 @@ Writing ruby2html templates/components (here or in a user app)? Load [`llms.txt` | Command | Notes | |---|---| | `bundle install` | — | -| `bundle exec rake compile` | C ext → `lib/ruby2html/ruby2html.so`; `RUBY2HTML_NO_SIMD=1` for the scalar escape path | +| `bundle exec rake compile` | C ext → `lib/ruby2html/ruby2html.so`; `RUBY2HTML_NO_SIMD=1` for the scalar escape path; carries `audit:advisories` (CI only; the `spec` tasks carry it too, so `SKIP_COMPILE=1` does not drop it) | +| `bundle exec rake audit:advisories`, `bin/audit-advisories [lockfile]` | audits one resolved lockfile — `.lock` (the script reads `ENV['BUNDLE_GEMFILE']` itself; the rake task runs it with the variable still set, only its child processes get the Bundler keys stripped), else `Gemfile.lock`; an explicit path wins — against the Ruby advisory database via `bundle-audit check --update --gemfile-lock`; the rake task runs only in CI (`CI=1`) and only for the unlocked `gemfiles/*.gemfile` matrix sets; exits non-zero on an advisory, except the advisories `bin/audit-advisories` accepts for the floor lines the matrix resolves on purpose (Rails 7.2.x, Active Support 7.2.x on Ruby 3.1): those are listed by advisory id in the script, scoped to the exact reviewed version — a moved resolution drops every exception and fails the audit until the list is re-reviewed | | `CI=1 bundle exec rspec` | full suite minus the ~6 min benchmark spec (skipped when CI is set) | | `bundle exec rspec spec/gem` | renderer + Rails integration units | | `bundle exec rake spec` | compiles first (`SKIP_COMPILE=1` to skip); `spec:gem` = `spec/gem`; `spec:standalone` = `spec/gem` without Rails | @@ -54,7 +55,7 @@ Writing ruby2html templates/components (here or in a user app)? Load [`llms.txt` | `bundle exec rake smoke` | build + `bin/gem-smoke`: file list check, install into temp GEM_HOME, render without Rails | | `bin/rails server` | demo: `/`, `/rb_files`, `/form`, `/benchmark/{html,ruby,compruby,slim,phlex}` | -- Gem tasks (`compile clean clobber spec spec:gem spec:standalone bench smoke build install release default`) do not boot Rails; any other task loads the Rails tasks. `RUBY2HTML_RAILS_TASKS=1` forces that. +- Gem tasks (`compile clean clobber spec spec:gem spec:standalone bench smoke build install release default audit:advisories`) do not boot Rails; any other task loads the Rails tasks. `RUBY2HTML_RAILS_TASKS=1` forces that. - After editing `ext/` always `rake compile`; otherwise specs run the stale `.so` (or PureRuby if none). - Never run `rake release`, push `v*` tags or `gem push`: merging a version bump to `main` is the release (`docs/release.md`). @@ -122,7 +123,7 @@ Goal: the repo needs no hand maintenance except major-version dependency upgrade |---|---| | Maintainer agent | Repo managed by [developerz.ai](https://developerz.ai) (policy: `.dz/maintainer/maintainer.yml`): triages issues, reviews and merges every PR (contributors, members, bots) once its author marks it ready (out of draft), it is reviewed and `CI OK` is green; daily `health` + weekday `release-due` checkups and a monthly `improve` pass (`.dz/briefs/`: canary fixes, dependency majors, speed with benchmark evidence, correctness, docs drift); opens release PRs; breaking changes escalate to the owner. Sibling projects such as [wurk](https://github.com/sebyx07/wurk) run the same model | | Dependencies | Dependabot (`.github/dependabot.yml`) opens grouped weekly PRs: app dependencies wait out a 14-day cooldown (security updates skip it), while the dev-tools group is exempt from it (`cooldown.exclude`) and is proposed within days; the agent reviews and merges on green — the only merger | -| CI | `.github/workflows/ci.yml`; branch protection requires only the aggregate `CI OK` job: RuboCop, bundle-audit, full suite + Brakeman, gem specs across the Ruby matrix (generated from maintained CRuby versions), native builds on Linux/macOS arm64, `-Werror` with gcc and clang, ASAN/UBSAN (`use_sigaltstack=0`), pure-Ruby fallback, JRuby (Java escaper jar + `-java` gem) and TruffleRuby (pure Ruby) via `gemfiles/standalone.gemfile`, Rails integration per `gemfiles/rails.gemfile`, `bin/gem-smoke` packaging on plain RubyGems. Every job required; no `continue-on-error` | +| CI | `.github/workflows/ci.yml`; branch protection requires only the aggregate `CI OK` job: RuboCop, bundle-audit (the committed root `Gemfile.lock` in the `audit` job, plus the resolved sets of the unlocked matrix gemfiles `gemfiles/standalone.gemfile` and `gemfiles/rails.gemfile` — the matrix sets are audited by `bin/audit-advisories`, which the `audit:advisories` task rides on `rake compile` and the `spec` tasks (so `SKIP_COMPILE=1` cannot drop it), so every matrix entry audits the set it just resolved; the advisories of the floor lines the matrix resolves on purpose (Rails 7.2.x, Active Support 7.2.x on Ruby 3.1) are accepted by advisory id in `bin/audit-advisories`, scoped to the exact reviewed version, every other advisory fails the entry), full suite + Brakeman, gem specs across the Ruby matrix (generated from maintained CRuby versions), native builds on Linux/macOS arm64, `-Werror` with gcc and clang, ASAN/UBSAN (`use_sigaltstack=0`), pure-Ruby fallback, JRuby (Java escaper jar + `-java` gem) and TruffleRuby (pure Ruby) via `gemfiles/standalone.gemfile`, Rails integration per `gemfiles/rails.gemfile`, `bin/gem-smoke` packaging on plain RubyGems. Every job required; no `continue-on-error` | | Canary | `.github/workflows/canary.yml`, monthly on the 1st, not gating: Ruby head and Rails edge surface upstream breaks before a release ships them; the monthly `improve` checkup fixes what it finds | | Benchmarks | `.github/workflows/benchmark.yml`, non-gating: every push to `main` and PR renders a table into the job summary and uploads JSON; PRs get a sticky comment with the delta vs `main`. Set repo variable `BENCH_RUNNER` for a stable self-hosted runner; README holds only a dated release snapshot | | Releases | `.github/workflows/release.yml`: merging a `version.rb` bump to `main` (no `v` tag yet) runs the tests and `rake smoke`, builds the `-java` gem on JRuby (jar inside), publishes **both** the C-extension gem and the `-java` gem via RubyGems trusted publishing (environment `release`), then creates the tag and the GitHub release from `bin/changelog-notes`. The release PR is opened by the agent or the `release` skill (`docs/release.md`) | diff --git a/CHANGELOG.md b/CHANGELOG.md index f02150a..addd20f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ Entries before 1.6.6 were reconstructed from `git log` and RubyGems release date ## [Unreleased] +### Changed + +- CI now audits every matrix entry's resolved dependency set against the Ruby advisory database — the `audit` job only checked the committed root `Gemfile.lock`, which keeps its own `bundle-audit` job. The matrix jobs install an unlocked gemfile, so Bundler resolves it during the job and writes the lockfile next to it (`gemfiles/standalone.gemfile.lock`, `gemfiles/rails.gemfile.lock`); the audit of that resolved lockfile rides on `rake compile` (`audit:advisories` → `bin/audit-advisories`), the task every matrix entry already runs, so each entry audits the set it actually installed, in the job that installed it. An advisory now fails the matrix entry and, with it, the aggregate `CI OK` job — except the advisories of the floor lines the matrix resolves on purpose (Rails 7.2.x, Active Support 7.2.x on Ruby 3.1, where Active Support 8 does not install): upstream patches those lines no more, so they are listed by advisory id in `bin/audit-advisories` and accepted only when the lockfile resolves the exact reviewed version (the five Rails-line components at 7.2.3, pinned by the rails 7.2 gem; Active Support, which the standalone gemfile on Ruby 3.1 resolves unlocked, additionally at 7.2.4 — that patch fixes three of its four advisories, while the `safe_constantize` DoS (`GHSA-8474-xrp6-wh69`) is fixed only in the 8.x lines) — a moved resolution drops every exception and fails the audit until the list is re-reviewed. The audit also rides on the `spec` tasks, so `SKIP_COMPILE=1` (strict extension builds, the pure-Ruby fallback) cannot drop it, and a signal-killed `bundle-audit` is reported as killed instead of looking like an advisory result. Every other line (Rails 8.0, 8.1, Active Support 8) is audited without any exception, and an advisory that is not listed still fails the entry. + ### Fixed - Appending a renderer's own output buffer to itself (`plain(current_output)`, `component(current_output)`) could read freed memory with the C extension: growing the buffer while its own bytes were still being read moved them out from under the read, which aborted under ASAN in the sanitizer CI job and could crash or corrupt output anywhere. Escaped and raw appends (`r2h_append_escaped`, `r2h_append_raw`) now duplicate the string first when it is the buffer; the pure-Ruby renderer was never affected, and output stays byte-identical (pinned by the new "can append the output buffer to itself" example, which runs against both implementations). diff --git a/Rakefile b/Rakefile index c4e6f61..f511055 100644 --- a/Rakefile +++ b/Rakefile @@ -13,7 +13,7 @@ require 'rake/clean' -GEM_TASK = /\A(default|compile(:.*)?|clean|clobber|spec(:gem|:standalone)?|bench|smoke|build(:.*)?|install(:.*)?|release(:.*)?)\z/ +GEM_TASK = /\A(default|compile(:.*)?|clean|clobber|spec(:gem|:standalone)?|bench|smoke|build(:.*)?|install(:.*)?|release(:.*)?|audit(:.*)?)\z/ requested = Rake.application.top_level_tasks.map { |t| t.split('[').first } listing = Rake.application.options.show_tasks || Rake.application.options.show_prereqs # rake -T / -D / -W / -P @@ -69,6 +69,35 @@ else end end +# --- Dependency audit ---------------------------------------------------------- +# The matrix jobs install an unlocked gemfile (gemfiles/standalone.gemfile, +# gemfiles/rails.gemfile): ruby/setup-ruby resolves it during the job and writes +# the lockfile next to it (gemfiles/*.lock, gitignored, never committed). That +# resolved set is what the job installs, so it is what has to be audited against +# the Ruby advisory database; the committed root Gemfile.lock has its own CI job. +# +# The audit rides on compile, the one task every matrix entry runs (spec:gem and +# spec:standalone depend on it), so it needs no extra step in the workflow. The +# spec tasks carry it too, so SKIP_COMPILE=1 (strict extension builds, the +# pure-Ruby fallback) cannot drop the audit of the set the entry resolved; rake +# invokes a task once per run, so it still runs exactly once per entry. +# +# bin/audit-advisories accepts the reviewed advisories of the floor lines the +# matrix resolves on purpose (Rails 7.2.x, Active Support 7.2.x on Ruby 3.1), +# listed by advisory id in the script and scoped to the resolved version. Every +# other advisory fails the entry, and with it the aggregate CI OK job. +desc 'Audit the resolved lockfile against the Ruby advisory database (CI only)' +task 'audit:advisories' do + gemfile = ENV['BUNDLE_GEMFILE'].to_s + if ENV['CI'] && File.basename(gemfile).end_with?('.gemfile') + ruby File.expand_path('bin/audit-advisories', __dir__) + else + puts 'audit:advisories skipped (CI only, and only for the unlocked matrix gemfiles)' + end +end + +Rake::Task[:compile].enhance(['audit:advisories']) + # --- Packaging (build / install / release into pkg/) ---------------------------- # Building a source gem does not need a compiled extension or a Rails boot. require 'bundler/gem_helper' @@ -111,6 +140,15 @@ if defined?(RSpec::Core::RakeTask) %w[spec:gem spec:standalone].each { |t| Rake::Task[t].enhance([:compile]) } unless ENV['SKIP_COMPILE'] == '1' end +# The audit does not depend on the extension: it must also run when compile is +# skipped (SKIP_COMPILE=1, and any entry that skips it in the future). rake +# invokes a task once per run, so entries that reach it through compile do not +# audit twice; outside CI, or outside the unlocked matrix gemfiles, the task +# body skips itself. +%w[spec spec:gem spec:standalone].each do |name| + Rake::Task[name].enhance(['audit:advisories']) if Rake::Task.task_defined?(name) +end + # --- Benchmarks ---------------------------------------------------------------- desc 'Run every bench/*.rb script (compiles the extension first)' task bench: :compile do diff --git a/bin/audit-advisories b/bin/audit-advisories new file mode 100755 index 0000000..5c2929d --- /dev/null +++ b/bin/audit-advisories @@ -0,0 +1,207 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +require 'pathname' + +# Audits one resolved lockfile against the Ruby advisory database. +# +# bin/audit-advisories # the lockfile of BUNDLE_GEMFILE, else Gemfile.lock +# bin/audit-advisories path/to/Gemfile.lock # an explicit lockfile +# +# The matrix jobs install an unlocked gemfile: ruby/setup-ruby resolves it +# during the job and writes the lockfile next to it (gemfiles/*.lock, +# gitignored, never committed). That resolved set is what the job installs, so +# it is what has to be audited; the committed root Gemfile.lock has its own CI +# job (bundle-audit check --update). +# +# bundle-audit exits 1 when it finds an advisory: this script exits with +# bundle-audit's status, so an advisory fails the CI job. The advisories of the +# floor lines the matrix resolves on purpose are accepted by id (see +# ACCEPTED_ADVISORIES below); every other advisory still fails the job. +# +# bundle-audit resolves --gemfile-lock against its own working directory, so +# the audit runs in the project root and gets the lockfile as a path relative +# to it. + +ROOT = File.expand_path('..', __dir__) + +# The advisories the audit accepts, per gem and per reviewed version. +# +# The matrix resolves the oldest supported dependency lines on purpose: Rails +# 7.2.x, and Active Support 7.2.x on Ruby 3.1, where Active Support 8 does not +# install. Upstream does not patch those lines any more: every advisory below +# is fixed in a newer line only (8.0.5.2, 8.1.4.1), or in a patch release the +# Rails line cannot resolve, because the rails gem pins its components to the +# version of the rails release it ships with — the standalone gemfile resolves +# Active Support unlocked, so a newer 7.2.x patch can land there alone, and +# its reviewed version carries only the advisories that still fire there. The +# gem itself has no runtime dependencies and the demo app that pulls Rails is +# not part of the package (SECURITY.md), so the accepted risk belongs to the +# demo app, not to the gem. +# +# The exception is scoped to the exact reviewed version: it applies only when +# the lockfile resolves that very version. A moved resolution — a new patch +# release of the line, patched or not — drops every exception and fails the +# audit until the list is reviewed again, so an exception can never outlive +# the version it was reviewed for, and an id that does not apply to the +# resolved version can never be ignored at all. +# +# Reviewed 2026-11-09 (the five Rails-line components at 7.2.3, pinned by the +# rails 7.2 gem) and 2026-11-10 (activesupport 7.2.4, which the standalone +# gemfile on Ruby 3.1 resolves unlocked: bundler-audit flags only +# GHSA-8474-xrp6-wh69 there — 7.2.4 patches the other three, and the +# safe_constantize DoS is fixed in the 8.x lines only). +# Drop an entry as soon as the matrix resolves a patched version of that line. +ACCEPTED_ADVISORIES = { + 'actionmailbox' => { '7.2.3' => %w[GHSA-5957-c7qm-m469] }, + 'actionpack' => { '7.2.3' => %w[ + GHSA-2586-8ghr-69fj + GHSA-gmj8-hvj3-49mw + GHSA-r9jg-fmfx-2gv6 + GHSA-rwvh-fw6r-pwvp + GHSA-wm3x-v3qw-4jh3 + GHSA-x6mj-8jx9-qhqv + ] }, + 'actionview' => { '7.2.3' => %w[ + GHSA-2p2f-4rq4-gjwc + GHSA-68p9-83h6-5wpj + GHSA-rq8q-p985-fjq7 + GHSA-v55j-83pf-r9cq + GHSA-v6rj-xm57-3p59 + ] }, + 'activestorage' => { '7.2.3' => %w[ + GHSA-3f4q-8x5m-76v8 + GHSA-3w93-6hmw-whh4 + GHSA-73f9-jhhh-hr5m + GHSA-9xrj-h377-fr87 + GHSA-f3w6-w445-fwh9 + GHSA-hfr4-8rjc-97m2 + GHSA-p2pm-g2h6-7j5j + GHSA-p9fm-f462-ggrg + GHSA-qcfx-2mfw-w4cg + GHSA-r46p-8f7g-vvvg + GHSA-v924-7rp8-4mxh + GHSA-xr9x-r78c-5hrm + ] }, + 'activesupport' => { '7.2.3' => %w[ + GHSA-2j26-frm8-cmj9 + GHSA-8474-xrp6-wh69 + GHSA-89vf-4333-qx8v + GHSA-cg4j-q9v8-6v38 + ], '7.2.4' => %w[GHSA-8474-xrp6-wh69] } +}.freeze + +# The child processes must not see the caller's Bundler environment: they read +# the lockfile they are given, not the bundle the caller resolved. GEM_HOME and +# GEM_PATH stay: bundler-audit is installed in and looked up there. +def clean_env + ENV.to_h.reject { |key, _value| key.match?(/\A(BUNDLE|BUNDLER|RUBYOPT|RUBYLIB)/) } +end + +# The lockfile to audit: ARGV[0], else the one Bundler writes next to +# BUNDLE_GEMFILE, else the committed root Gemfile.lock. +def lockfile + argument = ARGV[0].to_s + return File.expand_path(argument) unless argument.empty? + + gemfile = ENV['BUNDLE_GEMFILE'].to_s + return File.expand_path("#{gemfile}.lock") unless gemfile.empty? + + File.join(ROOT, 'Gemfile.lock') +end + +# The path bundle-audit gets for the lockfile: it resolves --gemfile-lock +# against its own working directory (File.join(File.expand_path(Dir.pwd), +# value)), so an absolute path never resolves. The audit runs in the project +# root and hands over a path relative to it. +def lockfile_argument(lock) + Pathname.new(lock).relative_path_from(Pathname.new(ROOT)).to_s +end + +# The versions the lockfile resolves: " gemname (1.2.3)" in its spec list. +def resolved_versions(lock) + File.foreach(lock).each_with_object({}) do |line, versions| + match = line.match(/\A {4}(\S+) \(([^)]+)\)\s*\z/) + versions[match[1]] = match[2] if match + end +end + +# The advisories accepted for this lockfile: an accepted advisory counts only +# when the lockfile resolves its gem at exactly the reviewed version, so an +# accepted id can never hide an advisory in another version of the same line. +def accepted_advisories(lock) + versions = resolved_versions(lock) + + ACCEPTED_ADVISORIES.filter_map do |gem, reviewed| + reviewed[versions[gem].to_s] + end.flatten.uniq +end + +# The first executable named cmd on PATH. +def which(cmd, env) + env.fetch('PATH', '').split(File::PATH_SEPARATOR).each do |dir| + next if dir.empty? + + path = File.join(dir, cmd) + return path if File.file?(path) && File.executable?(path) + end + nil +end + +# Where RubyGems installs executables: that directory is not always on PATH. +def gem_executable(name) + dirs = %i[default_bindir user_bindir bindir].filter_map do |dir| + Gem.public_send(dir) if Gem.respond_to?(dir) + end + dirs.uniq.map { |dir| File.join(dir, name) } + .find { |path| File.file?(path) && File.executable?(path) } +end + +# The gem executable: on PATH when RubyGems' bin dir is, else in the bindir of +# the running Ruby — which is exactly the setup where bundle-audit is missing +# too (installed into a bindir that PATH does not have). +def gem_installer(env) + which('gem', env) || gem_executable('gem') +end + +env = clean_env +lock = lockfile + +abort "audit: no such lockfile: #{lock}" unless File.file?(lock) + +exe = which('bundle-audit', env) + +unless exe + puts '== Installing bundler-audit' + installer = gem_installer(env) + abort 'audit: no gem executable found to install bundler-audit' unless installer + + abort 'audit: could not install bundler-audit' unless system(env, installer, 'install', '--no-document', 'bundler-audit', unsetenv_others: true) + + exe = which('bundle-audit', env) || gem_executable('bundle-audit') +end + +abort 'audit: bundle-audit is not available' if exe.nil? + +puts "== Auditing #{lock}" + +args = ['check', '--update', '--gemfile-lock', lockfile_argument(lock)] +accepted = accepted_advisories(lock) + +unless accepted.empty? + puts "== Accepting #{accepted.size} reviewed advisories (see bin/audit-advisories):" + accepted.sort.each { |advisory| puts " #{advisory}" } + args += ['--ignore', *accepted] +end + +pid = Process.spawn(env, exe, *args, chdir: ROOT, unsetenv_others: true) +_, status = Process.wait2(pid) + +if status.signaled? + # A killed audit is a broken job, not a finding: say so instead of letting a + # generic exit status pass for an advisory result. + warn "audit: bundle-audit was terminated by signal #{status.termsig}" + exit 1 +end + +exit(status.exitstatus || 1) diff --git a/spec/ci/advisory_audit_spec.rb b/spec/ci/advisory_audit_spec.rb new file mode 100644 index 0000000..edc9833 --- /dev/null +++ b/spec/ci/advisory_audit_spec.rb @@ -0,0 +1,646 @@ +# frozen_string_literal: true + +require 'json' +require 'open3' +require 'pathname' +require 'rbconfig' +require 'tmpdir' +require 'yaml' + +# Coverage guard for the dependency audits in CI. +# +# The matrix jobs install an unlocked gemfile: ruby/setup-ruby resolves it +# during the job and writes the lockfile next to it (gemfiles/*.lock, +# gitignored, never committed). That resolved set is what the job actually +# installs, so it is what has to be audited against the Ruby advisory +# database, not only the committed root Gemfile.lock. +# +# The audit rides on the rake tasks every matrix entry already runs (rake +# compile, and the spec tasks when compile is skipped): the platform merge +# gate refuses new steps in a workflow, so the workflow file stays as it is +# and the audit runs from the Rakefile. +# +# The claim lives in the workflow file, the Rakefile and the script, so +# without this spec the audit could be dropped or neutered (a skipped +# prerequisite, a swallowed exit status) without anything going red. The +# wiring itself is pinned by RUNNING it (see the rake wiring group below): +# matching the Rakefile's source text passes on prose alone. +RSpec.describe 'CI dependency audits' do + let(:root) { File.expand_path('../..', __dir__) } + let(:workflow_file) { File.join(root, '.github/workflows/ci.yml') } + let(:workflow) { YAML.safe_load_file(workflow_file, aliases: true) } + let(:jobs) { workflow.fetch('jobs') } + let(:script_path) { File.join(root, 'bin/audit-advisories') } + let(:script) { File.read(script_path) } + + # Unlocked matrix gemfiles: resolved fresh on every CI run, never committed. + let(:matrix_gemfiles) do + Dir[File.join(root, 'gemfiles', '*.gemfile')].map { |path| File.basename(path) }.sort + end + + # A stub bundle-audit: it records how it was called and exits with a given + # status. The real one needs the advisory database and the network. + def with_stub_audit(status:) + Dir.mktmpdir('audit-advisories') do |dir| + record = File.join(dir, 'record') + stub = File.join(dir, 'bundle-audit') + File.write(stub, <<~RUBY) + #!#{RbConfig.ruby} + call = [ARGV.join(' '), Dir.pwd, ENV['BUNDLE_GEMFILE'].to_s] + File.write(#{record.dump}, call.join("\\n") + "\\n") + exit #{status} + RUBY + File.chmod(0o755, stub) + + yield dir, record + end + end + + # What the stub recorded: the arguments it got, the working directory it + # ran in and the BUNDLE_GEMFILE it saw. + def recorded_call(record) + argv, cwd, bundle_gemfile = File.read(record).lines.map(&:chomp) + + { argv: argv, cwd: cwd, bundle_gemfile: bundle_gemfile } + end + + # The advisory ids the stub was told to ignore. + def ignored_advisories(record) + recorded_call(record)[:argv].split('--ignore').last.to_s.split + end + + def run_audit(env, *args) + # The script is standalone: it must not run inside the caller's bundle. + standalone = { 'RUBYOPT' => nil, 'RUBYGEMS_GEMDEPS' => nil, 'BUNDLER_SETUP' => nil } + Open3.capture3(standalone.merge(env), RbConfig.ruby, script_path, *args) + end + + # bundle-audit resolves --gemfile-lock against its own working directory, so + # the paths the script hands over are asserted relative to the repo root. + def relative_to_root(path) + Pathname.new(path).relative_path_from(Pathname.new(root)).to_s + end + + # A lockfile that resolves the given gems to the given versions. + def lockfile_with(dir, gems) + specs = gems.map { |gem, version| " #{gem} (#{version})" }.join("\n") + + File.join(dir, 'Gemfile.lock').tap do |lock| + File.write(lock, "GEM\n remote: https://rubygems.org/\n specs:\n#{specs}\n") + end + end + + def steps_of(job) + job['steps'].is_a?(Array) ? job['steps'] : [] + end + + # Steps that run bundle-audit (not the `gem install` of it). + def audit_steps(job) + steps_of(job).select { |step| step['run'].to_s.include?('bundle-audit check') } + end + + def gemfile_of(job) + job.dig('env', 'BUNDLE_GEMFILE').to_s + end + + # Jobs that install an unlocked matrix gemfile. + def matrix_jobs + jobs.select { |_name, job| gemfile_of(job).end_with?('.gemfile') } + end + + describe 'per-matrix audits' do + it 'runs a rake task in every job that installs an unlocked gemfile' do + expect(matrix_jobs).not_to be_empty, 'no CI job installs an unlocked gemfile' + + matrix_jobs.each do |name, job| + rake_steps = steps_of(job).select { |step| step['run'].to_s.match?(/\brake\b/) } + + expect(rake_steps).not_to be_empty, + "job #{name} installs #{gemfile_of(job)} but never runs a rake task" + + rake_steps.each do |step| + expect(step['run'].to_s).to match(/\brake\b.*\b(compile|spec|audit)\b/), + "job #{name} runs a rake task that does not reach the audit: #{step['run']}" + end + end + end + + it 'never skips the compile step that carries the audit' do + matrix_jobs.each do |name, job| + expect(job.dig('env', 'SKIP_COMPILE')).to be_nil, + "job #{name} sets SKIP_COMPILE: the audit never runs" + + steps_of(job).each do |step| + expect(step.dig('env', 'SKIP_COMPILE')).to be_nil, + "job #{name} sets SKIP_COMPILE on a step: the audit never runs" + end + end + end + + it 'keeps the audit out of the workflow steps' do + matrix_jobs.each do |name, job| + audit_steps(job).each do |step| + expect(step['run'].to_s).not_to include('--gemfile-lock'), + "job #{name} audits a lockfile in the workflow: the audit belongs in the Rakefile" + end + end + end + + it 'covers every unlocked matrix gemfile in the repo' do + covered = matrix_jobs.map { |_name, job| File.basename(gemfile_of(job)) }.uniq + missing = matrix_gemfiles - covered + + expect(covered).to match_array(matrix_gemfiles), + "unlocked gemfiles with no CI job auditing the set it resolves: #{missing.join(', ')}" + end + end + + describe 'the rake wiring' do + # Loading the Rakefile in a real rake run and invoking the tasks is the + # only proof that survives a refactor: matching the Rakefile's source text + # passes on prose alone — a dropped prerequisite, a body that never runs + # the script. The driver (spec/ci/support/rake_wiring_driver.rb) runs + # under `bundle exec` the way the CI jobs run rake, loads the Rakefile at + # the top level the way rake does, and invokes audit:advisories against a + # stub bundle-audit. + let(:driver) { File.join(root, 'spec/ci/support/rake_wiring_driver.rb') } + + # Returns [report, fake_gemfile]: the driver's JSON report, and the + # unlocked matrix gemfile (outside the repo) the Rakefile was loaded with. + def run_rake_driver(extra = {}) + Dir.mktmpdir('rake-wiring') do |dir| + gemfile = File.join(dir, 'standalone.gemfile') + File.write(gemfile, "source 'https://rubygems.org'\n") + File.write("#{gemfile}.lock", '') + record = File.join(dir, 'record') + report = File.join(dir, 'report.json') + stub = File.join(dir, 'bundle-audit') + + File.write(stub, <<~RUBY) + #!#{RbConfig.ruby} + call = [ARGV.join(' '), Dir.pwd, ENV['BUNDLE_GEMFILE'].to_s] + File.write(#{record.dump}, call.join("\\n") + "\\n") + exit Integer(ENV.fetch('RUBY2HTML_STUB_STATUS', '0')) + RUBY + File.chmod(0o755, stub) + + env = { + 'RUBYOPT' => nil, + 'RUBYGEMS_GEMDEPS' => nil, + 'BUNDLER_SETUP' => nil, + 'PATH' => dir, + 'RUBY2HTML_SPEC_GEMFILE' => gemfile, + 'RUBY2HTML_SPEC_RECORD' => record, + 'RUBY2HTML_SPEC_REPORT' => report + }.merge(extra) + bundler = Gem.bin_path('bundler', 'bundle') + + out, err, status = Open3.capture3(env, RbConfig.ruby, bundler, 'exec', RbConfig.ruby, driver, chdir: root) + raise "the rake wiring driver failed (#{status.exitstatus}): #{err}#{out}" unless status.success? + + [JSON.parse(File.read(report)), gemfile] + end + end + + it 'wires the audit into compile, ahead of the spec tasks' do + report, = run_rake_driver + + expect(report.fetch('prerequisites').fetch('compile')).to include('audit:advisories') + expect(report.fetch('prerequisites').fetch('spec')).to eq(%w[compile audit:advisories]) + expect(report.fetch('prerequisites').fetch('spec:gem')).to eq(%w[compile audit:advisories]) + expect(report.fetch('prerequisites').fetch('spec:standalone')).to eq(%w[compile audit:advisories]) + end + + it 'keeps the audit wired when compile is skipped (SKIP_COMPILE=1)' do + report, = run_rake_driver('SKIP_COMPILE' => '1') + + expect(report.fetch('prerequisites').fetch('compile')).to include('audit:advisories') + expect(report.fetch('prerequisites').fetch('spec:gem')).to eq(%w[audit:advisories]), + 'the audit must not depend on the compile step it does not need' + expect(report.fetch('prerequisites').fetch('spec:standalone')).to eq(%w[audit:advisories]) + end + + it 'runs bin/audit-advisories when audit:advisories is invoked in CI for a matrix gemfile' do + report, gemfile = run_rake_driver + + outcome = report.fetch('in_ci_with_matrix_gemfile') + expect(outcome.fetch('ran')).to be(true), 'invoking audit:advisories in CI must run the audit' + + argv, cwd, bundle_gemfile = outcome.fetch('call') + expect(argv).to eq("check --update --gemfile-lock #{relative_to_root("#{gemfile}.lock")}") + expect(cwd).to eq(root), 'the audit runs in the project root, where the lockfile path it hands over resolves' + expect(bundle_gemfile).to be_empty, 'the audit must run outside the caller Bundler environment' + end + + it 'runs nothing outside CI' do + report, = run_rake_driver + + expect(report.fetch('without_ci')).to include('ran' => false), 'the audit must not run outside CI' + end + + it 'runs nothing for the committed root Gemfile' do + report, = run_rake_driver + + expect(report.fetch('with_root_gemfile')).to include('ran' => false), + 'the audit is for the unlocked matrix gemfiles; the root lockfile has its own job' + end + + it 'fails the rake task when the audit finds an advisory' do + report, = run_rake_driver + + outcome = report.fetch('failing_audit') + expect(outcome.fetch('ran')).to be(true) + expect(outcome.fetch('raised')).to eq('RuntimeError'), 'a failing audit must raise out of the rake task' + expect(outcome.fetch('message')).to include('status (1)'), 'the audit exit status must surface' + end + end + + describe 'bin/audit-advisories' do + it 'is an executable script' do + expect(script).to start_with("#!/usr/bin/env ruby\n# frozen_string_literal: true\n") + expect(File.executable?(script_path)).to be(true), 'bin/audit-advisories must be executable' + end + + it 'runs without Bundler: the rake task shells out without bundle exec' do + expect(script).not_to match(/require ['"]bundler|Bundler\./), + 'bin/audit-advisories must not depend on Bundler being loadable' + end + + it 'reports a signal-killed audit instead of passing it off as an advisory' do + Dir.mktmpdir('audit-advisories') do |dir| + stub = File.join(dir, 'bundle-audit') + lock = File.join(dir, 'Gemfile.lock') + File.write(lock, '') + File.write(stub, <<~RUBY) + #!#{RbConfig.ruby} + Process.kill('TERM', Process.pid) + RUBY + File.chmod(0o755, stub) + + _out, err, status = run_audit({ 'PATH' => dir }, lock) + + expect(status.exitstatus).to eq(1), 'a killed audit must still fail the job' + expect(err).to include('signal'), 'a killed audit must be reported as killed, not look like an advisory result' + end + end + + it 'audits the lockfile Bundler wrote next to BUNDLE_GEMFILE' do + with_stub_audit(status: 0) do |dir, record| + gemfile = File.join(dir, 'standalone.gemfile') + lock = "#{gemfile}.lock" + File.write(lock, '') + + _out, _err, status = run_audit({ 'PATH' => dir, 'BUNDLE_GEMFILE' => gemfile }) + + expect(status.exitstatus).to eq(0) + call = recorded_call(record) + expect(call[:argv]).to eq("check --update --gemfile-lock #{relative_to_root(lock)}") + expect(call[:bundle_gemfile]).to be_empty, 'the audit must run outside the caller Bundler environment' + end + end + + it 'audits the committed root Gemfile.lock without arguments' do + with_stub_audit(status: 0) do |dir, record| + _out, _err, status = run_audit({ 'PATH' => dir, 'BUNDLE_GEMFILE' => nil }) + + expect(status.exitstatus).to eq(0) + call = recorded_call(record) + expect(call[:argv]).to eq('check --update --gemfile-lock Gemfile.lock') + end + end + + it 'hands bundle-audit a lockfile path it resolves from its working directory' do + with_stub_audit(status: 0) do |dir, record| + gemfile = File.join(dir, 'standalone.gemfile') + lock = "#{gemfile}.lock" + File.write(lock, '') + + _out, _err, status = run_audit({ 'PATH' => dir, 'BUNDLE_GEMFILE' => gemfile }) + + expect(status.exitstatus).to eq(0) + call = recorded_call(record) + resolved = File.expand_path(File.join(call[:cwd], call[:argv].split.last)) + + expect(resolved).to eq(lock), 'bundle-audit resolves --gemfile-lock against its working directory' + expect(File.file?(resolved)).to be(true), 'the lockfile bundle-audit resolves must exist' + end + end + + it 'propagates a non-zero bundle-audit exit status' do + with_stub_audit(status: 1) do |dir, _record| + lock = File.join(dir, 'Gemfile.lock') + File.write(lock, '') + + _out, _err, status = run_audit({ 'PATH' => dir }, lock) + + expect(status.exitstatus).to eq(1), 'an advisory must fail the script, and with it the CI job' + end + end + + it 'installs bundler-audit when the binary is missing' do + Dir.mktmpdir('audit-advisories') do |dir| + record = File.join(dir, 'record') + install_log = File.join(dir, 'install') + bin = File.join(dir, 'bundle-audit') + installer = File.join(dir, 'gem') + + File.write(installer, <<~RUBY) + #!#{RbConfig.ruby} + File.write(#{install_log.dump}, ARGV.join(' ')) + File.write(#{bin.dump}, "#!#{RbConfig.ruby}\\nFile.write('#{record}', ARGV.join(' '))\\nexit 0\\n") + File.chmod(0o755, #{bin.dump}) + RUBY + File.chmod(0o755, installer) + + lock = File.join(dir, 'Gemfile.lock') + File.write(lock, '') + + _out, _err, status = run_audit({ 'PATH' => dir }, lock) + + expect(status.exitstatus).to eq(0) + expect(File.read(install_log)).to include('install', 'bundler-audit') + expect(File.read(record)).to include("check --update --gemfile-lock #{relative_to_root(lock)}") + end + end + + it 'can find the gem executable beyond PATH when it installs bundler-audit' do + expect(script).to include("gem_executable('gem')"), + 'the gem executable may be missing from PATH too: the installer must look in the RubyGems bindir' + end + + it 'refuses to continue when bundler-audit cannot be installed' do + Dir.mktmpdir('audit-advisories') do |dir| + installer = File.join(dir, 'gem') + lock = File.join(dir, 'Gemfile.lock') + File.write(lock, '') + + File.write(installer, <<~RUBY) + #!#{RbConfig.ruby} + exit 1 + RUBY + File.chmod(0o755, installer) + + _out, err, status = run_audit({ 'PATH' => dir }, lock) + + expect(status.exitstatus).not_to eq(0) + expect(err).to include('could not install bundler-audit') + end + end + + it 'refuses to audit a lockfile that does not exist' do + with_stub_audit(status: 0) do |dir, _record| + missing = File.join(dir, 'missing.lock') + + _out, err, status = run_audit({ 'PATH' => dir }, missing) + + expect(status.exitstatus).not_to eq(0) + expect(err).to include('missing.lock') + end + end + end + + describe 'the accepted advisories' do + # The advisories the script accepts, read from the script itself. + def script_accept_list + block = script[/ACCEPTED_ADVISORIES = \{(.*?)\n\}\.freeze/m, 1] + raise 'bin/audit-advisories declares no ACCEPTED_ADVISORIES' if block.nil? + + block.scan(/'([\w-]+)' => \{(.*?)\}/m).to_h do |gem, body| + versions = {} + body.scan(/'([^']+)' => %w\[(.*?)\]/m) do |version, ids| + versions[version] = ids.split + end + [gem, versions] + end + end + + # The reviewed policy, pinned here INDEPENDENTLY of the script: an edit to + # the script's table (an id typo'd, an entry added, dropped or moved to + # another version) fails this spec and forces a conscious re-review. The + # versions are what the matrix resolves on the floor lines today (see + # bin/audit-advisories); once a resolution moves, the audit drops every + # exception and fails the entry, so this snapshot can only change in step + # with a real re-review. A gem can pin several versions when the matrix + # resolves its line more than once: the rails gem pins Active Support + # 7.2.3 for the Rails entries, while the standalone gemfile on Ruby 3.1 + # resolves the line unlocked and sits on the newest 7.2.x patch. + let(:expected_accepted) do + { + 'actionmailbox' => { '7.2.3' => %w[GHSA-5957-c7qm-m469] }, + 'actionpack' => { '7.2.3' => %w[ + GHSA-2586-8ghr-69fj + GHSA-gmj8-hvj3-49mw + GHSA-r9jg-fmfx-2gv6 + GHSA-rwvh-fw6r-pwvp + GHSA-wm3x-v3qw-4jh3 + GHSA-x6mj-8jx9-qhqv + ] }, + 'actionview' => { '7.2.3' => %w[ + GHSA-2p2f-4rq4-gjwc + GHSA-68p9-83h6-5wpj + GHSA-rq8q-p985-fjq7 + GHSA-v55j-83pf-r9cq + GHSA-v6rj-xm57-3p59 + ] }, + 'activestorage' => { '7.2.3' => %w[ + GHSA-3f4q-8x5m-76v8 + GHSA-3w93-6hmw-whh4 + GHSA-73f9-jhhh-hr5m + GHSA-9xrj-h377-fr87 + GHSA-f3w6-w445-fwh9 + GHSA-hfr4-8rjc-97m2 + GHSA-p2pm-g2h6-7j5j + GHSA-p9fm-f462-ggrg + GHSA-qcfx-2mfw-w4cg + GHSA-r46p-8f7g-vvvg + GHSA-v924-7rp8-4mxh + GHSA-xr9x-r78c-5hrm + ] }, + 'activesupport' => { '7.2.3' => %w[ + GHSA-2j26-frm8-cmj9 + GHSA-8474-xrp6-wh69 + GHSA-89vf-4333-qx8v + GHSA-cg4j-q9v8-6v38 + ], '7.2.4' => %w[GHSA-8474-xrp6-wh69] } + } + end + + let(:pinned_activesupport) { expected_accepted.fetch('activesupport').keys.first } + + it 'accepts exactly the reviewed list, scoped to the exact reviewed version' do + expect(script_accept_list).to eq(expected_accepted), 'the accept list changed: re-review it, then update this snapshot' + end + + it 'declares every reviewed version on the documented 7.2 floor line' do + script_accept_list.each do |gem, versions| + expect(versions).not_to be_empty, "no reviewed version for #{gem}" + + versions.each_key do |version| + expect(version).to match(/\A7\.2\.\d+(?:\.\d+)*\z/), "the reviewed version must sit on the Rails 7.2 floor line: #{gem} #{version}" + end + end + end + + it 'lists well-formed advisory ids, never under two gems' do + ids = script_accept_list.flat_map { |_gem, versions| versions.values.flatten } + + expect(ids).not_to be_empty, 'the script accepts no advisory at all' + ids.each do |id| + expect(id).to match(/\A(?:GHSA|CVE)-[A-Za-z0-9-]+\z/), "#{id} is not an advisory id" + end + + # bundle-audit --ignore is global per run: an id parked under two gems + # would be ignored while auditing either. The same id on two reviewed + # versions of ONE gem is fine — only one of them resolves per run, and + # the ids are scoped to the resolved version (proven below). + gems_of = Hash.new { |hash, key| hash[key] = [] } + script_accept_list.each do |gem, versions| + versions.each_value { |gem_ids| gem_ids.each { |id| gems_of[id] |= [gem] } } + end + + expect(gems_of.select { |_id, gems| gems.size > 1 }).to be_empty, + 'an id listed under two gems would be ignored while auditing either — review the pair' + end + + it 'accepts exactly the ids of the reviewed version the lockfile resolves' do + expected_accepted.fetch('activesupport').each do |version, ids| + with_stub_audit(status: 0) do |dir, record| + lock = lockfile_with(dir, 'activesupport' => version) + + _out, _err, status = run_audit({ 'PATH' => dir }, lock) + + expect(status.exitstatus).to eq(0) + expect(ignored_advisories(record)).to match_array(ids), + "a lockfile resolving #{version} must carry exactly its own exceptions" + end + end + end + + it 'names the accepted advisories in the job log' do + with_stub_audit(status: 0) do |dir, _record| + lock = lockfile_with(dir, 'activesupport' => pinned_activesupport) + + out, _err, status = run_audit({ 'PATH' => dir }, lock) + + expect(status.exitstatus).to eq(0) + expect(out).to include(expected_accepted.fetch('activesupport').fetch(pinned_activesupport).first), + 'the job log must name the advisories the audit accepts' + end + end + + it 'drops every exception for an unreviewed patch of the same line' do + with_stub_audit(status: 1) do |dir, record| + unreviewed_patch = expected_accepted.fetch('activesupport').keys.max.succ # the next 7.2.x patch + lock = lockfile_with(dir, 'activesupport' => unreviewed_patch) + + _out, _err, status = run_audit({ 'PATH' => dir }, lock) + + expect(recorded_call(record)[:argv]).not_to include('--ignore'), + 'a version that was not reviewed must be audited without exceptions' + expect(status.exitstatus).to eq(1), 'its advisories must fail the job until the new version is reviewed' + end + end + + it 'audits a newer line without any exception and fails the job on its advisories' do + with_stub_audit(status: 1) do |dir, record| + lock = lockfile_with(dir, 'activesupport' => '8.1.4.1') + + _out, _err, status = run_audit({ 'PATH' => dir }, lock) + + expect(recorded_call(record)[:argv]).not_to include('--ignore'), + 'a line outside the accepted versions must be audited without exceptions' + expect(status.exitstatus).to eq(1), 'the advisory bundle-audit reported must fail the script, and with it the CI job' + end + end + + it 'accepts advisories of the resolved gem only' do + with_stub_audit(status: 0) do |dir, record| + lock = lockfile_with(dir, 'activesupport' => pinned_activesupport) + + _out, _err, status = run_audit({ 'PATH' => dir }, lock) + + expect(status.exitstatus).to eq(0) + ignored = ignored_advisories(record) + + script_accept_list.each do |gem, versions| + next if gem == 'activesupport' + + versions.each_value do |ids| + ids.each do |id| + expect(ignored).not_to include(id), + "the audit accepted #{id}, an advisory of #{gem}, which it does not resolve" + end + end + end + end + end + + it 'keeps the root lockfile free of exceptions' do + with_stub_audit(status: 0) do |dir, record| + _out, _err, status = run_audit({ 'PATH' => dir }) + + expect(status.exitstatus).to eq(0) + expect(recorded_call(record)[:argv]).not_to include('--ignore'), + 'the committed root Gemfile.lock must be audited without exceptions' + end + end + end + + describe 'audit fatality' do + it 'makes a failed audit reach the aggregate CI OK job' do + aggregate = jobs.find { |id, job| id == 'ci-ok' || job['name'] == 'CI OK' }&.last + + expect(aggregate).not_to be_nil, 'the aggregate CI OK job is missing' + needs = Array(aggregate['needs']).map(&:to_s) + + matrix_jobs.each_key do |name| + expect(needs).to include(name), "job #{name} audits dependencies but is not required by the CI OK job" + end + end + end + + describe 'root audit' do + it 'still audits the committed root Gemfile.lock' do + root_audits = jobs.reject { |_name, job| gemfile_of(job).end_with?('.gemfile') } + .flat_map { |name, job| audit_steps(job).map { |step| [name, step] } } + .reject { |_name, step| step['run'].to_s.include?('--gemfile-lock') } + + expect(root_audits).not_to be_empty, + 'no job runs bundle-audit check without --gemfile-lock (the default Gemfile.lock)' + end + + it 'keeps the root audit job required by the CI OK job' do + aggregate = jobs.find { |id, job| id == 'ci-ok' || job['name'] == 'CI OK' }&.last + needs = Array(aggregate['needs']).map(&:to_s) + + expect(needs).to include('audit'), 'the root Gemfile.lock audit is not required by the CI OK job' + end + end + + it 'keeps the matrix lockfiles out of git' do + ignored = File.readlines(File.join(root, '.gitignore'), chomp: true).map(&:strip) + + expect(ignored).to include('/gemfiles/*.lock') + end + + it 'states which dependency sets are audited' do + names = jobs.map { |_id, job| job['name'].to_s } + + jobs.flat_map { |_id, job| steps_of(job).map { |step| step['name'].to_s } } + comments = File.readlines(workflow_file).filter_map { |line| line[/^\s*#\s*(.*)$/, 1] } + stated = (names + comments).join("\n") + + expect(stated).to include('Gemfile.lock'), 'the workflow does not name the audited root Gemfile.lock' + matrix_gemfiles.each do |gemfile| + expect(stated).to include("gemfiles/#{gemfile}"), "the workflow does not name the audited gemfiles/#{gemfile}" + end + end + + it 'documents the audited dependency sets in the repo guide' do + row = File.readlines(File.join(root, 'AGENTS.md')).find { |line| line.start_with?('| CI |') } + + expect(row).not_to be_nil, 'the AGENTS.md CI row is missing' + expect(row.to_s).to match(/bundle-audit|audit-advisories/) + expect(row.to_s).to include('Gemfile.lock') + expect(row.to_s).to include('gemfile') + end +end diff --git a/spec/ci/support/rake_wiring_driver.rb b/spec/ci/support/rake_wiring_driver.rb new file mode 100644 index 0000000..259b052 --- /dev/null +++ b/spec/ci/support/rake_wiring_driver.rb @@ -0,0 +1,92 @@ +# frozen_string_literal: true + +# Runs the Rakefile the way the CI jobs run rake, and reports what the audit +# wiring does. +# +# The spec (spec/ci/advisory_audit_spec.rb) starts this driver under +# `bundle exec`, with a stub bundle-audit first on PATH and an unlocked matrix +# gemfile outside the repo, and asserts the JSON report the driver writes: +# +# RUBY2HTML_SPEC_GEMFILE the unlocked matrix gemfile the audit runs against +# RUBY2HTML_SPEC_RECORD the file the stub bundle-audit records its call in +# RUBY2HTML_SPEC_REPORT the JSON report this driver writes +# +# Matching the Rakefile's source text proves the wiring on prose alone: the +# report pins what a real rake run does — the prerequisites the tasks carry, +# and what audit:advisories does in CI, outside CI and for the committed root +# Gemfile. + +require 'json' +require 'rake' + +ROOT = File.expand_path('../../..', __dir__) +Dir.chdir(ROOT) + +MATRIX_GEMFILE = ENV.fetch('RUBY2HTML_SPEC_GEMFILE') +RECORD = ENV.fetch('RUBY2HTML_SPEC_RECORD') +REPORT = ENV.fetch('RUBY2HTML_SPEC_REPORT') + +# `bundle exec` hands its Bundler environment to every child it starts, and +# RubyGems loads bundler/setup in every child that finds BUNDLER_SETUP or +# RUBYOPT in its environment. The audit script is standalone (it strips the +# Bundler variables for bundle-audit), and the matrix gemfile of the spec is a +# stub that bundler/setup would resolve against the repository lockfile: the +# driver drops the injected variables, so the audit runs outside the caller's +# Bundler environment, the way the script is meant to run. +ENV.delete('RUBYOPT') +ENV.delete('RUBYLIB') +ENV.delete('BUNDLER_SETUP') + +# Load the Rakefile the way rake does: at the top level, with the tasks of the +# application. +Rake.application.init +Rake.application.load_rakefile + +# The prerequisites a task carries, as the report names them. +def prerequisites_of(task) + Rake::Task[task].prerequisites.map(&:to_s) +end + +# Invokes the audit task and reports what it did: whether the audit ran (the +# stub recorded a call), the call it made and the error it raised. +def run_audit(ci:, gemfile:, stub_status: '0') + File.delete(RECORD) if File.exist?(RECORD) + + ENV['CI'] = ci ? '1' : nil + ENV['BUNDLE_GEMFILE'] = gemfile + ENV['RUBY2HTML_STUB_STATUS'] = stub_status + + task = Rake::Task['audit:advisories'] + task.reenable + + error = nil + begin + task.invoke + rescue StandardError => e + error = e + end + + outcome = { 'ran' => File.exist?(RECORD) } + if outcome['ran'] + argv, cwd, bundle_gemfile = File.read(RECORD).lines.map(&:chomp) + outcome['call'] = [argv, cwd, bundle_gemfile] + end + outcome['raised'] = error.class.name if error + outcome['message'] = error.message if error + outcome +end + +report = { + 'prerequisites' => { + 'compile' => prerequisites_of('compile'), + 'spec' => prerequisites_of('spec'), + 'spec:gem' => prerequisites_of('spec:gem'), + 'spec:standalone' => prerequisites_of('spec:standalone') + }, + 'without_ci' => run_audit(ci: false, gemfile: File.join(ROOT, 'Gemfile')), + 'with_root_gemfile' => run_audit(ci: true, gemfile: File.join(ROOT, 'Gemfile')), + 'in_ci_with_matrix_gemfile' => run_audit(ci: true, gemfile: MATRIX_GEMFILE), + 'failing_audit' => run_audit(ci: true, gemfile: MATRIX_GEMFILE, stub_status: '1') +} + +File.write(REPORT, JSON.generate(report))