Skip to content

fix(gates): gap suite could neither pass nor fail on macOS - #7568

Merged
proggeramlug merged 2 commits into
mainfrom
fix/7566-macos-gap-snapshot
Aug 7, 2026
Merged

fix(gates): gap suite could neither pass nor fail on macOS#7568
proggeramlug merged 2 commits into
mainfrom
fix/7566-macos-gap-snapshot

Conversation

@proggeramlug

@proggeramlug proggeramlug commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

run_gap_tests.sh selects test-parity/gap_snapshot.${platform}.json for non-Linux hosts. There is no gap_snapshot.macos.json in the repo, so every local macOS run did all ~490 tests' worth of work and then died in a FileNotFoundError — an exit 1 indistinguishable from "here is your regression list".

This is the fifth gate pathology found this campaign, and a notable one: the useful per-test results were only recoverable by reading the log by hand, which is literally how this session's verification runs were read.

A missing platform baseline now says so and falls back to the shared baseline — the one required CI uses, so a local run is measured against the same bar. Announced on stderr rather than applied silently, because the platforms genuinely differ (macOS carries known host-local failures) and quietly comparing against another platform's expectations would be its own kind of lie.

An explicitly-passed GAP_SNAPSHOT= that does not exist stays a hard error (exit 2) — the fallback is for the unset default, not for a typo.

Found by an agent whose gap run failed this way while validating #7565.

Summary by CodeRabbit

  • Bug Fixes

    • macOS gap tests now fall back to the shared baseline when a platform-specific snapshot is unavailable.
    • A warning is shown when this fallback occurs.
    • Explicitly requested but missing snapshots continue to produce an error.
  • Documentation

    • Added release notes describing snapshot fallback behavior.
  • Chores

    • Updated the application version to 0.5.1321.

@proggeramlug
proggeramlug merged commit b5a2954 into main Aug 7, 2026
27 of 46 checks passed
@proggeramlug
proggeramlug deleted the fix/7566-macos-gap-snapshot branch August 7, 2026 03:37
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f1b15bf3-aa58-4f2e-bd0a-8dc2160c3df4

📥 Commits

Reviewing files that changed from the base of the PR and between af1f739 and 58faea3.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (4)
  • CLAUDE.md
  • Cargo.toml
  • changelog.d/7566-macos-gap-snapshot-fallback.md
  • scripts/run_gap_tests.sh

📝 Walkthrough

Walkthrough

The gap test runner now falls back to the shared baseline when an implicit platform snapshot is missing. Explicit missing snapshots remain errors with exit code 2. Version metadata and the changelog are updated.

Changes

Gap snapshot fallback and release metadata

Layer / File(s) Summary
Snapshot selection and failure handling
scripts/run_gap_tests.sh, changelog.d/7566-macos-gap-snapshot-fallback.md
The runner validates explicit snapshot paths, reports missing explicit snapshots as errors, and uses the shared baseline for missing implicit platform snapshots. The changelog records the behavior.
Version metadata update
CLAUDE.md, Cargo.toml
The documented and workspace package versions change from 0.5.1320 to 0.5.1321.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested labels: bug, parity, tooling

Suggested reviewers: thehypnoo

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/7566-macos-gap-snapshot

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant