Skip to content

Commit 1472333

Browse files
committed
ci: check every family's Snowflake obligations nightly and report failed nightly runs
Pull requests and queue groups check only the families their change touches, so a scheduled run, whose base is the commit it tests, checked none. SYNTAX_OBLIGATIONS_SINCE, empty outside pull requests and queue groups, now makes those runs check every family. A failed scheduled run opens an issue labelled nightly-ci-failure, or comments on the open one.
1 parent 946a8fa commit 1472333

3 files changed

Lines changed: 39 additions & 6 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,9 @@ name: CI
66
# - code changes run check-ci, the Rust and Python tests, and the replay of the recorded cases,
77
# which reuses main's judgements for every case whose conversion and recording are unchanged;
88
# - a change under a full-replay path replays every case.
9-
# The nightly run on main replays every case, refreshes the replay cache pull requests reuse,
10-
# converts DuckShop within its time budget and fuzzes. Release Please dispatches a full run on
9+
# The nightly run on main replays every case, checks every family's syntax obligations, refreshes
10+
# the replay cache pull requests reuse, converts DuckShop within its time budget and fuzzes; when
11+
# it fails it opens an issue labelled nightly-ci-failure, or comments on the one already open. Release Please dispatches a full run on
1112
# each release pull request's branch, so every release replays every case first.
1213
# A merge queue group runs the same checks as a pull request, scoped against the group's base.
1314
on:
@@ -79,6 +80,9 @@ jobs:
7980
UV_NO_SYNC: "1"
8081
# The evidence pin check and the result baselines compare with the pull request's base.
8182
EVIDENCE_BASE: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha || 'origin/main' }}
83+
# Pull requests and queue groups check the families their change touches; other runs, which
84+
# have no base to compare with, check every family's syntax obligations.
85+
SYNTAX_OBLIGATIONS_SINCE: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha || '' }}
8286
REPLAY: ${{ needs.changes.outputs.replay }}
8387
# The Python tests and the replay take the module and binary from check-merge's gate build.
8488
PYTHON_MODULE_BUILD: gate-build
@@ -243,3 +247,27 @@ jobs:
243247
name: fuzz-artifacts
244248
path: fuzz/artifacts
245249
if-no-files-found: ignore
250+
251+
nightly-report:
252+
name: Report a failed nightly run
253+
if: failure() && github.event_name == 'schedule'
254+
needs: [changes, site, checks, verify, duckshop, fuzz]
255+
runs-on: ubuntu-24.04
256+
permissions:
257+
issues: write
258+
steps:
259+
- name: Open an issue, or comment on the one already open
260+
env:
261+
GH_TOKEN: ${{ github.token }}
262+
GH_REPO: ${{ github.repository }}
263+
LABEL: nightly-ci-failure
264+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
265+
run: |
266+
body="The nightly CI run on ${GITHUB_SHA} failed: ${RUN_URL}"
267+
open=$(gh issue list --label "${LABEL}" --state open --json number --jq '.[0].number // empty')
268+
if [[ -n "${open}" ]]; then
269+
gh issue comment "${open}" --body "${body}"
270+
else
271+
gh label create "${LABEL}" --color B60205 --description "The nightly CI run failed" --force
272+
gh issue create --title "Nightly CI failed" --label "${LABEL}" --body "${body}"
273+
fi

‎Makefile‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -196,10 +196,14 @@ generate-syntax-cases:
196196
uv run python -m scripts.build_syntax_cases $(SYNTAX_CASE_ARGS)
197197

198198
# Fails when a Snowflake obligations, withdrawn or elisions file differs from what the builder writes.
199-
# Only the families whose inputs changed since EVIDENCE_BASE are rebuilt (every family when the
200-
# builder changed); with none changed the target prints that it skipped them all and passes.
199+
# Only the families whose inputs changed since SYNTAX_OBLIGATIONS_SINCE are rebuilt (every family
200+
# when the builder changed); with none changed the target prints that it skipped them all and
201+
# passes. An empty SYNTAX_OBLIGATIONS_SINCE checks every family, as CI's nightly run does.
202+
SYNTAX_OBLIGATIONS_SINCE ?= $(EVIDENCE_BASE)
203+
201204
syntax-obligations:
202-
uv run python -m scripts.build_syntax_obligations_snowflake --check --since $(EVIDENCE_BASE) $(SYNTAX_FAMILIES)
205+
uv run python -m scripts.build_syntax_obligations_snowflake --check \
206+
$(if $(SYNTAX_OBLIGATIONS_SINCE),--since $(SYNTAX_OBLIGATIONS_SINCE)) $(SYNTAX_FAMILIES)
203207

204208
# Fails when a syntax case, mark or arity file differs from what the committed receipts regenerate.
205209
syntax-cases:

‎coverage/syntax/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,8 @@ python -m scripts.build_syntax_obligations_snowflake ddl query # obligations, w
4545
Rebuild a family's obligations whenever its catalogue, its rows in `limits.toml` or the builder
4646
changes. `make syntax-obligations`, part of `make check-ci`, fails when a committed obligations,
4747
withdrawn or elisions file differs from what the builder writes; it rebuilds only the families
48-
whose inputs or generated files changed since `EVIDENCE_BASE`, and passes when none did.
48+
whose inputs or generated files changed since `EVIDENCE_BASE`, and passes when none did. With
49+
`SYNTAX_OBLIGATIONS_SINCE=` (empty) it checks every family, as CI's nightly run does.
4950

5051
Pages with no Markdown export (the per-category command hubs) are fetched as HTML and catalogue
5152
no form. A rebuild keeps every existing id whose (page, grammar) is unchanged. A form whose

0 commit comments

Comments
 (0)