RFC-0056: CRCR Support for Nightly & Periodic CI

[subinz1]

Summary

Proposes extending the Cross-Repository CI Relay (CRCR) to support nightly and periodic CI for downstream repositories using an authenticated self-report model.

Currently, CRCR only dispatches on pull_request and push events. Nightly/periodic runs have no upstream trigger — the state machine rejects callbacks without a prior DISPATCHED record. This RFC proposes a simpler path: downstream repos self-schedule via cron, fetch the target SHA, run CI, and report results directly to the relay in a single callback (no in_progress state, no Redis, no state machine).

Nightly Flow

The diagram below illustrates the nightly CI flow. Periodic CI follows the same pattern but fetches the SHA from main or viable/strict instead of the nightly branch. This will ensure that the results are aligned with upstream nightly results on HUD page.

flowchart TD
    A["⏰ Downstream cron schedule<br/>(e.g., daily 02:00 UTC)"] --> B["Fetch HEAD SHA from<br/>pytorch/pytorch nightly branch"]
    B --> C["Build & test PyTorch<br/>at that SHA"]
    C --> D["Single callback to relay<br/>dispatch_id = SHA<br/>event_type = nightly<br/>conclusion = success/failure"]
    D --> E{"Relay validates"}
    E -->|"OIDC token → repo on allowlist"| F["✅ Valid"]
    E -->|"SHA exists on pytorch/pytorch"| F
    F --> G["Direct upsert to DynamoDB<br/>(no Redis, no state machine)"]
    G --> H["DynamoDB → ClickHouse"]
    H --> I["📊 HUD display<br/>hud.pytorch.org/crcr"]

Loading

Callback model comparison

	PR / push (existing)	Nightly / periodic (new)
Trigger	Upstream webhook → relay dispatches	Downstream cron (self-triggered)
State machine	DISPATCHED → IN_PROGRESS → COMPLETED	No state machine — single callback
Callbacks	Two: in_progress then completed	One: completed only
Redis	Required	Not used
Validation	GitHub webhook signature	OIDC token + SHA existence

SHA sources

Event type	Branch	Source
nightly	nightly	Top-of-tree (updated daily by trigger_nightly_core.yml)
periodic	main or viable/strict	Top-of-tree on target branch

The dispatch_id is the commit SHA itself — idempotent, correlatable, and maps directly to github.com/pytorch/pytorch/commit/<sha> on HUD.

Replay & Recovery

Nightly/periodic callbacks are idempotent by design: the delivery_id is the upstream commit SHA, and the callback upserts into DynamoDB, so re-running the same workflow for the same SHA is safe and produces no duplicates.

Manual replay: Navigate to the downstream repo's Actions tab → select the nightly workflow → click "Run workflow" (workflow_dispatch). The workflow fetches the current nightly branch HEAD SHA, runs CI, and reports results via the callback action — identical to a cron-triggered run.

Automated missed-nightly detection (self-healing re-triggers) may be considered in a future iteration based on WG feedback.

Implementation PRs

• Phase 1: #8302 — Add nightly/periodic callback handler (callback Lambda)
• Phase 1.2: #8304 — SHA validator with TTL cache (utils)
• Phase 2: #8303 — Add delivery-id and event-type inputs to callback action (action YAML)
• Phase 3: TorchedHat/pytorch-redhat-ci#4 — CRCR nightly workflow (downstream cron) ✅ Merged

Related RFCs

• RFC-0050: Cross-Repository CI Relay (original dispatch + callback pipeline)
• RFC-0054: HUD Integration for Out-of-Tree CI Results

Status

Draft — for CRCR Working Group discussion.

[atalman]

Suggestion: use the pytorch/pytorch main HEAD commit SHA as the dispatch/correlation ID instead of a random uuid — and avoid any trigger from pytorch/pytorch.

Option 2 already fetches the main HEAD SHA here (step 1). That fetch is a pull via the GitHub API on the relay's own EventBridge schedule — it needs no workflow_dispatch/repository_dispatch/webhook emitted by pytorch/pytorch. So the relay never depends on an event from torch; it just reads GET /repos/pytorch/pytorch/commits/main. That's a strong reason to prefer Option 2 over Option 1 (Option 1's whole premise is torch emitting the event).

Given the SHA is in hand, make it the ID rather than nightly-{date}-{uuid}:

• Meaningful & correlatable — the ID is the tested commit, so HUD (and humans) can map a nightly run straight to github.com/pytorch/pytorch/commit/<sha>.
• No torch coupling — derived entirely on the relay side.

Uniqueness for the Redis state machine (DISPATCHED → IN_PROGRESS → COMPLETED): a bare SHA can collide if main hasn't advanced between two scheduled runs, or on a manual re-trigger. Two clean options:

1. SHA-keyed + idempotent (recommended): key on the bare SHA and skip re-dispatch if a DISPATCHED/COMPLETED record already exists for it. Bonus — you don't re-run nightlies when main didn't move.
2. Composite if you want exactly one nightly per calendar day regardless of main movement: nightly-{sha}-{YYYYMMDD} (or -{run_id}). Keeps the SHA as the correlation key while guaranteeing a fresh state key per run.

HUD grouping (see the "HUD Changes" note near the end): this also lets you avoid the pr_number = 0 sentinel, which collapses every nightly into one bucket. Carry the SHA (already in payload.head_sha) and group non-PR runs by SHA — each nightly stays distinguishable and links to its commit. pr_number = 0 throws away that per-commit granularity.

Net design: EventBridge cron → relay reads main HEAD SHA → delivery_id = SHA (idempotent) → HUD correlates by commit — no event from pytorch/pytorch anywhere in the path.

[jewelkm89]

Sure @atalman

Also noting the differences in terms of changes between the option 1 and option 2 for clarity

Option A: EventBridge → Lambda

1. Requires provisioning new AWS resources (EventBridge rules, targets, Lambda permissions, CloudWatch alarms) via change in Terraform.
2. Adds a new Lambda entry point and a GitHub API dependency to fetch the HEAD SHA.
3. Add Redis idempotency logic to avoid duplicate dispatches.
4. Schedule changes require a Terraform deployment.

Option B: Reuse Existing Events from pytorch/pytorch workflow

1. Adds a periodic trigger workflow in pytorch/pytorch
2. Adds the event types in the lambda.
3. Nightly uses the existing nightly branch push that already generates a webhook daily.
4. The Lambda maps the ref name to a dispatch type; the SHA, delivery ID, and signature verification are all used from the current workflow.

[jewelkm89]

Also added more in-depth detail in https://docs.google.com/document/d/1VQLi45b08x_fd9_ndACtp805EThyBKajbHCO_YtGygM/edit?tab=t.0

[atalman]

Following up on the trigger discussion — I'd like to put a third option on the table that I think fits the nightly/periodic use case better than either Option A (EventBridge cron) or Option B (reuse existing events), because it removes the scheduled trigger from the critical path entirely.

Option C: Authenticated self-report

Instead of a central scheduler dispatching to downstream repos, let each downstream repo drive its own schedule and report results back. The relay stops being a trigger and becomes a validating ingest endpoint.

downstream cron → runs CI against pytorch/pytorch HEAD
  → dispatch_id = the tested pytorch/pytorch commit SHA
  → calls back to the relay
  → relay accepts IF:  (1) originating repo is on the allowlist
                       (2) the SHA is a real pytorch/pytorch commit
  → upsert record keyed by (repo, SHA) → HUD

This drops the "callback requires a prior DISPATCHED record" precondition and replaces it with authorization + hash-validity at callback time.

Why this is attractive

• No trigger coupling anywhere — nothing in pytorch/pytorch emits an event, and there is no central EventBridge cron. Each downstream owns its own schedule (and gets workflow_dispatch manual re-trigger for free).
• No new AWS infrastructure — no EventBridge rules, no Terraform, no scheduler Lambda path.
• Coordination-free correlation — because the dispatch ID is the pytorch/pytorch HEAD SHA, any actor derives the same ID from the same commit with no central handoff. HUD groups by commit SHA (no pr_number=0 sentinel).

Two requirements to make it safe

1. Repo identity must be cryptographic, not self-declared. Once we accept unsolicited callbacks, the relay's only security boundary is proving the caller really is an allowlisted repo. Use the GitHub Actions OIDC token — the signed repository claim maps to the allowlist, is short-lived, and is audience-scoped (also kills replay). A repo name in the payload or a shared secret is spoofable and should not be accepted.
2. Validate the SHA exists in pytorch/pytorch (GET /repos/pytorch/pytorch/commits/{sha}, cached to avoid rate limits). Note this proves the SHA is real, not that CI actually ran against it — that's inherent to self-reporting and acceptable in a cooperative-partner model, but worth stating explicitly.

Explicit non-goal: missing-run detection

Unlike central dispatch, the relay never "expects" N callbacks, so it cannot tell "a repo's cron silently broke" from "a repo isn't participating." That's fine by design here — we only care about the signals that arrive; absence of a signal is not an error condition. This is what lets us drop all the completeness/liveness bookkeeping.

On SHA alignment

If we want rows from different backends to line up for side-by-side comparison, downstreams should pin the same stable daily ref (nightly / viable/strict) rather than resolving raw main (which drifts minute-to-minute). If each backend's signal stands on its own, this is optional. Composite key {sha}-{YYYYMMDD} only if we want to allow intentional re-runs on the same commit.

Net

This keeps the SHA-as-correlation-ID improvement already applied in this PR, and it satisfies "nightly/periodic must not require a workflow to be initiated near pytorch/pytorch" more completely than the reuse approach (which still needs a periodic trigger workflow in pytorch/pytorch). The cost is moving the trust boundary to OIDC-verified callbacks and accepting that missing signals are silent — both of which are acceptable for this use case.