Skip to content

fix(cicd): reject incomplete boot outputs before upload - #4657

Open
chet wants to merge 1 commit into
NVIDIA:mainfrom
chet:gh-issue-4651
Open

fix(cicd): reject incomplete boot outputs before upload#4657
chet wants to merge 1 commit into
NVIDIA:mainfrom
chet:gh-issue-4651

Conversation

@chet

@chet chet commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

The reusable boot workflow used a warning-only file list, so a missing or empty required output could survive until an ephemeral-image or carrier job downloaded the bundle. This gives each active build and architecture an explicit contract and validates it at the producer before upload.

  • Expected green-run effect: Approximately neutral, with a small file-validation cost before upload.
  • What it really buys us: An incomplete boot bundle fails where it was produced, before we spend time and storage uploading it or start a downstream job that cannot succeed.

The four contracts cover x86 boot, ARM/BFB boot, x86 ephemeral, and ARM ephemeral production uploads. Required outputs must be nonempty regular files rather than symlinks, optional diagnostics stay optional, and the clean CI builders fail if the unused admin CLI appears. That last check catches a redundant full-workspace build that no downstream job needs.

Related issues

This supports #4651

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)
python3 -B -m unittest scripts.test_validate_boot_outputs
python3 -B -m unittest discover -s scripts -p 'test_validate_boot_outputs.py'
python3 -B scripts/validate_boot_outputs.py --help
cargo make format-nightly
cargo make clippy
cargo make carbide-lints

Additional Notes

Production upload is now success-only and uses if-no-files-found: error; the separate failure-only ephemeral build.log upload remains best-effort. Boot failures remain in Actions step logs, and partial production bundles are deliberately not retained. Artifact names, retention, compression, and downstream download paths are unchanged.

The unused artifacts_json output and its second hand-maintained path list were removed after a repository-wide and NVIDIA code search found no consumers. #4579 still owns checksums, architecture inspection, realized manifests, and consumer-side verification.

@chet
chet requested review from a team as code owners August 6, 2026 07:05
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Summary by CodeRabbit

  • New Features

    • Added validation for required boot and ephemeral build outputs across supported architectures.
    • Build artifacts are uploaded only after successful validation.
    • Missing, empty, disabled, symlinked, or invalid outputs now produce clear failures.
  • Bug Fixes

    • Prevented incomplete artifact sets from being uploaded.
  • Tests

    • Added automated coverage for output contracts and validation failure scenarios.
    • Integrated validation tests into continuous integration.

Walkthrough

Changes

The change adds output contracts for supported build types and architectures. It validates required files before upload, rejects invalid outputs, updates upload behavior, and adds unit-test coverage.

Boot output validation

Layer / File(s) Summary
Validation contract and CLI
scripts/validate_boot_outputs.py
Defines required, optional, and disabled outputs. Validates regular, non-empty files and reports status.
Validation test coverage
scripts/test_validate_boot_outputs.py
Tests contract classification, missing and empty files, disabled outputs, directories, symlinks, and CLI status handling.
Workflow enforcement
.github/workflows/build-boot-artifacts.yml, .github/workflows/ci.yaml
Runs validation before uploads, removes artifact metadata outputs, restricts uploaded paths, and runs validator tests during CI preparation.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

  • NVIDIA/infra-controller issue 4651: Covers producer-side boot-output validation and failure before upload.
  • NVIDIA/infra-controller issue 4579: Covers fail-closed boot-artifact validation before upload.

Possibly related PRs

  • NVIDIA/infra-controller#4513: Both changes modify boot-artifact validation and upload handling in .github/workflows/build-boot-artifacts.yml.

Sequence Diagram(s)

sequenceDiagram
  participant BuildWorkflow
  participant validate_boot_outputs.py
  participant BuildOutputs
  participant ArtifactUpload
  BuildWorkflow->>validate_boot_outputs.py: Validate build type, architecture, and repository root
  validate_boot_outputs.py->>BuildOutputs: Check required, optional, and disabled outputs
  BuildOutputs-->>validate_boot_outputs.py: Return filesystem validation results
  validate_boot_outputs.py-->>BuildWorkflow: Return success or failure status
  BuildWorkflow->>ArtifactUpload: Upload selected artifacts after successful validation
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the primary change: rejecting incomplete boot outputs before upload.
Description check ✅ Passed The description directly explains the validation contracts, upload behavior, removed outputs, testing, and intended fix.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@chet

chet commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

@chet: I will perform a complete review of this pull request.

✅ Action performed

Full review finished.

@chet chet changed the title fix: reject incomplete boot outputs before upload fix(cicd): reject incomplete boot outputs before upload Aug 6, 2026
The reusable boot workflow used a warning-only file list, so a missing or empty required output could survive until an ephemeral-image or carrier job downloaded the bundle. This gives each active build and architecture an explicit contract and validates it at the producer before upload.

- **Expected green-run effect:** Approximately neutral, with a small file-validation cost before upload.
- **What it really buys us:** An incomplete boot bundle fails where it was produced, before we spend time and storage uploading it or start a downstream job that cannot succeed.

Required outputs must be nonempty regular files, optional diagnostics stay optional, and the clean CI builders fail if the unused admin CLI appears, catching a redundant full-workspace build before it becomes normal again.

Tests added!

This supports NVIDIA#4651

Signed-off-by: Chet Nichols III <chetn@nvidia.com>

@nv-dmendoza nv-dmendoza left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but with 1 question on removal of an if

# `compression-level: 0` avoids re-deflating squashfs, BFB, and EFI files
# that are already compressed.
- name: Upload artifacts
if: ${{ !cancelled() }} # Upload artifacts even if build fails (matches GitLab CI 'when: always')

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we sure we want to upload artifacts always even on cancelled runs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants