Skip to content

Separate GCP test jobs and preserve coverage reporting - #129

Closed
danielmawhirter wants to merge 1 commit into
dm-test-credential-profilesfrom
dm-provider-test-routing
Closed

danielmawhirter wants to merge 1 commit into
dm-test-credential-profilesfrom
dm-provider-test-routing

Conversation

@danielmawhirter

@danielmawhirter danielmawhirter commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Depends on #128.

Summary

Tests that need Google Cloud currently share credentials with unrelated general and Snowflake shards. Route the selected GCP tests into dedicated jobs and remove unrelated Google authentication from general/Snowflake jobs. Preserve existing check names and run integrations for all changes except proven documentation-only changes, including when history is unavailable.

Three isolated GCP jobs in the main testing workflow retain the former general-suite coverage contribution. The launcher instruments the inner pytest process, uploads its parallel coverage files, and the existing combined report waits for these jobs to succeed. The GCS workflow retains its three existing integration rows and adds the previously omitted public API test. Only the BigQuery training row receives AWS credentials for its S3 dataplane.

Change map

File-by-file change map.

Testing Done

51 focused runner/workflow tests pass (37 runner, 14 workflow); the combined helper/profile/workflow suite independently passes 131 tests. Actionlint passes with shellcheck disabled; full shellcheck reports an existing SC2086 finding on an unchanged artifact-name command. An independent coverage smoke check combines general/GCP data with branch and multiprocessing settings intact. The node partition audit preserves all 154 nodes without overlap or omissions. Live authentication and permissions remain unverified.

General and Snowflake jobs retain transitional credentials for the remaining Azure/Databricks cases; this change does not yet narrow every hosted-suite provider identity. GCP provider-policy inspection was skipped at the user's direction; live GCP validation remains necessary before considering authentication failures resolved.

Current CI: all seven dedicated GCP groups reach authentication but Google rejects the token's attribute condition (representative job). Unrelated jobs run independently. No cloud policy changes were made; this draft is blocked on successful live federation.

Checklist

  • Described credential routing, failure behavior, and remaining migration limits.
  • Preserved existing check names and coverage publication policy.
  • Added workflow contracts, impact checks, and real subprocess coverage verification.

@copy-pr-bot

copy-pr-bot Bot commented Oct 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@danielmawhirter
danielmawhirter temporarily deployed to gha-kumo-workflow-tests October 27, 2026 20:17 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Oct 27, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@danielmawhirter

danielmawhirter commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

Change map

File Function / scope What changed Why
.github/workflows/testing.yml General pytest matrix, new pytest-gcp matrix, impact gate, coverage aggregator Remove Google authentication from general shards; run three isolated GCP groups; upload uniquely named parallel coverage data and wait for GCP success before aggregation. Google setup failures must not prevent unrelated test execution, and moving tests must preserve their contribution to the existing same-run coverage report.
.github/workflows/testing_snowflake.yml Snowflake matrix, setup gates, pytest commands Keep displayed labels and existing test selections, remove unrelated Google authentication, and use shared conservative impact and non-GCP selection. Snowflake tests do not need Google credentials; code changes outside narrow directory filters can still affect integrations.
.github/workflows/testing_gcs.yml Dedicated integration matrix and setup/authentication order Keep the three original labels, add public API integration coverage, authenticate after package setup, scope AWS to S3-backed BigQuery training, and make selected setup failures fail the job. Each dedicated group needs its declared credentials; authentication failures must be visible without masking them through continue-on-error.
test/scripts/test_ci_workflow_profiles.py Fourteen workflow contract tests Verify original labels, exact target ownership across both workflows, credential gates, conservative impact behavior, unique coverage artifacts, and aggregation dependency/success requirements. Prevent lost tests, dropped coverage, incidental check renames, or unrelated authentication dependencies during subsequent workflow edits.

Observed failure: Google authentication introduced into mixed general/Snowflake shards prevented unrelated pytest execution. Existing GCS job failures were hidden by continue-on-error. Moving GCP tests without report integration would also remove their measured code coverage.

Smallest sufficient change: retain existing workflows and displayed checks, move GCP execution into seven isolated groups (three main-workflow groups and four GCS integration groups), use the preceding profile launcher, and merge migrated coverage into the existing same-run report. No cloud-policy or repository coverage-policy changes are made.

Verification: 51 focused runner/workflow tests passed; root independently reran 131 combined focused tests. Actual child coverage measurement, the 154-node partition audit, flake8, isort, whitespace checks, and actionlint syntax validation passed. Full actionlint's sole SC2086 finding is pre-existing in the unchanged general artifact-name command; actionlint with shellcheck disabled passes. Live authentication is not verified.

Stopping condition: every inventoried GCP target has exactly one dedicated group, unrelated suites run independently, existing names remain stable, and migrated measured data joins the current report without publishing partial results after GCP failure. Azure/Databricks credential narrowing remains an explicit later task; provider-policy inspection was skipped at user direction.

@danielmawhirter

Copy link
Copy Markdown
Collaborator Author

Superseded by #224 and #225 after the GCP trust fix. Worth revisiting in smaller follow-ups: dedicated GCP-only and GCP+AWS jobs with preserved coverage and check names; and restoring the omitted public-API GCS integration test after fixing its prediction-job creation failure. Broader Azure/Databricks isolation also remains deferred. These improvements are separate from the minimal CI enablement and credential cleanup.

This branch was previously deployed

1 inactive deployment
gha-kumo-workflow-tests — 407894da Deployed Oct 27, 2026 by danielmawhirter via import #875
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant