ci: automate SynapseML releases for Spark 3.5 and 4.1

[Rana Singh (ranadeepsingh)]

Summary

Automate repeatable public SynapseML releases for master / Spark 3.5 and
spark4.1. Spark 4.0 remains an explicit opt-in.

What this automates

• Prepare version changes, release branches and versioned documentation.
• Bind plans to reviewed commits, create immutable tags, and coordinate
  publication with a durable release ledger.
• Generate runtime-matched Databricks notebook archives from each approved
  source commit. Strip saved outputs and verify every notebook through native
  Databricks export and reimport.
• Publish DBCs without overwrite to the existing public mmlspark/dbcs
  container. Check anonymous downloads before PyPI and ESRP publication, then
  include archive hashes in producer receipts and verified release notes.
• Keep existing approved schema-2 plans unchanged. New schema-4 plans explicitly
  approve notebook publication and require archive evidence for completion.
• Support approved pre-merge trials, read-only dry runs and recovery after
  partial or ambiguous submissions. Private integrations are not prerequisites.
• Refuse release-note publication on failed GitHub lookups; only a confirmed
  HTTP 404 authorizes creating a missing GitHub Release.

For 1.2.0, new plans require SynapseMLExamplesv1.2.0.dbc and
SynapseMLExamplesv1.2.0-spark4.1.dbc. Opting into Spark 4.0 adds its matching
archive. Future versions derive their archive names from the approved Maven
versions. This addresses the recurring missing archives in
#2346.

The operator guide
documents access prerequisites, approval, notebook preflight and recovery.
Plan approval, signing approval and PR merges remain human actions.
Default version bumps leave optional Spark 4.0 examples on their last release.

Validation and limits

The DBC integration was rebased onto master bccec7e73d. Range comparison
preserved all four earlier PR commits.

• 954 Linux release-tooling tests passed. One optional live SBT check was
  skipped. The current-checkout notebook test passed separately with Windows
  Git because WSL Git cannot open Windows-created worktree pointers.
• All 12 installation-documentation tests passed, and changed Python follows
  pinned Black 22.3.0. An earlier pipeline-only run passed 124 checks.
• Real Databricks Workspace APIs round-tripped all 56 current notebooks,
  preserving all 1,108 nonempty commands. This used a synthetic 0.0.0 archive
  without blob publication or production tags.
• Four available independent review rounds completed and all 13 findings were
  addressed. Two unavailable Gemini rounds were skipped at maintainer request.
  Review evidence
  records findings, fixes and limitations.

Notebook code was not executed on Databricks compute. Native archive
compatibility is not proof that every example runs without service setup.
The integration is pushed as f4b9473e5751d9ec9c157ac34565405952c17f61.
Azure build 238110760
has started for this head after /azp run. Hosted validation is still pending;
earlier-head checks do not validate this integration. Spark 4.1 PR compatibility
replay is advisory, but actual release candidates still require runtime-specific
validation.

Before publishing 1.2.0

This work has not published 1.2.0. Production publication remains blocked
until Python distribution works on both selected runtimes.

The primary PyPI wheel contains master's Python code, while spark4.1 has
additional compatibility fixes. Port-built wrapper tests do not validate the
primary wheel. Select and validate a cross-runtime wheel or matching
runtime-specific wheels, then update consumer instructions and inventory.
This DBC integration adds notebook publication to new approved plans; it does
not resolve the consumer-wheel packaging blocker.

Keep the PR unmerged until the requested pre-merge release trial is complete.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR adds release-engineering automation to make SynapseML-to-Fabric releases more deterministic and reviewable, including a GitHub Actions “prepare → tag merged commit → publish notes” flow plus supporting Python tooling to derive/verify release identifiers and safely bump BBC-VHD pins.

Changes:

• Introduces a release matrix generator (release_matrix.py) and an end-to-end verifier (verify_release.py) with accompanying tests.
• Adds bump_bbcvhd.py (plus tests) to safely update BBC-VHD component pins while preserving line endings and rolling back on failure.
• Adds/updates GitHub workflows for release prepare/tagging and manual, artifact-gated release notes; enables workflow_dispatch for PR validation so release PRs created via GITHUB_TOKEN can be validated.

Show a summary per file

File	Description
scripts/test_bump_version.py	Extends bump-version tests for denylisted repo-relative paths; normalizes path keys to POSIX.
scripts/bump-version.py	Improves Windows console encoding robustness; adds path-based denylist support; normalizes path handling via as_posix().
scripts/release/release_matrix.py	Adds a single-source-of-truth release matrix for tags and artifact versions + CLI rendering.
scripts/release/verify_release.py	Adds live verification of GitHub/ADO tags and Maven/PyPI/UPack/Azure Artifacts presence.
scripts/release/bump_bbcvhd.py	Adds deterministic, rollback-safe BBC-VHD component pin updater with CRLF/LF preservation.
scripts/release/README.md	Documents how to use the new release tooling and where it fits in the guide.
scripts/release/test_release_matrix.py	Adds regression/contract tests for the release matrix derivations and CLI validation.
scripts/release/test_verify_release.py	Adds unit tests for verifier networking/error handling and run plan behavior.
scripts/release/test_bump_bbcvhd.py	Adds tests for BBC-VHD bump idempotency, rollback, and newline preservation.
scripts/release/test_release_workflows.py	Adds tests asserting key workflow contract properties (manual gating, dispatch behavior).
scripts/release/test_prev_tag.sh	Adds a repo-taglist regression script for “previous primary tag” selection logic.
.github/workflows/release-prepare.yml	Adds a reviewed release PR generator and a merged-commit tagger that dispatches downstream orchestration.
.github/workflows/release-notes.yml	Adds a manual, artifact-gated GitHub Release publisher with explicit previous-tag selection.
.github/workflows/pr-validation.yml	Enables workflow_dispatch so validations can be dispatched for bot-opened release PR branches.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Suppressed comments (1)

scripts/release/verify_release.py:260

• --skip internal does not currently skip the internal PyPI feed checks (the synapseml_internal package); it only skips internal git tag checks. This makes --skip internal behave inconsistently compared with --skip public.

    def pip(self, package: str, version: str) -> str:
        if "pip" in self.skip or "ado" in self.skip:
            return SKIPPED
        # Azure Artifacts normalises pypi names: synapseml_internal -> synapseml-internal
        return (
            OK
            if version
            in self._feed_versions("Synapse-Conda", "pypi", package.replace("_", "-"))
            else MISSING
        )

• Files reviewed: 14/14 changed files
• Comments generated: 2
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/release/verify_release.py:75

• On Windows, subprocess.run(..., shell=True) expects a command string, but this code passes a list. That combination can fail (or behave differently) on win32, which would break the fallback ADO auth path even when az is installed. Build a platform-appropriate command (string for shell=True, list otherwise).

    out = subprocess.run(
        [
            "az",
            "account",
            "get-access-token",
            "--resource",
            ADO_RESOURCE,
            "--query",
            "accessToken",
            "-o",
            "tsv",
        ],
        capture_output=True,
        text=True,
        shell=(sys.platform == "win32"),
    )

• Files reviewed: 14/14 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Addressed the current-head suppressed Windows auth finding in 6e80f27.

• Windows now gives subprocess.run(..., shell=True) a command string; non-Windows retains an argument list with shell=False.
• Parameterized regression coverage locks both command shapes.
• A token-less Windows fallback replay authenticated through �z and found all 12 selected v1.1.3 master artifacts; the complete release-tooling suite is 279 passed.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/release/verify_release.py:313

• The Maven publication gate here verifies only synapseml-core_{scala} (via public_maven()), but the workflow-generated release notes advertise com.microsoft.azure:synapseml_2.12:${VERSION} (see .github/workflows/release-notes.yml:137) and the docs use the synapseml_2.12 coordinate (website/doctest.py:25). This can let verify_release.py report COMPLETE even if the primary install coordinate is missing (or vice versa). Consider aligning the verification to check the same Maven artifact(s) that users are instructed to consume (e.g., verify both synapseml_{scala} and synapseml-core_{scala}, or switch to the canonical one).

            "maven",
            tp.key,
            f"synapseml-core_{tp.scala}",
            tp.oss_maven_version,
            c.public_maven(tp.scala, tp.oss_maven_version),

• Files reviewed: 14/14 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved release-safety findings remain in release-note error handling and immutable-container validation.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Fail closed for unknown blob containers · New

Resolved since last review (8)

• Maven-only plans trigger unauthorized non-Maven publication tasks
• Required staging arguments break the Spark 4.1 release pipeline
• Reject stale versioned files during ESRP staging
• Optional skipped dependencies can prevent release publication
• Review artifact contains checkout-specific commands
• Review artifact contains an absolute machine-local path
• Review artifact exposes an absolute local session path
• Review artifact uses a session-workspace path

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🟡 Changes recommended

Three critical workflow issues and one moderate test-coverage issue remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3

Open (3)

• Empty tag list aborts release-note generation · New
• Unverified SBT launcher enables arbitrary code execution · New
• Primary tag check does not authorize merged release content · New

Resolved since last review (1)

• Fail closed for unknown blob containers

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🔵 Needs a closer look

Address the two moderate release-test coverage gaps before approval.

Review effort: Lite
Findings: None

Resolved since last review (3)

• Primary tag check does not authorize merged release content
• Unverified SBT launcher enables arbitrary code execution
• Empty tag list aborts release-note generation

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🔵 Needs a closer look

Unresolved release-workflow guard issues remain, and production publication is explicitly blocked pending Python wheel validation.

Review effort: Lite
Findings: 1

Open (1)

• Check all derivative tags before creating a new preparation PR · New

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🔵 Needs a closer look

Unresolved moderate release-safety findings remain in the guarded publication and documentation paths.

Review effort: Lite
Findings: None

Resolved since last review (1)

• Check all derivative tags before creating a new preparation PR

Previously missed (1)

In code that hasn't changed since last review

Distinguish missing releases from GitHub errors

.github/​workflows/​release-notes.yml:125

gh release view returns nonzero for both a confirmed missing release and transport/authentication failures. This if treats every failure as found=false, so a transient GitHub error can fall through to note generation and a write attempt instead of failing closed (and can race with an existing release that could not be read). Distinguish a confirmed 404 from other failures and stop on all other errors before publishing.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🟡 Changes recommended

Release-note integration, failure handling, and rerun idempotence issues remain unresolved, alongside the documented wheel-validation release blocker.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Accept squash/rebase merge commits in release verification · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

Moderate unresolved findings remain, CI is not green, and final human review is required.

Review effort: Lite
Findings: 1

Open (1)

• Treating all release lookup failures as release-not-found · New

Resolved since last review (1)

• Accept squash/rebase merge commits in release verification

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🔵 Needs a closer look

CI is not green, and the release trial and cross-runtime publication validation remain outstanding.

Review effort: Lite
Findings: None

Resolved since last review (1)

• Treating all release lookup failures as release-not-found

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved archive traversal, dependency pinning, and release-note integrity findings block safe approval.

Review effort: Lite
Findings: 1

Open (1)

• Directory entries bypass archive traversal validation · New