Skip to content

Route signed-binary packages through a per-artifact release path - #48966

Draft
Daniel Jurek (danieljurek) wants to merge 25 commits into
mainfrom
djurek/bdist-sign-release
Draft

Daniel Jurek (danieljurek) wants to merge 25 commits into
mainfrom
djurek/bdist-sign-release

Conversation

@danieljurek

Copy link
Copy Markdown
Member

Packages that ship compiled binaries need ESRP-signed wheels before release, but signing was keyed off ServiceDirectory, which pulled every storage package onto the signed path. Make it a per-artifact opt-in instead.

  • An artifact sets signBinaries: true in its service ci.yml to get a Sign_<safeName> stage and release from packages_<safeName>_signed
  • Every other artifact releases from packages_extended off the build stage as before, so a service can mix both kinds
  • Release jobs move verbatim into release-artifact.yml so the two paths stay single-sourced
  • Signing runs on release and scheduled builds, never on PR builds
  • Integration publishes the signed wheels for opted-in artifacts

Daniel Jurek (danieljurek) and others added 20 commits July 23, 2026 09:54
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restores the azure-sdk-build-tools repository resource referenced by the
mac/win signing templates (@azure-sdk-build-tools), pinned to tag
azure-sdk-build-tools_20260702.2 to match azure-dev. This seeds a ref line
that the tools-repo-versioning auto-updater will keep current.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8225ba73-f36e-4078-b21a-a7a166d261f9
Restores the ESRP publish path (isolate + esrp-publish.yml) gated on
PublicFeed == 'PyPi'. Reads from packages_all_signed/<artifact>, so the
signed wheels/sdist are what get published.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8225ba73-f36e-4078-b21a-a7a166d261f9
ESRP Code Signing used minimatch pattern '*.pyd' (non-recursive) against
win-sign-input/, but extract_sign_inputs.py places .pyd in numbered
subfolders (00000/crc64.pyd). The glob matched 0 of 4 files, the task
reported success signing nothing, and repackage reinserted unsigned .pyd.
Mac was unaffected because it zips the payload dir and signs the zip.

Change WinPattern to '**/*.pyd' so ESRP recurses into the subfolders,
matching the mac path's recursive behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8225ba73-f36e-4078-b21a-a7a166d261f9
Uncomment the esrp-publish.yml template call that was temporarily disabled
during signing-pipeline testing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8225ba73-f36e-4078-b21a-a7a166d261f9
# Conflicts:
#	eng/pipelines/templates/stages/archetype-python-release.yml
#	sdk/storage/azure-storage-extensions/CHANGELOG.md
Packages that ship compiled binaries need their wheels signed by ESRP before
release, but the previous approach keyed signing off ServiceDirectory, which
dragged every storage package onto the signed path.

Make it a per-artifact opt-in instead. An artifact sets `signBinaries: true` in
its service ci.yml and gets a dedicated Sign_<safeName> stage plus a
Release_<safeName> stage fed from the resulting packages_<safeName>_signed
artifact. Every other artifact keeps releasing from packages_extended off the
build stage exactly as before, so a service can mix both kinds of packages.

The signed artifact mirrors the layout of packages_extended, so the release jobs
work against it unchanged. To keep those jobs single-sourced they move verbatim
into release-artifact.yml, which parameterizes only the artifact name and the
upstream stage. packages_extended itself is untouched.

Signing runs on release and scheduled builds and never on PR builds: the public
project cannot reach internal/azure-sdk-build-tools, and ESRP must not be handed
unreviewed code. Scheduled runs are included because the Integration stage now
publishes signed alpha packages to the dev feed for opted-in artifacts.

The signing stage deliberately ignores Skip.Release and SetDevVersion. Those
belong on the consumers, which already carry them; checking them here would
propagate through Integration's dependency and silently break the nightly and
manual dev version alpha publishes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9079636e-8478-4697-ae82-538136cd41b5
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions github-actions Bot added the Storage Storage Service (Queues, Blobs, Files) label Sep 10, 2026
Testing the scheduled path directly, rather than simulating it with a
queue-time SetDevVersion, showed the signing condition was only ever being
satisfied by its 'Manual' term. Two gaps that hid behind that:

The 'Schedule' term is coupled to daily-dev-build-variable.yml, which sets
SetDevVersion only when Build.Reason is exactly 'Schedule'. Alpha wheels exist
only when that is true, so the two must widen together. Record that, along with
why the reasons left out of the list need no signing.

A missing signed artifact skips one package's alpha publish by design, so a
signing failure cannot take down the dev feed publish for a whole service. Raise
it as a build issue rather than Write-Warning so that degradation shows up in
the run summary instead of only in the task log.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9079636e-8478-4697-ae82-538136cd41b5
ENABLE_EXTENSION_BUILD gated the mac and windows builds on a hardcoded
azure-storage-extensions name check, so any other package opting into
signBinaries got a signing stage with no wheels to sign. Resolve it from
the artifact property instead, which Save-Package-Properties already
copies into each package info file as ArtifactDetails.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9079636e-8478-4697-ae82-538136cd41b5
Adds a minimal C extension package so the template service exercises both
kinds of release artifact: azure-template stays pure Python and unsigned,
while azure-template-two ships platform wheels routed through signing via
the signBinaries artifact property.

The extension exposes a single is_true() function and is compiled against
the limited API, so one cp310 abi3 wheel per platform covers 3.10+.
cibuildwheel is configured for x86_64 Linux, x86_64 Windows and arm64
macOS.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9079636e-8478-4697-ae82-538136cd41b5
Doc publishing resolves exactly one wheel per package, which does not hold
for a package that ships one wheel per platform, so the namespace step fails
converting the match list to a string. azure-storage-extensions already opts
out of doc publishing for the same reason.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9079636e-8478-4697-ae82-538136cd41b5
@github-actions

This comment has been minimized.

Property lookups on a declared object parameter are case sensitive, unlike
the each-loop variable this stage was extracted from, so Artifact.safename
resolved to empty and every artifact compiled to a stage literally named
Release_. A single artifact service still compiled, but any service with two
or more artifacts failed with a duplicate stage name.

The same spelling was also used to look up the auto-release output variable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9079636e-8478-4697-ae82-538136cd41b5
@github-actions

Copy link
Copy Markdown
Contributor
[Pilot] PR Pipeline Failure Analysis

What failed

Build 6834566 — check python - pullrequest (Build Analyze) — failed on two independent validation steps, both triggered by the new sdk/template/azure-template-two package added in this PR:

1. CSpell spell-check (log /tmp/log-analysis-fcd81ec1407343c4aed539c0b0b99062.txt:232) — 17 unknown-word errors across 6 files in the new package, all variants of the identifiers chosen for the fixture (istrue, istruemodule, SSIZE):

  • sdk/template/azure-template-two/CHANGELOG.md:7:45
  • sdk/template/azure-template-two/README.md:10:32, 12:1
  • sdk/template/azure-template-two/azure/template/two/__init__.py:7:15, 9:13
  • sdk/template/azure-template-two/azure/template/two/src/istruemodule.c:4:12 (SSIZE), 14:20, 19:20, 21:6, 24:5, 28:8
  • sdk/template/azure-template-two/setup.py:38:14, 39:38
  • sdk/template/azure-template-two/tests/test_istrue.py:7:32, 11:12, 18:9, 24:12

2. README structure validation (ward scan) (log /tmp/log-analysis-de43af534c544db484b985552be5d4c6.txt:174) — sdk/template/azure-template-two/README.md is missing required headers:
^Azure (.+ client library for Python|Smoke Test for Python|AI Agent Server Adapter for .*Python), Getting started, Key concepts, Examples, Troubleshooting, Next steps.

The python - storage and python - template checks were cancelled as a side effect of the Build Analyze failure, not independent failures.

Relevant pipeline output
2026-09-14T21:53:10.1146325Z ##[error]sdk/template/azure-template-two/CHANGELOG.md:7:45 - Unknown word (istrue)
2026-09-14T21:53:10.1150159Z ##[error]sdk/template/azure-template-two/README.md:10:32 - Unknown word (istrue)
2026-09-14T21:53:10.1155880Z ##[error]sdk/template/azure-template-two/README.md:12:1 - Unknown word (istrue)
2026-09-14T21:53:10.1161750Z ##[error]sdk/template/azure-template-two/azure/template/two/__init__.py:7:15 - Unknown word (istrue)
2026-09-14T21:53:10.1168077Z ##[error]sdk/template/azure-template-two/azure/template/two/__init__.py:9:13 - Unknown word (istrue)
2026-09-14T21:53:10.1173160Z ##[error]sdk/template/azure-template-two/azure/template/two/src/istruemodule.c:4:12 - Unknown word (SSIZE)
2026-09-14T21:53:10.1179520Z ##[error]sdk/template/azure-template-two/azure/template/two/src/istruemodule.c:14:20 - Unknown word (istrue)
2026-09-14T21:53:10.1185496Z ##[error]sdk/template/azure-template-two/azure/template/two/src/istruemodule.c:19:20 - Unknown word (istrue)
2026-09-14T21:53:10.1191320Z ##[error]sdk/template/azure-template-two/azure/template/two/src/istruemodule.c:21:6 - Unknown word (istrue)
2026-09-14T21:53:10.1196524Z ##[error]sdk/template/azure-template-two/azure/template/two/src/istruemodule.c:24:5 - Unknown word (istrue)
2026-09-14T21:53:10.1202356Z ##[error]sdk/template/azure-template-two/azure/template/two/src/istruemodule.c:28:8 - Unknown word (istrue)
2026-09-14T21:53:10.1207780Z ##[error]sdk/template/azure-template-two/setup.py:38:14 - Unknown word (istrue)
2026-09-14T21:53:10.1213888Z ##[error]sdk/template/azure-template-two/setup.py:39:38 - Unknown word (istruemodule)
2026-09-14T21:53:10.1219004Z ##[error]sdk/template/azure-template-two/tests/test_istrue.py:7:32 - Unknown word (istrue)
2026-09-14T21:53:10.1224837Z ##[error]sdk/template/azure-template-two/tests/test_istrue.py:11:12 - Unknown word (istrue)
2026-09-14T21:53:10.1230765Z ##[error]sdk/template/azure-template-two/tests/test_istrue.py:18:9 - Unknown word (istrue)
2026-09-14T21:53:10.1254564Z ##[error]sdk/template/azure-template-two/tests/test_istrue.py:24:12 - Unknown word (istrue)
...
2026-09-14T21:55:02.0491070Z 1 readme has at least one missing required section.
2026-09-14T21:55:02.0491733Z /sdk/template/azure-template-two/README.md is missing headers with patterns:
2026-09-14T21:55:02.0492178Z  * ^Azure (.+ client library for Python|Smoke Test for Python|AI Agent Server Adapter for .*Python)
2026-09-14T21:55:02.0492557Z  * ^Getting started$
2026-09-14T21:55:02.0492878Z  * ^Key concepts$
2026-09-14T21:55:02.0493171Z  * ^Examples$
2026-09-14T21:55:02.0493482Z  * ^Troubleshooting$
2026-09-14T21:55:02.0493736Z  * ^Next steps$

Recommended next steps

  • Add istrue, istruemodule, and SSIZE to the repo cspell allow-list (typically .vscode/cspell.json) so the new fixture's identifiers are accepted, or rename the identifiers if a common word is preferred.
  • Update sdk/template/azure-template-two/README.md to include the required section headers (## Getting started, ## Key concepts, ## Examples, ## Troubleshooting, ## Next steps) and a title matching the expected client-library pattern, following the format used by other package READMEs (e.g. sdk/template/azure-template).
  • See https://aka.ms/ci-fix

Automated fix: Requested

Generated by Pipeline Analysis Next Steps · auto · 104.1 AIC · ⌖ 2.56 AIC · ⊞ 9.2K ·

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Storage Storage Service (Queues, Blobs, Files)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants