Route generation module installs through the private CFS feed (PS Gallery follow-up) [draft] - #3720
Draft
Gavin Barron (gavinbarron) wants to merge 17 commits into
Draft
Route generation module installs through the private CFS feed (PS Gallery follow-up) [draft]#3720Gavin Barron (gavinbarron) wants to merge 17 commits into
Gavin Barron (gavinbarron) wants to merge 17 commits into
Conversation
…hell Gallery follow-up) Phase 2 of the PowerShell CFSClean remediation: eliminate the CFSClean2-tier egress to www.powershellgallery.com / cdn.powershellgallery.com in the generation pipelines (187/221/663) by routing every generation-time Install-Module/Find-Module through the private Azure Artifacts feed PowerShell_V2_Build, which already has a PowerShell Gallery upstream (so it serves both the internally published Graph modules and public tooling modules). Mechanism: - Expose $(System.AccessToken) as a process-wide SYSTEM_ACCESSTOKEN variable in weekly-generation.yml, command-metadata-refresh.yml and ci-build.yml, so credentials are available to every step and to the ForEach-Object -Parallel runspaces used by GenerateModules.ps1. - install-tools.yml: new "Register private module feed" step registers PowerShell_V2_Build as a Trusted PSRepository (persisted for the job). - tools/Get-CfsFeedCredential.ps1: shared helper (Get-CfsFeedName / Get-CfsFeedCredential / Register-CfsFeed) that builds a PSCredential from SYSTEM_ACCESSTOKEN; returns $null locally so behaviour is unchanged off-CI. - Repoint sources from PSGallery to PowerShell_V2_Build and inject the credential via $PSDefaultParameterValues (runspace-local, so it works inside the parallel generation): ValidateUpdatedModuleVersion, GenerateRollUpModule, GenerateMetaModule, GenerateAuthenticationModule, BuildModule, Versions/BumpModuleVersion, plus the ad-hoc tooling installs (PlatyPS, Pester, powershell-yaml, PowerHTML) in GenerateHelp/TestModule/ImportExamples/ UpdateOpenApi. DRAFT: requires CI validation. Open questions to confirm in a pipeline run: - ValidateUpdatedModuleVersion's version gate now queries the private feed (with PSGallery upstream) instead of public PSGallery directly; confirm the published-version comparison still behaves as intended (the upstream proxies public versions, but the feed may also expose internally published versions). - Confirm www.powershellgallery.com / cdn.powershellgallery.com drop to 0 in the generation jobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d3f8fec7-b00b-46be-ba39-7e1f3e7f7188
The 'Register private module feed (CFSClean)' step failed with 'The name specified has already been added to the list of available package sources' because a PackageManagement source named PowerShell_V2_Build already exists in the build job (not surfaced by Get-PSRepository). The repository still ends up registered, so wrap Register-PSRepository in try/catch and treat the already-exists collision as success when Get-PSRepository confirms the repo is present. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
…lery-private-feed
The 'already added' error is non-terminating and bypasses -ErrorAction Stop (so try/catch never fires) yet fails the PowerShell task under its default Stop preference. Pre-check Get-PackageSource (surfaces the collision that Get-PSRepository lazily misses in a fresh session), and suppress all streams on Register-PSRepository with a post-verify instead of relying on the call to not error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
…d step The step still exited 1 despite a successful registration because PackageManagement emits benign non-terminating errors that trip the PowerShell task's default Stop preference. Set the task errorActionPreference to continue; Register-CfsFeed still throws (terminating) on genuine failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
Registration succeeds (repo is created + printed) but the step exited 1 because PowerShellGet's NuGet-provider bootstrap leaves a non-zero \0 that the task checks. Reset it at the end of the step; Register-CfsFeed throws on genuine failure before this line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
BuildModule.ps1 read the existing module GUID from Find-Module's AdditionalMetadata.GUID, which the private Azure Artifacts feed does not populate (public PS Gallery does), producing a null GUID and failing Update-ModuleManifest. Add Get-CfsModuleGuid: Save-Package the published nupkg from the feed (NuGet provider, no dependency resolution) and read GUID from its .psd1; fall back to a fresh GUID only when the module is unpublished. Keeps the GUID lock working without any public PS Gallery call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
…vate feed Generation failed installing PlatyPS: 'multiple modules matched platyPS. Please specify a single -Repository' because PSGallery and the private feed were both registered. Unregister PSGallery in the register step so the private feed (which has a PS Gallery upstream) is the sole source - eliminating install ambiguity and any residual public PS Gallery egress. Persists per-user for later generation steps/runspaces. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
Install PlatyPS/Pester/powershell-yaml/PowerHTML once in install-tools (controlled credential context) so the lazy, already-guarded Install-Module calls during generation are skipped via their Get-Module -ListAvailable checks. Avoids repeating fragile authenticated private-feed installs across the parallel generation runspaces. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
…m feed URL The GUID lock returned null in CI (random GUID -> 'Should lock GUID' test failure) because Get-CfsModuleGuid first did Find-Module -Repository PowerShell_V2_Build, which returns null in the generation runspace where the PSRepository registration isn't visible. Drop that lookup and Save-Package the latest package straight from the feed URL (no registration needed; GUID is version-independent). Verified locally in a fresh process with no repo registered: returns the correct locked GUID. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
…SClean2) Attempt to eliminate the 3 residual www.powershellgallery.com connections during the register step (CFSClean2 violations) by pre-seeding the NuGet package provider from the private feed before any PowerShellGet operation, falling back to importing the agent's existing provider. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
…y first - Get-CfsModuleGuid now downloads the published package over HTTP from the feed's NuGet v2 OData endpoint and reads the GUID from the .psd1, bypassing PowerShellGet/PackageManagement source resolution (which returns null in generation runspaces -> random GUID -> 'Should lock GUID' test failure). Verified locally. - Unregister-PublicPSGallery now runs FIRST (before any other PowerShellGet call) via the low-level Unregister-PackageSource, so PSGallery is gone before an enumeration resolves its location and egresses to www.powershellgallery.com (CFSClean2). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
…on step Add temporary diagnostics to Get-CfsModuleGuid (token presence, OData status, GUID match) and map env SYSTEM_ACCESSTOKEN into the Generate Authentication Module step - the generation steps did not expose the token, so Get-CfsModuleGuid returned null (random GUID -> lock test failure). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
Diagnostic strings used an unbraced variable before a colon, which PowerShell parses as a drive-qualified reference (InvalidVariableReferenceWithDrive), breaking the helper parse and the generation step. Use braces to delimit the name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
…eta generation - Connect-MgGraph.Tests.ps1 installed Az.Accounts from public PSGallery, which fails now that PSGallery is unregistered. Pre-install Az.Accounts from the private feed (tooling list) and guard the test install with Get-Module -ListAvailable so it resolves without PSGallery. - Map env SYSTEM_ACCESSTOKEN into the workload/meta generation + pack steps so Get-CfsModuleGuid and the private-feed installs have the build token (previously only install-tools steps exposed it). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
The provider pre-seed did not eliminate the PowerShellGet-init PS Gallery pings during the register step (and its failed private-feed bootstrap caused a PS Gallery module download), so remove it and keep only the unregister-first behaviour. Also strip the temporary Get-CfsModuleGuid diagnostics now that the HTTP GUID retrieval is confirmed working in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ea00b50-c719-469b-bf9c-72120184bfe0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Generation runs
Install-Module/Find-Moduleagainst the public PowerShell Gallery (www.powershellgallery.com+cdn.powershellgallery.com), which trips CFSClean2. Telemetry (e.g. 221 build 230408, 663 build 232382) shows these hits in every workload generation job.Approach
The private Azure Artifacts feed
PowerShell_V2_Buildalready has a PowerShell Gallery upstream (confirmed via the feed's upstream sources), so it can serve both the internally published Graph modules and the public tooling modules (Pester, PlatyPS, powershell-yaml, PowerHTML). Route all generation module operations through it.Changes
SYSTEM_ACCESSTOKEN: $(System.AccessToken)tovariables:inweekly-generation.yml,command-metadata-refresh.yml,ci-build.yml— so credentials are available to every step and to theForEach-Object -Parallelrunspaces inGenerateModules.ps1.install-tools.yml: new Register private module feed step registersPowerShell_V2_Buildas a TrustedPSRepository(persisted for the job).tools/Get-CfsFeedCredential.ps1(new): shared helper —Get-CfsFeedName,Get-CfsFeedCredential(builds aPSCredentialfromSYSTEM_ACCESSTOKEN; returns$nulllocally so off-CI behaviour is unchanged),Register-CfsFeed.PSGallery→PowerShell_V2_Build, injecting the credential via$PSDefaultParameterValues(runspace-local, so it works inside parallel generation):ValidateUpdatedModuleVersion,GenerateRollUpModule,GenerateMetaModule,GenerateAuthenticationModule,BuildModule,Versions/BumpModuleVersion, and the ad-hoc tooling installs inGenerateHelp/TestModule/ImportExamples/UpdateOpenApi.Validation (in CI)
www.powershellgallery.com/cdn.powershellgallery.com= 0 in the generation jobs (187/221/663).ValidateUpdatedModuleVersionnow queries the private feed (PSGallery upstream) rather than public PSGallery directly — confirm the published-version comparison still gates correctly (the upstream proxies public versions, but the feed may also surface internally published versions).Related
s360-cfs/pipeline-cfs-compliance-status.md.