[main] Update common Docker engineering infrastructure with latest

eng/docker-tools/CHANGELOG.md

@@ -4,6 +4,71 @@ All breaking changes and new features in `eng/docker-tools` will be documented i
 
 ---
 
+## 2026-10-05: Lifecycle metadata commands combined
+
+- Pull request: [#2252](https://github.com/dotnet/docker-tools/pull/2252)
+- Issues: [#2153](https://github.com/dotnet/docker-tools/issues/2153), [#2066](https://github.com/dotnet/docker-tools/issues/2066)
+
+ImageBuilder now uses one command, `attachLifecycleMetadata`, to add lifecycle metadata to images.
+The shared publish and cleanup templates use the new command. Repositories that only use these
+templates do not need to make any changes.
+
+Repositories that call ImageBuilder commands directly must use these replacements:
+
+| Old command | New command |
+| --- | --- |
+| `generateEolAnnotationDataForPublish` and `annotateEolDigests` | `attachLifecycleMetadata unsupported` |
+| `generateEolAnnotationDataForAllImages` and `annotateEolDigests` | `attachLifecycleMetadata all` |
+| `annotateEolDigests` with a data file | `attachLifecycleMetadata file` |
+| `waitForMarAnnotationIngestion` | Add `--wait-for-ingestion` to an `attachLifecycleMetadata` command |
+
+The publish job no longer creates lifecycle metadata data files before it attaches the metadata.
+Update any automation that reads those intermediate files.
+
+Lifecycle metadata can now be marked as internal. Internal metadata is never published.
+
+---
+
+## 2026-10-05: Image-level syndication
+
+- Issue: [#2240](https://github.com/dotnet/docker-tools/issues/2240)
+
+Syndication has moved from tag-level to image-level. All platform tags, shared
+tags, signatures, and referrers are replicated 1:1 during publishing.
+
+`Build` and `Post_Build` stages no longer create syndicated tags or manifest
+lists. Image info records the destination as `syndicatedRepo` instead of
+recording separate `syndicatedDigests`, because syndicated images have the same
+digests as the primary images. Custom destination tag names and selective
+platform syndication are no longer supported.
+
+---
+
+## 2026-08-10: Pre-ImageBuilder build customization
+
+Build pipeline templates now accept `customPreImageBuilderBuildSteps`. These steps run after
+ImageBuilder is available but before repository content is copied into the Linux ImageBuilder
+image. Repositories can use the hook to stage files into Docker build contexts, such as shared
+`eng/common` content required by Dockerfiles.
+
+---
+
+## 2026-07-28: Publish stage artifacts consolidated
+
+The Publish stage now uploads `$(Build.ArtifactStagingDirectory)` once as
+`publish-attempt-$(System.JobAttempt)`. This replaces these separate artifacts:
+
+- `image-info-final-$(System.JobAttempt)`
+- `eol-annotation-data-$(System.JobAttempt)`
+- `annotation-digests-<registry>-$(System.JobAttempt)`
+- `source-build-id`
+
+Consumers of those artifact names must download the consolidated artifact instead. Files retain
+their staging-directory paths, including `imageInfo/`, `eol-annotation-data/`,
+`annotation-digests/`, and `sourceBuildId/source-build-id.txt`.
+
+---
+
 ## 2026-06-11: Configurable per-registry referrer-lookup rate limit
 
 - Issue: [#2141](https://github.com/dotnet/docker-tools/issues/2141)

eng/docker-tools/DEV-GUIDE.md

@@ -302,7 +302,7 @@ The publish stage does more than just push images. Here's the sequence:
 5. **Wait for Doc Ingestion** — Ensures README changes are live
 6. **Merge & Publish Image Info** — Updates the versions repo with new image metadata
 7. **Ingest Kusto Image Info** — Sends telemetry to Kusto for analytics
-8. **Generate & Apply EOL Annotations** — Marks images with end-of-life dates
+8. **Attach Lifecycle Metadata** - `attachLifecycleMetadata unsupported` marks unsupported images with end-of-life dates
 9. **Post Publish Notification** — Creates GitHub issues/notifications about the publish
 
 ### Dry-Run Mode
@@ -389,6 +389,22 @@ To force a rebuild regardless of cache state, set the `noCache` parameter to `tr
 
 ## Common Customization Patterns
 
+### Pattern: Staging Files into Docker Build Contexts
+
+Use `customPreImageBuilderBuildSteps` to modify repository content immediately before the
+repository is copied into the Linux ImageBuilder image. For example, a repository can stage
+shared `eng/common` files next to Dockerfiles whose build contexts cannot access the repository
+root:
+
+```yaml
+customPreImageBuilderBuildSteps:
+- powershell: ./eng/Stage-EngCommon.ps1
+  displayName: Stage eng/common in Docker Build Contexts
+```
+
+The steps run once per Linux build job, after ImageBuilder is available and before the
+`Dockerfile.WithRepo` image is built.
+
 ### Pattern: Adding Build Arguments
 
 Pass Dockerfile `ARG` values via ImageBuilder:

eng/docker-tools/templates/jobs/build-images.yml

@@ -10,6 +10,8 @@ parameters:
   # Custom steps that run after ImageBuilder is set up but before the build starts.
   # Use for build-specific initialization (e.g., setting variables, additional setup).
   customBuildInitSteps: []
+  # Custom steps that modify repository content before it is copied into the ImageBuilder image.
+  customPreImageBuilderBuildSteps: []
   publishConfig: null
   versionsRepoRef: ""
   noCache: false
@@ -41,6 +43,7 @@ jobs:
       versionsRepoRef: ${{ parameters.versionsRepoRef }}
       cleanupDocker: true
       customInitSteps: ${{ parameters.customInitSteps }}
+      customPreImageBuilderBuildSteps: ${{ parameters.customPreImageBuilderBuildSteps }}
   - ${{ parameters.customBuildInitSteps }}
   - template: /eng/docker-tools/templates/steps/reference-service-connections.yml@self
     parameters:
@@ -140,4 +143,4 @@ jobs:
         displayName: Publish SBOM
         internalProjectName: ${{ parameters.internalProjectName }}
         publicProjectName: ${{ parameters.publicProjectName }}
-        condition: ne(variables['BuildImages.builtImages'], '')
+        condition: and(succeeded(), ne(variables['BuildImages.builtImages'], ''))

eng/docker-tools/templates/jobs/post-build.yml

@@ -11,7 +11,6 @@ jobs:
   variables:
     imageInfosSubDir: "/image-infos"
     imageInfosHostDir: "$(Build.ArtifactStagingDirectory)$(imageInfosSubDir)"
-    imageInfosContainerDir: "$(artifactsPath)$(imageInfosSubDir)"
     imageInfosOutputSubDir: "/output"
     sbomOutputDir: "$(Build.ArtifactStagingDirectory)/sbom"
   steps:
@@ -73,11 +72,10 @@ jobs:
         exit 0
       }
 
-      New-Item -ItemType Directory -Path $(imageInfosHostDir)$(imageInfosOutputSubDir) -Force
       $(runImageBuilderCmd) mergeImageInfo `
         --manifest $(manifest) `
-        $(imageInfosContainerDir) `
-        $(imageInfosContainerDir)$(imageInfosOutputSubDir)/image-info.json `
+        image-infos `
+        image-infos/output/image-info.json `
         $(manifestVariables)
     name: MergeImageInfoFiles
     displayName: Merge Image Info Files
@@ -89,7 +87,7 @@ jobs:
         condition: and(succeeded(), ne(variables['MergeImageInfoFiles.noImageInfos'], 'true'), ne(variables['Build.Reason'], 'PullRequest'))
         args: >-
           createManifestList
-          '$(imageInfosContainerDir)$(imageInfosOutputSubDir)/image-info.json'
+          'image-infos/output/image-info.json'
           --repo-prefix '${{ parameters.publishConfig.BuildRegistry.repoPrefix }}'
           --os-type '*'
           --architecture '*'

eng/docker-tools/templates/jobs/publish.yml

@@ -37,15 +37,15 @@ jobs:
       value: $[ replace(variables['System.PullRequest.SourceBranch'], 'refs/heads/', '') ]
   - name: imageInfoHostDir
     value: $(Build.ArtifactStagingDirectory)/imageInfo
-  - name: imageInfoContainerDir
-    value: $(artifactsPath)/imageInfo
   - name: sourceBuildIdOutputDir
     value: $(Build.ArtifactStagingDirectory)/sourceBuildId
   - name: commitOverrideArg
     ${{ if eq(parameters.overrideImageInfoCommit, true) }}:
       value: --commit-override $(Build.SourceVersion)
     ${{ else }}:
       value: ''
+  - name: eolAnnotationsWaitArg
+    value: $[ iif(eq(variables['waitForIngestionEnabled'], 'true'), '--wait-for-ingestion', '') ]
   - ${{ parameters.customPublishVariables }}
 
   steps:
@@ -99,7 +99,7 @@ jobs:
 
   - script: >
       $(runImageBuilderCmd) trimUnchangedPlatforms
-      '$(imageInfoContainerDir)/image-info.json'
+      'imageInfo/image-info.json'
     displayName: Trim Unchanged Images
 
   - template: /eng/docker-tools/templates/steps/run-imagebuilder.yml@self
@@ -113,23 +113,15 @@ jobs:
         --os-type '*'
         --architecture '*'
         --repo-prefix '${{ parameters.publishConfig.PublishRegistry.repoPrefix }}'
-        --image-info '$(imageInfoContainerDir)/image-info.json'
+        --image-info 'imageInfo/image-info.json'
         $(dryRunArg)
         $(imageBuilder.pathArgs)
         $(imageBuilder.commonCmdArgs)
 
-  - template: /eng/docker-tools/templates/steps/publish-artifact.yml@self
-    parameters:
-      path: $(imageInfoHostDir)
-      artifactName: image-info-final-$(System.JobAttempt)
-      displayName: Publish Image Info File Artifact
-      internalProjectName: ${{ parameters.internalProjectName }}
-      publicProjectName: ${{ parameters.publicProjectName }}
-
   - template: /eng/docker-tools/templates/steps/wait-for-mcr-image-ingestion.yml@self
     parameters:
       publishConfig: ${{ parameters.publishConfig }}
-      imageInfoPath: '$(imageinfoContainerDir)/image-info.json'
+      imageInfoPath: 'imageInfo/image-info.json'
       minQueueTime: $(imageQueueTime)
       dryRunArg: $(dryRunArg)
       condition: succeeded()
@@ -139,9 +131,6 @@ jobs:
       dryRunArg: $(dryRunArg)
       condition: and(succeeded(), eq(variables['publishReadme'], 'true'))
 
-  - script: mkdir -p $(Build.ArtifactStagingDirectory)/eol-annotation-data
-    displayName: Create EOL Annotation Data Directory
-
   - script: |-
       cd $(versionsRepoRoot)
       git pull origin $(gitHubVersionsRepoInfo.branch)
@@ -155,13 +144,13 @@ jobs:
 
   - script: >
       $(runImageBuilderCmd) mergeImageInfo
-      $(imageInfoContainerDir)
-      $(imageInfoContainerDir)/full-image-info-new.json
+      imageInfo
+      imageInfo/full-image-info-new.json
       $(manifestVariables)
       $(dryRunArg)
       --manifest $(manifest)
       --publish
-      --initial-image-info-path $(imageInfoContainerDir)/full-image-info-orig.json
+      --initial-image-info-path imageInfo/full-image-info-orig.json
       $(commitOverrideArg)
     condition: and(succeeded(), eq(variables['publishImageInfo'], 'true'))
     displayName: Merge Image Info
@@ -178,7 +167,7 @@ jobs:
       condition: and(succeeded(), eq(variables['ingestKustoImageInfo'], 'true'))
       args: >-
         ingestKustoImageInfo
-        '$(imageInfoContainerDir)/image-info.json'
+        'imageInfo/image-info.json'
         '$(kusto.cluster)'
         '$(kusto.database)'
         '$(kusto.imageTable)'
@@ -190,36 +179,26 @@ jobs:
 
   - template: /eng/docker-tools/templates/steps/run-imagebuilder.yml@self
     parameters:
-      displayName: Generate EOL Annotation Data
+      displayName: Attach Lifecycle Metadata
+      serviceConnections:
+      - name: mar
+        id: $(marStatus.serviceConnection.id)
+        tenantId: $(marStatus.serviceConnection.tenantId)
+        clientId: $(marStatus.serviceConnection.clientId)
       internalProjectName: internal
       condition: and(succeeded(), eq(variables['publishEolAnnotations'], 'true'))
       args: >-
-        generateEolAnnotationDataForPublish
+        attachLifecycleMetadata unsupported
         '${{ parameters.publishConfig.PublishRegistry.server }}'
         '${{ parameters.publishConfig.PublishRegistry.repoPrefix }}'
-        '$(artifactsPath)/eol-annotation-data/eol-annotation-data.json'
-        '$(imageInfoContainerDir)/full-image-info-orig.json'
-        '$(imageInfoContainerDir)/full-image-info-new.json'
-        $(generateEolAnnotationDataExtraOptions)
+        'imageInfo/full-image-info-orig.json'
+        'imageInfo/full-image-info-new.json'
+        $(eolAnnotationsWaitArg)
         $(dryRunArg)
 
-  - template: /eng/docker-tools/templates/steps/publish-artifact.yml@self
-    parameters:
-      path: $(Build.ArtifactStagingDirectory)/eol-annotation-data
-      artifactName: eol-annotation-data-$(System.JobAttempt)
-      displayName: Publish EOL Annotation Data Artifact
-      internalProjectName: internal
-      publicProjectName: public
-      condition: and(succeeded(), eq(variables['publishEolAnnotations'], 'true'))
-
-  - template: /eng/docker-tools/templates/steps/annotate-eol-digests.yml@self
-    parameters:
-      acr: ${{ parameters.publishConfig.PublishRegistry }}
-      dataFile: $(artifactsPath)/eol-annotation-data/eol-annotation-data.json
-
   - script: >
       $(runImageBuilderCmd) publishImageInfo
-      '$(imageInfoContainerDir)/full-image-info-new.json'
+      'imageInfo/full-image-info-new.json'
       '$(gitHubVersionsRepoInfo.userName)'
       '$(gitHubVersionsRepoInfo.email)'
       $(gitHubVersionsRepoInfo.authArgs)
@@ -248,7 +227,7 @@ jobs:
       $(runImageBuilderCmd) postPublishNotification
       '$(publishNotificationRepoName)'
       '$(branchName)'
-      '$(imageInfoContainerDir)/image-info.json'
+      'imageInfo/image-info.json'
       $(Build.BuildId)
       '$(System.AccessToken)'
       '$(azdoOrgName)'
@@ -263,9 +242,7 @@ jobs:
       --task "🟪 Wait for MCR Doc Ingestion"
       --task "🟪 Publish Image Info"
       --task "🟪 Ingest Kusto Image Info"
-      --task "🟪 Generate EOL Annotation Data"
-      --task "🟪 Annotate EOL Images (${{ parameters.publishConfig.PublishRegistry.server }})"
-      --task "🟪 Wait for Annotation Ingestion (${{ parameters.publishConfig.PublishRegistry.server }})"
+      --task "🟪 Attach Lifecycle Metadata"
       $(dryRunArg)
       $(imageBuilder.commonCmdArgs)
     displayName: Post Publish Notification
@@ -285,8 +262,10 @@ jobs:
 
   - template: /eng/docker-tools/templates/steps/publish-artifact.yml@self
     parameters:
-      path: $(sourceBuildIdOutputDir)
-      artifactName: source-build-id
-      displayName: Publish Source Build ID Artifact
+      path: $(Build.ArtifactStagingDirectory)
+      artifactName: publish-attempt-$(System.JobAttempt)
+      displayName: Publish Artifacts
       internalProjectName: ${{ parameters.internalProjectName }}
       publicProjectName: ${{ parameters.publicProjectName }}
+      # Always upload, even if the pipeline fails or is canceled.
+      condition: always()

eng/docker-tools/templates/stages/build-and-test.yml

@@ -13,6 +13,8 @@ parameters:
   # Custom steps that run after ImageBuilder is set up but before the build starts.
   # Use for build-specific initialization (e.g., setting variables, additional setup).
   customBuildInitSteps: []
+  # Custom steps that modify repository content before it is copied into the ImageBuilder image.
+  customPreImageBuilderBuildSteps: []
   customTestInitSteps: []
   sourceBuildPipelineRunId: ""
   # When true, the Post-Build stage runs even if the Build stage failed (succeededOrFailed).
@@ -113,6 +115,7 @@ stages:
       versionsRepoRef: ${{ parameters.versionsRepoRef }}
       customInitSteps: ${{ parameters.customInitSteps }}
       customBuildInitSteps: ${{ parameters.customBuildInitSteps }}
+      customPreImageBuilderBuildSteps: ${{ parameters.customPreImageBuilderBuildSteps }}
       noCache: ${{ parameters.noCache }}
       publishConfig: ${{ parameters.publishConfig }}
       internalProjectName: ${{ parameters.internalProjectName }}
@@ -128,6 +131,7 @@ stages:
       versionsRepoRef: ${{ parameters.versionsRepoRef }}
       customInitSteps: ${{ parameters.customInitSteps }}
       customBuildInitSteps: ${{ parameters.customBuildInitSteps }}
+      customPreImageBuilderBuildSteps: ${{ parameters.customPreImageBuilderBuildSteps }}
       noCache: ${{ parameters.noCache }}
       publishConfig: ${{ parameters.publishConfig }}
       internalProjectName: ${{ parameters.internalProjectName }}
@@ -143,6 +147,7 @@ stages:
       versionsRepoRef: ${{ parameters.versionsRepoRef }}
       customInitSteps: ${{ parameters.customInitSteps }}
       customBuildInitSteps: ${{ parameters.customBuildInitSteps }}
+      customPreImageBuilderBuildSteps: ${{ parameters.customPreImageBuilderBuildSteps }}
       noCache: ${{ parameters.noCache }}
       publishConfig: ${{ parameters.publishConfig }}
       internalProjectName: ${{ parameters.internalProjectName }}
@@ -158,6 +163,7 @@ stages:
       versionsRepoRef: ${{ parameters.versionsRepoRef }}
       customInitSteps: ${{ parameters.customInitSteps }}
       customBuildInitSteps: ${{ parameters.customBuildInitSteps }}
+      customPreImageBuilderBuildSteps: ${{ parameters.customPreImageBuilderBuildSteps }}
       noCache: ${{ parameters.noCache }}
       publishConfig: ${{ parameters.publishConfig }}
       internalProjectName: ${{ parameters.internalProjectName }}
@@ -173,6 +179,7 @@ stages:
       versionsRepoRef: ${{ parameters.versionsRepoRef }}
       customInitSteps: ${{ parameters.customInitSteps }}
       customBuildInitSteps: ${{ parameters.customBuildInitSteps }}
+      customPreImageBuilderBuildSteps: ${{ parameters.customPreImageBuilderBuildSteps }}
       noCache: ${{ parameters.noCache }}
       publishConfig: ${{ parameters.publishConfig }}
       internalProjectName: ${{ parameters.internalProjectName }}
@@ -188,6 +195,7 @@ stages:
       versionsRepoRef: ${{ parameters.versionsRepoRef }}
       customInitSteps: ${{ parameters.customInitSteps }}
       customBuildInitSteps: ${{ parameters.customBuildInitSteps }}
+      customPreImageBuilderBuildSteps: ${{ parameters.customPreImageBuilderBuildSteps }}
       noCache: ${{ parameters.noCache }}
       publishConfig: ${{ parameters.publishConfig }}
       internalProjectName: ${{ parameters.internalProjectName }}
@@ -203,6 +211,7 @@ stages:
       versionsRepoRef: ${{ parameters.versionsRepoRef }}
       customInitSteps: ${{ parameters.customInitSteps }}
       customBuildInitSteps: ${{ parameters.customBuildInitSteps }}
+      customPreImageBuilderBuildSteps: ${{ parameters.customPreImageBuilderBuildSteps }}
       noCache: ${{ parameters.noCache }}
       publishConfig: ${{ parameters.publishConfig }}
       internalProjectName: ${{ parameters.internalProjectName }}