Add agentic workflows that keep Microsoft.Extensions.AI aligned with OpenTelemetry GenAI semantic-conventions#7611
Open
jeffhandley wants to merge 7 commits into
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…scans Move ownership of upstream-scan tracking-PR maintenance out of the skill and onto the maintaining workflow: the skill no longer documents a refresh procedure and instead continues past the tracking PR during its preflight. Switch the tracking-state block delimiters from HTML comments to fenced yaml comments (# otel-genai-tracking:begin/end) so GitHub Actions safe-output processing, which strips HTML comments from PR bodies, preserves them. List every open upstream PR in the in-flight applicability table, and clarify that the defer-constant rule applies only to brand-new attributes with no emission site, not to changes to already-emitted conventions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a plain (non-agentic) orchestration workflow whose discovery job resolves the open-telemetry/semantic-conventions-genai ref to scan and emits a single self-contained integration target. The discovery script resolves the requested upstream ref (or the default-branch HEAD) to a commit SHA with retries and hard-fails rather than emitting an empty target. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the agentic worker and wire the orchestrator's produce job to invoke it as a reusable workflow (workflow_call) for a single discovery target, so it runs in the orchestrator's run context and inherits the orchestrator's actor -- satisfying the worker's activation role check -- and grant it the contents, pull-requests, issues, and discussions writes it requests. A dry_run dispatch input gates the produce job so a manual run can compute and print the target without invoking the worker. A host setup step resolves the target, discovers and classifies the maintained draft PR (ours / blocked / none), reads the recorded scan state, and computes the recommended lifecycle action, writing target.json for the agent to consume. The agent applies the update-otel-genai-conventions skill to produce the integration and maintains the tracking PR through native gh-aw safe outputs (create, push, update, comment, mark-ready, no-op). A post-step validates that every full-body PR write carries the tracking identity before it can publish. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Let a person hand-start an Otel GenAI integration PR that the automation then takes over and maintains. When a PR on the evergreen branch is labeled automation + area-ai but carries no tracking marker yet, classify it as adopt: the worker writes the tracking block into its body on this run and maintains it thereafter, exactly as if it had opened the PR itself. The automation + area-ai labels are the opt-in -- any other PR occupying the branch stays human-owned and blocks the automation. Recompiled meai-otel-genai-worker.lock.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extend the worker to address reviewer feedback on the maintained draft PR on the next scheduled run, without triggering on review events. The agent collects the PR's submitted reviews and inline review comments itself through GitHub integrity filtering: tools.github sets min-integrity: approved, and features.integrity-reactions lets a maintainer promote an external contributor's comment to approved with an endorsement reaction. Only write-access and endorsed feedback reaches the agent, so it applies no host-side trust handling of its own. The host setup records a feedback-processed-through watermark and this run's run_started_at in target.json, and computes a body-free wake gate that counts review activity newer than the watermark so a caught-up PR still wakes the agent when new review activity exists. If any of the three activity queries fails, the gate opens automatically, keeping a transient API error from masking real, unprocessed feedback. The agent acts on feedback created after the watermark, folds in-scope feedback into the differential commit(s), rejects out-of-scope requests, resolves contradictions by recency, and advances the watermark to run_started_at. Fetch the maintained PR body with a three-attempt retry and gate the watermark read and wake gate on a successful read, so a transient blip never drops the tracking marker and replays feedback from an empty watermark. If a persistent failure still leaves the PR classified ours with an empty recorded scan ref and watermark, the agent re-reads the body itself once to recover them before scoping feedback or computing the differential, and if that read also fails treats the watermark as this run's start rather than reprocessing every earlier comment. Strip trailing inline # comments when reading the tracking block's feedback-processed-through, upstream-scan-ref, and upstream-release values. Require every iterative update to the maintained PR -- the incremental, feedback-only, and release mark-ready paths -- to fully regenerate the PR description via update-pull-request so it always reflects the current integrated state, including a feedback-only run that only advances the watermark. The behind non-draft advisory path is the exception: it only comments, so a PR a human has marked Ready for review keeps its body. Scope tracking-block extraction to the machine-managed block so human prose in the PR body cannot shadow the recorded state: read every field through tracking_block(), which emits only the last `# meai-otel-genai-worker:state:begin` .. `:state:end` range (the skill writes the live block at the very bottom, so the body ends with it). Otherwise a "> upstream-scan-ref:" line, a future "> feedback-processed-through:", or an older block a maintainer pastes above the live one would win tracking_value's first match and could force a false caught-up no-op or silently suppress real reviewer feedback. Apply the same within-block scoping in the publish guardrail so the guard and the extractor stay symmetric. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds GitHub Agentic Workflows (gh-aw) automation to continuously scan open-telemetry/semantic-conventions-genai and maintain a single evergreen draft PR that keeps Microsoft.Extensions.AI’s GenAI OpenTelemetry instrumentation aligned, including PAT-pool support and supporting deterministic discovery/setup scripts.
Changes:
- Introduces an orchestrator workflow that deterministically resolves an upstream ref to a concrete commit SHA and invokes an agentic worker as a reusable workflow.
- Adds an agentic worker workflow (plus lock file) with safe-outputs, threat-detection, and deterministic host-side setup logic for PR lifecycle management.
- Adds shared PAT-pool infrastructure and a scheduled validation workflow, plus supporting gh-aw scaffolding/skills.
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/validate-pat-pool.yml | Scheduled/manual validation of PAT pool entries via Copilot CLI and step-summary reporting. |
| .github/workflows/shared/pat_pool.README.md | Documentation for configuring/using the Copilot PAT pool in agentic workflows. |
| .github/workflows/shared/pat_pool.md | Shared gh-aw import that selects a PAT slot and emits pat_number. |
| .github/workflows/meai-otel-genai-worker.md | Agentic worker definition: safe-outputs, network/tooling constraints, lifecycle instructions, PAT pool integration. |
| .github/workflows/meai-otel-genai-worker.lock.yml | Compiled reusable workflow invoked by the orchestrator. |
| .github/workflows/meai-otel-genai-orchestrator.yml | Deterministic orchestrator resolving target SHA and calling the worker via workflow_call. |
| .github/workflows/copilot-setup-steps.yml | Copilot cloud-agent setup workflow to install gh-aw CLI extension. |
| .github/skills/update-otel-genai-conventions/SKILL.md | Updates skill guidance (notably release-signal analysis) for GenAI semconv alignment. |
| .github/skills/update-otel-genai-conventions/references/pr-description.md | Updates tracking PR body template and tracking block format (yaml-comment sentinels). |
| .github/skills/agentic-workflows/SKILL.md | Adds a router/dispatcher skill for gh-aw workflow tasks. |
| .github/skills/agentic-workflow-designer/SKILL.md | Adds an interview-based workflow designer skill (no model invocation). |
| .github/scripts/meai-otel-genai-worker-setup.sh | Deterministic worker setup: resolves upstream SHA, discovers/classifies PR, computes action, wake-gates feedback. |
| .github/scripts/meai-otel-genai-orchestrator-discover.sh | Deterministic discovery: resolve upstream ref to SHA and emit at-most-one target JSON. |
| .github/mcp.json | Configures local gh-aw MCP server tool exposure. |
| .github/aw/actions-lock.json | Pins action SHAs used by gh-aw workflows/actions. |
| .github/agents/agentic-workflows.md | Adds repo agent dispatcher guidance + repo-specific PAT pool/schedule seed/frontmatter ordering requirements. |
| .gitattributes | Marks lock workflows as generated and configures merge strategy for .lock.yml files. |
Collaborator
🎉 Good job! The coverage increased 🎉
Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1497852&view=codecoverage-tab |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Microsoft.Extensions.AIemits OpenTelemetry GenAI spans/metrics, and the upstreamopen-telemetry/semantic-conventions-genaispec evolves continuously. Keeping our instrumentation current is manual and easy to miss.This PR adds a pair of GitHub Agentic Workflows (built on
gh-aw) that continuously scan the upstream conventions and maintain a single evergreen draft PR proposing the compensating changes toMicrosoft.Extensions.AI. A human still reviews and merges -- the automation only ever produces a draft PR and never touches anything outsidesrc/Libraries/Microsoft.Extensions.AI*,test/Libraries/Microsoft.Extensions.AI*, anddocs/.It adds only CI/automation assets (
.github/**) -- no product code or public API changes.How it works
The system follows the OrchestratorOps pattern (one deterministic orchestrator, one scoped agentic worker):
meai-otel-genai-orchestrator.yml) -- a plain, non-agentic workflow. On a daily schedule (or manual dispatch), it resolves the upstream ref to a concrete commit SHA and hands the worker a single, fully-specified target. It invokes the worker as a reusable workflow (workflow_call) so the run keeps the orchestrator's actor and passes gh-aw's activation role check.meai-otel-genai-worker.md) -- the agentic workflow. A deterministic host step (meai-otel-genai-worker-setup.sh) classifies the maintained PR (fresh / incremental / advisory / adopt / no-op) and reads prior state from a machine-readable block in the PR body. The agent then applies theupdate-otel-genai-conventionsskill to implement/refresh the integration, builds + tests it, and maintains the PR through gh-aw safe outputs (create / push / update / comment / mark-ready / no-op).Safety model: read-only agent job; all writes go through allow-listed safe outputs; threat-detection screens the untrusted upstream content before anything is published; a PAT pool avoids single-token rate limits; the maintained PR stays a draft requiring human review.
Lifecycle handled: first-run creation, daily incremental updates (append-only commits), behind-non-draft advisory (comment only), adoption of a human-started PR, reviewer-feedback incorporation, and release mark-ready.
What is included
meai-otel-genai-orchestrator.yml,meai-otel-genai-worker.md(+ compiled.lock.yml)meai-otel-genai-orchestrator-discover.sh,meai-otel-genai-worker-setup.sh.github/skills/update-otel-genai-conventions/**.github/aw/,.github/mcp.json,.github/agents/,copilot-setup-steps.yml.github/workflows/shared/pat_pool.*,validate-pat-pool.yml17 files,
.github/**only.Testing
In brief: Validated end-to-end on a dedicated simulation repository by "time-traveling" through pinned upstream commits. The full lifecycle -- clean-state creation and an incremental update -- runs green, including a real restore/build/test and threat-detection. The simulation also uncovered and fixed two workflow bugs, after which every reachable scenario passed.
Details:
Method -- Because the workflow reacts to upstream state, it was exercised against a standalone test repo by forcing
upstream_refto specificsemantic-conventions-genaicommits (T0 baseline, T1 a later commit), driving the orchestrator -> worker chain end-to-end on real GitHub Actions runners.Fresh, from a clean state (@ T0): all jobs green (discover -> worker activation -> agent -> threat-detection -> safe outputs). A draft PR was created on the evergreen branch with
automation+area-ailabels, a real code change (documentmodality inOtelMessageSerializer.cs+ tests), a clean build across net8.0/net9.0/net10.0/netstandard2.0 (0 warnings) with tests passing, and the machine-readable tracking block.Incremental update (@ T1): the worker correctly detected the PR was behind, integrated the delta (
gen_ai.request.reasoning.level), and appended a second commit -- the original commit was preserved (no force-push, no amend), and the tracking state advanced. This validates the "every update is an incremental commit" guarantee.Bugs found and fixed during testing (folded into their originating commits):
HEADbefore committing.create-pull-request: allow-emptycaused gh-aw to skip patch generation entirely, shipping a content-free PR and failing threat-detection -- removed it and emitno-opwhen a scan yields nothing to integrate.Deterministic host scripts (
*-discover.sh,*-setup.sh) were separately exercised with a chaos/property/exhaustive test harness (classification, state parsing, release gating, wake-gate, idempotence), which caught and fixed additional edge cases in the setup logic.Notes / follow-ups
needs: pat_pool.Microsoft Reviewers: Open in CodeFlow