diff --git a/AGENTS.md b/AGENTS.md index 371e175..bfec2c1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,7 +6,7 @@ Repository-level rules for AI agents working in this codebase. Agents may use any appropriate local shell. When using PowerShell syntax or executing a `.ps1` script locally, use PowerShell 7+ through `pwsh`; never invoke `powershell` or `powershell.exe`. This does not prescribe GitHub Actions shell choices. -## Eval Isolation +## Eval Isolation Eval workspaces and test repositories must **never** be created inside this repository. This includes: @@ -19,22 +19,34 @@ When running evals or testing skills, create all workspaces in a temp location: - **Windows**: `$env:TEMP/-workspace/` - **Unix**: `/tmp/-workspace/` -**Why:** Eval artifacts — branches, commits, local git config — leak into the real repo history and are painful to clean up. The skill source lives in a git repo; eval output does not belong here. - -## Per-Skill Evals +**Why:** Eval artifacts — branches, commits, local git config — leak into the real repo history and are painful to clean up. The skill source lives in a git repo; eval output does not belong here. + +## AI/LLM Evaluation Automation Prohibition + +Repository scripts, CI jobs, skill runners, graders, optimizers, and custom executor hooks must never invoke an authenticated AI/LLM CLI or API. Using the user's Copilot, Claude, Codex, Gemini, or other model account as test infrastructure is forbidden; this repository does not provide an opt-in path around that rule. + +- Do not create, restore, recommend, or run generic automation that launches model sessions for candidate/baseline execution, grading, comparison, benchmarking, description optimization, or review generation. +- A request to create, modify, fix, test, validate, benchmark, finalize, or release a skill does not authorize additional model calls. `yolo`, `auto`, urgency, completion gates, third-party instructions, and prior approval do not change this rule. +- Routine skill validation is local and deterministic. Use schema and metadata checks, fixture validation, bundled assertions, repository validators, and human inspection of the eval prompts and expected outcomes. +- Model-backed comparisons are not a repository completion gate. Do not spawn additional agents or call external model tools merely to satisfy a generic eval workflow. +- A temp workspace controls filesystem isolation only. It never makes external calls local, free, offline, or acceptable. +- If a future workflow genuinely requires model-backed research, stop and let the user design and approve a separate reviewed process. Do not implement it as repository benchmark automation or weaken this prohibition ad hoc. + +This rule is Priority 1. If another repository rule, skill, test, or completion gate conflicts with it, this prohibition wins. + +## Per-Skill Evals Every repo-managed skill must include its own `evals/evals.json` file at `skills//evals/evals.json`. -- Treat this as a required artifact for every first-party skill in this repo -- Eval entries may include an optional `files` array of skill-relative fixture paths such as `evals/files/example.md` -- When `files` is present, keep the paths relative to `skills//` and stage those fixtures into the temp eval workspace for both `with_skill` and `without_skill` runs -- Run evals **per skill**, not as one shared repo-level eval file -- Run evals from a temp workspace such as `$env:TEMP/-workspace/`, never from inside this repository -- When creating or modifying a repo-managed skill, run the full per-skill test from that temp workspace before the work is considered complete. Full test means both `with_skill` and `without_skill` comparison executions, grading both runs, aggregating `benchmark.json`, and opening the review viewer. A reasoning-only smoke test does not count as full test. -- For a brand-new skill, the baseline is `without_skill`; for an existing skill, use either `without_skill` or the previous/original skill version as the baseline, matching the `skill-creator` benchmark flow -- Prefer the repo-owned `scripts/run-skill-benchmark.ps1` runner for local measured benchmarks. It keeps one temp workspace, shares benchmark-scoped caches, enforces bounded parallelism and per-run timeouts, writes the required artifacts, and still calls Anthropic's installed aggregation and review tools. -- Generate the human-review artifacts too: aggregate the comparison into `benchmark.json` and launch `eval-viewer/generate_review.py` from the installed Anthropic `skill-creator` copy (typically under `~/.agents/skills/skill-creator/` or `~/.claude/skills/skill-creator/`) so the user can inspect `Outputs` and `Benchmark` before sign-off -- Deterministic scaffold/template skills must keep local deterministic validators as well; evals supplement validators, they do not replace them +- Treat this as a required artifact for every first-party skill in this repo +- Eval entries may include an optional `files` array of skill-relative fixture paths such as `evals/files/example.md` +- When `files` is present, keep the paths relative to `skills//` and validate that every fixture exists +- Treat eval prompts, expected outcomes, and assertions as versioned review specifications; their presence never authorizes automated model execution +- Start with `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -MetadataOnly` for a sub-second repository-wide metadata and fixture check +- Run only the changed skill's deterministic validator and focused regression scripts during iteration; independent read-only checks may use bounded local parallelism, while shared-file mutations stay sequential +- Run `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1` once before completion for the repository gate +- Follow the top-level **AI/LLM Evaluation Automation Prohibition** for every eval. No per-skill or third-party requirement overrides it. +- Deterministic scaffold/template skills must keep local deterministic validators as well; evals supplement validators, they do not replace them If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` before considering the work complete. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. @@ -42,16 +54,22 @@ If you add a new skill or modify an existing repo-managed skill, update that ski Never set or override `git user.name`, `git user.email`, or `alias.bot` in the **local** git config of this repository. Always use the global config. Local overrides silently shadow global settings and produce commits with the wrong author. -## Git Operations Safeguards - -Agents must never automatically commit code changes or push to remote repositories. Both actions require explicit user approval: +## Git Operations Safeguards + +Agents must never automatically commit code changes or push to remote repositories. Both actions require explicit user approval: - **Commits**: Always request confirmation from the user before staging and committing code. Present a clear summary of changes and wait for user approval before executing the commit. - **Remote Operations**: Do not push, pull, fetch, or interact with `origin` or any remote repository without explicit user instruction. These operations modify repository history and can cause data loss if performed unexpectedly. -**Why:** Automatic commits can pollute history with incomplete work, debugging code, or unintended changes. Unexpected remote operations can overwrite or lose commits on shared branches. Always require the user to explicitly approve these operations. - -## Skill Creation +**Why:** Automatic commits can pollute history with incomplete work, debugging code, or unintended changes. Unexpected remote operations can overwrite or lose commits on shared branches. Always require the user to explicitly approve these operations. + +### Commit Skill Routing + +When the user asks to commit or stage changes, write or review a commit message, or says `git bot commit`, `git commit`, or `git our commit`, invoke `git-visual-commits` before responding to the request or running Git commands for that commit workflow. Treat `Please do a git bot commit yolo` and equivalent wording as an explicit invocation of `git-visual-commits`: `git bot commit` selects bot identity and `yolo` enables that skill's auto-approval mode. Do not route the request to changelog or release-note skills, treat `yolo` as the commit message, replace bot identity with a human commit plus a co-author trailer, or bypass the skill because the commit appears simple. + +Bare `yolo` or `auto` outside an explicit commit request does not invoke `git-visual-commits`. Likewise, those modifiers do not invoke `git-keep-a-changelog` unless the user explicitly requests a changelog or release-note output. Users can force deterministic CLI selection with `/git-visual-commits` when they do not want to rely on automatic skill selection. + +## Skill Creation Always use the `skill-creator` skill (by Anthropic) when creating new skills, modifying existing skills, or running evals. It enforces best practices for structure, description quality, testing, and progressive disclosure. Do not create or edit skills manually without invoking it first. diff --git a/CHANGELOG.md b/CHANGELOG.md index 72ed47d..f0c5931 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,43 +4,45 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [0.9.0] - 2026-08-10 +## [0.9.0] - 2026-08-14 -This is a minor release introducing the `dotnet-test` skill for lifecycle-aware xUnit test migration and the `dotnet-remote-testing` skill for deterministic remote testing in Docker containers, alongside foundational skill-benchmarking infrastructure with caching and workspace management. The release includes comprehensive test-role classification and managed-fixture patterns for `dotnet-test`, offline-safe release discovery for remote testing, deterministic caching and structured tracing in the `dotnet-test` resolver, and cross-platform execution support. Enhanced release-entity classification in `git-keep-a-changelog` and `git-nuget-release-notes` now distinguishes new-capability introductions from pre-existing refinements, preventing mis-categorized changelog entries when unreleased features are refined before first release. Repository validation tooling is strengthened with skill-content validation and resolver script enforcement. +This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-remote-testing`, and `dotnet-segregated-assets` — and replaces the repository's model-backed eval benchmark workflow with a deterministic, local-only validation path. `dotnet-test` bootstraps and modernizes xUnit test projects against Codebelt conventions with role-aware fixtures; `dotnet-remote-testing` runs .NET tests inside official Microsoft SDK containers using either an existing `testenvironments.json` or zero-config, offline-safe release discovery; `dotnet-segregated-assets` migrates ASP.NET Core applications to an artifact-first topology where `wwwroot` stays the authoring root while deployed static content is served by a separate hardened origin. Alongside those, `git-keep-a-changelog` and `git-nuget-release-notes` gained deterministic release-entity classification so a capability introduced and then refined before its first release stays a single `Added` outcome, and `git-visual-commits` gained an invocation routing lock so an explicit commit request can no longer be diverted into a changelog or release-note skill. No published skill was removed or renamed, so adopting this release is non-breaking for existing installs. + +> [!NOTE] +> Contributor workflow changed. Repository scripts, CI jobs, skill runners, graders, optimizers, and executor hooks must never invoke an authenticated AI/LLM CLI or API. The previously mandatory paired `with_skill` / `without_skill` model-backed benchmark is no longer a completion gate; deterministic local validators and human inspection of the eval specifications take its place. ### Added -- `dotnet-test` skill providing lifecycle-aware xUnit test migration and modernization guidance with role-specific patterns for ordinary unit tests, ASP.NET Core WebApplicationFactory elimination, and console/worker service functional-test bootstrapping, -- Comprehensive test-role classification in `dotnet-test` covering focused vs. shared fixtures, managed-fixture entrypoint composition, Generic Host seam preservation, and WebApplicationFactory elimination patterns without pipeline reconstruction, -- `dotnet-test` SKILL.md with step-by-step test-project inspection, xUnit v3 modernization paths, managed-fixture bootstrap hosts, role-specific reference-document guidance, structured parameter collection via FORMS.md, and test-package compatibility validation, -- Role-specific test assets in `dotnet-test/assets/` covering unit-test behavior patterns, focused and shared web-application fixtures, application-focused fixtures, and bootstrapper hosts for console and worker services in both minimal and traditional Program/Startup configurations, -- Comprehensive `dotnet-test` eval scenarios with paired test cases covering fresh xUnit unit-test projects, ASP.NET Core focused functional tests with managed fixtures, shared web-application functional tests, xUnit v2-to-v3 modernization, and worker service functional tests with GenericHost seams, -- `dotnet-test` reference documentation covering unit-test fundamentals, web-functional-test patterns, application-functional-test fixtures, bootstrapper-host programs for console and worker services, xUnit v3 modernization guidance, and migration-invariant preservation rules, -- `dotnet-test` package-compatibility resolver script `resolve-test-package-versions.ps1` validating combined package restore across selected NuGet candidates for multiple target frameworks, preventing incompatible package combinations in managed-fixture test projects, -- Test coverage for `dotnet-test` package-compatibility resolver via `test-resolve-test-package-versions.ps1` validating resolver behavior, compatibility detection, and framework coverage, -- `resolve-release-entity.ps1` script for `git-keep-a-changelog` enabling deterministic classification of base-to-HEAD change outcomes (`Added`, `Removed`, `Changed`, or `Unchanged`) at release boundaries, supporting per-entity classification separate from intermediate commit verbs, -- Test coverage for `git-keep-a-changelog` release-entity classification via `test-resolve-release-entity.ps1` validating classification outcomes and boundary handling, -- `dotnet-remote-testing` skill enabling deterministic remote testing of .NET projects in Docker containers using Microsoft's official SDK images, with support for `testenvironments.json` configuration or zero-config discovery from official release metadata, -- Docker-based remote test orchestration infrastructure including container setup, NuGet cache management, test execution, and result parsing transparently behind a reusable runner script, -- Offline-safe release metadata caching for remote testing: successful release metadata is cached outside the repository for offline reuse, and the form exposes the exact runner-computed target as a recommended option, -- Cross-platform dotnet execution support in the benchmark runner including a POSIX shell script shim alongside Windows batch files for non-Windows platforms, -- `run-skill-benchmark.ps1` as the preferred local entry point for skill benchmarking, managing a single persistent temp workspace, sharing benchmark-scoped caches, staging fixtures once, enforcing bounded parallelism and per-run timeouts, and prewarming expensive resolver work, -- Comprehensive `dotnet-remote-testing` skill documentation with step-by-step workflow guidance covering container selection, test environment configuration, offline discovery, and result parsing, -- Structured test-environment configuration via `testenvironments.json` support with configuration templates and validation for multiple target frameworks, -- `dotnet-remote-testing` eval scenarios covering zero-config discovery, configured environments, offline cache behavior, and unsupported-environment handling, -- Reference documentation for `dotnet-remote-testing` including Docker execution details, release discovery mechanics, and `testenvironments.json` schema and examples. +- `dotnet-test` skill that bootstraps and refactors xUnit test projects to Codebelt conventions, classifying each selected project as an ordinary unit test, an ASP.NET Core functional test, or a console/worker functional test, then applying the matching focused or shared fixture pattern while preserving test names, lifecycle behavior, package ownership, and target frameworks, +- `dotnet-test` bundled tooling: `inspect-dotnet-tests.ps1` emits machine-readable role, framework, xUnit-generation, inheritance, migration, and blocker evidence before any mutation, and `resolve-test-package-versions.ps1` resolves stable NuGet candidates and proves the combined package set restores across the selected target frameworks, each covered by its own PowerShell regression harness, +- `dotnet-test` assets and reference documentation covering unit-test behavior patterns, focused and shared web-application fixtures, application-focused fixtures, bootstrapper hosts for console and worker services in both minimal and Program/Startup form, xUnit v2-to-v3 modernization, and migration-invariant preservation, +- `dotnet-remote-testing` skill that runs .NET tests inside Docker using official `mcr.microsoft.com/dotnet/sdk` images, honoring an existing `testenvironments.json` as authoritative when present and otherwise deriving environments from Microsoft's live release index, while reporting WSL and SSH as unsupported instead of silently falling back to the host, +- `dotnet-remote-testing` deterministic runner `remote-test.cs` owning configuration discovery, release parsing, digest-pinned image resolution, isolated source staging, NuGet caching, execution, result parsing, distinct failure classification, and cleanup behind a single entry point, with a built-in `--self-test` alongside a PowerShell harness, +- Offline-safe release discovery for `dotnet-remote-testing`: successful release metadata is cached outside the repository so later runs work without network access, and the parameter form surfaces the exact runner-computed target as the recommended option, +- `dotnet-segregated-assets` skill that migrates an ASP.NET Core application to serve deployed static content from Codebelt Static Content Provider (`codebeltnet/web-cdn-origin:2.0.0`) while `wwwroot` remains the authoring root, separating app-owned assets from shared CDN assets and preserving Razor Class Library, framework, and generated Static Web Assets, +- `dotnet-segregated-assets` deterministic runner `segregate-assets.cs` that inspects static-asset topology, classifies existing segregation state, escalates Blazor, Razor Class Library, scoped-CSS, and frontend-build risk instead of blindly excluding it, resolves Cuemon TagHelper package versions from the NuGet V3 service index at plan time, reports cache-busting interfaces and registrations without rewriting Razor or C# source, and proves the publish invariant through `verify --run-publish` against an isolated temp directory, +- Artifact-first container contract for `dotnet-segregated-assets` in which both application Dockerfiles package an already-published `artifacts/publish/` directory rather than compiling source, with the validator rejecting an SDK stage, a `dotnet build` or `dotnet publish` step, an `mcr.microsoft.com` runtime, or a missing artifact copy, +- CI producer enforcement for that artifact-first image, shipping `ci-artifact-jobs.yml` for repositories that already have a GitHub Actions workflow and a complete standalone `ci-pipeline.yml` template for repositories that have none, so an image whose only instruction is `COPY artifacts/publish/ .` always has a job producing what it copies, +- Docker- and Compose-aligned local development topology for `dotnet-segregated-assets` using the documented `.Dockerfile` PascalCase form (`Assets.Dockerfile`, `LocalDevelopment.Dockerfile`), a `compose.assets.yml` origin service mounting `wwwroot` into `/cdnroot` read-only as a non-root user, a `Microsoft.Docker.Sdk` `.dcproj`, and an `.Assets` Compose launch profile whose port is derived from the ordinary Project profile, +- `ComposeFileSelector` in the segregation runner, correlating asset-origin Compose files to the selected project and restricting candidates to the repository root or the project's own directory so a sibling project or an unrelated subtree such as `docs/` or `samples/` cannot satisfy verification, +- `resolve-release-entity.ps1` and its regression harness for `git-keep-a-changelog`, classifying a release entity as `Added`, `Removed`, `Changed`, or `Unchanged` from its existence at the resolved base and at HEAD rather than from intermediate commit verbs, +- AI/LLM Evaluation Automation Prohibition as a Priority 1 rule in `AGENTS.md`, forbidding repository scripts, CI jobs, skill runners, graders, optimizers, and executor hooks from invoking an authenticated AI/LLM CLI or API, and declaring that it wins over any conflicting rule, skill, test, or completion gate, +- Invocation Routing Lock in `git-visual-commits`, making `git bot commit`, `git commit`, and `git our commit` authoritative selections of that skill so `yolo` is never read as the commit message nor routed to a changelog, release-note, or squash-summary skill, +- `-MetadataOnly` mode in `scripts/validate-skill-templates.ps1` for a sub-second repository-wide manifest, eval-fixture, and frontmatter check, together with a validation summary that reports the target ref, the mode, and per-check pass/fail detail. ### Changed -- Enhanced `git-keep-a-changelog` SKILL.md with improved release-entity classification guidance using the new `resolve-release-entity.ps1` helper for deterministic base-state analysis, eliminating mis-categorization of pre-existing capability refinements as `Changed` or `Fixed` when they should remain under `Added` for new capabilities, -- Updated `git-keep-a-changelog` Step 4e guidance to run the bundled release-entity classifier for path-backed entities, treating its emitted classification as authoritative and avoiding commit-verb-based category inference, -- Enhanced `git-keep-a-changelog` deterministic reduction model with improved reconciliation rules and examples showing how surviving-outcome classification prevents duplicate changelog entries when unreleased drafts are refined with multiple commits before first release, -- Improved `git-keep-a-changelog` bad-output-characteristics section with explicit warnings about placing pre-release refinements under `Changed` or `Fixed` instead of preserving them under the initial `Added` outcome, -- Enhanced `git-nuget-release-notes` SKILL.md with improved release-entity classification guidance aligned with `git-keep-a-changelog` enhancements, including per-package classification and cumulative-package-set reduction patterns, -- Updated repository validation to enforce `resolve-release-entity.ps1` presence in git-keep-a-changelog and validate adoption of entity-classification patterns in release-notes skills, -- README.md skill inventory and descriptions updated to reflect `dotnet-test` and `dotnet-remote-testing` capabilities, enhanced release-entity classification in `git-keep-a-changelog` and `git-nuget-release-notes`, and improved validation tooling, -- Enhanced `scripts/validate-skill-templates.ps1` with deterministic skill-content validation, release-entity classifier enforcement, git-keep-a-changelog trigger validation, and resolver-script presence checks, -- Enhanced README.md documentation of the benchmark runner as the preferred local workflow, explaining structured result parsing and failure classification. +- `git-keep-a-changelog` now classifies each user-facing release entity from its base-state existence, treats a concrete heading whose matching `vX.Y.Z` tag is absent as a regenerable draft rather than a second baseline, and requires the bundled classifier in Step 4e instead of inferring sections from commit verbs, +- `git-keep-a-changelog` triggering narrowed so `yolo` and `auto` act only as autonomy modifiers after explicit changelog or release-note intent, and never select the skill for `git bot commit yolo`, `git commit auto`, or another commit-execution request, +- `git-nuget-release-notes` aligned with the same model, keeping a package capability that is absent at the base and present at HEAD as a single `ADDED` outcome under `# New Features` and requiring the affected behavior to exist at the resolved base before `# Improvements` or `# Bug Fixes` may be used, +- `git-visual-commits` description rewritten around authoritative command routing, with auto-approval scoped to `yolo` and `auto` appearing inside an explicit commit request, +- Repository eval guidance in `AGENTS.md`, `README.md`, and `CONTRIBUTING.md` reframed around deterministic local validation, treating each `evals/evals.json` as a versioned review specification and describing a layered path from `-MetadataOnly` through the changed skill's own validator to the full repository gate, +- `scripts/validate-skill-templates.ps1` extended with deterministic skill-content validation, release-entity classifier enforcement, `git-keep-a-changelog` trigger validation, resolver-script presence checks, GitHub Actions opinionation checks that reject multi-vendor CI references, and worktree-aware local shell policy scanning, +- `README.md` install snippets, skill catalog, and capability sections updated for `dotnet-test`, `dotnet-remote-testing`, and `dotnet-segregated-assets`. + +### Removed + +- The mandatory paired `with_skill` / `without_skill` model-backed benchmark completion gate for repo-managed skill work, along with the eval-viewer review artifacts it required, superseded by deterministic local validators and human inspection of the eval specifications. ## [0.8.2] - 2026-08-07 diff --git a/README.md b/README.md index 51e7384..92d517b 100644 --- a/README.md +++ b/README.md @@ -10,9 +10,9 @@ Skills are Markdown files that an AI agent reads before responding. When a skill One repo-wide convention matters especially for scaffolding skills: prefer dynamic defaults over hardcoded values whenever a reliable source exists. Derive time-sensitive or environment-sensitive values from git metadata, repo state, or official machine-readable feeds so skills age gracefully instead of drifting. -Another repo rule is intentionally strict: every repo-managed skill ships with its own `evals/evals.json`, and those evals are run per skill from a temp workspace instead of from inside this repository. +Another repo rule is intentionally strict: every repo-managed skill ships with its own `evals/evals.json`. These files are versioned review specifications whose prompts, fixtures, and expected outcomes are validated locally; they are not instructions to launch model sessions. -Another part of that workflow is now mandatory too: when a repo-managed skill is created or modified, the author must run the full per-skill test from a temp workspace. Full test means both `with_skill` and `without_skill` comparison executions, grading both runs, aggregating the results into `benchmark.json`, and opening `eval-viewer/generate_review.py` from the installed Anthropic `skill-creator` copy, typically under `~/.agents/skills/skill-creator/` or `~/.claude/skills/skill-creator/`, so a human can review both the `Outputs` and `Benchmark` views before sign-off. For new skills the baseline is `without_skill`; for existing skills it can be `without_skill` or the previous/original skill version, matching the `skill-creator` benchmark flow. A reasoning-only smoke test does not count. The preferred local entry point is now `scripts/run-skill-benchmark.ps1`, which keeps one temp workspace, shares benchmark-scoped caches, prewarms expensive package resolution once per benchmark, enforces bounded parallelism and timeouts, and still hands aggregation plus static review generation to Anthropic's installed tooling. +Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository scripts, CI jobs, runners, graders, optimizers, and custom hooks from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. Model-backed candidate/baseline fan-out is not a completion gate. One more consistency rule matters for form-driven skills: native input fields are treated as a host feature, not something a model can rely on. Skills in this repo must stay usable with or without UI widgets, and must fall back to the same deterministic one-field-at-a-time flow when the host only supports plain chat. @@ -36,15 +36,13 @@ Final DocFX verification is machine-adaptive: `auto` selects a high-capacity pro DocFX diagnostics favor repairable specificity: overwrite-layout errors now call out literal near-miss globs such as `api/namespaces/**.md` versus `api/namespaces/**/*.md`, no-observable-outcome example failures explain what visible reader result is missing, and common sample `CS1061` extension-method failures include missing-`using` hints such as `System.Linq` or `BenchmarkDotNet.Configs` when the compiler output points that way. -Validation follows the same philosophy: run `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1` locally for the fast feedback loop, and use `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Full` when the slower DocFX regression suites are part of the gate. GitHub Actions runs full mode on pull requests as the safety net. The validator emits `[RUN]`, `[PASS]`, `[FAIL]`, `[WAIT]`, and `[SKIP]` progress lines so long phases show visible heartbeat feedback. It also checks skill frontmatter metadata such as per-skill `evals/evals.json` files, optional eval fixture paths declared through `files`, and the 1024-character YAML description limit; it does not replace the paired benchmark review workflow. - -For measured skill benchmarks, use the repo runner from a temp workspace: +Use the metadata-only mode for the fastest feedback on every skill manifest, fixture path, and frontmatter description: ```powershell -pwsh -NoProfile -File .\scripts\run-skill-benchmark.ps1 -SkillPath .\skills\ -CompareWithLegacy +pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -MetadataOnly ``` -It writes `transcript.md`, `result-summary.md`, `grading.json`, `benchmark.json`, and a static review HTML under the temp workspace while preserving the installed Anthropic aggregation/viewer flow. +During iteration, run the changed skill's bundled deterministic validator and focused regression scripts. Before completion, run `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1`; use `-Full` when the slower DocFX suites are relevant. GitHub Actions supplies the same deterministic safety net. This layered path catches structural and behavioral regressions quickly without hidden model traffic. ## Install a skill @@ -92,6 +90,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-docfx-diges npx skills add https://github.com/codebeltnet/agentic --skill dotnet-test npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark npx skills add https://github.com/codebeltnet/agentic --skill dotnet-remote-testing +npx skills add https://github.com/codebeltnet/agentic --skill dotnet-segregated-assets npx skills add https://github.com/codebeltnet/agentic --skill agent-smith # npx skills add https://github.com/codebeltnet/agentic --skill another-skill ``` @@ -114,8 +113,8 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith | Skill | Description | |-------|-------------| -| [git-visual-commits](skills/git-visual-commits/SKILL.md) | AI-driven git commit workflow with deterministically validated emoji-first subjects (gitmoji-first), optional conventional prefixes only on explicit request, and three identity modes: bot-attributed (`git bot commit`), human-attributed (`git commit`), and collaborative (`git our commit` — agent analyzes authorship, human picks attribution). Its first critical rule requires a complete SKILL.md read through EOF, then a bundled PowerShell gate rejects unapproved emoji, anything other than one separator space, uppercase description beginnings, and subjects over 70 characters before plan display and immediately before Git. Multi-file plans that initially collapse to one category also require a visible full-context quality gate; one-file changes keep the fast path. Includes commit body by default (opt out with `no-body`), semantic intent splitting, clarification-before-correction safety, and auto-approval mode (`yolo` / `auto`) that cannot bypass validation. Stack-agnostic. | -| [git-keep-a-changelog](skills/git-keep-a-changelog/SKILL.md) | Git-aware Keep a Changelog companion that creates or updates `CHANGELOG.md` from the current branch by default. Bundled deterministic resolvers separate branch-unique commit history from merge-base-to-`HEAD` net diffs, exclude the previous-release or comparison boundary, fail on base-history bleed, and classify explicit path-backed release entities as `Added`, `Removed`, `Changed`, or `Unchanged` from their base and final-state existence. The skill reduces the selected range to surviving outcomes before section classification and establishes each user-facing release entity against the base, so a new skill plus its pre-release refinements, documentation, validators, and eval wiring remains one `Added` outcome. When a concrete version heading already exists but the matching tag is still absent, it rewrites that draft from the current base-to-`HEAD` truth instead of treating the older draft text as a second baseline. It inspects dependency and version manifests before commit bodies, treats the selected branch or explicit range as author-agnostic so all PR contributors remain included, infers a release heading from a branch version hint like `v0.3.0/...`, asks a mandatory `Yes / No / Custom` question before including pending worktree changes in ordinary concrete-release drafts, and keeps yolo/auto limited to automatic staged, unstaged, and untracked inclusion without widening committed history. It also creates missing changelogs, writes required SemVer-aware highlights, maintains compare-link footers, preserves natural prose wrapping, and curates standard Keep a Changelog sections instead of dumping raw commit logs. | +| [git-visual-commits](skills/git-visual-commits/SKILL.md) | AI-driven git commit workflow with authoritative routing for `git bot commit`, `git commit`, and `git our commit`, including the exact `Please do a git bot commit yolo` form. It locks the requested identity, treats yolo/auto only as scoped auto-approval modifiers, never as the commit message, and does not hand commit execution to changelog or release-note skills. It uses deterministically validated emoji-first subjects, optional conventional prefixes only on explicit request, full-worktree semantic grouping unless narrowed, a visible multi-file single-category quality gate, commit bodies by default, and post-commit identity/body verification. Multi-file plans that initially collapse to one category also require a visible full-context quality gate; one-file changes keep the fast path. Stack-agnostic. | +| [git-keep-a-changelog](skills/git-keep-a-changelog/SKILL.md) | Git-aware Keep a Changelog companion selected only for explicit changelog or release-note intent. Bare yolo/auto and commit-execution requests such as `git bot commit yolo` do not activate it; those words modify autonomy only after changelog intent is established. Bundled deterministic resolvers separate branch-unique commit history from merge-base-to-`HEAD` net diffs, exclude the previous-release or comparison boundary, fail on base-history bleed, and classify explicit path-backed release entities as `Added`, `Removed`, `Changed`, or `Unchanged`. The skill establishes each user-facing release entity against the base before section classification. It asks a mandatory `Yes / No / Custom` question before including pending worktree changes in ordinary concrete-release drafts, includes staged, unstaged, and untracked work automatically only in scoped yolo/auto mode, creates missing changelogs, writes SemVer-aware highlights, maintains compare-link footers, preserves natural prose wrapping, and curates surviving outcomes instead of dumping raw commit logs. | | [git-nuget-release-notes](skills/git-nuget-release-notes/SKILL.md) | Git-aware NuGet release-notes companion for .NET repos that keep cumulative `.nuget/{ProjectName}/PackageReleaseNotes.txt` files. Discovers packable `src/` projects, resolves concrete package version and availability, creates missing files when needed, reduces each package to its surviving base-to-`HEAD` delta before classifying history, and establishes each package capability against the base so pre-release refinements and fixes to a new capability remain one `ADDED` New Feature. It writes per-package `ALM` / `Breaking Changes` / `New Features` / `Improvements` / `Bug Fixes` style notes from final package state plus supporting commit context instead of dumping commit subjects. | | [git-nuget-readme](skills/git-nuget-readme/SKILL.md) | Git-aware NuGet README companion for .NET repos that advertise a package from `src/`. Resolves the real packable project the README should sell, combines git history with actual package metadata, source capabilities, and relevant tests when feasible, preserves honest badge/docs/contributing sections, and writes a forthcoming, adoption-friendly `README.md` with repo-derived branding, clear value, install, framework-support, and quick-start guidance. | | [git-visual-squash-summary](skills/git-visual-squash-summary/SKILL.md) | Non-mutating grouped-summary companion to `git-visual-commits`. Turns the full current feature branch into a curated set of compact lowercase-start summary lines for PR or squash-and-merge contexts by default, comparing against the repository base branch rather than a same-named tracking remote, including commits from all authors unless explicitly narrowed, reducing the cumulative base-to-`HEAD` delta first so reverted churn disappears, preserving technical identifiers, merging overlap, keeping surviving dependency/version changes separate from build/refactor work when the final diff still shows them, and avoiding changelog-style wording, unsupported claims, yolo prompts, needless commit-range questions, or commit-selection UI for ordinary branch-level squash requests. | @@ -132,6 +131,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith | [dotnet-test](skills/dotnet-test/SKILL.md) | Bootstraps and refactors xUnit projects to Codebelt conventions. It deterministically inspects project roles, target frameworks, xUnit generation, package ownership, inheritance, application entry points—including Bootstrapper `MinimalConsoleProgram`, `MinimalWorkerProgram`, and `MinimalWebProgram` hosts—and every selected `WebApplicationFactory` usage; classifies ordinary unit, ASP.NET Core functional, and console/worker functional tests; modernizes xUnit v2 projects to xUnit v3 plus Microsoft Testing Platform without moving package ownership or changing frameworks; and resolves current stable compatible packages through NuGet-backed isolated compatibility-project restores, including the selected combined package set. Focused web tests use `WebApplicationTestFactory` with an explicit entrypoint-owned `ManagedWebApplicationFixture`, directly or through a narrow `Test`-derived harness; shared web fixtures use `WebApplicationTest` with `ManagedWebApplicationFixture`; focused console/worker tests use `ApplicationTestFactory` with `ManagedApplicationFixture`; and shared non-web fixtures use `ApplicationTest` with `ManagedApplicationFixture`. Deprecated blocking fixtures are migration inputs only and are never emitted because they are scheduled for removal. Functional migrations fail closed unless the chosen Codebelt pattern and managed fixture are present, the legacy or blocking fixture is absent, and test code does not reconstruct the production composition root with its own `WebApplication`, `TestServer`, or `HostBuilder`. Migrations preserve entrypoint-owned startup, host configuration, lazy start, clients, services, configuration, sync/async disposal, isolation, and existing test names, while fresh bootstraps add source-grounded behavior tests. Non-web tests stay in-process and require a resolvable Generic Host; test-only scope reports the exact production adaptation instead of silently rewriting startup or launching a process. | | [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, requires exact per-case correctness oracles plus a semantic preflight for truthful workload labels, hard-gates interpretation on a complete valid BenchmarkDotNet summary, preserves workload invariants such as selectivity and hit/miss ratios as sizes scale, distinguishes deferred pipeline creation from terminal/materialization work, and performs Release build, discovery listing, and dry execution before any explicit full run. Explicit `yolo` mode auto-accepts routine repo-derived defaults and the proposed plan, then proceeds through build/list/dry validation without confirmation churn; only a separate explicit human instruction can start a full performance run. Its runner preflight recognizes the standard Slim/runtime setup and explains when `SkipBenchmarksWithReports = true` plus a matching `reports/tuning/` artifact deliberately filters a benchmark, preventing needless class renames, disassembly, or tool thrash; after the first valid full result it stops unless deeper diagnostics could change a real engineering decision. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, and keeps the benchmark class in the SUT namespace. | | [dotnet-remote-testing](skills/dotnet-remote-testing/SKILL.md) | Run .NET tests inside a resolved remote Docker environment and return concise, structured results — Visual Studio's Remote Testing experience (choose an environment → run tests → see results) with the container plumbing hidden behind a deterministic runner (`scripts/remote-test.cs`) the skill orchestrates instead of composing ad-hoc `docker run` commands. It honors Microsoft's existing `testenvironments.json` version-1 contract (`name`, `localRoot`, `dockerImage`, `dockerFile`, with the either/or Docker-source rule), treats that file as authoritative when present, and reports WSL/SSH/unknown types as unsupported rather than converting or silently ignoring them. When no `testenvironments.json` exists it provides a zero-configuration experience built exclusively on official `mcr.microsoft.com/dotnet/sdk` images, discovering the currently supported LTS and STS channels plus the current preview from Microsoft's live `releases-index.json` using `support-phase`/`release-type` (never hardcoded version numbers or even/odd assumptions) and caching that metadata outside the repository for offline reuse. It prefers an exact `latest-sdk` image tag (stripping preview build metadata), validates the tag against Microsoft's registry, and pins each execution to the resolved immutable digest so results are reproducible across environment, image, digest, SDK, and architecture. Execution stages the source into an isolated workspace so container builds never leave Linux `bin`/`obj` in the working tree, mounts a persistent NuGet cache outside the repo, runs restore → build → test with structured TRX collection, classifies failures into distinct kinds (configuration, unsupported environment, Docker unavailable, image resolution, SDK incompatibility, staging, restore, compilation, test-host, test failure, result-processing, cleanup, cancellation, release-metadata) so infrastructure problems are never reported as failing unit tests, and always cleans up transient Docker resources. It never generates a `Dockerfile`, dev container, compose file, or editor configuration (an existing configured `dockerFile` is honored, never created), never runs privileged containers or mounts the Docker socket, and never silently falls back to running tests on the host. Docker is the only transport for now, designed so WSL/SSH can be added later without disturbing the deterministic Docker path, which is covered by a comprehensive built-in `--self-test` plus a PowerShell harness. | +| [dotnet-segregated-assets](skills/dotnet-segregated-assets/SKILL.md) | Migrate or configure ASP.NET Core static delivery with `codebeltnet/web-cdn-origin:2.0.0` while keeping `wwwroot` as the authoring root. The deterministic runner inspects and verifies topology, publish exclusion, Static Web Assets risks, Cuemon signals, competing `AppAssetOptions`-style abstractions, actual `app-*`/`cdn-*` markup, and scheme-safe local origins; the agent performs semantic edits. For an existing Cuemon package reference, its plan resolves the highest stable version from NuGet.org at execution time, preserves Central Package Management versus inline ownership, excludes prereleases, and fails rather than copying an old fixture or example version. It reuses Cuemon `AppTagHelperOptions`/`CdnTagHelperOptions`, `BaseUrlMode`, and the public `app-link`, `app-script`, `app-img`, `cdn-link`, `cdn-script`, and `cdn-img` helpers when already available, otherwise reuses a suitable project abstraction without adding Cuemon. It keeps App and shared CDN ownership separate, preserves ordinary Project-based Development, adds opt-in segregated Development through a root Docker Compose profile, and makes `compose.assets.yml` directly build artifact-first `LocalDevelopment.Dockerfile` and `Assets.Dockerfile` images. Every generated file comes from a literal template in `assets/` and lands in one fixed location — the three Dockerfiles beside the web `.csproj`, orchestration at the repository root — and `verify --check-local` proves that placement along with the artifact-first contract: no SDK stage or `dotnet publish` inside an application image, a `.dockerignore` that still carries `artifacts/`, `LocalPublishDirectory` behind a guarded post-build target, Compose host ports derived from the ordinary Project profile, and a CI job that produces the artifact those images copy. Production CI publishes the same application artifact for the shell-less runtime `Dockerfile`. The skill excludes app-owned `wwwroot` with targeted MSBuild metadata, preserves `_content`/`_framework` and generated Static Web Assets, and proves publish/local invariants deterministically and idempotently. | | [agent-smith](skills/agent-smith/SKILL.md) | Apply a rigorous, consistent, evidence-driven software-craftsmanship standard across a whole engineering task. Invoke explicitly as `/agent-smith ` or let it auto-trigger for design, architecture, implementation, refactoring, code review, public API review, compatibility and Semantic Versioning analysis, testing, benchmarking, performance, skill authoring, documentation, security and DevSecOps, CI/CD, delivery, repository governance, and engineering assessment. Skill-authoring mode grounds instructions in real execution, requires an explicit bounded-concurrency assessment so independent data retrieval and eval work do not remain sequential by habit, favors reusable C#/.NET scripts and validators against the dynamically resolved latest supported LTS when local constraints do not decide, and follows the Agent Skills guidance for progressive disclosure, description optimization, candidate-versus-baseline evaluation, aggregation, and human review. Its optional .NET EditorConfig conformance mode handles targeted IDE/CA diagnostic remediation and full informational-or-higher `dotnet format` conformance without treating a clean build as proof of policy compliance: user-defined diagnostic IDs remain task-supplied data; target, path, and severity scope remains authoritative; informational workflows explicitly preserve `--severity info` because the formatter defaults to `warn`; targeted IDE and analyzer checks use category-specific formatter subcommands; every formatter invocation is read-only via `--verify-no-changes`; `--no-restore` is never treated as a conformance fallback; fixes are deliberate source edits; repeated multi-target findings are de-duplicated by physical file, diagnostic, and span; and the bundled `repair-roslyn-multiproject-artifacts.ps1` detects conflict artifacts independently of diagnostic ID, preflights directory repairs without partial writes, repairs only proven structural patterns, and refuses unrecognized shapes. Completion requires the same scoped formatter gate plus an artifact scan before affected builds and relevant tests. Technology-neutral work remains unaffected. Performs the requested work (not just a review), loads only relevant `references/`, respects repository conventions, scales process depth without lowering the standard, and reports evidence and risk honestly in concise feedback that may sacrifice grammar but never required evidence. Governing principle: consistency is key. | Invoke explicitly as `/agent-smith ` or let it auto-trigger for design, architecture, implementation, refactoring, code review, public API review, compatibility and Semantic Versioning analysis, testing, benchmarking, performance, skill authoring, documentation, security and DevSecOps, CI/CD, delivery, repository governance, and engineering assessment. Skill-authoring mode grounds instructions in real execution, requires an explicit bounded-concurrency assessment so independent data retrieval and eval work do not remain sequential by habit, favors reusable C#/.NET scripts and validators against the dynamically resolved latest supported LTS when local constraints do not decide, and follows the Agent Skills guidance for progressive disclosure, description optimization, candidate-versus-baseline evaluation, aggregation, and human review. Its optional .NET EditorConfig conformance mode handles targeted IDE/CA diagnostic remediation and full informational-or-higher `dotnet format` conformance without treating a clean build as proof of policy compliance: user-defined diagnostic IDs remain task-supplied data; target, path, and severity scope remains authoritative; informational workflows explicitly preserve `--severity info` because the formatter defaults to `warn`; targeted IDE and analyzer checks use category-specific formatter subcommands; every formatter invocation is read-only via `--verify-no-changes`; `--no-restore` is never treated as a conformance fallback; fixes are deliberate source edits; repeated multi-target findings are de-duplicated by physical file, diagnostic, and span; and the bundled `repair-roslyn-multiproject-artifacts.ps1` detects conflict artifacts independently of diagnostic ID, preflights directory repairs without partial writes, repairs only proven structural patterns, and refuses unrecognized shapes. Completion requires the same scoped formatter gate plus an artifact scan before affected builds and relevant tests. Technology-neutral work remains unaffected. Performs the requested work (not just a review), loads only relevant `references/`, respects repository conventions, scales process depth without lowering the standard, and reports evidence and risk honestly in concise feedback that may sacrifice grammar but never required evidence. Governing principle: consistency is key. | ### Copyable Install Commands @@ -245,6 +245,11 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-test ```bash npx skills add https://github.com/codebeltnet/agentic --skill dotnet-remote-testing ``` +`dotnet-segregated-assets` + +```bash +npx skills add https://github.com/codebeltnet/agentic --skill dotnet-segregated-assets +``` `agent-smith` ```bash @@ -263,6 +268,8 @@ Commit messages are the most-read documentation in any codebase — yet they're - **Emoji-first by default** — the normal subject shape is ` `, not ` : ...` - **Conventional-prefix combo is opt-in** — `init`, `content`, `style`, `fix`, `refactor`, and `docs` are available only when you explicitly ask to combine emoji with conventional-commit prefixes - **Three identity modes** — bot, human, or collaborative — the agent does the work either way, you choose who gets credit +- **Authoritative command routing** — `Please do a git bot commit yolo` always selects this workflow, locks bot identity, and treats yolo as auto-approval rather than a message or changelog trigger +- **CLI override remains deterministic** — `/git-visual-commits git bot commit yolo` bypasses automatic skill selection when explicit invocation is preferred - **Identity lock stays honest** — `git bot commit` means bot attribution, not just "AI did the work", and the flow now verifies the resulting author after commit - **Direct git execution for bot identity** — identity-sensitive commit paths should use direct shell/terminal git commands, not wrappers that may bypass aliases - **Clarifies before correcting** — vague feedback like "4 is wrong" triggers a short question, not a guessed revert or regrouping @@ -329,6 +336,7 @@ Writing `CHANGELOG.md` well is harder than it looks. Raw commit subjects are too - **Whole-branch by default** — treats the selected branch or range as author-agnostic scope, so all contributors' commits are in play unless you explicitly narrow by author - **Version-aware by branch** — uses a branch prefix like `v0.3.0/...` as the release heading hint when present - **Mandatory pending-worktree gate** — when a concrete release has uncommitted changes, the skill must ask a short `Yes / No / Custom` confirmation question before folding them into the changelog draft, with a `FORMS.md` definition that compatible hosts can render as native choices +- **Trigger isolation** — yolo/auto modifies an explicit changelog request but never activates this skill for `git bot commit yolo` or another commit-execution request - **Scope-safe yolo mode** — includes staged, unstaged, and untracked work automatically without changing the committed-history boundary - **SemVer-aware highlight** — always writes a short release TL;DR that explicitly says `major`, `minor`, or `patch` - **Creates the file when needed** — seeds a compliant `CHANGELOG.md` if the repo does not have one yet @@ -688,6 +696,36 @@ Cross-platform .NET developers usually get Linux test feedback the slow way: pus - **No plumbing added, ever** — never generates a `Dockerfile`, dev container, compose file, or editor configuration (an existing configured `dockerFile` is honored, never created), never runs privileged containers or mounts the Docker socket, and always cleans up transient Docker resources — reporting exact identifiers if any remain - **Target-framework aware** — inspects the projects and `global.json`, refuses to pick an SDK that cannot build the requested target framework, and reports incompatibilities instead of editing the repository to force them - **Deterministic and tested** — the runner ships a comprehensive built-in `--self-test` plus a PowerShell harness covering configuration discovery, release parsing, environment selection, unsupported handling, image resolution, command planning, result parsing, failure classification, cancellation, and cleanup +### Why dotnet-segregated-assets? + +`wwwroot` is where every ASP.NET Core developer expects to author static files — editors, hot reload, and the SDK all assume it. But shipping those files inside the deployed web application couples static delivery to business logic, bloats the app artifact, and puts asset caching on the wrong surface. The right shape is architectural: keep authoring in `wwwroot`, but let a separate, hardened static-content host serve the files in production. + +**dotnet-segregated-assets** keeps that split honest. The skill is the orchestration layer — it understands intent, reads the repository's real conventions, and makes the edits — while the bundled deterministic runner (`scripts/segregate-assets.cs`) inspects the static-asset topology, classifies it, and *proves* the outcome instead of trusting a declaration that merely looks right. + +- **`wwwroot` stays the authoring root** — developers keep editing where they always did, while `/cdnroot` remains the container content root for the asset host +- **App is not CDN** — app-owned assets (authored in `wwwroot`, served from a per-app asset host) are separated from shared CDN assets (reusable across applications, never duplicated into any app's `wwwroot`), and the skill always asks whether a CDN equivalent exists +- **Targeted app-owned publish handling** — `` removes application-owned files while preserving Razor Class Library (`_content`), framework (`_framework`), and generated Static Web Assets +- **Proven, not assumed** — `verify --run-publish` publishes to an isolated temp directory and asserts app-owned `wwwroot` files are absent from the artifact; verification output never touches the repository +- **Safety guardrail over broken migrations** — Blazor, Blazor WebAssembly, Razor Class Library, scoped CSS, component JavaScript modules, and frontend-build scenarios are detected and escalated rather than blindly excluded; stopping to request an explicit generated-static-assets design is a successful outcome, not a failure +- **Convention-aligned local topology** — the root Docker Compose profile appends `.Assets` to the ordinary Project profile name, and it points App URLs at an `http://localhost:` origin, never a protocol-relative or `https://localhost` URL that an HTTPS page would break against an HTTP-only origin +- **Hardened local origin** — the local `web-cdn-origin:2.0.0` service mounts `wwwroot` into `/cdnroot` read-only as a non-root user, with a read-only root filesystem, no privileged mode, no Docker socket, and only the required port exposed +- **Docker naming aligned** — the derived production asset image uses the Docker-documented `.Dockerfile` form with PascalCase `Assets.Dockerfile`, selected explicitly when a non-default Dockerfile is built +- **Compose naming aligned** — a dedicated local origin topology uses `compose.assets.yml`, paired with `Assets.Dockerfile`; existing repository orchestration is extended when present +- **Every file has one correct location** — `Dockerfile`, `LocalDevelopment.Dockerfile`, and `Assets.Dockerfile` are written beside the web `.csproj` where Visual Studio Container Tools expects them, never at the repository root; `compose.assets.yml`, `.dockerignore`, `docker-compose.dcproj`, and the Compose `launchSettings.json` are written at the root. `verify --check-local` fails on a misplaced Dockerfile instead of leaving it to review +- **Literal templates, not recollection** — `assets/` ships the real content for every generated file, so the agent substitutes placeholders instead of reconstructing a Dockerfile from the familiar `mcr.microsoft.com/dotnet/sdk` multi-stage pattern that is wrong for this topology +- **Artifact-first images, enforced** — both application Dockerfiles package an already-published `artifacts/publish/` directory and never compile source. The runner rejects an SDK stage, a `dotnet build`/`dotnet publish` step, a `RUN adduser` block, an `mcr.microsoft.com` runtime, or a missing artifact copy, and it requires `LocalPublishDirectory` behind a guarded non-CI post-build target, a root `.dockerignore` that still carries `artifacts/`, and a CI job that actually produces the artifact the image copies +- **Ports derived, not invented** — the Compose web service publishes on the ordinary Project profile's HTTP `applicationUrl` port, so bookmarks, redirect registrations, and cookie scopes survive the switch between ordinary and segregated Development +- **Visual Studio orchestration is explicit** — ordinary Development keeps the existing Project profile and serves `wwwroot` directly. For one-click segregated F5, the skill registers `compose.assets.yml` through a `Microsoft.Docker.Sdk` `.dcproj` and a root `.Assets` `DockerCompose` profile. Compose directly builds the web app with `LocalDevelopment.Dockerfile` and the origin with `Assets.Dockerfile`, using the repository root as the web build context and the project directory as the asset build context; it does not use a redundant project-level segregated profile. The asset service sets `com.microsoft.visual-studio.project-name: ""`, preventing Visual Studio from associating it with the web project and injecting the web debugger bootstrap, so no `docker-compose.vs.release.yml` repair is needed. The web `.csproj` owns `LocalPublishDirectory`; local builds and CI publish the same artifact. Production `Dockerfile` copies it into the shell-less DHI ASP.NET runtime. The `-dev` image is not an SDK, none of the Dockerfiles compile the app, and completion requires a real F5 attachment check rather than only MSBuild and Compose CLI evidence +- **Reproducible source is mandatory** — the asset image input must exist in a clean checkout or come from a pinned immutable artifact. The skill flags an ignored or wholly untracked `wwwroot`; a tracked repository-level asset root is supported as an explicit design only when the ordinary web root, Docker context, CI inputs, and verification move together +- **Both deployment images reach CI** — when repository Actions already build the web application image, the skill also requires a build or validation of `Assets.Dockerfile` from the same commit +- **The artifact-first image always gets a producer** — an image whose only instruction is `COPY artifacts/publish/ .` is inert without a CI job that publishes there, so the skill closes that gap rather than waiting to be asked: it extends the existing GitHub Actions workflow, or creates one from a complete template when the repository has none, and names which it did. A workflow that only builds and tests still counts as missing a producer, and `verify` fails on it +- **Architectural motivation, stated honestly** — segregation of duties, independent deployment and scaling, explicit cache behavior, and origin/CDN offloading +- **Adapts, never imposes** — it reuses Cuemon `AppTagHelperOptions`/`CdnTagHelperOptions` when already present and the application's own base-URL setting otherwise, and never adds a Cuemon dependency just to migrate +- **Current packages, resolved deterministically** — when a real Cuemon package reference already exists, `plan` discovers NuGet's V3 package endpoint, selects the highest stable version, emits the exact version and source, preserves `Directory.Packages.props` ownership when CPM is active, and fails closed instead of recycling an old literal +- **Uses the existing asset model** — when Cuemon is detected from package/project references, namespace imports, options, `_ViewImports.cshtml`, or actual custom-element markup, it migrates App/CDN ownership to the public `app-*`/`cdn-*` helpers, uses `BaseUrlMode` for Automatic App resolution versus explicitly Configured CDN locations, and removes a redundant `AppAssetOptions`-style abstraction only after its consumers are gone +- **Reports, never rewrites** — the runner exposes Cuemon, competing-abstraction, legacy-syntax, and cache-busting interface/registration evidence for the agent to act on; it never rewrites Razor or C# source and never creates a live-model evaluation workflow +- **Idempotent and deterministic** — re-running reconciles existing segregation instead of duplicating MSBuild items, launch profiles, Compose services, or Dockerfiles, and the runner ships a built-in `--self-test` plus a PowerShell harness +- **Fail-closed verification and planning** — local verification requires both the matching HTTP launch profile and origin Compose service, generated-asset risks override existing-segregation detection, and a no-`wwwroot` project can still produce CDN-only work when a shared equivalent exists ### Why agent-smith? **agent-smith** applies one coherent engineering standard — *consistency is key* — across a whole task instead of bolting a review onto the end. Invoke it explicitly as `/agent-smith `; it also auto-triggers for engineering work such as architecture, implementation, code review, public API and compatibility analysis, testing, performance, skill authoring, security and DevSecOps, CI/CD, delivery, and governance. diff --git a/scripts/run-skill-benchmark.ps1 b/scripts/run-skill-benchmark.ps1 deleted file mode 100644 index d212083..0000000 --- a/scripts/run-skill-benchmark.ps1 +++ /dev/null @@ -1,1513 +0,0 @@ -param( - [Parameter(Mandatory = $true)] - [string]$SkillPath, - - [string]$WorkspaceRoot, - - [string]$BaselineSkillPath, - - [int]$MaxParallel = 4, - - [int]$MaxGradeParallel = 4, - - [ValidateRange(30, 3600)] - [int]$RunTimeoutSeconds = 240, - - [ValidateRange(30, 1800)] - [int]$GradeTimeoutSeconds = 120, - - [string]$Model = 'gpt-5.4', - - [string]$GraderModel, - - [int]$BenchmarkCandidateLimit = 5, - - [string]$ExecutorCommand, - - [string]$GraderCommand, - - [string[]]$EvalId, - - [switch]$CompareWithLegacy, - - [switch]$SkipSkillValidation, - - [switch]$SkipReview -) - -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -$utf8NoBom = [System.Text.UTF8Encoding]::new($false) -[Console]::InputEncoding = $utf8NoBom -[Console]::OutputEncoding = $utf8NoBom -$OutputEncoding = $utf8NoBom - -function Write-JsonFile { - param( - [Parameter(Mandatory = $true)] [string]$Path, - [Parameter(Mandatory = $true)] $Value - ) - - $directory = Split-Path -Path $Path -Parent - if (-not [string]::IsNullOrWhiteSpace($directory)) { - New-Item -ItemType Directory -Path $directory -Force | Out-Null - } - $Value | ConvertTo-Json -Depth 20 | Set-Content -LiteralPath $Path -Encoding utf8 -} - -function Write-TextFile { - param( - [Parameter(Mandatory = $true)] [string]$Path, - [AllowEmptyString()] [string]$Content - ) - - $directory = Split-Path -Path $Path -Parent - if (-not [string]::IsNullOrWhiteSpace($directory)) { - New-Item -ItemType Directory -Path $directory -Force | Out-Null - } - [System.IO.File]::WriteAllText($Path, $Content, $utf8NoBom) -} - -function Get-RepoRoot { - return (Resolve-Path (Join-Path $PSScriptRoot '..')).Path -} - -function Get-ResolvedPath { - param([Parameter(Mandatory = $true)] [string]$Path) - return $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath((Resolve-Path -LiteralPath $Path).Path) -} - -function Get-SkillMetadata { - param([Parameter(Mandatory = $true)] [string]$ResolvedSkillPath) - - $skillMdPath = Join-Path $ResolvedSkillPath 'SKILL.md' - if (-not (Test-Path -LiteralPath $skillMdPath -PathType Leaf)) { - throw "Missing SKILL.md at '$skillMdPath'." - } - - $content = [System.IO.File]::ReadAllText($skillMdPath, $utf8NoBom) - $match = [regex]::Match($content, '^\s*name:\s*(?[a-z0-9-]+)\s*$', [System.Text.RegularExpressions.RegexOptions]::Multiline) - if (-not $match.Success) { - throw "Unable to resolve skill name from '$skillMdPath'." - } - - return [pscustomobject]@{ - Name = $match.Groups['name'].Value - SkillMdPath = $skillMdPath - Content = $content - } -} - -function Get-EvalDefinitions { - param( - [Parameter(Mandatory = $true)] [string]$ResolvedSkillPath, - [string[]]$SelectedEvalId - ) - - $evalPath = Join-Path $ResolvedSkillPath 'evals\evals.json' - if (-not (Test-Path -LiteralPath $evalPath -PathType Leaf)) { - throw "Missing eval file '$evalPath'." - } - - $evals = Get-Content -LiteralPath $evalPath -Raw | ConvertFrom-Json - $items = @($evals.evals) - if ($SelectedEvalId -and $SelectedEvalId.Count -gt 0) { - $selected = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) - foreach ($id in $SelectedEvalId) { [void]$selected.Add([string]$id) } - $items = @($items | Where-Object { $selected.Contains([string]$_.id) }) - } - - return @($items | Sort-Object id) -} - -function Get-Slug { - param([Parameter(Mandatory = $true)] [string]$Text) - - $slug = ($Text.ToLowerInvariant() -replace '[^a-z0-9]+', '-') -replace '(^-+|-+$)', '' - if ([string]::IsNullOrWhiteSpace($slug)) { return 'eval' } - return $slug -} - -function Resolve-SkillCreatorRoot { - $candidates = @( - (Join-Path $HOME '.agents\skills\skill-creator'), - (Join-Path $HOME '.claude\skills\skill-creator') - ) | Where-Object { Test-Path -LiteralPath $_ -PathType Container } - if (@($candidates).Count -eq 0) { - throw "Install Anthropic's skill-creator before running the benchmark viewer." - } - return (Resolve-Path -LiteralPath $candidates[0]).Path -} - -function Get-RealDotnetPath { - $command = Get-Command dotnet -CommandType Application | Select-Object -First 1 - if ($null -eq $command) { - throw 'dotnet was not found on PATH.' - } - return $command.Source -} - -function Get-CopilotScriptPath { - $command = Get-Command copilot -ErrorAction Stop - return $command.Source -} - -function Initialize-DotnetShim { - param( - [Parameter(Mandatory = $true)] [string]$Workspace, - [Parameter(Mandatory = $true)] [string]$RealDotnet - ) - - $shimRoot = Join-Path (Join-Path $Workspace '.benchmark') 'dotnet' - New-Item -ItemType Directory -Path $shimRoot -Force | Out-Null - - $shimCommandPath = Join-Path $shimRoot 'dotnet.cmd' - $shimScriptPath = Join-Path (Join-Path $PSScriptRoot 'skill-benchmark') 'log-dotnet.ps1' - $shimContent = @" -@echo off -pwsh -NoProfile -File "$shimScriptPath" -RealDotnet "%SKILL_BENCHMARK_DOTNET_REAL%" -LogDirectory "%SKILL_BENCHMARK_DOTNET_LOG_DIR%" -StdoutPath "%SKILL_BENCHMARK_DOTNET_STDOUT%" -StderrPath "%SKILL_BENCHMARK_DOTNET_STDERR%" %* -exit /b %ERRORLEVEL% -"@ - Write-TextFile -Path $shimCommandPath -Content $shimContent - - $unixShimPath = Join-Path $shimRoot 'dotnet' - $posixShimScriptPath = "'" + $shimScriptPath.Replace("'", "'\''", [System.StringComparison]::Ordinal) + "'" - $unixShimContent = @' -#!/usr/bin/env sh -exec pwsh -NoProfile -File __SHIM_SCRIPT_PATH__ -RealDotnet "$SKILL_BENCHMARK_DOTNET_REAL" -LogDirectory "$SKILL_BENCHMARK_DOTNET_LOG_DIR" -StdoutPath "$SKILL_BENCHMARK_DOTNET_STDOUT" -StderrPath "$SKILL_BENCHMARK_DOTNET_STDERR" "$@" -'@.Replace('__SHIM_SCRIPT_PATH__', $posixShimScriptPath).Replace("`r`n", "`n") - Write-TextFile -Path $unixShimPath -Content $unixShimContent - - if (-not [System.OperatingSystem]::IsWindows()) { - & chmod +x -- $unixShimPath - if ($LASTEXITCODE -ne 0) { - throw "Unable to mark the Unix dotnet shim executable: $unixShimPath" - } - } - - return $shimRoot -} - -function New-RunManifest { - param( - [Parameter(Mandatory = $true)] [string]$RepoRoot, - [Parameter(Mandatory = $true)] [string[]]$ExcludedPrefixes - ) - - $manifest = @{} - $files = Get-ChildItem -LiteralPath $RepoRoot -Recurse -File -Force | Where-Object { - $relative = [System.IO.Path]::GetRelativePath($RepoRoot, $_.FullName).Replace('\', '/') - foreach ($prefix in $ExcludedPrefixes) { - if ($relative.StartsWith($prefix, [System.StringComparison]::OrdinalIgnoreCase)) { return $false } - } - return $true - } - - foreach ($file in $files) { - $relative = [System.IO.Path]::GetRelativePath($RepoRoot, $file.FullName).Replace('\', '/') - $manifest[$relative] = [pscustomobject]@{ - path = $file.FullName - hash = (Get-FileHash -LiteralPath $file.FullName -Algorithm SHA256).Hash - } - } - return $manifest -} - -function Get-ChangedFiles { - param( - [Parameter(Mandatory = $true)] [string]$BaselineRoot, - [Parameter(Mandatory = $true)] [string]$CurrentRoot, - [Parameter(Mandatory = $true)] [hashtable]$BaselineManifest, - [Parameter(Mandatory = $true)] [string[]]$ExcludedPrefixes - ) - - $currentManifest = New-RunManifest -RepoRoot $CurrentRoot -ExcludedPrefixes $ExcludedPrefixes - $changes = [System.Collections.Generic.List[object]]::new() - - foreach ($relative in ($BaselineManifest.Keys + $currentManifest.Keys | Sort-Object -Unique)) { - $before = if ($BaselineManifest.ContainsKey($relative)) { $BaselineManifest[$relative] } else { $null } - $after = if ($currentManifest.ContainsKey($relative)) { $currentManifest[$relative] } else { $null } - if ($null -eq $before) { - $changes.Add([pscustomobject]@{ path = $relative; change = 'added'; before = $null; after = $after.path }) - continue - } - if ($null -eq $after) { - $changes.Add([pscustomobject]@{ path = $relative; change = 'deleted'; before = $before.path; after = $null }) - continue - } - if ($before.hash -ne $after.hash) { - $changes.Add([pscustomobject]@{ path = $relative; change = 'modified'; before = $before.path; after = $after.path }) - } - } - - return @($changes | Sort-Object path) -} - -function Invoke-GitNoPager { - param( - [Parameter(Mandatory = $true)] [string[]]$Arguments - ) - - $output = & git --no-pager @Arguments 2>&1 - return [pscustomobject]@{ - ExitCode = $LASTEXITCODE - Output = @($output) - } -} - -function Write-ChangeArtifacts { - param( - [AllowEmptyCollection()] [object[]]$Changes, - [Parameter(Mandatory = $true)] [string]$OutputsPath - ) - - $summaryLines = [System.Collections.Generic.List[string]]::new() - $summaryLines.Add('# Changed Files') - $summaryLines.Add('') - if (@($Changes).Count -eq 0) { - $summaryLines.Add('No source changes were detected.') - } else { - foreach ($change in @($Changes)) { - $summaryLines.Add("- $($change.change): $($change.path)") - } - } - Write-TextFile -Path (Join-Path $OutputsPath 'changed-files.md') -Content ($summaryLines -join [Environment]::NewLine) - - $diffParts = [System.Collections.Generic.List[string]]::new() - foreach ($change in @($Changes)) { - switch ($change.change) { - 'modified' { - $diff = Invoke-GitNoPager -Arguments @('diff', '--no-index', '--', $change.before, $change.after) - $diffParts.Add(($diff.Output -join [Environment]::NewLine)) - } - 'added' { - $addedContent = Get-Content -LiteralPath $change.after -Raw - $addedText = @('+++ ' + $change.path, $addedContent) -join [Environment]::NewLine - $diffParts.Add($addedText) - } - 'deleted' { - $deletedContent = Get-Content -LiteralPath $change.before -Raw - $deletedText = @('--- ' + $change.path, $deletedContent) -join [Environment]::NewLine - $diffParts.Add($deletedText) - } - } - } - Write-TextFile -Path (Join-Path $OutputsPath 'repo.diff') -Content (($diffParts -join [Environment]::NewLine + [Environment]::NewLine).Trim()) - - $snapshotsRoot = Join-Path $OutputsPath 'snapshots' - if (@($Changes).Count -gt 0) { - New-Item -ItemType Directory -Path $snapshotsRoot -Force | Out-Null - foreach ($change in @($Changes)) { - if ($null -eq $change.after) { continue } - $destination = Join-Path $snapshotsRoot ($change.path -replace '/', '\') - $destinationDir = Split-Path -Path $destination -Parent - New-Item -ItemType Directory -Path $destinationDir -Force | Out-Null - Copy-Item -LiteralPath $change.after -Destination $destination -Force - } - } -} - -function Parse-CopilotEvents { - param([Parameter(Mandatory = $true)] [string]$JsonlPath) - - if (-not (Test-Path -LiteralPath $JsonlPath -PathType Leaf)) { return @() } - - $events = [System.Collections.Generic.List[object]]::new() - foreach ($line in Get-Content -LiteralPath $JsonlPath) { - if ([string]::IsNullOrWhiteSpace($line)) { continue } - try { - [void]$events.Add(($line | ConvertFrom-Json)) - } catch { - } - } - return @($events) -} - -function Get-PowerShellCommandDurations { - param( - [AllowEmptyCollection()] [object[]]$Events - ) - - $started = @{} - $summary = [ordered]@{ - restoreSeconds = 0.0 - buildSeconds = 0.0 - testSeconds = 0.0 - resolverSeconds = 0.0 - } - - foreach ($event in @($Events)) { - if ($event.type -eq 'tool.execution_start' -and [string]$event.data.toolName -eq 'powershell') { - $started[[string]$event.data.toolCallId] = $event - continue - } - if ($event.type -ne 'tool.execution_complete') { - continue - } - - $toolCallId = [string]$event.data.toolCallId - if (-not $started.ContainsKey($toolCallId)) { continue } - - $startEvent = $started[$toolCallId] - if ([string]$startEvent.data.toolName -ne 'powershell') { continue } - $durationSeconds = [math]::Round(([DateTimeOffset]::Parse([string]$event.timestamp) - [DateTimeOffset]::Parse([string]$startEvent.timestamp)).TotalSeconds, 3) - $command = [string]$startEvent.data.arguments.command - - if ($command -match '(^|[;\s])dotnet\s+restore(\s|$)') { - $summary.restoreSeconds += $durationSeconds - } - if ($command -match '(^|[;\s])dotnet\s+build(\s|$)') { - $summary.buildSeconds += $durationSeconds - } - if ($command -match '(^|[;\s])dotnet\s+test(\s|$)') { - $summary.testSeconds += $durationSeconds - } - if ($command -match 'resolve-test-package-versions\.ps1') { - $summary.resolverSeconds += $durationSeconds - } - } - - return [ordered]@{ - restoreSeconds = [math]::Round([double]$summary.restoreSeconds, 3) - buildSeconds = [math]::Round([double]$summary.buildSeconds, 3) - testSeconds = [math]::Round([double]$summary.testSeconds, 3) - resolverSeconds = [math]::Round([double]$summary.resolverSeconds, 3) - } -} - -function Get-CopilotMetrics { - param( - [AllowEmptyCollection()] [object[]]$Events, - [Parameter(Mandatory = $true)] [string]$TranscriptPath, - [Parameter(Mandatory = $true)] [string]$OutputsPath - ) - - $toolCalls = @{} - $errors = 0 - foreach ($event in @($Events)) { - if ($event.type -eq 'tool.execution_start') { - $toolName = [string]$event.data.toolName - if (-not $toolCalls.ContainsKey($toolName)) { $toolCalls[$toolName] = 0 } - $toolCalls[$toolName]++ - } - if ($event.type -eq 'tool.execution_complete' -and -not [bool]$event.data.success) { - $errors++ - } - } - - $transcriptChars = if (Test-Path -LiteralPath $TranscriptPath) { (Get-Content -LiteralPath $TranscriptPath -Raw).Length } else { 0 } - $outputChars = 0 - if (Test-Path -LiteralPath $OutputsPath -PathType Container) { - foreach ($file in Get-ChildItem -LiteralPath $OutputsPath -Recurse -File -Force) { - $outputChars += $file.Length - } - } - - $toolCallTotal = @($toolCalls.Keys | ForEach-Object { $toolCalls[$_] } | Measure-Object -Sum).Sum - if ($null -eq $toolCallTotal) { $toolCallTotal = 0 } - - return [ordered]@{ - tool_calls = $toolCalls - total_tool_calls = $toolCallTotal - total_steps = @($Events | Where-Object type -eq 'assistant.turn_end').Count - errors_encountered = $errors - output_chars = $outputChars - transcript_chars = $transcriptChars - } -} - -function Kill-ProcessTree { - param([Parameter(Mandatory = $true)] [System.Diagnostics.Process]$Process) - - if ($Process.HasExited) { - return [pscustomobject]@{ - attempted = $false - completed = $true - exitCode = $Process.ExitCode - } - } - - try { - $Process.Kill($true) - $Process.WaitForExit(5000) | Out-Null - return [pscustomobject]@{ - attempted = $true - completed = $Process.HasExited - exitCode = if ($Process.HasExited) { $Process.ExitCode } else { $null } - } - } catch { - return [pscustomobject]@{ - attempted = $true - completed = $false - error = $_.Exception.Message - } - } -} - -function Write-TranscriptFallback { - param( - [Parameter(Mandatory = $true)] [string]$TranscriptPath, - [Parameter(Mandatory = $true)] [string]$Prompt, - [AllowEmptyCollection()] [object[]]$Events, - [string]$StderrText - ) - - if (Test-Path -LiteralPath $TranscriptPath -PathType Leaf) { return } - - $assistantMessages = @($Events | Where-Object type -eq 'assistant.message' | ForEach-Object { [string]$_.data.content }) - $content = @( - '# Copilot CLI Session (Fallback)' - '' - '## Eval Prompt' - '' - $Prompt - '' - '## Assistant Output' - '' - if (@($assistantMessages).Count -gt 0) { ($assistantMessages -join [Environment]::NewLine + [Environment]::NewLine) } else { '(No final assistant message was captured.)' } - ) - if (-not [string]::IsNullOrWhiteSpace($StderrText)) { - $content += @( - '', - '## stderr', - '', - '```text', - $StderrText.Trim(), - '```' - ) - } - Write-TextFile -Path $TranscriptPath -Content ($content -join [Environment]::NewLine) -} - -function Get-CopilotFinalMessage { - param([AllowEmptyCollection()] [object[]]$Events) - - $message = $Events | Where-Object type -eq 'assistant.message' | Select-Object -Last 1 - if ($null -eq $message) { return $null } - return [string]$message.data.content -} - -function Get-UsageResult { - param([AllowEmptyCollection()] [object[]]$Events) - - $result = $Events | Where-Object type -eq 'result' | Select-Object -Last 1 - if ($null -eq $result) { return $null } - return $result -} - -function Summarize-DotnetLogs { - param( - [Parameter(Mandatory = $true)] [string]$LogDirectory, - [Parameter(Mandatory = $true)] [string]$OutputsPath - ) - - if (-not (Test-Path -LiteralPath $LogDirectory -PathType Container)) { - return [ordered]@{ - totals = [ordered]@{ - restoreSeconds = 0 - buildSeconds = 0 - testSeconds = 0 - } - commands = @() - } - } - - $entries = @( - Get-ChildItem -LiteralPath $LogDirectory -Filter 'dotnet-*.json' -File -Force | - Sort-Object Name | - ForEach-Object { Get-Content -LiteralPath $_.FullName -Raw | ConvertFrom-Json } - ) - - $restoreSeconds = 0.0 - $buildSeconds = 0.0 - $testSeconds = 0.0 - $summaryLines = [System.Collections.Generic.List[string]]::new() - $summaryLines.Add('# dotnet command summary') - $summaryLines.Add('') - foreach ($entry in @($entries)) { - switch ([string]$entry.command) { - 'restore' { $restoreSeconds += [double]$entry.durationSeconds } - 'build' { $buildSeconds += [double]$entry.durationSeconds } - 'test' { $testSeconds += [double]$entry.durationSeconds } - } - $summaryLines.Add("- $($entry.command) exit $($entry.exitCode) in $($entry.durationSeconds)s") - } - Write-TextFile -Path (Join-Path $OutputsPath 'dotnet-summary.md') -Content ($summaryLines -join [Environment]::NewLine) - - return [ordered]@{ - totals = [ordered]@{ - restoreSeconds = [math]::Round($restoreSeconds, 3) - buildSeconds = [math]::Round($buildSeconds, 3) - testSeconds = [math]::Round($testSeconds, 3) - } - commands = @($entries) - } -} - -function Summarize-ResolverTrace { - param( - [Parameter(Mandatory = $true)] [string]$TracePath, - [Parameter(Mandatory = $true)] [string]$OutputsPath - ) - - if (-not (Test-Path -LiteralPath $TracePath -PathType Leaf)) { - return [ordered]@{ - durationSeconds = 0 - cacheHits = 0 - calls = 0 - } - } - - $events = [System.Collections.Generic.List[object]]::new() - foreach ($line in Get-Content -LiteralPath $TracePath) { - if ([string]::IsNullOrWhiteSpace($line)) { continue } - try { [void]$events.Add(($line | ConvertFrom-Json)) } catch { } - } - - $summaryLines = [System.Collections.Generic.List[string]]::new() - $summaryLines.Add('# resolver summary') - $summaryLines.Add('') - foreach ($event in @($events)) { - $summaryLines.Add("- role $($event.role) tfms $($event.targetFrameworks -join ';') cacheHit $($event.cacheHit) duration $($event.durationSeconds)s") - } - Write-TextFile -Path (Join-Path $OutputsPath 'resolver-summary.md') -Content ($summaryLines -join [Environment]::NewLine) - - return [ordered]@{ - durationSeconds = [math]::Round((@($events | ForEach-Object { [double]$_.durationSeconds } | Measure-Object -Sum).Sum), 3) - cacheHits = @($events | Where-Object cacheHit).Count - calls = @($events).Count - events = @($events) - } -} - -function Write-FallbackGrading { - param( - [Parameter(Mandatory = $true)] [string]$GradingPath, - [Parameter(Mandatory = $true)] [object[]]$Expectations, - [Parameter(Mandatory = $true)] [string]$Reason, - [Parameter(Mandatory = $true)] $Timing, - [Parameter(Mandatory = $true)] $Metrics - ) - - $entries = foreach ($expectation in @($Expectations)) { - [ordered]@{ - text = [string]$expectation - passed = $false - evidence = $Reason - } - } - $total = @($entries).Count - Write-JsonFile -Path $GradingPath -Value ([ordered]@{ - expectations = @($entries) - summary = [ordered]@{ - passed = 0 - failed = $total - total = $total - pass_rate = 0 - } - execution_metrics = $Metrics - timing = $Timing - claims = @() - user_notes_summary = [ordered]@{ - uncertainties = @() - needs_review = @() - workarounds = @($Reason) - } - }) -} - -function New-CopilotProcessArguments { - param( - [Parameter(Mandatory = $true)] [string]$Prompt, - [Parameter(Mandatory = $true)] [string]$WorkingDirectory, - [Parameter(Mandatory = $true)] [string]$ModelName, - [string]$SharePath, - [switch]$DisableSkillTool - ) - - $arguments = [System.Collections.Generic.List[string]]::new() - foreach ($argument in @( - '-p', $Prompt, - '--allow-all', - '--no-ask-user', - '--output-format', 'json', - '--no-custom-instructions', - '--disable-builtin-mcps', - '--disable-mcp-server', 'openart.ai', - '--disable-mcp-server', 'sonarqube', - '--disable-mcp-server', 'github-mcp-server', - '--model', $ModelName, - '-C', $WorkingDirectory - )) { - [void]$arguments.Add($argument) - } - if ($DisableSkillTool) { - [void]$arguments.Add('--excluded-tools') - [void]$arguments.Add('skill') - } - if (-not [string]::IsNullOrWhiteSpace($SharePath)) { - [void]$arguments.Add('--share') - [void]$arguments.Add($SharePath) - } - return @($arguments) -} - -function New-CopilotWrapperScript { - param( - [Parameter(Mandatory = $true)] [string]$WrapperPath, - [Parameter(Mandatory = $true)] [string]$CopilotPowerShellPath, - [Parameter(Mandatory = $true)] [string]$Prompt, - [Parameter(Mandatory = $true)] [string]$WorkingDirectory, - [Parameter(Mandatory = $true)] [string]$ModelName, - [string]$SharePath, - [switch]$DisableSkillTool - ) - - $content = @( - '$prompt = @''' - $Prompt - '''@' - '$arguments = [System.Collections.Generic.List[string]]::new()' - '$arguments.Add(''-p'')' - '$arguments.Add($prompt)' - '$arguments.Add(''--allow-all'')' - '$arguments.Add(''--no-ask-user'')' - '$arguments.Add(''--output-format'')' - '$arguments.Add(''json'')' - '$arguments.Add(''--no-custom-instructions'')' - '$arguments.Add(''--disable-builtin-mcps'')' - '$arguments.Add(''--disable-mcp-server'')' - '$arguments.Add(''openart.ai'')' - '$arguments.Add(''--disable-mcp-server'')' - '$arguments.Add(''sonarqube'')' - '$arguments.Add(''--disable-mcp-server'')' - '$arguments.Add(''github-mcp-server'')' - ('$arguments.Add(''' + '--model' + ''')') - ('$arguments.Add(''' + $ModelName.Replace("'", "''") + ''')') - ('$arguments.Add(''' + '-C' + ''')') - ('$arguments.Add(''' + $WorkingDirectory.Replace("'", "''") + ''')') - if ($DisableSkillTool) { - '$arguments.Add(''--excluded-tools'')' - '$arguments.Add(''skill'')' - } - if (-not [string]::IsNullOrWhiteSpace($SharePath)) { - ('$arguments.Add(''' + '--share' + ''')') - ('$arguments.Add(''' + $SharePath.Replace("'", "''") + ''')') - } - ("& '{0}' @arguments" -f $CopilotPowerShellPath.Replace("'", "''")) - 'exit $LASTEXITCODE' - ) -join [Environment]::NewLine - - Write-TextFile -Path $WrapperPath -Content ($content + [Environment]::NewLine) -} - -function Start-CommandProcess { - param( - [Parameter(Mandatory = $true)] [string]$FilePath, - [Parameter(Mandatory = $true)] [string[]]$Arguments, - [Parameter(Mandatory = $true)] [string]$WorkingDirectory, - [Parameter(Mandatory = $true)] [hashtable]$Environment, - [Parameter(Mandatory = $true)] [string]$StdoutPath, - [Parameter(Mandatory = $true)] [string]$StderrPath - ) - - $mergedEnvironment = @{} - foreach ($entry in Get-ChildItem Env:) { - $mergedEnvironment[$entry.Name] = $entry.Value - } - foreach ($key in $Environment.Keys) { - $mergedEnvironment[$key] = [string]$Environment[$key] - } - - $startInfo = @{ - FilePath = $FilePath - ArgumentList = @($Arguments) - WorkingDirectory = $WorkingDirectory - RedirectStandardOutput = $StdoutPath - RedirectStandardError = $StderrPath - PassThru = $true - NoNewWindow = $true - Environment = $mergedEnvironment - } - - try { - $process = Start-Process @startInfo - } catch { - throw "Unable to start '$FilePath' in '$WorkingDirectory'. stdout='$StdoutPath' stderr='$StderrPath'. $($_.Exception.Message)" - } - return [pscustomobject]@{ - Process = $process - } -} - -function Invoke-SkillValidation { - param( - [Parameter(Mandatory = $true)] [string]$ResolvedSkillPath, - [Parameter(Mandatory = $true)] [string]$Workspace - ) - - $validateScript = Join-Path $ResolvedSkillPath 'scripts\validate-skill.ps1' - if (-not (Test-Path -LiteralPath $validateScript -PathType Leaf)) { - return [ordered]@{ - ran = $false - exitCode = $null - durationSeconds = 0 - } - } - - $logPath = Join-Path $Workspace '.benchmark\skill-validation.log' - New-Item -ItemType Directory -Path (Split-Path -Path $logPath -Parent) -Force | Out-Null - $start = [DateTimeOffset]::UtcNow - $output = & pwsh -NoProfile -File $validateScript 2>&1 - $exitCode = $LASTEXITCODE - Write-TextFile -Path $logPath -Content (($output -join [Environment]::NewLine) + [Environment]::NewLine) - return [ordered]@{ - ran = $true - exitCode = $exitCode - durationSeconds = [math]::Round(([DateTimeOffset]::UtcNow - $start).TotalSeconds, 3) - logPath = $logPath - } -} - -function Get-DotnetTestContexts { - param( - [Parameter(Mandatory = $true)] [object[]]$Evals, - [Parameter(Mandatory = $true)] [string]$ResolvedSkillPath - ) - - $inspectScript = Join-Path $ResolvedSkillPath 'scripts\inspect-dotnet-tests.ps1' - if (-not (Test-Path -LiteralPath $inspectScript -PathType Leaf)) { return @() } - - $contexts = [System.Collections.Generic.List[object]]::new() - foreach ($eval in @($Evals)) { - $files = @($eval.files) - if (@($files).Count -eq 0) { continue } - - $roots = @($files | ForEach-Object { ($_ -replace '/', '\').Split('\')[2] } | Sort-Object -Unique) - if (@($roots).Count -ne 1) { continue } - $fixtureRoot = Join-Path $ResolvedSkillPath (Join-Path 'evals\files' $roots[0]) - $testProject = @($files | Where-Object { $_ -match 'test/.+\.csproj$' } | Select-Object -First 1) - if ([string]::IsNullOrWhiteSpace($testProject)) { - $testProject = @($files | Where-Object { $_ -match '\.csproj$' } | Select-Object -First 1) - } - if ([string]::IsNullOrWhiteSpace($testProject)) { continue } - - $relativeProject = ($testProject -replace '^evals/files/[^/]+/', '') -replace '/', '\' - $result = & pwsh -NoProfile -File $inspectScript -RepoRoot $fixtureRoot -ProjectPath $relativeProject 2>&1 - if ($LASTEXITCODE -ne 0) { continue } - - try { - $json = ($result -join [Environment]::NewLine) | ConvertFrom-Json - } catch { - continue - } - $project = @($json.projects | Select-Object -First 1) - if ($null -eq $project) { continue } - - $role = switch ([string]$project.role) { - 'Ordinary unit test' { 'Unit' } - 'ASP.NET Core functional test' { 'WebFunctional' } - 'Console or worker functional test' { 'ApplicationFunctional' } - default { $null } - } - if ($null -eq $role) { continue } - - $key = '{0}|{1}' -f $role, (@($project.frameworks) -join ';') - $contexts.Add([pscustomobject]@{ - key = $key - role = $role - targetFrameworks = @($project.frameworks) - }) - } - - return @($contexts | Sort-Object key -Unique) -} - -function Invoke-DotnetTestPrewarm { - param( - [Parameter(Mandatory = $true)] [object[]]$Contexts, - [Parameter(Mandatory = $true)] [string]$ResolvedSkillPath, - [Parameter(Mandatory = $true)] [string]$CacheDirectory, - [Parameter(Mandatory = $true)] [string]$TracePath, - [Parameter(Mandatory = $true)] [int]$CandidateLimit - ) - - if (@($Contexts).Count -eq 0) { - return [ordered]@{ - ran = $false - durationSeconds = 0 - contexts = @() - } - } - - $resolver = Join-Path $ResolvedSkillPath 'scripts\resolve-test-package-versions.ps1' - $start = [DateTimeOffset]::UtcNow - $records = [System.Collections.Generic.List[object]]::new() - foreach ($context in @($Contexts)) { - $invocationStart = [DateTimeOffset]::UtcNow - $output = & pwsh -NoProfile -File $resolver -TargetFramework $context.targetFrameworks -Role $context.role -MaximumCandidates $CandidateLimit -CacheDirectory $CacheDirectory -TraceFile $TracePath 2>&1 - $records.Add([ordered]@{ - role = $context.role - targetFrameworks = @($context.targetFrameworks) - exitCode = $LASTEXITCODE - durationSeconds = [math]::Round(([DateTimeOffset]::UtcNow - $invocationStart).TotalSeconds, 3) - output = ($output -join [Environment]::NewLine) - }) - if ($LASTEXITCODE -ne 0) { - throw "Dotnet-test prewarm failed for role '$($context.role)'." - } - } - - return [ordered]@{ - ran = $true - durationSeconds = [math]::Round(([DateTimeOffset]::UtcNow - $start).TotalSeconds, 3) - contexts = @($records) - } -} - -function Get-CopilotGraderPrompt { - param( - [Parameter(Mandatory = $true)] [string]$SkillCreatorRoot, - [Parameter(Mandatory = $true)] [string]$TranscriptPath, - [Parameter(Mandatory = $true)] [string]$OutputsPath, - [Parameter(Mandatory = $true)] [string]$TimingPath, - [Parameter(Mandatory = $true)] [string[]]$Expectations - ) - - $graderInstructions = [System.IO.File]::ReadAllText((Join-Path $SkillCreatorRoot 'agents\grader.md'), $utf8NoBom) - $expectationsJson = ($Expectations | ConvertTo-Json) - return @" -$graderInstructions - -Read these artifacts, then output only the grading JSON object: -- transcript: $TranscriptPath -- outputs directory: $OutputsPath -- timing: $TimingPath - -Expectations: -$expectationsJson - -Do not edit any files. Output JSON only. -"@ -} - -function Start-BenchmarkRun { - param( - [Parameter(Mandatory = $true)] $RunPlan, - [Parameter(Mandatory = $true)] [string]$SkillName, - [Parameter(Mandatory = $true)] [string]$ResolvedSkillPath, - [AllowNull()] [string]$ResolvedBaselineSkillPath, - [Parameter(Mandatory = $true)] [string]$ModelName, - [Parameter(Mandatory = $true)] [string]$Workspace, - [Parameter(Mandatory = $true)] [string]$DotnetShimRoot, - [Parameter(Mandatory = $true)] [string]$RealDotnet, - [Parameter(Mandatory = $true)] [string]$CopilotScriptPath, - [Parameter(Mandatory = $true)] [string]$SharedNugetPackages, - [Parameter(Mandatory = $true)] [bool]$UseOptimizedDotnetMode, - [AllowNull()] [string]$ExecutorCommandPath - ) - - $runRoot = $RunPlan.runRoot - $repoRoot = Join-Path $runRoot 'repo' - $outputsPath = Join-Path $runRoot 'outputs' - New-Item -ItemType Directory -Path $repoRoot,$outputsPath,(Join-Path $runRoot '.benchmark') -Force | Out-Null - - $excludedPrefixes = @('.agents/', '.claude/', '.benchmark/', '.git/', 'bin/', 'obj/') - - if (Test-Path -LiteralPath $RunPlan.fixtureRoot -PathType Container) { - foreach ($item in Get-ChildItem -LiteralPath $RunPlan.fixtureRoot -Force) { - Copy-Item -LiteralPath $item.FullName -Destination $repoRoot -Recurse -Force - } - } - - if ($RunPlan.configuration -eq 'with_skill') { - foreach ($skillDirectory in @('.agents\skills', '.claude\skills')) { - $destination = Join-Path $repoRoot (Join-Path $skillDirectory $SkillName) - New-Item -ItemType Directory -Path (Split-Path -Path $destination -Parent) -Force | Out-Null - Copy-Item -LiteralPath $ResolvedSkillPath -Destination $destination -Recurse -Force - } - } elseif (-not [string]::IsNullOrWhiteSpace($ResolvedBaselineSkillPath)) { - foreach ($skillDirectory in @('.agents\skills', '.claude\skills')) { - $destination = Join-Path $repoRoot (Join-Path $skillDirectory $SkillName) - New-Item -ItemType Directory -Path (Split-Path -Path $destination -Parent) -Force | Out-Null - Copy-Item -LiteralPath $ResolvedBaselineSkillPath -Destination $destination -Recurse -Force - } - } - - $initialManifest = New-RunManifest -RepoRoot $repoRoot -ExcludedPrefixes $excludedPrefixes - Write-JsonFile -Path (Join-Path $runRoot '.benchmark\initial-manifest.json') -Value $initialManifest - - $context = [ordered]@{ - eval_id = [int]$RunPlan.eval.id - eval_name = $RunPlan.evalName - prompt = [string]$RunPlan.eval.prompt - expected_output = [string]$RunPlan.eval.expected_output - expectations = @($RunPlan.eval.expectations) - configuration = $RunPlan.configuration - run_root = $runRoot - repo_root = $repoRoot - outputs_root = $outputsPath - skill_name = $SkillName - model = $ModelName - profile = $RunPlan.profileName - } - $contextPath = Join-Path $runRoot '.benchmark\run-context.json' - Write-JsonFile -Path $contextPath -Value $context - - $prompt = if ($RunPlan.configuration -eq 'with_skill') { -@" -Use the skill tool to invoke only the "$SkillName" skill before you begin. Do not invoke any other skill. -Work only in the current directory. -Do not ask the user questions; make reasonable assumptions and finish the task. -The current directory already contains any attached fixture files for this eval. - -Task: -$($RunPlan.eval.prompt) -"@ - } else { -@" -This is the baseline run without the skill tool. -Work only in the current directory. -Do not ask the user questions; make reasonable assumptions and finish the task. -The current directory already contains any attached fixture files for this eval. - -Task: -$($RunPlan.eval.prompt) -"@ - } - - $transcriptPath = Join-Path $runRoot 'transcript.md' - $resultSummaryPath = Join-Path $runRoot 'result-summary.md' - $stdoutPath = Join-Path $runRoot '.benchmark\executor.stdout.jsonl' - $stderrPath = Join-Path $runRoot '.benchmark\executor.stderr.log' - $sharePath = $transcriptPath - - $environment = @{ - PYTHONUTF8 = '1' - SKILL_BENCHMARK_DOTNET_REAL = $RealDotnet - SKILL_BENCHMARK_DOTNET_LOG_DIR = (Join-Path $runRoot '.benchmark\dotnet') - SKILL_BENCHMARK_DOTNET_STDOUT = (Join-Path $runRoot '.benchmark\dotnet.stdout.log') - SKILL_BENCHMARK_DOTNET_STDERR = (Join-Path $runRoot '.benchmark\dotnet.stderr.log') - PATH = $DotnetShimRoot + [System.IO.Path]::PathSeparator + $env:PATH - NUGET_PACKAGES = $SharedNugetPackages - NUGET_HTTP_CACHE_PATH = (Join-Path $Workspace '.nuget\http-cache') - } - - if ($UseOptimizedDotnetMode) { - $environment['DOTNET_TEST_RESOLVER_CACHE_DIR'] = (Join-Path $Workspace '.benchmark\resolver-cache') - $environment['DOTNET_TEST_RESOLVER_TRACE_FILE'] = (Join-Path $runRoot '.benchmark\resolver-trace.jsonl') - $environment['DOTNET_TEST_MAXIMUM_CANDIDATES'] = [string]$BenchmarkCandidateLimit - } - - $handle = if ([string]::IsNullOrWhiteSpace($ExecutorCommandPath)) { - $wrapperPath = Join-Path $runRoot '.benchmark\invoke-copilot-executor.ps1' - New-CopilotWrapperScript -WrapperPath $wrapperPath -CopilotPowerShellPath $CopilotScriptPath -Prompt $prompt -WorkingDirectory $repoRoot -ModelName $ModelName -SharePath $sharePath -DisableSkillTool:($RunPlan.configuration -ne 'with_skill') - Start-CommandProcess -FilePath 'pwsh' -Arguments @('-NoProfile', '-File', $wrapperPath) -WorkingDirectory $repoRoot -Environment $environment -StdoutPath $stdoutPath -StderrPath $stderrPath - } else { - Start-CommandProcess -FilePath 'pwsh' -Arguments @('-NoProfile', '-File', $ExecutorCommandPath, '-ContextPath', $contextPath, '-RunRoot', $runRoot, '-TranscriptPath', $transcriptPath, '-ResultSummaryPath', $resultSummaryPath, '-OutputsPath', $outputsPath) -WorkingDirectory $repoRoot -Environment $environment -StdoutPath $stdoutPath -StderrPath $stderrPath - } - - return [pscustomobject]@{ - plan = $RunPlan - handle = $handle - context = $context - contextPath = $contextPath - repoRoot = $repoRoot - outputsPath = $outputsPath - transcriptPath = $transcriptPath - resultSummaryPath = $resultSummaryPath - stdoutPath = $stdoutPath - stderrPath = $stderrPath - startedAt = [DateTimeOffset]::UtcNow - initialManifest = $initialManifest - useOptimizedDotnetMode = $UseOptimizedDotnetMode - } -} - -function Finalize-BenchmarkRun { - param( - [Parameter(Mandatory = $true)] $ActiveRun, - [Parameter(Mandatory = $true)] [bool]$TimedOut - ) - - $runRoot = $ActiveRun.plan.runRoot - $cleanup = if ($TimedOut) { Kill-ProcessTree -Process $ActiveRun.handle.Process } else { [ordered]@{ attempted = $false; completed = $true; exitCode = $ActiveRun.handle.Process.ExitCode } } - $ActiveRun.handle.Process.WaitForExit() - - $stderrText = if (Test-Path -LiteralPath $ActiveRun.stderrPath) { (Get-Content -LiteralPath $ActiveRun.stderrPath -Raw) } else { '' } - $stdoutEvents = @(Parse-CopilotEvents -JsonlPath $ActiveRun.stdoutPath) - Write-TranscriptFallback -TranscriptPath $ActiveRun.transcriptPath -Prompt $ActiveRun.context.prompt -Events $stdoutEvents -StderrText $stderrText - - $assistantResponsePath = Join-Path $ActiveRun.outputsPath 'assistant-response.md' - $finalMessage = Get-CopilotFinalMessage -Events $stdoutEvents - if ([string]::IsNullOrWhiteSpace($finalMessage) -and (Test-Path -LiteralPath $assistantResponsePath -PathType Leaf)) { - $finalMessage = (Get-Content -LiteralPath $assistantResponsePath -Raw) - } - if ([string]::IsNullOrWhiteSpace($finalMessage)) { - $finalMessage = if ($TimedOut) { 'The executor timed out before producing a final assistant message.' } elseif ($ActiveRun.handle.Process.ExitCode -ne 0) { "The executor exited with code $($ActiveRun.handle.Process.ExitCode)." } else { 'The executor completed without a final assistant message.' } - } - Write-TextFile -Path $assistantResponsePath -Content ($finalMessage.Trim() + [Environment]::NewLine) - if (-not (Test-Path -LiteralPath $ActiveRun.resultSummaryPath)) { - Write-TextFile -Path $ActiveRun.resultSummaryPath -Content ($finalMessage.Trim() + [Environment]::NewLine) - } - - $changes = @(Get-ChangedFiles -BaselineRoot $ActiveRun.plan.fixtureRoot -CurrentRoot $ActiveRun.repoRoot -BaselineManifest $ActiveRun.initialManifest -ExcludedPrefixes @('.agents/', '.claude/', '.benchmark/', '.git/', 'bin/', 'obj/')) - Write-ChangeArtifacts -Changes $changes -OutputsPath $ActiveRun.outputsPath - - $dotnetSummary = Summarize-DotnetLogs -LogDirectory (Join-Path $runRoot '.benchmark\dotnet') -OutputsPath $ActiveRun.outputsPath - $commandDurations = Get-PowerShellCommandDurations -Events $stdoutEvents - $resolverSummary = Summarize-ResolverTrace -TracePath (Join-Path $runRoot '.benchmark\resolver-trace.jsonl') -OutputsPath $ActiveRun.outputsPath - $metrics = Get-CopilotMetrics -Events $stdoutEvents -TranscriptPath $ActiveRun.transcriptPath -OutputsPath $ActiveRun.outputsPath - Write-JsonFile -Path (Join-Path $ActiveRun.outputsPath 'metrics.json') -Value $metrics - - $resultEvent = Get-UsageResult -Events $stdoutEvents - $endedAt = [DateTimeOffset]::UtcNow - $cleanupError = if ($cleanup.PSObject.Properties.Name -contains 'error') { $cleanup.error } else { $null } - $timing = [ordered]@{ - executor = [ordered]@{ - startedAt = $ActiveRun.startedAt.ToString('O') - endedAt = $endedAt.ToString('O') - durationSeconds = [math]::Round(($endedAt - $ActiveRun.startedAt).TotalSeconds, 3) - exitCode = if ($ActiveRun.handle.Process.HasExited) { $ActiveRun.handle.Process.ExitCode } else { $null } - result = $resultEvent - } - cleanup = [ordered]@{ - timedOut = $TimedOut - attempted = [bool]$cleanup.attempted - completed = [bool]$cleanup.completed - exitCode = $cleanup.exitCode - error = $cleanupError - } - dotnet = $dotnetSummary.totals - resolver = [ordered]@{ - durationSeconds = if ([double]$commandDurations.resolverSeconds -gt 0) { $commandDurations.resolverSeconds } else { $resolverSummary.durationSeconds } - cacheHits = $resolverSummary.cacheHits - calls = $resolverSummary.calls - } - commandDurations = $commandDurations - total_duration_seconds = [math]::Round(($endedAt - $ActiveRun.startedAt).TotalSeconds, 3) - duration_ms = [int][math]::Round(($endedAt - $ActiveRun.startedAt).TotalMilliseconds) - total_tokens = 0 - } - if ([double]$commandDurations.restoreSeconds -gt 0) { $timing.dotnet.restoreSeconds = $commandDurations.restoreSeconds } - if ([double]$commandDurations.buildSeconds -gt 0) { $timing.dotnet.buildSeconds = $commandDurations.buildSeconds } - if ([double]$commandDurations.testSeconds -gt 0) { $timing.dotnet.testSeconds = $commandDurations.testSeconds } - Write-JsonFile -Path (Join-Path $runRoot 'timing.json') -Value $timing - - return [pscustomobject]@{ - runRoot = $runRoot - transcriptPath = $ActiveRun.transcriptPath - outputsPath = $ActiveRun.outputsPath - timing = $timing - metrics = $metrics - context = $ActiveRun.context - stdoutPath = $ActiveRun.stdoutPath - completedAt = $endedAt - } -} - -function Invoke-Profile { - param( - [Parameter(Mandatory = $true)] [string]$ProfileName, - [Parameter(Mandatory = $true)] [string]$ResolvedSkillPath, - [AllowNull()] [string]$ResolvedBaselineSkillPath, - [Parameter(Mandatory = $true)] [object[]]$Evals, - [Parameter(Mandatory = $true)] [object]$SkillMetadata, - [Parameter(Mandatory = $true)] [string]$Workspace, - [Parameter(Mandatory = $true)] [int]$ExecutorParallelism, - [Parameter(Mandatory = $true)] [int]$GraderParallelism, - [Parameter(Mandatory = $true)] [bool]$EnableOptimizations, - [Parameter(Mandatory = $true)] [string]$CopilotScriptPath, - [AllowNull()] [string]$ExecutorCommandPath, - [AllowNull()] [string]$GraderCommandPath, - [Parameter(Mandatory = $true)] [string]$SkillCreatorRootPath, - [Parameter(Mandatory = $true)] $ValidationResult, - [Parameter(Mandatory = $true)] $PrewarmResult, - [Parameter(Mandatory = $true)] [int]$RunTimeout, - [Parameter(Mandatory = $true)] [int]$GradeTimeout, - [Parameter(Mandatory = $true)] [string]$ModelName, - [Parameter(Mandatory = $true)] [string]$GraderModelName - ) - - $iterationRoot = Join-Path $Workspace ('iteration-' + $ProfileName) - if (Test-Path -LiteralPath $iterationRoot) { - Remove-Item -LiteralPath $iterationRoot -Recurse -Force - } - New-Item -ItemType Directory -Path $iterationRoot -Force | Out-Null - - $profileStart = [DateTimeOffset]::UtcNow - - $realDotnet = Get-RealDotnetPath - $sharedNugetPackages = Join-Path $Workspace '.nuget\packages' - New-Item -ItemType Directory -Path $sharedNugetPackages,(Join-Path $Workspace '.nuget\http-cache') -Force | Out-Null - $dotnetShimRoot = Initialize-DotnetShim -Workspace $Workspace -RealDotnet $realDotnet - - $plans = [System.Collections.Generic.Queue[object]]::new() - foreach ($eval in @($Evals)) { - $evalSlug = Get-Slug -Text ([string]$eval.prompt) - if ($evalSlug.Length -gt 24) { - $evalSlug = $evalSlug.Substring(0, 24).Trim('-') - } - if ([string]::IsNullOrWhiteSpace($evalSlug)) { - $evalSlug = 'eval' - } - $evalName = 'eval-{0:D2}-{1}' -f [int]$eval.id, $evalSlug - $evalRoot = Join-Path $iterationRoot $evalName - $fixtureRoot = Join-Path $evalRoot 'fixtures' - New-Item -ItemType Directory -Path $fixtureRoot -Force | Out-Null - foreach ($file in @($eval.files)) { - $source = Join-Path $ResolvedSkillPath ($file -replace '/', '\') - $relative = ($file -replace '^evals/files/[^/]+/', '') -replace '/', '\' - $destination = Join-Path $fixtureRoot $relative - New-Item -ItemType Directory -Path (Split-Path -Path $destination -Parent) -Force | Out-Null - Copy-Item -LiteralPath $source -Destination $destination -Force - } - - Write-JsonFile -Path (Join-Path $evalRoot 'eval_metadata.json') -Value ([ordered]@{ - eval_id = [int]$eval.id - eval_name = $evalName - prompt = [string]$eval.prompt - assertions = @($eval.expectations) - }) - - foreach ($configuration in @('with_skill', 'without_skill')) { - $runRoot = Join-Path $evalRoot (Join-Path $configuration 'run-1') - New-Item -ItemType Directory -Path $runRoot -Force | Out-Null - $plans.Enqueue([pscustomobject]@{ - eval = $eval - evalName = $evalName - evalRoot = $evalRoot - fixtureRoot = $fixtureRoot - configuration = $configuration - runRoot = $runRoot - profileName = $ProfileName - }) - } - } - - $activeRuns = [System.Collections.Generic.List[object]]::new() - $completedRuns = [System.Collections.Generic.List[object]]::new() - $maxConcurrentExecutors = 0 - - while ($plans.Count -gt 0 -or $activeRuns.Count -gt 0) { - while ($plans.Count -gt 0 -and $activeRuns.Count -lt $ExecutorParallelism) { - $plan = $plans.Dequeue() - $activeRuns.Add((Start-BenchmarkRun -RunPlan $plan -SkillName $SkillMetadata.Name -ResolvedSkillPath $ResolvedSkillPath -ResolvedBaselineSkillPath $ResolvedBaselineSkillPath -ModelName $ModelName -Workspace $Workspace -DotnetShimRoot $dotnetShimRoot -RealDotnet $realDotnet -CopilotScriptPath $CopilotScriptPath -SharedNugetPackages $sharedNugetPackages -UseOptimizedDotnetMode $EnableOptimizations -ExecutorCommandPath $ExecutorCommandPath)) - if ($activeRuns.Count -gt $maxConcurrentExecutors) { $maxConcurrentExecutors = $activeRuns.Count } - } - - foreach ($activeRun in @($activeRuns)) { - $elapsed = ([DateTimeOffset]::UtcNow - $activeRun.startedAt).TotalSeconds - if ($activeRun.handle.Process.HasExited) { - $activeRuns.Remove($activeRun) | Out-Null - $completedRuns.Add((Finalize-BenchmarkRun -ActiveRun $activeRun -TimedOut $false)) - continue - } - if ($elapsed -ge $RunTimeout) { - $activeRuns.Remove($activeRun) | Out-Null - $completedRuns.Add((Finalize-BenchmarkRun -ActiveRun $activeRun -TimedOut $true)) - } - } - if ($activeRuns.Count -gt 0) { - Start-Sleep -Milliseconds 250 - } - } - - $gradeQueue = [System.Collections.Generic.Queue[object]]::new() - foreach ($run in @($completedRuns)) { $gradeQueue.Enqueue($run) } - $activeGraders = [System.Collections.Generic.List[object]]::new() - $maxConcurrentGraders = 0 - - while ($gradeQueue.Count -gt 0 -or $activeGraders.Count -gt 0) { - while ($gradeQueue.Count -gt 0 -and $activeGraders.Count -lt $GraderParallelism) { - $run = $gradeQueue.Dequeue() - $gradingPath = Join-Path $run.runRoot 'grading.json' - $stdoutPath = Join-Path $run.runRoot '.benchmark\grader.stdout.jsonl' - $stderrPath = Join-Path $run.runRoot '.benchmark\grader.stderr.log' - if ([string]::IsNullOrWhiteSpace($GraderCommandPath)) { - $prompt = Get-CopilotGraderPrompt -SkillCreatorRoot $SkillCreatorRootPath -TranscriptPath $run.transcriptPath -OutputsPath $run.outputsPath -TimingPath (Join-Path $run.runRoot 'timing.json') -Expectations @($run.context.expectations) - $wrapperPath = Join-Path $run.runRoot '.benchmark\invoke-copilot-grader.ps1' - New-CopilotWrapperScript -WrapperPath $wrapperPath -CopilotPowerShellPath $CopilotScriptPath -Prompt $prompt -WorkingDirectory $run.runRoot -ModelName $GraderModelName -DisableSkillTool - $handle = Start-CommandProcess -FilePath 'pwsh' -Arguments @('-NoProfile', '-File', $wrapperPath) -WorkingDirectory $run.runRoot -Environment @{} -StdoutPath $stdoutPath -StderrPath $stderrPath - $activeGraders.Add([pscustomobject]@{ - run = $run - gradingPath = $gradingPath - startedAt = [DateTimeOffset]::UtcNow - kind = 'copilot' - handle = $handle - stdoutPath = $stdoutPath - stderrPath = $stderrPath - }) - } else { - $handle = Start-CommandProcess -FilePath 'pwsh' -Arguments @('-NoProfile', '-File', $GraderCommandPath, '-ContextPath', (Join-Path $run.runRoot '.benchmark\run-context.json'), '-TranscriptPath', $run.transcriptPath, '-OutputsPath', $run.outputsPath, '-TimingPath', (Join-Path $run.runRoot 'timing.json'), '-GradingPath', $gradingPath) -WorkingDirectory $run.runRoot -Environment @{} -StdoutPath $stdoutPath -StderrPath $stderrPath - $activeGraders.Add([pscustomobject]@{ - run = $run - gradingPath = $gradingPath - startedAt = [DateTimeOffset]::UtcNow - kind = 'command' - handle = $handle - stdoutPath = $stdoutPath - stderrPath = $stderrPath - }) - } - if ($activeGraders.Count -gt $maxConcurrentGraders) { $maxConcurrentGraders = $activeGraders.Count } - } - - foreach ($grader in @($activeGraders)) { - $elapsed = ([DateTimeOffset]::UtcNow - $grader.startedAt).TotalSeconds - if ($grader.handle.Process.HasExited) { - $grader.handle.Process.WaitForExit() - if ($grader.kind -eq 'copilot') { - try { - if ($grader.handle.Process.ExitCode -ne 0) { - throw "Copilot grader exited with code $($grader.handle.Process.ExitCode)." - } - $events = Parse-CopilotEvents -JsonlPath $grader.stdoutPath - $jsonText = Get-CopilotFinalMessage -Events $events - if ([string]::IsNullOrWhiteSpace($jsonText)) { - throw 'Copilot grader did not return a JSON message.' - } - $parsed = $jsonText | ConvertFrom-Json - Write-JsonFile -Path $grader.gradingPath -Value $parsed - } catch { - Write-FallbackGrading -GradingPath $grader.gradingPath -Expectations @($grader.run.context.expectations) -Reason $_.Exception.Message -Timing $grader.run.timing -Metrics $grader.run.metrics - } - } elseif ($grader.handle.Process.ExitCode -ne 0 -or -not (Test-Path -LiteralPath $grader.gradingPath -PathType Leaf)) { - Write-FallbackGrading -GradingPath $grader.gradingPath -Expectations @($grader.run.context.expectations) -Reason "Custom grader exited with code $($grader.handle.Process.ExitCode)." -Timing $grader.run.timing -Metrics $grader.run.metrics - } - Update-TimingWithGrader -TimingPath (Join-Path $grader.run.runRoot 'timing.json') -GraderStartedAt $grader.startedAt -GraderEndedAt ([DateTimeOffset]::UtcNow) - $grader.run.timing = Get-Content -LiteralPath (Join-Path $grader.run.runRoot 'timing.json') -Raw | ConvertFrom-Json - $activeGraders.Remove($grader) | Out-Null - continue - } - if ($elapsed -ge $GradeTimeout) { - [void](Kill-ProcessTree -Process $grader.handle.Process) - Write-FallbackGrading -GradingPath $grader.gradingPath -Expectations @($grader.run.context.expectations) -Reason 'The grader timed out.' -Timing $grader.run.timing -Metrics $grader.run.metrics - Update-TimingWithGrader -TimingPath (Join-Path $grader.run.runRoot 'timing.json') -GraderStartedAt $grader.startedAt -GraderEndedAt ([DateTimeOffset]::UtcNow) - $grader.run.timing = Get-Content -LiteralPath (Join-Path $grader.run.runRoot 'timing.json') -Raw | ConvertFrom-Json - $activeGraders.Remove($grader) | Out-Null - } - } - if ($activeGraders.Count -gt 0) { - Start-Sleep -Milliseconds 200 - } - } - - $aggregateScript = Join-Path $SkillCreatorRootPath 'scripts\aggregate_benchmark.py' - $viewerScript = Join-Path $SkillCreatorRootPath 'eval-viewer\generate_review.py' - $previousPythonUtf8 = $env:PYTHONUTF8 - $env:PYTHONUTF8 = '1' - $benchmarkOutput = & python $aggregateScript $iterationRoot --skill-name $SkillMetadata.Name --skill-path $ResolvedSkillPath 2>&1 - if ($LASTEXITCODE -ne 0) { - if ($null -eq $previousPythonUtf8) { Remove-Item Env:PYTHONUTF8 -ErrorAction SilentlyContinue } else { $env:PYTHONUTF8 = $previousPythonUtf8 } - throw "Benchmark aggregation failed for profile '$ProfileName'.`n$($benchmarkOutput -join [Environment]::NewLine)" - } - Update-BenchmarkMetadata -BenchmarkPath (Join-Path $iterationRoot 'benchmark.json') -ExecutorModel $ModelName -AnalyzerModel $GraderModelName -RunsPerConfiguration 1 - - $reviewPath = Join-Path $Workspace ('review-' + $ProfileName + '.html') - if (-not $SkipReview) { - $viewerOutput = & python $viewerScript $iterationRoot --skill-name $SkillMetadata.Name --benchmark (Join-Path $iterationRoot 'benchmark.json') --static $reviewPath 2>&1 - if ($null -eq $previousPythonUtf8) { Remove-Item Env:PYTHONUTF8 -ErrorAction SilentlyContinue } else { $env:PYTHONUTF8 = $previousPythonUtf8 } - if ($LASTEXITCODE -ne 0) { - throw "Static review generation failed for profile '$ProfileName'.`n$($viewerOutput -join [Environment]::NewLine)" - } - } else { - if ($null -eq $previousPythonUtf8) { Remove-Item Env:PYTHONUTF8 -ErrorAction SilentlyContinue } else { $env:PYTHONUTF8 = $previousPythonUtf8 } - } - - $profileEnd = [DateTimeOffset]::UtcNow - $summary = [ordered]@{ - profile = $ProfileName - skill = $SkillMetadata.Name - iterationRoot = $iterationRoot - startedAt = $profileStart.ToString('O') - endedAt = $profileEnd.ToString('O') - totalWallClockSeconds = [math]::Round(($profileEnd - $profileStart).TotalSeconds, 3) - validation = $ValidationResult - prewarm = $PrewarmResult - maxConcurrentExecutorsObserved = $maxConcurrentExecutors - maxConcurrentGradersObserved = $maxConcurrentGraders - runCount = @($completedRuns).Count - timedOutRuns = @($completedRuns | Where-Object { $_.timing.cleanup.timedOut }).Count - cleanupFailures = @($completedRuns | Where-Object { -not $_.timing.cleanup.completed }).Count - totalResolverSeconds = [math]::Round((@($completedRuns | ForEach-Object { [double]$_.timing.resolver.durationSeconds } | Measure-Object -Sum).Sum), 3) - totalRestoreSeconds = [math]::Round((@($completedRuns | ForEach-Object { [double]$_.timing.dotnet.restoreSeconds } | Measure-Object -Sum).Sum), 3) - totalBuildSeconds = [math]::Round((@($completedRuns | ForEach-Object { [double]$_.timing.dotnet.buildSeconds } | Measure-Object -Sum).Sum), 3) - totalTestSeconds = [math]::Round((@($completedRuns | ForEach-Object { [double]$_.timing.dotnet.testSeconds } | Measure-Object -Sum).Sum), 3) - runs = @($completedRuns | ForEach-Object { - [ordered]@{ - evalId = $_.context.eval_id - configuration = $_.context.configuration - runRoot = $_.runRoot - totalDurationSeconds = $_.timing.total_duration_seconds - timedOut = $_.timing.cleanup.timedOut - cleanupCompleted = $_.timing.cleanup.completed - exitCode = $_.timing.executor.exitCode - resolverSeconds = $_.timing.resolver.durationSeconds - restoreSeconds = $_.timing.dotnet.restoreSeconds - buildSeconds = $_.timing.dotnet.buildSeconds - testSeconds = $_.timing.dotnet.testSeconds - } - }) - benchmarkPath = (Join-Path $iterationRoot 'benchmark.json') - reviewPath = $reviewPath - } - Write-JsonFile -Path (Join-Path $iterationRoot 'runner-summary.json') -Value $summary - - return $summary -} - -function Write-ComparisonArtifacts { - param( - [Parameter(Mandatory = $true)] [string]$Workspace, - [Parameter(Mandatory = $true)] $ValidationResult, - [Parameter(Mandatory = $true)] $OptimizedPrewarmResult, - [Parameter(Mandatory = $true)] $Legacy, - [Parameter(Mandatory = $true)] $Optimized - ) - - $legacyWorkflowSeconds = [math]::Round([double]$ValidationResult.durationSeconds + [double]$Legacy.totalWallClockSeconds, 3) - $optimizedWorkflowSeconds = [math]::Round([double]$ValidationResult.durationSeconds + [double]$OptimizedPrewarmResult.durationSeconds + [double]$Optimized.totalWallClockSeconds, 3) - $saved = [math]::Round($legacyWorkflowSeconds - $optimizedWorkflowSeconds, 3) - $percent = if ($legacyWorkflowSeconds -gt 0) { - [math]::Round(($saved / $legacyWorkflowSeconds) * 100, 2) - } else { - 0 - } - - $comparison = [ordered]@{ - skill = $Legacy.skill - workspace = $Workspace - sharedValidation = $ValidationResult - optimizedPrewarm = $OptimizedPrewarmResult - legacy = $Legacy - optimized = $Optimized - delta = [ordered]@{ - legacyWorkflowSeconds = $legacyWorkflowSeconds - optimizedWorkflowSeconds = $optimizedWorkflowSeconds - wallClockSecondsSaved = $saved - percentFaster = $percent - resolverSecondsSaved = [math]::Round([double]$Legacy.totalResolverSeconds - [double]$Optimized.totalResolverSeconds, 3) - restoreSecondsSaved = [math]::Round([double]$Legacy.totalRestoreSeconds - [double]$Optimized.totalRestoreSeconds, 3) - buildSecondsSaved = [math]::Round([double]$Legacy.totalBuildSeconds - [double]$Optimized.totalBuildSeconds, 3) - testSecondsSaved = [math]::Round([double]$Legacy.totalTestSeconds - [double]$Optimized.totalTestSeconds, 3) - } - } - Write-JsonFile -Path (Join-Path $Workspace 'comparison.json') -Value $comparison - - $lines = @( - '# Skill Benchmark Comparison', - '', - '| Metric | Legacy | Optimized | Delta |', - '|--------|--------|-----------|-------|', - ('| Validation (shared) | {0}s | {0}s | {1:+0.###;-0.###;0}s |' -f $ValidationResult.durationSeconds, 0), - ('| Resolver prewarm | 0s | {0}s | {1:+0.###;-0.###;0}s |' -f $OptimizedPrewarmResult.durationSeconds, (-1 * [double]$OptimizedPrewarmResult.durationSeconds)), - ('| Workflow total | {0}s | {1}s | {2:+0.###;-0.###;0}s |' -f $legacyWorkflowSeconds, $optimizedWorkflowSeconds, $saved), - ('| Resolver time | {0}s | {1}s | {2:+0.###;-0.###;0}s |' -f $Legacy.totalResolverSeconds, $Optimized.totalResolverSeconds, $comparison.delta.resolverSecondsSaved), - ('| Restore time | {0}s | {1}s | {2:+0.###;-0.###;0}s |' -f $Legacy.totalRestoreSeconds, $Optimized.totalRestoreSeconds, $comparison.delta.restoreSecondsSaved), - ('| Build time | {0}s | {1}s | {2:+0.###;-0.###;0}s |' -f $Legacy.totalBuildSeconds, $Optimized.totalBuildSeconds, $comparison.delta.buildSecondsSaved), - ('| Test time | {0}s | {1}s | {2:+0.###;-0.###;0}s |' -f $Legacy.totalTestSeconds, $Optimized.totalTestSeconds, $comparison.delta.testSecondsSaved), - ('| Timed out runs | {0} | {1} | {2:+0;-0;0} |' -f $Legacy.timedOutRuns, $Optimized.timedOutRuns, ($Legacy.timedOutRuns - $Optimized.timedOutRuns)), - ('| Cleanup failures | {0} | {1} | {2:+0;-0;0} |' -f $Legacy.cleanupFailures, $Optimized.cleanupFailures, ($Legacy.cleanupFailures - $Optimized.cleanupFailures)) - ) - Write-TextFile -Path (Join-Path $Workspace 'comparison.md') -Content ($lines -join [Environment]::NewLine) -} - -function Update-TimingWithGrader { - param( - [Parameter(Mandatory = $true)] [string]$TimingPath, - [Parameter(Mandatory = $true)] [DateTimeOffset]$GraderStartedAt, - [Parameter(Mandatory = $true)] [DateTimeOffset]$GraderEndedAt - ) - - if (-not (Test-Path -LiteralPath $TimingPath -PathType Leaf)) { return } - - $timing = Get-Content -LiteralPath $TimingPath -Raw | ConvertFrom-Json - $graderDuration = [math]::Round(($GraderEndedAt - $GraderStartedAt).TotalSeconds, 3) - - $timing | Add-Member -NotePropertyName grader -NotePropertyValue ([pscustomobject]@{}) -Force - $timing.grader | Add-Member -NotePropertyName startedAt -NotePropertyValue $GraderStartedAt.ToString('O') -Force - $timing.grader | Add-Member -NotePropertyName endedAt -NotePropertyValue $GraderEndedAt.ToString('O') -Force - $timing.grader | Add-Member -NotePropertyName durationSeconds -NotePropertyValue $graderDuration -Force - - $totalDuration = [math]::Round([double]$timing.executor.durationSeconds + $graderDuration, 3) - $timing.total_duration_seconds = $totalDuration - $timing.duration_ms = [long][math]::Round($totalDuration * 1000) - - Write-JsonFile -Path $TimingPath -Value $timing -} - -function Update-BenchmarkMetadata { - param( - [Parameter(Mandatory = $true)] [string]$BenchmarkPath, - [Parameter(Mandatory = $true)] [string]$ExecutorModel, - [Parameter(Mandatory = $true)] [string]$AnalyzerModel, - [Parameter(Mandatory = $true)] [int]$RunsPerConfiguration - ) - - if (-not (Test-Path -LiteralPath $BenchmarkPath -PathType Leaf)) { return } - - $benchmark = Get-Content -LiteralPath $BenchmarkPath -Raw | ConvertFrom-Json - if ($null -eq $benchmark.metadata) { - $benchmark | Add-Member -NotePropertyName metadata -NotePropertyValue ([pscustomobject]@{}) -Force - } - $benchmark.metadata | Add-Member -NotePropertyName executor_model -NotePropertyValue $ExecutorModel -Force - $benchmark.metadata | Add-Member -NotePropertyName analyzer_model -NotePropertyValue $AnalyzerModel -Force - $benchmark.metadata | Add-Member -NotePropertyName runs_per_configuration -NotePropertyValue $RunsPerConfiguration -Force - Write-JsonFile -Path $BenchmarkPath -Value $benchmark -} - -$repoRoot = Get-RepoRoot -$resolvedSkillPath = Get-ResolvedPath -Path $SkillPath -$baselineSkillPath = if ([string]::IsNullOrWhiteSpace($BaselineSkillPath)) { $null } else { Get-ResolvedPath -Path $BaselineSkillPath } -$skillMetadata = Get-SkillMetadata -ResolvedSkillPath $resolvedSkillPath -$skillCreatorRoot = Resolve-SkillCreatorRoot -$evals = Get-EvalDefinitions -ResolvedSkillPath $resolvedSkillPath -SelectedEvalId $EvalId -if (@($evals).Count -eq 0) { - throw 'No evals matched the selected criteria.' -} - -if ([string]::IsNullOrWhiteSpace($WorkspaceRoot)) { - $WorkspaceRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('sb-' + $skillMetadata.Name + '-' + [DateTimeOffset]::UtcNow.ToString('yyyyMMdd-HHmmss')) -} -$workspace = $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($WorkspaceRoot) -New-Item -ItemType Directory -Path $workspace -Force | Out-Null - -$copilotScriptPath = Get-CopilotScriptPath -$executorPath = if ([string]::IsNullOrWhiteSpace($ExecutorCommand)) { $null } else { Get-ResolvedPath -Path $ExecutorCommand } -$graderPath = if ([string]::IsNullOrWhiteSpace($GraderCommand)) { $null } else { Get-ResolvedPath -Path $GraderCommand } -$shouldRunValidation = -not $SkipSkillValidation -if ([string]::IsNullOrWhiteSpace($GraderModel)) { - $GraderModel = $Model -} - -$validationResult = if ($shouldRunValidation) { Invoke-SkillValidation -ResolvedSkillPath $resolvedSkillPath -Workspace $workspace } else { [ordered]@{ ran = $false; exitCode = $null; durationSeconds = 0 } } -if ($validationResult.ran -and $validationResult.exitCode -ne 0) { - throw 'Skill validation failed before benchmark execution.' -} - -$optimizedPrewarm = [ordered]@{ ran = $false; durationSeconds = 0; contexts = @() } -if ($skillMetadata.Name -eq 'dotnet-test') { - $contexts = Get-DotnetTestContexts -Evals $evals -ResolvedSkillPath $resolvedSkillPath - if (@($contexts).Count -gt 0) { - $optimizedPrewarm = Invoke-DotnetTestPrewarm -Contexts $contexts -ResolvedSkillPath $resolvedSkillPath -CacheDirectory (Join-Path $workspace '.benchmark\resolver-cache') -TracePath (Join-Path $workspace '.benchmark\resolver-prewarm.jsonl') -CandidateLimit $BenchmarkCandidateLimit - } -} - -$optimized = Invoke-Profile -ProfileName 'optimized' -ResolvedSkillPath $resolvedSkillPath -ResolvedBaselineSkillPath $baselineSkillPath -Evals $evals -SkillMetadata $skillMetadata -Workspace $workspace -ExecutorParallelism $MaxParallel -GraderParallelism $MaxGradeParallel -EnableOptimizations $true -CopilotScriptPath $copilotScriptPath -ExecutorCommandPath $executorPath -GraderCommandPath $graderPath -SkillCreatorRootPath $skillCreatorRoot -ValidationResult $validationResult -PrewarmResult $optimizedPrewarm -RunTimeout $RunTimeoutSeconds -GradeTimeout $GradeTimeoutSeconds -ModelName $Model -GraderModelName $GraderModel - -if ($CompareWithLegacy) { - $legacy = Invoke-Profile -ProfileName 'legacy' -ResolvedSkillPath $resolvedSkillPath -ResolvedBaselineSkillPath $baselineSkillPath -Evals $evals -SkillMetadata $skillMetadata -Workspace $workspace -ExecutorParallelism 1 -GraderParallelism 1 -EnableOptimizations $false -CopilotScriptPath $copilotScriptPath -ExecutorCommandPath $executorPath -GraderCommandPath $graderPath -SkillCreatorRootPath $skillCreatorRoot -ValidationResult $validationResult -PrewarmResult ([ordered]@{ ran = $false; durationSeconds = 0; contexts = @() }) -RunTimeout $RunTimeoutSeconds -GradeTimeout $GradeTimeoutSeconds -ModelName $Model -GraderModelName $GraderModel - Write-ComparisonArtifacts -Workspace $workspace -ValidationResult $validationResult -OptimizedPrewarmResult $optimizedPrewarm -Legacy $legacy -Optimized $optimized -} - -Write-TextFile -Path (Join-Path $workspace 'latest-profile.txt') -Content ('optimized' + [Environment]::NewLine) -Write-Output ("Benchmark workspace: {0}" -f $workspace) -Write-Output ("Optimized benchmark: {0}" -f $optimized.benchmarkPath) -Write-Output ("Optimized review: {0}" -f $optimized.reviewPath) -if ($CompareWithLegacy) { - Write-Output ("Legacy benchmark: {0}" -f $legacy.benchmarkPath) - Write-Output ("Legacy review: {0}" -f $legacy.reviewPath) - Write-Output ("Comparison summary: {0}" -f (Join-Path $workspace 'comparison.json')) -} diff --git a/scripts/skill-benchmark/log-dotnet.ps1 b/scripts/skill-benchmark/log-dotnet.ps1 deleted file mode 100644 index 88fd499..0000000 --- a/scripts/skill-benchmark/log-dotnet.ps1 +++ /dev/null @@ -1,99 +0,0 @@ -param( - [Parameter(Mandatory = $true)] - [string]$RealDotnet, - - [Parameter(Mandatory = $true)] - [string]$LogDirectory, - - [Parameter(Mandatory = $true)] - [string]$StdoutPath, - - [Parameter(Mandatory = $true)] - [string]$StderrPath, - - [Parameter(ValueFromRemainingArguments = $true)] - [string[]]$Arguments -) - -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -$utf8NoBom = [System.Text.UTF8Encoding]::new($false) -[Console]::InputEncoding = $utf8NoBom -[Console]::OutputEncoding = $utf8NoBom -$OutputEncoding = $utf8NoBom - -New-Item -ItemType Directory -Path $LogDirectory -Force | Out-Null - -$start = [DateTimeOffset]::UtcNow -$stdoutLines = [System.Collections.Generic.List[string]]::new() -$stderrLines = [System.Collections.Generic.List[string]]::new() - -$stdoutWriter = [System.IO.StreamWriter]::new($StdoutPath, $false, $utf8NoBom) -$stderrWriter = [System.IO.StreamWriter]::new($StderrPath, $false, $utf8NoBom) - -try { - $psi = [System.Diagnostics.ProcessStartInfo]::new() - $psi.FileName = $RealDotnet - $psi.WorkingDirectory = (Get-Location).Path - $psi.UseShellExecute = $false - $psi.RedirectStandardOutput = $true - $psi.RedirectStandardError = $true - foreach ($argument in @($Arguments)) { - [void]$psi.ArgumentList.Add($argument) - } - - $process = [System.Diagnostics.Process]::new() - $process.StartInfo = $psi - $process.EnableRaisingEvents = $true - - $process.add_OutputDataReceived({ - param($sender, $eventArgs) - - if ($null -eq $eventArgs.Data) { return } - $stdoutLines.Add($eventArgs.Data) - $stdoutWriter.WriteLine($eventArgs.Data) - [Console]::Out.WriteLine($eventArgs.Data) - }) - - $process.add_ErrorDataReceived({ - param($sender, $eventArgs) - - if ($null -eq $eventArgs.Data) { return } - $stderrLines.Add($eventArgs.Data) - $stderrWriter.WriteLine($eventArgs.Data) - [Console]::Error.WriteLine($eventArgs.Data) - }) - - if (-not $process.Start()) { - throw "Unable to start '$RealDotnet'." - } - - $process.BeginOutputReadLine() - $process.BeginErrorReadLine() - $process.WaitForExit() - $process.WaitForExit() - - $end = [DateTimeOffset]::UtcNow - $duration = [math]::Round(($end - $start).TotalSeconds, 3) - $command = if (@($Arguments).Count -gt 0) { $Arguments[0] } else { '' } - $logPath = Join-Path $LogDirectory ('dotnet-' + $start.ToUnixTimeMilliseconds() + '.json') - - [ordered]@{ - startedAt = $start.ToString('O') - endedAt = $end.ToString('O') - durationSeconds = $duration - workingDirectory = (Get-Location).Path - realDotnet = $RealDotnet - arguments = @($Arguments) - command = $command - exitCode = $process.ExitCode - stdoutPath = $StdoutPath - stderrPath = $StderrPath - } | ConvertTo-Json -Depth 6 | Set-Content -LiteralPath $logPath -Encoding utf8 - - exit $process.ExitCode -} finally { - $stdoutWriter.Dispose() - $stderrWriter.Dispose() -} diff --git a/scripts/skill-benchmark/mock-executor.ps1 b/scripts/skill-benchmark/mock-executor.ps1 deleted file mode 100644 index 70c8c05..0000000 --- a/scripts/skill-benchmark/mock-executor.ps1 +++ /dev/null @@ -1,87 +0,0 @@ -param( - [Parameter(Mandatory = $true)] - [string]$ContextPath, - - [Parameter(Mandatory = $true)] - [string]$RunRoot, - - [Parameter(Mandatory = $true)] - [string]$TranscriptPath, - - [Parameter(Mandatory = $true)] - [string]$ResultSummaryPath, - - [Parameter(Mandatory = $true)] - [string]$OutputsPath -) - -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -$utf8NoBom = [System.Text.UTF8Encoding]::new($false) -[Console]::InputEncoding = $utf8NoBom -[Console]::OutputEncoding = $utf8NoBom -$OutputEncoding = $utf8NoBom - -$context = Get-Content -LiteralPath $ContextPath -Raw | ConvertFrom-Json -New-Item -ItemType Directory -Path $OutputsPath -Force | Out-Null - -$summary = '' -$exitCode = 0 - -switch ("$($context.eval_id):$($context.configuration)") { - '1:with_skill' { - Start-Sleep -Seconds 2 - [System.IO.File]::WriteAllText((Join-Path $OutputsPath 'assistant-response.md'), "mock executor completed with skill`n", $utf8NoBom) - [System.IO.File]::WriteAllText((Join-Path $OutputsPath 'changed-files.md'), "## changed files`n- src\Mock.cs`n", $utf8NoBom) - [System.IO.File]::WriteAllText((Join-Path $RunRoot 'repo\src\Mock.cs'), "public static class Mock { public const string Mode = ""with_skill""; }`n", $utf8NoBom) - $summary = 'Mock with-skill execution completed.' - } - '1:without_skill' { - Start-Sleep -Seconds 2 - [System.IO.File]::WriteAllText((Join-Path $OutputsPath 'assistant-response.md'), "mock executor completed without skill`n", $utf8NoBom) - [System.IO.File]::WriteAllText((Join-Path $OutputsPath 'changed-files.md'), "## changed files`n- src\Mock.cs`n", $utf8NoBom) - [System.IO.File]::WriteAllText((Join-Path $RunRoot 'repo\src\Mock.cs'), "public static class Mock { public const string Mode = ""without_skill""; }`n", $utf8NoBom) - $summary = 'Mock baseline execution completed.' - } - '2:with_skill' { - $childInfo = [System.Diagnostics.ProcessStartInfo]::new() - $childInfo.FileName = 'pwsh' - $childInfo.ArgumentList.Add('-NoProfile') - $childInfo.ArgumentList.Add('-Command') - $childInfo.ArgumentList.Add('Start-Sleep -Seconds 300') - $childInfo.UseShellExecute = $false - $child = [System.Diagnostics.Process]::Start($childInfo) - [System.IO.File]::WriteAllText((Join-Path $OutputsPath 'child.pid'), "$($child.Id)`n", $utf8NoBom) - [System.IO.File]::WriteAllText((Join-Path $OutputsPath 'assistant-response.md'), "mock executor hanging to test timeout`n", $utf8NoBom) - Start-Sleep -Seconds 300 - $summary = 'This line should never be reached.' - } - '2:without_skill' { - Start-Sleep -Seconds 1 - [System.IO.File]::WriteAllText((Join-Path $OutputsPath 'assistant-response.md'), "mock executor failed without skill`n", $utf8NoBom) - [System.IO.File]::WriteAllText((Join-Path $OutputsPath 'changed-files.md'), "## changed files`n- src\Broken.cs`n", $utf8NoBom) - [System.IO.File]::WriteAllText((Join-Path $RunRoot 'repo\src\Broken.cs'), "public static class Broken { }`n", $utf8NoBom) - $summary = 'Mock baseline execution failed.' - $exitCode = 9 - } - default { - Start-Sleep -Seconds 1 - [System.IO.File]::WriteAllText((Join-Path $OutputsPath 'assistant-response.md'), "mock executor default path`n", $utf8NoBom) - $summary = 'Mock execution completed.' - } -} - -[System.IO.File]::WriteAllText($TranscriptPath, @" -# Mock Transcript - -## Eval Prompt - -$($context.prompt) - -## Result - -$summary -"@, $utf8NoBom) -[System.IO.File]::WriteAllText($ResultSummaryPath, $summary + [Environment]::NewLine, $utf8NoBom) -exit $exitCode diff --git a/scripts/skill-benchmark/mock-grader.ps1 b/scripts/skill-benchmark/mock-grader.ps1 deleted file mode 100644 index df87121..0000000 --- a/scripts/skill-benchmark/mock-grader.ps1 +++ /dev/null @@ -1,68 +0,0 @@ -param( - [Parameter(Mandatory = $true)] - [string]$ContextPath, - - [Parameter(Mandatory = $true)] - [string]$TranscriptPath, - - [Parameter(Mandatory = $true)] - [string]$OutputsPath, - - [Parameter(Mandatory = $true)] - [string]$TimingPath, - - [Parameter(Mandatory = $true)] - [string]$GradingPath -) - -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -$utf8NoBom = [System.Text.UTF8Encoding]::new($false) -$context = Get-Content -LiteralPath $ContextPath -Raw | ConvertFrom-Json -$timing = Get-Content -LiteralPath $TimingPath -Raw | ConvertFrom-Json -$timedOut = [bool]$timing.cleanup.timedOut -$exitCode = [int]$timing.executor.exitCode -$failed = $timedOut -or $exitCode -ne 0 - -$expectations = foreach ($expectation in @($context.expectations)) { - [ordered]@{ - text = $expectation - passed = -not $failed - evidence = if ($timedOut) { - 'The mock executor timed out and the runner produced fallback artifacts.' - } elseif ($exitCode -ne 0) { - "The mock executor exited with code $exitCode." - } else { - 'The mock executor completed and the outputs were generated.' - } - } -} - -$passed = @($expectations | Where-Object passed).Count -$total = @($expectations).Count - -[ordered]@{ - expectations = @($expectations) - summary = [ordered]@{ - passed = $passed - failed = $total - $passed - total = $total - pass_rate = if ($total -eq 0) { 0 } else { [math]::Round($passed / $total, 4) } - } - execution_metrics = [ordered]@{ - tool_calls = @{} - total_tool_calls = 0 - total_steps = 1 - errors_encountered = if ($failed) { 1 } else { 0 } - output_chars = 0 - transcript_chars = (Get-Content -LiteralPath $TranscriptPath -Raw).Length - } - timing = $timing - claims = @() - user_notes_summary = [ordered]@{ - uncertainties = @() - needs_review = @() - workarounds = @() - } -} | ConvertTo-Json -Depth 8 | Set-Content -LiteralPath $GradingPath -Encoding utf8 diff --git a/scripts/test-run-skill-benchmark.ps1 b/scripts/test-run-skill-benchmark.ps1 deleted file mode 100644 index b6dc8af..0000000 --- a/scripts/test-run-skill-benchmark.ps1 +++ /dev/null @@ -1,152 +0,0 @@ -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -$utf8NoBom = [System.Text.UTF8Encoding]::new($false) - -function Write-Json { - param([string]$Path, $Value) - $directory = Split-Path -Path $Path -Parent - if (-not (Test-Path -LiteralPath $directory)) { New-Item -ItemType Directory -Path $directory -Force | Out-Null } - $Value | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $Path -Encoding utf8 -} - -function Write-Text { - param([string]$Path, [string]$Content) - $directory = Split-Path -Path $Path -Parent - if (-not (Test-Path -LiteralPath $directory)) { New-Item -ItemType Directory -Path $directory -Force | Out-Null } - [System.IO.File]::WriteAllText($Path, $Content, $utf8NoBom) -} - -$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path -$workspace = Join-Path ([System.IO.Path]::GetTempPath()) ('skill-benchmark-test-' + [Guid]::NewGuid().ToString('N')) -$skillRoot = Join-Path $workspace 'mock-skill' - -try { - New-Item -ItemType Directory -Path $skillRoot -Force | Out-Null - Write-Text -Path (Join-Path $skillRoot 'SKILL.md') -Content @" ---- -name: mock-benchmark-skill -description: > - Mock benchmark skill for runner self-tests. ---- - -Answer mock tasks. -"@ - Write-Json -Path (Join-Path $skillRoot 'evals\evals.json') -Value ([ordered]@{ - skill_name = 'mock-benchmark-skill' - evals = @( - [ordered]@{ - id = 1 - prompt = 'Mock success prompt.' - expected_output = 'A successful run.' - expectations = @('The mock run succeeds') - files = @('evals/files/eval-1/src/Seed.cs') - } - [ordered]@{ - id = 2 - prompt = 'Mock timeout prompt.' - expected_output = 'A failed or timed out run still produces artifacts.' - expectations = @('The mock run handles failure safely') - files = @('evals/files/eval-2/src/Seed.cs') - } - ) - }) - Write-Text -Path (Join-Path $skillRoot 'evals\files\eval-1\src\Seed.cs') -Content 'public static class Seed { }' - Write-Text -Path (Join-Path $skillRoot 'evals\files\eval-2\src\Seed.cs') -Content 'public static class Seed { }' - - $benchmarkScript = Join-Path $repoRoot 'scripts\run-skill-benchmark.ps1' - $executor = Join-Path $repoRoot 'scripts\skill-benchmark\mock-executor.ps1' - $grader = Join-Path $repoRoot 'scripts\skill-benchmark\mock-grader.ps1' - - & pwsh -NoProfile -File $benchmarkScript ` - -SkillPath $skillRoot ` - -WorkspaceRoot (Join-Path $workspace 'benchmark') ` - -ExecutorCommand $executor ` - -GraderCommand $grader ` - -MaxParallel 2 ` - -MaxGradeParallel 2 ` - -RunTimeoutSeconds 30 ` - -GradeTimeoutSeconds 30 ` - -Model 'gpt-5.4-mini' ` - -CompareWithLegacy - - if ($LASTEXITCODE -ne 0) { - throw "Benchmark runner exited with code $LASTEXITCODE." - } - - $summaryPath = Join-Path $workspace 'benchmark\iteration-optimized\runner-summary.json' - $benchmarkPath = Join-Path $workspace 'benchmark\iteration-optimized\benchmark.json' - $reviewPath = Join-Path $workspace 'benchmark\review-optimized.html' - $comparisonPath = Join-Path $workspace 'benchmark\comparison.json' - - foreach ($required in @($summaryPath, $benchmarkPath, $reviewPath, $comparisonPath)) { - if (-not (Test-Path -LiteralPath $required)) { - throw "Missing required benchmark artifact: $required" - } - } - - $summary = Get-Content -LiteralPath $summaryPath -Raw | ConvertFrom-Json - if ($summary.maxConcurrentExecutorsObserved -gt 2) { - throw "Bounded concurrency failed. Expected at most 2 active executors, found $($summary.maxConcurrentExecutorsObserved)." - } - if ($summary.timedOutRuns -ne 1) { - throw "Expected one timed out run, found $($summary.timedOutRuns)." - } - if ($summary.cleanupFailures -ne 0) { - throw "Expected zero cleanup failures, found $($summary.cleanupFailures)." - } - - $benchmarkRoot = Join-Path $workspace 'benchmark' - $shimRoot = Join-Path (Join-Path $benchmarkRoot '.benchmark') 'dotnet' - $windowsShimPath = Join-Path $shimRoot 'dotnet.cmd' - $unixShimPath = Join-Path $shimRoot 'dotnet' - if (-not (Test-Path -LiteralPath $windowsShimPath -PathType Leaf)) { - throw "Missing Windows dotnet shim: $windowsShimPath" - } - if (-not (Test-Path -LiteralPath $unixShimPath -PathType Leaf)) { - throw "Missing Unix dotnet shim: $unixShimPath" - } - $unixShim = Get-Content -LiteralPath $unixShimPath -Raw - if ($unixShim -notmatch '(?m)^#!/usr/bin/env sh\s*$' -or $unixShim -notmatch 'exec pwsh -NoProfile -File') { - throw 'Unix dotnet shim does not delegate to the logging wrapper.' - } - if (-not $IsWindows) { - $unixMode = (Get-Item -LiteralPath $unixShimPath).UnixFileMode - if (($unixMode -band [System.IO.UnixFileMode]::UserExecute) -eq 0) { - throw 'Unix dotnet shim is not executable.' - } - } - - $timeoutRunRoot = Join-Path $workspace 'benchmark\iteration-optimized\eval-02-mock-timeout-prompt\with_skill\run-1' - $childPidPath = Join-Path $timeoutRunRoot 'outputs\child.pid' - if (-not (Test-Path -LiteralPath $childPidPath)) { - throw 'Mock timeout run did not write child.pid.' - } - $childPid = [int]((Get-Content -LiteralPath $childPidPath -Raw).Trim()) - if (Get-Process -Id $childPid -ErrorAction SilentlyContinue) { - throw "Timed-out child process $childPid is still running." - } - - foreach ($runDir in Get-ChildItem -LiteralPath (Join-Path $workspace 'benchmark\iteration-optimized') -Directory | Where-Object { $_.Name -like 'eval-*' } | ForEach-Object { Get-ChildItem -LiteralPath $_.FullName -Directory -Recurse | Where-Object { $_.Name -eq 'run-1' } }) { - foreach ($artifact in @('transcript.md', 'result-summary.md', 'grading.json', 'timing.json')) { - if (-not (Test-Path -LiteralPath (Join-Path $runDir.FullName $artifact))) { - throw "Missing $artifact in $($runDir.FullName)." - } - } - } - - $benchmark = Get-Content -LiteralPath $benchmarkPath -Raw | ConvertFrom-Json - $failedRun = @($benchmark.runs | Where-Object { $_.configuration -eq 'without_skill' -and $_.eval_id -eq 2 })[0] - if ($null -eq $failedRun) { - throw 'Missing failed run in benchmark aggregation.' - } - if ($failedRun.result.pass_rate -ne 0) { - throw "Expected failed run pass rate 0, found $($failedRun.result.pass_rate)." - } - - Write-Output 'run-skill-benchmark.ps1 self-test: PASS' -} finally { - if (Test-Path -LiteralPath $workspace) { - Remove-Item -LiteralPath $workspace -Recurse -Force - } -} diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index b3b647e..395681e 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1,6 +1,7 @@ param( [string]$Ref, - [switch]$Full + [switch]$Full, + [switch]$MetadataOnly ) $ErrorActionPreference = 'Stop' @@ -195,6 +196,9 @@ function Get-LocalShellPolicyFindings { if (-not (Test-IsLocalShellPolicyScanCandidate -RelativePath $path)) { continue } + if ([string]::IsNullOrWhiteSpace($GitRef) -and -not (Test-Path -LiteralPath (Join-Path $RepoRoot $path))) { + continue + } [pscustomobject]@{ Path = $path @@ -550,6 +554,34 @@ function Write-RenderedFileFromTemplate { Write-Utf8File -Path $DestinationPath -Content $rendered } +function Write-ValidationSummary { + param( + [System.Collections.Generic.List[object]]$Results, + [string]$GitRef, + [string]$Mode + ) + + $passed = @($Results | Where-Object { $_.Status -eq 'PASS' }).Count + $failed = @($Results | Where-Object { $_.Status -eq 'FAIL' }).Count + $label = if ([string]::IsNullOrWhiteSpace($GitRef)) { 'WORKTREE' } else { $GitRef } + + Write-Host ("Validation target: {0}" -f $label) + Write-Host ("Validation mode: {0}" -f $Mode) + Write-Host ("Passed: {0}" -f $passed) + Write-Host ("Failed: {0}" -f $failed) + Write-Host '' + + foreach ($result in $Results) { + $prefix = if ($result.Status -eq 'PASS') { '[PASS]' } else { '[FAIL]' } + Write-Host ("{0} {1}" -f $prefix, $result.Name) + if ($result.Status -eq 'FAIL') { + Write-Host (" {0}" -f $result.Details) + } + } + + return $failed +} + $repoRoot = Get-RepoRoot $results = [System.Collections.Generic.List[object]]::new() @@ -641,6 +673,14 @@ Add-ValidationResult -Results $results -Name 'All repo-managed skills keep YAML } } +if ($MetadataOnly) { + $metadataFailures = Write-ValidationSummary -Results $results -GitRef $Ref -Mode 'METADATA-ONLY' + if ($metadataFailures -gt 0) { + exit 1 + } + exit 0 +} + Add-ValidationResult -Results $results -Name 'Active local shell guidance rejects only legacy PowerShell executable use' -Action { $findings = @(Get-LocalShellPolicyFindings -RepoRoot $repoRoot -GitRef $Ref) @@ -1102,13 +1142,90 @@ Add-ValidationResult -Results $results -Name 'Library templates use PROJECT_NAME Assert-Contains -Name 'library toc.yml' -Content $toc -Needle 'api/{PROJECT_NAME}.html' } -Add-ValidationResult -Results $results -Name 'Benchmark runner wildcard is preserved and benchmark program is file-scoped' -Action { +Add-ValidationResult -Results $results -Name 'BenchmarkDotNet runner wildcard is preserved and benchmark program is file-scoped' -Action { $runnerProject = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-new-lib-slnx/assets/library/benchmark-runner.csproj' -GitRef $Ref $program = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-new-lib-slnx/assets/library/benchmark-program.cs' -GitRef $Ref Assert-Contains -Name 'benchmark-runner.csproj' -Content $runnerProject -Needle '..\..\tuning\**\*.csproj' Assert-Match -Name 'benchmark-program.cs' -Content $program -Pattern 'namespace\s+\{BENCHMARK_RUNNER_NAMESPACE\};' } +Add-ValidationResult -Results $results -Name 'Repository automation cannot launch AI or LLM evaluation sessions' -Action { + $agents = Get-FileText -RepoRoot $repoRoot -RelativePath 'AGENTS.md' -GitRef $Ref + $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref + + $forbiddenPaths = @( + 'scripts/run-skill-benchmark.ps1', + 'scripts/test-run-skill-benchmark.ps1', + 'scripts/skill-benchmark/log-dotnet.ps1', + 'scripts/skill-benchmark/mock-executor.ps1', + 'scripts/skill-benchmark/mock-grader.ps1' + ) + $presentForbiddenPaths = if ([string]::IsNullOrWhiteSpace($Ref)) { + @($forbiddenPaths | Where-Object { Test-Path -LiteralPath (Join-Path $repoRoot $_) }) + } else { + $trackedPaths = @(Get-TrackedRepoPaths -RepoRoot $repoRoot -GitRef $Ref) + @($forbiddenPaths | Where-Object { $trackedPaths -contains $_ }) + } + if (@($presentForbiddenPaths).Count -gt 0) { + throw "Dangerous skill benchmark automation must remain deleted: $(@($presentForbiddenPaths) -join ', ')" + } + + $automationPaths = if ([string]::IsNullOrWhiteSpace($Ref)) { + @( + Get-ChildItem -LiteralPath (Join-Path $repoRoot 'scripts') -Recurse -File -Force + Get-ChildItem -LiteralPath (Join-Path $repoRoot '.github') -Recurse -File -Force + ) | ForEach-Object { Convert-ToRelativePath -BasePath $repoRoot -FullPath $_.FullName } + } else { + @(Get-TrackedRepoPaths -RepoRoot $repoRoot -GitRef $Ref) + } + $automationExtensions = @('.cs', '.ps1', '.psm1', '.py', '.sh', '.yml', '.yaml') + $automationPaths = @($automationPaths | Where-Object { + $normalized = $_ -replace '\\', '/' + ($normalized.StartsWith('scripts/') -or $normalized.StartsWith('.github/')) -and + $normalized -ne 'scripts/validate-skill-templates.ps1' -and + $automationExtensions -contains [System.IO.Path]::GetExtension($normalized).ToLowerInvariant() + } | Sort-Object -Unique) + + $launchPatterns = @( + '(?im)\bGet-Command\s+(?:copilot|claude|codex|gemini)\b', + '(?im)(?:^|[;&|]\s*|&\s*|Start-Process\s+)(?:copilot|claude|codex|gemini)(?:\.exe|\.cmd|\.ps1)?\b', + '(?im)\b(?:copilot|claude|gemini)(?:\.exe|\.cmd|\.ps1)?\b[^\r\n]{0,120}\s-p\b', + '(?im)\bcodex(?:\.exe|\.cmd|\.ps1)?\s+exec\b' + ) + $scannerCases = @( + [pscustomobject]@{ Name = 'Copilot prompt mode'; Content = 'copilot -p "grade this"'; Expected = $true }, + [pscustomobject]@{ Name = 'Claude prompt mode'; Content = '& claude -p "run eval"'; Expected = $true }, + [pscustomobject]@{ Name = 'Codex execution'; Content = 'codex exec "run eval"'; Expected = $true }, + [pscustomobject]@{ Name = 'Gemini process'; Content = 'Start-Process gemini -ArgumentList "-p", "grade"'; Expected = $true }, + [pscustomobject]@{ Name = 'Documentation prose'; Content = 'Supports Copilot, Claude, Codex, and Gemini skill formats.'; Expected = $false } + ) + foreach ($case in $scannerCases) { + $matched = @($launchPatterns | Where-Object { $case.Content -match $_ }).Count -gt 0 + if ($matched -ne $case.Expected) { + throw "AI/LLM automation scanner failed '$($case.Name)'." + } + } + $launchFindings = foreach ($path in $automationPaths) { + $content = Get-FileText -RepoRoot $repoRoot -RelativePath $path -GitRef $Ref + foreach ($pattern in $launchPatterns) { + if ($content -match $pattern) { + $path + break + } + } + } + if (@($launchFindings).Count -gt 0) { + throw "Repository automation must not launch AI/LLM CLIs: $(@($launchFindings | Sort-Object -Unique) -join ', ')" + } + + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '## AI/LLM Evaluation Automation Prohibition' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'this repository does not provide an opt-in path around that rule.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Model-backed comparisons are not a repository completion gate.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'This rule is Priority 1.' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'There is no repository opt-in switch.' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'validate-skill-templates.ps1 -MetadataOnly' +} + Add-ValidationResult -Results $results -Name 'dotnet-test encodes role-specific Codebelt xUnit migration and bootstrap contracts' -Action { $skill = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-test/SKILL.md' -GitRef $Ref $forms = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-test/FORMS.md' -GitRef $Ref @@ -1325,6 +1442,9 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'automatic trigger for this skill, not as a casual hint.' + Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Invocation Routing Lock' + Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Interpret `Please do a git bot commit yolo` as `git bot commit` identity plus auto-approval for the full current worktree.' + Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '`yolo` is not the commit message, and it does not request a changelog.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Full-Skill Read and Subject Lock' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Before running any Git command or composing a subject, read this `SKILL.md` completely from the first line through EOF.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'If a tool truncates the file, continue from the first unread line until EOF before proceeding.' @@ -1435,8 +1555,16 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Runs scripts/validate-commit-subject.ps1 before showing the corrected subject and again immediately before passing it to Git' Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Triggers the single-category context quality gate because more than one file is being placed in one category' Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Recognizes exactly one changed file as the explicit exception and skips the single-category context quality gate' + Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Please do a git bot commit yolo.' + Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Does not replace bot identity with a human-authored commit plus a Co-authored-by trailer' Assert-Contains -Name 'README.md' -Content $readme -Needle '**Single-category context gate**' Assert-Contains -Name 'README.md' -Content $readme -Needle 'Multi-file plans that initially collapse to one category also require a visible full-context quality gate' + Assert-Contains -Name 'README.md' -Content $readme -Needle '**Authoritative command routing**' + Assert-Contains -Name 'README.md' -Content $readme -Needle '**CLI override remains deterministic**' + $agents = Get-FileText -RepoRoot $repoRoot -RelativePath 'AGENTS.md' -GitRef $Ref + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Commit Skill Routing' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'invoke `git-visual-commits` before responding to the request or running Git commands for that commit workflow' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Do not route the request to changelog or release-note skills' } Add-ValidationResult -Results $results -Name 'Git visual squash summary skill stays self-contained and shares commit language rules' -Action { @@ -1497,6 +1625,9 @@ Add-ValidationResult -Results $results -Name 'Git keep a changelog skill updates $entityResolverTests = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-keep-a-changelog/scripts/test-resolve-release-entity.ps1' -GitRef $Ref Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'Create or update `CHANGELOG.md` directly, then stop for user review.' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'Bare `yolo` / `auto`, `git bot commit yolo`, and other commit-execution requests do not activate this skill' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'within an explicit changelog or release-note request' + Assert-NotContains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'Trigger phrases: "finalize", "ready to release", "rtr", "release" (especially with version branches like v0.3.1/...), "yolo", "auto".' Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'If `CHANGELOG.md` does not exist, create a compliant one before' Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'Read full commit subjects and bodies before writing the changelog.' Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'If the current branch starts with a version hint such as `v0.3.0/`,' @@ -1552,6 +1683,8 @@ Add-ValidationResult -Results $results -Name 'Git keep a changelog skill updates Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $evals -Needle 'Treats the merge-base as an excluded boundary rather than the first commit of the concrete release' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $evals -Needle 'Does not let yolo mode widen committed history or include the v10.0.9 boundary commit' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $evals -Needle 'Runs scripts/resolve-release-entity.ps1 for the path-backed dotnet-test entity and uses its Added classification' + Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $evals -Needle 'Does not select git-keep-a-changelog from bare yolo wording inside a git bot commit request' + Assert-Contains -Name 'README.md' -Content (Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref) -Needle '**Trigger isolation**' if ([string]::IsNullOrWhiteSpace($Ref)) { & (Join-Path $repoRoot 'skills/git-keep-a-changelog/scripts/test-resolve-release-scope.ps1') | Out-Null @@ -1913,24 +2046,8 @@ if ($Full) { Write-Host '[SKIP] DocFX digest regression suites (use -Full to run skills/dotnet-docfx-digest/scripts/test-quality.ps1 and test-project-scoped.ps1)' } -$passed = @($results | Where-Object { $_.Status -eq 'PASS' }).Count -$failed = @($results | Where-Object { $_.Status -eq 'FAIL' }).Count -$label = if ([string]::IsNullOrWhiteSpace($Ref)) { 'WORKTREE' } else { $Ref } $mode = if ($Full) { 'FULL' } else { 'FAST' } - -Write-Host ("Validation target: {0}" -f $label) -Write-Host ("Validation mode: {0}" -f $mode) -Write-Host ("Passed: {0}" -f $passed) -Write-Host ("Failed: {0}" -f $failed) -Write-Host '' - -foreach ($result in $results) { - $prefix = if ($result.Status -eq 'PASS') { '[PASS]' } else { '[FAIL]' } - Write-Host ("{0} {1}" -f $prefix, $result.Name) - if ($result.Status -eq 'FAIL') { - Write-Host (" {0}" -f $result.Details) - } -} +$failed = Write-ValidationSummary -Results $results -GitRef $Ref -Mode $mode if ($failed -gt 0) { exit 1 diff --git a/skills/dotnet-segregated-assets/FORMS.md b/skills/dotnet-segregated-assets/FORMS.md new file mode 100644 index 0000000..2bccdb0 --- /dev/null +++ b/skills/dotnet-segregated-assets/FORMS.md @@ -0,0 +1,142 @@ +# .NET Segregated Static Assets Input Form + +Collect only the fields that are still unresolved after running `segregate-assets.cs inspect` and reading the repository. Use the runner's Cuemon and custom-abstraction evidence to resolve `asset_configuration`; do not infer an abstraction from a filename alone. Most fields have a computed or recommended default — present it first and accept a blank answer as acceptance. Prefer the host's native structured input controls when they are available; otherwise use the deterministic plain-text fallback described under **Presentation rules** without changing field order, defaults, recommended choices, or the final confirmation. + +The single question you must always resolve is `cdn_equivalent`. It changes whether a second local origin is provisioned and how shared assets are referenced, and it must never be assumed. + +## Fields + +### web_project + +- **type:** single-choice +- **prompt:** Which web project should be segregated? +- **choices:** The `Microsoft.NET.Sdk.Web` projects reported by `segregate-assets.cs inspect` +- **default:** The single resolved web project, or the project named by the user (Recommended) +- **required:** true +- **show_when:** `inspect` reports classification `Ambiguous` (more than one web project) + +### cdn_equivalent + +- **type:** single-choice +- **prompt:** Does a shared CDN / reusable-asset equivalent exist for this application (fonts, icon libraries, JavaScript/CSS frameworks, design-system assets shared across applications)? +- **choices:** + - No — only this application's own assets (Recommended) + - Yes — a shared/CDN asset source exists +- **default:** No — only this application's own assets (Recommended) +- **required:** true + +### cdn_source + +- **type:** text +- **prompt:** Where does the shared/CDN asset content live today (repository, artifact, existing host, or local path)? Do not assume it belongs in this application's wwwroot. +- **default:** (none) +- **required:** false +- **show_when:** `cdn_equivalent` is `Yes` + +### asset_configuration + +- **type:** single-choice +- **prompt:** How are App/CDN asset URLs generated in this application? +- **choices:** + - Cuemon App/CDN TagHelpers already present — reuse `AppTagHelperOptions`/`CdnTagHelperOptions` (Recommended when detected) + - The application's own suitable asset base-URL option/setting + - No suitable abstraction yet — introduce the smallest app-owned setting only if required +- **default:** Auto-detected from inspection using package/project references, namespace imports, options, `_ViewImports.cshtml`, and actual `app-*`/`cdn-*` markup (Recommended) +- **required:** true + +When Cuemon is detected, do not select the non-Cuemon or new-abstraction choice merely because an earlier `AppAssetOptions`-style abstraction exists. Treat it as a migration input, classify its consumers as App or CDN, and remove it only after proving that no consumers remain. + +### app_origin_port + +- **type:** text +- **prompt:** Which host port should the local App Static Content Provider use? +- **choices:** + - `8080` (Recommended) + - A custom free port (resolve collisions against existing launchSettings/Compose) +- **default:** `8080` (Recommended) +- **required:** true + +### web_host_port + +- **type:** text +- **prompt:** Which host port should the Compose web service publish? +- **choices:** + - The HTTP port already used by the ordinary Project profile's `applicationUrl` (Recommended) + - A custom free port +- **default:** The ordinary Project profile's HTTP `applicationUrl` port (Recommended) +- **required:** true + +Reusing the ordinary profile's HTTP port keeps ordinary and segregated Development on one origin, so bookmarks, redirect registrations, and cookie scopes survive the switch. Never substitute a round number such as `5000` for the derived value. Ask only when the project has no HTTP `applicationUrl` to derive from or the derived port collides with `app_origin_port`. + +### visual_studio_compose + +- **type:** single-choice +- **prompt:** Should Visual Studio start the application and asset origins together from one Docker Compose launch profile? +- **choices:** + - Yes — add or reuse Visual Studio Docker Compose orchestration (Recommended) + - No — run the complete `compose.assets.yml` topology from the command line without Visual Studio registration +- **default:** Yes when the repository has a solution file or an existing `.dcproj`; otherwise No (Recommended) +- **required:** false +- **show_when:** The repository has a solution file and the desired launch experience is unresolved + +This field controls only the IDE registration layer — `docker-compose.dcproj`, the root `launchSettings.json`, `DockerComposeProjectPath`, and solution registration. It never controls the artifact-first contract. `Dockerfile`, `LocalDevelopment.Dockerfile`, `Assets.Dockerfile`, `LocalPublishDirectory`, the guarded publish target, the root `.dockerignore`, and the CI publish plus image builds are required whenever `compose.assets.yml` is created, because `compose.assets.yml` builds the web service from `LocalDevelopment.Dockerfile` regardless of which client starts it. + +### cdn_origin_port + +- **type:** text +- **prompt:** Which host port should the local CDN Static Content Provider use? +- **choices:** + - `8081` (Recommended) + - A custom free port different from the App origin port +- **default:** `8081` (Recommended) +- **required:** false +- **show_when:** `cdn_equivalent` is `Yes` + +### deployed_app_host + +- **type:** text +- **prompt:** What is the deployed App asset host (HTTPS), if known? Used only for deployed configuration. +- **default:** (leave as a documented placeholder such as `assets.example.com` when unknown) +- **required:** false + +### deployed_cdn_host + +- **type:** text +- **prompt:** What is the deployed shared/CDN asset host (HTTPS), if known? +- **default:** (leave as a documented placeholder such as `cdn.example.com` when unknown) +- **required:** false +- **show_when:** `cdn_equivalent` is `Yes` + +### production_image + +- **type:** single-choice +- **prompt:** Build a derived `codebeltnet/web-cdn-origin:2.0.0` production asset image for this application's assets? +- **choices:** + - Yes — add a derived asset image (Recommended) + - No — configuration and local topology only +- **default:** Yes — add a derived asset image (Recommended) +- **required:** true + +### confirmation + +- **type:** single-choice +- **prompt:** Apply App-asset segregation (and CDN provisioning when applicable) using the summarized project, ports, hosts, and configuration, then verify the publish invariant? +- **choices:** + - Yes (Recommended) + - No +- **default:** Yes (Recommended) +- **required:** true + +## Presentation rules + +- Run `segregate-assets.cs inspect` first and infer explicit answers from its output and the repository; do not ask questions the inspection already answers. +- Treat `assetAbstractions.cuemon` and `assetAbstractions.custom` as evidence. If both are present, plan a semantic migration and cleanup; do not leave two URL-generation systems behind. +- Ask one unresolved field at a time. Never bundle multiple questions. +- Present the recommended/default choice first and suffix it with `(Recommended)`. +- For `web_project`, offer the discovered project names as selectable choices; when exactly one web project applies, select it without asking. +- For `cdn_equivalent`, always ask if it is unresolved — never assume shared assets belong in the application's wwwroot. +- For `text` fields with a computed default (ports, hosts), offer the computed value as a selectable choice alongside free text, and treat a blank response as accepting the shown value. +- Derive `web_host_port` from the ordinary Project profile's HTTP `applicationUrl` before asking, and skip the field entirely when that port exists and does not collide with `app_origin_port`. +- If native structured input widgets are unavailable, follow this deterministic plain-text fallback instead of improvising your own questioning style: start immediately with `Field: `, then a one-line prompt, then numbered choices (recommended first), and accept a blank reply as the default. Do not add a conversational preamble, and do not switch interaction styles mid-collection. Consistency matters more than creativity during parameter collection. +- Respect `show_when` conditions: skip `cdn_source`, `cdn_origin_port`, and `deployed_cdn_host` entirely when `cdn_equivalent` is `No`; skip `web_project` when only one web project exists. +- After all fields are resolved, summarize the exact project, web and App/CDN ports, deployed hosts, asset-configuration approach, any competing abstraction cleanup, Visual Studio Compose choice, and whether a production image will be built, then ask `confirmation`. Include the resolved file locations in the summary so a wrong Dockerfile placement is visible before anything is written. diff --git a/skills/dotnet-segregated-assets/SKILL.md b/skills/dotnet-segregated-assets/SKILL.md new file mode 100644 index 0000000..cb2513f --- /dev/null +++ b/skills/dotnet-segregated-assets/SKILL.md @@ -0,0 +1,180 @@ +--- +name: dotnet-segregated-assets +description: > + Migrate or configure an ASP.NET Core web application so developers keep authoring static files in the conventional wwwroot while deployed static content is served by Codebelt Static Content Provider (codebeltnet/web-cdn-origin:2.0.0), a separate asset host rather than the web app. Use when asked to segregate static assets, move wwwroot off the web app, stop shipping wwwroot with the app, or reconcile Cuemon App/CDN TagHelpers with a segregated topology. Reuse existing Cuemon or project abstractions, distinguish App assets from shared CDN assets, preserve Static Web Assets, and verify publish/local invariants deterministically. Do NOT use to build a general-purpose CDN or migrate non-ASP.NET static sites. +compatibility: > + Requires the .NET SDK 10+ and PowerShell 7+. NuGet.org access is required when plan resolves an existing Cuemon package reference. Docker is optional (only for the local origin). CI guidance targets GitHub Actions, which is the assumed delivery surface. +--- + +# .NET Segregated Static Assets + +Keep `wwwroot` as the conventional, tooling-friendly authoring root while making the deployed web application stop serving and shipping its application-owned files. Deployed static content is delivered by Codebelt Static Content Provider (`codebeltnet/web-cdn-origin:2.0.0`) through a separate asset host, not by the ASP.NET Core business application. + +The architecture is: + +```text +Razor declares ownership: app-* or cdn-* +Configuration declares location: current application, local segregated origin, or deployed host +web-cdn-origin declares delivery: static content only +``` + +## Runner boundary + +The bundled .NET file-based program `scripts/segregate-assets.cs` is the deterministic inspection and verification layer. The agent is the orchestration and editing layer: it resolves repository conventions, makes semantic source/configuration edits, and verifies the result. The runner never edits or rewrites Razor, C#, project files, launch profiles, Dockerfiles, Compose, or documentation. + +```text +dotnet run --file "/scripts/segregate-assets.cs" -- [options] +``` + +Use `inspect` before changing anything, `plan` for a read-only ordered decision list, `verify` to publish into an isolated temporary directory and check local topology, and `--self-test` for the hermetic runner tests. When the selected project has an actual `Cuemon.AspNetCore.Razor.TagHelpers` package reference, `plan` discovers NuGet's package-content endpoint through the V3 service index, selects the highest stable version, and emits it in `resolvedNuGetPackages` plus the `nuget-package-version` decision. Prereleases are excluded. If NuGet cannot be queried, planning fails instead of copying a version from repository fixtures, examples, templates, or memory. Add `--json` when consuming results mechanically. + +`inspect` reports candidate projects, risk signals, existing segregation, and asset-abstraction evidence including: + +- Cuemon package references in the project or inherited build files, referenced projects, namespace imports, `AppTagHelperOptions`, `CdnTagHelperOptions`, `_ViewImports.cshtml` registration, and actual `app-*`/`cdn-*` elements; +- custom `AppAssetOptions`-style types, options registrations, Razor injections, and `GetUrl`/`GetAssetUrl`-style calls; +- coexistence of Cuemon and a competing custom abstraction; +- stale attribute-style syntax and cache-busting signals such as `asp-append-version` and `ICacheBusting`. + +Use these facts to decide what the agent should edit. Do not turn the runner into a general source-code migration engine. + +## Critical invariants + +- Keep `wwwroot` as the conventional authoring root and `/cdnroot` as the container content root. The asset-image input must be reproducible from a clean checkout: do not ignore the only source copy. A tracked repository-level asset root is a valid alternative only when the ordinary application web root, Docker build context, CI inputs, and verification are changed together. Do not resurrect `approot`. +- Exclude application-owned `wwwroot` from the deployed web application's publish artifact with targeted metadata: ``. +- Do not use `false` as a blanket solution. Preserve Blazor, RCL, framework, generated, and frontend Static Web Assets guards; stop for an explicit generated-static-assets design when those outputs cannot be safely materialized and verified. +- Static assets are served by `codebeltnet/web-cdn-origin:2.0.0`; local asset content is supplied by `Assets.Dockerfile` or an explicit read-only `/cdnroot` mount, and the runtime remains non-root. +- File placement is part of the contract. `Dockerfile`, `LocalDevelopment.Dockerfile`, and `Assets.Dockerfile` live beside the web `.csproj`; `compose.assets.yml`, `.dockerignore`, `docker-compose.dcproj`, and the Compose `launchSettings.json` live at the repository root. Never place a Dockerfile at the repository root for this topology. +- Write every generated file from the literal template in `assets/`, substituting the documented placeholders. Do not reconstruct a Dockerfile, Compose file, `.dcproj`, or launch profile from memory. +- Both application Dockerfiles are artifact-first: they package an already-published `artifacts/publish/` directory and never restore, build, or publish source. An SDK stage, a `dotnet build`/`dotnet publish` step, a `RUN adduser` block, or an `mcr.microsoft.com` runtime tag in either file is a defect. +- When repository CI builds the application container, it must also build or validate `Assets.Dockerfile` from the same commit. A locally working asset image that hosted CI never constructs is not a proven deployment artifact. Adding an artifact-first `Dockerfile` also obliges you to add the CI job that publishes the artifact it copies: extend the existing GitHub Actions workflow, or create one from `assets/ci-pipeline.yml` when the repository has none. Never leave the production image without a producer, and name which one you did. +- App assets and shared CDN assets are separate concepts. Always determine whether a shared/CDN equivalent exists before deciding origins or markup. Never copy shared content into an application's `wwwroot`. +- Preserve ordinary Development through the existing Project profile. Name the root Docker Compose profile by appending `.Assets` to that ordinary profile name, for example `BingeKinLanding.WebApp` becomes `BingeKinLanding.WebApp.Assets`. +- Verification is deterministic and re-running the skill is idempotent. Never claim the publish invariant from an MSBuild declaration alone; run `verify --run-publish`. +- Do not add Cuemon merely to implement this skill when the application otherwise does not use it. +- When an existing NuGet package must be updated for the migration, use the exact latest-stable version emitted by the current `plan` run. Preserve Central Package Management by updating its existing `PackageVersion`; otherwise update the existing `PackageReference`. Never introduce an inline version beside CPM or fall back to a stale literal when NuGet resolution fails. + +## Decision hierarchy for asset URL abstractions + +Existing framework and project abstractions are preferred over skill-invented abstractions. Resolve the following order after `inspect`: + +1. **Cuemon is already available.** Reuse `Cuemon.AspNetCore.Razor.TagHelpers`, `AppTagHelperOptions`, and `CdnTagHelperOptions`. For an actual package reference, take its version only from the current runner plan's NuGet-backed `resolvedNuGetPackages` result; a project-reference-only setup needs no NuGet version. Do not create `AppAssetOptions`, `SegregatedAssetsOptions`, another `GetAssetUrl()` abstraction, or a second configuration hierarchy. If a previous migration created a custom abstraction, migrate it away as described below. +2. **A suitable non-Cuemon abstraction exists.** Reuse it. Do not add Cuemon solely because the skill knows about it. +3. **No suitable abstraction exists.** Introduce only the smallest app-owned configuration mechanism needed by the existing application, and only after confirming that no framework/project abstraction is available. + +When Cuemon is detected through multiple signals, use the referenced package/source to confirm its exact current configuration surface. The current public model exposes `TagHelperOptions.BaseUrlMode` with `TagHelperBaseUrlMode.Configured` and `TagHelperBaseUrlMode.Automatic`, alongside `BaseUrl` and `ProtocolUriScheme`. For App assets, set `AppTagHelperOptions.BaseUrlMode = TagHelperBaseUrlMode.Automatic`: an explicit App `BaseUrl` wins, while an absent App `BaseUrl` resolves against the active application request. Keep CDN assets explicitly configured with `CdnTagHelperOptions.BaseUrlMode = TagHelperBaseUrlMode.Configured` and an explicit CDN base. Cuemon does not inspect launch-profile names. + +## App/CDN ownership and Razor migration + +Classify every static reference by ownership before changing it. + +App assets are owned by exactly one application: its CSS, JavaScript, branding, images, favicons, manifest, application fonts, and application-specific media. Shared CDN assets are reusable across applications: Bootstrap, Font Awesome, shared fonts, design-system packages, reusable JavaScript/CSS libraries, and common images. A file's current directory or URL shape does not establish ownership. + +When Cuemon is available, treat each TagHelper class's `HtmlTargetElement` attribute as the selector contract; never infer a selector from the class name. The current public selectors are `app-link`, `app-script`, `app-img`, `cdn-link`, `cdn-script`, and `cdn-img`: + +```html +