diff --git a/.github/skills/aw-value/SKILL.md b/.github/skills/aw-value/SKILL.md new file mode 100644 index 00000000000..b8ad59834ae --- /dev/null +++ b/.github/skills/aw-value/SKILL.md @@ -0,0 +1,134 @@ +--- +name: aw-value +description: "Design and verify a deterministic operational-value grader for a GitHub Agentic Workflow. Use for per-run operational value, evidence attribution, maturation, baselines, and operational-value evaluators. Usage: /aw-value OWNER/REPO WORKFLOW-NAME." +argument-hint: "OWNER/REPO WORKFLOW-NAME" +allowed-tools: bash jq gh +metadata: + version: "1.0.0" +--- + +# Operational Value Grader + +Design one deterministic `operational-value` grader that reports absolute operational attainment for each workflow run. + +Operational value is the degree to which the workflow's intended repository outcome is attained for the opportunity assigned to a run, demonstrated by accepted repository evidence under a frozen contract. It is not execution quality, output volume, safe-output creation, or an agent's assessment. + +## Output + +Create one executable evaluator at: + +```text +.github/graders/WORKFLOW-NAME-operational-value.sh +``` + +Configure the workflow: + +```yaml +graders: + operational-value: + run: .github/graders/WORKFLOW-NAME-operational-value.sh +``` + +The grader's primary `value` is absolute attainment in `[0,1]`. A comparable frozen baseline may be reported separately as `baselineValue`; gh-aw derives `deltaFromBaseline`. Never define the primary value as a difference from baseline. + +## Design Procedure + +1. Validate `OWNER/REPO` and resolve `.github/workflows/WORKFLOW-NAME.md`. Do not infer inputs from the workspace or remotes. +2. Recover adoption-time intent from the workflow's first commit and first parent. Use only adoption-time workflow content and pre-adoption evidence to choose opportunities, accepted evidence, formulas, targets, or a baseline. +3. Define how every workflow run binds to one operational case: + - produce a stable `opportunityKey`; + - prevent overlapping ownership where possible; + - preserve repeated keys when duplicate runs target the same opportunity so downstream analysis can cluster or deduplicate them; + - treat reruns with the same GitHub run ID as the same subject. +4. Freeze accepted evidence, evidence repositories, matching rules, zero-versus-missing behavior, and `maturesAt` computation. + - Declare only the workflow permission scopes required to collect that evidence. The evaluator receives `GH_TOKEN` with the agent job's declared permissions; gh-aw does not add evidence permissions automatically. +5. Choose exactly one direct primary metric in `[0,1]`. Higher must always mean greater attainment. Keep trace graders and activity counts separate. +6. If comparable pre-adoption evidence exists, score it with the same metric and freeze it under `baseline`. Otherwise use `attainment-only` with a null baseline value. +7. Implement the evaluator interface below and run: + + ```bash + .github/skills/aw-value/scripts/verify-operational-value-evaluator.sh .github/graders/WORKFLOW-NAME-operational-value.sh + gh aw compile .github/workflows/WORKFLOW-NAME.md + ``` + +## Evaluator Interface + +The evaluator uses Bash 3.2-compatible Bash plus `jq` and supports: + +- `--definition`: print the frozen schema-version 4 contract. +- `--metric`: read one evidence object on stdin and print a deterministic number in `[0,1]` or `null`. +- `--grade-run`: read a run request on stdin and print one value observation. + +`--grade-run` receives: + +```json +{ + "schemaVersion": 1, + "run": { + "id": "12345", + "attempt": 1, + "repository": "OWNER/REPO", + "workflow": "Workflow name", + "ref": "refs/heads/main", + "sha": "...", + "eventName": "schedule", + "createdAt": "2026-08-23T11:58:00Z" + }, + "evidenceAt": "2026-08-23T12:00:00.000Z", + "case": null, + "event": null, + "config": {} +} +``` + +It returns: + +```json +{ + "value": 0.75, + "opportunityKey": "issue:42", + "case": {"issue": 42}, + "evidenceCutoff": "2026-08-23T12:00:00.000Z", + "maturesAt": "2026-08-30T12:00:00.000Z", + "provenance": [ + {"repository": "OWNER/REPO", "kind": "issue", "ref": "42"} + ], + "diagnostics": {} +} +``` + +The function must cap `evidenceCutoff` at the earlier of `evidenceAt` and `maturesAt`. A run is never intrinsically pending: the value is an as-of observation and may be recomputed until maturity. After maturity, the cap makes the result stable. + +## Regrade a Historical Run + +Recompute a run at an explicit evidence time with the same evaluator used by the original run: + +```bash +gh aw graders operational-value RUN-ID \ + --evidence-at 2026-08-30T12:00:00.000Z \ + --json +``` + +Add `--repo [HOST/]OWNER/REPO` to select the GitHub host for the current repository checkout. The command downloads the original grader artifact, reuses its operational case and complete run subject, and refuses to execute unless the archived evaluator matches both digest records and the evaluator at the recorded commit in the trusted checkout. It prints a new observation and never modifies the original artifact. + +## Definition Contract + +`--definition` must contain: + +- `schemaVersion: 4` and `grader: "operational-value"`; +- repository, workflow name, source path, and adoption commit/time; +- operational-value statement; +- evidence opportunity, assignment, accepted evidence, repositories, collection, maturation, zero rule, and missing rule; +- one primary metric with formula and validation examples; +- baseline mode, value, cutoff, and provenance. + +For `baseline-comparable`, baseline value must be in `[0,1]` and have immutable provenance. For `attainment-only`, baseline value and cutoff must be null. + +## Interpretation Rules + +- `value` answers “how fully was this run's assigned opportunity attained?” +- `deltaFromBaseline` answers “how far is this observation above or below the frozen pre-adoption reference?” +- Neither establishes that the workflow caused the outcome. +- Compare runs only under the same evaluator digest and evidence horizon. +- Identify a replayed observation by `(runId, evaluatorDigest, evidenceAt)`. +- Do not treat repeated observations of one run, duplicate opportunity keys, or overlapping state windows as independent samples. \ No newline at end of file diff --git a/.github/skills/aw-value/scripts/operational-value-evaluator-path.sh b/.github/skills/aw-value/scripts/operational-value-evaluator-path.sh new file mode 100755 index 00000000000..b43f0ca2d72 --- /dev/null +++ b/.github/skills/aw-value/scripts/operational-value-evaluator-path.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash + +set -euo pipefail + +fail() { + printf 'error: %s\n' "$*" >&2 + exit 1 +} + +[[ $# -eq 1 ]] || fail "usage: operational-value-evaluator-path.sh WORKFLOW-NAME" + +workflow_name=$1 +[[ $workflow_name =~ ^[a-z0-9]+(-[a-z0-9]+)*$ ]] \ + || fail "workflow name must contain lowercase letters, numbers, and single hyphens" + +printf '.github/graders/%s-operational-value.sh\n' "$workflow_name" \ No newline at end of file diff --git a/.github/skills/aw-value/scripts/verify-operational-value-evaluator.sh b/.github/skills/aw-value/scripts/verify-operational-value-evaluator.sh new file mode 100755 index 00000000000..6415383d639 --- /dev/null +++ b/.github/skills/aw-value/scripts/verify-operational-value-evaluator.sh @@ -0,0 +1,122 @@ +#!/usr/bin/env bash + +set -euo pipefail + +fail() { + printf 'error: %s\n' "$*" >&2 + exit 1 +} + +[[ $# -eq 1 ]] || fail "usage: verify-operational-value-evaluator.sh " + +evaluator=$1 +[[ -f $evaluator ]] || fail "operational-value evaluator not found: $evaluator" +[[ -x $evaluator ]] || fail "operational-value evaluator is not executable: $evaluator" +command -v jq >/dev/null 2>&1 || fail "jq is required" +bash -n "$evaluator" + +definition=$("$evaluator" --definition) +printf '%s\n' "$definition" | jq -e ' + .schemaVersion == 4 + and .grader == "operational-value" + and (.repository | type == "string" and test("^[^/]+/[^/]+$")) + and (.workflowName | type == "string" and length > 0) + and (.sourcePath | type == "string" and startswith(".github/workflows/") and endswith(".md")) + and (.adoption.commit | type == "string" and test("^[0-9a-f]{40}$")) + and (.adoption.adoptedAt | type == "string" and test("^[0-9]{4}-[0-9]{2}-[0-9]{2}T")) + and (.operationalValue | type == "string" and length > 0) + and (.evidence.opportunity | type == "string" and length > 0) + and (.evidence.assignment | type == "string" and length > 0) + and (.evidence.accepted | type == "string" and length > 0) + and (.evidence.repositories | type == "array" and length > 0) + and (all(.evidence.repositories[]; type == "string" and test("^[^/]+/[^/]+$"))) + and (.evidence.collection | type == "string" and length > 0) + and (.evidence.maturation | type == "string" and length > 0) + and (.evidence.zeroRule | type == "string" and length > 0) + and (.evidence.missingRule | type == "string" and length > 0) + and (.primaryMetric.id | type == "string" and length > 0) + and (.primaryMetric.formula | type == "string" and length > 0) + and (.primaryMetric.direction == "higher_is_better") + and (.validationExamples | has("targetAttained") and has("targetMissed") and has("missing") and has("malformed")) + and (.baseline.mode == "baseline-comparable" or .baseline.mode == "attainment-only") + and (if .baseline.mode == "baseline-comparable" then + (.baseline.value | type == "number" and . >= 0 and . <= 1) + and (.baseline.evidenceCutoff | type == "string" and length > 0) + and (.baseline.provenance | type == "array" and length > 0) + else + .baseline.value == null + and .baseline.evidenceCutoff == null + end) +' >/dev/null || fail "operational-value evaluator definition is invalid" + +for example_name in targetAttained targetMissed missing malformed; do + evidence=$(printf '%s\n' "$definition" | jq -c --arg name "$example_name" '.validationExamples[$name]') + result=$(printf '%s\n' "$evidence" | "$evaluator" --metric) + printf '%s\n' "$result" | jq -e '. == null or (type == "number" and . >= 0 and . <= 1)' >/dev/null \ + || fail "--metric returned an invalid score for $example_name" + case $example_name in + targetAttained) target_attained=$result ;; + targetMissed) target_missed=$result ;; + missing|malformed) + [[ $result == null ]] || fail "--metric must return null for $example_name" + ;; + esac +done + +jq -en --argjson attained "$target_attained" --argjson missed "$target_missed" \ + '$attained != null and $missed != null and $attained > $missed' >/dev/null \ + || fail "targetAttained must score higher than targetMissed" + +repository=$(printf '%s\n' "$definition" | jq -r '.repository') +workflow_name=$(printf '%s\n' "$definition" | jq -r '.workflowName') +adoption_commit=$(printf '%s\n' "$definition" | jq -r '.adoption.commit') +created_at=$(printf '%s\n' "$definition" | jq -r '.adoption.adoptedAt') +evidence_at=2099-01-01T00:00:00Z +request=$(jq -cn \ + --arg repository "$repository" \ + --arg workflow "$workflow_name" \ + --arg sha "$adoption_commit" \ + --arg createdAt "$created_at" \ + --arg evidenceAt "$evidence_at" \ + '{ + schemaVersion: 1, + run: { + id: "1", + attempt: 1, + repository: $repository, + workflow: $workflow, + ref: "refs/heads/main", + sha: $sha, + eventName: "workflow_dispatch", + createdAt: $createdAt + }, + evidenceAt: $evidenceAt, + case: null, + event: {}, + config: {verification: true} + }') +grade_run=$(printf '%s\n' "$request" | "$evaluator" --grade-run) +printf '%s\n' "$grade_run" | jq -e --arg evidenceAt "$evidence_at" ' + def timestamp: + type == "string" + and test("^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\\.[0-9]{3})?Z$"); + def epoch: sub("\\.[0-9]{3}Z$"; "Z") | fromdateiso8601; + type == "object" + and (.value == null or (.value | type == "number" and isfinite and . >= 0 and . <= 1)) + and (.opportunityKey | type == "string" and length > 0) + and (.case | type == "object") + and (.evidenceCutoff | timestamp) + and (.maturesAt | timestamp) + and ((.evidenceCutoff | epoch) <= ($evidenceAt | epoch)) + and ((.evidenceCutoff | epoch) <= (.maturesAt | epoch)) + and (.provenance | type == "array") + and (if .value == null then true else (.provenance | length > 0) end) + and (all(.provenance[]; type == "object" + and (.repository | type == "string" and length > 0) + and (.kind | type == "string" and length > 0) + and (.ref | type == "string" and length > 0))) + and ((has("diagnostics") | not) or (.diagnostics | type == "object")) + and ((has("message") | not) or (.message | type == "string")) +' >/dev/null || fail "--grade-run returned an invalid operational-value observation" + +printf 'verified %s\n' "$evaluator" \ No newline at end of file diff --git a/.github/skills/aw-value/tests/test.sh b/.github/skills/aw-value/tests/test.sh new file mode 100755 index 00000000000..96401dd5d6e --- /dev/null +++ b/.github/skills/aw-value/tests/test.sh @@ -0,0 +1,98 @@ +#!/usr/bin/env bash + +set -euo pipefail + +skill_dir=$(CDPATH='' cd -- "$(dirname -- "$0")/.." && pwd) +repo_root=$(CDPATH='' cd -- "$skill_dir/../../.." && pwd) +work_dir=$(mktemp -d "$repo_root/.aw-value-test.XXXXXX") +trap 'rm -rf "$work_dir"' EXIT HUP INT TERM + +path=$("$skill_dir/scripts/operational-value-evaluator-path.sh" daily-file-diet) +[[ $path == .github/graders/daily-file-diet-operational-value.sh ]] +if "$skill_dir/scripts/operational-value-evaluator-path.sh" ../escape >/dev/null 2>&1; then + printf 'invalid workflow name was accepted\n' >&2 + exit 1 +fi + +evaluator_path="$work_dir/operational-value.sh" +cat > "$evaluator_path" <<'EOF' +#!/usr/bin/env bash + +set -euo pipefail + +case ${1:-} in + --definition) + cat <<'JSON' +{ + "schemaVersion": 4, + "grader": "operational-value", + "repository": "owner/repo", + "workflowName": "Example", + "sourcePath": ".github/workflows/example.md", + "adoption": { + "commit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "adoptedAt": "2026-01-01T00:00:00Z" + }, + "operationalValue": "For eligible issues, attain closure demonstrated by a closed issue.", + "evidence": { + "opportunity": "The issue assigned to the workflow run.", + "assignment": "Bind the triggering issue number to the run ID.", + "accepted": "The issue is closed by the evidence cutoff.", + "repositories": ["owner/repo"], + "collection": "Read immutable issue events through the capped cutoff.", + "maturation": "Seven days after the run was created.", + "zeroRule": "An eligible open issue at the cutoff scores zero.", + "missingRule": "An unavailable issue or event history scores null." + }, + "primaryMetric": { + "id": "issue-closure", + "formula": "1 when closed, otherwise 0", + "direction": "higher_is_better" + }, + "baseline": { + "mode": "baseline-comparable", + "value": 0.4, + "evidenceCutoff": "2025-12-31T00:00:00Z", + "provenance": [{"repository": "owner/repo", "kind": "issue-events", "ref": "baseline"}] + }, + "validationExamples": { + "targetAttained": {"eligible": true, "closed": true}, + "targetMissed": {"eligible": true, "closed": false}, + "missing": {"eligible": false}, + "malformed": {"eligible": "yes"} + } +} +JSON + ;; + --metric) + jq 'if (.eligible | type) != "boolean" or .eligible == false or (.closed | type) != "boolean" then null elif .closed then 1 else 0 end' + ;; + --grade-run) + request=$(cat) + printf '%s\n' "$request" | jq -e ' + .schemaVersion == 1 + and .run.id == "1" + and .run.repository == "owner/repo" + and .run.eventName == "workflow_dispatch" + and .case == null + and .config.verification == true + ' >/dev/null + printf '%s\n' "$request" | jq -c '{ + value: 1, + opportunityKey: "verification:1", + case: {verification: true}, + evidenceCutoff: .evidenceAt, + maturesAt: .evidenceAt, + provenance: [{repository: .run.repository, kind: "verification", ref: .run.id}], + diagnostics: {} + }' + ;; + *) + exit 1 + ;; +esac +EOF +chmod +x "$evaluator_path" + +"$skill_dir/scripts/verify-operational-value-evaluator.sh" "$evaluator_path" >/dev/null +printf 'aw-value skill tests passed\n' \ No newline at end of file diff --git a/actions/setup/js/generate_aw_info.cjs b/actions/setup/js/generate_aw_info.cjs index d3d0bf994f1..28e298ad19c 100644 --- a/actions/setup/js/generate_aw_info.cjs +++ b/actions/setup/js/generate_aw_info.cjs @@ -24,9 +24,10 @@ const { ERR_CONFIG, ERR_SYSTEM } = require("./error_codes.cjs"); * * @param {typeof import('@actions/core')} core - GitHub Actions core library * @param {any} ctx - GitHub Actions context object + * @param {any} [githubClient] - Authenticated GitHub client; falls back to global.github * @returns {Promise} */ -async function main(core, ctx) { +async function main(core, ctx, githubClient) { // Validate numeric context variables before processing run info. // This prevents malicious payloads from hiding special text or code in numeric fields. await validateContextVariables(core, ctx); @@ -97,6 +98,23 @@ async function main(core, ctx) { created_at: new Date().toISOString(), }; + if (process.env.GH_AW_INFO_FETCH_RUN_CREATED_AT === "true") { + try { + // @ts-ignore - global.github is set by setupGlobals() from github-script context + const github = githubClient || global.github; + const response = await github.rest.actions.getWorkflowRun({ + owner: ctx.repo.owner, + repo: ctx.repo.repo, + run_id: ctx.runId, + }); + const runCreatedAt = response.data.created_at || ""; + core.setOutput("run_created_at", runCreatedAt); + } catch (err) { + core.warning(`Unable to load workflow-run creation time: ${getErrorMessage(err)}`); + core.setOutput("run_created_at", ""); + } + } + const frontmatterSource = process.env.GH_AW_INFO_FRONTMATTER_SOURCE || ""; if (frontmatterSource) { awInfo.frontmatter_source = frontmatterSource; diff --git a/actions/setup/js/generate_aw_info.test.cjs b/actions/setup/js/generate_aw_info.test.cjs index 942b0c347e3..96d91088be5 100644 --- a/actions/setup/js/generate_aw_info.test.cjs +++ b/actions/setup/js/generate_aw_info.test.cjs @@ -28,6 +28,14 @@ const mockContext = { repo: { owner: "github", repo: "my-repo" }, }; +const mockGithub = { + rest: { + actions: { + getWorkflowRun: vi.fn(), + }, + }, +}; + describe("generate_aw_info.cjs", () => { let main; let awInfoPath; @@ -64,6 +72,7 @@ describe("generate_aw_info.cjs", () => { process.env.GH_AW_INFO_BODY_MODIFIED = ""; process.env.GH_AW_INFO_FEATURES = ""; process.env.GH_AW_INFO_SKILLS = ""; + delete process.env.GH_AW_INFO_FETCH_RUN_CREATED_AT; // Dynamic import to get fresh module state const module = await import("./generate_aw_info.cjs"); @@ -106,6 +115,20 @@ describe("generate_aw_info.cjs", () => { expect(awInfo.created_at).toBeTruthy(); }); + it("should expose the authoritative run creation time when requested", async () => { + process.env.GH_AW_INFO_FETCH_RUN_CREATED_AT = "true"; + mockGithub.rest.actions.getWorkflowRun.mockResolvedValue({ data: { created_at: "2026-08-24T12:00:00Z" } }); + + await main(mockCore, mockContext, mockGithub); + + expect(mockGithub.rest.actions.getWorkflowRun).toHaveBeenCalledWith({ + owner: "github", + repo: "my-repo", + run_id: 12345, + }); + expect(mockCore.setOutput).toHaveBeenCalledWith("run_created_at", "2026-08-24T12:00:00Z"); + }); + it("should include features from GH_AW_INFO_FEATURES and preserve value types", async () => { process.env.GH_AW_INFO_FEATURES = JSON.stringify({ "gh-aw-detection": true, diff --git a/actions/setup/js/operational_value_grader.cjs b/actions/setup/js/operational_value_grader.cjs new file mode 100644 index 00000000000..9d76bc05ac6 --- /dev/null +++ b/actions/setup/js/operational_value_grader.cjs @@ -0,0 +1,224 @@ +// @ts-check + +const cp = require("child_process"); +const fs = require("fs"); +const path = require("path"); +const { getErrorMessage } = require("./error_helpers.cjs"); + +const OPERATIONAL_VALUE_EVALUATOR_TIMEOUT_MS = 120000; +const OPERATIONAL_VALUE_EVALUATOR_MAX_OUTPUT = 1024 * 1024; +const OPERATIONAL_VALUE_EVENT_MAX_SIZE = 1024 * 1024; +const OPERATIONAL_VALUE_EVALUATOR_TEMP_ROOT = "/tmp/gh-aw/agent"; + +/** @param {unknown} value @returns {value is Record} */ +function isRecord(value) { + return value !== null && typeof value === "object" && !Array.isArray(value); +} + +/** @param {string} value @param {string} label @returns {number} */ +function parseTimestamp(value, label) { + if (typeof value !== "string" || !/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?Z$/.test(value)) { + throw new Error(`${label} must be a UTC ISO-8601 timestamp`); + } + const timestamp = Date.parse(value); + if (!Number.isFinite(timestamp)) { + throw new Error(`${label} must be a valid timestamp`); + } + return timestamp; +} + +/** @param {NodeJS.ProcessEnv} env @param {{createdAt?: string}} [metadata] */ +function buildRunSubject(env, metadata = {}) { + const runId = String(env.GITHUB_RUN_ID || ""); + if (!/^\d+$/.test(runId) || runId === "0") { + throw new Error("GITHUB_RUN_ID must identify the workflow run"); + } + return { + id: runId, + attempt: Number(env.GITHUB_RUN_ATTEMPT) || 1, + repository: String(env.GITHUB_REPOSITORY || ""), + workflow: String(env.GITHUB_WORKFLOW || ""), + ref: String(env.GITHUB_REF || ""), + sha: String(env.GITHUB_SHA || ""), + eventName: String(env.GITHUB_EVENT_NAME || ""), + createdAt: metadata.createdAt || null, + }; +} + +/** @param {NodeJS.ProcessEnv} env */ +function readEventPayload(env) { + const eventPath = env.GITHUB_EVENT_PATH; + if (!eventPath) return null; + try { + const stat = fs.statSync(eventPath); + if (!stat.isFile() || stat.size > OPERATIONAL_VALUE_EVENT_MAX_SIZE) return null; + const event = JSON.parse(fs.readFileSync(eventPath, "utf8")); + return isRecord(event) ? event : null; + } catch { + return null; + } +} + +/** @param {NodeJS.ProcessEnv} env */ +function safeFunctionEnv(env) { + /** @type {NodeJS.ProcessEnv} */ + const result = {}; + for (const key of ["PATH", "HOME", "TMPDIR", "TEMP", "TMP", "SystemRoot", "ComSpec", "GH_TOKEN", "GH_HOST", "GITHUB_API_URL", "GITHUB_GRAPHQL_URL", "GITHUB_SERVER_URL"]) { + if (env[key]) result[key] = env[key]; + } + return result; +} + +function parseOperationalValueBaselineDefinition(rawDefinition) { + let definition; + try { + definition = JSON.parse(rawDefinition || "{}"); + } catch (err) { + throw new Error(`operational-value evaluator returned an invalid definition: ${getErrorMessage(err)}`, { cause: err }); + } + if (!isRecord(definition) || definition.schemaVersion !== 4 || definition.grader !== "operational-value" || !isRecord(definition.baseline)) { + throw new Error("operational-value evaluator definition must use schemaVersion 4 and grader 'operational-value'"); + } + if (definition.baseline.mode === "attainment-only") { + if (definition.baseline.value !== null) throw new Error("attainment-only operational-value evaluators must have a null baseline value"); + return null; + } + if (definition.baseline.mode !== "baseline-comparable") { + throw new Error("operational-value evaluator baseline mode must be 'baseline-comparable' or 'attainment-only'"); + } + const baselineValue = definition.baseline.value; + if (typeof baselineValue !== "number" || !Number.isFinite(baselineValue) || baselineValue < 0 || baselineValue > 1) { + throw new Error("baseline-comparable operational-value evaluators require a baseline value in [0,1]"); + } + return baselineValue; +} + +/** + * Execute and validate one trusted, frozen operational-value evaluator. + * @param {string} evaluatorContent + * @param {{digest?: string, config?: object}} meta + * @param {{evidenceAt?: string, env?: NodeJS.ProcessEnv, event?: object|null, case?: object|null, runMetadata?: {createdAt?: string}, bashPath?: string}} [options] + */ +function executeOperationalValueEvaluator(evaluatorContent, meta, options = {}) { + const env = options.env || process.env; + const evidenceAt = options.evidenceAt || new Date().toISOString(); + const evidenceAtMs = parseTimestamp(evidenceAt, "evidenceAt"); + const run = buildRunSubject(env, options.runMetadata); + const request = { + schemaVersion: 1, + run, + evidenceAt, + case: options.case || null, + event: options.event === undefined ? readEventPayload(env) : options.event, + config: meta.config || {}, + }; + + fs.mkdirSync(OPERATIONAL_VALUE_EVALUATOR_TEMP_ROOT, { recursive: true, mode: 0o700 }); + const tempDir = fs.mkdtempSync(path.join(OPERATIONAL_VALUE_EVALUATOR_TEMP_ROOT, "operational-value-grader-")); + const evaluatorPath = path.join(tempDir, "operational-value.sh"); + const bashPath = options.bashPath || "/bin/bash"; + try { + fs.writeFileSync(evaluatorPath, evaluatorContent, { encoding: "utf8", mode: 0o700 }); + const syntax = cp.spawnSync(bashPath, ["-n", evaluatorPath], { + encoding: "utf8", + timeout: 5000, + env: safeFunctionEnv(env), + }); + if (syntax.error || syntax.status !== 0) { + throw new Error(`operational-value evaluator has invalid Bash syntax: ${syntax.stderr?.trim() || getErrorMessage(syntax.error)}`); + } + + const definitionExecution = cp.spawnSync(bashPath, [evaluatorPath, "--definition"], { + encoding: "utf8", + timeout: 5000, + maxBuffer: OPERATIONAL_VALUE_EVALUATOR_MAX_OUTPUT, + env: safeFunctionEnv(env), + }); + if (definitionExecution.error) throw definitionExecution.error; + if (definitionExecution.status !== 0) { + throw new Error(definitionExecution.stderr?.trim() || `operational-value evaluator --definition exited with status ${String(definitionExecution.status)}`); + } + const baselineValue = parseOperationalValueBaselineDefinition(definitionExecution.stdout); + + const execution = cp.spawnSync(bashPath, [evaluatorPath, "--grade-run"], { + input: JSON.stringify(request), + encoding: "utf8", + timeout: OPERATIONAL_VALUE_EVALUATOR_TIMEOUT_MS, + maxBuffer: OPERATIONAL_VALUE_EVALUATOR_MAX_OUTPUT, + env: safeFunctionEnv(env), + }); + if (execution.error) throw execution.error; + if (execution.status !== 0) { + throw new Error(execution.stderr?.trim() || `operational-value evaluator exited with status ${String(execution.status)}`); + } + + let output; + try { + output = JSON.parse(execution.stdout || "{}"); + } catch (err) { + throw new Error(`operational-value evaluator returned invalid JSON: ${getErrorMessage(err)}`, { cause: err }); + } + if (!isRecord(output)) throw new Error("operational-value evaluator output must be an object"); + if (output.value !== null && (typeof output.value !== "number" || !Number.isFinite(output.value) || output.value < 0 || output.value > 1)) { + throw new Error("operational-value evaluator result.value must be null or a finite number in [0,1]"); + } + if (!isRecord(output.case)) throw new Error("operational-value evaluator output.case must be an object"); + if (typeof output.opportunityKey !== "string" || output.opportunityKey.trim() === "") { + throw new Error("operational-value evaluator opportunityKey must be a non-empty string"); + } + const evidenceCutoffMs = parseTimestamp(output.evidenceCutoff, "evidenceCutoff"); + const maturesAtMs = parseTimestamp(output.maturesAt, "maturesAt"); + if (evidenceCutoffMs > evidenceAtMs) throw new Error("operational-value evaluator evidenceCutoff cannot follow evidenceAt"); + if (evidenceCutoffMs > maturesAtMs) throw new Error("operational-value evaluator evidenceCutoff cannot follow maturesAt"); + if (!Array.isArray(output.provenance) || (output.value !== null && output.provenance.length === 0)) { + throw new Error("operational-value evaluator must return provenance for a numeric value"); + } + for (const provenance of output.provenance) { + if (!isRecord(provenance) || !["repository", "kind", "ref"].every(key => typeof provenance[key] === "string" && provenance[key].length > 0)) { + throw new Error("operational-value evaluator provenance entries require repository, kind, and ref"); + } + } + return { + value: output.value, + ...(typeof output.message === "string" ? { message: output.message } : {}), + ...(isRecord(output.diagnostics) ? { diagnostics: output.diagnostics } : {}), + observation: { + subject: { + type: "workflow-run", + runId: run.id, + attempt: run.attempt, + repository: run.repository, + workflow: run.workflow, + ref: run.ref, + sha: run.sha, + eventName: run.eventName, + createdAt: run.createdAt, + }, + opportunityKey: output.opportunityKey, + evidenceAt, + evidenceCutoff: output.evidenceCutoff, + maturesAt: output.maturesAt, + mature: evidenceAtMs >= maturesAtMs, + case: output.case, + provenance: output.provenance, + }, + baselineValue, + deltaFromBaseline: typeof output.value === "number" && baselineValue !== null ? output.value - baselineValue : null, + }; + } finally { + fs.rmSync(tempDir, { recursive: true, force: true }); + } +} + +module.exports = { + executeOperationalValueEvaluator, + buildRunSubject, + readEventPayload, + parseTimestamp, + parseOperationalValueBaselineDefinition, + safeFunctionEnv, + OPERATIONAL_VALUE_EVALUATOR_TEMP_ROOT, + OPERATIONAL_VALUE_EVALUATOR_TIMEOUT_MS, + OPERATIONAL_VALUE_EVALUATOR_MAX_OUTPUT, + OPERATIONAL_VALUE_EVENT_MAX_SIZE, +}; diff --git a/actions/setup/js/operational_value_grader.test.cjs b/actions/setup/js/operational_value_grader.test.cjs new file mode 100644 index 00000000000..9c80f033e4c --- /dev/null +++ b/actions/setup/js/operational_value_grader.test.cjs @@ -0,0 +1,153 @@ +// @ts-check + +const { executeOperationalValueEvaluator, buildRunSubject, safeFunctionEnv, OPERATIONAL_VALUE_EVALUATOR_TEMP_ROOT } = require("./operational_value_grader.cjs"); + +const TEST_ENV = { + PATH: process.env.PATH, + HOME: process.env.HOME, + TMPDIR: process.env.TMPDIR, + GITHUB_RUN_ID: "12345", + GITHUB_RUN_ATTEMPT: "2", + GITHUB_REPOSITORY: "github/gh-aw", + GITHUB_WORKFLOW: "Example", + GITHUB_REF: "refs/heads/main", + GITHUB_SHA: "0123456789abcdef", + GITHUB_EVENT_NAME: "schedule", +}; + +function operationalValueEvaluator(output, baseline = { mode: "baseline-comparable", value: 0.25 }) { + return `#!/usr/bin/env bash +set -euo pipefail +case \${1:-} in +--definition) +cat <<'DEFINITION' +${JSON.stringify({ schemaVersion: 4, grader: "operational-value", baseline })} +DEFINITION +;; +--grade-run) +cat >/dev/null +cat <<'RESULT' +${JSON.stringify(output)} +RESULT +;; +*) exit 1 ;; +esac +`; +} + +describe("operational_value_grader", () => { + it("uses the gh-aw agent temp root and forwards the GitHub GraphQL URL", () => { + expect(OPERATIONAL_VALUE_EVALUATOR_TEMP_ROOT).toBe("/tmp/gh-aw/agent"); + expect(safeFunctionEnv({ GITHUB_GRAPHQL_URL: "https://api.github.com/graphql" })).toEqual({ + GITHUB_GRAPHQL_URL: "https://api.github.com/graphql", + }); + }); + + it("builds a stable workflow-run subject", () => { + expect(buildRunSubject(TEST_ENV)).toEqual({ + id: "12345", + attempt: 2, + repository: "github/gh-aw", + workflow: "Example", + ref: "refs/heads/main", + sha: "0123456789abcdef", + eventName: "schedule", + createdAt: null, + }); + }); + + it("returns absolute value with a secondary baseline delta", () => { + const output = executeOperationalValueEvaluator( + operationalValueEvaluator({ + value: 0.75, + opportunityKey: "schedule:2026-08-23", + case: { key: "schedule:2026-08-23" }, + evidenceCutoff: "2026-08-23T12:00:00Z", + maturesAt: "2026-08-30T12:00:00Z", + provenance: [{ repository: "github/gh-aw", kind: "git-commit", ref: "abc123" }], + }), + { digest: "abc" }, + { evidenceAt: "2026-08-24T12:00:00Z", env: TEST_ENV } + ); + + expect(output.value).toBe(0.75); + expect(output.baselineValue).toBe(0.25); + expect(output.deltaFromBaseline).toBe(0.5); + expect(output.observation.subject).toEqual({ + type: "workflow-run", + runId: "12345", + attempt: 2, + repository: "github/gh-aw", + workflow: "Example", + ref: "refs/heads/main", + sha: "0123456789abcdef", + eventName: "schedule", + createdAt: null, + }); + expect(output.observation.mature).toBe(false); + }); + + it("caps evidence at maturation and marks mature observations", () => { + const output = executeOperationalValueEvaluator( + operationalValueEvaluator( + { + value: 1, + opportunityKey: "issue:42", + case: { issue: 42 }, + evidenceCutoff: "2026-08-30T12:00:00Z", + maturesAt: "2026-08-30T12:00:00Z", + provenance: [{ repository: "github/gh-aw", kind: "issue", ref: "42" }], + }, + { mode: "attainment-only", value: null } + ), + {}, + { evidenceAt: "2026-09-01T12:00:00Z", env: TEST_ENV } + ); + + expect(output.observation.mature).toBe(true); + expect(output.observation.evidenceCutoff).toBe("2026-08-30T12:00:00Z"); + expect(output.baselineValue).toBeNull(); + expect(output.deltaFromBaseline).toBeNull(); + }); + + it("rejects invalid values and uncapped evidence", () => { + expect(() => + executeOperationalValueEvaluator( + operationalValueEvaluator({ + value: 2, + opportunityKey: "issue:42", + case: { issue: 42 }, + evidenceCutoff: "2026-09-01T12:00:00Z", + maturesAt: "2026-08-30T12:00:00Z", + provenance: [], + }), + {}, + { evidenceAt: "2026-09-01T12:00:00Z", env: TEST_ENV } + ) + ).toThrow("result.value must be null or a finite number in [0,1]"); + }); + + it("rejects invalid Bash", () => { + expect(() => executeOperationalValueEvaluator("#!/usr/bin/env bash\nif", {}, { evidenceAt: "2026-08-24T12:00:00Z", env: TEST_ENV })).toThrow("invalid Bash syntax"); + }); + + it("rejects an invalid frozen baseline", () => { + expect(() => + executeOperationalValueEvaluator( + operationalValueEvaluator( + { + value: 1, + opportunityKey: "issue:42", + case: { issue: 42 }, + evidenceCutoff: "2026-08-24T12:00:00Z", + maturesAt: "2026-08-30T12:00:00Z", + provenance: [{ repository: "github/gh-aw", kind: "issue", ref: "42" }], + }, + { mode: "baseline-comparable", value: 2 } + ), + {}, + { evidenceAt: "2026-08-24T12:00:00Z", env: TEST_ENV } + ) + ).toThrow("baseline value in [0,1]"); + }); +}); diff --git a/actions/setup/js/pi_provider.test.cjs b/actions/setup/js/pi_provider.test.cjs index a4530f2019c..609211e85f8 100644 --- a/actions/setup/js/pi_provider.test.cjs +++ b/actions/setup/js/pi_provider.test.cjs @@ -212,10 +212,9 @@ describe("pi_provider.cjs", () => { module.default(pi); await handlers.agent_start(); + const reflectOutputPath = path.join(process.env.RUNNER_TEMP || os.tmpdir(), "awf-reflect.json"); expect( - stderrOutput.some(line => - line.includes('reflect_failure phase=agent_start provider=copilot model=copilot/claude-sonnet-4 url=http://api-proxy:10000/reflect output=/tmp/gh-aw/sandbox/firewall/awf-reflect.json reason=request_failed error="ECONNREFUSED"') - ) + stderrOutput.some(line => line.includes(`reflect_failure phase=agent_start provider=copilot model=copilot/claude-sonnet-4 url=http://api-proxy:10000/reflect output=${reflectOutputPath} reason=request_failed error="ECONNREFUSED"`)) ).toBe(true); }); diff --git a/actions/setup/js/trace_graders.cjs b/actions/setup/js/trace_graders.cjs index 2348670bff8..4e9c0462249 100644 --- a/actions/setup/js/trace_graders.cjs +++ b/actions/setup/js/trace_graders.cjs @@ -8,12 +8,14 @@ const crypto = require("crypto"); const { getErrorMessage } = require("./error_helpers.cjs"); const { readExperimentAssignments } = require("./experiment_helpers.cjs"); const { calculateWorkingSetFromEntries } = require("./working_set_metrics.cjs"); +const { executeOperationalValueEvaluator } = require("./operational_value_grader.cjs"); // --- Constants --- const TMP_GH_AW = "/tmp/gh-aw"; const GRADERS_DIR = path.join(TMP_GH_AW, "agent", "graders"); const MANIFEST_PATH = path.join(GRADERS_DIR, "grader_manifest.json"); const RESULTS_PATH = path.join(GRADERS_DIR, "grader_results.json"); +const OPERATIONAL_VALUE_EVALUATOR_PATH = path.join(GRADERS_DIR, "operational_value_evaluator.sh"); // Trace source file paths const TOKEN_USAGE_PATHS = [ @@ -442,7 +444,11 @@ function evaluateThreshold(value, direction, threshold) { * @property {string} [details] * @property {string} [message] * @property {string} [error] - * @property {string} source - "builtin" | "inline" + * @property {string} source - "builtin" | "inline" | "value" + * @property {object} [observation] + * @property {object} [diagnostics] + * @property {number|null} [baselineValue] + * @property {number|null} [deltaFromBaseline] * @property {{id: string, version: number, digest?: string}} implementation */ @@ -493,10 +499,20 @@ function normalizeResult(id, rawResult, meta) { if (rawResult.details) base.details = String(rawResult.details); if (rawResult.message) base.message = String(rawResult.message); if (typeof rawResult.passed === "boolean") base.passed = rawResult.passed; + if (isRecord(rawResult.observation)) base.observation = deepClone(rawResult.observation); + if (isRecord(rawResult.diagnostics)) base.diagnostics = deepClone(rawResult.diagnostics); + if (typeof rawResult.baselineValue === "number" || rawResult.baselineValue === null) base.baselineValue = rawResult.baselineValue; + if (typeof rawResult.deltaFromBaseline === "number" || rawResult.deltaFromBaseline === null) base.deltaFromBaseline = rawResult.deltaFromBaseline; } else { value = rawResult; } + if (value === null || value === undefined) { + base.status = "unavailable"; + base.message ||= "grader returned no value"; + return base; + } + if (typeof value !== "number" || !isFinite(value)) { base.status = "error"; base.error = `grader ${id} returned non-finite value: ${value}`; @@ -612,6 +628,27 @@ function runCustomGrader(id, script, trace, meta) { } } +function runOperationalValueGrader(id, evaluatorContent, meta, options) { + try { + const rawResult = executeOperationalValueEvaluator(evaluatorContent, meta, options); + return normalizeResult(id, rawResult, meta); + } catch (err) { + const result = normalizeResult(id, null, meta); + result.status = "error"; + result.error = `grader ${id} runtime error: ${getErrorMessage(err)}`; + return result; + } +} + +function archiveOperationalValueEvaluator(evaluatorContent, expectedDigest, outputPath = OPERATIONAL_VALUE_EVALUATOR_PATH) { + const actualDigest = crypto.createHash("sha256").update(evaluatorContent, "utf8").digest("hex"); + if (!expectedDigest || actualDigest !== expectedDigest) { + throw new Error(`operational-value evaluator digest mismatch: expected ${expectedDigest || "none"}, got ${actualDigest}`); + } + fs.mkdirSync(path.dirname(outputPath), { recursive: true }); + fs.writeFileSync(outputPath, evaluatorContent, { encoding: "utf8", mode: 0o600 }); +} + /** * Legacy adapter for existing tests. Runs a grader by id. * @param {string} id @@ -638,7 +675,7 @@ function runGrader(id, builtin, script, trace, config) { /** * Main entry point. Called from the github-script step with base64 manifest and exec spec. * @param {string} manifestB64 - Base64-encoded JSON manifest - * @param {string} [execSpecB64] - Base64-encoded JSON array of {id, script} + * @param {string} [execSpecB64] - Base64-encoded JSON array of {id, script|run} */ async function main(manifestB64, execSpecB64) { /** @type {{version: number, graders: any[]}} */ @@ -651,15 +688,15 @@ async function main(manifestB64, execSpecB64) { return; } - // Decode execution spec (custom scripts) - /** @type {Record} */ - const scriptMap = {}; + // Decode trusted executable payloads for custom graders. + /** @type {Record} */ + const executionMap = {}; if (execSpecB64) { try { const specJson = Buffer.from(execSpecB64, "base64").toString("utf-8"); const specs = JSON.parse(specJson); for (const s of specs) { - if (s.id && s.script) scriptMap[s.id] = s.script; + if (s.id && (s.script || s.run)) executionMap[s.id] = { script: s.script, run: s.run }; } } catch (err) { core.warning(`Graders: failed to parse exec spec: ${getErrorMessage(err)}`); @@ -682,9 +719,24 @@ async function main(manifestB64, execSpecB64) { return; } + let operationalValueEvaluatorArchiveError; + const operationalValueManifest = enabledGraders.find(grader => grader.source === "operational-value"); + if (operationalValueManifest) { + try { + const evaluatorContent = executionMap[operationalValueManifest.id]?.run; + if (!evaluatorContent) throw new Error("operational-value evaluator is missing from the execution specification"); + archiveOperationalValueEvaluator(evaluatorContent, operationalValueManifest.digest); + } catch (err) { + operationalValueEvaluatorArchiveError = getErrorMessage(err); + core.warning(`Graders: unable to archive operational-value evaluator: ${operationalValueEvaluatorArchiveError}`); + } + } + // Single preprocessing pass core.info(`Graders: preprocessing trace files for ${enabledGraders.length} grader(s)...`); const trace = preprocessTrace(); + const runCreatedAt = process.env.GH_AW_RUN_CREATED_AT; + const operationalValueRunMetadata = runCreatedAt ? { createdAt: runCreatedAt } : undefined; // Run all graders /** @type {GraderResult[]} */ @@ -704,8 +756,14 @@ async function main(manifestB64, execSpecB64) { let result; if (grader.source === "builtin" && BUILTIN_GRADERS[grader.id]) { result = runBuiltinGrader(grader.id, trace, meta); - } else if (scriptMap[grader.id]) { - result = runCustomGrader(grader.id, scriptMap[grader.id], trace, meta); + } else if (grader.source === "operational-value" && operationalValueEvaluatorArchiveError) { + result = normalizeResult(grader.id, null, meta); + result.status = "error"; + result.error = `grader ${grader.id} runtime error: ${operationalValueEvaluatorArchiveError}`; + } else if (grader.source === "operational-value" && executionMap[grader.id]?.run) { + result = runOperationalValueGrader(grader.id, executionMap[grader.id].run, meta, { runMetadata: operationalValueRunMetadata }); + } else if (executionMap[grader.id]?.script) { + result = runCustomGrader(grader.id, executionMap[grader.id].script, trace, meta); } else { result = normalizeResult(grader.id, null, meta); result.status = "unavailable"; @@ -725,6 +783,8 @@ async function main(manifestB64, execSpecB64) { const output = { version: GRADER_VERSION, run: { + id: String(process.env.GITHUB_RUN_ID || ""), + attempt: Number(process.env.GITHUB_RUN_ATTEMPT) || 1, graderCount: results.length, passed, failed, @@ -788,6 +848,7 @@ module.exports = { runGrader, runBuiltinGrader, runCustomGrader, + runOperationalValueGrader, normalizeResult, evaluateThreshold, BUILTIN_GRADERS, @@ -797,6 +858,8 @@ module.exports = { GRADERS_DIR, MANIFEST_PATH, RESULTS_PATH, + OPERATIONAL_VALUE_EVALUATOR_PATH, + archiveOperationalValueEvaluator, MAX_FILE_SIZE, MAX_LINE_LENGTH, SCRIPT_TIMEOUT_MS, diff --git a/actions/setup/js/trace_graders.test.cjs b/actions/setup/js/trace_graders.test.cjs index 062bac1fc30..7b4aa60e201 100644 --- a/actions/setup/js/trace_graders.test.cjs +++ b/actions/setup/js/trace_graders.test.cjs @@ -4,6 +4,7 @@ const fs = require("fs"); const path = require("path"); const os = require("os"); +const crypto = require("crypto"); const { main, @@ -26,6 +27,7 @@ const { IMPLEMENTATION_ID, MANIFEST_PATH, RESULTS_PATH, + archiveOperationalValueEvaluator, MAX_FILE_SIZE, MAX_LINE_LENGTH, SCRIPT_TIMEOUT_MS, @@ -66,6 +68,22 @@ function makeTrace(overrides = {}) { } describe("trace_graders", () => { + describe("archiveOperationalValueEvaluator", () => { + it("writes only evaluator bytes matching the frozen digest", () => { + const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "operational-value-evaluator-archive-")); + const outputPath = path.join(tempDir, "operational_value_evaluator.sh"); + const content = "#!/usr/bin/env bash\nprintf 'ok\\n'\n"; + const digest = crypto.createHash("sha256").update(content, "utf8").digest("hex"); + try { + archiveOperationalValueEvaluator(content, digest, outputPath); + expect(fs.readFileSync(outputPath, "utf8")).toBe(content); + expect(() => archiveOperationalValueEvaluator(content, "invalid", outputPath)).toThrow("digest mismatch"); + } finally { + fs.rmSync(tempDir, { recursive: true, force: true }); + } + }); + }); + // --- safeParseJsonl --- describe("safeParseJsonl", () => { it("parses valid JSONL", () => { diff --git a/cmd/gh-aw/main.go b/cmd/gh-aw/main.go index 63a591399f7..9ae1ed08df3 100644 --- a/cmd/gh-aw/main.go +++ b/cmd/gh-aw/main.go @@ -556,7 +556,7 @@ type commandSet struct { addCmd, addWizardCmd, updateCmd, deployCmd, trialCmd, initCmd, statusCmd, listCmd *cobra.Command mcpCmd, logsCmd, auditCmd, viewCmd, healthCmd, outcomesCmd, mcpServerCmd, prCmd, secretsCmd *cobra.Command fixCmd, upgradeCmd, completionCmd, hashCmd, projectCmd, doctorCmd, checksCmd, validateCmd, lintCmd *cobra.Command - domainsCmd, experimentsCmd, forecastCmd, envCmd, modelsCmd *cobra.Command + domainsCmd, experimentsCmd, forecastCmd, gradersCmd, modelsCmd, envCmd *cobra.Command } func fixPathForCommand(s string) string { @@ -735,6 +735,7 @@ func createCommandSet() commandSet { domainsCmd: cli.NewDomainsCommand(), experimentsCmd: cli.NewExperimentsCommand(), forecastCmd: cli.NewForecastCommand(), + gradersCmd: cli.NewGradersCommand(), envCmd: cli.NewEnvCommand(), modelsCmd: cli.NewModelsCommand(), } @@ -852,6 +853,7 @@ func assignCommandGroups(cmds commandSet) { cmds.logsCmd.GroupID, cmds.auditCmd.GroupID, cmds.viewCmd.GroupID = "analysis", "analysis", "analysis" cmds.healthCmd.GroupID, cmds.outcomesCmd.GroupID, cmds.checksCmd.GroupID = "analysis", "analysis", "analysis" cmds.statusCmd.GroupID, cmds.listCmd.GroupID, cmds.experimentsCmd.GroupID, cmds.forecastCmd.GroupID, cmds.modelsCmd.GroupID = "analysis", "analysis", "analysis", "analysis", "analysis" + cmds.gradersCmd.GroupID = "analysis" cmds.mcpServerCmd.GroupID, cmds.prCmd.GroupID, cmds.completionCmd.GroupID, cmds.hashCmd.GroupID, cmds.projectCmd.GroupID = "utilities", "utilities", "utilities", "utilities", "utilities" } @@ -861,7 +863,7 @@ func addCommandsToRoot(cmds commandSet) { runCmd, removeCmd, cmds.statusCmd, cmds.listCmd, enableCmd, disableCmd, cmds.logsCmd, cmds.auditCmd, cmds.viewCmd, cmds.healthCmd, cmds.outcomesCmd, cmds.checksCmd, cmds.mcpCmd, cmds.mcpServerCmd, cmds.prCmd, versionCmd, cmds.secretsCmd, cmds.fixCmd, cmds.validateCmd, cmds.lintCmd, cmds.completionCmd, cmds.hashCmd, cmds.projectCmd, cmds.doctorCmd, - cmds.domainsCmd, cmds.experimentsCmd, cmds.forecastCmd, cmds.modelsCmd, cmds.envCmd, + cmds.domainsCmd, cmds.experimentsCmd, cmds.forecastCmd, cmds.gradersCmd, cmds.modelsCmd, cmds.envCmd, ) } diff --git a/docs/adr/55155-operational-value-grader.md b/docs/adr/55155-operational-value-grader.md new file mode 100644 index 00000000000..61a2618efc9 --- /dev/null +++ b/docs/adr/55155-operational-value-grader.md @@ -0,0 +1,52 @@ +# ADR-55155: Introduce a Dedicated Operational-Value Grader Type + +**Date**: 2026-08-24 +**Status**: Draft +**Deciders**: gh-aw maintainers + +--- + +### Context + +The existing built-in and inline graders measure execution quality from a run's trace, but operational value depends on repository evidence, maturation windows, and a stable baseline contract. That evidence must remain attributable to the workflow run and evaluator version while supporting deterministic replay after the original run artifact is sealed. Allowing arbitrary evaluator locations or mutable evaluator code would weaken provenance and make historical results difficult to reproduce. Authoritative workflow-run creation time is useful for assigning time-based opportunities, but fetching it must not broaden the token available to the agent. + +### Decision + +We will reserve the `operational-value` grader ID for a repository-relative Bash evaluator under `.github/graders/*.sh`. At compile time, gh-aw will reject traversal, symlinks, and non-regular files, validate Bash syntax, and freeze the evaluator bytes and SHA-256 digest into the compiled workflow. At runtime, gh-aw will execute deterministic `--definition` and `--grade-run` modes in the gh-aw temporary area with a curated environment, producing absolute attainment in `[0,1]` or `null` plus evidence provenance, maturity, a frozen baseline, and an optional derived delta. The activation job will fetch authoritative workflow-run creation time with its compiler-owned `actions: read` permission and pass it to the agent job as non-secret metadata. Enabling the grader will not modify the agent job's permissions; evaluator evidence access must be declared explicitly by the workflow. + +The unified agent artifact will contain `grader_manifest.json`, `grader_results.json`, and the frozen evaluator for replay. Historical regrading will execute only after verifying the archived bytes against the manifest/result digests and the evaluator at the recorded commit in a trusted checkout, while preserving the original run identity, attempt, subject, and operational case. + +### Alternatives Considered + +#### Alternative 1: Use Ordinary Inline Execution-Quality Graders + +Represent operational value as another inline JavaScript grader over the preprocessed execution trace. This would reuse the existing isolated worker and artifact schema, but traces describe how the agent executed rather than whether repository-level outcomes were attained. Inline graders also lack the evidence cutoff, maturity, baseline, provenance, and trusted-checkout replay contract required for operational value. + +#### Alternative 2: Embed Evaluator Logic in Workflow Frontmatter or JavaScript + +Place the complete evaluator directly in frontmatter or implement it as gh-aw-owned JavaScript. Frontmatter would make non-trivial evidence contracts difficult to review and maintain, while a built-in JavaScript implementation would couple repository-specific value definitions to gh-aw releases. A repository file keeps the evaluator versioned with the workflow while allowing the compiler to freeze and validate the exact bytes used. + +#### Alternative 3: Compute Operational Value Only Asynchronously + +Run value evaluation later in an external service or periodic process, outside the original run artifacts. This could wait naturally for mature evidence and avoid adding Bash execution to the agent job, but it would separate observations from their original run identity and frozen evaluator unless a parallel provenance system were built. It would also make immediate attainment unavailable and weaken self-contained replay from the run artifact. + +### Consequences + +#### Positive +- Each observation is tied to a run, operational case, evidence cutoff, provenance, and evaluator digest, enabling reproducible historical regrading. +- The manifest, results, and frozen evaluator form a self-contained artifact set for audit and replay without mutating the original run. +- Baseline-comparable and attainment-only definitions share one normalized result contract while preserving `null` for unavailable or immature evidence. + +#### Negative +- The activation job requires `actions: read` to obtain authoritative run creation time; a failed lookup leaves that optional subject field empty. +- The feature adds Bash availability, syntax validation, process timeout, output validation, curated-environment, and temporary-file complexity to compiler and runtime maintenance. +- Run artifacts contain trusted executable bytes; consumers must continue verifying digests and checkout provenance before executing them. + +#### Neutral +- Operational value is absolute attainment, not proof that the workflow caused the observed outcome; the optional baseline delta remains descriptive rather than causal. +- Evaluators may access only the curated runtime environment, including the workflow token and GitHub host variables, and execute from the gh-aw temporary area rather than inheriting the full workflow environment. +- Historical regrading creates a new observation keyed by run ID, evaluator digest, and evidence time while preserving the original run artifact and case. + +--- + +*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.* diff --git a/docs/src/content/docs/reference/trace-graders.md b/docs/src/content/docs/reference/trace-graders.md index 51bd527135f..bd52770cf07 100644 --- a/docs/src/content/docs/reference/trace-graders.md +++ b/docs/src/content/docs/reference/trace-graders.md @@ -1,9 +1,9 @@ --- title: Graders -description: Deterministic metrics computed from agent execution traces +description: Deterministic execution and operational value metrics --- -Graders compute deterministic metrics from post-agent execution trace files (token usage, MCP gateway logs, agent output) without LLM calls or network access. Results are persisted in the agent artifact for downstream consumption by detection jobs and reporting tools. +Graders compute deterministic metrics without LLM calls. Built-in and custom inline graders inspect post-agent execution traces. The reserved `value` grader evaluates operational repository outcomes under a frozen function and explicit evidence cutoff. Results are persisted in the agent artifact for downstream tools. For normative requirements, see the [Graders Specification](/gh-aw/specs/graders-specification/). @@ -56,14 +56,41 @@ graders: Custom scripts must return a value and stay within 4096 characters (no `require`, `import`, `fetch`, `eval`, or `process.exit`). +## Operational value grader + +Configure the reserved `operational-value` grader with a repository-relative Bash evaluator: + +```aw wrap +graders: + operational-value: + run: .github/graders/daily-file-diet-operational-value.sh +``` + +The compiler freezes the evaluator bytes and records their SHA-256 digest. The evaluator returns absolute operational attainment in `[0,1]` for the run's assigned case. A frozen baseline is optional metadata; when present, gh-aw derives `deltaFromBaseline` without changing the primary value. + +Each result records the complete run subject, operational case, evidence time, maturity, and provenance. Operational-value evaluators may query the repositories declared by their frozen evidence contract. They receive the workflow token through `GH_TOKEN` with the agent job's explicitly declared permissions, but do not receive workflow secrets. Enabling the grader does not add evidence permissions to the agent job. + +Use the `aw-value` skill to design and verify an operational-value evaluator. + +### Regrade a historical run + +```bash +gh aw graders operational-value 123456789 \ + --evidence-at 2026-08-30T12:00:00.000Z \ + --json +``` + +The command downloads the original grader artifact and reuses its case, run subject, and frozen evaluator. The archived evaluator must match the digest recorded by both the original manifest and result and the evaluator at the recorded commit in the current repository checkout. Regrading emits a new observation identified by `(runId, evaluatorDigest, evidenceAt)` and never modifies the original artifact. Use `--repo [HOST/]OWNER/REPO` to select the host for the checked-out repository. + ## Output files | File | Description | |---|---| | `grader_manifest.json` | Which graders were configured and their enabled state | -| `grader_results.json` | Normalized metric values with trace summary | +| `grader_results.json` | Normalized values, status, implementation identity, and value observations | +| `operational_value_evaluator.sh` | Exact frozen operational-value evaluator used for initial grading and historical replay | -Both files are included in the unified `agent` artifact. +All files are included in the unified `agent` artifact. ## Execution diff --git a/docs/src/content/docs/specs/graders-specification.md b/docs/src/content/docs/specs/graders-specification.md index e0cd469ea62..a7821b7c56c 100644 --- a/docs/src/content/docs/specs/graders-specification.md +++ b/docs/src/content/docs/specs/graders-specification.md @@ -7,17 +7,17 @@ sidebar: # Graders Specification -**Version**: 0.1.0 -**Status**: Draft Specification -**Feature Status**: Experimental -**Latest Version**: [graders-specification](/gh-aw/specs/graders-specification/) +**Version**: 0.2.0 +**Status**: Draft Specification +**Feature Status**: Experimental +**Latest Version**: [graders-specification](/gh-aw/specs/graders-specification/) **Editor**: GitHub Agentic Workflows Team --- ## Abstract -This specification defines the `graders` feature in gh-aw: deterministic, post-agent metrics computed from execution traces and persisted as structured artifacts. It specifies configuration, built-in grader behavior, custom inline grader constraints, execution ordering, artifact outputs, experiment metric references, and conformance requirements. +This specification defines the `graders` feature in gh-aw: deterministic execution metrics and operational value observations persisted as structured artifacts. It specifies configuration, built-in grader behavior, custom inline grader constraints, operational-value grader behavior, execution ordering, artifact outputs, historical regrading, experiment metric references, and conformance requirements. ## Status of This Document @@ -27,19 +27,20 @@ This feature is experimental and implementations SHOULD expect iteration before ## Table of Contents -1. [Introduction](#1-introduction) -2. [Conformance](#2-conformance) -3. [Architecture](#3-architecture) -4. [Configuration Model](#4-configuration-model) -5. [Built-in Graders](#5-built-in-graders) -6. [Custom Inline Graders](#6-custom-inline-graders) -7. [Execution and Artifacts](#7-execution-and-artifacts) -8. [Experiment Metric References](#8-experiment-metric-references) -9. [Security and Isolation](#9-security-and-isolation) -10. [Compliance Testing](#10-compliance-testing) -11. [Norms](#11-norms) -12. [References](#12-references) -13. [Change Log](#13-change-log) +1. [Introduction](#1-introduction) +2. [Conformance](#2-conformance) +3. [Architecture](#3-architecture) +4. [Configuration Model](#4-configuration-model) +5. [Built-in Graders](#5-built-in-graders) +6. [Custom Inline Graders](#6-custom-inline-graders) +7. [Operational Value Grader](#7-operational-value-grader) +8. [Execution and Artifacts](#8-execution-and-artifacts) +9. [Experiment Metric References](#9-experiment-metric-references) +10. [Security and Isolation](#10-security-and-isolation) +11. [Compliance Testing](#11-compliance-testing) +12. [Norms](#12-norms) +13. [References](#13-references) +14. [Change Log](#14-change-log) --- @@ -47,7 +48,7 @@ This feature is experimental and implementations SHOULD expect iteration before ### 1.1 Purpose -The `graders` feature provides deterministic quality and behavior metrics derived from workflow trace artifacts without issuing additional LLM calls. +The `graders` feature provides deterministic execution metrics and operational value observations without issuing additional LLM calls. ### 1.2 Scope @@ -56,6 +57,7 @@ This specification covers: - Frontmatter configuration under `graders` - Built-in grader identifiers and semantics - Custom inline grader script requirements +- Operational-value evaluator and replay requirements - Output artifact contracts - Experiment metric integration for grader references @@ -69,9 +71,9 @@ This specification does NOT cover: A conforming implementation: -1. MUST compute grader values deterministically from run artifacts. +1. MUST compute grader values deterministically for the same inputs and evidence cutoff. 2. MUST preserve stable grader IDs for experiment references. -3. SHOULD keep grading isolated from network-dependent behavior. +3. MUST keep trace grading isolated from network-dependent behavior. 4. MUST emit machine-readable grader artifacts for downstream tooling. --- @@ -102,7 +104,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S 1. Parse and validate frontmatter `graders`. 2. Build grader manifest and execution spec. 3. Preprocess trace artifacts once. -4. Execute enabled graders (built-in and custom inline). +4. Execute enabled graders (built-in, custom inline, and operational value). 5. Write normalized outputs to grader artifact files. The grading step MUST run with `if: always()` semantics and SHOULD continue even when individual graders fail, recording per-grader errors in results. @@ -127,8 +129,9 @@ The configuration key MUST be `graders`. - Built-in grader entries MAY be `null` to enable defaults. - Custom grader entries MUST be objects and MUST include `script`. +- The reserved `value` entry MUST be an object and MUST include `function`. -Supported object fields include `enabled`, `name`, `description`, `unit`, `direction`, `threshold`, `min`, `max`, `config`, and `script`. +Supported object fields include `enabled`, `name`, `description`, `unit`, `direction`, `threshold`, `min`, `max`, `config`, `script`, and `function`. Only the reserved `value` grader accepts `function`. --- @@ -181,32 +184,54 @@ Inline scripts MUST be rejected if they contain any forbidden pattern, including --- -## 7. Execution and Artifacts +## 7. Operational Value Grader -### 7.1 Output Directory +The reserved grader ID MUST be `operational-value`. It MUST NOT accept an inline `script`. + +The compiler MUST resolve `run` within the repository, reject symlinks and non-regular files, validate Bash syntax prerequisites, freeze the evaluator bytes, and record their SHA-256 digest in the grader manifest and result implementation. + +The evaluator MUST implement `--definition` and `--grade-run`. Its primary `value` MUST be absolute operational attainment in `[0,1]` or `null`. A baseline MAY be frozen separately; gh-aw MUST derive `deltaFromBaseline` and MUST NOT replace the primary value with that delta. + +An operational-value observation MUST include: + +- the complete workflow run subject and run attempt; +- a stable opportunity key and replayable operational case; +- requested evidence time, effective evidence cutoff, and maturity time; +- accepted evidence provenance for every numeric value. + +The effective evidence cutoff MUST NOT follow either the requested evidence time or the maturity time. A replayed observation MUST be identified by `(runId, evaluatorDigest, evidenceAt)`. + +Historical regrading MUST reuse the original case, run subject, and archived evaluator. It MUST verify that the archived evaluator matches the digest recorded by both the original manifest and result and the evaluator at the recorded commit in a trusted local checkout before execution. It MUST emit a new observation and MUST NOT mutate the original run artifact. + +--- + +## 8. Execution and Artifacts + +### 8.1 Output Directory Graders output MUST be written under: `/tmp/gh-aw/agent/graders` -### 7.2 Required Files +### 8.2 Required Files The implementation MUST produce: - `grader_manifest.json` - `grader_results.json` +- `operational_value_evaluator.sh` when the `operational-value` grader is enabled -### 7.3 Artifact Inclusion +### 8.3 Artifact Inclusion -Both files MUST be included in the unified `agent` artifact. +All applicable files MUST be included in the unified `agent` artifact. -### 7.4 Deterministic Output Contract +### 8.4 Deterministic Output Contract `grader_results.json` SHOULD include normalized run/result structures suitable for downstream programmatic reads, including per-grader value/status and run-level pass/fail/error counts. --- -## 8. Experiment Metric References +## 9. Experiment Metric References Experiment metric fields MAY reference grader outputs. @@ -241,18 +266,20 @@ semantic task correctness. The normative readiness, decision, and JSON contracts --- -## 9. Security and Isolation +## 10. Security and Isolation - Grading MUST operate on local run artifacts and MUST NOT require outbound network access for built-ins. - Custom inline graders MUST execute in a restricted context with blocked dangerous primitives. +- Operational-value graders MAY access declared repository evidence using `GH_TOKEN`; implementations MUST NOT add agent-job permission scopes on behalf of the evaluator, and evaluators MUST NOT receive workflow secrets. +- Historical regrading MUST verify archived evaluator bytes against both digest records and a trusted local checkout at the recorded commit before execution. - Implementations SHOULD enforce bounded execution time for inline scripts. - Implementations SHOULD redact grader outputs when custom scripts are enabled to reduce secret leakage risk. --- -## 10. Compliance Testing +## 11. Compliance Testing -### 10.1 Test Suite Requirements +### 11.1 Test Suite Requirements - **T-GRD-001**: Omitted `graders` key disables grading step emission. - **T-GRD-002**: `graders: {}` enables all built-ins. @@ -265,8 +292,11 @@ semantic task correctness. The normative readiness, decision, and JSON contracts - **T-GRD-009**: Grader files are present in `agent` artifact. - **T-GRD-010**: `experiments.*.metric` with `grader:` validates declared enabled grader. - **T-GRD-011**: `experiments.*.metric` with `graders..value` validates declared enabled grader. +- **T-GRD-012**: `graders.operational-value.run` is frozen and its digest is recorded. +- **T-GRD-013**: Operational-value output, evidence cutoff, maturity, and provenance are validated. +- **T-GRD-014**: Historical regrading rejects evaluator or run identity mismatches. -### 10.2 Compliance Checklist +### 11.2 Compliance Checklist | Requirement | Test ID | Level | Status | |---|---|---|---| @@ -276,20 +306,23 @@ semantic task correctness. The normative readiness, decision, and JSON contracts | Script safety constraints enforced | T-GRD-004, T-GRD-005 | 2 | Required | | Required artifact files emitted | T-GRD-007, T-GRD-008 | 1 | Required | | Experiment grader references validate | T-GRD-010, T-GRD-011 | 3 | Required | +| Operational-value evaluators and observations validate | T-GRD-012, T-GRD-013 | 2 | Required | +| Historical regrading preserves identity | T-GRD-014 | 2 | Required | --- -## 11. Norms +## 12. Norms - **N-GRD-001**: Implementations MUST treat `graders` as experimental. - **N-GRD-002**: Implementations MUST preserve built-in grader ID stability across patch releases. - **N-GRD-003**: Implementations SHOULD preserve deterministic output for identical trace inputs. - **N-GRD-004**: Implementations MUST fail fast on invalid custom grader scripts. - **N-GRD-005**: Implementations MUST keep grader artifact paths stable unless a major version change is issued. +- **N-GRD-006**: Implementations MUST keep operational value separate from execution quality metrics. --- -## 12. References +## 13. References ### Normative References @@ -303,7 +336,12 @@ semantic task correctness. The normative readiness, decision, and JSON contracts --- -## 13. Change Log +## 14. Change Log + +### Version 0.2.0 (Draft Specification) + +- Defines the operational value grader and absolute attainment semantics. +- Defines evidence-bounded historical regrading and observation identity. ### Version 0.1.0 (Draft Specification) diff --git a/pkg/cli/experiments_grader_observations.go b/pkg/cli/experiments_grader_observations.go index 6288c1a975a..11e9f939dec 100644 --- a/pkg/cli/experiments_grader_observations.go +++ b/pkg/cli/experiments_grader_observations.go @@ -62,13 +62,50 @@ type graderMetricObservationSet struct { type graderResultsArtifact struct { Version int `json:"version"` + Run graderArtifactRun `json:"run"` Results []graderArtifactResult `json:"results"` } +type graderArtifactRun struct { + ID string `json:"id"` + Attempt int `json:"attempt"` +} + type graderArtifactResult struct { - ID string `json:"id"` - Status string `json:"status"` - Value json.RawMessage `json:"value"` + ID string `json:"id"` + Status string `json:"status"` + Value json.RawMessage `json:"value"` + Observation *graderArtifactObservation `json:"observation,omitempty"` + Implementation graderArtifactImplementation `json:"implementation"` +} + +type graderArtifactObservation struct { + Subject graderArtifactSubject `json:"subject"` + OpportunityKey string `json:"opportunityKey"` + EvidenceAt string `json:"evidenceAt"` + EvidenceCutoff string `json:"evidenceCutoff"` + MaturesAt string `json:"maturesAt"` + Mature bool `json:"mature"` + Case map[string]any `json:"case"` + Provenance []map[string]any `json:"provenance"` +} + +type graderArtifactSubject struct { + Type string `json:"type"` + RunID string `json:"runId"` + Attempt int `json:"attempt"` + Repository string `json:"repository"` + Workflow string `json:"workflow"` + Ref string `json:"ref"` + SHA string `json:"sha"` + EventName string `json:"eventName"` + CreatedAt *string `json:"createdAt"` +} + +type graderArtifactImplementation struct { + ID string `json:"id"` + Version int `json:"version"` + Digest string `json:"digest,omitempty"` } type graderRunData struct { diff --git a/pkg/cli/graders_command.go b/pkg/cli/graders_command.go new file mode 100644 index 00000000000..f5d02c540a0 --- /dev/null +++ b/pkg/cli/graders_command.go @@ -0,0 +1,52 @@ +package cli + +import ( + "errors" + "strconv" + + "github.com/github/gh-aw/pkg/constants" + "github.com/spf13/cobra" +) + +// NewGradersCommand creates commands for inspecting and replaying workflow graders. +func NewGradersCommand() *cobra.Command { + cmd := &cobra.Command{ + Use: "graders", + Short: "Inspect and replay workflow graders", + } + cmd.AddCommand(newGradersOperationalValueCommand()) + return cmd +} + +func newGradersOperationalValueCommand() *cobra.Command { + cmd := &cobra.Command{ + Use: "operational-value ", + Short: "Regrade a workflow run's operational value", + Long: `Regrade the operational-value observation from a completed workflow run at an explicit +evidence cutoff. The command verifies and executes the exact evaluator archived +by the run. The original artifact is not modified.`, + Example: ` ` + string(constants.CLIExtensionPrefix) + ` graders operational-value 123456789 \ + --evidence-at 2026-08-30T12:00:00.000Z --json`, + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + runID, err := strconv.ParseInt(args[0], 10, 64) + if err != nil || runID <= 0 { + return errors.New("run ID must be a positive integer") + } + evidenceAt, _ := cmd.Flags().GetString("evidence-at") + repoOverride, _ := cmd.Flags().GetString("repo") + jsonOutput, _ := cmd.Flags().GetBool("json") + return RunOperationalValueRegrade(cmd.Context(), OperationalValueRegradeConfig{ + RunID: runID, + EvidenceAt: evidenceAt, + RepoOverride: repoOverride, + JSONOutput: jsonOutput, + }) + }, + } + cmd.Flags().String("evidence-at", "", "UTC evidence cutoff for this observation") + _ = cmd.MarkFlagRequired("evidence-at") + addRepoFlag(cmd) + addJSONFlag(cmd) + return cmd +} diff --git a/pkg/cli/graders_operational_value_regrade.go b/pkg/cli/graders_operational_value_regrade.go new file mode 100644 index 00000000000..b85254b29f6 --- /dev/null +++ b/pkg/cli/graders_operational_value_regrade.go @@ -0,0 +1,717 @@ +package cli + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "math" + "net/url" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "time" + "unicode/utf8" + + "github.com/github/gh-aw/pkg/console" + "github.com/github/gh-aw/pkg/constants" + "github.com/github/gh-aw/pkg/gitutil" + "github.com/github/gh-aw/pkg/repoutil" + "github.com/github/gh-aw/pkg/stringutil" +) + +const ( + maxOperationalValueRegradeEvaluatorBytes = 64 * 1024 + maxOperationalValueRegradeOutputBytes = 1024 * 1024 + operationalValueDefinitionTimeout = 5 * time.Second + operationalValueEvaluatorTimeout = 2 * time.Minute +) + +// OperationalValueRegradeConfig configures historical operational-value regrading. +type OperationalValueRegradeConfig struct { + RunID int64 + EvidenceAt string + RepoOverride string + JSONOutput bool +} + +type operationalValueGraderManifest struct { + Version int `json:"version"` + Graders []operationalValueGraderManifestEntry `json:"graders"` +} + +type operationalValueGraderManifestEntry struct { + ID string `json:"id"` + Name string `json:"name"` + Source string `json:"source"` + Enabled bool `json:"enabled"` + Unit string `json:"unit,omitempty"` + Direction string `json:"direction,omitempty"` + Threshold *float64 `json:"threshold,omitempty"` + Digest string `json:"digest"` + Run string `json:"run"` + Config map[string]any `json:"config,omitempty"` +} + +type operationalValueRunSubject struct { + ID string `json:"id"` + Attempt int `json:"attempt"` + Repository string `json:"repository"` + Workflow string `json:"workflow"` + Ref string `json:"ref"` + SHA string `json:"sha"` + EventName string `json:"eventName"` + CreatedAt *string `json:"createdAt"` +} + +type operationalValueRunRequest struct { + SchemaVersion int `json:"schemaVersion"` + Run operationalValueRunSubject `json:"run"` + EvidenceAt string `json:"evidenceAt"` + Case map[string]any `json:"case"` + Event any `json:"event"` + Config map[string]any `json:"config"` +} + +type operationalValueRegradeObservation struct { + Subject graderArtifactSubject `json:"subject"` + OpportunityKey string `json:"opportunityKey"` + EvidenceAt string `json:"evidenceAt"` + EvidenceCutoff string `json:"evidenceCutoff"` + MaturesAt string `json:"maturesAt"` + Mature bool `json:"mature"` + Case map[string]any `json:"case"` + Provenance []map[string]any `json:"provenance"` +} + +type operationalValueRegradeResult struct { + ID string `json:"id"` + Name string `json:"name"` + Value *float64 `json:"value"` + Unit string `json:"unit"` + Passed *bool `json:"passed"` + Status string `json:"status"` + Source string `json:"source"` + Message string `json:"message,omitempty"` + Observation operationalValueRegradeObservation `json:"observation"` + Diagnostics map[string]any `json:"diagnostics,omitempty"` + BaselineValue *float64 `json:"baselineValue"` + DeltaFromBaseline *float64 `json:"deltaFromBaseline"` + Implementation graderArtifactImplementation `json:"implementation"` +} + +type operationalValueRegradeMetadata struct { + Identity operationalValueRegradeIdentity `json:"identity"` + OriginalEvidenceAt string `json:"originalEvidenceAt"` +} + +type operationalValueRegradeIdentity struct { + RunID string `json:"runId"` + EvaluatorDigest string `json:"evaluatorDigest"` + EvidenceAt string `json:"evidenceAt"` +} + +type operationalValueRegradeArtifact struct { + Version int `json:"version"` + Run graderArtifactRun `json:"run"` + Regrade operationalValueRegradeMetadata `json:"regrade"` + Results []operationalValueRegradeResult `json:"results"` +} + +type operationalValueEvaluatorExecution struct { + Value *float64 + Message string + Diagnostics map[string]any + Observation operationalValueRegradeObservation + BaselineValue *float64 + DeltaFromBaseline *float64 +} + +type boundedCommandBuffer struct { + bytes.Buffer + limit int + exceeded bool +} + +func (b *boundedCommandBuffer) Write(data []byte) (int, error) { + written := len(data) + remaining := b.limit - b.Len() + if remaining > 0 { + if len(data) < remaining { + remaining = len(data) + } + _, _ = b.Buffer.Write(data[:remaining]) + } + if written > remaining { + b.exceeded = true + } + return written, nil +} + +// RunOperationalValueRegrade downloads a historical grader observation and recomputes it as of EvidenceAt. +func RunOperationalValueRegrade(ctx context.Context, config OperationalValueRegradeConfig) error { + evidenceAt, err := parseOperationalValueTimestamp(config.EvidenceAt, "evidence-at") + if err != nil { + return err + } + repoSlug, artifactRepo, evaluatorHost, err := resolveOperationalValueRegradeRepo(config.RepoOverride) + if err != nil { + return err + } + + tempDir, err := os.MkdirTemp("", "gh-aw-operational-value-regrade-*") + if err != nil { + return fmt.Errorf("failed to create operational-value regrade directory: %w", err) + } + defer os.RemoveAll(tempDir) + + runIDText := strconv.FormatInt(config.RunID, 10) + source := newGitHubGraderRunArtifactSource(tempDir, artifactRepo) + runData := source.downloadGraderArtifact(ctx, config.RunID, runIDText) + if runData.ExclusionReason != "" { + return fmt.Errorf("cannot regrade run %d: grader artifact %s", config.RunID, runData.ExclusionReason) + } + runDir := filepath.Join(tempDir, runIDText) + evaluatorContent, evaluatorDigest, err := readArchivedOperationalValueEvaluator(runDir) + if err != nil { + return err + } + manifest, err := readOperationalValueGraderManifest(runDir) + if err != nil { + return err + } + manifestEntry, originalResult, err := selectHistoricalOperationalValueGrader(manifest, runData.Artifact, runIDText) + if err != nil { + return err + } + if err := verifyHistoricalOperationalValueIdentity(repoSlug, evaluatorDigest, manifestEntry, originalResult, runData.Artifact.Run, runIDText); err != nil { + return err + } + currentRepoSlug, err := GetCurrentRepoSlug() + if err != nil { + return fmt.Errorf("cannot establish a trusted checkout for operational-value replay: %w", err) + } + gitRoot, err := gitutil.FindGitRoot() + if err != nil { + return fmt.Errorf("cannot establish a trusted checkout for operational-value replay: %w", err) + } + if err := verifyArchivedOperationalValueEvaluatorSource(gitRoot, currentRepoSlug, repoSlug, evaluatorContent, evaluatorDigest, *manifestEntry, originalResult.Observation.Subject); err != nil { + return err + } + + execution, err := executeHistoricalOperationalValueEvaluator(ctx, evaluatorContent, *manifestEntry, *originalResult.Observation, config.EvidenceAt, evidenceAt, evaluatorHost) + if err != nil { + return err + } + artifact := buildOperationalValueRegradeArtifact(runData.Artifact.Run, *manifestEntry, *originalResult, evaluatorDigest, execution) + return renderOperationalValueRegradeArtifact(artifact, config.JSONOutput) +} + +func resolveOperationalValueRegradeRepo(repoOverride string) (repoSlug, artifactRepo, evaluatorHost string, err error) { + if repoOverride == "" { + repoSlug, err = GetCurrentRepoSlug() + return repoSlug, "", getGitHubHostForRepo(repoSlug), err + } + ownerRepo, host := repoutil.NormalizeRepoForAPI(repoOverride) + owner, repo, splitErr := repoutil.SplitRepoSlug(ownerRepo) + if splitErr != nil { + return "", "", "", fmt.Errorf("invalid --repo %q: expected [HOST/]owner/repo", repoOverride) + } + evaluatorHost = getGitHubHostForRepo(ownerRepo) + if host != "" { + evaluatorHost = stringutil.NormalizeGitHubHostURL(host) + } + return strings.Join([]string{owner, repo}, "/"), repoOverride, evaluatorHost, nil +} + +func readArchivedOperationalValueEvaluator(runDir string) (string, string, error) { + evaluatorPath := filepath.Join(runDir, "agent", "graders", constants.OperationalValueEvaluatorFilename) + info, err := os.Lstat(evaluatorPath) + if err != nil { + evaluatorPath = filepath.Join(runDir, "graders", constants.OperationalValueEvaluatorFilename) + info, err = os.Lstat(evaluatorPath) + } + if err != nil { + return "", "", fmt.Errorf("cannot inspect archived operational-value evaluator: %w", err) + } + if info.Mode()&os.ModeSymlink != 0 { + return "", "", errors.New("archived operational-value evaluator must not be a symbolic link") + } + if !info.Mode().IsRegular() { + return "", "", errors.New("archived operational-value evaluator must be a regular file") + } + file, err := os.Open(evaluatorPath) + if err != nil { + return "", "", fmt.Errorf("cannot read archived operational-value evaluator: %w", err) + } + defer file.Close() + content, err := io.ReadAll(io.LimitReader(file, maxOperationalValueRegradeEvaluatorBytes+1)) + if err != nil { + return "", "", fmt.Errorf("cannot read archived operational-value evaluator: %w", err) + } + if len(content) > maxOperationalValueRegradeEvaluatorBytes { + return "", "", fmt.Errorf("archived operational-value evaluator exceeds the %d-byte limit", maxOperationalValueRegradeEvaluatorBytes) + } + if !utf8.Valid(content) { + return "", "", errors.New("archived operational-value evaluator must be valid UTF-8") + } + evaluatorContent := string(content) + if !strings.HasPrefix(evaluatorContent, "#!/usr/bin/env bash\n") && !strings.HasPrefix(evaluatorContent, "#!/bin/bash\n") { + return "", "", errors.New("archived operational-value evaluator must start with a Bash shebang") + } + digest := sha256.Sum256(content) + return evaluatorContent, hex.EncodeToString(digest[:]), nil +} + +func readOperationalValueGraderManifest(runDir string) (*operationalValueGraderManifest, error) { + manifestPath := filepath.Join(runDir, "agent", "graders", constants.GraderManifestFilename) + if _, err := os.Stat(manifestPath); err != nil { + manifestPath = filepath.Join(runDir, "graders", constants.GraderManifestFilename) + } + file, err := os.Open(manifestPath) + if err != nil { + return nil, fmt.Errorf("cannot read grader manifest: %w", err) + } + defer file.Close() + data, err := io.ReadAll(io.LimitReader(file, maxGraderResultsBytes+1)) + if err != nil { + return nil, fmt.Errorf("cannot read grader manifest: %w", err) + } + if len(data) > maxGraderResultsBytes { + return nil, fmt.Errorf("grader manifest exceeds the %d-byte limit", maxGraderResultsBytes) + } + var manifest operationalValueGraderManifest + if err := json.Unmarshal(data, &manifest); err != nil || manifest.Version <= 0 { + return nil, errors.New("grader manifest is malformed") + } + return &manifest, nil +} + +func selectHistoricalOperationalValueGrader(manifest *operationalValueGraderManifest, artifact *graderResultsArtifact, runID string) (*operationalValueGraderManifestEntry, *graderArtifactResult, error) { + if manifest == nil || artifact == nil { + return nil, nil, fmt.Errorf("run %s has no grader data", runID) + } + var manifestEntry *operationalValueGraderManifestEntry + for index := range manifest.Graders { + if manifest.Graders[index].ID != "operational-value" { + continue + } + if manifestEntry != nil { + return nil, nil, fmt.Errorf("run %s grader manifest contains duplicate operational-value graders", runID) + } + manifestEntry = &manifest.Graders[index] + } + var result *graderArtifactResult + for index := range artifact.Results { + if artifact.Results[index].ID != "operational-value" { + continue + } + if result != nil { + return nil, nil, fmt.Errorf("run %s grader artifact contains duplicate operational-value results", runID) + } + result = &artifact.Results[index] + } + if manifestEntry == nil || !manifestEntry.Enabled || manifestEntry.Source != "operational-value" { + return nil, nil, fmt.Errorf("run %s did not use an enabled operational-value grader", runID) + } + if result == nil || result.Observation == nil { + return nil, nil, fmt.Errorf("run %s has no replayable operational-value observation", runID) + } + return manifestEntry, result, nil +} + +func verifyHistoricalOperationalValueIdentity(repoSlug, evaluatorDigest string, manifest *operationalValueGraderManifestEntry, result *graderArtifactResult, run graderArtifactRun, runID string) error { + if run.ID != runID || run.Attempt <= 0 { + return fmt.Errorf("grader artifact run identity does not match run %s", runID) + } + if manifest.Digest == "" || result.Implementation.Digest == "" || manifest.Digest != result.Implementation.Digest { + return fmt.Errorf("run %s has inconsistent operational-value evaluator provenance", runID) + } + if evaluatorDigest != manifest.Digest { + return fmt.Errorf("operational-value evaluator digest mismatch: run %s recorded %s, local evaluator is %s", runID, manifest.Digest, evaluatorDigest) + } + subject := result.Observation.Subject + if subject.Type != "workflow-run" || subject.RunID != runID || subject.Attempt != run.Attempt { + return fmt.Errorf("operational-value observation subject does not match run %s attempt %d", runID, run.Attempt) + } + if subject.Repository == "" || subject.Repository != repoSlug { + return fmt.Errorf("operational-value observation repository %q does not match %q", subject.Repository, repoSlug) + } + if result.Observation.Case == nil { + return fmt.Errorf("run %s operational-value observation has no replayable case", runID) + } + return nil +} + +func verifyArchivedOperationalValueEvaluatorSource(gitRoot, currentRepoSlug, requestedRepoSlug, evaluatorContent, evaluatorDigest string, manifest operationalValueGraderManifestEntry, subject graderArtifactSubject) error { + if !strings.EqualFold(currentRepoSlug, requestedRepoSlug) { + return fmt.Errorf("refusing to execute an operational-value evaluator from %q without a trusted local checkout of that repository", requestedRepoSlug) + } + objectArg, err := buildSafeGitShowObjectArg(subject.SHA, manifest.Run) + if err != nil { + return errors.New("operational-value evaluator provenance contains an unsafe commit or path") + } + cmd := exec.Command("git", "-C", gitRoot, "show", "--no-ext-diff", "--no-textconv", objectArg) + trustedContent, err := cmd.Output() + if err != nil { + return fmt.Errorf("cannot establish operational-value evaluator from trusted commit %s: %w", subject.SHA, err) + } + trustedDigest := sha256.Sum256(trustedContent) + if hex.EncodeToString(trustedDigest[:]) != evaluatorDigest || !bytes.Equal(trustedContent, []byte(evaluatorContent)) { + return fmt.Errorf("archived operational-value evaluator does not match %s at trusted commit %s", manifest.Run, subject.SHA) + } + return nil +} + +func executeHistoricalOperationalValueEvaluator(ctx context.Context, evaluatorContent string, manifest operationalValueGraderManifestEntry, original graderArtifactObservation, evidenceAtText string, evidenceAt time.Time, evaluatorHost string) (*operationalValueEvaluatorExecution, error) { + bashPath := "/bin/bash" + if _, err := os.Stat(bashPath); err != nil { + return nil, fmt.Errorf("bash is required to regrade operational value: %w", err) + } + tempDir, err := os.MkdirTemp("", "gh-aw-operational-value-evaluator-*") + if err != nil { + return nil, fmt.Errorf("failed to create operational-value evaluator directory: %w", err) + } + defer os.RemoveAll(tempDir) + evaluatorPath := filepath.Join(tempDir, "operational-value.sh") + if err := os.WriteFile(evaluatorPath, []byte(evaluatorContent), constants.FilePermExecutable); err != nil { + return nil, fmt.Errorf("failed to stage operational-value evaluator: %w", err) + } + if _, err := runOperationalValueEvaluatorBash(ctx, bashPath, evaluatorPath, []string{"-n", evaluatorPath}, nil, operationalValueDefinitionTimeout, evaluatorHost); err != nil { + return nil, fmt.Errorf("operational-value evaluator has invalid Bash syntax: %w", err) + } + definitionJSON, err := runOperationalValueEvaluatorBash(ctx, bashPath, evaluatorPath, []string{evaluatorPath, "--definition"}, nil, operationalValueDefinitionTimeout, evaluatorHost) + if err != nil { + return nil, fmt.Errorf("operational-value evaluator --definition failed: %w", err) + } + baselineValue, err := parseOperationalValueDefinition(definitionJSON) + if err != nil { + return nil, err + } + evaluatorConfig := manifest.Config + if evaluatorConfig == nil { + evaluatorConfig = map[string]any{} + } + request := operationalValueRunRequest{ + SchemaVersion: 1, + Run: operationalValueRunSubject{ + ID: original.Subject.RunID, + Attempt: original.Subject.Attempt, + Repository: original.Subject.Repository, + Workflow: original.Subject.Workflow, + Ref: original.Subject.Ref, + SHA: original.Subject.SHA, + EventName: original.Subject.EventName, + CreatedAt: original.Subject.CreatedAt, + }, + EvidenceAt: evidenceAtText, + Case: original.Case, + Event: nil, + Config: evaluatorConfig, + } + requestJSON, err := json.Marshal(request) + if err != nil { + return nil, fmt.Errorf("failed to encode operational-value regrade request: %w", err) + } + outputJSON, err := runOperationalValueEvaluatorBash(ctx, bashPath, evaluatorPath, []string{evaluatorPath, "--grade-run"}, requestJSON, operationalValueEvaluatorTimeout, evaluatorHost) + if err != nil { + return nil, fmt.Errorf("operational-value evaluator --grade-run failed: %w", err) + } + return parseOperationalValueEvaluatorOutput(outputJSON, original.Subject, evidenceAtText, evidenceAt, baselineValue) +} + +func runOperationalValueEvaluatorBash(ctx context.Context, bashPath, evaluatorPath string, args []string, input []byte, timeout time.Duration, evaluatorHost string) ([]byte, error) { + commandCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + cmd := exec.CommandContext(commandCtx, bashPath, args...) + cmd.Dir = filepath.Dir(evaluatorPath) + cmd.Env = operationalValueEvaluatorEnvironment(os.Environ(), evaluatorHost) + cmd.Stdin = bytes.NewReader(input) + stdout := &boundedCommandBuffer{limit: maxOperationalValueRegradeOutputBytes} + stderr := &boundedCommandBuffer{limit: maxOperationalValueRegradeOutputBytes} + cmd.Stdout = stdout + cmd.Stderr = stderr + err := cmd.Run() + if errors.Is(commandCtx.Err(), context.DeadlineExceeded) { + return nil, fmt.Errorf("timed out after %s", timeout) + } + if stdout.exceeded || stderr.exceeded { + return nil, fmt.Errorf("output exceeded the %d-byte limit", maxOperationalValueRegradeOutputBytes) + } + if err != nil { + message := strings.TrimSpace(stderr.String()) + if message != "" { + return nil, errors.New(message) + } + return nil, err + } + return stdout.Bytes(), nil +} + +func operationalValueEvaluatorEnvironment(environ []string, evaluatorHost string) []string { + keys := []string{ + "PATH", "HOME", "TMPDIR", "TEMP", "TMP", "SystemRoot", "ComSpec", + "GH_TOKEN", "GH_HOST", "GITHUB_API_URL", "GITHUB_GRAPHQL_URL", "GITHUB_SERVER_URL", + } + values := make(map[string]string, len(environ)) + for _, entry := range environ { + key, value, ok := strings.Cut(entry, "=") + if ok { + values[key] = value + } + } + hostURL, err := url.Parse(evaluatorHost) + if err == nil && hostURL.Scheme != "" && hostURL.Host != "" { + serverURL := strings.TrimSuffix(hostURL.String(), "/") + values["GH_HOST"] = hostURL.Host + values["GITHUB_SERVER_URL"] = serverURL + if strings.EqualFold(hostURL.Hostname(), "github.com") { + values["GITHUB_API_URL"] = "https://api.github.com" + values["GITHUB_GRAPHQL_URL"] = "https://api.github.com/graphql" + } else { + values["GITHUB_API_URL"] = serverURL + "/api/v3" + values["GITHUB_GRAPHQL_URL"] = serverURL + "/api/graphql" + } + } + env := make([]string, 0, len(keys)) + for _, key := range keys { + if value := values[key]; value != "" { + env = append(env, key+"="+value) + } + } + return env +} + +func parseOperationalValueDefinition(data []byte) (*float64, error) { + var definition struct { + SchemaVersion int `json:"schemaVersion"` + Grader string `json:"grader"` + Baseline struct { + Mode string `json:"mode"` + Value json.RawMessage `json:"value"` + } `json:"baseline"` + } + if err := json.Unmarshal(data, &definition); err != nil { + return nil, fmt.Errorf("operational-value evaluator returned an invalid definition: %w", err) + } + if definition.SchemaVersion != 4 || definition.Grader != "operational-value" { + return nil, errors.New("operational-value evaluator definition must use schemaVersion 4 and grader \"operational-value\"") + } + valueJSON := bytes.TrimSpace(definition.Baseline.Value) + switch definition.Baseline.Mode { + case "attainment-only": + if !bytes.Equal(valueJSON, []byte("null")) { + return nil, errors.New("attainment-only operational-value evaluators must have a null baseline value") + } + return nil, nil + case "baseline-comparable": + value, err := parseNullableOperationalValue(valueJSON) + if err != nil || value == nil || *value < 0 || *value > 1 { + return nil, errors.New("baseline-comparable operational-value evaluators require a baseline value in [0,1]") + } + return value, nil + default: + return nil, errors.New("operational-value evaluator baseline mode must be \"baseline-comparable\" or \"attainment-only\"") + } +} + +func parseOperationalValueEvaluatorOutput(data []byte, subject graderArtifactSubject, evidenceAtText string, evidenceAt time.Time, baselineValue *float64) (*operationalValueEvaluatorExecution, error) { + var fields map[string]json.RawMessage + if err := json.Unmarshal(data, &fields); err != nil || fields == nil { + return nil, errors.New("operational-value evaluator returned invalid JSON") + } + value, err := parseNullableOperationalValue(fields["value"]) + if err != nil || (value != nil && (*value < 0 || *value > 1)) { + return nil, errors.New("operational-value evaluator result.value must be null or a finite number in [0,1]") + } + observation, err := parseOperationalValueObservation(fields, subject, evidenceAtText, evidenceAt, value) + if err != nil { + return nil, err + } + var message string + if rawMessage, ok := fields["message"]; ok { + if err := json.Unmarshal(rawMessage, &message); err != nil { + message = "" + } + } + var diagnostics map[string]any + if rawDiagnostics, ok := fields["diagnostics"]; ok { + if err := json.Unmarshal(rawDiagnostics, &diagnostics); err != nil { + diagnostics = nil + } + } + var delta *float64 + if value != nil && baselineValue != nil { + computed := *value - *baselineValue + delta = &computed + } + return &operationalValueEvaluatorExecution{ + Value: value, + Message: message, + Diagnostics: diagnostics, + Observation: observation, + BaselineValue: baselineValue, + DeltaFromBaseline: delta, + }, nil +} + +func parseOperationalValueObservation(fields map[string]json.RawMessage, subject graderArtifactSubject, evidenceAtText string, evidenceAt time.Time, value *float64) (operationalValueRegradeObservation, error) { + var caseValue map[string]any + if err := json.Unmarshal(fields["case"], &caseValue); err != nil || caseValue == nil { + return operationalValueRegradeObservation{}, errors.New("operational-value evaluator output.case must be an object") + } + var opportunityKey, evidenceCutoffText, maturesAtText string + if err := json.Unmarshal(fields["opportunityKey"], &opportunityKey); err != nil || strings.TrimSpace(opportunityKey) == "" { + return operationalValueRegradeObservation{}, errors.New("operational-value evaluator opportunityKey must be a non-empty string") + } + if err := json.Unmarshal(fields["evidenceCutoff"], &evidenceCutoffText); err != nil { + return operationalValueRegradeObservation{}, errors.New("operational-value evaluator evidenceCutoff must be a UTC ISO-8601 timestamp") + } + if err := json.Unmarshal(fields["maturesAt"], &maturesAtText); err != nil { + return operationalValueRegradeObservation{}, errors.New("operational-value evaluator maturesAt must be a UTC ISO-8601 timestamp") + } + evidenceCutoff, err := parseOperationalValueTimestamp(evidenceCutoffText, "evidenceCutoff") + if err != nil { + return operationalValueRegradeObservation{}, err + } + maturesAt, err := parseOperationalValueTimestamp(maturesAtText, "maturesAt") + if err != nil { + return operationalValueRegradeObservation{}, err + } + if evidenceCutoff.After(evidenceAt) { + return operationalValueRegradeObservation{}, errors.New("operational-value evaluator evidenceCutoff cannot follow evidenceAt") + } + if evidenceCutoff.After(maturesAt) { + return operationalValueRegradeObservation{}, errors.New("operational-value evaluator evidenceCutoff cannot follow maturesAt") + } + var provenance []map[string]any + if err := json.Unmarshal(fields["provenance"], &provenance); err != nil || (value != nil && len(provenance) == 0) { + return operationalValueRegradeObservation{}, errors.New("operational-value evaluator must return provenance for a numeric value") + } + for _, item := range provenance { + for _, key := range []string{"repository", "kind", "ref"} { + text, ok := item[key].(string) + if !ok || text == "" { + return operationalValueRegradeObservation{}, errors.New("operational-value evaluator provenance entries require repository, kind, and ref") + } + } + } + return operationalValueRegradeObservation{ + Subject: subject, + OpportunityKey: opportunityKey, + EvidenceAt: evidenceAtText, + EvidenceCutoff: evidenceCutoffText, + MaturesAt: maturesAtText, + Mature: !evidenceAt.Before(maturesAt), + Case: caseValue, + Provenance: provenance, + }, nil +} + +func parseNullableOperationalValue(data []byte) (*float64, error) { + data = bytes.TrimSpace(data) + if bytes.Equal(data, []byte("null")) { + return nil, nil + } + var value float64 + if len(data) == 0 || json.Unmarshal(data, &value) != nil || math.IsNaN(value) || math.IsInf(value, 0) { + return nil, errors.New("expected a finite number or null") + } + return &value, nil +} + +func parseOperationalValueTimestamp(value, label string) (time.Time, error) { + for _, layout := range []string{"2006-01-02T15:04:05Z", "2006-01-02T15:04:05.000Z"} { + if parsed, err := time.Parse(layout, value); err == nil { + return parsed, nil + } + } + return time.Time{}, fmt.Errorf("%s must be a UTC ISO-8601 timestamp", label) +} + +func buildOperationalValueRegradeArtifact(run graderArtifactRun, manifest operationalValueGraderManifestEntry, original graderArtifactResult, evaluatorDigest string, execution *operationalValueEvaluatorExecution) operationalValueRegradeArtifact { + passed := evaluateOperationalValueThreshold(execution.Value, manifest.Direction, manifest.Threshold) + status := "unavailable" + if execution.Value != nil { + status = "pass" + if passed != nil && !*passed { + status = "fail" + } + } + return operationalValueRegradeArtifact{ + Version: 1, + Run: run, + Regrade: operationalValueRegradeMetadata{ + Identity: operationalValueRegradeIdentity{ + RunID: run.ID, + EvaluatorDigest: evaluatorDigest, + EvidenceAt: execution.Observation.EvidenceAt, + }, + OriginalEvidenceAt: original.Observation.EvidenceAt, + }, + Results: []operationalValueRegradeResult{{ + ID: "operational-value", + Name: manifest.Name, + Value: execution.Value, + Unit: manifest.Unit, + Passed: passed, + Status: status, + Source: "operational-value", + Message: execution.Message, + Observation: execution.Observation, + Diagnostics: execution.Diagnostics, + BaselineValue: execution.BaselineValue, + DeltaFromBaseline: execution.DeltaFromBaseline, + Implementation: graderArtifactImplementation{ + ID: "gh-aw-graders-operational-value-regrade", + Version: 1, + Digest: evaluatorDigest, + }, + }}, + } +} + +func evaluateOperationalValueThreshold(value *float64, direction string, threshold *float64) *bool { + if value == nil || threshold == nil { + return nil + } + passed := *value >= *threshold + if direction == "lower_is_better" { + passed = *value <= *threshold + } + return &passed +} + +func renderOperationalValueRegradeArtifact(artifact operationalValueRegradeArtifact, jsonOutput bool) error { + result := artifact.Results[0] + if jsonOutput { + data, err := marshalIndentJSONOrWrap(artifact, "operational-value regrade observation") + if err != nil { + return err + } + fmt.Fprintln(os.Stdout, string(data)) + return nil + } + value := "null" + if result.Value != nil { + value = strconv.FormatFloat(*result.Value, 'f', -1, 64) + } + fmt.Fprintln(os.Stdout, console.FormatSuccessMessage(fmt.Sprintf("Regraded operational value for run %s: %s", artifact.Run.ID, value))) + fmt.Fprintf(os.Stdout, "Evidence cutoff: %s\n", result.Observation.EvidenceCutoff) + fmt.Fprintf(os.Stdout, "Mature: %t\n", result.Observation.Mature) + if result.BaselineValue != nil { + fmt.Fprintf(os.Stdout, "Baseline value: %s\n", strconv.FormatFloat(*result.BaselineValue, 'f', -1, 64)) + delta := "null" + if result.DeltaFromBaseline != nil { + delta = strconv.FormatFloat(*result.DeltaFromBaseline, 'f', -1, 64) + } + fmt.Fprintf(os.Stdout, "Delta from baseline: %s\n", delta) + } + return nil +} diff --git a/pkg/cli/graders_operational_value_regrade_test.go b/pkg/cli/graders_operational_value_regrade_test.go new file mode 100644 index 00000000000..307580dc77f --- /dev/null +++ b/pkg/cli/graders_operational_value_regrade_test.go @@ -0,0 +1,258 @@ +package cli + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "io" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" + "testing" + "time" +) + +func historicalOperationalValueFixture() (operationalValueGraderManifestEntry, graderArtifactResult, graderArtifactRun) { + digest := strings.Repeat("a", 64) + createdAt := "2026-08-23T11:58:00Z" + manifest := operationalValueGraderManifestEntry{ + ID: "operational-value", + Name: "Operational value", + Source: "operational-value", + Enabled: true, + Direction: "higher_is_better", + Digest: digest, + Config: map[string]any{"window": "7d"}, + } + result := graderArtifactResult{ + ID: "operational-value", + Implementation: graderArtifactImplementation{ + ID: "gh-aw-graders", + Version: 1, + Digest: digest, + }, + Observation: &graderArtifactObservation{ + Subject: graderArtifactSubject{ + Type: "workflow-run", + RunID: "12345", + Attempt: 2, + Repository: "github/gh-aw", + Workflow: "Example", + Ref: "refs/heads/main", + SHA: "0123456789abcdef", + EventName: "schedule", + CreatedAt: &createdAt, + }, + EvidenceAt: "2026-08-24T12:00:00Z", + Case: map[string]any{"issue": float64(42)}, + }, + } + return manifest, result, graderArtifactRun{ID: "12345", Attempt: 2} +} + +func TestVerifyHistoricalOperationalValueIdentity(t *testing.T) { + manifest, result, run := historicalOperationalValueFixture() + if err := verifyHistoricalOperationalValueIdentity("github/gh-aw", manifest.Digest, &manifest, &result, run, run.ID); err != nil { + t.Fatalf("expected valid identity, got %v", err) + } + + t.Run("digest mismatch", func(t *testing.T) { + err := verifyHistoricalOperationalValueIdentity("github/gh-aw", strings.Repeat("b", 64), &manifest, &result, run, run.ID) + if err == nil || !strings.Contains(err.Error(), "digest mismatch") { + t.Fatalf("expected digest mismatch, got %v", err) + } + }) + + t.Run("repository mismatch", func(t *testing.T) { + err := verifyHistoricalOperationalValueIdentity("github/other", manifest.Digest, &manifest, &result, run, run.ID) + if err == nil || !strings.Contains(err.Error(), "repository") { + t.Fatalf("expected repository mismatch, got %v", err) + } + }) +} + +func TestExecuteHistoricalOperationalValueEvaluator(t *testing.T) { + manifest, result, _ := historicalOperationalValueFixture() + manifest.Config = nil + evaluatorContent := `#!/usr/bin/env bash +set -euo pipefail +case ${1:-} in +--definition) + printf '%s\n' '{"schemaVersion":4,"grader":"operational-value","baseline":{"mode":"baseline-comparable","value":0.25}}' + ;; +--grade-run) + request=$(cat) + [[ "$request" == *'"evidenceAt":"2026-09-01T12:00:00Z"'* ]] + [[ "$request" == *'"case":{"issue":42}'* ]] + [[ "$request" == *'"config":{}'* ]] + printf '%s\n' '{"value":0.75,"opportunityKey":"issue:42","case":{"issue":42},"evidenceCutoff":"2026-08-30T12:00:00Z","maturesAt":"2026-08-30T12:00:00Z","provenance":[{"repository":"github/gh-aw","kind":"issue","ref":"42"}]}' + ;; +*) exit 1 ;; +esac +` + evidenceAt, err := parseOperationalValueTimestamp("2026-09-01T12:00:00Z", "evidence-at") + if err != nil { + t.Fatal(err) + } + execution, err := executeHistoricalOperationalValueEvaluator( + context.Background(), evaluatorContent, manifest, *result.Observation, + "2026-09-01T12:00:00Z", evidenceAt, "https://github.com", + ) + if err != nil { + t.Fatalf("executeHistoricalOperationalValueEvaluator() error = %v", err) + } + if execution.Value == nil || *execution.Value != 0.75 { + t.Fatalf("value = %v, want 0.75", execution.Value) + } + if execution.DeltaFromBaseline == nil || *execution.DeltaFromBaseline != 0.5 { + t.Fatalf("delta = %v, want 0.5", execution.DeltaFromBaseline) + } + if !execution.Observation.Mature || execution.Observation.Subject.RunID != "12345" { + t.Fatalf("unexpected replay observation: %+v", execution.Observation) + } +} + +func TestVerifyArchivedOperationalValueEvaluatorSource(t *testing.T) { + repoRoot := t.TempDir() + evaluatorPath := filepath.Join(repoRoot, ".github", "graders", "example.sh") + if err := os.MkdirAll(filepath.Dir(evaluatorPath), 0o755); err != nil { + t.Fatal(err) + } + content := []byte("#!/usr/bin/env bash\nprintf 'trusted\\n'\n") + if err := os.WriteFile(evaluatorPath, content, 0o755); err != nil { + t.Fatal(err) + } + for _, args := range [][]string{ + {"init"}, + {"config", "user.email", "test@example.com"}, + {"config", "user.name", "Test"}, + {"add", ".github/graders/example.sh"}, + {"commit", "-m", "add evaluator"}, + } { + cmd := exec.Command("git", append([]string{"-C", repoRoot}, args...)...) + if output, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("git %v failed: %v: %s", args, err, output) + } + } + shaOutput, err := exec.Command("git", "-C", repoRoot, "rev-parse", "HEAD").Output() + if err != nil { + t.Fatal(err) + } + digest := sha256.Sum256(content) + manifest := operationalValueGraderManifestEntry{Run: ".github/graders/example.sh"} + subject := graderArtifactSubject{SHA: strings.TrimSpace(string(shaOutput))} + if err := verifyArchivedOperationalValueEvaluatorSource(repoRoot, "owner/repo", "owner/repo", string(content), hex.EncodeToString(digest[:]), manifest, subject); err != nil { + t.Fatalf("expected trusted evaluator, got %v", err) + } + if err := verifyArchivedOperationalValueEvaluatorSource(repoRoot, "owner/repo", "other/repo", string(content), hex.EncodeToString(digest[:]), manifest, subject); err == nil || !strings.Contains(err.Error(), "trusted local checkout") { + t.Fatalf("expected repository trust error, got %v", err) + } + if err := verifyArchivedOperationalValueEvaluatorSource(repoRoot, "owner/repo", "owner/repo", string(content)+"# changed\n", hex.EncodeToString(digest[:]), manifest, subject); err == nil || !strings.Contains(err.Error(), "does not match") { + t.Fatalf("expected source mismatch, got %v", err) + } +} + +func TestReadArchivedOperationalValueEvaluatorRejectsSymlink(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("symlink creation requires additional privileges on Windows") + } + runDir := t.TempDir() + gradersDir := filepath.Join(runDir, "agent", "graders") + if err := os.MkdirAll(gradersDir, 0o755); err != nil { + t.Fatal(err) + } + targetPath := filepath.Join(runDir, "evaluator.sh") + if err := os.WriteFile(targetPath, []byte("#!/usr/bin/env bash\n"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.Symlink(targetPath, filepath.Join(gradersDir, "operational_value_evaluator.sh")); err != nil { + t.Fatal(err) + } + + _, _, err := readArchivedOperationalValueEvaluator(runDir) + if err == nil || !strings.Contains(err.Error(), "must not be a symbolic link") { + t.Fatalf("expected symbolic-link rejection, got %v", err) + } +} + +func TestOperationalValueEvaluatorEnvironmentUsesRequestedHost(t *testing.T) { + env := operationalValueEvaluatorEnvironment([]string{ + "PATH=/usr/bin", + "GH_TOKEN=token", + "GH_HOST=stale.example.com", + "GITHUB_API_URL=https://stale.example.com/api/v3", + }, "https://ghe.example.com") + joined := strings.Join(env, "\n") + for _, expected := range []string{ + "GH_HOST=ghe.example.com", + "GITHUB_SERVER_URL=https://ghe.example.com", + "GITHUB_API_URL=https://ghe.example.com/api/v3", + "GITHUB_GRAPHQL_URL=https://ghe.example.com/api/graphql", + } { + if !strings.Contains(joined, expected) { + t.Fatalf("expected %q in evaluator environment: %v", expected, env) + } + } +} + +func TestRenderOperationalValueRegradeArtifactWithNullDelta(t *testing.T) { + baseline := 0.25 + artifact := operationalValueRegradeArtifact{ + Run: graderArtifactRun{ID: "12345"}, + Results: []operationalValueRegradeResult{{ + BaselineValue: &baseline, + Observation: operationalValueRegradeObservation{}, + }}, + } + oldStdout := os.Stdout + readOutput, writeOutput, err := os.Pipe() + if err != nil { + t.Fatal(err) + } + os.Stdout = writeOutput + t.Cleanup(func() { os.Stdout = oldStdout }) + if err := renderOperationalValueRegradeArtifact(artifact, false); err != nil { + t.Fatal(err) + } + _ = writeOutput.Close() + output, err := io.ReadAll(readOutput) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(output), "Delta from baseline: null") { + t.Fatalf("unexpected output: %s", output) + } +} + +func TestParseOperationalValueEvaluatorOutputRejectsFutureEvidence(t *testing.T) { + evidenceAt, err := time.Parse(time.RFC3339, "2026-08-24T12:00:00Z") + if err != nil { + t.Fatal(err) + } + _, err = parseOperationalValueEvaluatorOutput([]byte(`{ + "value": 1, + "opportunityKey": "issue:42", + "case": {"issue": 42}, + "evidenceCutoff": "2026-08-25T12:00:00Z", + "maturesAt": "2026-08-30T12:00:00Z", + "provenance": [{"repository":"github/gh-aw","kind":"issue","ref":"42"}] +}`), graderArtifactSubject{}, "2026-08-24T12:00:00Z", evidenceAt, nil) + if err == nil || !strings.Contains(err.Error(), "cannot follow evidenceAt") { + t.Fatalf("expected future evidence rejection, got %v", err) + } +} + +func TestNewGradersCommand(t *testing.T) { + command := NewGradersCommand() + operationalValueCommand, _, err := command.Find([]string{"operational-value"}) + if err != nil { + t.Fatal(err) + } + for _, name := range []string{"evidence-at", "repo", "json"} { + if operationalValueCommand.Flags().Lookup(name) == nil { + t.Fatalf("operational-value command missing --%s", name) + } + } +} diff --git a/pkg/constants/job_constants.go b/pkg/constants/job_constants.go index 4a2c316f899..b32f80dbc22 100644 --- a/pkg/constants/job_constants.go +++ b/pkg/constants/job_constants.go @@ -140,6 +140,9 @@ const GraderManifestFilename = "grader_manifest.json" // by trace_graders.cjs. Contains deterministic metric values computed from trace files. const GraderResultsFilename = "grader_results.json" +// OperationalValueEvaluatorFilename is the filename of the frozen operational-value evaluator archived for replay. +const OperationalValueEvaluatorFilename = "operational_value_evaluator.sh" + // GradersDir is the subdirectory under TmpGhAwAgentDir where grader output files are written. const GradersDir = TmpGhAwDir + "/agent/graders" diff --git a/pkg/parser/schema_test.go b/pkg/parser/schema_test.go index 572edf34215..e9adfe828a0 100644 --- a/pkg/parser/schema_test.go +++ b/pkg/parser/schema_test.go @@ -1044,6 +1044,23 @@ func TestValidateMainWorkflowFrontmatterWithSchemaAndLocation_SandboxAgentPlatfo }) } +func TestValidateMainWorkflowFrontmatterWithSchemaAndLocation_OperationalValueGrader(t *testing.T) { + t.Parallel() + + frontmatter := map[string]any{ + "on": "workflow_dispatch", + "graders": map[string]any{ + "operational-value": map[string]any{ + "run": ".github/graders/example-operational-value.sh", + }, + }, + } + + if err := ValidateMainWorkflowFrontmatterWithSchemaAndLocation(frontmatter, "/tmp/gh-aw/operational-value-grader-test.md"); err != nil { + t.Fatalf("expected operational-value evaluator to pass schema validation, got: %v", err) + } +} + func TestMainWorkflowSchema_WorkflowDispatchNumberTypeDocumentation(t *testing.T) { t.Parallel() diff --git a/pkg/parser/schemas/main_workflow_schema.json b/pkg/parser/schemas/main_workflow_schema.json index 50315afa473..a12f33ce937 100644 --- a/pkg/parser/schemas/main_workflow_schema.json +++ b/pkg/parser/schemas/main_workflow_schema.json @@ -12849,7 +12849,7 @@ "examples": ["gpt-5.4", "claude-3-5-sonnet-20241022", "gpt-4"] }, "graders": { - "description": "\u26a0\ufe0f Experimental. Deterministic graders to compute post-agent metrics from execution artifacts. Map keys are grader IDs. Built-in graders can be configured by ID; custom graders require a script.", + "description": "\u26a0\ufe0f Experimental. Deterministic graders for workflow-run observations. Built-in graders use execution artifacts, custom graders use inline scripts, and the operational-value grader uses a repository evaluator.", "type": "object", "propertyNames": { "pattern": "^[a-z][a-z0-9-]{0,63}$", @@ -12908,6 +12908,11 @@ "minLength": 1, "maxLength": 4096, "description": "Custom grader JavaScript script body (trusted workflows only)." + }, + "run": { + "type": "string", + "pattern": "^\\.github/graders/.+\\.sh$", + "description": "Repository-relative Bash script for the operational-value evaluator. Supported only for the reserved operational-value grader ID." } } } diff --git a/pkg/workflow/aw_info_tmp_test.go b/pkg/workflow/aw_info_tmp_test.go index 64ce9ff53f4..427064e80d2 100644 --- a/pkg/workflow/aw_info_tmp_test.go +++ b/pkg/workflow/aw_info_tmp_test.go @@ -62,7 +62,7 @@ This workflow tests that aw_info.json is generated in /tmp directory. t.Error("Expected step to require generate_aw_info.cjs module") } - if !strings.Contains(lockStr, "await main(core, context)") { + if !strings.Contains(lockStr, "await main(core, context);") { t.Error("Expected step to call main(core, context) from generate_aw_info.cjs") } diff --git a/pkg/workflow/compiler.go b/pkg/workflow/compiler.go index 3851e5d3768..a252ddcdb9b 100644 --- a/pkg/workflow/compiler.go +++ b/pkg/workflow/compiler.go @@ -99,6 +99,10 @@ func (c *Compiler) configureGHESCompatibility() { // - validatePermissions: permissions parsing, MCP tool constraints, workflow_run security // - validateToolConfiguration: safe-outputs, GitHub tools, dispatches, and resources func (c *Compiler) validateWorkflowData(workflowData *WorkflowData, markdownPath string) error { + if err := c.prepareOperationalValueGrader(workflowData, markdownPath); err != nil { + return formatCompilerError(markdownPath, "error", err.Error(), err) + } + if err := validateRunnerConfig(workflowData.RunnerConfig); err != nil { return formatCompilerError(markdownPath, "error", err.Error(), err) } diff --git a/pkg/workflow/compiler_activation_context.go b/pkg/workflow/compiler_activation_context.go index 813726cbf36..c966692e52d 100644 --- a/pkg/workflow/compiler_activation_context.go +++ b/pkg/workflow/compiler_activation_context.go @@ -186,6 +186,9 @@ func (c *Compiler) addActivationEngineOutputs(ctx *activationJobBuildContext, en ctx.steps = append(ctx.steps, awInfoYAML.String()) ctx.outputs["engine_id"] = "${{ steps.generate_aw_info.outputs.engine_id }}" ctx.outputs["model"] = "${{ steps.generate_aw_info.outputs.model }}" + if operationalValueGraderEnabled(ctx.data) { + ctx.outputs["run_created_at"] = "${{ steps.generate_aw_info.outputs.run_created_at }}" + } ctx.outputs["lockdown_check_failed"] = "${{ steps.generate_aw_info.outputs.lockdown_check_failed == 'true' }}" ctx.outputs["oauth_token_check_failed"] = "${{ steps.check-oauth-tokens.outputs.oauth_token_check_failed == 'true' }}" if !ctx.data.StaleCheckDisabled { diff --git a/pkg/workflow/compiler_activation_job_test.go b/pkg/workflow/compiler_activation_job_test.go index 47bcd10092f..8aeaf50a615 100644 --- a/pkg/workflow/compiler_activation_job_test.go +++ b/pkg/workflow/compiler_activation_job_test.go @@ -42,6 +42,28 @@ func TestActivationArtifactUploadRunsAfterSuccessOrFailure(t *testing.T) { assert.NotContains(t, uploadStep, "if: always()") } +func TestOperationalValueGraderScopesActionsReadToActivation(t *testing.T) { + compiler := NewCompiler() + data := operationalValueGraderWorkflowData(".github/graders/example-operational-value.sh") + data.Name = "Operational Value" + data.StaleCheckDisabled = true + data.Permissions = "permissions:\n contents: read" + + job, err := compiler.buildActivationJob(data, false, "", "operational-value.lock.yml") + require.NoError(t, err) + require.NotNil(t, job) + assert.Contains(t, job.Permissions, "actions: read") + assert.Equal(t, "${{ steps.generate_aw_info.outputs.run_created_at }}", job.Outputs["run_created_at"]) + + steps := strings.Join(job.Steps, "") + assert.Contains(t, steps, "GH_AW_INFO_FETCH_RUN_CREATED_AT: \"true\"") + assert.Contains(t, steps, "await main(core, context)") + + mainPermissions, err := compiler.buildMainJobPermissions(data) + require.NoError(t, err) + assert.NotContains(t, mainPermissions, "actions: read") +} + func TestGenerateCheckoutGitHubFolderForActivation_WorkflowCall(t *testing.T) { tests := []struct { name string diff --git a/pkg/workflow/compiler_activation_permissions.go b/pkg/workflow/compiler_activation_permissions.go index 11800fc3c14..f903aa8abe3 100644 --- a/pkg/workflow/compiler_activation_permissions.go +++ b/pkg/workflow/compiler_activation_permissions.go @@ -136,7 +136,7 @@ func (c *Compiler) buildActivationBasePermissions(ctx *activationJobBuildContext permsMap := map[PermissionScope]PermissionLevel{ PermissionContents: PermissionRead, } - if !ctx.data.StaleCheckDisabled || hasMaxDailyAICGuardrail(ctx.data) { + if !ctx.data.StaleCheckDisabled || hasMaxDailyAICGuardrail(ctx.data) || operationalValueGraderEnabled(ctx.data) { permsMap[PermissionActions] = PermissionRead } if isPreCreatePullRequestEnabled(ctx.data) { diff --git a/pkg/workflow/compiler_artifacts_test.go b/pkg/workflow/compiler_artifacts_test.go index 15ccb0562f9..9d0e73e41be 100644 --- a/pkg/workflow/compiler_artifacts_test.go +++ b/pkg/workflow/compiler_artifacts_test.go @@ -9,6 +9,7 @@ import ( "strings" "testing" + "github.com/github/gh-aw/pkg/constants" "github.com/github/gh-aw/pkg/stringutil" "github.com/github/gh-aw/pkg/testutil" @@ -602,6 +603,9 @@ permissions: contents: read engine: copilot strict: false +graders: + custom: + script: return 1 safe-outputs: create-issue: --- @@ -632,6 +636,8 @@ Body. "name: agent-output-fallback\n", "/tmp/gh-aw/agent_output.json", "/tmp/gh-aw/safeoutputs.jsonl", + "/tmp/gh-aw/agent/graders/grader_manifest.json", + "/tmp/gh-aw/agent/graders/grader_results.json", "if-no-files-found: ignore", "continue-on-error: true", } { @@ -639,6 +645,9 @@ Body. t.Errorf("Expected %q in fallback upload step, got:\n%s", expected, uploadSection) } } + if strings.Contains(uploadSection, constants.OperationalValueEvaluatorFilename) { + t.Errorf("Fallback upload must not include an operational-value evaluator for inline-only graders, got:\n%s", uploadSection) + } // The fallback artifact must be uploaded before the (large, failure-prone) agent artifact. uploadIdx := strings.Index(lockYAML, "- name: Upload agent output fallback artifact") diff --git a/pkg/workflow/compiler_main_job_helpers.go b/pkg/workflow/compiler_main_job_helpers.go index 49542220a9f..b90a1261f93 100644 --- a/pkg/workflow/compiler_main_job_helpers.go +++ b/pkg/workflow/compiler_main_job_helpers.go @@ -380,6 +380,14 @@ func (c *Compiler) buildMainJobPermissions(data *WorkflowData) (string, error) { return permissions, nil } +func operationalValueGraderEnabled(data *WorkflowData) bool { + if data == nil || data.Graders == nil { + return false + } + grader, ok := data.Graders.Graders["operational-value"] + return ok && (grader.Enabled == nil || *grader.Enabled) +} + // augmentPermissionsForDevMode adds contents: read to permissions when the compiler is in // dev or script mode and the actions folder checkout is needed. // diff --git a/pkg/workflow/compiler_main_job_helpers_test.go b/pkg/workflow/compiler_main_job_helpers_test.go index 7920e4a93c2..aff52cbb2ec 100644 --- a/pkg/workflow/compiler_main_job_helpers_test.go +++ b/pkg/workflow/compiler_main_job_helpers_test.go @@ -341,6 +341,35 @@ func TestBuildMainJobPermissions(t *testing.T) { _, err := c.buildMainJobPermissions(data) require.NoError(t, err) }) + + t.Run("operational value preserves declared permissions", func(t *testing.T) { + c := NewCompiler() + data := operationalValueGraderWorkflowData(".github/graders/example-operational-value.sh") + data.Permissions = "permissions:\n contents: read" + perms, err := c.buildMainJobPermissions(data) + require.NoError(t, err) + assert.NotContains(t, perms, "actions: read") + assert.Contains(t, perms, "contents: read") + }) + + t.Run("operational value allows explicit empty permissions", func(t *testing.T) { + c := NewCompiler() + data := operationalValueGraderWorkflowData(".github/graders/example-operational-value.sh") + data.Permissions = "permissions: {}" + _, err := c.buildMainJobPermissions(data) + require.NoError(t, err) + }) + + t.Run("disabled operational value does not add actions read", func(t *testing.T) { + c := NewCompiler() + disabled := false + data := operationalValueGraderWorkflowData(".github/graders/example-operational-value.sh") + data.Permissions = "permissions:\n contents: read" + data.Graders.Graders["operational-value"].Enabled = &disabled + perms, err := c.buildMainJobPermissions(data) + require.NoError(t, err) + assert.NotContains(t, perms, "actions: read") + }) } // TestWarnBuiltinJobEnvReferences tests warning emission for built-in job references in engine.env. diff --git a/pkg/workflow/compiler_yaml_artifacts.go b/pkg/workflow/compiler_yaml_artifacts.go index 2f91bb04eb1..c083f3512c8 100644 --- a/pkg/workflow/compiler_yaml_artifacts.go +++ b/pkg/workflow/compiler_yaml_artifacts.go @@ -78,10 +78,7 @@ func (c *Compiler) generateAgentOutputFallbackUpload(yaml *strings.Builder, data // Include grader manifest/results in the fallback so detection and downstream // jobs have reliable access even when the large unified artifact times out. if data.Graders != nil && data.Graders.HasGraders() { - paths = append(paths, - constants.GradersDirSlash+constants.GraderManifestFilename, - constants.GradersDirSlash+constants.GraderResultsFilename, - ) + paths = append(paths, collectGraderArtifactPaths(data.Graders)...) } c.stepOrderTracker.RecordArtifactUpload("Upload agent output fallback artifact", paths) diff --git a/pkg/workflow/compiler_yaml_graders.go b/pkg/workflow/compiler_yaml_graders.go index 9002fcffae4..577cb88dec3 100644 --- a/pkg/workflow/compiler_yaml_graders.go +++ b/pkg/workflow/compiler_yaml_graders.go @@ -53,6 +53,11 @@ func (c *Compiler) generateGradersStep(yaml *strings.Builder, data *WorkflowData yaml.WriteString(" setupGlobals(core, github, context, exec, io, getOctokit);\n") yaml.WriteString(" const { main } = require('" + SetupActionDestination + "/trace_graders.cjs');\n") fmt.Fprintf(yaml, " await main('%s', '%s');\n", manifestB64, execB64) + if operationalValueGrader, ok := data.Graders.Graders["operational-value"]; ok && (operationalValueGrader.Enabled == nil || *operationalValueGrader.Enabled) { + yaml.WriteString(" env:\n") + yaml.WriteString(" GH_TOKEN: ${{ github.token }}\n") + yaml.WriteString(" GH_AW_RUN_CREATED_AT: ${{ needs.activation.outputs.run_created_at }}\n") + } compilerYamlGradersLog.Print("Generated graders step") } @@ -63,14 +68,15 @@ type graderManifestEntry struct { ID string `json:"id"` Name string `json:"name"` Description string `json:"description,omitempty"` - Source string `json:"source"` // "builtin" or "inline" + Source string `json:"source"` // "builtin", "inline", or "operational-value" Enabled bool `json:"enabled"` Unit string `json:"unit,omitempty"` Direction string `json:"direction,omitempty"` Threshold *float64 `json:"threshold,omitempty"` Max *float64 `json:"max,omitempty"` Min *float64 `json:"min,omitempty"` - Digest string `json:"digest,omitempty"` // SHA-256 of inline script + Digest string `json:"digest,omitempty"` // SHA-256 of inline script or operational-value evaluator + Run string `json:"run,omitempty"` Config map[string]any `json:"config,omitempty"` } @@ -80,10 +86,11 @@ type graderManifest struct { Graders []graderManifestEntry `json:"graders"` } -// graderExecEntry carries the script body for a custom grader, keyed by ID. +// graderExecEntry carries trusted executable content for a custom grader, keyed by ID. type graderExecEntry struct { ID string `json:"id"` - Script string `json:"script"` + Script string `json:"script,omitempty"` + Run string `json:"run,omitempty"` } // buildGraderManifest constructs the manifest for the JS runtime. @@ -115,6 +122,13 @@ func buildGraderManifest(cfg *GradersConfig) *graderManifest { if _, ok := builtinSet[id]; !ok { source = "inline" } + if id == "operational-value" { + source = "operational-value" + } + digest := g.ScriptDigest() + if source == "operational-value" { + digest = g.EvaluatorDigest() + } name := g.Name if name == "" { name = id @@ -130,7 +144,8 @@ func buildGraderManifest(cfg *GradersConfig) *graderManifest { Threshold: g.Threshold, Max: g.Max, Min: g.Min, - Digest: g.ScriptDigest(), + Digest: digest, + Run: g.Run, Config: g.Config, }) } @@ -159,7 +174,9 @@ func buildGraderExecSpec(cfg *GradersConfig) []graderExecEntry { var specs []graderExecEntry for _, id := range cfg.EnabledGraderIDs() { g := cfg.Graders[id] - if _, ok := builtinSet[id]; !ok && g.Script != "" { + if id == "operational-value" && g.evaluatorContent != "" { + specs = append(specs, graderExecEntry{ID: id, Run: g.evaluatorContent}) + } else if _, ok := builtinSet[id]; !ok && g.Script != "" { specs = append(specs, graderExecEntry{ID: id, Script: g.Script}) } } @@ -207,9 +224,16 @@ func (c *Compiler) generateGraderRedactionStep(yaml *strings.Builder, yamlConten } // collectGraderArtifactPaths returns artifact paths for grader output files. -func collectGraderArtifactPaths() []string { - return []string{ +func collectGraderArtifactPaths(graders *GradersConfig) []string { + paths := []string{ constants.GradersDirSlash + constants.GraderManifestFilename, constants.GradersDirSlash + constants.GraderResultsFilename, } + if graders != nil { + grader := graders.Graders["operational-value"] + if grader != nil && (grader.Enabled == nil || *grader.Enabled) && grader.evaluatorContent != "" { + paths = append(paths, constants.GradersDirSlash+constants.OperationalValueEvaluatorFilename) + } + } + return paths } diff --git a/pkg/workflow/compiler_yaml_post_agent.go b/pkg/workflow/compiler_yaml_post_agent.go index 806e8eb451a..91796be8f2e 100644 --- a/pkg/workflow/compiler_yaml_post_agent.go +++ b/pkg/workflow/compiler_yaml_post_agent.go @@ -62,7 +62,7 @@ func (c *Compiler) collectArtifactPaths(data *WorkflowData, engine CodingAgentEn // Collect grader manifest and results when graders are configured. if data.Graders != nil && data.Graders.HasGraders() { - paths = append(paths, collectGraderArtifactPaths()...) + paths = append(paths, collectGraderArtifactPaths(data.Graders)...) } // Collect safe outputs and agent output paths for the unified artifact. diff --git a/pkg/workflow/compiler_yaml_step_lifecycle.go b/pkg/workflow/compiler_yaml_step_lifecycle.go index 723cb771c7d..b14eb917481 100644 --- a/pkg/workflow/compiler_yaml_step_lifecycle.go +++ b/pkg/workflow/compiler_yaml_step_lifecycle.go @@ -183,6 +183,9 @@ func (c *Compiler) generateCreateAwInfo(yaml *strings.Builder, data *WorkflowDat fmt.Fprintf(yaml, " GH_AW_INFO_AWMG_VERSION: \"%s\"\n", mcpGatewayVersion) fmt.Fprintf(yaml, " GH_AW_INFO_FIREWALL_TYPE: \"%s\"\n", firewallType) fmt.Fprintf(yaml, " GH_AW_INFO_AGENT_RUNTIME: \"%s\"\n", agentRuntime) + if operationalValueGraderEnabled(data) { + yaml.WriteString(" GH_AW_INFO_FETCH_RUN_CREATED_AT: \"true\"\n") + } // Only emit the cache-memory flag when at least one cache is configured. if data.CacheMemoryConfig != nil && len(data.CacheMemoryConfig.Caches) > 0 { yaml.WriteString(" GH_AW_INFO_CACHE_MEMORY: \"true\"\n") diff --git a/pkg/workflow/graders_config.go b/pkg/workflow/graders_config.go index b4e08993c75..d01cee44ba7 100644 --- a/pkg/workflow/graders_config.go +++ b/pkg/workflow/graders_config.go @@ -8,6 +8,7 @@ import ( "errors" "fmt" "maps" + "math" "regexp" "sort" "strings" @@ -64,17 +65,19 @@ var builtinGraderMetaByID = func() map[string]*BuiltinGraderMeta { // GraderDefinition represents a single grader entry in the graders map. type GraderDefinition struct { - ID string // grader identifier (must be unique) - Enabled *bool // explicit enable/disable; nil means use default (true for built-ins) - Name string // human-readable name (defaults from registry for built-ins) - Description string // description of the metric - Unit string // e.g. "ratio", "count", "ms", "factor" - Direction string // "higher_is_better" or "lower_is_better" - Threshold *float64 // quality threshold (pass/fail boundary) - Max *float64 // theoretical maximum - Min *float64 // theoretical minimum - Script string // inline JS body for trusted custom graders (built-ins leave empty) - Config map[string]any // arbitrary config passed to grader at runtime + ID string // grader identifier (must be unique) + Enabled *bool // explicit enable/disable; nil means use default (true for built-ins) + Name string // human-readable name (defaults from registry for built-ins) + Description string // description of the metric + Unit string // e.g. "ratio", "count", "ms", "factor" + Direction string // "higher_is_better" or "lower_is_better" + Threshold *float64 // quality threshold (pass/fail boundary) + Max *float64 // theoretical maximum + Min *float64 // theoretical minimum + Run string // repository-relative operational-value evaluator script + Script string // inline JS body for trusted custom graders (built-ins leave empty) + Config map[string]any // arbitrary config passed to grader at runtime + evaluatorContent string } // ScriptDigest returns the SHA-256 hex digest of the script, or "" if no script. @@ -86,6 +89,15 @@ func (g *GraderDefinition) ScriptDigest() string { return hex.EncodeToString(h[:]) } +// EvaluatorDigest returns the SHA-256 hex digest of the frozen operational-value evaluator. +func (g *GraderDefinition) EvaluatorDigest() string { + if g.evaluatorContent == "" { + return "" + } + h := sha256.Sum256([]byte(g.evaluatorContent)) + return hex.EncodeToString(h[:]) +} + // GradersConfig holds the configuration for deterministic graders declared // in workflow frontmatter. Graders run as an always() post-agent step in the agent job. type GradersConfig struct { @@ -106,13 +118,13 @@ func (gc *GradersConfig) HasGraders() bool { return false } -// HasCustomScripts returns true if any enabled grader has a custom script. +// HasCustomScripts returns true if any enabled grader has trusted custom code. func (gc *GradersConfig) HasCustomScripts() bool { if gc == nil { return false } for _, g := range gc.Graders { - if (g.Enabled == nil || *g.Enabled) && g.Script != "" { + if (g.Enabled == nil || *g.Enabled) && (g.Script != "" || g.evaluatorContent != "") { return true } } @@ -226,10 +238,19 @@ func (c *Compiler) parseGradersFromFrontmatter(frontmatter map[string]any) (*Gra // Apply built-in defaults if this is a built-in if meta, ok := builtinGraderMetaByID[id]; ok { def = builtinDefFromMeta(meta) + } else if id == "operational-value" { + def.Name = "Operational Value" + def.Unit = "ratio" + def.Direction = "higher_is_better" + def.Min = new(0.0) + def.Max = new(1.0) } _, isBuiltin := builtinSet[id] if entryRaw == nil { + if id == "operational-value" { + return nil, errors.New("graders.operational-value requires a 'run' field") + } if !isBuiltin { return nil, fmt.Errorf("graders.%s is not a built-in grader and requires a 'script' field. Built-in graders: %s", id, strings.Join(BuiltinGraderIDs, ", ")) } @@ -246,8 +267,11 @@ func (c *Compiler) parseGradersFromFrontmatter(frontmatter map[string]any) (*Gra return nil, err } - // Custom graders must have a script - if !isBuiltin && def.Script == "" && (def.Enabled == nil || *def.Enabled) { + // The operational-value grader uses a repository evaluator; other custom graders use inline scripts. + if id == "operational-value" && def.Run == "" && (def.Enabled == nil || *def.Enabled) { + return nil, errors.New("graders.operational-value requires a 'run' field") + } + if !isBuiltin && id != "operational-value" && def.Script == "" && (def.Enabled == nil || *def.Enabled) { return nil, fmt.Errorf("graders.%s is not a built-in grader and requires a 'script' field. Built-in graders: %s", id, strings.Join(BuiltinGraderIDs, ", ")) } @@ -363,6 +387,21 @@ func parseGraderEntryFields(def *GraderDefinition, entry map[string]any, id stri def.Config = m } + if runRaw, ok := entry["run"]; ok { + runPath, ok := runRaw.(string) + if !ok { + return fmt.Errorf("graders.%s.run must be a string, got %T", id, runRaw) + } + runPath = strings.TrimSpace(runPath) + if id != "operational-value" { + return fmt.Errorf("graders.%s.run is only supported by the operational-value grader", id) + } + if !isValidOperationalValueEvaluatorPath(runPath) { + return fmt.Errorf("graders.operational-value.run must be a repository-relative .sh file under .github/graders, got %q", runPath) + } + def.Run = runPath + } + if scriptRaw, ok := entry["script"]; ok { s, ok := scriptRaw.(string) if !ok { @@ -375,6 +414,9 @@ func parseGraderEntryFields(def *GraderDefinition, entry map[string]any, id stri if isBuiltin { return fmt.Errorf("graders.%s is a built-in grader and cannot have a custom script", id) } + if id == "operational-value" { + return errors.New("graders.operational-value cannot have an inline script; use 'run'") + } scriptCharCount := utf8.RuneCountInString(s) if scriptCharCount > 4096 { return fmt.Errorf("graders.%s.script exceeds maximum length of 4096 characters (%d)", id, scriptCharCount) @@ -391,6 +433,22 @@ func parseGraderEntryFields(def *GraderDefinition, entry map[string]any, id stri return nil } +func isValidOperationalValueEvaluatorPath(evaluatorPath string) bool { + if evaluatorPath == "" || strings.Contains(evaluatorPath, "\\") { + return false + } + parts := strings.Split(evaluatorPath, "/") + if len(parts) < 3 || parts[0] != ".github" || parts[1] != "graders" { + return false + } + for _, part := range parts { + if part == "" || part == "." || part == ".." { + return false + } + } + return strings.HasSuffix(evaluatorPath, ".sh") +} + // parseOptionalFloat parses an optional float64 field from a map. func parseOptionalFloat(m map[string]any, key string, graderID string, target **float64) error { v, ok := m[key] @@ -414,6 +472,17 @@ func validateGraders(cfg *GradersConfig) error { if cfg == nil { return nil } + if grader, ok := cfg.Graders["operational-value"]; ok { + if grader.Direction != "higher_is_better" { + return errors.New("graders.operational-value.direction must be 'higher_is_better'") + } + if grader.Min == nil || *grader.Min != 0 || grader.Max == nil || *grader.Max != 1 { + return errors.New("graders.operational-value range must be min: 0 and max: 1") + } + if grader.Threshold != nil && (math.IsNaN(*grader.Threshold) || math.IsInf(*grader.Threshold, 0) || *grader.Threshold < 0 || *grader.Threshold > 1) { + return errors.New("graders.operational-value.threshold must be between 0 and 1") + } + } if !cfg.HasGraders() { return errors.New("graders configuration has no enabled graders. Remove the graders field to disable grading, or set enabled: true on at least one grader") } diff --git a/pkg/workflow/graders_config_test.go b/pkg/workflow/graders_config_test.go index 40fa29baa77..2f5079b44c0 100644 --- a/pkg/workflow/graders_config_test.go +++ b/pkg/workflow/graders_config_test.go @@ -10,6 +10,7 @@ import ( "testing" "github.com/github/gh-aw/pkg/stringutil" + "github.com/stretchr/testify/assert" ) // TestParseGradersFromFrontmatter_Absent verifies nil return when graders absent. @@ -122,6 +123,74 @@ func TestParseGradersFromFrontmatter_CustomGrader(t *testing.T) { } } +func TestParseGradersFromFrontmatter_OperationalValueGrader(t *testing.T) { + var c Compiler + cfg, err := c.parseGradersFromFrontmatter(map[string]any{ + "graders": map[string]any{ + "operational-value": map[string]any{ + "run": ".github/graders/example-operational-value.sh", + }, + }, + }) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + grader := cfg.Graders["operational-value"] + if grader.Run != ".github/graders/example-operational-value.sh" { + t.Fatalf("unexpected operational-value run path: %q", grader.Run) + } + if grader.Unit != "ratio" || grader.Direction != "higher_is_better" { + t.Fatalf("unexpected operational-value defaults: unit=%q direction=%q", grader.Unit, grader.Direction) + } + if grader.Min == nil || *grader.Min != 0 || grader.Max == nil || *grader.Max != 1 { + t.Fatalf("expected operational-value range [0,1], got min=%v max=%v", grader.Min, grader.Max) + } +} + +func TestParseGradersFromFrontmatter_OperationalValueGraderValidation(t *testing.T) { + var c Compiler + tests := []struct { + name string + entry map[string]any + errText string + }{ + {name: "missing run", entry: map[string]any{}, errText: "requires a 'run' field"}, + {name: "path traversal", entry: map[string]any{"run": ".github/graders/../secret.sh"}, errText: "repository-relative"}, + {name: "wrong directory", entry: map[string]any{"run": "scripts/operational-value.sh"}, errText: "repository-relative"}, + {name: "wrong extension", entry: map[string]any{"run": ".github/graders/operational-value.js"}, errText: "repository-relative"}, + {name: "inline script", entry: map[string]any{"run": ".github/graders/operational-value.sh", "script": "return 1"}, errText: "cannot have an inline script"}, + {name: "direction", entry: map[string]any{"run": ".github/graders/operational-value.sh", "direction": "lower_is_better"}, errText: "direction must be 'higher_is_better'"}, + {name: "minimum", entry: map[string]any{"run": ".github/graders/operational-value.sh", "min": 0.1}, errText: "range must be min: 0 and max: 1"}, + {name: "maximum", entry: map[string]any{"run": ".github/graders/operational-value.sh", "max": 2.0}, errText: "range must be min: 0 and max: 1"}, + {name: "threshold below range", entry: map[string]any{"run": ".github/graders/operational-value.sh", "threshold": -0.1}, errText: "threshold must be between 0 and 1"}, + {name: "threshold above range", entry: map[string]any{"run": ".github/graders/operational-value.sh", "threshold": 1.1}, errText: "threshold must be between 0 and 1"}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + _, err := c.parseGradersFromFrontmatter(map[string]any{ + "graders": map[string]any{"operational-value": test.entry}, + }) + if err == nil || !strings.Contains(err.Error(), test.errText) { + t.Fatalf("expected error containing %q, got %v", test.errText, err) + } + }) + } +} + +func TestParseGradersFromFrontmatter_RunRejectedForOtherGraders(t *testing.T) { + var c Compiler + _, err := c.parseGradersFromFrontmatter(map[string]any{ + "graders": map[string]any{ + "custom": map[string]any{ + "run": ".github/graders/operational-value.sh", + }, + }, + }) + if err == nil { + t.Fatal("expected run to be rejected for a non-operational-value grader") + } +} + func TestMergeImportedGradersFrontmatter(t *testing.T) { frontmatter := map[string]any{ "graders": map[string]any{ @@ -444,6 +513,48 @@ func TestBuildGraderManifest(t *testing.T) { } } +func TestBuildGraderManifest_OperationalValueGrader(t *testing.T) { + grader := &GraderDefinition{ + ID: "operational-value", + Run: ".github/graders/example-operational-value.sh", + } + grader.evaluatorContent = "#!/usr/bin/env bash\necho '{}'\n" + cfg := &GradersConfig{Graders: map[string]*GraderDefinition{"operational-value": grader}} + + manifest := buildGraderManifest(cfg) + if len(manifest.Graders) != 1 { + t.Fatalf("expected one grader, got %d", len(manifest.Graders)) + } + if manifest.Graders[0].Source != "operational-value" { + t.Fatalf("expected operational-value source, got %q", manifest.Graders[0].Source) + } + if manifest.Graders[0].Digest != grader.EvaluatorDigest() { + t.Fatalf("expected frozen evaluator digest, got %q", manifest.Graders[0].Digest) + } + manifestJSON, err := json.Marshal(manifest) + if err != nil { + t.Fatalf("marshal operational-value manifest: %v", err) + } + if !strings.Contains(string(manifestJSON), `"run":".github/graders/example-operational-value.sh"`) || strings.Contains(string(manifestJSON), `"evaluator"`) { + t.Fatalf("expected manifest to use run field, got %s", manifestJSON) + } + + execSpec := buildGraderExecSpec(cfg) + if len(execSpec) != 1 || execSpec[0].Run != grader.evaluatorContent { + t.Fatal("expected frozen evaluator in execution spec") + } + if execSpec[0].Script != "" { + t.Fatal("operational-value grader must not be serialized as an inline script") + } + execSpecJSON, err := json.Marshal(execSpec) + if err != nil { + t.Fatalf("marshal operational-value execution spec: %v", err) + } + if !strings.Contains(string(execSpecJSON), `"run":`) || strings.Contains(string(execSpecJSON), `"evaluator"`) { + t.Fatalf("expected execution spec to use run field, got %s", execSpecJSON) + } +} + // TestGenerateGradersStep_Absent verifies no step when graders nil. func TestGenerateGradersStep_Absent(t *testing.T) { c := &Compiler{} @@ -489,6 +600,20 @@ func TestGenerateGradersStep_Present(t *testing.T) { } } +func TestGenerateGradersStep_OperationalValueUsesActivationRunMetadata(t *testing.T) { + c := &Compiler{} + initActionPinCacheForTest(c) + var yaml strings.Builder + data := operationalValueGraderWorkflowData(".github/graders/example-operational-value.sh") + + c.generateGradersStep(&yaml, data) + + output := yaml.String() + assert.Contains(t, output, "GH_AW_RUN_CREATED_AT: ${{ needs.activation.outputs.run_created_at }}") + assert.Contains(t, output, "GH_TOKEN: ${{ github.token }}") + assert.NotContains(t, output, "getWorkflowRun") +} + // TestGenerateGradersStep_BeforeArtifactUpload verifies ordering. func TestGenerateGradersStep_BeforeArtifactUpload(t *testing.T) { c := &Compiler{} @@ -518,11 +643,15 @@ func TestGenerateGradersStep_BeforeArtifactUpload(t *testing.T) { } } -// TestCollectGraderArtifactPaths verifies paths include manifest and results. +// TestCollectGraderArtifactPaths verifies paths include all replay artifacts. func TestCollectGraderArtifactPaths(t *testing.T) { - paths := collectGraderArtifactPaths() - if len(paths) != 2 { - t.Fatalf("expected 2 paths, got %d", len(paths)) + grader := &GraderDefinition{ID: "operational-value"} + grader.evaluatorContent = "#!/usr/bin/env bash\n" + paths := collectGraderArtifactPaths(&GradersConfig{ + Graders: map[string]*GraderDefinition{"operational-value": grader}, + }) + if len(paths) != 3 { + t.Fatalf("expected 3 paths, got %d", len(paths)) } if !strings.Contains(paths[0], "grader_manifest.json") { t.Fatal("expected grader_manifest.json in paths") @@ -530,6 +659,18 @@ func TestCollectGraderArtifactPaths(t *testing.T) { if !strings.Contains(paths[1], "grader_results.json") { t.Fatal("expected grader_results.json in paths") } + if !strings.Contains(paths[2], "operational_value_evaluator.sh") { + t.Fatal("expected operational_value_evaluator.sh in paths") + } +} + +func TestCollectGraderArtifactPathsWithoutOperationalValue(t *testing.T) { + paths := collectGraderArtifactPaths(&GradersConfig{ + Graders: map[string]*GraderDefinition{"retries": {ID: "retries"}}, + }) + if len(paths) != 2 { + t.Fatalf("expected manifest and results paths, got %v", paths) + } } // initActionPinCacheForTest sets up minimal action pin resolution for tests. @@ -540,7 +681,7 @@ func initActionPinCacheForTest(c *Compiler) { // TestCollectGraderArtifactPaths_AgentGradersDir verifies paths use the agent/graders subdirectory. func TestCollectGraderArtifactPaths_AgentGradersDir(t *testing.T) { - paths := collectGraderArtifactPaths() + paths := collectGraderArtifactPaths(nil) for _, p := range paths { if !strings.Contains(p, "agent/graders/") { t.Errorf("expected path to contain agent/graders/, got %q", p) diff --git a/pkg/workflow/graders_operational_value.go b/pkg/workflow/graders_operational_value.go new file mode 100644 index 00000000000..98d30a36960 --- /dev/null +++ b/pkg/workflow/graders_operational_value.go @@ -0,0 +1,83 @@ +package workflow + +import ( + "errors" + "fmt" + "io" + "os" + "path/filepath" + "strings" + "unicode/utf8" + + "github.com/github/gh-aw/pkg/fileutil" + "github.com/github/gh-aw/pkg/gitutil" +) + +const maxOperationalValueEvaluatorSize = 64 * 1024 + +func (c *Compiler) prepareOperationalValueGrader(data *WorkflowData, markdownPath string) error { + if data == nil || data.Graders == nil { + return nil + } + grader, ok := data.Graders.Graders["operational-value"] + if !ok || (grader.Enabled != nil && !*grader.Enabled) { + return nil + } + if grader.Run == "" { + return errors.New("graders.operational-value requires a 'run' field") + } + + repoRoot, err := gitutil.FindGitRootFrom(filepath.Dir(markdownPath)) + if err != nil { + return fmt.Errorf("cannot resolve graders.operational-value.run %q: workflow is not inside a Git repository", grader.Run) + } + evaluatorPath := filepath.Join(repoRoot, filepath.FromSlash(grader.Run)) + if err := fileutil.ValidatePathWithinBase(repoRoot, evaluatorPath); err != nil { + return fmt.Errorf("graders.operational-value.run %q escapes the Git repository", grader.Run) + } + evaluatorInfo, err := os.Lstat(evaluatorPath) + if err != nil { + return fmt.Errorf("cannot inspect graders.operational-value.run %q: %w", grader.Run, err) + } + if evaluatorInfo.Mode()&os.ModeSymlink != 0 { + return fmt.Errorf("graders.operational-value.run %q must not be a symbolic link", grader.Run) + } + + file, err := os.Open(evaluatorPath) + if err != nil { + return fmt.Errorf("cannot read graders.operational-value.run %q: %w", grader.Run, err) + } + defer file.Close() + + info, err := file.Stat() + if err != nil { + return fmt.Errorf("cannot inspect graders.operational-value.run %q: %w", grader.Run, err) + } + if !info.Mode().IsRegular() { + return fmt.Errorf("graders.operational-value.run %q must be a regular file", grader.Run) + } + if info.Size() > maxOperationalValueEvaluatorSize { + return fmt.Errorf("graders.operational-value.run %q exceeds the %d-byte limit", grader.Run, maxOperationalValueEvaluatorSize) + } + + content, err := io.ReadAll(io.LimitReader(file, maxOperationalValueEvaluatorSize+1)) + if err != nil { + return fmt.Errorf("cannot read graders.operational-value.run %q: %w", grader.Run, err) + } + if len(content) > maxOperationalValueEvaluatorSize { + return fmt.Errorf("graders.operational-value.run %q exceeds the %d-byte limit", grader.Run, maxOperationalValueEvaluatorSize) + } + if !utf8.Valid(content) { + return fmt.Errorf("graders.operational-value.run %q must be valid UTF-8", grader.Run) + } + evaluatorContent := string(content) + if !strings.HasPrefix(evaluatorContent, "#!/usr/bin/env bash\n") && !strings.HasPrefix(evaluatorContent, "#!/bin/bash\n") { + return fmt.Errorf("graders.operational-value.run %q must start with a Bash shebang", grader.Run) + } + if err := validateOperationalValueEvaluatorBash(evaluatorContent); err != nil { + return fmt.Errorf("graders.operational-value.run %q has invalid Bash syntax: %w", grader.Run, err) + } + + grader.evaluatorContent = evaluatorContent + return nil +} diff --git a/pkg/workflow/graders_operational_value_bash.go b/pkg/workflow/graders_operational_value_bash.go new file mode 100644 index 00000000000..9d8e0a79e54 --- /dev/null +++ b/pkg/workflow/graders_operational_value_bash.go @@ -0,0 +1,23 @@ +//go:build !js && !wasm + +package workflow + +import ( + "errors" + "os/exec" + "strings" +) + +func validateOperationalValueEvaluatorBash(evaluatorContent string) error { + cmd := exec.Command("bash", "-n") + cmd.Stdin = strings.NewReader(evaluatorContent) + output, err := cmd.CombinedOutput() + if err == nil { + return nil + } + message := strings.TrimSpace(string(output)) + if message == "" { + message = err.Error() + } + return errors.New(message) +} diff --git a/pkg/workflow/graders_operational_value_bash_wasm.go b/pkg/workflow/graders_operational_value_bash_wasm.go new file mode 100644 index 00000000000..234703b75be --- /dev/null +++ b/pkg/workflow/graders_operational_value_bash_wasm.go @@ -0,0 +1,9 @@ +//go:build js || wasm + +package workflow + +import "errors" + +func validateOperationalValueEvaluatorBash(string) error { + return errors.New("Bash syntax validation is not available in Wasm") +} diff --git a/pkg/workflow/graders_operational_value_test.go b/pkg/workflow/graders_operational_value_test.go new file mode 100644 index 00000000000..5cbd16fc4a4 --- /dev/null +++ b/pkg/workflow/graders_operational_value_test.go @@ -0,0 +1,150 @@ +package workflow + +import ( + "os" + "path/filepath" + "runtime" + "strings" + "testing" +) + +func TestPrepareOperationalValueGrader(t *testing.T) { + repoRoot := t.TempDir() + if err := os.Mkdir(filepath.Join(repoRoot, ".git"), 0o755); err != nil { + t.Fatal(err) + } + workflowPath := filepath.Join(repoRoot, ".github", "workflows", "example.md") + evaluatorPath := filepath.Join(repoRoot, ".github", "graders", "example-operational-value.sh") + if err := os.MkdirAll(filepath.Dir(workflowPath), 0o755); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Dir(evaluatorPath), 0o755); err != nil { + t.Fatal(err) + } + content := "#!/usr/bin/env bash\nset -euo pipefail\n" + if err := os.WriteFile(evaluatorPath, []byte(content), 0o755); err != nil { + t.Fatal(err) + } + data := operationalValueGraderWorkflowData(".github/graders/example-operational-value.sh") + + if err := (&Compiler{}).prepareOperationalValueGrader(data, workflowPath); err != nil { + t.Fatalf("unexpected error: %v", err) + } + grader := data.Graders.Graders["operational-value"] + if grader.evaluatorContent != content { + t.Fatal("expected operational-value evaluator content to be frozen") + } + if len(grader.EvaluatorDigest()) != 64 { + t.Fatalf("expected SHA-256 digest, got %q", grader.EvaluatorDigest()) + } +} + +func TestPrepareOperationalValueGraderRejectsInvalidFiles(t *testing.T) { + tests := []struct { + name string + content string + errText string + }{ + {name: "missing", errText: "cannot inspect"}, + {name: "not bash", content: "echo operational value\n", errText: "Bash shebang"}, + {name: "invalid bash", content: "#!/usr/bin/env bash\nif true; then\n", errText: "invalid Bash syntax"}, + {name: "oversized", content: "#!/usr/bin/env bash\n" + strings.Repeat("x", maxOperationalValueEvaluatorSize), errText: "exceeds"}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + repoRoot := t.TempDir() + if err := os.Mkdir(filepath.Join(repoRoot, ".git"), 0o755); err != nil { + t.Fatal(err) + } + workflowPath := filepath.Join(repoRoot, ".github", "workflows", "example.md") + evaluatorPath := filepath.Join(repoRoot, ".github", "graders", "example-operational-value.sh") + if err := os.MkdirAll(filepath.Dir(workflowPath), 0o755); err != nil { + t.Fatal(err) + } + if test.content != "" { + if err := os.MkdirAll(filepath.Dir(evaluatorPath), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(evaluatorPath, []byte(test.content), 0o755); err != nil { + t.Fatal(err) + } + } + + err := (&Compiler{}).prepareOperationalValueGrader(operationalValueGraderWorkflowData(".github/graders/example-operational-value.sh"), workflowPath) + if err == nil || !strings.Contains(err.Error(), test.errText) { + t.Fatalf("expected error containing %q, got %v", test.errText, err) + } + }) + } +} + +func TestPrepareOperationalValueGraderRejectsSymlinkEscape(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("symlink creation requires additional privileges on Windows") + } + repoRoot := t.TempDir() + outside := filepath.Join(t.TempDir(), "operational-value.sh") + if err := os.WriteFile(outside, []byte("#!/usr/bin/env bash\n"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.Mkdir(filepath.Join(repoRoot, ".git"), 0o755); err != nil { + t.Fatal(err) + } + workflowPath := filepath.Join(repoRoot, ".github", "workflows", "example.md") + evaluatorPath := filepath.Join(repoRoot, ".github", "graders", "example-operational-value.sh") + if err := os.MkdirAll(filepath.Dir(workflowPath), 0o755); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Dir(evaluatorPath), 0o755); err != nil { + t.Fatal(err) + } + if err := os.Symlink(outside, evaluatorPath); err != nil { + t.Fatal(err) + } + + err := (&Compiler{}).prepareOperationalValueGrader(operationalValueGraderWorkflowData(".github/graders/example-operational-value.sh"), workflowPath) + if err == nil || !strings.Contains(err.Error(), "escapes") { + t.Fatalf("expected symlink escape error, got %v", err) + } +} + +func TestPrepareOperationalValueGraderRejectsRepositorySymlink(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("symlink creation requires additional privileges on Windows") + } + repoRoot := t.TempDir() + if err := os.Mkdir(filepath.Join(repoRoot, ".git"), 0o755); err != nil { + t.Fatal(err) + } + workflowPath := filepath.Join(repoRoot, ".github", "workflows", "example.md") + gradersDir := filepath.Join(repoRoot, ".github", "graders") + targetPath := filepath.Join(gradersDir, "target.sh") + evaluatorPath := filepath.Join(gradersDir, "example-operational-value.sh") + if err := os.MkdirAll(filepath.Dir(workflowPath), 0o755); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(gradersDir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(targetPath, []byte("#!/usr/bin/env bash\n"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.Symlink(targetPath, evaluatorPath); err != nil { + t.Fatal(err) + } + + err := (&Compiler{}).prepareOperationalValueGrader(operationalValueGraderWorkflowData(".github/graders/example-operational-value.sh"), workflowPath) + if err == nil || !strings.Contains(err.Error(), "must not be a symbolic link") { + t.Fatalf("expected symlink rejection error, got %v", err) + } +} + +func operationalValueGraderWorkflowData(evaluatorPath string) *WorkflowData { + return &WorkflowData{ + Graders: &GradersConfig{ + Graders: map[string]*GraderDefinition{ + "operational-value": {ID: "operational-value", Run: evaluatorPath}, + }, + }, + } +} diff --git a/pkg/workflow/step_order_validation.go b/pkg/workflow/step_order_validation.go index 5364e69e623..fbd8d69b83f 100644 --- a/pkg/workflow/step_order_validation.go +++ b/pkg/workflow/step_order_validation.go @@ -222,17 +222,16 @@ func isPathScannedBySecretRedaction(path string) bool { // that is NOT text-scanned by the redact_secrets step, but is nevertheless explicitly // permitted in artifact uploads. // -// .bundle files are binary git bundles produced when patch-format: bundle is -// configured. They are generated from the same git commit range as the accompanying -// .patch file (see generateGitBundle in actions/setup/js/generate_git_bundle.cjs), -// so the .patch scanning already covers the underlying diff content. Bundles cannot -// be safely scanned as UTF-8 text, but they are required downstream to apply changes -// while preserving merge topology, so they are intentionally allowed through -// artifact uploads unscanned. +// In addition to binary git bundles, the archived operational-value evaluator is +// allowed because the compiler freezes its trusted repository bytes and records +// their digest for replay. Redacting that archive would invalidate its provenance. func isKnownUnscannedButAllowedForUpload(path string) bool { isUnderGhAwDir := strings.HasPrefix(path, constants.TmpGhAwDirSlash) || strings.HasPrefix(path, constants.GhAwRootDirShellSlash) || strings.HasPrefix(path, constants.GhAwRootDirSlash) || strings.Contains(path, "${{ env.") - return isUnderGhAwDir && filepath.Ext(path) == ".bundle" + if !isUnderGhAwDir { + return false + } + return filepath.Ext(path) == ".bundle" || path == constants.GradersDirSlash+constants.OperationalValueEvaluatorFilename } diff --git a/pkg/workflow/step_order_validation_test.go b/pkg/workflow/step_order_validation_test.go index b42d1ed06de..4378160ca75 100644 --- a/pkg/workflow/step_order_validation_test.go +++ b/pkg/workflow/step_order_validation_test.go @@ -33,6 +33,19 @@ func TestStepOrderTracker_ValidateOrdering_SecretRedactionBeforeUploads(t *testi } } +func TestStepOrderTracker_ValidateOrdering_ArchivedOperationalValueEvaluator(t *testing.T) { + tracker := NewStepOrderTracker() + tracker.MarkAgentExecutionComplete() + tracker.RecordSecretRedaction("Redact grader outputs") + tracker.RecordArtifactUpload("Upload agent artifacts", []string{ + "/tmp/gh-aw/agent/graders/operational_value_evaluator.sh", + }) + + if err := tracker.ValidateStepOrdering(); err != nil { + t.Errorf("Expected archived operational-value evaluator to be allowed, got: %v", err) + } +} + func TestStepOrderTracker_ValidateOrdering_UploadBeforeSecretRedaction(t *testing.T) { tracker := NewStepOrderTracker() tracker.MarkAgentExecutionComplete()