feat: add value grader

.github/skills/aw-value/SKILL.md

@@ -0,0 +1,134 @@
+---
+name: aw-value
+description: "Design and verify a deterministic operational-value grader for a GitHub Agentic Workflow. Use for per-run operational value, evidence attribution, maturation, baselines, and operational-value evaluators. Usage: /aw-value OWNER/REPO WORKFLOW-NAME."
+argument-hint: "OWNER/REPO WORKFLOW-NAME"
+allowed-tools: bash jq gh
+metadata:
+  version: "1.0.0"
+---
+
+# Operational Value Grader
+
+Design one deterministic `operational-value` grader that reports absolute operational attainment for each workflow run.
+
+Operational value is the degree to which the workflow's intended repository outcome is attained for the opportunity assigned to a run, demonstrated by accepted repository evidence under a frozen contract. It is not execution quality, output volume, safe-output creation, or an agent's assessment.
+
+## Output
+
+Create one executable evaluator at:
+
+```text
+.github/graders/WORKFLOW-NAME-operational-value.sh
+```
+
+Configure the workflow:
+
+```yaml
+graders:
+  operational-value:
+    run: .github/graders/WORKFLOW-NAME-operational-value.sh
+```
+
+The grader's primary `value` is absolute attainment in `[0,1]`. A comparable frozen baseline may be reported separately as `baselineValue`; gh-aw derives `deltaFromBaseline`. Never define the primary value as a difference from baseline.
+
+## Design Procedure
+
+1. Validate `OWNER/REPO` and resolve `.github/workflows/WORKFLOW-NAME.md`. Do not infer inputs from the workspace or remotes.
+2. Recover adoption-time intent from the workflow's first commit and first parent. Use only adoption-time workflow content and pre-adoption evidence to choose opportunities, accepted evidence, formulas, targets, or a baseline.
+3. Define how every workflow run binds to one operational case:
+   - produce a stable `opportunityKey`;
+   - prevent overlapping ownership where possible;
+   - preserve repeated keys when duplicate runs target the same opportunity so downstream analysis can cluster or deduplicate them;
+   - treat reruns with the same GitHub run ID as the same subject.
+4. Freeze accepted evidence, evidence repositories, matching rules, zero-versus-missing behavior, and `maturesAt` computation.
+  - Declare only the workflow permission scopes required to collect that evidence. The evaluator receives `GH_TOKEN` with the agent job's declared permissions; gh-aw does not add evidence permissions automatically.
+5. Choose exactly one direct primary metric in `[0,1]`. Higher must always mean greater attainment. Keep trace graders and activity counts separate.
+6. If comparable pre-adoption evidence exists, score it with the same metric and freeze it under `baseline`. Otherwise use `attainment-only` with a null baseline value.
+7. Implement the evaluator interface below and run:
+
+   ```bash
+  .github/skills/aw-value/scripts/verify-operational-value-evaluator.sh .github/graders/WORKFLOW-NAME-operational-value.sh
+   gh aw compile .github/workflows/WORKFLOW-NAME.md
+   ```
+
+## Evaluator Interface
+
+The evaluator uses Bash 3.2-compatible Bash plus `jq` and supports:
+
+- `--definition`: print the frozen schema-version 4 contract.
+- `--metric`: read one evidence object on stdin and print a deterministic number in `[0,1]` or `null`.
+- `--grade-run`: read a run request on stdin and print one value observation.
+
+`--grade-run` receives:
+
+```json
+{
+  "schemaVersion": 1,
+  "run": {
+    "id": "12345",
+    "attempt": 1,
+    "repository": "OWNER/REPO",
+    "workflow": "Workflow name",
+    "ref": "refs/heads/main",
+    "sha": "...",
+		"eventName": "schedule",
+		"createdAt": "2026-08-23T11:58:00Z"
+  },
+  "evidenceAt": "2026-08-23T12:00:00.000Z",
+  "case": null,
+  "event": null,
+  "config": {}
+}
+```
+
+It returns:
+
+```json
+{
+  "value": 0.75,
+  "opportunityKey": "issue:42",
+  "case": {"issue": 42},
+  "evidenceCutoff": "2026-08-23T12:00:00.000Z",
+  "maturesAt": "2026-08-30T12:00:00.000Z",
+  "provenance": [
+    {"repository": "OWNER/REPO", "kind": "issue", "ref": "42"}
+  ],
+  "diagnostics": {}
+}
+```
+
+The function must cap `evidenceCutoff` at the earlier of `evidenceAt` and `maturesAt`. A run is never intrinsically pending: the value is an as-of observation and may be recomputed until maturity. After maturity, the cap makes the result stable.
+
+## Regrade a Historical Run
+
+Recompute a run at an explicit evidence time with the same evaluator used by the original run:
+
+```bash
+gh aw graders operational-value RUN-ID \
+  --evidence-at 2026-08-30T12:00:00.000Z \
+  --json
+```
+
+Add `--repo [HOST/]OWNER/REPO` to select the GitHub host for the current repository checkout. The command downloads the original grader artifact, reuses its operational case and complete run subject, and refuses to execute unless the archived evaluator matches both digest records and the evaluator at the recorded commit in the trusted checkout. It prints a new observation and never modifies the original artifact.
+
+## Definition Contract
+
+`--definition` must contain:
+
+- `schemaVersion: 4` and `grader: "operational-value"`;
+- repository, workflow name, source path, and adoption commit/time;
+- operational-value statement;
+- evidence opportunity, assignment, accepted evidence, repositories, collection, maturation, zero rule, and missing rule;
+- one primary metric with formula and validation examples;
+- baseline mode, value, cutoff, and provenance.
+
+For `baseline-comparable`, baseline value must be in `[0,1]` and have immutable provenance. For `attainment-only`, baseline value and cutoff must be null.
+
+## Interpretation Rules
+
+- `value` answers “how fully was this run's assigned opportunity attained?”
+- `deltaFromBaseline` answers “how far is this observation above or below the frozen pre-adoption reference?”
+- Neither establishes that the workflow caused the outcome.
+- Compare runs only under the same evaluator digest and evidence horizon.
+- Identify a replayed observation by `(runId, evaluatorDigest, evidenceAt)`.
+- Do not treat repeated observations of one run, duplicate opportunity keys, or overlapping state windows as independent samples.

.github/skills/aw-value/scripts/operational-value-evaluator-path.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+fail() {
+    printf 'error: %s\n' "$*" >&2
+    exit 1
+}
+
+[[ $# -eq 1 ]] || fail "usage: operational-value-evaluator-path.sh WORKFLOW-NAME"
+
+workflow_name=$1
+[[ $workflow_name =~ ^[a-z0-9]+(-[a-z0-9]+)*$ ]] \
+    || fail "workflow name must contain lowercase letters, numbers, and single hyphens"
+
+printf '.github/graders/%s-operational-value.sh\n' "$workflow_name"

.github/skills/aw-value/scripts/verify-operational-value-evaluator.sh

@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+fail() {
+    printf 'error: %s\n' "$*" >&2
+    exit 1
+}
+
+[[ $# -eq 1 ]] || fail "usage: verify-operational-value-evaluator.sh <operational-value-evaluator.sh>"
+
+evaluator=$1
+[[ -f $evaluator ]] || fail "operational-value evaluator not found: $evaluator"
+[[ -x $evaluator ]] || fail "operational-value evaluator is not executable: $evaluator"
+command -v jq >/dev/null 2>&1 || fail "jq is required"
+bash -n "$evaluator"
+
+definition=$("$evaluator" --definition)
+printf '%s\n' "$definition" | jq -e '
+    .schemaVersion == 4
+    and .grader == "operational-value"
+    and (.repository | type == "string" and test("^[^/]+/[^/]+$"))
+    and (.workflowName | type == "string" and length > 0)
+    and (.sourcePath | type == "string" and startswith(".github/workflows/") and endswith(".md"))
+    and (.adoption.commit | type == "string" and test("^[0-9a-f]{40}$"))
+    and (.adoption.adoptedAt | type == "string" and test("^[0-9]{4}-[0-9]{2}-[0-9]{2}T"))
+    and (.operationalValue | type == "string" and length > 0)
+    and (.evidence.opportunity | type == "string" and length > 0)
+    and (.evidence.assignment | type == "string" and length > 0)
+    and (.evidence.accepted | type == "string" and length > 0)
+    and (.evidence.repositories | type == "array" and length > 0)
+    and (all(.evidence.repositories[]; type == "string" and test("^[^/]+/[^/]+$")))
+    and (.evidence.collection | type == "string" and length > 0)
+    and (.evidence.maturation | type == "string" and length > 0)
+    and (.evidence.zeroRule | type == "string" and length > 0)
+    and (.evidence.missingRule | type == "string" and length > 0)
+    and (.primaryMetric.id | type == "string" and length > 0)
+    and (.primaryMetric.formula | type == "string" and length > 0)
+    and (.primaryMetric.direction == "higher_is_better")
+    and (.validationExamples | has("targetAttained") and has("targetMissed") and has("missing") and has("malformed"))
+    and (.baseline.mode == "baseline-comparable" or .baseline.mode == "attainment-only")
+    and (if .baseline.mode == "baseline-comparable" then
+        (.baseline.value | type == "number" and . >= 0 and . <= 1)
+        and (.baseline.evidenceCutoff | type == "string" and length > 0)
+        and (.baseline.provenance | type == "array" and length > 0)
+      else
+        .baseline.value == null
+        and .baseline.evidenceCutoff == null
+      end)
+' >/dev/null || fail "operational-value evaluator definition is invalid"
+
+for example_name in targetAttained targetMissed missing malformed; do
+    evidence=$(printf '%s\n' "$definition" | jq -c --arg name "$example_name" '.validationExamples[$name]')
+    result=$(printf '%s\n' "$evidence" | "$evaluator" --metric)
+    printf '%s\n' "$result" | jq -e '. == null or (type == "number" and . >= 0 and . <= 1)' >/dev/null \
+        || fail "--metric returned an invalid score for $example_name"
+    case $example_name in
+        targetAttained) target_attained=$result ;;
+        targetMissed) target_missed=$result ;;
+        missing|malformed)
+            [[ $result == null ]] || fail "--metric must return null for $example_name"
+            ;;
+    esac
+done
+
+jq -en --argjson attained "$target_attained" --argjson missed "$target_missed" \
+    '$attained != null and $missed != null and $attained > $missed' >/dev/null \
+    || fail "targetAttained must score higher than targetMissed"
+
+repository=$(printf '%s\n' "$definition" | jq -r '.repository')
+workflow_name=$(printf '%s\n' "$definition" | jq -r '.workflowName')
+adoption_commit=$(printf '%s\n' "$definition" | jq -r '.adoption.commit')
+created_at=$(printf '%s\n' "$definition" | jq -r '.adoption.adoptedAt')
+evidence_at=2099-01-01T00:00:00Z
+request=$(jq -cn \
+        --arg repository "$repository" \
+        --arg workflow "$workflow_name" \
+        --arg sha "$adoption_commit" \
+        --arg createdAt "$created_at" \
+        --arg evidenceAt "$evidence_at" \
+        '{
+            schemaVersion: 1,
+            run: {
+                id: "1",
+                attempt: 1,
+                repository: $repository,
+                workflow: $workflow,
+                ref: "refs/heads/main",
+                sha: $sha,
+                eventName: "workflow_dispatch",
+                createdAt: $createdAt
+            },
+            evidenceAt: $evidenceAt,
+            case: null,
+            event: {},
+            config: {verification: true}
+        }')
+grade_run=$(printf '%s\n' "$request" | "$evaluator" --grade-run)
+printf '%s\n' "$grade_run" | jq -e --arg evidenceAt "$evidence_at" '
+        def timestamp:
+            type == "string"
+            and test("^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\\.[0-9]{3})?Z$");
+        def epoch: sub("\\.[0-9]{3}Z$"; "Z") | fromdateiso8601;
+        type == "object"
+        and (.value == null or (.value | type == "number" and isfinite and . >= 0 and . <= 1))
+        and (.opportunityKey | type == "string" and length > 0)
+        and (.case | type == "object")
+        and (.evidenceCutoff | timestamp)
+        and (.maturesAt | timestamp)
+        and ((.evidenceCutoff | epoch) <= ($evidenceAt | epoch))
+        and ((.evidenceCutoff | epoch) <= (.maturesAt | epoch))
+        and (.provenance | type == "array")
+        and (if .value == null then true else (.provenance | length > 0) end)
+        and (all(.provenance[]; type == "object"
+            and (.repository | type == "string" and length > 0)
+            and (.kind | type == "string" and length > 0)
+            and (.ref | type == "string" and length > 0)))
+        and ((has("diagnostics") | not) or (.diagnostics | type == "object"))
+        and ((has("message") | not) or (.message | type == "string"))
+' >/dev/null || fail "--grade-run returned an invalid operational-value observation"
+
+printf 'verified %s\n' "$evaluator"

.github/skills/aw-value/tests/test.sh

@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+skill_dir=$(CDPATH='' cd -- "$(dirname -- "$0")/.." && pwd)
+repo_root=$(CDPATH='' cd -- "$skill_dir/../../.." && pwd)
+work_dir=$(mktemp -d "$repo_root/.aw-value-test.XXXXXX")
+trap 'rm -rf "$work_dir"' EXIT HUP INT TERM
+
+path=$("$skill_dir/scripts/operational-value-evaluator-path.sh" daily-file-diet)
+[[ $path == .github/graders/daily-file-diet-operational-value.sh ]]
+if "$skill_dir/scripts/operational-value-evaluator-path.sh" ../escape >/dev/null 2>&1; then
+    printf 'invalid workflow name was accepted\n' >&2
+    exit 1
+fi
+
+evaluator_path="$work_dir/operational-value.sh"
+cat > "$evaluator_path" <<'EOF'
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+case ${1:-} in
+    --definition)
+        cat <<'JSON'
+{
+  "schemaVersion": 4,
+  "grader": "operational-value",
+  "repository": "owner/repo",
+  "workflowName": "Example",
+  "sourcePath": ".github/workflows/example.md",
+  "adoption": {
+    "commit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+    "adoptedAt": "2026-01-01T00:00:00Z"
+  },
+  "operationalValue": "For eligible issues, attain closure demonstrated by a closed issue.",
+  "evidence": {
+    "opportunity": "The issue assigned to the workflow run.",
+    "assignment": "Bind the triggering issue number to the run ID.",
+    "accepted": "The issue is closed by the evidence cutoff.",
+    "repositories": ["owner/repo"],
+    "collection": "Read immutable issue events through the capped cutoff.",
+    "maturation": "Seven days after the run was created.",
+    "zeroRule": "An eligible open issue at the cutoff scores zero.",
+    "missingRule": "An unavailable issue or event history scores null."
+  },
+  "primaryMetric": {
+    "id": "issue-closure",
+    "formula": "1 when closed, otherwise 0",
+    "direction": "higher_is_better"
+  },
+  "baseline": {
+    "mode": "baseline-comparable",
+    "value": 0.4,
+    "evidenceCutoff": "2025-12-31T00:00:00Z",
+    "provenance": [{"repository": "owner/repo", "kind": "issue-events", "ref": "baseline"}]
+  },
+  "validationExamples": {
+    "targetAttained": {"eligible": true, "closed": true},
+    "targetMissed": {"eligible": true, "closed": false},
+    "missing": {"eligible": false},
+    "malformed": {"eligible": "yes"}
+  }
+}
+JSON
+        ;;
+    --metric)
+        jq 'if (.eligible | type) != "boolean" or .eligible == false or (.closed | type) != "boolean" then null elif .closed then 1 else 0 end'
+        ;;
+    --grade-run)
+      request=$(cat)
+      printf '%s\n' "$request" | jq -e '
+        .schemaVersion == 1
+        and .run.id == "1"
+        and .run.repository == "owner/repo"
+        and .run.eventName == "workflow_dispatch"
+        and .case == null
+        and .config.verification == true
+      ' >/dev/null
+      printf '%s\n' "$request" | jq -c '{
+        value: 1,
+        opportunityKey: "verification:1",
+        case: {verification: true},
+        evidenceCutoff: .evidenceAt,
+        maturesAt: .evidenceAt,
+        provenance: [{repository: .run.repository, kind: "verification", ref: .run.id}],
+        diagnostics: {}
+      }'
+      ;;
+    *)
+        exit 1
+        ;;
+esac
+EOF
+chmod +x "$evaluator_path"
+
+"$skill_dir/scripts/verify-operational-value-evaluator.sh" "$evaluator_path" >/dev/null
+printf 'aw-value skill tests passed\n'