Capstone AI checker and track certificates for the Dev Weekends levels

docs/16-capstone-checker-and-certificates.mdx

@@ -0,0 +1,273 @@
+---
+title: Capstone checker and certificates
+status: approved 2026-11-06, built
+---
+
+# Capstone checker and certificates
+
+**Reads:** [`05-review-queue.mdx`](05-review-queue.mdx), [`09-security.mdx`](09-security.mdx), [`15-content-separation.mdx`](15-content-separation.mdx)
+
+For the Dev Weekends guided-lab tracks (Level 1, 2 and 3). A learner submits a capstone, our
+own AI says how complete it is and what is left, a mentor confirms, and a passed track earns a
+certificate the learner can add to LinkedIn.
+
+## What the learner sees
+
+1. On the capstone chapter, a **Capstone check** block asks for a GitHub repo URL (and a live URL
+   for Level 2).
+2. They press **Check my capstone**. Within about a minute they see:
+   - **Complete** or **Not complete yet**, and a percentage.
+   - **What is left**: one card per unmet criterion, each with numbered steps.
+   - **Quality notes**: what is good and what would make it better.
+   - A line saying a mentor will confirm the result.
+3. They fix things and press **Check again**. Every attempt is kept as history.
+
+## What the mentor sees
+
+A check reaches the existing review queue when the AI says it is complete, or when the learner
+presses **Ask my mentor to look**. There is one waiting item per learner per capstone: a newer
+check replaces the one already waiting. The review page shows the full report: automated check
+results, per-criterion scores with the evidence the model cited, the commit, the model, tokens
+and cost, plus every earlier check. The review form is prefilled from the AI result, so
+**Approve** confirms it, and changing the feedback, score or verdict overrides it. This writes
+the ordinary `Review` row, so the queue, notifications and needs-revision loop work as today. **The AI never passes a learner on
+its own.** Only a mentor-confirmed pass counts toward a certificate.
+
+## How a check runs
+
+```
+submit -> automated checks -> (stop early if a hard gate fails) -> Sonnet grade -> report
+```
+
+**Automated checks** (no model, no cost), defined per pack:
+
+| Check | How |
+| --- | --- |
+| Repo reachable | GitHub API: repo exists and is public, default branch, head commit SHA |
+| Builds and tests pass | Status of the latest GitHub Actions run on that commit. We never run learner code on our servers. Each capstone template ships a CI workflow. |
+| Tags | Required tag pattern exists (for example `v1.*`) |
+| Files | Required paths exist (README, workflow, tests folder) |
+| Live URL responds | HTTPS GET returns 2xx within 5 s (Level 2) |
+| Live page contains | Optional text on the live page (for example the analytics snippet) |
+| Course files unchanged (`files_match`) | Named files (workflow, supplied tests) are byte-identical, line endings aside, to the template at a tag: `against: https://github.com/dw/template@v1`. Stops the run by default, so an edited workflow or test cannot fake a green build. |
+| File contains (`file_contains`) | A file has an expected line, for example a pinned `uses:` of a course-owned reusable workflow |
+| Named check run (`check_run`) | One named GitHub check concluded with success on the commit, so a number can come from CI rather than a committed file |
+
+Any check can set `stop: true` or `stop: false`; by default the repo, CI and course-file checks stop the run.
+If the repo is unreachable, CI is red or a course file was changed, the learner gets instant steps and **no model call is
+made**.
+
+**AI grade.** We send the rubric pack, the automated results, and a bounded snapshot of the repo
+(file tree, README, and the files the pack names, trimmed to a token budget) to
+`claude-sonnet-5-5` with a JSON output schema. The output contract:
+
+```ts
+{
+  completed: boolean,          // all required criteria met and percent >= pack.passPercent
+  percent: number,             // 0..100, weighted from criterion scores
+  criteria: [{ id, score, max, met, evidence: string[], missing: string[] }],
+  whatIsLeft: [{ criterionId, title, steps: string[] }],
+  qualityNotes: { strengths: string[], improvements: string[] },
+  confidence: "high" | "medium" | "low",
+  integrityFlags: string[]     // mentor only; any flag sends the check to the mentor
+}
+```
+
+The server recomputes `percent` and `completed` from the criterion scores and the hard gates,
+so the model cannot claim 100% with a failing gate. A criterion tied to a failed automated
+check is capped at 0.
+
+**Same commit, same pack version: no new call.** The previous result is returned free.
+
+## Cost
+
+Sonnet 5.5 is $2 per million input tokens and $10 per million output tokens. A typical check is
+about 30k input and 4k output, so **about $0.10**. Caps, all counted in the database like the
+chapter Q&A limits:
+
+| Limit | Default | Env |
+| --- | --- | --- |
+| Input budget per check | 40k tokens (snapshot trimmed to fit) | `CAPSTONE_MAX_INPUT_TOKENS` |
+| Output per check | 8k tokens | `CAPSTONE_MAX_OUTPUT_TOKENS` |
+| Worst case per check | about $0.16 | |
+| Checks per learner per block per day | 3 | `CAPSTONE_LEARNER_DAILY_LIMIT` |
+| Checks platform-wide per day | 150 (about $15 worst case) | `CAPSTONE_GLOBAL_DAILY_LIMIT` |
+
+It uses the same `ANTHROPIC_API_KEY` as the chapter questions and is on whenever that key is set
+(`CAPSTONE_CHECK_ENABLED=false` switches it off). Only courses with a capstone block use it. Tokens and cost are
+stored on every check.
+
+## Rubric packs
+
+Packs live with the course in Edorax-core and travel through the normal publish path. The
+importer reads them, validates them, and stores them in the block's server-only config.
+
+```
+courses/<course-slug>/capstone/
+  rubric.yaml
+  calibration/<example>.yaml
+  calibration/<example>/         # frozen repo snapshot for the example (optional)
+```
+
+```yaml
+# rubric.yaml
+pack: dw-level-2
+version: 1
+title: Personal site and outreach engine
+passPercent: 70
+submission: { repo: required, liveUrl: required }
+automated:
+  - { check: repo_public }
+  - { check: ci_green }
+  - { check: tag, pattern: "v1.*" }
+  - { check: files, paths: [README.md, ".github/workflows/*.yml"] }
+  - { check: live_ok }
+  - { check: live_contains, text: "data-analytics" }
+snapshot:                       # which files the grader reads, in priority order
+  include: ["README.md", "src/**", "docs/**"]
+  exclude: ["**/*.lock", "**/node_modules/**"]
+criteria:
+  - id: proposal-links
+    title: Tracked proposal links
+    weight: 20
+    required: true
+    gate: [live_ok]             # capped at 0 if these automated checks fail
+    lookFor: Each proposal has its own URL and visits are recorded per link.
+    levels:
+      0: No proposal links.
+      2: Links exist but visits are not tracked.
+      4: Links tracked and visible in an analytics view.
+oralExam: { enabled: false }    # Level 3 hook, see below
+```
+
+```yaml
+# calibration/strong-submission.yaml
+repo: https://github.com/example/site@3f2a9c1   # pinned commit, or
+snapshot: ./strong-submission/                    # frozen files plus automated.json
+liveUrl: https://example.dev
+expected: { completed: true, percent: 85, tolerance: 10 }
+note: Clean site, tracked links, weak README.
+```
+
+The block in a chapter is a fence; the importer reads the pack file, validates it and stores it
+in the block's server-only config (calibration files never ship to learners):
+
+````md
+```capstone
+title: Ship your personal site
+intro: Paste your repo and your live site. The check takes about a minute.
+pack: capstone/rubric.yaml
+```
+````
+
+Check ids default to the check name (`live_ok`), so a criterion gates on `[live_ok]`. Give an
+id when a pack has two checks of the same kind. `passLevel` (default: full marks) sets the level
+that counts as met. A snapshot example can include `live.html` so calibration never touches the
+network.
+
+## Calibration test
+
+- **CI (free):** loader and schema tests for every pack, plus grading-pipeline tests with a
+  recorded model response, so gates, recomputed percent and caps are covered without the API.
+- A course with several packs (`capstone/capstone-1.yaml`, `capstone-2.yaml`) keeps each pack's
+  examples in `calibration/<pack name>/`; a lone `capstone/rubric.yaml` uses `calibration/`.
+- **Live (paid, run by hand):** `pnpm --dir server capstone:calibrate --pack <dir>` runs every
+  calibration example through the real checker and fails if `completed` differs or `percent`
+  falls outside `tolerance`. It prints the cost before and after. Run it when a pack changes.
+
+## Certificates
+
+**When:** a learner's enrollment is complete and every capstone check in the course has a
+mentor-confirmed pass. Issued once per enrollment; a mentor or admin can revoke.
+
+**What it says:** the learner's name, the track title (for example "Level 2: Personal Site and
+Outreach Engine"), "Dev Weekends", the issue date, the skills covered, a unique id such as
+`DW-L2-7K3F9QXM`, and the course's trainer as issuer: the instructor name the catalog shows for
+the course (Zeeshan Adil for the Dev Weekends tracks). `certificate.issuer.name` in course.yaml
+overrides it, for example Aqib Nawab on the multi-vendor marketplace course.
+
+**Configured per course** in `course.yaml`:
+
+```yaml
+certificate:
+  program: Dev Weekends Mentorship
+  title: "Level 2: Personal Site and Outreach Engine"
+  code: L2
+  skills: [Personal site, Outreach engine, Link tracking, Web analytics]
+```
+
+**Public verify page** at `/verify/<id>`: no sign-in, shows the certificate, its status (valid
+or revoked), skills, and the capstone repo link. It sets Open Graph tags so a shared link shows
+the certificate image.
+
+**Add to LinkedIn:** one button that opens LinkedIn's add-to-profile form pre-filled:
+
+```
+https://www.linkedin.com/profile/add?startTask=CERTIFICATION_NAME
+  &name=<title>&organizationName=Edorax&issueYear=2026&issueMonth=11
+  &certUrl=<verify url>&certId=<id>
+```
+
+Plus **Share on LinkedIn** (post with the verify link) and **Download image** (1200 x 627 PNG
+rendered on the server with `next/og`).
+
+**Look:** Edorax type (Unbounded headings, DM Sans body), white card with a 1px line border and
+square corners, ink text on paper, no gradients, no ribbons, no seals, no script fonts. The Dev
+Weekends wordmark with its seven week bars carries the issuer mark in black and white. One
+cobalt element at most (the verify link). It should read like a well set document, not a
+template.
+
+## Level 3 oral exam (later phase)
+
+Each check is a list of stages: `automated`, `ai_grade`, and later `oral_exam`. A pack with
+`oralExam.enabled: true` will require a passed oral stage, built on the existing AI interview
+block, before the mentor can confirm. Phase 1 ships the stage interface and the pack field only.
+
+## Security
+
+- Fetching learner URLs is new for Edorax (the review queue doc says we do not). Rules: HTTPS
+  only, DNS resolved and private, loopback, link-local and metadata ranges refused (also after
+  redirects), GET only, no cookies, 5 s timeout, 512 KB cap.
+- GitHub is read through its API (repo, commit, tags, CI) and its raw file host (file contents), with no token by default.
+- Repo text is untrusted. The prompt treats it as data, output is schema-bound, the server
+  recomputes the score, gates are enforced in code, and a mentor confirms every pass.
+- Rubric packs stay server-only, like every block config.
+
+## Data model
+
+- `BlockType.CAPSTONE_CHECK`.
+- `CapstoneCheck`: one row per attempt, linked to the `Response` it creates. Holds repo URL,
+  live URL, commit SHA, pack id and version, status (`QUEUED`, `RUNNING`, `DONE`, `FAILED`),
+  automated results, AI report, percent, completed, model, tokens, cost. Mentor decisions use the
+  existing `Review`.
+- `Certificate`: public id, user, course, enrollment (unique), recipient name and title snapshot,
+  skills, issued at, revoked at and reason.
+
+## Needs keys or approval
+
+| Item | Who |
+| --- | --- |
+| Nothing for the AI key: the checker uses the `ANTHROPIC_API_KEY` already set for chapter Q&A | None |
+| `GITHUB_TOKEN` is optional. Files come from GitHub's raw host, so a check uses about five API calls and the 60 an hour limit without a token covers about 12 checks an hour. A read-only token raises that to 5,000 calls if needed | Optional |
+| LinkedIn lists Edorax, never Dev Weekends, as the issuer. When the platform has its own LinkedIn page, set its id as `CERTIFICATE_LINKEDIN_ORG_ID`; until then LinkedIn gets "Edorax" as plain text | Later |
+| Optional title under the trainer's name: `certificate.issuer.title` in course.yaml | Optional |
+| Daily caps above (env, defaults in the cost table) | User |
+| One live calibration run per level pack (`pnpm capstone:calibrate --pack <rubric.yaml> --live --record`, about $0.20 per example) | Level sessions, with the API key |
+| Each level's capstone template ships a CI workflow that builds and runs tests | Level sessions |
+
+## Known limits and follow-ups
+
+- The desktop app's reader does not draw the capstone block yet; it needs the same block in
+  `packages/learning-ui` (the web app has it). Learners can use the web app meanwhile.
+- Certificate eligibility counts every capstone block in the course. It does not yet filter by the
+  learner's course variant or release.
+- Claude's server-side refusal fallback is not enabled. A refused grade fails the check with a
+  message telling the learner to send it to their mentor.
+- Checks run inside the API process. A restart marks running checks as interrupted after five
+  minutes so the learner can check again. A job queue can replace this if volume grows.
+- Level 3's oral exam stage is the next phase (see above).
+- Asked for by Level 2, not built yet: the checker's own screenshots of the live URL at 375 and
+  1280 px for the grader. It needs a headless browser outside the API process.
+- Asked for by Level 1, not built yet: reading CI for the commit a release tag points at, instead of
+  the default branch head.
+

frontend/src/app/(teach)/review/[responseId]/page.tsx

@@ -6,6 +6,7 @@ import { serverApiFetchOrNotFound } from "@/lib/api-server";
 import { submitReview } from "@/server/actions/reviews";
 import { ReviewForm } from "@/components/teach/review-form";
 import { QuizIntegrity } from "@/components/teach/evaluation/quiz-integrity";
+import { CapstoneReport, capstoneDraftFeedback } from "@/components/teach/evaluation/capstone-report";
 
 export default async function ReviewDetailPage({
   params,
@@ -21,6 +22,8 @@ export default async function ReviewDetailPage({
   const { course, question, history } = response;
   const alreadyReviewed = response.status !== "PENDING_REVIEW";
   const reviewResponseId = response.id;
+  const capstone = response.capstone;
+  const capstonePercent = capstone?.current?.report?.percent ?? null;
 
   async function submitReviewAction(formData: FormData) {
     "use server";
@@ -63,8 +66,9 @@ export default async function ReviewDetailPage({
       </div>
 
       <div className="mt-6 grid gap-6 lg:grid-cols-2">
-        {/* Left: the question, rubric, and reference answer — mentor-only config */}
-        <section className="rounded-xl border bg-card p-5 text-card-foreground shadow-sm">
+        {capstone ? <CapstoneReport capstone={capstone} /> : null}
+        {/* Left: the question, rubric, and reference answer, mentor-only config */}
+        {capstone ? null : <section className="rounded-xl border bg-card p-5 text-card-foreground shadow-sm">
           <h2 className="text-sm font-semibold">Question</h2>
           <p className="mt-2 text-[15px] font-medium leading-snug">
             {question?.prompt ?? "(prompt unavailable)"}
@@ -108,11 +112,11 @@ export default async function ReviewDetailPage({
               ))}
             </div>
           )}
-        </section>
+        </section>}
 
         {/* Right: the mentee's answer + review form */}
         <section className="flex flex-col gap-4">
-          <div className="rounded-xl border bg-card p-5 text-card-foreground shadow-sm">
+          {capstone ? null : <div className="rounded-xl border bg-card p-5 text-card-foreground shadow-sm">
             <h2 className="text-sm font-semibold">Mentee&apos;s answer</h2>
             <p className="mt-2 whitespace-pre-wrap text-[15px] leading-relaxed">
               {response.answerText}
@@ -125,7 +129,7 @@ export default async function ReviewDetailPage({
                 </a>
               </p>
             ) : null}
-          </div>
+          </div>}
 
           {alreadyReviewed ? (
             <div className="rounded-xl border border-emerald-500/30 bg-emerald-500/10 p-4 text-sm text-emerald-700 dark:text-emerald-300">
@@ -141,6 +145,14 @@ export default async function ReviewDetailPage({
               responseId={reviewResponseId}
               maxScore={response.block.points}
               action={submitReviewAction}
+              {...(capstone
+                ? {
+                    aiDraft: false,
+                    initialFeedback: capstoneDraftFeedback(capstone.current),
+                    initialScore: capstonePercent === null ? null : Math.round((capstonePercent / 100) * response.block.points),
+                    note: "Prefilled from the AI check. Approve to confirm it, or change the feedback, score or verdict to override it. Only an approval counts toward the certificate.",
+                  }
+                : {})}
             />
           )}
         </section>

frontend/src/app/verify/[code]/certificate-data.ts

@@ -0,0 +1,24 @@
+import "server-only";
+import { cache } from "react";
+import { ApiError } from "@/lib/api-client";
+import { serverApiFetch } from "@/lib/api-server";
+import type { CertificateView } from "@/shared/lib/certificates";
+
+/** Public lookup, never with the visitor's session. Null for an unknown id. */
+export const loadCertificate = cache(async (code: string): Promise<CertificateView | null> => {
+  try {
+    const result = await serverApiFetch<{ certificate: CertificateView }>(`/api/v1/certificates/verify/${encodeURIComponent(code)}`, { token: null });
+    return result.certificate;
+  } catch (error) {
+    if (error instanceof ApiError && error.status < 500) return null;
+    throw error;
+  }
+});
+
+export function issuedOn(iso: string): string {
+  return new Date(iso).toLocaleDateString("en-GB", { day: "numeric", month: "long", year: "numeric", timeZone: "UTC" });
+}
+
+export function issuerLine(certificate: Pick<CertificateView, "issuerName" | "issuerTitle" | "organization">): string {
+  return certificate.issuerTitle ? `${certificate.issuerName}, ${certificate.issuerTitle}` : certificate.issuerName;
+}