diff --git a/openspec/changes/runtime-demo-path/.openspec.yaml b/openspec/changes/runtime-demo-path/.openspec.yaml new file mode 100644 index 00000000..b4b3ece7 --- /dev/null +++ b/openspec/changes/runtime-demo-path/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-09-01 diff --git a/openspec/changes/runtime-demo-path/design.md b/openspec/changes/runtime-demo-path/design.md new file mode 100644 index 00000000..5f7ec9f4 --- /dev/null +++ b/openspec/changes/runtime-demo-path/design.md @@ -0,0 +1,349 @@ +## Context + +The runtime tier has three parts, and only two of them can be tested from one +repository. Discovery, signing, validation and writing are ours. Generation is +the service's. The handshake between them — a signature the service blessed +over bytes this client then wrote — belongs to neither and is tested by +neither. + +The payload alignment work made that concrete. Every defect it surfaced lived +in the seam and was found by a person noticing: a runtime rule below the +file-set floor admitted rather than withheld, a signature attached to the wrong +rule, a capture dropped for want of a name, a rule written and never executed. +An empty scan reports success, so the symptom was almost always "no findings" +rather than an error. + +`check` is the wrong place to close that gap. It reports on a user's project, +and the whole of the last change was about it not doing anything else. + +## Goals / Non-Goals + +**Goals:** + +- One deliberately-invoked command that walks generation, delivery, writing and + verification, and says what happened at each step. +- Run on the paths a user runs on, so what it proves is what users get. +- Work with no login, so it can be shown to someone who does not have an + account yet. +- Leave the project recoverable: whatever it writes can be removed. + +**Non-Goals:** + +- Not a correctness test of the generated rule. The rule may be poor; that is + acceptable and `rule delete` removes it. +- Not a replacement for reconcile, and not a second way to get rules. +- Not reachable by routing. An agent deciding how to author a rule must never + land here. +- Not a demonstration of execution. See D3. + +## Decisions + +### D1 — The demo uses the well-known ticket and retrieval formats + +`POST /cli/api/demo/request` returning `{ requestId, status }`, then +`GET /cli/api/demo/request/{requestId}` returning `{ requestId, status, rules[] }`. The +same two-stage flow as `rule create`, the same status enum, the same file-set +variant. + +The alternative — one endpoint returning a rule directly — is smaller and +wrong. A demo exists to show the real path, so a payload shape that only the +demo uses would demonstrate something no user is on. Mirroring also means the +demo covers polling and the terminal `failed` and `unsupported` states for +free, because it is the same client code reaching them. + +Concretely, and the three are not equal — saying "reuse" without splitting them +would make the failure test below fire for a reason it was not built to catch. + +**`writeRuleFile` is reused byte for byte, and it is what the test is about.** +It carries the path checks, the completeness rules, the layout validation and +the refusal to write a runtime rule with no captures. **If the demo needs a +different writer, the payload shapes have diverged**, and the demo has done its +job by failing. That is the signal worth having, and it is the whole reason for +mirroring the delivery variant. + +**`submitRule` and `pollRuleStatus` need work first, and that is a task rather +than a divergence signal.** Both hardcode their endpoint as a literal, and both +take a required `token`, which `createApiClient` turns unconditionally into an +`Authorization: Bearer` header. The demo is unauthenticated by requirement (D3), +so neither is usable as it stands: the endpoint has to be a parameter, and the +token has to be optional. + +**An earlier draft said the path half "resolves itself when the client adopts +the renamed family". It did not, and the correction is worth keeping.** The +client has since adopted it — those two functions now call `/cli/api/request` +and `/cli/api/request/{requestId}` — and they are no less hardcoded for it. The +demo's endpoint is `/cli/api/demo/request` either way, so what stood between the +demo and reuse was never which noun the literal spelled; it was that the literal +is not a parameter. Both halves are ours, and both are planned in task 1.2a +rather than discovered during implementation. + +Neither is evidence of anything having gone wrong. They are two small changes to +functions written when every caller was authenticated and every caller wanted +the same route. + +The distinction matters because D1's failure test is a **claim about payload +shapes**. A requester that needs a second argument says nothing about whether +the demo and the mainline agree on what a rule looks like. + +### D2 — The demo tracks the mainline's naming, and follows rather than leads + +The paths above mirror the mainline's naming, whatever it currently is. The +demo must never be where a second convention lives. Its value is being +indistinguishable from the path users take; a demo that is tidier than +production is a demo that stops proving anything. If the mainline is renamed +again, these are renamed with it. + +**That principle already cost this change a wait, which is the evidence it is +real.** The generator team renamed the request resource from `rule`/`ruleId` to +`request`/`requestId` as its own change (**N10**, the inconsistency being that +`ruleId` named a ticket, by the service's own field description). That rename +has landed and is verified live, so the demo takes the settled noun above. +Naming the demo endpoints before it landed would have made the demo the debut +of the new convention, which is precisely what this decision forbids. + +Adopting the new noun in the _ordinary_ client was a separate change and was +never this one's dependency. `/cli/api/rule/*` still serves, marked +`deprecated`, so the mainline client was not broken by having not moved yet, and +the demo, being new, had no reason to be born on the deprecated spelling. That +separate change has since landed as well, so the two agree; the Risks section +records what the move taught. + +### D2a — The fixed input is data the service holds, not a repository + +An earlier draft of this proposal said "a public repository the service +controls". That was wrong, and the generator team corrected it: CLI-bound +generation never clones. The clone is deferred to the git-bound push, which +happens only for a pull request, and `repositoryUrl` on a CLI request is +authorization scoping and a ticket field rather than model input. + +So the fixture is a prompt and its examples, held as data alongside the +rule-hash vectors. Naming a repository would have been decoration, and +decoration is what a later reader tries to make load-bearing. + +### D3 — It stops at `verify`, and that is stated rather than worked around + +A runtime rule executes only when an authenticated reconcile returns its +signature in `run`. The demo is unauthenticated, so it cannot reach execution: +`check` skips the rule and reports "not authenticated — runtime rules were not +verified and did not run." + +Three options existed. Stopping at `verify` is honest and shows generation, +delivery, the file set, and validation. `--dangerously-run-scripts` reaches +execution and prints the warning it always prints, which is truthful and +conspicuously not the signature story. A client-side bypass keyed on the demo +rule id would show the real thing and is refused. + +That third option is the one worth writing down, because it is the tempting +one. The signature gate is the only thing between a payload downloaded over the +network and arbitrary code running on a developer's machine. A hole in it that +exists "only for the demo" is a hole, and the id it keys on is attacker-visible +in the recipe. No demo is worth that. + +The consequence is that this change demonstrates everything up to the gate. + +**An earlier draft of this section said logging in was "the next beat". That is +wrong, and the correction matters more than the wording.** Blessing is +recording: a signature exists for a rule because that rule was written into an +organization's corpus. The demo rule is one fixed rule, served to every caller +from a Taskless-owned installation, and it is never recorded for the caller. So +there is no signature for it in any caller's `run` set, and authenticating adds +nothing — the gate is not a step the demo stops just short of, it is unreachable +by construction from a shared fixture. + +That also disposes of the idea that the demo half-serves the blessed-execution +handshake and should be extended to serve it fully. It cannot. Exercising "we +bless, you run" needs a rule recorded for the organization that runs it, which +is what an ordinary authored rule already is. If that seam wants automated +coverage, the honest shape is an integration test against an organization we +own — not a demo — and it should be argued on those terms rather than folded in +here. Until someone does, the seam is covered by unit tests on each side and by +nothing that spans them, and this change records that rather than obscuring it. + +### D4 — Hidden by omission, not by a new mechanism + +`RECIPE_TOPICS` in `commands/agent.ts` is a hand-maintained literal; `getRecipe` +looks up by filename. A recipe file that is not in the list is therefore +fetchable by name and absent from the index already. + +Adding a hiding mechanism — a naming convention the loader understands, a +metadata field — would be new code to serve a property the existing design +already has. The `__` prefix in the topic name is for human readers, not for +the loader. + +The cost is that the index is curated by hand and can drift from the files on +disk. That is already true, and `cli-agent` gains a requirement saying it is +intended rather than an oversight. + +### D4a — The demo command nests under `rule`, so the second listing never sees it + +`taskless agent` prints **two** lists, and omission from `RECIPE_TOPICS` only +governs one of them. The first, headed `Topics:`, is built by iterating +`Object.entries(subCommands)` — the top-level command tree from `src/index.ts`, +gated by `SUBCOMMAND_NAMES` in `commands/names.ts` — and it filters exactly one +name, `agent` itself. Nothing in it consults `RECIPE_TOPICS`. The second, +headed `Authoring recipes:`, is `RECIPE_TOPICS`, and that is the list D4 is +about. + +So a demo registered as a new top-level verb would be hidden as a recipe and +advertised as a command in the same output, which is the opposite of the +property this change wants. **The demo command is therefore `rule demo`, a +subcommand of the existing `rule` command**, alongside `create`, `improve`, +`meta` and `delete`. It writes a rule, so it reads correctly there, and it adds +no entry to `SUBCOMMAND_NAMES` or `subCommands` — meaning the `Topics:` listing +is untouched and no filtering code has to be written to keep it that way. + +This is the same reasoning as D4 rather than an exception to it: the property is +obtained by not registering the thing, not by teaching a renderer to skip it. +The consequence is that `taskless rule --help` does list `demo`, from citty's +own usage output. That is intended. The requirement is that **routing** never +sends an agent to the demo, not that the demo is a secret — a person reading +`rule --help` has already chosen to look, and D3's risks say the same about the +recipe. + +### D5 — Failure is reported, never fabricated + +If generation cannot be served, the service answers with a terminal status and +a reason, and the demo prints it. The client already prefers the service's +reason over its own text for `unsupported`, so this needs nothing new. + +A demo that invents a rule when generation fails would be worse than a demo +that fails, because it would look like success. + +### D6 — The findings ship as `Finding[]`, grouped by the example that produced them + +**This shape is under revision and is not yet agreed with the service.** Task +0.4 — telling the service what findings shape we want — is open, so what follows +is the ask rather than the contract. The retrieval shape the two sides have +settled is D1's, and it does not include this. + +The service offered either its harness's `Finding[]` or something narrower, and +left the shape to us since we render it. + +`Finding[]` as-is, because it is not the service's internal type: `Finding` is +declared in `types/runtime-rule.ts` as part of the runtime-rule contract, and is +what a check returns on this side. Choosing a narrower demo-only object would be +introducing a payload only the demo consumes, which is the thing D1 exists to +forbid. The same reasoning that reuses `submitRule`, `pollRuleStatus` and +`writeRuleFile` applies to the type a check's results already have. + +What the demo does add is a wrapper, and the wrapper is the part carrying the +demonstration: + +``` +examples: [ { name, expectation: "fails" | "passes", findings: Finding[] } ] +``` + +A flat list of findings cannot distinguish a rule that catches the failing +examples from one that fires on everything it is shown, and those two render +identically as "3 findings". The second is the recurring defect in this area +wearing a success costume — an empty or indiscriminate scan reporting as a pass. +Attributing each finding to an example, and stating what that example was +expected to do, makes the asymmetry the thing a reader sees: findings on the +examples that should fail, none on the examples that should pass. + +That also makes the demo checkable rather than merely viewable. The CLI can +assert the asymmetry instead of printing whatever arrives, so a demo that +silently stops finding anything fails rather than looking clean. + +### D7 — Each thing the demo cannot do is the behaviour we want, not a shortfall + +D3 and D6 describe the demo negatively: it cannot reach the gate, and it must +fail rather than render an indiscriminate rule. Read together they invite the +wrong conclusion, that the demo is a reduced version of something better. It is +not. Every limit is a correct behaviour arriving through the ordinary path. + +**The rule is real and inspectable.** A developer gets `check.ts` and its +captures on disk and can read what a runtime rule actually is. That is the +demonstration. Note what is _not_ on disk: a signature. `run-set.ts` computes +one from the check's bytes at reconcile time; delivery never writes a signature +artifact. So there is nothing to inspect there and nothing to tamper with, and +the demo's value is the rule's content rather than a token. + +**It is inert, and inert is the correct state.** An ordinary `check` skips it +because no signature for it is in any organization's `run` set. A developer who +runs `check` and sees the rule skipped is watching the gate work, on the +ordinary code path, with the reason `check` already prints. Nothing about the +demo is special-cased to produce that. + +**It is removable, through the delete flow we already own.** No demo-specific +teardown, no residue `verify` or `check` reports afterwards. + +**`improve` does not work on it, and should not.** Improvement resolves a +request the caller's organization holds. The demo request is held by a +Taskless-owned installation and by no caller's organization, so improvement is +refused and nothing is written. That behaviour is the part this design rests on. + +**Which status the service returns for it is not settled, and this design no +longer asserts one.** An earlier draft said `RULE_NOT_FOUND`. That is a claim +about the service rather than about us, and the client's own handling makes it +consequential: `iterateRule` turns a 404 `request_not_found` into a `CLIError` +carrying `RULE_NOT_FOUND`, and turns a 403 `access_denied` into a plain `Error`, +which `improveCommand` reports as `NETWORK_ERROR`. A request that genuinely +exists and merely is not yours is 403-shaped rather than 404-shaped. If that is +what the service sends, `improve` on the demo rule reports `NETWORK_ERROR` and +tells an agent to retry an id that will never resolve — the exact +miscategorisation `iterateRule`'s own comment was written to prevent. Asserting +`NETWORK_ERROR` instead would trade one unconfirmed claim for another, so task +0.5 asks the service which status a ticket outside the caller's organization +returns, and the requirement states the behaviour until that answer exists. + +The deeper reason for refusing is worth stating separately, because it holds +whatever the status turns out to be: a pre-generated rule built against a shared +fixture is a bad base to iterate on. +Nothing generated in advance can match a rule authored against this developer's +own repository and context, so offering the demo rule as a starting point would +be offering a worse starting point wearing the demo's credibility. + +**This is correct-by-accident until it is pinned.** Each of these follows from +existing behaviour rather than from code written for the demo, which is exactly +the situation the `metadata.taskless` test was written for: behaviour we depend +on, produced by code that never knew about us. Tasks 3.4 and 5.2 pin the two +that would be silent if they broke. + +## Risks / Trade-offs + +**The demo was born on `request`/`requestId` ahead of the ordinary client, and +that skew has since closed.** It was deliberate (D2) and bounded while it +lasted: the `rule/*` family still serves, marked `deprecated`, so nothing was +broken by the mainline not having moved yet. The mainline has now moved. +`submitRule` posts to `/cli/api/request`, `pollRuleStatus` gets +`/cli/api/request/{requestId}`, `iterateRule` and `restoreRule` follow, and no +hand-written call site targets the `rule/*` family. + +**How that move landed is the part worth keeping, because this section's +expectation was wrong.** It assumed regenerating the types would catch the +typed call sites at compile time, leaving only the two that build their URL as a +template string to change by hand. It caught none of them. A deprecated path is +still IN the OpenAPI document, so `openapi-typescript` emits it as an ordinary +entry and `client.GET("/cli/api/rule/{ruleId}")` type-checks exactly as well as +the canonical spelling; all four typed call sites were reverted and `tsc +--noEmit` passed clean. What closes that gap is a test rather than the type +system: `generate:api` vendors the OpenAPI document next to the types it +generates, and `test/api-deprecated-paths.test.ts` reads that document to fail +if any source file still calls a path it marks deprecated. Deprecation is data +in the schema, not a type error, so catching it takes something that reads the +schema. + +That is the standing lesson for D2. When the mainline is renamed again and the +demo follows it, the rename is not self-enforcing, and the vendored document is +what makes it so. + +**The service half does not exist yet.** The endpoints are the service team's +to build, raised as N9. This change assumes the shapes in D1; if they land +differently, the client work is the adapter and the tests move with it. Nothing +here is worth building against a guess, so implementation waits on their +answer. + +**An unauthenticated generation endpoint is abuse surface.** Fixed input and a +regeneration window make it cheap to serve repeatedly, but the exposure is real +and it is the service's to bound. Worth naming here because the client asked +for the no-auth property and therefore owns half the reason it exists. + +**A demo that stops before execution may underwhelm.** Accepted, and preferable +to the alternative. The step it stops at is a signature gate, and explaining +why it stops is a better demonstration of the product than bypassing it would +be. + +**A hidden topic is discoverable by anyone who reads the source.** It is not a +secret and does not need to be. It is out of the index so routing cannot reach +it, not so that people cannot find it. diff --git a/openspec/changes/runtime-demo-path/proposal.md b/openspec/changes/runtime-demo-path/proposal.md new file mode 100644 index 00000000..c5e8423d --- /dev/null +++ b/openspec/changes/runtime-demo-path/proposal.md @@ -0,0 +1,121 @@ +## Why + +The runtime tier cannot be demonstrated, and it cannot be smoke-tested end to +end. Everything we can exercise alone covers the pieces — discovery, signing, +the file-set writer, delivery validation, the repair path — and none of it +covers the two things that only exist between the client and the service: +generation producing a runtime rule, and delivery handing it over in a shape +this client accepts. + +That gap is why the payload alignment work took the shape it did. Both sides +held the same assumptions separately, and every defect the exchange surfaced +lived in the seam: two floors that could disagree, a signature attached to the +wrong rule, a capture silently dropped, a rule written and never run. Each was +found by hand, by someone noticing. There is still no single command that walks +the whole path and shows what happened. + +A demo path closes that. Run one command, get a real generated runtime rule +written to disk, and see it verified. It is a live demo for a person, and the +same command is the closest thing we can have to an integration test across the +repository boundary. + +**Delivery shape: single PR.** The client half is one recipe, one command, and +their tests. Splitting it would separate a command from the recipe that names +it, which is the seam this change exists to remove. + +## What Changes + +**A hidden agent topic, `__undocumented-sample-runtime`.** Absent from the +`agent` index on purpose: `RECIPE_TOPICS` in `commands/agent.ts` is a +hand-maintained literal and lookup is by filename, so a topic that is not in +the list is fetchable by name and invisible otherwise. No new mechanism. This +is something we run deliberately, not something routing sends an agent to. + +**A command that walks the path, as `rule demo`.** Request, poll, retrieve, write, verify — +through the existing `submitRule` / `pollRuleStatus` client code and the +existing `writeRuleFile`, against demo endpoints that mirror the mainline +shapes: + +``` +POST /cli/api/demo/request -> { requestId, status } +GET /cli/api/demo/request/{requestId} -> { requestId, status, rules[] } +``` + +That is the shape both sides have agreed (task 0.1). What the retrieval carries +about what the rule found is a separate question and is still open: the findings +payload is an ask rather than a contract, sketched in design D6 and tracked by +task 0.4, which has not been answered. + +**The endpoints take no authentication at all.** We asked that an +unauthenticated call send no `Authorization` header rather than one carrying an +empty token. The service went further: with nothing scoped to a caller there is +nothing for authentication to decide, so the demo endpoints are public, like +`rule-hash-vectors`. There is no header to send and none for the service to +decide how to ignore. Making `submitRule` and `pollRuleStatus` tolerate an +absent token remains ours (task 1.2a); the server half is now a property rather +than a convention. + +**The retrieval also serves what the rule found.** The service's verification +gate already executes the generated `check.ts` against the fixture's failing and +passing examples on its way to deciding whether to accept the rule, and was +discarding the result. What we asked for is a per-example carrier: for each +fixture example, its name, whether it is expected to fail or pass, and the +`Finding[]` the check produced against it. That shape is under revision and is +not part of the agreed contract above; task 0.4 tracks settling it. + +Mirroring is the point. A bespoke demo payload would exercise a path no user is +on, which is the one thing a demo must not do. Reusing the well-known formats +means the demo also covers polling and the terminal `failed` / `unsupported` +states rather than a happy path built for the occasion. + +**The demo cannot reach the gate, and that is structural rather than a +choice.** A runtime rule executes only when an authenticated reconcile returns +its signature in `run`, and a signature exists because the rule was recorded +into an organization's corpus — recording is what blessing is. The demo rule is +one fixed rule, generated under a Taskless-owned installation and never recorded +for the caller, so no caller's `run` set can contain it. Authenticating would +not change that. The demo reaches a complete, well-formed, verified rule on disk +and stops, and `check` explains that state in the words it already uses. + +**No client-side bypass.** Not for a known demo id, not behind a flag of its +own. The signature gate is what stands between a downloaded payload and +arbitrary code execution on a developer's machine, and a demo is not worth a +hole in it. Executing the demo rule uses the documented +`--dangerously-run-scripts`, which prints the warning it always prints. + +## Capabilities + +### New Capabilities + +- `cli-runtime-demo`: A deliberately-invoked path that generates, delivers, + writes and verifies one runtime rule from a fixed, service-held input, so the + client-service seam can be shown working rather than described. + +### Modified Capabilities + +- `cli-agent`: The topic index is a curated list rather than every embedded + recipe. A topic may exist and be fetchable by name while staying out of the + index, which is already how the code behaves and is not yet stated as a + requirement. The index's command listing is separately sourced from the + top-level command tree, so staying out of the curated recipe list says + nothing about that half. + +## Impact + +- `packages/cli/src/agent/__undocumented-sample-runtime.txt` — new recipe. +- `packages/cli/src/commands/rules.ts` — the demo as a `rule demo` subcommand, + reusing `submitRule`, `pollRuleStatus` and `writeRuleFile` rather than + duplicating them. Nested rather than top-level so it adds nothing to + `SUBCOMMAND_NAMES` and therefore nothing to the `agent` index's command + listing (design D4a). +- `packages/cli/src/api/` — the demo endpoints, alongside the existing client. +- Depends on the service half (raised as **N9** in the cross-team document). + The endpoints are theirs to build; the shapes above are what we asked for and + what this change assumes. +- Sequenced behind the `request`/`requestId` rename (**N10**), which has + shipped and is verified live. The demo takes that noun. Adopting it in the + ordinary client was a separate change, was never this one's dependency, and + has since landed as well: no hand-written call site targets the `rule/*` + family any more. +- No change to the runtime gate, to reconcile, or to `check`'s execution + policy. diff --git a/openspec/changes/runtime-demo-path/specs/cli-agent/spec.md b/openspec/changes/runtime-demo-path/specs/cli-agent/spec.md new file mode 100644 index 00000000..2224d026 --- /dev/null +++ b/openspec/changes/runtime-demo-path/specs/cli-agent/spec.md @@ -0,0 +1,45 @@ +## ADDED Requirements + +### Requirement: The topic index is curated, and a topic may be absent from it + +The `agent` index SHALL list a curated set of topics rather than every embedded +recipe. A recipe MAY exist and be retrievable by name while being absent from +that index. + +**Rationale.** Some topics are invoked deliberately rather than discovered — a +demonstration path, for instance, which routing must never send an agent to. +The index is a hand-maintained list and lookup is by name, so absence from the +index is the existing behaviour; this states it as intended rather than leaving +it to be read as an oversight and "fixed". + +#### Scenario: An unlisted topic is still retrievable by name + +- **WHEN** an agent requests a topic that exists as a recipe but is not in the + index +- **THEN** the CLI SHALL return that recipe + +#### Scenario: An unlisted topic does not appear in the index + +- **WHEN** the topic index is rendered +- **THEN** a topic that is not in the curated list SHALL NOT appear in it + +### Requirement: The rendered index has two independently-sourced listings + +The `agent` index SHALL be treated as two separate listings: a command listing +built from the registered top-level subcommands, and the curated recipe +listing. Absence from the curated recipe listing SHALL NOT be relied on to keep +an entry out of the command listing, because the two share no source. + +**Rationale.** The command listing iterates the top-level command tree and +filters only `agent` itself; it never consults the curated recipe list. A +deliberately-invoked capability that registers a new top-level command is +therefore advertised there no matter what the recipe list says. Stating the two +listings separately keeps a later reader from assuming one requirement covers +both, which is how a topic ends up hidden in one half of its own output. + +#### Scenario: A deliberately-invoked capability adds no top-level command + +- **WHEN** a capability is meant to be invoked deliberately rather than + discovered from the index +- **THEN** it SHALL NOT be registered as a new top-level subcommand +- **AND** the command listing SHALL be unchanged by its addition diff --git a/openspec/changes/runtime-demo-path/specs/cli-runtime-demo/spec.md b/openspec/changes/runtime-demo-path/specs/cli-runtime-demo/spec.md new file mode 100644 index 00000000..7e9268b6 --- /dev/null +++ b/openspec/changes/runtime-demo-path/specs/cli-runtime-demo/spec.md @@ -0,0 +1,183 @@ +## ADDED Requirements + +### Requirement: A deliberately-invoked demo path + +The CLI SHALL provide a path that generates one runtime rule from a fixed +input the service holds, writes it, and verifies it, so the boundary between +this client and the rule service can be shown working rather than described. + +It SHALL be invoked deliberately. Routing SHALL NOT reach it, and no recipe +that decides how to author a rule SHALL name it as a destination. + +#### Scenario: The demo produces a complete runtime rule + +- **WHEN** the demo path is invoked in a project +- **THEN** a runtime rule directory SHALL exist containing `check.ts` and at + least one capture under `captures/` +- **AND** `verify` SHALL report that rule as valid + +#### Scenario: Routing never sends an agent to the demo + +- **WHEN** an agent fetches any authoring or routing recipe +- **THEN** the demo topic SHALL NOT appear as a destination in that recipe + +### Requirement: The demo is invoked as a subcommand of an existing command + +The demo SHALL be dispatched as `rule demo`, a subcommand of the existing +`rule` command. It SHALL NOT be registered as a new top-level subcommand, and +SHALL add no entry to the CLI's top-level command name list. + +**Rationale.** The `agent` index renders a command listing built from the +top-level command tree, separately from its curated recipe listing. A new +top-level verb would be advertised there regardless of the recipe being +unlisted, so the demo would be hidden in one half of the same output and +announced in the other. Nesting under `rule` obtains the property by not +registering the command, which is the same reasoning that keeps the recipe out +of the curated list rather than teaching a renderer to skip it. + +#### Scenario: The demo adds nothing to the top-level command listing + +- **WHEN** the `agent` index is rendered +- **THEN** the command listing SHALL NOT contain a demo entry +- **AND** the listing SHALL be identical to the one rendered before the demo + existed + +### Requirement: The demo uses the well-known request and retrieval formats + +The demo SHALL obtain its rule through the same two-stage flow as an ordinary +generated rule: a request that returns a ticket and a status, then a retrieval +by that ticket returning the status and the delivered rules. + +The retrieval response SHALL use the published file-set variant. The demo SHALL +NOT introduce a payload shape that only the demo consumes. + +**Rationale.** A demo exists to show the path a user is on. A bespoke shape +would demonstrate something no user receives, and would let the two shapes +drift without anything detecting it. + +#### Scenario: The demo reuses the ordinary delivery writer + +- **WHEN** the demo receives its rule +- **THEN** the rule SHALL be written by the same writer that writes a generated + rule, subject to the same path, completeness and layout validation + +#### Scenario: A malformed demo payload is refused, not written + +- **WHEN** the demo receives a rule that fails delivery validation +- **THEN** nothing SHALL be written for that rule +- **AND** the failure SHALL name what was wrong with the payload + +### Requirement: The demo shows what the rule found, per example + +The retrieval response SHALL carry, for each example in the service's fixture, +that example's name, whether it is expected to fail or pass, and the findings +the generated check produced against it. Findings SHALL use the published +`Finding` shape rather than a demo-only object. + +The CLI SHALL render findings grouped by example, and SHALL report the demo as +failed when the expected asymmetry does not hold: findings against an example +expected to pass, or no findings against an example expected to fail. + +**Rationale.** A flat list of findings renders identically for a rule that +catches the failing examples and a rule that fires on everything it is shown. +The second is the failure this area keeps producing — an indiscriminate or empty +scan reporting as a pass — so the demonstration is the asymmetry, not the count. +Asserting it makes the demo a check rather than a picture. + +#### Scenario: Findings are attributed to the example that produced them + +- **WHEN** the demo renders its result +- **THEN** each finding SHALL be shown under the example it came from +- **AND** each example SHALL state whether it was expected to fail or pass + +#### Scenario: A rule that fires on a passing example fails the demo + +- **WHEN** the response carries a finding against an example expected to pass +- **THEN** the demo SHALL report failure rather than rendering the findings as + a successful demonstration + +#### Scenario: A rule that finds nothing fails the demo + +- **WHEN** the response carries no findings against an example expected to fail +- **THEN** the demo SHALL report failure + +### Requirement: The demo reports a terminal failure rather than inventing a rule + +The CLI SHALL report a terminal status the service returns for a demo request, +together with the reason the service sent, and SHALL write nothing for that +request. + +**Rationale.** A demo that invents a rule when generation fails is worse than a +demo that fails, because it looks like success. + +#### Scenario: Generation cannot be served + +- **WHEN** the demo request terminates as unsupported or failed +- **THEN** the CLI SHALL surface the service's reason +- **AND** SHALL NOT write a rule + +### Requirement: The demo does not weaken the runtime execution gate + +The demo SHALL NOT cause a runtime rule to execute without the verification +`check` already requires. No demo rule identifier, and no flag introduced for +the demo, SHALL bypass signature verification. + +**Rationale.** Signature verification is what stands between a payload fetched +over the network and arbitrary code executing on a developer's machine. An +exception that exists for a demo is an exception, and the identifier it keys on +is visible to anyone who reads the recipe. + +#### Scenario: A demo rule is not executed by an unauthenticated check + +- **WHEN** `check` runs with no authentication in a project holding a demo rule +- **THEN** the demo rule SHALL be skipped with the reason `check` already gives + for an unverified runtime rule + +#### Scenario: Executing a demo rule uses the documented escape + +- **WHEN** a user runs `check` with `--dangerously-run-scripts` +- **THEN** the demo rule SHALL execute under that flag's existing warning, and + under no other mechanism + +### Requirement: The demo rule cannot be used as a base for improvement + +The CLI SHALL NOT offer the demo rule as an input to rule improvement. An +attempt to improve it SHALL terminate with the result the service returns for a +request the caller's organization does not hold, and SHALL write nothing. The +reported failure SHALL be terminal rather than one that invites a retry of the +same identifier. + +**Rationale.** Improvement resolves a request the caller's organization holds, +and the demo is held by a Taskless-owned installation instead, so refusal is the +accurate answer rather than a missing feature. Which status the service sends +for such a request is not yet confirmed (task 0.5), and the two candidates map to +different CLI codes — a 404 to `RULE_NOT_FOUND`, a 403 to `NETWORK_ERROR` — so +this requirement names the behaviour and leaves the code to that answer. The +retry distinction is the part that matters: a transport-shaped failure tells an +agent to retry an identifier that will never resolve. + +Refusal is also the answer we want on its merits: a rule pre-generated against a +shared fixture is a poor base to iterate on, and nothing generated in advance can +match a rule authored against this developer's own repository and context. +Offering it as a starting point would lend a worse starting point the demo's +credibility. + +#### Scenario: Improving the demo rule is refused + +- **WHEN** rule improvement is invoked against the demo rule +- **THEN** the CLI SHALL report the terminal result the service returns +- **AND** SHALL NOT present it as a failure worth retrying with the same + identifier +- **AND** SHALL write no rule + +### Requirement: What the demo writes can be removed + +A demo rule that has been written SHALL be removable by rule identifier through +the ordinary deletion path, leaving no residue that `verify` or `check` would +later report. + +#### Scenario: Deleting the demo rule + +- **WHEN** the user deletes the demo rule by its identifier +- **THEN** its rule directory SHALL be removed +- **AND** a subsequent `check` SHALL NOT report it diff --git a/openspec/changes/runtime-demo-path/tasks.md b/openspec/changes/runtime-demo-path/tasks.md new file mode 100644 index 00000000..9350aa3b --- /dev/null +++ b/openspec/changes/runtime-demo-path/tasks.md @@ -0,0 +1,75 @@ +# Tasks + +One PR. The command and the recipe that names it are the same seam this change +exists to remove, so they land together. + +## 0. Blocked on the rename, then on the service half + +The generator team renamed the request resource first, as its own change: +`request`/`requestId` rather than `rule`/`ruleId`. Their own +`openspec/specs/cli/spec.md` already specified it, and their live `meta` block +already carries `ticketId` per delivered rule, so the route was the outlier. + +- [x] 0.0 Wait for the rename to land and the new schema to publish. **Done**: verified from `GET /cli/api/__schema` that `/cli/api/request/*` is canonical and the `rule/*` family is `deprecated: true` while still serving. Adopting it in our client was its own change, tracked separately, and it has since landed +- [x] 0.0a The demo therefore takes the settled noun: `POST /cli/api/demo/request` and `GET /cli/api/demo/request/{requestId}` + +The demo did not start before that, which was the point. Naming these endpoints +earlier would have made the demo the place a new convention debuted, which is +what design D2 rules out. The mainline took the noun first; the demo follows it. + +## 0b. Blocked on the service half + +- [x] 0.1 Confirm the endpoint shapes with the generator team (**N9**): `POST /cli/api/demo/request` returning `{ requestId, status }`, `GET /cli/api/demo/request/{requestId}` returning `{ requestId, status, rules[] }` with the published file-set variant. Agreed both sides +- [x] 0.2 Confirm the service can serve the demo endpoints unauthenticated, and get their answer on whether reconcile can bless the sample signature for an anonymous caller. **Answered, both halves.** The endpoints take no authentication at all — public, like `rule-hash-vectors` — because nothing in the demo is scoped to a caller: no ticket, no corpus entry, no bill, and one Taskless-owned installation behind every request. And no blessing, for the reason D3 now records: blessing is recording, and a shared fixed rule is never recorded for the caller, so authenticating would not reach execution either +- [ ] 0.3 Agree what the demo rule is for — the scenario the generated rule addresses — so the demo shows something a person recognises rather than an arbitrary rule. The fixture now has a second job: its examples are what the served findings are attributed to, so it needs examples that are expected to fail AND examples that are expected to pass +- [ ] 0.4 Tell the service the findings shape we want (D6): `Finding[]` verbatim, grouped per fixture example with that example's name and whether it is expected to fail or pass. `Finding` is our published runtime-rule type, so this is the well-known format rather than a demo-only one +- [ ] 0.5 Ask the service which status `iterate` returns for a request outside the caller's organization: 404 `request_not_found` or 403 `access_denied`. It decides what `rule improve` against the demo rule reports — `iterateRule` maps the 404 to a `CLIError` carrying `RULE_NOT_FOUND` and the 403 to a plain `Error`, which `improveCommand` reports as `NETWORK_ERROR`, telling an agent to retry an id that will never resolve. The demo request exists under a Taskless-owned installation and is not the caller's, which is 403-shaped. D7 and the spec state the behaviour rather than a code until this is answered + +**Nothing below starts until 0.0 and 0.1 land.** Building against a guessed shape is +what produced the seam this change is closing. + +## 1. The client path + +- [ ] 1.1 Add the demo endpoints to `src/api/`, reusing the existing client rather than a second fetch layer +- [ ] 1.2 Drive them through `submitRule`/`pollRuleStatus`, so polling and the terminal `failed`/`unsupported` states are the same code an ordinary generation uses +- [ ] 1.2a Make that reuse possible first: `submitRule` and `pollRuleStatus` hardcode their endpoint as a literal (now `/cli/api/request` and `/cli/api/request/{requestId}`, since the client adopted the rename) and take a required `token` that `createApiClient` turns into an `Authorization` header unconditionally. Parameterize the endpoint and make the token optional. **Not a divergence signal** — see D1; these were written when every caller was authenticated, and the demo is the first that is not +- [ ] 1.2b Confirm an unauthenticated call sends no `Authorization` header at all, rather than one with an empty token. A header the service must then decide how to ignore is a worse contract than its absence +- [ ] 1.3 Write through `writeRuleFile`. If the demo needs its own writer, stop: the shapes have diverged and that is the finding +- [ ] 1.4 Report a terminal status by surfacing the service's reason, which `unsupportedMessage` already prefers over our own text +- [ ] 1.5 Regenerate `src/generated/api.d.ts` once the endpoints are published +- [ ] 1.6 Render the served findings grouped by example, showing each example's name, whether it was expected to fail or pass, and what the check returned against it +- [ ] 1.7 Assert the asymmetry rather than printing whatever arrives: findings on the examples expected to fail, none on the examples expected to pass. A demo whose rule has stopped finding anything, or that fires on everything, SHALL fail rather than render cleanly (D6) + +## 2. The hidden topic + +- [ ] 2.0 Register the command as `rule demo` under the existing `rule` command in `src/commands/rules.ts`. Not a top-level verb: `SUBCOMMAND_NAMES` and `subCommands` are unchanged, so the `agent` index's command listing gains nothing (design D4a) +- [ ] 2.1 Add `src/agent/__undocumented-sample-runtime.txt`, absent from `RECIPE_TOPICS` +- [ ] 2.2 The recipe names the command, says the rule may be imperfect, and says how to delete it +- [ ] 2.3 The recipe states where the demo stops and why, so a reader meets the signature gate as a design rather than as a failure +- [ ] 2.4 Test that the topic is fetchable by name and absent from the index +- [ ] 2.5 Test that the `agent` index's command listing is unchanged by the demo, so the second listing is covered and not just `RECIPE_TOPICS` + +## 3. Prove it does not weaken the gate + +- [ ] 3.1 Test that an unauthenticated `check` skips a demo rule, with the reason it already gives +- [ ] 3.2 Test that the demo rule executes under `--dangerously-run-scripts` and under nothing else +- [ ] 3.3 Test that no demo rule identifier is special-cased anywhere on the execution path +- [ ] 3.4 Test that `rule improve` against the demo rule terminates without writing anything, and reports a terminal failure rather than one that invites retrying the same id. Pin the behaviour so it cannot regress into a partial improve flow (D7). Pin the specific code once 0.5 answers which status the service sends; asserting one before that would encode a guess + +## 4. Prove it does not write a broken rule + +- [ ] 4.1 Test that a delivered demo rule missing `check.ts` or its captures is refused and nothing is written +- [ ] 4.2 Test that a terminal `unsupported` surfaces the service's reason and writes nothing +- [ ] 4.3 Test the whole path against a mock serving both endpoints, asserting the rule lands and `verify` reports it valid +- [ ] 4.4 Test that a payload whose passing examples carry findings is reported as a failed demonstration, not rendered as success — the indiscriminate-rule case D6 names +- [ ] 4.5 Test that a payload whose failing examples carry no findings is likewise reported as failed, since an empty scan is the shape a silent regression takes + +## 5. Prove it can be undone + +- [ ] 5.1 Test that deleting the demo rule by id removes its directory and that a later `check` does not report it +- [ ] 5.2 Test that an ordinary `check` skips the demo rule through the existing unverified-runtime path, asserting the reason comes from that path rather than from anything the demo added (D7) + +## 6. Close out + +- [ ] 6.1 Tell the generator team the release that ships this +- [ ] 6.2 Archive the change