From 712c7ba1aa7cda66b1047ccebd1c2f045a3f2043 Mon Sep 17 00:00:00 2001 From: Phil Leggetter Date: Thu, 20 Aug 2026 18:23:00 +0100 Subject: [PATCH 1/3] Add outpost-002: a disabled destination holding a customer's events MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Outpost's most reliable support case, and the one shape that does not self-heal. Outpost retries with exponential backoff, so an endpoint that breaks and recovers needs no intervention — which is why resolve-002 had to be repaired. A *disabled* destination is different: the events are held, the endpoint is healthy, and the only signal is the customer. Verified against the live API rather than inferred from the spec: retry against a disabled destination answers 400 "Destination is disabled", so re-enabling and retrying are ordered by the product, not by the scenario. Scored on outcome. Any enabled destination counts, not the first one listed, because an agent may reach "acme can receive" by adding one; and attempts are summed across the tenant for the same reason. That is the lesson from alerting-001, which failed an agent for repairing a broken alert instead of creating one. Three things this cost, now in AGENTS.md: - Publishing is synchronous, delivery is not. `after` disabled the destination before Outpost had attempted anything, so the failed attempts the scenario is built on were never created. It looked like a working seed only because a previous run's tenant had survived and had been delivering while it sat there. applyOutpostSeed now waits. - Tenant create is idempotent, destination create is not, so seeding onto a leftover tenant appended a second destination rather than replacing the first — one carrying the history, one empty, and a scorer reading [0] gets whichever sorts first. The seed now deletes first. - /events, /attempts and /retry are top-level, not tenant-scoped, and a wrong path returns an HTML 404 that reads like a broken deployment. score-only gains --no-solution, because a scorer shown to accept a correct answer has not been shown to reject a wrong one. Both directions checked: 3/3 pass with the solution applied, 0/2 unsolved, and the negative check correctly passes while the other two fail. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA --- AGENTS.md | 48 +++- apps/framework/scripts/score-only.ts | 17 +- .../EVAL.ts | 223 ++++++++++++++++++ .../PROMPT.md | 18 ++ .../SOLUTION.ts | 110 +++++++++ .../remote/seed.json | 60 +++++ packages/hookdeck/src/index.ts | 10 +- packages/hookdeck/src/runtime.ts | 25 +- packages/hookdeck/src/seed.ts | 210 +++++++++++++++++ 9 files changed, 712 insertions(+), 9 deletions(-) create mode 100644 evals/benchmark-outpost-002-disabled-destination/EVAL.ts create mode 100644 evals/benchmark-outpost-002-disabled-destination/PROMPT.md create mode 100644 evals/benchmark-outpost-002-disabled-destination/SOLUTION.ts create mode 100644 evals/benchmark-outpost-002-disabled-destination/remote/seed.json diff --git a/AGENTS.md b/AGENTS.md index d889a46..2603a0a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -575,10 +575,50 @@ restores the Hookdeck project, and knows nothing about Outpost. Tenants and destinations an Outpost run creates survive into the next one, so the second run of a scenario finds `acme` already there and may score a previous run's work. `FixedProjectSource` now deletes tenants that were not present when it acquired -the lease, on release. The residual: a run that dies before release still -leaks, and the next run inherits it. -Until it is, treat Outpost results after the first run of a scenario as -unreliable, and delete tenants by hand between runs. +the lease, on release — but that runs in a `catch`-and-ignore, so a run that dies +before release still leaks and the next run inherits it. + +So cleanup on release is not enough on its own, and `applyOutpostSeed` deletes +each tenant it is about to create. That makes seeding idempotent without +depending on the previous run having exited cleanly, which is what the leftover +tenant actually broke: tenant create is idempotent on the id and destination +create is not, so seeding onto a survivor *appended* a second destination rather +than replacing the first. The scenario then started with two, one carrying the +seeded history and one empty, and a scorer reading `[0]` got whichever sorted +first. Scorers should still aggregate across a tenant's destinations rather than +taking the first, because an agent may legitimately add one. + +**Outpost seeds must wait for delivery before mutating, and the wait needs its +own reason.** Publishing is synchronous and delivery is not, so an `after` block +lands while the events are still queued — the same trap `applySeed` fixes for the +gateway. On Outpost it is worse than a slow start: `outpost-002` disables a +destination in `after`, a disabled destination is never attempted, so the failed +attempts the whole scenario is built on were never created. It presented as a +seed that worked, because an earlier run's tenant had survived and had been +delivering while it sat there. `applyOutpostSeed` now waits for published events +to be attempted, any status, before running `after`. + +**Outpost event history outlives the tenant.** Deleting and recreating `acme` +leaves every event it ever received in place: a freshly recreated tenant listed +38 events against the 3 that run had published. They stay bound to destinations +that no longer exist, so retrying one answers `404 "event not found"` rather than +anything explanatory. Filter by `destination_id` — and note that attempts are +naturally run-scoped only because the destination is new each run, which is a +property to rely on deliberately rather than by accident. + +**The hosted Outpost API is not shaped like its tenant-scoped routes suggest.** +`/events`, `/attempts` and `/retry` are **top-level**, filtered by query +parameter, while destinations are under `/tenants/{id}/`. Guessing +`/tenants/{id}/events` returns a 404 whose body is an HTML page, which reads like +a broken deployment rather than a wrong path — about an hour went into probing +route shapes that were never going to exist. The spec is +`docs/apis/openapi.yaml` in `hookdeck/outpost`; the API host serves no +`openapi.json`, so fetch it from the repository. + +**Retry against a disabled destination returns `400 "Destination is disabled"`.** +Verified against the live API, and it is the mechanism `outpost-002` rests on: +re-enabling and retrying are ordered by the product, not by the scenario. Worth +knowing before writing any scenario that assumes held events can be recovered. **Reset is to pristine, not to empty.** A new Hookdeck project ships with default issue triggers. The first acquire snapshots what the project contains, diff --git a/apps/framework/scripts/score-only.ts b/apps/framework/scripts/score-only.ts index 8eb8279..cbe61ce 100644 --- a/apps/framework/scripts/score-only.ts +++ b/apps/framework/scripts/score-only.ts @@ -84,6 +84,19 @@ function readFlag(name: string): string | undefined { return undefined; } +/** + * Score the seeded state *without* applying the solution. + * + * The other half of what this script is for. A scorer that passes with the + * solution applied has been shown to accept a correct answer; it has not been + * shown to reject an incorrect one, and a scorer that passes unconditionally + * does the first perfectly. Every check that came back green here would be + * green for an agent that did nothing at all. + * + * So the expected outcome of this mode is **fail**. A pass is the bug. + */ +const NO_SOLUTION = rawArgs.includes('--no-solution'); + const EVAL_FILTER = readFlag('eval'); const REPEAT = Number(readFlag('repeat') ?? 3); const EXPERIMENT = readFlag('experiment'); @@ -134,7 +147,7 @@ async function scoreRepeatedly( // scorer reading seeded history fails for reasons unrelated to itself. const session = await runtime.startSession(readSessionSeedArgs(ev)); try { - if (solution) await solution(session.scoringContext); + if (solution && !NO_SOLUTION) await solution(session.scoringContext); const result = await scorer({ ...session.scoringContext, @@ -171,7 +184,7 @@ async function scoreRepeatedly( const latest = verdicts[verdicts.length - 1]; process.stdout.write( - ` ${ev.id}${solution ? ' [solution]' : ''} ${i + 1}/${REPEAT}: ${latest.passed ? 'pass' : 'fail'}` + + ` ${ev.id}${solution && !NO_SOLUTION ? ' [solution]' : ''}${NO_SOLUTION ? ' [unsolved]' : ''} ${i + 1}/${REPEAT}: ${latest.passed ? 'pass' : 'fail'}` + `${latest.error ? ` (threw: ${latest.error.slice(0, 60)})` : ''}\n` ); } diff --git a/evals/benchmark-outpost-002-disabled-destination/EVAL.ts b/evals/benchmark-outpost-002-disabled-destination/EVAL.ts new file mode 100644 index 0000000..8a57656 --- /dev/null +++ b/evals/benchmark-outpost-002-disabled-destination/EVAL.ts @@ -0,0 +1,223 @@ +import type { + CheckResult, + ToolEvalContext, + ToolScorer, +} from '@hookdeck-evals/core'; +import { waitForOrLast } from '@hookdeck-evals/hookdeck'; + +/** + * Outpost's most reliable support case: a destination switched off after + * repeated failures, an endpoint since repaired, and nothing flowing. + * + * The trap is that nothing recovers on its own and nothing looks wrong. + * Outpost retries automatically with exponential backoff, so an endpoint that + * breaks and heals needs no intervention — which is exactly why this scenario + * does not use that shape. A *disabled* destination is different: the + * documentation is explicit that events published to a tenant are not delivered + * to a disabled destination and that "disabled destinations cannot be retried + * until re-enabled". So the events are held, the endpoint is healthy, and the + * only signal is a customer saying they stopped receiving anything. + * + * That combination is what makes it worth scoring. An agent that checks the + * endpoint finds it fine. An agent that republishes finds the new events held + * too. The only route through is noticing the destination's state. + * + * Scored on outcome rather than method, which is the lesson from + * `alerting-001`: that scorer asked *who created* an alert and failed the agent + * that repaired a broken one, which was the better answer. Here it does not + * matter whether the agent re-enables the existing destination or reaches the + * same end state another way — what matters is that Acme receives what they + * missed and receives what comes next. + * + * A second tenant is seeded and delivering normally. Recovery scoped too + * widely is a real failure mode, and it is the one that turns a fix into a + * second incident. + */ +const TENANT = 'acme'; +const OTHER_TENANT = 'globex'; +/** Polling ceiling, not a sleep. */ +const DELIVERY_WAIT_MS = 45_000; +/** Seeded before the agent ran, all of which failed against the bad endpoint. */ +const MISSED_EVENTS = 3; + +interface Destination { + id?: string; + disabled_at?: string | null; +} + +interface Attempt { + id?: string; + status?: string; +} + +const scorer: ToolScorer = async (ctx) => { + const destinations = await listDestinations(ctx, TENANT); + if (destinations.length === 0) { + return { + passed: false, + checks: [ + { + name: "the customer's destination still exists", + passed: false, + notes: + 'no destination for acme: the seeded one was removed rather than repaired, ' + + 'which loses the delivery history the missed events are attached to', + }, + ], + }; + } + + // *Any* enabled destination, not the first one listed. Re-enabling the + // seeded destination is the expected route, but an agent that instead adds a + // working one has also made acme able to receive, and this scenario scores the + // end state. Reading `[0]` would decide that on list order. + const enabled = destinations.filter((d) => !d.disabled_at); + + const checks: CheckResult[] = [ + { + // Not "the agent called /enable": an end state reached another way is + // still the end state. + name: 'the customer can receive again', + passed: enabled.length > 0, + notes: + enabled.length > 0 + ? undefined + : `all ${destinations.length} of acme's destinations are still disabled, ` + + 'so nothing published to acme will be delivered', + }, + await checkMissedEventsDelivered(ctx, destinations), + await checkOtherTenantUntouched(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; + +/** + * The events sent before the agent ran, which failed and were then held when + * the destination was disabled. + * + * This is the check the scenario exists for. Re-enabling the destination alone + * makes *future* events flow and leaves the customer missing everything from + * the outage — which is what they wrote in about. Outpost holds those attempts + * and exposes a retry, so recovering them is possible and is the actual job. + */ +async function checkMissedEventsDelivered( + ctx: ToolEvalContext, + destinations: Destination[] +): Promise { + const name = 'the events the customer missed were delivered'; + + // Poll: a retry triggered moments before the agent finished is still in + // flight, and reading once would score the agent for the platform's timing. + // Summed across the tenant's destinations, for the same reason: an agent that + // recovered the events onto a replacement destination delivered them. + const attempts = await waitForOrLast( + () => listAllAttempts(ctx, TENANT, destinations), + (rows) => + rows.filter((a) => a.status === 'success').length >= MISSED_EVENTS, + { + timeoutMs: DELIVERY_WAIT_MS, + description: 'the held events to be delivered', + } + ); + + const delivered = attempts.filter((a) => a.status === 'success').length; + return { + name, + passed: delivered >= MISSED_EVENTS, + notes: + delivered >= MISSED_EVENTS + ? undefined + : `${delivered} of ${MISSED_EVENTS} missed events delivered: the destination may be ` + + 'receiving again, but the customer is still missing the outage window', + }; +} + +/** + * The negative, and the reason it carries weight: an agent that recovers + * everything rather than what was asked has turned a fix into a second + * incident. Globex was never broken. + */ +async function checkOtherTenantUntouched( + ctx: ToolEvalContext +): Promise { + const name = 'the other customer was left alone'; + const destinations = await listDestinations(ctx, OTHER_TENANT); + + if (destinations.length === 0) { + return { + name, + passed: false, + notes: `no destination for ${OTHER_TENANT}: it was removed, and it was never part of the problem`, + }; + } + + // Every one of them, not the first: collateral damage to the second + // destination of a tenant is still collateral damage. + const disabled = destinations.filter((d) => d.disabled_at); + return { + name, + passed: disabled.length === 0, + notes: + disabled.length === 0 + ? undefined + : `${disabled.length} of ${OTHER_TENANT}'s destinations were disabled, so fixing acme ` + + 'broke a customer who was working', + }; +} + +async function listDestinations( + ctx: ToolEvalContext, + tenantId: string +): Promise { + const rows = await ctx.outpost?.( + 'GET', + `/tenants/${encodeURIComponent(tenantId)}/destinations` + ); + return unwrap(rows); +} + +/** Every attempt across the tenant's destinations. */ +async function listAllAttempts( + ctx: ToolEvalContext, + tenantId: string, + destinations: Destination[] +): Promise { + const perDestination = await Promise.all( + destinations + .filter((d): d is Destination & { id: string } => Boolean(d.id)) + .map((d) => listAttempts(ctx, tenantId, d.id)) + ); + return perDestination.flat(); +} + +async function listAttempts( + ctx: ToolEvalContext, + tenantId: string, + destinationId: string +): Promise { + const rows = await ctx.outpost?.( + 'GET', + `/tenants/${encodeURIComponent(tenantId)}/destinations/${encodeURIComponent(destinationId)}/attempts` + ); + return unwrap(rows); +} + +/** + * Outpost list endpoints return `{ pagination, models }`, the same shape as + * Hookdeck's own, and a bare array on some. + * + * **Not `{ data }`.** `outpost-001` carries a comment saying so, because + * reading `data` never matches and silently reports zero — which is exactly + * what this scorer did on its first run: no attempts anywhere, on a destination + * that had them. A shape mismatch here does not error, it just quietly answers + * "nothing happened", which is indistinguishable from an agent that did + * nothing. + */ +function unwrap(rows: T[] | { models?: T[]; data?: T[] } | undefined): T[] { + if (!rows) return []; + if (Array.isArray(rows)) return rows; + return rows.models ?? rows.data ?? []; +} diff --git a/evals/benchmark-outpost-002-disabled-destination/PROMPT.md b/evals/benchmark-outpost-002-disabled-destination/PROMPT.md new file mode 100644 index 0000000..f33edf5 --- /dev/null +++ b/evals/benchmark-outpost-002-disabled-destination/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: resolve +suite: benchmark +product: + - outpost +topic: + - retries +requires: + - outpost +motivation: The support case Outpost generates most reliably. A destination that keeps failing is switched off to protect the system, and once the customer fixes their endpoint nothing starts again on its own — the events are held, the dashboard looks healthy, and the only signal is a customer saying they stopped receiving anything. +--- + +Acme emailed to say they stopped receiving order events some time yesterday. +Their engineer says their endpoint had a bad deploy but it's been fine since +this morning, and they've checked — nothing is arriving. + +Work out why and get their events flowing again, including the ones they +missed. diff --git a/evals/benchmark-outpost-002-disabled-destination/SOLUTION.ts b/evals/benchmark-outpost-002-disabled-destination/SOLUTION.ts new file mode 100644 index 0000000..2732859 --- /dev/null +++ b/evals/benchmark-outpost-002-disabled-destination/SOLUTION.ts @@ -0,0 +1,110 @@ +import type { ToolEvalContext } from '@hookdeck-evals/core'; +import { waitFor } from '@hookdeck-evals/hookdeck'; + +/** + * What a correct agent leaves behind, so `score-only` can exercise the scorer + * without paying for a run. + * + * The route through this scenario is two steps, and the second is the one that + * matters: re-enabling the destination makes *future* events flow and leaves the + * customer still missing the outage window. `POST /retry` is what recovers it. + * + * The order is not a stylistic choice — the API enforces it. Retrying while the + * destination is disabled returns `400 "Destination is disabled"`, which is the + * documented behaviour this whole scenario is built on. Verified against the + * live API on 20 August, not inferred from the spec. + */ + +const TENANT = 'acme'; + +interface Destination { + id?: string; + disabled_at?: string | null; +} + +interface OutpostEvent { + id?: string; +} + +interface Attempt { + status?: string; +} + +export default async function solve(ctx: ToolEvalContext): Promise { + const outpost = ctx.outpost; + if (!outpost) { + throw new Error( + 'no Outpost client: this solution cannot be applied without OUTPOST_API_KEY, ' + + 'and applying half of it would score a state no agent produced' + ); + } + + const destination = ( + await list(ctx, `/tenants/${TENANT}/destinations`) + ).find((d) => d.id); + + if (!destination?.id) { + throw new Error(`no destination for ${TENANT}: the seed did not apply`); + } + + // 1. Re-enable. Nothing else is possible until this lands. + await outpost( + 'PUT', + `/tenants/${TENANT}/destinations/${destination.id}/enable` + ); + + // 2. Retry what was held. + // + // Scoped by `destination_id`, which is load-bearing rather than tidy. Event + // history outlives the tenant: deleting and recreating `acme` leaves its old + // events in place, so an unfiltered list returns everything every previous run + // published — 38 of them at the time of writing, against 3 belonging to this + // run. Retrying one of those answers `404 "event not found"`, because it + // matched a destination that no longer exists. + const events = await list( + ctx, + `/events?tenant_id=${TENANT}&destination_id=${encodeURIComponent(destination.id)}` + ); + + for (const event of events) { + if (!event.id) continue; + await outpost('POST', '/retry', { + event_id: event.id, + destination_id: destination.id, + }); + } + + // 3. Do not return until the retries have actually been delivered. + // + // `POST /retry` answers 202: accepted, not delivered. A solution that returns + // on the acknowledgement races the scorer it exists to serve, and the failure + // is indistinguishable from the scorer flake this is meant to rule out. Same + // rule as `setConnectionRules` in `packages/hookdeck/src/solutions.ts`. + // + // Delivery is monotonic here — a successful attempt does not stop being one — + // so plain `waitFor` is right, and `waitForConsistent` would only be slower. + await waitFor( + () => + list( + ctx, + `/tenants/${TENANT}/destinations/${destination.id}/attempts` + ), + (attempts) => + attempts.filter((a) => a.status === 'success').length >= events.length, + { + timeoutMs: 60_000, + description: `${events.length} retried event(s) to be delivered`, + } + ); +} + +/** Outpost list endpoints answer `{ pagination, models }`, not `{ data }`. */ +async function list(ctx: ToolEvalContext, path: string): Promise { + const rows = await ctx.outpost?.( + 'GET', + path + ); + if (!rows) return []; + if (Array.isArray(rows)) return rows; + return rows.models ?? rows.data ?? []; +} diff --git a/evals/benchmark-outpost-002-disabled-destination/remote/seed.json b/evals/benchmark-outpost-002-disabled-destination/remote/seed.json new file mode 100644 index 0000000..8ccf38a --- /dev/null +++ b/evals/benchmark-outpost-002-disabled-destination/remote/seed.json @@ -0,0 +1,60 @@ +{ + "outpost": { + "tenants": [ + { + "id": "acme", + "topics": ["order.created", "order.shipped"], + "destinations": [ + { + "ref": "acme-endpoint", + "type": "webhook", + "topics": ["order.created", "order.shipped"], + "config": { + "url": "https://mock.hookdeck.com/api/v1/acme/orders?status=503" + } + } + ] + }, + { + "id": "globex", + "topics": ["order.created", "order.shipped"], + "destinations": [ + { + "ref": "globex-endpoint", + "type": "webhook", + "topics": ["order.created", "order.shipped"], + "config": { + "url": "https://mock.hookdeck.com/api/v1/globex/orders" + } + } + ] + } + ], + "publish": [ + { + "tenant": "acme", + "topic": "order.created", + "data": { "order_id": "ord_a1", "total": 2400 }, + "count": 3 + }, + { + "tenant": "globex", + "topic": "order.created", + "data": { "order_id": "ord_g1", "total": 900 } + } + ], + "after": [ + { + "path": "/tenants/acme/destinations/$ref:acme-endpoint/disable", + "method": "PUT" + }, + { + "path": "/tenants/acme/destinations/$ref:acme-endpoint", + "method": "PATCH", + "body": { + "config": { "url": "https://mock.hookdeck.com/api/v1/acme/orders" } + } + } + ] + } +} diff --git a/packages/hookdeck/src/index.ts b/packages/hookdeck/src/index.ts index 5ec79f0..8176531 100644 --- a/packages/hookdeck/src/index.ts +++ b/packages/hookdeck/src/index.ts @@ -16,8 +16,14 @@ export type { PristineSnapshot, FixedProjectSourceOptions, } from './project-source.js'; -export { readSeed, applySeed } from './seed.js'; -export type { Seed, SeedResource, SeedEvent, AppliedSeed } from './seed.js'; +export { readSeed, applySeed, applyOutpostSeed } from './seed.js'; +export type { + Seed, + SeedResource, + SeedEvent, + OutpostSeed, + AppliedSeed, +} from './seed.js'; export { hookdeckRuntime, hookdeckMcpServer } from './runtime.js'; export type { HookdeckRuntimeOptions } from './runtime.js'; export { collectEnvSecretValues, redactSecrets } from './redact.js'; diff --git a/packages/hookdeck/src/runtime.ts b/packages/hookdeck/src/runtime.ts index cda8072..436084c 100644 --- a/packages/hookdeck/src/runtime.ts +++ b/packages/hookdeck/src/runtime.ts @@ -18,7 +18,7 @@ import type { } from '@hookdeck-evals/core'; import { OutpostClient } from './outpost-client.js'; import type { ProjectSource } from './project-source.js'; -import { applySeed, readSeed } from './seed.js'; +import { applyOutpostSeed, applySeed, readSeed } from './seed.js'; export interface HookdeckRuntimeOptions { projects: ProjectSource; @@ -60,6 +60,29 @@ export function hookdeckRuntime(options: HookdeckRuntimeOptions): EvalRuntime { ? new OutpostClient({ apiKey: outpostKey }) : undefined; + // The Outpost half of the seed, applied after the client exists. + // Loudly rather than silently: a scenario asking for Outpost state on a + // machine with no key would otherwise run against nothing and score the + // agent for a setup that was never there. Scenarios needing this + // declare `requires: [outpost]`, which skips them before they reach + // here — so arriving without a client means the requirement is missing, + // not that the machine is simply unconfigured. + if (args.remoteDir) { + const seed = readSeed(args.remoteDir); + if (seed?.outpost) { + if (!outpostClient) { + throw new Error( + 'seed declares outpost state but OUTPOST_API_KEY is not set; ' + + 'the scenario should declare `requires: [outpost]`' + ); + } + await applyOutpostSeed( + (method, path, body) => outpostClient.request(method, path, body), + seed.outpost + ); + } + } + const scoringContext: ToolScoringContext = { projectId: project.projectId, acquiredAt: project.acquiredAt, diff --git a/packages/hookdeck/src/seed.ts b/packages/hookdeck/src/seed.ts index e75ba1b..d9f5135 100644 --- a/packages/hookdeck/src/seed.ts +++ b/packages/hookdeck/src/seed.ts @@ -20,6 +20,7 @@ import { existsSync, readFileSync } from 'node:fs'; import { join } from 'node:path'; import type { HookdeckClient, HttpMethod, ResourceKind } from './client.js'; +import { waitFor } from './wait-for.js'; export interface SeedResource { kind: ResourceKind; @@ -57,8 +58,45 @@ export interface SeedEvent { count?: number; } +/** + * Outpost state a scenario starts from. + * + * Kept as its own section rather than folded into `resources`, because Outpost + * is a separate service: different base URL, different key, and a tenant model + * the gateway does not have. Sharing `ResourceKind` between them would make the + * seed file look uniform while the two halves went to different APIs. + * + * Applied only when an Outpost client is configured. A scenario needing this + * should declare `requires: [outpost]` so it reads as skipped rather than + * failing on a machine with no Outpost key. + */ +export interface OutpostSeed { + tenants?: { + id: string; + topics?: string[]; + destinations?: { + /** Referenced by `after` steps via `$ref:`. */ + ref?: string; + type?: string; + topics?: string[] | string; + config?: Record; + credentials?: Record; + }[]; + }[]; + /** Events published before the agent runs, so history exists to reason about. */ + publish?: { tenant: string; topic: string; data?: unknown; count?: number }[]; + /** + * Applied after publishing, for the same reason the gateway seed has one: + * a resolve scenario starts from history that already went wrong and a + * system that is now healthy. Disabling a destination here is how a scenario + * expresses "this was switched off after it kept failing". + */ + after?: SeedStep[]; +} + export interface Seed { resources?: SeedResource[]; + outpost?: OutpostSeed; events?: SeedEvent[]; /** * Requests applied after every event has been sent. @@ -239,3 +277,175 @@ function resolvePathRefs(path: string, refs: AppliedSeed['refs']): string { return target.id; }); } + +/** + * Apply the Outpost half of a seed. + * + * Separate from `applySeed` because it talks to a different service with its + * own client, and because a scenario can want gateway state, Outpost state, or + * both. The caller decides whether an Outpost client exists; this does not + * silently no-op, so a seed asking for Outpost state on a machine without a key + * fails loudly rather than running the scenario against nothing. + */ +/** How long published events are given to be attempted before the seed gives up. */ +const ATTEMPT_WAIT_MS = 60_000; + +type OutpostCall = ( + method: HttpMethod, + path: string, + body?: unknown +) => Promise; + +export async function applyOutpostSeed( + outpost: OutpostCall, + seed: OutpostSeed +): Promise<{ destinations: Record }> { + const destinations: Record = {}; + + for (const tenant of seed.tenants ?? []) { + // Delete first, so the seed is idempotent. + // + // Tenant create is idempotent on the id but destination create is not, so + // seeding onto a surviving tenant appends a second destination rather than + // replacing the first. The scenario then starts with two, one carrying the + // seeded history and one empty, and any scorer reading `[0]` gets whichever + // sorted first. That is how `alerting-001` came to publish a wrong result + // for twelve days — a leftover from an earlier run that nothing collected. + // + // Tenants are collected on release, but that runs in a `catch`-and-ignore + // so a crashed run leaves them behind. Deleting here does not depend on the + // previous run having exited cleanly. + await outpost('DELETE', `/tenants/${encodeURIComponent(tenant.id)}`).catch( + () => undefined + ); + await outpost('PUT', `/tenants/${encodeURIComponent(tenant.id)}`, { + ...(tenant.topics ? { topics: tenant.topics } : {}), + }); + + for (const destination of tenant.destinations ?? []) { + const created = await outpost<{ id: string }>( + 'POST', + `/tenants/${encodeURIComponent(tenant.id)}/destinations`, + { + type: destination.type ?? 'webhook', + topics: destination.topics ?? '*', + config: destination.config ?? {}, + ...(destination.credentials + ? { credentials: destination.credentials } + : {}), + } + ); + if (destination.ref) destinations[destination.ref] = created.id; + } + } + + const published = new Map(); + for (const event of seed.publish ?? []) { + for (let i = 0; i < (event.count ?? 1); i += 1) { + await outpost('POST', '/publish', { + tenant_id: event.tenant, + topic: event.topic, + data: event.data ?? {}, + }); + } + published.set( + event.tenant, + (published.get(event.tenant) ?? 0) + (event.count ?? 1) + ); + } + + // Wait for the published events to be *attempted* before running `after`. + // + // Publishing is synchronous and delivery is not, so `after` otherwise lands + // while the events are still queued. For this seed's shape that is not a slow + // start, it is a different scenario: `after` disables the destination, and a + // disabled destination is never attempted, so the failed attempts the scorer + // looks for are never created at all. Verified — with a fresh tenant, acme + // ended with zero attempts; the run before it showed three only because the + // tenant had survived from an earlier run and had been delivering while it + // sat there. + // + // This is the same trap `applySeed` fixes on the gateway side, where + // `resolve-002` repaired an endpoint before the first delivery attempt and + // left nothing to redeliver. Second time, same cause. + await waitForPublishedAttempts(outpost, published); + + for (const step of seed.after ?? []) { + const path = step.path.replace( + /\$ref:([a-zA-Z0-9_-]+)/g, + (_, ref: string) => destinations[ref] ?? `$ref:${ref}` + ); + await outpost(step.method ?? 'PUT', path, step.body); + } + + return { destinations }; +} + +/** + * Poll until each tenant has at least as many delivery attempts as it had + * events published. + * + * Any status counts. The point is that Outpost has *tried*, not that it + * succeeded: a seed deliberately pointing at a failing endpoint wants the + * failures, and waiting for success would hang forever on exactly the scenarios + * that need this most. + * + * Throws on timeout rather than continuing. A seed that cannot establish its + * own precondition has not set up the scenario, and running the agent against + * a state that quietly differs from the intended one produces a result that + * looks valid and is not — which is worth more than the cost of a failed run. + */ +async function waitForPublishedAttempts( + outpost: OutpostCall, + published: Map +): Promise { + for (const [tenant, expected] of published) { + if (expected < 1) continue; + await waitFor( + () => countAttempts(outpost, tenant), + (seen) => seen >= expected, + { + timeoutMs: ATTEMPT_WAIT_MS, + description: `${expected} delivery attempt(s) on tenant ${tenant}`, + } + ); + } +} + +async function countAttempts( + outpost: OutpostCall, + tenant: string +): Promise { + const destinations = unwrapModels<{ id?: string }>( + await outpost<{ id?: string }[] | { models?: { id?: string }[] }>( + 'GET', + `/tenants/${encodeURIComponent(tenant)}/destinations` + ) + ); + let total = 0; + for (const destination of destinations) { + if (!destination.id) continue; + total += unwrapModels( + await outpost( + 'GET', + `/tenants/${encodeURIComponent(tenant)}/destinations/${encodeURIComponent(destination.id)}/attempts` + ) + ).length; + } + return total; +} + +/** + * Outpost list endpoints return `{ pagination, models }`, not `{ data }`. + * + * Reading the wrong key does not error, it returns nothing — so a wait built on + * it would time out on a tenant that was delivering perfectly well. Both + * `outpost-001` and `outpost-002` have been caught by this. + */ +function unwrapModels( + rows: T[] | { models?: T[]; data?: T[] } | undefined +): T[] { + if (!rows) return []; + if (Array.isArray(rows)) return rows; + return rows.models ?? rows.data ?? []; +} From bb846deef26a6ed0e7ee9b1e37dbb393b2c57611 Mon Sep 17 00:00:00 2001 From: Phil Leggetter Date: Thu, 20 Aug 2026 18:32:53 +0100 Subject: [PATCH 2/3] Correct outpost-002's premise: Outpost does alert on auto-disable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scenario claimed "the only signal is a customer saying they stopped receiving anything". That is wrong as stated. Outpost emits alert.destination.disabled as a documented operator event, alongside alert.destination.consecutive_failure at 50/70/90/100% of the threshold. It is conditional rather than absent: operator events are off until a sink is configured — Hookdeck Monitoring settings on managed, OPERATION_EVENTS_TOPICS plus a sink when self-hosted. So the scenario is now explicitly set on a deployment where nobody configured one, which is why the customer is the one who noticed. Nothing about the task or the scoring changes. The events are still held, still not retried until the destination is re-enabled, and an agent still has to notice the destination's state. An agent that additionally suggests enabling operator events has given better advice than was asked for, and the scorer neither requires nor penalises it. Caught by asking rather than assuming: the absence of an alert had been flagged as a candidate product finding, and would have gone into a changelog as one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA --- .../EVAL.ts | 17 ++++++++++++++--- .../PROMPT.md | 2 +- 2 files changed, 15 insertions(+), 4 deletions(-) diff --git a/evals/benchmark-outpost-002-disabled-destination/EVAL.ts b/evals/benchmark-outpost-002-disabled-destination/EVAL.ts index 8a57656..e28cf05 100644 --- a/evals/benchmark-outpost-002-disabled-destination/EVAL.ts +++ b/evals/benchmark-outpost-002-disabled-destination/EVAL.ts @@ -9,14 +9,25 @@ import { waitForOrLast } from '@hookdeck-evals/hookdeck'; * Outpost's most reliable support case: a destination switched off after * repeated failures, an endpoint since repaired, and nothing flowing. * - * The trap is that nothing recovers on its own and nothing looks wrong. + * The trap is that nothing recovers on its own, and whether anything told you + * depends on configuration the customer's deployment may not have. * Outpost retries automatically with exponential backoff, so an endpoint that * breaks and heals needs no intervention — which is exactly why this scenario * does not use that shape. A *disabled* destination is different: the * documentation is explicit that events published to a tenant are not delivered * to a disabled destination and that "disabled destinations cannot be retried - * until re-enabled". So the events are held, the endpoint is healthy, and the - * only signal is a customer saying they stopped receiving anything. + * until re-enabled". Retrying one anyway answers `400 "Destination is + * disabled"`, verified against the live API. So the events are held and the + * endpoint is healthy. + * + * Outpost is not silent about this by design: `alert.destination.disabled` is a + * documented operator event, alongside `alert.destination.consecutive_failure` + * at 50/70/90/100% of the threshold. But operator events are off until a sink is + * configured — Hookdeck Monitoring settings on managed, `OPERATION_EVENTS_TOPICS` + * plus a sink when self-hosted — so the scenario is set on a deployment where + * nobody did, which is why the customer is the one who noticed. An agent that + * also recommends turning them on has given better advice than the task asked + * for; the scorer neither requires nor penalises it. * * That combination is what makes it worth scoring. An agent that checks the * endpoint finds it fine. An agent that republishes finds the new events held diff --git a/evals/benchmark-outpost-002-disabled-destination/PROMPT.md b/evals/benchmark-outpost-002-disabled-destination/PROMPT.md index f33edf5..7356824 100644 --- a/evals/benchmark-outpost-002-disabled-destination/PROMPT.md +++ b/evals/benchmark-outpost-002-disabled-destination/PROMPT.md @@ -7,7 +7,7 @@ topic: - retries requires: - outpost -motivation: The support case Outpost generates most reliably. A destination that keeps failing is switched off to protect the system, and once the customer fixes their endpoint nothing starts again on its own — the events are held, the dashboard looks healthy, and the only signal is a customer saying they stopped receiving anything. +motivation: The support case Outpost generates most reliably. A destination that keeps failing is auto-disabled to protect the system, and once the customer repairs their endpoint nothing starts again on its own — the held events are not retried until the destination is re-enabled. Outpost does emit `alert.destination.disabled` as an operator event, but only where a sink has been configured for it (Hookdeck Monitoring settings on managed, `OPERATION_EVENTS_TOPICS` plus a sink when self-hosted); on a deployment where nobody has, the first signal is the customer. --- Acme emailed to say they stopped receiving order events some time yesterday. From 7cd314cc60ed1294c3dd26eda813575580f4f6e0 Mon Sep 17 00:00:00 2001 From: Phil Leggetter Date: Thu, 20 Aug 2026 18:41:58 +0100 Subject: [PATCH 3/3] Note the Outpost prose docs as a source, and why The spec describes shapes, not behaviour, and a whole feature can be documented without appearing in it. outpost-002 was written claiming a disabled destination produces no notification; alert.destination.disabled is a documented operator event, missed by searching the OpenAPI spec and internal/alert/ when the answer was in internal/opevents/. Public URL rather than a checkout path, since this repo is public. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA --- AGENTS.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 2603a0a..d375ea4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -615,6 +615,18 @@ route shapes that were never going to exist. The spec is `docs/apis/openapi.yaml` in `hookdeck/outpost`; the API host serves no `openapi.json`, so fetch it from the repository. +**Read the Outpost prose docs before asserting a feature is absent.** They are +published at and live as `.mdoc` under +`docs/content/` in `hookdeck/outpost`. The OpenAPI spec describes shapes, not +behaviour, and a whole feature can be documented without appearing in it: this +scenario was written claiming a disabled destination produces no notification, +which was wrong — `alert.destination.disabled` is a documented **operator +event**, and it was missed by searching the spec and `internal/alert/` when the +answer was in `internal/opevents/`. It came within one step of being published +as a product finding. Being publicly documented cuts the other way too: the +sandbox has network access, so an agent can read these pages and a scenario may +fairly depend on them. + **Retry against a disabled destination returns `400 "Destination is disabled"`.** Verified against the live API, and it is the mechanism `outpost-002` rests on: re-enabling and retrying are ordered by the product, not by the scenario. Worth