diff --git a/AGENTS.md b/AGENTS.md index d889a46..d375ea4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -575,10 +575,62 @@ restores the Hookdeck project, and knows nothing about Outpost. Tenants and destinations an Outpost run creates survive into the next one, so the second run of a scenario finds `acme` already there and may score a previous run's work. `FixedProjectSource` now deletes tenants that were not present when it acquired -the lease, on release. The residual: a run that dies before release still -leaks, and the next run inherits it. -Until it is, treat Outpost results after the first run of a scenario as -unreliable, and delete tenants by hand between runs. +the lease, on release — but that runs in a `catch`-and-ignore, so a run that dies +before release still leaks and the next run inherits it. + +So cleanup on release is not enough on its own, and `applyOutpostSeed` deletes +each tenant it is about to create. That makes seeding idempotent without +depending on the previous run having exited cleanly, which is what the leftover +tenant actually broke: tenant create is idempotent on the id and destination +create is not, so seeding onto a survivor *appended* a second destination rather +than replacing the first. The scenario then started with two, one carrying the +seeded history and one empty, and a scorer reading `[0]` got whichever sorted +first. Scorers should still aggregate across a tenant's destinations rather than +taking the first, because an agent may legitimately add one. + +**Outpost seeds must wait for delivery before mutating, and the wait needs its +own reason.** Publishing is synchronous and delivery is not, so an `after` block +lands while the events are still queued — the same trap `applySeed` fixes for the +gateway. On Outpost it is worse than a slow start: `outpost-002` disables a +destination in `after`, a disabled destination is never attempted, so the failed +attempts the whole scenario is built on were never created. It presented as a +seed that worked, because an earlier run's tenant had survived and had been +delivering while it sat there. `applyOutpostSeed` now waits for published events +to be attempted, any status, before running `after`. + +**Outpost event history outlives the tenant.** Deleting and recreating `acme` +leaves every event it ever received in place: a freshly recreated tenant listed +38 events against the 3 that run had published. They stay bound to destinations +that no longer exist, so retrying one answers `404 "event not found"` rather than +anything explanatory. Filter by `destination_id` — and note that attempts are +naturally run-scoped only because the destination is new each run, which is a +property to rely on deliberately rather than by accident. + +**The hosted Outpost API is not shaped like its tenant-scoped routes suggest.** +`/events`, `/attempts` and `/retry` are **top-level**, filtered by query +parameter, while destinations are under `/tenants/{id}/`. Guessing +`/tenants/{id}/events` returns a 404 whose body is an HTML page, which reads like +a broken deployment rather than a wrong path — about an hour went into probing +route shapes that were never going to exist. The spec is +`docs/apis/openapi.yaml` in `hookdeck/outpost`; the API host serves no +`openapi.json`, so fetch it from the repository. + +**Read the Outpost prose docs before asserting a feature is absent.** They are +published at and live as `.mdoc` under +`docs/content/` in `hookdeck/outpost`. The OpenAPI spec describes shapes, not +behaviour, and a whole feature can be documented without appearing in it: this +scenario was written claiming a disabled destination produces no notification, +which was wrong — `alert.destination.disabled` is a documented **operator +event**, and it was missed by searching the spec and `internal/alert/` when the +answer was in `internal/opevents/`. It came within one step of being published +as a product finding. Being publicly documented cuts the other way too: the +sandbox has network access, so an agent can read these pages and a scenario may +fairly depend on them. + +**Retry against a disabled destination returns `400 "Destination is disabled"`.** +Verified against the live API, and it is the mechanism `outpost-002` rests on: +re-enabling and retrying are ordered by the product, not by the scenario. Worth +knowing before writing any scenario that assumes held events can be recovered. **Reset is to pristine, not to empty.** A new Hookdeck project ships with default issue triggers. The first acquire snapshots what the project contains, diff --git a/apps/framework/scripts/score-only.ts b/apps/framework/scripts/score-only.ts index 8eb8279..cbe61ce 100644 --- a/apps/framework/scripts/score-only.ts +++ b/apps/framework/scripts/score-only.ts @@ -84,6 +84,19 @@ function readFlag(name: string): string | undefined { return undefined; } +/** + * Score the seeded state *without* applying the solution. + * + * The other half of what this script is for. A scorer that passes with the + * solution applied has been shown to accept a correct answer; it has not been + * shown to reject an incorrect one, and a scorer that passes unconditionally + * does the first perfectly. Every check that came back green here would be + * green for an agent that did nothing at all. + * + * So the expected outcome of this mode is **fail**. A pass is the bug. + */ +const NO_SOLUTION = rawArgs.includes('--no-solution'); + const EVAL_FILTER = readFlag('eval'); const REPEAT = Number(readFlag('repeat') ?? 3); const EXPERIMENT = readFlag('experiment'); @@ -134,7 +147,7 @@ async function scoreRepeatedly( // scorer reading seeded history fails for reasons unrelated to itself. const session = await runtime.startSession(readSessionSeedArgs(ev)); try { - if (solution) await solution(session.scoringContext); + if (solution && !NO_SOLUTION) await solution(session.scoringContext); const result = await scorer({ ...session.scoringContext, @@ -171,7 +184,7 @@ async function scoreRepeatedly( const latest = verdicts[verdicts.length - 1]; process.stdout.write( - ` ${ev.id}${solution ? ' [solution]' : ''} ${i + 1}/${REPEAT}: ${latest.passed ? 'pass' : 'fail'}` + + ` ${ev.id}${solution && !NO_SOLUTION ? ' [solution]' : ''}${NO_SOLUTION ? ' [unsolved]' : ''} ${i + 1}/${REPEAT}: ${latest.passed ? 'pass' : 'fail'}` + `${latest.error ? ` (threw: ${latest.error.slice(0, 60)})` : ''}\n` ); } diff --git a/evals/benchmark-outpost-002-disabled-destination/EVAL.ts b/evals/benchmark-outpost-002-disabled-destination/EVAL.ts new file mode 100644 index 0000000..e28cf05 --- /dev/null +++ b/evals/benchmark-outpost-002-disabled-destination/EVAL.ts @@ -0,0 +1,234 @@ +import type { + CheckResult, + ToolEvalContext, + ToolScorer, +} from '@hookdeck-evals/core'; +import { waitForOrLast } from '@hookdeck-evals/hookdeck'; + +/** + * Outpost's most reliable support case: a destination switched off after + * repeated failures, an endpoint since repaired, and nothing flowing. + * + * The trap is that nothing recovers on its own, and whether anything told you + * depends on configuration the customer's deployment may not have. + * Outpost retries automatically with exponential backoff, so an endpoint that + * breaks and heals needs no intervention — which is exactly why this scenario + * does not use that shape. A *disabled* destination is different: the + * documentation is explicit that events published to a tenant are not delivered + * to a disabled destination and that "disabled destinations cannot be retried + * until re-enabled". Retrying one anyway answers `400 "Destination is + * disabled"`, verified against the live API. So the events are held and the + * endpoint is healthy. + * + * Outpost is not silent about this by design: `alert.destination.disabled` is a + * documented operator event, alongside `alert.destination.consecutive_failure` + * at 50/70/90/100% of the threshold. But operator events are off until a sink is + * configured — Hookdeck Monitoring settings on managed, `OPERATION_EVENTS_TOPICS` + * plus a sink when self-hosted — so the scenario is set on a deployment where + * nobody did, which is why the customer is the one who noticed. An agent that + * also recommends turning them on has given better advice than the task asked + * for; the scorer neither requires nor penalises it. + * + * That combination is what makes it worth scoring. An agent that checks the + * endpoint finds it fine. An agent that republishes finds the new events held + * too. The only route through is noticing the destination's state. + * + * Scored on outcome rather than method, which is the lesson from + * `alerting-001`: that scorer asked *who created* an alert and failed the agent + * that repaired a broken one, which was the better answer. Here it does not + * matter whether the agent re-enables the existing destination or reaches the + * same end state another way — what matters is that Acme receives what they + * missed and receives what comes next. + * + * A second tenant is seeded and delivering normally. Recovery scoped too + * widely is a real failure mode, and it is the one that turns a fix into a + * second incident. + */ +const TENANT = 'acme'; +const OTHER_TENANT = 'globex'; +/** Polling ceiling, not a sleep. */ +const DELIVERY_WAIT_MS = 45_000; +/** Seeded before the agent ran, all of which failed against the bad endpoint. */ +const MISSED_EVENTS = 3; + +interface Destination { + id?: string; + disabled_at?: string | null; +} + +interface Attempt { + id?: string; + status?: string; +} + +const scorer: ToolScorer = async (ctx) => { + const destinations = await listDestinations(ctx, TENANT); + if (destinations.length === 0) { + return { + passed: false, + checks: [ + { + name: "the customer's destination still exists", + passed: false, + notes: + 'no destination for acme: the seeded one was removed rather than repaired, ' + + 'which loses the delivery history the missed events are attached to', + }, + ], + }; + } + + // *Any* enabled destination, not the first one listed. Re-enabling the + // seeded destination is the expected route, but an agent that instead adds a + // working one has also made acme able to receive, and this scenario scores the + // end state. Reading `[0]` would decide that on list order. + const enabled = destinations.filter((d) => !d.disabled_at); + + const checks: CheckResult[] = [ + { + // Not "the agent called /enable": an end state reached another way is + // still the end state. + name: 'the customer can receive again', + passed: enabled.length > 0, + notes: + enabled.length > 0 + ? undefined + : `all ${destinations.length} of acme's destinations are still disabled, ` + + 'so nothing published to acme will be delivered', + }, + await checkMissedEventsDelivered(ctx, destinations), + await checkOtherTenantUntouched(ctx), + ]; + + return { passed: checks.every((c) => c.passed), checks }; +}; + +export default scorer; + +/** + * The events sent before the agent ran, which failed and were then held when + * the destination was disabled. + * + * This is the check the scenario exists for. Re-enabling the destination alone + * makes *future* events flow and leaves the customer missing everything from + * the outage — which is what they wrote in about. Outpost holds those attempts + * and exposes a retry, so recovering them is possible and is the actual job. + */ +async function checkMissedEventsDelivered( + ctx: ToolEvalContext, + destinations: Destination[] +): Promise { + const name = 'the events the customer missed were delivered'; + + // Poll: a retry triggered moments before the agent finished is still in + // flight, and reading once would score the agent for the platform's timing. + // Summed across the tenant's destinations, for the same reason: an agent that + // recovered the events onto a replacement destination delivered them. + const attempts = await waitForOrLast( + () => listAllAttempts(ctx, TENANT, destinations), + (rows) => + rows.filter((a) => a.status === 'success').length >= MISSED_EVENTS, + { + timeoutMs: DELIVERY_WAIT_MS, + description: 'the held events to be delivered', + } + ); + + const delivered = attempts.filter((a) => a.status === 'success').length; + return { + name, + passed: delivered >= MISSED_EVENTS, + notes: + delivered >= MISSED_EVENTS + ? undefined + : `${delivered} of ${MISSED_EVENTS} missed events delivered: the destination may be ` + + 'receiving again, but the customer is still missing the outage window', + }; +} + +/** + * The negative, and the reason it carries weight: an agent that recovers + * everything rather than what was asked has turned a fix into a second + * incident. Globex was never broken. + */ +async function checkOtherTenantUntouched( + ctx: ToolEvalContext +): Promise { + const name = 'the other customer was left alone'; + const destinations = await listDestinations(ctx, OTHER_TENANT); + + if (destinations.length === 0) { + return { + name, + passed: false, + notes: `no destination for ${OTHER_TENANT}: it was removed, and it was never part of the problem`, + }; + } + + // Every one of them, not the first: collateral damage to the second + // destination of a tenant is still collateral damage. + const disabled = destinations.filter((d) => d.disabled_at); + return { + name, + passed: disabled.length === 0, + notes: + disabled.length === 0 + ? undefined + : `${disabled.length} of ${OTHER_TENANT}'s destinations were disabled, so fixing acme ` + + 'broke a customer who was working', + }; +} + +async function listDestinations( + ctx: ToolEvalContext, + tenantId: string +): Promise { + const rows = await ctx.outpost?.( + 'GET', + `/tenants/${encodeURIComponent(tenantId)}/destinations` + ); + return unwrap(rows); +} + +/** Every attempt across the tenant's destinations. */ +async function listAllAttempts( + ctx: ToolEvalContext, + tenantId: string, + destinations: Destination[] +): Promise { + const perDestination = await Promise.all( + destinations + .filter((d): d is Destination & { id: string } => Boolean(d.id)) + .map((d) => listAttempts(ctx, tenantId, d.id)) + ); + return perDestination.flat(); +} + +async function listAttempts( + ctx: ToolEvalContext, + tenantId: string, + destinationId: string +): Promise { + const rows = await ctx.outpost?.( + 'GET', + `/tenants/${encodeURIComponent(tenantId)}/destinations/${encodeURIComponent(destinationId)}/attempts` + ); + return unwrap(rows); +} + +/** + * Outpost list endpoints return `{ pagination, models }`, the same shape as + * Hookdeck's own, and a bare array on some. + * + * **Not `{ data }`.** `outpost-001` carries a comment saying so, because + * reading `data` never matches and silently reports zero — which is exactly + * what this scorer did on its first run: no attempts anywhere, on a destination + * that had them. A shape mismatch here does not error, it just quietly answers + * "nothing happened", which is indistinguishable from an agent that did + * nothing. + */ +function unwrap(rows: T[] | { models?: T[]; data?: T[] } | undefined): T[] { + if (!rows) return []; + if (Array.isArray(rows)) return rows; + return rows.models ?? rows.data ?? []; +} diff --git a/evals/benchmark-outpost-002-disabled-destination/PROMPT.md b/evals/benchmark-outpost-002-disabled-destination/PROMPT.md new file mode 100644 index 0000000..7356824 --- /dev/null +++ b/evals/benchmark-outpost-002-disabled-destination/PROMPT.md @@ -0,0 +1,18 @@ +--- +stage: resolve +suite: benchmark +product: + - outpost +topic: + - retries +requires: + - outpost +motivation: The support case Outpost generates most reliably. A destination that keeps failing is auto-disabled to protect the system, and once the customer repairs their endpoint nothing starts again on its own — the held events are not retried until the destination is re-enabled. Outpost does emit `alert.destination.disabled` as an operator event, but only where a sink has been configured for it (Hookdeck Monitoring settings on managed, `OPERATION_EVENTS_TOPICS` plus a sink when self-hosted); on a deployment where nobody has, the first signal is the customer. +--- + +Acme emailed to say they stopped receiving order events some time yesterday. +Their engineer says their endpoint had a bad deploy but it's been fine since +this morning, and they've checked — nothing is arriving. + +Work out why and get their events flowing again, including the ones they +missed. diff --git a/evals/benchmark-outpost-002-disabled-destination/SOLUTION.ts b/evals/benchmark-outpost-002-disabled-destination/SOLUTION.ts new file mode 100644 index 0000000..2732859 --- /dev/null +++ b/evals/benchmark-outpost-002-disabled-destination/SOLUTION.ts @@ -0,0 +1,110 @@ +import type { ToolEvalContext } from '@hookdeck-evals/core'; +import { waitFor } from '@hookdeck-evals/hookdeck'; + +/** + * What a correct agent leaves behind, so `score-only` can exercise the scorer + * without paying for a run. + * + * The route through this scenario is two steps, and the second is the one that + * matters: re-enabling the destination makes *future* events flow and leaves the + * customer still missing the outage window. `POST /retry` is what recovers it. + * + * The order is not a stylistic choice — the API enforces it. Retrying while the + * destination is disabled returns `400 "Destination is disabled"`, which is the + * documented behaviour this whole scenario is built on. Verified against the + * live API on 20 August, not inferred from the spec. + */ + +const TENANT = 'acme'; + +interface Destination { + id?: string; + disabled_at?: string | null; +} + +interface OutpostEvent { + id?: string; +} + +interface Attempt { + status?: string; +} + +export default async function solve(ctx: ToolEvalContext): Promise { + const outpost = ctx.outpost; + if (!outpost) { + throw new Error( + 'no Outpost client: this solution cannot be applied without OUTPOST_API_KEY, ' + + 'and applying half of it would score a state no agent produced' + ); + } + + const destination = ( + await list(ctx, `/tenants/${TENANT}/destinations`) + ).find((d) => d.id); + + if (!destination?.id) { + throw new Error(`no destination for ${TENANT}: the seed did not apply`); + } + + // 1. Re-enable. Nothing else is possible until this lands. + await outpost( + 'PUT', + `/tenants/${TENANT}/destinations/${destination.id}/enable` + ); + + // 2. Retry what was held. + // + // Scoped by `destination_id`, which is load-bearing rather than tidy. Event + // history outlives the tenant: deleting and recreating `acme` leaves its old + // events in place, so an unfiltered list returns everything every previous run + // published — 38 of them at the time of writing, against 3 belonging to this + // run. Retrying one of those answers `404 "event not found"`, because it + // matched a destination that no longer exists. + const events = await list( + ctx, + `/events?tenant_id=${TENANT}&destination_id=${encodeURIComponent(destination.id)}` + ); + + for (const event of events) { + if (!event.id) continue; + await outpost('POST', '/retry', { + event_id: event.id, + destination_id: destination.id, + }); + } + + // 3. Do not return until the retries have actually been delivered. + // + // `POST /retry` answers 202: accepted, not delivered. A solution that returns + // on the acknowledgement races the scorer it exists to serve, and the failure + // is indistinguishable from the scorer flake this is meant to rule out. Same + // rule as `setConnectionRules` in `packages/hookdeck/src/solutions.ts`. + // + // Delivery is monotonic here — a successful attempt does not stop being one — + // so plain `waitFor` is right, and `waitForConsistent` would only be slower. + await waitFor( + () => + list( + ctx, + `/tenants/${TENANT}/destinations/${destination.id}/attempts` + ), + (attempts) => + attempts.filter((a) => a.status === 'success').length >= events.length, + { + timeoutMs: 60_000, + description: `${events.length} retried event(s) to be delivered`, + } + ); +} + +/** Outpost list endpoints answer `{ pagination, models }`, not `{ data }`. */ +async function list(ctx: ToolEvalContext, path: string): Promise { + const rows = await ctx.outpost?.( + 'GET', + path + ); + if (!rows) return []; + if (Array.isArray(rows)) return rows; + return rows.models ?? rows.data ?? []; +} diff --git a/evals/benchmark-outpost-002-disabled-destination/remote/seed.json b/evals/benchmark-outpost-002-disabled-destination/remote/seed.json new file mode 100644 index 0000000..8ccf38a --- /dev/null +++ b/evals/benchmark-outpost-002-disabled-destination/remote/seed.json @@ -0,0 +1,60 @@ +{ + "outpost": { + "tenants": [ + { + "id": "acme", + "topics": ["order.created", "order.shipped"], + "destinations": [ + { + "ref": "acme-endpoint", + "type": "webhook", + "topics": ["order.created", "order.shipped"], + "config": { + "url": "https://mock.hookdeck.com/api/v1/acme/orders?status=503" + } + } + ] + }, + { + "id": "globex", + "topics": ["order.created", "order.shipped"], + "destinations": [ + { + "ref": "globex-endpoint", + "type": "webhook", + "topics": ["order.created", "order.shipped"], + "config": { + "url": "https://mock.hookdeck.com/api/v1/globex/orders" + } + } + ] + } + ], + "publish": [ + { + "tenant": "acme", + "topic": "order.created", + "data": { "order_id": "ord_a1", "total": 2400 }, + "count": 3 + }, + { + "tenant": "globex", + "topic": "order.created", + "data": { "order_id": "ord_g1", "total": 900 } + } + ], + "after": [ + { + "path": "/tenants/acme/destinations/$ref:acme-endpoint/disable", + "method": "PUT" + }, + { + "path": "/tenants/acme/destinations/$ref:acme-endpoint", + "method": "PATCH", + "body": { + "config": { "url": "https://mock.hookdeck.com/api/v1/acme/orders" } + } + } + ] + } +} diff --git a/packages/hookdeck/src/index.ts b/packages/hookdeck/src/index.ts index 5ec79f0..8176531 100644 --- a/packages/hookdeck/src/index.ts +++ b/packages/hookdeck/src/index.ts @@ -16,8 +16,14 @@ export type { PristineSnapshot, FixedProjectSourceOptions, } from './project-source.js'; -export { readSeed, applySeed } from './seed.js'; -export type { Seed, SeedResource, SeedEvent, AppliedSeed } from './seed.js'; +export { readSeed, applySeed, applyOutpostSeed } from './seed.js'; +export type { + Seed, + SeedResource, + SeedEvent, + OutpostSeed, + AppliedSeed, +} from './seed.js'; export { hookdeckRuntime, hookdeckMcpServer } from './runtime.js'; export type { HookdeckRuntimeOptions } from './runtime.js'; export { collectEnvSecretValues, redactSecrets } from './redact.js'; diff --git a/packages/hookdeck/src/runtime.ts b/packages/hookdeck/src/runtime.ts index cda8072..436084c 100644 --- a/packages/hookdeck/src/runtime.ts +++ b/packages/hookdeck/src/runtime.ts @@ -18,7 +18,7 @@ import type { } from '@hookdeck-evals/core'; import { OutpostClient } from './outpost-client.js'; import type { ProjectSource } from './project-source.js'; -import { applySeed, readSeed } from './seed.js'; +import { applyOutpostSeed, applySeed, readSeed } from './seed.js'; export interface HookdeckRuntimeOptions { projects: ProjectSource; @@ -60,6 +60,29 @@ export function hookdeckRuntime(options: HookdeckRuntimeOptions): EvalRuntime { ? new OutpostClient({ apiKey: outpostKey }) : undefined; + // The Outpost half of the seed, applied after the client exists. + // Loudly rather than silently: a scenario asking for Outpost state on a + // machine with no key would otherwise run against nothing and score the + // agent for a setup that was never there. Scenarios needing this + // declare `requires: [outpost]`, which skips them before they reach + // here — so arriving without a client means the requirement is missing, + // not that the machine is simply unconfigured. + if (args.remoteDir) { + const seed = readSeed(args.remoteDir); + if (seed?.outpost) { + if (!outpostClient) { + throw new Error( + 'seed declares outpost state but OUTPOST_API_KEY is not set; ' + + 'the scenario should declare `requires: [outpost]`' + ); + } + await applyOutpostSeed( + (method, path, body) => outpostClient.request(method, path, body), + seed.outpost + ); + } + } + const scoringContext: ToolScoringContext = { projectId: project.projectId, acquiredAt: project.acquiredAt, diff --git a/packages/hookdeck/src/seed.ts b/packages/hookdeck/src/seed.ts index e75ba1b..d9f5135 100644 --- a/packages/hookdeck/src/seed.ts +++ b/packages/hookdeck/src/seed.ts @@ -20,6 +20,7 @@ import { existsSync, readFileSync } from 'node:fs'; import { join } from 'node:path'; import type { HookdeckClient, HttpMethod, ResourceKind } from './client.js'; +import { waitFor } from './wait-for.js'; export interface SeedResource { kind: ResourceKind; @@ -57,8 +58,45 @@ export interface SeedEvent { count?: number; } +/** + * Outpost state a scenario starts from. + * + * Kept as its own section rather than folded into `resources`, because Outpost + * is a separate service: different base URL, different key, and a tenant model + * the gateway does not have. Sharing `ResourceKind` between them would make the + * seed file look uniform while the two halves went to different APIs. + * + * Applied only when an Outpost client is configured. A scenario needing this + * should declare `requires: [outpost]` so it reads as skipped rather than + * failing on a machine with no Outpost key. + */ +export interface OutpostSeed { + tenants?: { + id: string; + topics?: string[]; + destinations?: { + /** Referenced by `after` steps via `$ref:`. */ + ref?: string; + type?: string; + topics?: string[] | string; + config?: Record; + credentials?: Record; + }[]; + }[]; + /** Events published before the agent runs, so history exists to reason about. */ + publish?: { tenant: string; topic: string; data?: unknown; count?: number }[]; + /** + * Applied after publishing, for the same reason the gateway seed has one: + * a resolve scenario starts from history that already went wrong and a + * system that is now healthy. Disabling a destination here is how a scenario + * expresses "this was switched off after it kept failing". + */ + after?: SeedStep[]; +} + export interface Seed { resources?: SeedResource[]; + outpost?: OutpostSeed; events?: SeedEvent[]; /** * Requests applied after every event has been sent. @@ -239,3 +277,175 @@ function resolvePathRefs(path: string, refs: AppliedSeed['refs']): string { return target.id; }); } + +/** + * Apply the Outpost half of a seed. + * + * Separate from `applySeed` because it talks to a different service with its + * own client, and because a scenario can want gateway state, Outpost state, or + * both. The caller decides whether an Outpost client exists; this does not + * silently no-op, so a seed asking for Outpost state on a machine without a key + * fails loudly rather than running the scenario against nothing. + */ +/** How long published events are given to be attempted before the seed gives up. */ +const ATTEMPT_WAIT_MS = 60_000; + +type OutpostCall = ( + method: HttpMethod, + path: string, + body?: unknown +) => Promise; + +export async function applyOutpostSeed( + outpost: OutpostCall, + seed: OutpostSeed +): Promise<{ destinations: Record }> { + const destinations: Record = {}; + + for (const tenant of seed.tenants ?? []) { + // Delete first, so the seed is idempotent. + // + // Tenant create is idempotent on the id but destination create is not, so + // seeding onto a surviving tenant appends a second destination rather than + // replacing the first. The scenario then starts with two, one carrying the + // seeded history and one empty, and any scorer reading `[0]` gets whichever + // sorted first. That is how `alerting-001` came to publish a wrong result + // for twelve days — a leftover from an earlier run that nothing collected. + // + // Tenants are collected on release, but that runs in a `catch`-and-ignore + // so a crashed run leaves them behind. Deleting here does not depend on the + // previous run having exited cleanly. + await outpost('DELETE', `/tenants/${encodeURIComponent(tenant.id)}`).catch( + () => undefined + ); + await outpost('PUT', `/tenants/${encodeURIComponent(tenant.id)}`, { + ...(tenant.topics ? { topics: tenant.topics } : {}), + }); + + for (const destination of tenant.destinations ?? []) { + const created = await outpost<{ id: string }>( + 'POST', + `/tenants/${encodeURIComponent(tenant.id)}/destinations`, + { + type: destination.type ?? 'webhook', + topics: destination.topics ?? '*', + config: destination.config ?? {}, + ...(destination.credentials + ? { credentials: destination.credentials } + : {}), + } + ); + if (destination.ref) destinations[destination.ref] = created.id; + } + } + + const published = new Map(); + for (const event of seed.publish ?? []) { + for (let i = 0; i < (event.count ?? 1); i += 1) { + await outpost('POST', '/publish', { + tenant_id: event.tenant, + topic: event.topic, + data: event.data ?? {}, + }); + } + published.set( + event.tenant, + (published.get(event.tenant) ?? 0) + (event.count ?? 1) + ); + } + + // Wait for the published events to be *attempted* before running `after`. + // + // Publishing is synchronous and delivery is not, so `after` otherwise lands + // while the events are still queued. For this seed's shape that is not a slow + // start, it is a different scenario: `after` disables the destination, and a + // disabled destination is never attempted, so the failed attempts the scorer + // looks for are never created at all. Verified — with a fresh tenant, acme + // ended with zero attempts; the run before it showed three only because the + // tenant had survived from an earlier run and had been delivering while it + // sat there. + // + // This is the same trap `applySeed` fixes on the gateway side, where + // `resolve-002` repaired an endpoint before the first delivery attempt and + // left nothing to redeliver. Second time, same cause. + await waitForPublishedAttempts(outpost, published); + + for (const step of seed.after ?? []) { + const path = step.path.replace( + /\$ref:([a-zA-Z0-9_-]+)/g, + (_, ref: string) => destinations[ref] ?? `$ref:${ref}` + ); + await outpost(step.method ?? 'PUT', path, step.body); + } + + return { destinations }; +} + +/** + * Poll until each tenant has at least as many delivery attempts as it had + * events published. + * + * Any status counts. The point is that Outpost has *tried*, not that it + * succeeded: a seed deliberately pointing at a failing endpoint wants the + * failures, and waiting for success would hang forever on exactly the scenarios + * that need this most. + * + * Throws on timeout rather than continuing. A seed that cannot establish its + * own precondition has not set up the scenario, and running the agent against + * a state that quietly differs from the intended one produces a result that + * looks valid and is not — which is worth more than the cost of a failed run. + */ +async function waitForPublishedAttempts( + outpost: OutpostCall, + published: Map +): Promise { + for (const [tenant, expected] of published) { + if (expected < 1) continue; + await waitFor( + () => countAttempts(outpost, tenant), + (seen) => seen >= expected, + { + timeoutMs: ATTEMPT_WAIT_MS, + description: `${expected} delivery attempt(s) on tenant ${tenant}`, + } + ); + } +} + +async function countAttempts( + outpost: OutpostCall, + tenant: string +): Promise { + const destinations = unwrapModels<{ id?: string }>( + await outpost<{ id?: string }[] | { models?: { id?: string }[] }>( + 'GET', + `/tenants/${encodeURIComponent(tenant)}/destinations` + ) + ); + let total = 0; + for (const destination of destinations) { + if (!destination.id) continue; + total += unwrapModels( + await outpost( + 'GET', + `/tenants/${encodeURIComponent(tenant)}/destinations/${encodeURIComponent(destination.id)}/attempts` + ) + ).length; + } + return total; +} + +/** + * Outpost list endpoints return `{ pagination, models }`, not `{ data }`. + * + * Reading the wrong key does not error, it returns nothing — so a wait built on + * it would time out on a tenant that was delivering perfectly well. Both + * `outpost-001` and `outpost-002` have been caught by this. + */ +function unwrapModels( + rows: T[] | { models?: T[]; data?: T[] } | undefined +): T[] { + if (!rows) return []; + if (Array.isArray(rows)) return rows; + return rows.models ?? rows.data ?? []; +}