Add outpost-002: a disabled destination holding a customer's events

AGENTS.md

@@ -575,10 +575,62 @@ restores the Hookdeck project, and knows nothing about Outpost. Tenants and
 destinations an Outpost run creates survive into the next one, so the second run
 of a scenario finds `acme` already there and may score a previous run's work.
 `FixedProjectSource` now deletes tenants that were not present when it acquired
-the lease, on release. The residual: a run that dies before release still
-leaks, and the next run inherits it.
-Until it is, treat Outpost results after the first run of a scenario as
-unreliable, and delete tenants by hand between runs.
+the lease, on release — but that runs in a `catch`-and-ignore, so a run that dies
+before release still leaks and the next run inherits it.
+
+So cleanup on release is not enough on its own, and `applyOutpostSeed` deletes
+each tenant it is about to create. That makes seeding idempotent without
+depending on the previous run having exited cleanly, which is what the leftover
+tenant actually broke: tenant create is idempotent on the id and destination
+create is not, so seeding onto a survivor *appended* a second destination rather
+than replacing the first. The scenario then started with two, one carrying the
+seeded history and one empty, and a scorer reading `[0]` got whichever sorted
+first. Scorers should still aggregate across a tenant's destinations rather than
+taking the first, because an agent may legitimately add one.
+
+**Outpost seeds must wait for delivery before mutating, and the wait needs its
+own reason.** Publishing is synchronous and delivery is not, so an `after` block
+lands while the events are still queued — the same trap `applySeed` fixes for the
+gateway. On Outpost it is worse than a slow start: `outpost-002` disables a
+destination in `after`, a disabled destination is never attempted, so the failed
+attempts the whole scenario is built on were never created. It presented as a
+seed that worked, because an earlier run's tenant had survived and had been
+delivering while it sat there. `applyOutpostSeed` now waits for published events
+to be attempted, any status, before running `after`.
+
+**Outpost event history outlives the tenant.** Deleting and recreating `acme`
+leaves every event it ever received in place: a freshly recreated tenant listed
+38 events against the 3 that run had published. They stay bound to destinations
+that no longer exist, so retrying one answers `404 "event not found"` rather than
+anything explanatory. Filter by `destination_id` — and note that attempts are
+naturally run-scoped only because the destination is new each run, which is a
+property to rely on deliberately rather than by accident.
+
+**The hosted Outpost API is not shaped like its tenant-scoped routes suggest.**
+`/events`, `/attempts` and `/retry` are **top-level**, filtered by query
+parameter, while destinations are under `/tenants/{id}/`. Guessing
+`/tenants/{id}/events` returns a 404 whose body is an HTML page, which reads like
+a broken deployment rather than a wrong path — about an hour went into probing
+route shapes that were never going to exist. The spec is
+`docs/apis/openapi.yaml` in `hookdeck/outpost`; the API host serves no
+`openapi.json`, so fetch it from the repository.
+
+**Read the Outpost prose docs before asserting a feature is absent.** They are
+published at <https://hookdeck.com/docs/outpost> and live as `.mdoc` under
+`docs/content/` in `hookdeck/outpost`. The OpenAPI spec describes shapes, not
+behaviour, and a whole feature can be documented without appearing in it: this
+scenario was written claiming a disabled destination produces no notification,
+which was wrong — `alert.destination.disabled` is a documented **operator
+event**, and it was missed by searching the spec and `internal/alert/` when the
+answer was in `internal/opevents/`. It came within one step of being published
+as a product finding. Being publicly documented cuts the other way too: the
+sandbox has network access, so an agent can read these pages and a scenario may
+fairly depend on them.
+
+**Retry against a disabled destination returns `400 "Destination is disabled"`.**
+Verified against the live API, and it is the mechanism `outpost-002` rests on:
+re-enabling and retrying are ordered by the product, not by the scenario. Worth
+knowing before writing any scenario that assumes held events can be recovered.
 
 **Reset is to pristine, not to empty.** A new Hookdeck project ships with
 default issue triggers. The first acquire snapshots what the project contains,

apps/framework/scripts/score-only.ts

@@ -84,6 +84,19 @@ function readFlag(name: string): string | undefined {
   return undefined;
 }
 
+/**
+ * Score the seeded state *without* applying the solution.
+ *
+ * The other half of what this script is for. A scorer that passes with the
+ * solution applied has been shown to accept a correct answer; it has not been
+ * shown to reject an incorrect one, and a scorer that passes unconditionally
+ * does the first perfectly. Every check that came back green here would be
+ * green for an agent that did nothing at all.
+ *
+ * So the expected outcome of this mode is **fail**. A pass is the bug.
+ */
+const NO_SOLUTION = rawArgs.includes('--no-solution');
+
 const EVAL_FILTER = readFlag('eval');
 const REPEAT = Number(readFlag('repeat') ?? 3);
 const EXPERIMENT = readFlag('experiment');
@@ -134,7 +147,7 @@ async function scoreRepeatedly(
     // scorer reading seeded history fails for reasons unrelated to itself.
     const session = await runtime.startSession(readSessionSeedArgs(ev));
     try {
-      if (solution) await solution(session.scoringContext);
+      if (solution && !NO_SOLUTION) await solution(session.scoringContext);
 
       const result = await scorer({
         ...session.scoringContext,
@@ -171,7 +184,7 @@ async function scoreRepeatedly(
 
     const latest = verdicts[verdicts.length - 1];
     process.stdout.write(
-      `  ${ev.id}${solution ? ' [solution]' : ''} ${i + 1}/${REPEAT}: ${latest.passed ? 'pass' : 'fail'}` +
+      `  ${ev.id}${solution && !NO_SOLUTION ? ' [solution]' : ''}${NO_SOLUTION ? ' [unsolved]' : ''} ${i + 1}/${REPEAT}: ${latest.passed ? 'pass' : 'fail'}` +
         `${latest.error ? ` (threw: ${latest.error.slice(0, 60)})` : ''}\n`
     );
   }

evals/benchmark-outpost-002-disabled-destination/EVAL.ts

@@ -0,0 +1,234 @@
+import type {
+  CheckResult,
+  ToolEvalContext,
+  ToolScorer,
+} from '@hookdeck-evals/core';
+import { waitForOrLast } from '@hookdeck-evals/hookdeck';
+
+/**
+ * Outpost's most reliable support case: a destination switched off after
+ * repeated failures, an endpoint since repaired, and nothing flowing.
+ *
+ * The trap is that nothing recovers on its own, and whether anything told you
+ * depends on configuration the customer's deployment may not have.
+ * Outpost retries automatically with exponential backoff, so an endpoint that
+ * breaks and heals needs no intervention — which is exactly why this scenario
+ * does not use that shape. A *disabled* destination is different: the
+ * documentation is explicit that events published to a tenant are not delivered
+ * to a disabled destination and that "disabled destinations cannot be retried
+ * until re-enabled". Retrying one anyway answers `400 "Destination is
+ * disabled"`, verified against the live API. So the events are held and the
+ * endpoint is healthy.
+ *
+ * Outpost is not silent about this by design: `alert.destination.disabled` is a
+ * documented operator event, alongside `alert.destination.consecutive_failure`
+ * at 50/70/90/100% of the threshold. But operator events are off until a sink is
+ * configured — Hookdeck Monitoring settings on managed, `OPERATION_EVENTS_TOPICS`
+ * plus a sink when self-hosted — so the scenario is set on a deployment where
+ * nobody did, which is why the customer is the one who noticed. An agent that
+ * also recommends turning them on has given better advice than the task asked
+ * for; the scorer neither requires nor penalises it.
+ *
+ * That combination is what makes it worth scoring. An agent that checks the
+ * endpoint finds it fine. An agent that republishes finds the new events held
+ * too. The only route through is noticing the destination's state.
+ *
+ * Scored on outcome rather than method, which is the lesson from
+ * `alerting-001`: that scorer asked *who created* an alert and failed the agent
+ * that repaired a broken one, which was the better answer. Here it does not
+ * matter whether the agent re-enables the existing destination or reaches the
+ * same end state another way — what matters is that Acme receives what they
+ * missed and receives what comes next.
+ *
+ * A second tenant is seeded and delivering normally. Recovery scoped too
+ * widely is a real failure mode, and it is the one that turns a fix into a
+ * second incident.
+ */
+const TENANT = 'acme';
+const OTHER_TENANT = 'globex';
+/** Polling ceiling, not a sleep. */
+const DELIVERY_WAIT_MS = 45_000;
+/** Seeded before the agent ran, all of which failed against the bad endpoint. */
+const MISSED_EVENTS = 3;
+
+interface Destination {
+  id?: string;
+  disabled_at?: string | null;
+}
+
+interface Attempt {
+  id?: string;
+  status?: string;
+}
+
+const scorer: ToolScorer = async (ctx) => {
+  const destinations = await listDestinations(ctx, TENANT);
+  if (destinations.length === 0) {
+    return {
+      passed: false,
+      checks: [
+        {
+          name: "the customer's destination still exists",
+          passed: false,
+          notes:
+            'no destination for acme: the seeded one was removed rather than repaired, ' +
+            'which loses the delivery history the missed events are attached to',
+        },
+      ],
+    };
+  }
+
+  // *Any* enabled destination, not the first one listed. Re-enabling the
+  // seeded destination is the expected route, but an agent that instead adds a
+  // working one has also made acme able to receive, and this scenario scores the
+  // end state. Reading `[0]` would decide that on list order.
+  const enabled = destinations.filter((d) => !d.disabled_at);
+
+  const checks: CheckResult[] = [
+    {
+      // Not "the agent called /enable": an end state reached another way is
+      // still the end state.
+      name: 'the customer can receive again',
+      passed: enabled.length > 0,
+      notes:
+        enabled.length > 0
+          ? undefined
+          : `all ${destinations.length} of acme's destinations are still disabled, ` +
+            'so nothing published to acme will be delivered',
+    },
+    await checkMissedEventsDelivered(ctx, destinations),
+    await checkOtherTenantUntouched(ctx),
+  ];
+
+  return { passed: checks.every((c) => c.passed), checks };
+};
+
+export default scorer;
+
+/**
+ * The events sent before the agent ran, which failed and were then held when
+ * the destination was disabled.
+ *
+ * This is the check the scenario exists for. Re-enabling the destination alone
+ * makes *future* events flow and leaves the customer missing everything from
+ * the outage — which is what they wrote in about. Outpost holds those attempts
+ * and exposes a retry, so recovering them is possible and is the actual job.
+ */
+async function checkMissedEventsDelivered(
+  ctx: ToolEvalContext,
+  destinations: Destination[]
+): Promise<CheckResult> {
+  const name = 'the events the customer missed were delivered';
+
+  // Poll: a retry triggered moments before the agent finished is still in
+  // flight, and reading once would score the agent for the platform's timing.
+  // Summed across the tenant's destinations, for the same reason: an agent that
+  // recovered the events onto a replacement destination delivered them.
+  const attempts = await waitForOrLast(
+    () => listAllAttempts(ctx, TENANT, destinations),
+    (rows) =>
+      rows.filter((a) => a.status === 'success').length >= MISSED_EVENTS,
+    {
+      timeoutMs: DELIVERY_WAIT_MS,
+      description: 'the held events to be delivered',
+    }
+  );
+
+  const delivered = attempts.filter((a) => a.status === 'success').length;
+  return {
+    name,
+    passed: delivered >= MISSED_EVENTS,
+    notes:
+      delivered >= MISSED_EVENTS
+        ? undefined
+        : `${delivered} of ${MISSED_EVENTS} missed events delivered: the destination may be ` +
+          'receiving again, but the customer is still missing the outage window',
+  };
+}
+
+/**
+ * The negative, and the reason it carries weight: an agent that recovers
+ * everything rather than what was asked has turned a fix into a second
+ * incident. Globex was never broken.
+ */
+async function checkOtherTenantUntouched(
+  ctx: ToolEvalContext
+): Promise<CheckResult> {
+  const name = 'the other customer was left alone';
+  const destinations = await listDestinations(ctx, OTHER_TENANT);
+
+  if (destinations.length === 0) {
+    return {
+      name,
+      passed: false,
+      notes: `no destination for ${OTHER_TENANT}: it was removed, and it was never part of the problem`,
+    };
+  }
+
+  // Every one of them, not the first: collateral damage to the second
+  // destination of a tenant is still collateral damage.
+  const disabled = destinations.filter((d) => d.disabled_at);
+  return {
+    name,
+    passed: disabled.length === 0,
+    notes:
+      disabled.length === 0
+        ? undefined
+        : `${disabled.length} of ${OTHER_TENANT}'s destinations were disabled, so fixing acme ` +
+          'broke a customer who was working',
+  };
+}
+
+async function listDestinations(
+  ctx: ToolEvalContext,
+  tenantId: string
+): Promise<Destination[]> {
+  const rows = await ctx.outpost?.<Destination[] | { models?: Destination[] }>(
+    'GET',
+    `/tenants/${encodeURIComponent(tenantId)}/destinations`
+  );
+  return unwrap<Destination>(rows);
+}
+
+/** Every attempt across the tenant's destinations. */
+async function listAllAttempts(
+  ctx: ToolEvalContext,
+  tenantId: string,
+  destinations: Destination[]
+): Promise<Attempt[]> {
+  const perDestination = await Promise.all(
+    destinations
+      .filter((d): d is Destination & { id: string } => Boolean(d.id))
+      .map((d) => listAttempts(ctx, tenantId, d.id))
+  );
+  return perDestination.flat();
+}
+
+async function listAttempts(
+  ctx: ToolEvalContext,
+  tenantId: string,
+  destinationId: string
+): Promise<Attempt[]> {
+  const rows = await ctx.outpost?.<Attempt[] | { models?: Attempt[] }>(
+    'GET',
+    `/tenants/${encodeURIComponent(tenantId)}/destinations/${encodeURIComponent(destinationId)}/attempts`
+  );
+  return unwrap<Attempt>(rows);
+}
+
+/**
+ * Outpost list endpoints return `{ pagination, models }`, the same shape as
+ * Hookdeck's own, and a bare array on some.
+ *
+ * **Not `{ data }`.** `outpost-001` carries a comment saying so, because
+ * reading `data` never matches and silently reports zero — which is exactly
+ * what this scorer did on its first run: no attempts anywhere, on a destination
+ * that had them. A shape mismatch here does not error, it just quietly answers
+ * "nothing happened", which is indistinguishable from an agent that did
+ * nothing.
+ */
+function unwrap<T>(rows: T[] | { models?: T[]; data?: T[] } | undefined): T[] {
+  if (!rows) return [];
+  if (Array.isArray(rows)) return rows;
+  return rows.models ?? rows.data ?? [];
+}

evals/benchmark-outpost-002-disabled-destination/PROMPT.md

@@ -0,0 +1,18 @@
+---
+stage: resolve
+suite: benchmark
+product:
+  - outpost
+topic:
+  - retries
+requires:
+  - outpost
+motivation: The support case Outpost generates most reliably. A destination that keeps failing is auto-disabled to protect the system, and once the customer repairs their endpoint nothing starts again on its own — the held events are not retried until the destination is re-enabled. Outpost does emit `alert.destination.disabled` as an operator event, but only where a sink has been configured for it (Hookdeck Monitoring settings on managed, `OPERATION_EVENTS_TOPICS` plus a sink when self-hosted); on a deployment where nobody has, the first signal is the customer.
+---
+
+Acme emailed to say they stopped receiving order events some time yesterday.
+Their engineer says their endpoint had a bad deploy but it's been fine since
+this morning, and they've checked — nothing is arriving.
+
+Work out why and get their events flowing again, including the ones they
+missed.