Skip to content

Commit 38e78f8

Browse files
authored
feat(webapp): deployment lifecycle telemetry events (#4778)
Deployments currently leave little analytical trace. This PR makes every deployment emit two analytics events to enable useful queries. It also enables comparing deployments across build paths, CLI versions, runtimes, and orgs. ### Where the events come from ``` trigger deploy │ ▼ initialize ─────────────────────────────▶ ✨ deployment.initialized │ createdAt ▼ PENDING waiting for a build slot ┐ │ startedAt │ queue time ▼ ┘ INSTALLING build server installs deps ┐ │ installedAt (native paths only) │ install time ▼ ┘ BUILDING the image is built ┐ │ builtAt │ building time ▼ ┘ DEPLOYING indexing + registry push ┐ │ deployedAt / failedAt / canceledAt │ deploying time ▼ ┘ DEPLOYED · FAILED · TIMED_OUT · CANCELED │ └───────────────────────────────────▶ ✨ deployment.finished ``` `deployment.finished` fires exactly once, whichever way the deployment ends, and is backdated to cover the deployment's real lifetime. Not every path visits every state (Depot deploys skip PENDING/INSTALLING, for example) — a phase duration is simply omitted when its state was never entered. ### What each event carries - **Which path built it**: `depot`, `native`, or `native_local_bundle` - **How it ended**: status, plus an error class and message when it failed - **How long each phase took**: queue, install, building, deploying, and total — derived from the timestamps above - **Who and with what**: org, project, environment, runtime, CLI version, and how the deploy was triggered (CLI, GitHub, Vercel) With that, one query gives failure rate per build path, duration percentiles per phase, adoption per CLI version, or a per-org health table. ### Fixes that ride along - The old `deployment.outcome` span was silently dropped ~95% of the time (it was subject to trace sampling). The new events opt out of sampling explicitly, so every deployment is counted. - The fail/timeout/finalize transitions were racy: a late timeout could overwrite a successful deployment. They now use guarded writes, so exactly one caller wins the terminal transition — and exactly one event is emitted. - Canceled deployments previously recorded nothing; they do now. - The deployment's CLI version is now stored at initialization (new nullable column), so even deploys that fail early are attributable to a CLI release. - Telemetry is flushed on shutdown (the last batch used to be lost on every webapp deploy), and an optional second exporter can mirror just these events into a dedicated dataset.
1 parent 00e3c15 commit 38e78f8

17 files changed

Lines changed: 661 additions & 113 deletions

.changeset/violet-buses-tease.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"trigger.dev": patch
3+
---
4+
5+
Send the CLI version header on all API requests so deployments are attributable to a CLI version

apps/webapp/app/env.server.ts

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -940,6 +940,10 @@ const EnvironmentSchema = z
940940
DISABLE_HTTP_INSTRUMENTATION: BoolEnv.default(false),
941941

942942
INTERNAL_OTEL_LOG_EXPORTER_URL: z.string().optional(),
943+
944+
// Second trace exporter receiving only `deployment.*` spans; they still flow to the main one
945+
INTERNAL_OTEL_DEPLOYMENT_EVENT_EXPORTER_URL: z.string().optional(),
946+
INTERNAL_OTEL_DEPLOYMENT_EVENT_EXPORTER_AUTH_HEADERS: z.string().optional(),
943947
INTERNAL_OTEL_METRIC_EXPORTER_URL: z.string().optional(),
944948
INTERNAL_OTEL_METRIC_EXPORTER_AUTH_HEADERS: z.string().optional(),
945949
INTERNAL_OTEL_METRIC_EXPORTER_ENABLED: z.string().default("0"),

apps/webapp/app/routes/api.v1.deployments.ts

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,9 @@ export async function action({ request, params }: ActionFunctionArgs) {
4242
const service = new InitializeDeploymentService();
4343

4444
try {
45-
const result = await service.call(authenticatedEnv, body.data);
45+
const result = await service.call(authenticatedEnv, body.data, {
46+
cliVersion: parseCliVersionHeader(request),
47+
});
4648
const { deployment, imageRef } = result;
4749

4850
const responseBody: InitializeDeploymentResponseBody = {
@@ -75,6 +77,14 @@ export async function action({ request, params }: ActionFunctionArgs) {
7577
}
7678
}
7779

80+
// Client-controlled and persisted, so cap what we accept
81+
const CLI_VERSION_MAX_LENGTH = 128;
82+
83+
function parseCliVersionHeader(request: Request): string | undefined {
84+
const value = request.headers.get("x-trigger-cli-version");
85+
return value && value.length <= CLI_VERSION_MAX_LENGTH ? value : undefined;
86+
}
87+
7888
export const loader = createLoaderApiRoute(
7989
{
8090
searchParams: ApiDeploymentListSearchParams,
Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
import { BuildServerMetadata } from "@trigger.dev/core/v3";
2+
3+
/**
4+
* Attribute names for the `deployment.finished` and `deployment.initialized`
5+
* telemetry events (emitted by services/recordDeploymentFinished.server.ts).
6+
* This module is the single owner of these names — external queries,
7+
* dashboards, and monitors reference them, so treat renames as breaking.
8+
*
9+
* Query gotchas: dedup with `arg_max(_time, *) by deployment.id` (job retries
10+
* can double-emit); the span's `_time` is the deployment's createdAt, so a
11+
* TIMED_OUT event lands backdated by up to the full deploy timeout — monitor
12+
* windows must exceed it; phase durations are omitted (not zero) when a
13+
* boundary timestamp is missing, and `total_ms` excludes local-bundle's
14+
* pre-init client work (esbuild + upload) until the CLI reports timings.
15+
*/
16+
export const DeploymentTelemetryAttributes = {
17+
ORG_ID: "$trigger.org.id",
18+
PROJECT_ID: "$trigger.project.id",
19+
// Project external ref ("proj_…")
20+
PROJECT_REF: "$trigger.project.ref",
21+
ENV_ID: "$trigger.env.id",
22+
// PRODUCTION / STAGING / PREVIEW / DEVELOPMENT
23+
ENV_TYPE: "$trigger.env.type",
24+
// Deployment friendly id — the dedup key
25+
DEPLOYMENT_ID: "deployment.id",
26+
VERSION: "deployment.version",
27+
// finished: terminal status; initialized: initial status (PENDING/BUILDING)
28+
STATUS: "deployment.status",
29+
// status === DEPLOYED; CANCELED is excluded from failure rates
30+
SUCCESS: "deployment.success",
31+
// depot / native / native_local_bundle (see deriveBuildPath)
32+
BUILD_PATH: "deployment.build_path",
33+
// V1 / MANAGED (run engine)
34+
WORKER_TYPE: "deployment.worker_type",
35+
RUNTIME: "deployment.runtime",
36+
// Set at indexing; null for pre-index failures
37+
RUNTIME_VERSION: "deployment.runtime_version",
38+
// From x-trigger-cli-version at init; null for pre-column history
39+
CLI_VERSION: "deployment.cli_version",
40+
TRIGGERED_VIA: "deployment.triggered_via",
41+
COMMIT_SHA: "deployment.commit_sha",
42+
// error.* only on FAILED/TIMED_OUT; CANCELED uses canceled_reason
43+
ERROR_NAME: "deployment.error.name",
44+
ERROR_MESSAGE: "deployment.error.message",
45+
CANCELED_REASON: "deployment.canceled_reason",
46+
// createdAt → terminal (also the span's own duration)
47+
DURATION_TOTAL_MS: "deployment.duration.total_ms",
48+
// createdAt → startedAt; ≈0 when created directly in BUILDING (depot)
49+
DURATION_QUEUE_MS: "deployment.duration.queue_ms",
50+
// startedAt → installedAt; build-server paths only (depot never sets it)
51+
DURATION_INSTALL_MS: "deployment.duration.install_ms",
52+
// (installedAt ?? startedAt) → builtAt
53+
DURATION_BUILDING_MS: "deployment.duration.building_ms",
54+
// builtAt → terminal; for depot dominated by the server-side registry push
55+
DURATION_DEPLOYING_MS: "deployment.duration.deploying_ms",
56+
} as const;
57+
58+
export type DeploymentBuildPath = "native_local_bundle" | "native" | "depot";
59+
60+
/**
61+
* Everything that is not a native-build-server deployment falls into the depot
62+
* bucket, including rare `--local-build` deploys (their flag is not persisted).
63+
* `externalBuildData` is NOT a usable depot signal: init writes a placeholder
64+
* for every path.
65+
*/
66+
export function deriveBuildPath(buildServerMetadata: unknown): DeploymentBuildPath {
67+
const metadata = BuildServerMetadata.safeParse(buildServerMetadata);
68+
69+
if (metadata.success && metadata.data.isNativeBuild) {
70+
return metadata.data.fromBundle ? "native_local_bundle" : "native";
71+
}
72+
73+
return "depot";
74+
}
75+
76+
export type DeploymentTimestamps = {
77+
createdAt: Date;
78+
startedAt?: Date | null;
79+
installedAt?: Date | null;
80+
builtAt?: Date | null;
81+
};
82+
83+
export type DeploymentDurations = {
84+
totalMs: number;
85+
queueMs?: number;
86+
installMs?: number;
87+
buildingMs?: number;
88+
deployingMs?: number;
89+
};
90+
91+
/**
92+
* Timestamp chains are path-shaped (e.g. depot never sets installedAt), so
93+
* each phase is derived only when both of its boundary timestamps exist and
94+
* are ordered.
95+
*/
96+
export function deriveDeploymentDurations(
97+
timestamps: DeploymentTimestamps,
98+
terminalAt: Date
99+
): DeploymentDurations {
100+
const { createdAt, startedAt, installedAt, builtAt } = timestamps;
101+
const buildingFrom = installedAt ?? startedAt;
102+
103+
return {
104+
totalMs: Math.max(terminalAt.getTime() - createdAt.getTime(), 0),
105+
queueMs: msBetween(createdAt, startedAt),
106+
installMs: msBetween(startedAt, installedAt),
107+
buildingMs: msBetween(buildingFrom, builtAt),
108+
deployingMs: msBetween(builtAt, terminalAt),
109+
};
110+
}
111+
112+
function msBetween(from?: Date | null, to?: Date | null): number | undefined {
113+
if (!from || !to) return undefined;
114+
const ms = to.getTime() - from.getTime();
115+
return ms >= 0 ? ms : undefined;
116+
}

apps/webapp/app/v3/services/createDeploymentBackgroundWorkerV4.server.ts

Lines changed: 18 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ import {
1818
} from "./createBackgroundWorker.server";
1919
import { findOrCreateBackgroundWorker } from "./createDeploymentBackgroundWorkerV4/findOrCreateBackgroundWorker.server";
2020
import { TimeoutDeploymentService } from "./timeoutDeployment.server";
21-
import { recordDeploymentOutcome } from "./recordDeploymentOutcome.server";
21+
import { recordDeploymentFinished } from "./recordDeploymentFinished.server";
2222
import { env } from "~/env.server";
2323
import { webhookPrisma } from "~/db.server";
2424

@@ -298,6 +298,12 @@ export class CreateDeploymentBackgroundWorkerServiceV4 extends BaseService {
298298
error: Error,
299299
environment: AuthenticatedEnvironment
300300
) {
301+
const failedAt = new Date();
302+
const errorData = {
303+
name: error.name,
304+
message: error.message,
305+
};
306+
301307
// Guarded BUILDING → FAILED transition, symmetric with the BUILDING → DEPLOYING
302308
// transition in `call()`. With idempotent retries, two attempts can run side-by-side;
303309
// without the predicate, one attempt's failure could downgrade the deployment after
@@ -309,11 +315,8 @@ export class CreateDeploymentBackgroundWorkerServiceV4 extends BaseService {
309315
},
310316
data: {
311317
status: "FAILED",
312-
failedAt: new Date(),
313-
errorData: {
314-
name: error.name,
315-
message: error.message,
316-
},
318+
failedAt,
319+
errorData,
317320
buildEnvVars: Prisma.DbNull,
318321
},
319322
});
@@ -332,13 +335,16 @@ export class CreateDeploymentBackgroundWorkerServiceV4 extends BaseService {
332335
// BUILDING → DEPLOYING transition.
333336
await TimeoutDeploymentService.dequeue(deployment.id, this._prisma);
334337

335-
recordDeploymentOutcome({
338+
recordDeploymentFinished({
336339
status: "FAILED",
337-
deploymentFriendlyId: deployment.friendlyId,
338-
organizationId: environment.organizationId,
339-
projectId: environment.projectId,
340-
environmentId: environment.id,
341-
environmentType: environment.type,
340+
deployment: { ...deployment, status: "FAILED", failedAt, errorData },
341+
environment: {
342+
organizationId: environment.organizationId,
343+
projectId: environment.projectId,
344+
projectRef: environment.project.externalRef,
345+
environmentId: environment.id,
346+
environmentType: environment.type,
347+
},
342348
reason: error.message,
343349
});
344350
}

apps/webapp/app/v3/services/deployment.server.ts

Lines changed: 46 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import {
99
type DeploymentEvent,
1010
} from "@trigger.dev/core/v3";
1111
import { TimeoutDeploymentService } from "./timeoutDeployment.server";
12+
import { recordDeploymentFinished } from "./recordDeploymentFinished.server";
1213
import { env } from "~/env.server";
1314
import { createRemoteImageBuild } from "../remoteImageBuilder.server";
1415
import { FINAL_DEPLOYMENT_STATUSES } from "./failDeployment.server";
@@ -195,10 +196,8 @@ export class DeploymentService extends BaseService {
195196
friendlyId: string,
196197
data?: Partial<Pick<WorkerDeployment, "canceledReason">>
197198
) {
198-
const validateDeployment = (
199-
deployment: Pick<WorkerDeployment, "id" | "status" | "shortCode"> & {
200-
environment: { project: { externalRef: string } };
201-
}
199+
const validateDeployment = <T extends Pick<WorkerDeployment, "id" | "status">>(
200+
deployment: T
202201
) => {
203202
if (FINAL_DEPLOYMENT_STATUSES.includes(deployment.status)) {
204203
logger.warn("Attempted cancelling deployment in a final state", {
@@ -210,11 +209,7 @@ export class DeploymentService extends BaseService {
210209
return okAsync(deployment);
211210
};
212211

213-
const cancelDeployment = (
214-
deployment: Pick<WorkerDeployment, "id" | "shortCode"> & {
215-
environment: { project: { externalRef: string } };
216-
}
217-
) =>
212+
const cancelDeployment = <T extends Pick<WorkerDeployment, "id">>(deployment: T) =>
218213
fromPromise(
219214
this._prisma.workerDeployment.updateMany({
220215
where: {
@@ -250,7 +245,25 @@ export class DeploymentService extends BaseService {
250245
return this.getDeployment(authenticatedEnv.id, friendlyId)
251246
.andThen(validateDeployment)
252247
.andThen(cancelDeployment)
253-
.andThen(({ deployment }) =>
248+
.andTee(({ deployment }) =>
249+
recordDeploymentFinished({
250+
status: "CANCELED",
251+
deployment: {
252+
...deployment,
253+
status: "CANCELED",
254+
canceledAt: new Date(),
255+
canceledReason: data?.canceledReason ?? null,
256+
},
257+
environment: {
258+
organizationId: deployment.environment.project.organizationId,
259+
projectId: deployment.environment.project.id,
260+
projectRef: deployment.environment.project.externalRef,
261+
environmentId: deployment.environment.id,
262+
environmentType: deployment.environment.type,
263+
},
264+
})
265+
)
266+
.andTee(({ deployment }) =>
254267
this.appendToEventLog(deployment.environment.project, deployment, [
255268
{
256269
type: "finalized",
@@ -259,14 +272,11 @@ export class DeploymentService extends BaseService {
259272
message: data?.canceledReason ?? undefined,
260273
},
261274
},
262-
])
263-
.orElse((error) => {
264-
logger.error("Failed to append event to deployment event log", { error });
265-
return okAsync(deployment);
266-
})
267-
.map(() => deployment)
275+
]).orTee((error) => {
276+
logger.error("Failed to append event to deployment event log", { error });
277+
})
268278
)
269-
.andThen(deleteTimeout)
279+
.andThen(({ deployment }) => deleteTimeout(deployment))
270280
.map(() => undefined);
271281
}
272282

@@ -484,13 +494,32 @@ export class DeploymentService extends BaseService {
484494
select: {
485495
status: true,
486496
id: true,
497+
friendlyId: true,
498+
version: true,
499+
type: true,
500+
createdAt: true,
501+
startedAt: true,
502+
installedAt: true,
503+
builtAt: true,
504+
deployedAt: true,
505+
failedAt: true,
506+
canceledAt: true,
507+
canceledReason: true,
508+
errorData: true,
509+
runtime: true,
510+
runtimeVersion: true,
511+
cliVersion: true,
512+
triggeredVia: true,
513+
commitSHA: true,
487514
buildServerMetadata: true,
488515
imageReference: true,
489516
shortCode: true,
490517
environment: {
491518
include: {
492519
project: {
493520
select: {
521+
id: true,
522+
organizationId: true,
494523
externalRef: true,
495524
},
496525
},

0 commit comments

Comments
 (0)