Skip to content

Document Cloud Run OTel and Worker identity setup #5292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
lennessyy wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Document Cloud Run OTel and Worker identity setup #5292

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 101 additions & 4 deletions docs/develop/dotnet/workers/serverless-workers/cloud-run.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import { ReleaseNoteHeader } from '@site/src/components';
On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker.
Register Workflows and Activities the same way you would with any other .NET Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific package.
A Cloud Run Worker needs no Cloud Run-specific runtime or handler.
The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run).
Expand Down Expand Up @@ -135,7 +135,104 @@ public static string Process(IReadOnlyList<string> items)

For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle).

## Add observability {/* #add-observability */}
## Configure OpenTelemetry {/* #opentelemetry */}

Configure the .NET OpenTelemetry exporter and the Temporal Runtime to send telemetry by OTLP to `localhost:4317`. The following code sample configures the Worker only. You must separately run an OTLP-compatible receiver at that address. In a Cloud Run Worker Pool, run that receiver as a sidecar.

<!--SNIPSTART dotnet-cloud-run-otel-->
[src/OpenTelemetry/CoreSdkForwarding/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/OpenTelemetry/CoreSdkForwarding/Program.cs)
```cs
var resourceBuilder = ResourceBuilder.
CreateDefault().
AddService("TemporalioSamples.OpenTelemetry", serviceInstanceId: instanceId);

using var tracerProvider = Sdk.
CreateTracerProviderBuilder().
SetResourceBuilder(resourceBuilder).
AddSource(TracingInterceptor.ClientSource.Name, TracingInterceptor.WorkflowsSource.Name, TracingInterceptor.ActivitiesSource.Name).
AddOtlpExporter().
Build();

// Shared by the client and by Core SDK log forwarding below. The OpenTelemetry provider exports
// logs to the dashboard alongside the traces and metrics.
using var loggerFactory = LoggerFactory.Create(builder =>
builder.
AddSimpleConsole(options => options.TimestampFormat = "[HH:mm:ss] ").
AddOpenTelemetry(options =>
{
options.SetResourceBuilder(resourceBuilder);
options.IncludeFormattedMessage = true;
options.IncludeScopes = true;
options.AddOtlpExporter();
}).
SetMinimumLevel(LogLevel.Information));

// Create a client to localhost on default namespace
var connectOptions = ClientEnvConfig.LoadClientConnectOptions();
connectOptions.TargetHost ??= "localhost:7233";
connectOptions.LoggerFactory = loggerFactory;
connectOptions.Interceptors = new[] { new TracingInterceptor() };
connectOptions.Runtime = new TemporalRuntime(new TemporalRuntimeOptions()
{
Telemetry = new TelemetryOptions()
{
Metrics = new MetricsOptions()
{
OpenTelemetry = new OpenTelemetryOptions()
{
Url = new Uri("http://localhost:4317"),
},
},
Logging = new LoggingOptions()
{
// Core SDK logs default to WARN; lowered here so there is more to see.
Filter = new TelemetryFilterOptions(core: TelemetryFilterOptions.Level.Info),

// The Core SDK writes its logs to the console itself unless Forwarding is set, in
// which case they go to this ILogger instead.
Forwarding = new LogForwardingOptions(loggerFactory.CreateLogger("Temporalio.Core")),
},
},
});
var client = await TemporalClient.ConnectAsync(connectOptions);
```
<!--SNIPEND-->

The [OpenTelemetry sample](https://github.com/temporalio/samples-dotnet/tree/main/src/OpenTelemetry) shows the tracing, metrics, and log-export configuration. Its Docker Compose file runs the .NET Aspire Dashboard locally and exposes its OTLP endpoint on port `4317`; it does not define a Cloud Run sidecar. Configure your Cloud Run sidecar to export the received telemetry to your backend.

## Set a Worker identity {/* #worker-identity */}

Use `WorkerIdPlugin` to identify each Worker instance as `<instance-id>@<revision>`. The plugin reads Cloud Run environment variables and instance metadata when the Client connects, then Workers created from that Client inherit the identity.

<!--SNIPSTART dotnet-cloud-run-worker-id-->
[src/Gcp/CloudRun/WorkerId/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/Gcp/CloudRun/WorkerId/Program.cs)
```cs
var address = GetEnvironmentVariable("TEMPORAL_ADDRESS") ?? "localhost:7233";
var temporalNamespace = GetEnvironmentVariable("TEMPORAL_NAMESPACE") ?? "default";
var taskQueue = GetEnvironmentVariable("TEMPORAL_TASK_QUEUE") ?? "cloud-run-worker-sample";

using var loggerFactory = LoggerFactory.Create(builder => builder.
AddSimpleConsole(options => options.TimestampFormat = "[HH:mm:ss] ").
SetMinimumLevel(LogLevel.Information));
var logger = loggerFactory.CreateLogger("CloudRunWorkerId");

// Register the Cloud Run plugin once on the client. At connect time it reads the Cloud Run instance
// id from the metadata server, and the worker pool / service name and revision from the environment,
// then sets the client Identity to the worker identity "{instanceId}@{revision}" (unless one was
// already configured). Every worker created from this client inherits that identity. The plugin only
// sets the worker identity; it does not configure anything else.
//
// NOTE: this requires the process to be running on a Cloud Run worker pool or service. Running it
// elsewhere throws at connect time because the metadata server is unreachable.
var clientOptions = new TemporalClientConnectOptions(address)
{
Namespace = temporalNamespace,
LoggerFactory = loggerFactory,
Plugins = new[] { new WorkerIdPlugin() },
};

var client = await TemporalClient.ConnectAsync(clientOptions);
```
<!--SNIPEND-->

A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else.
For how to configure metrics export and OpenTelemetry tracing interceptors, see [Observability - .NET SDK](/develop/dotnet/platform/observability) and the [SDK metrics reference](/references/sdk-metrics).
The plugin requires the Cloud Run metadata server, so it fails when the Worker runs outside Cloud Run. For a complete example, see the [.NET Cloud Run Worker Id sample](https://github.com/temporalio/samples-dotnet/pull/219).
123 changes: 119 additions & 4 deletions docs/develop/go/workers/serverless-workers/cloud-run.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import { ReleaseNoteHeader } from '@site/src/components';
On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker.
Register Workflows and Activities the same way you would with any other Go Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific package.
A Cloud Run Worker needs no Cloud Run-specific runtime or handler.
The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run).
Expand Down Expand Up @@ -112,7 +112,122 @@ func MyActivity(ctx context.Context, input MyInput) (string, error) {

For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle).

## Add observability {/* #add-observability */}
## Configure OpenTelemetry {/* #opentelemetry */}

A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else.
For how to configure metrics export and OpenTelemetry tracing interceptors, see [Observability - Go SDK](/develop/go/platform/observability) and the [SDK metrics reference](/references/sdk-metrics).
Run an OpenTelemetry Collector as a sidecar in the Worker Pool. The Cloud Run OpenTelemetry plugin exports metrics and traces by OTLP gRPC to the Collector at `localhost:4317`. By default, it derives the service name from `OTEL_SERVICE_NAME`, `CLOUD_RUN_WORKER_POOL`, or `K_SERVICE`.

Create the plugin before connecting, then add it to the Client options. Client plugins that implement `worker.Plugin` also apply to Workers created from that Client:

<!--SNIPSTART go-cloud-run-worker {"selectedLines": ["27-44"]}-->
[gcp/cloudrun/otel/worker/main.go](https://github.com/temporalio/samples-go/blob/gcp-cloud-run-otel/gcp/cloudrun/otel/worker/main.go)
```go
// ...
otelPlugin, err := otel.NewPlugin(ctx, otel.PluginOptions{})
if err != nil {
log.Fatalln("Unable to create OpenTelemetry plugin", err)
}

// Load the Temporal connection from the environment (see temporal.toml or the
// TEMPORAL_* environment variables) and install the plugin. Client plugins
// that also implement worker.Plugin are applied to workers automatically.
clientOptions, err := envconfig.LoadDefaultClientOptions()
if err != nil {
log.Fatalln("Unable to load Temporal client options", err)
}
clientOptions.Plugins = append(clientOptions.Plugins, otelPlugin)

c, err := client.Dial(clientOptions)
if err != nil {
log.Fatalln("Unable to create Temporal client", err)
}
```
<!--SNIPEND-->

Configure the Collector sidecar to receive OTLP gRPC on `localhost:4317`, export traces to Google Cloud, and export metrics to Google Managed Service for Prometheus:

<!--SNIPSTART go-cloud-run-worker-otel-collector-config-->
[gcp/cloudrun/otel/otel-collector-config.yaml](https://github.com/temporalio/samples-go/blob/main/gcp/cloudrun/otel/otel-collector-config.yaml)
```yaml
# Google-Built OpenTelemetry Collector configuration for a Cloud Run worker pool
# sidecar. The Temporal worker exports OTLP gRPC to localhost:4317; this collector
# adds GCP resource attributes and exports to Google Cloud.
#
# Metrics use googlemanagedprometheus with NO batch processor: batching can merge
# periodic and forced-shutdown snapshots of the same cumulative series into a
# single Google Monitoring write, which Managed Service for Prometheus rejects as
# duplicate data. Traces may be batched independently.
receivers:
otlp:
protocols:
grpc:
endpoint: localhost:4317

processors:
# Guard the sidecar against unbounded memory growth.
memory_limiter:
check_interval: 1s
limit_percentage: 65
spike_limit_percentage: 20
# Detect Google Cloud resource attributes (project, region, revision, ...).
resourcedetection:
detectors: [gcp]
timeout: 10s
# Batch is used ONLY for traces.
batch:
send_batch_size: 200
timeout: 5s

exporters:
debug:
googlemanagedprometheus:
googlecloud:

extensions:
# Health check used as the container startup probe. Bind on all interfaces
# so the Cloud Run startup probe can reach it.
health_check:
endpoint: 0.0.0.0:13133

service:
extensions: [health_check]
pipelines:
# No batch processor in the metrics pipeline.
metrics:
receivers: [otlp]
processors: [memory_limiter, resourcedetection]
exporters: [googlemanagedprometheus, debug]
traces:
receivers: [otlp]
processors: [memory_limiter, resourcedetection, batch]
exporters: [googlecloud, debug]
telemetry:
logs:
level: info
```
<!--SNIPEND-->

On shutdown, stop the Worker and call `otelPlugin.Shutdown` with a deadline shorter than Cloud Run's termination window so telemetry can flush. For a Collector configuration that exports traces to Google Cloud and metrics to Google Managed Service for Prometheus, see the [Go Cloud Run OpenTelemetry sample](https://github.com/temporalio/samples-go/pull/528).

## Set a Worker identity {/* #worker-identity */}

Use the Cloud Run Worker Id plugin to identify each Worker instance as `<instance-id>@<revision>`. The plugin reads Cloud Run environment variables and instance metadata once when the Client connects, then Workers created from that Client inherit the identity.

<!--SNIPSTART go-cloud-run-worker-id {"selectedLines": ["30-40"]}-->
[gcp/cloudrun/workerid/worker/main.go](https://github.com/temporalio/samples-go/blob/main/gcp/cloudrun/workerid/worker/main.go)
```go
// ...
plugin := workerid.NewPlugin(workerid.PluginOptions{})
clientOptions := client.Options{
HostPort: getenv("TEMPORAL_ADDRESS", client.DefaultHostPort),
Namespace: getenv("TEMPORAL_NAMESPACE", client.DefaultNamespace),
Plugins: []client.Plugin{plugin},
}

c, err := client.Dial(clientOptions)
if err != nil {
log.Fatalf("Unable to create Temporal client (is this running on a Cloud Run worker pool or service?): %v", err)
}
```
<!--SNIPEND-->

The plugin requires the Cloud Run metadata server, so it fails when the Worker runs outside Cloud Run. For a complete example, see the [Go Cloud Run Worker Id sample](https://github.com/temporalio/samples-go/pull/531).
Loading