Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion CHANGELOG.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@

== 3.1.0 (unreleased)

=== Kotlin 2.3 or later

Bootable's modules are compiled for Kotlin API 2.3, rather than 2.1, so an application needs Kotlin 2.3 or later: Bootable supports the last two Kotlin versions.
The Metro flavors already required it.

=== Service start order: phases and milestones replace priorities

Services no longer order themselves with a global `priority()` number.
Expand Down Expand Up @@ -39,7 +44,8 @@ With Kodein, `bootableModule` uses `StartOrder` and `StartOrder.Tiebreaker` bind

`LifecycleController.priority()` is unchanged.

`LifecycleController.aroundAppStart(serviceContext, startupMark, start)` runs code around the application's start, which starts every service, and `aroundServiceStart(appService, start)` around each service's start, the highest-priority controller's outermost; the defaults run the start alone.
`LifecycleController.aroundAppStart(serviceContext, timeline, start)` runs code around the application's start, which starts every service, and `aroundServiceStart(appService, start)` around each service's start, the highest-priority controller's outermost; the defaults run the start alone.
The `StartupTimeline` marks when the boot began and its stages before the services start: with `boot`, initializing the JVM classes for logging and configuration (setting up logging and running the `boot` lambda), planning the start, and initializing.

=== Telemetry with OpenTelemetry

Expand All @@ -57,6 +63,9 @@ Its exemplars carry only the trace and span ids, within OpenMetrics's 128-charac
* `boot-db-pool-hikari-otel`: records HikariCP's pool metrics through the application's OpenTelemetry, as a `HikariConfigCustomizer` the Kodein and Metro flavors add to the pool's customizers.
* `boot-telemetry-otel` adds `ServiceStartTracing`, a lifecycle controller that traces the application's start as one trace: a root span, from when `boot` was called until the application is running, with a span for each service's start, and what it does as it starts, such as a migration's queries, within that.
Kodein's `serviceStartTracingModule` and Metro's `ServiceStartTracingBindings`, named in the graph, add it.
* `ServiceStartTracing` records each stage of the `StartupTimeline` as a span under the root, and `OpenTelemetryService`'s sampler always samples the start's trace, whatever `sampleRatio` is.
* `withRootSpan(name, sample = Sample.Always)` always samples the trace it starts too: a root span with the `ALWAYS_SAMPLE` attribute, `bootable.sample.always`, is always sampled.
* `OpenTelemetryService` and `ServiceStartTracing` take a `kotlin.time.Clock`, `Clock.System` by default, which the SDK and the root span take their time from; with Kodein and Metro, a `Clock` binding, if there is one.
* `boot-db-jdbc-otel`: traces each JDBC query as a client span, under the span current where it runs, such as a request's, as a `DataSourceWrapper` the Kodein and Metro flavors add to `boot-db-jdbc`'s wrappers.
`jdbcTracing`'s `configure` lambda, Kodein's `jdbcTracingModule { ... }`, or a Metro `JdbcTelemetryConfigurer` binding changes the instrumentation's defaults.
`JdbcDataSourceService` takes the `DataSourceWrapper`, as its third argument; Kodein's `jdbcDataSourceModule` and Metro's `JdbcDataSourceBindings`, now an interface, collect a set of them.
Expand Down
5 changes: 5 additions & 0 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -171,11 +171,16 @@ Ship them with the platform's log collection, such as Cloud Logging's, or an Ope
`Tracer.withSpan(name) { span -> … }` runs a suspending block in a child span of the current one, which stays current across dispatchers, and records an exception as the span's error.
A span from `withSpan` is expected to have a parent: without one, it starts a trace of its own and logs a warning, once for each span name, since its caller's trace context usually didn't reach it, such as across a new `CoroutineScope` or a Java callback.
To start a trace on purpose, such as for a scheduled job or a consumed message, use `Tracer.withRootSpan(name) { span -> … }`, which can also link the trace to the operation that kicked it off.
`withRootSpan(name, sample = Sample.Always)` samples that trace whatever `sampleRatio` is, for a trace too rare or valuable to lose, such as a nightly job's; the application's start is always sampled the same way.
`ServiceStartTracing(openTelemetry)` is a lifecycle controller that traces the application's start as one trace, so its total time, and each service's share of it, can be seen: a root span, `start application`, with a child span for each service's start, such as `start FlywayMigrationService`, within which is what the service does as it starts, such as its queries.
The root span begins when `boot` was called, so it covers building the application, such as decoding its config, as well as starting its services, and ends when the application is running; it doesn't cover the JVM's own startup, before `main`.
The stages before the first service starts are spans under it too: `initialize JVM classes for logging and configuration`, `plan start`, and `initialize`.
The first covers setting up logging and the `boot` lambda, including the config and the DI graph: on a cold JVM, most of its time is loading and initializing the classes of the libraries these use, which a JVM AOT cache can cut sharply.
The start's trace is always sampled, whatever `sampleRatio` is, so a slow or failed start is never missing from the traces.
A start that fails records the exception, and an error status, on the service's span and on the root span.
Add it to the lifecycle controllers: `defaultLifecycleControllers() + ServiceStartTracing(openTelemetry)` with vanilla wiring, `serviceStartTracingModule` with Kodein, and `ServiceStartTracingBindings` in the graph's `bindingContainers` with Metro.
The span is current on the thread that calls `start`: a `start` that runs coroutines carries it into them with `runBlocking(Context.current().asContextElement()) { … }`.
The SDK times its spans and measurements, and `ServiceStartTracing` its root span, with a `kotlin.time.Clock`: `Clock.System` unless one is given to `OpenTelemetryService` and `ServiceStartTracing`, or bound, with Kodein or Metro, such as a fixed clock in a test.
For SDK settings the config doesn't cover, such as a span processor, a metric view, or another sampler, add an `OpenTelemetryCustomizer` to its set binding (Kodein `inBindSet`, Metro `@IntoSet`); this is also how the modules below plug in.
`excludeUrlPathsFromTraces(paths)` is one, which keeps requests such as health checks out of the traces.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,19 @@ import org.kodein.di.bindSet
import org.kodein.di.bindSingleton
import org.kodein.di.inBindSet
import org.kodein.di.instance
import org.kodein.di.instanceOrNull
import org.kodein.di.singleton
import kotlin.time.Clock

/**
* A Kodein module that binds the application's [OpenTelemetry] and registers its [OpenTelemetryService] into the
* app-service set, so the last telemetry is exported, and the SDK shut down, when the application stops. Its
* [TelemetryConfig] is decoded by [telemetryConfig], from the `telemetry.sdk` config section. Requires a
* `ConfigBinder` binding (e.g. from boot-config-hoplite's `hopliteConfigModule`).
*
* A `kotlin.time.Clock` binding, if there is one, is the SDK's clock; otherwise it is `Clock.System`. Bind one, e.g. a
* fixed clock in a test, to set the time of the spans and measurements.
*
* It opens two sets, so import it before the modules that add to them:
* - `Set<OpenTelemetryCustomizer>`: how other modules, such as `prometheusTelemetryModule` from
* `boot-telemetry-otel-prometheus`, plug into the SDK. Add your own for what [TelemetryConfig] doesn't cover:
Expand All @@ -27,7 +32,9 @@ val openTelemetryModule by DI.Module {
bindSet<OpenTelemetryCustomizer>()
bindSet<MetricsSnapshotProvider>()
bindSingleton { telemetryConfig(instance()) }
bindSingleton { OpenTelemetryService(instance(), instance<Set<OpenTelemetryCustomizer>>()) }
bindSingleton {
OpenTelemetryService(instance(), instance<Set<OpenTelemetryCustomizer>>(), instanceOrNull<Clock>() ?: Clock.System)
}
bindSingleton<OpenTelemetry> { instance<OpenTelemetryService>().openTelemetry }
inBindSet<AppService> { add { singleton { instance<OpenTelemetryService>() } } }
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,16 @@ import com.github.rocketraman.bootable.boot.LifecycleController
import org.kodein.di.DI
import org.kodein.di.inBindSet
import org.kodein.di.instance
import org.kodein.di.instanceOrNull
import org.kodein.di.singleton
import kotlin.time.Clock

/**
* A Kodein module that traces the application's start as one trace, with a span for each service's start, by adding
* [ServiceStartTracing] to the [LifecycleController] set of `bootableModule` (from boot-kodein). Import it alongside
* [openTelemetryModule], which provides the `OpenTelemetry`.
* [openTelemetryModule], which provides the `OpenTelemetry`. Like that module, it uses the `kotlin.time.Clock` binding,
* if there is one, or else `Clock.System`.
*/
val serviceStartTracingModule by DI.Module {
inBindSet<LifecycleController> { add { singleton { ServiceStartTracing(instance()) } } }
inBindSet<LifecycleController> { add { singleton { ServiceStartTracing(instance(), instanceOrNull<Clock>() ?: Clock.System) } } }
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,22 +8,28 @@ import io.opentelemetry.api.OpenTelemetry
import io.opentelemetry.api.baggage.propagation.W3CBaggagePropagator
import io.opentelemetry.api.common.AttributeKey
import io.opentelemetry.api.common.Attributes
import io.opentelemetry.api.trace.SpanKind
import io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator
import io.opentelemetry.context.Context
import io.opentelemetry.context.propagation.ContextPropagators
import io.opentelemetry.context.propagation.TextMapPropagator
import io.opentelemetry.exporter.otlp.http.metrics.OtlpHttpMetricExporter
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter
import io.opentelemetry.instrumentation.runtimetelemetry.RuntimeTelemetry
import io.opentelemetry.sdk.OpenTelemetrySdk
import io.opentelemetry.sdk.common.Clock as OtelClock
import io.opentelemetry.sdk.metrics.SdkMeterProvider
import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader
import io.opentelemetry.sdk.resources.Resource
import io.opentelemetry.sdk.trace.SdkTracerProvider
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor
import io.opentelemetry.sdk.trace.data.LinkData
import io.opentelemetry.sdk.trace.samplers.Sampler
import io.opentelemetry.sdk.trace.samplers.SamplingResult
import org.apache.logging.log4j.kotlin.logger
import java.util.UUID
import java.util.concurrent.TimeUnit
import kotlin.time.Clock
import kotlin.time.toJavaDuration

/**
Expand All @@ -37,12 +43,22 @@ import kotlin.time.toJavaDuration
* When the OpenTelemetry Java agent is attached, it has already built and registered the process's SDK, and owns its
* lifecycle; [openTelemetry] is then the agent's, and [config] and the [customizers] are ignored.
*
* Traces are sampled at [TracesConfig.sampleRatio], or as the customizers' sampler decides, except a trace whose root
* span carries [ALWAYS_SAMPLE], which is always sampled: the application's start, as [ServiceStartTracing] traces it,
* which happens once per process, and a [withRootSpan] with [Sample.Always].
*
* The SDK takes the time of its spans and measurements from [clock], so a test can fix it, and [ServiceStartTracing],
* given the same clock, dates its spans consistently with them. With [Clock.System], the default, the SDK keeps its
* own clock, which measures durations with the JVM's monotonic `System.nanoTime`. Any other clock is the SDK's
* source of both the time and durations.
*
* The service is in the [DefaultPhase.Telemetry] phase, so every other service, the database pool included, starts
* after the SDK and shuts down before it, and its last spans and measurements are in the final export.
*/
class OpenTelemetryService(
private val config: TelemetryConfig = TelemetryConfig(),
customizers: Collection<OpenTelemetryCustomizer> = emptyList(),
clock: Clock = Clock.System,
) : AppService {

// the agent has already built and registered the process's SDK
Expand All @@ -62,7 +78,7 @@ class OpenTelemetryService(

private val sdk: OpenTelemetrySdk? =
if (agentAttached) null
else buildSdk(config, customizers, resource).also { if (config.registerGlobal) registerGlobal(it) }
else buildSdk(config, customizers, resource, clock).also { if (config.registerGlobal) registerGlobal(it) }

/** The application's OpenTelemetry: get tracers and meters from it, and hand it to instrumentation. */
val openTelemetry: OpenTelemetry = sdk ?: GlobalOpenTelemetry.get()
Expand Down Expand Up @@ -128,7 +144,10 @@ private fun buildSdk(
config: TelemetryConfig,
customizers: Collection<OpenTelemetryCustomizer>,
resource: Resource,
clock: Clock,
): OpenTelemetrySdk {
// the SDK's own clock, unless another was given
val sdkClock = if (clock === Clock.System) null else SdkClock(clock)
val endpoint = config.otlp.endpoint?.trimEnd('/')
val headers = config.otlp.headers
val builder = OpenTelemetrySdk.builder().setPropagators(
Expand All @@ -138,10 +157,13 @@ private fun buildSdk(
)

if (config.traces.enabled) {
val sampler = customizers.fold(Sampler.parentBased(Sampler.traceIdRatioBased(config.traces.sampleRatio))) { sampler, customizer ->
customizer.customizeSampler(sampler)
}
val sampler = SampleMarked(
customizers.fold(Sampler.parentBased(Sampler.traceIdRatioBased(config.traces.sampleRatio))) { sampler, customizer ->
customizer.customizeSampler(sampler)
}
)
val tracerProvider = SdkTracerProvider.builder().setResource(resource).setSampler(sampler)
sdkClock?.let { tracerProvider.setClock(it) }
if (endpoint != null && config.traces.otlp) {
val url = "$endpoint/v1/traces"
val exporter = OtlpHttpSpanExporter.builder().setEndpoint(url).apply {
Expand All @@ -162,6 +184,7 @@ private fun buildSdk(

if (config.metrics.enabled) {
val meterProvider = SdkMeterProvider.builder().setResource(resource)
sdkClock?.let { meterProvider.setClock(it) }
if (endpoint != null && config.metrics.otlp) {
val url = "$endpoint/v1/metrics"
val exporter = OtlpHttpMetricExporter.builder().setEndpoint(url).apply {
Expand Down Expand Up @@ -191,3 +214,31 @@ private fun exportHeaders(
if (suppliers.isEmpty()) return null
return { suppliers.fold(emptyMap()) { headers, supplier -> headers + supplier() } }
}

/** [clock] as the SDK's clock, for both the time and durations, in nanoseconds. */
private class SdkClock(private val clock: Clock) : OtelClock {
override fun now(): Long = clock.now().let { it.epochSeconds * NANOS_PER_SECOND + it.nanosecondsOfSecond }

override fun nanoTime(): Long = now()
}

private const val NANOS_PER_SECOND = 1_000_000_000L

/**
* Samples a span marked [ALWAYS_SAMPLE], such as the application's start, and leaves every other span to [delegate].
* The spans within its trace follow it if [delegate] samples by the parent, as the default sampler does.
*/
private class SampleMarked(private val delegate: Sampler) : Sampler {
override fun shouldSample(
parentContext: Context,
traceId: String,
name: String,
spanKind: SpanKind,
attributes: Attributes,
parentLinks: List<LinkData>,
): SamplingResult =
if (attributes.get(ALWAYS_SAMPLE) == true) SamplingResult.recordAndSample()
else delegate.shouldSample(parentContext, traceId, name, spanKind, attributes, parentLinks)

override fun getDescription(): String = "SampleMarked{${delegate.description}}"
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,17 @@ package com.github.rocketraman.bootable.telemetry.otel
import com.github.rocketraman.bootable.boot.AppService
import com.github.rocketraman.bootable.boot.LifecycleController
import com.github.rocketraman.bootable.boot.ServiceContext
import com.github.rocketraman.bootable.boot.StartupTimeline
import io.opentelemetry.api.OpenTelemetry
import io.opentelemetry.api.trace.Span
import io.opentelemetry.api.trace.SpanBuilder
import io.opentelemetry.api.trace.StatusCode
import io.opentelemetry.api.trace.Tracer
import java.time.Instant
import kotlin.time.TimeMark
import kotlin.time.toJavaDuration
import kotlin.time.Clock
import kotlin.time.ComparableTimeMark
import kotlin.time.TimeSource
import kotlin.time.toJavaInstant

/**
* A [LifecycleController] that traces the application's start as one trace: a root span, `start application`, with a
Expand All @@ -19,11 +22,23 @@ import kotlin.time.toJavaDuration
* `boot-db-jdbc-otel`. A start that throws has the exception recorded on its span, with an error status, and so does
* the root span, when that stops the application's start.
*
* The root span has a child span for each stage of the boot before the services start, from the [StartupTimeline]:
* `initialize JVM classes for logging and configuration`, `plan start`, and `initialize`, so the time before the first
* service starts is accounted for too. The SDK doesn't exist yet in those stages, so their spans are recorded
* afterwards, with the times the timeline marked.
*
* [OpenTelemetryService]'s sampler always samples the start's trace, whatever the configured ratio, so a start is
* never missing from the traces: the root span carries [ALWAYS_SAMPLE], which the sampler looks for.
*
* The root span begins when the application began booting: with Bootable's `boot`, before logging is initialized and
* the application built, so it covers building the application, such as decoding its config and creating its
* services, as well as starting them, the telemetry phase's included. It ends once every service has started, when
* the application is running. It doesn't cover the JVM's own startup, before `main`.
*
* The root span's start time is [clock]'s time, less how long the application has been booting, as measured by the
* JVM's monotonic clock. Give it the clock [OpenTelemetryService] has, so its time is consistent with the SDK's time
* for the other spans; both default to [Clock.System].
*
* Add it to the application's lifecycle controllers:
*
* ```
Expand All @@ -44,12 +59,26 @@ import kotlin.time.toJavaDuration
* A service started later, outside the application's start, such as by a controller that starts services on command,
* gets a span of its own, a child of whatever span is current then.
*/
class ServiceStartTracing(openTelemetry: OpenTelemetry) : LifecycleController {
class ServiceStartTracing(
openTelemetry: OpenTelemetry,
private val clock: Clock = Clock.System,
) : LifecycleController {
private val tracer: Tracer = openTelemetry.getTracer(INSTRUMENTATION_SCOPE)

override fun aroundAppStart(serviceContext: ServiceContext, startupMark: TimeMark, start: () -> Unit) {
val startedAt = Instant.now().minus(startupMark.elapsedNow().toJavaDuration())
tracer.spanBuilder(APP_START_SPAN).setNoParent().setStartTimestamp(startedAt).runInSpan(start)
override fun aroundAppStart(serviceContext: ServiceContext, timeline: StartupTimeline, start: () -> Unit) {
// the wall-clock time of each mark, relative to one reading of the clock, so the spans are consistent
val reference = TimeSource.Monotonic.markNow()
val now = clock.now()
fun at(mark: ComparableTimeMark): Instant = (now - (reference - mark)).toJavaInstant()

tracer.spanBuilder(APP_START_SPAN).setNoParent().setAttribute(ALWAYS_SAMPLE, true)
.setStartTimestamp(at(timeline.start))
.runInSpan {
timeline.stages.forEach { stage ->
tracer.spanBuilder(stage.name).setStartTimestamp(at(stage.start)).startSpan().end(at(stage.end))
}
start()
}
}

override fun aroundServiceStart(appService: AppService, start: () -> Unit) {
Expand Down
Loading
Loading