Feature Description
Propagate the request correlation id to the downstream sinks, so consumers of the fan-out targets can join their processing logs to the EventGate request that produced the message:
- Kafka — add the id as a record header (e.g.
correlation_id) in WriterKafka.write(). Record headers keep the payload untouched, and consumers can read the header without schema changes.
- EventBridge — include the id in the
put_events entry, either as a top-level field of Detail or via TraceHeader, in WriterEventBridge.write().
- Postgres — optional; would require a schema change (extra column), so likely out of scope for the first iteration.
Problem / Opportunity
Since #193 / PR #204, every EventGate request resolves a correlation id (X-Correlation-ID → X-Request-ID → API Gateway request id), logs it on every line and returns it in the X-Correlation-ID response header. The id currently stops at EventGate: once a message is fanned out, downstream consumers have no way to correlate their processing with the originating request.
Beneficiaries: teams consuming the Kafka topics and EventBridge events, and anyone debugging a cross-system flow end-to-end — one id would then trace a message from the caller, through EventGate's logs, into every sink's consumer logs.
Acceptance Criteria
- A message published to Kafka carries the request's correlation id as a record header; the message payload (value) is byte-for-byte unchanged.
- An event published to EventBridge carries the request's correlation id in its entry metadata (
TraceHeader or a documented Detail field); the schema-validated payload is unchanged.
- When no correlation id is resolved (empty id), no header/field is added and writes behave exactly as today.
- Existing topic schema validation and all existing consumers are unaffected — propagation uses transport metadata only, never the message body.
- Unit tests cover both writers with and without a bound correlation id; integration tests prove the header/field arrives in the sink.
- README "Logging & Correlation" documents the downstream propagation contract.
Proposed Solution
The writers do not currently receive the correlation id; it is bound in src/utils/observability.py. Two options:
- a small accessor (e.g.
current_correlation_id()) in src/utils/observability.py, read by the writers, or
- an explicit parameter threaded through
HandlerTopic._write_to_all() into Writer.write().
The explicit parameter is more testable and keeps writers free of hidden state; the accessor avoids touching the Writer interface. Alternative considered and rejected: putting the id into the message body — breaks topic schema validation and changes consumer contracts.
The id is already validated against ^[A-Za-z0-9._:-]{1,128}$ before use, so it is safe to forward verbatim.
Dependencies / Related
Additional Context
Follow-up proposed during the second-opinion review of PR #204 (item P7). Postgres propagation can be a separate issue if a schema change is ever justified.
Feature Description
Propagate the request correlation id to the downstream sinks, so consumers of the fan-out targets can join their processing logs to the EventGate request that produced the message:
correlation_id) inWriterKafka.write(). Record headers keep the payload untouched, and consumers can read the header without schema changes.put_eventsentry, either as a top-level field ofDetailor viaTraceHeader, inWriterEventBridge.write().Problem / Opportunity
Since #193 / PR #204, every EventGate request resolves a correlation id (
X-Correlation-ID→X-Request-ID→ API Gateway request id), logs it on every line and returns it in theX-Correlation-IDresponse header. The id currently stops at EventGate: once a message is fanned out, downstream consumers have no way to correlate their processing with the originating request.Beneficiaries: teams consuming the Kafka topics and EventBridge events, and anyone debugging a cross-system flow end-to-end — one id would then trace a message from the caller, through EventGate's logs, into every sink's consumer logs.
Acceptance Criteria
TraceHeaderor a documentedDetailfield); the schema-validated payload is unchanged.Proposed Solution
The writers do not currently receive the correlation id; it is bound in
src/utils/observability.py. Two options:current_correlation_id()) insrc/utils/observability.py, read by the writers, orHandlerTopic._write_to_all()intoWriter.write().The explicit parameter is more testable and keeps writers free of hidden state; the accessor avoids touching the
Writerinterface. Alternative considered and rejected: putting the id into the message body — breaks topic schema validation and changes consumer contracts.The id is already validated against
^[A-Za-z0-9._:-]{1,128}$before use, so it is safe to forward verbatim.Dependencies / Related
adr/001-observability/001-observability.md)Additional Context
Follow-up proposed during the second-opinion review of PR #204 (item P7). Postgres propagation can be a separate issue if a schema change is ever justified.