Kafka FUP Final

[CagriYonca]

Fixed trace correlation issues, aligned with other tracers through the tracer-test-suite.

[CagriYonca]

A couple of words behind the need for the changes.

Let's say we have two distributed environments: env A as a producer and env B as a consumer, and say we have a flask endpoint named send_message(), which receives the call and sends a message to a kafka topic, after sending the message we also make an http call to let's say ibm-producer-api. the consumer checks the topic every 10 seconds, reads the message if any, then makes another http call to ibm-consumer-api.

So the pseudo code is here, trace IDs are given as examples:

env A - producer

• send_message() endpoint receives a call, WSGI Span is created, trace ID: 100
  • message is sent to kafka topic, kafka-produce span is created, trace ID: 100
  • http call made to ibm.com, urllib3 span is created, trace ID: 100

env B - consumer

• consumer checks the topic every 10 seconds for any new messages, if any:
  • message is consumed from the kafka topic, kafka-consume span is created, trace ID: 100
  • since there is no active span after the message has been consumed, kafka-consume span is ended, and since the span context was not saved, we lose the trace ID, so the new urllib3 span will have the trace ID: 101

What we want is whether there is a parent span or not, to keep the incoming trace ID alive.

So after this PR, what will happen in the example above is as follows:

env B - consumer

• consumer checks the topic every 10 seconds for any new messages, if any:
  • message is consumed from the kafka topic, kafka-consume span is created, trace ID: 100
    • kafka-consume span is still alive until we call kafka_consumer.close() or manually close the span using close_consumer_span(). now since the span context information is still there, urllib3 span will also have the trace ID: 100

[pvital]

It looks good to me.