Skip to content

Rework kafka-avro example: drop Confluent Schema Registry, fix OOM - #200

Open
Croway wants to merge 1 commit into
apache:mainfrom
Croway:fix-kafka-avro
Open

Rework kafka-avro example: drop Confluent Schema Registry, fix OOM#200
Croway wants to merge 1 commit into
apache:mainfrom
Croway:fix-kafka-avro

Conversation

@Croway

@Croway Croway commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

The kafka-avro example was broken in several ways:

  • CachedSchemaRegistryClient(schemas, Integer.MAX_VALUE) in both the custom serializer and deserializer caused an OOM crash on startup, regardless of whether a schema registry was reachable.
  • Real Avro object creation was commented out, so the example actually sent a plain String, not Avro.
  • The avro-maven-plugin codegen was commented out in the pom.
  • The Confluent schema registry URL was hardcoded.
  • The README documented two contradictory setup paths (manual Confluent vs. camel infra run kafka) - neither worked alone, since infra kafka has no schema registry.

This rewrites the example to use Camel's native AvroDataFormat instead of Confluent's schema-registry-backed serializers. This needs only a plain Kafka broker (no schema registry at all), which removes the OOM-causing code outright and lets the README document a single, working setup path via camel infra run kafka.

Also fixes two latent bugs found while verifying end-to-end:

  • KafkaConstants.KAFKA_RECORD_META needed recordMetadata=true on the producer endpoint to actually populate (was silently null), so the previously-unwired KafkaAvroProcessor is now wired into the producer route.
  • consumer.group was defined in application.properties but never referenced in the consumer endpoint URI.

Test plan

  • mvn clean package (checkstyle/RAT included) passes
  • Started a broker with camel infra run kafka, ran mvn spring-boot:run, and confirmed the producer builds and sends real Employee Avro records, the consumer unmarshals and logs them back, and record metadata logging works with no errors/OOM.

The example was broken: CachedSchemaRegistryClient(schemas,
Integer.MAX_VALUE) in both serializer/deserializer caused an OOM
crash on startup regardless of schema registry reachability, real
Avro object creation was commented out (it sent a plain String, not
Avro), the avro-maven-plugin codegen was commented out in the pom,
the schema registry URL was hardcoded, and the README documented two
contradictory setup paths (manual Confluent vs. camel infra run
kafka) - neither worked alone since infra kafka has no schema
registry.

Replace the Confluent client-based serializers with Camel's native
AvroDataFormat, which needs only a plain Kafka broker and no schema
registry at all. This removes the OOM-causing code outright, wires
up real Employee Avro object creation, uncomments and fixes the
avro-maven-plugin codegen, and lets the README document a single,
working setup path via `camel infra run kafka`. Also fixes two
latent bugs found while verifying end-to-end: the KAFKA_RECORD_META
header needed recordMetadata=true to populate, and consumer.group
was defined but never wired into the consumer endpoint.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HseDjsECuZhsCXxxioiD1k
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant