Rework kafka-avro example: drop Confluent Schema Registry, fix OOM - #200
Open
Croway wants to merge 1 commit into
Open
Rework kafka-avro example: drop Confluent Schema Registry, fix OOM#200Croway wants to merge 1 commit into
Croway wants to merge 1 commit into
Conversation
The example was broken: CachedSchemaRegistryClient(schemas, Integer.MAX_VALUE) in both serializer/deserializer caused an OOM crash on startup regardless of schema registry reachability, real Avro object creation was commented out (it sent a plain String, not Avro), the avro-maven-plugin codegen was commented out in the pom, the schema registry URL was hardcoded, and the README documented two contradictory setup paths (manual Confluent vs. camel infra run kafka) - neither worked alone since infra kafka has no schema registry. Replace the Confluent client-based serializers with Camel's native AvroDataFormat, which needs only a plain Kafka broker and no schema registry at all. This removes the OOM-causing code outright, wires up real Employee Avro object creation, uncomments and fixes the avro-maven-plugin codegen, and lets the README document a single, working setup path via `camel infra run kafka`. Also fixes two latent bugs found while verifying end-to-end: the KAFKA_RECORD_META header needed recordMetadata=true to populate, and consumer.group was defined but never wired into the consumer endpoint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HseDjsECuZhsCXxxioiD1k
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
kafka-avroexample was broken in several ways:CachedSchemaRegistryClient(schemas, Integer.MAX_VALUE)in both the custom serializer and deserializer caused an OOM crash on startup, regardless of whether a schema registry was reachable.String, not Avro.avro-maven-plugincodegen was commented out in the pom.camel infra run kafka) - neither worked alone, since infra kafka has no schema registry.This rewrites the example to use Camel's native
AvroDataFormatinstead of Confluent's schema-registry-backed serializers. This needs only a plain Kafka broker (no schema registry at all), which removes the OOM-causing code outright and lets the README document a single, working setup path viacamel infra run kafka.Also fixes two latent bugs found while verifying end-to-end:
KafkaConstants.KAFKA_RECORD_METAneededrecordMetadata=trueon the producer endpoint to actually populate (was silentlynull), so the previously-unwiredKafkaAvroProcessoris now wired into the producer route.consumer.groupwas defined inapplication.propertiesbut never referenced in the consumer endpoint URI.Test plan
mvn clean package(checkstyle/RAT included) passescamel infra run kafka, ranmvn spring-boot:run, and confirmed the producer builds and sends realEmployeeAvro records, the consumer unmarshals and logs them back, and record metadata logging works with no errors/OOM.