fix(deps): update module github.com/twmb/franz-go to v1.22.0

[renovate]

ℹ️ Note

This PR body was truncated due to platform limits.

This PR contains the following updates:

Package	Change	Age	Confidence
github.com/twmb/franz-go	v1.3.1 → v1.22.0

---

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

Release Notes

twmb/franz-go (github.com/twmb/franz-go)

v1.22.0

Compare Source

===

This release supports Kafka 4.3 and 4.4, has a few new APIs, and has a few
big internal improvements. In particular, I recommend checking out the new
StreamingCompression option, as well as evaluating if you'd like to use
RackAwarePartitioning. There are some behavior changes that you should read
about below. The "next gen" rebalancer is now usable via the new
ServerSideBalancer option. It's had a few releases to shake out bugs
internally (via integration tests and LLM audits), but if you do experience a
bug, please open an issue straightaway.

Some minor bug fixes (that were never reported) were found during the
implementation that are not worth mentioning.

kfake has also been significantly extended and I recommend checking out the
new APIs, in particular:

• A new Fault type to make it easier to inject errors without Control functions
• Group introspection cluster APIs
• BlackholeProduce and SyntheticFetch APIs for benchmarking / play testing

My kcl CLI has been significantly expanded as well and is worth checking
out. It supports essentially everything you can do with a cluster, and now
allows you to run a full broker locally via kcl fake (in memory or a dumb
disk backed localhost broker) - as well as setup the fake broker with fault
injection. I've been running LLM audits and extensions to kcl in particular
to try to shape it up to a "finalized" CLI shape. If you use it and have ideas
for improvements, please open an issue.

Behavior changes

• Rack aware group partition assignment (KIP-881) now requires BalanceRacks.
  v1.21.0 enabled group balancers to assign partitions based on the rack that
  members were in if you used the range or sticky/cooperative-sticky balancers.
  Well, Rack is also used to opt into preferred read replica assignment
  when fetching by the broker itself. These two decisions conflict with each
  other. Now, BalanceRacks() is required to opt into group balancers using
  the rack while balancing. The client warns when balancing if BalanceRacks
  is on and the brokers have preferred read replicas enabled.

• ConsumeResetOffset defaults to RewindOffset(time.Minute) rather
  than NewOffset().AtStart(). Setting only ConsumeStartOffset no longer
  sets ConsumeResetOffset. I introduced ConsumeStartOffset a while back
  because it was really weird IMO to use a reset offset for both how a consumer
  starts and for how it recovers in the event of data loss or falling behind.
  They were bidirectional since introduction, but since start is newer and much
  less commonly used and you often don't want to recover from the start, I've
  removed the start -> reset mapping when you only set the start. I recommend
  reading the docs on both options for an updated understanding of when and
  how they apply. As well, I've introduced RewindOffset(d) which is only
  relevant to the reset offset (rewind by d duration from the last consumed
  offset on data loss we cannot exactly recover from) and LookbackOffset(d)
  which is relevant to both options but more useful for the start offset
  (start consuming d before the newest record; before Kafka 3.0 it is d
  before the current time). If a committed offset has fallen below the log
  start, the first fetch answers OFFSET_OUT_OF_RANGE and the reset offset
  decides where to resume. Before, a start offset of AtEnd was copied into
  the reset offset, so the consumer skipped to the end. Now, with the
  defaults, it resumes at the log start.
  ConsumeResetOffset's new default is RewindOffset(time.Minute).

• Topic recreation is now a hard failure. The client always
  produces to and consumes from the first instance of a topic. If you delete
  and recreate a topic, the client refuses the new version: buffered records
  fail with UNKNOWN_TOPIC_ID, fetches stop, offsets from the old topic cannot
  be committed to the new one, and transactions on the old topic fail. This
  needs a broker that reports topic IDs (Kafka 2.8+). Previously, some things
  in the client continued to accidentally work, and the behavior was
  unreliable and usually not good. If you want your application to stay alive
  across topic recreations, you can PurgeTopicsFromClient and, for
  consumers, AddConsumeTopics. More details about topic recreation are now in
  a new section in the README.

• MaxDecompressBatchBytes now blocks decompression if a batch would
  decompress too large (default 1GiB). Fetches when consuming can only
  specify to the broker "give me X bytes of batches", but they cannot control
  how large those batches decompress into. A hostile or buggy batch could OOM
  your program. Now, a batch over the limit causes the partition to enter
  a fatal state and return ErrDecompressTooLarge once from polling.
  The application can recover by manually skipping the batch with SetOffsets
  (with the fields in the error; see the docs), or by restarting the client
  with a higher limit. This option does not apply to custom decompressors,
  but, custom decompressors can still return ErrMaxDecompress to stop
  the partition. This option is also closely related to streaming compression,
  which is described below.

Improvements

• gzip now uses klauspost/compress (same format). Its default level is
  1.7x faster than stdlib's with a slightly better ratio; klauspost's default
  maps to its level 5 where stdlib's mapped to 6, and level for level it is
  1.1x to 1.2x faster. WithLevel(n) now selects klauspost's level n, so
  the bytes a given level produces differ from before.

• The sticky balancers are now exactly optimal on balance, then rack
  placement (with BalanceRacks), then stickiness. Balancing was already
  load optimal but had some very niche edge cases where maximal stickiness
  was not preserved, especially if balancing used racks. Rack placement
  outranks stickiness: turning BalanceRacks on in a running group
  reassigns, at its next rebalance, every partition held by a member in a
  different zone from the partition's leader.

• Sticky balancing is much faster, most of all on rejoins and on groups whose
  members subscribe to different topics. Against v1.21.7: a rejoin of 100
  members over 1600 topics of 100 partitions goes from 351ms to 24ms; a regex
  shaped group of 500 members over 20,000 topics from 176ms and 810MB to 12ms
  and 9MB; 2001 members over 500 topics of 2000 partitions with one narrow
  subscriber from 3.4s to 0.3s. Fresh uniform balances are unchanged.

Features

Streaming compression

StreamingCompression is an opt-in producer option that compresses a
partition's backlog of batches together, bounded by their compressed size.
By default a batch is cut at ProducerBatchMaxBytes measured on uncompressed
records. Streaming compression will help reduce traffic to the broker and
increase how effective compression actually is (by pulling more data in at
once). A custom compressor makes this option a no-op.

The client is implemented such that each compression codec's worst case
overhead is tracked internally, which should avoid a compressed batch ever
exceeding ProducerBatchMaxBytes. If this ever does happen, the client
discards the merge, logs a warning, disables streaming compression for the
client going forward (records are still compressed batch by batch), and asks
you to file an issue.

The client has a new option MaxDecompressBatchBytes to bound both (a) how
much the producer can stuff into a merged batch (i.e. how much it will
decompress into), and (b) the maximum size a consumer will decompress a batch
to; the consumer never decompresses past the bound (preventing a zip bomb).
The default is 1GiB.

Rack aware producer partitioning (KIP-1123)

RackAwarePartitioning sends unkeyed records to partitions whose leader is in
the client's Rack (which must also be set), falling back to all partitions
when no leader is. Keyed records are never affected. Unlike the Java client,
this works with any partitioner, since the eligible-broker filtering happens
before your partitioner is consulted. Note that this option skews which
partitions receive records if your producers are not spread across racks in
proportion to partition leaders.

ServerSideBalancer (KIP-848)

ServerSideBalancer opts into KIP-848 "next-gen" consumer groups, where the
broker's group coordinator assigns partitions rather than the client. This
requires Kafka 4.0+ and either a range or sticky / cooperative-sticky
balancer. This replaces the hidden opt_in_kafka_next_gen_balancer_beta
context key from v1.19.0; the key still works in this release and is removed
in the next. The default remains the classic protocol, matching the Java
client. I still think the classic client side balancers are better (and this
client's implementation is way faster than the Java client), but if you want
to use server side balancing, it is strongly recommended to only use it if
your cluster is Kafka 4.3+. Before 4.3 (before KIP-1251), an offset commit
that races with a heartbeat epoch bump can fail with STALE_MEMBER_EPOCH,
which the client cannot detect nor handle.

BalanceInfo for custom balancers

A balancer that implements GroupMemberBalancerInfo receives a BalanceInfo
before balancing: the group, generation, leader member ID, and lazily built
topic and broker metadata. ConsumerBalancer implements it, so balancers
built on NewConsumerBalancer can call Info(). This allows, for example, a
balancer that assigns every partition to the leader with the other members as
hot standbys. Thanks @​michaelwilner!

API additions

// Producing
func StreamingCompression() ProducerOpt
func RackAwarePartitioning() ProducerOpt

// Consuming
func BalanceRacks() ConsumerOpt
func ServerSideBalancer() GroupOpt
func RewindOffset(d time.Duration) Offset
func LookbackOffset(d time.Duration) Offset

// Decompression bound
func MaxDecompressBatchBytes(n int) Opt
var ErrMaxDecompress error
type ErrDecompressTooLarge struct {
    Topic      string
    Partition  int32
    Offset     int64
    Epoch      int32
    NextOffset int64
}

// Custom balancers
type BalanceInfo struct {
    Group      string
    Generation int32
    LeaderID   string
    Topics     func() map[string]TopicMetadata
    Brokers    func() map[int32]BrokerMetadata
}
type GroupMemberBalancerInfo interface {
    GroupMemberBalancer
    SetBalanceInfo(BalanceInfo)
}
func (*ConsumerBalancer) Info() BalanceInfo
type TopicMetadata struct { ... }
type PartitionMetadata struct { ... }

// Records
type RecordAttrsOpts struct {
    Codec         CompressionCodecType
    TimestampType int8
    Transactional bool
    Control       bool
}
func NewRecordAttrs(RecordAttrsOpts) RecordAttrs

// kversion
func (*Versions) EachSupportedFeature(fn func(name string, min, max int16))
func (*Versions) EachFinalizedFeature(fn func(name string, level int16))
func FeatureLevelDescription(name string, level int16) string

Relevant commits

There are many commits, but some of the more notable ones:

• 27d11286 feature kversion: FeatureLevelDescription
• 73358f62 feature kversion: supported and finalized feature levels per release
• 7be0be16 behavior change kgo: add MaxDecompressedBatchBytes
• b37f1041 feature kgo: add ServerSideBalancer to opt into KIP-848
• 033a46c7 improvement kgo: begin ApiVersions at the max a broker told us, for an hour
• 46a9b2ad behavior change kgo: use ConsumeResetOffset when the broker loses data we cannot locate
• 8b33e43d improvement kgo: speed up compression on both the legacy and the merge path
• 9de0fa36 feature kgo: add StreamingCompression, compressed-size-bound batch merging
• de7327e6 feature kgo: detect misrouted connections (KIP-1242)
• 8ad36ec7 feature kgo: support TxnOffsetCommit v6
• e4f7bc43 feature kgo: add rack-aware producer partitioning (KIP-1123)
• 123f2ffa improvement kgo: repair the sticky plan to the best balance, rack, and stickiness
• 23ab9a0e behavior change kgo: add BalanceRacks, gate rack aware balancing behind it
• d4f6db2f improvement kgo: drop reassigned partitions in one pass in AdjustCooperative
• 35efafc8 behavior change kgo: fail records for a recreated topic instead of producing by name
• 4f10346a feature kgo: expose BalanceInfo for custom balancer implementations (thanks @​michaelwilner!)
• cd7f9b4e feature kgo: add NewRecordAttrs constructor (thanks @​pracucci!)

v1.21.7

Compare Source

===

A handful of bug fixes and improvements found by users and while working on
v1.22. Rather than enumerating the relevant commits, you can check the git log
between v1.21.6 and this release - there are many minor commits. As well, kfake
has been improved significantly and has more API surface to aid in writing
tests.

• A rare, very niche panic while producing has been fixed. Thanks
  @​PumpkinDemo for the report, see
  #​1385 for more details.

• If retention deleted the segment a consumer was reading, the
  OffsetOutOfRange reset listed by the last consumed timestamp and could skip
  surviving records, or jump to the log end and skip everything. The reset
  now resumes at the log start when below it.

• Improved KIP-951 handling (the broker returning where a partition should move
  with the produce response if the partition changed leadership). Previously, a
  broker could return NotLeaderForPartition and hint the leader the client
  was already using, at the same or an older epoch. These hints are now
  ignored and the client backs off, rather than spinning. Thanks
  @​3AceShowHand for the report and
  @​jjj-n for a fix, see
  #​1412.

• Rack aware balancers ignored rack for any topic the group leader did not
  itself consume. A client now loads the rack for all partitions in the
  group, even if the leader does not consume some of the topics.

• The metadata cache has been improved (there were a few cases where it was
  emptied erroneously).

• Regex consuming now consistently never matches internal topics such as
  __consumer_offsets. As well, the regex log no longer reports an excluded
  topic as both added and skipped (thanks @​lahsivjar).

• Decompression allocates less (thanks @​scunningham).
  If you use pools, slices are now reliably returned if decompression errors.

• The client now starts at a random seed broker rather than always the
  first, so many clients starting at once no longer all hit the same seed.
  Thanks @​chailuecha!

• A producer that receives RequestTimedOut or NotEnoughReplicasAfterAppend now
  retries after the produce backoff rather than waiting for a metadata refresh.

• A few other minor improvements and bug fixes.

v1.21.6

Compare Source

===

Some bug fixes (mostly minor - hence the delay for the release) found by users
and further Claude audits. I am gearing up for a 1.22 release but some of the
features I am planning for are more complicated to review, so it may take a bit
of time. Anyway:

• Previously, rollback from a cooperative group to an eager group was
  deliberately not supported and there was a data race condition if this
  happened. It is now technically supported, although you will experience
  duplicate data. If you want a safe non-duplicate-causing rollback, you need
  to turn off the entire group, remove the cooperative consumer, and swap the
  whole group to eager rebalancing.

• Fixed a panic: close of closed channel on an acks=0 produce connection
  in a specific edge case (a broker connection dying before the connection
  was fully established caused the panic).

• If EndTransaction failed with an unconfirmed outcome (a transport error,
  exhausted retries, or UNKNOWN_SERVER_ERROR), the documented abort retry
  was a wire no-op and the next transaction could silently commit the prior
  "failed" transaction's records under KIP-890 part 2. The producer ID is now
  flagged for reload, which fence-aborts anything still ongoing broker-side.

• GroupTransactSession.End could hang forever, ignoring its context, if the
  group had never joined (e.g. the consumed topic did not exist yet) and the
  transaction committed no offsets.

• Previously, if a broker replied to ApiVersions with an error, we ignored it
  and you would eventually see an unclear error (usually a bare io.EOF, since
  anything that rejects ApiVersions hangs up right after replying). These
  errors are now handled correctly.

• Some niche edge case bugs that are only worth reading about if you're super
  interested were found in repeated Claude audits and were fixed (check the PR
  / git history). This includes further KIP-848 "next gen consumer group" fixes.

Relevant commits

• 582e0f21 bugfix kgo: surface error codes in ApiVersions responses
• 67ef4c61 bugfix kgo: fix double close of a connection's deadCh on acks=0 produce
• 3ac2fff1 bugfix kgo: revoke everything when the group protocol downgrades from cooperative to eager
• 795d5b61 improvement kgo: flatten topic/partition maps in group rebalance logs (thanks @​constanca-m!)
• 70addc1e improvement kgo: classify retired broker reads as broker dead (thanks @​tomplarge!)
• 18f9a10f improvement deps: replace golang.org/x/crypto/pbkdf2 with stdlib crypto/pbkdf2 (thanks @​macdewee!)
• 6ecd2f9f bugfix kgo: recover when an attempted EndTxn outcome is unconfirmed
• 821f879e bugfix kgo: fix GroupTransactSession.End hanging when the group never joined

v1.21.5

Compare Source

===

Three bug fixes:

• Fixed a nil-pointer panic when building a group OffsetFetch: if the group
  was assigned a topic that was no longer in the client's tracked set --
  reachable when a topic is purged from consuming while still assigned, for
  example PurgeTopicsFromConsuming overlapping AddConsumeTopics, or the
  automatic regex missing-topic purge -- loadTopic returned nil and
  dereferencing it for the topic ID crashed the client. The topic ID is now
  only set when the topic is known. Thanks @​iwittkau!

• Fixed a data race on a coordinator's cached node ID. When a broker
  disconnected while a FindCoordinator load for that broker was still in
  flight, deleteStaleCoordinatorsByNode could read the in-flight load's
  node field before the loading goroutine published it (via closing the
  load's wait channel), which go test -race flagged. The node read now
  happens only after the load has been observed as complete. Thanks
  @​nikolauspschuetz!

• A share partition that was listed in a ShareFetch only to carry a
  piggybacked acknowledgement -- for a cursor that was revoked, paused, or
  migrated to a new leader after its records were drained -- was added to the
  broker's share session but never tracked client-side, so it could never be
  forgotten. The broker would re-acquire and redeliver that partition's
  records indefinitely while the client discarded them ("broker returned
  partition ... we did not ask for"), spinning the share fetch loop. The
  client now tracks every partition it sends, matching the broker's session
  bookkeeping.

Relevant commits

• ab185e42 bugfix kgo: add nil check when loading topics in groupConsumer (thanks @​iwittkau!)
• aca084ed bugfix kgo: fix data race on coordinatorLoad.node (thanks @​nikolauspschuetz!)
• 754bc349 bugfix kgo: forget piggyback-only partitions from the share session

v1.21.4

Compare Source

===

This release is a "large" (many commits) release that has many small or
hard to encounter bugs fixed. I pointed Claude's Fable at this repo and
ran some audit rounds while available and thankfully got through the highest
value audit rounds before Fable was removed.

For once, I will not be describing every bug fixed nor calling out every
relevant commit. Instead, if you are curious, look at
#​1348. Some worthwhile
description is below.

Three important bug fixes to call out:

• In transactional exactly-once consuming, a SetOffsets seek (which happens
  during GroupTransactSession.End after an aborted transaction) could be
  undone by a concurrent offset load (via a background list or epoch load) that
  completed slightly later. This could happen when the client discovers a
  partition leader moved while you are aborting, which could result in missed
  records.

• Consuming with read_committed against a broker that returns a partition's
  aborted-transaction list out of offset order could surface aborted,
  rolled-back records as if they were committed. Apache Kafka always returns
  them in order so this was never observed there, but Redpanda does not (when
  an aborted transaction is still in memory and an earlier one is already on
  disk). The list is now sorted client-side, matching the Java client,
  librdkafka, and Sarama.

• GroupTransactSession.End no longer reports a successful commit when the
  broker answers EndTxn with UNKNOWN_SERVER_ERROR (seen from Redpanda in
  some older versions). Previously the consumer's offsets were advanced past a
  transaction that may have aborted; now the commit is reported as failing
  and the session rewinds for reprocessing.

Beyond those, by area:

• Many transaction-path fixes for coordinator churn and KIP-890 part 2 that
  would have resulted in not-working (hard client fail) or hung transactions:
  InitProducerID retries CONCURRENT_TRANSACTIONS when taking over a
  crashed producer's transaction, retriable producer-id load failures are no
  longer treated as fatal, KIP-890p2 is opted into only when the negotiated
  versions actually support it (fixing spurious INVALID_TXN_STATE on 4.0+
  clusters running older semantics), a transaction whose every produce failed
  now aborts instead of hanging until the transaction timeout, and a failed
  AddPartitionsToTxn no longer drops partitions added by an earlier request.

• GzipCompression().WithLevel(...) was completely broken and would panic.

• More KIP-848 (next-gen consumer group) robustness fixes under coordinator
  and leader churn.

• Stale consumer-group member rejoining fixes: a member that rejoins claiming a
  partition at an old generation no longer panics the group leader or causes
  two members to consume the same partition. Malformed member metadata,
  duplicate member ids, and negative claimed partitions in a join are now
  rejected or sanitized rather than mis-balancing or panicking the leader.

• Metadata and topic recreation: a stale per-broker metadata view that
  momentarily omits a just-added partition (the window right after
  CreatePartitions) no longer fails buffered producer records or leaves a
  newly assigned consumer / share partition silently unconsumed; both heal
  once metadata catches up.

• Share consumer: fetch errors are now classified like the classic consumer
  (retriable errors stripped, a metadata refresh triggered to heal a leader
  move, top-level errors backed off) instead of stalling for up to
  MetadataMaxAge or hot-looping, and leader-move migrations are tracked so
  that leaving or closing cannot strand un-acked records.

• SASL: KIP-368 re-authentication no longer races the connection's other
  reader, which could corrupt pipelined traffic on brokers that set a session
  lifetime (e.g. AWS MSK IAM); requests now park and replay across a re-auth.
  The Azure Event Hubs ApiVersions reset retry no longer leaks the abandoned
  connection or silently downgrades it to v0.

• KIP-714 client telemetry: the terminating push is now actually delivered on
  Close, the .rate and .avg rollups are computed correctly (they were
  constant / wrong before), and an unsupported user-metric attribute no longer
  corrupts the OTLP payload (which had disabled metrics for the rest of the
  client's life).

• Smaller consumer fixes: overlapping manual CommitOffsets no longer reopen
  autocommit early (which could rewind the committed offset), a conformant
  UNDEFINED_EPOCH_OFFSET epoch response no longer raises a false
  ErrDataLoss, and Fetches.EachTopic now preserves TopicID across
  multi-broker responses (it was zero whenever more than one broker replied,
  i.e. normally).

• RecordReader / RecordFormatter no longer panic on truncated or malformed
  layouts, accept \xNN escapes for bytes above 0x7f, and reject layouts that
  would read nothing and loop forever.

• WithPools: decompression no longer produces garbage when a pool hands back
  a non-zero-length sized slice, and pooled slices are no longer leaked for
  batches that keep no records (e.g. aborted-transaction data under
  read_committed).

• Other producer fixes: EnsureProduceConnectionIsOpen dials the right broker
  for filtered ids and no longer breaks an acks=0 connection, the adaptive
  LeastBackupPartitioner now actually picks the least-backed-up partition,
  and producing during or after Close fails cleanly instead of hanging a
  later Flush.

• A broad set of guards against malformed or hostile broker responses that
  could previously panic the fetcher, hot-loop, or mis-consume: negative or
  oversized record counts and batch lengths, decompression bombs, duplicate or
  omitted partitions, negative offsets, and unexpected top-level fetch errors.
  The client is also more resilient when its own API contracts are violated
  (e.g. AllowRebalance called while a poll is in flight).

v1.21.3

Compare Source

===

This patch release contains a few bug fixes and a few internal improvements.

• PollRecords / PollFetches could permanently hang since v1.21.0 if a
  consumer session stopped (usually via metadata updates) while
  fetches to more than four brokers were pending and no poll was in
  flight. This could only affect users that deliberately set MaxConcurrentFetches(0),
  or that were using ShareMaxRecordsStrict.

• Producing to a topic whose partitions ALL have a retriable load error
  (e.g. a rolling restart of an RF=1 broker briefly leaving every
  partition leaderless) no longer fails records up front with "unable to
  partition record due to no usable partitions". Instead, the records
  remain buffered and retried as metadata reloads.

• Classic consumer groups now rejoin immediately when an offset commit
  returns UNKNOWN_MEMBER_ID or ILLEGAL_GENERATION (the broker lost
  the member, e.g. a session expired during a network blip), rather than
  consuming as a zombie until the heartbeat loop notices the dead session.

• DescribeShareGroupOffsets, AlterShareGroupOffsets, and
  DeleteShareGroupOffsets are now routed to the group coordinator
  rather than the share coordinator (which would reject the requests
  for being misrouted).

• The client-internal metadata cache now deeply clones the cached response
  before putting it into the cache and before returning it via
  RequestCachedMetadata (which is now used by default in kadm), eliminating
  data race possibilities.

• Various next-gen rebalancer session improvements.

Relevant commits

• f8842170 improvement kgo: fall back to all partitions when no partition is writable (thanks @​ericsg666!)
• 824e34d2 improvement kgo: rejoin a classic group when a commit returns a fatal member error (thanks @​v14dis14v!)
• f520e820 bugfix kgo: do not exit manageFetchConcurrency while sources are pending in wantFetch (thanks @​SLoeuillet!)
• 8d9c836b bugfix kgo: isolate metadata cache from broker response
• 19f7dbb2 bugfix kgo,kfake: route share group offset RPCs to the group coordinator

v1.21.2

Compare Source

===

This patch release contains two narrow bug fixes and one small feature.
Deps are also bumped so that you are force-pinned to a klauspost/compress
version that has a stack-splitting bugfix that sometimes affected franz-go.

• PurgeTopicsFromConsuming now correctly persists deleted topics if
  you also had specific topics paused. Previously, when a topic had
  partition-level pauses, unpausing the topic itself (while keeping
  specific partitions paused) was bugged and the topic was stuck in
  an "all paused" state (thanks @​gorakdev!).

• PollFetches no longer surfaces a spurious UNSTABLE_OFFSET_COMMIT
  fetch error when an OffsetFetch retry is canceled mid-wait by a
  rebalance or client close. Observed flaking TestTxnEtl/sticky/848
  on KIP-848 consumer groups under transactional load.

• The MSK IAM SASL mechanism now honors AWS_REGION when the broker
  hostname does not match the standard MSK URL format, allowing
  connections through custom DNS names (e.g. private link endpoints)
  (thanks @​janmoritzmeyer0210!).

Relevant commits

• 22a17320 bugfix kgo: do not inject fake fetch error when OffsetFetch retry is canceled
• 95d74ab3 bugfix kgo: fix delTopics not writing back modified pausedPartitions struct (thanks @​gorakdev!)
• 40e5a0e5 feature sasl/aws: support custom AWS MSK DNS names (thanks @​janmoritzmeyer0210!)

v1.21.1

Compare Source

===

This patch release contains a few bug fixes in the new share consumer and a
few internal improvements. The highlights are noted below; if you are
interested in all the improvements, check the commits.

• Share consuming no longer panics on empty batches from the broker.

• The share acking previously did not work in tandem with Record.Recycle;
  that has been fixed. Records that are recycled without being ack'd are
  now also auto-accepted.

• PurgeFetchTopics no longer panics on KIP-848 consumer groups.

• ListGroups and ListTransactions fan out to every broker and
  merge the responses. During a group-coordinator or
  transaction state leader migration, the same group or
  transactional ID can transiently appear in both the old and new
  coordinator's response, producing duplicates in the merged result.
  The merges now dedupe by key (first shard response wins).

• Two narrow producer logic races have been fixed; the code previously
  auto-fixed after some time and there were no correctness issues, but
  now the logic race has been eliminated.

• Azure Event Hubs (and any other broker that advertises high
  Produce / OffsetFetch / OffsetCommit versions while capping Metadata
  below v10) is now handled correctly. Previously, OffsetFetch v8/v9
  responses were spuriously stamped with UNKNOWN_TOPIC_OR_PARTITION
  client-side, and Produce v13 / OffsetFetch v10+ / OffsetCommit v10+
  could be sent with zero TopicIDs on the wire. The Produce path now
  caps at v12 if any partition lacks a TopicID, and OffsetFetch /
  OffsetCommit now pin to v9 in the same situation. Fixes #​1312.

Relevant commits

• 3c1d0d2e b7463fd8 7bb7ccbc improvement kgo: add guards across Produce / OffsetFetch / OffsetCommit for brokers that advertise high request versions while capping Metadata below v10 (Azure Event Hubs); fixes #​1312
• c606410e bugfix (and other stuff) kgo: simplify share-consumer ack tracking, fix pool-reuse race
• 2ff493a1 kgo: consumer-path audit fixes
• 1623daba kgo: produce path audit fixes
• e7bed5fc improvement kgo: dedupe groups and txns in fan-out sharders
• 96a14015 improvement kgo: consume-path alloc reductions and log-level gates

v1.21.0

Compare Source

===

This is a relatively "major" minor release. It adds support for Kafka 4.2,
adds full support for KIP-932 share groups, adds
KIP-881 rack-aware partition assignment, adds a handful of other
features / options, and fixes several niche bugs.

The companion kfake package has also been significantly extended; it now
supports everything except delegation tokens, streams APIs, and broker
internal APIs. kfake can be used as a dumb localhost broker; it has an option
to persist to disk to tolerate restarts (and it even handles quick restarts
without interrupting any client state). See the run_tests.sh script and the
main.go file in pkg/kfake if you want to see about bootstrapping this yourself.
I may create some tiny 'dumbkafka' binary that supports running on localhost
with a few options. Regardless, kfake is quite neat.

The kadm package has been extended with new share APIs. See the
incoming kadm tag for full details.

As a meta note, this was a significant time investment (>4w most evenings and
weekends for KIP-932 alone). I hope future releases require less work; 932 is
the last major feature this library has been missing for a while, and of
upcoming KIPs, only transactional support for 932 looks to maybe be some
effort. That said, if you get a lot of value from this and have a spare
quarter, please consider sponsoring.

API additions

Share groups (KIP-932)

franz-go now fully supports KIP-932 share groups for consuming.
Share groups are the "queue-like" alternative to consumer groups: many
consumers can share a single partition, records are individually
acknowledged, and unacknowledged records are automatically redelivered.

The new share group API mirrors the existing consumer group shape; see
full documentation on pkg.go.dev:

type AckStatus int8

const (
    AckAccept  AckStatus = 1
    AckRelease AckStatus = 2
    AckReject  AckStatus = 3
    AckRenew   AckStatus = 4
)

type ShareAckResult struct { Topic string; Partition int32; Err error }
type ShareAckResults []ShareAckResult

func (ShareAckResults) Ok() bool
func (ShareAckResults) Error() error

func ShareGroup(group string) GroupOpt
func ShareMaxRecords(n int32) GroupOpt
func ShareMaxRecordsStrict() GroupOpt
func ShareAckCallback(fn func(*Client, ShareAckResults)) GroupOpt

func (*Client) MarkAcks(status AckStatus, rs ...*Record)
func (*Client) FlushAcks(ctx context.Context) error

func (*Record) Ack(status AckStatus)
func (*Record) DeliveryCount() int32
func (*Record) AcquisitionDeadline() time.Time

Two new RecordFormatter verbs were added alongside this:

• %D - share group delivery count
• %A - share group acquisition deadline (timestamp; supports the same
  strftime/Go formatting as %d)

Rack-aware partition assignment (KIP-881)

Both the range and sticky balancers now understand consumer racks. If you
set kgo.Rack, the leader will preferentially assign you partitions whose
leader is in the same rack, preserving the existing priority of balance
over locality over stickiness. I was originally not planning to support this
in franz-go since the next generation rebalancer was releasing at a similar
time, but I suspect it's worth it to keep the client-driven rebalancing for
a while.

Custom balancers that use the consumer protocol can use the new
(*ConsumerBalancer).PartitionRacks() method to access the computed
partition-rack map.

Other new kgo APIs

func AllowIdempotentProduceCancellation() ProducerOpt
func ProducerBatchMaxBytesFn(fn func(string) int32) ProducerOpt
func AlwaysRetryEOF() Opt

type HookPollStart interface {
    OnPollStart(ctx context.Context)
}

• AllowIdempotentProduceCancellation permits cancellation of in-flight
  idempotent records, at the cost of breaking idempotency's duplicate
  guarantee. When a record is in-flight, the client cannot tell "never
  written" from "written but reply lost". Cancelling leaves the client's
  sequence window inconsistent with the broker: the next produce either
  silently gap-accepts (broker wrote the cancelled records) or hits
  OUT_OF_ORDER_SEQUENCE and forces a producer ID reload (broker did
  not). Any application-level retry of a cancelled record that the
  broker actually stored will duplicate on the broker - idempotent
  dedupe cannot help because the window has reset. By default the
  client refuses to cancel in this state and waits for the record's
  outcome. Use this when time-bounded delivery matters more than
  duplicate-avoidance. Incompatible with a transactional id.
• ProducerBatchMaxBytesFn takes a topic name and returns the max batch
  size for that topic, following the RetryTimeout / RetryTimeoutFn
  pattern. Useful when you produce to multiple topics with different
  broker-side max.message.bytes.
• AlwaysRetryEOF keeps retrying EOF errors indefinitely for users whose
  infrastructure considers EOF always transient. This option is actually
  generally recommended for all users, BUT you really have to ensure your
  SASL and TLS is setup correctly when using this option. Invalid SASL or
  TLS is only visible as an EOF error, so the client by default uses
  heuristics to hard fail requests without retrying if the first write triggers
  an EOF. This has bit some users over time due to an EOF ALSO being seen
  during restarts; this new option allows you to say "trust me, I know
  my configuration is correct: keep retrying".
• HookPollStart fires at the start of each PollFetches /
  PollRecords call. Thanks @​rarguellof91!

Misc additions

• kadm gained a RequireStable option for offset fetching.
• kfake.VirtualNetwork plus a new kgo xsync package enable full
  testing/synctest support against an in-process kfake cluster. See
  examples/testing_with_kfake_and_synctest. Thanks
  @​cupcicm!

Behavior changes

• MaxConcurrentFetches(0) previously meant "unbounded"; it now means
  "no background fetches: a single fetch is only issued while you are
  polling". This was a deliberate change to support
  ShareMaxRecordsStrict, where no buffering is recommended because the
  broker-side acquisition timer starts as soon as records are
  returned. Use a negative value (e.g. -1) for the prior unbounded
  behavior (or just don't use the option; the default is unbounded).
  This also means you can now use 0 to disable pre-fetching entirely,
  even outside of share groups.

• ApiVersions is now sent on every new broker connection, not just
  the first one per broker. The broker uses the ApiVersions request's
  ClientSoftwareName / ClientSoftwareVersion fields to scope KIP-714
  metric subscriptions; caching and skipping ApiVersions meant later
  connections (fetch, group, etc.) registered as software=unknown, and
  one kgo.Client appeared as two software entities to the broker.
  Versions are still cached for request-version selection, so this is
  only a one-extra-request-per-connection cost.

• Group consumer OffsetFetch now hardcodes RequireStable=true. The
  prior behavior could return stale committed offsets during a
  rebalance, and in the worst case, result in duplicate messages.
  You no longer need to use the RequireStableFetchOffsets option.

• ErrRecordTimeout now wraps the last retry error seen while waiting
  on metadata in waitUnknownTopic. errors.Is continues to work for
  ErrRecordTimeout; you can now also errors.Is the underlying cause
  (e.g. SASL auth errors that previously disappeared).

• BlockRebalanceOnPoll no longer gates the assign-side callback if the
  user did not register OnPartitionsAssigned. Assign only adds
  partitions, so a user's in-flight commit cannot reference anything
  they don't still own; the gate only existed to serialize user
  callbacks with poll.

• MessageTooLarge errors now include the uncompressed and compressed
  message sizes. Thanks @​anubhav21sharma!

Bug fixes

• Fixed a KIP-951 bug where ensureBrokers could destroy unrelated
  broker objects when the broker set changed. This would manifest to you
  as "the broker chosen for the request is dead" (or does not exist).

• listOffsets had two related bugs: a nil cursor panic in the drain
  path, and a race where a metadata signal could be lost. These were
  possible to encounter on extremely fast setups (the bugs were years
  old, and I only encountered via localhost kfake testing).

• The KIP-848 heartbeat path had several bugs that were shaken out by
  long spin loops against the new kfake share/848 support (that said,
  I am still not opting into KIP-848 by default, and will only allow
  an option-based opt-in once KIP-1251 is released in Kafka 4.3).

• fetchOffsets now validates OffsetFetch responses against what was
  requested (brokers have been seen to omit partitions) and correctly
  handles the group-level error code.

• Fixed a TOCTOU race between producerID and createReq that could
  construct a produce request with a stale producer ID.

• failDial now actually clears the stale coordinator / controller
  cache, so a single bad dial does not poison subsequent discovery.

• Various smaller fixes: retrying when broker.go wrote 0 bytes (treat
  as "didn't try"), transient dial errors now retry for the full
  configured retry budget rather than the hard-coded ~1.5s cap (which
  previously caused failures across 5-30s rolling restarts).

Improvements

• batchPromises is now backpressured: the ring is a dynamically-sized
  circular buffer that can block pushes at max(maxBufferedRecords, 8192). Previously it could grow unboundedly and OOM the client when
  records failed faster than finishPromises could process them.

• Two rounds of allocation reductions in hot paths: the produce blocking
  path no longer forces its closure onto the heap, recBuf's linger
  timer is reused, brokerCxn's 4-byte read buffer is reused,
  headerless records no longer allocate an empty header slice, and some
  deprecated APIs were swapped out.

• InitProducerID failures caused by transient broker errors (dial
  refused, EOF across a broker restart, etc.) no longer surface as a
  fatal "unrecoverable producer ID" error. The client now marks the
  producer ID for reload on these errors so the next produce or begin
  re-runs InitProducerID against the (probably now-available) broker.

Relevant commits

• 8854973c improvement kgo: recover producer ID from transient broker errors in maybeRecoverProducerID
• c9a91d11 feature kgo: add xsync package + kfake: add VirtualNetwork for synctest (thanks @​manuc-conf!)
• a5a7c6f2 kgo: send ApiVersions on every new broker connection
• 14346ddc feature kgo: add rack-aware partition assignment (KIP-881)
• 40bf3e52 feature kgo: add HookPollStart hook (thanks @​rarguellof91!)
• 68c85147 behavior change kgo: skip BlockRebalanceOnPoll gate on assign with no OnPartitionsAssigned
• 6d9a0188 feature kgo: add %D (delivery count) and %A (acquisition deadline) to RecordFormatter
• 0f0bca22 feature kgo: add support for share groups (KIP-932)
• 8762d567 kgo: retry if we failed at 0 bytes written
• 3e2f61a8 kgo: retry transient dial errors for the configured retry budget
• 857ed6dc behavior change kgo: wrap last retry error into ErrRecordTimeout
• d922b883 kgo: fix failDial to actually clear stale coordinator/controller cache
• af5abf66 bugfix kgo: fix KIP-951 ensureBrokers destroying unrelated broker objects
• 6c7aab54 bugfix kgo: fix listOrEpoch drain and nil cursor panic in listOffsets
• 416e8269 kgo: do not cancel prior in-flight offset commits
• 2f666797 kgo: simplify STALE_MEMBER_EPOCH retry in offset commit
• 9dde6ad8 behavior change kgo: hardcode RequireStable for group consumer OffsetFetch
• 8a9400df bugfix kgo: fix listOrEpoch race where metadata signal is lost
• 0338467d feature kgo: add AllowIdempotentProduceCancellation option
• bc46151a feature kgo: add ProducerBatchMaxBytesFn for per-topic batch size limits
• de2dff52 bugfix kgo: handle group-level error code in fetchOffsets
• 45ec9cfb bugfix kgo: fix TOCTOU race between producerID and createReq
• 63cf8fa6 kgo: add backpressure to batchPromises ring buffer
• 9f15841b feature kgo: add AlwaysRetryEOF option
• 1de163c3 include uncompressed/compressed message size in MessageTooLarge error (thanks @​anubhav21sharma!)

v1.20.7

Compare Source

===

This patch release fixes numerous niche bugs - some user reported, some found
while investigating other things - contains a few behavior improvements,
extensive kfake additions, and many test additions / improvements.

There have been extensive additions to kfake over the course of the past
month; kfake now supports transactions, the next generation consumer group
protocol, and more Kafka APIs. franz-go integration tests now run and pass
against kfake in CI.

Testing has further been extensively improved: integration tests now run
against the latest patch version of Kafka for all major Kafka versions going
back to 0.11.0. The existing test suite has been extended to run while opting
into the next generation consumer group. An integration test against Kerberos
has been added. For ~roughly the past year (maybe half year), all bugs found
have had regression tests added in kfake. This release massively extends the
kfake test suite -- both with behavior tests that were ported via Claude
directly to franz-go (with license attribution!) and with behavior tests that
Claude generated specifically for kfake.

The "next generation" (KIP-848) consumer group code in franz-go itself has some
improvements. These improvements were found after adding 848 code to kfake,
which allowed for much faster integration test looping. This looping was still
very slow for what it's worth; towards the end, integration tests would pass
~40+ times with race mode over the course of two hours before failing once.
Claude was instrumental with adding appropriate log lines and tracing logs for
diagnosing extremely niche failures; things also got slower when a few specific
log lines that would've helped weren't added the first time...

Anyway,

Bug fixes

• Returns from PollRecords / PollFetches that contained ONLY an error (context
  cancellation or something) previously did not block rebalances, even if you
  opted into BlockRebalanceOnPoll.

• If, while idempotently producing, the client encountered TIMED_OUT while
  producing (retryable), the client considered this a "we definitively did not
  produce" state, and allowed you to cancel the records. Well, maybe the records
  actually did get produced broker side eventually, and now you re-produce new
  records - the NEW records could be "deduplicated" due to how idempotency works.
  This one is a bit niche, if you're interested, you should read #​1217 and the
  two PRs that address it.

• My original implementation of how Kerberos handled authentication was
  correct... for the time. I missed how it should have been touched up years
  ago and now 4.0 hard deprecates the old auth flow. So, that's been found,
  reported, and now fixed.

• If a partition returned a retryable error in a metadata request (odd behavior
  already) the first time the client is discovering the partition (i.e. on
  startup), the client would not retry loading the partition right away (it
  would, but much later on standard metadata refresh).

• There was a very subtle, basically un-encounterable data race while
  consuming. That was fixed in 5caaa1e0. It's so niche it's not worth
  writing more about here.

• RequestCachedMetadata, broken for a few releases, has been fixed. I plan to
  switch kadm back to using it in the next kadm release.

• There was a data race on client shutdown while leaving the group if you canceled
  the client context.

Improvements

• ConnIdleTimeout is now obeyed more exactly, allowing you to more reliably reason
  about the maximum idle time. Thanks @​carsonip!

• At the end of a group transact session, I force a heartbeat to kinda "force
  detect" the group is still alive (with some timing bounds). If the heartbeat
  detected any error - including REBALANCE_IN_PROGRESS - the session would
  abort no matter what. This has been changed to actually allow a commit (if
  that's what you're doing) in certain scenarios (notably: the client detects
  KIP-447 support on the broker and you are using RequireStable).

• SetOffsets now does not let you set offsets for partitions that are not being
  consumed. Previously, you could, but it'd just create entries in a map that were
  never used. Those entries are no longer created.

• ProduceSync now automatically un-lingers any partition that is produced to,
  causing more immediate flushes. This should resolve lag that was introduced
  from v1.20.0 where linger was set to 10ms by default. There are many usages of
  franz-go I've seen in the wild where ProduceSync is used in random places to
  produce one message before going back to other things.

• 764eb29d improvement kgo: unlinger

❗ Important

✂ PR body was truncated to here.

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[renovate]

ℹ️ Artifact update notice

File name: go.mod

In order to perform the update(s) described in the table above, Renovate ran the go get command, which resulted in the following additional change(s):

• 3 additional dependencies were updated
• The go directive was updated for compatibility reasons

Details:

Package	Change
go	1.17 -> 1.26.0
github.com/twmb/franz-go/pkg/kmsg	v0.0.0-20220114004744-91b30863ac2f -> v1.14.0
github.com/klauspost/compress	v1.13.6 -> v1.20.0
github.com/pierrec/lz4/v4	v4.1.11 -> v4.1.30