Characterize delayed queue inserts below retired boundaries

[taylan-oai]

Summary

Characterize Cassandra queue inserts that succeed below a retired read boundary, so a publication contract can be agreed before changing the protocol.

Problem

QueueStore and QueueV2 select the next ID from the current physical maximum, then insert with IF NOT EXISTS. A producer can pause after selecting ID 0. Another producer writes IDs 0 and 1; a reader processes them; cleanup removes ID 0 while retaining ID 1. When the delayed producer resumes, its insert succeeds because ID 0 is absent. Its new payload is behind the acknowledged cursor or QueueV2 minimum readable ID.

QueueV2 also has a gap between physical deletion and the metadata update. A delayed insert can succeed during that gap, and a subsequent metadata read still permits its ID. Cleanup then advances the minimum past it. Checking metadata after insertion alone would therefore miss this interleaving.

Approach

Add comments at the allocation/insertion gap and deterministic Cassandra characterization tests for both queues and the QueueV2 metadata-update gap. The tests assert the current adverse outcome and explicitly identify the future desired contract: an acknowledged insert must not newly publish its payload behind a retired boundary.

Two possible next steps need evaluation:

1. Validate the physical maximum after insertion in a bounded scope. QueueV2 retains its maximum row; under suitable consistency, a greater maximum could detect a recreated lower ID. This needs a defined uncertain-outcome error, proof for supported consistency settings, and tests for healthy writers overtaking one another. An error may follow an already consumed insertion, so retries can duplicate payloads or repeatedly fail. Legacy main-queue retention is caller-owned; legacy DLQ deletion does not preserve the same anchor.
2. Prototype a QueueV2 publication/retirement marker. Limit the experiment to one QueueV2 message partition, with participating writers and cleanup using one agreed protocol. Its initialization, atomic operations, retention, failure recovery and mixed-version behavior require design and testing before extending it to other queues. This is an option, not an established requirement.

The namespace queue retry wrapper retries Unavailable and ConditionFailedError; the History task queue manager has no corresponding wrapper, and DLQWriter wraps enqueue failures under ErrSendTaskToDLQ. A proposed error change needs caller-level validation. Process-local producer locks do not coordinate different processes with cleanup.

Validation

• All three characterization scenarios and the existing Cassandra QueuePersistence/QueueV2Persistence suites passed on Cassandra 5.0.9 (go test -p 4 -tags test_dep, 13.177s).
• Native changed-package lint passed with no reported new issues; git diff --check passed.
• All three scenarios passed with the race detector across three repetitions (12.372s), with no race reports.

The tests use real Cassandra conditional inserts, reads, deletes and metadata updates, with channel gates immediately before the selected driver query. They do not pause an already submitted Cassandra operation.

Risks, rollout, and scope

This draft changes comments and tests only. The passing tests document a defect; they do not validate either repair option. Evidence is limited to the persistence boundary and does not establish production frequency, end-to-end workflow loss, or a distributed consistency proof. Legacy main queue is covered; legacy DLQ retirement is outside this reproduction.

References

• QueueV2 caching proposal changes metadata lookup cost, with the same ID allocation/insertion sequence.
• DLQ queue creation optimization preserves process-local serialization and explicitly leaves cross-process message-ID contention out of scope.
• Legacy replication DLQ merge fix concerns a separate task store and retirement protocol; it does not change QueueStore or QueueV2 publication.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}