storage-cassandra: a concurrent list_append test found one entry more than its calls #893

tests/test_concurrency.py::TestConcurrentListAppend::test_concurrent_list_append seeds an item with an empty list. Then 50 threads, each with its own client, send one UpdateItem with SET events = list_append(events, :items). Each call appends the entry of its thread. _retry_on_internal_error sends a call again after InternalServerError or TransactionConflictException. The test expects 50 entries.

It failed in one cassandra-pytest job:

• job 127864622393, fix/console-iam-user-own-pages (fix(console): serve an IAM user its own user page and access keys only #844), 6 November 11:21 UTC

>       assert len(events) == NUM_THREADS
E       AssertionError: assert 51 == 50

The check before it, total_ops == NUM_THREADS, passed, so all 50 calls returned success. The list holds one entry more than the calls, so at least one call's entry appears twice. Either the client sent it again after an attempt that the backend applied, or the backend applied one attempt twice. The job log has no server output, so it does not show which.

#342 was the same kind of failure on MongoDB: ADD applied twice when the backend reported a committed write as InternalServerError. #878 is about UpdateItem on Cassandra answering TransactionConflictException after 21 version checks, with nothing applied. That does not explain an extra entry.

This was 1 failure in 297 completed cassandra-pytest jobs between 3 November 00:00 UTC and 6 November 15:07 UTC. I have not reproduced it.