CASSANDRA-19576 Write compaction history asynchronously through a bounded executor

[cheeeee]

CASSANDRA-19576: SystemKeyspace.updateCompactionHistory writes to system.compaction_history on the
compaction thread. When memtable memory is exhausted that write blocks until a flush frees space, so
compactions cannot complete and fall further behind.

Changes:

• History records are written by a dedicated single-threaded executor ("CompactionHistory", JMX
  visible) with a bounded queue of 1000 records and AbortPolicy. When the queue is full or the executor
  is shut down the newest record is dropped with a rate-limited WARN; the compaction thread never
  blocks on the write and never runs it inline. History is diagnostic, so dropping a record is
  preferred to stalling compaction (this is the trade-off the ticket proposes).
• updateCompactionHistory returns a Future that completes after insertion or exceptionally on
  write failure or rejection. Because the write is deferred, the rowsMerged and compaction_properties
  maps are copied before submission.
• Shutdown ordering: StorageService.drain shuts the writer down and waits for accepted records
  (cassandra.drain_executor_timeout_ms) after CompactionManager.forceShutdown and before the system
  tables are flushed and the commitlog is shut down, so accepted records are not lost and no history
  mutation is still running while the commitlog closes. If draining exceeds the timeout, drain fails
  rather than proceeding. The in-JVM dtest Instance.shutdown, which does not go through drain, stops
  the writer the same way.
• Existing tests that read history right after a compaction now wait for the record to appear
  (CompactionTaskTest, nodetool CompactionHistoryTest, SystemKeyspaceTest).

Testing:

• CompactionHistoryWriterTest: with a queue limit of 1, a blocked write plus one queued record causes
  the next submission to be rejected with a failed future while the accepted ones complete in FIFO
  order; shutdownAndWait drains accepted records and rejects submissions made afterwards.
• SystemKeyspaceTest.testCompactionHistoryDefensiveCopy: mutating the caller's maps after the call does
  not change the stored row.
• CompactionHistoryShutdownTest (in-JVM dtest): a record written through the executor is visible, and
  Instance.shutdown stops the writer and passes the harness thread-leak check.
• The memtable-pressure scenario from the ticket was not reproduced; the tests pin the executor's
  contract and the shutdown ordering, not the original stall.

CASSANDRA-19576

[cheeeee]

Withdrawing this patch.

It does remove the stall: with the memtable pool exhausted and flush blocked, an unpatched
compaction thread parks in MemtableAllocator.allocate inside the compaction_history write and the
compaction does not finish in 45 s, while with the patch it completes in about 200 ms.

But deferring the write to a dedicated thread does not remove the dependency, it moves it: that
thread blocks on the same pool, so under the pressure the ticket describes the history is still not
written. On top of that the patch makes drain wait up to cassandra.drain_executor_timeout_ms and
then fail if the writer is parked, and turns a write failure into a log-only event.

Compaction is a path that frees resources, and its last step allocates from the pool it is trying to
relieve. A fix that holds up needs the diagnostic record to be independent of that pool, which is a
design decision for the owners of this area rather than something to land as a bug fix. I will put
the reproduction, the stack and the trade-offs in CASSANDRA-19576 so the analysis is not lost.