[Dataset][CI] TestReservationListener crashes the test JVM by closing a pool still in use #1455

Describe the bug, including details regarding any error messages, version, and platform.

The RC workflow's Binaries job intermittently fails because the JVM running the arrow-dataset tests crashes. Surefire reports TestAllTypes as the crashed test (Process Exit Code: 134, "The forked VM terminated without properly saying goodbye"), and sometimes TestAllTypes fails with a memory leak instead. It has failed about 1 in 8 push runs on main since at least Aug 25. Recent examples:

• main: https://github.com/apache/arrow-java/actions/runs/42614743276/job/129174599773
• MINOR: Bump com.nimbusds:oauth2-oidc-sdk from 11.38.2 to 11.39 #1453: https://github.com/apache/arrow-java/actions/runs/42576225131/job/129026485158
• MINOR: Bump com.diffplug.spotless:spotless-maven-plugin from 3.10.4 to 3.11.0 #1452 (leak variant): https://github.com/apache/arrow-java/actions/runs/42576170397/job/129025668681

The crash comes from TestReservationListener#testErrorThrownFromReservationListener, which runs just before TestAllTypes in the same JVM. The three hs_err files uploaded so far (#1428) show the same stack on an Arrow C++ thread-pool thread. Symbolized against the job's libarrow_dataset_jni.so:

SIGSEGV (0xb) ... si_code: 2 (SEGV_ACCERR)   <- call to a heap address
arrow::PoolBuffer::Reserve(long)+0x2d3
arrow::PoolBuffer::Resize(long, bool)+0x5f
arrow::PoolBuffer::MakeUnique(arrow::MemoryPool*, long)+0x67
arrow::AllocateBuffer(long, long, arrow::MemoryPool*)+0x3b

pool->Allocate() calls through the vtable of a MemoryPool that has been freed. In each file, the last JVM event before the crash, 2 to 5 ms earlier, is the IllegalArgumentException that the test's listener throws, rethrown through JNI.

What happens:

1. The listener throws from reserve(), so the scan fails.
2. TestDataset#collectResultFromFactory rethrows without closing the scanner or the dataset.
3. The test closes the NativeMemoryPool, and releaseMemoryPool deletes the native ReservationListenableMemoryPool.
4. Arrow C++ is still running the failed scan's async work (Parquet file opens, read-ahead) on its thread pools. That work holds a raw pointer to the pool and allocates from it or frees to it after it's deleted.

I reproduced this on macOS arm64 with the jni-macos-aarch_64 libraries from the RC workflow, looping the test's steps up to 1,000 times per JVM:

• Pool closed after each scan, as the test does now: 5 of 6 runs failed. The JVM crashed (in PoolBuffer::Resize, or an abort in ReservationListenableMemoryPool::Impl::Free while a Parquet reader was destroyed on an IO thread), or a scan whose listener always throws completed without an error.
• Scanner and dataset closed before the pool: 6 of 8 runs failed the same ways, including a crash with the PoolBuffer stack above. Closing the scanner doesn't wait for that work.
• Pool left open, with or without closing the scanner: 10 of 10 runs passed.

This issue covers the test. A listenable pool being freed while native scan work still uses it is a library problem, and I'll add the native-side details to #246, which reports sporadic failures and crashes in this test class.