[CI][Java] Sporadic failure in {{testDirectReservationListener}} #246

Activity

1. asfimport commented on Sep 13, 2022

   

   asfimport

   on Sep 13, 2022

   Author

   More actions

   Antoine Pitrou / @pitrou:
   cc @zhztheplayer @lwhite1  
2. laurentgo commented on Jun 5, 2024

   

   laurentgo

   on Jun 5, 2024

   Last edited by laurentgo

   Contributor

   More actions

   I also observe sporadic crashes in the same test (see conversation in apache/arrow#41772)

   I managed to get a run and capture the jvm output, the hotspot error file and a core dump. The archive is located at https://github.com/laurentgo/arrow-crossbow/actions/runs/9387573961/artifacts/1571969955 (1.3GB)

   Relevant files are:

   • Users/runner/work/arrow-crossbow/arrow-crossbow/arrow/java/dataset/target/surefire-reports/2024-06-05T16-54-43_206-jvmRun1.dumpstream
   • Users/runner/work/arrow-crossbow/arrow-crossbow/arrow/java/dataset/hs_err_pid12700.log
   • cores/core.12700
3. assignUser

   transferred this issue fromapache/arrowon Nov 26, 2024
4. jbonofre commented on Nov 9, 2026

   

   jbonofre

   on Nov 9, 2026

   Member

   More actions

   The crash files that the RC workflow uploads since #1428 show the cause of the crashes in this test class. Details are in #1455.

   NativeMemoryPool#close() calls releaseMemoryPool, which immediately deletes the native ReservationListenableMemoryPool and the global reference to the Java listener. Scanners get the pool as a raw arrow::MemoryPool* through ScannerBuilder::Pool. After a scan fails, Arrow C++ can still run work for it on its thread pools: Parquet file opens, read-ahead, and destroying readers whose buffers came from the pool. That work then calls Allocate or Free on the deleted pool. Closing the scanner first doesn't help, because the close doesn't wait for that work. I only reproduced the failed-scan case, but a scanner closed before the end of the data may leave work running too.

   In testErrorThrownFromReservationListener, this crashed the test JVM in about 1 in 8 Binaries runs on main. #1457 stops that test from closing the pool, but the library problem remains. The original failure here (expected:<8194> but was:<8193> in testDirectReservationListener) looks like a reservation update arriving after the scan returned, which could have the same cause, but I haven't confirmed that.

   A fix in the library could be:

   • releaseMemoryPool marks the pool closed instead of deleting it.
   • After close, Allocate and Reallocate return an error status without calling the listener, and Free returns the memory to the parent pool without calling it. Today a failed listener call in Impl::Free hits ARROW_LOG(FATAL).
   • The listener's global reference is deleted and the native object freed only once no call can still be in progress, for example with a count of in-flight calls. Waiting for bytes_allocated() to reach 0 isn't enough, since an allocation can start after the check. Leaking the small native object is a simpler option.

   Making a scanner's close wait for its outstanding work would also fix it, but I don't think the Arrow C++ scanner API offers a way to wait for that work.