Core: Fix race condition in CachingCatalog loadTable (#17338) - #17375
Core: Fix race condition in CachingCatalog loadTable (#17338)#17375fightBoxing wants to merge 1 commit into
Conversation
Replace non-atomic getIfPresent + get + put sequence in loadTable() with asMap().compute() to eliminate race condition between concurrent loadTable() and invalidateTable() that could resurrect stale entries in the cache. The race: Thread A checks cache (miss) and starts loading; Thread B invalidates the cache; Thread A finishes loading and put-s a stale value. With asMap().compute(), ConcurrentHashMap serializes compute and remove on the same key, preventing the stale put. Also add regression test testConcurrentLoadAndInvalidateDoesNotResurrectStaleEntry that stress-tests 200 iterations of concurrent loadTable + invalidateTable.
| Table originTable = | ||
| tableCache | ||
| .asMap() | ||
| .computeIfAbsent(originTableIdentifier, catalog::loadTable); |
There was a problem hiding this comment.
The metadata branch performs this nested call from inside the outer compute(canonicalized, …) remapping function, mutating the same cache during its own computation. This is the anti-pattern PR #3801 deliberately removed after production deadlock #3791; both Caffeine's asMap() view and the CHM it wraps document that the mapping function must not update the map during computation. Depending on interleaving and key hashing, it can deadlock or throw a "Recursive update" IllegalStateException, and the still-synchronous RemovalListener firing invalidateAll(...) during eviction compounds the liveness surface. The branch is reachable by any ordinary metadata-table load (e.g. table.snapshots, table.files), so this is a common path, not an edge case. Merging as-is risks regressing #3791.
uros-b
left a comment
There was a problem hiding this comment.
Please see the comment above, the approach might need to be redesigned a bit
Summary
Fix the race condition in
CachingCatalog.loadTable()as reported in #17338,其中Cache.put与Cache.invalidate/Cache.invalidateAll之间存在竞态条件,可能导致陈旧(stale)的表引用被重新放回缓存。Race Condition
这在启用 OpenLineage(后台线程调用
loadTable())与写入操作并发时尤其容易被触发。Fix
将
loadTable()中原有的三步非原子操作替换为单次tableCache.asMap().compute()原子调用:ConcurrentHashMap(Caffeine 的底层实现)保证同一 key 上的compute和remove(即invalidate)是串行化的,从而消除了竞态窗口。Test
新增回归测试
testConcurrentLoadAndInvalidateDoesNotResurrectStaleEntry:CountDownLatch精确同步loadTable和invalidateTable同时启动loadTable始终返回有效引用所有原有
TestCachingCatalog测试(13 个用例)全部通过,无回归。Files Changed
core/src/main/java/org/apache/iceberg/CachingCatalog.java— 重构 loadTable() 使用原子 computecore/src/test/java/org/apache/iceberg/hadoop/TestCachingCatalog.java— 新增并发回归测试Closes #17338.