[Caideyipi]

Description

With realtime file extraction and region-level downgrading enabled, a failed in-flight TsFile transfer can leave a completion barrier ahead of later periodic heartbeats. The source stops scanning at the barrier, preventing those heartbeats from reaching the sink to trigger retries when the receiver becomes available again.

Continue scanning past completion barriers when supplying ordinary heartbeats, while keeping barriers queued and preserving file ordering. Strengthen the existing regression to cover coalesced and subsequent heartbeats while a file is in flight, verify that heartbeats do not advance progress or complete the barrier, and check ordered completion after both files commit.

Validation

• Focused PipeRealtimeExtractTest#testTsFileSourceRegionLevelDowngradingSuppliesHeartbeatWhileTsFileIsInFlight: 1 test passed; all 28 reactor modules succeeded.
• Spotless and git diff --check passed.
• Receiver restart cluster integration testing and the original fix's pre-fix/post-fix comparison were not completed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 48.91%. Comparing base (6620933) to head (aeb1685).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19219      +/-   ##
============================================
+ Coverage     48.87%   48.91%   +0.03%     
  Complexity      689      689              
============================================
  Files          5588     5588              
  Lines        413425   413446      +21     
  Branches      54525    54538      +13     
============================================
+ Hits         202074   202221     +147     
+ Misses       211351   211225     -126     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Caideyipi]

Follow-up review summary for aeb1685cd9a726da486a640ee440209e7aa13d42:

• Finding: An ordinary retry heartbeat could be coalesced when a completion barrier was the last queued item, both in the realtime source pending queue and in the default realtime-first sink queue. This could leave a failed in-flight transfer without a heartbeat reaching the sink to trigger another attempt.
• Fix: Retain the first ordinary heartbeat after a completion barrier in each queue, then continue coalescing subsequent ordinary heartbeats. Added sink-queue coverage and strengthened the source regression assertions.
• Validation: The 28-module reactor succeeded; PipeRealtimeExtractTest (20 tests) and PipeEventCollectorTest (3 tests) passed, 23 total. Spotless and git diff --check passed.
• Not run: Receiver DataNode restart cluster integration testing.

[sonarqubecloud]

Quality Gate failed

Failed conditions
B Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

[Caideyipi]

Independent re-review at PR head aeb1685cd9a726da486a640ee440209e7aa13d42 against current origin/master 6e7448d9bebd644259b695dfc8774d3d9f91252e found no actionable behavior or test-logic issues; no code changes were needed. The source pending queue and default realtime-first sink queue retain one ordinary heartbeat after a completion barrier and coalesce subsequent ordinary heartbeats, while the downgraded TsFile source leaves barriers in place and preserves TsFile order; the regression assertions confirm heartbeat commits do not advance progress or complete the barrier before TsFile commits. Verification: the prior remote 28-module reactor succeeded and 23 focused tests passed (20 PipeRealtimeExtractTest, 3 PipeEventCollectorTest); Spotless and git diff --check passed. The sink-queue regression was not run locally, and the receiver restart cluster IT was not run. The first remote attempt failed during Maven Central dependency resolution; the later focused test run succeeded. Reviewed, final local, and confirmed remote head: aeb1685cd9a726da486a640ee440209e7aa13d42.