Repository navigation
[To dev/1.3] Prevent deleted values from reappearing after a local LOAD crash - #19188
Merged
Merged
Conversation
…che#19136) * Prevent partial local LOAD installation after process crashes * Keep committed LOAD successful when source cleanup fails * Avoid counting converted LOAD mods twice * Record LOAD mods metrics only after publication * Align LOAD cleanup tests with source group retention (cherry picked from commit c6e229a)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport #19136 (
c6e229aef50608cb67bfb4fd94717d05f4ef3d0f) todev/1.3, based oncb70919de5e82f184ae60df49172889f7966c2c2.The 1.3 local whole-file LOAD path also exposes the final
.tsfilename before its resource and deletions are installed. A DataNode process crash in that window can let recovery make deleted values visible again.Stage the data under
.tsfile.load-pending, install the resource and.mods, and publish with a same-directory atomic rename. Keep the complete source group until publication, clean unpublished installations on failure and restart, and record mods metrics only after publication. A committed LOAD remains successful when source cleanup fails; retained source data keeps its sidecars.Compatibility adaptations use 1.3's synchronous
.modsandgetModFileAPIs, copy-based staging, and TsFile 1.1 tablet/device/schema APIs. The source's mods2 conversion changes have no counterpart in 1.3's single-format deletion storage. The unit and process-crash tests cover the equivalent supported scenarios, including both source deletion policies. Required messages are provided in both locales.Validation:
with-integration-tests,ClusterIT, andwith-zh-locale.git diff --checkpassed.