Repository navigation
feat(files): on_overlapping_files = "last_wins" lets a later file replace values an earlier one holds - #339
Merged
Merged
Conversation
…ile replace a replace_key value an earlier one holds
…file's values; source-level setting skips other strategies
…rlier one; docs scope the rewritten case to order = path
…specs.rs stay under 2,000 lines
Deploying sqlload with
|
| Latest commit: |
9884152
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://048a71af.sqlload.pages.dev |
| Branch Preview URL: | https://files-last-wins.sqlload.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some feeds restate the end of the previous delivery on purpose:
sales_2026-11-14.csvrepeats Nov 5–7 fromsales_2026-11-07.csv. Withdelete_insertbyreplace_key, two files in the folder holding one value used to fail the load, which is right when the files are parts of one export but wrong for such a feed.A files source now takes
on_overlapping_files = "fail" | "last_wins"(default"fail", the behaviour so far). With"last_wins", the file last inorderreplaces the values it holds, even while an earlier file holding them is still in the folder:order = "path", a file rewritten while the later one isn't.plan -vsays so on the table'sbatchline.The setting applies only to a files source's
delete_insertwith areplace_key.planrefuses it onmerge, where the last file already wins each key, onappendand ondelete_insertby cursor. When it is set on the source, tables of that source that set another strategy of their own ignore it, as they ignore a source-levelreplace_key.The files source page has a new section, "When newer files restate older ones", with an example.
Verification
order = "path", a value split across part files with"fail", an older file sent late, both files rewritten, a source-level setting on a merge table, and the three refusals. The late-file and rewritten cases each failed before their fix.make verifyandmake test-pythonpass on the final head.