Skip to content

feat(files): on_overlapping_files = "last_wins" lets a later file replace values an earlier one holds - #339

Merged
kvlonge merged 6 commits into
mainfrom
files-last-wins
Nov 18, 2026
Merged

kvlonge merged 6 commits into
mainfrom
files-last-wins

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Some feeds restate the end of the previous delivery on purpose: sales_2026-11-14.csv repeats Nov 5–7 from sales_2026-11-07.csv. With delete_insert by replace_key, two files in the folder holding one value used to fail the load, which is right when the files are parts of one export but wrong for such a feed.

A files source now takes on_overlapping_files = "fail" | "last_wins" (default "fail", the behaviour so far). With "last_wins", the file last in order replaces the values it holds, even while an earlier file holding them is still in the folder:

  • Each file still loads in its own commit, so the earlier file's other values stay.
  • A file that comes before one already loaded fails instead of taking back the later file's values, and the error names both files and how to sort the correction after the later one. Examples are an older file sent late, or, with order = "path", a file rewritten while the later one isn't.
  • When the later file is read again in the same run, for example because both files were rewritten, the earlier one doesn't fail: the later one loads after it and replaces the values again.
  • plan -v says so on the table's batch line.

The setting applies only to a files source's delete_insert with a replace_key. plan refuses it on merge, where the last file already wins each key, on append and on delete_insert by cursor. When it is set on the source, tables of that source that set another strategy of their own ignore it, as they ignore a source-level replace_key.

The files source page has a new section, "When newer files restate older ones", with an example.

Verification

  • A loader test runs DuckDB and Postgres targets for each case: one run, two runs, order = "path", a value split across part files with "fail", an older file sent late, both files rewritten, a source-level setting on a merge table, and the three refusals. The late-file and rewritten cases each failed before their fix.
  • fensu check is clean.
  • make verify and make test-python pass on the final head.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Deploying sqlload with  Cloudflare Pages  Cloudflare Pages

Latest commit: 9884152
Status: ✅  Deploy successful!
Preview URL: https://048a71af.sqlload.pages.dev
Branch Preview URL: https://files-last-wins.sqlload.pages.dev

View logs

@kvlonge
kvlonge merged commit cc0f325 into main Nov 18, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant