Skip to content

fix(dora): stream the diff-based deployment fetch in calculateChangeLeadTime - #9281

Merged
klesh merged 1 commit into
apache:mainfrom
Avantio:fix/dora-stream-deployment-fetch-main
Oct 30, 2026
Merged

klesh merged 1 commit into
apache:mainfrom
Avantio:fix/dora-stream-deployment-fetch-main

Conversation

@mdelacruz-avantio

Copy link
Copy Markdown
Contributor

Summary

batchFetchDeployments loaded the whole cicd_deployment_commits x commits_diffs join into a slice of full CicdDeploymentCommit rows before keeping the first row per merge commit. The join has one row per commit between consecutive production deployments for every merge commit in that range, so its size has no relation to the result. On a project with 14k merge commits it returned 936k rows and the lake container was OOMKilled on every nightly run.

Item Change
Phase 2 query Read through db.Cursor and db.Fetch one row at a time into a struct with id, finished_date and merge_sha, keeping the first row per merge commit. Joins, filters and ORDER BY cd.commit_sha, dc.started_date ASC, dc.id ASC are unchanged, so the first row seen is still the earliest deployment
Phase 1 query Selects dc.id, dc.finished_date, dc.commit_sha instead of dc.*, the fields the conversion reads
Error handling cursor.Err() is checked after the loop, so a failed Next returns an error instead of a silently truncated map
Map contract The function comment states that map values carry only Id and FinishedDate
e2e New pull request pr8 whose merge commit is in the diff range of two production deployments, where the deployment with the lower id started later. The expected deployment is the earlier-started one, which an order-by-id or last-wins implementation gets wrong

Memory of the second phase on the instance above goes from about 1.6 GiB to a few MiB. The database work is the same as before.

Does this close any open issues?

Closes #9280

Screenshots

No UI change.

Other Information

  • Results are identical: same query, same ordering, same first-wins rule, phase 1 still takes precedence over phase 2. The dora e2e passes on MySQL 8.4 and PostgreSQL 18.4 with the existing fixtures and the new case.
  • No schema change, no migration, no option change.

…eadTime

Signed-off-by: mdelacruz-avantio <mdelacruz@avantio.com>

@klesh klesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job.
Thank you for your contribution.

@klesh
klesh merged commit 9309e7e into apache:main Oct 30, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][dora] calculateChangeLeadTime loads the whole commits_diffs deployment join into memory

2 participants