fix: sweep stale kustomization-* tmp dirs at startup - #1726
Open
AruneshDwivedi wants to merge 1 commit into
Open
fix: sweep stale kustomization-* tmp dirs at startup#1726AruneshDwivedi wants to merge 1 commit into
AruneshDwivedi wants to merge 1 commit into
Conversation
Deferred os.RemoveAll calls in reconcile do not run on SIGKILL or os.Exit(1), leaving orphaned kustomization-* directories in /tmp. These accumulate across restarts and can consume significant ephemeral storage (or memory when /tmp is tmpfs). Add sweepStaleTmpDirs() called before mgr.Start() to clean leftover dirs from ungraceful exits. Signed-off-by: Arunesh Dwivedi <arunesh@example.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: sweep stale kustomization-* tmp dirs at startup
The kustomize-controller creates a temp dir per reconcile and registers a deferred
os.RemoveAll. Deferred functions do not run when the process exits ungracefully (SIGKILL, OOMKill, or leader lease lost causingos.Exit(1)), and nothing cleans stale dirs on startup. This leaves orphanedkustomization-*directories in/tmpthat accumulate across restarts.In production, this was observed consuming 1.5GB of orphaned artifacts sitting in a pod's /tmp five days after a crash. When
/tmpis mounted asemptyDir: {medium: Memory}, the leak becomes an OOM loop: the pages stay charged to the pod cgroup, so each OOMKill leaves less headroom for the next start.This PR adds a
sweepStaleTmpDirs()function that scans/tmpforkustomization-*directories and removes them before the manager starts reconciling. It runs on the main goroutine beforemgr.Start(), so it does not block reconciliation.Fixes #1723