fix(hotblocks-retain): act on SIGTERM instead of waiting for the kill#104
Merged
Conversation
The process runs as PID 1 in its container, and Linux gives PID 1 no default signal dispositions -- a signal with no handler installed is ignored outright rather than being fatal. So SIGTERM did nothing here, and the loop sleeps up to an hour between passes, which left SIGKILL at the termination grace as the only way this ever stopped. Measured on testnet-dev by polling a pod through its termination: this container exits 137 while its neighbours exit 0 and 1. It cost 5s per rotation when the grace was 5s and nobody noticed; now that hotblocks needs a real grace it holds every pod for the full 60s, long after the drain is done at ~25s. Exiting straight away is safe: retention is applied over HTTP and nothing is held locally, so a pass cut mid-flight is simply recomputed from the scheduler status on the next start. SIGINT is left on the default handler, matching the hotblocks binary -- a dev Ctrl-C is not PID 1 and already works.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sqd-hotblocks-retainhas no signal handling, and it runs as PID 1 in its container. Linux gives PID 1 no default signal dispositions: a signal with no handler installed is ignored, not fatal. SIGTERM therefore did nothing, and since the loop sleeps up to an hour between passes, SIGKILL at the end of the shutdown grace was the only thing that ever stopped this process.That shows up as the container outliving its neighbours by the whole grace window and dying by signal, while the rest of the pod exits cleanly — pure dead time on every restart, and it grows with whatever grace the deployment needs.
The fix
Race the loop against a SIGTERM handler. Exiting immediately is safe here: retention is applied over HTTP and nothing is held locally, so a pass cut mid-flight is recomputed from the scheduler status on the next start.
SIGINT stays on the default handler, matching the hotblocks binary — a dev Ctrl-C is not PID 1, so it already works.
Verification
Ran the built binary against dead upstreams and signalled it: exits 0 having logged
SIGTERM received, exiting, where an uninstalled handler would give 143 and no log line. The signal is caught mid-sleep, so the select races the timer correctly.🤖 Generated with Claude Code