Repository navigation
fix(attestor,releaser): Recover from nonce gaps, stuck transactions and refused log ranges - #589
Merged
Merged
Conversation
An own approval stays at its nonce until it is mined. When the account's mined nonce stays below it and the node holds nothing in between (a mined approval whose block was reorged away, or a skipped foreign transaction that later left the pool), nothing will ever fill the empty nonce, and every later approval queues behind it until the process restarts. The follow-up now detects that case and moves the approval down to the empty nonce, or, when the migration no longer needs it, leaves the nonce to the next approval. The attempt left behind merely reverts if it is ever mined.
A checkpoint records the attestor address and its vault attestor generation together. One without either still loads as written before the fields existed and is bound to the current key on startup, but one with only one of them was edited, and binding it would skip the rewind the identity check asks for. Refuse such a checkpoint with a message pointing at the runbook.
The "own approval not mined" page went out once per transaction. While an attestor waits on such a transaction it processes nothing else, so no other alert follows, and a page whose delivery failed left it silent. Page again 10 minutes after the first page, then at doubling intervals up to every 4 hours, for as long as the transaction stays unmined. Repeat pages say "still not mined (page N)".
A page whose webhook delivery failed (relay down, rate limited, timed out) was logged and dropped. Try a failed delivery again after 2 s and after 10 s more. A 4xx answer other than 408 and 429 is not retried: the relay refused the request itself, which repeating cannot change.
PENDULUM_GENESIS_HASH lets rehearsals run against a local test chain. Nothing tied the override to a test Base chain, so an environment carried over from a rehearsal could pin a test chain's genesis while pointing at the mainnet vault. Refuse any genesis other than Pendulum's when BASE_CHAIN_ID is 8453.
… running Membership and attestor generation were read once, at startup. A key removed and re-added while the daemon ran kept making progress although its earlier approvals no longer counted, and only the next restart refused the checkpoint. Read both again on every finalized head (two calls) and exit when either differs from startup; the next start then asks for the checkpoint to be rewound. The startup check uses the same reads.
The ReleasePending scan split its blocks into ranges of MAX_BLOCK_RANGE (default 10,000) and failed the whole cycle when the endpoint refused that size. Base Sepolia's public endpoint caps eth_getLogs at 200 blocks, so every cycle failed and paged until the setting was lowered by hand. A range the endpoint refuses for its size is now read again at half the size, down to a single block. After ten full-size reads in a row succeed the size doubles again, never above MAX_BLOCK_RANGE, which stays the maximum. Other failures propagate as before. MAX_BLOCK_RANGE must now be a whole number of at least 1, and each cycle's log line shows the current size.
Cover the rule that only a consumption read at the finality boundary drops an entry: a release consumed only at the latest state, or a NonceAlreadyConsumed from a latest-state simulation, keeps the entry; a boundary read drops it. Also cover the interval throttle of blocked pages and the vault binding of the state file.
An unexpected release failure paged on every cycle for every affected entry, so one persistent condition produced a critical page per entry per minute on the webhook the attestors share. Throttle it per release like a blocked one: the same page about a release goes out again only after BLOCKED_ALERT_INTERVAL_MS, and a different page about it at once.
A release transaction not mined within viem's 180 s receipt wait paged "unexpected release failure", and the next cycle signed another release at the next nonce, queued behind the stuck one, until fees fell again. Sign each release at the pending nonce with the capped fee estimate and wait FEE_BUMP_AFTER_SECONDS for it. One not mined by then stays in flight: no other release is sent behind it, and from cycle to cycle it is re-sent as the identical call at the same nonce with both fees raised, at most FEE_BUMP_MAX times and never above MAX_FEE_PER_GAS_GWEI, then it pages "release transaction not mined", throttled like the other release pages. Once its nonce is used by another transaction, or nothing below it can be mined, the release is simulated and sent anew. The settings and defaults match the attestor's.
POLL_INTERVAL_MS=0 turned the cycle loop into a busy loop against the RPC, and READ_BATCH_SIZE=0 failed every cycle that had anything pending. Require a poll interval of at least one second and a whole read batch size of at least 1 at startup.
… empty The follow-up took a pending count at or below the mined count as proof that the nonce in between was empty, and started the approval over there: its bumps, pages and earlier hashes were reset, and a refused send moved it straight back up. When the backend answering the count did not hold a transaction stuck at that nonce, the approval went down and back up every bump interval and was never bumped or paged. A latest count that lagged also dropped the earlier attempts from the follow-up. Re-send the latest attempt at the empty nonce instead, keeping the bumps, pages and hashes. The approval moves there only once the node takes it; a refusal leaves it at its nonce, bumped and paged as before. Receipts are polled newest first, so once mined, the attempt that moved down is reported rather than one it left behind.
A checkpoint with neither attestorAddress nor attestorGeneration loaded as one from before the fields existed and was bound to whichever key started on it, so that key resumed from progress nothing tied to its approvals. Refuse it like a checkpoint with only one of the fields. The error says to restore removed fields, or else to backfill as a new key does (runbook RB-6).
Once its bumps were used up, or the fee cap stopped the next one, a release transaction in flight was never sent again. One the pool then dropped, leaving the account's latest and pending counts both at its nonce, was neither settled nor sent again, so every deferred release waited behind it until a restart. When it is not bumped, send the latest attempt again unchanged each cycle, as the attestor does, so a pool that dropped it takes it back. The answers "already known" and "nonce too low" are expected and not logged; any other failure is.
…bject The page throttle kept one entry per release, compared by subject only. An unexpected release failure whose reason changed (another revert, or an RPC error instead of a revert) stayed muted for BLOCKED_ALERT_INTERVAL_MS, and a transient RPC error that paged first muted the real failure that followed it for as long. Key each page by its subject and a normalized reason, each with its own interval: for an unexpected failure the revert (its reason, error name or selector), or else the names of the errors it was raised through, never the message, which names requests and endpoints. A blocked page is keyed by its error name, and the not-mined page by why bumping stopped.
A page about a release was recorded as sent before its webhook delivery, so one whose delivery failed (relay down, rate limited, or timed out after the retries) muted the same page for BLOCKED_ALERT_INTERVAL_MS. Record whether each page was delivered. One that was not goes out again after 10 minutes, or after BLOCKED_ALERT_INTERVAL_MS when that is shorter, rather than on the next cycle: each cycle waits on a page's delivery attempts, up to about 42 s, which would hold back the releases that can go while the webhook is down. Without a webhook, the logged alert counts as delivered.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Attestor
PENDULUM_GENESIS_HASHother than Pendulum's is refused whenBASE_CHAIN_IDis 8453.Attestor and releaser
Releaser
ReleasePendinglog range the endpoint refuses as too large is read again at half the size, down to one block. It doubles again after ten full-size reads, never aboveMAX_BLOCK_RANGE.FEE_BUMP_AFTER_SECONDSstays in flight. It is re-sent at the same nonce with both fees raised byFEE_BUMP_PERCENT, at mostFEE_BUMP_MAXtimes and never aboveMAX_FEE_PER_GAS_GWEI, and then pages "release transaction not mined". From then on the latest attempt is sent again unchanged every cycle, so a transaction the pool dropped comes back. The settings and defaults match the attestor's and are documented inreleaser/README.md.BLOCKED_ALERT_INTERVAL_MS, so a different reason pages at once. Unexpected release failures are now throttled too, like blocked releases. A page whose delivery failed goes out again after 10 minutes.POLL_INTERVAL_MSbelow one second and aREAD_BATCH_SIZEbelow 1.Testing
npm testinattestor/(96 pass) andreleaser/(36 pass).eth_getLogslimit, the releaser shrank its range from 10,000 by itself and completed every cycle.